libs/math/doc/sf_and_dist/html/math_toolkit/dist/stat_tut/weg/st_eg/tut_mean_test.html

   1 <html>
   2 <head>
   3 <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
   4 <title>Testing a sample mean for difference from a "true" mean</title>
   5 <link rel="stylesheet" href="../../../../../../../../../../doc/src/boostbook.css" type="text/css">
   6 <meta name="generator" content="DocBook XSL Stylesheets V1.76.1">
   7 <link rel="home" href="../../../../../index.html" title="Math Toolkit">
   8 <link rel="up" href="../st_eg.html" title="Student's t Distribution Examples">
   9 <link rel="prev" href="tut_mean_intervals.html" title="Calculating confidence intervals on the mean with the Students-t distribution">
  10 <link rel="next" href="tut_mean_size.html" title="Estimating how large a sample size would have to become in order to give a significant Students-t test result with a single sample test">
  11 </head>
  12 <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
  13 <table cellpadding="2" width="100%"><tr>
  14 <td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../../../../boost.png"></td>
  15 <td align="center"><a href="../../../../../../../../../../index.html">Home</a></td>
  16 <td align="center"><a href="../../../../../../../../../../libs/libraries.htm">Libraries</a></td>
  17 <td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
  18 <td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
  19 <td align="center"><a href="../../../../../../../../../../more/index.htm">More</a></td>
  20 </tr></table>
  21 <hr>
  22 <div class="spirit-nav">
  23 <a accesskey="p" href="tut_mean_intervals.html"><img src="../../../../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../st_eg.html"><img src="../../../../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../../../index.html"><img src="../../../../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="tut_mean_size.html"><img src="../../../../../../../../../../doc/src/images/next.png" alt="Next"></a>
  24 </div>
  25 <div class="section math_toolkit_dist_stat_tut_weg_st_eg_tut_mean_test">
  26 <div class="titlepage"><div><div><h6 class="title">
  27 <a name="math_toolkit.dist.stat_tut.weg.st_eg.tut_mean_test"></a><a class="link" href="tut_mean_test.html" title='Testing a sample mean for difference from a "true" mean'>Testing
  28             a sample mean for difference from a "true" mean</a>
  29 </h6></div></div></div>
  30 <p>
  31               When calibrating or comparing a scientific instrument or measurement
  32               method of some kind, we want to be answer the question "Does an
  33               observed sample mean differ from the "true" mean in any significant
  34               way?". If it does, then we have evidence of a systematic difference.
  35               This question can be answered with a Students-t test: more information
  36               can be found <a href="http://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm" target="_top">on
  37               the NIST site</a>.
  38             </p>
  39 <p>
  40               Of course, the assignment of "true" to one mean may be quite
  41               arbitrary, often this is simply a "traditional" method of
  42               measurement.
  43             </p>
  44 <p>
  45               The following example code is taken from the example program <a href="../../../../../../../../example/students_t_single_sample.cpp" target="_top">students_t_single_sample.cpp</a>.
  46             </p>
  47 <p>
  48               We'll begin by defining a procedure to determine which of the possible
  49               hypothesis are rejected or not-rejected at a given significance level:
  50             </p>
  51 <div class="note"><table border="0" summary="Note">
  52 <tr>
  53 <td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../../../../doc/src/images/note.png"></td>
  54 <th align="left">Note</th>
  55 </tr>
  56 <tr><td align="left" valign="top"><p>
  57                 Non-statisticians might say 'not-rejected' means 'accepted', (often
  58                 of the null-hypothesis) implying, wrongly, that there really <span class="bold"><strong>IS</strong></span> no difference, but statisticans eschew
  59                 this to avoid implying that there is positive evidence of 'no difference'.
  60                 'Not-rejected' here means there is <span class="bold"><strong>no evidence</strong></span>
  61                 of difference, but there still might well be a difference. For example,
  62                 see <a href="http://en.wikipedia.org/wiki/Argument_from_ignorance" target="_top">argument
  63                 from ignorance</a> and <a href="http://www.bmj.com/cgi/content/full/311/7003/485" target="_top">Absence
  64                 of evidence does not constitute evidence of absence.</a>
  65               </p></td></tr>
  66 </table></div>
  67 <pre class="programlisting"><span class="comment">// Needed includes:</span>
  68 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">math</span><span class="special">/</span><span class="identifier">distributions</span><span class="special">/</span><span class="identifier">students_t</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  69 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">iostream</span><span class="special">&gt;</span>
  70 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">iomanip</span><span class="special">&gt;</span>
  71 <span class="comment">// Bring everything into global namespace for ease of use:</span>
  72 <span class="keyword">using</span> <span class="keyword">namespace</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">math</span><span class="special">;</span>
  73 <span class="keyword">using</span> <span class="keyword">namespace</span> <span class="identifier">std</span><span class="special">;</span>
  74
  75 <span class="keyword">void</span> <span class="identifier">single_sample_t_test</span><span class="special">(</span><span class="keyword">double</span> <span class="identifier">M</span><span class="special">,</span> <span class="keyword">double</span> <span class="identifier">Sm</span><span class="special">,</span> <span class="keyword">double</span> <span class="identifier">Sd</span><span class="special">,</span> <span class="keyword">unsigned</span> <span class="identifier">Sn</span><span class="special">,</span> <span class="keyword">double</span> <span class="identifier">alpha</span><span class="special">)</span>
  76 <span class="special">{</span>
  77    <span class="comment">//</span>
  78    <span class="comment">// M = true mean.</span>
  79    <span class="comment">// Sm = Sample Mean.</span>
  80    <span class="comment">// Sd = Sample Standard Deviation.</span>
  81    <span class="comment">// Sn = Sample Size.</span>
  82    <span class="comment">// alpha = Significance Level.</span>
  83 </pre>
  84 <p>
  85               Most of the procedure is pretty-printing, so let's just focus on the
  86               calculation, we begin by calculating the t-statistic:
  87             </p>
  88 <pre class="programlisting"><span class="comment">// Difference in means:</span>
  89 <span class="keyword">double</span> <span class="identifier">diff</span> <span class="special">=</span> <span class="identifier">Sm</span> <span class="special">-</span> <span class="identifier">M</span><span class="special">;</span>
  90 <span class="comment">// Degrees of freedom:</span>
  91 <span class="keyword">unsigned</span> <span class="identifier">v</span> <span class="special">=</span> <span class="identifier">Sn</span> <span class="special">-</span> <span class="number">1</span><span class="special">;</span>
  92 <span class="comment">// t-statistic:</span>
  93 <span class="keyword">double</span> <span class="identifier">t_stat</span> <span class="special">=</span> <span class="identifier">diff</span> <span class="special">*</span> <span class="identifier">sqrt</span><span class="special">(</span><span class="keyword">double</span><span class="special">(</span><span class="identifier">Sn</span><span class="special">))</span> <span class="special">/</span> <span class="identifier">Sd</span><span class="special">;</span>
  94 </pre>
  95 <p>
  96               Finally calculate the probability from the t-statistic. If we're interested
  97               in simply whether there is a difference (either less or greater) or
  98               not, we don't care about the sign of the t-statistic, and we take the
  99               complement of the probability for comparison to the significance level:
 100             </p>
 101 <pre class="programlisting"><span class="identifier">students_t</span> <span class="identifier">dist</span><span class="special">(</span><span class="identifier">v</span><span class="special">);</span>
 102 <span class="keyword">double</span> <span class="identifier">q</span> <span class="special">=</span> <span class="identifier">cdf</span><span class="special">(</span><span class="identifier">complement</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span> <span class="identifier">fabs</span><span class="special">(</span><span class="identifier">t_stat</span><span class="special">)));</span>
 103 </pre>
 104 <p>
 105               The procedure then prints out the results of the various tests that
 106               can be done, these can be summarised in the following table:
 107             </p>
 108 <div class="informaltable"><table class="table">
 109 <colgroup>
 110 <col>
 111 <col>
 112 </colgroup>
 113 <thead><tr>
 114 <th>
 115                       <p>
 116                         Hypothesis
 117                       </p>
 118                     </th>
 119 <th>
 120                       <p>
 121                         Test
 122                       </p>
 123                     </th>
 124 </tr></thead>
 125 <tbody>
 126 <tr>
 127 <td>
 128                       <p>
 129                         The Null-hypothesis: there is <span class="bold"><strong>no difference</strong></span>
 130                         in means
 131                       </p>
 132                     </td>
 133 <td>
 134                       <p>
 135                         Reject if complement of CDF for |t| &lt; significance level
 136                         / 2:
 137                       </p>
 138                       <p>
 139                         <code class="computeroutput"><span class="identifier">cdf</span><span class="special">(</span><span class="identifier">complement</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span>
 140                         <span class="identifier">fabs</span><span class="special">(</span><span class="identifier">t</span><span class="special">)))</span>
 141                         <span class="special">&lt;</span> <span class="identifier">alpha</span>
 142                         <span class="special">/</span> <span class="number">2</span></code>
 143                       </p>
 144                     </td>
 145 </tr>
 146 <tr>
 147 <td>
 148                       <p>
 149                         The Alternative-hypothesis: there <span class="bold"><strong>is
 150                         difference</strong></span> in means
 151                       </p>
 152                     </td>
 153 <td>
 154                       <p>
 155                         Reject if complement of CDF for |t| &gt; significance level
 156                         / 2:
 157                       </p>
 158                       <p>
 159                         <code class="computeroutput"><span class="identifier">cdf</span><span class="special">(</span><span class="identifier">complement</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span>
 160                         <span class="identifier">fabs</span><span class="special">(</span><span class="identifier">t</span><span class="special">)))</span>
 161                         <span class="special">&gt;</span> <span class="identifier">alpha</span>
 162                         <span class="special">/</span> <span class="number">2</span></code>
 163                       </p>
 164                     </td>
 165 </tr>
 166 <tr>
 167 <td>
 168                       <p>
 169                         The Alternative-hypothesis: the sample mean <span class="bold"><strong>is
 170                         less</strong></span> than the true mean.
 171                       </p>
 172                     </td>
 173 <td>
 174                       <p>
 175                         Reject if CDF of t &gt; significance level:
 176                       </p>
 177                       <p>
 178                         <code class="computeroutput"><span class="identifier">cdf</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span>
 179                         <span class="identifier">t</span><span class="special">)</span>
 180                         <span class="special">&gt;</span> <span class="identifier">alpha</span></code>
 181                       </p>
 182                     </td>
 183 </tr>
 184 <tr>
 185 <td>
 186                       <p>
 187                         The Alternative-hypothesis: the sample mean <span class="bold"><strong>is
 188                         greater</strong></span> than the true mean.
 189                       </p>
 190                     </td>
 191 <td>
 192                       <p>
 193                         Reject if complement of CDF of t &gt; significance level:
 194                       </p>
 195                       <p>
 196                         <code class="computeroutput"><span class="identifier">cdf</span><span class="special">(</span><span class="identifier">complement</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span>
 197                         <span class="identifier">t</span><span class="special">))</span>
 198                         <span class="special">&gt;</span> <span class="identifier">alpha</span></code>
 199                       </p>
 200                     </td>
 201 </tr>
 202 </tbody>
 203 </table></div>
 204 <div class="note"><table border="0" summary="Note">
 205 <tr>
 206 <td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../../../../doc/src/images/note.png"></td>
 207 <th align="left">Note</th>
 208 </tr>
 209 <tr><td align="left" valign="top"><p>
 210                 Notice that the comparisons are against <code class="computeroutput"><span class="identifier">alpha</span>
 211                 <span class="special">/</span> <span class="number">2</span></code>
 212                 for a two-sided test and against <code class="computeroutput"><span class="identifier">alpha</span></code>
 213                 for a one-sided test
 214               </p></td></tr>
 215 </table></div>
 216 <p>
 217               Now that we have all the parts in place, let's take a look at some
 218               sample output, first using the <a href="http://www.itl.nist.gov/div898/handbook/eda/section4/eda428.htm" target="_top">Heat
 219               flow data</a> from the NIST site. The data set was collected by
 220               Bob Zarr of NIST in January, 1990 from a heat flow meter calibration
 221               and stability analysis. The corresponding dataplot output for this
 222               test can be found in <a href="http://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm" target="_top">section
 223               3.5.2</a> of the <a href="http://www.itl.nist.gov/div898/handbook/" target="_top">NIST/SEMATECH
 224               e-Handbook of Statistical Methods.</a>.
 225             </p>
 226 <pre class="programlisting">   __________________________________
 227    Student t test for a single sample
 228    __________________________________
 229
 230    Number of Observations                                 =  195
 231    Sample Mean                                            =  9.26146
 232    Sample Standard Deviation                              =  0.02279
 233    Expected True Mean                                     =  5.00000
 234
 235    Sample Mean - Expected Test Mean                       =  4.26146
 236    Degrees of Freedom                                     =  194
 237    T Statistic                                            =  2611.28380
 238    Probability that difference is due to chance           =  0.000e+000
 239
 240    Results for Alternative Hypothesis and alpha           =  0.0500
 241
 242    Alternative Hypothesis     Conclusion
 243    Mean != 5.000              NOT REJECTED
 244    Mean  &lt; 5.000              REJECTED
 245    Mean  &gt; 5.000              NOT REJECTED
 246 </pre>
 247 <p>
 248               You will note the line that says the probability that the difference
 249               is due to chance is zero. From a philosophical point of view, of course,
 250               the probability can never reach zero. However, in this case the calculated
 251               probability is smaller than the smallest representable double precision
 252               number, hence the appearance of a zero here. Whatever its "true"
 253               value is, we know it must be extraordinarily small, so the alternative
 254               hypothesis - that there is a difference in means - is not rejected.
 255             </p>
 256 <p>
 257               For comparison the next example data output is taken from <span class="emphasis"><em>P.K.Hou,
 258               O. W. Lau &amp; M.C. Wong, Analyst (1983) vol. 108, p 64. and from
 259               Statistics for Analytical Chemistry, 3rd ed. (1994), pp 54-55 J. C.
 260               Miller and J. N. Miller, Ellis Horwood ISBN 0 13 0309907.</em></span>
 261               The values result from the determination of mercury by cold-vapour
 262               atomic absorption.
 263             </p>
 264 <pre class="programlisting">   __________________________________
 265    Student t test for a single sample
 266    __________________________________
 267
 268    Number of Observations                                 =  3
 269    Sample Mean                                            =  37.80000
 270    Sample Standard Deviation                              =  0.96437
 271    Expected True Mean                                     =  38.90000
 272
 273    Sample Mean - Expected Test Mean                       =  -1.10000
 274    Degrees of Freedom                                     =  2
 275    T Statistic                                            =  -1.97566
 276    Probability that difference is due to chance           =  1.869e-001
 277
 278    Results for Alternative Hypothesis and alpha           =  0.0500
 279
 280    Alternative Hypothesis     Conclusion
 281    Mean != 38.900             REJECTED
 282    Mean  &lt; 38.900             REJECTED
 283    Mean  &gt; 38.900             REJECTED
 284 </pre>
 285 <p>
 286               As you can see the small number of measurements (3) has led to a large
 287               uncertainty in the location of the true mean. So even though there
 288               appears to be a difference between the sample mean and the expected
 289               true mean, we conclude that there is no significant difference, and
 290               are unable to reject the null hypothesis. However, if we were to lower
 291               the bar for acceptance down to alpha = 0.1 (a 90% confidence level)
 292               we see a different output:
 293             </p>
 294 <pre class="programlisting">__________________________________
 295 Student t test for a single sample
 296 __________________________________
 297
 298 Number of Observations                                 =  3
 299 Sample Mean                                            =  37.80000
 300 Sample Standard Deviation                              =  0.96437
 301 Expected True Mean                                     =  38.90000
 302
 303 Sample Mean - Expected Test Mean                       =  -1.10000
 304 Degrees of Freedom                                     =  2
 305 T Statistic                                            =  -1.97566
 306 Probability that difference is due to chance           =  1.869e-001
 307
 308 Results for Alternative Hypothesis and alpha           =  0.1000
 309
 310 Alternative Hypothesis     Conclusion
 311 Mean != 38.900            REJECTED
 312 Mean  &lt; 38.900            NOT REJECTED
 313 Mean  &gt; 38.900            REJECTED
 314 </pre>
 315 <p>
 316               In this case, we really have a borderline result, and more data (and/or
 317               more accurate data), is needed for a more convincing conclusion.
 318             </p>
 319 </div>
 320 <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
 321 <td align="left"></td>
 322 <td align="right"><div class="copyright-footer">Copyright &#169; 2006-2010 John Maddock, Paul A. Bristow, Hubert Holin, Xiaogang Zhang, Bruno
 323       Lalande, Johan R&#229;de, Gautam Sewani, Thijs van den Berg and Benjamin Sobotta<p>
 324         Distributed under the Boost Software License, Version 1.0. (See accompanying
 325         file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
 326       </p>
 327 </div></td>
 328 </tr></table>
 329 <hr>
 330 <div class="spirit-nav">
 331 <a accesskey="p" href="tut_mean_intervals.html"><img src="../../../../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../st_eg.html"><img src="../../../../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../../../index.html"><img src="../../../../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="tut_mean_size.html"><img src="../../../../../../../../../../doc/src/images/next.png" alt="Next"></a>
 332 </div>
 333 </body>
 334 </html>