libs/test/doc/html/boost_test/testing_tools/reports.html

   1 <html>
   2 <head>
   3 <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
   4 <title>Reported information</title>
   5 <link rel="stylesheet" href="../../boostbook.css" type="text/css">
   6 <meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
   7 <link rel="home" href="../../index.html" title="Boost.Test">
   8 <link rel="up" href="../testing_tools.html" title="Writing unit tests">
   9 <link rel="prev" href="boost_test_universal_macro.html" title="BOOST_TEST: universal and general purpose assertions">
  10 <link rel="next" href="extended_comparison.html" title="Extended comparisons support">
  11 </head>
  12 <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
  13 <table cellpadding="2" width="100%"><tr>
  14 <td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
  15 <td align="center"><a href="../../../../../../index.html">Home</a></td>
  16 <td align="center"><a href="../../../../../../libs/libraries.htm">Libraries</a></td>
  17 <td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
  18 <td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
  19 <td align="center"><a href="../../../../../../more/index.htm">More</a></td>
  20 </tr></table>
  21 <hr>
  22 <div class="spirit-nav">
  23 <a accesskey="p" href="boost_test_universal_macro.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../testing_tools.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="extended_comparison.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
  24 </div>
  25 <div class="section">
  26 <div class="titlepage"><div><div><h3 class="title">
  27 <a name="boost_test.testing_tools.reports"></a><a class="link" href="reports.html" title="Reported information">Reported information</a>
  28 </h3></div></div></div>
  29 <h4>
  30 <a name="boost_test.testing_tools.reports.h0"></a>
  31         <span class="phrase"><a name="boost_test.testing_tools.reports.failure_message_why"></a></span><a class="link" href="reports.html#boost_test.testing_tools.reports.failure_message_why">Failure message,
  32         why?</a>
  33       </h4>
  34 <p>
  35         When an assertion fails, a message is logged containing:
  36       </p>
  37 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
  38 <li class="listitem">
  39             the body of the statement that failed
  40           </li>
  41 <li class="listitem">
  42             the name of the file and the line of the failed assertion
  43           </li>
  44 <li class="listitem">
  45             the name of the test case containing this assertion
  46           </li>
  47 </ul></div>
  48 <p>
  49         The purpose of all these information is to isolate as quickly as possible
  50         the test that failed from the others. The <span class="bold"><strong>feedback</strong></span>
  51         that the execution of the test case provides is an important cue, for the
  52         following reasons:
  53       </p>
  54 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
  55 <li class="listitem">
  56             within the scheme of a continuous build/test, the logs available from
  57             the server contain this information, which points to a particular statement
  58             in the code
  59           </li>
  60 <li class="listitem">
  61             the <span class="bold"><strong>cost</strong></span> for reproducing an error is
  62             induced by the following steps:
  63             <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; ">
  64 <li class="listitem">
  65                   identify the test module that failed in case there are many
  66                 </li>
  67 <li class="listitem">
  68                   compile and run the test module to reproduce the error
  69                 </li>
  70 <li class="listitem">
  71                   identify the line of the code that failed,
  72                 </li>
  73 <li class="listitem">
  74                   fix the test directly if all the information is enough, or start
  75                   a debug session
  76                 </li>
  77 </ul></div>
  78           </li>
  79 </ul></div>
  80 <p>
  81         We can see from the scheme above that reproduction of an error is <span class="emphasis"><em>costly</em></span>,
  82         since usually one tends to reproduce the error, which in turn induces at
  83         least the compilation of the test module. Also, a hidden cost is the lookup
  84         at the line of code that contains the failing statement, which triggers a
  85         sequence of back and forth lookup between the log on one hand and the code
  86         on the other hand.
  87       </p>
  88 <p>
  89         The information extracted from the logs suggests the following fact:
  90       </p>
  91 <div class="tip"><table border="0" summary="Tip">
  92 <tr>
  93 <td rowspan="2" align="center" valign="top" width="25"><img alt="[Tip]" src="../../../../../../doc/src/images/tip.png"></td>
  94 <th align="left">Tip</th>
  95 </tr>
  96 <tr><td align="left" valign="top"><p>
  97           Richness of the information contained in the logs is a key for the rapid
  98           understanding and the resolution of a failed statement
  99         </p></td></tr>
 100 </table></div>
 101 <h4>
 102 <a name="boost_test.testing_tools.reports.h1"></a>
 103         <span class="phrase"><a name="boost_test.testing_tools.reports.default_reporting"></a></span><a class="link" href="reports.html#boost_test.testing_tools.reports.default_reporting">Default
 104         reporting</a>
 105       </h4>
 106 <p>
 107         When an assertion fails, <a class="link" href="../utf_reference/testing_tool_ref/assertion_boost_test_universal_macro.html" title="BOOST_TEST"><code class="computeroutput"><span class="identifier">BOOST_TEST</span></code></a> reports details and values
 108         on the operands of <code class="computeroutput"><span class="identifier">statement</span></code>
 109         that lead to the failure.
 110       </p>
 111 <h6>
 112 <a name="boost_test.testing_tools.reports.h2"></a>
 113         <span class="phrase"><a name="boost_test.testing_tools.reports.example_descr"></a></span><a class="link" href="reports.html#boost_test.testing_tools.reports.example_descr">Example:
 114         BOOST_TEST reporting</a>
 115       </h6>
 116 <div class="informaltable"><table class="table">
 117 <colgroup><col></colgroup>
 118 <thead><tr><th>
 119                 <p>
 120                   Code
 121                 </p>
 122               </th></tr></thead>
 123 <tbody><tr><td>
 124 <pre xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" class="table-programlisting"><span class="preprocessor">#define</span> <span class="identifier">BOOST_TEST_MODULE</span> <span class="identifier">boost_test_macro3</span>
 125 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">test</span><span class="special">/</span><span class="identifier">included</span><span class="special">/</span><span class="identifier">unit_test</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
 126
 127 <span class="identifier">BOOST_AUTO_TEST_CASE</span><span class="special">(</span> <span class="identifier">test_op_reportings</span> <span class="special">)</span>
 128 <span class="special">{</span>
 129   <span class="keyword">int</span> <span class="identifier">a</span> <span class="special">=</span> <span class="number">13</span><span class="special">,</span> <span class="identifier">b</span> <span class="special">=</span> <span class="number">12</span><span class="special">;</span>
 130   <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="identifier">a</span> <span class="special">==</span> <span class="identifier">b</span><span class="special">);</span>
 131   <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="identifier">a</span> <span class="special">&lt;</span> <span class="identifier">b</span><span class="special">);</span>
 132   <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="identifier">a</span> <span class="special">-</span> <span class="number">1</span> <span class="special">&lt;</span> <span class="identifier">b</span><span class="special">);</span>
 133   <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="identifier">b</span> <span class="special">&gt;</span> <span class="identifier">a</span> <span class="special">-</span> <span class="number">1</span><span class="special">);</span>
 134 <span class="special">}</span>
 135 </pre>
 136               </td></tr></tbody>
 137 </table></div>
 138 <div class="informaltable"><table class="table">
 139 <colgroup><col></colgroup>
 140 <thead><tr><th>
 141                 <p>
 142                   Output
 143                 </p>
 144               </th></tr></thead>
 145 <tbody><tr><td>
 146 <pre xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" class="table-programlisting"><span class="special">&gt;</span> <span class="special">./</span><span class="identifier">boost_test_macro3</span> <span class="special">--</span><span class="identifier">log_level</span><span class="special">=</span><span class="identifier">all</span>
 147 <span class="identifier">Running</span> <span class="number">1</span> <span class="identifier">test</span> <span class="keyword">case</span><span class="special">...</span>
 148 <span class="identifier">Entering</span> <span class="identifier">test</span> <span class="identifier">module</span> <span class="string">"boost_test_macro3"</span>
 149 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">12</span><span class="special">):</span> <span class="identifier">Entering</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test_op_reportings"</span>
 150 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">15</span><span class="special">):</span> <span class="identifier">error</span><span class="special">:</span> <span class="identifier">in</span> <span class="string">"test_op_reportings"</span><span class="special">:</span> <span class="identifier">check</span> <span class="identifier">a</span> <span class="special">==</span> <span class="identifier">b</span> <span class="identifier">has</span> <span class="identifier">failed</span> <span class="special">[</span><span class="number">13</span> <span class="special">!=</span> <span class="number">12</span><span class="special">]</span>
 151 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">16</span><span class="special">):</span> <span class="identifier">error</span><span class="special">:</span> <span class="identifier">in</span> <span class="string">"test_op_reportings"</span><span class="special">:</span> <span class="identifier">check</span> <span class="identifier">a</span> <span class="special">&lt;</span> <span class="identifier">b</span> <span class="identifier">has</span> <span class="identifier">failed</span> <span class="special">[</span><span class="number">13</span> <span class="special">&gt;=</span> <span class="number">12</span><span class="special">]</span>
 152 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">17</span><span class="special">):</span> <span class="identifier">error</span><span class="special">:</span> <span class="identifier">in</span> <span class="string">"test_op_reportings"</span><span class="special">:</span> <span class="identifier">check</span> <span class="identifier">a</span> <span class="special">-</span> <span class="number">1</span> <span class="special">&lt;</span> <span class="identifier">b</span> <span class="identifier">has</span> <span class="identifier">failed</span> <span class="special">[</span><span class="number">13</span> <span class="special">-</span> <span class="number">1</span> <span class="special">&gt;=</span> <span class="number">12</span><span class="special">]</span>
 153 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">18</span><span class="special">):</span> <span class="identifier">error</span><span class="special">:</span> <span class="identifier">in</span> <span class="string">"test_op_reportings"</span><span class="special">:</span> <span class="identifier">check</span> <span class="identifier">b</span> <span class="special">&gt;</span> <span class="identifier">a</span> <span class="special">-</span> <span class="number">1</span> <span class="identifier">has</span> <span class="identifier">failed</span> <span class="special">[</span><span class="number">12</span> <span class="special">&lt;=</span> <span class="number">12</span><span class="special">]</span>
 154 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">12</span><span class="special">):</span> <span class="identifier">Leaving</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test_op_reportings"</span><span class="special">;</span> <span class="identifier">testing</span> <span class="identifier">time</span><span class="special">:</span> <span class="number">484u</span><span class="identifier">s</span>
 155 <span class="identifier">Leaving</span> <span class="identifier">test</span> <span class="identifier">module</span> <span class="string">"boost_test_macro3"</span><span class="special">;</span> <span class="identifier">testing</span> <span class="identifier">time</span><span class="special">:</span> <span class="number">588u</span><span class="identifier">s</span>
 156
 157 <span class="special">***</span> <span class="number">2</span> <span class="identifier">failures</span> <span class="identifier">are</span> <span class="identifier">detected</span> <span class="identifier">in</span> <span class="identifier">the</span> <span class="identifier">test</span> <span class="identifier">module</span> <span class="string">"boost_test_macro3"</span>
 158 </pre>
 159               </td></tr></tbody>
 160 </table></div>
 161 <p>
 162         In the above example, the values of the operands are reported for inspection,
 163         which is more valuable as a copy of the full statement. However, we can observe
 164         that they are not treated symmetrically:
 165       </p>
 166 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
 167 <li class="listitem">
 168             "<code class="computeroutput"><span class="identifier">a</span> <span class="special">-</span>
 169             <span class="number">1</span> <span class="special">&lt;</span>
 170             <span class="identifier">b</span></code>" reports <code class="computeroutput"><span class="string">"13 - 1 &gt;= 12"</span> <span class="identifier">failed</span></code>
 171           </li>
 172 <li class="listitem">
 173             "<code class="computeroutput"><span class="identifier">b</span> <span class="special">&gt;</span>
 174             <span class="identifier">a</span> <span class="special">-</span>
 175             <span class="number">1</span></code>" reports <code class="computeroutput"><span class="string">"12
 176             &lt;= 12"</span> <span class="identifier">failed</span></code>
 177           </li>
 178 </ul></div>
 179 <p>
 180         More details on how the <span class="emphasis"><em>Unit Test Framework</em></span> parses the
 181         statement are given in <a class="link" href="internal_details.html" title="BOOST_TEST: details on expressions">this</a>
 182         section.
 183       </p>
 184 <h4>
 185 <a name="boost_test.testing_tools.reports.h3"></a>
 186         <span class="phrase"><a name="boost_test.testing_tools.reports.custom_messages"></a></span><a class="link" href="reports.html#boost_test.testing_tools.reports.custom_messages">Custom
 187         messages</a>
 188       </h4>
 189 <p>
 190         While perfectly exact and precise, the file name, test case name, line number
 191         of a failed statement carry an information that is partial with regards to
 192         the meaning of the failed statement. Sometimes these information are not
 193         informative enough. The <code class="computeroutput"><span class="identifier">BOOST_TEST</span></code>
 194         macro let you override the default message by the use of a second argument,
 195         as shown on the following example.
 196       </p>
 197 <h6>
 198 <a name="boost_test.testing_tools.reports.h4"></a>
 199         <span class="phrase"><a name="boost_test.testing_tools.reports.example_descr0"></a></span><a class="link" href="reports.html#boost_test.testing_tools.reports.example_descr0">Example:
 200         BOOST_TEST optional failure message</a>
 201       </h6>
 202 <div class="informaltable"><table class="table">
 203 <colgroup><col></colgroup>
 204 <thead><tr><th>
 205                 <p>
 206                   Code
 207                 </p>
 208               </th></tr></thead>
 209 <tbody><tr><td>
 210 <pre xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" class="table-programlisting"><span class="preprocessor">#define</span> <span class="identifier">BOOST_TEST_MODULE</span> <span class="identifier">boost_test_message</span>
 211 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">test</span><span class="special">/</span><span class="identifier">included</span><span class="special">/</span><span class="identifier">unit_test</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
 212
 213 <span class="identifier">BOOST_AUTO_TEST_CASE</span><span class="special">(</span> <span class="identifier">test_message</span> <span class="special">)</span>
 214 <span class="special">{</span>
 215   <span class="keyword">const</span> <span class="keyword">int</span> <span class="identifier">a</span><span class="special">(</span><span class="number">1</span><span class="special">),</span> <span class="identifier">b</span><span class="special">(</span><span class="number">2</span><span class="special">);</span>
 216   <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="identifier">a</span> <span class="special">==</span> <span class="identifier">b</span><span class="special">,</span> <span class="string">"a should be equal to b: "</span> <span class="special">&lt;&lt;</span> <span class="identifier">a</span> <span class="special">&lt;&lt;</span> <span class="string">"!="</span> <span class="special">&lt;&lt;</span> <span class="identifier">b</span><span class="special">);</span>
 217   <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="identifier">a</span> <span class="special">!=</span> <span class="number">10</span><span class="special">,</span> <span class="string">"value of a="</span> <span class="special">&lt;&lt;</span> <span class="identifier">a</span><span class="special">);</span>
 218 <span class="special">}</span>
 219 </pre>
 220               </td></tr></tbody>
 221 </table></div>
 222 <div class="informaltable"><table class="table">
 223 <colgroup><col></colgroup>
 224 <thead><tr><th>
 225                 <p>
 226                   Output
 227                 </p>
 228               </th></tr></thead>
 229 <tbody><tr><td>
 230 <pre xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" class="table-programlisting"><span class="special">&gt;</span> <span class="special">./</span><span class="identifier">boost_test_message</span> <span class="special">--</span><span class="identifier">log_level</span><span class="special">=</span><span class="identifier">all</span>
 231 <span class="identifier">Running</span> <span class="number">1</span> <span class="identifier">test</span> <span class="keyword">case</span><span class="special">...</span>
 232 <span class="identifier">Entering</span> <span class="identifier">test</span> <span class="identifier">module</span> <span class="string">"boost_test_message"</span>
 233 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">12</span><span class="special">):</span> <span class="identifier">Entering</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test_message"</span>
 234 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">15</span><span class="special">):</span> <span class="identifier">error</span><span class="special">:</span> <span class="identifier">in</span> <span class="string">"test_message"</span><span class="special">:</span> <span class="identifier">a</span> <span class="identifier">should</span> <span class="identifier">be</span> <span class="identifier">equal</span> <span class="identifier">to</span> <span class="identifier">b</span><span class="special">:</span> <span class="number">1</span><span class="special">!=</span><span class="number">2</span>
 235 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">16</span><span class="special">):</span> <span class="identifier">info</span><span class="special">:</span> <span class="identifier">check</span> <span class="char">'value of a=1'</span> <span class="identifier">has</span> <span class="identifier">passed</span>
 236 <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">(</span><span class="number">12</span><span class="special">):</span> <span class="identifier">Leaving</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test_message"</span><span class="special">;</span> <span class="identifier">testing</span> <span class="identifier">time</span><span class="special">:</span> <span class="number">219u</span><span class="identifier">s</span>
 237 <span class="identifier">Leaving</span> <span class="identifier">test</span> <span class="identifier">module</span> <span class="string">"boost_test_message"</span><span class="special">;</span> <span class="identifier">testing</span> <span class="identifier">time</span><span class="special">:</span> <span class="number">318u</span><span class="identifier">s</span>
 238
 239 <span class="special">***</span> <span class="number">1</span> <span class="identifier">failure</span> <span class="identifier">is</span> <span class="identifier">detected</span> <span class="identifier">in</span> <span class="identifier">the</span> <span class="identifier">test</span> <span class="identifier">module</span> <span class="string">"boost_test_message"</span>
 240 </pre>
 241               </td></tr></tbody>
 242 </table></div>
 243 </div>
 244 <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
 245 <td align="left"></td>
 246 <td align="right"><div class="copyright-footer">Copyright &#169; 2001-2019 Boost.Test
 247       contributors<p>
 248         Distributed under the Boost Software License, Version 1.0. (See accompanying
 249         file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
 250       </p>
 251 </div></td>
 252 </tr></table>
 253 <hr>
 254 <div class="spirit-nav">
 255 <a accesskey="p" href="boost_test_universal_macro.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../testing_tools.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="extended_comparison.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
 256 </div>
 257 </body>
 258 </html>