libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart2.html

   1 <html>
   2 <head>
   3 <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
   4 <title>Quickstart 2 - A better word counter using Spirit.Lex</title>
   5 <link rel="stylesheet" href="../../../../../../../doc/src/boostbook.css" type="text/css">
   6 <meta name="generator" content="DocBook XSL Stylesheets V1.78.1">
   7 <link rel="home" href="../../../index.html" title="Spirit 2.5.2">
   8 <link rel="up" href="../tutorials.html" title="Spirit.Lex Tutorials">
   9 <link rel="prev" href="lexer_quickstart1.html" title="Quickstart 1 - A word counter using Spirit.Lex">
  10 <link rel="next" href="lexer_quickstart3.html" title="Quickstart 3 - Counting Words Using a Parser">
  11 </head>
  12 <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
  13 <table cellpadding="2" width="100%"><tr>
  14 <td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
  15 <td align="center"><a href="../../../../../../../index.html">Home</a></td>
  16 <td align="center"><a href="../../../../../../../libs/libraries.htm">Libraries</a></td>
  17 <td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
  18 <td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
  19 <td align="center"><a href="../../../../../../../more/index.htm">More</a></td>
  20 </tr></table>
  21 <hr>
  22 <div class="spirit-nav">
  23 <a accesskey="p" href="lexer_quickstart1.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_quickstart3.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
  24 </div>
  25 <div class="section">
  26 <div class="titlepage"><div><div><h4 class="title">
  27 <a name="spirit.lex.tutorials.lexer_quickstart2"></a><a class="link" href="lexer_quickstart2.html" title="Quickstart 2 - A better word counter using Spirit.Lex">Quickstart
  28         2 - A better word counter using <span class="emphasis"><em>Spirit.Lex</em></span></a>
  29 </h4></div></div></div>
  30 <p>
  31           People familiar with <a href="http://flex.sourceforge.net/" target="_top">Flex</a>
  32           will probably complain about the example from the section <a class="link" href="lexer_quickstart1.html" title="Quickstart 1 - A word counter using Spirit.Lex">Lex
  33           Quickstart 1 - A word counter using <span class="emphasis"><em>Spirit.Lex</em></span></a>
  34           as being overly complex and not being written to leverage the possibilities
  35           provided by this tool. In particular the previous example did not directly
  36           use the lexer actions to count the lines, words, and characters. So the
  37           example provided in this step of the tutorial will show how to use semantic
  38           actions in <span class="emphasis"><em>Spirit.Lex</em></span>. Even though this examples still
  39           counts textual elements, the purpose is to introduce new concepts and configuration
  40           options along the lines (for the full example code see here: <a href="../../../../../example/lex/word_count_lexer.cpp" target="_top">word_count_lexer.cpp</a>).
  41         </p>
  42 <h6>
  43 <a name="spirit.lex.tutorials.lexer_quickstart2.h0"></a>
  44           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart2.prerequisites"></a></span><a class="link" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.prerequisites">Prerequisites</a>
  45         </h6>
  46 <p>
  47           In addition to the only required <code class="computeroutput"><span class="preprocessor">#include</span></code>
  48           specific to <span class="emphasis"><em>Spirit.Lex</em></span> this example needs to include
  49           a couple of header files from the <a href="../../../../../../../libs/phoenix/doc/html/index.html" target="_top">Boost.Phoenix</a>
  50           library. This example shows how to attach functors to token definitions,
  51           which could be done using any type of C++ technique resulting in a callable
  52           object. Using <a href="../../../../../../../libs/phoenix/doc/html/index.html" target="_top">Boost.Phoenix</a>
  53           for this task simplifies things and avoids adding dependencies to other
  54           libraries (<a href="../../../../../../../libs/phoenix/doc/html/index.html" target="_top">Boost.Phoenix</a>
  55           is already in use for <a href="http://boost-spirit.com" target="_top">Spirit</a>
  56           anyway).
  57         </p>
  58 <p>
  59 </p>
  60 <pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">lex_lexertl</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  61 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">phoenix_operator</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  62 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">phoenix_statement</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  63 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">phoenix_algorithm</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  64 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">phoenix_core</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  65 </pre>
  66 <p>
  67         </p>
  68 <p>
  69           To make all the code below more readable we introduce the following namespaces.
  70         </p>
  71 <p>
  72 </p>
  73 <pre class="programlisting"><span class="keyword">namespace</span> <span class="identifier">lex</span> <span class="special">=</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">;</span>
  74 </pre>
  75 <p>
  76         </p>
  77 <p>
  78           To give a preview at what to expect from this example, here is the flex
  79           program which has been used as the starting point. The useful code is directly
  80           included inside the actions associated with each of the token definitions.
  81         </p>
  82 <p>
  83 </p>
  84 <pre class="programlisting"><span class="special">%{</span>
  85     <span class="keyword">int</span> <span class="identifier">c</span> <span class="special">=</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">w</span> <span class="special">=</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">l</span> <span class="special">=</span> <span class="number">0</span><span class="special">;</span>
  86 <span class="special">%}</span>
  87 <span class="special">%%</span>
  88 <span class="special">[^</span> <span class="special">\</span><span class="identifier">t</span><span class="special">\</span><span class="identifier">n</span><span class="special">]+</span>  <span class="special">{</span> <span class="special">++</span><span class="identifier">w</span><span class="special">;</span> <span class="identifier">c</span> <span class="special">+=</span> <span class="identifier">yyleng</span><span class="special">;</span> <span class="special">}</span>
  89 <span class="special">\</span><span class="identifier">n</span>         <span class="special">{</span> <span class="special">++</span><span class="identifier">c</span><span class="special">;</span> <span class="special">++</span><span class="identifier">l</span><span class="special">;</span> <span class="special">}</span>
  90 <span class="special">.</span>          <span class="special">{</span> <span class="special">++</span><span class="identifier">c</span><span class="special">;</span> <span class="special">}</span>
  91 <span class="special">%%</span>
  92 <span class="identifier">main</span><span class="special">()</span>
  93 <span class="special">{</span>
  94     <span class="identifier">yylex</span><span class="special">();</span>
  95     <span class="identifier">printf</span><span class="special">(</span><span class="string">"%d %d %d\n"</span><span class="special">,</span> <span class="identifier">l</span><span class="special">,</span> <span class="identifier">w</span><span class="special">,</span> <span class="identifier">c</span><span class="special">);</span>
  96 <span class="special">}</span>
  97 </pre>
  98 <p>
  99         </p>
 100 <h6>
 101 <a name="spirit.lex.tutorials.lexer_quickstart2.h1"></a>
 102           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart2.semantic_actions_in__emphasis_spirit_lex__emphasis_"></a></span><a class="link" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.semantic_actions_in__emphasis_spirit_lex__emphasis_">Semantic
 103           Actions in <span class="emphasis"><em>Spirit.Lex</em></span></a>
 104         </h6>
 105 <p>
 106           <span class="emphasis"><em>Spirit.Lex</em></span> uses a very similar way of associating
 107           actions with the token definitions (which should look familiar to anybody
 108           knowledgeable with <a href="http://boost-spirit.com" target="_top">Spirit</a>
 109           as well): specifying the operations to execute inside of a pair of <code class="computeroutput"><span class="special">[]</span></code> brackets. In order to be able to attach
 110           semantic actions to token definitions for each of them there is defined
 111           an instance of a <code class="computeroutput"><span class="identifier">token_def</span><span class="special">&lt;&gt;</span></code>.
 112         </p>
 113 <p>
 114 </p>
 115 <pre class="programlisting"><span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">&gt;</span>
 116 <span class="keyword">struct</span> <span class="identifier">word_count_tokens</span> <span class="special">:</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexer</span><span class="special">&lt;</span><span class="identifier">Lexer</span><span class="special">&gt;</span>
 117 <span class="special">{</span>
 118     <span class="identifier">word_count_tokens</span><span class="special">()</span>
 119       <span class="special">:</span> <span class="identifier">c</span><span class="special">(</span><span class="number">0</span><span class="special">),</span> <span class="identifier">w</span><span class="special">(</span><span class="number">0</span><span class="special">),</span> <span class="identifier">l</span><span class="special">(</span><span class="number">0</span><span class="special">)</span>
 120       <span class="special">,</span> <span class="identifier">word</span><span class="special">(</span><span class="string">"[^ \t\n]+"</span><span class="special">)</span>     <span class="comment">// define tokens</span>
 121       <span class="special">,</span> <span class="identifier">eol</span><span class="special">(</span><span class="string">"\n"</span><span class="special">)</span>
 122       <span class="special">,</span> <span class="identifier">any</span><span class="special">(</span><span class="string">"."</span><span class="special">)</span>
 123     <span class="special">{</span>
 124         <span class="keyword">using</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">_start</span><span class="special">;</span>
 125         <span class="keyword">using</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">_end</span><span class="special">;</span>
 126         <span class="keyword">using</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">phoenix</span><span class="special">::</span><span class="identifier">ref</span><span class="special">;</span>
 127
 128         <span class="comment">// associate tokens with the lexer</span>
 129         <span class="keyword">this</span><span class="special">-&gt;</span><span class="identifier">self</span>
 130             <span class="special">=</span>   <span class="identifier">word</span>  <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">w</span><span class="special">),</span> <span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)</span> <span class="special">+=</span> <span class="identifier">distance</span><span class="special">(</span><span class="identifier">_start</span><span class="special">,</span> <span class="identifier">_end</span><span class="special">)]</span>
 131             <span class="special">|</span>   <span class="identifier">eol</span>   <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">),</span> <span class="special">++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">l</span><span class="special">)]</span>
 132             <span class="special">|</span>   <span class="identifier">any</span>   <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)]</span>
 133             <span class="special">;</span>
 134     <span class="special">}</span>
 135
 136     <span class="identifier">std</span><span class="special">::</span><span class="identifier">size_t</span> <span class="identifier">c</span><span class="special">,</span> <span class="identifier">w</span><span class="special">,</span> <span class="identifier">l</span><span class="special">;</span>
 137     <span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special">&lt;&gt;</span> <span class="identifier">word</span><span class="special">,</span> <span class="identifier">eol</span><span class="special">,</span> <span class="identifier">any</span><span class="special">;</span>
 138 <span class="special">};</span>
 139 </pre>
 140 <p>
 141         </p>
 142 <p>
 143           The semantics of the shown code is as follows. The code inside the <code class="computeroutput"><span class="special">[]</span></code> brackets will be executed whenever the
 144           corresponding token has been matched by the lexical analyzer. This is very
 145           similar to <a href="http://flex.sourceforge.net/" target="_top">Flex</a>, where
 146           the action code associated with a token definition gets executed after
 147           the recognition of a matching input sequence. The code above uses function
 148           objects constructed using <a href="../../../../../../../libs/phoenix/doc/html/index.html" target="_top">Boost.Phoenix</a>,
 149           but it is possible to insert any C++ function or function object as long
 150           as it exposes the proper interface. For more details on please refer to
 151           the section <a class="link" href="../abstracts/lexer_semantic_actions.html" title="Lexer Semantic Actions">Lexer
 152           Semantic Actions</a>.
 153         </p>
 154 <h6>
 155 <a name="spirit.lex.tutorials.lexer_quickstart2.h2"></a>
 156           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart2.associating_token_definitions_with_the_lexer"></a></span><a class="link" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.associating_token_definitions_with_the_lexer">Associating
 157           Token Definitions with the Lexer</a>
 158         </h6>
 159 <p>
 160           If you compare this code to the code from <a class="link" href="lexer_quickstart1.html" title="Quickstart 1 - A word counter using Spirit.Lex">Lex
 161           Quickstart 1 - A word counter using <span class="emphasis"><em>Spirit.Lex</em></span></a>
 162           with regard to the way how token definitions are associated with the lexer,
 163           you will notice a different syntax being used here. In the previous example
 164           we have been using the <code class="computeroutput"><span class="identifier">self</span><span class="special">.</span><span class="identifier">add</span><span class="special">()</span></code> style of the API, while we here directly
 165           assign the token definitions to <code class="computeroutput"><span class="identifier">self</span></code>,
 166           combining the different token definitions using the <code class="computeroutput"><span class="special">|</span></code>
 167           operator. Here is the code snippet again:
 168         </p>
 169 <pre class="programlisting"><span class="keyword">this</span><span class="special">-&gt;</span><span class="identifier">self</span>
 170     <span class="special">=</span>   <span class="identifier">word</span>  <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">w</span><span class="special">),</span> <span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)</span> <span class="special">+=</span> <span class="identifier">distance</span><span class="special">(</span><span class="identifier">_1</span><span class="special">)]</span>
 171     <span class="special">|</span>   <span class="identifier">eol</span>   <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">),</span> <span class="special">++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">l</span><span class="special">)]</span>
 172     <span class="special">|</span>   <span class="identifier">any</span>   <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)]</span>
 173     <span class="special">;</span>
 174 </pre>
 175 <p>
 176           This way we have a very powerful and natural way of building the lexical
 177           analyzer. If translated into English this may be read as: The lexical analyzer
 178           will recognize ('<code class="computeroutput"><span class="special">=</span></code>') tokens
 179           as defined by any of ('<code class="computeroutput"><span class="special">|</span></code>')
 180           the token definitions <code class="computeroutput"><span class="identifier">word</span></code>,
 181           <code class="computeroutput"><span class="identifier">eol</span></code>, and <code class="computeroutput"><span class="identifier">any</span></code>.
 182         </p>
 183 <p>
 184           A second difference to the previous example is that we do not explicitly
 185           specify any token ids to use for the separate tokens. Using semantic actions
 186           to trigger some useful work has freed us from the need to define those.
 187           To ensure every token gets assigned a id the <span class="emphasis"><em>Spirit.Lex</em></span>
 188           library internally assigns unique numbers to the token definitions, starting
 189           with the constant defined by <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">min_token_id</span></code>.
 190         </p>
 191 <h6>
 192 <a name="spirit.lex.tutorials.lexer_quickstart2.h3"></a>
 193           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart2.pulling_everything_together"></a></span><a class="link" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.pulling_everything_together">Pulling
 194           everything together</a>
 195         </h6>
 196 <p>
 197           In order to execute the code defined above we still need to instantiate
 198           an instance of the lexer type, feed it from some input sequence and create
 199           a pair of iterators allowing to iterate over the token sequence as created
 200           by the lexer. This code shows how to achieve these steps:
 201         </p>
 202 <p>
 203 </p>
 204 <pre class="programlisting"><span class="keyword">int</span> <span class="identifier">main</span><span class="special">(</span><span class="keyword">int</span> <span class="identifier">argc</span><span class="special">,</span> <span class="keyword">char</span><span class="special">*</span> <span class="identifier">argv</span><span class="special">[])</span>
 205 <span class="special">{</span>
 206
 207 <a class="co" name="spirit.lex.tutorials.lexer_quickstart2.c0" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.c1"><img src="../../../../../../../doc/src/images/callouts/1.png" alt="1" border="0"></a>  <span class="keyword">typedef</span>
 208         <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special">&lt;</span><span class="keyword">char</span> <span class="keyword">const</span><span class="special">*,</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">omit</span><span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">false_</span><span class="special">&gt;</span>
 209      <span class="identifier">token_type</span><span class="special">;</span>
 210
 211 <a class="co" name="spirit.lex.tutorials.lexer_quickstart2.c2" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.c3"><img src="../../../../../../../doc/src/images/callouts/2.png" alt="2" border="0"></a>  <span class="keyword">typedef</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">actor_lexer</span><span class="special">&lt;</span><span class="identifier">token_type</span><span class="special">&gt;</span> <span class="identifier">lexer_type</span><span class="special">;</span>
 212
 213 <a class="co" name="spirit.lex.tutorials.lexer_quickstart2.c4" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.c5"><img src="../../../../../../../doc/src/images/callouts/3.png" alt="3" border="0"></a>  <span class="identifier">word_count_tokens</span><span class="special">&lt;</span><span class="identifier">lexer_type</span><span class="special">&gt;</span> <span class="identifier">word_count_lexer</span><span class="special">;</span>
 214
 215 <a class="co" name="spirit.lex.tutorials.lexer_quickstart2.c6" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.c7"><img src="../../../../../../../doc/src/images/callouts/4.png" alt="4" border="0"></a>  <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span> <span class="identifier">str</span> <span class="special">(</span><span class="identifier">read_from_file</span><span class="special">(</span><span class="number">1</span> <span class="special">==</span> <span class="identifier">argc</span> <span class="special">?</span> <span class="string">"word_count.input"</span> <span class="special">:</span> <span class="identifier">argv</span><span class="special">[</span><span class="number">1</span><span class="special">]));</span>
 216     <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">first</span> <span class="special">=</span> <span class="identifier">str</span><span class="special">.</span><span class="identifier">c_str</span><span class="special">();</span>
 217     <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">last</span> <span class="special">=</span> <span class="special">&amp;</span><span class="identifier">first</span><span class="special">[</span><span class="identifier">str</span><span class="special">.</span><span class="identifier">size</span><span class="special">()];</span>
 218
 219 <a class="co" name="spirit.lex.tutorials.lexer_quickstart2.c8" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.c9"><img src="../../../../../../../doc/src/images/callouts/5.png" alt="5" border="0"></a>  <span class="identifier">lexer_type</span><span class="special">::</span><span class="identifier">iterator_type</span> <span class="identifier">iter</span> <span class="special">=</span> <span class="identifier">word_count_lexer</span><span class="special">.</span><span class="identifier">begin</span><span class="special">(</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">);</span>
 220     <span class="identifier">lexer_type</span><span class="special">::</span><span class="identifier">iterator_type</span> <span class="identifier">end</span> <span class="special">=</span> <span class="identifier">word_count_lexer</span><span class="special">.</span><span class="identifier">end</span><span class="special">();</span>
 221
 222 <a class="co" name="spirit.lex.tutorials.lexer_quickstart2.c10" href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.c11"><img src="../../../../../../../doc/src/images/callouts/6.png" alt="6" border="0"></a>  <span class="keyword">while</span> <span class="special">(</span><span class="identifier">iter</span> <span class="special">!=</span> <span class="identifier">end</span> <span class="special">&amp;&amp;</span> <span class="identifier">token_is_valid</span><span class="special">(*</span><span class="identifier">iter</span><span class="special">))</span>
 223         <span class="special">++</span><span class="identifier">iter</span><span class="special">;</span>
 224
 225     <span class="keyword">if</span> <span class="special">(</span><span class="identifier">iter</span> <span class="special">==</span> <span class="identifier">end</span><span class="special">)</span> <span class="special">{</span>
 226         <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="string">"lines: "</span> <span class="special">&lt;&lt;</span> <span class="identifier">word_count_lexer</span><span class="special">.</span><span class="identifier">l</span>
 227                   <span class="special">&lt;&lt;</span> <span class="string">", words: "</span> <span class="special">&lt;&lt;</span> <span class="identifier">word_count_lexer</span><span class="special">.</span><span class="identifier">w</span>
 228                   <span class="special">&lt;&lt;</span> <span class="string">", characters: "</span> <span class="special">&lt;&lt;</span> <span class="identifier">word_count_lexer</span><span class="special">.</span><span class="identifier">c</span>
 229                   <span class="special">&lt;&lt;</span> <span class="string">"\n"</span><span class="special">;</span>
 230     <span class="special">}</span>
 231     <span class="keyword">else</span> <span class="special">{</span>
 232         <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span> <span class="identifier">rest</span><span class="special">(</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">);</span>
 233         <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="string">"Lexical analysis failed\n"</span> <span class="special">&lt;&lt;</span> <span class="string">"stopped at: \""</span>
 234                   <span class="special">&lt;&lt;</span> <span class="identifier">rest</span> <span class="special">&lt;&lt;</span> <span class="string">"\"\n"</span><span class="special">;</span>
 235     <span class="special">}</span>
 236     <span class="keyword">return</span> <span class="number">0</span><span class="special">;</span>
 237 <span class="special">}</span>
 238 </pre>
 239 <p>
 240         </p>
 241 <div class="calloutlist"><table border="0" summary="Callout list">
 242 <tr>
 243 <td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart2.c1"></a><a href="#spirit.lex.tutorials.lexer_quickstart2.c0"><img src="../../../../../../../doc/src/images/callouts/1.png" alt="1" border="0"></a> </p></td>
 244 <td valign="top" align="left"><p>
 245               Specifying <code class="computeroutput"><span class="identifier">omit</span></code> as
 246               the token attribute type generates a token class not holding any token
 247               attribute at all (not even the iterator range of the matched input
 248               sequence), therefore optimizing the token, the lexer, and possibly
 249               the parser implementation as much as possible. Specifying <code class="computeroutput"><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">false_</span></code> as the 3rd template parameter
 250               generates a token type and an iterator, both holding no lexer state,
 251               allowing for even more aggressive optimizations. As a result the token
 252               instances contain the token ids as the only data member.
 253             </p></td>
 254 </tr>
 255 <tr>
 256 <td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart2.c3"></a><a href="#spirit.lex.tutorials.lexer_quickstart2.c2"><img src="../../../../../../../doc/src/images/callouts/2.png" alt="2" border="0"></a> </p></td>
 257 <td valign="top" align="left"><p>
 258               This defines the lexer type to use
 259             </p></td>
 260 </tr>
 261 <tr>
 262 <td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart2.c5"></a><a href="#spirit.lex.tutorials.lexer_quickstart2.c4"><img src="../../../../../../../doc/src/images/callouts/3.png" alt="3" border="0"></a> </p></td>
 263 <td valign="top" align="left"><p>
 264               Create the lexer object instance needed to invoke the lexical analysis
 265             </p></td>
 266 </tr>
 267 <tr>
 268 <td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart2.c7"></a><a href="#spirit.lex.tutorials.lexer_quickstart2.c6"><img src="../../../../../../../doc/src/images/callouts/4.png" alt="4" border="0"></a> </p></td>
 269 <td valign="top" align="left"><p>
 270               Read input from the given file, tokenize all the input, while discarding
 271               all generated tokens
 272             </p></td>
 273 </tr>
 274 <tr>
 275 <td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart2.c9"></a><a href="#spirit.lex.tutorials.lexer_quickstart2.c8"><img src="../../../../../../../doc/src/images/callouts/5.png" alt="5" border="0"></a> </p></td>
 276 <td valign="top" align="left"><p>
 277               Create a pair of iterators returning the sequence of generated tokens
 278             </p></td>
 279 </tr>
 280 <tr>
 281 <td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart2.c11"></a><a href="#spirit.lex.tutorials.lexer_quickstart2.c10"><img src="../../../../../../../doc/src/images/callouts/6.png" alt="6" border="0"></a> </p></td>
 282 <td valign="top" align="left"><p>
 283               Here we simply iterate over all tokens, making sure to break the loop
 284               if an invalid token gets returned from the lexer
 285             </p></td>
 286 </tr>
 287 </table></div>
 288 </div>
 289 <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
 290 <td align="left"></td>
 291 <td align="right"><div class="copyright-footer">Copyright &#169; 2001-2011 Joel de Guzman, Hartmut Kaiser<p>
 292         Distributed under the Boost Software License, Version 1.0. (See accompanying
 293         file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
 294       </p>
 295 </div></td>
 296 </tr></table>
 297 <hr>
 298 <div class="spirit-nav">
 299 <a accesskey="p" href="lexer_quickstart1.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_quickstart3.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
 300 </div>
 301 </body>
 302 </html>