libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart1.html

   1 <html>
   2 <head>
   3 <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
   4 <title>Quickstart 1 - A word counter using Spirit.Lex</title>
   5 <link rel="stylesheet" href="../../../../../../../doc/src/boostbook.css" type="text/css">
   6 <meta name="generator" content="DocBook XSL Stylesheets V1.78.1">
   7 <link rel="home" href="../../../index.html" title="Spirit 2.5.2">
   8 <link rel="up" href="../tutorials.html" title="Spirit.Lex Tutorials">
   9 <link rel="prev" href="lexer_tutorials.html" title="Spirit.Lex Tutorials Overview">
  10 <link rel="next" href="lexer_quickstart2.html" title="Quickstart 2 - A better word counter using Spirit.Lex">
  11 </head>
  12 <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
  13 <table cellpadding="2" width="100%"><tr>
  14 <td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
  15 <td align="center"><a href="../../../../../../../index.html">Home</a></td>
  16 <td align="center"><a href="../../../../../../../libs/libraries.htm">Libraries</a></td>
  17 <td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
  18 <td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
  19 <td align="center"><a href="../../../../../../../more/index.htm">More</a></td>
  20 </tr></table>
  21 <hr>
  22 <div class="spirit-nav">
  23 <a accesskey="p" href="lexer_tutorials.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_quickstart2.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
  24 </div>
  25 <div class="section">
  26 <div class="titlepage"><div><div><h4 class="title">
  27 <a name="spirit.lex.tutorials.lexer_quickstart1"></a><a class="link" href="lexer_quickstart1.html" title="Quickstart 1 - A word counter using Spirit.Lex">Quickstart
  28         1 - A word counter using <span class="emphasis"><em>Spirit.Lex</em></span></a>
  29 </h4></div></div></div>
  30 <p>
  31           <span class="emphasis"><em>Spirit.Lex</em></span> is very modular, which follows the general
  32           building principle of the <a href="http://boost-spirit.com" target="_top">Spirit</a>
  33           libraries. You never pay for features you don't use. It is nicely integrated
  34           with the other parts of <a href="http://boost-spirit.com" target="_top">Spirit</a>
  35           but nevertheless can be used separately to build stand alone lexical analyzers.
  36           The first quick start example describes a stand alone application: counting
  37           characters, words, and lines in a file, very similar to what the well known
  38           Unix command <code class="computeroutput"><span class="identifier">wc</span></code> is doing
  39           (for the full example code see here: <a href="../../../../../example/lex/word_count_functor.cpp" target="_top">word_count_functor.cpp</a>).
  40         </p>
  41 <h6>
  42 <a name="spirit.lex.tutorials.lexer_quickstart1.h0"></a>
  43           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart1.prerequisites"></a></span><a class="link" href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.prerequisites">Prerequisites</a>
  44         </h6>
  45 <p>
  46           The only required <code class="computeroutput"><span class="preprocessor">#include</span></code>
  47           specific to <span class="emphasis"><em>Spirit.Lex</em></span> follows. It is a wrapper for
  48           all necessary definitions to use <span class="emphasis"><em>Spirit.Lex</em></span> in a stand
  49           alone fashion, and on top of the <a href="http://www.benhanson.net/lexertl.html" target="_top">Lexertl</a>
  50           library. Additionally we <code class="computeroutput"><span class="preprocessor">#include</span></code>
  51           two of the Boost headers to define <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">bind</span><span class="special">()</span></code> and <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">ref</span><span class="special">()</span></code>.
  52         </p>
  53 <p>
  54 </p>
  55 <pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">lex_lexertl</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  56 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">bind</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  57 <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">ref</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  58 </pre>
  59 <p>
  60         </p>
  61 <p>
  62           To make all the code below more readable we introduce the following namespaces.
  63         </p>
  64 <p>
  65 </p>
  66 <pre class="programlisting"><span class="keyword">namespace</span> <span class="identifier">lex</span> <span class="special">=</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">;</span>
  67 </pre>
  68 <p>
  69         </p>
  70 <h6>
  71 <a name="spirit.lex.tutorials.lexer_quickstart1.h1"></a>
  72           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart1.defining_tokens"></a></span><a class="link" href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.defining_tokens">Defining
  73           Tokens</a>
  74         </h6>
  75 <p>
  76           The most important step while creating a lexer using <span class="emphasis"><em>Spirit.Lex</em></span>
  77           is to define the tokens to be recognized in the input sequence. This is
  78           normally done by defining the regular expressions describing the matching
  79           character sequences, and optionally their corresponding token ids. Additionally
  80           the defined tokens need to be associated with an instance of a lexer object
  81           as provided by the library. The following code snippet shows how this can
  82           be done using <span class="emphasis"><em>Spirit.Lex</em></span>.
  83         </p>
  84 <p>
  85           The template <code class="computeroutput"><span class="identifier">word_count_tokens</span></code>
  86           defines three different tokens: <code class="computeroutput"><span class="identifier">ID_WORD</span></code>,
  87           <code class="computeroutput"><span class="identifier">ID_EOL</span></code>, and <code class="computeroutput"><span class="identifier">ID_CHAR</span></code>, representing a word (anything
  88           except a whitespace or a newline), a newline character, and any other character
  89           (<code class="computeroutput"><span class="identifier">ID_WORD</span></code>, <code class="computeroutput"><span class="identifier">ID_EOL</span></code>, and <code class="computeroutput"><span class="identifier">ID_CHAR</span></code>
  90           are enum values representing the token ids, but could be anything else
  91           convertible to an integer as well). The direct base class of any token
  92           definition class needs to be the template <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexer</span><span class="special">&lt;&gt;</span></code>, where the corresponding template
  93           parameter (here: <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">lexer</span><span class="special">&lt;</span><span class="identifier">BaseIterator</span><span class="special">&gt;</span></code>)
  94           defines which underlying lexer engine has to be used to provide the required
  95           state machine functionality. In this example we use the Lexertl based lexer
  96           engine as the underlying lexer type.
  97         </p>
  98 <p>
  99 </p>
 100 <pre class="programlisting"><span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">&gt;</span>
 101 <span class="keyword">struct</span> <span class="identifier">word_count_tokens</span> <span class="special">:</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexer</span><span class="special">&lt;</span><span class="identifier">Lexer</span><span class="special">&gt;</span>
 102 <span class="special">{</span>
 103     <span class="identifier">word_count_tokens</span><span class="special">()</span>
 104     <span class="special">{</span>
 105         <span class="comment">// define tokens (the regular expression to match and the corresponding</span>
 106         <span class="comment">// token id) and add them to the lexer </span>
 107         <span class="keyword">this</span><span class="special">-&gt;</span><span class="identifier">self</span><span class="special">.</span><span class="identifier">add</span>
 108             <span class="special">(</span><span class="string">"[^ \t\n]+"</span><span class="special">,</span> <span class="identifier">ID_WORD</span><span class="special">)</span> <span class="comment">// words (anything except ' ', '\t' or '\n')</span>
 109             <span class="special">(</span><span class="string">"\n"</span><span class="special">,</span> <span class="identifier">ID_EOL</span><span class="special">)</span>         <span class="comment">// newline characters</span>
 110             <span class="special">(</span><span class="string">"."</span><span class="special">,</span> <span class="identifier">ID_CHAR</span><span class="special">)</span>         <span class="comment">// anything else is a plain character</span>
 111         <span class="special">;</span>
 112     <span class="special">}</span>
 113 <span class="special">};</span>
 114 </pre>
 115 <p>
 116         </p>
 117 <h6>
 118 <a name="spirit.lex.tutorials.lexer_quickstart1.h2"></a>
 119           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart1.doing_the_useful_work"></a></span><a class="link" href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.doing_the_useful_work">Doing
 120           the Useful Work</a>
 121         </h6>
 122 <p>
 123           We will use a setup, where we want the <span class="emphasis"><em>Spirit.Lex</em></span>
 124           library to invoke a given function after any of the generated tokens is
 125           recognized. For this reason we need to implement a functor taking at least
 126           the generated token as an argument and returning a boolean value allowing
 127           to stop the tokenization process. The default token type used in this example
 128           carries a token value of the type <a href="../../../../../../../libs/range/doc/html/range/reference/utilities/iterator_range.html" target="_top"><code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">iterator_range</span></code></a><code class="computeroutput"><span class="special">&lt;</span><span class="identifier">BaseIterator</span><span class="special">&gt;</span></code>
 129           pointing to the matched range in the underlying input sequence.
 130         </p>
 131 <p>
 132           In this example the struct 'counter' is used as a functor counting the
 133           characters, words and lines in the analyzed input sequence by identifying
 134           the matched tokens as passed from the <span class="emphasis"><em>Spirit.Lex</em></span> library.
 135         </p>
 136 <p>
 137 </p>
 138 <pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">counter</span>
 139 <span class="special">{</span>
 140     <span class="comment">// the function operator gets called for each of the matched tokens</span>
 141     <span class="comment">// c, l, w are references to the counters used to keep track of the numbers</span>
 142     <span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Token</span><span class="special">&gt;</span>
 143     <span class="keyword">bool</span> <span class="keyword">operator</span><span class="special">()(</span><span class="identifier">Token</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">t</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">size_t</span><span class="special">&amp;</span> <span class="identifier">c</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">size_t</span><span class="special">&amp;</span> <span class="identifier">w</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">size_t</span><span class="special">&amp;</span> <span class="identifier">l</span><span class="special">)</span> <span class="keyword">const</span>
 144     <span class="special">{</span>
 145         <span class="keyword">switch</span> <span class="special">(</span><span class="identifier">t</span><span class="special">.</span><span class="identifier">id</span><span class="special">())</span> <span class="special">{</span>
 146         <span class="keyword">case</span> <span class="identifier">ID_WORD</span><span class="special">:</span>       <span class="comment">// matched a word</span>
 147         <span class="comment">// since we're using a default token type in this example, every </span>
 148         <span class="comment">// token instance contains a `iterator_range&lt;BaseIterator&gt;` as its token</span>
 149         <span class="comment">// attribute pointing to the matched character sequence in the input </span>
 150             <span class="special">++</span><span class="identifier">w</span><span class="special">;</span> <span class="identifier">c</span> <span class="special">+=</span> <span class="identifier">t</span><span class="special">.</span><span class="identifier">value</span><span class="special">().</span><span class="identifier">size</span><span class="special">();</span>
 151             <span class="keyword">break</span><span class="special">;</span>
 152         <span class="keyword">case</span> <span class="identifier">ID_EOL</span><span class="special">:</span>        <span class="comment">// matched a newline character</span>
 153             <span class="special">++</span><span class="identifier">l</span><span class="special">;</span> <span class="special">++</span><span class="identifier">c</span><span class="special">;</span>
 154             <span class="keyword">break</span><span class="special">;</span>
 155         <span class="keyword">case</span> <span class="identifier">ID_CHAR</span><span class="special">:</span>       <span class="comment">// matched something else</span>
 156             <span class="special">++</span><span class="identifier">c</span><span class="special">;</span>
 157             <span class="keyword">break</span><span class="special">;</span>
 158         <span class="special">}</span>
 159         <span class="keyword">return</span> <span class="keyword">true</span><span class="special">;</span>        <span class="comment">// always continue to tokenize</span>
 160     <span class="special">}</span>
 161 <span class="special">};</span>
 162 </pre>
 163 <p>
 164         </p>
 165 <p>
 166           All what is left is to write some boilerplate code helping to tie together
 167           the pieces described so far. To simplify this example we call the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize</span><span class="special">()</span></code>
 168           function implemented in <span class="emphasis"><em>Spirit.Lex</em></span> (for a more detailed
 169           description of this function see here: <span class="bold"><strong>FIXME</strong></span>),
 170           even if we could have written a loop to iterate over the lexer iterators
 171           [<code class="computeroutput"><span class="identifier">first</span></code>, <code class="computeroutput"><span class="identifier">last</span></code>)
 172           as well.
 173         </p>
 174 <h6>
 175 <a name="spirit.lex.tutorials.lexer_quickstart1.h3"></a>
 176           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart1.pulling_everything_together"></a></span><a class="link" href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.pulling_everything_together">Pulling
 177           Everything Together</a>
 178         </h6>
 179 <p>
 180           The main function simply loads the given file into memory (as a <code class="computeroutput"><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span></code>), instantiates an instance of
 181           the token definition template using the correct iterator type (<code class="computeroutput"><span class="identifier">word_count_tokens</span><span class="special">&lt;</span><span class="keyword">char</span> <span class="keyword">const</span><span class="special">*&gt;</span></code>), and finally calls <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize</span></code>, passing an instance of the
 182           counter function object. The return value of <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize</span><span class="special">()</span></code> will be <code class="computeroutput"><span class="keyword">true</span></code>
 183           if the whole input sequence has been successfully tokenized, and <code class="computeroutput"><span class="keyword">false</span></code> otherwise.
 184         </p>
 185 <p>
 186 </p>
 187 <pre class="programlisting"><span class="keyword">int</span> <span class="identifier">main</span><span class="special">(</span><span class="keyword">int</span> <span class="identifier">argc</span><span class="special">,</span> <span class="keyword">char</span><span class="special">*</span> <span class="identifier">argv</span><span class="special">[])</span>
 188 <span class="special">{</span>
 189     <span class="comment">// these variables are used to count characters, words and lines</span>
 190     <span class="identifier">std</span><span class="special">::</span><span class="identifier">size_t</span> <span class="identifier">c</span> <span class="special">=</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">w</span> <span class="special">=</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">l</span> <span class="special">=</span> <span class="number">0</span><span class="special">;</span>
 191
 192     <span class="comment">// read input from the given file</span>
 193     <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span> <span class="identifier">str</span> <span class="special">(</span><span class="identifier">read_from_file</span><span class="special">(</span><span class="number">1</span> <span class="special">==</span> <span class="identifier">argc</span> <span class="special">?</span> <span class="string">"word_count.input"</span> <span class="special">:</span> <span class="identifier">argv</span><span class="special">[</span><span class="number">1</span><span class="special">]));</span>
 194
 195     <span class="comment">// create the token definition instance needed to invoke the lexical analyzer</span>
 196     <span class="identifier">word_count_tokens</span><span class="special">&lt;</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">lexer</span><span class="special">&lt;&gt;</span> <span class="special">&gt;</span> <span class="identifier">word_count_functor</span><span class="special">;</span>
 197
 198     <span class="comment">// tokenize the given string, the bound functor gets invoked for each of </span>
 199     <span class="comment">// the matched tokens</span>
 200     <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">first</span> <span class="special">=</span> <span class="identifier">str</span><span class="special">.</span><span class="identifier">c_str</span><span class="special">();</span>
 201     <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">last</span> <span class="special">=</span> <span class="special">&amp;</span><span class="identifier">first</span><span class="special">[</span><span class="identifier">str</span><span class="special">.</span><span class="identifier">size</span><span class="special">()];</span>
 202     <span class="keyword">bool</span> <span class="identifier">r</span> <span class="special">=</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize</span><span class="special">(</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">,</span> <span class="identifier">word_count_functor</span><span class="special">,</span>
 203         <span class="identifier">boost</span><span class="special">::</span><span class="identifier">bind</span><span class="special">(</span><span class="identifier">counter</span><span class="special">(),</span> <span class="identifier">_1</span><span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">),</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">w</span><span class="special">),</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">l</span><span class="special">)));</span>
 204
 205     <span class="comment">// print results</span>
 206     <span class="keyword">if</span> <span class="special">(</span><span class="identifier">r</span><span class="special">)</span> <span class="special">{</span>
 207         <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="string">"lines: "</span> <span class="special">&lt;&lt;</span> <span class="identifier">l</span> <span class="special">&lt;&lt;</span> <span class="string">", words: "</span> <span class="special">&lt;&lt;</span> <span class="identifier">w</span>
 208                   <span class="special">&lt;&lt;</span> <span class="string">", characters: "</span> <span class="special">&lt;&lt;</span> <span class="identifier">c</span> <span class="special">&lt;&lt;</span> <span class="string">"\n"</span><span class="special">;</span>
 209     <span class="special">}</span>
 210     <span class="keyword">else</span> <span class="special">{</span>
 211         <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span> <span class="identifier">rest</span><span class="special">(</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">);</span>
 212         <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="string">"Lexical analysis failed\n"</span> <span class="special">&lt;&lt;</span> <span class="string">"stopped at: \""</span>
 213                   <span class="special">&lt;&lt;</span> <span class="identifier">rest</span> <span class="special">&lt;&lt;</span> <span class="string">"\"\n"</span><span class="special">;</span>
 214     <span class="special">}</span>
 215     <span class="keyword">return</span> <span class="number">0</span><span class="special">;</span>
 216 <span class="special">}</span>
 217 </pre>
 218 <p>
 219         </p>
 220 <h6>
 221 <a name="spirit.lex.tutorials.lexer_quickstart1.h4"></a>
 222           <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart1.comparing__emphasis_spirit_lex__emphasis__with__ulink_url__http___flex_sourceforge_net___flex__ulink_"></a></span><a class="link" href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.comparing__emphasis_spirit_lex__emphasis__with__ulink_url__http___flex_sourceforge_net___flex__ulink_">Comparing
 223           <span class="emphasis"><em>Spirit.Lex</em></span> with Flex</a>
 224         </h6>
 225 <p>
 226           This example was deliberately chosen to be as much as possible similar
 227           to the equivalent <a href="http://flex.sourceforge.net/" target="_top">Flex</a>
 228           program (see below), which isn't too different from what has to be written
 229           when using <span class="emphasis"><em>Spirit.Lex</em></span>.
 230         </p>
 231 <div class="note"><table border="0" summary="Note">
 232 <tr>
 233 <td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../images/note.png"></td>
 234 <th align="left">Note</th>
 235 </tr>
 236 <tr><td align="left" valign="top"><p>
 237             Interestingly enough, performance comparisons of lexical analyzers written
 238             using <span class="emphasis"><em>Spirit.Lex</em></span> with equivalent programs generated
 239             by <a href="http://flex.sourceforge.net/" target="_top">Flex</a> show that both
 240             have comparable execution speeds! Generally, thanks to the highly optimized
 241             <a href="http://www.benhanson.net/lexertl.html" target="_top">Lexertl</a> library
 242             and due its carefully designed integration with <a href="http://boost-spirit.com" target="_top">Spirit</a>
 243             the abstraction penalty to be paid for using <span class="emphasis"><em>Spirit.Lex</em></span>
 244             is negligible.
 245           </p></td></tr>
 246 </table></div>
 247 <p>
 248           The remaining examples in this tutorial will use more sophisticated features
 249           of <span class="emphasis"><em>Spirit.Lex</em></span>, mainly to allow further simplification
 250           of the code to be written, while maintaining the similarity with corresponding
 251           features of <a href="http://flex.sourceforge.net/" target="_top">Flex</a>. <span class="emphasis"><em>Spirit.Lex</em></span>
 252           has been designed to be as similar to <a href="http://flex.sourceforge.net/" target="_top">Flex</a>
 253           as possible. That is why this documentation will provide the corresponding
 254           <a href="http://flex.sourceforge.net/" target="_top">Flex</a> code for the shown
 255           <span class="emphasis"><em>Spirit.Lex</em></span> examples almost everywhere. So consequently,
 256           here is the <a href="http://flex.sourceforge.net/" target="_top">Flex</a> code
 257           corresponding to the example as shown above.
 258         </p>
 259 <p>
 260 </p>
 261 <pre class="programlisting"><span class="special">%{</span>
 262     <span class="preprocessor">#define</span> <span class="identifier">ID_WORD</span> <span class="number">1000</span>
 263     <span class="preprocessor">#define</span> <span class="identifier">ID_EOL</span>  <span class="number">1001</span>
 264     <span class="preprocessor">#define</span> <span class="identifier">ID_CHAR</span> <span class="number">1002</span>
 265     <span class="keyword">int</span> <span class="identifier">c</span> <span class="special">=</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">w</span> <span class="special">=</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">l</span> <span class="special">=</span> <span class="number">0</span><span class="special">;</span>
 266 <span class="special">%}</span>
 267 <span class="special">%%</span>
 268 <span class="special">[^</span> <span class="special">\</span><span class="identifier">t</span><span class="special">\</span><span class="identifier">n</span><span class="special">]+</span>  <span class="special">{</span> <span class="keyword">return</span> <span class="identifier">ID_WORD</span><span class="special">;</span> <span class="special">}</span>
 269 <span class="special">\</span><span class="identifier">n</span>         <span class="special">{</span> <span class="keyword">return</span> <span class="identifier">ID_EOL</span><span class="special">;</span> <span class="special">}</span>
 270 <span class="special">.</span>          <span class="special">{</span> <span class="keyword">return</span> <span class="identifier">ID_CHAR</span><span class="special">;</span> <span class="special">}</span>
 271 <span class="special">%%</span>
 272 <span class="keyword">bool</span> <span class="identifier">count</span><span class="special">(</span><span class="keyword">int</span> <span class="identifier">tok</span><span class="special">)</span>
 273 <span class="special">{</span>
 274     <span class="keyword">switch</span> <span class="special">(</span><span class="identifier">tok</span><span class="special">)</span> <span class="special">{</span>
 275     <span class="keyword">case</span> <span class="identifier">ID_WORD</span><span class="special">:</span> <span class="special">++</span><span class="identifier">w</span><span class="special">;</span> <span class="identifier">c</span> <span class="special">+=</span> <span class="identifier">yyleng</span><span class="special">;</span> <span class="keyword">break</span><span class="special">;</span>
 276     <span class="keyword">case</span> <span class="identifier">ID_EOL</span><span class="special">:</span>  <span class="special">++</span><span class="identifier">l</span><span class="special">;</span> <span class="special">++</span><span class="identifier">c</span><span class="special">;</span> <span class="keyword">break</span><span class="special">;</span>
 277     <span class="keyword">case</span> <span class="identifier">ID_CHAR</span><span class="special">:</span> <span class="special">++</span><span class="identifier">c</span><span class="special">;</span> <span class="keyword">break</span><span class="special">;</span>
 278     <span class="keyword">default</span><span class="special">:</span>
 279         <span class="keyword">return</span> <span class="keyword">false</span><span class="special">;</span>
 280     <span class="special">}</span>
 281     <span class="keyword">return</span> <span class="keyword">true</span><span class="special">;</span>
 282 <span class="special">}</span>
 283 <span class="keyword">void</span> <span class="identifier">main</span><span class="special">()</span>
 284 <span class="special">{</span>
 285     <span class="keyword">int</span> <span class="identifier">tok</span> <span class="special">=</span> <span class="identifier">EOF</span><span class="special">;</span>
 286     <span class="keyword">do</span> <span class="special">{</span>
 287         <span class="identifier">tok</span> <span class="special">=</span> <span class="identifier">yylex</span><span class="special">();</span>
 288         <span class="keyword">if</span> <span class="special">(!</span><span class="identifier">count</span><span class="special">(</span><span class="identifier">tok</span><span class="special">))</span>
 289             <span class="keyword">break</span><span class="special">;</span>
 290     <span class="special">}</span> <span class="keyword">while</span> <span class="special">(</span><span class="identifier">EOF</span> <span class="special">!=</span> <span class="identifier">tok</span><span class="special">);</span>
 291     <span class="identifier">printf</span><span class="special">(</span><span class="string">"%d %d %d\n"</span><span class="special">,</span> <span class="identifier">l</span><span class="special">,</span> <span class="identifier">w</span><span class="special">,</span> <span class="identifier">c</span><span class="special">);</span>
 292 <span class="special">}</span>
 293 </pre>
 294 <p>
 295         </p>
 296 </div>
 297 <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
 298 <td align="left"></td>
 299 <td align="right"><div class="copyright-footer">Copyright &#169; 2001-2011 Joel de Guzman, Hartmut Kaiser<p>
 300         Distributed under the Boost Software License, Version 1.0. (See accompanying
 301         file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
 302       </p>
 303 </div></td>
 304 </tr></table>
 305 <hr>
 306 <div class="spirit-nav">
 307 <a accesskey="p" href="lexer_tutorials.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_quickstart2.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
 308 </div>
 309 </body>
 310 </html>