Updates to the scanner section. Gave the semantic condition figures bounding

boxes.

git-svn-id: http://svn.complang.org/ragel/trunk@31 052ea7fc-9027-0410-9066-f65837a77df0

diff --git a/doc/conds1.fig b/doc/conds1.fig
index 24e82bf..3436e89 100644 (file)
--- a/doc/conds1.fig
+++ b/doc/conds1.fig
@@ -1,8 +1,8 @@
-#FIG 3.2
+#FIG 3.2  Produced by xfig version 3.2.5-alpha5
 Portrait
 Center
 Metric
-A4
+A4      
 100.00
 Single
 -2
@@ -17,64 +17,80 @@ Single
 # 0
 1 1 0 1 0 32 0 0 -1 0.000 0 0.0000 1400 4650 383 383 1400 4650 1783 5033
 1 1 0 1 0 32 0 0 -1 0.000 0 0.0000 1400 4650 450 450 1400 4650 1850 5100
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 1400 4733 0\001
-# ENTRY -> 0
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 7
-  66 4650 132 4650 225 4650 341 4650 474 4650 617 4650 766 4650
- 0 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 766 4583 933 4650 766 4700 766 4583
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 500 4600 IN\001
 # 1
 1 1 0 1 0 0 0 0 -1 0.000 0 0.0000 3233 4650 383 383 3233 4650 3616 5033
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 3233 4733 1\001
-# 0 -> 1
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 7
-  1866 4650 1995 4650 2129 4650 2266 4650 2403 4650 2537 4650 2666 4650
- 0 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 2666 4583 2833 4650 2666 4700 2666 4583
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 2350 4600 100\001
 # 3
 1 1 0 1 0 0 0 0 -1 0.000 0 0.0000 8466 4650 383 383 8466 4650 8849 5033
 1 1 0 1 0 0 0 0 -1 0.000 0 0.0000 8466 4650 450 450 8466 4650 8916 5100
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 8466 4733 3\001
-# 3 -> 3
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 13
-  8783 4316 8808 4210 8801 4112 8762 4027 8693 3959 8594 3915 8466 3900 8372 3908 8291 3931 8225 3968 8175 4018 8144 4079 8133 4150
- 0 1 1 1 1 1 1 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 8183 4150 8150 4316 8083 4150 8183 4150
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 8466 3850 97..122(test_len)\001
-# 3 -> 1
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 19
-  8016 4633 7959 4632 7904 4629 7852 4625 7800 4620 7750 4617 7700 4616 7136 4601 6679 4589 6272 4581 5865 4577 5403 4577 4833 4583 4661 4591 4483 4600 4302 4608 4122 4616 3947 4625 3783 4633
- 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 3783 4683 3616 4633 3783 4566 3783 4683
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 5500 4533 100(!test_len)\001
 # 2
 1 1 0 1 0 0 0 0 -1 0.000 0 0.0000 5500 5683 383 383 5500 5683 5883 6066
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 5500 5766 2\001
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        766 4583 933 4650 766 4700 766 4583
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        2666 4583 2833 4650 2666 4700 2666 4583
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        8183 4150 8150 4316 8083 4150 8183 4150
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        3783 4683 3616 4633 3783 4566 3783 4683
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        5016 5400 5150 5516 4966 5500 5016 5400
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        7850 4800 8033 4800 7883 4900 7850 4800
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        5266 5233 5233 5400 5166 5233 5266 5233
+2 2 0 0 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+        9360 3645 -90 3645 -90 6165 9360 6165 9360 3645
+# ENTRY -> 0
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 7
+        66 4650 132 4650 225 4650 341 4650 474 4650 617 4650
+        766 4650
+        0.000 1.000 1.000 1.000 1.000 1.000 0.000
+# 0 -> 1
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 7
+        1866 4650 1995 4650 2129 4650 2266 4650 2403 4650 2537 4650
+        2666 4650
+        0.000 1.000 1.000 1.000 1.000 1.000 0.000
+# 3 -> 3
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 13
+        8783 4316 8808 4210 8801 4112 8762 4027 8693 3959 8594 3915
+        8466 3900 8372 3908 8291 3931 8225 3968 8175 4018 8144 4079
+        8133 4150
+        0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 1.000 1.000 0.000
+# 3 -> 1
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 19
+        8016 4633 7959 4632 7904 4629 7852 4625 7800 4620 7750 4617
+        7700 4616 7136 4601 6679 4589 6272 4581 5865 4577 5403 4577
+        4833 4583 4661 4591 4483 4600 4302 4608 4122 4616 3947 4625
+        3783 4633
+        0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 0.000
 # 1 -> 2
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 7
-  3600 4816 3800 4908 4029 5014 4275 5127 4525 5241 4771 5351 5000 5450
- 0 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 5016 5400 5150 5516 4966 5500 5016 5400
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 4233 4966 48..57\001
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 7
+        3600 4816 3800 4908 4029 5014 4275 5127 4525 5241 4771 5351
+        5000 5450
+        0.000 1.000 1.000 1.000 1.000 1.000 0.000
 # 2 -> 3
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 7
-  5866 5550 6147 5455 6477 5342 6835 5218 7200 5090 7550 4965 7866 4850
- 0 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 7850 4800 8033 4800 7883 4900 7850 4800
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 7083 4900 58 / rec_num\001
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 7
+        5866 5550 6147 5455 6477 5342 6835 5218 7200 5090 7550 4965
+        7866 4850
+        0.000 1.000 1.000 1.000 1.000 1.000 0.000
 # 2 -> 2
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 13
-  5766 5400 5787 5300 5782 5207 5752 5125 5695 5059 5611 5015 5500 5000 5416 5008 5348 5030 5295 5066 5256 5113 5230 5169 5216 5233
- 0 1 1 1 1 1 1 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 5266 5233 5233 5400 5166 5233 5266 5233
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 5500 4950 48..57\001
-# end of FIG file
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 13
+        5766 5400 5787 5300 5782 5207 5752 5125 5695 5059 5611 5015
+        5500 5000 5416 5008 5348 5030 5295 5066 5256 5113 5230 5169
+        5216 5233
+        0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 1.000 1.000 0.000
+4 1 0 0 0 0 14 0.0000 2 150 120 1400 4733 0\001
+4 1 0 0 0 0 14 0.0000 2 150 255 500 4600 IN\001
+4 1 0 0 0 0 14 0.0000 2 165 120 3233 4733 1\001
+4 1 0 0 0 0 14 0.0000 2 165 360 2350 4600 100\001
+4 1 0 0 0 0 14 0.0000 2 150 120 8466 4733 3\001
+4 1 0 0 0 0 14 0.0000 2 210 1590 8466 3850 97..122(test_len)\001
+4 1 0 0 0 0 14 0.0000 2 210 1305 5500 4533 100(!test_len)\001
+4 1 0 0 0 0 14 0.0000 2 150 120 5500 5766 2\001
+4 1 0 0 0 0 14 0.0000 2 150 600 4233 4966 48..57\001
+4 1 0 0 0 0 14 0.0000 2 180 1245 7083 4900 58 / rec_num\001
+4 1 0 0 0 0 14 0.0000 2 150 600 5500 4950 48..57\001

diff --git a/doc/conds2.fig b/doc/conds2.fig
index d1cf959..d5a5557 100644 (file)
--- a/doc/conds2.fig
+++ b/doc/conds2.fig
@@ -1,8 +1,8 @@
-#FIG 3.2
+#FIG 3.2  Produced by xfig version 3.2.5-alpha5
 Portrait
 Center
 Metric
-A4
+A4      
 100.00
 Single
 -2
@@ -16,71 +16,92 @@ Single
 1 1 0 1 0 0 0 0 20 0.000 0 0.0000 33 4350 33 33 33 4350 66 4383
 # 0
 1 1 0 1 0 32 0 0 -1 0.000 0 0.0000 1333 4350 383 383 1333 4350 1716 4733
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 1333 4433 0\001
-# ENTRY -> 0
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 7
-  66 4350 139 4350 237 4350 354 4350 485 4350 624 4350 766 4350
- 0 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 766 4283 933 4350 766 4400 766 4283
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 500 4300 IN\001
 # 2
 1 1 0 1 0 0 0 0 -1 0.000 0 0.0000 11633 5016 383 383 11633 5016 12016 5399
 1 1 0 1 0 0 0 0 -1 0.000 0 0.0000 11633 5016 450 450 11633 5016 12083 5466
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 11633 5100 2\001
 # 1
 1 1 0 1 0 0 0 0 -1 0.000 0 0.0000 9183 4433 383 383 9183 4433 9566 4816
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 9183 4516 1\001
-# 0 -> 1
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 25
-  1716 4283 1773 4282 1828 4278 1881 4272 1932 4266 1983 4258 2033 4250 2352 4232 2611 4227 2839 4231 3066 4238 3321 4246 3633 4250 4551 4259 5297 4260 5956 4260 6613 4267 7355 4288 8266 4333 8325 4340 8383 4345 8441 4350 8499 4354 8558 4359 8616 4366
- 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 8616 4316 8783 4383 8616 4416 8616 4316
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 4766 4216 97..122(!test_len)\001
 # 3
 1 1 0 1 0 0 0 0 -1 0.000 0 0.0000 4766 5383 383 383 4766 5383 5149 5766
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 4766 5466 3\001
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        766 4283 933 4350 766 4400 766 4283
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        8616 4316 8783 4383 8616 4416 8616 4316
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        4233 5166 4383 5266 4200 5266 4233 5166
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        11033 4816 11183 4916 11000 4916 11033 4816
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        8950 3983 8916 4150 8850 3983 8950 3983
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        11016 5050 11183 5083 11016 5150 11016 5050
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        8633 4500 8800 4516 8650 4600 8633 4500
+2 3 0 1 0 0 0 0 20 0.000 0 0 0 0 0 4
+        4533 4933 4500 5100 4433 4933 4533 4933
+2 2 0 0 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+        12285 3465 -135 3465 -135 5895 12285 5895 12285 3465
+# ENTRY -> 0
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 7
+        66 4350 139 4350 237 4350 354 4350 485 4350 624 4350
+        766 4350
+        0.000 1.000 1.000 1.000 1.000 1.000 0.000
+# 0 -> 1
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 25
+        1716 4283 1773 4282 1828 4278 1881 4272 1932 4266 1983 4258
+        2033 4250 2352 4232 2611 4227 2839 4231 3066 4238 3321 4246
+        3633 4250 4551 4259 5297 4260 5956 4260 6613 4267 7355 4288
+        8266 4333 8325 4340 8383 4345 8441 4350 8499 4354 8558 4359
+        8616 4366
+        0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        0.000
 # 0 -> 3
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 7
-  1716 4466 2062 4570 2479 4694 2935 4829 3398 4966 3835 5098 4216 5216
- 0 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 4233 5166 4383 5266 4200 5266 4233 5166
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 2833 4533 97..122(test_len)\001
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 7
+        1716 4466 2062 4570 2479 4694 2935 4829 3398 4966 3835 5098
+        4216 5216
+        0.000 1.000 1.000 1.000 1.000 1.000 0.000
 # 1 -> 2
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 7
-  9566 4516 9774 4564 10009 4622 10260 4685 10518 4750 10773 4811 11016 4866
- 0 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 11033 4816 11183 4916 11000 4916 11033 4816
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 10483 4633 10 / two\001
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 7
+        9566 4516 9774 4564 10009 4622 10260 4685 10518 4750 10773 4811
+        11016 4866
+        0.000 1.000 1.000 1.000 1.000 1.000 0.000
 # 1 -> 1
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 13
-  9450 4150 9470 4050 9466 3957 9435 3875 9378 3809 9294 3765 9183 3750 9099 3758 9032 3780 8979 3816 8940 3863 8914 3919 8900 3983
- 0 1 1 1 1 1 1 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 8950 3983 8916 4150 8850 3983 8950 3983
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 9183 3700 48..57, 97..122\001
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 13
+        9450 4150 9470 4050 9466 3957 9435 3875 9378 3809 9294 3765
+        9183 3750 9099 3758 9032 3780 8979 3816 8940 3863 8914 3919
+        8900 3983
+        0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 1.000 1.000 0.000
 # 3 -> 2
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 13
-  5166 5383 5685 5380 6364 5370 7158 5352 8024 5324 8919 5284 9800 5233 10010 5215 10222 5195 10433 5172 10638 5149 10833 5124 11016 5100
- 0 1 1 1 1 1 1 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 11016 5050 11183 5083 11016 5150 11016 5050
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 9183 5183 10 / one, two\001
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 13
+        5166 5383 5685 5380 6364 5370 7158 5352 8024 5324 8919 5284
+        9800 5233 10010 5215 10222 5195 10433 5172 10638 5149 10833 5124
+        11016 5100
+        0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 1.000 1.000 0.000
 # 3 -> 1
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 7
-  5150 5300 5623 5200 6208 5075 6854 4937 7508 4796 8118 4663 8633 4550
- 0 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 8633 4500 8800 4516 8650 4600 8633 4500
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 7083 4616 48..57, 97..122(!test_len)\001
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 7
+        5150 5300 5623 5200 6208 5075 6854 4937 7508 4796 8118 4663
+        8633 4550
+        0.000 1.000 1.000 1.000 1.000 1.000 0.000
 # 3 -> 3
-3 4 0 1 0 0 0 0 -1 0.0 0 0 0 13
-  5033 5100 5054 5000 5049 4907 5018 4825 4961 4759 4877 4715 4766 4700 4683 4708 4615 4730 4562 4766 4523 4813 4497 4869 4483 4933
- 0 1 1 1 1 1 1 1 1 1 1 1 0
-2 3 0 1 0 0 0 0 20 0.0 0 0 0 0 0 4
- 4533 4933 4500 5100 4433 4933 4533 4933
-4 1 0 0 0 0 14.0 0.0000 2 0.0 0.0 4766 4650 97..122(test_len)\001
-# end of FIG file
+3 4 0 1 0 0 0 0 -1 0.000 0 0 0 13
+        5033 5100 5054 5000 5049 4907 5018 4825 4961 4759 4877 4715
+        4766 4700 4683 4708 4615 4730 4562 4766 4523 4813 4497 4869
+        4483 4933
+        0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
+        1.000 1.000 1.000 1.000 0.000
+4 1 0 0 0 0 14 0.0000 2 150 120 1333 4433 0\001
+4 1 0 0 0 0 14 0.0000 2 150 255 500 4300 IN\001
+4 1 0 0 0 0 14 0.0000 2 150 120 11633 5100 2\001
+4 1 0 0 0 0 14 0.0000 2 165 120 9183 4516 1\001
+4 1 0 0 0 0 14 0.0000 2 210 1665 4766 4216 97..122(!test_len)\001
+4 1 0 0 0 0 14 0.0000 2 150 120 4766 5466 3\001
+4 1 0 0 0 0 14 0.0000 2 210 1590 2833 4533 97..122(test_len)\001
+4 1 0 0 0 0 14 0.0000 2 165 780 10483 4633 10 / two\001
+4 1 0 0 0 0 14 0.0000 2 195 1440 9183 3700 48..57, 97..122\001
+4 1 0 0 0 0 14 0.0000 2 195 1245 9183 5183 10 / one, two\001
+4 1 0 0 0 0 14 0.0000 2 210 2385 7083 4616 48..57, 97..122(!test_len)\001
+4 1 0 0 0 0 14 0.0000 2 210 1590 4766 4650 97..122(test_len)\001

diff --git a/doc/ragel-guide.tex b/doc/ragel-guide.tex
index 28e80f1..5b091ae 100644 (file)
--- a/doc/ragel-guide.tex
+++ b/doc/ragel-guide.tex
@@ -2845,19 +2845,25 @@ name tree. However, it can always be renamed.
 % banned.
 
 
-
 \section{Scanners}
 
-Though the overall language may be represented by regular expressions, it may
-be the case that a language
-contains sub-regions where the input is best represented as a sequence
-of tokens. To support the scanning of sub-regions of a language, Ragel allows
-the definition of longest-match machines, also known as scanners. The generated
-code will repeatedly attempt to match patterns from a list, favouring longer
-patterns over shorter patterns.  In the case of equal length matches, the
-generated code will favour patterns that appear ahead of others. When a scanner
-makes a match it executes the user code associated with the match, consumes the
-input then resumes scanning.
+Scanners are very much intertwinded with regular-languages and their
+corresponding processors. For this reason Ragel supports the definition of
+Scanners.  The generated code will repeatedly attempt to match patterns from a
+list, favouring longer patterns over shorter patterns.  In the case of
+equal-length matches, the generated code will favour patterns that appear ahead
+of others. When a scanner makes a match it executes the user code associated
+with the match, consumes the input then resumes scanning.
+
+\verbspace
+\begin{verbatim}
+<machine_name> := |* 
+        pattern1 => action1;
+        pattern2 => action2;
+        ...
+    *|;
+\end{verbatim}
+\verbspace
 
 On the surface, Ragel scanners are similar to those defined by Lex. Though
 there is a key distinguishing feature: patterns may be arbitrary Ragel
@@ -2865,19 +2871,15 @@ expressions and can therefore contain embedded code. With a Ragel-based scanner
 the user need not wait until the end of a pattern before user code can be
 executed.
 
-The longest-match construction is not a pure state machine construction. It
-relies on several variables which enable it to backtrack and make pointers to the
-matched input text available to the user. 
-For this reason scanners must be immediately instantiated. They cannot be defined inline or
-referenced by another expression. Scanners must be jumped to or called.
+Scanners can be used to processes sub-languages, as well as for tokenizing
+programming languages. In the following example a scanner is used to tokenize
+the contents of header field.
 
-% GENERATE: scanner
-% %%{
-%      machine scanner;
-%      word = 'foo';
-%      head_name = 'bar';
 \begin{inline_code}
 \begin{verbatim}
+word = [a-z]+;
+head_name = 'Header';
+
 header := |*
     word;
     ' ';
@@ -2887,42 +2889,24 @@ header := |*
 main := ( head_name ':' @{ fcall header; } )*;
 \end{verbatim}
 \end{inline_code}
-% }%%
-% %% write data;
-% void f()
-% {
-%      %% write init;
-%      %% write exec;
-% }
-% END GENERATE
+\verbspace
 
 The scanner construction has a purpose similar to the longest-match kleene star
-operator \verb|**| when used in conjunction with the union operator. The key
-difference is that a scanner is able to backtrack to match a previously
-matched shorter string when the pursuit of a longer string fails.
-
-The longest-match operator can be used to construct scanners.  The generated
-machine repeatedly attempts to match one of the given patterns, first favouring
-longer pattern matches over shorter ones. If there is a choice between equal
-length matches, the match of the pattern which appears first is chosen.
-
-\verbspace
-\begin{verbatim}
-<machine_name> := |* 
-        pattern1 => action1;
-        pattern2 => action2;
-        ...
-    *|;
-\end{verbatim}
-\verbspace
+operator \verb|**|. The key
+difference is that a scanner is able to backtrack to match a previously matched
+shorter string when the pursuit of a longer string fails.  For this reason the
+scanner construction operator is not a pure state machine construction
+operator. It relies on several variables which enable it to backtrack and make
+pointers to the matched input text available to the user.  For this reason
+scanners must be immediately instantiated. They cannot be defined inline or
+referenced by another expression. Scanners must be jumped to or called.
 
-The longest-match construction operator is not a pure state machine operator.
-It relies on the \verb|tokstart|, \verb|tokend| and \verb|act| variables to be
-present so that it can backtrack and make pointers to the matched text
-available to the user. If input is processed using multiple calls to the
-execute code then the user must ensure that when a token is only partially
-matched that the prefix is preserved on the subsequent invocation of the
-execute code.
+Scanners rely on the \verb|tokstart|, \verb|tokend| and \verb|act|
+variables to be present so that it can backtrack and make pointers to the
+matched text available to the user. If input is processed using multiple calls
+to the execute code then the user must ensure that when a token is only
+partially matched that the prefix is preserved on the subsequent invocation of
+the execute code.
 
 The \verb|tokstart| variable must be defined as a pointer to the input data.
 It is used for recording where the current token match begins. This variable
@@ -3068,7 +3052,7 @@ An example scanner processing loop is given in Figure \ref{scanner-loop}.
 \section{State Charts}
 
 In addition to supporting the construction of state machines using regular
-languages, Ragel also provides a way to manually specify state machines using
+languages, Ragel provides a way to manually specify state machines using
 state charts.  The comma operator wombines machines together without any
 implied transitions. The user can then manually link machines by specifying
 epsilon transitions with the \verb|->| operator.  Epsilon transitions are drawn
@@ -3155,7 +3139,7 @@ wrap up a collection of states using a regular expression we do not loose
 access to the states and transitions. We can still execute code on the
 transitions that we have encapsulated.
 
-\subsection{Down One Level of Abstraction}
+\subsection{Dropping Down One Level of Abstraction}
 \label{down}
 
 The second benefit of incorporating state charts into Ragel is that it permits
@@ -3272,8 +3256,6 @@ example.  Two intersecting patterns are unioned, one with a condition and one
 without. The condition embedded in the first pattern does not affect the second
 pattern.
 
-\newpage
-
 % GENERATE: conds2
 % OPT: -p
 % %%{