Change Log
==========
+Version 2.4.0 - April, 2019
+---------------------------
+- Well, it looks like the API change that was introduced in 2.3.1 was more
+ drastic than expected, so for a friendlier forward upgrade path, this
+ release:
+ . Bumps the current version number to 2.4.0, to reflect this
+ incompatible change.
+ . Adds a pyparsing.__compat__ object for specifying compatibility with
+ future breaking changes.
+ . Conditionalizes the API-breaking behavior, based on the value
+ pyparsing.__compat__.collect_all_And_tokens. By default, this value
+ will be set to True, reflecting the new bugfixed behavior. To set this
+ value to False, add to your code:
+
+ import pyparsing
+ pyparsing.__compat__.collect_all_And_tokens = False
+
+ . User code that is dependent on the pre-bugfix behavior can restore
+ it by setting this value to False.
+
+ In 2.5 and later versions, the conditional code will be removed and
+ setting the flag to True or False in these later versions will have no
+ effect.
+
+- Updated unitTests.py and simple_unit_tests.py to be compatible with
+ "python setup.py test". To run tests using setup, do:
+
+ python setup.py test
+ python setup.py test -s unitTests.suite
+ python setup.py test -s simple_unit_tests.suite
+
+ Prompted by issue #83 and PR submitted by bdragon28, thanks.
+
+- Fixed bug in runTests handling '\n' literals in quoted strings.
+
+- Added tag_body attribute to the start tag expressions generated by
+ makeHTMLTags, so that you can avoid using SkipTo to roll your own
+ tag body expression:
+
+ a, aEnd = pp.makeHTMLTags('a')
+ link = a + a.tag_body("displayed_text") + aEnd
+ for t in s.searchString(html_page):
+ print(t.displayed_text, '->', t.startA.href)
+
+- indentedBlock failure handling was improved; PR submitted by TMiguelT,
+ thanks!
+
+- Address Py2 incompatibility in simpleUnitTests, plus explain() and
+ Forward str() cleanup; PRs graciously provided by eswald.
+
+- Fixed docstring with embedded '\w', which creates SyntaxWarnings in
+ Py3.8, issue #80.
+
+- Examples:
+
+ - Added example parser for rosettacode.org tutorial compiler.
+
+ - Added example to show how an HTML table can be parsed into a
+ collection of Python lists or dicts, one per row.
+
+ - Updated SimpleSQL.py example to handle nested selects, reworked
+ 'where' expression to use infixNotation.
+
+ - Added include_preprocessor.py, similar to macroExpander.py.
+
+ - Examples using makeHTMLTags use new tag_body expression when
+ retrieving a tag's body text.
+
+ - Updated examples that are runnable as unit tests:
+
+ python setup.py test -s examples.antlr_grammar_tests
+ python setup.py test -s examples.test_bibparse
+
+
Version 2.3.1 - January, 2019
-----------------------------
- POSSIBLE API CHANGE: this release fixes a bug when results names were
# parse a string with a numeric second value instead of alpha
expr.parseString("123 355")
except pp.ParseException as pe:
- print_(pp.ParseException.explain(pe))
+ print(pp.ParseException.explain(pe))
Prints:
123 355
include pyparsing.py
include HowToUsePyparsing.html pyparsingClassDiagram.*
include README.md CODE_OF_CONDUCT.md CHANGES LICENSE
-include examples/*.py examples/Setup.ini examples/*.dfm examples/*.ics examples/*.html
+include examples/*.py examples/Setup.ini examples/*.dfm examples/*.ics examples/*.html examples/*.h
recursive-include docs *
prune docs/_build/*
recursive-include test *
-Metadata-Version: 1.1
+Metadata-Version: 1.2
Name: pyparsing
-Version: 2.3.1
+Version: 2.4.0
Summary: Python parsing module
Home-page: https://github.com/pyparsing/pyparsing/
Author: Paul McGuire
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
+Requires-Python: >=2.6, !=3.0.*, !=3.1.*, !=3.2.*
-<HTML>\r
-<title>pyparsing Examples</title>\r
-<body>\r
-<h1>pyparsing Examples</h1>\r
-<p>\r
-This directory contains a number of Python scripts that can get you started in learning to use pyparsing.\r
-\r
-<ul>\r
-<li><a href="greeting.py">greeting.py</a><br>\r
-Parse "Hello, World!".\r
-</li>\r
-<p>\r
-\r
-<li><a href="greetingInKorean.py">greetingInKorean.py</a> <i>~ submission by June Kim</i><br>\r
-Unicode example to parse "Hello, World!" in Korean.\r
-</li>\r
-<p>\r
-\r
-<li><a href="greetingInGreek.py">greetingInGreek.py</a> <i>~ submission by ???</i><br>\r
-Unicode example to parse "Hello, World!" in Greek.\r
-</li>\r
-<p>\r
-\r
-<li><a href="holaMundo.py">holaMundo.py</a> <i>~ submission by Marco Alfonso</i><br>\r
-"Hello, World!" example translated to Spanish, from Marco Alfonso's blog.\r
-</li>\r
-<p>\r
-\r
-<li><a href="chemicalFormulas.py">chemicalFormulas.py</a><br>\r
-Simple example to demonstrate the use of ParseResults returned from parseString().\r
-Parses a chemical formula (such as "H2O" or "C6H5OH"), and walks the returned list of tokens to calculate the molecular weight.\r
-</li>\r
-<p>\r
-\r
-<li><a href="wordsToNum.py">wordsToNum.py</a><br>\r
-A sample program that reads a number in words (such as "fifteen hundred and sixty four"), and returns the actual number (1564).\r
-Also demonstrates some processing of ParseExceptions, including marking where the parse failure was found.\r
-</li>\r
-<p>\r
-\r
-<li><a href="pythonGrammarparser.py">pythonGrammarparser.py</a> <i>~ suggested by JH Stovall</i><br>\r
-A sample program that parses the EBNF used in the Python source code to define the Python grammar. From this parser,\r
-one can generate Python grammar documentation tools, such as railroad track diagrams. Also demonstrates use of\r
-Dict class.\r
-</li>\r
-<p>\r
-\r
-<li><a href="commasep.py">commasep.py</a><br>\r
-Demonstration of the use of the commaSeparatedList helper. Shows examples of\r
-proper handling of commas within quotes, trimming of whitespace around delimited entries, and handling of consecutive commas (null arguments). Includes comparison with simple string.split(',').\r
-</li>\r
-<p>\r
-\r
-<li><a href="dictExample.py">dictExample.py</a><br>\r
-A demonstration of using the Dict class, to parse a table of ASCII tabulated data.\r
-</li>\r
-<p>\r
-\r
-<li><a href="dictExample2.py">dictExample2.py</a> <i>~ submission by Mike Kelly</i><br>\r
-An extended version of dictExample.py, in which Mike Kelly also parses the column headers, and generates a transposed version of the original table!\r
-</li>\r
-<p>\r
-\r
-<li><a href="scanExamples.py">scanExamples.py</a><br>\r
-Some examples of using scanString and transformString, as alternative parsing methods to parseString, to do macro substitution, and selection and/or removal of matching strings within a source file.\r
-</li>\r
-<p>\r
-\r
-<li><a href="urlExtractor.py">urlExtractor.py</a><br>\r
-Another example using scanString, this time to extract all HREF references found on Yahoo!'s home page, and return them as a dictionary.\r
-</li>\r
-<p>\r
-\r
-<li><a href="makeHTMLTagExample.py">makeHTMLTagExample.py</a><br>\r
-A sample program showing sample definitions and applications of HTML tag expressions\r
-created using makeHTMLTags helper function. Very useful for scraping data from HTML pages.\r
-</li>\r
-<p>\r
-\r
-<li><a href="urlExtractorNew.py">urlExtractorNew.py</a><br>\r
-Another updated version of urlExtractor.py, using the new makeHTMLTags() method.\r
-</li>\r
-<p>\r
-\r
-<li><a href="fourFn.py">fourFn.py</a><br>\r
-A simple algebraic expression parser, that performs +,-,*,/, and ^ arithmetic operations. (With suggestions and bug-fixes graciously offered by Andrea Griffini.)\r
-</li>\r
-<p>\r
-\r
-<li><a href="SimpleCalc.py">SimpleCalc.py</a> <i>~ submission by Steven Siew</i><br>\r
-An interactive version of fourFn.py, with support for variables.\r
-</li>\r
-<p>\r
-\r
-<li><a href="LAParser.py">LAParser.py</a> <i>~ submission by Mike Ellis</i><br>\r
-An interactive Linear Algebra Parser, an extension of SimpleCalc.py. Supports linear algebra (LA) notation for vectors, matrices, and scalars,\r
-including matrix operations such as inversion and determinants. Converts LA expressions to C code - uses a separate C library for runtime\r
-evaluation of results.\r
-</li>\r
-<p>\r
-\r
-<li><a href="configParse.py">configParse.py</a><br>\r
-A simple alternative to Python's ConfigParse module, demonstrating the use of the Dict class to return nested dictionary access to configuration values.\r
-</li>\r
-<p>\r
-\r
-<li><a href="getNTPservers.py">getNTPservers.py</a><br>\r
-Yet another scanString example, to read/extract the list of NTP servers from NIST's web site.\r
-</li>\r
-<p>\r
-\r
-<li><a href="getNTPserversNew.py">getNTPserversNew.py</a><br>\r
-An updated version of getNTPservers.py, using the new makeHTMLTags() method.\r
-</li>\r
-<p>\r
-\r
-<li><a href="httpServerLogParser.py">httpServerLogParser.py</a><br>\r
-Parser for Apache server log files.\r
-</li>\r
-<p>\r
-\r
-<li><a href="idlParse.py">idlParse.py</a><br>\r
-Parser for CORBA IDL files.\r
-</li>\r
-<p>\r
-\r
-<li><a href="mozillaCalendarParser.py">mozillaCalendarParser.py</a>\r
-<i>~ submission by Petri Savolainen</i><br>\r
-Parser for Mozilla calendar (*.ics) files.\r
-</li>\r
-<p>\r
-\r
-<li><a href="pgn.py">pgn.py</a> <i>~ submission by Alberto Santini</i><br>\r
-Parser for PGN (Portable Game Notation) files, the standard form for documenting the moves in chess games.\r
-</li>\r
-<p>\r
-\r
-<li><a href="simpleSQL.py">simpleSQL.py</a><br>\r
-A simple parser that will extract table and column names from SQL SELECT statements..\r
-</li>\r
-<p>\r
-\r
-<li><a href="dfmparse.py">dfmparse.py</a> <i>~ submission by Dan Griffith</i><br>\r
-Parser for Delphi forms.\r
-</li>\r
-<p>\r
-\r
-<li><a href="ebnf.py">ebnf.py / ebnftest.py</a> <i>~ submission by Seo Sanghyeon</i><br>\r
-An EBNF-compiler that reads EBNF and generates a pyparsing grammar! Including a test that compiles... EBNF itself!\r
-</li>\r
-<p>\r
-\r
-<li><a href="searchparser.py">searchparser.py</a> <i>~ submission by Steven Mooij and Rudolph Froger</i><br>\r
-An expression parser that parses search strings, with special keyword and expression operations using (), not, and, or, and quoted strings.\r
-</li>\r
-<p>\r
-\r
-<li><a href="sparser.py">sparser.py</a> <i>~ submission by Tim Cera</i><br>\r
-A configurable parser module that can be configured with a list of tuples, giving a high-level definition for parsing common sets\r
-of water table data files. Tim had to contend with several different styles of data file formats, each with slight variations of its own.\r
-Tim created a configurable parser (or "SPECIFIED parser" - hence the name "sparser"), that simply works from a config variable listing\r
-the field names and data types, and implicitly, their order in the source data file.\r
-<p>\r
-See <a href="mayport_florida_8720220_data_def.txt">mayport_florida_8720220_data_def.txt</a> for an\r
-example configuration file.\r
-</li>\r
-<p>\r
-\r
-<li><a href="romanNumerals.py">romanNumerals.py</a><br>\r
-A Roman numeral generator and parser example, showing the power of parse actions\r
-to compile Roman numerals into their integer values.\r
-</li>\r
-<p>\r
-\r
-<li><a href="removeLineBreaks.py">removeLineBreaks.py</a><br>\r
-A string transformer that converts text files with hard line-breaks into one with line breaks\r
-only between paragraphs. Useful when converting downloads from\r
-<a href="https://www.gutenberg.org/">Project Gutenberg</a> to import to word processing apps\r
-that can reformat paragraphs once hard line-breaks are removed, or for loading into your Palm Pilot for portable perusal.\r
-<p>\r
-See <a href="Successful Methods of Public Speaking.txt">Successful Methods of Public Speaking.txt</a> and\r
-<a href="Successful Methods of Public Speaking(2).txt">Successful Methods of Public Speaking(2).txt</a> for a sample\r
-before and after (text file courtesy of Project Gutenberg).\r
-</li>\r
-<p>\r
-\r
-<li><a href="listAllMatches.py">listAllMatches.py</a><br>\r
-An example program showing the utility of the listAllMatches option when specifying results naming.\r
-</li>\r
-<p>\r
-\r
-<li><a href="linenoExample.py">linenoExample.py</a><br>\r
-An example program showing how to use the string location to extract line and column numbers, or the\r
-source line of text.\r
-</li>\r
-<p>\r
-\r
-<li><a href="parseListString.py">parseListString.py</a><br>\r
-An example program showing a progression of steps, how to parse a string representation of a Python\r
-list back into a true list.\r
-</li>\r
-<p>\r
-\r
-<li><a href="parsePythonValue.py">parsePythonValue.py</a><br>\r
-An extension of parseListString.py to parse tuples and dicts, including nested values,\r
-returning a Python value of the original type.\r
-</li>\r
-<p>\r
-\r
-<li><a href="indentedGrammarExample.py">indentedGrammarExample.py</a><br>\r
-An example program showing how to parse a grammar using indentation for grouping,\r
-such as is done in Python.\r
-</li>\r
-<p>\r
-\r
-<li><a href="simpleArith.py">simpleArith.py</a><br>\r
-An example program showing how to use the new operatorPrecedence helper method to define a 6-function\r
-(+, -, *, /, ^, and !) arithmetic expression parser, with unary plus and minus signs.\r
-</li>\r
-<p>\r
-\r
-<li><a href="simpleBool.py">simpleBool.py</a><br>\r
-An example program showing how to use the new operatorPrecedence helper method to define a\r
-boolean expression parser, with parse actions associated with each operator to "compile" the expression\r
-into a data structure that will evaluate the expression's boolean value.\r
-</li>\r
-<p>\r
-\r
-<li><a href="simpleWiki.py">simpleWiki.py</a><br>\r
-An example program showing how to use transformString to implement a simple Wiki markup parser.\r
-</li>\r
-<p>\r
-\r
-<li><a href="sql2dot.py">sql2dot.py</a><i>~ submission by EnErGy [CSDX]</i><br>\r
-A nice graphing program that generates schema diagrams from SQL table definition statements.\r
-</li>\r
-<p>\r
-\r
-<li><a href="htmlStripper.py">htmlStripper.py</a><br>\r
-An example implementation of a common application, removing HTML markup tags from an HTML page,\r
-leaving just the text content.\r
-</li>\r
-<p>\r
-\r
-<li><a href="macroExpansion.py">macroExpansion.py</a><br>\r
-An example implementation of a simple preprocessor, that will read embedded macro definitions\r
-and replace macro references with the defined substitution string.\r
-</li>\r
-<p>\r
-\r
-<li><a href="sexpParser.py">sexpParser.py</a><br>\r
-A parser that uses a recursive grammar to parse S-expressions.\r
-</li>\r
-<p>\r
-\r
-<li><a href="nested.py">nested.py</a><br>\r
-An example using nestedExpr, a helper method to simplify definitions of expressions of nested lists.\r
-</li>\r
-<p>\r
-\r
-<li><a href="withAttribute.py">withAttribute.py</a><br>\r
-An example using withAttribute, a helper method to define parse actions to validate matched HTML tags\r
-using additional attributes. Especially helpful for matching common tags such as <DIV> and <TD>.\r
-</li>\r
-<p>\r
-\r
-<li><a href="stackish.py">stackish.py</a><br>\r
-A parser for the data representation format, Stackish.\r
-</li>\r
-<p>\r
-\r
-<li><a href="builtin_parse_action_demo.py">builtin_parse_action_demo.py</a><br>\r
-<b>New in version 1.5.7</b><br>\r
-Demonstration of using builtins (min, max, sum, len, etc.) as parse actions.\r
-</li>\r
-<p>\r
-\r
-<li><a href="antlr_grammar.py">antlr_grammar.py</a><i>~ submission by Luca DellOlio</i><br>\r
-<b>New in version 1.5.7</b><br>\r
-Pyparsing example parsing ANTLR .a files and generating a working pyparsing parser.\r
-</li>\r
-<p>\r
-\r
-<li><a href="shapes.py">shapes.py</a><br>\r
-<b>New in version 1.5.7</b><br>\r
-Parse actions example simple shape definition syntax, and returning the matched tokens as\r
-domain objects instead of just strings.\r
-</li>\r
-<p>\r
-\r
-<li><a href="datetimeParseActions.py">datetimeParseActions.py</a><br>\r
-<b>New in version 1.5.7</b><br>\r
-Parse actions example showing a parse action returning a datetime object instead of\r
-string tokens, and doing validation of the tokens, raising a ParseException if the\r
-given YYYY/MM/DD string does not represent a valid date.\r
-</li>\r
-<p>\r
-\r
-<li><a href="position.py">position.py</a><br>\r
-<b>New in version 1.5.7</b><br>\r
-Demonstration of a couple of different ways to capture the location a particular\r
-expression was found within the overall input string.\r
-</li>\r
-<p>\r
-\r
-\r
-</ul>\r
-\r
-</body></html>\r
+<HTML>
+<title>pyparsing Examples</title>
+<body>
+<h1>pyparsing Examples</h1>
+<p>
+This directory contains a number of Python scripts that can get you started in learning to use pyparsing.
+
+<ul>
+<li><a href="greeting.py">greeting.py</a><br>
+Parse "Hello, World!".
+</li>
+<p>
+
+<li><a href="greetingInKorean.py">greetingInKorean.py</a> <i>~ submission by June Kim</i><br>
+Unicode example to parse "Hello, World!" in Korean.
+</li>
+<p>
+
+<li><a href="greetingInGreek.py">greetingInGreek.py</a> <i>~ submission by ???</i><br>
+Unicode example to parse "Hello, World!" in Greek.
+</li>
+<p>
+
+<li><a href="holaMundo.py">holaMundo.py</a> <i>~ submission by Marco Alfonso</i><br>
+"Hello, World!" example translated to Spanish, from Marco Alfonso's blog.
+</li>
+<p>
+
+<li><a href="chemicalFormulas.py">chemicalFormulas.py</a><br>
+Simple example to demonstrate the use of ParseResults returned from parseString().
+Parses a chemical formula (such as "H2O" or "C6H5OH"), and walks the returned list of tokens to calculate the molecular weight.
+</li>
+<p>
+
+<li><a href="wordsToNum.py">wordsToNum.py</a><br>
+A sample program that reads a number in words (such as "fifteen hundred and sixty four"), and returns the actual number (1564).
+Also demonstrates some processing of ParseExceptions, including marking where the parse failure was found.
+</li>
+<p>
+
+<li><a href="pythonGrammarparser.py">pythonGrammarparser.py</a> <i>~ suggested by JH Stovall</i><br>
+A sample program that parses the EBNF used in the Python source code to define the Python grammar. From this parser,
+one can generate Python grammar documentation tools, such as railroad track diagrams. Also demonstrates use of
+Dict class.
+</li>
+<p>
+
+<li><a href="commasep.py">commasep.py</a><br>
+Demonstration of the use of the commaSeparatedList helper. Shows examples of
+proper handling of commas within quotes, trimming of whitespace around delimited entries, and handling of consecutive commas (null arguments). Includes comparison with simple string.split(',').
+</li>
+<p>
+
+<li><a href="dictExample.py">dictExample.py</a><br>
+A demonstration of using the Dict class, to parse a table of ASCII tabulated data.
+</li>
+<p>
+
+<li><a href="dictExample2.py">dictExample2.py</a> <i>~ submission by Mike Kelly</i><br>
+An extended version of dictExample.py, in which Mike Kelly also parses the column headers, and generates a transposed version of the original table!
+</li>
+<p>
+
+<li><a href="scanExamples.py">scanExamples.py</a><br>
+Some examples of using scanString and transformString, as alternative parsing methods to parseString, to do macro substitution, and selection and/or removal of matching strings within a source file.
+</li>
+<p>
+
+<li><a href="urlExtractorNew.py">urlExtractorNew.py</a><br>
+A sample program showing sample definitions and applications of HTML tag expressions
+created using makeHTMLTags helper function. Very useful for scraping data from HTML pages.
+</li>
+<p>
+
+<li><a href="fourFn.py">fourFn.py</a><br>
+A simple algebraic expression parser, that performs +,-,*,/, and ^ arithmetic operations. (With suggestions and bug-fixes graciously offered by Andrea Griffini.)
+</li>
+<p>
+
+<li><a href="SimpleCalc.py">SimpleCalc.py</a> <i>~ submission by Steven Siew</i><br>
+An interactive version of fourFn.py, with support for variables.
+</li>
+<p>
+
+<li><a href="LAParser.py">LAParser.py</a> <i>~ submission by Mike Ellis</i><br>
+An interactive Linear Algebra Parser, an extension of SimpleCalc.py. Supports linear algebra (LA) notation for vectors, matrices, and scalars,
+including matrix operations such as inversion and determinants. Converts LA expressions to C code - uses a separate C library for runtime
+evaluation of results.
+</li>
+<p>
+
+<li><a href="configParse.py">configParse.py</a><br>
+A simple alternative to Python's ConfigParse module, demonstrating the use of the Dict class to return nested dictionary access to configuration values.
+</li>
+<p>
+
+<li><a href="getNTPserversNew.py">getNTPserversNew.py</a><br>
+Yet another scanString example, to read/extract the list of NTP servers from NIST's web site.
+Uses the new makeHTMLTags() method.
+</li>
+<p>
+
+<li><a href="httpServerLogParser.py">httpServerLogParser.py</a><br>
+Parser for Apache server log files.
+</li>
+<p>
+
+<li><a href="idlParse.py">idlParse.py</a><br>
+Parser for CORBA IDL files.
+</li>
+<p>
+
+<li><a href="mozillaCalendarParser.py">mozillaCalendarParser.py</a>
+<i>~ submission by Petri Savolainen</i><br>
+Parser for Mozilla calendar (*.ics) files.
+</li>
+<p>
+
+<li><a href="pgn.py">pgn.py</a> <i>~ submission by Alberto Santini</i><br>
+Parser for PGN (Portable Game Notation) files, the standard form for documenting the moves in chess games.
+</li>
+<p>
+
+<li><a href="simpleSQL.py">simpleSQL.py</a><br>
+A simple parser that will extract table and column names from SQL SELECT statements..
+</li>
+<p>
+
+<li><a href="dfmparse.py">dfmparse.py</a> <i>~ submission by Dan Griffith</i><br>
+Parser for Delphi forms.
+</li>
+<p>
+
+<li><a href="ebnf.py">ebnf.py / ebnftest.py</a> <i>~ submission by Seo Sanghyeon</i><br>
+An EBNF-compiler that reads EBNF and generates a pyparsing grammar! Including a test that compiles... EBNF itself!
+</li>
+<p>
+
+<li><a href="searchparser.py">searchparser.py</a> <i>~ submission by Steven Mooij and Rudolph Froger</i><br>
+An expression parser that parses search strings, with special keyword and expression operations using (), not, and, or, and quoted strings.
+</li>
+<p>
+
+<li><a href="sparser.py">sparser.py</a> <i>~ submission by Tim Cera</i><br>
+A configurable parser module that can be configured with a list of tuples, giving a high-level definition for parsing common sets
+of water table data files. Tim had to contend with several different styles of data file formats, each with slight variations of its own.
+Tim created a configurable parser (or "SPECIFIED parser" - hence the name "sparser"), that simply works from a config variable listing
+the field names and data types, and implicitly, their order in the source data file.
+<p>
+See <a href="mayport_florida_8720220_data_def.txt">mayport_florida_8720220_data_def.txt</a> for an
+example configuration file.
+</li>
+<p>
+
+<li><a href="romanNumerals.py">romanNumerals.py</a><br>
+A Roman numeral generator and parser example, showing the power of parse actions
+to compile Roman numerals into their integer values.
+</li>
+<p>
+
+<li><a href="removeLineBreaks.py">removeLineBreaks.py</a><br>
+A string transformer that converts text files with hard line-breaks into one with line breaks
+only between paragraphs. Useful when converting downloads from
+<a href="https://www.gutenberg.org/">Project Gutenberg</a> to import to word processing apps
+that can reformat paragraphs once hard line-breaks are removed, or for loading into your Palm Pilot for portable perusal.
+<p>
+See <a href="Successful Methods of Public Speaking.txt">Successful Methods of Public Speaking.txt</a> and
+<a href="Successful Methods of Public Speaking(2).txt">Successful Methods of Public Speaking(2).txt</a> for a sample
+before and after (text file courtesy of Project Gutenberg).
+</li>
+<p>
+
+<li><a href="listAllMatches.py">listAllMatches.py</a><br>
+An example program showing the utility of the listAllMatches option when specifying results naming.
+</li>
+<p>
+
+<li><a href="linenoExample.py">linenoExample.py</a><br>
+An example program showing how to use the string location to extract line and column numbers, or the
+source line of text.
+</li>
+<p>
+
+<li><a href="parseListString.py">parseListString.py</a><br>
+An example program showing a progression of steps, how to parse a string representation of a Python
+list back into a true list.
+</li>
+<p>
+
+<li><a href="parsePythonValue.py">parsePythonValue.py</a><br>
+An extension of parseListString.py to parse tuples and dicts, including nested values,
+returning a Python value of the original type.
+</li>
+<p>
+
+<li><a href="indentedGrammarExample.py">indentedGrammarExample.py</a><br>
+An example program showing how to parse a grammar using indentation for grouping,
+such as is done in Python.
+</li>
+<p>
+
+<li><a href="simpleArith.py">simpleArith.py</a><br>
+An example program showing how to use the new operatorPrecedence helper method to define a 6-function
+(+, -, *, /, ^, and !) arithmetic expression parser, with unary plus and minus signs.
+</li>
+<p>
+
+<li><a href="simpleBool.py">simpleBool.py</a><br>
+An example program showing how to use the new operatorPrecedence helper method to define a
+boolean expression parser, with parse actions associated with each operator to "compile" the expression
+into a data structure that will evaluate the expression's boolean value.
+</li>
+<p>
+
+<li><a href="simpleWiki.py">simpleWiki.py</a><br>
+An example program showing how to use transformString to implement a simple Wiki markup parser.
+</li>
+<p>
+
+<li><a href="sql2dot.py">sql2dot.py</a><i>~ submission by EnErGy [CSDX]</i><br>
+A nice graphing program that generates schema diagrams from SQL table definition statements.
+</li>
+<p>
+
+<li><a href="htmlStripper.py">htmlStripper.py</a><br>
+An example implementation of a common application, removing HTML markup tags from an HTML page,
+leaving just the text content.
+</li>
+<p>
+
+<li><a href="macroExpansion.py">macroExpansion.py</a><br>
+An example implementation of a simple preprocessor, that will read embedded macro definitions
+and replace macro references with the defined substitution string.
+</li>
+<p>
+
+<li><a href="sexpParser.py">sexpParser.py</a><br>
+A parser that uses a recursive grammar to parse S-expressions.
+</li>
+<p>
+
+<li><a href="nested.py">nested.py</a><br>
+An example using nestedExpr, a helper method to simplify definitions of expressions of nested lists.
+</li>
+<p>
+
+<li><a href="withAttribute.py">withAttribute.py</a><br>
+An example using withAttribute, a helper method to define parse actions to validate matched HTML tags
+using additional attributes. Especially helpful for matching common tags such as <DIV> and <TD>.
+</li>
+<p>
+
+<li><a href="stackish.py">stackish.py</a><br>
+A parser for the data representation format, Stackish.
+</li>
+<p>
+
+<li><a href="builtin_parse_action_demo.py">builtin_parse_action_demo.py</a><br>
+<b>New in version 1.5.7</b><br>
+Demonstration of using builtins (min, max, sum, len, etc.) as parse actions.
+</li>
+<p>
+
+<li><a href="antlr_grammar.py">antlr_grammar.py</a><i>~ submission by Luca DellOlio</i><br>
+<b>New in version 1.5.7</b><br>
+Pyparsing example parsing ANTLR .a files and generating a working pyparsing parser.
+</li>
+<p>
+
+<li><a href="shapes.py">shapes.py</a><br>
+<b>New in version 1.5.7</b><br>
+Parse actions example simple shape definition syntax, and returning the matched tokens as
+domain objects instead of just strings.
+</li>
+<p>
+
+<li><a href="datetimeParseActions.py">datetimeParseActions.py</a><br>
+<b>New in version 1.5.7</b><br>
+Parse actions example showing a parse action returning a datetime object instead of
+string tokens, and doing validation of the tokens, raising a ParseException if the
+given YYYY/MM/DD string does not represent a valid date.
+</li>
+<p>
+
+<li><a href="position.py">position.py</a><br>
+<b>New in version 1.5.7</b><br>
+Demonstration of a couple of different ways to capture the location a particular
+expression was found within the overall input string.
+</li>
+<p>
+
+
+</ul>
+
+</body></html>
-'''\r
-antlr_grammar.py\r
-\r
-Created on 4 sept. 2010\r
-\r
-@author: luca\r
-\r
-Submitted by Luca DallOlio, September, 2010\r
-(Minor updates by Paul McGuire, June, 2012)\r
-'''\r
-from pyparsing import Word, ZeroOrMore, printables, Suppress, OneOrMore, Group, \\r
- LineEnd, Optional, White, originalTextFor, hexnums, nums, Combine, Literal, Keyword, \\r
- cStyleComment, Regex, Forward, MatchFirst, And, oneOf, alphas, alphanums, \\r
- delimitedList\r
-\r
-# http://www.antlr.org/grammar/ANTLR/ANTLRv3.g\r
-\r
-# Tokens\r
-EOL = Suppress(LineEnd()) # $\r
-singleTextString = originalTextFor(ZeroOrMore(~EOL + (White(" \t") | Word(printables)))).leaveWhitespace()\r
-XDIGIT = hexnums\r
-INT = Word(nums)\r
-ESC = Literal('\\') + (oneOf(list(r'nrtbf\">'+"'")) | ('u' + Word(hexnums, exact=4)) | Word(printables, exact=1))\r
-LITERAL_CHAR = ESC | ~(Literal("'") | Literal('\\')) + Word(printables, exact=1)\r
-CHAR_LITERAL = Suppress("'") + LITERAL_CHAR + Suppress("'")\r
-STRING_LITERAL = Suppress("'") + Combine(OneOrMore(LITERAL_CHAR)) + Suppress("'")\r
-DOUBLE_QUOTE_STRING_LITERAL = '"' + ZeroOrMore(LITERAL_CHAR) + '"'\r
-DOUBLE_ANGLE_STRING_LITERAL = '<<' + ZeroOrMore(Word(printables, exact=1)) + '>>'\r
-TOKEN_REF = Word(alphas.upper(), alphanums+'_')\r
-RULE_REF = Word(alphas.lower(), alphanums+'_')\r
-ACTION_ESC = (Suppress("\\") + Suppress("'")) | Suppress('\\"') | Suppress('\\') + (~(Literal("'") | Literal('"')) + Word(printables, exact=1))\r
-ACTION_CHAR_LITERAL = Suppress("'") + (ACTION_ESC | ~(Literal('\\') | Literal("'")) + Word(printables, exact=1)) + Suppress("'")\r
-ACTION_STRING_LITERAL = Suppress('"') + ZeroOrMore(ACTION_ESC | ~(Literal('\\') | Literal('"')) + Word(printables, exact=1)) + Suppress('"')\r
-SRC = Suppress('src') + ACTION_STRING_LITERAL("file") + INT("line")\r
-id = TOKEN_REF | RULE_REF\r
-SL_COMMENT = Suppress('//') + Suppress('$ANTLR') + SRC | ZeroOrMore(~EOL + Word(printables)) + EOL\r
-ML_COMMENT = cStyleComment\r
-WS = OneOrMore(Suppress(' ') | Suppress('\t') | (Optional(Suppress('\r')) + Literal('\n')))\r
-WS_LOOP = ZeroOrMore(SL_COMMENT | ML_COMMENT)\r
-NESTED_ARG_ACTION = Forward()\r
-NESTED_ARG_ACTION << Suppress('[') + ZeroOrMore(NESTED_ARG_ACTION | ACTION_STRING_LITERAL | ACTION_CHAR_LITERAL) + Suppress(']')\r
-ARG_ACTION = NESTED_ARG_ACTION\r
-NESTED_ACTION = Forward()\r
-NESTED_ACTION << Suppress('{') + ZeroOrMore(NESTED_ACTION | SL_COMMENT | ML_COMMENT | ACTION_STRING_LITERAL | ACTION_CHAR_LITERAL) + Suppress('}')\r
-ACTION = NESTED_ACTION + Optional('?')\r
-SCOPE = Suppress('scope')\r
-OPTIONS = Suppress('options') + Suppress('{') # + WS_LOOP + Suppress('{')\r
-TOKENS = Suppress('tokens') + Suppress('{') # + WS_LOOP + Suppress('{')\r
-FRAGMENT = 'fragment';\r
-TREE_BEGIN = Suppress('^(')\r
-ROOT = Suppress('^')\r
-BANG = Suppress('!')\r
-RANGE = Suppress('..')\r
-REWRITE = Suppress('->')\r
-\r
-# General Parser Definitions\r
-\r
-# Grammar heading\r
-optionValue = id | STRING_LITERAL | CHAR_LITERAL | INT | Literal('*').setName("s")\r
-\r
-option = Group(id("id") + Suppress('=') + optionValue("value"))("option")\r
-optionsSpec = OPTIONS + Group(OneOrMore(option + Suppress(';')))("options") + Suppress('}')\r
-tokenSpec = Group(TOKEN_REF("token_ref") + (Suppress('=') + (STRING_LITERAL | CHAR_LITERAL)("lit")))("token") + Suppress(';')\r
-tokensSpec = TOKENS + Group(OneOrMore(tokenSpec))("tokens") + Suppress('}')\r
-attrScope = Suppress('scope') + id + ACTION\r
-grammarType = Keyword('lexer') + Keyword('parser') + Keyword('tree')\r
-actionScopeName = id | Keyword('lexer')("l") | Keyword('parser')("p")\r
-action = Suppress('@') + Optional(actionScopeName + Suppress('::')) + id + ACTION\r
-\r
-grammarHeading = Optional(ML_COMMENT("ML_COMMENT")) + Optional(grammarType) + Suppress('grammar') + id("grammarName") + Suppress(';') + Optional(optionsSpec) + Optional(tokensSpec) + ZeroOrMore(attrScope) + ZeroOrMore(action)\r
-\r
-modifier = Keyword('protected') | Keyword('public') | Keyword('private') | Keyword('fragment')\r
-ruleAction = Suppress('@') + id + ACTION\r
-throwsSpec = Suppress('throws') + delimitedList(id)\r
-ruleScopeSpec = (Suppress('scope') + ACTION) | (Suppress('scope') + delimitedList(id) + Suppress(';')) | (Suppress('scope') + ACTION + Suppress('scope') + delimitedList(id) + Suppress(';'))\r
-unary_op = oneOf("^ !")\r
-notTerminal = CHAR_LITERAL | TOKEN_REF | STRING_LITERAL\r
-terminal = (CHAR_LITERAL | TOKEN_REF + Optional(ARG_ACTION) | STRING_LITERAL | '.') + Optional(unary_op)\r
-block = Forward()\r
-notSet = Suppress('~') + (notTerminal | block)\r
-rangeNotPython = CHAR_LITERAL("c1") + RANGE + CHAR_LITERAL("c2")\r
-atom = Group(rangeNotPython + Optional(unary_op)("op")) | terminal | (notSet + Optional(unary_op)("op")) | (RULE_REF + Optional(ARG_ACTION("arg")) + Optional(unary_op)("op"))\r
-element = Forward()\r
-treeSpec = Suppress('^(') + element*(2,) + Suppress(')')\r
-ebnfSuffix = oneOf("? * +")\r
-ebnf = block + Optional(ebnfSuffix("op") | '=>')\r
-elementNoOptionSpec = (id("result_name") + oneOf('= +=')("labelOp") + atom("atom") + Optional(ebnfSuffix)) | (id("result_name") + oneOf('= +=')("labelOp") + block + Optional(ebnfSuffix)) | atom("atom") + Optional(ebnfSuffix) | ebnf | ACTION | (treeSpec + Optional(ebnfSuffix)) # | SEMPRED ( '=>' -> GATED_SEMPRED | -> SEMPRED )\r
-element << Group(elementNoOptionSpec)("element")\r
-alternative = Group(Group(OneOrMore(element))("elements")) # Do not ask me why group is needed twice... seems like the xml that you see is not always the real structure?\r
-rewrite = Optional(Literal('TODO REWRITE RULES TODO'))\r
-block << Suppress('(') + Optional(Optional(optionsSpec("opts")) + Suppress(':')) + Group(alternative('a1') + rewrite + Group(ZeroOrMore(Suppress('|') + alternative('a2') + rewrite))("alternatives"))("block") + Suppress(')')\r
-altList = alternative('a1') + rewrite + Group(ZeroOrMore(Suppress('|') + alternative('a2') + rewrite))("alternatives")\r
-exceptionHandler = Suppress('catch') + ARG_ACTION + ACTION\r
-finallyClause = Suppress('finally') + ACTION\r
-exceptionGroup = (OneOrMore(exceptionHandler) + Optional(finallyClause)) | finallyClause\r
-\r
-ruleHeading = Optional(ML_COMMENT)("ruleComment") + Optional(modifier)("modifier") + id("ruleName") + Optional("!") + Optional(ARG_ACTION("arg")) + Optional(Suppress('returns') + ARG_ACTION("rt")) + Optional(throwsSpec) + Optional(optionsSpec) + Optional(ruleScopeSpec) + ZeroOrMore(ruleAction)\r
-rule = Group(ruleHeading + Suppress(':') + altList + Suppress(';') + Optional(exceptionGroup))("rule")\r
-\r
-grammarDef = grammarHeading + Group(OneOrMore(rule))("rules")\r
-\r
-def grammar():\r
- return grammarDef\r
-\r
-def __antlrAlternativesConverter(pyparsingRules, antlrBlock):\r
- rule = None\r
- if hasattr(antlrBlock, 'alternatives') and antlrBlock.alternatives != '' and len(antlrBlock.alternatives) > 0:\r
- alternatives = []\r
- alternatives.append(__antlrAlternativeConverter(pyparsingRules, antlrBlock.a1))\r
- for alternative in antlrBlock.alternatives:\r
- alternatives.append(__antlrAlternativeConverter(pyparsingRules, alternative))\r
- rule = MatchFirst(alternatives)("anonymous_or")\r
- elif hasattr(antlrBlock, 'a1') and antlrBlock.a1 != '':\r
- rule = __antlrAlternativeConverter(pyparsingRules, antlrBlock.a1)\r
- else:\r
- raise Exception('Not yet implemented')\r
- assert rule != None\r
- return rule\r
-\r
-def __antlrAlternativeConverter(pyparsingRules, antlrAlternative):\r
- elementList = []\r
- for element in antlrAlternative.elements:\r
- rule = None\r
- if hasattr(element.atom, 'c1') and element.atom.c1 != '':\r
- regex = r'['+str(element.atom.c1[0])+'-'+str(element.atom.c2[0]+']')\r
- rule = Regex(regex)("anonymous_regex")\r
- elif hasattr(element, 'block') and element.block != '':\r
- rule = __antlrAlternativesConverter(pyparsingRules, element.block)\r
- else:\r
- ruleRef = element.atom\r
- assert ruleRef in pyparsingRules\r
- rule = pyparsingRules[element.atom](element.atom)\r
- if hasattr(element, 'op') and element.op != '':\r
- if element.op == '+':\r
- rule = Group(OneOrMore(rule))("anonymous_one_or_more")\r
- elif element.op == '*':\r
- rule = Group(ZeroOrMore(rule))("anonymous_zero_or_more")\r
- elif element.op == '?':\r
- rule = Optional(rule)\r
- else:\r
- raise Exception('rule operator not yet implemented : ' + element.op)\r
- rule = rule\r
- elementList.append(rule)\r
- if len(elementList) > 1:\r
- rule = Group(And(elementList))("anonymous_and")\r
- else:\r
- rule = elementList[0]\r
- assert rule != None\r
- return rule\r
-\r
-def __antlrRuleConverter(pyparsingRules, antlrRule):\r
- rule = None\r
- rule = __antlrAlternativesConverter(pyparsingRules, antlrRule)\r
- assert rule != None\r
- rule(antlrRule.ruleName)\r
- return rule\r
-\r
-def antlrConverter(antlrGrammarTree):\r
- pyparsingRules = {}\r
- antlrTokens = {}\r
- for antlrToken in antlrGrammarTree.tokens:\r
- antlrTokens[antlrToken.token_ref] = antlrToken.lit\r
- for antlrTokenName, antlrToken in list(antlrTokens.items()):\r
- pyparsingRules[antlrTokenName] = Literal(antlrToken)\r
- antlrRules = {}\r
- for antlrRule in antlrGrammarTree.rules:\r
- antlrRules[antlrRule.ruleName] = antlrRule\r
- pyparsingRules[antlrRule.ruleName] = Forward() # antlr is a top down grammar\r
- for antlrRuleName, antlrRule in list(antlrRules.items()):\r
- pyparsingRule = __antlrRuleConverter(pyparsingRules, antlrRule)\r
- assert pyparsingRule != None\r
- pyparsingRules[antlrRuleName] << pyparsingRule\r
- return pyparsingRules\r
-\r
-if __name__ == "__main__":\r
-\r
- text = """grammar SimpleCalc;\r
-\r
-options {\r
- language = Python;\r
-}\r
-\r
-tokens {\r
- PLUS = '+' ;\r
- MINUS = '-' ;\r
- MULT = '*' ;\r
- DIV = '/' ;\r
-}\r
-\r
-/*------------------------------------------------------------------\r
- * PARSER RULES\r
- *------------------------------------------------------------------*/\r
-\r
-expr : term ( ( PLUS | MINUS ) term )* ;\r
-\r
-term : factor ( ( MULT | DIV ) factor )* ;\r
-\r
-factor : NUMBER ;\r
-\r
-\r
-/*------------------------------------------------------------------\r
- * LEXER RULES\r
- *------------------------------------------------------------------*/\r
-\r
-NUMBER : (DIGIT)+ ;\r
-\r
-/* WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+ { $channel = HIDDEN; } ; */\r
-\r
-fragment DIGIT : '0'..'9' ;\r
-\r
-"""\r
-\r
- grammar().validate()\r
- antlrGrammarTree = grammar().parseString(text)\r
- print(antlrGrammarTree.asXML("antlrGrammarTree"))\r
- pyparsingRules = antlrConverter(antlrGrammarTree)\r
- pyparsingRule = pyparsingRules["expr"]\r
- pyparsingTree = pyparsingRule.parseString("2 - 5 * 42 + 7 / 25")\r
- print(pyparsingTree.asXML("pyparsingTree"))\r
+'''
+antlr_grammar.py
+
+Created on 4 sept. 2010
+
+@author: luca
+
+Submitted by Luca DallOlio, September, 2010
+(Minor updates by Paul McGuire, June, 2012)
+(Code idiom updates by Paul McGuire, April, 2019)
+'''
+from pyparsing import (Word, ZeroOrMore, printables, Suppress, OneOrMore, Group,
+ LineEnd, Optional, White, originalTextFor, hexnums, nums, Combine, Literal, Keyword,
+ cStyleComment, Regex, Forward, MatchFirst, And, oneOf, alphas, alphanums,
+ delimitedList, Char)
+
+# http://www.antlr.org/grammar/ANTLR/ANTLRv3.g
+
+QUOTE,APOS,EQ,LBRACK,RBRACK,LBRACE,RBRACE,LPAR,RPAR,ROOT,BANG,AT,TIL,SEMI,COLON,VERT = map(Suppress,
+ '"\'=[]{}()^!@~;:|')
+BSLASH = Literal('\\')
+keywords = (SRC_, SCOPE_, OPTIONS_, TOKENS_, FRAGMENT, ID, LEXER, PARSER, GRAMMAR, TREE, CATCH, FINALLY,
+ THROWS, PROTECTED, PUBLIC, PRIVATE, ) = map(Keyword,
+ """src scope options tokens fragment id lexer parser grammar tree catch finally throws protected
+ public private """.split())
+KEYWORD = MatchFirst(keywords)
+
+# Tokens
+EOL = Suppress(LineEnd()) # $
+SGL_PRINTABLE = Char(printables)
+singleTextString = originalTextFor(ZeroOrMore(~EOL + (White(" \t") | Word(printables)))).leaveWhitespace()
+XDIGIT = hexnums
+INT = Word(nums)
+ESC = BSLASH + (oneOf(list(r'nrtbf\">'+"'")) | ('u' + Word(hexnums, exact=4)) | SGL_PRINTABLE)
+LITERAL_CHAR = ESC | ~(APOS | BSLASH) + SGL_PRINTABLE
+CHAR_LITERAL = APOS + LITERAL_CHAR + APOS
+STRING_LITERAL = APOS + Combine(OneOrMore(LITERAL_CHAR)) + APOS
+DOUBLE_QUOTE_STRING_LITERAL = '"' + ZeroOrMore(LITERAL_CHAR) + '"'
+DOUBLE_ANGLE_STRING_LITERAL = '<<' + ZeroOrMore(SGL_PRINTABLE) + '>>'
+TOKEN_REF = Word(alphas.upper(), alphanums+'_')
+RULE_REF = Word(alphas.lower(), alphanums+'_')
+ACTION_ESC = (BSLASH.suppress() + APOS
+ | BSLASH.suppress()
+ | BSLASH.suppress() + (~(APOS | QUOTE) + SGL_PRINTABLE)
+ )
+ACTION_CHAR_LITERAL = (APOS + (ACTION_ESC | ~(BSLASH | APOS) + SGL_PRINTABLE) + APOS)
+ACTION_STRING_LITERAL = (QUOTE + ZeroOrMore(ACTION_ESC | ~(BSLASH | QUOTE) + SGL_PRINTABLE) + QUOTE)
+
+SRC = SRC_.suppress() + ACTION_STRING_LITERAL("file") + INT("line")
+id = TOKEN_REF | RULE_REF
+SL_COMMENT = Suppress('//') + Suppress('$ANTLR') + SRC | ZeroOrMore(~EOL + Word(printables)) + EOL
+ML_COMMENT = cStyleComment
+WS = OneOrMore(Suppress(' ') | Suppress('\t') | (Optional(Suppress('\r')) + Literal('\n')))
+WS_LOOP = ZeroOrMore(SL_COMMENT | ML_COMMENT)
+NESTED_ARG_ACTION = Forward()
+NESTED_ARG_ACTION << (LBRACK
+ + ZeroOrMore(NESTED_ARG_ACTION
+ | ACTION_STRING_LITERAL
+ | ACTION_CHAR_LITERAL)
+ + RBRACK)
+ARG_ACTION = NESTED_ARG_ACTION
+NESTED_ACTION = Forward()
+NESTED_ACTION << (LBRACE
+ + ZeroOrMore(NESTED_ACTION
+ | SL_COMMENT
+ | ML_COMMENT
+ | ACTION_STRING_LITERAL
+ | ACTION_CHAR_LITERAL)
+ + RBRACE)
+ACTION = NESTED_ACTION + Optional('?')
+SCOPE = SCOPE_.suppress()
+OPTIONS = OPTIONS_.suppress() + LBRACE # + WS_LOOP + Suppress('{')
+TOKENS = TOKENS_.suppress() + LBRACE # + WS_LOOP + Suppress('{')
+TREE_BEGIN = ROOT + LPAR
+RANGE = Suppress('..')
+REWRITE = Suppress('->')
+
+# General Parser Definitions
+
+# Grammar heading
+optionValue = id | STRING_LITERAL | CHAR_LITERAL | INT | Literal('*').setName("s")
+
+option = Group(id("id") + EQ + optionValue("value"))("option")
+optionsSpec = OPTIONS + Group(OneOrMore(option + SEMI))("options") + RBRACE
+tokenSpec = Group(TOKEN_REF("token_ref")
+ + (EQ + (STRING_LITERAL | CHAR_LITERAL)("lit")))("token") + SEMI
+tokensSpec = TOKENS + Group(OneOrMore(tokenSpec))("tokens") + RBRACE
+attrScope = SCOPE_.suppress() + id + ACTION
+grammarType = LEXER + PARSER + TREE
+actionScopeName = id | LEXER("l") | PARSER("p")
+action = AT + Optional(actionScopeName + Suppress('::')) + id + ACTION
+
+grammarHeading = (Optional(ML_COMMENT("ML_COMMENT"))
+ + Optional(grammarType)
+ + GRAMMAR
+ + id("grammarName") + SEMI
+ + Optional(optionsSpec)
+ + Optional(tokensSpec)
+ + ZeroOrMore(attrScope)
+ + ZeroOrMore(action))
+
+modifier = PROTECTED | PUBLIC | PRIVATE | FRAGMENT
+ruleAction = AT + id + ACTION
+throwsSpec = THROWS.suppress() + delimitedList(id)
+ruleScopeSpec = ((SCOPE_.suppress() + ACTION)
+ | (SCOPE_.suppress() + delimitedList(id) + SEMI)
+ | (SCOPE_.suppress() + ACTION + SCOPE_.suppress() + delimitedList(id) + SEMI))
+unary_op = oneOf("^ !")
+notTerminal = CHAR_LITERAL | TOKEN_REF | STRING_LITERAL
+terminal = (CHAR_LITERAL | TOKEN_REF + Optional(ARG_ACTION) | STRING_LITERAL | '.') + Optional(unary_op)
+block = Forward()
+notSet = TIL + (notTerminal | block)
+rangeNotPython = CHAR_LITERAL("c1") + RANGE + CHAR_LITERAL("c2")
+atom = Group((rangeNotPython + Optional(unary_op)("op"))
+ | terminal
+ | (notSet + Optional(unary_op)("op"))
+ | (RULE_REF + Optional(ARG_ACTION("arg")) + Optional(unary_op)("op"))
+ )
+element = Forward()
+treeSpec = ROOT + LPAR + element*(2,) + RPAR
+ebnfSuffix = oneOf("? * +")
+ebnf = block + Optional(ebnfSuffix("op") | '=>')
+elementNoOptionSpec = ((id("result_name") + oneOf('= +=')("labelOp") + atom("atom") + Optional(ebnfSuffix))
+ | (id("result_name") + oneOf('= +=')("labelOp") + block + Optional(ebnfSuffix))
+ | atom("atom") + Optional(ebnfSuffix)
+ | ebnf
+ | ACTION
+ | (treeSpec + Optional(ebnfSuffix))
+ ) # | SEMPRED ( '=>' -> GATED_SEMPRED | -> SEMPRED )
+element <<= Group(elementNoOptionSpec)("element")
+# Do not ask me why group is needed twice... seems like the xml that you see is not always the real structure?
+alternative = Group(Group(OneOrMore(element))("elements"))
+rewrite = Optional(Literal('TODO REWRITE RULES TODO'))
+block <<= (LPAR
+ + Optional(Optional(optionsSpec("opts")) + COLON)
+ + Group(alternative('a1')
+ + rewrite
+ + Group(ZeroOrMore(VERT
+ + alternative('a2')
+ + rewrite))("alternatives"))("block")
+ + RPAR)
+altList = alternative('a1') + rewrite + Group(ZeroOrMore(VERT + alternative('a2') + rewrite))("alternatives")
+exceptionHandler = CATCH.suppress() + ARG_ACTION + ACTION
+finallyClause = FINALLY.suppress() + ACTION
+exceptionGroup = (OneOrMore(exceptionHandler) + Optional(finallyClause)) | finallyClause
+
+ruleHeading = (Optional(ML_COMMENT)("ruleComment")
+ + Optional(modifier)("modifier")
+ + id("ruleName")
+ + Optional("!")
+ + Optional(ARG_ACTION("arg"))
+ + Optional(Suppress('returns') + ARG_ACTION("rt"))
+ + Optional(throwsSpec)
+ + Optional(optionsSpec)
+ + Optional(ruleScopeSpec)
+ + ZeroOrMore(ruleAction))
+rule = Group(ruleHeading + COLON + altList + SEMI + Optional(exceptionGroup))("rule")
+
+grammarDef = grammarHeading + Group(OneOrMore(rule))("rules")
+
+def grammar():
+ return grammarDef
+
+def __antlrAlternativesConverter(pyparsingRules, antlrBlock):
+ rule = None
+ if hasattr(antlrBlock, 'alternatives') and antlrBlock.alternatives != '' and len(antlrBlock.alternatives) > 0:
+ alternatives = []
+ alternatives.append(__antlrAlternativeConverter(pyparsingRules, antlrBlock.a1))
+ for alternative in antlrBlock.alternatives:
+ alternatives.append(__antlrAlternativeConverter(pyparsingRules, alternative))
+ rule = MatchFirst(alternatives)("anonymous_or")
+ elif hasattr(antlrBlock, 'a1') and antlrBlock.a1 != '':
+ rule = __antlrAlternativeConverter(pyparsingRules, antlrBlock.a1)
+ else:
+ raise Exception('Not yet implemented')
+ assert rule != None
+ return rule
+
+def __antlrAlternativeConverter(pyparsingRules, antlrAlternative):
+ elementList = []
+ for element in antlrAlternative.elements:
+ rule = None
+ if hasattr(element.atom, 'c1') and element.atom.c1 != '':
+ regex = r'['+str(element.atom.c1[0])+'-'+str(element.atom.c2[0]+']')
+ rule = Regex(regex)("anonymous_regex")
+ elif hasattr(element, 'block') and element.block != '':
+ rule = __antlrAlternativesConverter(pyparsingRules, element.block)
+ else:
+ ruleRef = element.atom[0]
+ assert ruleRef in pyparsingRules
+ rule = pyparsingRules[ruleRef](ruleRef)
+ if hasattr(element, 'op') and element.op != '':
+ if element.op == '+':
+ rule = Group(OneOrMore(rule))("anonymous_one_or_more")
+ elif element.op == '*':
+ rule = Group(ZeroOrMore(rule))("anonymous_zero_or_more")
+ elif element.op == '?':
+ rule = Optional(rule)
+ else:
+ raise Exception('rule operator not yet implemented : ' + element.op)
+ rule = rule
+ elementList.append(rule)
+ if len(elementList) > 1:
+ rule = Group(And(elementList))("anonymous_and")
+ else:
+ rule = elementList[0]
+ assert rule is not None
+ return rule
+
+def __antlrRuleConverter(pyparsingRules, antlrRule):
+ rule = None
+ rule = __antlrAlternativesConverter(pyparsingRules, antlrRule)
+ assert rule != None
+ rule(antlrRule.ruleName)
+ return rule
+
+def antlrConverter(antlrGrammarTree):
+ pyparsingRules = {}
+
+ antlrTokens = {}
+ for antlrToken in antlrGrammarTree.tokens:
+ antlrTokens[antlrToken.token_ref] = antlrToken.lit
+ for antlrTokenName, antlrToken in list(antlrTokens.items()):
+ pyparsingRules[antlrTokenName] = Literal(antlrToken)
+
+ antlrRules = {}
+ for antlrRule in antlrGrammarTree.rules:
+ antlrRules[antlrRule.ruleName] = antlrRule
+ pyparsingRules[antlrRule.ruleName] = Forward() # antlr is a top down grammar
+ for antlrRuleName, antlrRule in list(antlrRules.items()):
+ pyparsingRule = __antlrRuleConverter(pyparsingRules, antlrRule)
+ assert pyparsingRule != None
+ pyparsingRules[antlrRuleName] <<= pyparsingRule
+
+ return pyparsingRules
+
+if __name__ == "__main__":
+
+ text = """\
+grammar SimpleCalc;
+
+options {
+ language = Python;
+}
+
+tokens {
+ PLUS = '+' ;
+ MINUS = '-' ;
+ MULT = '*' ;
+ DIV = '/' ;
+}
+
+/*------------------------------------------------------------------
+ * PARSER RULES
+ *------------------------------------------------------------------*/
+
+expr : term ( ( PLUS | MINUS ) term )* ;
+
+term : factor ( ( MULT | DIV ) factor )* ;
+
+factor : NUMBER ;
+
+
+/*------------------------------------------------------------------
+ * LEXER RULES
+ *------------------------------------------------------------------*/
+
+NUMBER : (DIGIT)+ ;
+
+/* WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+ { $channel = HIDDEN; } ; */
+
+fragment DIGIT : '0'..'9' ;
+
+"""
+
+ grammar().validate()
+ antlrGrammarTree = grammar().parseString(text)
+ print(antlrGrammarTree.dump())
+ pyparsingRules = antlrConverter(antlrGrammarTree)
+ pyparsingRule = pyparsingRules["expr"]
+ pyparsingTree = pyparsingRule.parseString("2 - 5 * 42 + 7 / 25")
+ print(pyparsingTree.dump())
pyparsingRules = antlr_grammar.antlrConverter(antlrGrammarTree)
pyparsingRule = pyparsingRules["expr"]
pyparsingTree = pyparsingRule.parseString("2 - 5 * 42 + 7 / 25")
- self.assertNotEqual(None, pyparsingTree)
+ pyparsingTreeList = pyparsingTree.asList()
+ print(pyparsingTreeList)
+ self.assertEqual(pyparsingTreeList,
+ [[[['2'], []], [['-', [['5'], [['*', ['4', '2']]]]], ['+', [['7'], [['/', ['2', '5']]]]]]]]
+ )
if __name__ == "__main__":
#import sys;sys.argv = ['', 'Test.testOptionsSpec']
-# commasep.py\r
-#\r
-# comma-separated list example, to illustrate the advantages of using\r
-# the pyparsing commaSeparatedList as opposed to string.split(","):\r
-# - leading and trailing whitespace is implicitly trimmed from list elements\r
-# - list elements can be quoted strings, which can safely contain commas without breaking\r
-# into separate elements\r
-#\r
-# Copyright (c) 2004-2016, Paul McGuire\r
-#\r
-\r
-from pyparsing import commaSeparatedList\r
-\r
-testData = [\r
- "a,b,c,100.2,,3",\r
- "d, e, j k , m ",\r
- "'Hello, World', f, g , , 5.1,x",\r
- "John Doe, 123 Main St., Cleveland, Ohio",\r
- "Jane Doe, 456 St. James St., Los Angeles , California ",\r
- "",\r
- ]\r
-\r
-for line in testData:\r
- print(commaSeparatedList.parseString(line))\r
- print(line.split(","))\r
- print()\r
+# commasep.py
+#
+# comma-separated list example, to illustrate the advantages of using
+# the pyparsing commaSeparatedList as opposed to string.split(","):
+# - leading and trailing whitespace is implicitly trimmed from list elements
+# - list elements can be quoted strings, which can safely contain commas without breaking
+# into separate elements
+#
+# Copyright (c) 2004-2016, Paul McGuire
+#
+
+from pyparsing import commaSeparatedList
+
+testData = [
+ "a,b,c,100.2,,3",
+ "d, e, j k , m ",
+ "'Hello, World', f, g , , 5.1,x",
+ "John Doe, 123 Main St., Cleveland, Ohio",
+ "Jane Doe, 456 St. James St., Los Angeles , California ",
+ "",
+ ]
+
+commaSeparatedList.runTests(testData)
# Copyright 2012, Paul T. McGuire\r
#\r
from datetime import datetime\r
-from pyparsing import *\r
+import pyparsing as pp\r
+from pyparsing import pyparsing_common as ppc\r
\r
# define an integer string, and a parse action to convert it\r
# to an integer at parse time\r
-integer = Word(nums).setName("integer")\r
+integer = pp.Word(pp.nums).setName("integer")\r
def convertToInt(tokens):\r
# no need to test for validity - we can't get here\r
# unless tokens[0] contains all numeric digits\r
\r
# define a pattern for a year/month/day date\r
date_expr = integer('year') + '/' + integer('month') + '/' + integer('day')\r
-date_expr.ignore(pythonStyleComment)\r
+date_expr.ignore(pp.pythonStyleComment)\r
\r
def convertToDatetime(s,loc,tokens):\r
try:\r
except Exception as ve:\r
errmsg = "'%s/%s/%s' is not a valid date, %s" % \\r
(tokens.year, tokens.month, tokens.day, ve)\r
- raise ParseException(s, loc, errmsg)\r
+ raise pp.ParseException(s, loc, errmsg)\r
date_expr.setParseAction(convertToDatetime)\r
\r
\r
\r
\r
# if dates conform to ISO8601, use definitions in pyparsing_common\r
-date_expr = pyparsing_common.iso8601_date.setParseAction(pyparsing_common.convertToDate())\r
-date_expr.ignore(pythonStyleComment)\r
+date_expr = ppc.iso8601_date.setParseAction(ppc.convertToDate())\r
+date_expr.ignore(pp.pythonStyleComment)\r
\r
date_expr.runTests("""\\r
2000-01-01\r
Constant ::= intConstant | doubleConstant | boolConstant | stringConstant | null
"""
import pyparsing as pp
+from pyparsing import pyparsing_common as ppc
pp.ParserElement.enablePackrat()
# keywords
LPAR, RPAR, LBRACE, RBRACE, LBRACK, RBRACK, DOT, EQ, COMMA, SEMI = map(pp.Suppress, "(){}[].=,;")
hexConstant = pp.Regex(r"0[xX][0-9a-fA-F]+").addParseAction(lambda t: int(t[0][2:], 16))
-intConstant = hexConstant | pp.pyparsing_common.integer
-doubleConstant = pp.pyparsing_common.real
+intConstant = hexConstant | ppc.integer
+doubleConstant = ppc.real
boolConstant = TRUE | FALSE
stringConstant = pp.dblQuotedString
null = NULL
#\r
# Copyright (c) 2003, Paul McGuire\r
#\r
-from pyparsing import Literal, Word, Group, Dict, ZeroOrMore, alphas, nums, delimitedList\r
-import pprint\r
+import pyparsing as pp\r
\r
testData = """\r
+-------+------+------+------+------+------+------+------+------+\r
"""\r
\r
# define grammar for datatable\r
-heading = (Literal(\r
+heading = (pp.Literal(\r
"+-------+------+------+------+------+------+------+------+------+") +\r
"| | A1 | B1 | C1 | D1 | A2 | B2 | C2 | D2 |" +\r
"+=======+======+======+======+======+======+======+======+======+").suppress()\r
-vert = Literal("|").suppress()\r
-number = Word(nums)\r
-rowData = Group( vert + Word(alphas) + vert + delimitedList(number,"|") + vert )\r
-trailing = Literal(\r
+vert = pp.Literal("|").suppress()\r
+number = pp.Word(pp.nums)\r
+rowData = pp.Group( vert + pp.Word(pp.alphas) + vert + pp.delimitedList(number,"|") + vert )\r
+trailing = pp.Literal(\r
"+-------+------+------+------+------+------+------+------+------+").suppress()\r
\r
-datatable = heading + Dict( ZeroOrMore(rowData) ) + trailing\r
+datatable = heading + pp.Dict(pp.ZeroOrMore(rowData)) + trailing\r
\r
# now parse data and print results\r
data = datatable.parseString(testData)\r
print(data)\r
-pprint.pprint(data.asList())\r
+\r
+# shortcut for import pprint; pprint.pprint(data.asList())\r
+data.pprint()\r
+\r
+# access all data keys\r
print("data keys=", list(data.keys()))\r
+\r
+# use dict-style access to values\r
print("data['min']=", data['min'])\r
+\r
+# use attribute-style access to values (if key is a valid Python identifier)\r
print("data.max", data.max)\r
#\r
# Copyright (c) 2004, Paul McGuire\r
#\r
-from pyparsing import Literal, Word, Group, Dict, ZeroOrMore, alphas, nums, delimitedList, pyparsing_common\r
+from pyparsing import Literal, Word, Group, Dict, ZeroOrMore, alphas, nums, delimitedList, pyparsing_common as ppc\r
\r
testData = """\r
+-------+------+------+------+------+------+------+------+------+\r
\r
# define grammar for datatable\r
underline = Word("-=")\r
-number = pyparsing_common.integer\r
+number = ppc.integer\r
\r
vert = Literal("|").suppress()\r
\r
from pyparsing import (CaselessKeyword, Suppress, Word, alphas,\r
alphanums, nums, Optional, Group, oneOf, Forward,\r
infixNotation, opAssoc, dblQuotedString, delimitedList,\r
- Combine, Literal, QuotedString, ParserElement, pyparsing_common)\r
+ Combine, Literal, QuotedString, ParserElement, pyparsing_common as ppc)\r
ParserElement.enablePackrat()\r
\r
EQ,LPAR,RPAR,COLON,COMMA = map(Suppress, '=():,')\r
COMPARISON_OP = oneOf("< = > >= <= != <>")\r
condExpr = expr + COMPARISON_OP + expr\r
\r
-ifFunc = (CaselessKeyword("if") -\r
- LPAR +\r
- Group(condExpr)("condition") +\r
- COMMA + Group(expr)("if_true") +\r
- COMMA + Group(expr)("if_false") + RPAR)\r
+ifFunc = (CaselessKeyword("if")\r
+ - LPAR\r
+ + Group(condExpr)("condition")\r
+ + COMMA + Group(expr)("if_true")\r
+ + COMMA + Group(expr)("if_false")\r
+ + RPAR)\r
\r
statFunc = lambda name : Group(CaselessKeyword(name) + Group(LPAR + delimitedList(expr) + RPAR))\r
sumFunc = statFunc("sum")\r
\r
multOp = oneOf("* /")\r
addOp = oneOf("+ -")\r
-numericLiteral = pyparsing_common.number\r
+numericLiteral = ppc.number\r
operand = numericLiteral | funcCall | cellRange | cellRef\r
arithExpr = infixNotation(operand,\r
[\r
+++ /dev/null
-# getNTPservers.py\r
-#\r
-# Demonstration of the parsing module, implementing a HTML page scanner,\r
-# to extract a list of NTP time servers from the NIST web site.\r
-#\r
-# Copyright 2004, by Paul McGuire\r
-#\r
-from pyparsing import Word, Combine, Suppress, CharsNotIn, nums\r
-import urllib.request, urllib.parse, urllib.error\r
-\r
-integer = Word(nums)\r
-ipAddress = Combine( integer + "." + integer + "." + integer + "." + integer )\r
-tdStart = Suppress("<td>")\r
-tdEnd = Suppress("</td>")\r
-timeServerPattern = tdStart + ipAddress.setResultsName("ipAddr") + tdEnd + \\r
- tdStart + CharsNotIn("<").setResultsName("loc") + tdEnd\r
-\r
-# get list of time servers\r
-nistTimeServerURL = "http://www.boulder.nist.gov/timefreq/service/time-servers.html"\r
-serverListPage = urllib.request.urlopen( nistTimeServerURL )\r
-serverListHTML = serverListPage.read()\r
-serverListPage.close()\r
-\r
-addrs = {}\r
-for srvr,startloc,endloc in timeServerPattern.scanString( serverListHTML ):\r
- print(srvr.ipAddr, "-", srvr.loc)\r
- addrs[srvr.ipAddr] = srvr.loc\r
- # or do this:\r
- #~ addr,loc = srvr\r
- #~ print addr, "-", loc\r
-# getNTPserversNew.py\r
-#\r
-# Demonstration of the parsing module, implementing a HTML page scanner,\r
-# to extract a list of NTP time servers from the NIST web site.\r
-#\r
-# Copyright 2004-2010, by Paul McGuire\r
-# September, 2010 - updated to more current use of setResultsName, new NIST URL\r
-#\r
-from pyparsing import (Word, Combine, SkipTo, nums, makeHTMLTags,\r
- delimitedList, alphas, alphanums)\r
-try:\r
- import urllib.request\r
- urlopen = urllib.request.urlopen\r
-except ImportError:\r
- import urllib\r
- urlopen = urllib.urlopen\r
-\r
-integer = Word(nums)\r
-ipAddress = Combine( integer + "." + integer + "." + integer + "." + integer )\r
-hostname = delimitedList(Word(alphas,alphanums+"-_"),".",combine=True)\r
-tdStart,tdEnd = makeHTMLTags("td")\r
-timeServerPattern = (tdStart + hostname("hostname") + tdEnd +\r
- tdStart + ipAddress("ipAddr") + tdEnd +\r
- tdStart + SkipTo(tdEnd)("loc") + tdEnd)\r
-\r
-# get list of time servers\r
-nistTimeServerURL = "https://tf.nist.gov/tf-cgi/servers.cgi#"\r
-serverListPage = urlopen( nistTimeServerURL )\r
-serverListHTML = serverListPage.read().decode("UTF-8")\r
-serverListPage.close()\r
-\r
-addrs = {}\r
-for srvr,startloc,endloc in timeServerPattern.scanString( serverListHTML ):\r
- print("{0} ({1}) - {2}".format(srvr.ipAddr, srvr.hostname.strip(), srvr.loc.strip()))\r
- addrs[srvr.ipAddr] = srvr.loc\r
+# getNTPserversNew.py
+#
+# Demonstration of the parsing module, implementing a HTML page scanner,
+# to extract a list of NTP time servers from the NIST web site.
+#
+# Copyright 2004-2010, by Paul McGuire
+# September, 2010 - updated to more current use of setResultsName, new NIST URL
+#
+import pyparsing as pp
+ppc = pp.pyparsing_common
+from contextlib import closing
+
+try:
+ import urllib.request
+ urlopen = urllib.request.urlopen
+except ImportError:
+ import urllib
+ urlopen = urllib.urlopen
+
+integer = pp.Word(pp.nums)
+ipAddress = ppc.ipv4_address()
+hostname = pp.delimitedList(pp.Word(pp.alphas, pp.alphanums+"-_"), ".", combine=True)
+tdStart, tdEnd = pp.makeHTMLTags("td")
+timeServerPattern = (tdStart + hostname("hostname") + tdEnd
+ + tdStart + ipAddress("ipAddr") + tdEnd
+ + tdStart + tdStart.tag_body("loc") + tdEnd)
+
+# get list of time servers
+nistTimeServerURL = "https://tf.nist.gov/tf-cgi/servers.cgi#"
+with closing(urlopen(nistTimeServerURL)) as serverListPage:
+ serverListHTML = serverListPage.read().decode("UTF-8")
+
+addrs = {}
+for srvr, startloc, endloc in timeServerPattern.scanString(serverListHTML):
+ print("{0} ({1}) - {2}".format(srvr.ipAddr, srvr.hostname.strip(), srvr.loc.strip()))
+ addrs[srvr.ipAddr] = srvr.loc
-# greeting.py\r
-#\r
-# Demonstration of the pyparsing module, on the prototypical "Hello, World!"\r
-# example\r
-#\r
-# Copyright 2003, by Paul McGuire\r
-#\r
-from pyparsing import Word, alphas\r
-\r
-# define grammar\r
-greet = Word( alphas ) + "," + Word( alphas ) + "!"\r
-\r
-# input string\r
-hello = "Hello, World!"\r
-\r
-# parse input string\r
-print(hello, "->", greet.parseString( hello ))\r
+# greeting.py
+#
+# Demonstration of the pyparsing module, on the prototypical "Hello, World!"
+# example
+#
+# Copyright 2003, 2019 by Paul McGuire
+#
+import pyparsing as pp
+
+# define grammar
+greet = pp.Word(pp.alphas) + "," + pp.Word(pp.alphas) + pp.oneOf("! ? .")
+
+# input string
+hello = "Hello, World!"
+
+# parse input string
+print(hello, "->", greet.parseString( hello ))
+
+# parse a bunch of input strings
+greet.runTests("""\
+ Hello, World!
+ Ahoy, Matey!
+ Howdy, Pardner!
+ Morning, Neighbor!
+ """)
\ No newline at end of file
#\r
# Copyright 2004-2016, by Paul McGuire\r
#\r
-from pyparsing import Word, pyparsing_unicode\r
+from pyparsing import Word, pyparsing_unicode as ppu\r
\r
# define grammar\r
-alphas = pyparsing_unicode.Greek.alphas\r
+alphas = ppu.Greek.alphas\r
greet = Word(alphas) + ',' + Word(alphas) + '!'\r
\r
# input string\r
#\r
# Copyright 2004-2016, by Paul McGuire\r
#\r
-from pyparsing import Word, pyparsing_unicode\r
+from pyparsing import Word, pyparsing_unicode as ppu\r
\r
-koreanChars = pyparsing_unicode.Korean.alphas\r
+koreanChars = ppu.Korean.alphas\r
koreanWord = Word(koreanChars, min=2)\r
\r
# define grammar\r
# escrito por Marco Alfonso, 2004 Noviembre\r
\r
# importamos los sÃmbolos requeridos desde el módulo\r
-from pyparsing import Word, alphas, oneOf, pyparsing_unicode, nums, Group, OneOrMore\r
+from pyparsing import Word, alphas, oneOf, nums, Group, OneOrMore, pyparsing_unicode as ppu\r
\r
# usamos las letras en latin1, que incluye las como 'ñ', 'á', 'é', etc.\r
-alphas = pyparsing_unicode.Latin1.alphas\r
+alphas = ppu.Latin1.alphas\r
\r
# Aqui decimos que la gramatica "saludo" DEBE contener\r
# una palabra compuesta de caracteres alfanumericos\r
-#\r
-# htmlStripper.py\r
-#\r
-# Sample code for stripping HTML markup tags and scripts from\r
-# HTML source files.\r
-#\r
-# Copyright (c) 2006, 2016, Paul McGuire\r
-#\r
-from contextlib import closing\r
-import urllib.request, urllib.parse, urllib.error\r
-from pyparsing import (makeHTMLTags, SkipTo, commonHTMLEntity, replaceHTMLEntity,\r
- htmlComment, anyOpenTag, anyCloseTag, LineEnd, OneOrMore, replaceWith)\r
-\r
-scriptOpen,scriptClose = makeHTMLTags("script")\r
-scriptBody = scriptOpen + SkipTo(scriptClose) + scriptClose\r
-commonHTMLEntity.setParseAction(replaceHTMLEntity)\r
-\r
-# get some HTML\r
-targetURL = "https://wiki.python.org/moin/PythonDecoratorLibrary"\r
-with closing(urllib.request.urlopen( targetURL )) as targetPage:\r
- targetHTML = targetPage.read().decode("UTF-8")\r
-\r
-# first pass, strip out tags and translate entities\r
-firstPass = (htmlComment | scriptBody | commonHTMLEntity |\r
- anyOpenTag | anyCloseTag ).suppress().transformString(targetHTML)\r
-\r
-# first pass leaves many blank lines, collapse these down\r
-repeatedNewlines = LineEnd() + OneOrMore(LineEnd())\r
-repeatedNewlines.setParseAction(replaceWith("\n\n"))\r
-secondPass = repeatedNewlines.transformString(firstPass)\r
-\r
-print(secondPass)\r
+#
+# htmlStripper.py
+#
+# Sample code for stripping HTML markup tags and scripts from
+# HTML source files.
+#
+# Copyright (c) 2006, 2016, Paul McGuire
+#
+from contextlib import closing
+import urllib.request, urllib.parse, urllib.error
+from pyparsing import (makeHTMLTags, commonHTMLEntity, replaceHTMLEntity,
+ htmlComment, anyOpenTag, anyCloseTag, LineEnd, OneOrMore, replaceWith)
+
+scriptOpen, scriptClose = makeHTMLTags("script")
+scriptBody = scriptOpen + scriptOpen.tag_body + scriptClose
+commonHTMLEntity.setParseAction(replaceHTMLEntity)
+
+# get some HTML
+targetURL = "https://wiki.python.org/moin/PythonDecoratorLibrary"
+with closing(urllib.request.urlopen( targetURL )) as targetPage:
+ targetHTML = targetPage.read().decode("UTF-8")
+
+# first pass, strip out tags and translate entities
+firstPass = (htmlComment | scriptBody | commonHTMLEntity |
+ anyOpenTag | anyCloseTag ).suppress().transformString(targetHTML)
+
+# first pass leaves many blank lines, collapse these down
+repeatedNewlines = LineEnd()*(2,)
+repeatedNewlines.setParseAction(replaceWith("\n\n"))
+secondPass = repeatedNewlines.transformString(firstPass)
+
+print(secondPass)
--- /dev/null
+#
+# htmlTableParser.py
+#
+# Example of parsing a simple HTML table into a list of rows, and optionally into a little database
+#
+# Copyright 2019, Paul McGuire
+#
+
+import pyparsing as pp
+import urllib.request
+
+
+# define basic HTML tags, and compose into a Table
+table, table_end = pp.makeHTMLTags('table')
+thead, thead_end = pp.makeHTMLTags('thead')
+tbody, tbody_end = pp.makeHTMLTags('tbody')
+tr, tr_end = pp.makeHTMLTags('tr')
+th, th_end = pp.makeHTMLTags('th')
+td, td_end = pp.makeHTMLTags('td')
+a, a_end = pp.makeHTMLTags('a')
+
+# method to strip HTML tags from a string - will be used to clean up content of table cells
+strip_html = (pp.anyOpenTag | pp.anyCloseTag).suppress().transformString
+
+# expression for parsing <a href="url">text</a> links, returning a (text, url) tuple
+link = pp.Group(a + a.tag_body('text') + a_end.suppress())
+link.addParseAction(lambda t: (t[0].text, t[0].href))
+
+# method to create table rows of header and data tags
+def table_row(start_tag, end_tag):
+ body = start_tag.tag_body
+ body.addParseAction(pp.tokenMap(str.strip),
+ pp.tokenMap(strip_html))
+ row = pp.Group(tr.suppress()
+ + pp.ZeroOrMore(start_tag.suppress()
+ + body
+ + end_tag.suppress())
+ + tr_end.suppress())
+ return row
+
+th_row = table_row(th, th_end)
+td_row = table_row(td, td_end)
+
+# define expression for overall table - may vary slightly for different pages
+html_table = table + tbody + pp.Optional(th_row('headers')) + pp.ZeroOrMore(td_row)('rows') + tbody_end + table_end
+
+
+# read in a web page containing an interesting HTML table
+with urllib.request.urlopen("https://en.wikipedia.org/wiki/List_of_tz_database_time_zones") as page:
+ page_html = page.read().decode()
+
+tz_table = html_table.searchString(page_html)[0]
+
+# convert rows to dicts
+rows = [dict(zip(tz_table.headers, row)) for row in tz_table.rows]
+
+# make a dict keyed by TZ database name
+tz_db = {row['TZ database name']: row for row in rows}
+
+from pprint import pprint
+pprint(tz_db['America/Chicago'])
--- /dev/null
+#
+# include_preprocessor.py
+#
+# Short pyparsing script to perform #include inclusions similar to the C preprocessor
+#
+# Copyright 2019, Paul McGuire
+#
+import pyparsing as pp
+from pathlib import Path
+
+# parser elements to be used to assemble into #include parser
+SEMI = pp.Suppress(';')
+INCLUDE = pp.Keyword("#include")
+quoted_string = pp.quotedString.addParseAction(pp.removeQuotes)
+file_ref = (quoted_string
+ | pp.Word(pp.printables, excludeChars=';'))
+
+# parser for parsing "#include xyz.dat;" directives
+include_directive = (INCLUDE + file_ref("include_file_name") + SEMI)
+
+# add parse action that will recursively pull in included files - when
+# using transformString, the value returned from the parse action will replace
+# the text matched by the attached expression
+seen = set()
+def read_include_contents(s, l, t):
+ include_file_ref = t.include_file_name
+ include_echo = "/* {} */".format(pp.line(l, s).strip())
+
+ # guard against recursive includes
+ if include_file_ref not in seen:
+ seen.add(include_file_ref)
+ included_file_contents = Path(include_file_ref).read_text()
+ return (include_echo + '\n'
+ + include_directive.transformString(included_file_contents))
+ else:
+ lead = ' '*(pp.col(l, s) - 1)
+ return "/* recursive include! */\n{}{}".format(lead, include_echo)
+
+# attach include processing method as parse action (parse-time callback)
+# to include_directive expression
+include_directive.addParseAction(read_include_contents)
+
+
+if __name__ == '__main__':
+
+ # demo
+
+ # create test files:
+ # - a.txt includes b.txt
+ # - b.txt includes c.txt
+ # - c.txt includes b.txt (must catch infinite recursion)
+ Path('a.txt').write_text("""\
+ /* a.txt */
+ int i;
+
+ /* sometimes included files aren't in quotes */
+ #include b.txt;
+ """)
+
+ Path('b.txt').write_text("""\
+ i = 100;
+ #include 'c.txt';
+ """)
+
+ Path('c.txt').write_text("""\
+ i += 1;
+
+ /* watch out! this might be recursive if this file included by b.txt */
+ #include b.txt;
+ """)
+
+
+ # use include_directive.transformString to perform includes
+
+ # read contents of original file
+ initial_file = Path('a.txt').read_text()
+
+ # print original file
+ print(initial_file)
+ print('-----------------')
+
+ # expand includes in source file (and any included files) and print the result
+ expanded_source = include_directive.transformString(initial_file)
+ print(expanded_source)
+
+ # clean up
+ for fname in "a.txt b.txt c.txt".split():
+ Path(fname).unlink()
null\r
"""\r
\r
-from pyparsing import *\r
+import pyparsing as pp\r
+from pyparsing import pyparsing_common as ppc\r
\r
def make_keyword(kwd_str, kwd_value):\r
- return Keyword(kwd_str).setParseAction(replaceWith(kwd_value))\r
+ return pp.Keyword(kwd_str).setParseAction(pp.replaceWith(kwd_value))\r
TRUE = make_keyword("true", True)\r
FALSE = make_keyword("false", False)\r
NULL = make_keyword("null", None)\r
\r
-LBRACK, RBRACK, LBRACE, RBRACE, COLON = map(Suppress, "[]{}:")\r
+LBRACK, RBRACK, LBRACE, RBRACE, COLON = map(pp.Suppress, "[]{}:")\r
\r
-jsonString = dblQuotedString().setParseAction(removeQuotes)\r
-jsonNumber = pyparsing_common.number()\r
+jsonString = pp.dblQuotedString().setParseAction(pp.removeQuotes)\r
+jsonNumber = ppc.number()\r
\r
-jsonObject = Forward()\r
-jsonValue = Forward()\r
-jsonElements = delimitedList( jsonValue )\r
-jsonArray = Group(LBRACK + Optional(jsonElements, []) + RBRACK)\r
-jsonValue << (jsonString | jsonNumber | Group(jsonObject) | jsonArray | TRUE | FALSE | NULL)\r
-memberDef = Group(jsonString + COLON + jsonValue)\r
-jsonMembers = delimitedList(memberDef)\r
-jsonObject << Dict(LBRACE + Optional(jsonMembers) + RBRACE)\r
+jsonObject = pp.Forward()\r
+jsonValue = pp.Forward()\r
+jsonElements = pp.delimitedList( jsonValue )\r
+jsonArray = pp.Group(LBRACK + pp.Optional(jsonElements, []) + RBRACK)\r
+jsonValue << (jsonString | jsonNumber | pp.Group(jsonObject) | jsonArray | TRUE | FALSE | NULL)\r
+memberDef = pp.Group(jsonString + COLON + jsonValue)\r
+jsonMembers = pp.delimitedList(memberDef)\r
+jsonObject << pp.Dict(LBRACE + pp.Optional(jsonMembers) + RBRACE)\r
\r
-jsonComment = cppStyleComment\r
+jsonComment = pp.cppStyleComment\r
jsonObject.ignore(jsonComment)\r
\r
\r
}\r
"""\r
\r
- import pprint\r
results = jsonObject.parseString(testdata)\r
- pprint.pprint( results.asList() )\r
+ results.pprint()\r
print()\r
def testPrint(x):\r
- print(type(x),repr(x))\r
+ print(type(x), repr(x))\r
print(list(results.glossary.GlossDiv.GlossList.keys()))\r
testPrint( results.glossary.title )\r
testPrint( results.glossary.GlossDiv.GlossList.ID )\r
# at http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/docs/queryparsersyntax.html\r
#\r
\r
-from pyparsing import (Literal, CaselessKeyword, Forward, Regex, QuotedString, Suppress,\r
- Optional, Group, infixNotation, opAssoc, ParserElement,\r
- pyparsing_common)\r
-ParserElement.enablePackrat()\r
+import pyparsing as pp\r
+from pyparsing import pyparsing_common as ppc\r
+pp.ParserElement.enablePackrat()\r
\r
-COLON,LBRACK,RBRACK,LBRACE,RBRACE,TILDE,CARAT = map(Literal,":[]{}~^")\r
-LPAR,RPAR = map(Suppress,"()")\r
-and_, or_, not_, to_ = map(CaselessKeyword, "AND OR NOT TO".split())\r
+COLON,LBRACK,RBRACK,LBRACE,RBRACE,TILDE,CARAT = map(pp.Literal,":[]{}~^")\r
+LPAR,RPAR = map(pp.Suppress,"()")\r
+and_, or_, not_, to_ = map(pp.CaselessKeyword, "AND OR NOT TO".split())\r
keyword = and_ | or_ | not_ | to_\r
\r
-expression = Forward()\r
+expression = pp.Forward()\r
\r
-valid_word = Regex(r'([a-zA-Z0-9*_+.-]|\\\\|\\([+\-!(){}\[\]^"~*?:]|\|\||&&))+').setName("word")\r
+valid_word = pp.Regex(r'([a-zA-Z0-9*_+.-]|\\\\|\\([+\-!(){}\[\]^"~*?:]|\|\||&&))+').setName("word")\r
valid_word.setParseAction(\r
lambda t : t[0].replace('\\\\',chr(127)).replace('\\','').replace(chr(127),'\\')\r
)\r
\r
-string = QuotedString('"')\r
+string = pp.QuotedString('"')\r
\r
-required_modifier = Literal("+")("required")\r
-prohibit_modifier = Literal("-")("prohibit")\r
-integer = Regex(r"\d+").setParseAction(lambda t:int(t[0]))\r
-proximity_modifier = Group(TILDE + integer("proximity"))\r
-number = pyparsing_common.fnumber()\r
-fuzzy_modifier = TILDE + Optional(number, default=0.5)("fuzzy")\r
+required_modifier = pp.Literal("+")("required")\r
+prohibit_modifier = pp.Literal("-")("prohibit")\r
+integer = ppc.integer()\r
+proximity_modifier = pp.Group(TILDE + integer("proximity"))\r
+number = ppc.fnumber()\r
+fuzzy_modifier = TILDE + pp.Optional(number, default=0.5)("fuzzy")\r
\r
-term = Forward()\r
+term = pp.Forward()\r
field_name = valid_word().setName("fieldname")\r
-incl_range_search = Group(LBRACK + term("lower") + to_ + term("upper") + RBRACK)\r
-excl_range_search = Group(LBRACE + term("lower") + to_ + term("upper") + RBRACE)\r
+incl_range_search = pp.Group(LBRACK + term("lower") + to_ + term("upper") + RBRACK)\r
+excl_range_search = pp.Group(LBRACE + term("lower") + to_ + term("upper") + RBRACE)\r
range_search = incl_range_search("incl_range") | excl_range_search("excl_range")\r
boost = (CARAT + number("boost"))\r
\r
-string_expr = Group(string + proximity_modifier) | string\r
-word_expr = Group(valid_word + fuzzy_modifier) | valid_word\r
-term << (Optional(field_name("field") + COLON) +\r
- (word_expr | string_expr | range_search | Group(LPAR + expression + RPAR)) +\r
- Optional(boost))\r
+string_expr = pp.Group(string + proximity_modifier) | string\r
+word_expr = pp.Group(valid_word + fuzzy_modifier) | valid_word\r
+term << (pp.Optional(field_name("field") + COLON)\r
+ + (word_expr | string_expr | range_search | pp.Group(LPAR + expression + RPAR))\r
+ + pp.Optional(boost))\r
term.setParseAction(lambda t:[t] if 'field' in t or 'boost' in t else None)\r
\r
-expression << infixNotation(term,\r
+expression << pp.infixNotation(term,\r
[\r
- (required_modifier | prohibit_modifier, 1, opAssoc.RIGHT),\r
- ((not_ | '!').setParseAction(lambda: "NOT"), 1, opAssoc.RIGHT),\r
- ((and_ | '&&').setParseAction(lambda: "AND"), 2, opAssoc.LEFT),\r
- (Optional(or_ | '||').setParseAction(lambda: "OR"), 2, opAssoc.LEFT),\r
+ (required_modifier | prohibit_modifier, 1, pp.opAssoc.RIGHT),\r
+ ((not_ | '!').setParseAction(lambda: "NOT"), 1, pp.opAssoc.RIGHT),\r
+ ((and_ | '&&').setParseAction(lambda: "AND"), 2, pp.opAssoc.LEFT),\r
+ (pp.Optional(or_ | '||').setParseAction(lambda: "OR"), 2, pp.opAssoc.LEFT),\r
])\r
\r
# test strings taken from grammar description doc, and TestQueryParser.java\r
+++ /dev/null
-import urllib.request, urllib.parse, urllib.error\r
-\r
-from pyparsing import makeHTMLTags, SkipTo\r
-\r
-# read HTML from a web page\r
-serverListPage = urllib.request.urlopen( "https://www.yahoo.com/" )\r
-htmlText = serverListPage.read()\r
-serverListPage.close()\r
-\r
-# using makeHTMLTags to define opening and closing tags\r
-anchorStart,anchorEnd = makeHTMLTags("a")\r
-\r
-# compose an expression for an anchored reference\r
-anchor = anchorStart + SkipTo(anchorEnd)("body") + anchorEnd\r
-\r
-# use scanString to scan through the HTML source, extracting\r
-# just the anchor tags and their associated body text\r
-# (note the href attribute of the opening A tag is available\r
-# as an attribute in the returned parse results)\r
-for tokens,start,end in anchor.scanString(htmlText):\r
- print(tokens.body,'->',tokens.href)\r
-# parsePythonValue.py\r
-#\r
-# Copyright, 2006, by Paul McGuire\r
-#\r
-from __future__ import print_function\r
-from pyparsing import *\r
-\r
-\r
-cvtBool = lambda t:t[0]=='True'\r
-cvtInt = lambda toks: int(toks[0])\r
-cvtReal = lambda toks: float(toks[0])\r
-cvtTuple = lambda toks : tuple(toks.asList())\r
-cvtDict = lambda toks: dict(toks.asList())\r
-cvtList = lambda toks: [toks.asList()]\r
-\r
-# define punctuation as suppressed literals\r
-lparen,rparen,lbrack,rbrack,lbrace,rbrace,colon = \\r
- map(Suppress,"()[]{}:")\r
-\r
-integer = Regex(r"[+-]?\d+")\\r
- .setName("integer")\\r
- .setParseAction( cvtInt )\r
-real = Regex(r"[+-]?\d+\.\d*([Ee][+-]?\d+)?")\\r
- .setName("real")\\r
- .setParseAction( cvtReal )\r
-tupleStr = Forward()\r
-listStr = Forward()\r
-dictStr = Forward()\r
-\r
-unicodeString.setParseAction(lambda t:t[0][2:-1].decode('unicode-escape'))\r
-quotedString.setParseAction(lambda t:t[0][1:-1].decode('string-escape'))\r
-boolLiteral = oneOf("True False").setParseAction(cvtBool)\r
-noneLiteral = Literal("None").setParseAction(replaceWith(None))\r
-\r
-listItem = real|integer|quotedString|unicodeString|boolLiteral|noneLiteral| \\r
- Group(listStr) | tupleStr | dictStr\r
-\r
-tupleStr << ( Suppress("(") + Optional(delimitedList(listItem)) +\r
- Optional(Suppress(",")) + Suppress(")") )\r
-tupleStr.setParseAction( cvtTuple )\r
-\r
-listStr << (lbrack + Optional(delimitedList(listItem) +\r
- Optional(Suppress(","))) + rbrack)\r
-listStr.setParseAction(cvtList, lambda t: t[0])\r
-\r
-dictEntry = Group( listItem + colon + listItem )\r
-dictStr << (lbrace + Optional(delimitedList(dictEntry) + \\r
- Optional(Suppress(","))) + rbrace)\r
-dictStr.setParseAction( cvtDict )\r
-\r
-tests = """['a', 100, ('A', [101,102]), 3.14, [ +2.718, 'xyzzy', -1.414] ]\r
- [{0: [2], 1: []}, {0: [], 1: [], 2: []}, {0: [1, 2]}]\r
- { 'A':1, 'B':2, 'C': {'a': 1.2, 'b': 3.4} }\r
- 3.14159\r
- 42\r
- 6.02E23\r
- 6.02e+023\r
- 1.0e-7\r
- 'a quoted string'""".split("\n")\r
-\r
-for test in tests:\r
- print("Test:", test.strip())\r
- result = listItem.parseString(test)[0]\r
- print("Result:", result)\r
- try:\r
- for dd in result:\r
- if isinstance(dd,dict): print(list(dd.items()))\r
- except TypeError as te:\r
- pass\r
- print()\r
+# parsePythonValue.py
+#
+# Copyright, 2006, by Paul McGuire
+#
+from __future__ import print_function
+import pyparsing as pp
+
+
+cvtBool = lambda t:t[0]=='True'
+cvtInt = lambda toks: int(toks[0])
+cvtReal = lambda toks: float(toks[0])
+cvtTuple = lambda toks : tuple(toks.asList())
+cvtDict = lambda toks: dict(toks.asList())
+cvtList = lambda toks: [toks.asList()]
+
+# define punctuation as suppressed literals
+lparen, rparen, lbrack, rbrack, lbrace, rbrace, colon, comma = map(pp.Suppress,"()[]{}:,")
+
+integer = pp.Regex(r"[+-]?\d+").setName("integer").setParseAction(cvtInt )
+real = pp.Regex(r"[+-]?\d+\.\d*([Ee][+-]?\d+)?").setName("real").setParseAction(cvtReal)
+tupleStr = pp.Forward()
+listStr = pp.Forward()
+dictStr = pp.Forward()
+
+pp.unicodeString.setParseAction(lambda t:t[0][2:-1])
+pp.quotedString.setParseAction(lambda t:t[0][1:-1])
+boolLiteral = pp.oneOf("True False").setParseAction(cvtBool)
+noneLiteral = pp.Literal("None").setParseAction(pp.replaceWith(None))
+
+listItem = (real
+ | integer
+ | pp.quotedString
+ | pp.unicodeString
+ | boolLiteral
+ | noneLiteral
+ | pp.Group(listStr)
+ | tupleStr
+ | dictStr)
+
+tupleStr << (lparen
+ + pp.Optional(pp.delimitedList(listItem))
+ + pp.Optional(comma)
+ + rparen)
+tupleStr.setParseAction(cvtTuple)
+
+listStr << (lbrack
+ + pp.Optional(pp.delimitedList(listItem) + pp.Optional(comma))
+ + rbrack)
+listStr.setParseAction(cvtList, lambda t: t[0])
+
+dictEntry = pp.Group(listItem + colon + listItem)
+dictStr << (lbrace
+ + pp.Optional(pp.delimitedList(dictEntry) + pp.Optional(comma))
+ + rbrace)
+dictStr.setParseAction(cvtDict)
+
+tests = """['a', 100, ('A', [101,102]), 3.14, [ +2.718, 'xyzzy', -1.414] ]
+ [{0: [2], 1: []}, {0: [], 1: [], 2: []}, {0: [1, 2]}]
+ { 'A':1, 'B':2, 'C': {'a': 1.2, 'b': 3.4} }
+ 3.14159
+ 42
+ 6.02E23
+ 6.02e+023
+ 1.0e-7
+ 'a quoted string'"""
+
+listItem.runTests(tests)
--- /dev/null
+#
+# rosettacode.py
+#
+# parser for language used by rosettacode.org (http://rosettacode.org/wiki/Compiler/syntax_analyzer)
+#
+# Copyright Paul McGuire, 2019
+#
+BNF = """
+ stmt_list = {stmt} ;
+
+ stmt = ';'
+ | Identifier '=' expr ';'
+ | 'while' paren_expr stmt
+ | 'if' paren_expr stmt ['else' stmt]
+ | 'print' '(' prt_list ')' ';'
+ | 'putc' paren_expr ';'
+ | '{' stmt_list '}'
+ ;
+
+ paren_expr = '(' expr ')' ;
+
+ prt_list = string | expr {',' String | expr} ;
+
+ expr = and_expr {'||' and_expr} ;
+ and_expr = equality_expr {'&&' equality_expr} ;
+ equality_expr = relational_expr [('==' | '!=') relational_expr] ;
+ relational_expr = addition_expr [('<' | '<=' | '>' | '>=') addition_expr] ;
+ addition_expr = multiplication_expr {('+' | '-') multiplication_expr} ;
+ multiplication_expr = primary {('*' | '/' | '%') primary } ;
+ primary = Identifier
+ | Integer
+ | '(' expr ')'
+ | ('+' | '-' | '!') primary
+ ;
+"""
+
+import pyparsing as pp
+pp.ParserElement.enablePackrat()
+
+LBRACE, RBRACE, LPAR, RPAR, SEMI = map(pp.Suppress, "{}();")
+EQ = pp.Literal('=')
+
+keywords = (WHILE, IF, PRINT, PUTC, ELSE) = map(pp.Keyword, "while if print putc else".split())
+identifier = ~(pp.MatchFirst(keywords)) + pp.pyparsing_common.identifier
+integer = pp.pyparsing_common.integer
+string = pp.QuotedString('"', convertWhitespaceEscapes=False).setName("quoted string")
+char = pp.Regex(r"'\\?.'")
+
+expr = pp.infixNotation(identifier | integer | char,
+ [
+ (pp.oneOf("+ - !"), 1, pp.opAssoc.RIGHT,),
+ (pp.oneOf("* / %"), 2, pp.opAssoc.LEFT, ),
+ (pp.oneOf("+ -"), 2, pp.opAssoc.LEFT,),
+ (pp.oneOf("< <= > >="), 2, pp.opAssoc.LEFT,),
+ (pp.oneOf("== !="), 2, pp.opAssoc.LEFT,),
+ (pp.oneOf("&&"), 2, pp.opAssoc.LEFT,),
+ (pp.oneOf("||"), 2, pp.opAssoc.LEFT,),
+ ])
+
+prt_list = pp.Group(pp.delimitedList(string | expr))
+paren_expr = pp.Group(LPAR + expr + RPAR)
+
+stmt = pp.Forward()
+assignment_stmt = pp.Group(identifier + EQ + expr + SEMI)
+while_stmt = pp.Group(WHILE - paren_expr + stmt)
+if_stmt = pp.Group(IF - paren_expr + stmt + pp.Optional(ELSE + stmt))
+print_stmt = pp.Group(PRINT - pp.Group(LPAR + prt_list + RPAR) + SEMI)
+putc_stmt = pp.Group(PUTC - paren_expr + SEMI)
+stmt_list = pp.Group(LBRACE + pp.ZeroOrMore(stmt) + RBRACE)
+stmt <<= (pp.Group(SEMI)
+ | assignment_stmt
+ | while_stmt
+ | if_stmt
+ | print_stmt
+ | putc_stmt
+ | stmt_list
+ ).setName("statement")
+
+code = pp.ZeroOrMore(stmt)
+code.ignore(pp.cppStyleComment)
+
+
+tests = [
+ r'''
+ count = 1;
+ while (count < 10) {
+ print("count is: ", count, "\n");
+ count = count + 1;
+ }
+ ''',
+ r'''
+ /*
+ Simple prime number generator
+ */
+ count = 1;
+ n = 1;
+ limit = 100;
+ while (n < limit) {
+ k=3;
+ p=1;
+ n=n+2;
+ while ((k*k<=n) && (p)) {
+ p=n/k*k!=n;
+ k=k+2;
+ }
+ if (p) {
+ print(n, " is prime\n");
+ count = count + 1;
+ }
+ }
+ print("Total primes found: ", count, "\n");
+ ''',
+ r'''
+ /*
+ Hello world
+ */
+ print("Hello, World!\n");
+ ''',
+ r'''
+ /*
+ Show Ident and Integers
+ */
+ phoenix_number = 142857;
+ print(phoenix_number, "\n");
+ ''',
+ r'''
+ /*** test printing, embedded \n and comments with lots of '*' ***/
+ print(42);
+ print("\nHello World\nGood Bye\nok\n");
+ print("Print a slash n - \\n.\n");
+ ''',
+ r'''
+ /* 100 Doors */
+ i = 1;
+ while (i * i <= 100) {
+ print("door ", i * i, " is open\n");
+ i = i + 1;
+ }
+ ''',
+ r'''
+ a = (-1 * ((-1 * (5 * 15)) / 10));
+ print(a, "\n");
+ b = -a;
+ print(b, "\n");
+ print(-b, "\n");
+ print(-(1), "\n");
+ ''',
+ r'''
+ print(---------------------------------+++5, "\n");
+ print(((((((((3 + 2) * ((((((2))))))))))))), "\n");
+
+ if (1) { if (1) { if (1) { if (1) { if (1) { print(15, "\n"); } } } } }
+ ''',
+ r'''
+ /* Compute the gcd of 1071, 1029: 21 */
+
+ a = 1071;
+ b = 1029;
+
+ while (b != 0) {
+ new_a = b;
+ b = a % b;
+ a = new_a;
+ }
+ print(a);
+ ''',
+ r'''
+ /* 12 factorial is 479001600 */
+
+ n = 12;
+ result = 1;
+ i = 1;
+ while (i <= n) {
+ result = result * i;
+ i = i + 1;
+ }
+ print(result);
+ ''',
+ r'''
+ /* fibonacci of 44 is 701408733 */
+
+ n = 44;
+ i = 1;
+ a = 0;
+ b = 1;
+ while (i < n) {
+ w = a + b;
+ a = b;
+ b = w;
+ i = i + 1;
+ }
+ print(w, "\n");
+ ''',
+ r'''
+ /* FizzBuzz */
+ i = 1;
+ while (i <= 100) {
+ if (!(i % 15))
+ print("FizzBuzz");
+ else if (!(i % 3))
+ print("Fizz");
+ else if (!(i % 5))
+ print("Buzz");
+ else
+ print(i);
+
+ print("\n");
+ i = i + 1;
+ }
+ ''',
+ r'''
+ /* 99 bottles */
+ bottles = 99;
+ while (bottles > 0) {
+ print(bottles, " bottles of beer on the wall\n");
+ print(bottles, " bottles of beer\n");
+ print("Take one down, pass it around\n");
+ bottles = bottles - 1;
+ print(bottles, " bottles of beer on the wall\n\n");
+ }
+ ''',
+ r'''
+ {
+ /*
+ This is an integer ascii Mandelbrot generator
+ */
+ left_edge = -420;
+ right_edge = 300;
+ top_edge = 300;
+ bottom_edge = -300;
+ x_step = 7;
+ y_step = 15;
+
+ max_iter = 200;
+
+ y0 = top_edge;
+ while (y0 > bottom_edge) {
+ x0 = left_edge;
+ while (x0 < right_edge) {
+ y = 0;
+ x = 0;
+ the_char = ' ';
+ i = 0;
+ while (i < max_iter) {
+ x_x = (x * x) / 200;
+ y_y = (y * y) / 200;
+ if (x_x + y_y > 800 ) {
+ the_char = '0' + i;
+ if (i > 9) {
+ the_char = '@';
+ }
+ i = max_iter;
+ }
+ y = x * y / 100 + y0;
+ x = x_x - y_y + x0;
+ i = i + 1;
+ }
+ putc(the_char);
+ x0 = x0 + x_step;
+ }
+ putc('\n');
+ y0 = y0 - y_step;
+ }
+ }
+ ''',
+]
+
+import sys
+sys.setrecursionlimit(2000)
+
+for test in tests:
+ try:
+ results = code.parseString(test)
+ except pp.ParseException as pe:
+ pp.ParseException.explain(pe)
+ else:
+ results.pprint()
+ print()
+++ /dev/null
-from pyparsing import makeHTMLTags,SkipTo,htmlComment\r
-import urllib.request, urllib.parse, urllib.error\r
-\r
-serverListPage = urllib.request.urlopen( "https://www.yahoo.com/" )\r
-htmlText = serverListPage.read()\r
-serverListPage.close()\r
-\r
-aStart,aEnd = makeHTMLTags("A")\r
-\r
-link = aStart + SkipTo(aEnd).setResultsName("link") + aEnd\r
-link.ignore(htmlComment)\r
-\r
-for toks,start,end in link.scanString(htmlText):\r
- print(toks.link, "->", toks.startA.href)
\ No newline at end of file
-# simpleSQL.py\r
-#\r
-# simple demo of using the parsing library to do simple-minded SQL parsing\r
-# could be extended to include where clauses etc.\r
-#\r
-# Copyright (c) 2003,2016, Paul McGuire\r
-#\r
-from pyparsing import Word, delimitedList, Optional, \\r
- Group, alphas, alphanums, Forward, oneOf, quotedString, \\r
- ZeroOrMore, restOfLine, CaselessKeyword, pyparsing_common\r
-\r
-# define SQL tokens\r
-selectStmt = Forward()\r
-SELECT, FROM, WHERE = map(CaselessKeyword, "select from where".split())\r
-\r
-ident = Word( alphas, alphanums + "_$" ).setName("identifier")\r
-columnName = delimitedList(ident, ".", combine=True).setName("column name")\r
-columnName.addParseAction(pyparsing_common.upcaseTokens)\r
-columnNameList = Group( delimitedList(columnName))\r
-tableName = delimitedList(ident, ".", combine=True).setName("table name")\r
-tableName.addParseAction(pyparsing_common.upcaseTokens)\r
-tableNameList = Group(delimitedList(tableName))\r
-\r
-whereExpression = Forward()\r
-and_, or_, in_ = map(CaselessKeyword, "and or in".split())\r
-\r
-binop = oneOf("= != < > >= <= eq ne lt le gt ge", caseless=True)\r
-realNum = pyparsing_common.real()\r
-intNum = pyparsing_common.signed_integer()\r
-\r
-columnRval = realNum | intNum | quotedString | columnName # need to add support for alg expressions\r
-whereCondition = Group(\r
- ( columnName + binop + columnRval ) |\r
- ( columnName + in_ + "(" + delimitedList( columnRval ) + ")" ) |\r
- ( columnName + in_ + "(" + selectStmt + ")" ) |\r
- ( "(" + whereExpression + ")" )\r
- )\r
-whereExpression << whereCondition + ZeroOrMore( ( and_ | or_ ) + whereExpression )\r
-\r
-# define the grammar\r
-selectStmt <<= (SELECT + ('*' | columnNameList)("columns") +\r
- FROM + tableNameList( "tables" ) +\r
- Optional(Group(WHERE + whereExpression), "")("where"))\r
-\r
-simpleSQL = selectStmt\r
-\r
-# define Oracle comment format, and ignore them\r
-oracleSqlComment = "--" + restOfLine\r
-simpleSQL.ignore( oracleSqlComment )\r
-\r
-if __name__ == "__main__":\r
- simpleSQL.runTests("""\\r
-\r
- # multiple tables\r
- SELECT * from XYZZY, ABC\r
-\r
- # dotted table name\r
- select * from SYS.XYZZY\r
-\r
- Select A from Sys.dual\r
-\r
- Select A,B,C from Sys.dual\r
-\r
- Select A, B, C from Sys.dual, Table2\r
-\r
- # FAIL - invalid SELECT keyword\r
- Xelect A, B, C from Sys.dual\r
-\r
- # FAIL - invalid FROM keyword\r
- Select A, B, C frox Sys.dual\r
-\r
- # FAIL - incomplete statement\r
- Select\r
-\r
- # FAIL - incomplete statement\r
- Select * from\r
-\r
- # FAIL - invalid column\r
- Select &&& frox Sys.dual\r
-\r
- # where clause\r
- Select A from Sys.dual where a in ('RED','GREEN','BLUE')\r
-\r
- # compound where clause\r
- Select A from Sys.dual where a in ('RED','GREEN','BLUE') and b in (10,20,30)\r
-\r
- # where clause with comparison operator\r
- Select A,b from table1,table2 where table1.id eq table2.id""")\r
+# simpleSQL.py
+#
+# simple demo of using the parsing library to do simple-minded SQL parsing
+# could be extended to include where clauses etc.
+#
+# Copyright (c) 2003,2016, Paul McGuire
+#
+from pyparsing import Word, delimitedList, Optional, \
+ Group, alphas, alphanums, Forward, oneOf, quotedString, \
+ infixNotation, opAssoc, \
+ ZeroOrMore, restOfLine, CaselessKeyword, pyparsing_common as ppc
+
+# define SQL tokens
+selectStmt = Forward()
+SELECT, FROM, WHERE, AND, OR, IN, IS, NOT, NULL = map(CaselessKeyword,
+ "select from where and or in is not null".split())
+NOT_NULL = NOT + NULL
+
+ident = Word( alphas, alphanums + "_$" ).setName("identifier")
+columnName = delimitedList(ident, ".", combine=True).setName("column name")
+columnName.addParseAction(ppc.upcaseTokens)
+columnNameList = Group( delimitedList(columnName))
+tableName = delimitedList(ident, ".", combine=True).setName("table name")
+tableName.addParseAction(ppc.upcaseTokens)
+tableNameList = Group(delimitedList(tableName))
+
+binop = oneOf("= != < > >= <= eq ne lt le gt ge", caseless=True)
+realNum = ppc.real()
+intNum = ppc.signed_integer()
+
+columnRval = realNum | intNum | quotedString | columnName # need to add support for alg expressions
+whereCondition = Group(
+ ( columnName + binop + columnRval ) |
+ ( columnName + IN + Group("(" + delimitedList( columnRval ) + ")" )) |
+ ( columnName + IN + Group("(" + selectStmt + ")" )) |
+ ( columnName + IS + (NULL | NOT_NULL))
+ )
+
+whereExpression = infixNotation(whereCondition,
+ [
+ (NOT, 1, opAssoc.RIGHT),
+ (AND, 2, opAssoc.LEFT),
+ (OR, 2, opAssoc.LEFT),
+ ])
+
+# define the grammar
+selectStmt <<= (SELECT + ('*' | columnNameList)("columns") +
+ FROM + tableNameList( "tables" ) +
+ Optional(Group(WHERE + whereExpression), "")("where"))
+
+simpleSQL = selectStmt
+
+# define Oracle comment format, and ignore them
+oracleSqlComment = "--" + restOfLine
+simpleSQL.ignore( oracleSqlComment )
+
+if __name__ == "__main__":
+ simpleSQL.runTests("""\
+
+ # multiple tables
+ SELECT * from XYZZY, ABC
+
+ # dotted table name
+ select * from SYS.XYZZY
+
+ Select A from Sys.dual
+
+ Select A,B,C from Sys.dual
+
+ Select A, B, C from Sys.dual, Table2
+
+ # FAIL - invalid SELECT keyword
+ Xelect A, B, C from Sys.dual
+
+ # FAIL - invalid FROM keyword
+ Select A, B, C frox Sys.dual
+
+ # FAIL - incomplete statement
+ Select
+
+ # FAIL - incomplete statement
+ Select * from
+
+ # FAIL - invalid column
+ Select &&& frox Sys.dual
+
+ # where clause
+ Select A from Sys.dual where a in ('RED','GREEN','BLUE')
+
+ # compound where clause
+ Select A from Sys.dual where a in ('RED','GREEN','BLUE') and b in (10,20,30)
+
+ # where clause with comparison operator
+ Select A,b from table1,table2 where table1.id eq table2.id
+ """)
--- /dev/null
+#ifndef SNMP_API_H\r
+#define SNMP_API_H\r
+\r
+/*\r
+ * snmp_api.h - API for access to snmp.\r
+ *\r
+ * Caution: when using this library in a multi-threaded application,\r
+ * the values of global variables "snmp_errno" and "snmp_detail"\r
+ * cannot be reliably determined. Suggest using snmp_error()\r
+ * to obtain the library error codes.\r
+ */\r
+\r
+#ifndef DONT_SHARE_ERROR_WITH_OTHER_THREADS\r
+#define SET_SNMP_ERROR(x) snmp_errno=(x)\r
+#else\r
+#define SET_SNMP_ERROR(x)\r
+#endif\r
+\r
+\r
+#ifdef __cplusplus\r
+extern "C" {\r
+#endif\r
+\r
+/***********************************************************\r
+ Copyright 1989 by Carnegie Mellon University\r
+\r
+ All Rights Reserved\r
+\r
+Permission to use, copy, modify, and distribute this software and its\r
+documentation for any purpose and without fee is hereby granted,\r
+provided that the above copyright notice appear in all copies and that\r
+both that copyright notice and this permission notice appear in\r
+supporting documentation, and that the name of CMU not be\r
+used in advertising or publicity pertaining to distribution of the\r
+software without specific, written prior permission.\r
+\r
+CMU DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING\r
+ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL\r
+CMU BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR\r
+ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,\r
+WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,\r
+ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS\r
+SOFTWARE.\r
+******************************************************************/\r
+\r
+\r
+struct variable_list;\r
+struct timeval;\r
+\r
+\r
+ /*\r
+ * Mimic size and alignment of 'struct sockaddr_storage' (see RFC 2553)\r
+ * But retain field names of traditional 'struct sockaddr'\r
+ */\r
+\r
+#define _UCD_SS_MAXSIZE 92 /* <= sizeof( sockaddr_un ) */\r
+#define _UCD_SS_ALIGNSIZE (sizeof (long))\r
+\r
+#define _UCD_SS_PAD1SIZE (_UCD_SS_ALIGNSIZE - sizeof( unsigned short ))\r
+#define _UCD_SS_PAD2SIZE (_UCD_SS_MAXSIZE - \\r
+ (sizeof( unsigned short ) + _UCD_SS_PAD1SIZE + _UCD_SS_ALIGNSIZE ))\r
+\r
+typedef struct {\r
+\r
+#ifdef STRUCT_SOCKADDR_HAS_SA_UNION_SA_GENERIC_SA_FAMILY2\r
+ /*\r
+ * Certain systems (notably Irix 6.x) have a non-traditional\r
+ * socket structure, and #define the traditional field names.\r
+ * This local definition should reproduce this structure, and still\r
+ * be large enough to handle any necessary Unix domain addresses.\r
+ */\r
+ union {\r
+ struct {\r
+#ifdef _HAVE_SA_LEN\r
+ unsigned char sa_len2;\r
+ unsigned char sa_family2;\r
+#else\r
+ unsigned short sa_family2;\r
+#endif\r
+ char sa_data2[ _UCD_SS_PAD1SIZE ];\r
+ } sa_generic;\r
+ long sa_align;\r
+ char sa_pad2[ _UCD_SS_PAD2SIZE ];\r
+ } sa_union;\r
+\r
+#else\r
+\r
+#ifdef STRUCT_SOCKADDR_HAS_SA_LEN\r
+ unsigned char sa_len;\r
+ unsigned char sa_family;\r
+#else\r
+ unsigned short sa_family;\r
+#endif\r
+ char sa_data[ _UCD_SS_PAD1SIZE ];\r
+ long sa_align;\r
+ char sa_pad2[ _UCD_SS_PAD2SIZE ];\r
+#endif\r
+\r
+} snmp_ipaddr;\r
+\r
+#define USM_AUTH_KU_LEN 32\r
+#define USM_PRIV_KU_LEN 32\r
+\r
+struct snmp_pdu {\r
+\r
+ /*\r
+ * Protocol-version independent fields\r
+ */\r
+ long version;\r
+ int command; /* Type of this PDU */\r
+ long reqid; /* Request id - note: not incremented on retries */\r
+ long msgid; /* Message id for V3 messages\r
+ * note: incremented for each retry */\r
+ long transid; /* Unique ID for incoming transactions */\r
+ long sessid; /* Session id for AgentX messages */\r
+ long errstat; /* Error status (non_repeaters in GetBulk) */\r
+ long errindex; /* Error index (max_repetitions in GetBulk) */\r
+ u_long time; /* Uptime */\r
+ u_long flags;\r
+\r
+ int securityModel;\r
+ int securityLevel; /* noAuthNoPriv, authNoPriv, authPriv */\r
+ int msgParseModel;\r
+\r
+ snmp_ipaddr address; /* Address of peer or trap destination */\r
+\r
+ struct variable_list *variables;\r
+\r
+\r
+ /*\r
+ * SNMPv1 & SNMPv2c fields\r
+ */\r
+ u_char *community; /* community for outgoing requests. */\r
+ size_t community_len; /* Length of community name. */\r
+\r
+ /*\r
+ * Trap information\r
+ */\r
+ oid *enterprise; /* System OID */\r
+ size_t enterprise_length;\r
+ long trap_type; /* trap type */\r
+ long specific_type; /* specific type */\r
+ snmp_ipaddr agent_addr;\r
+\r
+ /*\r
+ * SNMPv3 fields\r
+ */\r
+ u_char *contextEngineID; /* context snmpEngineID */\r
+ size_t contextEngineIDLen; /* Length of contextEngineID */\r
+ char *contextName; /* authoritative contextName */\r
+ size_t contextNameLen; /* Length of contextName */\r
+ u_char *securityEngineID; /* authoritative snmpEngineID for security */\r
+ size_t securityEngineIDLen;/* Length of securityEngineID */\r
+ char *securityName; /* on behalf of this principal */\r
+ size_t securityNameLen; /* Length of securityName. */\r
+\r
+ /*\r
+ * AgentX fields\r
+ * (also uses SNMPv1 community field)\r
+ */\r
+ int priority;\r
+ int range_subid;\r
+\r
+ void * securityStateRef;\r
+};\r
+\r
+struct snmp_session;\r
+typedef int (*snmp_callback) (int, struct snmp_session *, int, struct snmp_pdu *, void *);\r
+\r
+struct snmp_session {\r
+ /*\r
+ * Protocol-version independent fields\r
+ */\r
+ long version;\r
+ int retries; /* Number of retries before timeout. */\r
+ long timeout; /* Number of uS until first timeout, then exponential backoff */\r
+ u_long flags;\r
+ struct snmp_session *subsession;\r
+ struct snmp_session *next;\r
+\r
+ char *peername; /* Domain name or dotted IP address of default peer */\r
+ u_short remote_port;/* UDP port number of peer. */\r
+ u_short local_port; /* My UDP port number, 0 for default, picked randomly */\r
+ /* Authentication function or NULL if null authentication is used */\r
+ u_char *(*authenticator) (u_char *, size_t *, u_char *, size_t);\r
+ snmp_callback callback; /* Function to interpret incoming data */\r
+ /* Pointer to data that the callback function may consider important */\r
+ void *callback_magic;\r
+\r
+ int s_errno; /* copy of system errno */\r
+ int s_snmp_errno; /* copy of library errno */\r
+ long sessid; /* Session id - AgentX only */\r
+\r
+ /*\r
+ * SNMPv1 & SNMPv2c fields\r
+ */\r
+ u_char *community; /* community for outgoing requests. */\r
+ size_t community_len; /* Length of community name. */\r
+\r
+ /*\r
+ * SNMPv3 fields\r
+ */\r
+ u_char isAuthoritative; /* are we the authoritative engine? */\r
+ u_char *contextEngineID; /* authoritative snmpEngineID */\r
+ size_t contextEngineIDLen; /* Length of contextEngineID */\r
+ u_int engineBoots; /* initial engineBoots for remote engine */\r
+ u_int engineTime; /* initial engineTime for remote engine */\r
+ char *contextName; /* authoritative contextName */\r
+ size_t contextNameLen; /* Length of contextName */\r
+ u_char *securityEngineID; /* authoritative snmpEngineID */\r
+ size_t securityEngineIDLen; /* Length of contextEngineID */\r
+ char *securityName; /* on behalf of this principal */\r
+ size_t securityNameLen; /* Length of securityName. */\r
+ oid *securityAuthProto; /* auth protocol oid */\r
+ size_t securityAuthProtoLen; /* Length of auth protocol oid */\r
+ u_char securityAuthKey[USM_AUTH_KU_LEN]; /* Ku for auth protocol XXX */\r
+ size_t securityAuthKeyLen; /* Length of Ku for auth protocol */\r
+ oid *securityPrivProto; /* priv protocol oid */\r
+ size_t securityPrivProtoLen; /* Length of priv protocol oid */\r
+ u_char securityPrivKey[USM_PRIV_KU_LEN]; /* Ku for privacy protocol XXX */\r
+ size_t securityPrivKeyLen; /* Length of Ku for priv protocol */\r
+ int securityModel;\r
+ int securityLevel; /* noAuthNoPriv, authNoPriv, authPriv */\r
+};\r
+\r
+/*\r
+ * A list of all the outstanding requests for a particular session.\r
+ */\r
+#ifdef SNMP_NEED_REQUEST_LIST\r
+struct request_list {\r
+ struct request_list *next_request;\r
+ long request_id; /* request id */\r
+ long message_id; /* message id */\r
+ snmp_callback callback; /* user callback per request (NULL if unused) */\r
+ void *cb_data; /* user callback data per request (NULL if unused) */\r
+ int retries; /* Number of retries */\r
+ u_long timeout; /* length to wait for timeout */\r
+ struct timeval time; /* Time this request was made */\r
+ struct timeval expire; /* time this request is due to expire */\r
+ struct snmp_session *session;\r
+ struct snmp_pdu *pdu; /* The pdu for this request\r
+ (saved so it can be retransmitted */\r
+};\r
+#endif /* SNMP_NEED_REQUEST_LIST */\r
+\r
+/*\r
+ * Set fields in session and pdu to the following to get a default or unconfigured value.\r
+ */\r
+#define SNMP_DEFAULT_COMMUNITY_LEN 0 /* to get a default community name */\r
+#define SNMP_DEFAULT_RETRIES -1\r
+#define SNMP_DEFAULT_TIMEOUT -1\r
+#define SNMP_DEFAULT_REMPORT 0\r
+#define SNMP_DEFAULT_REQID -1\r
+#define SNMP_DEFAULT_MSGID -1\r
+#define SNMP_DEFAULT_ERRSTAT -1\r
+#define SNMP_DEFAULT_ERRINDEX -1\r
+#define SNMP_DEFAULT_ADDRESS 0\r
+#define SNMP_DEFAULT_PEERNAME NULL\r
+#define SNMP_DEFAULT_ENTERPRISE_LENGTH 0\r
+#define SNMP_DEFAULT_TIME 0\r
+#define SNMP_DEFAULT_VERSION -1\r
+#define SNMP_DEFAULT_CONTEXT ""\r
+#define SNMP_DEFAULT_AUTH_PROTO usmHMACMD5AuthProtocol\r
+#define SNMP_DEFAULT_AUTH_PROTOLEN USM_LENGTH_OID_TRANSFORM\r
+#define SNMP_DEFAULT_PRIV_PROTO usmDESPrivProtocol\r
+#define SNMP_DEFAULT_PRIV_PROTOLEN USM_LENGTH_OID_TRANSFORM\r
+\r
+extern const char *snmp_api_errstring (int);\r
+extern void snmp_perror (const char *);\r
+extern void snmp_set_detail (const char *);\r
+\r
+#define SNMP_MAX_MSG_SIZE 1472 /* ethernet MTU minus IP/UDP header */\r
+#define SNMP_MAX_MSG_V3_HDRS (4+3+4+7+7+3+7+16) /* fudge factor=16 */\r
+#define SNMP_MAX_ENG_SIZE 32\r
+#define SNMP_MAX_SEC_NAME_SIZE 256\r
+#define SNMP_MAX_CONTEXT_SIZE 256\r
+#define SNMP_SEC_PARAM_BUF_SIZE 256\r
+\r
+/* set to one to ignore unauthenticated Reports */\r
+#define SNMPV3_IGNORE_UNAUTH_REPORTS 0\r
+\r
+/* authoritative engine definitions */\r
+#define SNMP_SESS_NONAUTHORITATIVE 0 /* should be 0 to default to this */\r
+#define SNMP_SESS_AUTHORITATIVE 1 /* don't learn engineIDs */\r
+#define SNMP_SESS_UNKNOWNAUTH 2 /* sometimes (like NRs) */\r
+\r
+/* to determine type of Report from varbind_list */\r
+#define REPORT_STATS_LEN 9\r
+#define REPORT_snmpUnknownSecurityModels_NUM 1\r
+#define REPORT_snmpInvalidMsgs_NUM 2\r
+#define REPORT_usmStatsUnsupportedSecLevels_NUM 1\r
+#define REPORT_usmStatsNotInTimeWindows_NUM 2\r
+#define REPORT_usmStatsUnknownUserNames_NUM 3\r
+#define REPORT_usmStatsUnknownEngineIDs_NUM 4\r
+#define REPORT_usmStatsWrongDigests_NUM 5\r
+#define REPORT_usmStatsDecryptionErrors_NUM 6\r
+\r
+#define SNMP_DETAIL_SIZE 512\r
+\r
+#define SNMP_FLAGS_DONT_PROBE 0x100 /* don't probe for an engineID */\r
+#define SNMP_FLAGS_STREAM_SOCKET 0x80\r
+#define SNMP_FLAGS_LISTENING 0x40 /* Server stream sockets only */\r
+#define SNMP_FLAGS_SUBSESSION 0x20\r
+#define SNMP_FLAGS_STRIKE2 0x02\r
+#define SNMP_FLAGS_STRIKE1 0x01\r
+\r
+#define CLEAR_SNMP_STRIKE_FLAGS(x) \\r
+ x &= ~(SNMP_FLAGS_STRIKE2|SNMP_FLAGS_STRIKE1)\r
+\r
+ /*\r
+ * returns '1' if the session is to be regarded as dead,\r
+ * otherwise set the strike flags appropriately, and return 0\r
+ */\r
+#define SET_SNMP_STRIKE_FLAGS(x) \\r
+ (( x & SNMP_FLAGS_STRIKE2 ) ? 1 : \\r
+ ((( x & SNMP_FLAGS_STRIKE1 ) ? ( x |= SNMP_FLAGS_STRIKE2 ) : \\r
+ ( x |= SNMP_FLAGS_STRIKE1 )), \\r
+ 0))\r
+\r
+/*\r
+ * Error return values.\r
+ *\r
+ * SNMPERR_SUCCESS is the non-PDU "success" code.\r
+ *\r
+ * XXX These should be merged with SNMP_ERR_* defines and confined\r
+ * to values < 0. ???\r
+ */\r
+#define SNMPERR_SUCCESS (0) /* XXX Non-PDU "success" code. */\r
+#define SNMPERR_GENERR (-1)\r
+#define SNMPERR_BAD_LOCPORT (-2)\r
+#define SNMPERR_BAD_ADDRESS (-3)\r
+#define SNMPERR_BAD_SESSION (-4)\r
+#define SNMPERR_TOO_LONG (-5)\r
+#define SNMPERR_NO_SOCKET (-6)\r
+#define SNMPERR_V2_IN_V1 (-7)\r
+#define SNMPERR_V1_IN_V2 (-8)\r
+#define SNMPERR_BAD_REPEATERS (-9)\r
+#define SNMPERR_BAD_REPETITIONS (-10)\r
+#define SNMPERR_BAD_ASN1_BUILD (-11)\r
+#define SNMPERR_BAD_SENDTO (-12)\r
+#define SNMPERR_BAD_PARSE (-13)\r
+#define SNMPERR_BAD_VERSION (-14)\r
+#define SNMPERR_BAD_SRC_PARTY (-15)\r
+#define SNMPERR_BAD_DST_PARTY (-16)\r
+#define SNMPERR_BAD_CONTEXT (-17)\r
+#define SNMPERR_BAD_COMMUNITY (-18)\r
+#define SNMPERR_NOAUTH_DESPRIV (-19)\r
+#define SNMPERR_BAD_ACL (-20)\r
+#define SNMPERR_BAD_PARTY (-21)\r
+#define SNMPERR_ABORT (-22)\r
+#define SNMPERR_UNKNOWN_PDU (-23)\r
+#define SNMPERR_TIMEOUT (-24)\r
+#define SNMPERR_BAD_RECVFROM (-25)\r
+#define SNMPERR_BAD_ENG_ID (-26)\r
+#define SNMPERR_BAD_SEC_NAME (-27)\r
+#define SNMPERR_BAD_SEC_LEVEL (-28)\r
+#define SNMPERR_ASN_PARSE_ERR (-29)\r
+#define SNMPERR_UNKNOWN_SEC_MODEL (-30)\r
+#define SNMPERR_INVALID_MSG (-31)\r
+#define SNMPERR_UNKNOWN_ENG_ID (-32)\r
+#define SNMPERR_UNKNOWN_USER_NAME (-33)\r
+#define SNMPERR_UNSUPPORTED_SEC_LEVEL (-34)\r
+#define SNMPERR_AUTHENTICATION_FAILURE (-35)\r
+#define SNMPERR_NOT_IN_TIME_WINDOW (-36)\r
+#define SNMPERR_DECRYPTION_ERR (-37)\r
+#define SNMPERR_SC_GENERAL_FAILURE (-38)\r
+#define SNMPERR_SC_NOT_CONFIGURED (-39)\r
+#define SNMPERR_KT_NOT_AVAILABLE (-40)\r
+#define SNMPERR_UNKNOWN_REPORT (-41)\r
+#define SNMPERR_USM_GENERICERROR (-42)\r
+#define SNMPERR_USM_UNKNOWNSECURITYNAME (-43)\r
+#define SNMPERR_USM_UNSUPPORTEDSECURITYLEVEL (-44)\r
+#define SNMPERR_USM_ENCRYPTIONERROR (-45)\r
+#define SNMPERR_USM_AUTHENTICATIONFAILURE (-46)\r
+#define SNMPERR_USM_PARSEERROR (-47)\r
+#define SNMPERR_USM_UNKNOWNENGINEID (-48)\r
+#define SNMPERR_USM_NOTINTIMEWINDOW (-49)\r
+#define SNMPERR_USM_DECRYPTIONERROR (-50)\r
+#define SNMPERR_NOMIB (-51)\r
+#define SNMPERR_RANGE (-52)\r
+#define SNMPERR_MAX_SUBID (-53)\r
+#define SNMPERR_BAD_SUBID (-54)\r
+#define SNMPERR_LONG_OID (-55)\r
+#define SNMPERR_BAD_NAME (-56)\r
+#define SNMPERR_VALUE (-57)\r
+#define SNMPERR_UNKNOWN_OBJID (-58)\r
+#define SNMPERR_NULL_PDU (-59)\r
+#define SNMPERR_NO_VARS (-60)\r
+#define SNMPERR_VAR_TYPE (-61)\r
+#define SNMPERR_MALLOC (-62)\r
+\r
+#define SNMPERR_MAX (-62)\r
+\r
+#define non_repeaters errstat\r
+#define max_repetitions errindex\r
+\r
+\r
+struct variable_list {\r
+ struct variable_list *next_variable; /* NULL for last variable */\r
+ oid *name; /* Object identifier of variable */\r
+ size_t name_length; /* number of subid's in name */\r
+ u_char type; /* ASN type of variable */\r
+ union { /* value of variable */\r
+ long *integer;\r
+ u_char *string;\r
+ oid *objid;\r
+ u_char *bitstring;\r
+ struct counter64 *counter64;\r
+#ifdef OPAQUE_SPECIAL_TYPES\r
+ float *floatVal;\r
+ double *doubleVal;\r
+/* t_union *unionVal; */\r
+#endif /* OPAQUE_SPECIAL_TYPES */\r
+ } val;\r
+ size_t val_len;\r
+ oid name_loc[MAX_OID_LEN]; /* 90 percentile < 24. */\r
+ u_char buf[40]; /* 90 percentile < 40. */\r
+ void *data; /* (Opaque) hook for additional data */\r
+ int index;\r
+};\r
+\r
+\r
+\r
+/*\r
+ * struct snmp_session *snmp_open(session)\r
+ * struct snmp_session *session;\r
+ *\r
+ * Sets up the session with the snmp_session information provided\r
+ * by the user. Then opens and binds the necessary UDP port.\r
+ * A handle to the created session is returned (this is different than\r
+ * the pointer passed to snmp_open()). On any error, NULL is returned\r
+ * and snmp_errno is set to the appropriate error code.\r
+ */\r
+struct snmp_session *snmp_open (struct snmp_session *);\r
+\r
+/*\r
+ * int snmp_close(session)\r
+ * struct snmp_session *session;\r
+ *\r
+ * Close the input session. Frees all data allocated for the session,\r
+ * dequeues any pending requests, and closes any sockets allocated for\r
+ * the session. Returns 0 on error, 1 otherwise.\r
+ *\r
+ * snmp_close_sessions() does the same thing for all open sessions\r
+ */\r
+int snmp_close (struct snmp_session *);\r
+int snmp_close_sessions (void);\r
+\r
+\r
+/*\r
+ * int snmp_send(session, pdu)\r
+ * struct snmp_session *session;\r
+ * struct snmp_pdu *pdu;\r
+ *\r
+ * Sends the input pdu on the session after calling snmp_build to create\r
+ * a serialized packet. If necessary, set some of the pdu data from the\r
+ * session defaults. Add a request corresponding to this pdu to the list\r
+ * of outstanding requests on this session, then send the pdu.\r
+ * Returns the request id of the generated packet if applicable, otherwise 1.\r
+ * On any error, 0 is returned.\r
+ * The pdu is freed by snmp_send() unless a failure occured.\r
+ */\r
+int snmp_send (struct snmp_session *, struct snmp_pdu *);\r
+\r
+/*\r
+ * int snmp_async_send(session, pdu, callback, cb_data)\r
+ * struct snmp_session *session;\r
+ * struct snmp_pdu *pdu;\r
+ * snmp_callback callback;\r
+ * void *cb_data;\r
+ *\r
+ * Sends the input pdu on the session after calling snmp_build to create\r
+ * a serialized packet. If necessary, set some of the pdu data from the\r
+ * session defaults. Add a request corresponding to this pdu to the list\r
+ * of outstanding requests on this session and store callback and data,\r
+ * then send the pdu.\r
+ * Returns the request id of the generated packet if applicable, otherwise 1.\r
+ * On any error, 0 is returned.\r
+ * The pdu is freed by snmp_send() unless a failure occured.\r
+ */\r
+int snmp_async_send (struct snmp_session *, struct snmp_pdu *,\r
+ snmp_callback, void *);\r
+\r
+\r
+/*\r
+ * void snmp_read(fdset)\r
+ * fd_set *fdset;\r
+ *\r
+ * Checks to see if any of the fd's set in the fdset belong to\r
+ * snmp. Each socket with it's fd set has a packet read from it\r
+ * and snmp_parse is called on the packet received. The resulting pdu\r
+ * is passed to the callback routine for that session. If the callback\r
+ * routine returns successfully, the pdu and it's request are deleted.\r
+ */\r
+void snmp_read (fd_set *);\r
+\r
+\r
+\r
+/*\r
+ * void\r
+ * snmp_free_pdu(pdu)\r
+ * struct snmp_pdu *pdu;\r
+ *\r
+ * Frees the pdu and any malloc'd data associated with it.\r
+ */\r
+void snmp_free_pdu (struct snmp_pdu *);\r
+\r
+void snmp_free_var (struct variable_list *); /* frees just this one */\r
+\r
+void snmp_free_varbind(struct variable_list *var); /* frees all in list */\r
+\r
+/*\r
+ * int snmp_select_info(numfds, fdset, timeout, block)\r
+ * int *numfds;\r
+ * fd_set *fdset;\r
+ * struct timeval *timeout;\r
+ * int *block;\r
+ *\r
+ * Returns info about what snmp requires from a select statement.\r
+ * numfds is the number of fds in the list that are significant.\r
+ * All file descriptors opened for SNMP are OR'd into the fdset.\r
+ * If activity occurs on any of these file descriptors, snmp_read\r
+ * should be called with that file descriptor set.\r
+ *\r
+ * The timeout is the latest time that SNMP can wait for a timeout. The\r
+ * select should be done with the minimum time between timeout and any other\r
+ * timeouts necessary. This should be checked upon each invocation of select.\r
+ * If a timeout is received, snmp_timeout should be called to check if the\r
+ * timeout was for SNMP. (snmp_timeout is idempotent)\r
+ *\r
+ * Block is 1 if the select is requested to block indefinitely, rather than\r
+ * time out. If block is input as 1, the timeout value will be treated as\r
+ * undefined, but it must be available for setting in snmp_select_info. On\r
+ * return, if block is true, the value of timeout will be undefined.\r
+ *\r
+ * snmp_select_info returns the number of open sockets. (i.e. The number\r
+ * of sessions open)\r
+ */\r
+int snmp_select_info (int *, fd_set *, struct timeval *, int *);\r
+\r
+\r
+\r
+/*\r
+ * void snmp_timeout();\r
+ *\r
+ * snmp_timeout should be called whenever the timeout from snmp_select_info\r
+ * expires, but it is idempotent, so snmp_timeout can be polled (probably a\r
+ * cpu expensive proposition). snmp_timeout checks to see if any of the\r
+ * sessions have an outstanding request that has timed out. If it finds one\r
+ * (or more), and that pdu has more retries available, a new packet is formed\r
+ * from the pdu and is resent. If there are no more retries available, the\r
+ * callback for the session is used to alert the user of the timeout.\r
+ */\r
+\r
+void snmp_timeout (void);\r
+\r
+\r
+/*\r
+ * This routine must be supplied by the application:\r
+ *\r
+ * u_char *authenticator(pdu, length, community, community_len)\r
+ * u_char *pdu; The rest of the PDU to be authenticated\r
+ * int *length; The length of the PDU (updated by the authenticator)\r
+ * u_char *community; The community name to authenticate under.\r
+ * int community_len The length of the community name.\r
+ *\r
+ * Returns the authenticated pdu, or NULL if authentication failed.\r
+ * If null authentication is used, the authenticator in snmp_session can be\r
+ * set to NULL(0).\r
+ */\r
+\r
+\r
+\r
+/*\r
+ * This routine must be supplied by the application:\r
+ *\r
+ * int callback(operation, session, reqid, pdu, magic)\r
+ * int operation;\r
+ * struct snmp_session *session; The session authenticated under.\r
+ * int reqid; The request id of this pdu (0 for TRAP)\r
+ * struct snmp_pdu *pdu; The pdu information.\r
+ * void *magic A link to the data for this routine.\r
+ *\r
+ * Returns 1 if request was successful, 0 if it should be kept pending.\r
+ * Any data in the pdu must be copied because it will be freed elsewhere.\r
+ * Operations are defined below:\r
+ */\r
+\r
+#define RECEIVED_MESSAGE 1\r
+#define TIMED_OUT 2\r
+#define SEND_FAILED 3\r
+\r
+long snmp_get_next_msgid(void);\r
+long snmp_get_next_reqid(void);\r
+long snmp_get_next_sessid(void);\r
+long snmp_get_next_transid(void);\r
+/* provide for backwards compatibility */\r
+void snmp_set_dump_packet(int);\r
+int snmp_get_dump_packet(void);\r
+void snmp_set_quick_print(int);\r
+int snmp_get_quick_print(void);\r
+void snmp_set_suffix_only(int);\r
+int snmp_get_suffix_only(void);\r
+void snmp_set_full_objid(int);\r
+int snmp_get_full_objid(void);\r
+void snmp_set_random_access(int);\r
+int snmp_get_random_access(void);\r
+\r
+int snmp_oid_compare (const oid *, size_t, const oid *, size_t);\r
+void init_snmp (const char *);\r
+u_char *snmp_pdu_build (struct snmp_pdu *, u_char *, size_t *);\r
+#ifdef USE_REVERSE_ASNENCODING\r
+u_char *snmp_pdu_rbuild (struct snmp_pdu *, u_char *, size_t *);\r
+#endif\r
+int snmpv3_parse(struct snmp_pdu *, u_char *, size_t *, u_char **, struct snmp_session *);\r
+int snmpv3_dparse(struct snmp_pdu *, u_char *, size_t *, u_char **, int);\r
+int snmpv3_packet_build(struct snmp_pdu *pdu, u_char *packet, size_t *out_length, u_char *pdu_data, size_t pdu_data_len);\r
+int snmpv3_packet_rbuild(struct snmp_pdu *pdu, u_char *packet, size_t *out_length, u_char *pdu_data, size_t pdu_data_len);\r
+int snmpv3_make_report(struct snmp_pdu *pdu, int error);\r
+int snmpv3_get_report_type(struct snmp_pdu *pdu);\r
+int snmp_pdu_parse(struct snmp_pdu *pdu, u_char *data, size_t *length);\r
+int snmp_pdu_dparse(struct snmp_pdu *pdu, u_char *data, size_t *length, int);\r
+u_char* snmpv3_scopedPDU_parse(struct snmp_pdu *pdu, u_char *cp, size_t *length);\r
+u_char* snmpv3_scopedPDU_dparse(struct snmp_pdu *pdu, u_char *cp, size_t *length, int);\r
+void snmp_store(const char *type);\r
+void snmp_shutdown(const char *type);\r
+struct variable_list *snmp_pdu_add_variable (struct snmp_pdu *, oid *, size_t, u_char, u_char *, size_t);\r
+struct variable_list *snmp_varlist_add_variable(struct variable_list **varlist,\r
+ oid *name, size_t name_length, u_char type, u_char *value, size_t len);\r
+int hex_to_binary (const char *, u_char *);\r
+int ascii_to_binary (const char *, u_char *);\r
+int snmp_add_var (struct snmp_pdu *, oid*, size_t, char, const char *);\r
+oid *snmp_duplicate_objid(oid *objToCopy, size_t);\r
+u_int snmp_increment_statistic(int which);\r
+u_int snmp_increment_statistic_by(int which, int count);\r
+u_int snmp_get_statistic(int which);\r
+void snmp_init_statistics(void);\r
+int create_user_from_session(struct snmp_session *session);\r
+\r
+/* extended open */\r
+struct snmp_session *snmp_open_ex (struct snmp_session *,\r
+ int (*fpre_parse) (struct snmp_session *, snmp_ipaddr),\r
+ int (*fparse) (struct snmp_session *, struct snmp_pdu *, u_char *, size_t),\r
+ int (*fpost_parse) (struct snmp_session *, struct snmp_pdu *, int),\r
+ int (*fbuild) (struct snmp_session *, struct snmp_pdu *, u_char *, size_t *),\r
+ int (*fcheck) (u_char *, size_t)\r
+);\r
+\r
+/* provided for backwards compatability. Don't use these functions.\r
+ See snmp_debug.h and snmp_debug.c instead.\r
+*/\r
+#if HAVE_STDARG_H\r
+void DEBUGP (const char *, ...);\r
+#else\r
+void DEBUGP (va_alist);\r
+#endif\r
+void DEBUGPOID(oid *, size_t);\r
+void snmp_set_do_debugging (int);\r
+int snmp_get_do_debugging (void);\r
+\r
+#ifdef CMU_COMPATIBLE\r
+extern int snmp_dump_packet;\r
+extern int quick_print;\r
+#endif\r
+\r
+size_t snmp_socket_length (int family);\r
+\r
+/*\r
+ * snmp_error - return error data\r
+ * Inputs : address of errno, address of snmp_errno, address of string\r
+ * Caller must free the string returned after use.\r
+ */\r
+void snmp_error (struct snmp_session *, int *, int *, char **);\r
+/*\r
+ * single session API.\r
+ *\r
+ * These functions perform similar actions as snmp_XX functions,\r
+ * but operate on a single session only.\r
+ *\r
+ * Synopsis:\r
+\r
+ void * sessp;\r
+ struct snmp_session session, *ss;\r
+ struct snmp_pdu *pdu, *response;\r
+\r
+ snmp_sess_init(&session);\r
+ session.retries = ...\r
+ session.remote_port = ...\r
+ sessp = snmp_sess_open(&session);\r
+ ss = snmp_sess_session(sessp);\r
+ if (ss == NULL)\r
+ exit(1);\r
+ ...\r
+ if (ss->community) free(ss->community);\r
+ ss->community = strdup(gateway);\r
+ ss->community_len = strlen(gateway);\r
+ ...\r
+ snmp_sess_synch_response(sessp, pdu, &response);\r
+ ...\r
+ snmp_sess_close(sessp);\r
+\r
+ * See also:\r
+ * snmp_sess_synch_response, in snmp_client.h.\r
+\r
+ * Notes:\r
+ * 1. Invoke snmp_sess_session after snmp_sess_open.\r
+ * 2. snmp_sess_session return value is an opaque pointer.\r
+ * 3. Do NOT free memory returned by snmp_sess_session.\r
+ * 4. Replace snmp_send(ss,pdu) with snmp_sess_send(sessp,pdu)\r
+ */\r
+\r
+void snmp_sess_init (struct snmp_session *);\r
+void * snmp_sess_open (struct snmp_session *);\r
+struct snmp_session * snmp_sess_session (void *);\r
+\r
+/* use return value from snmp_sess_open as void * parameter */\r
+\r
+int snmp_sess_send (void *, struct snmp_pdu *);\r
+int snmp_sess_async_send (void *, struct snmp_pdu *,\r
+ snmp_callback, void *);\r
+int snmp_sess_select_info (void *, int *, fd_set *,\r
+ struct timeval *, int *);\r
+int snmp_sess_read (void *, fd_set *);\r
+void snmp_sess_timeout (void *);\r
+int snmp_sess_close (void *);\r
+\r
+void snmp_sess_error (void *, int *, int *, char **);\r
+void snmp_sess_perror (const char *prog_string, struct snmp_session *ss);\r
+\r
+/* end single session API */\r
+\r
+/* generic statistic counters */\r
+\r
+/* snmpv3 statistics */\r
+\r
+/* mpd stats */\r
+#define STAT_SNMPUNKNOWNSECURITYMODELS 0\r
+#define STAT_SNMPINVALIDMSGS 1\r
+#define STAT_SNMPUNKNOWNPDUHANDLERS 2\r
+#define STAT_MPD_STATS_START STAT_SNMPUNKNOWNSECURITYMODELS\r
+#define STAT_MPD_STATS_END STAT_SNMPUNKNOWNPDUHANDLERS\r
+\r
+/* usm stats */\r
+#define STAT_USMSTATSUNSUPPORTEDSECLEVELS 3\r
+#define STAT_USMSTATSNOTINTIMEWINDOWS 4\r
+#define STAT_USMSTATSUNKNOWNUSERNAMES 5\r
+#define STAT_USMSTATSUNKNOWNENGINEIDS 6\r
+#define STAT_USMSTATSWRONGDIGESTS 7\r
+#define STAT_USMSTATSDECRYPTIONERRORS 8\r
+#define STAT_USM_STATS_START STAT_USMSTATSUNSUPPORTEDSECLEVELS\r
+#define STAT_USM_STATS_END STAT_USMSTATSDECRYPTIONERRORS\r
+\r
+/* snmp counters */\r
+#define STAT_SNMPINPKTS 9\r
+#define STAT_SNMPOUTPKTS 10\r
+#define STAT_SNMPINBADVERSIONS 11\r
+#define STAT_SNMPINBADCOMMUNITYNAMES 12\r
+#define STAT_SNMPINBADCOMMUNITYUSES 13\r
+#define STAT_SNMPINASNPARSEERRS 14\r
+/* #define STAT_SNMPINBADTYPES 15 */\r
+#define STAT_SNMPINTOOBIGS 16\r
+#define STAT_SNMPINNOSUCHNAMES 17\r
+#define STAT_SNMPINBADVALUES 18\r
+#define STAT_SNMPINREADONLYS 19\r
+#define STAT_SNMPINGENERRS 20\r
+#define STAT_SNMPINTOTALREQVARS 21\r
+#define STAT_SNMPINTOTALSETVARS 22\r
+#define STAT_SNMPINGETREQUESTS 23\r
+#define STAT_SNMPINGETNEXTS 24\r
+#define STAT_SNMPINSETREQUESTS 25\r
+#define STAT_SNMPINGETRESPONSES 26\r
+#define STAT_SNMPINTRAPS 27\r
+#define STAT_SNMPOUTTOOBIGS 28\r
+#define STAT_SNMPOUTNOSUCHNAMES 29\r
+#define STAT_SNMPOUTBADVALUES 30\r
+/* #define STAT_SNMPOUTREADONLYS 31 */\r
+#define STAT_SNMPOUTGENERRS 32\r
+#define STAT_SNMPOUTGETREQUESTS 33\r
+#define STAT_SNMPOUTGETNEXTS 34\r
+#define STAT_SNMPOUTSETREQUESTS 35\r
+#define STAT_SNMPOUTGETRESPONSES 36\r
+#define STAT_SNMPOUTTRAPS 37\r
+/* AUTHTRAPENABLE 38 */\r
+#define STAT_SNMPSILENTDROPS 39\r
+#define STAT_SNMPPROXYDROPS 40\r
+#define STAT_SNMP_STATS_START STAT_SNMPINPKTS\r
+#define STAT_SNMP_STATS_END STAT_SNMPOUTTRAPS\r
+\r
+#define MAX_STATS 41\r
+\r
+#ifdef __cplusplus\r
+}\r
+#endif\r
+\r
+#endif /* SNMP_API_H */\r
-# URL extractor\r
-# Copyright 2004, Paul McGuire\r
-from pyparsing import makeHTMLTags, SkipTo, pyparsing_common\r
-import urllib.request\r
-from contextlib import closing\r
-import pprint\r
-\r
-linkOpenTag, linkCloseTag = makeHTMLTags('a')\r
-\r
-linkBody = SkipTo(linkCloseTag)\r
-linkBody.setParseAction(pyparsing_common.stripHTMLTags)\r
-linkBody.addParseAction(lambda toks: ' '.join(toks[0].strip().split()))\r
-\r
-link = linkOpenTag + linkBody("body") + linkCloseTag.suppress()\r
-\r
-# Go get some HTML with some links in it.\r
-with closing(urllib.request.urlopen("https://www.yahoo.com/")) as serverListPage:\r
- htmlText = serverListPage.read().decode("UTF-8")\r
-\r
-# scanString is a generator that loops through the input htmlText, and for each\r
-# match yields the tokens and start and end locations (for this application, we are\r
-# not interested in the start and end values).\r
-for toks,strt,end in link.scanString(htmlText):\r
- print(toks.asList())\r
-\r
-# Create dictionary from list comprehension, assembled from each pair of tokens returned\r
-# from a matched URL.\r
-pprint.pprint(\r
- {toks.body: toks.href for toks,strt,end in link.scanString(htmlText)}\r
- )\r
+# URL extractor
+# Copyright 2004, Paul McGuire
+from pyparsing import makeHTMLTags, pyparsing_common as ppc
+import urllib.request
+from contextlib import closing
+import pprint
+
+linkOpenTag, linkCloseTag = makeHTMLTags('a')
+
+linkBody = linkOpenTag.tag_body
+linkBody.setParseAction(ppc.stripHTMLTags)
+linkBody.addParseAction(lambda toks: ' '.join(toks[0].strip().split()))
+
+link = linkOpenTag + linkBody("body") + linkCloseTag.suppress()
+
+# Go get some HTML with some links in it.
+with closing(urllib.request.urlopen("https://www.cnn.com/")) as serverListPage:
+ htmlText = serverListPage.read().decode("UTF-8")
+
+# scanString is a generator that loops through the input htmlText, and for each
+# match yields the tokens and start and end locations (for this application, we are
+# not interested in the start and end values).
+for toks, strt, end in link.scanString(htmlText):
+ print(toks.asList())
+
+# Create dictionary from list comprehension, assembled from each pair of tokens returned
+# from a matched URL.
+pprint.pprint(
+ {toks.body: toks.href for toks, strt, end in link.scanString(htmlText)}
+ )
-# URL extractor\r
-# Copyright 2004, Paul McGuire\r
-from pyparsing import SkipTo, makeHTMLTags\r
-import urllib.request, urllib.parse, urllib.error\r
-import pprint\r
-\r
-# Define the pyparsing grammar for a URL, that is:\r
-# URLlink ::= <a href= URL>linkText</a>\r
-# URL ::= doubleQuotedString | alphanumericWordPath\r
-# Note that whitespace may appear just about anywhere in the link. Note also\r
-# that it is not necessary to explicitly show this in the pyparsing grammar; by default,\r
-# pyparsing skips over whitespace between tokens.\r
-linkOpenTag,linkCloseTag = makeHTMLTags("a")\r
-link = linkOpenTag + SkipTo(linkCloseTag)("body") + linkCloseTag.suppress()\r
-\r
-# Go get some HTML with some links in it.\r
-serverListPage = urllib.request.urlopen( "https://www.google.com/" )\r
-htmlText = serverListPage.read()\r
-serverListPage.close()\r
-\r
-# scanString is a generator that loops through the input htmlText, and for each\r
-# match yields the tokens and start and end locations (for this application, we are\r
-# not interested in the start and end values).\r
-for toks,strt,end in link.scanString(htmlText):\r
- print(toks.startA.href,"->",toks.body)\r
-\r
-# Create dictionary from list comprehension, assembled from each pair of tokens returned\r
-# from a matched URL.\r
-pprint.pprint(\r
- { toks.body:toks.startA.href for toks,strt,end in link.scanString(htmlText) }\r
- )\r
+# URL extractor
+# Copyright 2004, Paul McGuire
+from pyparsing import makeHTMLTags
+from contextlib import closing
+import urllib.request, urllib.parse, urllib.error
+import pprint
+
+# Define the pyparsing grammar for a URL, that is:
+# URLlink ::= <a href= URL>linkText</a>
+# URL ::= doubleQuotedString | alphanumericWordPath
+# Note that whitespace may appear just about anywhere in the link. Note also
+# that it is not necessary to explicitly show this in the pyparsing grammar; by default,
+# pyparsing skips over whitespace between tokens.
+linkOpenTag, linkCloseTag = makeHTMLTags("a")
+link = linkOpenTag + linkOpenTag.tag_body("body") + linkCloseTag.suppress()
+
+# Go get some HTML with some links in it.
+with closing(urllib.request.urlopen("https://www.cnn.com/")) as serverListPage:
+ htmlText = serverListPage.read()
+
+# scanString is a generator that loops through the input htmlText, and for each
+# match yields the tokens and start and end locations (for this application, we are
+# not interested in the start and end values).
+for toks, strt, end in link.scanString(htmlText):
+ print(toks.startA.href, "->", toks.body)
+
+# Create dictionary from list comprehension, assembled from each pair of tokens returned
+# from a matched URL.
+pprint.pprint(
+ {toks.body: toks.startA.href for toks, strt, end in link.scanString(htmlText)}
+ )
-#\r
-# withAttribute.py\r
-# Copyright, 2007 - Paul McGuire\r
-#\r
-# Simple example of using withAttribute parse action helper\r
-# to define\r
-#\r
-data = """\\r
- <td align=right width=80><font size=2 face="New Times Roman,Times,Serif"> 49.950 </font></td>\r
- <td align=left width=80><font size=2 face="New Times Roman,Times,Serif"> 50.950 </font></td>\r
- <td align=right width=80><font size=2 face="New Times Roman,Times,Serif"> 51.950 </font></td>\r
- """\r
-\r
-from pyparsing import *\r
-\r
-tdS,tdE = makeHTMLTags("TD")\r
-fontS,fontE = makeHTMLTags("FONT")\r
-realNum = Combine( Word(nums) + "." + Word(nums) ).setParseAction(lambda t:float(t[0]))\r
-NBSP = Literal(" ")\r
-patt = tdS + fontS + NBSP + realNum("value") + NBSP + fontE + tdE\r
-\r
-tdS.setParseAction( withAttribute(align="right",width="80") )\r
-for s in patt.searchString(data):\r
- print(s.value)\r
+#
+# withAttribute.py
+# Copyright, 2007 - Paul McGuire
+#
+# Simple example of using withAttribute parse action helper
+# to define
+#
+import pyparsing as pp
+
+data = """\
+ <td align=right width=80><font size=2 face="New Times Roman,Times,Serif"> 49.950 </font></td>
+ <td align=left width=80><font size=2 face="New Times Roman,Times,Serif"> 50.950 </font></td>
+ <td align=right width=80><font size=2 face="New Times Roman,Times,Serif"> 51.950 </font></td>
+ """
+
+td, tdEnd = pp.makeHTMLTags("TD")
+font, fontEnd = pp.makeHTMLTags("FONT")
+realNum = pp.pyparsing_common.real
+NBSP = pp.Literal(" ")
+patt = td + font + NBSP + realNum("value") + NBSP + fontEnd + tdEnd
+
+# always use addParseAction when adding withAttribute as a parse action to a start tag
+td.addParseAction(pp.withAttribute(align="right", width="80"))
+
+for s in patt.searchString(data):
+ print(s.value)
-Metadata-Version: 1.1
+Metadata-Version: 1.2
Name: pyparsing
-Version: 2.3.1
+Version: 2.4.0
Summary: Python parsing module
Home-page: https://github.com/pyparsing/pyparsing/
Author: Paul McGuire
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
+Requires-Python: >=2.6, !=3.0.*, !=3.1.*, !=3.2.*
examples/excelExpr.py
examples/fourFn.py
examples/gen_ctypes.py
-examples/getNTPservers.py
examples/getNTPserversNew.py
examples/greeting.py
examples/greetingInGreek.py
examples/groupUsingListAllMatches.py
examples/holaMundo.py
examples/htmlStripper.py
+examples/htmlTableParser.py
examples/httpServerLogParser.py
examples/idlParse.py
+examples/include_preprocessor.py
examples/indentedGrammarExample.py
examples/invRegex.py
examples/jsonParser.py
examples/listAllMatches.py
examples/lucene_grammar.py
examples/macroExpander.py
-examples/makeHTMLTagExample.py
examples/matchPreviousDemo.py
examples/mozilla.ics
examples/mozillaCalendarParser.py
examples/readJson.py
examples/removeLineBreaks.py
examples/romanNumerals.py
+examples/rosettacode.py
examples/scanExamples.py
-examples/scanYahoo.py
examples/searchParserAppDemo.py
examples/searchparser.py
examples/select_parser.py
examples/simpleBool.py
examples/simpleSQL.py
examples/simpleWiki.py
+examples/snmp_api.h
examples/sparser.py
examples/sql2dot.py
examples/stackish.py
namespace class
"""
-__version__ = "2.3.1"
-__versionTime__ = "09 Jan 2019 23:26 UTC"
+__version__ = "2.4.0"
+__versionTime__ = "07 Apr 2019 18:28 UTC"
__author__ = "Paul McGuire <ptmcg@users.sourceforge.net>"
import string
except ImportError:
class SimpleNamespace: pass
+# version compatibility configuration
+__compat__ = SimpleNamespace()
+__compat__.__doc__ = """
+ A cross-version compatibility configuration for pyparsing features that will be
+ released in a future version. By setting values in this configuration to True,
+ those features can be enabled in prior versions for compatibility development
+ and testing.
+
+ - collect_all_And_tokens - flag to enable fix for Issue #63 that fixes erroneous grouping
+ of results names when an And expression is nested within an Or or MatchFirst; set to
+ True to enable bugfix to be released in pyparsing 2.4
+"""
+__compat__.collect_all_And_tokens = True
+
#~ sys.stderr.write( "testing pyparsing module, version %s, %s\n" % (__version__,__versionTime__ ) )
-__all__ = [
+__all__ = [ '__version__', '__versionTime__', '__author__', '__compat__',
'And', 'CaselessKeyword', 'CaselessLiteral', 'CharsNotIn', 'Combine', 'Dict', 'Each', 'Empty',
'FollowedBy', 'Forward', 'GoToColumn', 'Group', 'Keyword', 'LineEnd', 'LineStart', 'Literal',
'PrecededBy', 'MatchFirst', 'NoMatch', 'NotAny', 'OneOrMore', 'OnlyOnce', 'Optional', 'Or',
callers = inspect.getinnerframes(exc.__traceback__, context=depth)
seen = set()
for i, ff in enumerate(callers[-depth:]):
- frm = ff.frame
+ frm = ff[0]
f_self = frm.f_locals.get('self', None)
if isinstance(f_self, ParserElement):
print(patt.addParseAction(make_palindrome).parseString("lskdj sdlkjf lksd")) # -> 'lskdjsdlkjflksddsklfjkldsjdksl'
"""
if isinstance(itemseq, ParseResults):
- self += itemseq
+ self.__iadd__(itemseq)
else:
self.__toklist.extend(itemseq)
comments = []
try:
# convert newline marks to actual newlines, and strip leading BOM if present
- t = t.replace(r'\n','\n').lstrip('\ufeff')
+ NL = Literal(r'\n').addParseAction(replaceWith('\n')).ignore(quotedString)
+ BOM = '\ufeff'
+ t = NL.transformString(t.lstrip(BOM))
result = self.parseString(t, parseAll=parseAll)
out.append(result.dump(full=fullDump))
success = success and not failureTests
def __init__( self, initChars, bodyChars=None, min=1, max=0, exact=0, asKeyword=False, excludeChars=None ):
super(Word,self).__init__()
if excludeChars:
+ excludeChars = set(excludeChars)
initChars = ''.join(c for c in initChars if c not in excludeChars)
if bodyChars:
bodyChars = ''.join(c for c in bodyChars if c not in excludeChars)
loc = result.end()
return loc, result.group()
- if not(instring[ loc ] in self.initChars):
+ if instring[loc] not in self.initChars:
raise ParseException(instring, loc, self.errmsg, self)
start = loc
throwException = False
if loc - start < self.minLen:
throwException = True
- if self.maxSpecified and loc < instrlen and instring[loc] in bodychars:
+ elif self.maxSpecified and loc < instrlen and instring[loc] in bodychars:
throwException = True
- if self.asKeyword:
+ elif self.asKeyword:
if (start>0 and instring[start-1] in bodychars) or (loc<instrlen and instring[loc] in bodychars):
throwException = True
when defining a match of any single character in a string of
characters.
"""
- def __init__(self, charset):
- super(Char, self).__init__(charset, exact=1)
+ def __init__(self, charset, asKeyword=False, excludeChars=None):
+ super(Char, self).__init__(charset, exact=1, asKeyword=asKeyword, excludeChars=excludeChars)
self.reString = "[%s]" % _escapeRegexRangeChars(self.initCharsOrig)
self.re = re.compile( self.reString )
self.mayReturnEmpty = True
self.asGroupList = asGroupList
self.asMatch = asMatch
+ if self.asGroupList:
+ self.parseImpl = self.parseImplAsGroupList
+ if self.asMatch:
+ self.parseImpl = self.parseImplAsMatch
- def parseImpl( self, instring, loc, doActions=True ):
+ def parseImpl(self, instring, loc, doActions=True):
result = self.re.match(instring,loc)
if not result:
raise ParseException(instring, loc, self.errmsg, self)
loc = result.end()
- if self.asMatch:
- ret = result
- elif self.asGroupList:
- ret = result.groups()
- else:
- ret = ParseResults(result.group())
- d = result.groupdict()
- if d:
- for k, v in d.items():
- ret[k] = v
- return loc,ret
+ ret = ParseResults(result.group())
+ d = result.groupdict()
+ if d:
+ for k, v in d.items():
+ ret[k] = v
+ return loc, ret
+
+ def parseImplAsGroupList(self, instring, loc, doActions=True):
+ result = self.re.match(instring,loc)
+ if not result:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ loc = result.end()
+ ret = result.groups()
+ return loc, ret
+
+ def parseImplAsMatch(self, instring, loc, doActions=True):
+ result = self.re.match(instring,loc)
+ if not result:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ loc = result.end()
+ ret = result
+ return loc, ret
def __str__( self ):
try:
return self.strRepr
def sub(self, repl):
- """
+ r"""
Return Regex with an attached parse action to transform the parsed
result as if called using `re.sub(expr, repl, string) <https://docs.python.org/3/library/re.html#re.sub>`_.
self.minLen = exact
def parseImpl( self, instring, loc, doActions=True ):
- if not(instring[ loc ] in self.matchWhite):
+ if instring[loc] not in self.matchWhite:
raise ParseException(instring, loc, self.errmsg, self)
start = loc
loc += 1
class LineStart(_PositionToken):
- """Matches if current position is at the beginning of a line within
+ r"""Matches if current position is at the beginning of a line within
the parse string
Example::
return self
- def setResultsName( self, name, listAllMatches=False ):
- ret = super(ParseExpression,self).setResultsName(name,listAllMatches)
- return ret
-
def validate( self, validateTrace=[] ):
tmp = validateTrace[:]+[self]
for e in self.exprs:
def streamline(self):
super(Or, self).streamline()
- self.saveAsList = any(e.saveAsList for e in self.exprs)
+ if __compat__.collect_all_And_tokens:
+ self.saveAsList = any(e.saveAsList for e in self.exprs)
return self
def parseImpl( self, instring, loc, doActions=True ):
super(MatchFirst,self).__init__(exprs, savelist)
if self.exprs:
self.mayReturnEmpty = any(e.mayReturnEmpty for e in self.exprs)
- # self.saveAsList = any(e.saveAsList for e in self.exprs)
else:
self.mayReturnEmpty = True
def streamline(self):
super(MatchFirst, self).streamline()
- self.saveAsList = any(e.saveAsList for e in self.exprs)
+ if __compat__.collect_all_And_tokens:
+ self.saveAsList = any(e.saveAsList for e in self.exprs)
return self
def parseImpl( self, instring, loc, doActions=True ):
def __str__( self ):
if hasattr(self,"name"):
return self.name
- return self.__class__.__name__ + ": ..."
- # stubbed out for now - creates awful memory and perf issues
- self._revertClass = self.__class__
- self.__class__ = _ForwardNoRecurse
+ # Avoid infinite recursion by setting a temporary name
+ self.name = self.__class__.__name__ + ": ..."
+
+ # Use the string representation of main expression.
try:
if self.expr is not None:
retString = _ustr(self.expr)
else:
retString = "None"
finally:
- self.__class__ = self._revertClass
+ del self.name
return self.__class__.__name__ + ": " + retString
def copy(self):
ret <<= self
return ret
-class _ForwardNoRecurse(Forward):
- def __str__( self ):
- return "..."
-
class TokenConverter(ParseElementEnhance):
"""
Abstract subclass of :class:`ParseExpression`, for converting parsed results.
"""
def __init__( self, expr ):
super(Group,self).__init__( expr )
- self.saveAsList = expr.saveAsList
+ self.saveAsList = True
def postParse( self, instring, loc, tokenlist ):
return [ tokenlist ]
"""Helper to undo pyparsing's default grouping of And expressions,
even if all but one are non-empty.
"""
- return TokenConverter(expr).setParseAction(lambda t:t[0])
+ return TokenConverter(expr).addParseAction(lambda t:t[0])
def locatedExpr(expr):
"""Helper to decorate a returned token with its starting and ending
"""(Deprecated) Helper parse action to convert tokens to lower case.
Deprecated in favor of :class:`pyparsing_common.downcaseTokens`"""
-def _makeTags(tagStr, xml):
+def _makeTags(tagStr, xml,
+ suppress_LT=Suppress("<"),
+ suppress_GT=Suppress(">")):
"""Internal helper to construct opening and closing tag expressions, given a tag name"""
if isinstance(tagStr,basestring):
resname = tagStr
tagAttrName = Word(alphas,alphanums+"_-:")
if (xml):
tagAttrValue = dblQuotedString.copy().setParseAction( removeQuotes )
- openTag = Suppress("<") + tagStr("tag") + \
- Dict(ZeroOrMore(Group( tagAttrName + Suppress("=") + tagAttrValue ))) + \
- Optional("/",default=[False]).setResultsName("empty").setParseAction(lambda s,l,t:t[0]=='/') + Suppress(">")
+ openTag = (suppress_LT
+ + tagStr("tag")
+ + Dict(ZeroOrMore(Group(tagAttrName + Suppress("=") + tagAttrValue )))
+ + Optional("/", default=[False])("empty").setParseAction(lambda s,l,t:t[0]=='/')
+ + suppress_GT)
else:
- printablesLessRAbrack = "".join(c for c in printables if c not in ">")
- tagAttrValue = quotedString.copy().setParseAction( removeQuotes ) | Word(printablesLessRAbrack)
- openTag = Suppress("<") + tagStr("tag") + \
- Dict(ZeroOrMore(Group( tagAttrName.setParseAction(downcaseTokens) + \
- Optional( Suppress("=") + tagAttrValue ) ))) + \
- Optional("/",default=[False]).setResultsName("empty").setParseAction(lambda s,l,t:t[0]=='/') + Suppress(">")
- closeTag = Combine(_L("</") + tagStr + ">")
-
- openTag = openTag.setResultsName("start"+"".join(resname.replace(":"," ").title().split())).setName("<%s>" % resname)
- closeTag = closeTag.setResultsName("end"+"".join(resname.replace(":"," ").title().split())).setName("</%s>" % resname)
+ tagAttrValue = quotedString.copy().setParseAction( removeQuotes ) | Word(printables, excludeChars=">")
+ openTag = (suppress_LT
+ + tagStr("tag")
+ + Dict(ZeroOrMore(Group(tagAttrName.setParseAction(downcaseTokens)
+ + Optional(Suppress("=") + tagAttrValue))))
+ + Optional("/",default=[False])("empty").setParseAction(lambda s,l,t:t[0]=='/')
+ + suppress_GT)
+ closeTag = Combine(_L("</") + tagStr + ">", adjacent=False)
+
+ openTag.setName("<%s>" % resname)
+ # add start<tagname> results name in parse action now that ungrouped names are not reported at two levels
+ openTag.addParseAction(lambda t: t.__setitem__("start"+"".join(resname.replace(":"," ").title().split()), t.copy()))
+ closeTag = closeTag("end"+"".join(resname.replace(":"," ").title().split())).setName("</%s>" % resname)
openTag.tag = resname
closeTag.tag = resname
+ openTag.tag_body = SkipTo(closeTag())
return openTag, closeTag
def makeHTMLTags(tagStr):
':',
[[['def', 'eggs', ['(', 'z', ')'], ':', [['pass']]]]]]]
"""
+ backup_stack = indentStack[:]
+
+ def reset_stack():
+ indentStack[:] = backup_stack
+
def checkPeerIndent(s,l,t):
if l >= len(s): return
curCol = col(l,s)
if curCol != indentStack[-1]:
if curCol > indentStack[-1]:
- raise ParseFatalException(s,l,"illegal nesting")
+ raise ParseException(s,l,"illegal nesting")
raise ParseException(s,l,"not a peer entry")
def checkSubIndent(s,l,t):
else:
smExpr = Group( Optional(NL) +
(OneOrMore( PEER + Group(blockStatementExpr) + Optional(NL) )) )
+ smExpr.setFailAction(lambda a, b, c, d: reset_stack())
blockStatementExpr.ignore(_bslash + LineEnd())
return smExpr.setName('indented block')
[egg_info]
tag_build =
tag_date = 0
-tag_svn_revision = 0
-#!/usr/bin/env python\r
-\r
-"""Setup script for the pyparsing module distribution."""\r
-\r
-from setuptools import setup\r
-from pyparsing import __version__ as pyparsing_version\r
-\r
-modules = ["pyparsing",]\r
-\r
-setup(# Distribution meta-data\r
- name = "pyparsing",\r
- version = pyparsing_version,\r
- description = "Python parsing module",\r
- author = "Paul McGuire",\r
- author_email = "ptmcg@users.sourceforge.net",\r
- url = "https://github.com/pyparsing/pyparsing/",\r
- download_url = "https://pypi.org/project/pyparsing/",\r
- license = "MIT License",\r
- py_modules = modules,\r
- python_requires='>=2.6, !=3.0.*, !=3.1.*, !=3.2.*',\r
- classifiers=[\r
- 'Development Status :: 5 - Production/Stable',\r
- 'Intended Audience :: Developers',\r
- 'Intended Audience :: Information Technology',\r
- 'License :: OSI Approved :: MIT License',\r
- 'Operating System :: OS Independent',\r
- 'Programming Language :: Python',\r
- 'Programming Language :: Python :: 2',\r
- 'Programming Language :: Python :: 2.6',\r
- 'Programming Language :: Python :: 2.7',\r
- 'Programming Language :: Python :: 3',\r
- 'Programming Language :: Python :: 3.3',\r
- 'Programming Language :: Python :: 3.4',\r
- 'Programming Language :: Python :: 3.5',\r
- 'Programming Language :: Python :: 3.6',\r
- 'Programming Language :: Python :: 3.7',\r
- ]\r
- )\r
+#!/usr/bin/env python
+
+"""Setup script for the pyparsing module distribution."""
+
+from setuptools import setup
+from pyparsing import __version__ as pyparsing_version
+
+modules = ["pyparsing",]
+
+setup(# Distribution meta-data
+ name = "pyparsing",
+ version = pyparsing_version,
+ description = "Python parsing module",
+ author = "Paul McGuire",
+ author_email = "ptmcg@users.sourceforge.net",
+ url = "https://github.com/pyparsing/pyparsing/",
+ download_url = "https://pypi.org/project/pyparsing/",
+ license = "MIT License",
+ py_modules = modules,
+ python_requires='>=2.6, !=3.0.*, !=3.1.*, !=3.2.*',
+ test_suite="unitTests.suite",
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'Intended Audience :: Developers',
+ 'Intended Audience :: Information Technology',
+ 'License :: OSI Approved :: MIT License',
+ 'Operating System :: OS Independent',
+ 'Programming Language :: Python',
+ 'Programming Language :: Python :: 2',
+ 'Programming Language :: Python :: 2.6',
+ 'Programming Language :: Python :: 2.7',
+ 'Programming Language :: Python :: 3',
+ 'Programming Language :: Python :: 3.3',
+ 'Programming Language :: Python :: 3.4',
+ 'Programming Language :: Python :: 3.5',
+ 'Programming Language :: Python :: 3.6',
+ 'Programming Language :: Python :: 3.7',
+ ]
+ )
given text strings. Subclasses must define a class attribute 'tests' which
is a list of PpTestSpec instances.
"""
+
+ if not hasattr(unittest.TestCase, 'subTest'):
+ # Python 2 compatibility
+ from contextlib import contextmanager
+ @contextmanager
+ def subTest(self, **params):
+ print('subTest:', params)
+ yield
+
tests = []
def runTest(self):
if self.__class__ is PyparsingExpressionTestCase:
# the location against an expected value
with self.subTest(test_spec=test_spec):
test_spec.expr.streamline()
- print("\n{} - {}({})".format(test_spec.desc,
- type(test_spec.expr).__name__,
- test_spec.expr))
+ print("\n{0} - {1}({2})".format(test_spec.desc,
+ type(test_spec.expr).__name__,
+ test_spec.expr))
parsefn = getattr(test_spec.expr, test_spec.parse_fn)
if test_spec.expected_fail_locn is None:
# compare results against given list and/or dict
if test_spec.expected_list is not None:
self.assertEqual([result], test_spec.expected_list)
-
else:
# expect fail
try:
parsefn(test_spec.text)
except Exception as exc:
+ if not hasattr(exc, '__traceback__'):
+ # Python 2 compatibility
+ from sys import exc_info
+ etype, value, traceback = exc_info()
+ exc.__traceback__ = traceback
print(pp.ParseException.explain(exc))
self.assertEqual(exc.loc, test_spec.expected_fail_locn)
else:
self.assertTrue(False, "failed to raise expected exception")
-#=========== TEST DEFINITIONS START HERE ==============
+# =========== TEST DEFINITIONS START HERE ==============
class TestLiteral(PyparsingExpressionTestCase):
tests = [
),
]
+class TestCombine(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Parsing real numbers - fail, parsed numbers are in pieces",
+ expr=pp.OneOrMore(pp.Word(pp.nums) + '.' + pp.Word(pp.nums)),
+ text="1.2 2.3 3.1416 98.6",
+ expected_list=['1', '.', '2', '2', '.', '3', '3', '.', '1416', '98', '.', '6'],
+ ),
+ PpTestSpec(
+ desc="Parsing real numbers - better, use Combine to combine multiple tokens into one",
+ expr=pp.OneOrMore(pp.Combine(pp.Word(pp.nums) + '.' + pp.Word(pp.nums))),
+ text="1.2 2.3 3.1416 98.6",
+ expected_list=['1.2', '2.3', '3.1416', '98.6'],
+ ),
+ ]
+
class TestRepetition(PyparsingExpressionTestCase):
tests = [
PpTestSpec(
class TestParseAction(PyparsingExpressionTestCase):
tests = [
+ PpTestSpec(
+ desc="Parsing real numbers - use parse action to convert to float at parse time",
+ expr=pp.OneOrMore(pp.Combine(pp.Word(pp.nums) + '.' + pp.Word(pp.nums)).addParseAction(lambda t: float(t[0]))),
+ text="1.2 2.3 3.1416 98.6",
+ expected_list= [1.2, 2.3, 3.1416, 98.6], # note, these are now floats, not strs
+ ),
PpTestSpec(
desc = "Match with numeric string converted to int",
expr = pp.Word("0123456789").addParseAction(lambda t: int(t[0])),
),
]
+class TestRegex(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Parsing real numbers - using Regex instead of Combine",
+ expr=pp.OneOrMore(pp.Regex(r'\d+\.\d+').addParseAction(lambda t: float(t[0]))),
+ text="1.2 2.3 3.1416 98.6",
+ expected_list=[1.2, 2.3, 3.1416, 98.6], # note, these are now floats, not strs
+ ),
+ ]
+
class TestParseCondition(PyparsingExpressionTestCase):
tests = [
PpTestSpec(
}
def markup_convert(t):
htmltag = TestTransformStringUsingParseActions.markup_convert_map[t.markup_symbol]
- return "<{}>{}</{}>".format(htmltag, t.body, htmltag)
+ return "<{0}>{1}</{2}>".format(htmltag, t.body, htmltag)
tests = [
PpTestSpec(
]
-#============ MAIN ================
+def _get_decl_line_no(cls):
+ import inspect
+ return inspect.getsourcelines(cls)[1]
-if __name__ == '__main__':
- # we use unittest features that are in Py3 only, bail out if run on Py2
- import sys
- if sys.version_info[0] < 3:
- print("simple_unit_tests.py runs on Python 3 only")
- sys.exit(0)
- import inspect
- def get_decl_line_no(cls):
- return inspect.getsourcelines(cls)[1]
+# get all test case classes defined in this module and sort them by decl line no
+test_case_classes = list(PyparsingExpressionTestCase.__subclasses__())
+test_case_classes.sort(key=_get_decl_line_no)
- # get all test case classes defined in this module and sort them by decl line no
- test_case_classes = list(PyparsingExpressionTestCase.__subclasses__())
- test_case_classes.sort(key=get_decl_line_no)
+# make into a suite and run it - this will run the tests in the same order
+# they are declared in this module
+#
+# runnable from setup.py using "python setup.py test -s simple_unit_tests.suite"
+#
+suite = unittest.TestSuite(cls() for cls in test_case_classes)
+
+
+# ============ MAIN ================
+
+if __name__ == '__main__':
+ result = unittest.TextTestRunner().run(suite)
- # make into a suite and run it - this will run the tests in the same order
- # they are declared in this module
- suite = unittest.TestSuite(cls() for cls in test_case_classes)
- unittest.TextTestRunner().run(suite)
+ exit(0 if result.wasSuccessful() else 1)
class InfixNotationGrammarTest5(ParseTestCase):
def runTest(self):
- from pyparsing import infixNotation, opAssoc, pyparsing_common, Literal, oneOf
+ from pyparsing import infixNotation, opAssoc, pyparsing_common as ppc, Literal, oneOf
expop = Literal('**')
signop = oneOf('+ -')
import operator
opn_map = {'+': operator.add, '-': operator.sub}
- from pyparsing import pyparsing_common, infixNotation
-
- operand = pyparsing_common.number().setParseAction(NumberNode)
+ operand = ppc.number().setParseAction(NumberNode)
expr = infixNotation(operand,
[
(expop, 2, opAssoc.LEFT, (lambda pr: [pr[0][::-1]], ExpOp)),
print_(test[s:e], "->", t.asList())
(expectedType, expectedEmpty, expectedBG, expectedFG) = next(resIter)
- tType = t.getName()
- #~ print tType,"==",expectedType,"?"
- self.assertTrue(tType in "startBody endBody".split(), "parsed token of unknown type '%s'" % tType)
- self.assertEqual(tType, expectedType, "expected token of type %s, got %s" % (expectedType, tType))
- if tType == "startBody":
+ print_(t.dump())
+ if "startBody" in t:
self.assertEqual(bool(t.empty), expectedEmpty,
"expected %s token, got %s" % (expectedEmpty and "empty" or "not empty",
t.empty and "empty" or "not empty"))
"failed to match BGCOLOR, expected %s, got %s" % (expectedBG, t.bgcolor))
self.assertEqual(t.fgcolor, expectedFG,
"failed to match FGCOLOR, expected %s, got %s" % (expectedFG, t.bgcolor))
- elif tType == "endBody":
- #~ print "end tag"
+ elif "endBody" in t:
+ print_("end tag")
pass
else:
print_("BAD!!!")
+
class UpcaseDowncaseUnicode(ParseTestCase):
def runTest(self):
import pyparsing as pp
+ from pyparsing import pyparsing_unicode as ppu
import sys
if PY_3:
unichr = chr
a = u'\u00bfC\u00f3mo esta usted?'
if not JYTHON_ENV:
- ualphas = pp.pyparsing_unicode.alphas
+ ualphas = ppu.alphas
else:
ualphas = "".join( unichr(i) for i in list(range(0xd800)) + list(range(0xe000,sys.maxunicode))
if unichr(i).isalpha() )
<a B="x">3</a>
<a b="X">4</a>
<a b="y">5</a>
- <a class="boo">8</a>
+ <a class="boo">8</ a>
"""
tagStart, tagEnd = makeHTMLTags("a")
class SetNameTest(ParseTestCase):
def runTest(self):
from pyparsing import (oneOf,infixNotation,Word,nums,opAssoc,delimitedList,countedArray,
- nestedExpr,makeHTMLTags,anyOpenTag,anyCloseTag,commonHTMLEntity,replaceHTMLEntity)
+ nestedExpr,makeHTMLTags,anyOpenTag,anyCloseTag,commonHTMLEntity,replaceHTMLEntity,
+ Forward,ZeroOrMore)
a = oneOf("a b c")
b = oneOf("d e f")
[
(('?',':'),3,opAssoc.LEFT),
])
+ recursive = Forward()
+ recursive <<= a + ZeroOrMore(b + recursive)
tests = [
a,
arith_expr.expr,
arith_expr2,
arith_expr2.expr,
+ recursive,
delimitedList(Word(nums).setName("int")),
countedArray(Word(nums).setName("int")),
nestedExpr(),
a | b | c
d | e | f
{a | b | c | d | e | f}
- Forward: ...
+ Forward: + | - term
+ | - term
- Forward: ...
+ Forward: ?: term
?: term
+ Forward: {a | b | c [{d | e | f Forward: ...}]...}
int [, int]...
(len) int...
nested () expression
class ParseResultsNamesInGroupWithDictTest(ParseTestCase):
def runTest(self):
import pyparsing as pp
+ from pyparsing import pyparsing_common as ppc
- key = pp.pyparsing_common.identifier()
- value = pp.pyparsing_common.integer()
- lat = pp.pyparsing_common.real()
- long = pp.pyparsing_common.real()
+ key = ppc.identifier()
+ value = ppc.integer()
+ lat = ppc.real()
+ long = ppc.real()
EQ = pp.Suppress('=')
data = lat("lat") + long("long") + pp.Dict(pp.OneOrMore(pp.Group(key + EQ + value)))
# U = list_num.parseString(test_string)
# self.assertTrue("LIT_NUM" not in U.LIST.LIST_VALUES, "results name retained as sub in ungrouped named result")
+ a, aEnd = pp.makeHTMLTags('a')
+ attrs = a.parseString("<a href='blah'>")
+ print_(attrs.dump())
+ self.assertEqual(attrs.startA.href, 'blah')
+ self.assertEqual(attrs.asDict(), {'startA': {'href': 'blah', 'tag': 'a', 'empty': False},
+ 'href': 'blah', 'tag': 'a', 'empty': False})
+
+
class FollowedByTest(ParseTestCase):
def runTest(self):
- expr = pp.Word(pp.alphas)("item") + pp.FollowedBy(pp.pyparsing_common.integer("qty"))
+ import pyparsing as pp
+ from pyparsing import pyparsing_common as ppc
+ expr = pp.Word(pp.alphas)("item") + pp.FollowedBy(ppc.integer("qty"))
result = expr.parseString("balloon 99")
print_(result.dump())
self.assertTrue('qty' in result, "failed to capture results name in FollowedBy")
class UnicodeTests(ParseTestCase):
def runTest(self):
import pyparsing as pp
- p_u = pp.pyparsing_unicode
+ ppu = pp.pyparsing_unicode
+ ppc = pp.pyparsing_common
# verify proper merging of ranges by addition
- kanji_printables = p_u.Japanese.Kanji.printables
- katakana_printables = p_u.Japanese.Katakana.printables
- hiragana_printables = p_u.Japanese.Hiragana.printables
- japanese_printables = p_u.Japanese.printables
+ kanji_printables = ppu.Japanese.Kanji.printables
+ katakana_printables = ppu.Japanese.Katakana.printables
+ hiragana_printables = ppu.Japanese.Hiragana.printables
+ japanese_printables = ppu.Japanese.printables
self.assertEqual(set(japanese_printables), set(kanji_printables
+ katakana_printables
+ hiragana_printables),
"failed to construct ranges by merging Japanese types")
# verify proper merging of ranges using multiple inheritance
- cjk_printables = p_u.CJK.printables
+ cjk_printables = ppu.CJK.printables
self.assertEqual(len(cjk_printables), len(set(cjk_printables)),
"CJK contains duplicate characters - all should be unique")
- chinese_printables = p_u.Chinese.printables
- korean_printables = p_u.Korean.printables
+ chinese_printables = ppu.Chinese.printables
+ korean_printables = ppu.Korean.printables
print_(len(cjk_printables), len(set(chinese_printables
+ korean_printables
+ japanese_printables)))
+ japanese_printables)),
"failed to construct ranges by merging Chinese, Japanese and Korean")
- alphas = pp.pyparsing_unicode.Greek.alphas
+ alphas = ppu.Greek.alphas
greet = pp.Word(alphas) + ',' + pp.Word(alphas) + '!'
# input string
"Failed to parse Greek 'Hello, World!' using pyparsing_unicode.Greek.alphas")
# define a custom unicode range using multiple inheritance
- class Turkish_set(pp.pyparsing_unicode.Latin1, pp.pyparsing_unicode.LatinA):
+ class Turkish_set(ppu.Latin1, ppu.LatinA):
pass
self.assertEqual(set(Turkish_set.printables),
- set(pp.pyparsing_unicode.Latin1.printables
- + pp.pyparsing_unicode.LatinA.printables),
+ set(ppu.Latin1.printables + ppu.LatinA.printables),
"failed to construct ranges by merging Latin1 and LatinA (printables)")
self.assertEqual(set(Turkish_set.alphas),
- set(pp.pyparsing_unicode.Latin1.alphas
- + pp.pyparsing_unicode.LatinA.alphas),
+ set(ppu.Latin1.alphas + ppu.LatinA.alphas),
"failed to construct ranges by merging Latin1 and LatinA (alphas)")
self.assertEqual(set(Turkish_set.nums),
- set(pp.pyparsing_unicode.Latin1.nums
- + pp.pyparsing_unicode.LatinA.nums),
+ set(ppu.Latin1.nums + ppu.LatinA.nums),
"failed to construct ranges by merging Latin1 and LatinA (nums)")
key = pp.Word(Turkish_set.alphas)
- value = pp.pyparsing_common.integer | pp.Word(Turkish_set.alphas, Turkish_set.alphanums)
+ value = ppc.integer | pp.Word(Turkish_set.alphas, Turkish_set.alphanums)
EQ = pp.Suppress('=')
key_value = key + EQ + value
self.assertEqual(result.c.c1, 200, "invalid indented block result")
self.assertEqual(result.c.c2.c21, 999, "invalid indented block result")
+
+class IndentedBlockScanTest(ParseTestCase):
+ def get_parser(self):
+ """
+ A valid statement is the word "block:", followed by an indent, followed by the letter A only, or another block
+ """
+ stack = [1]
+ block = pp.Forward()
+ body = pp.indentedBlock(pp.Literal('A') ^ block, indentStack=stack, indent=True)
+ block <<= pp.Literal('block:') + body
+ return block
+
+ def runTest(self):
+ from textwrap import dedent
+
+ # This input string is a perfect match for the parser, so a single match is found
+ p1 = self.get_parser()
+ r1 = list(p1.scanString(dedent("""\
+ block:
+ A
+ """)))
+ self.assertEqual(len(r1), 1)
+
+ # This input string is a perfect match for the parser, except for the letter B instead of A, so this will fail (and should)
+ p2 = self.get_parser()
+ r2 = list(p2.scanString(dedent("""\
+ block:
+ B
+ """)))
+ self.assertEqual(len(r2), 0)
+
+ # This input string contains both string A and string B, and it finds one match (as it should)
+ p3 = self.get_parser()
+ r3 = list(p3.scanString(dedent("""\
+ block:
+ A
+ block:
+ B
+ """)))
+ self.assertEqual(len(r3), 1)
+
+ # This input string contains both string A and string B, but in a different order.
+ p4 = self.get_parser()
+ r4 = list(p4.scanString(dedent("""\
+ block:
+ B
+ block:
+ A
+ """)))
+ self.assertEqual(len(r4), 1)
+
+ # This is the same as case 3, but with nesting
+ p5 = self.get_parser()
+ r5 = list(p5.scanString(dedent("""\
+ block:
+ block:
+ A
+ block:
+ block:
+ B
+ """)))
+ self.assertEqual(len(r5), 1)
+
+ # This is the same as case 4, but with nesting
+ p6 = self.get_parser()
+ r6 = list(p6.scanString(dedent("""\
+ block:
+ block:
+ B
+ block:
+ block:
+ A
+ """)))
+ self.assertEqual(len(r6), 1)
+
+
class ParseResultsWithNameMatchFirst(ParseTestCase):
def runTest(self):
import pyparsing as pp
self.assertEqual(list(expr.parseString('not the bird')['rexp']), 'not the bird'.split())
self.assertEqual(list(expr.parseString('the bird')['rexp']), 'the bird'.split())
+ # test compatibility mode, restoring pre-2.3.1 behavior
+ with AutoReset(pp.__compat__, "collect_all_And_tokens"):
+ pp.__compat__.collect_all_And_tokens = False
+ expr_a = pp.Literal('not') + pp.Literal('the') + pp.Literal('bird')
+ expr_b = pp.Literal('the') + pp.Literal('bird')
+ expr = (expr_a | expr_b)('rexp')
+ expr.runTests("""\
+ not the bird
+ the bird
+ """)
+ self.assertEqual(expr.parseString('not the bird')['rexp'], 'not')
+ self.assertEqual(expr.parseString('the bird')['rexp'], 'the')
+
+
class ParseResultsWithNameOr(ParseTestCase):
def runTest(self):
import pyparsing as pp
self.assertEqual(list(expr.parseString('not the bird')['rexp']), 'not the bird'.split())
self.assertEqual(list(expr.parseString('the bird')['rexp']), 'the bird'.split())
+ expr = (expr_a | expr_b)('rexp')
+ expr.runTests("""\
+ not the bird
+ the bird
+ """)
+ self.assertEqual(list(expr.parseString('not the bird')['rexp']), 'not the bird'.split())
+ self.assertEqual(list(expr.parseString('the bird')['rexp']), 'the bird'.split())
+
+ # test compatibility mode, restoring pre-2.3.1 behavior
+ with AutoReset(pp.__compat__, "collect_all_And_tokens"):
+ pp.__compat__.collect_all_And_tokens = False
+ expr_a = pp.Literal('not') + pp.Literal('the') + pp.Literal('bird')
+ expr_b = pp.Literal('the') + pp.Literal('bird')
+ expr = (expr_a ^ expr_b)('rexp')
+ expr.runTests("""\
+ not the bird
+ the bird
+ """)
+ self.assertEqual(expr.parseString('not the bird')['rexp'], 'not')
+ self.assertEqual(expr.parseString('the bird')['rexp'], 'the')
+
+
class EmptyDictDoesNotRaiseException(ParseTestCase):
def runTest(self):
- if not PY_3:
- print('explain() not supported in Py2')
- return
-
import pyparsing as pp
key = pp.Word(pp.alphas)
try:
print_(key_value_dict.parseString("").dump())
except pp.ParseException as pe:
+ exc = pe
+ if not hasattr(exc, '__traceback__'):
+ # Python 2 compatibility
+ etype, value, traceback = sys.exc_info()
+ exc.__traceback__ = traceback
print_(pp.ParseException.explain(pe))
else:
self.assertTrue(False, "failed to raise exception when matching empty string")
class ExplainExceptionTest(ParseTestCase):
def runTest(self):
- if not PY_3:
- print('explain() not supported in Py2')
- return
-
import pyparsing as pp
expr = pp.Word(pp.nums).setName("int") + pp.Word(pp.alphas).setName("word")
try:
expr.parseString("123 355")
except pp.ParseException as pe:
+ exc = pe
+ if not hasattr(exc, '__traceback__'):
+ # Python 2 compatibility
+ etype, value, traceback = sys.exc_info()
+ exc.__traceback__ = traceback
print_(pp.ParseException.explain(pe, depth=0))
expr = pp.Word(pp.nums).setName("int") - pp.Word(pp.alphas).setName("word")
try:
expr.parseString("123 355 (test using ErrorStop)")
except pp.ParseSyntaxException as pe:
+ exc = pe
+ if not hasattr(exc, '__traceback__'):
+ # Python 2 compatibility
+ etype, value, traceback = sys.exc_info()
+ exc.__traceback__ = traceback
print_(pp.ParseException.explain(pe))
integer = pp.Word(pp.nums).setName("int").addParseAction(lambda t: int(t[0]))
try:
expr.parseString("123 0")
except pp.ParseException as pe:
+ exc = pe
+ if not hasattr(exc, '__traceback__'):
+ # Python 2 compatibility
+ etype, value, traceback = sys.exc_info()
+ exc.__traceback__ = traceback
print_(pp.ParseException.explain(pe))
except Exception as exc:
+ if not hasattr(exc, '__traceback__'):
+ # Python 2 compatibility
+ etype, value, traceback = sys.exc_info()
+ exc.__traceback__ = traceback
print_(pp.ParseException.explain(exc))
raise
def makeTestSuite():
import inspect
suite = TestSuite()
- suite.addTest( PyparsingTestInit() )
+ suite.addTest(PyparsingTestInit())
test_case_classes = ParseTestCase.__subclasses__()
# put classes in order as they are listed in the source code
def makeTestSuiteTemp(classes):
suite = TestSuite()
suite.addTest(PyparsingTestInit())
- for cls in classes:
- suite.addTest(cls())
+ suite.addTests(cls() for cls in classes)
return suite
+# runnable from setup.py using "python setup.py test -s unitTests.suite"
+suite = makeTestSuite()
+
+
if __name__ == '__main__':
testRunner = TextTestRunner()
]
if not testclasses:
- testRunner.run(makeTestSuite())
+ result = testRunner.run(suite)
else:
BUFFER_OUTPUT = False
- testRunner.run(makeTestSuiteTemp(testclasses))
+ result = testRunner.run(makeTestSuiteTemp(testclasses))
+
+ exit(0 if result.wasSuccessful() else 1)
projects / platform / upstream / python3-pyparsing.git / commitdiff