Change Log
==========
-Version 2.4.7 - March, 2020
----------------------------
+Version 3.0.0 -
+---------------
+- A consolidated list of all the changes in the 3.0.0 release can be found in
+ docs/whats_new_in_3_0_0.rst.
+ (https://github.com/pyparsing/pyparsing/blob/master/docs/whats_new_in_3_0_0.rst)
+
+
+Version 3.0.0.final -
+---------------------
+- Added support for python -W warning option to call enable_all_warnings() at startup.
+ Also detects setting of PYPARSINGENABLEALLWARNINGS environment variable to any non-blank
+ value.
+
+- Fixed named results returned by `url` to match fields as they would be parsed
+ using urllib.parse.urlparse.
+
+- Early response to `with_line_numbers` was positive, with some requested enhancements:
+ . added a trailing "|" at the end of each line (to show presence of trailing spaces);
+ can be customized using `eol_mark` argument
+ . added expand_tabs argument, to control calling str.expandtabs (defaults to True
+ to match parseString)
+ . added mark_spaces argument to support display of a printing character in place of
+ spaces, or Unicode symbols for space and tab characters
+ . added mark_control argument to support highlighting of control characters using
+ '.' or Unicode symbols, such as "␍" and "␊".
+
+- Modified helpers common_html_entity and replace_html_entity() to use the HTML
+ entity definitions from html.entities.html5.
+
+- Updated the class diagram in the pyparsing docs directory, along with the supporting
+ .puml file (PlantUML markup) used to create the diagram.
+
+- Added global method `autoname_elements()` to call `set_name()` on all locally
+ defined `ParserElements` that haven't been explicitly named using `set_name()`, using
+ their local variable name. Useful for setting names on multiple elements when
+ creating a railroad diagram.
+
+ a = pp.Literal("a")
+ b = pp.Literal("b").set_name("bbb")
+ pp.autoname_elements()
+
+ `a` will get named "a", while `b` will keep its name "bbb".
+
+
+Version 3.0.0rc2 -
+------------------
+- Added `url` expression to `pyparsing_common`. (Sample code posted by Wolfgang Fahl,
+ very nice!)
+
+ This new expression has been added to the `urlExtractorNew.py` example, to show how
+ it extracts URL fields into separate results names.
+
+- Added method to `pyparsing_test` to help debugging, `with_line_numbers`.
+ Returns a string with line and column numbers corresponding to values shown
+ when parsing with expr.set_debug():
+
+ data = """\
+ A
+ 100"""
+ expr = pp.Word(pp.alphanums).set_name("word").set_debug()
+ print(ppt.with_line_numbers(data))
+ expr[...].parseString(data)
+
+ prints:
+
+ 1
+ 1234567890
+ 1: A
+ 2: 100
+ Match word at loc 3(1,4)
+ A
+ ^
+ Matched word -> ['A']
+ Match word at loc 11(2,7)
+ 100
+ ^
+ Matched word -> ['100']
+
+- Added new example `cuneiform_python.py` to demonstrate creating a new Unicode
+ range, and writing a Cuneiform->Python transformer (inspired by zhpy).
+
+- Fixed issue #272, reported by PhasecoreX, when LineStart() expressions would match
+ input text that was not necessarily at the beginning of a line.
+
+ As part of this fix, two new classes have been added: AtLineStart and AtStringStart.
+ The following expressions are equivalent:
+
+ LineStart() + expr and AtLineStart(expr)
+ StringStart() + expr and AtStringStart(expr)
+
+- Fixed ParseFatalExceptions failing to override normal exceptions or expression
+ matches in MatchFirst expressions. Addresses issue #251, reported by zyp-rgb.
+
+- Fixed bug in which ParseResults replaces a collection type value with an invalid
+ type annotation (as a result of changed behavior in Python 3.9). Addresses issue #276, reported by
+ Rob Shuler, thanks.
+
+- Fixed bug in ParseResults when calling `__getattr__` for special double-underscored
+ methods. Now raises AttributeError for non-existent results when accessing a
+ name starting with '__'. Addresses issue #208, reported by Joachim Metz.
+
+- Modified debug fail messages to include the expression name to make it easier to sync
+ up match vs success/fail debug messages.
+
+
+Version 3.0.0rc1 - September, 2021
+----------------------------------
+- Railroad diagrams have been reformatted:
+ . creating diagrams is easier - call
+
+ expr.create_diagram("diagram_output.html")
+
+ create_diagram() takes 3 arguments:
+ . the filename to write the diagram HTML
+ . optional 'vertical' argument, to specify the minimum number of items in a path
+ to be shown vertically; default=3
+ . optional 'show_results_names' argument, to specify whether results name
+ annotations should be shown; default=False
+ . every expression that gets a name using setName() gets separated out as
+ a separate subdiagram
+ . results names can be shown as annotations to diagram items
+ . Each, FollowedBy, and PrecededBy elements get [ALL], [LOOKAHEAD], and [LOOKBEHIND]
+ annotations
+ . removed annotations for Suppress elements
+ . some diagram cleanup when a grammar contains Forward elements
+ . check out the examples make_diagram.py and railroad_diagram_demo.py
+
+- Type annotations have been added to most public API methods and classes.
+
+- Better exception messages to show full word where an exception occurred.
+
+ Word(alphas, alphanums)[...].parseString("ab1 123", parseAll=True)
+
+ Was:
+ pyparsing.ParseException: Expected end of text, found '1' (at char 4), (line:1, col:5)
+ Now:
+ pyparsing.exceptions.ParseException: Expected end of text, found '123' (at char 4), (line:1, col:5)
+
+- Suppress can be used to suppress text skipped using "...".
+
+ source = "lead in START relevant text END trailing text"
+ start_marker = Keyword("START")
+ end_marker = Keyword("END")
+ find_body = Suppress(...) + start_marker + ... + end_marker
+ print(find_body.parseString(source).dump())
+
+ Prints:
+
+ ['START', 'relevant text ', 'END']
+ - _skipped: ['relevant text ']
+
+- New string constants `identchars` and `identbodychars` to help in defining identifier Word expressions
+
+ Two new module-level strings have been added to help when defining identifiers, `identchars` and `identbodychars`.
+
+ Instead of writing::
+
+ import pyparsing as pp
+ identifier = pp.Word(pp.alphas + "_", pp.alphanums + "_")
+
+ you will be able to write::
+
+ identifier = pp.Word(pp.identchars, pp.identbodychars)
+
+ Those constants have also been added to all the Unicode string classes::
+
+ import pyparsing as pp
+ ppu = pp.pyparsing_unicode
+
+ cjk_identifier = pp.Word(ppu.CJK.identchars, ppu.CJK.identbodychars)
+ greek_identifier = pp.Word(ppu.Greek.identchars, ppu.Greek.identbodychars)
+
+- Added a caseless parameter to the `CloseMatch` class to allow for casing to be
+ ignored when checking for close matches. (Issue #281) (PR by Adrian Edwards, thanks!)
+
+- Fixed bug in Located class when used with a results name. (Issue #294)
+
+- Fixed bug in QuotedString class when the escaped quote string is not a
+ repeated character. (Issue #263)
+
+- parseFile() and create_diagram() methods now will accept pathlib.Path
+ arguments.
+
+
+Version 3.0.0b3 - August, 2021
+------------------------------
+- PEP-8 compatible names are being introduced in pyparsing version 3.0!
+ All methods such as `parseString` have been replaced with the PEP-8
+ compliant name `parse_string`. In addition, arguments such as `parseAll`
+ have been renamed to `parse_all`. For backward-compatibility, synonyms for
+ all renamed methods and arguments have been added, so that existing
+ pyparsing parsers will not break. These synonyms will be removed in a future
+ release.
+
+ In addition, the Optional class has been renamed to Opt, since it clashes
+ with the common typing.Optional type specifier that is used in the Python
+ type annotations. A compatibility synonym is defined for now, but will be
+ removed in a future release.
+
+- HUGE NEW FEATURE - Support for left-recursive parsers!
+ Following the method used in Python's PEG parser, pyparsing now supports
+ left-recursive parsers when left recursion is enabled.
+
+ import pyparsing as pp
+ pp.ParserElement.enable_left_recursion()
+
+ # a common left-recursion definition
+ # define a list of items as 'list + item | item'
+ # BNF:
+ # item_list := item_list item | item
+ # item := word of alphas
+ item_list = pp.Forward()
+ item = pp.Word(pp.alphas)
+ item_list <<= item_list + item | item
+
+ item_list.run_tests("""\
+ To parse or not to parse that is the question
+ """)
+ Prints:
+
+ ['To', 'parse', 'or', 'not', 'to', 'parse', 'that', 'is', 'the', 'question']
+
+ Great work contributed by Max Fischer!
+
+- `delimited_list` now supports an additional flag `allow_trailing_delim`,
+ to optionally parse an additional delimiter at the end of the list.
+ Contributed by Kazantcev Andrey, thanks!
+
+- Removed internal comparison of results values against b"", which
+ raised a BytesWarning when run with `python -bb`. Fixes issue #271 reported
+ by Florian Bruhin, thank you!
+
+- Fixed STUDENTS table in sql2dot.py example, fixes issue #261 reported by
+ legrandlegrand - much better.
+
+- Python 3.5 will not be supported in the pyparsing 3 releases. This will allow
+ for future pyparsing releases to add parameter type annotations, and to take
+ advantage of dict key ordering in internal results name tracking.
+
+
+Version 3.0.0b2 - December, 2020
+--------------------------------
+- API CHANGE
+ `locatedExpr` is being replaced by the class `Located`. `Located` has the same
+ constructor interface as `locatedExpr`, but fixes bugs in the returned
+ `ParseResults` when the searched expression contains multiple tokens, or
+ has internal results names.
+
+ `locatedExpr` is deprecated, and will be removed in a future release.
+
+
+Version 3.0.0b1 - November, 2020
+--------------------------------
+- API CHANGE
+ Diagnostic flags have been moved to an enum, `pyparsing.Diagnostics`, and
+ they are enabled through module-level methods:
+ - `pyparsing.enable_diag()`
+ - `pyparsing.disable_diag()`
+ - `pyparsing.enable_all_warnings()`
+
+- API CHANGE
+ Most previous `SyntaxWarnings` that were warned when using pyparsing
+ classes incorrectly have been converted to `TypeError` and `ValueError` exceptions,
+ consistent with Python calling conventions. All warnings warned by diagnostic
+ flags have been converted from `SyntaxWarnings` to `UserWarnings`.
+
+- To support parsers that are intended to generate native Python collection
+ types such as lists and dicts, the `Group` and `Dict` classes now accept an
+ additional boolean keyword argument `aslist` and `asdict` respectively. See
+ the `jsonParser.py` example in the `pyparsing/examples` source directory for
+ how to return types as `ParseResults` and as Python collection types, and the
+ distinctions in working with the different types.
+
+ In addition parse actions that must return a value of list type (which would
+ normally be converted internally to a ParseResults) can override this default
+ behavior by returning their list wrapped in the new `ParseResults.List` class:
+
+ # this parse action tries to return a list, but pyparsing
+ # will convert to a ParseResults
+ def return_as_list_but_still_get_parse_results(tokens):
+ return tokens.asList()
+
+ # this parse action returns the tokens as a list, and pyparsing will
+ # maintain its list type in the final parsing results
+ def return_as_list(tokens):
+ return ParseResults.List(tokens.asList())
+
+ This is the mechanism used internally by the `Group` class when defined
+ using `aslist=True`.
+
+- A new `IndentedBlock` class is introduced, to eventually replace the
+ current `indentedBlock` helper method. The interface is largely the same,
+ however, the new class manages its own internal indentation stack, so
+ it is no longer necessary to maintain an external `indentStack` variable.
+
+- API CHANGE
+ Added `cache_hit` keyword argument to debug actions. Previously, if packrat
+ parsing was enabled, the debug methods were not called in the event of cache
+ hits. Now these methods will be called, with an added argument
+ `cache_hit=True`.
+
+ If you are using packrat parsing and enable debug on expressions using a
+ custom debug method, you can add the `cache_hit=False` keyword argument,
+ and your method will be called on packrat cache hits. If you choose not
+ to add this keyword argument, the debug methods will fail silently,
+ behaving as they did previously.
+
+- When using `setDebug` with packrat parsing enabled, packrat cache hits will
+ now be included in the output, shown with a leading '*'. (Previously, cache
+ hits and responses were not included in debug output.) For those using custom
+ debug actions, see the previous item regarding an optional API change
+ for those methods.
+
+- `setDebug` output will also show more details about what expression
+ is about to be parsed (the current line of text being parsed, and
+ the current parse position):
+
+ Match integer at loc 0(1,1)
+ 1 2 3
+ ^
+ Matched integer -> ['1']
+
+ The current debug location will also be indicated after whitespace
+ has been skipped (was previously inconsistent, reported in Issue #244,
+ by Frank Goyens, thanks!).
+
+- Modified the repr() output for `ParseResults` to include the class
+ name as part of the output. This is to clarify for new pyparsing users
+ who misread the repr output as a tuple of a list and a dict. pyparsing
+ results will now read like:
+
+ ParseResults(['abc', 'def'], {'qty': 100}]
+
+ instead of just:
+
+ (['abc', 'def'], {'qty': 100}]
+
+- Fixed bugs in Each when passed OneOrMore or ZeroOrMore expressions:
+ . first expression match could be enclosed in an extra nesting level
+ . out-of-order expressions now handled correctly if mixed with required
+ expressions
+ . results names are maintained correctly for these expressions
+
+- Fixed traceback trimming, and added `ParserElement.verbose_traceback`
+ save/restore to `reset_pyparsing_context()`.
+
+- Default string for `Word` expressions now also include indications of
+ `min` and `max` length specification, if applicable, similar to regex length
+ specifications:
+
+ Word(alphas) -> "W:(A-Za-z)"
+ Word(nums) -> "W:(0-9)"
+ Word(nums, exact=3) -> "W:(0-9){3}"
+ Word(nums, min=2) -> "W:(0-9){2,...}"
+ Word(nums, max=3) -> "W:(0-9){1,3}"
+ Word(nums, min=2, max=3) -> "W:(0-9){2,3}"
+
+ For expressions of the `Char` class (similar to `Word(..., exact=1)`, the expression
+ is simply the character range in parentheses:
+
+ Char(nums) -> "(0-9)"
+ Char(alphas) -> "(A-Za-z)"
+
+- Removed `copy()` override in `Keyword` class which did not preserve definition
+ of ident chars from the original expression. PR #233 submitted by jgrey4296,
+ thanks!
+
+- In addition to `pyparsing.__version__`, there is now also a `pyparsing.__version_info__`,
+ following the same structure and field names as in `sys.version_info`.
+
+
+Version 3.0.0a2 - June, 2020
+----------------------------
+- Summary of changes for 3.0.0 can be found in "What's New in Pyparsing 3.0.0"
+ documentation.
+
+- API CHANGE
+ Changed result returned when parsing using countedArray,
+ the array items are no longer returned in a doubly-nested
+ list.
+
+- An excellent new enhancement is the new railroad diagram
+ generator for documenting pyparsing parsers:
+
+ import pyparsing as pp
+ from pyparsing.diagram import to_railroad, railroad_to_html
+ from pathlib import Path
+
+ # define a simple grammar for parsing street addresses such
+ # as "123 Main Street"
+ # number word...
+ number = pp.Word(pp.nums).setName("number")
+ name = pp.Word(pp.alphas).setName("word")[1, ...]
+
+ parser = number("house_number") + name("street")
+ parser.setName("street address")
+
+ # construct railroad track diagram for this parser and
+ # save as HTML
+ rr = to_railroad(parser)
+ Path('parser_rr_diag.html').write_text(railroad_to_html(rr))
+
+ Very nice work provided by Michael Milton, thanks a ton!
+
+- Enhanced default strings created for Word expressions, now showing
+ string ranges if possible. `Word(alphas)` would formerly
+ print as `W:(ABCD...)`, now prints as `W:(A-Za-z)`.
+
+- Added ignoreWhitespace(recurse:bool = True) and added a
+ recurse argument to leaveWhitespace, both added to provide finer
+ control over pyparsing's whitespace skipping. Also contributed
+ by Michael Milton.
+
+- The unicode range definitions for the various languages were
+ recalculated by interrogating the unicodedata module by character
+ name, selecting characters that contained that language in their
+ Unicode name. (Issue #227)
+
+ Also, pyparsing_unicode.Korean was renamed to Hangul (Korean
+ is also defined as a synonym for compatibility).
+
+- Enhanced ParseResults dump() to show both results names and list
+ subitems. Fixes bug where adding a results name would hide
+ lower-level structures in the ParseResults.
+
+- Added new __diag__ warnings:
+
+ "warn_on_parse_using_empty_Forward" - warns that a Forward
+ has been included in a grammar, but no expression was
+ attached to it using '<<=' or '<<'
+
+ "warn_on_assignment_to_Forward" - warns that a Forward has
+ been created, but was probably later overwritten by
+ erroneously using '=' instead of '<<=' (this is a common
+ mistake when using Forwards)
+ (**currently not working on PyPy**)
+
+- Added ParserElement.recurse() method to make it simpler for
+ grammar utilities to navigate through the tree of expressions in
+ a pyparsing grammar.
+
+- Fixed bug in ParseResults repr() which showed all matching
+ entries for a results name, even if listAllMatches was set
+ to False when creating the ParseResults originally. Reported
+ by Nicholas42 on GitHub, good catch! (Issue #205)
+
+- Modified refactored modules to use relative imports, as
+ pointed out by setuptools project member jaraco, thank you!
+
+- Off-by-one bug found in the roman_numerals.py example, a bug
+ that has been there for about 14 years! PR submitted by
+ Jay Pedersen, nice catch!
+
+- A simplified Lua parser has been added to the examples
+ (lua_parser.py).
+
+- Added make_diagram.py to the examples directory to demonstrate
+ creation of railroad diagrams for selected pyparsing examples.
+ Also restructured some examples to make their parsers importable
+ without running their embedded tests.
+
+
+Version 3.0.0a1 - April, 2020
+-----------------------------
+- Removed Py2.x support and other deprecated features. Pyparsing
+ now requires Python 3.5 or later. If you are using an earlier
+ version of Python, you must use a Pyparsing 2.4.x version
+
+ Deprecated features removed:
+ . ParseResults.asXML() - if used for debugging, switch
+ to using ParseResults.dump(); if used for data transfer,
+ use ParseResults.asDict() to convert to a nested Python
+ dict, which can then be converted to XML or JSON or
+ other transfer format
+
+ . operatorPrecedence synonym for infixNotation -
+ convert to calling infixNotation
+
+ . commaSeparatedList - convert to using
+ pyparsing_common.comma_separated_list
+
+ . upcaseTokens and downcaseTokens - convert to using
+ pyparsing_common.upcaseTokens and downcaseTokens
+
+ . __compat__.collect_all_And_tokens will not be settable to
+ False to revert to pre-2.3.1 results name behavior -
+ review use of names for MatchFirst and Or expressions
+ containing And expressions, as they will return the
+ complete list of parsed tokens, not just the first one.
+ Use `__diag__.warn_multiple_tokens_in_named_alternation`
+ to help identify those expressions in your parsers that
+ will have changed as a result.
+
+- Removed support for running `python setup.py test`. The setuptools
+ maintainers consider the test command deprecated (see
+ <https://github.com/pypa/setuptools/issues/1684>). To run the Pyparsing test,
+ use the command `tox`.
+
+- API CHANGE:
+ The staticmethod `ParseException.explain` has been moved to
+ `ParseBaseException.explain_exception`, and a new `explain` instance
+ method added to ParseBaseException. This will make calls to `explain`
+ much more natural:
+
+ try:
+ expr.parseString("...")
+ except ParseException as pe:
+ print(pe.explain())
+
+- POTENTIAL API CHANGE:
+ ZeroOrMore expressions that have results names will now
+ include empty lists for their name if no matches are found.
+ Previously, no named result would be present. Code that tested
+ for the presence of any expressions using "if name in results:"
+ will now always return True. This code will need to change to
+ "if name in results and results[name]:" or just
+ "if results[name]:". Also, any parser unit tests that check the
+ asDict() contents will now see additional entries for parsers
+ having named ZeroOrMore expressions, whose values will be `[]`.
+
+- POTENTIAL API CHANGE:
+ Fixed a bug in which calls to ParserElement.setDefaultWhitespaceChars
+ did not change whitespace definitions on any pyparsing built-in
+ expressions defined at import time (such as quotedString, or those
+ defined in pyparsing_common). This would lead to confusion when
+ built-in expressions would not use updated default whitespace
+ characters. Now a call to ParserElement.setDefaultWhitespaceChars
+ will also go and update all pyparsing built-ins to use the new
+ default whitespace characters. (Note that this will only modify
+ expressions defined within the pyparsing module.) Prompted by
+ work on a StackOverflow question posted by jtiai.
+
+- Expanded __diag__ and __compat__ to actual classes instead of
+ just namespaces, to add some helpful behavior:
+ - enable() and .disable() methods to give extra
+ help when setting or clearing flags (detects invalid
+ flag names, detects when trying to set a __compat__ flag
+ that is no longer settable). Use these methods now to
+ set or clear flags, instead of directly setting to True or
+ False.
+
+ import pyparsing as pp
+ pp.__diag__.enable("warn_multiple_tokens_in_named_alternation")
+
+ - __diag__.enable_all_warnings() is another helper that sets
+ all "warn*" diagnostics to True.
+
+ pp.__diag__.enable_all_warnings()
+
+ - added new warning, "warn_on_match_first_with_lshift_operator" to
+ warn when using '<<' with a '|' MatchFirst operator, which will
+ create an unintended expression due to precedence of operations.
+
+ Example: This statement will erroneously define the `fwd` expression
+ as just `expr_a`, even though `expr_a | expr_b` was intended,
+ since '<<' operator has precedence over '|':
+
+ fwd << expr_a | expr_b
+
+ To correct this, use the '<<=' operator (preferred) or parentheses
+ to override operator precedence:
+
+ fwd <<= expr_a | expr_b
+ or
+ fwd << (expr_a | expr_b)
+
+- Cleaned up default tracebacks when getting a ParseException when calling
+ parseString. Exception traces should now stop at the call in parseString,
+ and not include the internal traceback frames. (If the full traceback
+ is desired, then set ParserElement.verbose_traceback to True.)
+
+- Fixed FutureWarnings that sometimes are raised when '[' passed as a
+ character to Word.
+
+- New namespace, assert methods and classes added to support writing
+ unit tests.
+ - assertParseResultsEquals
+ - assertParseAndCheckList
+ - assertParseAndCheckDict
+ - assertRunTestResults
+ - assertRaisesParseException
+ - reset_pyparsing_context context manager, to restore pyparsing
+ config settings
+
+- Enhanced error messages and error locations when parsing fails on
+ the Keyword or CaselessKeyword classes due to the presence of a
+ preceding or trailing keyword character. Surfaced while
+ working with metaperl on issue #201.
+
+- Enhanced the Regex class to be compatible with re's compiled with the
+ re-equivalent regex module. Individual expressions can be built with
+ regex compiled expressions using:
+
+ import pyparsing as pp
+ import regex
+
+ # would use regex for this expression
+ integer_parser = pp.Regex(regex.compile(r'\d+'))
+
+ Inspired by PR submitted by bjrnfrdnnd on GitHub, very nice!
+
+- Fixed handling of ParseSyntaxExceptions raised as part of Each
+ expressions, when sub-expressions contain '-' backtrack
+ suppression. As part of resolution to a question posted by John
+ Greene on StackOverflow.
+
+- Potentially *huge* performance enhancement when parsing Word
+ expressions built from pyparsing_unicode character sets. Word now
+ internally converts ranges of consecutive characters to regex
+ character ranges (converting "0123456789" to "0-9" for instance),
+ resulting in as much as 50X improvement in performance! Work
+ inspired by a question posted by Midnighter on StackOverflow.
+
+- Improvements in select_parser.py, to include new SQL syntax
+ from SQLite. PR submitted by Robert Coup, nice work!
+
+- Fixed bug in PrecededBy which caused infinite recursion, issue #127
+ submitted by EdwardJB.
+
+- Fixed bug in CloseMatch where end location was incorrectly
+ computed; and updated partial_gene_match.py example.
+
+- Fixed bug in indentedBlock with a parser using two different
+ types of nested indented blocks with different indent values,
+ but sharing the same indent stack, submitted by renzbagaporo.
+
+- Fixed bug in Each when using Regex, when Regex expression would
+ get parsed twice; issue #183 submitted by scauligi, thanks!
+
+- BigQueryViewParser.py added to examples directory, PR submitted
+ by Michael Smedberg, nice work!
+
+- booleansearchparser.py added to examples directory, PR submitted
+ by xecgr. Builds on searchparser.py, adding support for '*'
+ wildcards and non-Western alphabets.
+
+- Fixed bug in delta_time.py example, when using a quantity
+ of seconds/minutes/hours/days > 999.
+
+- Fixed bug in regex definitions for real and sci_real expressions in
+ pyparsing_common. Issue #194, reported by Michael Wayne Goodman, thanks!
+
+- Fixed FutureWarning raised beginning in Python 3.7 for Regex expressions
+ containing '[' within a regex set.
+
+- Minor reformatting of output from runTests to make embedded
+ comments more visible.
+
+- And finally, many thanks to those who helped in the restructuring
+ of the pyparsing code base as part of this release. Pyparsing now
+ has more standard package structure, more standard unit tests,
+ and more standard code formatting (using black). Special thanks
+ to jdufresne, klahnakoski, mattcarmody, and ckeygusuz, to name just
+ a few.
+
+
+Version 2.4.7 - March, 2020 (April, actually)
+---------------------------------------------
- Backport of selected fixes from 3.0.0 work:
. Each bug with Regex expressions
. And expressions not properly constructing with generator
Version 2.4.5 - November, 2019
------------------------------
-- Fixed encoding when setup.py reads README.rst to include the
- project long description when uploading to PyPI. A stray
- unicode space in README.rst prevented the source install on
- systems whose default encoding is not 'utf-8'.
+- NOTE: final release compatible with Python 2.x.
+
+- Fixed issue with reading README.rst as part of setup.py's
+ initialization of the project's long_description, with a
+ non-ASCII space character causing errors when installing from
+ source on platforms where UTF-8 is not the default encoding.
Version 2.4.4 - November, 2019
--------------------------------
- Unresolved symbol reference in 2.4.3 release was masked by stdout
- buffering in unit tests, thanks for the prompt heads-up, Ned
+ buffering in unit tests, thanks for the prompt heads-up, Ned
Batchelder!
Michael Clerx for the assist. (Addresses issue #123)
- Fixed bug in indentedBlock where a block that ended at the end
- of the input string could cause pyaprsing to loop forever. Raised
+ of the input string could cause pyparsing to loop forever. Raised
as part of discussion on StackOverflow with geckos.
- Backports from pyparsing 3.0.0:
- API change adding support for `expr[...]` - the original
code in 2.4.1 incorrectly implemented this as OneOrMore.
- Code using this feature under this relase should explicitly
+ Code using this feature under this release should explicitly
use `expr[0, ...]` for ZeroOrMore and `expr[1, ...]` for
OneOrMore. In 2.4.2 you will be able to write `expr[...]`
equivalent to `ZeroOrMore(expr)`.
- warn_ungrouped_named_tokens_in_collection - flag to enable warnings when a results
name is defined on a containing expression with ungrouped subexpressions that also
have results names (default=True)
- - warn_name_set_on_empty_Forward - flag to enable warnings whan a Forward is defined
+ - warn_name_set_on_empty_Forward - flag to enable warnings when a Forward is defined
with a results name, but has no contents defined (default=False)
- - warn_on_multiple_string_args_to_oneof - flag to enable warnings whan oneOf is
+ - warn_on_multiple_string_args_to_oneof - flag to enable warnings when oneOf is
incorrectly called with multiple str arguments (default=True)
- enable_debug_on_named_expressions - flag to auto-enable debug on all subsequent
calls to ParserElement.setName() (default=False)
repr form provides important information when debugging parse actions.
-Verison 2.1.5 - June, 2016
+Version 2.1.5 - June, 2016
------------------------------
- Added ParserElement.split() generator method, similar to re.split().
Includes optional arguments maxsplit (to limit the number of splits),
- Added "pprint()" method to ParseResults, to simplify troubleshooting
and prettified output. Now instead of importing the pprint module
and then writing "pprint.pprint(result)", you can just write
- "result.pprint()". This method also accepts addtional positional and
+ "result.pprint()". This method also accepts additional positional and
keyword arguments (such as indent, width, etc.), which get passed
through directly to the pprint method
(see https://docs.python.org/2/library/pprint.html#pprint.pprint).
- Fixed bug in srange when using '\x###' hex character codes.
-- Addeed optional 'intExpr' argument to countedArray, so that you
+- Added optional 'intExpr' argument to countedArray, so that you
can define your own expression that will evaluate to an integer,
to be used as the count for the following elements. Allows you
to define a countedArray with the count given in hex, for example,
programs, at some cost to performance (3-5%). Suggested by bca48150 on
the pyparsing wiki, thanks!
-- Enhanced the documentation describing the vagaries and idiosyncracies
+- Enhanced the documentation describing the vagaries and idiosyncrasies
of parsing strings with embedded tabs, and the impact on:
. parse actions
. scanString
and Python features.
- [submit an issue](https://github.com/pyparsing/pyparsing/issues) - If you have a problem with pyparsing that looks
- like an actual bug, or have an idea for a feature to add to pyaprsing please submit an issue on GitHub. Some
+ like an actual bug, or have an idea for a feature to add to pyparsing please submit an issue on GitHub. Some
pyparsing behavior may be counter-intuitive, so try to review some of the other resources first, or some of the
other open and closed issues. Or post your question on SO or reddit. But don't wait until you are desperate and
frustrated - just ask! :)
If you are considering proposing updates to pyparsing, please bear in mind the following guidelines.
-Please review [_The Zen of Pyparsing_ and _The Zen of Pyparsing
-Development_](https://github.com/pyparsing/pyparsing/wiki/Zen)
-article on the pyparsing wiki, to get a general feel for the historical and future approaches to pyparsing's
+Please review [_The Zen of Pyparsing_ and _The Zen of Pyparsing
+Development_](https://github.com/pyparsing/pyparsing/wiki/Zen)
+article on the pyparsing wiki, to get a general feel for the historical and future approaches to pyparsing's
design, and intended developer experience as an embedded DSL.
## Some design points
-- Minimize additions to the module namespace. Over time, pyparsing's namespace has acquired a *lot* of names.
- New features have been encapsulated into namespace classes to try to hold back the name flooding when importing
+- Minimize additions to the module namespace. Over time, pyparsing's namespace has acquired a *lot* of names.
+ New features have been encapsulated into namespace classes to try to hold back the name flooding when importing
pyparsing.
-- New operator overloads will need to show broad applicability.
+- New operator overloads for ParserElement will need to show broad applicability, and should be related to
+ parser construction.
- Performance tuning should focus on parse time performance. Optimizing parser definition performance is secondary.
-- New external dependencies will require substantial justification, and if included, will need to be guarded for
+- New external dependencies will require substantial justification, and if included, will need to be guarded for
`ImportError`s raised if the external module is not installed.
## Some coding points
These coding styles are encouraged whether submitting code for core pyparsing or for submitting an example.
-- PEP8 - at this time, pyparsing is very non-compliant with many PEP8 guidelines, especially those regarding
+- PEP8 - pyparsing has historically been very non-compliant with many PEP8 guidelines, especially those regarding
name casing. I had just finished several years of Java and Smalltalk development, and camel case seemed to be the
- future trend in coding styles. There are plans to convert these names to PEP8-conformant snake case, but this will
- be done over several releases to provide a migration path for current pyparsing-dependent applications. See more
- information at the [PEP8 wiki page](https://github.com/pyparsing/pyparsing/wiki/PEP-8-planning).
-
- If you wish to submit a new example, please follow PEP8 name and coding guidelines. Example code must be available
- for distribution with the rest of pyparsing under the MIT open source license.
+ future trend in coding styles. As of version 3.0.0, pyparsing is moving over to PEP8 naming, while maintaining
+ compatibility with existing parser code by defining synonyms using the legacy names. These names will be
+ retained until a future release (probably 4.0), to provide a migration path for current pyparsing-dependent
+ applications - DO NOT MODIFY OR REMOVE THESE NAMES.
+ See more information at the [PEP8 wiki page](https://github.com/pyparsing/pyparsing/wiki/PEP-8-planning).
+
+ If you wish to submit a new example, please follow PEP8 name and coding guidelines, and use the black formatter
+ to auto-format code. Example code must be available for distribution with the rest of pyparsing under the MIT
+ open source license.
- No backslashes for line continuations.
Continuation lines for expressions in ()'s should start with the continuing operator:
+ some_other_long_thing
+ even_another_long_thing)
-- Changes to core pyparsing must be compatible back to Py3.5 without conditionalizing. Later Py3 features may be
+- Maximum line length is 120 characters. (Black will override this.)
+
+- Changes to core pyparsing must be compatible back to Py3.6 without conditionalizing. Later Py3 features may be
used in examples by way of illustration.
- str.format() statements should use named format arguments (unless this proves to be a slowdown at parse time).
-- List, tuple, and dict literals should include a trailing comma after the last element, which reduces changeset
+- List, tuple, and dict literals should include a trailing comma after the last element, which reduces changeset
clutter when another element gets added to the end.
- Examples should import pyparsing and the common namespace classes as:
ppc = pp.pyparsing_common
ppu = pp.pyparsing_unicode
+ Submitted examples *must* be Python 3 compatible.
+
- Where possible use operators to create composite parse expressions:
expr = expr_a + expr_b | expr_c
-
+
instead of:
-
+
expr = pp.MatchFirst([pp.And([expr_a, expr_b]), expr_c])
Exception: if using a generator to create an expression:
-
+
import keyword
python_keywords = keyword.kwlist
- any_keyword = pp.MatchFirst(pp.Keyword(kw)
+ any_keyword = pp.MatchFirst(pp.Keyword(kw)
for kw in python_keywords))
-- Learn [The Classic Blunders](https://github.com/pyparsing/pyparsing/wiki/The-Classic-Blunders) and
+- Learn [Common Pitfalls When Writing Parsers](https://github.com/pyparsing/pyparsing/wiki/Common-Pitfalls-When-Writing-Parsers) and
how to avoid them when developing new examples.
-- New features should be accompanied with updates to unitTests.py and a bullet in the CHANGES file.
+- New features should be accompanied by updates to unitTests.py and a bullet in the CHANGES file.
-include pyparsing.py
-include HowToUsePyparsing.rst pyparsingClassDiagram.*
-include README.md CODE_OF_CONDUCT.rst CHANGES LICENSE CONTRIBUTING.md modules.rst
+include pyparsing/diagram/*.jinja2
+include README.rst CODE_OF_CONDUCT.rst CHANGES LICENSE CONTRIBUTING.md
include examples/*.py examples/Setup.ini examples/*.dfm examples/*.ics examples/*.html examples/*.h examples/*.g examples/statemachine/*
recursive-include docs *
prune docs/_build/*
-recursive-include test *
-include setup.py simple_unit_tests.py unitTests.py
+recursive-include tests *
+prune tests/__pycache__
+include setup.py tox.ini
-Metadata-Version: 1.2
+Metadata-Version: 2.1
Name: pyparsing
-Version: 2.4.7
+Version: 3.0.0
Summary: Python parsing module
Home-page: https://github.com/pyparsing/pyparsing/
Author: Paul McGuire
-Author-email: ptmcg@users.sourceforge.net
+Author-email: ptmcg.gm+pyparsing@gmail.com
License: MIT License
Download-URL: https://pypi.org/project/pyparsing/
Description: PyParsing -- A Python Parsing Module
====================================
- |Build Status|
+ |Build Status| |Coverage|
Introduction
============
*[Since first writing this description of pyparsing in late 2003, this
technique for developing parsers has become more widespread, under the
- name Parsing Expression Grammars - PEGs. See more information on PEGs at*
- https://en.wikipedia.org/wiki/Parsing_expression_grammar *.]*
+ name Parsing Expression Grammars - PEGs. See more information on PEGs*
+ `here <https://en.wikipedia.org/wiki/Parsing_expression_grammar>`__
+ *.]*
Here is a program to parse ``"Hello, World!"`` (or any greeting of the form
``"salutation, addressee!"``):
self-explanatory class names, and the use of '+', '|' and '^' operator
definitions.
- The parsed results returned from ``parseString()`` can be accessed as a
+ The parsed results returned from ``parseString()`` is a collection of type
+ ``ParseResults``, which can be accessed as a
nested list, a dictionary, or an object with named attributes.
The pyparsing module handles some of the problems that are typically
=============
There are many examples in the online docstrings of the classes
- and methods in pyparsing. You can find them compiled into online docs
- at https://pyparsing-docs.readthedocs.io/en/latest/. Additional
+ and methods in pyparsing. You can find them compiled into `online docs <https://pyparsing-docs.readthedocs.io/en/latest/>`__. Additional
documentation resources and project info are listed in the online
- GitHub wiki, at https://github.com/pyparsing/pyparsing/wiki. An
- entire directory of examples is at
- https://github.com/pyparsing/pyparsing/tree/master/examples.
+ `GitHub wiki <https://github.com/pyparsing/pyparsing/wiki>`__. An
+ entire directory of examples can be found `here <https://github.com/pyparsing/pyparsing/tree/master/examples>`__.
License
=======
- MIT License. See header of pyparsing.py
+ MIT License. See header of the `pyparsing.py <https://github.com/pyparsing/pyparsing/blob/master/pyparsing/__init__.py#L1-L23>`__ file.
History
=======
- See CHANGES file.
+ See `CHANGES <https://github.com/pyparsing/pyparsing/blob/master/CHANGES>`__ file.
- .. |Build Status| image:: https://travis-ci.org/pyparsing/pyparsing.svg?branch=master
- :target: https://travis-ci.org/pyparsing/pyparsing
+ .. |Build Status| image:: https://travis-ci.com/pyparsing/pyparsing.svg?branch=master
+ :target: https://travis-ci.com/pyparsing/pyparsing
+ .. |Coverage| image:: https://codecov.io/gh/pyparsing/pyparsing/branch/master/graph/badge.svg
+ :target: https://codecov.io/gh/pyparsing/pyparsing
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 2
-Classifier: Programming Language :: Python :: 2.6
-Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.3
-Classifier: Programming Language :: Python :: 3.4
-Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
-Requires-Python: >=2.6, !=3.0.*, !=3.1.*, !=3.2.*
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Requires-Python: >=3.6
+Description-Content-Type: text/x-rst
+Provides-Extra: diagrams
PyParsing -- A Python Parsing Module
====================================
-|Build Status|
+|Build Status| |Coverage|
Introduction
============
*[Since first writing this description of pyparsing in late 2003, this
technique for developing parsers has become more widespread, under the
-name Parsing Expression Grammars - PEGs. See more information on PEGs at*
-https://en.wikipedia.org/wiki/Parsing_expression_grammar *.]*
+name Parsing Expression Grammars - PEGs. See more information on PEGs*
+`here <https://en.wikipedia.org/wiki/Parsing_expression_grammar>`__
+*.]*
Here is a program to parse ``"Hello, World!"`` (or any greeting of the form
``"salutation, addressee!"``):
self-explanatory class names, and the use of '+', '|' and '^' operator
definitions.
-The parsed results returned from ``parseString()`` can be accessed as a
+The parsed results returned from ``parseString()`` is a collection of type
+``ParseResults``, which can be accessed as a
nested list, a dictionary, or an object with named attributes.
The pyparsing module handles some of the problems that are typically
=============
There are many examples in the online docstrings of the classes
-and methods in pyparsing. You can find them compiled into online docs
-at https://pyparsing-docs.readthedocs.io/en/latest/. Additional
+and methods in pyparsing. You can find them compiled into `online docs <https://pyparsing-docs.readthedocs.io/en/latest/>`__. Additional
documentation resources and project info are listed in the online
-GitHub wiki, at https://github.com/pyparsing/pyparsing/wiki. An
-entire directory of examples is at
-https://github.com/pyparsing/pyparsing/tree/master/examples.
+`GitHub wiki <https://github.com/pyparsing/pyparsing/wiki>`__. An
+entire directory of examples can be found `here <https://github.com/pyparsing/pyparsing/tree/master/examples>`__.
License
=======
-MIT License. See header of pyparsing.py
+MIT License. See header of the `pyparsing.py <https://github.com/pyparsing/pyparsing/blob/master/pyparsing/__init__.py#L1-L23>`__ file.
History
=======
-See CHANGES file.
+See `CHANGES <https://github.com/pyparsing/pyparsing/blob/master/CHANGES>`__ file.
-.. |Build Status| image:: https://travis-ci.org/pyparsing/pyparsing.svg?branch=master
- :target: https://travis-ci.org/pyparsing/pyparsing
+.. |Build Status| image:: https://travis-ci.com/pyparsing/pyparsing.svg?branch=master
+ :target: https://travis-ci.com/pyparsing/pyparsing
+.. |Coverage| image:: https://codecov.io/gh/pyparsing/pyparsing/branch/master/graph/badge.svg
+ :target: https://codecov.io/gh/pyparsing/pyparsing
==========================
:author: Paul McGuire
-:address: ptmcg@users.sourceforge.net
+:address: ptmcg.pm+pyparsing@gmail.com
-:revision: 2.0.1a
-:date: July, 2013 (minor update August, 2018)
+:revision: 3.0.0
+:date: October, 2021
-:copyright: Copyright |copy| 2003-2013 Paul McGuire.
+:copyright: Copyright |copy| 2003-2021 Paul McGuire.
.. |copy| unicode:: 0xA9
.. contents:: :depth: 4
Note: While this content is still valid, there are more detailed
-descriptions and examples at the online doc server at
-https://pythonhosted.org/pyparsing/pyparsing-module.html
+descriptions and extensive examples at the `online doc server
+<https://pyparsing-docs.readthedocs.io/en/latest/pyparsing.html>`_, and
+in the online help for the various pyparsing classes and methods (viewable
+using the Python interpreter's built-in ``help()`` function). You will also
+find many example scripts in the `examples <https://github.com/pyparsing/pyparsing/tree/master/examples>`_
+directory of the pyparsing GitHub repo.
+
+-----------
+
+**Note**: *In pyparsing 3.0, many method and function names which were
+originally written using camelCase have been converted to PEP8-compatible
+snake_case. So ``parseString()`` is being renamed to ``parse_string()``,
+``delimitedList`` to ``delimited_list``, and so on. You may see the old
+names in legacy parsers, and they will be supported for a time with
+synonyms, but the synonyms will be removed in a future release.*
+
+*If you are using this documentation, but working with a 2.4.x version of pyparsing,
+you'll need to convert methods and arguments from the documented snake_case
+names to the legacy camelCase names. In pyparsing 3.0.x, both forms are
+supported, but the legacy forms are deprecated; they will be dropped in a
+future release.*
+
+-----------
+
Steps to follow
===============
To parse an incoming data string, the client code must follow these steps:
1. First define the tokens and patterns to be matched, and assign
- this to a program variable. Optional results names or parsing
+ this to a program variable. Optional results names or parse
actions can also be defined at this time.
-2. Call ``parseString()`` or ``scanString()`` on this variable, passing in
+2. Call ``parse_string()`` or ``scan_string()`` on this variable, passing in
the string to
be parsed. During the matching process, whitespace between
tokens is skipped by default (although this can be changed).
When token matches occur, any defined parse action methods are
called.
-3. Process the parsed results, returned as a list of strings.
- Matching results may also be accessed as named attributes of
+3. Process the parsed results, returned as a ParseResults_ object.
+ The ParseResults_ object can be accessed as if it were a list of
+ strings. Matching results may also be accessed as named attributes of
the returned results, if names are defined in the definition of
- the token pattern, using ``setResultsName()``.
+ the token pattern, using ``set_results_name()``.
Hello, World!
-------------
-The following complete Python program will parse the greeting "Hello, World!",
+The following complete Python program will parse the greeting ``"Hello, World!"``,
or any other greeting of the form "<salutation>, <addressee>!"::
- from pyparsing import Word, alphas
+ import pyparsing as pp
- greet = Word(alphas) + "," + Word(alphas) + "!"
- greeting = greet.parseString("Hello, World!")
- print greeting
+ greet = pp.Word(pp.alphas) + "," + pp.Word(pp.alphas) + "!"
+ for greeting_str in [
+ "Hello, World!",
+ "Bonjour, Monde!",
+ "Hola, Mundo!",
+ "Hallo, Welt!",
+ ]:
+ greeting = greet.parse_string(greeting_str)
+ print(greeting)
The parsed tokens are returned in the following form::
['Hello', ',', 'World', '!']
+ ['Bonjour', ',', 'Monde', '!']
+ ['Hola', ',', 'Mundo', '!']
+ ['Hallo', ',', 'Welt', '!']
Usage notes
- To keep up the readability of your code, use operators_ such as ``+``, ``|``,
``^``, and ``~`` to combine expressions. You can also combine
- string literals with ParseExpressions - they will be
- automatically converted to Literal objects. For example::
+ string literals with ``ParseExpressions`` - they will be
+ automatically converted to Literal_ objects. For example::
integer = Word(nums) # simple unsigned integer
- variable = Word(alphas, max=1) # single letter variable, such as x, z, m, etc.
- arithOp = Word("+-*/", max=1) # arithmetic operators
- equation = variable + "=" + integer + arithOp + integer # will match "x=2+2", etc.
+ variable = Char(alphas) # single letter variable, such as x, z, m, etc.
+ arith_op = one_of("+ - * /") # arithmetic operators
+ equation = variable + "=" + integer + arith_op + integer # will match "x=2+2", etc.
In the definition of ``equation``, the string ``"="`` will get added as
a ``Literal("=")``, but in a more readable way.
Of course, it is quite simple to extend this example to support more elaborate expressions, with
nesting with parentheses, floating point numbers, scientific notation, and named constants
- (such as ``e`` or ``pi``). See ``fourFn.py``, included in the examples directory.
+ (such as ``e`` or ``pi``). See `fourFn.py <https://github.com/pyparsing/pyparsing/blob/master/examples/fourFn.py>`_,
+ and `simpleArith.py <https://github.com/pyparsing/pyparsing/blob/master/examples/simpleArith.py>`_
+ included in the examples directory.
- To modify pyparsing's default whitespace skipping, you can use one or
more of the following methods:
- - use the static method ``ParserElement.setDefaultWhitespaceChars``
- to override the normal set of whitespace chars (' \t\n'). For instance
+ - use the static method ``ParserElement.set_default_whitespace_chars``
+ to override the normal set of whitespace chars (``' \t\n'``). For instance
when defining a grammar in which newlines are significant, you should
- call ``ParserElement.setDefaultWhitespaceChars(' \t')`` to remove
+ call ``ParserElement.set_default_whitespace_chars(' \t')`` to remove
newline from the set of skippable whitespace characters. Calling
this method will affect all pyparsing expressions defined afterward.
- - call ``leaveWhitespace()`` on individual expressions, to suppress the
+ - call ``leave_whitespace()`` on individual expressions, to suppress the
skipping of whitespace before trying to match the expression
- use ``Combine`` to require that successive expressions must be
- ``expr*3`` is equivalent to ``expr + expr + expr``
- - ``expr[2, 3]`` is equivalent to ``expr + expr + Optional(expr)``
+ - ``expr[2, 3]`` is equivalent to ``expr + expr + Opt(expr)``
- ``expr[n, ...]`` or ``expr[n,]`` is equivalent
to ``expr*n + ZeroOrMore(expr)`` (read as "at least n instances of expr")
occurrences. If this behavior is desired, then write
``expr[..., n] + ~expr``.
-- ``MatchFirst`` expressions are matched left-to-right, and the first
+- MatchFirst_ expressions are matched left-to-right, and the first
match found will skip all later expressions within, so be sure
to define less-specific patterns after more-specific patterns.
- If you are not sure which expressions are most specific, use Or
+ If you are not sure which expressions are most specific, use Or_
expressions (defined using the ``^`` operator) - they will always
match the longest expression, although they are more
compute-intensive.
-- ``Or`` expressions will evaluate all of the specified subexpressions
+- Or_ expressions will evaluate all of the specified subexpressions
to determine which is the "best" match, that is, which matches
the longest string in the input data. In case of a tie, the
- left-most expression in the ``Or`` list will win.
+ left-most expression in the Or_ list will win.
- If parsing the contents of an entire file, pass it to the
- ``parseFile`` method using::
+ ``parse_file`` method using::
- expr.parseFile(sourceFile)
+ expr.parse_file(source_file)
- ``ParseExceptions`` will report the location where an expected token
or expression failed to match. For example, if we tried to use our
- Punctuation may be significant for matching, but is rarely of
much interest in the parsed results. Use the ``suppress()`` method
to keep these tokens from cluttering up your returned lists of
- tokens. For example, ``delimitedList()`` matches a succession of
+ tokens. For example, ``delimited_list()`` matches a succession of
one or more expressions, separated by delimiters (commas by
default), but only returns a list of the actual expressions -
the delimiters are used for parsing, but are suppressed from the
expressions. It is much easier to access a token using its field
name than using a positional index, especially if the expression
contains optional elements. You can also shortcut
- the ``setResultsName`` call::
+ the ``set_results_name`` call::
- stats = ("AVE:" + realNum.setResultsName("average")
- + "MIN:" + realNum.setResultsName("min")
- + "MAX:" + realNum.setResultsName("max"))
+ stats = ("AVE:" + real_num.set_results_name("average")
+ + "MIN:" + real_num.set_results_name("min")
+ + "MAX:" + real_num.set_results_name("max"))
- can now be written as this::
+ can more simply and cleanly be written as this::
- stats = ("AVE:" + realNum("average")
- + "MIN:" + realNum("min")
- + "MAX:" + realNum("max"))
+ stats = ("AVE:" + real_num("average")
+ + "MIN:" + real_num("min")
+ + "MAX:" + real_num("max"))
- Be careful when defining parse actions that modify global variables or
- data structures (as in ``fourFn.py``), especially for low level tokens
- or expressions that may occur within an ``And`` expression; an early element
- of an ``And`` may match, but the overall expression may fail.
+ data structures (as in fourFn.py_), especially for low level tokens
+ or expressions that may occur within an And_ expression; an early element
+ of an And_ may match, but the overall expression may fail.
Classes
``ParserElement`` - abstract base class for all pyparsing classes;
methods for code to use are:
-- ``parseString(sourceString, parseAll=False)`` - only called once, on the overall
+- ``parse_string(source_string, parse_all=False)`` - only called once, on the overall
matching pattern; returns a ParseResults_ object that makes the
matched tokens available as a list, and optionally as a dictionary,
- or as an object with named attributes; if parseAll is set to True, then
- parseString will raise a ParseException if the grammar does not process
+ or as an object with named attributes; if ``parse_all`` is set to True, then
+ ``parse_string`` will raise a ParseException_ if the grammar does not process
the complete input string.
-- ``parseFile(sourceFile)`` - a convenience function, that accepts an
+- ``parse_file(source_file)`` - a convenience function, that accepts an
input file object or filename. The file contents are passed as a
- string to ``parseString()``. ``parseFile`` also supports the ``parseAll`` argument.
+ string to ``parse_string()``. ``parse_file`` also supports the ``parse_all`` argument.
-- ``scanString(sourceString)`` - generator function, used to find and
+- ``scan_string(source_string)`` - generator function, used to find and
extract matching text in the given source string; for each matched text,
returns a tuple of:
- end location in the given source string
- ``scanString`` allows you to scan through the input source string for
+ ``scan_string`` allows you to scan through the input source string for
random matches, instead of exhaustively defining the grammar for the entire
- source text (as would be required with ``parseString``).
+ source text (as would be required with ``parse_string``).
-- ``transformString(sourceString)`` - convenience wrapper function for
- ``scanString``, to process the input source string, and replace matching
+- ``transform_string(source_string)`` - convenience wrapper function for
+ ``scan_string``, to process the input source string, and replace matching
text with the tokens returned from parse actions defined in the grammar
- (see setParseAction_).
+ (see set_parse_action_).
-- ``searchString(sourceString)`` - another convenience wrapper function for
- ``scanString``, returns a list of the matching tokens returned from each
- call to ``scanString``.
+- ``search_string(source_string)`` - another convenience wrapper function for
+ ``scan_string``, returns a list of the matching tokens returned from each
+ call to ``scan_string``.
-- ``setName(name)`` - associate a short descriptive name for this
+- ``set_name(name)`` - associate a short descriptive name for this
element, useful in displaying exceptions and trace information
-- ``setResultsName(string, listAllMatches=False)`` - name to be given
+- ``run_tests(tests_string)`` - useful development and testing method on
+ expressions, to pass a multiline string of sample strings to test against
+ the expression. Comment lines (beginning with ``#``) can be inserted
+ and they will be included in the test output::
+
+ digits = Word(nums).set_name("numeric digits")
+ real_num = Combine(digits + '.' + digits)
+ real_num.run_tests("""\
+ # valid number
+ 3.14159
+
+ # no integer part
+ .00001
+
+ # no decimal
+ 101
+
+ # no decimal value
+ 101.
+ """)
+
+ will print::
+
+ # valid number
+ 3.14159
+ ['3.14159']
+
+ # no integer part
+ .00001
+ ^
+ FAIL: Expected numeric digits, found '.' (at char 0), (line:1, col:1)
+
+ # no decimal
+ 101
+ ^
+ FAIL: Expected ".", found end of text (at char 3), (line:1, col:4)
+
+ # no decimal value
+ 101.
+ ^
+ FAIL: Expected numeric digits, found end of text (at char 4), (line:1, col:5)
+
+- ``set_results_name(string, list_all_matches=False)`` - name to be given
to tokens matching
the element; if multiple tokens within
- a repetition group (such as ``ZeroOrMore`` or ``delimitedList``) the
- default is to return only the last matching token - if listAllMatches
+ a repetition group (such as ``ZeroOrMore`` or ``delimited_list``) the
+ default is to return only the last matching token - if ``list_all_matches``
is set to True, then a list of all the matching tokens is returned.
- (New in 1.5.6 - a results name with a trailing '*' character will be
- interpreted as setting listAllMatches to True.)
+
+ ``expr.set_results_name("key")`` can also be written ``expr("key")``
+ (a results name with a trailing '*' character will be
+ interpreted as setting ``list_all_matches`` to ``True``).
+
Note:
- ``setResultsName`` returns a *copy* of the element so that a single
+ ``set_results_name`` returns a *copy* of the element so that a single
basic element can be referenced multiple times and given
different names within a complex grammar.
-.. _setParseAction:
+.. _set_parse_action:
-- ``setParseAction(*fn)`` - specify one or more functions to call after successful
+- ``set_parse_action(*fn)`` - specify one or more functions to call after successful
matching of the element; each function is defined as ``fn(s, loc, toks)``, where:
- ``s`` is the original parse string
- ``toks`` is the list of the matched tokens, packaged as a ParseResults_ object
- Multiple functions can be attached to a ParserElement by specifying multiple
- arguments to setParseAction, or by calling setParseAction multiple times.
+ Parse actions can have any of the following signatures::
+
+ fn(s: str, loc: int, tokens: ParseResults)
+ fn(loc: int, tokens: ParseResults)
+ fn(tokens: ParseResults)
+ fn()
+
+ Multiple functions can be attached to a ``ParserElement`` by specifying multiple
+ arguments to ``set_parse_action``, or by calling ``add_parse_action``. Calls to ``set_parse_action``
+ will replace any previously defined parse actions. ``set_parse_action(None)`` will clear
+ all previously defined parse actions.
Each parse action function can return a modified ``toks`` list, to perform conversion, or
string modifications. For brevity, ``fn`` may also be a
lambda - here is an example of using a parse action to convert matched
integer tokens from strings to integers::
- intNumber = Word(nums).setParseAction(lambda s,l,t: [int(t[0])])
+ int_number = Word(nums).set_parse_action(lambda s, l, t: [int(t[0])])
- If ``fn`` does not modify the ``toks`` list, it does not need to return
- anything at all.
+ If ``fn`` modifies the ``toks`` list in-place, it does not need to return
+ and pyparsing will use the modified ``toks`` list.
-- ``setBreak(breakFlag=True)`` - if breakFlag is True, calls pdb.set_break()
+- ``add_parse_action`` - similar to ``set_parse_action``, but instead of replacing any
+ previously defined parse actions, will append the given action or actions to the
+ existing defined parse actions.
+
+- ``add_condition`` - a simplified form of ``add_parse_action`` if the purpose
+ of the parse action is to simply do some validation, and raise an exception
+ if the validation fails. Takes a method that takes the same arguments,
+ but simply returns ``True`` or ``False``. If ``False`` is returned, an exception will be
+ raised.
+
+- ``set_break(break_flag=True)`` - if ``break_flag`` is ``True``, calls ``pdb.set_break()``
as this expression is about to be parsed
-- ``copy()`` - returns a copy of a ParserElement; can be used to use the same
+- ``copy()`` - returns a copy of a ``ParserElement``; can be used to use the same
parse expression in different places in a grammar, with different parse actions
- attached to each
+ attached to each; a short-form ``expr()`` is equivalent to ``expr.copy()``
-- ``leaveWhitespace()`` - change default behavior of skipping
+- ``leave_whitespace()`` - change default behavior of skipping
whitespace before starting matching (mostly used internally to the
pyparsing module, rarely used by client code)
-- ``setWhitespaceChars(chars)`` - define the set of chars to be ignored
- as whitespace before trying to match a specific ParserElement, in place of the
+- ``set_whitespace_chars(chars)`` - define the set of chars to be ignored
+ as whitespace before trying to match a specific ``ParserElement``, in place of the
default set of whitespace (space, tab, newline, and return)
-- ``setDefaultWhitespaceChars(chars)`` - class-level method to override
+- ``set_default_whitespace_chars(chars)`` - class-level method to override
the default set of whitespace chars for all subsequently created ParserElements
(including copies); useful when defining grammars that treat one or more of the
default whitespace characters as significant (such as a line-sensitive grammar, to
omit newline from the list of ignorable whitespace)
- ``suppress()`` - convenience function to suppress the output of the
- given element, instead of wrapping it with a Suppress object.
+ given element, instead of wrapping it with a ``Suppress`` object.
- ``ignore(expr)`` - function to specify parse expression to be
ignored while matching defined patterns; can be called
repeatedly to specify multiple expressions; useful to specify
patterns of comment syntax, for example
-- ``setDebug(dbgFlag=True)`` - function to enable/disable tracing output
+- ``set_debug(debug_flag=True)`` - function to enable/disable tracing output
when trying to match this element
- ``validate()`` - function to verify that the defined grammar does not
contain infinitely recursive constructs
-.. _parseWithTabs:
+.. _parse_with_tabs:
-- ``parseWithTabs()`` - function to override default behavior of converting
+- ``parse_with_tabs()`` - function to override default behavior of converting
tabs to spaces before parsing the input string; rarely used, except when
specifying whitespace-significant grammars using the White_ class.
-- ``enablePackrat()`` - a class-level static method to enable a memoizing
+- ``enable_packrat()`` - a class-level static method to enable a memoizing
performance enhancement, known as "packrat parsing". packrat parsing is
disabled by default, since it may conflict with some user programs that use
parse actions. To activate the packrat feature, your
- program must call the class method ParserElement.enablePackrat(). For best
- results, call enablePackrat() immediately after importing pyparsing.
+ program must call the class method ``ParserElement.enable_packrat()``. For best
+ results, call ``enable_packrat()`` immediately after importing pyparsing.
+- ``enable_left_recursion()`` - a class-level static method to enable
+ pyparsing with left-recursive (LR) parsers. Similar to ``ParserElement.enable_packrat()``,
+ your program must call the class method ``ParserElement.enable_left_recursion()`` to
+ enable this feature. ``enable_left_recursion()`` uses a separate packrat cache, and so
+ is incompatible with ``enable_packrat()``.
Basic ParserElement subclasses
------------------------------
+.. _Literal:
+
- ``Literal`` - construct with a string to be matched exactly
+.. _CaselessLiteral:
+
- ``CaselessLiteral`` - construct with a string to be matched, but
without case checking; results are always returned as the
defining literal, NOT as they are found in the input string
-- ``Keyword`` - similar to Literal, but must be immediately followed by
+.. _Keyword:
+
+- ``Keyword`` - similar to Literal_, but must be immediately followed by
whitespace, punctuation, or other non-keyword characters; prevents
accidental matching of a non-keyword that happens to begin with a
defined keyword
-- ``CaselessKeyword`` - similar to Keyword, but with caseless matching
+- ``CaselessKeyword`` - similar to Keyword_, but with caseless matching
behavior
.. _Word:
- ``Word`` - one or more contiguous characters; construct with a
string containing the set of allowed initial characters, and an
optional second string of allowed body characters; for instance,
- a common Word construct is to match a code identifier - in C, a
+ a common ``Word`` construct is to match a code identifier - in C, a
valid identifier must start with an alphabetic character or an
underscore ('_'), followed by a body that can also include numeric
digits. That is, ``a``, ``i``, ``MAX_LENGTH``, ``_a1``, ``b_109_``, and
``plan9FromOuterSpace``
are all valid identifiers; ``9b7z``, ``$a``, ``.section``, and ``0debug``
are not. To
- define an identifier using a Word, use either of the following::
+ define an identifier using a ``Word``, use either of the following::
+
+ Word(alphas+"_", alphanums+"_")
+ Word(srange("[a-zA-Z_]"), srange("[a-zA-Z0-9_]"))
- - Word(alphas+"_", alphanums+"_")
- - Word(srange("[a-zA-Z_]"), srange("[a-zA-Z0-9_]"))
+ Pyparsing also provides pre-defined strings ``identchars`` and
+ ``identbodychars`` so that you can also write::
+
+ Word(identchars, identbodychars)
If only one
string given, it specifies that the same character set defined
for the initial character is used for the word body; for instance, to
define an identifier that can only be composed of capital letters and
- underscores, use::
+ underscores, use one of::
- - Word("ABCDEFGHIJKLMNOPQRSTUVWXYZ_")
- - Word(srange("[A-Z_]"))
+ ``Word("ABCDEFGHIJKLMNOPQRSTUVWXYZ_")``
+ ``Word(srange("[A-Z_]"))``
- A Word may
+ A ``Word`` may
also be constructed with any of the following optional parameters:
- ``min`` - indicating a minimum length of matching characters
If ``exact`` is specified, it will override any values for ``min`` or ``max``.
- New in 1.5.6 - Sometimes you want to define a word using all
+ Sometimes you want to define a word using all
characters in a range except for one or two of them; you can do this
- with the new ``excludeChars`` argument. This is helpful if you want to define
- a word with all printables except for a single delimiter character, such
+ with the new ``exclude_chars`` argument. This is helpful if you want to define
+ a word with all ``printables`` except for a single delimiter character, such
as '.'. Previously, you would have to create a custom string to pass to Word.
- With this change, you can just create ``Word(printables, excludeChars='.')``.
+ With this change, you can just create ``Word(printables, exclude_chars='.')``.
+
+- ``Char`` - a convenience form of ``Word`` that will match just a single character from
+ a string of matching characters::
+
+ single_digit = Char(nums)
- ``CharsNotIn`` - similar to Word_, but matches characters not
in the given constructor string (accepts only one string for both
- ``Regex`` - a powerful construct, that accepts a regular expression
to be matched at the current parse position; accepts an optional
- ``flags`` parameter, corresponding to the flags parameter in the re.compile
+ ``flags`` parameter, corresponding to the flags parameter in the ``re.compile``
method; if the expression includes named sub-fields, they will be
- represented in the returned ParseResults_
+ represented in the returned ParseResults_.
- ``QuotedString`` - supports the definition of custom quoted string
- formats, in addition to pyparsing's built-in ``dblQuotedString`` and
- ``sglQuotedString``. ``QuotedString`` allows you to specify the following
+ formats, in addition to pyparsing's built-in ``dbl_quoted_string`` and
+ ``sgl_quoted_string``. ``QuotedString`` allows you to specify the following
parameters:
- - ``quoteChar`` - string of one or more characters defining the quote delimiting string
+ - ``quote_char`` - string of one or more characters defining the quote delimiting string
- - ``escChar`` - character to escape quotes, typically backslash (default=None)
+ - ``esc_char`` - character to escape quotes, typically backslash (default=None)
- - ``escQuote`` - special quote sequence to escape an embedded quote string (such as SQL's "" to escape an embedded ") (default=None)
+ - ``esc_quote`` - special quote sequence to escape an embedded quote string (such as SQL's "" to escape an embedded ") (default=None)
- ``multiline`` - boolean indicating whether quotes can span multiple lines (default=False)
- - ``unquoteResults`` - boolean indicating whether the matched text should be unquoted (default=True)
+ - ``unquote_results`` - boolean indicating whether the matched text should be unquoted (default=True)
+
+ - ``end_quote_char`` - string of one or more characters defining the end of the quote delimited string (default=None => same as ``quote_char``)
- - ``endQuoteChar`` - string of one or more characters defining the end of the quote delimited string (default=None => same as quoteChar)
+.. _SkipTo:
- ``SkipTo`` - skips ahead in the input string, accepting any
characters up to the specified pattern; may be constructed with
- ``ignore`` - allows the user to specify patterns to not be matched,
to prevent false matches
- - ``failOn`` - if a literal string or expression is given for this argument, it defines an expression that
- should cause the ``SkipTo`` expression to fail, and not skip over that expression
+ - ``fail_on`` - if a literal string or expression is given for this argument, it defines an expression that
+ should cause the SkipTo_ expression to fail, and not skip over that expression
+
+ ``SkipTo`` can also be written using ``...``::
+
+ LBRACE, RBRACE = map(Literal, "{}")
+
+ brace_expr = LBRACE + SkipTo(RBRACE) + RBRACE
+ # can also be written as
+ brace_expr = LBRACE + ... + RBRACE
.. _White:
ignored by pyparsing. However, some grammars are whitespace-sensitive,
such as those that use leading tabs or spaces to indicating grouping
or hierarchy. (If matching on tab characters, be sure to call
- parseWithTabs_ on the top-level parse element.)
+ parse_with_tabs_ on the top-level parse element.)
- ``Empty`` - a null expression, requiring no characters - will always
match; useful for debugging and for specialized grammars
-- ``NoMatch`` - opposite of Empty, will never match; useful for debugging
+- ``NoMatch`` - opposite of ``Empty``, will never match; useful for debugging
and for specialized grammars
Expression subclasses
---------------------
-- ``And`` - construct with a list of ParserElements, all of which must
- match for And to match; can also be created using the '+'
- operator; multiple expressions can be Anded together using the '*'
+.. _And:
+
+- ``And`` - construct with a list of ``ParserElements``, all of which must
+ match for ``And`` to match; can also be created using the '+'
+ operator; multiple expressions can be ``Anded`` together using the '*'
operator as in::
- ipAddress = Word(nums) + ('.' + Word(nums)) * 3
+ ip_address = Word(nums) + ('.' + Word(nums)) * 3
A tuple can be used as the multiplier, indicating a min/max::
- usPhoneNumber = Word(nums) + ('-' + Word(nums)) * (1,2)
+ us_phone_number = Word(nums) + ('-' + Word(nums)) * (1,2)
A special form of ``And`` is created if the '-' operator is used
- instead of the '+' operator. In the ipAddress example above, if
- no trailing '.' and Word(nums) are found after matching the initial
- Word(nums), then pyparsing will back up in the grammar and try other
- alternatives to ipAddress. However, if ipAddress is defined as::
+ instead of the '+' operator. In the ``ip_address`` example above, if
+ no trailing '.' and ``Word(nums)`` are found after matching the initial
+ ``Word(nums)``, then pyparsing will back up in the grammar and try other
+ alternatives to ``ip_address``. However, if ``ip_address`` is defined as::
- strictIpAddress = Word(nums) - ('.'+Word(nums))*3
+ strict_ip_address = Word(nums) - ('.'+Word(nums))*3
- then no backing up is done. If the first Word(nums) of strictIpAddress
- is matched, then any mismatch after that will raise a ParseSyntaxException,
+ then no backing up is done. If the first ``Word(nums)`` of ``strict_ip_address``
+ is matched, then any mismatch after that will raise a ``ParseSyntaxException``,
which will halt the parsing process immediately. By careful use of the
'-' operator, grammars can provide meaningful error messages close to
the location where the incoming text does not match the specified
grammar.
-- ``Or`` - construct with a list of ParserElements, any of which must
- match for Or to match; if more than one expression matches, the
+.. _Or:
+
+- ``Or`` - construct with a list of ``ParserElements``, any of which must
+ match for ``Or`` to match; if more than one expression matches, the
expression that makes the longest match will be used; can also
be created using the '^' operator
-- ``MatchFirst`` - construct with a list of ParserElements, any of
- which must match for MatchFirst to match; matching is done
+.. _MatchFirst:
+
+- ``MatchFirst`` - construct with a list of ``ParserElements``, any of
+ which must match for ``MatchFirst`` to match; matching is done
left-to-right, taking the first expression that matches; can
also be created using the '|' operator
-- ``Each`` - similar to And, in that all of the provided expressions
- must match; however, Each permits matching to be done in any order;
+.. _Each:
+
+- ``Each`` - similar to And_, in that all of the provided expressions
+ must match; however, ``Each`` permits matching to be done in any order;
can also be created using the '&' operator
-- ``Optional`` - construct with a ParserElement, but this element is
+- ``Opt`` - construct with a ``ParserElement``, but this element is
not required to match; can be constructed with an optional ``default`` argument,
containing a default string or object to be supplied if the given optional
parse element is not found in the input string; parse action will only
- be called if a match is found, or if a default is specified
+ be called if a match is found, or if a default is specified.
+
+ (``Opt`` was formerly named ``Optional``, but since the standard Python
+ library module ``typing`` now defines ``Optional``, the pyparsing class has
+ been renamed to ``Opt``. A compatibility synonym ``Optional`` is defined,
+ but will be removed in a future release.)
-- ``ZeroOrMore`` - similar to Optional, but can be repeated
+- ``ZeroOrMore`` - similar to ``Opt``, but can be repeated; ``ZeroOrMore(expr)``
+ can also be written as ``expr[...]``.
-- ``OneOrMore`` - similar to ZeroOrMore, but at least one match must
- be present
+- ``OneOrMore`` - similar to ``ZeroOrMore``, but at least one match must
+ be present; ``OneOrMore(expr)`` can also be written as ``expr[1, ...]``.
- ``FollowedBy`` - a lookahead expression, requires matching of the given
expressions, but does not advance the parsing position within the input string
+.. _NotAny:
+
- ``NotAny`` - a negative lookahead expression, prevents matching of named
expressions, does not advance the parsing position within the input string;
can also be created using the unary '~' operator
Expression operators
--------------------
-- ``~`` - creates NotAny using the expression after the operator
-
-- ``+`` - creates And using the expressions before and after the operator
+- ``+`` - creates And_ using the expressions before and after the operator
-- ``|`` - creates MatchFirst (first left-to-right match) using the expressions before and after the operator
+- ``|`` - creates MatchFirst_ (first left-to-right match) using the expressions before and after the operator
-- ``^`` - creates Or (longest match) using the expressions before and after the operator
+- ``^`` - creates Or_ (longest match) using the expressions before and after the operator
-- ``&`` - creates Each using the expressions before and after the operator
+- ``&`` - creates Each_ using the expressions before and after the operator
-- ``*`` - creates And by multiplying the expression by the integer operand; if
- expression is multiplied by a 2-tuple, creates an And of (min,max)
- expressions (similar to "{min,max}" form in regular expressions); if
- min is None, intepret as (0,max); if max is None, interpret as
- expr*min + ZeroOrMore(expr)
+- ``*`` - creates And_ by multiplying the expression by the integer operand; if
+ expression is multiplied by a 2-tuple, creates an And_ of ``(min,max)``
+ expressions (similar to ``{min,max}`` form in regular expressions); if
+ ``min`` is ``None``, interpret as ``(0,max)``; if ``max`` is ``None``, interpret as
+ ``expr*min + ZeroOrMore(expr)``
- ``-`` - like ``+`` but with no backup and retry of alternatives
-- ``*`` - repetition of expression
+- ``~`` - creates NotAny_ using the expression after the operator
-- ``==`` - matching expression to string; returns True if the string matches the given expression
+- ``==`` - matching expression to string; returns ``True`` if the string matches the given expression
- ``<<=`` - inserts the expression following the operator as the body of the
- Forward expression before the operator
+ ``Forward`` expression before the operator (``<<`` can also be used, but ``<<=`` is preferred
+ to avoid operator precedence misinterpretation of the pyparsing expression)
+- ``...`` - inserts a SkipTo_ expression leading to the next expression, as in
+ ``Keyword("start") + ... + Keyword("end")``.
+
+- ``[min, max]`` - specifies repetition similar to ``*`` with ``min`` and ``max`` specified
+ as the minimum and maximum number of repetitions. ``...`` can be used in place of ``None``.
+ For example ``expr[...]`` is equivalent to ``ZeroOrMore(expr)``, ``expr[1, ...]`` is
+ equivalent to ``OneOrMore(expr)``, and ``expr[..., 3]`` is equivalent to "up to 3 instances
+ of ``expr``".
Positional subclasses
--------------------
- ``Combine`` - joins all matched tokens into a single string, using
- specified joinString (default ``joinString=""``); expects
+ specified ``join_string`` (default ``join_string=""``); expects
all matching tokens to be adjacent, with no intervening
whitespace (can be overridden by specifying ``adjacent=False`` in constructor)
break up matched tokens into groups for each repeated pattern
- ``Dict`` - like ``Group``, but also constructs a dictionary, using the
- [0]'th elements of all enclosed token lists as the keys, and
+ ``[0]``'th elements of all enclosed token lists as the keys, and
each token list as the value
-- ``SkipTo`` - catch-all matching expression that accepts all characters
- up until the given pattern is found to match; useful for specifying
- incomplete grammars
-
- ``Forward`` - placeholder token used to define recursive token
patterns; when defining the actual expression later in the
- program, insert it into the ``Forward`` object using the ``<<``
- operator (see ``fourFn.py`` for an example).
+ program, insert it into the ``Forward`` object using the ``<<=``
+ operator (see fourFn.py_ for an example).
Other classes
- ``ParseResults`` - class used to contain and manage the lists of tokens
created from parsing the input using the user-defined parse
- expression. ParseResults can be accessed in a number of ways:
+ expression. ``ParseResults`` can be accessed in a number of ways:
- as a list
- - total list of elements can be found using len()
+ - total list of elements can be found using ``len()``
- - individual elements can be found using [0], [1], [-1], etc.
+ - individual elements can be found using ``[0], [1], [-1],`` etc.,
+ or retrieved using slices
- elements can be deleted using ``del``
- - the -1th element can be extracted and removed in a single operation
+ - the ``-1``th element can be extracted and removed in a single operation
using ``pop()``, or any element can be extracted and removed
using ``pop(n)``
+ - a nested ParseResults_ can be created by using the pyparsing ``Group`` class
+ around elements in an expression::
+
+ Word(alphas) + Group(Word(nums)[...]) + Word(alphas)
+
+ will parse the string "abc 100 200 300 end" as::
+
+ ['abc', ['100', '200', '300'], 'end']
+
+ If the ``Group`` is constructed using ``aslist=True``, the resulting tokens
+ will be a Python list instead of a ParseResults_.
+
- as a dictionary
- - if ``setResultsName()`` is used to name elements within the
+ - if ``set_results_name()`` is used to name elements within the
overall parse expression, then these fields can be referenced
as dictionary elements or as attributes
- - the Dict class generates dictionary entries using the data of the
- input text - in addition to ParseResults listed as ``[ [ a1, b1, c1, ...], [ a2, b2, c2, ...] ]``
+ - the ``Dict`` class generates dictionary entries using the data of the
+ input text - in addition to ParseResults_ listed as ``[ [ a1, b1, c1, ...], [ a2, b2, c2, ...] ]``
it also acts as a dictionary with entries defined as ``{ a1 : [ b1, c1, ... ] }, { a2 : [ b2, c2, ... ] }``;
this is especially useful when processing tabular data where the first column contains a key
- value for that line of data
+ value for that line of data; when constructed with ``aslist=True``, will
+ return an actual Python ``dict`` instead of a ParseResults_.
- list elements that are deleted using ``del`` will still be accessible by their
dictionary keys
- supports ``get()``, ``items()`` and ``keys()`` methods, similar to a dictionary
- a keyed item can be extracted and removed using ``pop(key)``. Here
- key must be non-numeric (such as a string), in order to use dict
+ ``key`` must be non-numeric (such as a string), in order to use dict
extraction instead of list extraction.
- new named elements can be added (in a parse action, for instance), using the same
- syntax as adding an item to a dict (``parseResults["X"] = "new item"``); named elements can be removed using ``del parseResults["X"]``
+ syntax as adding an item to a dict (``parse_results["X"] = "new item"``);
+ named elements can be removed using ``del parse_results["X"]``
- as a nested list
own list structure, so that the tokens can be handled as a hierarchical
tree
- ParseResults can also be converted to an ordinary list of strings
- by calling ``asList()``. Note that this will strip the results of any
+ - as an object
+
+ - named elements can be accessed as if they were attributes of an object:
+ if an element is referenced that does not exist, it will return ``""``.
+
+ ParseResults_ can also be converted to an ordinary list of strings
+ by calling ``as_list()``. Note that this will strip the results of any
field names that have been defined for any embedded parse elements.
(The ``pprint`` module is especially good at printing out the nested contents
- given by ``asList()``.)
+ given by ``as_list()``.)
- Finally, ParseResults can be viewed by calling ``dump()``. ``dump()` will first show
- the ``asList()`` output, followed by an indented structure listing parsed tokens that
+ Finally, ParseResults_ can be viewed by calling ``dump()``. ``dump()`` will first show
+ the ``as_list()`` output, followed by an indented structure listing parsed tokens that
have been assigned results names.
+ Here is sample code illustrating some of these methods::
+
+ >>> number = Word(nums)
+ >>> name = Combine(Word(alphas)[...], adjacent=False, join_string=" ")
+ >>> parser = number("house_number") + name("street_name")
+ >>> result = parser.parse_string("123 Main St")
+ >>> print(result)
+ ['123', 'Main St']
+ >>> print(type(result))
+ <class 'pyparsing.ParseResults'>
+ >>> print(repr(result))
+ (['123', 'Main St'], {'house_number': ['123'], 'street_name': ['Main St']})
+ >>> result.house_number
+ '123'
+ >>> result["street_name"]
+ 'Main St'
+ >>> result.as_list()
+ ['123', 'Main St']
+ >>> result.as_dict()
+ {'house_number': '123', 'street_name': 'Main St'}
+ >>> print(result.dump())
+ ['123', 'Main St']
+ - house_number: '123'
+ - street_name: 'Main St'
+
Exception classes and Troubleshooting
-------------------------------------
.. _ParseException:
- ``ParseException`` - exception returned when a grammar parse fails;
- ParseExceptions have attributes loc, msg, line, lineno, and column; to view the
+ ``ParseExceptions`` have attributes ``loc``, ``msg``, ``line``, ``lineno``, and ``column``; to view the
text line and location where the reported ParseException occurs, use::
- except ParseException, err:
- print err.line
- print " " * (err.column - 1) + "^"
- print err
+ except ParseException as err:
+ print(err.line)
+ print(" " * (err.column - 1) + "^")
+ print(err)
+
+ ``ParseExceptions`` also have an ``explain()`` method that gives this same information::
+
+ except ParseException as err:
+ print(err.explain())
- ``RecursiveGrammarException`` - exception returned by ``validate()`` if
the grammar contains a recursive infinite loop, such as::
- badGrammar = Forward()
- goodToken = Literal("A")
- badGrammar <<= Optional(goodToken) + badGrammar
+ bad_grammar = Forward()
+ good_token = Literal("A")
+ bad_grammar <<= Opt(good_token) + bad_grammar
- ``ParseFatalException`` - exception that parse actions can raise to stop parsing
immediately. Should be used when a semantic error is found in the input text, such
- ``ParseSyntaxException`` - subclass of ``ParseFatalException`` raised when a
syntax error is found, based on the use of the '-' operator when defining
- a sequence of expressions in an ``And`` expression.
+ a sequence of expressions in an And_ expression.
+
+- You can also get some insights into the parsing logic using diagnostic parse actions,
+ and ``set_debug()``, or test the matching of expression fragments by testing them using
+ ``search_string()`` or ``scan_string()``.
+
+- Use ``with_line_numbers`` from ``pyparsing_testing`` to display the input string
+ being parsed, with line and column numbers that correspond to the values reported
+ in set_debug() output::
+
+ import pyparsing as pp
+ ppt = pp.testing
+
+ data = """\
+ A
+ 100"""
+
+ expr = pp.Word(pp.alphanums).set_name("word").set_debug()
+ print(ppt.with_line_numbers(data))
+ expr[...].parseString(data)
+
+ prints::
+
+ . 1
+ 1234567890
+ 1: A|
+ 2: 100|
+
+ Match word at loc 3(1,4)
+ A
+ ^
+ Matched word -> ['A']
+ Match word at loc 11(2,7)
+ 100
+ ^
+ Matched word -> ['100']
+
+ `with_line_numbers` has several options for displaying control characters, end-of-line
+ and space markers, Unicode symbols for control characters - these are documented in the
+ function's docstring.
+
+- Diagnostics can be enabled using ``pyparsing.enable_diag`` and passing
+ one of the following enum values defined in ``pyparsing.Diagnostics``
+
+ - ``warn_multiple_tokens_in_named_alternation`` - flag to enable warnings when a results
+ name is defined on a MatchFirst_ or Or_ expression with one or more And_ subexpressions
+
+ - ``warn_ungrouped_named_tokens_in_collection`` - flag to enable warnings when a results
+ name is defined on a containing expression with ungrouped subexpressions that also
+ have results names
-You can also get some insights into the parsing logic using diagnostic parse actions,
-and setDebug(), or test the matching of expression fragments by testing them using
-scanString().
+ - ``warn_name_set_on_empty_Forward`` - flag to enable warnings when a ``Forward`` is defined
+ with a results name, but has no contents defined
+
+ - ``warn_on_parse_using_empty_Forward`` - flag to enable warnings when a ``Forward`` is
+ defined in a grammar but has never had an expression attached to it
+
+ - ``warn_on_assignment_to_Forward`` - flag to enable warnings when a ``Forward`` is defined
+ but is overwritten by assigning using ``'='`` instead of ``'<<='`` or ``'<<'``
+
+ - ``warn_on_multiple_string_args_to_oneof`` - flag to enable warnings when ``one_of`` is
+ incorrectly called with multiple str arguments
+
+ - ``enable_debug_on_named_expressions`` - flag to auto-enable debug on all subsequent
+ calls to ``ParserElement.set_name``
+
+ All warnings can be enabled by calling ``pyparsing.enable_all_warnings()``.
+ Sample::
+
+ import pyparsing as pp
+ pp.enable_all_warnings()
+
+ fwd = pp.Forward().set_results_name("recursive_expr")
+
+ >>> UserWarning: warn_name_set_on_empty_Forward: setting results name 'recursive_expr'
+ on Forward expression that has no contained expression
+
+ Warnings can also be enabled using the Python ``-W`` switch, or setting a non-empty
+ value to the environment variable ``PYPARSINGENABLEALLWARNINGS``
Miscellaneous attributes and methods
Helper methods
--------------
-- ``delimitedList(expr, delim=',')`` - convenience function for
+- ``delimited_list(expr, delim=',')`` - convenience function for
matching one or more occurrences of expr, separated by delim.
By default, the delimiters are suppressed, so the returned results contain
only the separate list elements. Can optionally specify ``combine=True``,
indicating that the expressions and delimiters should be returned as one
combined value (useful for scoped variables, such as ``"a.b.c"``, or
- ``"a::b::c"``, or paths such as ``"a/b/c"``).
+ ``"a::b::c"``, or paths such as ``"a/b/c"``). Can also optionally specify
+ ``allow_trailing_delim`` to accept a trailing delimiter at the end of the list.
-- ``countedArray(expr)`` - convenience function for a pattern where an list of
+- ``counted_array(expr)`` - convenience function for a pattern where an list of
instances of the given expression are preceded by an integer giving the count of
elements in the list. Returns an expression that parses the leading integer,
reads exactly that many expressions, and returns the array of expressions in the
parse results - the leading integer is suppressed from the results (although it
is easily reconstructed by using len on the returned array).
-- ``oneOf(string, caseless=False)`` - convenience function for quickly declaring an
- alternative set of ``Literal`` tokens, by splitting the given string on
- whitespace boundaries. The tokens are sorted so that longer
- matches are attempted first; this ensures that a short token does
+- ``one_of(choices, caseless=False, as_keyword=False)`` - convenience function for quickly declaring an
+ alternative set of Literal_ expressions. ``choices`` can be passed as a list of strings
+ or as a single string of values separated by spaces. The values are sorted so that longer
+ matches are attempted first; this ensures that a short value does
not mask a longer one that starts with the same characters. If ``caseless=True``,
- will create an alternative set of CaselessLiteral tokens.
+ will create an alternative set of CaselessLiteral_ tokens. If ``as_keyword=True``,
+ ``one_of`` will declare Keyword_ expressions instead of Literal_ expressions.
-- ``dictOf(key, value)`` - convenience function for quickly declaring a
+- ``dict_of(key, value)`` - convenience function for quickly declaring a
dictionary pattern of ``Dict(ZeroOrMore(Group(key + value)))``.
-- ``makeHTMLTags(tagName)`` and ``makeXMLTags(tagName)`` - convenience
+- ``make_html_tags(tag_str)`` and ``make_xml_tags(tag_str)`` - convenience
functions to create definitions of opening and closing tag expressions. Returns
- a pair of expressions, for the corresponding <tag> and </tag> strings. Includes
- support for attributes in the opening tag, such as <tag attr1="abc"> - attributes
- are returned as keyed tokens in the returned ParseResults. ``makeHTMLTags`` is less
- restrictive than ``makeXMLTags``, especially with respect to case sensitivity.
+ a pair of expressions, for the corresponding ``<tag>`` and ``</tag>`` strings. Includes
+ support for attributes in the opening tag, such as ``<tag attr1="abc">`` - attributes
+ are returned as named results in the returned ParseResults_. ``make_html_tags`` is less
+ restrictive than ``make_xml_tags``, especially with respect to case sensitivity.
-- ``infixNotation(baseOperand, operatorList)`` - (formerly named ``operatorPrecedence``)
+- ``infix_notation(base_operand, operator_list)`` -
convenience function to define a grammar for parsing infix notation
- expressions with a hierarchical precedence of operators. To use the ``infixNotation``
+ expressions with a hierarchical precedence of operators. To use the ``infix_notation``
helper:
1. Define the base "atom" operand term of the grammar.
For this simple grammar, the smallest operand is either
- and integer or a variable. This will be the first argument
- to the ``infixNotation`` method.
+ an integer or a variable. This will be the first argument
+ to the ``infix_notation`` method.
2. Define a list of tuples for each level of operator
- precendence. Each tuple is of the form
- ``(opExpr, numTerms, rightLeftAssoc, parseAction)``, where:
+ precedence. Each tuple is of the form
+ ``(operand_expr, num_operands, right_left_assoc, parse_action)``, where:
- - ``opExpr`` - the pyparsing expression for the operator;
- may also be a string, which will be converted to a Literal; if
- None, indicates an empty operator, such as the implied
+ - ``operand_expr`` - the pyparsing expression for the operator;
+ may also be a string, which will be converted to a Literal_; if
+ ``None``, indicates an empty operator, such as the implied
multiplication operation between 'm' and 'x' in "y = mx + b".
- - ``numTerms`` - the number of terms for this operator (must
+ - ``num_operands`` - the number of terms for this operator (must
be 1, 2, or 3)
- - ``rightLeftAssoc`` is the indicator whether the operator is
+ - ``right_left_assoc`` is the indicator whether the operator is
right or left associative, using the pyparsing-defined
- constants ``opAssoc.RIGHT`` and ``opAssoc.LEFT``.
+ constants ``OpAssoc.RIGHT`` and ``OpAssoc.LEFT``.
- - ``parseAction`` is the parse action to be associated with
+ - ``parse_action`` is the parse action to be associated with
expressions matching this operator expression (the
- ``parseAction`` tuple member may be omitted)
+ ``parse_action`` tuple member may be omitted)
- 3. Call ``infixNotation`` passing the operand expression and
+ 3. Call ``infix_notation`` passing the operand expression and
the operator precedence list, and save the returned value
as the generated pyparsing expression. You can then use
this expression to parse input strings, or incorporate it
into a larger, more complex grammar.
-- ``matchPreviousLiteral`` and ``matchPreviousExpr`` - function to define and
+- ``match_previous_literal`` and ``match_previous_expr`` - function to define an
expression that matches the same content
as was parsed in a previous parse expression. For instance::
first = Word(nums)
- matchExpr = first + ":" + matchPreviousLiteral(first)
+ match_expr = first + ":" + match_previous_literal(first)
will match "1:1", but not "1:2". Since this matches at the literal
level, this will also match the leading "1:1" in "1:10".
In contrast::
first = Word(nums)
- matchExpr = first + ":" + matchPreviousExpr(first)
+ match_expr = first + ":" + match_previous_expr(first)
will *not* match the leading "1:1" in "1:10"; the expressions are
evaluated first, and then compared, so "1" is compared with "10".
-- ``nestedExpr(opener, closer, content=None, ignoreExpr=quotedString)`` - method for defining nested
+- ``nested_expr(opener, closer, content=None, ignore_expr=quoted_string)`` - method for defining nested
lists enclosed in opening and closing delimiters.
- ``opener`` - opening character for a nested list (default="("); can also be a pyparsing expression
- ``content`` - expression for items within the nested lists (default=None)
- - ``ignoreExpr`` - expression for ignoring opening and closing delimiters (default=quotedString)
+ - ``ignore_expr`` - expression for ignoring opening and closing delimiters (default=``quoted_string``)
If an expression is not provided for the content argument, the nested
expression will capture all whitespace-delimited content between delimiters
as a list of separate values.
- Use the ignoreExpr argument to define expressions that may contain
+ Use the ``ignore_expr`` argument to define expressions that may contain
opening or closing characters that should not be treated as opening
- or closing characters for nesting, such as quotedString or a comment
- expression. Specify multiple expressions using an Or or MatchFirst.
- The default is quotedString, but if no expressions are to be ignored,
- then pass None for this argument.
+ or closing characters for nesting, such as ``quoted_string`` or a comment
+ expression. Specify multiple expressions using an Or_ or MatchFirst_.
+ The default is ``quoted_string``, but if no expressions are to be ignored,
+ then pass ``None`` for this argument.
-- ``indentedBlock(statementExpr, indentationStackVar, indent=True)`` -
+- ``IndentedBlock(statement_expr, recursive=True)`` -
function to define an indented block of statements, similar to
indentation-based blocking in Python source code:
- - ``statementExpr`` - the expression defining a statement that
- will be found in the indented block; a valid ``indentedBlock``
- must contain at least 1 matching ``statementExpr``
-
- - ``indentationStackVar`` - a Python list variable; this variable
- should be common to all ``indentedBlock`` expressions defined
- within the same grammar, and should be reinitialized to [1]
- each time the grammar is to be used
-
- - ``indent`` - a boolean flag indicating whether the expressions
- within the block must be indented from the current parse
- location; if using ``indentedBlock`` to define the left-most
- statements (all starting in column 1), set ``indent`` to False
+ - ``statement_expr`` - the expression defining a statement that
+ will be found in the indented block; a valid ``IndentedBlock``
+ must contain at least 1 matching ``statement_expr``
.. _originalTextFor:
-- ``originalTextFor(expr)`` - helper function to preserve the originally parsed text, regardless of any
+- ``original_text_for(expr)`` - helper function to preserve the originally parsed text, regardless of any
token processing or conversion done by the contained expression. For instance, the following expression::
- fullName = Word(alphas) + Word(alphas)
+ full_name = Word(alphas) + Word(alphas)
will return the parse of "John Smith" as ['John', 'Smith']. In some applications, the actual name as it
- was given in the input string is what is desired. To do this, use ``originalTextFor``::
+ was given in the input string is what is desired. To do this, use ``original_text_for``::
- fullName = originalTextFor(Word(alphas) + Word(alphas))
+ full_name = original_text_for(Word(alphas) + Word(alphas))
- ``ungroup(expr)`` - function to "ungroup" returned tokens; useful
- to undo the default behavior of And to always group the returned tokens, even
- if there is only one in the list. (New in 1.5.6)
+ to undo the default behavior of And_ to always group the returned tokens, even
+ if there is only one in the list.
- ``lineno(loc, string)`` - function to give the line number of the
location within the string; the first line is line 1, newlines
representing ``lineno(loc, string)``; useful when printing out diagnostic
messages for exceptions
-- ``srange(rangeSpec)`` - function to define a string of characters,
+- ``srange(range_spec)`` - function to define a string of characters,
given a string of the form used by regexp string ranges, such as ``"[0-9]"`` for
all numeric digits, ``"[A-Z_]"`` for uppercase characters plus underscore, and
- so on (note that rangeSpec does not include support for generic regular
+ so on (note that ``range_spec`` does not include support for generic regular
expressions, just string range specs)
-- ``getTokensEndLoc()`` - function to call from within a parse action to get
- the ending location for the matched tokens
-
-- ``traceParseAction(fn)`` - decorator function to debug parse actions. Lists
+- ``trace_parse_action(fn)`` - decorator function to debug parse actions. Lists
each call, called arguments, and return value or exception
Helper parse actions
--------------------
-- ``removeQuotes`` - removes the first and last characters of a quoted string;
+- ``remove_quotes`` - removes the first and last characters of a quoted string;
useful to remove the delimiting quotes from quoted strings
-- ``replaceWith(replString)`` - returns a parse action that simply returns the
- replString; useful when using transformString, or converting HTML entities, as in::
+- ``replace_with(repl_string)`` - returns a parse action that simply returns the
+ ``repl_string``; useful when using ``transform_string``, or converting HTML entities, as in::
- nbsp = Literal(" ").setParseAction( replaceWith("<BLANK>") )
+ nbsp = Literal(" ").set_parse_action(replace_with("<BLANK>"))
-- ``keepOriginalText``- (deprecated, use originalTextFor_ instead) restores any internal whitespace or suppressed
+- ``original_text_for``- restores any internal whitespace or suppressed
text within the tokens for a matched parse
expression. This is especially useful when defining expressions
- for scanString or transformString applications.
+ for ``scan_string`` or ``transform_string`` applications.
-- ``withAttribute( *args, **kwargs )`` - helper to create a validating parse action to be used with start tags created
- with ``makeXMLTags`` or ``makeHTMLTags``. Use ``withAttribute`` to qualify a starting tag
+- ``with_attribute(*args, **kwargs)`` - helper to create a validating parse action to be used with start tags created
+ with ``make_xml_tags`` or ``make_html_tags``. Use ``with_attribute`` to qualify a starting tag
with a required attribute value, to avoid false matches on common tags such as
``<TD>`` or ``<DIV>``.
- ``withAttribute`` can be called with:
+ ``with_attribute`` can be called with:
- keyword arguments, as in ``(class="Customer", align="right")``, or
- a list of name-value tuples, as in ``(("ns1:class", "Customer"), ("ns2:align", "right"))``
An attribute can be specified to have the special value
- ``withAttribute.ANY_VALUE``, which will match any value - use this to
+ ``with_attribute.ANY_VALUE``, which will match any value - use this to
ensure that an attribute is present but any attribute value is
acceptable.
-- ``downcaseTokens`` - converts all matched tokens to lowercase
-
-- ``upcaseTokens`` - converts all matched tokens to uppercase
-
-- ``matchOnlyAtCol(columnNumber)`` - a parse action that verifies that
+- ``match_only_at_col(column_number)`` - a parse action that verifies that
an expression was matched at a particular column, raising a
- ParseException if matching at a different column number; useful when parsing
+ ``ParseException`` if matching at a different column number; useful when parsing
tabular data
+- ``common.convert_to_integer()`` - converts all matched tokens to uppercase
+
+- ``common.convert_to_float()`` - converts all matched tokens to uppercase
+
+- ``common.convert_to_date()`` - converts matched token to a datetime.date
+
+- ``common.convert_to_datetime()`` - converts matched token to a datetime.datetime
+
+- ``common.strip_html_tags()`` - removes HTML tags from matched token
+
+- ``common.downcase_tokens()`` - converts all matched tokens to lowercase
+
+- ``common.upcase_tokens()`` - converts all matched tokens to uppercase
Common string and token constants
- ``empty`` - a global ``Empty()``; will always match
-- ``sglQuotedString`` - a string of characters enclosed in 's; may
+- ``sgl_quoted_string`` - a string of characters enclosed in 's; may
include whitespace, but not newlines
-- ``dblQuotedString`` - a string of characters enclosed in "s; may
+- ``dbl_quoted_string`` - a string of characters enclosed in "s; may
include whitespace, but not newlines
-- ``quotedString`` - ``sglQuotedString | dblQuotedString``
+- ``quoted_string`` - ``sgl_quoted_string | dbl_quoted_string``
-- ``cStyleComment`` - a comment block delimited by ``'/*'`` and ``'*/'`` sequences; can span
+- ``c_style_comment`` - a comment block delimited by ``'/*'`` and ``'*/'`` sequences; can span
multiple lines, but does not support nesting of comments
-- ``htmlComment`` - a comment block delimited by ``'<!--'`` and ``'-->'`` sequences; can span
+- ``html_comment`` - a comment block delimited by ``'<!--'`` and ``'-->'`` sequences; can span
multiple lines, but does not support nesting of comments
-- ``commaSeparatedList`` - similar to ``delimitedList``, except that the
+- ``comma_separated_list`` - similar to ``delimited_list``, except that the
list expressions can be any text value, or a quoted string; quoted strings can
safely include commas without incorrectly breaking the string into two tokens
-- ``restOfLine`` - all remaining printable characters up to but not including the next
+- ``rest_of_line`` - all remaining printable characters up to but not including the next
newline
+
+- ``common.integer`` - an integer with no leading sign; parsed token is converted to int
+
+- ``common.hex_integer`` - a hexadecimal integer; parsed token is converted to int
+
+- ``common.signed_integer`` - an integer with optional leading sign; parsed token is converted to int
+
+- ``common.fraction`` - signed_integer '/' signed_integer; parsed tokens are converted to float
+
+- ``common.mixed_integer`` - signed_integer '-' fraction; parsed tokens are converted to float
+
+- ``common.real`` - real number; parsed tokens are converted to float
+
+- ``common.sci_real`` - real number with optional scientific notation; parsed tokens are convert to float
+
+- ``common.number`` - any numeric expression; parsed tokens are returned as converted by the matched expression
+
+- ``common.fnumber`` - any numeric expression; parsed tokens are converted to float
+
+- ``common.identifier`` - a programming identifier (follows Python's syntax convention of leading alpha or "_",
+ followed by 0 or more alpha, num, or "_")
+
+- ``common.ipv4_address`` - IPv4 address
+
+- ``common.ipv6_address`` - IPv6 address
+
+- ``common.mac_address`` - MAC address (with ":", "-", or "." delimiters)
+
+- ``common.iso8601_date`` - date in ``YYYY-MM-DD`` format
+
+- ``common.iso8601_datetime`` - datetime in ``YYYY-MM-DDThh:mm:ss.s(Z|+-00:00)`` format; trailing seconds,
+ milliseconds, and timezone optional; accepts separating ``'T'`` or ``' '``
+
+- ``common.url`` - matches URL strings and returns a ParseResults with named fields like those returned
+ by ``urllib.parse.urlparse()``
+
+
+Generating Railroad Diagrams
+============================
+Grammars are conventionally represented in what are called "railroad diagrams", which allow you to visually follow
+the sequence of tokens in a grammar along lines which are a bit like train tracks. You might want to generate a
+railroad diagram for your grammar in order to better understand it yourself, or maybe to communicate it to others.
+
+Usage
+-----
+To generate a railroad diagram in pyparsing, you first have to install pyparsing with the ``diagrams`` extra.
+To do this, just run ``pip install pyparsing[diagrams]``, and make sure you add ``pyparsing[diagrams]`` to any
+``setup.py`` or ``requirements.txt`` that specifies pyparsing as a dependency.
+
+Create your parser as you normally would. Then call ``create_diagram()``, passing the name of an output HTML file.::
+
+ street_address = Word(nums).set_name("house_number") + Word(alphas)[1, ...].set_name("street_name")
+ street_address.set_name("street_address")
+ street_address.create_diagram("street_address_diagram.html")
+
+This will result in the railroad diagram being written to ``street_address_diagram.html``.
+
+Diagrams usually will vertically wrap expressions containing more than 3 terms. You can override this by
+passing the `vertical` argument to `create_diagram` with a larger value.
+
+Example
+-------
+You can view an example railroad diagram generated from `a pyparsing grammar for
+SQL SELECT statements <_static/sql_railroad.html>`_.
+
+Naming tip
+----------
+Parser elements that are separately named will be broken out as their own sub-diagrams. As a short-cut alternative
+to going through and adding ``.set_name()`` calls on all your sub-expressions, you can use ``autoname_elements()`` after
+defining your complete grammar. For example::
+
+ a = pp.Literal("a")
+ b = pp.Literal("b").set_name("bbb")
+ pp.autoname_elements()
+
+`a` will get named "a", while `b` will keep its name "bbb".
+
+Customization
+-------------
+You can customize the resulting diagram in a few ways.
+To do so, run ``pyparsing.diagrams.to_railroad`` to convert your grammar into a form understood by the
+`railroad-diagrams <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md>`_ module, and
+then ``pyparsing.diagrams.railroad_to_html`` to convert that into an HTML document. For example::
+
+ from pyparsing.diagram import to_railroad, railroad_to_html
+
+ with open('output.html', 'w') as fp:
+ railroad = to_railroad(my_grammar)
+ fp.write(railroad_to_html(railroad))
+
+This will result in the railroad diagram being written to ``output.html``
+
+You can then pass in additional keyword arguments to ``pyparsing.diagrams.to_railroad``, which will be passed
+into the ``Diagram()`` constructor of the underlying library,
+`as explained here <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md#diagrams>`_.
+
+In addition, you can edit global options in the underlying library, by editing constants::
+
+ from pyparsing.diagram import to_railroad, railroad_to_html
+ import railroad
+
+ railroad.DIAGRAM_CLASS = "my-custom-class"
+ my_railroad = to_railroad(my_grammar)
+
+These options `are documented here <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md#options>`_.
+
+Finally, you can edit the HTML produced by ``pyparsing.diagrams.railroad_to_html`` by passing in certain keyword
+arguments that will be used in the HTML template. Currently, these are:
+
+- ``head``: A string containing HTML to use in the ``<head>`` tag. This might be a stylesheet or other metadata
+
+- ``body``: A string containing HTML to use in the ``<body>`` tag, above the actual diagram. This might consist of a
+ heading, description, or JavaScript.
+
+If you want to provide a custom stylesheet using the ``head`` keyword, you can make use of the following CSS classes:
+
+- ``railroad-group``: A group containing everything relating to a given element group (ie something with a heading)
+
+- ``railroad-heading``: The title for each group
+
+- ``railroad-svg``: A div containing only the diagram SVG for each group
+
+- ``railroad-description``: A div containing the group description (unused)
--- /dev/null
+<!DOCTYPE html>
+<html>
+<head>
+
+ <style type="text/css">
+ .railroad-heading {
+ font-family: monospace;
+ }
+ </style>
+
+</head>
+<body>
+
+
+ <div class="railroad-group">
+ <h1 class="railroad-heading">Forward</h1>
+ <div class="railroad-description"></div>
+ <div class="railroad-svg">
+ <svg class="railroad-diagram" height="172" viewBox="0 0 1344.0 172" width="1344.0" xmlns="http://www.w3.org/2000/svg">
+<g transform="translate(.5 .5)">
+<style>/* <![CDATA[ */
+ svg.railroad-diagram {
+ background-color:hsl(30,20%,95%);
+ }
+ svg.railroad-diagram path {
+ stroke-width:3;
+ stroke:black;
+ fill:rgba(0,0,0,0);
+ }
+ svg.railroad-diagram text {
+ font:bold 14px monospace;
+ text-anchor:middle;
+ }
+ svg.railroad-diagram text.label{
+ text-anchor:start;
+ }
+ svg.railroad-diagram text.comment{
+ font:italic 12px monospace;
+ }
+ svg.railroad-diagram rect{
+ stroke-width:3;
+ stroke:black;
+ fill:hsl(120,100%,90%);
+ }
+ svg.railroad-diagram rect.group-box {
+ stroke: gray;
+ stroke-dasharray: 10 5;
+ fill: none;
+ }
+
+/* ]]> */
+</style><g>
+<path d="M20 53v20m10 -20v20m-10 -10h20"></path></g><path d="M40 63h10"></path><g>
+<path d="M50 63h0.0"></path><path d="M1294.0 63h0.0"></path><g>
+<path d="M50.0 63h0.0"></path><path d="M1048.0 63h0.0"></path><g>
+<path d="M50.0 63h0.0"></path><path d="M662.0 63h0.0"></path><g>
+<path d="M50.0 63h0.0"></path><path d="M571.0 63h0.0"></path><g class="terminal">
+<path d="M50.0 63h0.0"></path><path d="M138.0 63h0.0"></path><rect height="22" rx="10" ry="10" width="88.0" x="50.0" y="52"></rect><text x="94.0" y="67">'select'</text></g><path d="M138.0 63h10"></path><g>
+<path d="M148.0 63h0.0"></path><path d="M571.0 63h0.0"></path><path d="M148.0 63h20"></path><g class="terminal">
+<path d="M168.0 63h168.75"></path><path d="M382.25 63h168.75"></path><rect height="22" rx="10" ry="10" width="45.5" x="336.75" y="52"></rect><text x="359.5" y="67">'*'</text></g><path d="M551.0 63h20"></path><path d="M148.0 63a10 10 0 0 1 10 10v42a10 10 0 0 0 10 10"></path><g>
+<path d="M168.0 125h0.0"></path><path d="M551.0 125h0.0"></path><g class="non-terminal">
+<path d="M168.0 125h0.0"></path><path d="M281.5 125h0.0"></path><rect height="22" width="113.5" x="168.0" y="114"></rect><text x="224.75" y="129">column name</text></g><path d="M281.5 125h10"></path><g>
+<path d="M291.5 125h0.0"></path><path d="M551.0 125h0.0"></path><path d="M291.5 125a10 10 0 0 0 10 -10v-23a10 10 0 0 1 10 -10"></path><g>
+<path d="M311.5 82h219.5"></path></g><path d="M531.0 82a10 10 0 0 1 10 10v23a10 10 0 0 0 10 10"></path><path d="M291.5 125h20"></path><g>
+<path d="M311.5 125h0.0"></path><path d="M531.0 125h0.0"></path><path d="M311.5 125h10"></path><g>
+<path d="M321.5 125h0.0"></path><path d="M521.0 125h0.0"></path><g>
+<path d="M321.5 125h0.0"></path><path d="M387.5 125h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="66" x="321.5" y="106"></rect><g class="terminal">
+<path d="M321.5 125h10.25"></path><path d="M377.25 125h10.25"></path><rect height="22" rx="10" ry="10" width="45.5" x="331.75" y="114"></rect><text x="354.5" y="129">','</text></g><g>
+<path d="M321.5 98h0.0"></path><path d="M387.5 98h0.0"></path><text class="comment" x="354.5" y="103">Suppress</text></g></g><path d="M387.5 125h10"></path><path d="M397.5 125h10"></path><g class="non-terminal">
+<path d="M407.5 125h0.0"></path><path d="M521.0 125h0.0"></path><rect height="22" width="113.5" x="407.5" y="114"></rect><text x="464.25" y="129">column name</text></g></g><path d="M521.0 125h10"></path><path d="M321.5 125a10 10 0 0 0 -10 10v7a10 10 0 0 0 10 10"></path><g>
+<path d="M321.5 152h199.5"></path></g><path d="M521.0 152a10 10 0 0 0 10 -10v-7a10 10 0 0 0 -10 -10"></path></g><path d="M531.0 125h20"></path></g></g><path d="M551.0 125a10 10 0 0 0 10 -10v-42a10 10 0 0 1 10 -10"></path></g></g><path d="M571.0 63h10"></path><path d="M581.0 63h10"></path><g class="terminal">
+<path d="M591.0 63h0.0"></path><path d="M662.0 63h0.0"></path><rect height="22" rx="10" ry="10" width="71.0" x="591.0" y="52"></rect><text x="626.5" y="67">'from'</text></g></g><path d="M662.0 63h10"></path><path d="M672.0 63h10"></path><g>
+<path d="M682.0 63h0.0"></path><path d="M1048.0 63h0.0"></path><g class="non-terminal">
+<path d="M682.0 63h0.0"></path><path d="M787.0 63h0.0"></path><rect height="22" width="105.0" x="682.0" y="52"></rect><text x="734.5" y="67">table name</text></g><path d="M787.0 63h10"></path><g>
+<path d="M797.0 63h0.0"></path><path d="M1048.0 63h0.0"></path><path d="M797.0 63a10 10 0 0 0 10 -10v-23a10 10 0 0 1 10 -10"></path><g>
+<path d="M817.0 20h211.0"></path></g><path d="M1028.0 20a10 10 0 0 1 10 10v23a10 10 0 0 0 10 10"></path><path d="M797.0 63h20"></path><g>
+<path d="M817.0 63h0.0"></path><path d="M1028.0 63h0.0"></path><path d="M817.0 63h10"></path><g>
+<path d="M827.0 63h0.0"></path><path d="M1018.0 63h0.0"></path><g>
+<path d="M827.0 63h0.0"></path><path d="M893.0 63h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="66" x="827.0" y="44"></rect><g class="terminal">
+<path d="M827.0 63h10.25"></path><path d="M882.75 63h10.25"></path><rect height="22" rx="10" ry="10" width="45.5" x="837.25" y="52"></rect><text x="860.0" y="67">','</text></g><g>
+<path d="M827.0 36h0.0"></path><path d="M893.0 36h0.0"></path><text class="comment" x="860.0" y="41">Suppress</text></g></g><path d="M893.0 63h10"></path><path d="M903.0 63h10"></path><g class="non-terminal">
+<path d="M913.0 63h0.0"></path><path d="M1018.0 63h0.0"></path><rect height="22" width="105.0" x="913.0" y="52"></rect><text x="965.5" y="67">table name</text></g></g><path d="M1018.0 63h10"></path><path d="M827.0 63a10 10 0 0 0 -10 10v7a10 10 0 0 0 10 10"></path><g>
+<path d="M827.0 90h191.0"></path></g><path d="M1018.0 90a10 10 0 0 0 10 -10v-7a10 10 0 0 0 -10 -10"></path></g><path d="M1028.0 63h20"></path></g></g></g><path d="M1048.0 63h10"></path><g>
+<path d="M1058.0 63h0.0"></path><path d="M1294.0 63h0.0"></path><path d="M1058.0 63a10 10 0 0 0 10 -10v0a10 10 0 0 1 10 -10"></path><g>
+<path d="M1078.0 43h196.0"></path></g><path d="M1274.0 43a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M1058.0 63h20"></path><g>
+<path d="M1078.0 63h0.0"></path><path d="M1274.0 63h0.0"></path><g class="terminal">
+<path d="M1078.0 63h0.0"></path><path d="M1157.5 63h0.0"></path><rect height="22" rx="10" ry="10" width="79.5" x="1078.0" y="52"></rect><text x="1117.75" y="67">'where'</text></g><path d="M1157.5 63h10"></path><path d="M1167.5 63h10"></path><g class="non-terminal">
+<path d="M1177.5 63h0.0"></path><path d="M1274.0 63h0.0"></path><rect height="22" width="96.5" x="1177.5" y="52"></rect><text x="1225.75" y="67">'or' term</text></g></g><path d="M1274.0 63h20"></path></g></g><path d="M1294.0 63h10"></path><path d="M 1304.0 63 h 20 m -10 -10 v 20 m 10 -20 v 20"></path></g></svg>
+ </div>
+ </div>
+
+ <div class="railroad-group">
+ <h1 class="railroad-heading">column name</h1>
+ <div class="railroad-description"></div>
+ <div class="railroad-svg">
+ <svg class="railroad-diagram" height="110" viewBox="0 0 706.5 110" width="706.5" xmlns="http://www.w3.org/2000/svg">
+<g transform="translate(.5 .5)">
+<style>/* <![CDATA[ */
+ svg.railroad-diagram {
+ background-color:hsl(30,20%,95%);
+ }
+ svg.railroad-diagram path {
+ stroke-width:3;
+ stroke:black;
+ fill:rgba(0,0,0,0);
+ }
+ svg.railroad-diagram text {
+ font:bold 14px monospace;
+ text-anchor:middle;
+ }
+ svg.railroad-diagram text.label{
+ text-anchor:start;
+ }
+ svg.railroad-diagram text.comment{
+ font:italic 12px monospace;
+ }
+ svg.railroad-diagram rect{
+ stroke-width:3;
+ stroke:black;
+ fill:hsl(120,100%,90%);
+ }
+ svg.railroad-diagram rect.group-box {
+ stroke: gray;
+ stroke-dasharray: 10 5;
+ fill: none;
+ }
+
+/* ]]> */
+</style><g>
+<path d="M20 53v20m10 -20v20m-10 -10h20"></path></g><path d="M40 63h10"></path><g>
+<path d="M50 63h0.0"></path><path d="M656.5 63h0.0"></path><g>
+<path d="M50.0 63h0.0"></path><path d="M285.5 63h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="235.5" x="50.0" y="44"></rect><g class="terminal">
+<path d="M50.0 63h10.0"></path><path d="M275.5 63h10.0"></path><rect height="22" rx="10" ry="10" width="215.5" x="60.0" y="52"></rect><text x="167.75" y="67">W:(A-Za-z, $0-9A-Z_a-z)</text></g><g>
+<path d="M50.0 36h0.0"></path><path d="M130.0 36h0.0"></path><text class="comment" x="90.0" y="41">identifier</text></g></g><path d="M285.5 63h10"></path><g>
+<path d="M295.5 63h0.0"></path><path d="M656.5 63h0.0"></path><path d="M295.5 63a10 10 0 0 0 10 -10v-23a10 10 0 0 1 10 -10"></path><g>
+<path d="M315.5 20h321.0"></path></g><path d="M636.5 20a10 10 0 0 1 10 10v23a10 10 0 0 0 10 10"></path><path d="M295.5 63h20"></path><g>
+<path d="M315.5 63h0.0"></path><path d="M636.5 63h0.0"></path><path d="M315.5 63h10"></path><g>
+<path d="M325.5 63h0.0"></path><path d="M626.5 63h0.0"></path><g class="terminal">
+<path d="M325.5 63h0.0"></path><path d="M371.0 63h0.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="325.5" y="52"></rect><text x="348.25" y="67">'.'</text></g><path d="M371.0 63h10"></path><path d="M381.0 63h10"></path><g>
+<path d="M391.0 63h0.0"></path><path d="M626.5 63h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="235.5" x="391.0" y="44"></rect><g class="terminal">
+<path d="M391.0 63h10.0"></path><path d="M616.5 63h10.0"></path><rect height="22" rx="10" ry="10" width="215.5" x="401.0" y="52"></rect><text x="508.75" y="67">W:(A-Za-z, $0-9A-Z_a-z)</text></g><g>
+<path d="M391.0 36h0.0"></path><path d="M471.0 36h0.0"></path><text class="comment" x="431.0" y="41">identifier</text></g></g></g><path d="M626.5 63h10"></path><path d="M325.5 63a10 10 0 0 0 -10 10v7a10 10 0 0 0 10 10"></path><g>
+<path d="M325.5 90h301.0"></path></g><path d="M626.5 90a10 10 0 0 0 10 -10v-7a10 10 0 0 0 -10 -10"></path></g><path d="M636.5 63h20"></path></g></g><path d="M656.5 63h10"></path><path d="M 666.5 63 h 20 m -10 -10 v 20 m 10 -20 v 20"></path></g></svg>
+ </div>
+ </div>
+
+ <div class="railroad-group">
+ <h1 class="railroad-heading">table name</h1>
+ <div class="railroad-description"></div>
+ <div class="railroad-svg">
+ <svg class="railroad-diagram" height="110" viewBox="0 0 706.5 110" width="706.5" xmlns="http://www.w3.org/2000/svg">
+<g transform="translate(.5 .5)">
+<style>/* <![CDATA[ */
+ svg.railroad-diagram {
+ background-color:hsl(30,20%,95%);
+ }
+ svg.railroad-diagram path {
+ stroke-width:3;
+ stroke:black;
+ fill:rgba(0,0,0,0);
+ }
+ svg.railroad-diagram text {
+ font:bold 14px monospace;
+ text-anchor:middle;
+ }
+ svg.railroad-diagram text.label{
+ text-anchor:start;
+ }
+ svg.railroad-diagram text.comment{
+ font:italic 12px monospace;
+ }
+ svg.railroad-diagram rect{
+ stroke-width:3;
+ stroke:black;
+ fill:hsl(120,100%,90%);
+ }
+ svg.railroad-diagram rect.group-box {
+ stroke: gray;
+ stroke-dasharray: 10 5;
+ fill: none;
+ }
+
+/* ]]> */
+</style><g>
+<path d="M20 53v20m10 -20v20m-10 -10h20"></path></g><path d="M40 63h10"></path><g>
+<path d="M50 63h0.0"></path><path d="M656.5 63h0.0"></path><g>
+<path d="M50.0 63h0.0"></path><path d="M285.5 63h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="235.5" x="50.0" y="44"></rect><g class="terminal">
+<path d="M50.0 63h10.0"></path><path d="M275.5 63h10.0"></path><rect height="22" rx="10" ry="10" width="215.5" x="60.0" y="52"></rect><text x="167.75" y="67">W:(A-Za-z, $0-9A-Z_a-z)</text></g><g>
+<path d="M50.0 36h0.0"></path><path d="M130.0 36h0.0"></path><text class="comment" x="90.0" y="41">identifier</text></g></g><path d="M285.5 63h10"></path><g>
+<path d="M295.5 63h0.0"></path><path d="M656.5 63h0.0"></path><path d="M295.5 63a10 10 0 0 0 10 -10v-23a10 10 0 0 1 10 -10"></path><g>
+<path d="M315.5 20h321.0"></path></g><path d="M636.5 20a10 10 0 0 1 10 10v23a10 10 0 0 0 10 10"></path><path d="M295.5 63h20"></path><g>
+<path d="M315.5 63h0.0"></path><path d="M636.5 63h0.0"></path><path d="M315.5 63h10"></path><g>
+<path d="M325.5 63h0.0"></path><path d="M626.5 63h0.0"></path><g class="terminal">
+<path d="M325.5 63h0.0"></path><path d="M371.0 63h0.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="325.5" y="52"></rect><text x="348.25" y="67">'.'</text></g><path d="M371.0 63h10"></path><path d="M381.0 63h10"></path><g>
+<path d="M391.0 63h0.0"></path><path d="M626.5 63h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="235.5" x="391.0" y="44"></rect><g class="terminal">
+<path d="M391.0 63h10.0"></path><path d="M616.5 63h10.0"></path><rect height="22" rx="10" ry="10" width="215.5" x="401.0" y="52"></rect><text x="508.75" y="67">W:(A-Za-z, $0-9A-Z_a-z)</text></g><g>
+<path d="M391.0 36h0.0"></path><path d="M471.0 36h0.0"></path><text class="comment" x="431.0" y="41">identifier</text></g></g></g><path d="M626.5 63h10"></path><path d="M325.5 63a10 10 0 0 0 -10 10v7a10 10 0 0 0 10 10"></path><g>
+<path d="M325.5 90h301.0"></path></g><path d="M626.5 90a10 10 0 0 0 10 -10v-7a10 10 0 0 0 -10 -10"></path></g><path d="M636.5 63h20"></path></g></g><path d="M656.5 63h10"></path><path d="M 666.5 63 h 20 m -10 -10 v 20 m 10 -20 v 20"></path></g></svg>
+ </div>
+ </div>
+
+ <div class="railroad-group">
+ <h1 class="railroad-heading">'or' term</h1>
+ <div class="railroad-description"></div>
+ <div class="railroad-svg">
+ <svg class="railroad-diagram" height="125" viewBox="0 0 788.0 125" width="788.0" xmlns="http://www.w3.org/2000/svg">
+<g transform="translate(.5 .5)">
+<style>/* <![CDATA[ */
+ svg.railroad-diagram {
+ background-color:hsl(30,20%,95%);
+ }
+ svg.railroad-diagram path {
+ stroke-width:3;
+ stroke:black;
+ fill:rgba(0,0,0,0);
+ }
+ svg.railroad-diagram text {
+ font:bold 14px monospace;
+ text-anchor:middle;
+ }
+ svg.railroad-diagram text.label{
+ text-anchor:start;
+ }
+ svg.railroad-diagram text.comment{
+ font:italic 12px monospace;
+ }
+ svg.railroad-diagram rect{
+ stroke-width:3;
+ stroke:black;
+ fill:hsl(120,100%,90%);
+ }
+ svg.railroad-diagram rect.group-box {
+ stroke: gray;
+ stroke-dasharray: 10 5;
+ fill: none;
+ }
+
+/* ]]> */
+</style><g>
+<path d="M20 45v20m10 -20v20m-10 -10h20"></path></g><g>
+<path d="M40 55h0.0"></path><path d="M748.0 55h0.0"></path><path d="M40.0 55h20"></path><g>
+<path d="M60.0 55h0.0"></path><path d="M728.0 55h0.0"></path><g>
+<path d="M60.0 55h0.0"></path><path d="M384.0 55h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="324.0" x="60.0" y="36"></rect><g>
+<path d="M60.0 55h10.0"></path><path d="M374.0 55h10.0"></path><g>
+<path d="M70.0 55h0.0"></path><path d="M249.0 55h0.0"></path><g class="non-terminal">
+<path d="M70.0 55h0.0"></path><path d="M175.0 55h0.0"></path><rect height="22" width="105.0" x="70.0" y="44"></rect><text x="122.5" y="59">'and' term</text></g><path d="M175.0 55h10"></path><path d="M185.0 55h10"></path><g class="terminal">
+<path d="M195.0 55h0.0"></path><path d="M249.0 55h0.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="195.0" y="44"></rect><text x="222.0" y="59">'or'</text></g></g><path d="M249.0 55h10"></path><path d="M259.0 55h10"></path><g class="non-terminal">
+<path d="M269.0 55h0.0"></path><path d="M374.0 55h0.0"></path><rect height="22" width="105.0" x="269.0" y="44"></rect><text x="321.5" y="59">'and' term</text></g></g><g>
+<path d="M60.0 28h0.0"></path><path d="M91.0 28h0.0"></path><text class="comment" x="75.5" y="33">_FB</text></g></g><path d="M384.0 55h10"></path><path d="M394.0 55h10"></path><g>
+<path d="M404.0 55h0.0"></path><path d="M728.0 55h0.0"></path><g class="non-terminal">
+<path d="M404.0 55h0.0"></path><path d="M509.0 55h0.0"></path><rect height="22" width="105.0" x="404.0" y="44"></rect><text x="456.5" y="59">'and' term</text></g><path d="M509.0 55h10"></path><path d="M519.0 55h10"></path><g>
+<path d="M529.0 55h0.0"></path><path d="M728.0 55h0.0"></path><path d="M529.0 55h10"></path><g>
+<path d="M539.0 55h0.0"></path><path d="M718.0 55h0.0"></path><g class="terminal">
+<path d="M539.0 55h0.0"></path><path d="M593.0 55h0.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="539.0" y="44"></rect><text x="566.0" y="59">'or'</text></g><path d="M593.0 55h10"></path><path d="M603.0 55h10"></path><g class="non-terminal">
+<path d="M613.0 55h0.0"></path><path d="M718.0 55h0.0"></path><rect height="22" width="105.0" x="613.0" y="44"></rect><text x="665.5" y="59">'and' term</text></g></g><path d="M718.0 55h10"></path><path d="M539.0 55a10 10 0 0 0 -10 10v0a10 10 0 0 0 10 10"></path><g>
+<path d="M539.0 75h179.0"></path></g><path d="M718.0 75a10 10 0 0 0 10 -10v0a10 10 0 0 0 -10 -10"></path></g></g></g><path d="M728.0 55h20"></path><path d="M40.0 55a10 10 0 0 1 10 10v19a10 10 0 0 0 10 10"></path><g class="non-terminal">
+<path d="M60.0 94h281.5"></path><path d="M446.5 94h281.5"></path><rect height="22" width="105.0" x="341.5" y="83"></rect><text x="394.0" y="98">'and' term</text></g><path d="M728.0 94a10 10 0 0 0 10 -10v-19a10 10 0 0 1 10 -10"></path></g><path d="M 748.0 55 h 20 m -10 -10 v 20 m 10 -20 v 20"></path></g></svg>
+ </div>
+ </div>
+
+ <div class="railroad-group">
+ <h1 class="railroad-heading">'and' term</h1>
+ <div class="railroad-description"></div>
+ <div class="railroad-svg">
+ <svg class="railroad-diagram" height="125" viewBox="0 0 805.0 125" width="805.0" xmlns="http://www.w3.org/2000/svg">
+<g transform="translate(.5 .5)">
+<style>/* <![CDATA[ */
+ svg.railroad-diagram {
+ background-color:hsl(30,20%,95%);
+ }
+ svg.railroad-diagram path {
+ stroke-width:3;
+ stroke:black;
+ fill:rgba(0,0,0,0);
+ }
+ svg.railroad-diagram text {
+ font:bold 14px monospace;
+ text-anchor:middle;
+ }
+ svg.railroad-diagram text.label{
+ text-anchor:start;
+ }
+ svg.railroad-diagram text.comment{
+ font:italic 12px monospace;
+ }
+ svg.railroad-diagram rect{
+ stroke-width:3;
+ stroke:black;
+ fill:hsl(120,100%,90%);
+ }
+ svg.railroad-diagram rect.group-box {
+ stroke: gray;
+ stroke-dasharray: 10 5;
+ fill: none;
+ }
+
+/* ]]> */
+</style><g>
+<path d="M20 45v20m10 -20v20m-10 -10h20"></path></g><g>
+<path d="M40 55h0.0"></path><path d="M765.0 55h0.0"></path><path d="M40.0 55h20"></path><g>
+<path d="M60.0 55h0.0"></path><path d="M745.0 55h0.0"></path><g>
+<path d="M60.0 55h0.0"></path><path d="M392.5 55h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="332.5" x="60.0" y="36"></rect><g>
+<path d="M60.0 55h10.0"></path><path d="M382.5 55h10.0"></path><g>
+<path d="M70.0 55h0.0"></path><path d="M257.5 55h0.0"></path><g class="non-terminal">
+<path d="M70.0 55h0.0"></path><path d="M175.0 55h0.0"></path><rect height="22" width="105.0" x="70.0" y="44"></rect><text x="122.5" y="59">'not' term</text></g><path d="M175.0 55h10"></path><path d="M185.0 55h10"></path><g class="terminal">
+<path d="M195.0 55h0.0"></path><path d="M257.5 55h0.0"></path><rect height="22" rx="10" ry="10" width="62.5" x="195.0" y="44"></rect><text x="226.25" y="59">'and'</text></g></g><path d="M257.5 55h10"></path><path d="M267.5 55h10"></path><g class="non-terminal">
+<path d="M277.5 55h0.0"></path><path d="M382.5 55h0.0"></path><rect height="22" width="105.0" x="277.5" y="44"></rect><text x="330.0" y="59">'not' term</text></g></g><g>
+<path d="M60.0 28h0.0"></path><path d="M91.0 28h0.0"></path><text class="comment" x="75.5" y="33">_FB</text></g></g><path d="M392.5 55h10"></path><path d="M402.5 55h10"></path><g>
+<path d="M412.5 55h0.0"></path><path d="M745.0 55h0.0"></path><g class="non-terminal">
+<path d="M412.5 55h0.0"></path><path d="M517.5 55h0.0"></path><rect height="22" width="105.0" x="412.5" y="44"></rect><text x="465.0" y="59">'not' term</text></g><path d="M517.5 55h10"></path><path d="M527.5 55h10"></path><g>
+<path d="M537.5 55h0.0"></path><path d="M745.0 55h0.0"></path><path d="M537.5 55h10"></path><g>
+<path d="M547.5 55h0.0"></path><path d="M735.0 55h0.0"></path><g class="terminal">
+<path d="M547.5 55h0.0"></path><path d="M610.0 55h0.0"></path><rect height="22" rx="10" ry="10" width="62.5" x="547.5" y="44"></rect><text x="578.75" y="59">'and'</text></g><path d="M610.0 55h10"></path><path d="M620.0 55h10"></path><g class="non-terminal">
+<path d="M630.0 55h0.0"></path><path d="M735.0 55h0.0"></path><rect height="22" width="105.0" x="630.0" y="44"></rect><text x="682.5" y="59">'not' term</text></g></g><path d="M735.0 55h10"></path><path d="M547.5 55a10 10 0 0 0 -10 10v0a10 10 0 0 0 10 10"></path><g>
+<path d="M547.5 75h187.5"></path></g><path d="M735.0 75a10 10 0 0 0 10 -10v0a10 10 0 0 0 -10 -10"></path></g></g></g><path d="M745.0 55h20"></path><path d="M40.0 55a10 10 0 0 1 10 10v19a10 10 0 0 0 10 10"></path><g class="non-terminal">
+<path d="M60.0 94h290.0"></path><path d="M455.0 94h290.0"></path><rect height="22" width="105.0" x="350.0" y="83"></rect><text x="402.5" y="98">'not' term</text></g><path d="M745.0 94a10 10 0 0 0 10 -10v-19a10 10 0 0 1 10 -10"></path></g><path d="M 765.0 55 h 20 m -10 -10 v 20 m 10 -20 v 20"></path></g></svg>
+ </div>
+ </div>
+
+ <div class="railroad-group">
+ <h1 class="railroad-heading">'not' term</h1>
+ <div class="railroad-description"></div>
+ <div class="railroad-svg">
+ <svg class="railroad-diagram" height="372" viewBox="0 0 1622.5 372" width="1622.5" xmlns="http://www.w3.org/2000/svg">
+<g transform="translate(.5 .5)">
+<style>/* <![CDATA[ */
+ svg.railroad-diagram {
+ background-color:hsl(30,20%,95%);
+ }
+ svg.railroad-diagram path {
+ stroke-width:3;
+ stroke:black;
+ fill:rgba(0,0,0,0);
+ }
+ svg.railroad-diagram text {
+ font:bold 14px monospace;
+ text-anchor:middle;
+ }
+ svg.railroad-diagram text.label{
+ text-anchor:start;
+ }
+ svg.railroad-diagram text.comment{
+ font:italic 12px monospace;
+ }
+ svg.railroad-diagram rect{
+ stroke-width:3;
+ stroke:black;
+ fill:hsl(120,100%,90%);
+ }
+ svg.railroad-diagram rect.group-box {
+ stroke: gray;
+ stroke-dasharray: 10 5;
+ fill: none;
+ }
+
+/* ]]> */
+</style><g>
+<path d="M20 45v20m10 -20v20m-10 -10h20"></path></g><g>
+<path d="M40 55h0.0"></path><path d="M1582.5 55h0.0"></path><path d="M40.0 55h20"></path><g>
+<path d="M60.0 55h528.75"></path><path d="M1033.75 55h528.75"></path><g>
+<path d="M588.75 55h0.0"></path><path d="M796.25 55h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="207.5" x="588.75" y="36"></rect><g>
+<path d="M588.75 55h10.0"></path><path d="M786.25 55h10.0"></path><g class="terminal">
+<path d="M598.75 55h0.0"></path><path d="M661.25 55h0.0"></path><rect height="22" rx="10" ry="10" width="62.5" x="598.75" y="44"></rect><text x="630.0" y="59">'not'</text></g><path d="M661.25 55h10"></path><path d="M671.25 55h10"></path><g class="non-terminal">
+<path d="M681.25 55h0.0"></path><path d="M786.25 55h0.0"></path><rect height="22" width="105.0" x="681.25" y="44"></rect><text x="733.75" y="59">'not' term</text></g></g><g>
+<path d="M588.75 28h0.0"></path><path d="M619.75 28h0.0"></path><text class="comment" x="604.25" y="33">_FB</text></g></g><path d="M796.25 55h10"></path><path d="M806.25 55h10"></path><g>
+<path d="M816.25 55h0.0"></path><path d="M1033.75 55h0.0"></path><g>
+<path d="M816.25 55h0.0"></path><path d="M918.75 55h0.0"></path><path d="M816.25 55a10 10 0 0 0 10 -10v0a10 10 0 0 1 10 -10"></path><g>
+<path d="M836.25 35h62.5"></path></g><path d="M898.75 35a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M816.25 55h20"></path><g class="terminal">
+<path d="M836.25 55h0.0"></path><path d="M898.75 55h0.0"></path><rect height="22" rx="10" ry="10" width="62.5" x="836.25" y="44"></rect><text x="867.5" y="59">'not'</text></g><path d="M898.75 55h20"></path></g><path d="M918.75 55h10"></path><g class="non-terminal">
+<path d="M928.75 55h0.0"></path><path d="M1033.75 55h0.0"></path><rect height="22" width="105.0" x="928.75" y="44"></rect><text x="981.25" y="59">'not' term</text></g></g></g><path d="M1562.5 55h20"></path><path d="M40.0 55a10 10 0 0 1 10 10v27a10 10 0 0 0 10 10"></path><g>
+<path d="M60.0 102h0.0"></path><path d="M1562.5 102h0.0"></path><path d="M60.0 102h20"></path><g>
+<path d="M80.0 102h0.0"></path><path d="M1542.5 102h0.0"></path><path d="M80.0 102h20"></path><g>
+<path d="M100.0 102h0.0"></path><path d="M1522.5 102h0.0"></path><path d="M100.0 102h20"></path><g>
+<path d="M120.0 102h0.0"></path><path d="M1502.5 102h0.0"></path><path d="M120.0 102h20"></path><g>
+<path d="M140.0 102h0.0"></path><path d="M1482.5 102h0.0"></path><g>
+<path d="M140.0 102h0.0"></path><path d="M1366.0 102h0.0"></path><g class="non-terminal">
+<path d="M140.0 102h0.0"></path><path d="M253.5 102h0.0"></path><rect height="22" width="113.5" x="140.0" y="91"></rect><text x="196.75" y="106">column name</text></g><path d="M253.5 102h10"></path><g>
+<path d="M263.5 102h0.0"></path><path d="M1366.0 102h0.0"></path><path d="M263.5 102a10 10 0 0 0 10 -10v0a10 10 0 0 1 10 -10h978.5"></path><path d="M359.0 122h987.0a10 10 0 0 0 10 -10v0a10 10 0 0 1 10 -10"></path><path d="M263.5 102h10"></path><g class="terminal">
+<path d="M273.5 102h10.0"></path><path d="M329.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="283.5" y="91"></rect><text x="306.25" y="106">'='</text></g><path d="M339.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M339.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M359.0 102h10.0"></path><path d="M423.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="369.0" y="91"></rect><text x="396.0" y="106">'!='</text></g><path d="M433.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M433.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M453.0 102h10.0"></path><path d="M517.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="463.0" y="91"></rect><text x="490.0" y="106">'<='</text></g><path d="M527.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M527.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M547.0 102h10.0"></path><path d="M602.5 102h10.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="557.0" y="91"></rect><text x="579.75" y="106">'<'</text></g><path d="M612.5 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M612.5 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M632.5 102h10.0"></path><path d="M696.5 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="642.5" y="91"></rect><text x="669.5" y="106">'>='</text></g><path d="M706.5 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M706.5 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M726.5 102h10.0"></path><path d="M782.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="736.5" y="91"></rect><text x="759.25" y="106">'>'</text></g><path d="M792.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M792.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M812.0 102h10.0"></path><path d="M876.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="822.0" y="91"></rect><text x="849.0" y="106">'EQ'</text></g><path d="M886.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M886.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M906.0 102h10.0"></path><path d="M970.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="916.0" y="91"></rect><text x="943.0" y="106">'NE'</text></g><path d="M980.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M980.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M1000.0 102h10.0"></path><path d="M1064.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="1010.0" y="91"></rect><text x="1037.0" y="106">'LT'</text></g><path d="M1074.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M1074.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M1094.0 102h10.0"></path><path d="M1158.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="1104.0" y="91"></rect><text x="1131.0" y="106">'LE'</text></g><path d="M1168.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M1168.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M1188.0 102h10.0"></path><path d="M1252.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="1198.0" y="91"></rect><text x="1225.0" y="106">'GT'</text></g><path d="M1262.0 102a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><path d="M1262.0 82a10 10 0 0 1 10 10v0a10 10 0 0 0 10 10"></path><g class="terminal">
+<path d="M1282.0 102h10.0"></path><path d="M1346.0 102h10.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="1292.0" y="91"></rect><text x="1319.0" y="106">'GE'</text></g><path d="M1356.0 102h10"></path></g></g><path d="M1366.0 102h10"></path><path d="M1376.0 102h10"></path><g class="non-terminal">
+<path d="M1386.0 102h0.0"></path><path d="M1482.5 102h0.0"></path><rect height="22" width="96.5" x="1386.0" y="91"></rect><text x="1434.25" y="106">Unnamed 2</text></g></g><path d="M1482.5 102h20"></path><path d="M120.0 102a10 10 0 0 1 10 10v51a10 10 0 0 0 10 10"></path><g>
+<path d="M140.0 173h327.5"></path><path d="M1155.0 173h327.5"></path><g>
+<path d="M467.5 173h0.0"></path><path d="M655.0 173h0.0"></path><g class="non-terminal">
+<path d="M467.5 173h0.0"></path><path d="M581.0 173h0.0"></path><rect height="22" width="113.5" x="467.5" y="162"></rect><text x="524.25" y="177">column name</text></g><path d="M581.0 173h10"></path><path d="M591.0 173h10"></path><g class="terminal">
+<path d="M601.0 173h0.0"></path><path d="M655.0 173h0.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="601.0" y="162"></rect><text x="628.0" y="177">'in'</text></g></g><path d="M655.0 173h10"></path><path d="M665.0 173h10"></path><g>
+<path d="M675.0 173h0.0"></path><path d="M1155.0 173h0.0"></path><g>
+<path d="M675.0 173h0.0"></path><path d="M1089.5 173h0.0"></path><g class="terminal">
+<path d="M675.0 173h0.0"></path><path d="M720.5 173h0.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="675.0" y="162"></rect><text x="697.75" y="177">'('</text></g><path d="M720.5 173h10"></path><path d="M730.5 173h10"></path><g>
+<path d="M740.5 173h0.0"></path><path d="M1089.5 173h0.0"></path><g class="non-terminal">
+<path d="M740.5 173h0.0"></path><path d="M837.0 173h0.0"></path><rect height="22" width="96.5" x="740.5" y="162"></rect><text x="788.75" y="177">Unnamed 2</text></g><path d="M837.0 173h10"></path><g>
+<path d="M847.0 173h0.0"></path><path d="M1089.5 173h0.0"></path><path d="M847.0 173a10 10 0 0 0 10 -10v-23a10 10 0 0 1 10 -10"></path><g>
+<path d="M867.0 130h202.5"></path></g><path d="M1069.5 130a10 10 0 0 1 10 10v23a10 10 0 0 0 10 10"></path><path d="M847.0 173h20"></path><g>
+<path d="M867.0 173h0.0"></path><path d="M1069.5 173h0.0"></path><path d="M867.0 173h10"></path><g>
+<path d="M877.0 173h0.0"></path><path d="M1059.5 173h0.0"></path><g>
+<path d="M877.0 173h0.0"></path><path d="M943.0 173h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="66" x="877.0" y="154"></rect><g class="terminal">
+<path d="M877.0 173h10.25"></path><path d="M932.75 173h10.25"></path><rect height="22" rx="10" ry="10" width="45.5" x="887.25" y="162"></rect><text x="910.0" y="177">','</text></g><g>
+<path d="M877.0 146h0.0"></path><path d="M943.0 146h0.0"></path><text class="comment" x="910.0" y="151">Suppress</text></g></g><path d="M943.0 173h10"></path><path d="M953.0 173h10"></path><g class="non-terminal">
+<path d="M963.0 173h0.0"></path><path d="M1059.5 173h0.0"></path><rect height="22" width="96.5" x="963.0" y="162"></rect><text x="1011.25" y="177">Unnamed 2</text></g></g><path d="M1059.5 173h10"></path><path d="M877.0 173a10 10 0 0 0 -10 10v7a10 10 0 0 0 10 10"></path><g>
+<path d="M877.0 200h182.5"></path></g><path d="M1059.5 200a10 10 0 0 0 10 -10v-7a10 10 0 0 0 -10 -10"></path></g><path d="M1069.5 173h20"></path></g></g></g><path d="M1089.5 173h10"></path><path d="M1099.5 173h10"></path><g class="terminal">
+<path d="M1109.5 173h0.0"></path><path d="M1155.0 173h0.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="1109.5" y="162"></rect><text x="1132.25" y="177">')'</text></g></g></g><path d="M1482.5 173a10 10 0 0 0 10 -10v-51a10 10 0 0 1 10 -10"></path></g><path d="M1502.5 102h20"></path><path d="M100.0 102a10 10 0 0 1 10 10v97a10 10 0 0 0 10 10"></path><g>
+<path d="M120.0 219h482.25"></path><path d="M1020.25 219h482.25"></path><g>
+<path d="M602.25 219h0.0"></path><path d="M789.75 219h0.0"></path><g class="non-terminal">
+<path d="M602.25 219h0.0"></path><path d="M715.75 219h0.0"></path><rect height="22" width="113.5" x="602.25" y="208"></rect><text x="659.0" y="223">column name</text></g><path d="M715.75 219h10"></path><path d="M725.75 219h10"></path><g class="terminal">
+<path d="M735.75 219h0.0"></path><path d="M789.75 219h0.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="735.75" y="208"></rect><text x="762.75" y="223">'in'</text></g></g><path d="M789.75 219h10"></path><path d="M799.75 219h10"></path><g>
+<path d="M809.75 219h0.0"></path><path d="M1020.25 219h0.0"></path><g>
+<path d="M809.75 219h0.0"></path><path d="M954.75 219h0.0"></path><g class="terminal">
+<path d="M809.75 219h0.0"></path><path d="M855.25 219h0.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="809.75" y="208"></rect><text x="832.5" y="223">'('</text></g><path d="M855.25 219h10"></path><path d="M865.25 219h10"></path><g class="non-terminal">
+<path d="M875.25 219h0.0"></path><path d="M954.75 219h0.0"></path><rect height="22" width="79.5" x="875.25" y="208"></rect><text x="915.0" y="223">Forward</text></g></g><path d="M954.75 219h10"></path><path d="M964.75 219h10"></path><g class="terminal">
+<path d="M974.75 219h0.0"></path><path d="M1020.25 219h0.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="974.75" y="208"></rect><text x="997.5" y="223">')'</text></g></g></g><path d="M1502.5 219a10 10 0 0 0 10 -10v-97a10 10 0 0 1 10 -10"></path></g><path d="M1522.5 102h20"></path><path d="M80.0 102a10 10 0 0 1 10 10v127a10 10 0 0 0 10 10"></path><g>
+<path d="M100.0 249h515.75"></path><path d="M1006.75 249h515.75"></path><g>
+<path d="M615.75 249h0.0"></path><path d="M803.25 249h0.0"></path><g class="non-terminal">
+<path d="M615.75 249h0.0"></path><path d="M729.25 249h0.0"></path><rect height="22" width="113.5" x="615.75" y="238"></rect><text x="672.5" y="253">column name</text></g><path d="M729.25 249h10"></path><path d="M739.25 249h10"></path><g class="terminal">
+<path d="M749.25 249h0.0"></path><path d="M803.25 249h0.0"></path><rect height="22" rx="10" ry="10" width="54.0" x="749.25" y="238"></rect><text x="776.25" y="253">'is'</text></g></g><path d="M803.25 249h10"></path><g>
+<path d="M813.25 249h0.0"></path><path d="M1006.75 249h0.0"></path><path d="M813.25 249h20"></path><g class="terminal">
+<path d="M833.25 249h41.25"></path><path d="M945.5 249h41.25"></path><rect height="22" rx="10" ry="10" width="71.0" x="874.5" y="238"></rect><text x="910.0" y="253">'null'</text></g><path d="M986.75 249h20"></path><path d="M813.25 249a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10"></path><g>
+<path d="M833.25 279h0.0"></path><path d="M986.75 279h0.0"></path><g class="terminal">
+<path d="M833.25 279h0.0"></path><path d="M895.75 279h0.0"></path><rect height="22" rx="10" ry="10" width="62.5" x="833.25" y="268"></rect><text x="864.5" y="283">'not'</text></g><path d="M895.75 279h10"></path><path d="M905.75 279h10"></path><g class="terminal">
+<path d="M915.75 279h0.0"></path><path d="M986.75 279h0.0"></path><rect height="22" rx="10" ry="10" width="71.0" x="915.75" y="268"></rect><text x="951.25" y="283">'null'</text></g></g><path d="M986.75 279a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10"></path></g></g><path d="M1522.5 249a10 10 0 0 0 10 -10v-127a10 10 0 0 1 10 -10"></path></g><path d="M1542.5 102h20"></path><path d="M60.0 102a10 10 0 0 1 10 10v211a10 10 0 0 0 10 10"></path><g>
+<path d="M80.0 333h597.0"></path><path d="M945.5 333h597.0"></path><g>
+<path d="M677.0 333h0.0"></path><path d="M859.5 333h0.0"></path><g>
+<path d="M677.0 333h0.0"></path><path d="M743.0 333h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="66" x="677.0" y="314"></rect><g class="terminal">
+<path d="M677.0 333h10.25"></path><path d="M732.75 333h10.25"></path><rect height="22" rx="10" ry="10" width="45.5" x="687.25" y="322"></rect><text x="710.0" y="337">'('</text></g><g>
+<path d="M677.0 306h0.0"></path><path d="M743.0 306h0.0"></path><text class="comment" x="710.0" y="311">Suppress</text></g></g><path d="M743.0 333h10"></path><path d="M753.0 333h10"></path><g class="non-terminal">
+<path d="M763.0 333h0.0"></path><path d="M859.5 333h0.0"></path><rect height="22" width="96.5" x="763.0" y="322"></rect><text x="811.25" y="337">'or' term</text></g></g><path d="M859.5 333h10"></path><path d="M869.5 333h10"></path><g>
+<path d="M879.5 333h0.0"></path><path d="M945.5 333h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="66" x="879.5" y="314"></rect><g class="terminal">
+<path d="M879.5 333h10.25"></path><path d="M935.25 333h10.25"></path><rect height="22" rx="10" ry="10" width="45.5" x="889.75" y="322"></rect><text x="912.5" y="337">')'</text></g><g>
+<path d="M879.5 306h0.0"></path><path d="M945.5 306h0.0"></path><text class="comment" x="912.5" y="311">Suppress</text></g></g></g><path d="M1542.5 333a10 10 0 0 0 10 -10v-211a10 10 0 0 1 10 -10"></path></g><path d="M1562.5 102a10 10 0 0 0 10 -10v-27a10 10 0 0 1 10 -10"></path></g><path d="M 1582.5 55 h 20 m -10 -10 v 20 m 10 -20 v 20"></path></g></svg>
+ </div>
+ </div>
+
+ <div class="railroad-group">
+ <h1 class="railroad-heading">Unnamed 2</h1>
+ <div class="railroad-description"></div>
+ <div class="railroad-svg">
+ <svg class="railroad-diagram" height="278" viewBox="0 0 787.0 278" width="787.0" xmlns="http://www.w3.org/2000/svg">
+<g transform="translate(.5 .5)">
+<style>/* <![CDATA[ */
+ svg.railroad-diagram {
+ background-color:hsl(30,20%,95%);
+ }
+ svg.railroad-diagram path {
+ stroke-width:3;
+ stroke:black;
+ fill:rgba(0,0,0,0);
+ }
+ svg.railroad-diagram text {
+ font:bold 14px monospace;
+ text-anchor:middle;
+ }
+ svg.railroad-diagram text.label{
+ text-anchor:start;
+ }
+ svg.railroad-diagram text.comment{
+ font:italic 12px monospace;
+ }
+ svg.railroad-diagram rect{
+ stroke-width:3;
+ stroke:black;
+ fill:hsl(120,100%,90%);
+ }
+ svg.railroad-diagram rect.group-box {
+ stroke: gray;
+ stroke-dasharray: 10 5;
+ fill: none;
+ }
+
+/* ]]> */
+</style><g>
+<path d="M20 45v20m10 -20v20m-10 -10h20"></path></g><g>
+<path d="M40 55h0.0"></path><path d="M747.0 55h0.0"></path><path d="M40.0 55h20"></path><g>
+<path d="M60.0 55h0.0"></path><path d="M727.0 55h0.0"></path><path d="M60.0 55h20"></path><g>
+<path d="M80.0 55h146.0"></path><path d="M561.0 55h146.0"></path><path d="M226.0 55h20"></path><g>
+<path d="M246.0 55h0.0"></path><path d="M541.0 55h0.0"></path><rect class="group-box" height="38" rx="10" ry="10" width="295.0" x="246.0" y="36"></rect><g class="terminal">
+<path d="M246.0 55h10.0"></path><path d="M531.0 55h10.0"></path><rect height="22" rx="10" ry="10" width="275.0" x="256.0" y="44"></rect><text x="393.5" y="59">Re:('[+-]?(?:\d+\.\d*|\.\d+)')</text></g><g>
+<path d="M246.0 28h0.0"></path><path d="M333.0 28h0.0"></path><text class="comment" x="289.5" y="33">real number</text></g></g><path d="M541.0 55h20"></path><path d="M226.0 55a10 10 0 0 1 10 10v42a10 10 0 0 0 10 10"></path><g>
+<path d="M246.0 117h63.75"></path><path d="M477.25 117h63.75"></path><rect class="group-box" height="38" rx="10" ry="10" width="167.5" x="309.75" y="98"></rect><g class="terminal">
+<path d="M309.75 117h10.0"></path><path d="M467.25 117h10.0"></path><rect height="22" rx="10" ry="10" width="147.5" x="319.75" y="106"></rect><text x="393.5" y="121">Re:('[+-]?\d+')</text></g><g>
+<path d="M309.75 90h0.0"></path><path d="M417.75 90h0.0"></path><text class="comment" x="363.75" y="95">signed integer</text></g></g><path d="M541.0 117a10 10 0 0 0 10 -10v-42a10 10 0 0 1 10 -10"></path></g><path d="M707.0 55h20"></path><path d="M60.0 55a10 10 0 0 1 10 10v104a10 10 0 0 0 10 10"></path><g>
+<path d="M80.0 179h0.0"></path><path d="M707.0 179h0.0"></path><rect class="group-box" height="68" rx="10" ry="10" width="627.0" x="80.0" y="160"></rect><g>
+<path d="M80.0 179h0.0"></path><path d="M707.0 179h0.0"></path><path d="M80.0 179h20"></path><g>
+<path d="M100.0 179h0.0"></path><path d="M687.0 179h0.0"></path><g class="terminal">
+<path d="M100.0 179h0.0"></path><path d="M621.5 179h0.0"></path><rect height="22" rx="10" ry="10" width="521.5" x="100.0" y="168"></rect><text x="360.75" y="183">Re:('"(?:[^"\n\r\\]|(?:"")|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*')</text></g><path d="M621.5 179h10"></path><path d="M631.5 179h10"></path><g class="terminal">
+<path d="M641.5 179h0.0"></path><path d="M687.0 179h0.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="641.5" y="168"></rect><text x="664.25" y="183">'"'</text></g></g><path d="M687.0 179h20"></path><path d="M80.0 179a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10"></path><g>
+<path d="M100.0 209h0.0"></path><path d="M687.0 209h0.0"></path><g class="terminal">
+<path d="M100.0 209h0.0"></path><path d="M621.5 209h0.0"></path><rect height="22" rx="10" ry="10" width="521.5" x="100.0" y="198"></rect><text x="360.75" y="213">Re:("'(?:[^'\n\r\\]|(?:'')|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*")</text></g><path d="M621.5 209h10"></path><path d="M631.5 209h10"></path><g class="terminal">
+<path d="M641.5 209h0.0"></path><path d="M687.0 209h0.0"></path><rect height="22" rx="10" ry="10" width="45.5" x="641.5" y="198"></rect><text x="664.25" y="213">"'"</text></g></g><path d="M687.0 209a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10"></path></g><g>
+<path d="M80.0 152h0.0"></path><path d="M384.0 152h0.0"></path><text class="comment" x="232.0" y="157">quotedString using single or double quotes</text></g></g><path d="M707.0 179a10 10 0 0 0 10 -10v-104a10 10 0 0 1 10 -10"></path></g><path d="M727.0 55h20"></path><path d="M40.0 55a10 10 0 0 1 10 10v172a10 10 0 0 0 10 10"></path><g class="non-terminal">
+<path d="M60.0 247h276.75"></path><path d="M450.25 247h276.75"></path><rect height="22" width="113.5" x="336.75" y="236"></rect><text x="393.5" y="251">column name</text></g><path d="M727.0 247a10 10 0 0 0 10 -10v-172a10 10 0 0 1 10 -10"></path></g><path d="M 747.0 55 h 20 m -10 -10 v 20 m 10 -20 v 20"></path></g></svg>
+ </div>
+ </div>
+
+</body>
+</html>
\ No newline at end of file
-# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
#
import os
import sys
-sys.path.insert(0, os.path.abspath('..'))
+
+sys.path.insert(0, os.path.abspath(".."))
from pyparsing import __version__ as pyparsing_version
# -- Project information -----------------------------------------------------
-project = 'PyParsing'
-copyright = '2018, Paul T. McGuire'
-author = 'Paul T. McGuire'
+project = "PyParsing"
+copyright = "2018-2021, Paul T. McGuire"
+author = "Paul T. McGuire"
# The short X.Y version
version = pyparsing_version
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
- 'sphinx.ext.autodoc',
+ "sphinx.ext.autodoc",
]
# Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+source_suffix = ".rst"
# The master toctree document.
-master_doc = 'index'
+master_doc = "index"
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = "sphinx"
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
-html_theme = 'alabaster'
+html_theme = "alabaster"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = ["_static"]
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
-htmlhelp_basename = 'PyParsingdoc'
+htmlhelp_basename = "PyParsingdoc"
# -- Options for LaTeX output ------------------------------------------------
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
-
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
-
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
-
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
- (master_doc, 'PyParsing.tex', 'PyParsing Documentation',
- 'Paul T. McGuire', 'manual'),
+ (
+ master_doc,
+ "PyParsing.tex",
+ "PyParsing Documentation",
+ "Paul T. McGuire",
+ "manual",
+ ),
]
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
-man_pages = [
- (master_doc, 'pyparsing', 'PyParsing Documentation',
- [author], 1)
-]
+man_pages = [(master_doc, "pyparsing", "PyParsing Documentation", [author], 1)]
# -- Options for Texinfo output ----------------------------------------------
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
- (master_doc, 'PyParsing', 'PyParsing Documentation',
- author, 'PyParsing', 'One line description of project.',
- 'Miscellaneous'),
+ (
+ master_doc,
+ "PyParsing",
+ "PyParsing Documentation",
+ author,
+ "PyParsing",
+ "Python PEG parsing library.",
+ "Miscellaneous",
+ ),
]
# epub_uid = ''
# A list of files that should not be packed into the epub file.
-epub_exclude_files = ['search.html']
+epub_exclude_files = ["search.html"]
# -- Extension configuration -------------------------------------------------
:maxdepth: 2
:caption: Contents:
+ whats_new_in_3_0_0
HowToUsePyparsing
modules
CODE_OF_CONDUCT
.. automodule:: pyparsing
:members:
- :undoc-members:
+ :special-members:
:show-inheritance:
--- /dev/null
+@startuml
+'https://plantuml.com/class-diagram
+
+top to bottom direction
+hide circle
+hide empty members
+'hide empty methods
+skinparam groupInheritance 3
+
+note as N1
+Class Diagram
+---
+<size 18>pyparsing 3.0.0
+<size 18>October, 2021
+end note
+
+N1 <-[hidden]- unicode
+
+package core {
+
+class globals {
+quoted_string
+sgl_quoted_string
+dbl_quoted_string
+delimited_list()
+counted_array()
+match_previous_literal()
+match_previous_expr()
+one_of()
+dict_of()
+original_text_for()
+ungroup()
+nested_expr()
+make_html_tags()
+make_xml_tags()
+common_html_entity
+replace_html_entity()
+class OpAssoc
+infix_notation()
+class IndentedBlock
+c_style_comment
+html_comment
+rest_of_line
+dbl_slash_comment
+cpp_style_comment
+java_style_comment
+python_style_comment
+match_only_at_col()
+replace_with()
+remove_quotes()
+with_attribute()
+with_class()
+trace_parse_action()
+condition_as_parse_action()
+srange()
+token_map()
+autoname_elements()
+}
+
+class ParseResults {
+class List
+{static}from_dict()
+__getitem__()
+__setitem__()
+__contains__()
+__len__()
+__bool__()
+__iter__()
+__reversed__()
+__getattr__()
+__add__()
+__getstate__()
+__setstate__()
+__getnewargs__()
+__dir__()
+as_dict()
+as_list()
+dump()
+get_name()
+items()
+keys()
+values()
+haskeys()
+pop()
+get()
+insert()
+append()
+extend()
+clear()
+copy()
+get_name()
+pprint()
+}
+
+class ParseBaseException #ffffff {
+{static} explain_exception()
+explain()
+mark_input_line()
+line
+lineno
+column
+parser_element
+}
+class ParseException
+class ParseFatalException
+class ParseSyntaxException
+
+ParseBaseException <|-- ParseException
+ParseBaseException <|-- ParseFatalException
+ParseFatalException <|-- ParseSyntaxException
+
+class ParserElement {
+name: str
+results_name: str
+---
+{classifier} enable_packrat()
+{classifier} enable_left_recursion()
+{classifier} disable_memoization()
+{classifier} set_default_whitespace_chars()
+{classifier} inline_literals_using()
+{classifier} reset_cache()
+
+{static} verbose_stacktrace
+
+operator + () -> And
+operator - () -> And.ErrorStop
+operator | () -> MatchFirst
+operator ^ () -> Or
+operator & () -> Each
+operator ~ () -> NotAny
+operator [] () -> _MultipleMatch
+add_condition()
+add_parse_action()
+set_parse_action()
+copy()
+ignore(expr)
+leave_whitespace()
+parse_with_tabs()
+suppress()
+set_break()
+set_debug()
+set_debug_actions()
+set_name()
+set_results_name()
+parse_string()
+scan_string()
+search_string()
+transform_string()
+split()
+run_tests()
+create_diagram()
+}
+class Token #ffffff
+class ParseExpression #ffffff {
+exprs: list[ParserElement]
+}
+class ParseElementEnhance #ffffff {
+expr: ParserElement
+}
+class _PositionToken #ffffff
+class Char
+class Empty
+class White
+class Keyword {
+{static} set_default_keyword_chars(chars: str)
+}
+class CaselessKeyword
+class NoMatch
+class Literal
+class Regex
+class Word {
+'Word(init_chars: str, body_chars: str, min: int, \nmax: int, exact: int, as_keyword: bool, exclude_chars: str)
+}
+class CharsNotIn
+class QuotedString
+
+class And
+class Or
+class MatchFirst
+class Each
+
+class OneOrMore
+class ZeroOrMore
+class SkipTo
+class Group
+class Forward {
+operator <<= ()
+}
+
+class LineStart
+class LineEnd
+class StringStart
+class StringEnd
+class WordStart
+class WordEnd
+class AtLineStart
+class AtStringStart
+
+class FollowedBy
+class _MultipleMatch #ffffff
+class PrecededBy
+class Located
+class Opt
+class TokenConverter #ffffff
+
+class Combine
+class Group
+class Dict
+class Suppress
+
+ParserElement <|-- Token
+ParserElement <|----- ParseExpression
+Token <|-- _PositionToken
+ParserElement <|----- ParseElementEnhance
+
+'ParseElementEnhance ---> ParserElement
+'ParseExpression ---> "*" ParserElement
+
+
+Token <|-- Empty
+Token <|-- CloseMatch
+Token <|--- NoMatch
+Token <|---- Literal
+Token <|---- Word
+Token <|---- Keyword
+Token <|--- Regex
+Token <|--- CharsNotIn
+Token <|--- White
+Token <|---- QuotedString
+Word <|-- Char
+Literal <|-- CaselessLiteral
+Keyword <|-- CaselessKeyword
+
+ParseExpression <|-- And
+ParseExpression <|-- Or
+ParseExpression <|-- MatchFirst
+ParseExpression <|-- Each
+
+ParseElementEnhance <|-- SkipTo
+ParseElementEnhance <|--- Forward
+ParseElementEnhance <|-- Located
+ParseElementEnhance <|--- _MultipleMatch
+_MultipleMatch <|-- OneOrMore
+_MultipleMatch <|-- ZeroOrMore
+ParseElementEnhance <|-- NotAny
+ParseElementEnhance <|-- FollowedBy
+ParseElementEnhance <|-- PrecededBy
+ParseElementEnhance <|-- Opt
+ParseElementEnhance <|--- TokenConverter
+ParseElementEnhance <|-- AtStringStart
+ParseElementEnhance <|-- AtLineStart
+TokenConverter <|-- Group
+TokenConverter <|-- Dict
+TokenConverter <|-- Suppress
+TokenConverter <|-- Combine
+
+_PositionToken <|-- LineStart
+_PositionToken <|-- LineEnd
+_PositionToken <|-- WordStart
+_PositionToken <|-- WordEnd
+_PositionToken <|-- StringStart
+_PositionToken <|-- StringEnd
+
+}
+
+package common {
+class " " {
+comma_separated_list
+convert_to_integer()
+convert_to_float()
+integer
+hex_integer
+signed_integer
+fraction
+mixed_integer
+real
+sci_real
+number
+fnumber
+identifier
+ipv4_address
+ipv6_address
+mac_address
+convert_to_date()
+convert_to_datetime()
+iso8601_date
+iso8601_datetime
+uuid
+strip_html_tags()
+upcase_tokens()
+downcase_tokens()
+url
+}
+
+}
+package unicode {
+class unicode_set {
+printables: str
+alphas: str
+nums: str
+alphanums: str
+identchars: str
+identbodychars: str
+}
+class Latin1
+class LatinA
+class LatinB
+class Cyrillic
+class Chinese
+class Thai
+class Japanese {
+class Kanji
+class Hiragana
+class Katakana
+}
+class Greek
+class Hangul
+class Arabic
+class Devanagari
+class Hebrew
+unicode_set <|-- Latin1
+unicode_set <|--- LatinA
+unicode_set <|-- LatinB
+unicode_set <|-- Greek
+unicode_set <|--- Cyrillic
+unicode_set <|--- Chinese
+unicode_set <|--- Japanese
+unicode_set <|--- Hangul
+Chinese <|-- CJK
+Japanese <|-- CJK
+Hangul <|-- CJK
+unicode_set <|-- Thai
+unicode_set <|-- Arabic
+unicode_set <|-- Hebrew
+unicode_set <|--- Devanagari
+
+}
+
+ParserElement <-[hidden] ParseBaseException
+'ParseBaseException <-[hidden] globals
+'globals <-[hidden] ParserElement
+CJK <-[hidden]-- common
+
+@enduml
\ No newline at end of file
--- /dev/null
+=============================
+What's New in Pyparsing 3.0.0
+=============================
+
+:author: Paul McGuire
+
+:date: October, 2021
+
+:abstract: This document summarizes the changes made
+ in the 3.0.0 release of pyparsing.
+
+.. sectnum:: :depth: 4
+
+.. contents:: :depth: 4
+
+
+New Features
+============
+
+PEP-8 naming
+------------
+This release of pyparsing will (finally!) include PEP-8 compatible names and arguments.
+Backward-compatibility is maintained by defining synonyms using the old camelCase names
+pointing to the new snake_case names.
+
+This code written using non-PEP8 names::
+
+ wd = pp.Word(pp.printables, excludeChars="$")
+ wd_list = pp.delimitedList(wd, delim="$")
+ print(wd_list.parseString("dkls$134lkjk$lsd$$").asList())
+
+can now be written as::
+
+ wd = pp.Word(pp.printables, exclude_chars="$")
+ wd_list = pp.delimited_list(wd, delim="$")
+ print(wd_list.parse_string("dkls$134lkjk$lsd$$").as_list())
+
+Pyparsing 3.0 will run both versions of this example.
+
+New code should be written using the PEP-8 compatible names. The compatibility
+synonyms will be removed in a future version of pyparsing.
+
+
+Railroad diagramming
+--------------------
+An excellent new enhancement is the new railroad diagram
+generator for documenting pyparsing parsers.::
+
+ import pyparsing as pp
+
+ # define a simple grammar for parsing street addresses such
+ # as "123 Main Street"
+ # number word...
+ number = pp.Word(pp.nums).set_name("number")
+ name = pp.Word(pp.alphas).set_name("word")[1, ...]
+
+ parser = number("house_number") + name("street")
+ parser.set_name("street address")
+
+ # construct railroad track diagram for this parser and
+ # save as HTML
+ parser.create_diagram('parser_rr_diag.html')
+
+To use this new feature, install the supporting diagramming packages using::
+
+ pip install pyparsing[diagrams]
+
+See more in the examples directory: ``make_diagram.py`` and ``railroad_diagram_demo.py``.
+
+(Railroad diagram enhancement contributed by Michael Milton)
+
+Support for left-recursive parsers
+----------------------------------
+Another significant enhancement in 3.0 is support for left-recursive (LR)
+parsers. Previously, given a left-recursive parser, pyparsing would
+recurse repeatedly until hitting the Python recursion limit. Following
+the methods of the Python PEG parser, pyparsing uses a variation of
+packrat parsing to detect and handle left-recursion during parsing.::
+
+ import pyparsing as pp
+ pp.ParserElement.enable_left_recursion()
+
+ # a common left-recursion definition
+ # define a list of items as 'list + item | item'
+ # BNF:
+ # item_list := item_list item | item
+ # item := word of alphas
+ item_list = pp.Forward()
+ item = pp.Word(pp.alphas)
+ item_list <<= item_list + item | item
+
+ item_list.run_tests("""\
+ To parse or not to parse that is the question
+ """)
+
+Prints::
+
+ ['To', 'parse', 'or', 'not', 'to', 'parse', 'that', 'is', 'the', 'question']
+
+See more examples in ``left_recursion.py`` in the pyparsing examples directory.
+
+(LR parsing support contributed by Max Fischer)
+
+Packrat/memoization enable and disable methods
+----------------------------------------------
+As part of the implementation of left-recursion support, new methods have been added
+to enable and disable packrat parsing.
+
+====================== =======================================================
+Name Description
+---------------------- -------------------------------------------------------
+enable_packrat Enable packrat parsing (with specified cache size)
+enable_left_recursion Enable left-recursion cache
+disable_memoization Disable all internal parsing caches
+====================== =======================================================
+
+Type annotations on all public methods
+--------------------------------------
+Python 3.6 and upward compatible type annotations have been added to most of the
+public methods in pyparsing. This should facilitate developing pyparsing-based
+applications using IDEs for development-time type checking.
+
+New string constants ``identchars`` and ``identbodychars`` to help in defining identifier Word expressions
+----------------------------------------------------------------------------------------------------------
+Two new module-level strings have been added to help when defining identifiers,
+``identchars`` and ``identbodychars``.
+
+Instead of writing::
+
+ import pyparsing as pp
+ identifier = pp.Word(pp.alphas + "_", pp.alphanums + "_")
+
+you will be able to write::
+
+ identifier = pp.Word(pp.indentchars, pp.identbodychars)
+
+Those constants have also been added to all the Unicode string classes::
+
+ import pyparsing as pp
+ ppu = pp.pyparsing_unicode
+
+ cjk_identifier = pp.Word(ppu.CJK.identchars, ppu.CJK.identbodychars)
+ greek_identifier = pp.Word(ppu.Greek.identchars, ppu.Greek.identbodychars)
+
+
+Refactored/added diagnostic flags
+---------------------------------
+Expanded ``__diag__`` and ``__compat__`` to actual classes instead of
+just namespaces, to add some helpful behavior:
+
+- ``pyparsing.enable_diag()`` and ``pyparsing.disable_diag()`` methods to give extra
+ help when setting or clearing flags (detects invalid
+ flag names, detects when trying to set a ``__compat__`` flag
+ that is no longer settable). Use these methods now to
+ set or clear flags, instead of directly setting to ``True`` or
+ ``False``::
+
+ import pyparsing as pp
+ pp.enable_diag(pp.Diagnostics.warn_multiple_tokens_in_named_alternation)
+
+- ``pyparsing.enable_all_warnings()`` is another helper that sets
+ all "warn*" diagnostics to ``True``::
+
+ pp.enable_all_warnings()
+
+- added support for calling ``enable_all_warnings()`` if warnings are enabled
+ using the Python ``-W`` switch, or setting a non-empty value to the environment
+ variable ``PYPARSINGENABLEALLWARNINGS``.
+
+- added new warning, ``warn_on_match_first_with_lshift_operator`` to
+ warn when using ``'<<'`` with a ``'|'`` ``MatchFirst`` operator,
+ which will
+ create an unintended expression due to precedence of operations.
+
+ Example: This statement will erroneously define the ``fwd`` expression
+ as just ``expr_a``, even though ``expr_a | expr_b`` was intended,
+ since ``'<<'`` operator has precedence over ``'|'``::
+
+ fwd << expr_a | expr_b
+
+ To correct this, use the ``'<<='`` operator (preferred) or parentheses
+ to override operator precedence::
+
+ fwd <<= expr_a | expr_b
+
+ or::
+
+ fwd << (expr_a | expr_b)
+
+- ``warn_on_parse_using_empty_Forward`` - warns that a ``Forward``
+ has been included in a grammar, but no expression was
+ attached to it using ``'<<='`` or ``'<<'``
+
+- ``warn_on_assignment_to_Forward`` - warns that a ``Forward`` has
+ been created, but was probably later overwritten by
+ erroneously using ``'='`` instead of ``'<<='`` (this is a common
+ mistake when using Forwards)
+ (**currently not working on PyPy**)
+
+Support for yielding native Python ``list`` and ``dict`` types in place of ``ParseResults``
+-------------------------------------------------------------------------------
+To support parsers that are intended to generate native Python collection
+types such as lists and dicts, the ``Group`` and `Dict` classes now accept an
+additional boolean keyword argument ``aslist`` and `asdict` respectively. See
+the ``jsonParser.py`` example in the ``pyparsing/examples`` source directory for
+how to return types as ``ParseResults`` and as Python collection types, and the
+distinctions in working with the different types.
+
+In addition parse actions that must return a value of list type (which would
+normally be converted internally to a ``ParseResults``) can override this default
+behavior by returning their list wrapped in the new ``ParseResults.List`` class::
+
+ # this parse action tries to return a list, but pyparsing
+ # will convert to a ParseResults
+ def return_as_list_but_still_get_parse_results(tokens):
+ return tokens.asList()
+
+ # this parse action returns the tokens as a list, and pyparsing will
+ # maintain its list type in the final parsing results
+ def return_as_list(tokens):
+ return ParseResults.List(tokens.asList())
+
+This is the mechanism used internally by the ``Group`` class when defined
+using ``aslist=True``.
+
+New Located class to replace locatedExpr helper method
+------------------------------------------------------
+The new ``Located`` class will replace the current ``locatedExpr`` method for
+marking parsed results with the start and end locations of the parsed data in
+the input string. ``locatedExpr`` had several bugs, and returned its results
+in a hard-to-use format (location data and results names were mixed in with
+the located expression's parsed results, and wrapped in an unnecessary extra
+nesting level).
+
+For this code::
+
+ wd = Word(alphas)
+ for match in locatedExpr(wd).search_string("ljsdf123lksdjjf123lkkjj1222"):
+ print(match)
+
+the docs for ``locatedExpr`` show this output::
+
+ [[0, 'ljsdf', 5]]
+ [[8, 'lksdjjf', 15]]
+ [[18, 'lkkjj', 23]]
+
+The parsed values and the start and end locations are merged into a single
+nested ``ParseResults`` (and any results names in the parsed values are also
+merged in with the start and end location names).
+
+Using ``Located``, the output is::
+
+ [0, ['ljsdf'], 5]
+ [8, ['lksdjjf'], 15]
+ [18, ['lkkjj'], 23]
+
+With ``Located``, the parsed expression values and results names are kept
+separate in the second parsed value, and there is no extra grouping level
+on the whole result.
+
+The existing ``locatedExpr`` is retained for backward-compatibility, but will be
+deprecated in a future release.
+
+New AtLineStart and AtStringStart classes
+-----------------------------------------
+As part fixing some matching behavior in LineStart and StringStart, two new
+classes have been added: AtLineStart and AtStringStart.
+
+The following expressions are equivalent::
+
+ LineStart() + expr and AtLineStart(expr)
+ StringStart() + expr and AtStringStart(expr)
+
+LineStart and StringStart now will only match if their related expression is
+actually at the start of the string or current line, without skipping whitespace.::
+
+ (LineStart() + Word(alphas)).parseString("ABC") # passes
+ (LineStart() + Word(alphas)).parseString(" ABC") # fails
+
+LineStart is also smarter about matching at the beginning of the string.
+
+This was the intended behavior previously, but could be bypassed if wrapped
+in other ParserElements.
+
+New IndentedBlock class to replace indentedBlock helper method
+--------------------------------------------------------------
+The new ``IndentedBlock`` class will replace the current ``indentedBlock`` method
+for defining indented blocks of text, similar to Python source code. Using
+``IndentedBlock``, the expression instance itself keeps track of the indent stack,
+so a separate external ``indentStack`` variable is no longer required.
+
+Here is a simple example of an expression containing an alphabetic key, followed
+by an indented list of integers::
+
+ integer = pp.Word(pp.nums)
+ group = pp.Group(pp.Char(pp.alphas) + pp.Group(pp.IndentedBlock(integer)))
+
+parses::
+
+ A
+ 100
+ 101
+ B
+ 200
+ 201
+
+as::
+
+ [['A', [100, 101]], ['B', [200, 201]]]
+
+``IndentedBlock`` may also be used to define a recursive indented block (containing nested
+indented blocks).
+
+The existing ``indentedBlock`` is retained for backward-compatibility, but will be
+deprecated in a future release.
+
+Shortened tracebacks
+--------------------
+Cleaned up default tracebacks when getting a ``ParseException`` when calling
+``parse_string``. Exception traces should now stop at the call in ``parse_string``,
+and not include the internal pyparsing traceback frames. (If the full traceback
+is desired, then set ``ParserElement.verbose_traceback`` to ``True``.)
+
+Improved debug logging
+----------------------
+Debug logging has been improved by:
+
+- Including ``try/match/fail`` logging when getting results from the
+ packrat cache (previously cache hits did not show debug logging).
+ Values returned from the packrat cache are marked with an '*'.
+
+- Improved fail logging, showing the failed expression, text line, and marker where
+ the failure occurred.
+
+- Adding ``with_line_numbers`` to ``pyparsing_testing``. Use ``with_line_numbers``
+ to visualize the data being parsed, with line and column numbers corresponding
+ to the values output when enabling ``set_debug()`` on an expression::
+
+ data = """\
+ A
+ 100"""
+ expr = pp.Word(pp.alphanums).set_name("word").set_debug()
+ print(ppt.with_line_numbers(data))
+ expr[...].parseString(data)
+
+ prints::
+
+ . 1
+ 1234567890
+ 1: A
+ 2: 100
+ Match word at loc 3(1,4)
+ A
+ ^
+ Matched word -> ['A']
+ Match word at loc 11(2,7)
+ 100
+ ^
+ Matched word -> ['100']
+
+New / improved examples
+-----------------------
+- ``number_words.py`` includes a parser/evaluator to parse ``"forty-two"``
+ and return ``42``. Also includes example code to generate a railroad
+ diagram for this parser.
+
+- ``BigQueryViewParser.py`` added to examples directory, submitted
+ by Michael Smedberg.
+
+- ``booleansearchparser.py`` added to examples directory, submitted
+ by xecgr. Builds on searchparser.py, adding support for '*'
+ wildcards and non-Western alphabets.
+
+- Improvements in ``select_parser.py``, to include new SQL syntax
+ from SQLite, submitted by Robert Coup.
+
+- Off-by-one bug found in the ``roman_numerals.py`` example, a bug
+ that has been there for about 14 years! Submitted by
+ Jay Pedersen.
+
+- A simplified Lua parser has been added to the examples
+ (``lua_parser.py``).
+
+- Demonstration of defining a custom Unicode set for cuneiform
+ symbols, as well as simple Cuneiform->Python conversion is included
+ in ``cuneiform_python.py``.
+
+- Fixed bug in ``delta_time.py`` example, when using a quantity
+ of seconds/minutes/hours/days > 999.
+
+Other new features
+------------------
+- ``url`` expression added to ``pyparsing_common``, with named fields for
+ common fields in URLs. See the updated ``urlExtractorNew.py`` file in the
+ ``examples`` directory. Submitted by Wolfgang Fahl.
+
+- ``delimited_list`` now supports an additional flag ``allow_trailing_delim``,
+ to optionally parse an additional delimiter at the end of the list.
+ Submitted by Kazantcev Andrey.
+
+- Added global method ``autoname_elements()`` to call ``set_name()`` on all locally
+ defined ``ParserElements`` that haven't been explicitly named using ``set_name()``, using
+ their local variable name. Useful for setting names on multiple elements when
+ creating a railroad diagram::
+
+ a = pp.Literal("a")
+ b = pp.Literal("b").set_name("bbb")
+ pp.autoname_elements()
+
+ `a` will get named "a", while `b` will keep its name "bbb".
+
+- Enhanced default strings created for ``Word`` expressions, now showing
+ string ranges if possible. ``Word(alphas)`` would formerly
+ print as ``W:(ABCD...)``, now prints as ``W:(A-Za-z)``.
+
+- Better exception messages to show full word where an exception occurred.::
+
+ Word(alphas)[...].parse_string("abc 123", parse_all=True)
+
+ Was::
+
+ pyparsing.ParseException: Expected end of text, found '1' (at char 4), (line:1, col:5)
+
+ Now::
+
+ pyparsing.exceptions.ParseException: Expected end of text, found '123' (at char 4), (line:1, col:5)
+
+- Using ``...`` for ``SkipTo`` can now be wrapped in ``Suppress`` to suppress
+ the skipped text from the returned parse results.::
+
+ source = "lead in START relevant text END trailing text"
+ start_marker = Keyword("START")
+ end_marker = Keyword("END")
+ find_body = Suppress(...) + start_marker + ... + end_marker
+ print(find_body.parse_string(source).dump())
+
+ Prints::
+
+ ['START', 'relevant text ', 'END']
+ - _skipped: ['relevant text ']
+
+- Added ``ignore_whitespace(recurse:bool = True)`` and added a
+ ``recurse`` argument to ``leave_whitespace``, both added to provide finer
+ control over pyparsing's whitespace skipping. Contributed by
+ Michael Milton.
+
+- Added ``ParserElement.recurse()`` method to make it simpler for
+ grammar utilities to navigate through the tree of expressions in
+ a pyparsing grammar.
+
+- The ``repr()`` string for ``ParseResults`` is now of the form::
+
+ ParseResults([tokens], {named_results})
+
+ The previous form omitted the leading ``ParseResults`` class name,
+ and was easily misinterpreted as a ``tuple`` containing a ``list`` and
+ a ``dict``.
+
+- Minor reformatting of output from ``run_tests`` to make embedded
+ comments more visible.
+
+- New ``pyparsing_test`` namespace, assert methods and classes added to support writing
+ unit tests.
+
+ - ``assertParseResultsEquals``
+ - ``assertParseAndCheckList``
+ - ``assertParseAndCheckDict``
+ - ``assertRunTestResults``
+ - ``assertRaisesParseException``
+ - ``reset_pyparsing_context`` context manager, to restore pyparsing
+ config settings
+
+- Enhanced error messages and error locations when parsing fails on
+ the ``Keyword`` or ``CaselessKeyword`` classes due to the presence of a
+ preceding or trailing keyword character.
+
+- Enhanced the ``Regex`` class to be compatible with re's compiled with the
+ re-equivalent ``regex`` module. Individual expressions can be built with
+ regex compiled expressions using::
+
+ import pyparsing as pp
+ import regex
+
+ # would use regex for this expression
+ integer_parser = pp.Regex(regex.compile(r'\d+'))
+
+- Fixed handling of ``ParseSyntaxExceptions`` raised as part of ``Each``
+ expressions, when sub-expressions contain ``'-'`` backtrack
+ suppression.
+
+- Potential performance enhancement when parsing ``Word``
+ expressions built from ``pyparsing_unicode`` character sets. ``Word`` now
+ internally converts ranges of consecutive characters to regex
+ character ranges (converting ``"0123456789"`` to ``"0-9"`` for instance).
+
+- Added a caseless parameter to the `CloseMatch` class to allow for casing to be
+ ignored when checking for close matches. Contributed by Adrian Edwards.
+
+
+API Changes
+===========
+
+- ``enable_diag()`` and ``disable_diag()`` methods to
+ enable specific diagnostic values (instead of setting them
+ to ``True`` or ``False``). ``enable_all_warnings()`` has
+ also been added.
+
+- ``counted_array`` formerly returned its list of items nested
+ within another list, so that accessing the items required
+ indexing the 0'th element to get the actual list. This
+ extra nesting has been removed. In addition, if there are
+ other metadata fields parsed between the count and the
+ list items, they can be preserved in the resulting list
+ if given results names.
+
+- ``ParseException.explain()`` is now an instance method of
+ ``ParseException``::
+
+ expr = pp.Word(pp.nums) * 3
+ try:
+ expr.parse_string("123 456 A789")
+ except pp.ParseException as pe:
+ print(pe.explain(depth=0))
+
+ prints::
+
+ 123 456 A789
+ ^
+ ParseException: Expected W:(0-9), found 'A789' (at char 8), (line:1, col:9)
+
+ To run explain against other exceptions, use
+ ``ParseException.explain_exception()``.
+
+- Debug actions now take an added keyword argument ``cache_hit``.
+ Now that debug actions are called for expressions matched in the
+ packrat parsing cache, debug actions are now called with this extra
+ flag, set to True. For custom debug actions, it is necessary to add
+ support for this new argument.
+
+- ``ZeroOrMore`` expressions that have results names will now
+ include empty lists for their name if no matches are found.
+ Previously, no named result would be present. Code that tested
+ for the presence of any expressions using ``"if name in results:"``
+ will now always return ``True``. This code will need to change to
+ ``"if name in results and results[name]:"`` or just
+ ``"if results[name]:"``. Also, any parser unit tests that check the
+ ``as_dict()`` contents will now see additional entries for parsers
+ having named ``ZeroOrMore`` expressions, whose values will be ``[]``.
+
+- ``ParserElement.set_default_whitespace_chars`` will now update
+ whitespace characters on all built-in expressions defined
+ in the pyparsing module.
+
+- ``camelCase`` names have been converted to PEP-8 ``snake_case`` names.
+
+ Method names and arguments that were camel case (such as ``parseString``)
+ have been replaced with PEP-8 snake case versions (``parse_string``).
+
+ Backward-compatibility synonyms for all names and arguments have
+ been included, to allow parsers written using the old names to run
+ without change. The synonyms will be removed in a future release.
+ New parser code should be written using the new PEP-8 snake case names.
+
+ ============================== ================================
+ Name Previous name
+ ------------------------------ --------------------------------
+ ParserElement
+ - parse_string parseString
+ - scan_string scanString
+ - search_string searchString
+ - transform_string transformString
+ - add_condition addCondition
+ - add_parse_action addParseAction
+ - can_parse_next canParseNext
+ - default_name defaultName
+ - enable_left_recursion enableLeftRecursion
+ - enable_packrat enablePackrat
+ - ignore_whitespace ignoreWhitespace
+ - inline_literals_using inlineLiteralsUsing
+ - parse_file parseFile
+ - leave_whitespace leaveWhitespace
+ - parse_string parseString
+ - parse_with_tabs parseWithTabs
+ - reset_cache resetCache
+ - run_tests runTests
+ - scan_string scanString
+ - search_string searchString
+ - set_break setBreak
+ - set_debug setDebug
+ - set_debug_actions setDebugActions
+ - set_default_whitespace_chars setDefaultWhitespaceChars
+ - set_fail_action setFailAction
+ - set_name setName
+ - set_parse_action setParseAction
+ - set_results_name setResultsName
+ - set_whitespace_chars setWhitespaceChars
+ - transform_string transformString
+ - try_parse tryParse
+
+ ParseResults
+ - as_list asList
+ - as_dict asDict
+ - get_name getName
+
+ ParseBaseException
+ - parser_element parserElement
+
+ any_open_tag anyOpenTag
+ any_close_tag anyCloseTag
+ c_style_comment cStyleComment
+ common_html_entity commonHTMLEntity
+ condition_as_parse_action conditionAsParseAction
+ counted_array countedArray
+ cpp_style_comment cppStyleComment
+ dbl_quoted_string dblQuotedString
+ dbl_slash_comment dblSlashComment
+ delimited_list delimitedList
+ dict_of dictOf
+ html_comment htmlComment
+ infix_notation infixNotation
+ java_style_comment javaStyleComment
+ line_end lineEnd
+ line_start lineStart
+ make_html_tags makeHTMLTags
+ make_xml_tags makeXMLTags
+ match_only_at_col matchOnlyAtCol
+ match_previous_expr matchPreviousExpr
+ match_previous_literal matchPreviousLiteral
+ nested_expr nestedExpr
+ null_debug_action nullDebugAction
+ one_of oneOf
+ OpAssoc opAssoc
+ original_text_for originalTextFor
+ python_style_comment pythonStyleComment
+ quoted_string quotedString
+ remove_quotes removeQuotes
+ replace_html_entity replaceHTMLEntity
+ replace_with replaceWith
+ rest_of_line restOfLine
+ sgl_quoted_string sglQuotedString
+ string_end stringEnd
+ string_start stringStart
+ token_map tokenMap
+ trace_parse_action traceParseAction
+ unicode_string unicodeString
+ with_attribute withAttribute
+ with_class withClass
+ ============================== ================================
+
+Discontinued Features
+=====================
+
+Python 2.x no longer supported
+------------------------------
+Removed Py2.x support and other deprecated features. Pyparsing
+now requires Python 3.6 or later. If you are using an earlier
+version of Python, you must use a Pyparsing 2.4.x version.
+
+Other discontinued features
+---------------------------
+- ``ParseResults.asXML()`` - if used for debugging, switch
+ to using ``ParseResults.dump()``; if used for data transfer,
+ use ``ParseResults.as_dict()`` to convert to a nested Python
+ dict, which can then be converted to XML or JSON or
+ other transfer format
+
+- ``operatorPrecedence`` synonym for ``infixNotation`` -
+ convert to calling ``infix_notation``
+
+- ``commaSeparatedList`` - convert to using
+ ``pyparsing_common.comma_separated_list``
+
+- ``upcaseTokens`` and ``downcaseTokens`` - convert to using
+ ``pyparsing_common.upcase_tokens`` and ``downcase_tokens``
+
+- ``__compat__.collect_all_And_tokens`` will not be settable to
+ ``False`` to revert to pre-2.3.1 results name behavior -
+ review use of names for ``MatchFirst`` and Or expressions
+ containing ``And`` expressions, as they will return the
+ complete list of parsed tokens, not just the first one.
+ Use ``pyparsing.enable_diag(pyparsing.Diagnostics.warn_multiple_tokens_in_named_alternation)``
+ to help identify those expressions in your parsers that
+ will have changed as a result.
+
+- Removed support for running ``python setup.py test``. The setuptools
+ maintainers consider the ``test`` command deprecated (see
+ <https://github.com/pypa/setuptools/issues/1684>). To run the Pyparsing tests,
+ use the command ``tox``.
+
+
+Fixed Bugs
+==========
+
+- Fixed issue when LineStart() expressions would match input text that was not
+ necessarily at the beginning of a line.
+
+- Fixed bug in regex definitions for ``real`` and ``sci_real`` expressions in
+ ``pyparsing_common``.
+
+- Fixed ``FutureWarning`` raised beginning in Python 3.7 for ``Regex`` expressions
+ containing '[' within a regex set.
+
+- Fixed bug in ``PrecededBy`` which caused infinite recursion.
+
+- Fixed bug in ``CloseMatch`` where end location was incorrectly
+ computed; and updated ``partial_gene_match.py`` example.
+
+- Fixed bug in ``indentedBlock`` with a parser using two different
+ types of nested indented blocks with different indent values,
+ but sharing the same indent stack.
+
+- Fixed bug in ``Each`` when using ``Regex``, when ``Regex`` expression would
+ get parsed twice.
+
+- Fixed bugs in ``Each`` when passed ``OneOrMore`` or ``ZeroOrMore`` expressions:
+ . first expression match could be enclosed in an extra nesting level
+ . out-of-order expressions now handled correctly if mixed with required
+ expressions
+ . results names are maintained correctly for these expression
+
+- Fixed ``FutureWarning`` that sometimes is raised when ``'['`` passed as a
+ character to ``Word``.
+
+- Fixed debug logging to show failure location after whitespace skipping.
+
+- Fixed ``ParseFatalExceptions`` failing to override normal exceptions or expression
+ matches in ``MatchFirst`` expressions.
+
+- Fixed bug in which ``ParseResults`` replaces a collection type value with an invalid
+ type annotation (as a result of changed behavior in Python 3.9).
+
+- Fixed bug in ``ParseResults`` when calling ``__getattr__`` for special double-underscored
+ methods. Now raises ``AttributeError`` for non-existent results when accessing a
+ name starting with '__'.
+
+- Fixed bug in ``Located`` class when used with a results name.
+
+- Fixed bug in ``QuotedString`` class when the escaped quote string is not a
+ repeated character.
+
+Acknowledgments
+===============
+And finally, many thanks to those who helped in the restructuring
+of the pyparsing code base as part of this release. Pyparsing now
+has more standard package structure, more standard unit tests,
+and more standard code formatting (using black). Special thanks
+to jdufresne, klahnakoski, mattcarmody, ckeygusuz,
+tmiguelt, and toonarmycaptain to name just a few.
+
+Thanks also to Michael Milton and Max Fischer, who added some
+significant new features to pyparsing.
\ No newline at end of file
object SearchFindFirst1: TSearchFindFirst\r
Category = 'Search'\r
Caption = 'F&ind First'\r
- Hint = 'Find First|Finds the first occurance of specified text'\r
+ Hint = 'Find First|Finds the first occurrence of specified text'\r
end\r
object CustomizeActionBars1: TCustomizeActionBars\r
Category = 'Tools'\r
"""
-import re,sys
-from pyparsing import Word, alphas, ParseException, Literal, CaselessLiteral \
-, Combine, Optional, nums, Forward, ZeroOrMore, \
- StringEnd, alphanums
+import re, sys
+from pyparsing import (
+ Word,
+ alphas,
+ ParseException,
+ Literal,
+ CaselessLiteral,
+ Combine,
+ Optional,
+ nums,
+ Forward,
+ ZeroOrMore,
+ StringEnd,
+ alphanums,
+)
# Debugging flag can be set to either "debug_flag=True" or "debug_flag=False"
-debug_flag=False
+debug_flag = False
-#----------------------------------------------------------------------------
+# ----------------------------------------------------------------------------
# Variables that hold intermediate parsing results and a couple of
# helper functions.
-exprStack = [] # Holds operators and operands parsed from input.
-targetvar = None # Holds variable name to left of '=' sign in LA equation.
+exprStack = [] # Holds operators and operands parsed from input.
+targetvar = None # Holds variable name to left of '=' sign in LA equation.
-def _pushFirst( str, loc, toks ):
- if debug_flag: print("pushing ", toks[0], "str is ", str)
- exprStack.append( toks[0] )
+def _pushFirst(str, loc, toks):
+ if debug_flag:
+ print("pushing ", toks[0], "str is ", str)
+ exprStack.append(toks[0])
-def _assignVar( str, loc, toks ):
+
+def _assignVar(str, loc, toks):
global targetvar
- targetvar = toks[0]
+ targetvar = toks[0]
+
-#-----------------------------------------------------------------------------
+# -----------------------------------------------------------------------------
# The following statements define the grammar for the parser.
-point = Literal('.')
-e = CaselessLiteral('E')
-plusorminus = Literal('+') | Literal('-')
+point = Literal(".")
+e = CaselessLiteral("E")
+plusorminus = Literal("+") | Literal("-")
number = Word(nums)
-integer = Combine( Optional(plusorminus) + number )
-floatnumber = Combine( integer +
- Optional( point + Optional(number) ) +
- Optional( e + integer )
- )
+integer = Combine(Optional(plusorminus) + number)
+floatnumber = Combine(
+ integer + Optional(point + Optional(number)) + Optional(e + integer)
+)
lbracket = Literal("[")
rbracket = Literal("]")
## can include references to array elements, rows and columns, e.g., a = b[i] + 5.
## Expressions within []'s are not presently supported, so a = b[i+1] will raise
## a ParseException.
-ident = Combine(Word(alphas + '-',alphanums + '_') + \
- ZeroOrMore(lbracket + (Word(alphas + '-',alphanums + '_')|integer) + rbracket) \
- )
-
-plus = Literal( "+" )
-minus = Literal( "-" )
-mult = Literal( "*" )
-div = Literal( "/" )
-outer = Literal( "@" )
-lpar = Literal( "(" ).suppress()
-rpar = Literal( ")" ).suppress()
-addop = plus | minus
+ident = Combine(
+ Word(alphas + "-", alphanums + "_")
+ + ZeroOrMore(lbracket + (Word(alphas + "-", alphanums + "_") | integer) + rbracket)
+)
+
+plus = Literal("+")
+minus = Literal("-")
+mult = Literal("*")
+div = Literal("/")
+outer = Literal("@")
+lpar = Literal("(").suppress()
+rpar = Literal(")").suppress()
+addop = plus | minus
multop = mult | div | outer
-expop = Literal( "^" )
-assignop = Literal( "=" )
+expop = Literal("^")
+assignop = Literal("=")
expr = Forward()
-atom = ( ( e | floatnumber | integer | ident ).setParseAction(_pushFirst) |
- ( lpar + expr.suppress() + rpar )
- )
+atom = (e | floatnumber | integer | ident).setParseAction(_pushFirst) | (
+ lpar + expr.suppress() + rpar
+)
factor = Forward()
-factor << atom + ZeroOrMore( ( expop + factor ).setParseAction( _pushFirst ) )
+factor << atom + ZeroOrMore((expop + factor).setParseAction(_pushFirst))
-term = factor + ZeroOrMore( ( multop + factor ).setParseAction( _pushFirst ) )
-expr << term + ZeroOrMore( ( addop + term ).setParseAction( _pushFirst ) )
+term = factor + ZeroOrMore((multop + factor).setParseAction(_pushFirst))
+expr << term + ZeroOrMore((addop + term).setParseAction(_pushFirst))
equation = (ident + assignop).setParseAction(_assignVar) + expr + StringEnd()
# End of grammar definition
-#-----------------------------------------------------------------------------
+# -----------------------------------------------------------------------------
## The following are helper variables and functions used by the Binary Infix Operator
## Functions described below.
-vprefix = 'V3_'
+vprefix = "V3_"
vplen = len(vprefix)
-mprefix = 'M3_'
+mprefix = "M3_"
mplen = len(mprefix)
## We don't support unary negation for vectors and matrices
-class UnaryUnsupportedError(Exception): pass
+class UnaryUnsupportedError(Exception):
+ pass
+
def _isvec(ident):
- if ident[0] == '-' and ident[1:vplen+1] == vprefix:
- raise UnaryUnsupportedError
- else: return ident[0:vplen] == vprefix
+ if ident[0] == "-" and ident[1 : vplen + 1] == vprefix:
+ raise UnaryUnsupportedError
+ else:
+ return ident[0:vplen] == vprefix
+
def _ismat(ident):
- if ident[0] == '-' and ident[1:mplen+1] == mprefix:
- raise UnaryUnsupportedError
- else: return ident[0:mplen] == mprefix
+ if ident[0] == "-" and ident[1 : mplen + 1] == mprefix:
+ raise UnaryUnsupportedError
+ else:
+ return ident[0:mplen] == mprefix
+
+
+def _isscalar(ident):
+ return not (_isvec(ident) or _ismat(ident))
-def _isscalar(ident): return not (_isvec(ident) or _ismat(ident))
## Binary infix operator (BIO) functions. These are called when the stack evaluator
## pops a binary operator like '+' or '*". The stack evaluator pops the two operand, a and b,
## the appropriate prefix is placed on the outer function for removal later as the stack evaluation
## recurses toward the final assignment statement.
-def _addfunc(a,b):
- if _isscalar(a) and _isscalar(b): return "(%s+%s)"%(a,b)
- if _isvec(a) and _isvec(b): return "%svAdd(%s,%s)"%(vprefix,a[vplen:],b[vplen:])
- if _ismat(a) and _ismat(b): return "%smAdd(%s,%s)"%(mprefix,a[mplen:],b[mplen:])
- else: raise TypeError
-
-def _subfunc(a,b):
- if _isscalar(a) and _isscalar(b): return "(%s-%s)"%(a,b)
- if _isvec(a) and _isvec(b): return "%svSubtract(%s,%s)"%(vprefix,a[vplen:],b[vplen:])
- if _ismat(a) and _ismat(b): return "%smSubtract(%s,%s)"%(mprefix,a[mplen:],b[mplen:])
- else: raise TypeError
-
-def _mulfunc(a,b):
- if _isscalar(a) and _isscalar(b): return "%s*%s"%(a,b)
- if _isvec(a) and _isvec(b): return "vDot(%s,%s)"%(a[vplen:],b[vplen:])
- if _ismat(a) and _ismat(b): return "%smMultiply(%s,%s)"%(mprefix,a[mplen:],b[mplen:])
- if _ismat(a) and _isvec(b): return "%smvMultiply(%s,%s)"%(vprefix,a[mplen:],b[vplen:])
- if _ismat(a) and _isscalar(b): return "%smScale(%s,%s)"%(mprefix,a[mplen:],b)
- if _isvec(a) and _isscalar(b): return "%svScale(%s,%s)"%(vprefix,a[mplen:],b)
- else: raise TypeError
-
-def _outermulfunc(a,b):
- ## The '@' operator is used for the vector outer product.
- if _isvec(a) and _isvec(b):
- return "%svOuterProduct(%s,%s)"%(mprefix,a[vplen:],b[vplen:])
- else: raise TypeError
-
-def _divfunc(a,b):
- ## The '/' operator is used only for scalar division
- if _isscalar(a) and _isscalar(b): return "%s/%s"%(a,b)
- else: raise TypeError
-
-def _expfunc(a,b):
- ## The '^' operator is used for exponentiation on scalars and
- ## as a marker for unary operations on vectors and matrices.
- if _isscalar(a) and _isscalar(b): return "pow(%s,%s)"%(str(a),str(b))
- if _ismat(a) and b=='-1': return "%smInverse(%s)"%(mprefix,a[mplen:])
- if _ismat(a) and b=='T': return "%smTranspose(%s)"%(mprefix,a[mplen:])
- if _ismat(a) and b=='Det': return "mDeterminant(%s)"%(a[mplen:])
- if _isvec(a) and b=='Mag': return "sqrt(vMagnitude2(%s))"%(a[vplen:])
- if _isvec(a) and b=='Mag2': return "vMagnitude2(%s)"%(a[vplen:])
- else: raise TypeError
-
-def _assignfunc(a,b):
- ## The '=' operator is used for assignment
- if _isscalar(a) and _isscalar(b): return "%s=%s"%(a,b)
- if _isvec(a) and _isvec(b): return "vCopy(%s,%s)"%(a[vplen:],b[vplen:])
- if _ismat(a) and _ismat(b): return "mCopy(%s,%s)"%(a[mplen:],b[mplen:])
- else: raise TypeError
+
+def _addfunc(a, b):
+ if _isscalar(a) and _isscalar(b):
+ return "(%s+%s)" % (a, b)
+ if _isvec(a) and _isvec(b):
+ return "%svAdd(%s,%s)" % (vprefix, a[vplen:], b[vplen:])
+ if _ismat(a) and _ismat(b):
+ return "%smAdd(%s,%s)" % (mprefix, a[mplen:], b[mplen:])
+ else:
+ raise TypeError
+
+
+def _subfunc(a, b):
+ if _isscalar(a) and _isscalar(b):
+ return "(%s-%s)" % (a, b)
+ if _isvec(a) and _isvec(b):
+ return "%svSubtract(%s,%s)" % (vprefix, a[vplen:], b[vplen:])
+ if _ismat(a) and _ismat(b):
+ return "%smSubtract(%s,%s)" % (mprefix, a[mplen:], b[mplen:])
+ else:
+ raise TypeError
+
+
+def _mulfunc(a, b):
+ if _isscalar(a) and _isscalar(b):
+ return "%s*%s" % (a, b)
+ if _isvec(a) and _isvec(b):
+ return "vDot(%s,%s)" % (a[vplen:], b[vplen:])
+ if _ismat(a) and _ismat(b):
+ return "%smMultiply(%s,%s)" % (mprefix, a[mplen:], b[mplen:])
+ if _ismat(a) and _isvec(b):
+ return "%smvMultiply(%s,%s)" % (vprefix, a[mplen:], b[vplen:])
+ if _ismat(a) and _isscalar(b):
+ return "%smScale(%s,%s)" % (mprefix, a[mplen:], b)
+ if _isvec(a) and _isscalar(b):
+ return "%svScale(%s,%s)" % (vprefix, a[mplen:], b)
+ else:
+ raise TypeError
+
+
+def _outermulfunc(a, b):
+ ## The '@' operator is used for the vector outer product.
+ if _isvec(a) and _isvec(b):
+ return "%svOuterProduct(%s,%s)" % (mprefix, a[vplen:], b[vplen:])
+ else:
+ raise TypeError
+
+
+def _divfunc(a, b):
+ ## The '/' operator is used only for scalar division
+ if _isscalar(a) and _isscalar(b):
+ return "%s/%s" % (a, b)
+ else:
+ raise TypeError
+
+
+def _expfunc(a, b):
+ ## The '^' operator is used for exponentiation on scalars and
+ ## as a marker for unary operations on vectors and matrices.
+ if _isscalar(a) and _isscalar(b):
+ return "pow(%s,%s)" % (str(a), str(b))
+ if _ismat(a) and b == "-1":
+ return "%smInverse(%s)" % (mprefix, a[mplen:])
+ if _ismat(a) and b == "T":
+ return "%smTranspose(%s)" % (mprefix, a[mplen:])
+ if _ismat(a) and b == "Det":
+ return "mDeterminant(%s)" % (a[mplen:])
+ if _isvec(a) and b == "Mag":
+ return "sqrt(vMagnitude2(%s))" % (a[vplen:])
+ if _isvec(a) and b == "Mag2":
+ return "vMagnitude2(%s)" % (a[vplen:])
+ else:
+ raise TypeError
+
+
+def _assignfunc(a, b):
+ ## The '=' operator is used for assignment
+ if _isscalar(a) and _isscalar(b):
+ return "%s=%s" % (a, b)
+ if _isvec(a) and _isvec(b):
+ return "vCopy(%s,%s)" % (a[vplen:], b[vplen:])
+ if _ismat(a) and _ismat(b):
+ return "mCopy(%s,%s)" % (a[mplen:], b[mplen:])
+ else:
+ raise TypeError
+
## End of BIO func definitions
##----------------------------------------------------------------------------
# Map operator symbols to corresponding BIO funcs
-opn = { "+" : ( _addfunc ),
- "-" : ( _subfunc ),
- "*" : ( _mulfunc ),
- "@" : ( _outermulfunc ),
- "/" : ( _divfunc),
- "^" : ( _expfunc ), }
+opn = {
+ "+": (_addfunc),
+ "-": (_subfunc),
+ "*": (_mulfunc),
+ "@": (_outermulfunc),
+ "/": (_divfunc),
+ "^": (_expfunc),
+}
##----------------------------------------------------------------------------
# Recursive function that evaluates the expression stack
-def _evaluateStack( s ):
- op = s.pop()
- if op in "+-*/@^":
- op2 = _evaluateStack( s )
- op1 = _evaluateStack( s )
- result = opn[op]( op1, op2 )
- if debug_flag: print(result)
- return result
- else:
- return op
+def _evaluateStack(s):
+ op = s.pop()
+ if op in "+-*/@^":
+ op2 = _evaluateStack(s)
+ op1 = _evaluateStack(s)
+ result = opn[op](op1, op2)
+ if debug_flag:
+ print(result)
+ return result
+ else:
+ return op
+
##----------------------------------------------------------------------------
# The parse function that invokes all of the above.
calls that implement the expression.
"""
- global exprStack
+ global exprStack
global targetvar
# Start with a blank exprStack and a blank targetvar
exprStack = []
- targetvar=None
-
- if input_string != '':
- # try parsing the input string
- try:
- L=equation.parseString( input_string )
- except ParseException as err:
- print('Parse Failure', file=sys.stderr)
- print(err.line, file=sys.stderr)
- print(" "*(err.column-1) + "^", file=sys.stderr)
- print(err, file=sys.stderr)
- raise
-
- # show result of parsing the input string
- if debug_flag:
- print(input_string, "->", L)
- print("exprStack=", exprStack)
-
- # Evaluate the stack of parsed operands, emitting C code.
- try:
- result=_evaluateStack(exprStack)
- except TypeError:
- print("Unsupported operation on right side of '%s'.\nCheck for missing or incorrect tags on non-scalar operands."%input_string, file=sys.stderr)
- raise
- except UnaryUnsupportedError:
- print("Unary negation is not supported for vectors and matrices: '%s'"%input_string, file=sys.stderr)
- raise
-
- # Create final assignment and print it.
- if debug_flag: print("var=",targetvar)
- if targetvar != None:
- try:
- result = _assignfunc(targetvar,result)
- except TypeError:
- print("Left side tag does not match right side of '%s'"%input_string, file=sys.stderr)
- raise
- except UnaryUnsupportedError:
- print("Unary negation is not supported for vectors and matrices: '%s'"%input_string, file=sys.stderr)
+ targetvar = None
+
+ if input_string != "":
+ # try parsing the input string
+ try:
+ L = equation.parseString(input_string)
+ except ParseException as err:
+ print("Parse Failure", file=sys.stderr)
+ print(err.line, file=sys.stderr)
+ print(" " * (err.column - 1) + "^", file=sys.stderr)
+ print(err, file=sys.stderr)
raise
- return result
- else:
- print("Empty left side in '%s'"%input_string, file=sys.stderr)
- raise TypeError
-
-##-----------------------------------------------------------------------------------
-def fprocess(infilep,outfilep):
- """
- Scans an input file for LA equations between double square brackets,
- e.g. [[ M3_mymatrix = M3_anothermatrix^-1 ]], and replaces the expression
- with a comment containing the equation followed by nested function calls
- that implement the equation as C code. A trailing semi-colon is appended.
- The equation within [[ ]] should NOT end with a semicolon as that will raise
- a ParseException. However, it is ok to have a semicolon after the right brackets.
-
- Other text in the file is unaltered.
-
- The arguments are file objects (NOT file names) opened for reading and
- writing, respectively.
- """
- pattern = r'\[\[\s*(.*?)\s*\]\]'
- eqn = re.compile(pattern,re.DOTALL)
- s = infilep.read()
- def parser(mo):
- ccode = parse(mo.group(1))
- return "/* %s */\n%s;\nLAParserBufferReset();\n"%(mo.group(1),ccode)
-
- content = eqn.sub(parser,s)
- outfilep.write(content)
-
-##-----------------------------------------------------------------------------------
-def test():
- """
- Tests the parsing of various supported expressions. Raises
- an AssertError if the output is not what is expected. Prints the
- input, expected output, and actual output for all tests.
- """
- print("Testing LAParser")
- testcases = [
- ("Scalar addition","a = b+c","a=(b+c)"),
- ("Vector addition","V3_a = V3_b + V3_c","vCopy(a,vAdd(b,c))"),
- ("Vector addition","V3_a=V3_b+V3_c","vCopy(a,vAdd(b,c))"),
- ("Matrix addition","M3_a = M3_b + M3_c","mCopy(a,mAdd(b,c))"),
- ("Matrix addition","M3_a=M3_b+M3_c","mCopy(a,mAdd(b,c))"),
- ("Scalar subtraction","a = b-c","a=(b-c)"),
- ("Vector subtraction","V3_a = V3_b - V3_c","vCopy(a,vSubtract(b,c))"),
- ("Matrix subtraction","M3_a = M3_b - M3_c","mCopy(a,mSubtract(b,c))"),
- ("Scalar multiplication","a = b*c","a=b*c"),
- ("Scalar division","a = b/c","a=b/c"),
- ("Vector multiplication (dot product)","a = V3_b * V3_c","a=vDot(b,c)"),
- ("Vector multiplication (outer product)","M3_a = V3_b @ V3_c","mCopy(a,vOuterProduct(b,c))"),
- ("Matrix multiplication","M3_a = M3_b * M3_c","mCopy(a,mMultiply(b,c))"),
- ("Vector scaling","V3_a = V3_b * c","vCopy(a,vScale(b,c))"),
- ("Matrix scaling","M3_a = M3_b * c","mCopy(a,mScale(b,c))"),
- ("Matrix by vector multiplication","V3_a = M3_b * V3_c","vCopy(a,mvMultiply(b,c))"),
- ("Scalar exponentiation","a = b^c","a=pow(b,c)"),
- ("Matrix inversion","M3_a = M3_b^-1","mCopy(a,mInverse(b))"),
- ("Matrix transpose","M3_a = M3_b^T","mCopy(a,mTranspose(b))"),
- ("Matrix determinant","a = M3_b^Det","a=mDeterminant(b)"),
- ("Vector magnitude squared","a = V3_b^Mag2","a=vMagnitude2(b)"),
- ("Vector magnitude","a = V3_b^Mag","a=sqrt(vMagnitude2(b))"),
- ("Complicated expression", "myscalar = (M3_amatrix * V3_bvector)^Mag + 5*(-xyz[i] + 2.03^2)","myscalar=(sqrt(vMagnitude2(mvMultiply(amatrix,bvector)))+5*(-xyz[i]+pow(2.03,2)))"),
- ("Complicated Multiline", "myscalar = \n(M3_amatrix * V3_bvector)^Mag +\n 5*(xyz + 2.03^2)","myscalar=(sqrt(vMagnitude2(mvMultiply(amatrix,bvector)))+5*(xyz+pow(2.03,2)))")
-
- ]
-
-
- all_passed = [True]
-
- def post_test(test, parsed):
-
- # copy exprStack to evaluate and clear before running next test
- parsed_stack = exprStack[:]
- exprStack.clear()
-
- name, testcase, expected = next(tc for tc in testcases if tc[1] == test)
-
- this_test_passed = False
- try:
- try:
- result=_evaluateStack(parsed_stack)
- except TypeError:
- print("Unsupported operation on right side of '%s'.\nCheck for missing or incorrect tags on non-scalar operands."%input_string, file=sys.stderr)
+ # show result of parsing the input string
+ if debug_flag:
+ print(input_string, "->", L)
+ print("exprStack=", exprStack)
+
+ # Evaluate the stack of parsed operands, emitting C code.
+ try:
+ result = _evaluateStack(exprStack)
+ except TypeError:
+ print(
+ "Unsupported operation on right side of '%s'.\nCheck for missing or incorrect tags on non-scalar operands."
+ % input_string,
+ file=sys.stderr,
+ )
raise
- except UnaryUnsupportedError:
- print("Unary negation is not supported for vectors and matrices: '%s'"%input_string, file=sys.stderr)
+ except UnaryUnsupportedError:
+ print(
+ "Unary negation is not supported for vectors and matrices: '%s'"
+ % input_string,
+ file=sys.stderr,
+ )
raise
- # Create final assignment and print it.
- if debug_flag: print("var=",targetvar)
- if targetvar != None:
- try:
- result = _assignfunc(targetvar,result)
- except TypeError:
- print("Left side tag does not match right side of '%s'"%input_string, file=sys.stderr)
+ # Create final assignment and print it.
+ if debug_flag:
+ print("var=", targetvar)
+ if targetvar != None:
+ try:
+ result = _assignfunc(targetvar, result)
+ except TypeError:
+ print(
+ "Left side tag does not match right side of '%s'" % input_string,
+ file=sys.stderr,
+ )
raise
- except UnaryUnsupportedError:
- print("Unary negation is not supported for vectors and matrices: '%s'"%input_string, file=sys.stderr)
+ except UnaryUnsupportedError:
+ print(
+ "Unary negation is not supported for vectors and matrices: '%s'"
+ % input_string,
+ file=sys.stderr,
+ )
raise
- else:
- print("Empty left side in '%s'"%input_string, file=sys.stderr)
+ return result
+ else:
+ print("Empty left side in '%s'" % input_string, file=sys.stderr)
raise TypeError
- parsed['result'] = result
- parsed['passed'] = this_test_passed = result == expected
- finally:
- all_passed[0] = all_passed[0] and this_test_passed
- print('\n' + name)
+##-----------------------------------------------------------------------------------
+def fprocess(infilep, outfilep):
+ """
+ Scans an input file for LA equations between double square brackets,
+ e.g. [[ M3_mymatrix = M3_anothermatrix^-1 ]], and replaces the expression
+ with a comment containing the equation followed by nested function calls
+ that implement the equation as C code. A trailing semi-colon is appended.
+ The equation within [[ ]] should NOT end with a semicolon as that will raise
+ a ParseException. However, it is ok to have a semicolon after the right brackets.
+
+ Other text in the file is unaltered.
+
+ The arguments are file objects (NOT file names) opened for reading and
+ writing, respectively.
+ """
+ pattern = r"\[\[\s*(.*?)\s*\]\]"
+ eqn = re.compile(pattern, re.DOTALL)
+ s = infilep.read()
- equation.runTests((t[1] for t in testcases), postParse=post_test)
+ def parser(mo):
+ ccode = parse(mo.group(1))
+ return "/* %s */\n%s;\nLAParserBufferReset();\n" % (mo.group(1), ccode)
+ content = eqn.sub(parser, s)
+ outfilep.write(content)
- ##TODO: Write testcases with invalid expressions and test that the expected
- ## exceptions are raised.
- print("Tests completed!")
- print("PASSED" if all_passed[0] else "FAILED")
- assert all_passed[0]
+##-----------------------------------------------------------------------------------
+def test():
+ """
+ Tests the parsing of various supported expressions. Raises
+ an AssertError if the output is not what is expected. Prints the
+ input, expected output, and actual output for all tests.
+ """
+ print("Testing LAParser")
+ testcases = [
+ ("Scalar addition", "a = b+c", "a=(b+c)"),
+ ("Vector addition", "V3_a = V3_b + V3_c", "vCopy(a,vAdd(b,c))"),
+ ("Vector addition", "V3_a=V3_b+V3_c", "vCopy(a,vAdd(b,c))"),
+ ("Matrix addition", "M3_a = M3_b + M3_c", "mCopy(a,mAdd(b,c))"),
+ ("Matrix addition", "M3_a=M3_b+M3_c", "mCopy(a,mAdd(b,c))"),
+ ("Scalar subtraction", "a = b-c", "a=(b-c)"),
+ ("Vector subtraction", "V3_a = V3_b - V3_c", "vCopy(a,vSubtract(b,c))"),
+ ("Matrix subtraction", "M3_a = M3_b - M3_c", "mCopy(a,mSubtract(b,c))"),
+ ("Scalar multiplication", "a = b*c", "a=b*c"),
+ ("Scalar division", "a = b/c", "a=b/c"),
+ ("Vector multiplication (dot product)", "a = V3_b * V3_c", "a=vDot(b,c)"),
+ (
+ "Vector multiplication (outer product)",
+ "M3_a = V3_b @ V3_c",
+ "mCopy(a,vOuterProduct(b,c))",
+ ),
+ ("Matrix multiplication", "M3_a = M3_b * M3_c", "mCopy(a,mMultiply(b,c))"),
+ ("Vector scaling", "V3_a = V3_b * c", "vCopy(a,vScale(b,c))"),
+ ("Matrix scaling", "M3_a = M3_b * c", "mCopy(a,mScale(b,c))"),
+ (
+ "Matrix by vector multiplication",
+ "V3_a = M3_b * V3_c",
+ "vCopy(a,mvMultiply(b,c))",
+ ),
+ ("Scalar exponentiation", "a = b^c", "a=pow(b,c)"),
+ ("Matrix inversion", "M3_a = M3_b^-1", "mCopy(a,mInverse(b))"),
+ ("Matrix transpose", "M3_a = M3_b^T", "mCopy(a,mTranspose(b))"),
+ ("Matrix determinant", "a = M3_b^Det", "a=mDeterminant(b)"),
+ ("Vector magnitude squared", "a = V3_b^Mag2", "a=vMagnitude2(b)"),
+ ("Vector magnitude", "a = V3_b^Mag", "a=sqrt(vMagnitude2(b))"),
+ (
+ "Complicated expression",
+ "myscalar = (M3_amatrix * V3_bvector)^Mag + 5*(-xyz[i] + 2.03^2)",
+ "myscalar=(sqrt(vMagnitude2(mvMultiply(amatrix,bvector)))+5*(-xyz[i]+pow(2.03,2)))",
+ ),
+ (
+ "Complicated Multiline",
+ "myscalar = \n(M3_amatrix * V3_bvector)^Mag +\n 5*(xyz + 2.03^2)",
+ "myscalar=(sqrt(vMagnitude2(mvMultiply(amatrix,bvector)))+5*(xyz+pow(2.03,2)))",
+ ),
+ ]
+
+ all_passed = [True]
+
+ def post_test(test, parsed):
+
+ # copy exprStack to evaluate and clear before running next test
+ parsed_stack = exprStack[:]
+ exprStack.clear()
+
+ name, testcase, expected = next(tc for tc in testcases if tc[1] == test)
+
+ this_test_passed = False
+ try:
+ try:
+ result = _evaluateStack(parsed_stack)
+ except TypeError:
+ print(
+ "Unsupported operation on right side of '%s'.\nCheck for missing or incorrect tags on non-scalar operands."
+ % input_string,
+ file=sys.stderr,
+ )
+ raise
+ except UnaryUnsupportedError:
+ print(
+ "Unary negation is not supported for vectors and matrices: '%s'"
+ % input_string,
+ file=sys.stderr,
+ )
+ raise
+
+ # Create final assignment and print it.
+ if debug_flag:
+ print("var=", targetvar)
+ if targetvar != None:
+ try:
+ result = _assignfunc(targetvar, result)
+ except TypeError:
+ print(
+ "Left side tag does not match right side of '%s'"
+ % input_string,
+ file=sys.stderr,
+ )
+ raise
+ except UnaryUnsupportedError:
+ print(
+ "Unary negation is not supported for vectors and matrices: '%s'"
+ % input_string,
+ file=sys.stderr,
+ )
+ raise
+
+ else:
+ print("Empty left side in '%s'" % input_string, file=sys.stderr)
+ raise TypeError
+
+ parsed["result"] = result
+ parsed["passed"] = this_test_passed = result == expected
+
+ finally:
+ all_passed[0] = all_passed[0] and this_test_passed
+ print("\n" + name)
+
+ equation.runTests((t[1] for t in testcases), postParse=post_test)
+
+ ##TODO: Write testcases with invalid expressions and test that the expected
+ ## exceptions are raised.
+
+ print("Tests completed!")
+ print("PASSED" if all_passed[0] else "FAILED")
+ assert all_passed[0]
+
##----------------------------------------------------------------------------
## The following is executed only when this module is executed as
## and then enters an interactive loop where you
## can enter expressions and see the resulting C code as output.
-if __name__ == '__main__':
+if __name__ == "__main__":
- import sys
- if not sys.flags.interactive:
- # run testcases
- test()
- sys.exit(0)
+ import sys
- # input_string
- input_string=''
+ if not sys.flags.interactive:
+ # run testcases
+ test()
+ sys.exit(0)
- # Display instructions on how to use the program interactively
- interactiveusage = """
+ # input_string
+ input_string = ""
+
+ # Display instructions on how to use the program interactively
+ interactiveusage = """
Entering interactive mode:
Type in an equation to be parsed or 'quit' to exit the program.
Type 'debug on' to print parsing details as each string is processed.
Type 'debug off' to stop printing parsing details
"""
- print(interactiveusage)
- input_string = input("> ")
-
- while input_string != 'quit':
- if input_string == "debug on":
- debug_flag = True
- elif input_string == "debug off":
- debug_flag = False
- else:
- try:
- print(parse(input_string))
- except Exception:
- pass
-
- # obtain new input string
+ print(interactiveusage)
input_string = input("> ")
- # if user types 'quit' then say goodbye
- print("Good bye!")
- import os
- os._exit(0)
-
+ while input_string != "quit":
+ if input_string == "debug on":
+ debug_flag = True
+ elif input_string == "debug off":
+ debug_flag = False
+ else:
+ try:
+ print(parse(input_string))
+ except Exception:
+ pass
+
+ # obtain new input string
+ input_string = input("> ")
+
+ # if user types 'quit' then say goodbye
+ print("Good bye!")
+ import os
+
+ os._exit(0)
-# SimpleCalc.py\r
-#\r
-# Demonstration of the parsing module,\r
-# Sample usage\r
-#\r
-# $ python SimpleCalc.py\r
-# Type in the string to be parse or 'quit' to exit the program\r
-# > g=67.89 + 7/5\r
-# 69.29\r
-# > g\r
-# 69.29\r
-# > h=(6*g+8.8)-g\r
-# 355.25\r
-# > h + 1\r
-# 356.25\r
-# > 87.89 + 7/5\r
-# 89.29\r
-# > ans+10\r
-# 99.29\r
-# > quit\r
-# Good bye!\r
-#\r
-#\r
-\r
-\r
-\r
-# Uncomment the line below for readline support on interactive terminal\r
-# import readline\r
-from pyparsing import ParseException, Word, alphas, alphanums\r
-import math\r
-\r
-# Debugging flag can be set to either "debug_flag=True" or "debug_flag=False"\r
-debug_flag=False\r
-\r
-variables = {}\r
-\r
-from fourFn import BNF, exprStack, fn, opn\r
-def evaluateStack( s ):\r
- op = s.pop()\r
- if op == 'unary -':\r
- return -evaluateStack( s )\r
- if op in "+-*/^":\r
- op2 = evaluateStack( s )\r
- op1 = evaluateStack( s )\r
- return opn[op]( op1, op2 )\r
- elif op == "PI":\r
- return math.pi # 3.1415926535\r
- elif op == "E":\r
- return math.e # 2.718281828\r
- elif op in fn:\r
- return fn[op]( evaluateStack( s ) )\r
- elif op[0].isalpha():\r
- if op in variables:\r
- return variables[op]\r
- raise Exception("invalid identifier '%s'" % op)\r
- else:\r
- return float( op )\r
-\r
-arithExpr = BNF()\r
-ident = Word(alphas, alphanums).setName("identifier")\r
-assignment = ident("varname") + '=' + arithExpr\r
-pattern = assignment | arithExpr\r
-\r
-if __name__ == '__main__':\r
- # input_string\r
- input_string=''\r
-\r
- # Display instructions on how to quit the program\r
- print("Type in the string to be parsed or 'quit' to exit the program")\r
- input_string = input("> ")\r
-\r
- while input_string.strip().lower() != 'quit':\r
- if input_string.strip().lower() == 'debug':\r
- debug_flag=True\r
- input_string = input("> ")\r
- continue\r
-\r
- # Reset to an empty exprStack\r
- del exprStack[:]\r
-\r
- if input_string != '':\r
- # try parsing the input string\r
- try:\r
- L=pattern.parseString(input_string, parseAll=True)\r
- except ParseException as err:\r
- L=['Parse Failure', input_string, (str(err), err.line, err.column)]\r
-\r
- # show result of parsing the input string\r
- if debug_flag: print(input_string, "->", L)\r
- if len(L)==0 or L[0] != 'Parse Failure':\r
- if debug_flag: print("exprStack=", exprStack)\r
-\r
- # calculate result , store a copy in ans , display the result to user\r
- try:\r
- result=evaluateStack(exprStack)\r
- except Exception as e:\r
- print(str(e))\r
- else:\r
- variables['ans']=result\r
- print(result)\r
-\r
- # Assign result to a variable if required\r
- if L.varname:\r
- variables[L.varname] = result\r
- if debug_flag: print("variables=", variables)\r
- else:\r
- print('Parse Failure')\r
- err_str, err_line, err_col = L[-1]\r
- print(err_line)\r
- print(" "*(err_col-1) + "^")\r
- print(err_str)\r
-\r
- # obtain new input string\r
- input_string = input("> ")\r
-\r
- # if user type 'quit' then say goodbye\r
- print("Good bye!")\r
+# SimpleCalc.py
+#
+# Demonstration of the parsing module,
+# Sample usage
+#
+# $ python SimpleCalc.py
+# Type in the string to be parse or 'quit' to exit the program
+# > g=67.89 + 7/5
+# 69.29
+# > g
+# 69.29
+# > h=(6*g+8.8)-g
+# 355.25
+# > h + 1
+# 356.25
+# > 87.89 + 7/5
+# 89.29
+# > ans+10
+# 99.29
+# > quit
+# Good bye!
+#
+#
+
+
+# Uncomment the line below for readline support on interactive terminal
+# import readline
+from pyparsing import ParseException, Word, alphas, alphanums
+
+# Debugging flag can be set to either "debug_flag=True" or "debug_flag=False"
+debug_flag = False
+
+variables = {}
+
+from fourFn import BNF, exprStack, evaluate_stack
+
+# from fourFn import BNF, exprStack, fn, opn
+# def evaluateStack( s ):
+# op = s.pop()
+# if op == 'unary -':
+# return -evaluateStack( s )
+# if op in "+-*/^":
+# op2 = evaluateStack( s )
+# op1 = evaluateStack( s )
+# return opn[op]( op1, op2 )
+# elif op == "PI":
+# return math.pi # 3.1415926535
+# elif op == "E":
+# return math.e # 2.718281828
+# elif op in fn:
+# return fn[op]( evaluateStack( s ) )
+# elif op[0].isalpha():
+# if op in variables:
+# return variables[op]
+# raise Exception("invalid identifier '%s'" % op)
+# else:
+# return float( op )
+
+arithExpr = BNF()
+ident = Word(alphas, alphanums).setName("identifier")
+assignment = ident("varname") + "=" + arithExpr
+pattern = assignment | arithExpr
+
+if __name__ == "__main__":
+ # input_string
+ input_string = ""
+
+ # Display instructions on how to quit the program
+ print("Type in the string to be parsed or 'quit' to exit the program")
+ input_string = input("> ")
+
+ while input_string.strip().lower() != "quit":
+ if input_string.strip().lower() == "debug":
+ debug_flag = True
+ input_string = input("> ")
+ continue
+
+ # Reset to an empty exprStack
+ del exprStack[:]
+
+ if input_string != "":
+ # try parsing the input string
+ try:
+ L = pattern.parseString(input_string, parseAll=True)
+ except ParseException as err:
+ L = ["Parse Failure", input_string, (str(err), err.line, err.column)]
+
+ # show result of parsing the input string
+ if debug_flag:
+ print(input_string, "->", L)
+ if len(L) == 0 or L[0] != "Parse Failure":
+ if debug_flag:
+ print("exprStack=", exprStack)
+
+ for i, ob in enumerate(exprStack):
+ if isinstance(ob, str) and ob in variables:
+ exprStack[i] = str(variables[ob])
+
+ # calculate result , store a copy in ans , display the result to user
+ try:
+ result = evaluate_stack(exprStack)
+ except Exception as e:
+ print(str(e))
+ else:
+ variables["ans"] = result
+ print(result)
+
+ # Assign result to a variable if required
+ if L.varname:
+ variables[L.varname] = result
+ if debug_flag:
+ print("variables=", variables)
+ else:
+ print("Parse Failure")
+ err_str, err_line, err_col = L[-1]
+ print(err_line)
+ print(" " * (err_col - 1) + "^")
+ print(err_str)
+
+ # obtain new input string
+ input_string = input("> ")
+
+ # if user type 'quit' then say goodbye
+ print("Good bye!")
# Copyright 2008, by Paul McGuire\r
#\r
\r
-from pyparsing import ParserElement,LineEnd,Optional,Word,nums,Regex,\\r
- Literal,CaselessLiteral,Group,OneOrMore,Suppress,restOfLine,\\r
- FollowedBy,empty\r
-\r
-__all__ = ['tapOutputParser', 'TAPTest', 'TAPSummary']\r
+from pyparsing import (\r
+ ParserElement,\r
+ LineEnd,\r
+ Optional,\r
+ Word,\r
+ nums,\r
+ Regex,\r
+ Literal,\r
+ CaselessLiteral,\r
+ Group,\r
+ OneOrMore,\r
+ Suppress,\r
+ restOfLine,\r
+ FollowedBy,\r
+ empty,\r
+)\r
+\r
+__all__ = ["tapOutputParser", "TAPTest", "TAPSummary"]\r
\r
# newlines are significant whitespace, so set default skippable\r
# whitespace to just spaces and tabs\r
NL = LineEnd().suppress()\r
\r
integer = Word(nums)\r
-plan = '1..' + integer("ubound")\r
+plan = "1.." + integer("ubound")\r
\r
-OK,NOT_OK = map(Literal,['ok','not ok'])\r
-testStatus = (OK | NOT_OK)\r
+OK, NOT_OK = map(Literal, ["ok", "not ok"])\r
+testStatus = OK | NOT_OK\r
\r
description = Regex("[^#\n]+")\r
-description.setParseAction(lambda t:t[0].lstrip('- '))\r
-\r
-TODO,SKIP = map(CaselessLiteral,'TODO SKIP'.split())\r
-directive = Group(Suppress('#') + (TODO + restOfLine |\r
- FollowedBy(SKIP) +\r
- restOfLine.copy().setParseAction(lambda t:['SKIP',t[0]]) ))\r
+description.setParseAction(lambda t: t[0].lstrip("- "))\r
+\r
+TODO, SKIP = map(CaselessLiteral, "TODO SKIP".split())\r
+directive = Group(\r
+ Suppress("#")\r
+ + (\r
+ TODO + restOfLine\r
+ | FollowedBy(SKIP) + restOfLine.copy().setParseAction(lambda t: ["SKIP", t[0]])\r
+ )\r
+)\r
\r
commentLine = Suppress("#") + empty + restOfLine\r
\r
testLine = Group(\r
- Optional(OneOrMore(commentLine + NL))("comments") +\r
- testStatus("passed") +\r
- Optional(integer)("testNumber") +\r
- Optional(description)("description") +\r
- Optional(directive)("directive")\r
- )\r
-bailLine = Group(Literal("Bail out!")("BAIL") +\r
- empty + Optional(restOfLine)("reason"))\r
+ Optional(OneOrMore(commentLine + NL))("comments")\r
+ + testStatus("passed")\r
+ + Optional(integer)("testNumber")\r
+ + Optional(description)("description")\r
+ + Optional(directive)("directive")\r
+)\r
+bailLine = Group(Literal("Bail out!")("BAIL") + empty + Optional(restOfLine)("reason"))\r
+\r
+tapOutputParser = Optional(Group(plan)("plan") + NL) & Group(\r
+ OneOrMore((testLine | bailLine) + NL)\r
+)("tests")\r
\r
-tapOutputParser = Optional(Group(plan)("plan") + NL) & \\r
- Group(OneOrMore((testLine|bailLine) + NL))("tests")\r
\r
-class TAPTest(object):\r
- def __init__(self,results):\r
+class TAPTest:\r
+ def __init__(self, results):\r
self.num = results.testNumber\r
- self.passed = (results.passed=="ok")\r
+ self.passed = results.passed == "ok"\r
self.skipped = self.todo = False\r
if results.directive:\r
- self.skipped = (results.directive[0][0]=='SKIP')\r
- self.todo = (results.directive[0][0]=='TODO')\r
+ self.skipped = results.directive[0][0] == "SKIP"\r
+ self.todo = results.directive[0][0] == "TODO"\r
+\r
@classmethod\r
- def bailedTest(cls,num):\r
+ def bailedTest(cls, num):\r
ret = TAPTest(empty.parseString(""))\r
ret.num = num\r
ret.skipped = True\r
return ret\r
\r
-class TAPSummary(object):\r
- def __init__(self,results):\r
+\r
+class TAPSummary:\r
+ def __init__(self, results):\r
self.passedTests = []\r
self.failedTests = []\r
self.skippedTests = []\r
self.bonusTests = []\r
self.bail = False\r
if results.plan:\r
- expected = list(range(1, int(results.plan.ubound)+1))\r
+ expected = list(range(1, int(results.plan.ubound) + 1))\r
else:\r
- expected = list(range(1,len(results.tests)+1))\r
+ expected = list(range(1, len(results.tests) + 1))\r
\r
- for i,res in enumerate(results.tests):\r
+ for i, res in enumerate(results.tests):\r
# test for bail out\r
if res.BAIL:\r
- #~ print "Test suite aborted: " + res.reason\r
- #~ self.failedTests += expected[i:]\r
+ # ~ print "Test suite aborted: " + res.reason\r
+ # ~ self.failedTests += expected[i:]\r
self.bail = True\r
- self.skippedTests += [ TAPTest.bailedTest(ii) for ii in expected[i:] ]\r
+ self.skippedTests += [TAPTest.bailedTest(ii) for ii in expected[i:]]\r
self.bailReason = res.reason\r
break\r
\r
- #~ print res.dump()\r
- testnum = i+1\r
+ # ~ print res.dump()\r
+ testnum = i + 1\r
if res.testNumber != "":\r
if testnum != int(res.testNumber):\r
print("ERROR! test %(testNumber)s out of sequence" % res)\r
self.passedTests.append(test)\r
else:\r
self.failedTests.append(test)\r
- if test.skipped: self.skippedTests.append(test)\r
- if test.todo: self.todoTests.append(test)\r
- if test.todo and test.passed: self.bonusTests.append(test)\r
+ if test.skipped:\r
+ self.skippedTests.append(test)\r
+ if test.todo:\r
+ self.todoTests.append(test)\r
+ if test.todo and test.passed:\r
+ self.bonusTests.append(test)\r
\r
- self.passedSuite = not self.bail and (set(self.failedTests)-set(self.todoTests) == set())\r
+ self.passedSuite = not self.bail and (\r
+ set(self.failedTests) - set(self.todoTests) == set()\r
+ )\r
\r
def summary(self, showPassed=False, showAll=False):\r
- testListStr = lambda tl : "[" + ",".join(str(t.num) for t in tl) + "]"\r
+ testListStr = lambda tl: "[" + ",".join(str(t.num) for t in tl) + "]"\r
summaryText = []\r
if showPassed or showAll:\r
- summaryText.append( "PASSED: %s" % testListStr(self.passedTests) )\r
+ summaryText.append("PASSED: %s" % testListStr(self.passedTests))\r
if self.failedTests or showAll:\r
- summaryText.append( "FAILED: %s" % testListStr(self.failedTests) )\r
+ summaryText.append("FAILED: %s" % testListStr(self.failedTests))\r
if self.skippedTests or showAll:\r
- summaryText.append( "SKIPPED: %s" % testListStr(self.skippedTests) )\r
+ summaryText.append("SKIPPED: %s" % testListStr(self.skippedTests))\r
if self.todoTests or showAll:\r
- summaryText.append( "TODO: %s" % testListStr(self.todoTests) )\r
+ summaryText.append("TODO: %s" % testListStr(self.todoTests))\r
if self.bonusTests or showAll:\r
- summaryText.append( "BONUS: %s" % testListStr(self.bonusTests) )\r
+ summaryText.append("BONUS: %s" % testListStr(self.bonusTests))\r
if self.passedSuite:\r
- summaryText.append( "PASSED" )\r
+ summaryText.append("PASSED")\r
else:\r
- summaryText.append( "FAILED" )\r
+ summaryText.append("FAILED")\r
return "\n".join(summaryText)\r
\r
+\r
# create TAPSummary objects from tapOutput parsed results, by setting\r
# class as parse action\r
tapOutputParser.setParseAction(TAPSummary)\r
\r
\r
-if __name__ == "__main__":\r
+def main():\r
test1 = """\\r
1..4\r
ok 1 - Input file opened\r
1..7\r
"""\r
\r
- for test in (test1,test2,test3,test4,test5,test6):\r
+ for test in (test1, test2, test3, test4, test5, test6):\r
print(test)\r
tapResult = tapOutputParser.parseString(test)[0]\r
print(tapResult.summary(showAll=True))\r
print()\r
+\r
+\r
+if __name__ == "__main__":\r
+ main()\r
import random\r
import string\r
\r
-def aOrAn( item ):\r
+\r
+def aOrAn(item):\r
if item.desc[0] in "aeiou":\r
return "an " + item.desc\r
else:\r
return "a " + item.desc\r
\r
+\r
def enumerateItems(l):\r
- if len(l) == 0: return "nothing"\r
+ if len(l) == 0:\r
+ return "nothing"\r
out = []\r
if len(l) > 1:\r
- out.append(', '.join(aOrAn(item) for item in l[:-1]))\r
- out.append('and')\r
+ out.append(", ".join(aOrAn(item) for item in l[:-1]))\r
+ out.append("and")\r
out.append(aOrAn(l[-1]))\r
return " ".join(out)\r
\r
+\r
def enumerateDoors(l):\r
- if len(l) == 0: return ""\r
+ if len(l) == 0:\r
+ return ""\r
out = []\r
if len(l) > 1:\r
- out.append(', '.join(l[:-1]))\r
+ out.append(", ".join(l[:-1]))\r
out.append("and")\r
out.append(l[-1])\r
return " ".join(out)\r
\r
-class Room(object):\r
+\r
+class Room:\r
def __init__(self, desc):\r
self.desc = desc\r
self.inv = []\r
self.gameOver = False\r
- self.doors = [None,None,None,None]\r
+ self.doors = [None, None, None, None]\r
\r
- def __getattr__(self,attr):\r
- return \\r
- {\r
- "n":self.doors[0],\r
- "s":self.doors[1],\r
- "e":self.doors[2],\r
- "w":self.doors[3],\r
- }[attr]\r
+ def __getattr__(self, attr):\r
+ return {\r
+ "n": self.doors[0],\r
+ "s": self.doors[1],\r
+ "e": self.doors[2],\r
+ "w": self.doors[3],\r
+ }[attr]\r
\r
- def enter(self,player):\r
+ def enter(self, player):\r
if self.gameOver:\r
player.gameOver = True\r
\r
def addItem(self, it):\r
self.inv.append(it)\r
\r
- def removeItem(self,it):\r
+ def removeItem(self, it):\r
self.inv.remove(it)\r
\r
def describe(self):\r
print(self.desc)\r
- visibleItems = [ it for it in self.inv if it.isVisible ]\r
+ visibleItems = [it for it in self.inv if it.isVisible]\r
if random.random() > 0.5:\r
if len(visibleItems) > 1:\r
is_form = "are"\r
else:\r
is_form = "is"\r
- print("There {0} {1} here.".format(is_form, enumerateItems(visibleItems)))\r
+ print("There {} {} here.".format(is_form, enumerateItems(visibleItems)))\r
else:\r
print("You see %s." % (enumerateItems(visibleItems)))\r
\r
\r
class Exit(Room):\r
def __init__(self):\r
- super(Exit,self).__init__("")\r
+ super().__init__("")\r
\r
- def enter(self,player):\r
+ def enter(self, player):\r
player.gameOver = True\r
\r
\r
-\r
-class Item(object):\r
+class Item:\r
items = {}\r
+\r
def __init__(self, desc):\r
self.desc = desc\r
self.isDeadly = False\r
\r
def isUsable(self, player, target):\r
if self.usableConditionTest:\r
- return self.usableConditionTest( player, target )\r
+ return self.usableConditionTest(player, target)\r
else:\r
return False\r
\r
if self.useAction:\r
self.useAction(player, self, target)\r
\r
+\r
class OpenableItem(Item):\r
def __init__(self, desc, contents=None):\r
- super(OpenableItem,self).__init__(desc)\r
+ super().__init__(desc)\r
self.isOpenable = True\r
self.isOpened = False\r
if contents is not None:\r
if isinstance(contents, Item):\r
- self.contents = [contents,]\r
+ self.contents = [\r
+ contents,\r
+ ]\r
else:\r
self.contents = contents\r
else:\r
self.isOpened = not self.isOpened\r
if self.contents is not None:\r
for item in self.contents:\r
- player.room.addItem( item )\r
+ player.room.addItem(item)\r
self.contents = []\r
self.desc = "open " + self.desc\r
\r
self.desc = self.desc[5:]\r
\r
\r
-class Command(object):\r
+class Command:\r
"Base class for commands"\r
+\r
def __init__(self, verb, verbProg):\r
self.verb = verb\r
self.verbProg = verbProg\r
def _doCommand(self, player):\r
pass\r
\r
- def __call__(self, player ):\r
- print(self.verbProg.capitalize()+"...")\r
+ def __call__(self, player):\r
+ print(self.verbProg.capitalize() + "...")\r
self._doCommand(player)\r
\r
\r
class MoveCommand(Command):\r
def __init__(self, quals):\r
- super(MoveCommand,self).__init__("MOVE", "moving")\r
+ super().__init__("MOVE", "moving")\r
self.direction = quals.direction[0]\r
\r
@staticmethod\r
rm = player.room\r
nextRoom = rm.doors[\r
{\r
- "N":0,\r
- "S":1,\r
- "E":2,\r
- "W":3,\r
+ "N": 0,\r
+ "S": 1,\r
+ "E": 2,\r
+ "W": 3,\r
}[self.direction]\r
- ]\r
+ ]\r
if nextRoom:\r
- player.moveTo( nextRoom )\r
+ player.moveTo(nextRoom)\r
else:\r
print("Can't go that way.")\r
\r
\r
class TakeCommand(Command):\r
def __init__(self, quals):\r
- super(TakeCommand,self).__init__("TAKE", "taking")\r
+ super().__init__("TAKE", "taking")\r
self.subject = quals.item\r
\r
@staticmethod\r
\r
class DropCommand(Command):\r
def __init__(self, quals):\r
- super(DropCommand,self).__init__("DROP", "dropping")\r
+ super().__init__("DROP", "dropping")\r
self.subject = quals.item\r
\r
@staticmethod\r
else:\r
print("You don't have %s." % (aOrAn(subj)))\r
\r
+\r
class InventoryCommand(Command):\r
def __init__(self, quals):\r
- super(InventoryCommand,self).__init__("INV", "taking inventory")\r
+ super().__init__("INV", "taking inventory")\r
\r
@staticmethod\r
def helpDescription():\r
return "INVENTORY or INV or I - lists what items you have"\r
\r
def _doCommand(self, player):\r
- print("You have %s." % enumerateItems( player.inv ))\r
+ print("You have %s." % enumerateItems(player.inv))\r
+\r
\r
class LookCommand(Command):\r
def __init__(self, quals):\r
- super(LookCommand,self).__init__("LOOK", "looking")\r
+ super().__init__("LOOK", "looking")\r
\r
@staticmethod\r
def helpDescription():\r
def _doCommand(self, player):\r
player.room.describe()\r
\r
+\r
class DoorsCommand(Command):\r
def __init__(self, quals):\r
- super(DoorsCommand,self).__init__("DOORS", "looking for doors")\r
+ super().__init__("DOORS", "looking for doors")\r
\r
@staticmethod\r
def helpDescription():\r
reply = "There is a door to the "\r
else:\r
reply = "There are doors to the "\r
- doorNames = [ {0:"north", 1:"south", 2:"east", 3:"west"}[i]\r
- for i,d in enumerate(rm.doors) if d is not None ]\r
- #~ print doorNames\r
- reply += enumerateDoors( doorNames )\r
+ doorNames = [\r
+ {0: "north", 1: "south", 2: "east", 3: "west"}[i]\r
+ for i, d in enumerate(rm.doors)\r
+ if d is not None\r
+ ]\r
+ # ~ print doorNames\r
+ reply += enumerateDoors(doorNames)\r
reply += "."\r
print(reply)\r
\r
+\r
class UseCommand(Command):\r
def __init__(self, quals):\r
- super(UseCommand,self).__init__("USE", "using")\r
+ super().__init__("USE", "using")\r
self.subject = Item.items[quals.usedObj]\r
if quals.targetObj:\r
self.target = Item.items[quals.targetObj]\r
rm = player.room\r
availItems = rm.inv + player.inv\r
if self.subject in availItems:\r
- if self.subject.isUsable( player, self.target ):\r
- self.subject.useItem( player, self.target )\r
+ if self.subject.isUsable(player, self.target):\r
+ self.subject.useItem(player, self.target)\r
else:\r
print("You can't use that here.")\r
else:\r
print("There is no %s here to use." % self.subject)\r
\r
+\r
class OpenCommand(Command):\r
def __init__(self, quals):\r
- super(OpenCommand,self).__init__("OPEN", "opening")\r
+ super().__init__("OPEN", "opening")\r
self.subject = Item.items[quals.item]\r
\r
@staticmethod\r
\r
def _doCommand(self, player):\r
rm = player.room\r
- availItems = rm.inv+player.inv\r
+ availItems = rm.inv + player.inv\r
if self.subject in availItems:\r
if self.subject.isOpenable:\r
if not self.subject.isOpened:\r
- self.subject.openItem( player )\r
+ self.subject.openItem(player)\r
else:\r
print("It's already open.")\r
else:\r
else:\r
print("There is no %s here to open." % self.subject)\r
\r
+\r
class CloseCommand(Command):\r
def __init__(self, quals):\r
- super(CloseCommand,self).__init__("CLOSE", "closing")\r
+ super().__init__("CLOSE", "closing")\r
self.subject = Item.items[quals.item]\r
\r
@staticmethod\r
\r
def _doCommand(self, player):\r
rm = player.room\r
- availItems = rm.inv+player.inv\r
+ availItems = rm.inv + player.inv\r
if self.subject in availItems:\r
if self.subject.isOpenable:\r
if self.subject.isOpened:\r
- self.subject.closeItem( player )\r
+ self.subject.closeItem(player)\r
else:\r
print("You can't close that, it's not open.")\r
else:\r
else:\r
print("There is no %s here to close." % self.subject)\r
\r
+\r
class QuitCommand(Command):\r
def __init__(self, quals):\r
- super(QuitCommand,self).__init__("QUIT", "quitting")\r
+ super().__init__("QUIT", "quitting")\r
\r
@staticmethod\r
def helpDescription():\r
print("Ok....")\r
player.gameOver = True\r
\r
+\r
class HelpCommand(Command):\r
def __init__(self, quals):\r
- super(HelpCommand,self).__init__("HELP", "helping")\r
+ super().__init__("HELP", "helping")\r
\r
@staticmethod\r
def helpDescription():\r
DoorsCommand,\r
QuitCommand,\r
HelpCommand,\r
- ]:\r
+ ]:\r
print(" - %s" % cmd.helpDescription())\r
print()\r
\r
+\r
class AppParseException(ParseException):\r
pass\r
\r
-class Parser(object):\r
+\r
+class Parser:\r
def __init__(self):\r
self.bnf = self.makeBNF()\r
\r
def makeBNF(self):\r
invVerb = oneOf("INV INVENTORY I", caseless=True)\r
dropVerb = oneOf("DROP LEAVE", caseless=True)\r
- takeVerb = oneOf("TAKE PICKUP", caseless=True) | \\r
- (CaselessLiteral("PICK") + CaselessLiteral("UP") )\r
+ takeVerb = oneOf("TAKE PICKUP", caseless=True) | (\r
+ CaselessLiteral("PICK") + CaselessLiteral("UP")\r
+ )\r
moveVerb = oneOf("MOVE GO", caseless=True) | empty\r
useVerb = oneOf("USE U", caseless=True)\r
openVerb = oneOf("OPEN O", caseless=True)\r
quitVerb = oneOf("QUIT Q", caseless=True)\r
lookVerb = oneOf("LOOK L", caseless=True)\r
doorsVerb = CaselessLiteral("DOORS")\r
- helpVerb = oneOf("H HELP ?",caseless=True)\r
+ helpVerb = oneOf("H HELP ?", caseless=True)\r
\r
- itemRef = OneOrMore(Word(alphas)).setParseAction( self.validateItemName )\r
- nDir = oneOf("N NORTH",caseless=True).setParseAction(replaceWith("N"))\r
- sDir = oneOf("S SOUTH",caseless=True).setParseAction(replaceWith("S"))\r
- eDir = oneOf("E EAST",caseless=True).setParseAction(replaceWith("E"))\r
- wDir = oneOf("W WEST",caseless=True).setParseAction(replaceWith("W"))\r
+ itemRef = OneOrMore(Word(alphas)).setParseAction(self.validateItemName)\r
+ nDir = oneOf("N NORTH", caseless=True).setParseAction(replaceWith("N"))\r
+ sDir = oneOf("S SOUTH", caseless=True).setParseAction(replaceWith("S"))\r
+ eDir = oneOf("E EAST", caseless=True).setParseAction(replaceWith("E"))\r
+ wDir = oneOf("W WEST", caseless=True).setParseAction(replaceWith("W"))\r
moveDirection = nDir | sDir | eDir | wDir\r
\r
invCommand = invVerb\r
dropCommand = dropVerb + itemRef("item")\r
takeCommand = takeVerb + itemRef("item")\r
- useCommand = useVerb + itemRef("usedObj") + \\r
- Optional(oneOf("IN ON",caseless=True)) + \\r
- Optional(itemRef,default=None)("targetObj")\r
+ useCommand = (\r
+ useVerb\r
+ + itemRef("usedObj")\r
+ + Optional(oneOf("IN ON", caseless=True))\r
+ + Optional(itemRef, default=None)("targetObj")\r
+ )\r
openCommand = openVerb + itemRef("item")\r
closeCommand = closeVerb + itemRef("item")\r
moveCommand = moveVerb + moveDirection("direction")\r
helpCommand.setParseAction(HelpCommand)\r
\r
# define parser using all command expressions\r
- return ( invCommand |\r
- useCommand |\r
- openCommand |\r
- closeCommand |\r
- dropCommand |\r
- takeCommand |\r
- moveCommand |\r
- lookCommand |\r
- doorsCommand |\r
- helpCommand |\r
- quitCommand )("command") + LineEnd()\r
-\r
- def validateItemName(self,s,l,t):\r
+ return (\r
+ invCommand\r
+ | useCommand\r
+ | openCommand\r
+ | closeCommand\r
+ | dropCommand\r
+ | takeCommand\r
+ | moveCommand\r
+ | lookCommand\r
+ | doorsCommand\r
+ | helpCommand\r
+ | quitCommand\r
+ )("command") + LineEnd()\r
+\r
+ def validateItemName(self, s, l, t):\r
iname = " ".join(t)\r
if iname not in Item.items:\r
- raise AppParseException(s,l,"No such item '%s'." % iname)\r
+ raise AppParseException(s, l, "No such item '%s'." % iname)\r
return iname\r
\r
def parseCmd(self, cmdstr):\r
except AppParseException as pe:\r
print(pe.msg)\r
except ParseException as pe:\r
- print(random.choice([ "Sorry, I don't understand that.",\r
- "Huh?",\r
- "Excuse me?",\r
- "???",\r
- "What?" ] ))\r
-\r
-class Player(object):\r
+ print(\r
+ random.choice(\r
+ [\r
+ "Sorry, I don't understand that.",\r
+ "Huh?",\r
+ "Excuse me?",\r
+ "???",\r
+ "What?",\r
+ ]\r
+ )\r
+ )\r
+\r
+\r
+class Player:\r
def __init__(self, name):\r
self.name = name\r
self.gameOver = False\r
else:\r
rm.describe()\r
\r
- def take(self,it):\r
+ def take(self, it):\r
if it.isDeadly:\r
print("Aaaagh!...., the %s killed me!" % it)\r
self.gameOver = True\r
else:\r
self.inv.append(it)\r
\r
- def drop(self,it):\r
+ def drop(self, it):\r
self.inv.remove(it)\r
if it.isFragile:\r
it.breakItem()\r
\r
\r
-def createRooms( rm ):\r
+def createRooms(rm):\r
"""\r
create rooms, using multiline string showing map layout\r
string contains symbols for the following:\r
\r
# scan through input string looking for connections between rooms\r
rows = rm.split("\n")\r
- for row,line in enumerate(rows):\r
- for col,c in enumerate(line):\r
+ for row, line in enumerate(rows):\r
+ for col, c in enumerate(line):\r
if c in string.ascii_letters:\r
room = ret[c]\r
n = None\r
# look in neighboring cells for connection symbols (must take\r
# care to guard that neighboring cells exist before testing\r
# contents)\r
- if col > 0 and line[col-1] in "<-":\r
- other = line[col-2]\r
+ if col > 0 and line[col - 1] in "<-":\r
+ other = line[col - 2]\r
w = ret[other]\r
- if col < len(line)-1 and line[col+1] in "->":\r
- other = line[col+2]\r
+ if col < len(line) - 1 and line[col + 1] in "->":\r
+ other = line[col + 2]\r
e = ret[other]\r
- if row > 1 and col < len(rows[row-1]) and rows[row-1][col] in '|^':\r
- other = rows[row-2][col]\r
+ if row > 1 and col < len(rows[row - 1]) and rows[row - 1][col] in "|^":\r
+ other = rows[row - 2][col]\r
n = ret[other]\r
- if row < len(rows)-1 and col < len(rows[row+1]) and rows[row+1][col] in '|.':\r
- other = rows[row+2][col]\r
+ if (\r
+ row < len(rows) - 1\r
+ and col < len(rows[row + 1])\r
+ and rows[row + 1][col] in "|."\r
+ ):\r
+ other = rows[row + 2][col]\r
s = ret[other]\r
\r
# set connections to neighboring rooms\r
- room.doors=[n,s,e,w]\r
+ room.doors = [n, s, e, w]\r
\r
return ret\r
\r
+\r
# put items in rooms\r
-def putItemInRoom(i,r):\r
- if isinstance(r,str):\r
+def putItemInRoom(i, r):\r
+ if isinstance(r, str):\r
r = rooms[r]\r
- r.addItem( Item.items[i] )\r
+ r.addItem(Item.items[i])\r
\r
-def playGame(p,startRoom):\r
+\r
+def playGame(p, startRoom):\r
# create parser\r
parser = Parser()\r
- p.moveTo( startRoom )\r
+ p.moveTo(startRoom)\r
while not p.gameOver:\r
cmdstr = input(">> ")\r
cmd = parser.parseCmd(cmdstr)\r
if cmd is not None:\r
- cmd.command( p )\r
+ cmd.command(p)\r
print()\r
print("You ended the game with:")\r
for i in p.inv:\r
print(" -", aOrAn(i))\r
\r
\r
-#====================\r
+# ====================\r
# start game definition\r
roomMap = """\r
d-Z\r
|\r
A\r
"""\r
-rooms = createRooms( roomMap )\r
+rooms = createRooms(roomMap)\r
rooms["A"].desc = "You are standing on the front porch of a wooden shack."\r
rooms["b"].desc = "You are in a garden."\r
rooms["c"].desc = "You are in a kitchen."\r
\r
# define global variables for referencing rooms\r
frontPorch = rooms["A"]\r
-garden = rooms["b"]\r
-kitchen = rooms["c"]\r
-backPorch = rooms["d"]\r
-library = rooms["e"]\r
-patio = rooms["f"]\r
+garden = rooms["b"]\r
+kitchen = rooms["c"]\r
+backPorch = rooms["d"]\r
+library = rooms["e"]\r
+patio = rooms["f"]\r
\r
# create items\r
-itemNames = """sword.diamond.apple.flower.coin.shovel.book.mirror.telescope.gold bar""".split(".")\r
+itemNames = (\r
+ """sword.diamond.apple.flower.coin.shovel.book.mirror.telescope.gold bar""".split(\r
+ "."\r
+ )\r
+)\r
for itemName in itemNames:\r
- Item( itemName )\r
+ Item(itemName)\r
Item.items["apple"].isDeadly = True\r
Item.items["mirror"].isFragile = True\r
Item.items["coin"].isVisible = False\r
-Item.items["shovel"].usableConditionTest = ( lambda p,t: p.room is garden )\r
-def useShovel(p,subj,target):\r
+Item.items["shovel"].usableConditionTest = lambda p, t: p.room is garden\r
+\r
+\r
+def useShovel(p, subj, target):\r
coin = Item.items["coin"]\r
if not coin.isVisible and coin in p.room.inv:\r
coin.isVisible = True\r
+\r
+\r
Item.items["shovel"].useAction = useShovel\r
\r
Item.items["telescope"].isTakeable = False\r
-def useTelescope(p,subj,target):\r
+\r
+\r
+def useTelescope(p, subj, target):\r
print("You don't see anything.")\r
+\r
+\r
Item.items["telescope"].useAction = useTelescope\r
\r
OpenableItem("treasure chest", Item.items["gold bar"])\r
\r
# create player\r
plyr = Player("Bob")\r
-plyr.take( Item.items["sword"] )\r
+plyr.take(Item.items["sword"])\r
\r
# start game\r
-playGame( plyr, frontPorch )\r
+playGame(plyr, frontPorch)\r
-'''
+"""
antlr_grammar.py
Created on 4 sept. 2010
Submitted by Luca DallOlio, September, 2010
(Minor updates by Paul McGuire, June, 2012)
(Code idiom updates by Paul McGuire, April, 2019)
-'''
-from pyparsing import (Word, ZeroOrMore, printables, Suppress, OneOrMore, Group,
- LineEnd, Optional, White, originalTextFor, hexnums, nums, Combine, Literal, Keyword,
- cStyleComment, Regex, Forward, MatchFirst, And, oneOf, alphas, alphanums,
- delimitedList, Char)
+"""
+from pyparsing import (
+ Word,
+ ZeroOrMore,
+ printables,
+ Suppress,
+ OneOrMore,
+ Group,
+ LineEnd,
+ Optional,
+ White,
+ originalTextFor,
+ hexnums,
+ nums,
+ Combine,
+ Literal,
+ Keyword,
+ cStyleComment,
+ Regex,
+ Forward,
+ MatchFirst,
+ And,
+ oneOf,
+ alphas,
+ alphanums,
+ delimitedList,
+ Char,
+)
# http://www.antlr.org/grammar/ANTLR/ANTLRv3.g
-QUOTE,APOS,EQ,LBRACK,RBRACK,LBRACE,RBRACE,LPAR,RPAR,ROOT,BANG,AT,TIL,SEMI,COLON,VERT = map(Suppress,
- '"\'=[]{}()^!@~;:|')
-BSLASH = Literal('\\')
-keywords = (SRC_, SCOPE_, OPTIONS_, TOKENS_, FRAGMENT, ID, LEXER, PARSER, GRAMMAR, TREE, CATCH, FINALLY,
- THROWS, PROTECTED, PUBLIC, PRIVATE, ) = map(Keyword,
- """src scope options tokens fragment id lexer parser grammar tree catch finally throws protected
- public private """.split())
+(
+ QUOTE,
+ APOS,
+ EQ,
+ LBRACK,
+ RBRACK,
+ LBRACE,
+ RBRACE,
+ LPAR,
+ RPAR,
+ ROOT,
+ BANG,
+ AT,
+ TIL,
+ SEMI,
+ COLON,
+ VERT,
+) = map(Suppress, "\"'=[]{}()^!@~;:|")
+BSLASH = Literal("\\")
+keywords = (
+ SRC_,
+ SCOPE_,
+ OPTIONS_,
+ TOKENS_,
+ FRAGMENT,
+ ID,
+ LEXER,
+ PARSER,
+ GRAMMAR,
+ TREE,
+ CATCH,
+ FINALLY,
+ THROWS,
+ PROTECTED,
+ PUBLIC,
+ PRIVATE,
+) = map(
+ Keyword,
+ """src scope options tokens fragment id lexer parser grammar tree catch finally throws protected
+ public private """.split(),
+)
KEYWORD = MatchFirst(keywords)
# Tokens
-EOL = Suppress(LineEnd()) # $
+EOL = Suppress(LineEnd()) # $
SGL_PRINTABLE = Char(printables)
-singleTextString = originalTextFor(ZeroOrMore(~EOL + (White(" \t") | Word(printables)))).leaveWhitespace()
+singleTextString = originalTextFor(
+ ZeroOrMore(~EOL + (White(" \t") | Word(printables)))
+).leaveWhitespace()
XDIGIT = hexnums
INT = Word(nums)
-ESC = BSLASH + (oneOf(list(r'nrtbf\">'+"'")) | ('u' + Word(hexnums, exact=4)) | SGL_PRINTABLE)
+ESC = BSLASH + (
+ oneOf(list(r"nrtbf\">" + "'")) | ("u" + Word(hexnums, exact=4)) | SGL_PRINTABLE
+)
LITERAL_CHAR = ESC | ~(APOS | BSLASH) + SGL_PRINTABLE
CHAR_LITERAL = APOS + LITERAL_CHAR + APOS
STRING_LITERAL = APOS + Combine(OneOrMore(LITERAL_CHAR)) + APOS
DOUBLE_QUOTE_STRING_LITERAL = '"' + ZeroOrMore(LITERAL_CHAR) + '"'
-DOUBLE_ANGLE_STRING_LITERAL = '<<' + ZeroOrMore(SGL_PRINTABLE) + '>>'
-TOKEN_REF = Word(alphas.upper(), alphanums+'_')
-RULE_REF = Word(alphas.lower(), alphanums+'_')
-ACTION_ESC = (BSLASH.suppress() + APOS
- | BSLASH.suppress()
- | BSLASH.suppress() + (~(APOS | QUOTE) + SGL_PRINTABLE)
- )
-ACTION_CHAR_LITERAL = (APOS + (ACTION_ESC | ~(BSLASH | APOS) + SGL_PRINTABLE) + APOS)
-ACTION_STRING_LITERAL = (QUOTE + ZeroOrMore(ACTION_ESC | ~(BSLASH | QUOTE) + SGL_PRINTABLE) + QUOTE)
+DOUBLE_ANGLE_STRING_LITERAL = "<<" + ZeroOrMore(SGL_PRINTABLE) + ">>"
+TOKEN_REF = Word(alphas.upper(), alphanums + "_")
+RULE_REF = Word(alphas.lower(), alphanums + "_")
+ACTION_ESC = (
+ BSLASH.suppress() + APOS
+ | BSLASH.suppress()
+ | BSLASH.suppress() + (~(APOS | QUOTE) + SGL_PRINTABLE)
+)
+ACTION_CHAR_LITERAL = APOS + (ACTION_ESC | ~(BSLASH | APOS) + SGL_PRINTABLE) + APOS
+ACTION_STRING_LITERAL = (
+ QUOTE + ZeroOrMore(ACTION_ESC | ~(BSLASH | QUOTE) + SGL_PRINTABLE) + QUOTE
+)
SRC = SRC_.suppress() + ACTION_STRING_LITERAL("file") + INT("line")
id = TOKEN_REF | RULE_REF
-SL_COMMENT = Suppress('//') + Suppress('$ANTLR') + SRC | ZeroOrMore(~EOL + Word(printables)) + EOL
+SL_COMMENT = (
+ Suppress("//") + Suppress("$ANTLR") + SRC
+ | ZeroOrMore(~EOL + Word(printables)) + EOL
+)
ML_COMMENT = cStyleComment
-WS = OneOrMore(Suppress(' ') | Suppress('\t') | (Optional(Suppress('\r')) + Literal('\n')))
+WS = OneOrMore(
+ Suppress(" ") | Suppress("\t") | (Optional(Suppress("\r")) + Literal("\n"))
+)
WS_LOOP = ZeroOrMore(SL_COMMENT | ML_COMMENT)
NESTED_ARG_ACTION = Forward()
-NESTED_ARG_ACTION << (LBRACK
- + ZeroOrMore(NESTED_ARG_ACTION
- | ACTION_STRING_LITERAL
- | ACTION_CHAR_LITERAL)
- + RBRACK)
+NESTED_ARG_ACTION << (
+ LBRACK
+ + ZeroOrMore(NESTED_ARG_ACTION | ACTION_STRING_LITERAL | ACTION_CHAR_LITERAL)
+ + RBRACK
+)
ARG_ACTION = NESTED_ARG_ACTION
NESTED_ACTION = Forward()
-NESTED_ACTION << (LBRACE
- + ZeroOrMore(NESTED_ACTION
- | SL_COMMENT
- | ML_COMMENT
- | ACTION_STRING_LITERAL
- | ACTION_CHAR_LITERAL)
- + RBRACE)
-ACTION = NESTED_ACTION + Optional('?')
+NESTED_ACTION << (
+ LBRACE
+ + ZeroOrMore(
+ NESTED_ACTION
+ | SL_COMMENT
+ | ML_COMMENT
+ | ACTION_STRING_LITERAL
+ | ACTION_CHAR_LITERAL
+ )
+ + RBRACE
+)
+ACTION = NESTED_ACTION + Optional("?")
SCOPE = SCOPE_.suppress()
-OPTIONS = OPTIONS_.suppress() + LBRACE # + WS_LOOP + Suppress('{')
-TOKENS = TOKENS_.suppress() + LBRACE # + WS_LOOP + Suppress('{')
+OPTIONS = OPTIONS_.suppress() + LBRACE # + WS_LOOP + Suppress('{')
+TOKENS = TOKENS_.suppress() + LBRACE # + WS_LOOP + Suppress('{')
TREE_BEGIN = ROOT + LPAR
-RANGE = Suppress('..')
-REWRITE = Suppress('->')
+RANGE = Suppress("..")
+REWRITE = Suppress("->")
# General Parser Definitions
# Grammar heading
-optionValue = id | STRING_LITERAL | CHAR_LITERAL | INT | Literal('*').setName("s")
+optionValue = id | STRING_LITERAL | CHAR_LITERAL | INT | Literal("*").setName("s")
option = Group(id("id") + EQ + optionValue("value"))("option")
optionsSpec = OPTIONS + Group(OneOrMore(option + SEMI))("options") + RBRACE
-tokenSpec = Group(TOKEN_REF("token_ref")
- + (EQ + (STRING_LITERAL | CHAR_LITERAL)("lit")))("token") + SEMI
+tokenSpec = (
+ Group(TOKEN_REF("token_ref") + (EQ + (STRING_LITERAL | CHAR_LITERAL)("lit")))(
+ "token"
+ )
+ + SEMI
+)
tokensSpec = TOKENS + Group(OneOrMore(tokenSpec))("tokens") + RBRACE
attrScope = SCOPE_.suppress() + id + ACTION
grammarType = LEXER + PARSER + TREE
actionScopeName = id | LEXER("l") | PARSER("p")
-action = AT + Optional(actionScopeName + Suppress('::')) + id + ACTION
-
-grammarHeading = (Optional(ML_COMMENT("ML_COMMENT"))
- + Optional(grammarType)
- + GRAMMAR
- + id("grammarName") + SEMI
- + Optional(optionsSpec)
- + Optional(tokensSpec)
- + ZeroOrMore(attrScope)
- + ZeroOrMore(action))
+action = AT + Optional(actionScopeName + Suppress("::")) + id + ACTION
+
+grammarHeading = (
+ Optional(ML_COMMENT("ML_COMMENT"))
+ + Optional(grammarType)
+ + GRAMMAR
+ + id("grammarName")
+ + SEMI
+ + Optional(optionsSpec)
+ + Optional(tokensSpec)
+ + ZeroOrMore(attrScope)
+ + ZeroOrMore(action)
+)
modifier = PROTECTED | PUBLIC | PRIVATE | FRAGMENT
ruleAction = AT + id + ACTION
throwsSpec = THROWS.suppress() + delimitedList(id)
-ruleScopeSpec = ((SCOPE_.suppress() + ACTION)
- | (SCOPE_.suppress() + delimitedList(id) + SEMI)
- | (SCOPE_.suppress() + ACTION + SCOPE_.suppress() + delimitedList(id) + SEMI))
+ruleScopeSpec = (
+ (SCOPE_.suppress() + ACTION)
+ | (SCOPE_.suppress() + delimitedList(id) + SEMI)
+ | (SCOPE_.suppress() + ACTION + SCOPE_.suppress() + delimitedList(id) + SEMI)
+)
unary_op = oneOf("^ !")
notTerminal = CHAR_LITERAL | TOKEN_REF | STRING_LITERAL
-terminal = (CHAR_LITERAL | TOKEN_REF + Optional(ARG_ACTION) | STRING_LITERAL | '.') + Optional(unary_op)
+terminal = (
+ CHAR_LITERAL | TOKEN_REF + Optional(ARG_ACTION) | STRING_LITERAL | "."
+) + Optional(unary_op)
block = Forward()
notSet = TIL + (notTerminal | block)
rangeNotPython = CHAR_LITERAL("c1") + RANGE + CHAR_LITERAL("c2")
-atom = Group((rangeNotPython + Optional(unary_op)("op"))
- | terminal
- | (notSet + Optional(unary_op)("op"))
- | (RULE_REF + Optional(ARG_ACTION("arg")) + Optional(unary_op)("op"))
- )
+atom = Group(
+ (rangeNotPython + Optional(unary_op)("op"))
+ | terminal
+ | (notSet + Optional(unary_op)("op"))
+ | (RULE_REF + Optional(ARG_ACTION("arg")) + Optional(unary_op)("op"))
+)
element = Forward()
-treeSpec = ROOT + LPAR + element*(2,) + RPAR
+treeSpec = ROOT + LPAR + element * (2,) + RPAR
ebnfSuffix = oneOf("? * +")
-ebnf = block + Optional(ebnfSuffix("op") | '=>')
-elementNoOptionSpec = ((id("result_name") + oneOf('= +=')("labelOp") + atom("atom") + Optional(ebnfSuffix))
- | (id("result_name") + oneOf('= +=')("labelOp") + block + Optional(ebnfSuffix))
- | atom("atom") + Optional(ebnfSuffix)
- | ebnf
- | ACTION
- | (treeSpec + Optional(ebnfSuffix))
- ) # | SEMPRED ( '=>' -> GATED_SEMPRED | -> SEMPRED )
+ebnf = block + Optional(ebnfSuffix("op") | "=>")
+elementNoOptionSpec = (
+ (id("result_name") + oneOf("= +=")("labelOp") + atom("atom") + Optional(ebnfSuffix))
+ | (id("result_name") + oneOf("= +=")("labelOp") + block + Optional(ebnfSuffix))
+ | atom("atom") + Optional(ebnfSuffix)
+ | ebnf
+ | ACTION
+ | (treeSpec + Optional(ebnfSuffix))
+) # | SEMPRED ( '=>' -> GATED_SEMPRED | -> SEMPRED )
element <<= Group(elementNoOptionSpec)("element")
# Do not ask me why group is needed twice... seems like the xml that you see is not always the real structure?
alternative = Group(Group(OneOrMore(element))("elements"))
-rewrite = Optional(Literal('TODO REWRITE RULES TODO'))
-block <<= (LPAR
- + Optional(Optional(optionsSpec("opts")) + COLON)
- + Group(alternative('a1')
- + rewrite
- + Group(ZeroOrMore(VERT
- + alternative('a2')
- + rewrite))("alternatives"))("block")
- + RPAR)
-altList = alternative('a1') + rewrite + Group(ZeroOrMore(VERT + alternative('a2') + rewrite))("alternatives")
+rewrite = Optional(Literal("TODO REWRITE RULES TODO"))
+block <<= (
+ LPAR
+ + Optional(Optional(optionsSpec("opts")) + COLON)
+ + Group(
+ alternative("a1")
+ + rewrite
+ + Group(ZeroOrMore(VERT + alternative("a2") + rewrite))("alternatives")
+ )("block")
+ + RPAR
+)
+altList = (
+ alternative("a1")
+ + rewrite
+ + Group(ZeroOrMore(VERT + alternative("a2") + rewrite))("alternatives")
+)
exceptionHandler = CATCH.suppress() + ARG_ACTION + ACTION
finallyClause = FINALLY.suppress() + ACTION
exceptionGroup = (OneOrMore(exceptionHandler) + Optional(finallyClause)) | finallyClause
-ruleHeading = (Optional(ML_COMMENT)("ruleComment")
- + Optional(modifier)("modifier")
- + id("ruleName")
- + Optional("!")
- + Optional(ARG_ACTION("arg"))
- + Optional(Suppress('returns') + ARG_ACTION("rt"))
- + Optional(throwsSpec)
- + Optional(optionsSpec)
- + Optional(ruleScopeSpec)
- + ZeroOrMore(ruleAction))
+ruleHeading = (
+ Optional(ML_COMMENT)("ruleComment")
+ + Optional(modifier)("modifier")
+ + id("ruleName")
+ + Optional("!")
+ + Optional(ARG_ACTION("arg"))
+ + Optional(Suppress("returns") + ARG_ACTION("rt"))
+ + Optional(throwsSpec)
+ + Optional(optionsSpec)
+ + Optional(ruleScopeSpec)
+ + ZeroOrMore(ruleAction)
+)
rule = Group(ruleHeading + COLON + altList + SEMI + Optional(exceptionGroup))("rule")
grammarDef = grammarHeading + Group(OneOrMore(rule))("rules")
+
def grammar():
return grammarDef
+
def __antlrAlternativesConverter(pyparsingRules, antlrBlock):
rule = None
- if hasattr(antlrBlock, 'alternatives') and antlrBlock.alternatives != '' and len(antlrBlock.alternatives) > 0:
+ if (
+ hasattr(antlrBlock, "alternatives")
+ and antlrBlock.alternatives != ""
+ and len(antlrBlock.alternatives) > 0
+ ):
alternatives = []
alternatives.append(__antlrAlternativeConverter(pyparsingRules, antlrBlock.a1))
for alternative in antlrBlock.alternatives:
- alternatives.append(__antlrAlternativeConverter(pyparsingRules, alternative))
+ alternatives.append(
+ __antlrAlternativeConverter(pyparsingRules, alternative)
+ )
rule = MatchFirst(alternatives)("anonymous_or")
- elif hasattr(antlrBlock, 'a1') and antlrBlock.a1 != '':
+ elif hasattr(antlrBlock, "a1") and antlrBlock.a1 != "":
rule = __antlrAlternativeConverter(pyparsingRules, antlrBlock.a1)
else:
- raise Exception('Not yet implemented')
+ raise Exception("Not yet implemented")
assert rule != None
return rule
+
def __antlrAlternativeConverter(pyparsingRules, antlrAlternative):
elementList = []
for element in antlrAlternative.elements:
rule = None
- if hasattr(element.atom, 'c1') and element.atom.c1 != '':
- regex = r'['+str(element.atom.c1[0])+'-'+str(element.atom.c2[0]+']')
+ if hasattr(element.atom, "c1") and element.atom.c1 != "":
+ regex = r"[" + str(element.atom.c1[0]) + "-" + str(element.atom.c2[0] + "]")
rule = Regex(regex)("anonymous_regex")
- elif hasattr(element, 'block') and element.block != '':
+ elif hasattr(element, "block") and element.block != "":
rule = __antlrAlternativesConverter(pyparsingRules, element.block)
else:
ruleRef = element.atom[0]
assert ruleRef in pyparsingRules
rule = pyparsingRules[ruleRef](ruleRef)
- if hasattr(element, 'op') and element.op != '':
- if element.op == '+':
+ if hasattr(element, "op") and element.op != "":
+ if element.op == "+":
rule = Group(OneOrMore(rule))("anonymous_one_or_more")
- elif element.op == '*':
+ elif element.op == "*":
rule = Group(ZeroOrMore(rule))("anonymous_zero_or_more")
- elif element.op == '?':
+ elif element.op == "?":
rule = Optional(rule)
else:
- raise Exception('rule operator not yet implemented : ' + element.op)
+ raise Exception("rule operator not yet implemented : " + element.op)
rule = rule
elementList.append(rule)
if len(elementList) > 1:
assert rule is not None
return rule
+
def __antlrRuleConverter(pyparsingRules, antlrRule):
rule = None
rule = __antlrAlternativesConverter(pyparsingRules, antlrRule)
rule(antlrRule.ruleName)
return rule
+
def antlrConverter(antlrGrammarTree):
pyparsingRules = {}
antlrRules = {}
for antlrRule in antlrGrammarTree.rules:
antlrRules[antlrRule.ruleName] = antlrRule
- pyparsingRules[antlrRule.ruleName] = Forward() # antlr is a top down grammar
+ pyparsingRules[antlrRule.ruleName] = Forward() # antlr is a top down grammar
for antlrRuleName, antlrRule in list(antlrRules.items()):
pyparsingRule = __antlrRuleConverter(pyparsingRules, antlrRule)
assert pyparsingRule != None
return pyparsingRules
+
if __name__ == "__main__":
text = """\
-'''
+"""
Created on 4 sept. 2010
@author: luca
Submitted by Luca DallOlio, September, 2010
-'''
+"""
import unittest
from . import antlr_grammar
-class Test(unittest.TestCase):
-
+class Test(unittest.TestCase):
def testOptionsSpec(self):
text = """options {
language = Python;
}"""
- antlr_grammar.optionsSpec.parseString(text) #@UndefinedVariable
+ antlr_grammar.optionsSpec.parseString(text) # @UndefinedVariable
def testTokensSpec(self):
text = """tokens {
MULT = '*' ;
DIV = '/' ;
}"""
- antlr_grammar.tokensSpec.parseString(text) #@UndefinedVariable
+ antlr_grammar.tokensSpec.parseString(text) # @UndefinedVariable
def testBlock(self):
text = """( PLUS | MINUS )"""
- antlr_grammar.block.parseString(text) #@UndefinedVariable
+ antlr_grammar.block.parseString(text) # @UndefinedVariable
def testRule(self):
text = """expr : term ( ( PLUS | MINUS ) term )* ;"""
- antlr_grammar.rule.parseString(text) #@UndefinedVariable
+ antlr_grammar.rule.parseString(text) # @UndefinedVariable
def testLexerRule(self):
text = """fragment DIGIT : '0'..'9' ;"""
- antlr_grammar.rule.parseString(text) #@UndefinedVariable
+ antlr_grammar.rule.parseString(text) # @UndefinedVariable
def testLexerRule2(self):
text = """WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+ { $channel = HIDDEN; } ;"""
- #antlr_grammar.rule.parseString(text) #@UndefinedVariable
+ # antlr_grammar.rule.parseString(text) #@UndefinedVariable
def testGrammar(self):
text = """grammar SimpleCalc;
/* WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+ { $channel = HIDDEN; } ; */
fragment DIGIT : '0'..'9' ;"""
- antlrGrammarTree = antlr_grammar.grammarDef.parseString(text) #@UndefinedVariable
+ antlrGrammarTree = antlr_grammar.grammarDef.parseString(
+ text
+ ) # @UndefinedVariable
pyparsingRules = antlr_grammar.antlrConverter(antlrGrammarTree)
pyparsingRule = pyparsingRules["expr"]
pyparsingTree = pyparsingRule.parseString("2 - 5 * 42 + 7 / 25")
pyparsingTreeList = pyparsingTree.asList()
print(pyparsingTreeList)
- self.assertEqual(pyparsingTreeList,
- [[[['2'], []], [['-', [['5'], [['*', ['4', '2']]]]], ['+', [['7'], [['/', ['2', '5']]]]]]]]
- )
+ self.assertEqual(
+ pyparsingTreeList,
+ [
+ [
+ [["2"], []],
+ [
+ ["-", [["5"], [["*", ["4", "2"]]]]],
+ ["+", [["7"], [["/", ["2", "5"]]]]],
+ ],
+ ]
+ ],
+ )
+
if __name__ == "__main__":
- #import sys;sys.argv = ['', 'Test.testOptionsSpec']
+ # import sys;sys.argv = ['', 'Test.testOptionsSpec']
unittest.main()
from pyparsing import *\r
\r
# define punctuation and simple tokens for locating API calls\r
-LBRACK,RBRACK,LBRACE,RBRACE = map(Suppress,"[]{}")\r
-ident = Word(alphas,alphanums+"_") | QuotedString("{",endQuoteChar="}")\r
+LBRACK, RBRACK, LBRACE, RBRACE = map(Suppress, "[]{}")\r
+ident = Word(alphas, alphanums + "_") | QuotedString("{", endQuoteChar="}")\r
arg = "$" + ident\r
\r
# define an API call with a specific number of arguments - using '-'\r
# will ensure that after matching procname, an incorrect number of args will\r
# raise a ParseSyntaxException, which will interrupt the scanString\r
def apiProc(name, numargs):\r
- return LBRACK + Keyword(name)("procname") - arg*numargs + RBRACK\r
+ return LBRACK + Keyword(name)("procname") - arg * numargs + RBRACK\r
+\r
\r
# create an apiReference, listing all API functions to be scanned for, and\r
# their respective number of arguments. Beginning the overall expression\r
# with FollowedBy allows us to quickly rule out non-api calls while scanning,\r
# since all of the api calls begin with a "["\r
-apiRef = FollowedBy("[") + MatchFirst([\r
- apiProc("procname1", 2),\r
- apiProc("procname2", 1),\r
- apiProc("procname3", 2),\r
- ])\r
+apiRef = FollowedBy("[") + MatchFirst(\r
+ [\r
+ apiProc("procname1", 2),\r
+ apiProc("procname2", 1),\r
+ apiProc("procname3", 2),\r
+ ]\r
+)\r
\r
test = """[ procname1 $par1 $par2 ]\r
other code here\r
api_scanner = apiRef.scanString(test)\r
while 1:\r
try:\r
- t,s,e = next(api_scanner)\r
- print("found %s on line %d" % (t.procname, lineno(s,test)))\r
+ t, s, e = next(api_scanner)\r
+ print("found %s on line %d" % (t.procname, lineno(s, test)))\r
except ParseSyntaxException as pe:\r
print("invalid arg count on line", pe.lineno)\r
- print(pe.lineno,':',pe.line)\r
+ print(pe.lineno, ":", pe.line)\r
# reset api scanner to start after this exception location\r
- test = "\n"*(pe.lineno-1)+test[pe.loc+1:]\r
+ test = "\n" * (pe.lineno - 1) + test[pe.loc + 1 :]\r
api_scanner = apiRef.scanString(test)\r
except StopIteration:\r
break\r
--- /dev/null
+# bigquery_view_parser.py
+#
+# A parser to extract table names from BigQuery view definitions.
+# This is based on the `select_parser.py` sample in pyparsing:
+# https://github.com/pyparsing/pyparsing/blob/master/examples/select_parser.py
+#
+# Michael Smedberg
+#
+
+from pyparsing import ParserElement, Suppress, Forward, CaselessKeyword
+from pyparsing import MatchFirst, alphas, alphanums, Combine, Word
+from pyparsing import QuotedString, CharsNotIn, Optional, Group, ZeroOrMore
+from pyparsing import oneOf, delimitedList, restOfLine, cStyleComment
+from pyparsing import infixNotation, opAssoc, Regex, nums
+
+ParserElement.enablePackrat()
+
+
+class BigQueryViewParser:
+ """Parser to extract table info from BigQuery view definitions"""
+
+ _parser = None
+ _table_identifiers = set()
+ _with_aliases = set()
+
+ def get_table_names(self, sql_stmt):
+ table_identifiers, with_aliases = self._parse(sql_stmt)
+
+ # Table names and alias names might differ by case, but that's not
+ # relevant- aliases are not case sensitive
+ lower_aliases = BigQueryViewParser.lowercase_set_of_tuples(with_aliases)
+ tables = {
+ x
+ for x in table_identifiers
+ if not BigQueryViewParser.lowercase_of_tuple(x) in lower_aliases
+ }
+
+ # Table names ARE case sensitive as described at
+ # https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical#case_sensitivity
+ return tables
+
+ def _parse(self, sql_stmt):
+ BigQueryViewParser._table_identifiers.clear()
+ BigQueryViewParser._with_aliases.clear()
+ BigQueryViewParser._get_parser().parseString(sql_stmt)
+
+ return (BigQueryViewParser._table_identifiers, BigQueryViewParser._with_aliases)
+
+ @classmethod
+ def lowercase_of_tuple(cls, tuple_to_lowercase):
+ return tuple(x.lower() if x else None for x in tuple_to_lowercase)
+
+ @classmethod
+ def lowercase_set_of_tuples(cls, set_of_tuples):
+ return {BigQueryViewParser.lowercase_of_tuple(x) for x in set_of_tuples}
+
+ @classmethod
+ def _get_parser(cls):
+ if cls._parser is not None:
+ return cls._parser
+
+ ParserElement.enablePackrat()
+
+ LPAR, RPAR, COMMA, LBRACKET, RBRACKET, LT, GT = map(Suppress, "(),[]<>")
+ ungrouped_select_stmt = Forward().setName("select statement")
+
+ # keywords
+ (
+ UNION,
+ ALL,
+ AND,
+ INTERSECT,
+ EXCEPT,
+ COLLATE,
+ ASC,
+ DESC,
+ ON,
+ USING,
+ NATURAL,
+ INNER,
+ CROSS,
+ LEFT,
+ RIGHT,
+ OUTER,
+ FULL,
+ JOIN,
+ AS,
+ INDEXED,
+ NOT,
+ SELECT,
+ DISTINCT,
+ FROM,
+ WHERE,
+ GROUP,
+ BY,
+ HAVING,
+ ORDER,
+ BY,
+ LIMIT,
+ OFFSET,
+ OR,
+ CAST,
+ ISNULL,
+ NOTNULL,
+ NULL,
+ IS,
+ BETWEEN,
+ ELSE,
+ END,
+ CASE,
+ WHEN,
+ THEN,
+ EXISTS,
+ COLLATE,
+ IN,
+ LIKE,
+ GLOB,
+ REGEXP,
+ MATCH,
+ ESCAPE,
+ CURRENT_TIME,
+ CURRENT_DATE,
+ CURRENT_TIMESTAMP,
+ WITH,
+ EXTRACT,
+ PARTITION,
+ ROWS,
+ RANGE,
+ UNBOUNDED,
+ PRECEDING,
+ CURRENT,
+ ROW,
+ FOLLOWING,
+ OVER,
+ INTERVAL,
+ DATE_ADD,
+ DATE_SUB,
+ ADDDATE,
+ SUBDATE,
+ REGEXP_EXTRACT,
+ SPLIT,
+ ORDINAL,
+ FIRST_VALUE,
+ LAST_VALUE,
+ NTH_VALUE,
+ LEAD,
+ LAG,
+ PERCENTILE_CONT,
+ PRECENTILE_DISC,
+ RANK,
+ DENSE_RANK,
+ PERCENT_RANK,
+ CUME_DIST,
+ NTILE,
+ ROW_NUMBER,
+ DATE,
+ TIME,
+ DATETIME,
+ TIMESTAMP,
+ UNNEST,
+ INT64,
+ NUMERIC,
+ FLOAT64,
+ BOOL,
+ BYTES,
+ GEOGRAPHY,
+ ARRAY,
+ STRUCT,
+ SAFE_CAST,
+ ANY_VALUE,
+ ARRAY_AGG,
+ ARRAY_CONCAT_AGG,
+ AVG,
+ BIT_AND,
+ BIT_OR,
+ BIT_XOR,
+ COUNT,
+ COUNTIF,
+ LOGICAL_AND,
+ LOGICAL_OR,
+ MAX,
+ MIN,
+ STRING_AGG,
+ SUM,
+ CORR,
+ COVAR_POP,
+ COVAR_SAMP,
+ STDDEV_POP,
+ STDDEV_SAMP,
+ STDDEV,
+ VAR_POP,
+ VAR_SAMP,
+ VARIANCE,
+ TIMESTAMP_ADD,
+ TIMESTAMP_SUB,
+ GENERATE_ARRAY,
+ GENERATE_DATE_ARRAY,
+ GENERATE_TIMESTAMP_ARRAY,
+ FOR,
+ SYSTEMTIME,
+ AS,
+ OF,
+ WINDOW,
+ RESPECT,
+ IGNORE,
+ NULLS,
+ ) = map(
+ CaselessKeyword,
+ """
+ UNION, ALL, AND, INTERSECT, EXCEPT, COLLATE, ASC, DESC, ON, USING,
+ NATURAL, INNER, CROSS, LEFT, RIGHT, OUTER, FULL, JOIN, AS, INDEXED,
+ NOT, SELECT, DISTINCT, FROM, WHERE, GROUP, BY, HAVING, ORDER, BY,
+ LIMIT, OFFSET, OR, CAST, ISNULL, NOTNULL, NULL, IS, BETWEEN, ELSE,
+ END, CASE, WHEN, THEN, EXISTS, COLLATE, IN, LIKE, GLOB, REGEXP,
+ MATCH, ESCAPE, CURRENT_TIME, CURRENT_DATE, CURRENT_TIMESTAMP, WITH,
+ EXTRACT, PARTITION, ROWS, RANGE, UNBOUNDED, PRECEDING, CURRENT,
+ ROW, FOLLOWING, OVER, INTERVAL, DATE_ADD, DATE_SUB, ADDDATE,
+ SUBDATE, REGEXP_EXTRACT, SPLIT, ORDINAL, FIRST_VALUE, LAST_VALUE,
+ NTH_VALUE, LEAD, LAG, PERCENTILE_CONT, PRECENTILE_DISC, RANK,
+ DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE, ROW_NUMBER, DATE, TIME,
+ DATETIME, TIMESTAMP, UNNEST, INT64, NUMERIC, FLOAT64, BOOL, BYTES,
+ GEOGRAPHY, ARRAY, STRUCT, SAFE_CAST, ANY_VALUE, ARRAY_AGG,
+ ARRAY_CONCAT_AGG, AVG, BIT_AND, BIT_OR, BIT_XOR, COUNT, COUNTIF,
+ LOGICAL_AND, LOGICAL_OR, MAX, MIN, STRING_AGG, SUM, CORR,
+ COVAR_POP, COVAR_SAMP, STDDEV_POP, STDDEV_SAMP, STDDEV, VAR_POP,
+ VAR_SAMP, VARIANCE, TIMESTAMP_ADD, TIMESTAMP_SUB, GENERATE_ARRAY,
+ GENERATE_DATE_ARRAY, GENERATE_TIMESTAMP_ARRAY, FOR, SYSTEMTIME, AS,
+ OF, WINDOW, RESPECT, IGNORE, NULLS
+ """.replace(
+ ",", ""
+ ).split(),
+ )
+
+ keyword_nonfunctions = MatchFirst(
+ (
+ UNION,
+ ALL,
+ INTERSECT,
+ EXCEPT,
+ COLLATE,
+ ASC,
+ DESC,
+ ON,
+ USING,
+ NATURAL,
+ INNER,
+ CROSS,
+ LEFT,
+ RIGHT,
+ OUTER,
+ FULL,
+ JOIN,
+ AS,
+ INDEXED,
+ NOT,
+ SELECT,
+ DISTINCT,
+ FROM,
+ WHERE,
+ GROUP,
+ BY,
+ HAVING,
+ ORDER,
+ BY,
+ LIMIT,
+ OFFSET,
+ CAST,
+ ISNULL,
+ NOTNULL,
+ NULL,
+ IS,
+ BETWEEN,
+ ELSE,
+ END,
+ CASE,
+ WHEN,
+ THEN,
+ EXISTS,
+ COLLATE,
+ IN,
+ LIKE,
+ GLOB,
+ REGEXP,
+ MATCH,
+ STRUCT,
+ WINDOW,
+ )
+ )
+
+ keyword = keyword_nonfunctions | MatchFirst(
+ (
+ ESCAPE,
+ CURRENT_TIME,
+ CURRENT_DATE,
+ CURRENT_TIMESTAMP,
+ DATE_ADD,
+ DATE_SUB,
+ ADDDATE,
+ SUBDATE,
+ INTERVAL,
+ STRING_AGG,
+ REGEXP_EXTRACT,
+ SPLIT,
+ ORDINAL,
+ UNNEST,
+ SAFE_CAST,
+ PARTITION,
+ TIMESTAMP_ADD,
+ TIMESTAMP_SUB,
+ ARRAY,
+ GENERATE_ARRAY,
+ GENERATE_DATE_ARRAY,
+ GENERATE_TIMESTAMP_ARRAY,
+ )
+ )
+
+ identifier_word = Word(alphas + "_@#", alphanums + "@$#_")
+ identifier = ~keyword + identifier_word.copy()
+ collation_name = identifier.copy()
+ # NOTE: Column names can be keywords. Doc says they cannot, but in practice it seems to work.
+ column_name = identifier_word.copy()
+ qualified_column_name = Combine(
+ column_name
+ + (ZeroOrMore(" ") + "." + ZeroOrMore(" ") + column_name) * (0, 6)
+ )
+ # NOTE: As with column names, column aliases can be keywords, e.g. functions like `current_time`. Other
+ # keywords, e.g. `from` make parsing pretty difficult (e.g. "SELECT a from from b" is confusing.)
+ column_alias = ~keyword_nonfunctions + column_name.copy()
+ table_name = identifier.copy()
+ table_alias = identifier.copy()
+ index_name = identifier.copy()
+ function_name = identifier.copy()
+ parameter_name = identifier.copy()
+ # NOTE: The expression in a CASE statement can be an integer. E.g. this is valid SQL:
+ # select CASE 1 WHEN 1 THEN -1 ELSE -2 END from test_table
+ unquoted_case_identifier = ~keyword + Word(alphanums + "$_")
+ quoted_case_identifier = ~keyword + (
+ QuotedString('"') ^ Suppress("`") + CharsNotIn("`") + Suppress("`")
+ )
+ case_identifier = quoted_case_identifier | unquoted_case_identifier
+ case_expr = (
+ Optional(case_identifier + Suppress("."))
+ + Optional(case_identifier + Suppress("."))
+ + case_identifier
+ )
+
+ # expression
+ expr = Forward().setName("expression")
+
+ integer = Regex(r"[+-]?\d+")
+ numeric_literal = Regex(r"[+-]?\d*\.?\d+([eE][+-]?\d+)?")
+ string_literal = QuotedString("'") | QuotedString('"') | QuotedString("`")
+ regex_literal = "r" + string_literal
+ blob_literal = Regex(r"[xX]'[0-9A-Fa-f]+'")
+ date_or_time_literal = (DATE | TIME | DATETIME | TIMESTAMP) + string_literal
+ literal_value = (
+ numeric_literal
+ | string_literal
+ | regex_literal
+ | blob_literal
+ | date_or_time_literal
+ | NULL
+ | CURRENT_TIME + Optional(LPAR + Optional(string_literal) + RPAR)
+ | CURRENT_DATE + Optional(LPAR + Optional(string_literal) + RPAR)
+ | CURRENT_TIMESTAMP + Optional(LPAR + Optional(string_literal) + RPAR)
+ )
+ bind_parameter = Word("?", nums) | Combine(oneOf(": @ $") + parameter_name)
+ type_name = oneOf(
+ """TEXT REAL INTEGER BLOB NULL TIMESTAMP STRING DATE
+ INT64 NUMERIC FLOAT64 BOOL BYTES DATETIME GEOGRAPHY TIME ARRAY
+ STRUCT""",
+ caseless=True,
+ )
+ date_part = oneOf(
+ """DAY DAY_HOUR DAY_MICROSECOND DAY_MINUTE DAY_SECOND
+ HOUR HOUR_MICROSECOND HOUR_MINUTE HOUR_SECOND MICROSECOND MINUTE
+ MINUTE_MICROSECOND MINUTE_SECOND MONTH QUARTER SECOND
+ SECOND_MICROSECOND WEEK YEAR YEAR_MONTH""",
+ caseless=True,
+ )
+ datetime_operators = (
+ DATE_ADD | DATE_SUB | ADDDATE | SUBDATE | TIMESTAMP_ADD | TIMESTAMP_SUB
+ )
+
+ grouping_term = expr.copy()
+ ordering_term = Group(
+ expr("order_key")
+ + Optional(COLLATE + collation_name("collate"))
+ + Optional(ASC | DESC)("direction")
+ )("ordering_term")
+
+ function_arg = expr.copy()("function_arg")
+ function_args = Optional(
+ "*"
+ | Optional(DISTINCT)
+ + delimitedList(function_arg)
+ + Optional((RESPECT | IGNORE) + NULLS)
+ )("function_args")
+ function_call = (
+ (function_name | keyword)("function_name")
+ + LPAR
+ + Group(function_args)("function_args_group")
+ + RPAR
+ )
+
+ navigation_function_name = (
+ FIRST_VALUE
+ | LAST_VALUE
+ | NTH_VALUE
+ | LEAD
+ | LAG
+ | PERCENTILE_CONT
+ | PRECENTILE_DISC
+ )
+ aggregate_function_name = (
+ ANY_VALUE
+ | ARRAY_AGG
+ | ARRAY_CONCAT_AGG
+ | AVG
+ | BIT_AND
+ | BIT_OR
+ | BIT_XOR
+ | COUNT
+ | COUNTIF
+ | LOGICAL_AND
+ | LOGICAL_OR
+ | MAX
+ | MIN
+ | STRING_AGG
+ | SUM
+ )
+ statistical_aggregate_function_name = (
+ CORR
+ | COVAR_POP
+ | COVAR_SAMP
+ | STDDEV_POP
+ | STDDEV_SAMP
+ | STDDEV
+ | VAR_POP
+ | VAR_SAMP
+ | VARIANCE
+ )
+ numbering_function_name = (
+ RANK | DENSE_RANK | PERCENT_RANK | CUME_DIST | NTILE | ROW_NUMBER
+ )
+ analytic_function_name = (
+ navigation_function_name
+ | aggregate_function_name
+ | statistical_aggregate_function_name
+ | numbering_function_name
+ )("analytic_function_name")
+ partition_expression_list = delimitedList(grouping_term)(
+ "partition_expression_list"
+ )
+ window_frame_boundary_start = (
+ UNBOUNDED + PRECEDING
+ | numeric_literal + (PRECEDING | FOLLOWING)
+ | CURRENT + ROW
+ )
+ window_frame_boundary_end = (
+ UNBOUNDED + FOLLOWING
+ | numeric_literal + (PRECEDING | FOLLOWING)
+ | CURRENT + ROW
+ )
+ window_frame_clause = (ROWS | RANGE) + (
+ ((UNBOUNDED + PRECEDING) | (numeric_literal + PRECEDING) | (CURRENT + ROW))
+ | (BETWEEN + window_frame_boundary_start + AND + window_frame_boundary_end)
+ )
+ window_name = identifier.copy()("window_name")
+ window_specification = (
+ Optional(window_name)
+ + Optional(PARTITION + BY + partition_expression_list)
+ + Optional(ORDER + BY + delimitedList(ordering_term))
+ + Optional(window_frame_clause)("window_specification")
+ )
+ analytic_function = (
+ analytic_function_name
+ + LPAR
+ + function_args
+ + RPAR
+ + OVER
+ + (window_name | LPAR + Optional(window_specification) + RPAR)
+ )("analytic_function")
+
+ string_agg_term = (
+ STRING_AGG
+ + LPAR
+ + Optional(DISTINCT)
+ + expr
+ + Optional(COMMA + string_literal)
+ + Optional(
+ ORDER + BY + expr + Optional(ASC | DESC) + Optional(LIMIT + integer)
+ )
+ + RPAR
+ )("string_agg")
+ array_literal = (
+ Optional(ARRAY + Optional(LT + delimitedList(type_name) + GT))
+ + LBRACKET
+ + delimitedList(expr)
+ + RBRACKET
+ )
+ interval = INTERVAL + expr + date_part
+ array_generator = (
+ GENERATE_ARRAY
+ + LPAR
+ + numeric_literal
+ + COMMA
+ + numeric_literal
+ + COMMA
+ + numeric_literal
+ + RPAR
+ )
+ date_array_generator = (
+ (GENERATE_DATE_ARRAY | GENERATE_TIMESTAMP_ARRAY)
+ + LPAR
+ + expr("start_date")
+ + COMMA
+ + expr("end_date")
+ + Optional(COMMA + interval)
+ + RPAR
+ )
+
+ explicit_struct = (
+ STRUCT
+ + Optional(LT + delimitedList(type_name) + GT)
+ + LPAR
+ + Optional(delimitedList(expr + Optional(AS + identifier)))
+ + RPAR
+ )
+
+ case_when = WHEN + expr.copy()("when")
+ case_then = THEN + expr.copy()("then")
+ case_clauses = Group(ZeroOrMore(case_when + case_then))
+ case_else = ELSE + expr.copy()("else")
+ case_stmt = (
+ CASE
+ + Optional(case_expr.copy())
+ + case_clauses("case_clauses")
+ + Optional(case_else)
+ + END
+ )("case")
+
+ expr_term = (
+ (analytic_function)("analytic_function")
+ | (CAST + LPAR + expr + AS + type_name + RPAR)("cast")
+ | (SAFE_CAST + LPAR + expr + AS + type_name + RPAR)("safe_cast")
+ | (Optional(EXISTS) + LPAR + ungrouped_select_stmt + RPAR)("subselect")
+ | (literal_value)("literal")
+ | (bind_parameter)("bind_parameter")
+ | (EXTRACT + LPAR + expr + FROM + expr + RPAR)("extract")
+ | case_stmt
+ | (datetime_operators + LPAR + expr + COMMA + interval + RPAR)(
+ "date_operation"
+ )
+ | string_agg_term("string_agg_term")
+ | array_literal("array_literal")
+ | array_generator("array_generator")
+ | date_array_generator("date_array_generator")
+ | explicit_struct("explicit_struct")
+ | function_call("function_call")
+ | qualified_column_name("column")
+ ) + Optional(LBRACKET + (OFFSET | ORDINAL) + LPAR + expr + RPAR + RBRACKET)(
+ "offset_ordinal"
+ )
+
+ struct_term = LPAR + delimitedList(expr_term) + RPAR
+
+ UNARY, BINARY, TERNARY = 1, 2, 3
+ expr << infixNotation(
+ (expr_term | struct_term),
+ [
+ (oneOf("- + ~") | NOT, UNARY, opAssoc.RIGHT),
+ (ISNULL | NOTNULL | NOT + NULL, UNARY, opAssoc.LEFT),
+ ("||", BINARY, opAssoc.LEFT),
+ (oneOf("* / %"), BINARY, opAssoc.LEFT),
+ (oneOf("+ -"), BINARY, opAssoc.LEFT),
+ (oneOf("<< >> & |"), BINARY, opAssoc.LEFT),
+ (oneOf("= > < >= <= <> != !< !>"), BINARY, opAssoc.LEFT),
+ (
+ IS + Optional(NOT)
+ | Optional(NOT) + IN
+ | Optional(NOT) + LIKE
+ | GLOB
+ | MATCH
+ | REGEXP,
+ BINARY,
+ opAssoc.LEFT,
+ ),
+ ((BETWEEN, AND), TERNARY, opAssoc.LEFT),
+ (
+ Optional(NOT)
+ + IN
+ + LPAR
+ + Group(ungrouped_select_stmt | delimitedList(expr))
+ + RPAR,
+ UNARY,
+ opAssoc.LEFT,
+ ),
+ (AND, BINARY, opAssoc.LEFT),
+ (OR, BINARY, opAssoc.LEFT),
+ ],
+ )
+ quoted_expr = (
+ expr
+ ^ Suppress('"') + expr + Suppress('"')
+ ^ Suppress("'") + expr + Suppress("'")
+ ^ Suppress("`") + expr + Suppress("`")
+ )("quoted_expr")
+
+ compound_operator = (
+ UNION + Optional(ALL | DISTINCT)
+ | INTERSECT + DISTINCT
+ | EXCEPT + DISTINCT
+ | INTERSECT
+ | EXCEPT
+ )("compound_operator")
+
+ join_constraint = Group(
+ Optional(
+ ON + expr
+ | USING + LPAR + Group(delimitedList(qualified_column_name)) + RPAR
+ )
+ )("join_constraint")
+
+ join_op = (
+ COMMA
+ | Group(
+ Optional(NATURAL)
+ + Optional(
+ INNER
+ | CROSS
+ | LEFT + OUTER
+ | LEFT
+ | RIGHT + OUTER
+ | RIGHT
+ | FULL + OUTER
+ | OUTER
+ | FULL
+ )
+ + JOIN
+ )
+ )("join_op")
+
+ join_source = Forward()
+
+ # We support three kinds of table identifiers.
+ #
+ # First, dot delimited info like project.dataset.table, where
+ # each component follows the rules described in the BigQuery
+ # docs, namely:
+ # Contain letters (upper or lower case), numbers, and underscores
+ #
+ # Second, a dot delimited quoted string. Since it's quoted, we'll be
+ # liberal w.r.t. what characters we allow. E.g.:
+ # `project.dataset.name-with-dashes`
+ #
+ # Third, a series of quoted strings, delimited by dots, e.g.:
+ # `project`.`dataset`.`name-with-dashes`
+ #
+ # We also support combinations, like:
+ # project.dataset.`name-with-dashes`
+ # `project`.`dataset.name-with-dashes`
+
+ def record_table_identifier(t):
+ identifier_list = t.asList()
+ padded_list = [None] * (3 - len(identifier_list)) + identifier_list
+ cls._table_identifiers.add(tuple(padded_list))
+
+ standard_table_part = ~keyword + Word(alphanums + "_")
+ quoted_project_part = (
+ Suppress('"') + CharsNotIn('"') + Suppress('"')
+ | Suppress("'") + CharsNotIn("'") + Suppress("'")
+ | Suppress("`") + CharsNotIn("`") + Suppress("`")
+ )
+ quoted_table_part = (
+ Suppress('"') + CharsNotIn('".') + Suppress('"')
+ | Suppress("'") + CharsNotIn("'.") + Suppress("'")
+ | Suppress("`") + CharsNotIn("`.") + Suppress("`")
+ )
+ quoted_table_parts_identifier = (
+ Optional(
+ (quoted_project_part("project") | standard_table_part("project"))
+ + Suppress(".")
+ )
+ + Optional(
+ (quoted_table_part("dataset") | standard_table_part("dataset"))
+ + Suppress(".")
+ )
+ + (quoted_table_part("table") | standard_table_part("table"))
+ ).setParseAction(record_table_identifier)
+
+ def record_quoted_table_identifier(t):
+ identifier_list = t.asList()[0].split(".")
+ first = ".".join(identifier_list[0:-2]) or None
+ second = identifier_list[-2]
+ third = identifier_list[-1]
+ identifier_list = [first, second, third]
+ padded_list = [None] * (3 - len(identifier_list)) + identifier_list
+ cls._table_identifiers.add(tuple(padded_list))
+
+ quotable_table_parts_identifier = (
+ Suppress('"') + CharsNotIn('"') + Suppress('"')
+ | Suppress("'") + CharsNotIn("'") + Suppress("'")
+ | Suppress("`") + CharsNotIn("`") + Suppress("`")
+ ).setParseAction(record_quoted_table_identifier)
+
+ table_identifier = (
+ quoted_table_parts_identifier | quotable_table_parts_identifier
+ )
+ single_source = (
+ (
+ table_identifier
+ + Optional(Optional(AS) + table_alias("table_alias*"))
+ + Optional(FOR + SYSTEMTIME + AS + OF + string_literal)
+ + Optional(INDEXED + BY + index_name("name") | NOT + INDEXED)
+ )("index")
+ | (LPAR + ungrouped_select_stmt + RPAR)
+ | (LPAR + join_source + RPAR)
+ | (UNNEST + LPAR + expr + RPAR)
+ ) + Optional(Optional(AS) + table_alias)
+
+ join_source << single_source + ZeroOrMore(
+ join_op + single_source + join_constraint
+ )
+
+ over_partition = (PARTITION + BY + delimitedList(partition_expression_list))(
+ "over_partition"
+ )
+ over_order = ORDER + BY + delimitedList(ordering_term)
+ over_unsigned_value_specification = expr
+ over_window_frame_preceding = (
+ UNBOUNDED + PRECEDING
+ | over_unsigned_value_specification + PRECEDING
+ | CURRENT + ROW
+ )
+ over_window_frame_following = (
+ UNBOUNDED + FOLLOWING
+ | over_unsigned_value_specification + FOLLOWING
+ | CURRENT + ROW
+ )
+ over_window_frame_bound = (
+ over_window_frame_preceding | over_window_frame_following
+ )
+ over_window_frame_between = (
+ BETWEEN + over_window_frame_bound + AND + over_window_frame_bound
+ )
+ over_window_frame_extent = (
+ over_window_frame_preceding | over_window_frame_between
+ )
+ over_row_or_range = (ROWS | RANGE) + over_window_frame_extent
+ over = (
+ OVER
+ + LPAR
+ + Optional(over_partition)
+ + Optional(over_order)
+ + Optional(over_row_or_range)
+ + RPAR
+ )("over")
+
+ result_column = Optional(table_name + ".") + "*" + Optional(
+ EXCEPT + LPAR + delimitedList(column_name) + RPAR
+ ) | Group(quoted_expr + Optional(over) + Optional(Optional(AS) + column_alias))
+
+ window_select_clause = (
+ WINDOW + identifier + AS + LPAR + window_specification + RPAR
+ )
+
+ with_stmt = Forward().setName("with statement")
+ ungrouped_select_no_with = (
+ SELECT
+ + Optional(DISTINCT | ALL)
+ + Group(delimitedList(result_column))("columns")
+ + Optional(FROM + join_source("from*"))
+ + Optional(WHERE + expr)
+ + Optional(
+ GROUP + BY + Group(delimitedList(grouping_term))("group_by_terms")
+ )
+ + Optional(HAVING + expr("having_expr"))
+ + Optional(
+ ORDER + BY + Group(delimitedList(ordering_term))("order_by_terms")
+ )
+ + Optional(delimitedList(window_select_clause))
+ )
+ select_no_with = ungrouped_select_no_with | (
+ LPAR + ungrouped_select_no_with + RPAR
+ )
+ select_core = Optional(with_stmt) + select_no_with
+ grouped_select_core = select_core | (LPAR + select_core + RPAR)
+
+ ungrouped_select_stmt << (
+ grouped_select_core
+ + ZeroOrMore(compound_operator + grouped_select_core)
+ + Optional(
+ LIMIT
+ + (Group(expr + OFFSET + expr) | Group(expr + COMMA + expr) | expr)(
+ "limit"
+ )
+ )
+ )("select")
+ select_stmt = ungrouped_select_stmt | (LPAR + ungrouped_select_stmt + RPAR)
+
+ # define comment format, and ignore them
+ sql_comment = oneOf("-- #") + restOfLine | cStyleComment
+ select_stmt.ignore(sql_comment)
+
+ def record_with_alias(t):
+ identifier_list = t.asList()
+ padded_list = [None] * (3 - len(identifier_list)) + identifier_list
+ cls._with_aliases.add(tuple(padded_list))
+
+ with_clause = Group(
+ identifier.setParseAction(record_with_alias)
+ + AS
+ + LPAR
+ + select_stmt
+ + RPAR
+ )
+ with_stmt << (WITH + delimitedList(with_clause))
+ with_stmt.ignore(sql_comment)
+
+ cls._parser = select_stmt
+ return cls._parser
+
+ def test(self, sql_stmt, expected_tables, verbose=False):
+ def print_(*args):
+ if verbose:
+ print(*args)
+
+ print_(sql_stmt.strip())
+ found_tables = self.get_table_names(sql_stmt)
+ print_(found_tables)
+ expected_tables_set = set(expected_tables)
+
+ if expected_tables_set != found_tables:
+ raise Exception(
+ f"Test {test_index} failed- expected {expected_tables_set} but got {found_tables}"
+ )
+ print_()
+
+
+if __name__ == "__main__":
+ TEST_CASES = [
+ [
+ """
+ SELECT x FROM y.a, b
+ """,
+ [
+ (None, "y", "a"),
+ (
+ None,
+ None,
+ "b",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT x FROM y.a JOIN b
+ """,
+ [
+ (None, "y", "a"),
+ (None, None, "b"),
+ ],
+ ],
+ [
+ """
+ select * from xyzzy where z > 100
+ """,
+ [
+ (None, None, "xyzzy"),
+ ],
+ ],
+ [
+ """
+ select * from xyzzy where z > 100 order by zz
+ """,
+ [
+ (None, None, "xyzzy"),
+ ],
+ ],
+ [
+ """
+ select * from xyzzy
+ """,
+ [
+ (
+ None,
+ None,
+ "xyzzy",
+ ),
+ ],
+ ],
+ [
+ """
+ select z.* from xyzzy
+ """,
+ [
+ (
+ None,
+ None,
+ "xyzzy",
+ ),
+ ],
+ ],
+ [
+ """
+ select a, b from test_table where 1=1 and b='yes'
+ """,
+ [
+ (None, None, "test_table"),
+ ],
+ ],
+ [
+ """
+ select a, b from test_table where 1=1 and b in (select bb from foo)
+ """,
+ [
+ (None, None, "test_table"),
+ (None, None, "foo"),
+ ],
+ ],
+ [
+ """
+ select z.a, b from test_table where 1=1 and b in (select bb from foo)
+ """,
+ [
+ (None, None, "test_table"),
+ (None, None, "foo"),
+ ],
+ ],
+ [
+ """
+ select z.a, b from test_table where 1=1 and b in (select bb from foo) order by b,c desc,d
+ """,
+ [
+ (None, None, "test_table"),
+ (None, None, "foo"),
+ ],
+ ],
+ [
+ """
+ select z.a, b from test_table left join test2_table where 1=1 and b in (select bb from foo)
+ """,
+ [
+ (None, None, "test_table"),
+ (None, None, "test2_table"),
+ (None, None, "foo"),
+ ],
+ ],
+ [
+ """
+ select a, db.table.b as BBB from db.table where 1=1 and BBB='yes'
+ """,
+ [
+ (None, "db", "table"),
+ ],
+ ],
+ [
+ """
+ select a, db.table.b as BBB from test_table,db.table where 1=1 and BBB='yes'
+ """,
+ [
+ (None, None, "test_table"),
+ (None, "db", "table"),
+ ],
+ ],
+ [
+ """
+ select a, db.table.b as BBB from test_table,db.table where 1=1 and BBB='yes' limit 50
+ """,
+ [
+ (None, None, "test_table"),
+ (None, "db", "table"),
+ ],
+ ],
+ [
+ """
+ select a, b from test_table where (1=1 or 2=3) and b='yes' group by zx having b=2 order by 1
+ """,
+ [
+ (None, None, "test_table"),
+ ],
+ ],
+ [
+ """
+ select
+ a,
+ b
+ # this is a comment
+ from
+ test_table
+ # another comment
+ where (1=1 or 2=3) and b='yes'
+ #yup, a comment
+ group by zx having b=2 order by 1
+ """,
+ [
+ (None, None, "test_table"),
+ ],
+ ],
+ [
+ """
+ SELECT COUNT(DISTINCT foo) FROM bar JOIN baz ON bar.baz_id = baz.id
+ """,
+ [
+ (None, None, "bar"),
+ (None, None, "baz"),
+ ],
+ ],
+ [
+ """
+ SELECT COUNT(DISTINCT foo) FROM bar, baz WHERE bar.baz_id = baz.id
+ """,
+ [
+ (None, None, "bar"),
+ (None, None, "baz"),
+ ],
+ ],
+ [
+ """
+ WITH one AS (SELECT id FROM foo) SELECT one.id
+ """,
+ [
+ (None, None, "foo"),
+ ],
+ ],
+ [
+ """
+ WITH one AS (SELECT id FROM foo), two AS (select id FROM bar) SELECT one.id, two.id
+ """,
+ [
+ (None, None, "foo"),
+ (None, None, "bar"),
+ ],
+ ],
+ [
+ """
+ SELECT x,
+ RANK() OVER (ORDER BY x ASC) AS rank,
+ DENSE_RANK() OVER (ORDER BY x ASC) AS dense_rank,
+ ROW_NUMBER() OVER (PARTITION BY x ORDER BY y) AS row_num
+ FROM a
+ """,
+ [
+ (
+ None,
+ None,
+ "a",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT x, COUNT(*) OVER ( ORDER BY x
+ RANGE BETWEEN 2 PRECEDING AND 2 FOLLOWING ) AS count_x
+ FROM T
+ """,
+ [
+ (
+ None,
+ None,
+ "T",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT firstname, department, startdate,
+ RANK() OVER ( PARTITION BY department ORDER BY startdate ) AS rank
+ FROM Employees
+ """,
+ [
+ (None, None, "Employees"),
+ ],
+ ],
+ # A fragment from https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ SELECT 'Sophia Liu' as name,
+ TIMESTAMP '2016-10-18 2:51:45' as finish_time,
+ 'F30-34' as division
+ UNION ALL SELECT 'Lisa Stelzner', TIMESTAMP '2016-10-18 2:54:11', 'F35-39'
+ UNION ALL SELECT 'Nikki Leith', TIMESTAMP '2016-10-18 2:59:01', 'F30-34'
+ UNION ALL SELECT 'Lauren Matthews', TIMESTAMP '2016-10-18 3:01:17', 'F35-39'
+ UNION ALL SELECT 'Desiree Berry', TIMESTAMP '2016-10-18 3:05:42', 'F35-39'
+ UNION ALL SELECT 'Suzy Slane', TIMESTAMP '2016-10-18 3:06:24', 'F35-39'
+ UNION ALL SELECT 'Jen Edwards', TIMESTAMP '2016-10-18 3:06:36', 'F30-34'
+ UNION ALL SELECT 'Meghan Lederer', TIMESTAMP '2016-10-18 3:07:41', 'F30-34'
+ UNION ALL SELECT 'Carly Forte', TIMESTAMP '2016-10-18 3:08:58', 'F25-29'
+ UNION ALL SELECT 'Lauren Reasoner', TIMESTAMP '2016-10-18 3:10:14', 'F30-34'
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ WITH finishers AS
+ (SELECT 'Sophia Liu' as name,
+ TIMESTAMP '2016-10-18 2:51:45' as finish_time,
+ 'F30-34' as division
+ UNION ALL SELECT 'Lisa Stelzner', TIMESTAMP '2016-10-18 2:54:11', 'F35-39'
+ UNION ALL SELECT 'Nikki Leith', TIMESTAMP '2016-10-18 2:59:01', 'F30-34'
+ UNION ALL SELECT 'Lauren Matthews', TIMESTAMP '2016-10-18 3:01:17', 'F35-39'
+ UNION ALL SELECT 'Desiree Berry', TIMESTAMP '2016-10-18 3:05:42', 'F35-39'
+ UNION ALL SELECT 'Suzy Slane', TIMESTAMP '2016-10-18 3:06:24', 'F35-39'
+ UNION ALL SELECT 'Jen Edwards', TIMESTAMP '2016-10-18 3:06:36', 'F30-34'
+ UNION ALL SELECT 'Meghan Lederer', TIMESTAMP '2016-10-18 3:07:41', 'F30-34'
+ UNION ALL SELECT 'Carly Forte', TIMESTAMP '2016-10-18 3:08:58', 'F25-29'
+ UNION ALL SELECT 'Lauren Reasoner', TIMESTAMP '2016-10-18 3:10:14', 'F30-34')
+ SELECT name,
+ FORMAT_TIMESTAMP('%X', finish_time) AS finish_time,
+ division,
+ FORMAT_TIMESTAMP('%X', fastest_time) AS fastest_time,
+ TIMESTAMP_DIFF(finish_time, fastest_time, SECOND) AS delta_in_seconds
+ FROM (
+ SELECT name,
+ finish_time,
+ division,
+ FIRST_VALUE(finish_time)
+ OVER (PARTITION BY division ORDER BY finish_time ASC
+ ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS fastest_time
+ FROM finishers)
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ WITH finishers AS
+ (SELECT 'Sophia Liu' as name,
+ TIMESTAMP '2016-10-18 2:51:45' as finish_time,
+ 'F30-34' as division
+ UNION ALL SELECT 'Lisa Stelzner', TIMESTAMP '2016-10-18 2:54:11', 'F35-39'
+ UNION ALL SELECT 'Nikki Leith', TIMESTAMP '2016-10-18 2:59:01', 'F30-34'
+ UNION ALL SELECT 'Lauren Matthews', TIMESTAMP '2016-10-18 3:01:17', 'F35-39'
+ UNION ALL SELECT 'Desiree Berry', TIMESTAMP '2016-10-18 3:05:42', 'F35-39'
+ UNION ALL SELECT 'Suzy Slane', TIMESTAMP '2016-10-18 3:06:24', 'F35-39'
+ UNION ALL SELECT 'Jen Edwards', TIMESTAMP '2016-10-18 3:06:36', 'F30-34'
+ UNION ALL SELECT 'Meghan Lederer', TIMESTAMP '2016-10-18 3:07:41', 'F30-34'
+ UNION ALL SELECT 'Carly Forte', TIMESTAMP '2016-10-18 3:08:58', 'F25-29'
+ UNION ALL SELECT 'Lauren Reasoner', TIMESTAMP '2016-10-18 3:10:14', 'F30-34')
+ SELECT name,
+ FORMAT_TIMESTAMP('%X', finish_time) AS finish_time,
+ division,
+ FORMAT_TIMESTAMP('%X', slowest_time) AS slowest_time,
+ TIMESTAMP_DIFF(slowest_time, finish_time, SECOND) AS delta_in_seconds
+ FROM (
+ SELECT name,
+ finish_time,
+ division,
+ LAST_VALUE(finish_time)
+ OVER (PARTITION BY division ORDER BY finish_time ASC
+ ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS slowest_time
+ FROM finishers)
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ WITH finishers AS
+ (SELECT 'Sophia Liu' as name,
+ TIMESTAMP '2016-10-18 2:51:45' as finish_time,
+ 'F30-34' as division
+ UNION ALL SELECT 'Lisa Stelzner', TIMESTAMP '2016-10-18 2:54:11', 'F35-39'
+ UNION ALL SELECT 'Nikki Leith', TIMESTAMP '2016-10-18 2:59:01', 'F30-34'
+ UNION ALL SELECT 'Lauren Matthews', TIMESTAMP '2016-10-18 3:01:17', 'F35-39'
+ UNION ALL SELECT 'Desiree Berry', TIMESTAMP '2016-10-18 3:05:42', 'F35-39'
+ UNION ALL SELECT 'Suzy Slane', TIMESTAMP '2016-10-18 3:06:24', 'F35-39'
+ UNION ALL SELECT 'Jen Edwards', TIMESTAMP '2016-10-18 3:06:36', 'F30-34'
+ UNION ALL SELECT 'Meghan Lederer', TIMESTAMP '2016-10-18 3:07:41', 'F30-34'
+ UNION ALL SELECT 'Carly Forte', TIMESTAMP '2016-10-18 3:08:58', 'F25-29'
+ UNION ALL SELECT 'Lauren Reasoner', TIMESTAMP '2016-10-18 3:10:14', 'F30-34')
+ SELECT name,
+ FORMAT_TIMESTAMP('%X', finish_time) AS finish_time,
+ division,
+ FORMAT_TIMESTAMP('%X', fastest_time) AS fastest_time,
+ FORMAT_TIMESTAMP('%X', second_fastest) AS second_fastest
+ FROM (
+ SELECT name,
+ finish_time,
+ division,finishers,
+ FIRST_VALUE(finish_time)
+ OVER w1 AS fastest_time,
+ NTH_VALUE(finish_time, 2)
+ OVER w1 as second_fastest
+ FROM finishers
+ WINDOW w1 AS (
+ PARTITION BY division ORDER BY finish_time ASC
+ ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING))
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ WITH finishers AS
+ (SELECT 'Sophia Liu' as name,
+ TIMESTAMP '2016-10-18 2:51:45' as finish_time,
+ 'F30-34' as division
+ UNION ALL SELECT 'Lisa Stelzner', TIMESTAMP '2016-10-18 2:54:11', 'F35-39'
+ UNION ALL SELECT 'Nikki Leith', TIMESTAMP '2016-10-18 2:59:01', 'F30-34'
+ UNION ALL SELECT 'Lauren Matthews', TIMESTAMP '2016-10-18 3:01:17', 'F35-39'
+ UNION ALL SELECT 'Desiree Berry', TIMESTAMP '2016-10-18 3:05:42', 'F35-39'
+ UNION ALL SELECT 'Suzy Slane', TIMESTAMP '2016-10-18 3:06:24', 'F35-39'
+ UNION ALL SELECT 'Jen Edwards', TIMESTAMP '2016-10-18 3:06:36', 'F30-34'
+ UNION ALL SELECT 'Meghan Lederer', TIMESTAMP '2016-10-18 3:07:41', 'F30-34'
+ UNION ALL SELECT 'Carly Forte', TIMESTAMP '2016-10-18 3:08:58', 'F25-29'
+ UNION ALL SELECT 'Lauren Reasoner', TIMESTAMP '2016-10-18 3:10:14', 'F30-34')
+ SELECT name,
+ finish_time,
+ division,
+ LEAD(name)
+ OVER (PARTITION BY division ORDER BY finish_time ASC) AS followed_by
+ FROM finishers
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ WITH finishers AS
+ (SELECT 'Sophia Liu' as name,
+ TIMESTAMP '2016-10-18 2:51:45' as finish_time,
+ 'F30-34' as division
+ UNION ALL SELECT 'Lisa Stelzner', TIMESTAMP '2016-10-18 2:54:11', 'F35-39'
+ UNION ALL SELECT 'Nikki Leith', TIMESTAMP '2016-10-18 2:59:01', 'F30-34'
+ UNION ALL SELECT 'Lauren Matthews', TIMESTAMP '2016-10-18 3:01:17', 'F35-39'
+ UNION ALL SELECT 'Desiree Berry', TIMESTAMP '2016-10-18 3:05:42', 'F35-39'
+ UNION ALL SELECT 'Suzy Slane', TIMESTAMP '2016-10-18 3:06:24', 'F35-39'
+ UNION ALL SELECT 'Jen Edwards', TIMESTAMP '2016-10-18 3:06:36', 'F30-34'
+ UNION ALL SELECT 'Meghan Lederer', TIMESTAMP '2016-10-18 3:07:41', 'F30-34'
+ UNION ALL SELECT 'Carly Forte', TIMESTAMP '2016-10-18 3:08:58', 'F25-29'
+ UNION ALL SELECT 'Lauren Reasoner', TIMESTAMP '2016-10-18 3:10:14', 'F30-34')
+ SELECT name,
+ finish_time,
+ division,
+ LEAD(name, 2)
+ OVER (PARTITION BY division ORDER BY finish_time ASC) AS two_runners_back
+ FROM finishers
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ WITH finishers AS
+ (SELECT 'Sophia Liu' as name,
+ TIMESTAMP '2016-10-18 2:51:45' as finish_time,
+ 'F30-34' as division
+ UNION ALL SELECT 'Lisa Stelzner', TIMESTAMP '2016-10-18 2:54:11', 'F35-39'
+ UNION ALL SELECT 'Nikki Leith', TIMESTAMP '2016-10-18 2:59:01', 'F30-34'
+ UNION ALL SELECT 'Lauren Matthews', TIMESTAMP '2016-10-18 3:01:17', 'F35-39'
+ UNION ALL SELECT 'Desiree Berry', TIMESTAMP '2016-10-18 3:05:42', 'F35-39'
+ UNION ALL SELECT 'Suzy Slane', TIMESTAMP '2016-10-18 3:06:24', 'F35-39'
+ UNION ALL SELECT 'Jen Edwards', TIMESTAMP '2016-10-18 3:06:36', 'F30-34'
+ UNION ALL SELECT 'Meghan Lederer', TIMESTAMP '2016-10-18 3:07:41', 'F30-34'
+ UNION ALL SELECT 'Carly Forte', TIMESTAMP '2016-10-18 3:08:58', 'F25-29'
+ UNION ALL SELECT 'Lauren Reasoner', TIMESTAMP '2016-10-18 3:10:14', 'F30-34')
+ SELECT name,
+ finish_time,
+ division,
+ LAG(name)
+ OVER (PARTITION BY division ORDER BY finish_time ASC) AS preceding_runner
+ FROM finishers
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ SELECT
+ PERCENTILE_CONT(x, 0) OVER() AS min,
+ PERCENTILE_CONT(x, 0.01) OVER() AS percentile1,
+ PERCENTILE_CONT(x, 0.5) OVER() AS median,
+ PERCENTILE_CONT(x, 0.9) OVER() AS percentile90,
+ PERCENTILE_CONT(x, 1) OVER() AS max
+ FROM UNNEST([0, 3, NULL, 1, 2]) AS x LIMIT 1
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions
+ [
+ """
+ SELECT
+ x,
+ PERCENTILE_DISC(x, 0) OVER() AS min,
+ PERCENTILE_DISC(x, 0.5) OVER() AS median,
+ PERCENTILE_DISC(x, 1) OVER() AS max
+ FROM UNNEST(['c', NULL, 'b', 'a']) AS x
+ """,
+ [],
+ ],
+ # From https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions
+ [
+ """
+ SELECT
+ TIMESTAMP "2008-12-25 15:30:00 UTC" as original,
+ TIMESTAMP_ADD(TIMESTAMP "2008-12-25 15:30:00 UTC", INTERVAL 10 MINUTE) AS later
+ """,
+ [],
+ ],
+ # Previously hosted on https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions, but
+ # appears to no longer be there
+ [
+ """
+ WITH date_hour_slots AS (
+ SELECT
+ [
+ STRUCT(
+ " 00:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 01:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 02:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 03:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 04:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 05:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 06:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 07:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 08:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY ) as dt_range),
+ STRUCT(
+ " 09:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01', current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 10:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 11:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 12:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 13:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 14:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 15:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 16:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 17:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 18:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 19:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 20:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 21:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 22:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range),
+ STRUCT(
+ " 23:00:00 UTC" as hrs,
+ GENERATE_DATE_ARRAY('2016-01-01',current_date(), INTERVAL 1 DAY) as dt_range)
+ ]
+ AS full_timestamps)
+ SELECT
+ dt AS dates, hrs, CAST(CONCAT( CAST(dt as STRING), CAST(hrs as STRING)) as TIMESTAMP) as timestamp_value
+ FROM `date_hour_slots`, date_hour_slots.full_timestamps LEFT JOIN full_timestamps.dt_range as dt
+ """,
+ [
+ (None, "date_hour_slots", "full_timestamps"),
+ (None, "full_timestamps", "dt_range"),
+ ],
+ ],
+ [
+ """
+ SELECT
+ [foo],
+ ARRAY[foo],
+ ARRAY<int64, STRING>[foo, bar],
+ STRUCT(1, 3),
+ STRUCT<int64, STRING>(2, 'foo'),
+ current_date(),
+ GENERATE_ARRAY(5, NULL, 1),
+ GENERATE_DATE_ARRAY('2016-10-05', '2016-10-01', INTERVAL 1 DAY),
+ GENERATE_DATE_ARRAY('2016-10-05', NULL),
+ GENERATE_DATE_ARRAY('2016-01-01', '2016-12-31', INTERVAL 2 MONTH),
+ GENERATE_DATE_ARRAY('2000-02-01',current_date(), INTERVAL 1 DAY),
+ GENERATE_TIMESTAMP_ARRAY('2016-10-05 00:00:00', '2016-10-05 00:00:02', INTERVAL 1 SECOND)
+ FROM
+ bar
+ """,
+ [
+ (None, None, "bar"),
+ ],
+ ],
+ [
+ """
+ SELECT GENERATE_ARRAY(start, 5) AS example_array
+ FROM UNNEST([3, 4, 5]) AS start
+ """,
+ [],
+ ],
+ [
+ """
+ WITH StartsAndEnds AS (
+ SELECT DATE '2016-01-01' AS date_start, DATE '2016-01-31' AS date_end
+ UNION ALL SELECT DATE "2016-04-01", DATE "2016-04-30"
+ UNION ALL SELECT DATE "2016-07-01", DATE "2016-07-31"
+ UNION ALL SELECT DATE "2016-10-01", DATE "2016-10-31"
+ )
+ SELECT GENERATE_DATE_ARRAY(date_start, date_end, INTERVAL 1 WEEK) AS date_range
+ FROM StartsAndEnds
+ """,
+ [],
+ ],
+ [
+ """
+ SELECT GENERATE_TIMESTAMP_ARRAY(start_timestamp, end_timestamp, INTERVAL 1 HOUR)
+ AS timestamp_array
+ FROM
+ (SELECT
+ TIMESTAMP '2016-10-05 00:00:00' AS start_timestamp,
+ TIMESTAMP '2016-10-05 02:00:00' AS end_timestamp
+ UNION ALL
+ SELECT
+ TIMESTAMP '2016-10-05 12:00:00' AS start_timestamp,
+ TIMESTAMP '2016-10-05 14:00:00' AS end_timestamp
+ UNION ALL
+ SELECT
+ TIMESTAMP '2016-10-05 23:59:00' AS start_timestamp,
+ TIMESTAMP '2016-10-06 01:59:00' AS end_timestamp)
+ """,
+ [],
+ ],
+ [
+ """
+ SELECT DATE_SUB(current_date("-08:00")), INTERVAL 2 DAY)
+ """,
+ [],
+ ],
+ [
+ """
+ SELECT
+ case when (a) then b else c end
+ FROM d
+ """,
+ [
+ (
+ None,
+ None,
+ "d",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT
+ e,
+ case when (f) then g else h end
+ FROM i
+ """,
+ [
+ (
+ None,
+ None,
+ "i",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT
+ case when j then k else l end
+ FROM m
+ """,
+ [
+ (
+ None,
+ None,
+ "m",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT
+ n,
+ case when o then p else q end
+ FROM r
+ """,
+ [
+ (
+ None,
+ None,
+ "r",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT
+ case s when (t) then u else v end
+ FROM w
+ """,
+ [
+ (
+ None,
+ None,
+ "w",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT
+ x,
+ case y when (z) then aa else ab end
+ FROM ac
+ """,
+ [
+ (
+ None,
+ None,
+ "ac",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT
+ case ad when ae then af else ag end
+ FROM ah
+ """,
+ [
+ (
+ None,
+ None,
+ "ah",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT
+ ai,
+ case aj when ak then al else am end
+ FROM an
+ """,
+ [
+ (
+ None,
+ None,
+ "an",
+ ),
+ ],
+ ],
+ [
+ """
+ WITH
+ ONE AS (SELECT x FROM y),
+ TWO AS (select a FROM b)
+ SELECT y FROM onE JOIN TWo
+ """,
+ [
+ (
+ None,
+ None,
+ "y",
+ ),
+ (
+ None,
+ None,
+ "b",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT
+ a,
+ (SELECT b FROM oNE)
+ FROM OnE
+ """,
+ [
+ (
+ None,
+ None,
+ "oNE",
+ ),
+ (
+ None,
+ None,
+ "OnE",
+ ),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM `a.b.c`
+ """,
+ [
+ ("a", "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM `b.c`
+ """,
+ [
+ (None, "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM `c`
+ """,
+ [
+ (None, None, "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM a.b.c
+ """,
+ [
+ ("a", "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM "a"."b"."c"
+ """,
+ [
+ ("a", "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM 'a'.'b'.'c'
+ """,
+ [
+ ("a", "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM `a`.`b`.`c`
+ """,
+ [
+ ("a", "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM "a.b.c"
+ """,
+ [
+ ("a", "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM 'a.b.c'
+ """,
+ [
+ ("a", "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT * FROM `a.b.c`
+ """,
+ [
+ ("a", "b", "c"),
+ ],
+ ],
+ [
+ """
+ SELECT *
+ FROM t1
+ WHERE t1.a IN (SELECT t2.a
+ FROM t2 ) FOR SYSTEM_TIME AS OF t1.timestamp_column)
+ """,
+ [
+ (None, None, "t1"),
+ (None, None, "t2"),
+ ],
+ ],
+ [
+ """
+ WITH a AS (SELECT b FROM c)
+ SELECT d FROM A JOIN e ON f = g JOIN E ON h = i
+ """,
+ [
+ (None, None, "c"),
+ (None, None, "e"),
+ (None, None, "E"),
+ ],
+ ],
+ [
+ """
+ with
+ a as (
+ (
+ select b from
+ (
+ select c from d
+ )
+ Union all
+ (
+ select e from f
+ )
+ )
+ )
+
+ select g from h
+ """,
+ [
+ (None, None, "d"),
+ (None, None, "f"),
+ (None, None, "h"),
+ ],
+ ],
+ [
+ """
+ select
+ a AS ESCAPE,
+ b AS CURRENT_TIME,
+ c AS CURRENT_DATE,
+ d AS CURRENT_TIMESTAMP,
+ e AS DATE_ADD
+ FROM x
+ """,
+ [
+ (None, None, "x"),
+ ],
+ ],
+ [
+ """
+ WITH x AS (
+ SELECT a
+ FROM b
+ WINDOW w as (PARTITION BY a)
+ )
+ SELECT y FROM z
+ """,
+ [(None, None, "b"), (None, None, "z")],
+ ],
+ [
+ """
+ SELECT DISTINCT
+ FIRST_VALUE(x IGNORE NULLS) OVER (PARTITION BY y)
+ FROM z
+ """,
+ [(None, None, "z")],
+ ],
+ [
+ """
+ SELECT a . b . c
+ FROM d
+ """,
+ [(None, None, "d")],
+ ],
+ [
+ """
+ WITH a AS (
+ SELECT b FROM c
+ UNION ALL
+ (
+ WITH d AS (
+ SELECT e FROM f
+ )
+ SELECT g FROM d
+ )
+ )
+ SELECT h FROM a
+ """,
+ [(None, None, "c"), (None, None, "f")],
+ ],
+ [
+ """
+ WITH a AS (
+ SELECT b FROM c
+ UNION ALL
+ (
+ WITH d AS (
+ SELECT e FROM f
+ )
+ SELECT g FROM d
+ )
+ )
+ (SELECT h FROM a)
+ """,
+ [(None, None, "c"), (None, None, "f")],
+ ],
+ [
+ """
+ SELECT * FROM a.b.`c`
+ """,
+ [("a", "b", "c")],
+ ],
+ [
+ """
+ SELECT * FROM 'a'.b.`c`
+ """,
+ [("a", "b", "c")],
+ ],
+ ]
+
+ parser = BigQueryViewParser()
+ for test_index, test_case in enumerate(TEST_CASES):
+ sql, expected = test_case
+ parser.test(sql_stmt=sql, expected_tables=expected, verbose=True)
--- /dev/null
+"""
+Boolean Search query parser (Based on searchparser: https://github.com/pyparsing/pyparsing/blob/master/examples/searchparser.py)
+
+version 2018-07-22
+
+This search query parser uses the excellent Pyparsing module
+(http://pyparsing.sourceforge.net/) to parse search queries by users.
+It handles:
+
+* 'and', 'or' and implicit 'and' operators;
+* parentheses;
+* quoted strings;
+* wildcards at the end of a search term (help*);
+* wildcards at the beginning of a search term (*lp);
+* non-western languages
+
+Requirements:
+* Python
+* Pyparsing
+
+SAMPLE USAGE:
+from booleansearchparser import BooleanSearchParser
+from __future__ import print_function
+bsp = BooleanSearchParser()
+text = u"wildcards at the beginning of a search term "
+exprs= [
+ u"*cards and term", #True
+ u"wild* and term", #True
+ u"not terms", #True
+ u"terms or begin", #False
+]
+for expr in exprs:
+ print (bsp.match(text,expr))
+
+#non-western samples
+text = u"안녕하세요, 당신은 어떠세요?"
+exprs= [
+ u"*신은 and 어떠세요", #True
+ u"not 당신은", #False
+ u"당신 or 당", #False
+]
+for expr in exprs:
+ print (bsp.match(text,expr))
+-------------------------------------------------------------------------------
+Copyright (c) 2006, Estrate, the Netherlands
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+* Neither the name of Estrate nor the names of its contributors may be used
+ to endorse or promote products derived from this software without specific
+ prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+CONTRIBUTORS:
+- Steven Mooij
+- Rudolph Froger
+- Paul McGuire
+- Guiem Bosch
+- Francesc Garcia
+
+TODO:
+- add more docs
+- ask someone to check my English texts
+- add more kinds of wildcards ('*' at the beginning and '*' inside a word)?
+
+"""
+from pyparsing import (
+ Word,
+ alphanums,
+ Keyword,
+ Group,
+ Forward,
+ Suppress,
+ OneOrMore,
+ oneOf,
+)
+import re
+
+
+alphabet_ranges = [
+ ##CYRILIC: https://en.wikipedia.org/wiki/Cyrillic_(Unicode_block)
+ [int("0400", 16), int("04FF", 16)],
+ ##THAI: https://en.wikipedia.org/wiki/Thai_(Unicode_block)
+ [int("0E00", 16), int("0E7F", 16)],
+ ##ARABIC: https://en.wikipedia.org/wiki/Arabic_(Unicode_block) (Arabic (0600–06FF)+ Syriac (0700–074F)+ Arabic Supplement (0750–077F) )
+ [int("0600", 16), int("07FF", 16)],
+ ##CHINESE: https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
+ [int("0400", 16), int("09FF", 16)],
+ # JAPANESE : https://en.wikipedia.org/wiki/Japanese_writing_system
+ [int("3040", 16), int("30FF", 16)],
+ # KOREAN : https://en.wikipedia.org/wiki/Hangul
+ [int("AC00", 16), int("D7AF", 16)],
+ [int("1100", 16), int("11FF", 16)],
+ [int("3130", 16), int("318F", 16)],
+ [int("3200", 16), int("32FF", 16)],
+ [int("A960", 16), int("A97F", 16)],
+ [int("D7B0", 16), int("D7FF", 16)],
+ [int("FF00", 16), int("FFEF", 16)],
+]
+
+
+class BooleanSearchParser:
+ def __init__(self, only_parse=False):
+ self._methods = {
+ "and": self.evaluateAnd,
+ "or": self.evaluateOr,
+ "not": self.evaluateNot,
+ "parenthesis": self.evaluateParenthesis,
+ "quotes": self.evaluateQuotes,
+ "word": self.evaluateWord,
+ "wordwildcardprefix": self.evaluateWordWildcardPrefix,
+ "wordwildcardsufix": self.evaluateWordWildcardSufix,
+ }
+ self._parser = self.parser()
+ self.text = ""
+ self.words = []
+
+ def parser(self):
+ """
+ This function returns a parser.
+ The grammar should be like most full text search engines (Google, Tsearch, Lucene).
+
+ Grammar:
+ - a query consists of alphanumeric words, with an optional '*'
+ wildcard at the end or the beginning of a word
+ - a sequence of words between quotes is a literal string
+ - words can be used together by using operators ('and' or 'or')
+ - words with operators can be grouped with parenthesis
+ - a word or group of words can be preceded by a 'not' operator
+ - the 'and' operator precedes an 'or' operator
+ - if an operator is missing, use an 'and' operator
+ """
+ operatorOr = Forward()
+
+ alphabet = alphanums
+
+ # support for non-western alphabets
+ for r in alphabet_ranges:
+ alphabet += "".join(chr(c) for c in range(*r) if not chr(c).isspace())
+
+ operatorWord = Group(Word(alphabet + "*")).setResultsName("word*")
+
+ operatorQuotesContent = Forward()
+ operatorQuotesContent << ((operatorWord + operatorQuotesContent) | operatorWord)
+
+ operatorQuotes = (
+ Group(Suppress('"') + operatorQuotesContent + Suppress('"')).setResultsName(
+ "quotes"
+ )
+ | operatorWord
+ )
+
+ operatorParenthesis = (
+ Group(Suppress("(") + operatorOr + Suppress(")")).setResultsName(
+ "parenthesis"
+ )
+ | operatorQuotes
+ )
+
+ operatorNot = Forward()
+ operatorNot << (
+ Group(Suppress(Keyword("not", caseless=True)) + operatorNot).setResultsName(
+ "not"
+ )
+ | operatorParenthesis
+ )
+
+ operatorAnd = Forward()
+ operatorAnd << (
+ Group(
+ operatorNot + Suppress(Keyword("and", caseless=True)) + operatorAnd
+ ).setResultsName("and")
+ | Group(
+ operatorNot + OneOrMore(~oneOf("and or") + operatorAnd)
+ ).setResultsName("and")
+ | operatorNot
+ )
+
+ operatorOr << (
+ Group(
+ operatorAnd + Suppress(Keyword("or", caseless=True)) + operatorOr
+ ).setResultsName("or")
+ | operatorAnd
+ )
+
+ return operatorOr.parseString
+
+ def evaluateAnd(self, argument):
+ return all(self.evaluate(arg) for arg in argument)
+
+ def evaluateOr(self, argument):
+ return any(self.evaluate(arg) for arg in argument)
+
+ def evaluateNot(self, argument):
+ return self.GetNot(self.evaluate(argument[0]))
+
+ def evaluateParenthesis(self, argument):
+ return self.evaluate(argument[0])
+
+ def evaluateQuotes(self, argument):
+ """Evaluate quoted strings
+
+ First is does an 'and' on the indidual search terms, then it asks the
+ function GetQuoted to only return the subset of ID's that contain the
+ literal string.
+ """
+ # r = set()
+ r = False
+ search_terms = []
+ for item in argument:
+ search_terms.append(item[0])
+ r = r and self.evaluate(item)
+ return self.GetQuotes(" ".join(search_terms), r)
+
+ def evaluateWord(self, argument):
+ wildcard_count = argument[0].count("*")
+ if wildcard_count > 0:
+ if wildcard_count == 1 and argument[0].startswith("*"):
+ return self.GetWordWildcard(argument[0][1:], method="endswith")
+ if wildcard_count == 1 and argument[0].endswith("*"):
+ return self.GetWordWildcard(argument[0][:-1], method="startswith")
+ else:
+ _regex = argument[0].replace("*", ".+")
+ matched = False
+ for w in self.words:
+ matched = bool(re.search(_regex, w))
+ if matched:
+ break
+ return matched
+
+ return self.GetWord(argument[0])
+
+ def evaluateWordWildcardPrefix(self, argument):
+ return self.GetWordWildcard(argument[0], method="endswith")
+
+ def evaluateWordWildcardSufix(self, argument):
+ return self.GetWordWildcard(argument[0], method="startswith")
+
+ def evaluate(self, argument):
+ return self._methods[argument.getName()](argument)
+
+ def Parse(self, query):
+ return self.evaluate(self._parser(query)[0])
+
+ def GetWord(self, word):
+ return word in self.words
+
+ def GetWordWildcard(self, word, method="startswith"):
+ matched = False
+ for w in self.words:
+ matched = getattr(w, method)(word)
+ if matched:
+ break
+ return matched
+
+ """
+ def GetKeyword(self, name, value):
+ return set()
+
+ def GetBetween(self, min, max):
+ print (min,max)
+ return set()
+ """
+
+ def GetQuotes(self, search_string, tmp_result):
+ return search_string in self.text
+
+ def GetNot(self, not_set):
+ return not not_set
+
+ def _split_words(self, text):
+ words = []
+ """
+ >>> import string
+ >>> string.punctuation
+ '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'
+ """
+ # it will keep @, # and
+ # usernames and hashtags can contain dots, so a double check is done
+ r = re.compile(r"[\s{}]+".format(re.escape("!\"$%&'()*+,-/:;<=>?[\\]^`{|}~")))
+ _words = r.split(text)
+ for _w in _words:
+ if "." in _w and not _w.startswith("#") and not _w.startswith("@"):
+ for __w in _w.split("."):
+ words.append(__w)
+ continue
+
+ words.append(_w)
+
+ return words
+
+ def match(self, text, expr):
+ self.text = text
+ self.words = self._split_words(text)
+
+ return self.Parse(expr)
+
+
+class ParserTest(BooleanSearchParser):
+ """Tests the parser with some search queries
+ tests contains a dictionary with tests and expected results.
+ """
+
+ def Test(self):
+ exprs = {
+ "0": "help",
+ "1": "help or hulp",
+ "2": "help and hulp",
+ "3": "help hulp",
+ "4": "help and hulp or hilp",
+ "5": "help or hulp and hilp",
+ "6": "help or hulp or hilp or halp",
+ "7": "(help or hulp) and (hilp or halp)",
+ "8": "help and (hilp or halp)",
+ "9": "(help and (hilp or halp)) or hulp",
+ "10": "not help",
+ "11": "not hulp and halp",
+ "12": "not (help and halp)",
+ "13": '"help me please"',
+ "14": '"help me please" or hulp',
+ "15": '"help me please" or (hulp and halp)',
+ "16": "help*",
+ "17": "help or hulp*",
+ "18": "help* and hulp",
+ "19": "help and hulp* or hilp",
+ "20": "help* or hulp or hilp or halp",
+ "21": "(help or hulp*) and (hilp* or halp)",
+ "22": "help* and (hilp* or halp*)",
+ "23": "(help and (hilp* or halp)) or hulp*",
+ "24": "not help* and halp",
+ "25": "not (help* and helpe*)",
+ "26": '"help* me please"',
+ "27": '"help* me* please" or hulp*',
+ "28": '"help me please*" or (hulp and halp)',
+ "29": '"help me please" not (hulp and halp)',
+ "30": '"help me please" hulp',
+ "31": "help and hilp and not holp",
+ "32": "help hilp not holp",
+ "33": "help hilp and not holp",
+ "34": "*lp and halp",
+ "35": "*신은 and 어떠세요",
+ }
+
+ texts_matcheswith = {
+ "halp thinks he needs help": [
+ "25",
+ "22",
+ "20",
+ "21",
+ "11",
+ "17",
+ "16",
+ "23",
+ "34",
+ "1",
+ "0",
+ "5",
+ "7",
+ "6",
+ "9",
+ "8",
+ ],
+ "he needs halp": ["24", "25", "20", "11", "10", "12", "34", "6"],
+ "help": ["25", "20", "12", "17", "16", "1", "0", "5", "6"],
+ "help hilp": [
+ "25",
+ "22",
+ "20",
+ "32",
+ "21",
+ "12",
+ "17",
+ "16",
+ "19",
+ "31",
+ "23",
+ "1",
+ "0",
+ "5",
+ "4",
+ "7",
+ "6",
+ "9",
+ "8",
+ "33",
+ ],
+ "help me please hulp": [
+ "30",
+ "25",
+ "27",
+ "20",
+ "13",
+ "12",
+ "15",
+ "14",
+ "17",
+ "16",
+ "19",
+ "18",
+ "23",
+ "29",
+ "1",
+ "0",
+ "3",
+ "2",
+ "5",
+ "4",
+ "6",
+ "9",
+ ],
+ "helper": ["20", "10", "12", "16"],
+ "hulp hilp": [
+ "25",
+ "27",
+ "20",
+ "21",
+ "10",
+ "12",
+ "14",
+ "17",
+ "19",
+ "23",
+ "1",
+ "5",
+ "4",
+ "7",
+ "6",
+ "9",
+ ],
+ "nothing": ["25", "10", "12"],
+ "안녕하세요, 당신은 어떠세요?": ["10", "12", "25", "35"],
+ }
+
+ all_ok = True
+ for text, matches in texts_matcheswith.items():
+ _matches = []
+ for _id, expr in exprs.items():
+ if self.match(text, expr):
+ _matches.append(_id)
+
+ test_passed = sorted(matches) == sorted(_matches)
+ if not test_passed:
+ print("Failed", repr(text), "expected", matches, "matched", _matches)
+
+ all_ok = all_ok and test_passed
+
+ return all_ok
+
+
+if __name__ == "__main__":
+ if ParserTest().Test():
+ print("All tests OK")
+ exit(0)
+ else:
+ print("One or more tests FAILED")
+ exit(1)
Simplified BSD license\r
"""\r
\r
-from pyparsing import (Regex, Suppress, ZeroOrMore, Group, Optional, Forward,\r
- SkipTo, CaselessLiteral, Dict)\r
+from pyparsing import (\r
+ Regex,\r
+ Suppress,\r
+ ZeroOrMore,\r
+ Group,\r
+ Optional,\r
+ Forward,\r
+ SkipTo,\r
+ CaselessLiteral,\r
+ Dict,\r
+)\r
+\r
+\r
+class Macro:\r
+ """Class to encapsulate undefined macro references"""\r
\r
-\r
-class Macro(object):\r
- """ Class to encapsulate undefined macro references """\r
def __init__(self, name):\r
self.name = name\r
+\r
def __repr__(self):\r
return 'Macro("%s")' % self.name\r
+\r
def __eq__(self, other):\r
return self.name == other.name\r
- def __ne__(self, other):\r
- return self.name != other.name\r
\r
\r
# Character literals\r
-LCURLY,RCURLY,LPAREN,RPAREN,QUOTE,COMMA,AT,EQUALS,HASH = map(Suppress,'{}()",@=#')\r
+LCURLY, RCURLY, LPAREN, RPAREN, QUOTE, COMMA, AT, EQUALS, HASH = map(\r
+ Suppress, '{}()",@=#'\r
+)\r
\r
\r
def bracketed(expr):\r
- """ Return matcher for `expr` between curly brackets or parentheses """\r
+ """Return matcher for `expr` between curly brackets or parentheses"""\r
return (LPAREN + expr + RPAREN) | (LCURLY + expr + RCURLY)\r
\r
\r
quoted_string = QUOTE + ZeroOrMore(quoted_item) + QUOTE\r
\r
# Numbers can just be numbers. Only integers though.\r
-number = Regex('[0-9]+')\r
+number = Regex("[0-9]+")\r
\r
# Basis characters (by exclusion) for variable / field names. The following\r
# list of characters is from the btparse documentation\r
-any_name = Regex('[^\\s"#%\'(),={}]+')\r
+any_name = Regex("[^\\s\"#%'(),={}]+")\r
\r
# btparse says, and the test bibs show by experiment, that macro and field names\r
# cannot start with a digit. In fact entry type names cannot start with a digit\r
# either (see tests/bibs). Cite keys can start with a digit\r
-not_digname = Regex('[^\\d\\s"#%\'(),={}][^\\s"#%\'(),={}]*')\r
+not_digname = Regex("[^\\d\\s\"#%'(),={}][^\\s\"#%'(),={}]*")\r
\r
# Comment comments out to end of line\r
-comment = (AT + CaselessLiteral('comment') +\r
- Regex(r"[\s{(].*").leaveWhitespace())\r
+comment = AT + CaselessLiteral("comment") + Regex(r"[\s{(].*").leaveWhitespace()\r
\r
# The name types with their digiteyness\r
not_dig_lower = not_digname.copy().setParseAction(lambda t: t[0].lower())\r
macro_def = not_dig_lower.copy()\r
-macro_ref = not_dig_lower.copy().setParseAction(lambda t : Macro(t[0].lower()))\r
+macro_ref = not_dig_lower.copy().setParseAction(lambda t: Macro(t[0].lower()))\r
field_name = not_dig_lower.copy()\r
# Spaces in names mean they cannot clash with field names\r
-entry_type = not_dig_lower('entry_type')\r
-cite_key = any_name('cite_key')\r
+entry_type = not_dig_lower("entry_type")\r
+cite_key = any_name("cite_key")\r
# Number has to be before macro name\r
-string = (number | macro_ref | quoted_string | curly_string)\r
+string = number | macro_ref | quoted_string | curly_string\r
\r
# There can be hash concatenation\r
field_value = string + ZeroOrMore(HASH + string)\r
entry_contents = Dict(ZeroOrMore(field_def + COMMA) + Optional(field_def))\r
\r
# Entry is surrounded either by parentheses or curlies\r
-entry = (AT + entry_type + bracketed(cite_key + COMMA + entry_contents))\r
+entry = AT + entry_type + bracketed(cite_key + COMMA + entry_contents)\r
\r
# Preamble is a macro-like thing with no name\r
-preamble = AT + CaselessLiteral('preamble') + bracketed(field_value)\r
+preamble = AT + CaselessLiteral("preamble") + bracketed(field_value)\r
\r
# Macros (aka strings)\r
macro_contents = macro_def + EQUALS + field_value\r
-macro = AT + CaselessLiteral('string') + bracketed(macro_contents)\r
+macro = AT + CaselessLiteral("string") + bracketed(macro_contents)\r
\r
# Implicit comments\r
-icomment = SkipTo('@').setParseAction(lambda t : t.insert(0, 'icomment'))\r
+icomment = SkipTo("@").setParseAction(lambda t: t.insert(0, "icomment"))\r
\r
# entries are last in the list (other than the fallback) because they have\r
# arbitrary start patterns that would match comments, preamble or macro\r
-definitions = Group(comment |\r
- preamble |\r
- macro |\r
- entry |\r
- icomment)\r
+definitions = Group(comment | preamble | macro | entry | icomment)\r
\r
# Start symbol\r
bibfile = ZeroOrMore(definitions)\r
return bibfile.parseString(str)\r
\r
\r
-if __name__ == '__main__':\r
+if __name__ == "__main__":\r
# Run basic test\r
txt = """\r
Some introductory text\r
number = {2}\r
}\r
"""\r
- print('\n\n'.join(defn.dump() for defn in parse_str(txt)))\r
+ print("\n\n".join(defn.dump() for defn in parse_str(txt)))\r
\r
from pyparsing import *\r
\r
-integer = Word(nums).setParseAction(lambda t : int(t[0]))\r
+integer = Word(nums).setParseAction(lambda t: int(t[0]))\r
\r
# make an expression that will match a list of ints (which\r
# will be converted to actual ints by the parse action attached\r
fn_name = fn.__name__\r
if fn is reversed:\r
# reversed returns an iterator, we really want to show the list of items\r
- fn = lambda x : list(reversed(x))\r
+ fn = lambda x: list(reversed(x))\r
\r
# show how each builtin works as a free-standing parse action\r
print(fn_name, nums.setParseAction(fn).parseString(test))\r
# Copyright, 2012 - Paul McGuire\r
#\r
\r
-from pyparsing import Word, alphas, alphanums, Combine, oneOf, Optional, delimitedList, Group, Keyword\r
+from pyparsing import (\r
+ Word,\r
+ alphas,\r
+ alphanums,\r
+ Combine,\r
+ oneOf,\r
+ Optional,\r
+ delimitedList,\r
+ Group,\r
+ Keyword,\r
+)\r
\r
testdata = """\r
int func1(float *vec, int len, double arg1);\r
"""\r
\r
ident = Word(alphas, alphanums + "_")\r
-vartype = Combine( oneOf("float double int char") + Optional(Word("*")), adjacent = False)\r
+vartype = Combine(oneOf("float double int char") + Optional(Word("*")), adjacent=False)\r
arglist = delimitedList(Group(vartype("type") + ident("name")))\r
\r
functionCall = Keyword("int") + ident("name") + "(" + arglist("args") + ")" + ";"\r
\r
-for fn,s,e in functionCall.scanString(testdata):\r
+for fn, s, e in functionCall.scanString(testdata):\r
print(fn.name)\r
for a in fn.args:\r
print(" - %(name)s (%(type)s)" % a)\r
-# -*- coding: utf-8 -*-\r
#\r
# chemicalFormulas.py\r
#\r
import pyparsing as pp\r
\r
atomicWeight = {\r
- "O" : 15.9994,\r
- "H" : 1.00794,\r
- "Na" : 22.9897,\r
- "Cl" : 35.4527,\r
- "C" : 12.0107,\r
- }\r
+ "O": 15.9994,\r
+ "H": 1.00794,\r
+ "Na": 22.9897,\r
+ "Cl": 35.4527,\r
+ "C": 12.0107,\r
+}\r
\r
digits = "0123456789"\r
\r
# Version 1\r
-element = pp.Word(pp.alphas.upper(), pp.alphas.lower(), max=2)\r
+element = pp.Word(pp.alphas.upper(), pp.alphas.lower(), max=2).set_name("element")\r
# for stricter matching, use this Regex instead\r
# element = Regex("A[cglmrstu]|B[aehikr]?|C[adeflmorsu]?|D[bsy]|"\r
# "E[rsu]|F[emr]?|G[ade]|H[efgos]?|I[nr]?|Kr?|L[airu]|"\r
elementRef = pp.Group(element + pp.Optional(pp.Word(digits), default="1"))\r
formula = elementRef[...]\r
\r
-fn = lambda elemList : sum(atomicWeight[elem]*int(qty) for elem,qty in elemList)\r
-formula.runTests("""\\r
+\r
+def sum_atomic_weights(element_list):\r
+ return sum(atomicWeight[elem] * int(qty) for elem, qty in element_list)\r
+\r
+\r
+formula.runTests(\r
+ """\\r
H2O\r
C6H5OH\r
NaCl\r
""",\r
- fullDump=False, postParse=lambda _, tokens: "Molecular weight: {0}".format(fn(tokens)))\r
+ fullDump=False,\r
+ postParse=lambda _, tokens: "Molecular weight: {}".format(\r
+ sum_atomic_weights(tokens)\r
+ ),\r
+)\r
print()\r
\r
# Version 2 - access parsed items by results name\r
-elementRef = pp.Group(element("symbol") + pp.Optional(pp.Word(digits), default="1")("qty"))\r
+elementRef = pp.Group(\r
+ element("symbol") + pp.Optional(pp.Word(digits), default="1")("qty")\r
+)\r
formula = elementRef[...]\r
\r
-fn = lambda elemList : sum(atomicWeight[elem.symbol]*int(elem.qty) for elem in elemList)\r
-formula.runTests("""\\r
+\r
+def sum_atomic_weights_by_results_name(element_list):\r
+ return sum(atomicWeight[elem.symbol] * int(elem.qty) for elem in element_list)\r
+\r
+\r
+formula.runTests(\r
+ """\\r
H2O\r
C6H5OH\r
NaCl\r
""",\r
- fullDump=False, postParse=lambda _, tokens: "Molecular weight: {0}".format(fn(tokens)))\r
+ fullDump=False,\r
+ postParse=lambda _, tokens: "Molecular weight: {}".format(\r
+ sum_atomic_weights_by_results_name(tokens)\r
+ ),\r
+)\r
print()\r
\r
# Version 3 - convert integers during parsing process\r
-integer = pp.Word(digits).setParseAction(lambda t:int(t[0]))\r
+integer = pp.Word(digits).setParseAction(lambda t: int(t[0])).setName("integer")\r
elementRef = pp.Group(element("symbol") + pp.Optional(integer, default=1)("qty"))\r
-formula = elementRef[...]\r
+formula = elementRef[...].setName("chemical_formula")\r
+\r
\r
-fn = lambda elemList : sum(atomicWeight[elem.symbol]*elem.qty for elem in elemList)\r
-formula.runTests("""\\r
+def sum_atomic_weights_by_results_name_with_converted_ints(element_list):\r
+ return sum(atomicWeight[elem.symbol] * int(elem.qty) for elem in element_list)\r
+\r
+\r
+formula.runTests(\r
+ """\\r
H2O\r
C6H5OH\r
NaCl\r
""",\r
- fullDump=False, postParse=lambda _, tokens: "Molecular weight: {0}".format(fn(tokens)))\r
+ fullDump=False,\r
+ postParse=lambda _, tokens: "Molecular weight: {}".format(\r
+ sum_atomic_weights_by_results_name_with_converted_ints(tokens)\r
+ ),\r
+)\r
print()\r
\r
# Version 4 - parse and convert integers as subscript digits\r
subscript_digits = "₀₁₂₃₄₅₆₇₈₉"\r
-subscript_int_map = dict((e[1], e[0]) for e in enumerate(subscript_digits))\r
+subscript_int_map = {e[1]: e[0] for e in enumerate(subscript_digits)}\r
+\r
+\r
def cvt_subscript_int(s):\r
ret = 0\r
for c in s[0]:\r
- ret = ret*10 + subscript_int_map[c]\r
+ ret = ret * 10 + subscript_int_map[c]\r
return ret\r
-subscript_int = pp.Word(subscript_digits).addParseAction(cvt_subscript_int)\r
+\r
+\r
+subscript_int = pp.Word(subscript_digits).addParseAction(cvt_subscript_int).set_name("subscript")\r
\r
elementRef = pp.Group(element("symbol") + pp.Optional(subscript_int, default=1)("qty"))\r
-formula = elementRef[...]\r
-formula.runTests("""\\r
+formula = elementRef[1, ...].setName("chemical_formula")\r
+formula.runTests(\r
+ """\\r
H₂O\r
C₆H₅OH\r
NaCl\r
""",\r
- fullDump=False, postParse=lambda _, tokens: "Molecular weight: {0}".format(fn(tokens)))\r
+ fullDump=False,\r
+ postParse=lambda _, tokens: "Molecular weight: {}".format(\r
+ sum_atomic_weights_by_results_name_with_converted_ints(tokens)\r
+ ),\r
+)\r
print()\r
#
import pyparsing as pp
+
ppc = pp.pyparsing_common
testData = [
"John Doe, 123 Main St., Cleveland, Ohio",
"Jane Doe, 456 St. James St., Los Angeles , California ",
"",
- ]
+]
ppc.comma_separated_list.runTests(testData)
# Copyright (c) 2003, Paul McGuire\r
#\r
\r
-from pyparsing import \\r
- Literal, Word, ZeroOrMore, Group, Dict, Optional, \\r
- printables, ParseException, restOfLine, empty\r
+from pyparsing import (\r
+ Literal,\r
+ Word,\r
+ ZeroOrMore,\r
+ Group,\r
+ Dict,\r
+ Optional,\r
+ printables,\r
+ ParseException,\r
+ restOfLine,\r
+ empty,\r
+)\r
import pprint\r
\r
\r
inibnf = None\r
+\r
+\r
def inifile_BNF():\r
global inibnf\r
\r
lbrack = Literal("[").suppress()\r
rbrack = Literal("]").suppress()\r
equals = Literal("=").suppress()\r
- semi = Literal(";")\r
+ semi = Literal(";")\r
\r
- comment = semi + Optional( restOfLine )\r
+ comment = semi + Optional(restOfLine)\r
\r
- nonrbrack = "".join( [ c for c in printables if c != "]" ] ) + " \t"\r
- nonequals = "".join( [ c for c in printables if c != "=" ] ) + " \t"\r
+ nonrbrack = "".join([c for c in printables if c != "]"]) + " \t"\r
+ nonequals = "".join([c for c in printables if c != "="]) + " \t"\r
\r
- sectionDef = lbrack + Word( nonrbrack ) + rbrack\r
- keyDef = ~lbrack + Word( nonequals ) + equals + empty + restOfLine\r
+ sectionDef = lbrack + Word(nonrbrack) + rbrack\r
+ keyDef = ~lbrack + Word(nonequals) + equals + empty + restOfLine\r
# strip any leading or trailing blanks from key\r
def stripKey(tokens):\r
tokens[0] = tokens[0].strip()\r
+\r
keyDef.setParseAction(stripKey)\r
\r
# using Dict will allow retrieval of named data fields as attributes of the parsed results\r
- inibnf = Dict( ZeroOrMore( Group( sectionDef + Dict( ZeroOrMore( Group( keyDef ) ) ) ) ) )\r
+ inibnf = Dict(ZeroOrMore(Group(sectionDef + Dict(ZeroOrMore(Group(keyDef))))))\r
\r
- inibnf.ignore( comment )\r
+ inibnf.ignore(comment)\r
\r
return inibnf\r
\r
\r
pp = pprint.PrettyPrinter(2)\r
\r
-def test( strng ):\r
+\r
+def test(strng):\r
print(strng)\r
try:\r
iniFile = open(strng)\r
- iniData = "".join( iniFile.readlines() )\r
+ iniData = "".join(iniFile.readlines())\r
bnf = inifile_BNF()\r
- tokens = bnf.parseString( iniData )\r
- pp.pprint( tokens.asList() )\r
+ tokens = bnf.parseString(iniData)\r
+ pp.pprint(tokens.asList())\r
\r
except ParseException as err:\r
print(err.line)\r
- print(" "*(err.column-1) + "^")\r
+ print(" " * (err.column - 1) + "^")\r
print(err)\r
\r
iniFile.close()\r
print()\r
return tokens\r
\r
+\r
if __name__ == "__main__":\r
- ini = test("setup.ini")\r
- print("ini['Startup']['modemid'] =", ini['Startup']['modemid'])\r
- print("ini.Startup =", ini.Startup)\r
- print("ini.Startup.modemid =", ini.Startup.modemid)\r
+ ini = test("setup.ini")\r
+ print("ini['Startup']['modemid'] =", ini["Startup"]["modemid"])\r
+ print("ini.Startup =", ini.Startup)\r
+ print("ini.Startup.modemid =", ini.Startup.modemid)\r
#\r
\r
from pyparsing import *\r
+\r
# sample string with enums and other stuff\r
-sample = '''\r
+sample = """\r
stuff before\r
enum hello {\r
Zero,\r
zeta = 50\r
};\r
at the end\r
- '''\r
+ """\r
\r
# syntax we don't want to see in the final parse tree\r
-LBRACE,RBRACE,EQ,COMMA = map(Suppress,"{}=,")\r
-_enum = Suppress('enum')\r
-identifier = Word(alphas,alphanums+'_')\r
+LBRACE, RBRACE, EQ, COMMA = map(Suppress, "{}=,")\r
+_enum = Suppress("enum")\r
+identifier = Word(alphas, alphanums + "_")\r
integer = Word(nums)\r
-enumValue = Group(identifier('name') + Optional(EQ + integer('value')))\r
+enumValue = Group(identifier("name") + Optional(EQ + integer("value")))\r
enumList = Group(enumValue + ZeroOrMore(COMMA + enumValue))\r
-enum = _enum + identifier('enum') + LBRACE + enumList('names') + RBRACE\r
+enum = _enum + identifier("enum") + LBRACE + enumList("names") + RBRACE\r
\r
# find instances of enums ignoring other syntax\r
-for item,start,stop in enum.scanString(sample):\r
+for item, start, stop in enum.scanString(sample):\r
id = 0\r
for entry in item.names:\r
- if entry.value != '':\r
+ if entry.value != "":\r
id = int(entry.value)\r
- print('%s_%s = %d' % (item.enum.upper(),entry.name.upper(),id))\r
+ print("%s_%s = %d" % (item.enum.upper(), entry.name.upper(), id))\r
id += 1\r
--- /dev/null
+#
+# cuneiform_python.py
+#
+# Example showing how to create a custom Unicode set for parsing
+#
+# Copyright Paul McGuire, 2021
+#
+from typing import List, Tuple
+import pyparsing as pp
+
+
+class Cuneiform(pp.unicode_set):
+ """Unicode set for Cuneiform Character Range"""
+
+ _ranges: List[Tuple[int, ...]] = [
+ (0x10380, 0x103d5),
+ (0x12000, 0x123FF),
+ (0x12400, 0x1247F),
+ ]
+
+
+# list out all valid identifier characters
+# print(Cuneiform.identchars)
+
+
+"""
+Simple Cuneiform Python language transformer
+
+Define Cuneiform "words"
+ print: 𒄑𒉿𒅔𒋫
+ hello: 𒀄𒂖𒆷𒁎
+ world: 𒍟𒁎𒉿𒆷𒀳
+ def: 𒁴𒈫
+"""
+
+# uncomment to show parse-time debugging
+# pp.enable_diag(pp.Diagnostics.enable_debug_on_named_expressions)
+
+# define a MINIMAL Python parser
+LPAR, RPAR, COLON, EQ = map(pp.Suppress, "():=")
+def_ = pp.Keyword("𒁴𒈫", ident_chars=Cuneiform.identbodychars).set_name("def")
+any_keyword = def_
+ident = (~any_keyword) + pp.Word(
+ Cuneiform.identchars, Cuneiform.identbodychars, asKeyword=True
+)
+str_expr = pp.infix_notation(
+ pp.QuotedString('"') | pp.common.integer,
+ [
+ ("*", 2, pp.OpAssoc.LEFT),
+ ("+", 2, pp.OpAssoc.LEFT),
+ ],
+)
+
+rvalue = pp.Forward()
+fn_call = (ident + pp.Group(LPAR + pp.Optional(rvalue) + RPAR)).set_name("fn_call")
+
+rvalue <<= fn_call | ident | str_expr | pp.common.number
+assignment_stmt = ident + EQ + rvalue
+
+stmt = pp.Group(fn_call | assignment_stmt).set_name("stmt")
+
+fn_def = pp.Group(
+ def_ + ident + pp.Group(LPAR + pp.Optional(rvalue) + RPAR) + COLON
+).set_name("fn_def")
+fn_body = pp.IndentedBlock(stmt).set_name("fn_body")
+fn_expr = pp.Group(fn_def + pp.Group(fn_body))
+
+script = fn_expr[...] + stmt[...]
+
+
+# parse some Python written in Cuneiform
+cuneiform_hello_world = r"""
+𒁴𒈫 𒀄𒂖𒆷𒁎():
+ 𒀁 = "𒀄𒂖𒆷𒁎, 𒍟𒁎𒉿𒆷𒀳!\n" * 3
+ 𒄑𒉿𒅔𒋫(𒀁)
+
+𒀄𒂖𒆷𒁎()"""
+script.parseString(cuneiform_hello_world).pprint(width=40)
+
+
+# use transform_string to convert keywords and builtins to runnable Python
+names_map = {
+ "𒄑𒉿𒅔𒋫": "print",
+}
+ident.add_parse_action(lambda t: names_map.get(t[0], t[0]))
+def_.add_parse_action(lambda: "def")
+
+print("\nconvert Cuneiform Python to executable Python")
+transformed = (
+ # always put ident last
+ (def_ | ident)
+ .ignore(pp.quoted_string)
+ .transform_string(cuneiform_hello_world)
+ .strip()
+)
+print(
+ "=================\n"
+ + cuneiform_hello_world.strip()
+ + "\n=================\n"
+ + transformed
+ + "\n=================\n"
+)
+print("# run transformed Python")
+exec(transformed)
# define an integer string, and a parse action to convert it\r
# to an integer at parse time\r
integer = pp.Word(pp.nums).setName("integer")\r
+\r
+\r
def convertToInt(tokens):\r
# no need to test for validity - we can't get here\r
# unless tokens[0] contains all numeric digits\r
return int(tokens[0])\r
+\r
+\r
integer.setParseAction(convertToInt)\r
# or can be written as one line as\r
-#integer = Word(nums).setParseAction(lambda t: int(t[0]))\r
+# integer = Word(nums).setParseAction(lambda t: int(t[0]))\r
\r
# define a pattern for a year/month/day date\r
-date_expr = integer('year') + '/' + integer('month') + '/' + integer('day')\r
+date_expr = integer("year") + "/" + integer("month") + "/" + integer("day")\r
date_expr.ignore(pp.pythonStyleComment)\r
\r
-def convertToDatetime(s,loc,tokens):\r
+\r
+def convertToDatetime(s, loc, tokens):\r
try:\r
# note that the year, month, and day fields were already\r
# converted to ints from strings by the parse action defined\r
# on the integer expression above\r
return datetime(tokens.year, tokens.month, tokens.day).date()\r
except Exception as ve:\r
- errmsg = "'%s/%s/%s' is not a valid date, %s" % \\r
- (tokens.year, tokens.month, tokens.day, ve)\r
+ errmsg = "'%s/%s/%s' is not a valid date, %s" % (\r
+ tokens.year,\r
+ tokens.month,\r
+ tokens.day,\r
+ ve,\r
+ )\r
raise pp.ParseException(s, loc, errmsg)\r
+\r
+\r
date_expr.setParseAction(convertToDatetime)\r
\r
\r
-date_expr.runTests("""\\r
+date_expr.runTests(\r
+ """\\r
2000/1/1\r
\r
# invalid month\r
\r
# but 2000 was\r
2000/2/29\r
- """)\r
+ """\r
+)\r
\r
\r
# if dates conform to ISO8601, use definitions in pyparsing_common\r
date_expr = ppc.iso8601_date.setParseAction(ppc.convertToDate())\r
date_expr.ignore(pp.pythonStyleComment)\r
\r
-date_expr.runTests("""\\r
+date_expr.runTests(\r
+ """\\r
2000-01-01\r
\r
# invalid month\r
\r
# but 2000 was\r
2000-02-29\r
- """)\r
+ """\r
+)\r
"""
Program ::= Decl+
Decl ::= VariableDecl | FunctionDecl | ClassDecl | InterfaceDecl
- VariableDecl ::= Variable ;
- Variable ::= Type ident
- Type ::= int | double | bool | string | ident | Type []
- FunctionDecl ::= Type ident ( Formals ) StmtBlock | void ident ( Formals ) StmtBlock
- Formals ::= Variable+, | e
- ClassDecl ::= class ident <extends ident> <implements ident + ,> { Field* }
- Field ::= VariableDecl | FunctionDecl
- InterfaceDecl ::= interface ident { Prototype* }
- Prototype ::= Type ident ( Formals ) ; | void ident ( Formals ) ;
- StmtBlock ::= { VariableDecl* Stmt* }
- Stmt ::= <Expr> ; | IfStmt | WhileStmt | ForStmt | BreakStmt | ReturnStmt | PrintStmt | StmtBlock
- IfStmt ::= if ( Expr ) Stmt <else Stmt>
- WhileStmt ::= while ( Expr ) Stmt
- ForStmt ::= for ( <Expr> ; Expr ; <Expr> ) Stmt
- ReturnStmt ::= return <Expr> ;
- BreakStmt ::= break ;
- PrintStmt ::= Print ( Expr+, ) ;
+ VariableDecl ::= Variable ;
+ Variable ::= Type ident
+ Type ::= int | double | bool | string | ident | Type []
+ FunctionDecl ::= Type ident ( Formals ) StmtBlock | void ident ( Formals ) StmtBlock
+ Formals ::= Variable+, | e
+ ClassDecl ::= class ident <extends ident> <implements ident + ,> { Field* }
+ Field ::= VariableDecl | FunctionDecl
+ InterfaceDecl ::= interface ident { Prototype* }
+ Prototype ::= Type ident ( Formals ) ; | void ident ( Formals ) ;
+ StmtBlock ::= { VariableDecl* Stmt* }
+ Stmt ::= <Expr> ; | IfStmt | WhileStmt | ForStmt | BreakStmt | ReturnStmt | PrintStmt | StmtBlock
+ IfStmt ::= if ( Expr ) Stmt <else Stmt>
+ WhileStmt ::= while ( Expr ) Stmt
+ ForStmt ::= for ( <Expr> ; Expr ; <Expr> ) Stmt
+ ReturnStmt ::= return <Expr> ;
+ BreakStmt ::= break ;
+ PrintStmt ::= Print ( Expr+, ) ;
Expr ::= LValue = Expr | Constant | LValue | this | Call
- | ( Expr )
- | Expr + Expr | Expr - Expr | Expr * Expr | Expr / Expr | Expr % Expr | - Expr
- | Expr < Expr | Expr <= Expr | Expr > Expr | Expr >= Expr | Expr == Expr | Expr != Expr
- | Expr && Expr | Expr || Expr | ! Expr
- | ReadInteger ( ) | ReadLine ( ) | new ident | NewArray ( Expr , Typev)
- LValue ::= ident | Expr . ident | Expr [ Expr ]
- Call ::= ident ( Actuals ) | Expr . ident ( Actuals )
- Actuals ::= Expr+, | e
+ | ( Expr )
+ | Expr + Expr | Expr - Expr | Expr * Expr | Expr / Expr | Expr % Expr | - Expr
+ | Expr < Expr | Expr <= Expr | Expr > Expr | Expr >= Expr | Expr == Expr | Expr != Expr
+ | Expr && Expr | Expr || Expr | ! Expr
+ | ReadInteger ( ) | ReadLine ( ) | new ident | NewArray ( Expr , Typev)
+ LValue ::= ident | Expr . ident | Expr [ Expr ]
+ Call ::= ident ( Actuals ) | Expr . ident ( Actuals )
+ Actuals ::= Expr+, | e
Constant ::= intConstant | doubleConstant | boolConstant | stringConstant | null
"""
import pyparsing as pp
from pyparsing import pyparsing_common as ppc
+
pp.ParserElement.enablePackrat()
# keywords
-keywords = (VOID, INT, DOUBLE, BOOL, STRING, CLASS, INTERFACE, NULL, THIS, EXTENDS, IMPLEMENTS, FOR, WHILE,
- IF, ELSE, RETURN, BREAK, NEW, NEWARRAY, PRINT, READINTEGER, READLINE, TRUE, FALSE) = map(pp.Keyword,
- """void int double bool string class interface null this extends implements or while
- if else return break new NewArray Print ReadInteger ReadLine true false""".split())
+keywords = (
+ VOID,
+ INT,
+ DOUBLE,
+ BOOL,
+ STRING,
+ CLASS,
+ INTERFACE,
+ NULL,
+ THIS,
+ EXTENDS,
+ IMPLEMENTS,
+ FOR,
+ WHILE,
+ IF,
+ ELSE,
+ RETURN,
+ BREAK,
+ NEW,
+ NEWARRAY,
+ PRINT,
+ READINTEGER,
+ READLINE,
+ TRUE,
+ FALSE,
+) = map(
+ pp.Keyword,
+ """void int double bool string class interface null this extends implements or while
+ if else return break new NewArray Print ReadInteger ReadLine true false""".split(),
+)
keywords = pp.MatchFirst(list(keywords))
-LPAR, RPAR, LBRACE, RBRACE, LBRACK, RBRACK, DOT, EQ, COMMA, SEMI = map(pp.Suppress, "(){}[].=,;")
+LPAR, RPAR, LBRACE, RBRACE, LBRACK, RBRACK, DOT, EQ, COMMA, SEMI = map(
+ pp.Suppress, "(){}[].=,;"
+)
hexConstant = pp.Regex(r"0[xX][0-9a-fA-F]+").addParseAction(lambda t: int(t[0][2:], 16))
intConstant = hexConstant | ppc.integer
doubleConstant = ppc.real
stringConstant = pp.dblQuotedString
null = NULL
constant = doubleConstant | boolConstant | intConstant | stringConstant | null
-ident = ~keywords + pp.Word(pp.alphas, pp.alphanums+'_')
+ident = ~keywords + pp.Word(pp.alphas, pp.alphanums + "_")
type_ = pp.Group((INT | DOUBLE | BOOL | STRING | ident) + pp.ZeroOrMore("[]"))
variable = type_ + ident
expr = pp.Forward()
expr_parens = pp.Group(LPAR + expr + RPAR)
actuals = pp.Optional(pp.delimitedList(expr))
-call = pp.Group(ident("call_ident") + LPAR + actuals("call_args") + RPAR
- | (expr_parens + pp.ZeroOrMore(DOT + ident))("call_ident_expr") + LPAR + actuals("call_args") + RPAR)
-lvalue = ((ident | expr_parens)
- + pp.ZeroOrMore(DOT + (ident | expr_parens))
- + pp.ZeroOrMore(LBRACK + expr + RBRACK))
+call = pp.Group(
+ ident("call_ident") + LPAR + actuals("call_args") + RPAR
+ | (expr_parens + pp.ZeroOrMore(DOT + ident))("call_ident_expr")
+ + LPAR
+ + actuals("call_args")
+ + RPAR
+)
+lvalue = (
+ (ident | expr_parens)
+ + pp.ZeroOrMore(DOT + (ident | expr_parens))
+ + pp.ZeroOrMore(LBRACK + expr + RBRACK)
+)
assignment = pp.Group(lvalue("lhs") + EQ + expr("rhs"))
read_integer = pp.Group(READINTEGER + LPAR + RPAR)
read_line = pp.Group(READLINE + LPAR + RPAR)
new_statement = pp.Group(NEW + ident)
new_array = pp.Group(NEWARRAY + LPAR + expr + COMMA + type_ + RPAR)
rvalue = constant | call | read_integer | read_line | new_statement | new_array | ident
-arith_expr = pp.infixNotation(rvalue,
+arith_expr = pp.infixNotation(
+ rvalue,
[
- ('-', 1, pp.opAssoc.RIGHT,),
- (pp.oneOf("* / %"), 2, pp.opAssoc.LEFT,),
- (pp.oneOf("+ -"), 2, pp.opAssoc.LEFT,),
- ])
-comparison_expr = pp.infixNotation(arith_expr,
+ (
+ "-",
+ 1,
+ pp.opAssoc.RIGHT,
+ ),
+ (
+ pp.oneOf("* / %"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("+ -"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ ],
+)
+comparison_expr = pp.infixNotation(
+ arith_expr,
[
- ('!', 1, pp.opAssoc.RIGHT,),
- (pp.oneOf("< > <= >="), 2, pp.opAssoc.LEFT,),
- (pp.oneOf("== !="), 2, pp.opAssoc.LEFT,),
- (pp.oneOf("&&"), 2, pp.opAssoc.LEFT,),
- (pp.oneOf("||"), 2, pp.opAssoc.LEFT,),
- ])
-expr <<= (assignment
- | call
- | THIS
- | comparison_expr
- | arith_expr
- | lvalue
- | constant
- | read_integer
- | read_line
- | new_statement
- | new_array
- )
+ (
+ "!",
+ 1,
+ pp.opAssoc.RIGHT,
+ ),
+ (
+ pp.oneOf("< > <= >="),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("== !="),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("&&"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("||"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ ],
+)
+expr <<= (
+ assignment
+ | call
+ | THIS
+ | comparison_expr
+ | arith_expr
+ | lvalue
+ | constant
+ | read_integer
+ | read_line
+ | new_statement
+ | new_array
+)
stmt = pp.Forward()
-print_stmt = pp.Group(PRINT("statement") + LPAR + pp.Group(pp.Optional(pp.delimitedList(expr)))("args") + RPAR + SEMI)
+print_stmt = pp.Group(
+ PRINT("statement")
+ + LPAR
+ + pp.Group(pp.Optional(pp.delimitedList(expr)))("args")
+ + RPAR
+ + SEMI
+)
break_stmt = pp.Group(BREAK("statement") + SEMI)
return_stmt = pp.Group(RETURN("statement") + expr + SEMI)
-for_stmt = pp.Group(FOR("statement") + LPAR + pp.Optional(expr) + SEMI + expr + SEMI + pp.Optional(expr) + RPAR + stmt)
+for_stmt = pp.Group(
+ FOR("statement")
+ + LPAR
+ + pp.Optional(expr)
+ + SEMI
+ + expr
+ + SEMI
+ + pp.Optional(expr)
+ + RPAR
+ + stmt
+)
while_stmt = pp.Group(WHILE("statement") + LPAR + expr + RPAR + stmt)
-if_stmt = pp.Group(IF("statement")
- + LPAR + pp.Group(expr)("condition") + RPAR
- + pp.Group(stmt)("then_statement")
- + pp.Group(pp.Optional(ELSE + stmt))("else_statement"))
-stmt_block = pp.Group(LBRACE + pp.ZeroOrMore(variable_decl) + pp.ZeroOrMore(stmt) + RBRACE)
-stmt <<= (if_stmt
- | while_stmt
- | for_stmt
- | break_stmt
- | return_stmt
- | print_stmt
- | stmt_block
- | pp.Group(expr + SEMI)
- )
+if_stmt = pp.Group(
+ IF("statement")
+ + LPAR
+ + pp.Group(expr)("condition")
+ + RPAR
+ + pp.Group(stmt)("then_statement")
+ + pp.Group(pp.Optional(ELSE + stmt))("else_statement")
+)
+stmt_block = pp.Group(
+ LBRACE + pp.ZeroOrMore(variable_decl) + pp.ZeroOrMore(stmt) + RBRACE
+)
+stmt <<= (
+ if_stmt
+ | while_stmt
+ | for_stmt
+ | break_stmt
+ | return_stmt
+ | print_stmt
+ | stmt_block
+ | pp.Group(expr + SEMI)
+)
formals = pp.Optional(pp.delimitedList(variable))
-prototype = pp.Group((type_ | VOID)("return_type")
- + ident("function_name")
- + LPAR + formals("args") + RPAR + SEMI)("prototype")
-function_decl = pp.Group((type_ | VOID)("return_type") + ident("function_name")
- + LPAR + formals("args") + RPAR
- + stmt_block("body"))("function_decl")
+prototype = pp.Group(
+ (type_ | VOID)("return_type")
+ + ident("function_name")
+ + LPAR
+ + formals("args")
+ + RPAR
+ + SEMI
+)("prototype")
+function_decl = pp.Group(
+ (type_ | VOID)("return_type")
+ + ident("function_name")
+ + LPAR
+ + formals("args")
+ + RPAR
+ + stmt_block("body")
+)("function_decl")
-interface_decl = pp.Group(INTERFACE + ident("interface_name")
- + LBRACE + pp.ZeroOrMore(prototype)("prototypes") + RBRACE)("interface")
+interface_decl = pp.Group(
+ INTERFACE
+ + ident("interface_name")
+ + LBRACE
+ + pp.ZeroOrMore(prototype)("prototypes")
+ + RBRACE
+)("interface")
field = variable_decl | function_decl
-class_decl = pp.Group(CLASS + ident("class_name")
- + pp.Optional(EXTENDS + ident)("extends")
- + pp.Optional(IMPLEMENTS + pp.delimitedList(ident))("implements")
- + LBRACE + pp.ZeroOrMore(field)("fields") + RBRACE)("class_decl")
+class_decl = pp.Group(
+ CLASS
+ + ident("class_name")
+ + pp.Optional(EXTENDS + ident)("extends")
+ + pp.Optional(IMPLEMENTS + pp.delimitedList(ident))("implements")
+ + LBRACE
+ + pp.ZeroOrMore(field)("fields")
+ + RBRACE
+)("class_decl")
decl = variable_decl | function_decl | class_decl | interface_decl | prototype
program = pp.OneOrMore(pp.Group(decl))
decaf_parser = program
-stmt.runTests("""\
+stmt.runTests(
+ """\
sin(30);
a = 1;
b = 1 + 1;
a[100] = b;
a[0][0] = 2;
a = 0x1234;
-""")
+"""
+)
test_program = """
void getenv(string var);
__all__ = ["time_expression"]
+
# basic grammar definitions
def make_integer_word_expr(int_name, int_value):
return pp.CaselessKeyword(int_name).addParseAction(pp.replaceWith(int_value))
-integer_word = pp.MatchFirst(make_integer_word_expr(int_str, int_value)
- for int_value, int_str
- in enumerate("one two three four five six seven eight nine ten"
- " eleven twelve thirteen fourteen fifteen sixteen"
- " seventeen eighteen nineteen twenty".split(), start=1))
+
+
+integer_word = pp.MatchFirst(
+ make_integer_word_expr(int_str, int_value)
+ for int_value, int_str in enumerate(
+ "one two three four five six seven eight nine ten"
+ " eleven twelve thirteen fourteen fifteen sixteen"
+ " seventeen eighteen nineteen twenty".split(),
+ start=1,
+ )
+).setName("integer_word")
+
integer = pp.pyparsing_common.integer | integer_word
+integer.setName("numeric")
CK = pp.CaselessKeyword
CL = pp.CaselessLiteral
-today, tomorrow, yesterday, noon, midnight, now = map(CK, "today tomorrow yesterday noon midnight now".split())
+today, tomorrow, yesterday, noon, midnight, now = map(
+ CK, "today tomorrow yesterday noon midnight now".split()
+)
+
+
def plural(s):
- return CK(s) | CK(s + 's').addParseAction(pp.replaceWith(s))
+ return CK(s) | CK(s + "s").addParseAction(pp.replaceWith(s))
+
+
week, day, hour, minute, second = map(plural, "week day hour minute second".split())
time_units = hour | minute | second
-any_time_units = week | day | time_units
+any_time_units = (week | day | time_units).setName("time_units")
am = CL("am")
pm = CL("pm")
-COLON = pp.Suppress(':')
+COLON = pp.Suppress(":")
in_ = CK("in").setParseAction(pp.replaceWith(1))
from_ = CK("from").setParseAction(pp.replaceWith(1))
at_ = CK("at")
on_ = CK("on")
-couple = (pp.Optional(CK("a")) + CK("couple") + pp.Optional(CK("of"))).setParseAction(pp.replaceWith(2))
+couple = (
+ (pp.Optional(CK("a")) + CK("couple") + pp.Optional(CK("of")))
+ .setParseAction(pp.replaceWith(2))
+ .setName("couple")
+)
+
a_qty = (CK("a") | CK("an")).setParseAction(pp.replaceWith(1))
the_qty = CK("the").setParseAction(pp.replaceWith(1))
-qty = pp.ungroup(integer | couple | a_qty | the_qty).setName("qty")
-time_ref_present = pp.Empty().addParseAction(pp.replaceWith(True))('time_ref_present')
+qty = pp.ungroup(
+ (integer | couple | a_qty | the_qty).setName("qty_expression")
+).setName("qty")
+time_ref_present = pp.Empty().addParseAction(pp.replaceWith(True))("time_ref_present")
+
def fill_24hr_time_fields(t):
- t['HH'] = t[0]
- t['MM'] = t[1]
- t['SS'] = 0
- t['ampm'] = ('am','pm')[t.HH >= 12]
+ t["HH"] = t[0]
+ t["MM"] = t[1]
+ t["SS"] = 0
+ t["ampm"] = ("am", "pm")[t.HH >= 12]
+
def fill_default_time_fields(t):
- for fld in 'HH MM SS'.split():
+ for fld in "HH MM SS".split():
if fld not in t:
t[fld] = 0
+
weekday_name_list = list(calendar.day_name)
-weekday_name = pp.oneOf(weekday_name_list)
+weekday_name = pp.oneOf(weekday_name_list).setName("weekday_name")
-_24hour_time = (~(integer + any_time_units)
- + pp.Word(pp.nums, exact=4).addParseAction(lambda t: [int(t[0][:2]),int(t[0][2:])],
- fill_24hr_time_fields)
- )
+_24hour_time = ~(integer + any_time_units).setName("numbered_time_units") + pp.Word(pp.nums, exact=4).setName("HHMM").addParseAction(
+ lambda t: [int(t[0][:2]), int(t[0][2:])], fill_24hr_time_fields
+)
_24hour_time.setName("0000 time")
ampm = am | pm
-timespec = (integer("HH")
- + pp.Optional(CK("o'clock")
- |
- COLON + integer("MM")
- + pp.Optional(COLON + integer("SS"))
- )
- + (am | pm)("ampm")
- ).addParseAction(fill_default_time_fields)
+timespec = (
+ integer("HH")
+ + pp.Optional(
+ CK("o'clock") | COLON + integer("MM") + pp.Optional(COLON + integer("SS"))
+ )
+ + (am | pm)("ampm")
+).addParseAction(fill_default_time_fields)
absolute_time = _24hour_time | timespec
+absolute_time.setName("absolute time")
absolute_time_of_day = noon | midnight | now | absolute_time
+absolute_time_of_day.setName("time of day")
+
def add_computed_time(t):
- if t[0] in 'now noon midnight'.split():
- t['computed_time'] = {'now': datetime.now().time().replace(microsecond=0),
- 'noon': time(hour=12),
- 'midnight': time()}[t[0]]
+ if t[0] in "now noon midnight".split():
+ t["computed_time"] = {
+ "now": datetime.now().time().replace(microsecond=0),
+ "noon": time(hour=12),
+ "midnight": time(),
+ }[t[0]]
else:
- t['HH'] = {'am': int(t['HH']) % 12,
- 'pm': int(t['HH']) % 12 + 12}[t.ampm]
- t['computed_time'] = time(hour=t.HH, minute=t.MM, second=t.SS)
+ t["HH"] = {"am": int(t["HH"]) % 12, "pm": int(t["HH"]) % 12 + 12}[t.ampm]
+ t["computed_time"] = time(hour=t.HH, minute=t.MM, second=t.SS)
+
absolute_time_of_day.addParseAction(add_computed_time)
-# relative_time_reference ::= qty time_units ('from' | 'before' | 'after') absolute_time_of_day
-# | qty time_units 'ago'
+# relative_time_reference ::= qty time_units ('ago' | ('from' | 'before' | 'after') absolute_time_of_day)
# | 'in' qty time_units
-time_units = hour | minute | second
-relative_time_reference = (qty('qty') + time_units('units') + ago('dir')
- | qty('qty') + time_units('units')
- + (from_ | before | after)('dir')
- + pp.Group(absolute_time_of_day)('ref_time')
- | in_('dir') + qty('qty') + time_units('units')
- )
+time_units = (hour | minute | second).setName("time unit")
+relative_time_reference = (
+ (
+ qty("qty")
+ + time_units("units")
+ + (
+ ago("dir")
+ | (from_ | before | after)("dir")
+ + pp.Group(absolute_time_of_day)("ref_time")
+ )
+ )
+ | in_("dir") + qty("qty") + time_units("units")
+).setName("relative time")
+
def compute_relative_time(t):
- if 'ref_time' not in t:
- t['ref_time'] = datetime.now().time().replace(microsecond=0)
+ if "ref_time" not in t:
+ t["ref_time"] = datetime.now().time().replace(microsecond=0)
else:
- t['ref_time'] = t.ref_time.computed_time
- delta_seconds = {'hour': 3600,
- 'minute': 60,
- 'second': 1}[t.units] * t.qty
- t['time_delta'] = timedelta(seconds=t.dir * delta_seconds)
+ t["ref_time"] = t.ref_time.computed_time
+ delta_seconds = {"hour": 3600, "minute": 60, "second": 1}[t.units] * t.qty
+ t["time_delta"] = timedelta(seconds=t.dir * delta_seconds)
+
relative_time_reference.addParseAction(compute_relative_time)
time_reference = absolute_time_of_day | relative_time_reference
+time_reference.setName("time reference")
+
+
def add_default_time_ref_fields(t):
- if 'time_delta' not in t:
- t['time_delta'] = timedelta()
+ if "time_delta" not in t:
+ t["time_delta"] = timedelta()
+
+
time_reference.addParseAction(add_default_time_ref_fields)
# absolute_day_reference ::= 'today' | 'tomorrow' | 'yesterday' | ('next' | 'last') weekday_name
# day_units ::= 'days' | 'weeks'
day_units = day | week
-weekday_reference = pp.Optional(next_ | last_, 1)('dir') + weekday_name('day_name')
+weekday_reference = pp.Optional(next_ | last_, 1)("dir") + weekday_name("day_name")
+
def convert_abs_day_reference_to_date(t):
now = datetime.now().replace(microsecond=0)
# handle day reference by weekday name
- if 'day_name' in t:
+ if "day_name" in t:
todaynum = now.weekday()
daynames = [n.lower() for n in weekday_name_list]
nameddaynum = daynames.index(t.day_name.lower())
else:
name = t[0]
t["abs_date"] = {
- "now" : now,
- "today" : datetime(now.year, now.month, now.day),
- "yesterday" : datetime(now.year, now.month, now.day) + timedelta(days=-1),
- "tomorrow" : datetime(now.year, now.month, now.day) + timedelta(days=+1),
- }[name]
+ "now": now,
+ "today": datetime(now.year, now.month, now.day),
+ "yesterday": datetime(now.year, now.month, now.day) + timedelta(days=-1),
+ "tomorrow": datetime(now.year, now.month, now.day) + timedelta(days=+1),
+ }[name]
-absolute_day_reference = today | tomorrow | yesterday | now + time_ref_present | weekday_reference
-absolute_day_reference.addParseAction(convert_abs_day_reference_to_date)
+absolute_day_reference = (
+ today | tomorrow | yesterday | now + time_ref_present | weekday_reference
+)
+absolute_day_reference.addParseAction(convert_abs_day_reference_to_date)
+absolute_day_reference.setName("absolute day")
# relative_day_reference ::= 'in' qty day_units
-# | qty day_units 'ago'
-# | 'qty day_units ('from' | 'before' | 'after') absolute_day_reference
-relative_day_reference = (in_('dir') + qty('qty') + day_units('units')
- | qty('qty') + day_units('units') + ago('dir')
- | qty('qty') + day_units('units') + (from_ | before | after)('dir')
- + absolute_day_reference('ref_day')
- )
+# | qty day_units
+# ('ago'
+# | ('from' | 'before' | 'after') absolute_day_reference)
+relative_day_reference = in_("dir") + qty("qty") + day_units("units") | qty(
+ "qty"
+) + day_units("units") + (
+ ago("dir") | ((from_ | before | after)("dir") + absolute_day_reference("ref_day"))
+)
+relative_day_reference.setName("relative day")
+
def compute_relative_date(t):
now = datetime.now().replace(microsecond=0)
- if 'ref_day' in t:
- t['computed_date'] = t.ref_day
+ if "ref_day" in t:
+ t["computed_date"] = t.ref_day
else:
- t['computed_date'] = now.date()
- day_diff = t.dir * t.qty * {'week': 7, 'day': 1}[t.units]
- t['date_delta'] = timedelta(days=day_diff)
+ t["computed_date"] = now.date()
+ day_diff = t.dir * t.qty * {"week": 7, "day": 1}[t.units]
+ t["date_delta"] = timedelta(days=day_diff)
+
+
relative_day_reference.addParseAction(compute_relative_date)
# combine expressions for absolute and relative day references
day_reference = relative_day_reference | absolute_day_reference
+day_reference.setName("day reference")
+
+
def add_default_date_fields(t):
- if 'date_delta' not in t:
- t['date_delta'] = timedelta()
+ if "date_delta" not in t:
+ t["date_delta"] = timedelta()
+
+
day_reference.addParseAction(add_default_date_fields)
# combine date and time expressions into single overall parser
-time_and_day = (time_reference + time_ref_present + pp.Optional(pp.Optional(on_) + day_reference)
- | day_reference + pp.Optional(at_ + absolute_time_of_day + time_ref_present))
+time_and_day = time_reference + time_ref_present + pp.Optional(
+ pp.Optional(on_) + day_reference
+) | day_reference + pp.Optional(at_ + absolute_time_of_day + time_ref_present)
+time_and_day.setName("time and day")
# parse actions for total time_and_day expression
def save_original_string(s, l, t):
# save original input string and reference time
- t['original'] = ' '.join(s.strip().split())
- t['relative_to'] = datetime.now().replace(microsecond=0)
+ t["original"] = " ".join(s.strip().split())
+ t["relative_to"] = datetime.now().replace(microsecond=0)
+
def compute_timestamp(t):
# accumulate values from parsed time and day subexpressions - fill in defaults for omitted parts
now = datetime.now().replace(microsecond=0)
- if 'computed_time' not in t:
- t['computed_time'] = t.ref_time or now.time()
- if 'abs_date' not in t:
- t['abs_date'] = now
+ if "computed_time" not in t:
+ t["computed_time"] = t.ref_time or now.time()
+ if "abs_date" not in t:
+ t["abs_date"] = now
# roll up all fields and apply any time or day deltas
- t['computed_dt'] = (
- t.abs_date.replace(hour=t.computed_time.hour, minute=t.computed_time.minute, second=t.computed_time.second)
+ t["computed_dt"] = (
+ t.abs_date.replace(
+ hour=t.computed_time.hour,
+ minute=t.computed_time.minute,
+ second=t.computed_time.second,
+ )
+ (t.time_delta or timedelta(0))
+ (t.date_delta or timedelta(0))
)
# if time just given in terms of day expressions, zero out time fields
if not t.time_ref_present:
- t['computed_dt'] = t.computed_dt.replace(hour=0, minute=0, second=0)
+ t["computed_dt"] = t.computed_dt.replace(hour=0, minute=0, second=0)
# add results name compatible with previous version
- t['calculatedTime'] = t.computed_dt
+ t["calculatedTime"] = t.computed_dt
# add time_offset fields
- t['time_offset'] = t.computed_dt - t.relative_to
+ t["time_offset"] = t.computed_dt - t.relative_to
+
def remove_temp_keys(t):
# strip out keys that are just used internally
all_keys = list(t.keys())
for k in all_keys:
- if k not in ('computed_dt', 'original', 'relative_to', 'time_offset', 'calculatedTime'):
+ if k not in (
+ "computed_dt",
+ "original",
+ "relative_to",
+ "time_offset",
+ "calculatedTime",
+ ):
del t[k]
+
time_and_day.addParseAction(save_original_string, compute_timestamp, remove_temp_keys)
time_expression = time_and_day
-if __name__ == "__main__":
+def main():
current_time = datetime.now()
# test grammar
tests = """\
10000 seconds ago
"""
- time_of_day = timedelta(hours=current_time.hour,
- minutes=current_time.minute,
- seconds=current_time.second)
+ time_of_day = timedelta(
+ hours=current_time.hour,
+ minutes=current_time.minute,
+ seconds=current_time.second,
+ )
expected = {
- 'now' : timedelta(0),
+ "now": timedelta(0),
"10 seconds ago": timedelta(seconds=-10),
"100 seconds ago": timedelta(seconds=-100),
"1000 seconds ago": timedelta(seconds=-1000),
"10000 seconds ago": timedelta(seconds=-10000),
- '10 minutes ago': timedelta(minutes=-10),
- '10 minutes from now': timedelta(minutes=10),
- 'in 10 minutes': timedelta(minutes=10),
- 'in a minute': timedelta(minutes=1),
- 'in a couple of minutes': timedelta(minutes=2),
- '20 seconds ago': timedelta(seconds=-20),
- 'in 30 seconds': timedelta(seconds=30),
- 'in an hour': timedelta(hours=1),
- 'in a couple hours': timedelta(hours=2),
- 'a week from now': timedelta(days=7),
- '3 days from now': timedelta(days=3),
- 'a couple of days from now': timedelta(days=2),
- 'an hour ago': timedelta(hours=-1),
- 'in a couple days': timedelta(days=2) - time_of_day,
- 'a week from today': timedelta(days=7) - time_of_day,
- 'three weeks ago': timedelta(days=-21) - time_of_day,
- 'a day ago': timedelta(days=-1) - time_of_day,
- 'in a couple of days': timedelta(days=2) - time_of_day,
- 'a couple of days from today': timedelta(days=2) - time_of_day,
- '2 weeks after today': timedelta(days=14) - time_of_day,
- 'in 2 weeks': timedelta(days=14) - time_of_day,
- 'the day after tomorrow': timedelta(days=2) - time_of_day,
- 'tomorrow': timedelta(days=1) - time_of_day,
- 'the day before yesterday': timedelta(days=-2) - time_of_day,
- 'yesterday': timedelta(days=-1) - time_of_day,
- 'today': -time_of_day,
- 'midnight': -time_of_day,
- 'in a day': timedelta(days=1) - time_of_day,
- '3 days ago': timedelta(days=-3) - time_of_day,
- 'noon tomorrow': timedelta(days=1) - time_of_day + timedelta(hours=12),
- '6am tomorrow': timedelta(days=1) - time_of_day + timedelta(hours=6),
- '0800 yesterday': timedelta(days=-1) - time_of_day + timedelta(hours=8),
- '1700 tomorrow': timedelta(days=1) - time_of_day + timedelta(hours=17),
- '12:15 AM today': -time_of_day + timedelta(minutes=15),
- '3pm 2 days from today': timedelta(days=2) - time_of_day + timedelta(hours=15),
- 'ten seconds before noon tomorrow': timedelta(days=1) - time_of_day
- + timedelta(hours=12) + timedelta(seconds=-10),
- '20 seconds before noon': -time_of_day + timedelta(hours=12) + timedelta(seconds=-20),
- 'in 3 days at 5pm': timedelta(days=3) - time_of_day + timedelta(hours=17),
+ "10 minutes ago": timedelta(minutes=-10),
+ "10 minutes from now": timedelta(minutes=10),
+ "in 10 minutes": timedelta(minutes=10),
+ "in a minute": timedelta(minutes=1),
+ "in a couple of minutes": timedelta(minutes=2),
+ "20 seconds ago": timedelta(seconds=-20),
+ "in 30 seconds": timedelta(seconds=30),
+ "in an hour": timedelta(hours=1),
+ "in a couple hours": timedelta(hours=2),
+ "a week from now": timedelta(days=7),
+ "3 days from now": timedelta(days=3),
+ "a couple of days from now": timedelta(days=2),
+ "an hour ago": timedelta(hours=-1),
+ "in a couple days": timedelta(days=2) - time_of_day,
+ "a week from today": timedelta(days=7) - time_of_day,
+ "three weeks ago": timedelta(days=-21) - time_of_day,
+ "a day ago": timedelta(days=-1) - time_of_day,
+ "in a couple of days": timedelta(days=2) - time_of_day,
+ "a couple of days from today": timedelta(days=2) - time_of_day,
+ "2 weeks after today": timedelta(days=14) - time_of_day,
+ "in 2 weeks": timedelta(days=14) - time_of_day,
+ "the day after tomorrow": timedelta(days=2) - time_of_day,
+ "tomorrow": timedelta(days=1) - time_of_day,
+ "the day before yesterday": timedelta(days=-2) - time_of_day,
+ "8am the day after tomorrow": timedelta(days=+2)
+ - time_of_day
+ + timedelta(hours=8),
+ "yesterday": timedelta(days=-1) - time_of_day,
+ "today": -time_of_day,
+ "midnight": -time_of_day,
+ "in a day": timedelta(days=1) - time_of_day,
+ "3 days ago": timedelta(days=-3) - time_of_day,
+ "noon tomorrow": timedelta(days=1) - time_of_day + timedelta(hours=12),
+ "6am tomorrow": timedelta(days=1) - time_of_day + timedelta(hours=6),
+ "0800 yesterday": timedelta(days=-1) - time_of_day + timedelta(hours=8),
+ "1700 tomorrow": timedelta(days=1) - time_of_day + timedelta(hours=17),
+ "12:15 AM today": -time_of_day + timedelta(minutes=15),
+ "3pm 2 days from today": timedelta(days=2) - time_of_day + timedelta(hours=15),
+ "ten seconds before noon tomorrow": timedelta(days=1)
+ - time_of_day
+ + timedelta(hours=12)
+ + timedelta(seconds=-10),
+ "20 seconds before noon": -time_of_day
+ + timedelta(hours=12)
+ + timedelta(seconds=-20),
+ "in 3 days at 5pm": timedelta(days=3) - time_of_day + timedelta(hours=17),
}
def verify_offset(instring, parsed):
if instring in expected:
# allow up to a second time discrepancy due to test processing time
if (parsed.time_offset - expected[instring]) <= time_epsilon:
- parsed['verify_offset'] = 'PASS'
+ parsed["verify_offset"] = "PASS"
else:
- parsed['verify_offset'] = 'FAIL'
+ parsed["verify_offset"] = "FAIL"
print("(relative to %s)" % datetime.now())
time_expression.runTests(tests, postParse=verify_offset)
+
+
+if __name__ == "__main__":
+ main()
__author__ = "Daniel 'Dang' Griffith <pythondev - dang at lazytwinacres . net>"\r
\r
\r
-from pyparsing import Literal, CaselessLiteral, Word, delimitedList \\r
- , Optional, Combine, Group, alphas, nums, alphanums, Forward \\r
- , oneOf, OneOrMore, ZeroOrMore, CharsNotIn\r
+from pyparsing import (\r
+ Literal,\r
+ CaselessLiteral,\r
+ Word,\r
+ delimitedList,\r
+ Optional,\r
+ Combine,\r
+ Group,\r
+ alphas,\r
+ nums,\r
+ alphanums,\r
+ Forward,\r
+ oneOf,\r
+ OneOrMore,\r
+ ZeroOrMore,\r
+ CharsNotIn,\r
+)\r
\r
\r
# This converts DFM character constants into Python string (unicode) values.\r
def to_chr(x):\r
"""chr(x) if 0 < x < 128 ; unicode(x) if x > 127."""\r
- return 0 < x < 128 and chr(x) or eval("u'\\u%d'" % x )\r
+ return 0 < x < 128 and chr(x) or eval("u'\\u%d'" % x)\r
+\r
\r
#################\r
# BEGIN GRAMMAR\r
#################\r
\r
-COLON = Literal(":").suppress()\r
+COLON = Literal(":").suppress()\r
CONCAT = Literal("+").suppress()\r
EQUALS = Literal("=").suppress()\r
LANGLE = Literal("<").suppress()\r
RBRACE = Literal("]").suppress()\r
RPAREN = Literal(")").suppress()\r
\r
-CATEGORIES = CaselessLiteral("categories").suppress()\r
-END = CaselessLiteral("end").suppress()\r
-FONT = CaselessLiteral("font").suppress()\r
-HINT = CaselessLiteral("hint").suppress()\r
-ITEM = CaselessLiteral("item").suppress()\r
-OBJECT = CaselessLiteral("object").suppress()\r
+CATEGORIES = CaselessLiteral("categories").suppress()\r
+END = CaselessLiteral("end").suppress()\r
+FONT = CaselessLiteral("font").suppress()\r
+HINT = CaselessLiteral("hint").suppress()\r
+ITEM = CaselessLiteral("item").suppress()\r
+OBJECT = CaselessLiteral("object").suppress()\r
\r
-attribute_value_pair = Forward() # this is recursed in item_list_entry\r
+attribute_value_pair = Forward() # this is recursed in item_list_entry\r
\r
simple_identifier = Word(alphas, alphanums + "_")\r
-identifier = Combine( simple_identifier + ZeroOrMore( Literal(".") + simple_identifier ))\r
+identifier = Combine(simple_identifier + ZeroOrMore(Literal(".") + simple_identifier))\r
object_name = identifier\r
object_type = identifier\r
\r
# Integer and floating point values are converted to Python longs and floats, respectively.\r
-int_value = Combine(Optional("-") + Word(nums)).setParseAction(lambda s,l,t: [ int(t[0]) ] )\r
-float_value = Combine(Optional("-") + Optional(Word(nums)) + "." + Word(nums)).setParseAction(lambda s,l,t: [ float(t[0]) ] )\r
+int_value = Combine(Optional("-") + Word(nums)).setParseAction(\r
+ lambda s, l, t: [int(t[0])]\r
+)\r
+float_value = Combine(\r
+ Optional("-") + Optional(Word(nums)) + "." + Word(nums)\r
+).setParseAction(lambda s, l, t: [float(t[0])])\r
number_value = float_value | int_value\r
\r
# Base16 constants are left in string form, including the surrounding braces.\r
-base16_value = Combine(Literal("{") + OneOrMore(Word("0123456789ABCDEFabcdef")) + Literal("}"), adjacent=False)\r
+base16_value = Combine(\r
+ Literal("{") + OneOrMore(Word("0123456789ABCDEFabcdef")) + Literal("}"),\r
+ adjacent=False,\r
+)\r
\r
# This is the first part of a hack to convert the various delphi partial sglQuotedStrings\r
# into a single sglQuotedString equivalent. The gist of it is to combine\r
# all sglQuotedStrings (with their surrounding quotes removed (suppressed))\r
# with sequences of #xyz character constants, with "strings" concatenated\r
# with a '+' sign.\r
-unquoted_sglQuotedString = Combine( Literal("'").suppress() + ZeroOrMore( CharsNotIn("'\n\r") ) + Literal("'").suppress() )\r
+unquoted_sglQuotedString = Combine(\r
+ Literal("'").suppress() + ZeroOrMore(CharsNotIn("'\n\r")) + Literal("'").suppress()\r
+)\r
\r
# The parse action on this production converts repetitions of constants into a single string.\r
pound_char = Combine(\r
- OneOrMore((Literal("#").suppress()+Word(nums)\r
- ).setParseAction( lambda s, l, t: to_chr(int(t[0]) ))))\r
+ OneOrMore(\r
+ (Literal("#").suppress() + Word(nums)).setParseAction(\r
+ lambda s, l, t: to_chr(int(t[0]))\r
+ )\r
+ )\r
+)\r
\r
# This is the second part of the hack. It combines the various "unquoted"\r
# partial strings into a single one. Then, the parse action puts\r
# a single matched pair of quotes around it.\r
delphi_string = Combine(\r
- OneOrMore(CONCAT | pound_char | unquoted_sglQuotedString)\r
- , adjacent=False\r
- ).setParseAction(lambda s, l, t: "'%s'" % t[0])\r
+ OneOrMore(CONCAT | pound_char | unquoted_sglQuotedString), adjacent=False\r
+).setParseAction(lambda s, l, t: "'%s'" % t[0])\r
\r
string_value = delphi_string | base16_value\r
\r
-list_value = LBRACE + Optional(Group(delimitedList(identifier | number_value | string_value))) + RBRACE\r
-paren_list_value = LPAREN + ZeroOrMore(identifier | number_value | string_value) + RPAREN\r
+list_value = (\r
+ LBRACE\r
+ + Optional(Group(delimitedList(identifier | number_value | string_value)))\r
+ + RBRACE\r
+)\r
+paren_list_value = (\r
+ LPAREN + ZeroOrMore(identifier | number_value | string_value) + RPAREN\r
+)\r
\r
item_list_entry = ITEM + ZeroOrMore(attribute_value_pair) + END\r
item_list = LANGLE + ZeroOrMore(item_list_entry) + RANGLE\r
\r
generic_value = identifier\r
-value = item_list | number_value | string_value | list_value | paren_list_value | generic_value\r
+value = (\r
+ item_list\r
+ | number_value\r
+ | string_value\r
+ | list_value\r
+ | paren_list_value\r
+ | generic_value\r
+)\r
\r
category_attribute = CATEGORIES + PERIOD + oneOf("strings itemsvisibles visibles", True)\r
-event_attribute = oneOf("onactivate onclosequery onclose oncreate ondeactivate onhide onshow", True)\r
+event_attribute = oneOf(\r
+ "onactivate onclosequery onclose oncreate ondeactivate onhide onshow", True\r
+)\r
font_attribute = FONT + PERIOD + oneOf("charset color height name style", True)\r
hint_attribute = HINT\r
layout_attribute = oneOf("left top width height", True)\r
generic_attribute = identifier\r
-attribute = (category_attribute | event_attribute | font_attribute | hint_attribute | layout_attribute | generic_attribute)\r
+attribute = (\r
+ category_attribute\r
+ | event_attribute\r
+ | font_attribute\r
+ | hint_attribute\r
+ | layout_attribute\r
+ | generic_attribute\r
+)\r
\r
category_attribute_value_pair = category_attribute + EQUALS + paren_list_value\r
event_attribute_value_pair = event_attribute + EQUALS + value\r
layout_attribute_value_pair = layout_attribute + EQUALS + value\r
generic_attribute_value_pair = attribute + EQUALS + value\r
attribute_value_pair << Group(\r
- category_attribute_value_pair\r
+ category_attribute_value_pair\r
| event_attribute_value_pair\r
| font_attribute_value_pair\r
| hint_attribute_value_pair\r
| layout_attribute_value_pair\r
| generic_attribute_value_pair\r
- )\r
+)\r
\r
-object_declaration = Group((OBJECT + object_name + COLON + object_type))\r
+object_declaration = Group(OBJECT + object_name + COLON + object_type)\r
object_attributes = Group(ZeroOrMore(attribute_value_pair))\r
\r
nested_object = Forward()\r
-object_definition = object_declaration + object_attributes + ZeroOrMore(nested_object) + END\r
+object_definition = (\r
+ object_declaration + object_attributes + ZeroOrMore(nested_object) + END\r
+)\r
nested_object << Group(object_definition)\r
\r
#################\r
# END GRAMMAR\r
#################\r
\r
+\r
def printer(s, loc, tok):\r
- print(tok, end=' ')\r
+ print(tok, end=" ")\r
return tok\r
\r
+\r
def get_filename_list(tf):\r
import sys, glob\r
+\r
if tf == None:\r
if len(sys.argv) > 1:\r
tf = sys.argv[1:]\r
testfiles.extend(glob.glob(arg))\r
return testfiles\r
\r
+\r
def main(testfiles=None, action=printer):\r
"""testfiles can be None, in which case the command line arguments are used as filenames.\r
testfiles can be a string, in which case that file is parsed.\r
failures.append(f)\r
\r
if failures:\r
- print('\nfailed while processing %s' % ', '.join(failures))\r
- print('\nsucceeded on %d of %d files' %(success, len(testfiles)))\r
+ print("\nfailed while processing %s" % ", ".join(failures))\r
+ print("\nsucceeded on %d of %d files" % (success, len(testfiles)))\r
\r
if len(retval) == 1 and len(testfiles) == 1:\r
# if only one file is parsed, return the parseResults directly\r
# else, return a dictionary of parseResults\r
return retval\r
\r
+\r
if __name__ == "__main__":\r
main()\r
"""\r
\r
from pyparsing import *\r
-import datetime,time\r
+import datetime, time\r
\r
-LBRACE,RBRACE,SEMI,QUOTE = map(Suppress,'{};"')\r
-ipAddress = Combine(Word(nums) + ('.' + Word(nums))*3)\r
-hexint = Word(hexnums,exact=2)\r
-macAddress = Combine(hexint + (':'+hexint)*5)\r
+LBRACE, RBRACE, SEMI, QUOTE = map(Suppress, '{};"')\r
+ipAddress = Combine(Word(nums) + ("." + Word(nums)) * 3)\r
+hexint = Word(hexnums, exact=2)\r
+macAddress = Combine(hexint + (":" + hexint) * 5)\r
hdwType = Word(alphanums)\r
\r
-yyyymmdd = Combine((Word(nums,exact=4)|Word(nums,exact=2))+\r
- ('/'+Word(nums,exact=2))*2)\r
-hhmmss = Combine(Word(nums,exact=2)+(':'+Word(nums,exact=2))*2)\r
-dateRef = oneOf(list("0123456"))("weekday") + yyyymmdd("date") + \\r
- hhmmss("time")\r
+yyyymmdd = Combine(\r
+ (Word(nums, exact=4) | Word(nums, exact=2)) + ("/" + Word(nums, exact=2)) * 2\r
+)\r
+hhmmss = Combine(Word(nums, exact=2) + (":" + Word(nums, exact=2)) * 2)\r
+dateRef = oneOf(list("0123456"))("weekday") + yyyymmdd("date") + hhmmss("time")\r
+\r
\r
def utcToLocalTime(tokens):\r
- utctime = datetime.datetime.strptime("%(date)s %(time)s" % tokens,\r
- "%Y/%m/%d %H:%M:%S")\r
- localtime = utctime-datetime.timedelta(0,time.timezone,0)\r
- tokens["utcdate"],tokens["utctime"] = tokens["date"],tokens["time"]\r
- tokens["localdate"],tokens["localtime"] = str(localtime).split()\r
+ utctime = datetime.datetime.strptime(\r
+ "%(date)s %(time)s" % tokens, "%Y/%m/%d %H:%M:%S"\r
+ )\r
+ localtime = utctime - datetime.timedelta(0, time.timezone, 0)\r
+ tokens["utcdate"], tokens["utctime"] = tokens["date"], tokens["time"]\r
+ tokens["localdate"], tokens["localtime"] = str(localtime).split()\r
del tokens["date"]\r
del tokens["time"]\r
+\r
+\r
dateRef.setParseAction(utcToLocalTime)\r
\r
startsStmt = "starts" + dateRef + SEMI\r
uidStmt = "uid" + QuotedString('"')("uid") + SEMI\r
bindingStmt = "binding" + Word(alphanums) + Word(alphanums) + SEMI\r
\r
-leaseStatement = startsStmt | endsStmt | tstpStmt | tsfpStmt | hdwStmt | \\r
- uidStmt | bindingStmt\r
-leaseDef = "lease" + ipAddress("ipaddress") + LBRACE + \\r
- Dict(ZeroOrMore(Group(leaseStatement))) + RBRACE\r
+leaseStatement = (\r
+ startsStmt | endsStmt | tstpStmt | tsfpStmt | hdwStmt | uidStmt | bindingStmt\r
+)\r
+leaseDef = (\r
+ "lease"\r
+ + ipAddress("ipaddress")\r
+ + LBRACE\r
+ + Dict(ZeroOrMore(Group(leaseStatement)))\r
+ + RBRACE\r
+)\r
\r
for lease in leaseDef.searchString(sample):\r
print(lease.dump())\r
- print(lease.ipaddress,'->',lease.hardware.mac)\r
+ print(lease.ipaddress, "->", lease.hardware.mac)\r
print()\r
"""\r
\r
# define grammar for datatable\r
-heading = (pp.Literal(\r
-"+-------+------+------+------+------+------+------+------+------+") +\r
-"| | A1 | B1 | C1 | D1 | A2 | B2 | C2 | D2 |" +\r
-"+=======+======+======+======+======+======+======+======+======+").suppress()\r
+heading = (\r
+ pp.Literal("+-------+------+------+------+------+------+------+------+------+")\r
+ + "| | A1 | B1 | C1 | D1 | A2 | B2 | C2 | D2 |"\r
+ + "+=======+======+======+======+======+======+======+======+======+"\r
+).suppress()\r
vert = pp.Literal("|").suppress()\r
number = pp.Word(pp.nums)\r
-rowData = pp.Group( vert + pp.Word(pp.alphas) + vert + pp.delimitedList(number,"|") + vert )\r
+rowData = pp.Group(\r
+ vert + pp.Word(pp.alphas) + vert + pp.delimitedList(number, "|") + vert\r
+)\r
trailing = pp.Literal(\r
-"+-------+------+------+------+------+------+------+------+------+").suppress()\r
+ "+-------+------+------+------+------+------+------+------+------+"\r
+).suppress()\r
\r
datatable = heading + pp.Dict(pp.ZeroOrMore(rowData)) + trailing\r
\r
print("data keys=", list(data.keys()))\r
\r
# use dict-style access to values\r
-print("data['min']=", data['min'])\r
+print("data['min']=", data["min"])\r
\r
# use attribute-style access to values (if key is a valid Python identifier)\r
print("data.max", data.max)\r
#\r
# Copyright (c) 2004, Paul McGuire\r
#\r
-from pyparsing import Literal, Word, Group, Dict, ZeroOrMore, alphas, nums, delimitedList, pyparsing_common as ppc\r
+from pyparsing import (\r
+ Literal,\r
+ Word,\r
+ Group,\r
+ Dict,\r
+ ZeroOrMore,\r
+ alphas,\r
+ nums,\r
+ delimitedList,\r
+ pyparsing_common as ppc,\r
+)\r
\r
testData = """\r
+-------+------+------+------+------+------+------+------+------+\r
\r
vert = Literal("|").suppress()\r
\r
-rowDelim = ("+" + ZeroOrMore( underline + "+" ) ).suppress()\r
+rowDelim = ("+" + ZeroOrMore(underline + "+")).suppress()\r
columnHeader = Group(vert + vert + delimitedList(Word(alphas + nums), "|") + vert)\r
\r
heading = rowDelim + columnHeader("columns") + rowDelim\r
-rowData = Group( vert + Word(alphas) + vert + delimitedList(number,"|") + vert )\r
+rowData = Group(vert + Word(alphas) + vert + delimitedList(number, "|") + vert)\r
trailing = rowDelim\r
\r
-datatable = heading + Dict( ZeroOrMore(rowData) ) + trailing\r
+datatable = heading + Dict(ZeroOrMore(rowData)) + trailing\r
\r
# now parse data and print results\r
data = datatable.parseString(testData)\r
print(data.dump())\r
print("data keys=", list(data.keys()))\r
-print("data['min']=", data['min'])\r
-print("sum(data['min']) =", sum(data['min']))\r
+print("data['min']=", data["min"])\r
+print("sum(data['min']) =", sum(data["min"]))\r
print("data.max =", data.max)\r
print("sum(data.max) =", sum(data.max))\r
\r
# now print transpose of data table, using column labels read from table header and\r
# values from data lists\r
print()\r
-print(" " * 5, end=' ')\r
-for i in range(1,len(data)):\r
- print("|%5s" % data[i][0], end=' ')\r
+print(" " * 5, end=" ")\r
+for i in range(1, len(data)):\r
+ print("|%5s" % data[i][0], end=" ")\r
print()\r
-print(("-" * 6) + ("+------" * (len(data)-1)))\r
+print(("-" * 6) + ("+------" * (len(data) - 1)))\r
for i in range(len(data.columns)):\r
- print("%5s" % data.columns[i], end=' ')\r
+ print("%5s" % data.columns[i], end=" ")\r
for j in range(len(data) - 1):\r
- print('|%5s' % data[j + 1][i + 1], end=' ')\r
+ print("|%5s" % data[j + 1][i + 1], end=" ")\r
print()\r
from pyparsing import *\r
\r
\r
-all_names = '''\r
+all_names = """\r
integer\r
meta_identifier\r
terminal_string\r
definitions_list\r
syntax_rule\r
syntax\r
-'''.split()\r
+""".split()\r
\r
\r
integer = Word(nums)\r
-meta_identifier = Word(alphas, alphanums + '_')\r
-terminal_string = Suppress("'") + CharsNotIn("'") + Suppress("'") ^ \\r
- Suppress('"') + CharsNotIn('"') + Suppress('"')\r
+meta_identifier = Word(alphas, alphanums + "_")\r
+terminal_string = Suppress("'") + CharsNotIn("'") + Suppress("'") ^ Suppress(\r
+ '"'\r
+) + CharsNotIn('"') + Suppress('"')\r
definitions_list = Forward()\r
-optional_sequence = Suppress('[') + definitions_list + Suppress(']')\r
-repeated_sequence = Suppress('{') + definitions_list + Suppress('}')\r
-grouped_sequence = Suppress('(') + definitions_list + Suppress(')')\r
-syntactic_primary = optional_sequence ^ repeated_sequence ^ \\r
- grouped_sequence ^ meta_identifier ^ terminal_string\r
-syntactic_factor = Optional(integer + Suppress('*')) + syntactic_primary\r
-syntactic_term = syntactic_factor + Optional(Suppress('-') + syntactic_factor)\r
-single_definition = delimitedList(syntactic_term, ',')\r
-definitions_list << delimitedList(single_definition, '|')\r
-syntax_rule = meta_identifier + Suppress('=') + definitions_list + \\r
- Suppress(';')\r
-\r
-ebnfComment = ( "(*" +\r
- ZeroOrMore( CharsNotIn("*") | ( "*" + ~Literal(")") ) ) +\r
- "*)" ).streamline().setName("ebnfComment")\r
+optional_sequence = Suppress("[") + definitions_list + Suppress("]")\r
+repeated_sequence = Suppress("{") + definitions_list + Suppress("}")\r
+grouped_sequence = Suppress("(") + definitions_list + Suppress(")")\r
+syntactic_primary = (\r
+ optional_sequence\r
+ ^ repeated_sequence\r
+ ^ grouped_sequence\r
+ ^ meta_identifier\r
+ ^ terminal_string\r
+)\r
+syntactic_factor = Optional(integer + Suppress("*")) + syntactic_primary\r
+syntactic_term = syntactic_factor + Optional(Suppress("-") + syntactic_factor)\r
+single_definition = delimitedList(syntactic_term, ",")\r
+definitions_list << delimitedList(single_definition, "|")\r
+syntax_rule = meta_identifier + Suppress("=") + definitions_list + Suppress(";")\r
+\r
+ebnfComment = (\r
+ ("(*" + ZeroOrMore(CharsNotIn("*") | ("*" + ~Literal(")"))) + "*)")\r
+ .streamline()\r
+ .setName("ebnfComment")\r
+)\r
\r
syntax = OneOrMore(syntax_rule)\r
syntax.ignore(ebnfComment)\r
def do_integer(str, loc, toks):\r
return int(toks[0])\r
\r
+\r
def do_meta_identifier(str, loc, toks):\r
if toks[0] in symbol_table:\r
return symbol_table[toks[0]]\r
symbol_table[toks[0]] = Forward()\r
return symbol_table[toks[0]]\r
\r
+\r
def do_terminal_string(str, loc, toks):\r
return Literal(toks[0])\r
\r
+\r
def do_optional_sequence(str, loc, toks):\r
return Optional(toks[0])\r
\r
+\r
def do_repeated_sequence(str, loc, toks):\r
return ZeroOrMore(toks[0])\r
\r
+\r
def do_grouped_sequence(str, loc, toks):\r
return Group(toks[0])\r
\r
+\r
def do_syntactic_primary(str, loc, toks):\r
return toks[0]\r
\r
+\r
def do_syntactic_factor(str, loc, toks):\r
if len(toks) == 2:\r
# integer * syntactic_primary\r
return And([toks[1]] * toks[0])\r
else:\r
# syntactic_primary\r
- return [ toks[0] ]\r
+ return [toks[0]]\r
+\r
\r
def do_syntactic_term(str, loc, toks):\r
if len(toks) == 2:\r
return NotAny(toks[1]) + toks[0]\r
else:\r
# syntactic_factor\r
- return [ toks[0] ]\r
+ return [toks[0]]\r
+\r
\r
def do_single_definition(str, loc, toks):\r
toks = toks.asList()\r
return And(toks)\r
else:\r
# syntactic_term\r
- return [ toks[0] ]\r
+ return [toks[0]]\r
+\r
\r
def do_definitions_list(str, loc, toks):\r
toks = toks.asList()\r
return Or(toks)\r
else:\r
# single_definition\r
- return [ toks[0] ]\r
+ return [toks[0]]\r
+\r
\r
def do_syntax_rule(str, loc, toks):\r
# meta_identifier = definitions_list ;\r
assert toks[0].expr is None, "Duplicate definition"\r
forward_count.value -= 1\r
toks[0] << toks[1]\r
- return [ toks[0] ]\r
+ return [toks[0]]\r
+\r
\r
def do_syntax(str, loc, toks):\r
# syntax_rule syntax_rule ...\r
return symbol_table\r
\r
\r
-\r
symbol_table = {}\r
+\r
+\r
class forward_count:\r
pass\r
+\r
+\r
forward_count.value = 0\r
for name in all_names:\r
expr = vars()[name]\r
- action = vars()['do_' + name]\r
+ action = vars()["do_" + name]\r
expr.setName(name)\r
expr.setParseAction(action)\r
- #~ expr.setDebug()\r
+ # ~ expr.setDebug()\r
\r
\r
def parse(ebnf, given_table={}):\r
for name in table:\r
expr = table[name]\r
expr.setName(name)\r
- #~ expr.setDebug()\r
+ # ~ expr.setDebug()\r
return table\r
#\r
# Submitted 2004 by Seo Sanghyeon\r
#\r
-print('Importing pyparsing...')\r
+print("Importing pyparsing...")\r
from pyparsing import *\r
\r
-print('Constructing EBNF parser with pyparsing...')\r
+print("Constructing EBNF parser with pyparsing...")\r
import ebnf\r
\r
\r
-grammar = '''\r
+grammar = """\r
syntax = (syntax_rule), {(syntax_rule)};\r
syntax_rule = meta_identifier, '=', definitions_list, ';';\r
definitions_list = single_definition, {'|', single_definition};\r
meta_identifier = letter, {letter | digit};\r
integer = digit, {digit};\r
*)\r
-'''\r
+"""\r
\r
table = {}\r
-#~ table['character'] = Word(printables, exact=1)\r
-#~ table['letter'] = Word(alphas + '_', exact=1)\r
-#~ table['digit'] = Word(nums, exact=1)\r
-table['terminal_string'] = sglQuotedString\r
-table['meta_identifier'] = Word(alphas+"_", alphas+"_"+nums)\r
-table['integer'] = Word(nums)\r
+# ~ table['character'] = Word(printables, exact=1)\r
+# ~ table['letter'] = Word(alphas + '_', exact=1)\r
+# ~ table['digit'] = Word(nums, exact=1)\r
+table["terminal_string"] = sglQuotedString\r
+table["meta_identifier"] = Word(alphas + "_", alphas + "_" + nums)\r
+table["integer"] = Word(nums)\r
\r
-print('Parsing EBNF grammar with EBNF parser...')\r
+print("Parsing EBNF grammar with EBNF parser...")\r
parsers = ebnf.parse(grammar, table)\r
-ebnf_parser = parsers['syntax']\r
+ebnf_parser = parsers["syntax"]\r
\r
commentcharcount = 0\r
commentlocs = set()\r
-def tallyCommentChars(s,l,t):\r
- global commentcharcount,commentlocs\r
+\r
+\r
+def tallyCommentChars(s, l, t):\r
+ global commentcharcount, commentlocs\r
# only count this comment if we haven't seen it before\r
if l not in commentlocs:\r
- charCount = ( len(t[0]) - len(list(filter(str.isspace, t[0]))) )\r
+ charCount = len(t[0]) - len(list(filter(str.isspace, t[0])))\r
commentcharcount += charCount\r
commentlocs.add(l)\r
- return l,t\r
+ return l, t\r
+\r
\r
-#ordinarily, these lines wouldn't be necessary, but we are doing extra stuff with the comment expression\r
-ebnf.ebnfComment.setParseAction( tallyCommentChars )\r
-ebnf_parser.ignore( ebnf.ebnfComment )\r
+# ordinarily, these lines wouldn't be necessary, but we are doing extra stuff with the comment expression\r
+ebnf.ebnfComment.setParseAction(tallyCommentChars)\r
+ebnf_parser.ignore(ebnf.ebnfComment)\r
\r
-print('Parsing EBNF grammar with generated EBNF parser...\n')\r
+print("Parsing EBNF grammar with generated EBNF parser...\n")\r
parsed_chars = ebnf_parser.parseString(grammar)\r
parsed_char_len = len(parsed_chars)\r
\r
-print("],\n".join(str( parsed_chars.asList() ).split("],")))\r
+print("],\n".join(str(parsed_chars.asList()).split("],")))\r
\r
-#~ grammar_length = len(grammar) - len(filter(str.isspace, grammar))-commentcharcount\r
+# ~ grammar_length = len(grammar) - len(filter(str.isspace, grammar))-commentcharcount\r
\r
-#~ assert parsed_char_len == grammar_length\r
+# ~ assert parsed_char_len == grammar_length\r
\r
-print('Ok!')\r
+print("Ok!")\r
# Added support for exponentiation, using right-to-left evaluation of
# operands
#
-from pyparsing import Word, nums, alphas, Combine, oneOf, \
- opAssoc, infixNotation, Literal
+from pyparsing import (
+ Word,
+ nums,
+ alphas,
+ Combine,
+ oneOf,
+ opAssoc,
+ infixNotation,
+ Literal,
+ ParserElement,
+)
-class EvalConstant(object):
+ParserElement.enablePackrat()
+
+
+class EvalConstant:
"Class to evaluate a parsed constant or variable"
vars_ = {}
+
def __init__(self, tokens):
self.value = tokens[0]
+
def eval(self):
if self.value in EvalConstant.vars_:
return EvalConstant.vars_[self.value]
else:
return float(self.value)
-class EvalSignOp(object):
+
+class EvalSignOp:
"Class to evaluate expressions with a leading + or - sign"
+
def __init__(self, tokens):
self.sign, self.value = tokens[0]
+
def eval(self):
- mult = {'+':1, '-':-1}[self.sign]
+ mult = {"+": 1, "-": -1}[self.sign]
return mult * self.value.eval()
+
def operatorOperands(tokenlist):
"generator to extract operators and operands in pairs"
it = iter(tokenlist)
except StopIteration:
break
-class EvalPowerOp(object):
+
+class EvalPowerOp:
"Class to evaluate multiplication and division expressions"
+
def __init__(self, tokens):
self.value = tokens[0]
+
def eval(self):
res = self.value[-1].eval()
for val in self.value[-3::-2]:
- res = val.eval()**res
+ res = val.eval() ** res
return res
-class EvalMultOp(object):
+
+class EvalMultOp:
"Class to evaluate multiplication and division expressions"
+
def __init__(self, tokens):
self.value = tokens[0]
+
def eval(self):
prod = self.value[0].eval()
- for op,val in operatorOperands(self.value[1:]):
- if op == '*':
+ for op, val in operatorOperands(self.value[1:]):
+ if op == "*":
prod *= val.eval()
- if op == '/':
+ if op == "/":
prod /= val.eval()
return prod
-class EvalAddOp(object):
+
+class EvalAddOp:
"Class to evaluate addition and subtraction expressions"
+
def __init__(self, tokens):
self.value = tokens[0]
+
def eval(self):
sum = self.value[0].eval()
- for op,val in operatorOperands(self.value[1:]):
- if op == '+':
+ for op, val in operatorOperands(self.value[1:]):
+ if op == "+":
sum += val.eval()
- if op == '-':
+ if op == "-":
sum -= val.eval()
return sum
-class EvalComparisonOp(object):
+
+class EvalComparisonOp:
"Class to evaluate comparison expressions"
opMap = {
- "<" : lambda a,b : a < b,
- "<=" : lambda a,b : a <= b,
- ">" : lambda a,b : a > b,
- ">=" : lambda a,b : a >= b,
- "!=" : lambda a,b : a != b,
- "=" : lambda a,b : a == b,
- "LT" : lambda a,b : a < b,
- "LE" : lambda a,b : a <= b,
- "GT" : lambda a,b : a > b,
- "GE" : lambda a,b : a >= b,
- "NE" : lambda a,b : a != b,
- "EQ" : lambda a,b : a == b,
- "<>" : lambda a,b : a != b,
- }
+ "<": lambda a, b: a < b,
+ "<=": lambda a, b: a <= b,
+ ">": lambda a, b: a > b,
+ ">=": lambda a, b: a >= b,
+ "!=": lambda a, b: a != b,
+ "=": lambda a, b: a == b,
+ "LT": lambda a, b: a < b,
+ "LE": lambda a, b: a <= b,
+ "GT": lambda a, b: a > b,
+ "GE": lambda a, b: a >= b,
+ "NE": lambda a, b: a != b,
+ "EQ": lambda a, b: a == b,
+ "<>": lambda a, b: a != b,
+ }
+
def __init__(self, tokens):
self.value = tokens[0]
+
def eval(self):
val1 = self.value[0].eval()
- for op,val in operatorOperands(self.value[1:]):
+ for op, val in operatorOperands(self.value[1:]):
fn = EvalComparisonOp.opMap[op]
val2 = val.eval()
- if not fn(val1,val2):
+ if not fn(val1, val2):
break
val1 = val2
else:
# define the parser
integer = Word(nums)
real = Combine(Word(nums) + "." + Word(nums))
-variable = Word(alphas,exact=1)
+variable = Word(alphas, exact=1)
operand = real | integer | variable
-signop = oneOf('+ -')
-multop = oneOf('* /')
-plusop = oneOf('+ -')
-expop = Literal('**')
+signop = oneOf("+ -")
+multop = oneOf("* /")
+plusop = oneOf("+ -")
+expop = Literal("**")
# use parse actions to attach EvalXXX constructors to sub-expressions
operand.setParseAction(EvalConstant)
-arith_expr = infixNotation(operand,
+arith_expr = infixNotation(
+ operand,
[
- (signop, 1, opAssoc.RIGHT, EvalSignOp),
- (expop, 2, opAssoc.LEFT, EvalPowerOp),
- (multop, 2, opAssoc.LEFT, EvalMultOp),
- (plusop, 2, opAssoc.LEFT, EvalAddOp),
- ])
+ (signop, 1, opAssoc.RIGHT, EvalSignOp),
+ (expop, 2, opAssoc.LEFT, EvalPowerOp),
+ (multop, 2, opAssoc.LEFT, EvalMultOp),
+ (plusop, 2, opAssoc.LEFT, EvalAddOp),
+ ],
+)
comparisonop = oneOf("< <= > >= != = <> LT GT LE GE EQ NE")
-comp_expr = infixNotation(arith_expr,
+comp_expr = infixNotation(
+ arith_expr,
[
- (comparisonop, 2, opAssoc.LEFT, EvalComparisonOp),
- ])
-
-def main():
- # sample expressions posted on comp.lang.python, asking for advice
- # in safely evaluating them
- rules=[
- '( A - B ) = 0',
- '(A + B + C + D + E + F + G + H + I) = J',
- '(A + B + C + D + E + F + G + H) = I',
- '(A + B + C + D + E + F) = G',
- '(A + B + C + D + E) = (F + G + H + I + J)',
- '(A + B + C + D + E) = (F + G + H + I)',
- '(A + B + C + D + E) = F',
- '(A + B + C + D) = (E + F + G + H)',
- '(A + B + C) = (D + E + F)',
- '(A + B) = (C + D + E + F)',
- '(A + B) = (C + D)',
- '(A + B) = (C - D + E - F - G + H + I + J)',
- '(A + B) = C',
- '(A + B) = 0',
- '(A+B+C+D+E) = (F+G+H+I+J)',
- '(A+B+C+D) = (E+F+G+H)',
- '(A+B+C+D)=(E+F+G+H)',
- '(A+B+C)=(D+E+F)',
- '(A+B)=(C+D)',
- '(A+B)=C',
- '(A-B)=C',
- '(A/(B+C))',
- '(B/(C+D))',
- '(G + H) = I',
- '-0.99 LE ((A+B+C)-(D+E+F+G)) LE 0.99',
- '-0.99 LE (A-(B+C)) LE 0.99',
- '-1000.00 LE A LE 0.00',
- '-5000.00 LE A LE 0.00',
- 'A < B',
- 'A < 7000',
- 'A = -(B)',
- 'A = C',
- 'A = 0',
- 'A GT 0',
- 'A GT 0.00',
- 'A GT 7.00',
- 'A LE B',
- 'A LT -1000.00',
- 'A LT -5000',
- 'A LT 0',
- 'A=(B+C+D)',
- 'A=B',
- 'I = (G + H)',
- '0.00 LE A LE 4.00',
- '4.00 LT A LE 7.00',
- '0.00 LE A LE 4.00 LE E > D',
- '2**2**(A+3)',
- ]
- vars_={'A': 0, 'B': 1.1, 'C': 2.2, 'D': 3.3, 'E': 4.4, 'F': 5.5, 'G':
- 6.6, 'H':7.7, 'I':8.8, 'J':9.9}
-
- # define tests from given rules
- tests = []
- for t in rules:
- t_orig = t
- t = t.replace("=","==")
- t = t.replace("EQ","==")
- t = t.replace("LE","<=")
- t = t.replace("GT",">")
- t = t.replace("LT","<")
- t = t.replace("GE",">=")
- t = t.replace("LE","<=")
- t = t.replace("NE","!=")
- t = t.replace("<>","!=")
- tests.append( (t_orig,eval(t,vars_)) )
-
- # copy vars_ to EvalConstant lookup dict
- EvalConstant.vars_ = vars_
- failed = 0
- for test,expected in tests:
- ret = comp_expr.parseString(test)[0]
- parsedvalue = ret.eval()
- print(test, expected, parsedvalue)
- if parsedvalue != expected:
- print("<<< FAIL")
- failed += 1
- else:
- print('')
+ (comparisonop, 2, opAssoc.LEFT, EvalComparisonOp),
+ ],
+)
+
+
+# sample expressions posted on comp.lang.python, asking for advice
+# in safely evaluating them
+rules = [
+ "( A - B ) = 0",
+ "( B - C + B ) = 0",
+ "(A + B + C + D + E + F + G + H + I) = J",
+ "(A + B + C + D + E + F + G + H) = I",
+ "(A + B + C + D + E + F) = G",
+ "(A + B + C + D + E) = (F + G + H + I + J)",
+ "(A + B + C + D + E) = (F + G + H + I)",
+ "(A + B + C + D + E) = F",
+ "(A + B + C + D) = (E + F + G + H)",
+ "(A + B + C) = D",
+ "(A + B + C) = (D + E + F)",
+ "(A + B) = (C + D + E + F)",
+ "(A + B) = (C + D)",
+ "(A + B) = (C - D + E - F - G + H + I + J)",
+ "(A + B) = C",
+ "(A + B) = 0",
+ "(A+B+C+D+E) = (F+G+H+I+J)",
+ "(A+B+C+D) = (E+F+G+H)",
+ "(A+B+C+D)=(E+F+G+H)",
+ "(A+B+C)=(D+E+F)",
+ "(A+B)=(C+D)",
+ "(A+B)=C",
+ "(A-B)=C",
+ "(A/(B+C))",
+ "(B/(C+D))",
+ "(G + H) = I",
+ "-0.99 LE ((A+B+C)-(D+E+F+G)) LE 0.99",
+ "-0.99 LE (A-(B+C)) LE 0.99",
+ "-1000.00 LE A LE 0.00",
+ "-5000.00 LE A LE 0.00",
+ "A < B",
+ "A < 7000",
+ "A = -(B)",
+ "A = C",
+ "A = 0",
+ "A GT 0",
+ "A GT 0.00",
+ "A GT 7.00",
+ "A LE B",
+ "A LT -1000.00",
+ "A LT -5000",
+ "A LT 0",
+ "G=(B+C+D)",
+ "A=B",
+ "I = (G + H)",
+ "0.00 LE A LE 4.00",
+ "4.00 LT A LE 7.00",
+ "0.00 LE A LE 4.00 LE E > D",
+ "2**2**(A+3)",
+]
+vars_ = {
+ "A": 0,
+ "B": 1.1,
+ "C": 2.2,
+ "D": 3.3,
+ "E": 4.4,
+ "F": 5.5,
+ "G": 6.6,
+ "H": 7.7,
+ "I": 8.8,
+ "J": 9.9,
+}
+
+# define tests from given rules
+tests = []
+for t in rules:
+ t_orig = t
+ t = t.replace("=", "==")
+ t = t.replace("EQ", "==")
+ t = t.replace("LE", "<=")
+ t = t.replace("GT", ">")
+ t = t.replace("LT", "<")
+ t = t.replace("GE", ">=")
+ t = t.replace("LE", "<=")
+ t = t.replace("NE", "!=")
+ t = t.replace("<>", "!=")
+ tests.append((t_orig, eval(t, vars_)))
- print('')
- if failed:
- print(failed, "tests FAILED")
- return 1
+# copy vars_ to EvalConstant lookup dict
+EvalConstant.vars_ = vars_
+failed = 0
+for test, expected in tests:
+ ret = comp_expr.parseString(test)[0]
+ parsedvalue = ret.eval()
+ print(test, expected, parsedvalue)
+ if abs(parsedvalue - expected) > 1e-6:
+ print("<<< FAIL")
+ failed += 1
else:
- print("all tests PASSED")
- return 0
+ print("")
-if __name__=='__main__':
- exit(main())
+print("")
+if failed:
+ raise Exception("could not parse")
#\r
# A partial implementation of a parser of Excel formula expressions.\r
#\r
-from pyparsing import (CaselessKeyword, Suppress, Word, alphas,\r
- alphanums, nums, Optional, Group, oneOf, Forward,\r
- infixNotation, opAssoc, dblQuotedString, delimitedList,\r
- Combine, Literal, QuotedString, ParserElement, pyparsing_common as ppc)\r
+from pyparsing import (\r
+ CaselessKeyword,\r
+ Suppress,\r
+ Word,\r
+ alphas,\r
+ alphanums,\r
+ nums,\r
+ Optional,\r
+ Group,\r
+ oneOf,\r
+ Forward,\r
+ infixNotation,\r
+ opAssoc,\r
+ dblQuotedString,\r
+ delimitedList,\r
+ Combine,\r
+ Literal,\r
+ QuotedString,\r
+ ParserElement,\r
+ pyparsing_common as ppc,\r
+)\r
+\r
ParserElement.enablePackrat()\r
\r
-EQ,LPAR,RPAR,COLON,COMMA = map(Suppress, '=():,')\r
-EXCL, DOLLAR = map(Literal,"!$")\r
-sheetRef = Word(alphas, alphanums) | QuotedString("'",escQuote="''")\r
-colRef = Optional(DOLLAR) + Word(alphas,max=2)\r
+EQ, LPAR, RPAR, COLON, COMMA = map(Suppress, "=():,")\r
+EXCL, DOLLAR = map(Literal, "!$")\r
+sheetRef = Word(alphas, alphanums) | QuotedString("'", escQuote="''")\r
+colRef = Optional(DOLLAR) + Word(alphas, max=2)\r
rowRef = Optional(DOLLAR) + Word(nums)\r
-cellRef = Combine(Group(Optional(sheetRef + EXCL)("sheet") + colRef("col") +\r
- rowRef("row")))\r
+cellRef = Combine(\r
+ Group(Optional(sheetRef + EXCL)("sheet") + colRef("col") + rowRef("row"))\r
+)\r
\r
-cellRange = (Group(cellRef("start") + COLON + cellRef("end"))("range")\r
- | cellRef | Word(alphas,alphanums))\r
+cellRange = (\r
+ Group(cellRef("start") + COLON + cellRef("end"))("range")\r
+ | cellRef\r
+ | Word(alphas, alphanums)\r
+)\r
\r
expr = Forward()\r
\r
COMPARISON_OP = oneOf("< = > >= <= != <>")\r
condExpr = expr + COMPARISON_OP + expr\r
\r
-ifFunc = (CaselessKeyword("if")\r
- - LPAR\r
- + Group(condExpr)("condition")\r
- + COMMA + Group(expr)("if_true")\r
- + COMMA + Group(expr)("if_false")\r
- + RPAR)\r
+ifFunc = (\r
+ CaselessKeyword("if")\r
+ - LPAR\r
+ + Group(condExpr)("condition")\r
+ + COMMA\r
+ + Group(expr)("if_true")\r
+ + COMMA\r
+ + Group(expr)("if_false")\r
+ + RPAR\r
+)\r
+\r
+\r
+def stat_function(name):\r
+ return Group(CaselessKeyword(name) + Group(LPAR + delimitedList(expr) + RPAR))\r
+\r
\r
-statFunc = lambda name : Group(CaselessKeyword(name) + Group(LPAR + delimitedList(expr) + RPAR))\r
-sumFunc = statFunc("sum")\r
-minFunc = statFunc("min")\r
-maxFunc = statFunc("max")\r
-aveFunc = statFunc("ave")\r
+sumFunc = stat_function("sum")\r
+minFunc = stat_function("min")\r
+maxFunc = stat_function("max")\r
+aveFunc = stat_function("ave")\r
funcCall = ifFunc | sumFunc | minFunc | maxFunc | aveFunc\r
\r
multOp = oneOf("* /")\r
addOp = oneOf("+ -")\r
numericLiteral = ppc.number\r
operand = numericLiteral | funcCall | cellRange | cellRef\r
-arithExpr = infixNotation(operand,\r
+arithExpr = infixNotation(\r
+ operand,\r
[\r
- (multOp, 2, opAssoc.LEFT),\r
- (addOp, 2, opAssoc.LEFT),\r
- ])\r
+ (multOp, 2, opAssoc.LEFT),\r
+ (addOp, 2, opAssoc.LEFT),\r
+ ],\r
+)\r
\r
textOperand = dblQuotedString | cellRef\r
-textExpr = infixNotation(textOperand,\r
+textExpr = infixNotation(\r
+ textOperand,\r
[\r
- ('&', 2, opAssoc.LEFT),\r
- ])\r
+ ("&", 2, opAssoc.LEFT),\r
+ ],\r
+)\r
\r
-expr << (arithExpr | textExpr)\r
+expr <<= arithExpr | textExpr\r
\r
\r
-(EQ + expr).runTests("""\\r
+(EQ + expr).runTests(\r
+ """\\r
=3*A7+5\r
=3*Sheet1!$A$7+5\r
- =3*'Sheet 1'!$A$7+5"\r
+ =3*'Sheet 1'!$A$7+5\r
=3*'O''Reilly''s sheet'!$A$7+5\r
=if(Sum(A1:A25)>42,Min(B1:B25),if(Sum(C1:C25)>3.14, (Min(C1:C25)+3)*18,Max(B1:B25)))\r
=sum(a1:a25,10,min(b1,c2,d3))\r
=if("T"&a2="TTime", "Ready", "Not ready")\r
-""")\r
+"""\r
+)\r
-# fourFn.py\r
-#\r
-# Demonstration of the pyparsing module, implementing a simple 4-function expression parser,\r
-# with support for scientific notation, and symbols for e and pi.\r
-# Extended to add exponentiation and simple built-in functions.\r
-# Extended test cases, simplified pushFirst method.\r
-# Removed unnecessary expr.suppress() call (thanks Nathaniel Peterson!), and added Group\r
-# Changed fnumber to use a Regex, which is now the preferred method\r
-#\r
-# Copyright 2003-2009 by Paul McGuire\r
-#\r
-from pyparsing import Literal,Word,Group,\\r
- ZeroOrMore,Forward,alphas,alphanums,Regex,ParseException,\\r
- CaselessKeyword, Suppress\r
-import math\r
-import operator\r
-\r
-exprStack = []\r
-\r
-def pushFirst( strg, loc, toks ):\r
- exprStack.append( toks[0] )\r
-def pushUMinus( strg, loc, toks ):\r
- for t in toks:\r
- if t == '-':\r
- exprStack.append( 'unary -' )\r
- #~ exprStack.append( '-1' )\r
- #~ exprStack.append( '*' )\r
- else:\r
- break\r
-\r
-bnf = None\r
-def BNF():\r
- """\r
- expop :: '^'\r
- multop :: '*' | '/'\r
- addop :: '+' | '-'\r
- integer :: ['+' | '-'] '0'..'9'+\r
- atom :: PI | E | real | fn '(' expr ')' | '(' expr ')'\r
- factor :: atom [ expop factor ]*\r
- term :: factor [ multop factor ]*\r
- expr :: term [ addop term ]*\r
- """\r
- global bnf\r
- if not bnf:\r
- point = Literal( "." )\r
- # use CaselessKeyword for e and pi, to avoid accidentally matching\r
- # functions that start with 'e' or 'pi' (such as 'exp'); Keyword\r
- # and CaselessKeyword only match whole words\r
- e = CaselessKeyword( "E" )\r
- pi = CaselessKeyword( "PI" )\r
- #~ fnumber = Combine( Word( "+-"+nums, nums ) +\r
- #~ Optional( point + Optional( Word( nums ) ) ) +\r
- #~ Optional( e + Word( "+-"+nums, nums ) ) )\r
- fnumber = Regex(r"[+-]?\d+(?:\.\d*)?(?:[eE][+-]?\d+)?")\r
- ident = Word(alphas, alphanums+"_$")\r
-\r
- plus, minus, mult, div = map(Literal, "+-*/")\r
- lpar, rpar = map(Suppress, "()")\r
- addop = plus | minus\r
- multop = mult | div\r
- expop = Literal( "^" )\r
-\r
- expr = Forward()\r
- atom = ((0,None)*minus + ( pi | e | fnumber | ident + lpar + expr + rpar | ident ).setParseAction( pushFirst ) |\r
- Group( lpar + expr + rpar )).setParseAction(pushUMinus)\r
-\r
- # by defining exponentiation as "atom [ ^ factor ]..." instead of "atom [ ^ atom ]...", we get right-to-left exponents, instead of left-to-righ\r
- # that is, 2^3^2 = 2^(3^2), not (2^3)^2.\r
- factor = Forward()\r
- factor << atom + ZeroOrMore( ( expop + factor ).setParseAction( pushFirst ) )\r
-\r
- term = factor + ZeroOrMore( ( multop + factor ).setParseAction( pushFirst ) )\r
- expr << term + ZeroOrMore( ( addop + term ).setParseAction( pushFirst ) )\r
- bnf = expr\r
- return bnf\r
-\r
-# map operator symbols to corresponding arithmetic operations\r
-epsilon = 1e-12\r
-opn = { "+" : operator.add,\r
- "-" : operator.sub,\r
- "*" : operator.mul,\r
- "/" : operator.truediv,\r
- "^" : operator.pow }\r
-fn = { "sin" : math.sin,\r
- "cos" : math.cos,\r
- "tan" : math.tan,\r
- "exp" : math.exp,\r
- "abs" : abs,\r
- "trunc" : lambda a: int(a),\r
- "round" : round,\r
- "sgn" : lambda a: (a > epsilon) - (a < -epsilon) }\r
-def evaluateStack( s ):\r
- op = s.pop()\r
- if op == 'unary -':\r
- return -evaluateStack( s )\r
- if op in "+-*/^":\r
- op2 = evaluateStack( s )\r
- op1 = evaluateStack( s )\r
- return opn[op]( op1, op2 )\r
- elif op == "PI":\r
- return math.pi # 3.1415926535\r
- elif op == "E":\r
- return math.e # 2.718281828\r
- elif op in fn:\r
- return fn[op]( evaluateStack( s ) )\r
- elif op[0].isalpha():\r
- raise Exception("invalid identifier '%s'" % op)\r
- else:\r
- return float( op )\r
-\r
-if __name__ == "__main__":\r
-\r
- def test( s, expVal ):\r
- global exprStack\r
- exprStack[:] = []\r
- try:\r
- results = BNF().parseString( s, parseAll=True )\r
- val = evaluateStack( exprStack[:] )\r
- except ParseException as pe:\r
- print(s, "failed parse:", str(pe))\r
- except Exception as e:\r
- print(s, "failed eval:", str(e))\r
- else:\r
- if val == expVal:\r
- print(s, "=", val, results, "=>", exprStack)\r
- else:\r
- print(s+"!!!", val, "!=", expVal, results, "=>", exprStack)\r
-\r
- test( "9", 9 )\r
- test( "-9", -9 )\r
- test( "--9", 9 )\r
- test( "-E", -math.e )\r
- test( "9 + 3 + 6", 9 + 3 + 6 )\r
- test( "9 + 3 / 11", 9 + 3.0 / 11 )\r
- test( "(9 + 3)", (9 + 3) )\r
- test( "(9+3) / 11", (9+3.0) / 11 )\r
- test( "9 - 12 - 6", 9 - 12 - 6 )\r
- test( "9 - (12 - 6)", 9 - (12 - 6) )\r
- test( "2*3.14159", 2*3.14159 )\r
- test( "3.1415926535*3.1415926535 / 10", 3.1415926535*3.1415926535 / 10 )\r
- test( "PI * PI / 10", math.pi * math.pi / 10 )\r
- test( "PI*PI/10", math.pi*math.pi/10 )\r
- test( "PI^2", math.pi**2 )\r
- test( "round(PI^2)", round(math.pi**2) )\r
- test( "6.02E23 * 8.048", 6.02E23 * 8.048 )\r
- test( "e / 3", math.e / 3 )\r
- test( "sin(PI/2)", math.sin(math.pi/2) )\r
- test( "trunc(E)", int(math.e) )\r
- test( "trunc(-E)", int(-math.e) )\r
- test( "round(E)", round(math.e) )\r
- test( "round(-E)", round(-math.e) )\r
- test( "E^PI", math.e**math.pi )\r
- test( "exp(0)", 1 )\r
- test( "exp(1)", math.e )\r
- test( "2^3^2", 2**3**2 )\r
- test( "2^3+2", 2**3+2 )\r
- test( "2^3+5", 2**3+5 )\r
- test( "2^9", 2**9 )\r
- test( "sgn(-2)", -1 )\r
- test( "sgn(0)", 0 )\r
- test( "foo(0.1)", None )\r
- test( "sgn(0.1)", 1 )\r
-\r
-\r
-"""\r
-Test output:\r
->pythonw -u fourFn.py\r
-9 = 9.0 ['9'] => ['9']\r
-9 + 3 + 6 = 18.0 ['9', '+', '3', '+', '6'] => ['9', '3', '+', '6', '+']\r
-9 + 3 / 11 = 9.27272727273 ['9', '+', '3', '/', '11'] => ['9', '3', '11', '/', '+']\r
-(9 + 3) = 12.0 [] => ['9', '3', '+']\r
-(9+3) / 11 = 1.09090909091 ['/', '11'] => ['9', '3', '+', '11', '/']\r
-9 - 12 - 6 = -9.0 ['9', '-', '12', '-', '6'] => ['9', '12', '-', '6', '-']\r
-9 - (12 - 6) = 3.0 ['9', '-'] => ['9', '12', '6', '-', '-']\r
-2*3.14159 = 6.28318 ['2', '*', '3.14159'] => ['2', '3.14159', '*']\r
-3.1415926535*3.1415926535 / 10 = 0.986960440053 ['3.1415926535', '*', '3.1415926535', '/', '10'] => ['3.1415926535', '3.1415926535', '*', '10', '/']\r
-PI * PI / 10 = 0.986960440109 ['PI', '*', 'PI', '/', '10'] => ['PI', 'PI', '*', '10', '/']\r
-PI*PI/10 = 0.986960440109 ['PI', '*', 'PI', '/', '10'] => ['PI', 'PI', '*', '10', '/']\r
-PI^2 = 9.86960440109 ['PI', '^', '2'] => ['PI', '2', '^']\r
-6.02E23 * 8.048 = 4.844896e+024 ['6.02E23', '*', '8.048'] => ['6.02E23', '8.048', '*']\r
-e / 3 = 0.90609394282 ['E', '/', '3'] => ['E', '3', '/']\r
-sin(PI/2) = 1.0 ['sin', 'PI', '/', '2'] => ['PI', '2', '/', 'sin']\r
-trunc(E) = 2 ['trunc', 'E'] => ['E', 'trunc']\r
-E^PI = 23.1406926328 ['E', '^', 'PI'] => ['E', 'PI', '^']\r
-2^3^2 = 512.0 ['2', '^', '3', '^', '2'] => ['2', '3', '2', '^', '^']\r
-2^3+2 = 10.0 ['2', '^', '3', '+', '2'] => ['2', '3', '^', '2', '+']\r
-2^9 = 512.0 ['2', '^', '9'] => ['2', '9', '^']\r
-sgn(-2) = -1 ['sgn', '-2'] => ['-2', 'sgn']\r
-sgn(0) = 0 ['sgn', '0'] => ['0', 'sgn']\r
-sgn(0.1) = 1 ['sgn', '0.1'] => ['0.1', 'sgn']\r
->Exit code: 0\r
-"""\r
+# fourFn.py
+#
+# Demonstration of the pyparsing module, implementing a simple 4-function expression parser,
+# with support for scientific notation, and symbols for e and pi.
+# Extended to add exponentiation and simple built-in functions.
+# Extended test cases, simplified pushFirst method.
+# Removed unnecessary expr.suppress() call (thanks Nathaniel Peterson!), and added Group
+# Changed fnumber to use a Regex, which is now the preferred method
+# Reformatted to latest pypyparsing features, support multiple and variable args to functions
+#
+# Copyright 2003-2019 by Paul McGuire
+#
+from pyparsing import (
+ Literal,
+ Word,
+ Group,
+ Forward,
+ alphas,
+ alphanums,
+ Regex,
+ ParseException,
+ CaselessKeyword,
+ Suppress,
+ delimitedList,
+)
+import math
+import operator
+
+exprStack = []
+
+
+def push_first(toks):
+ exprStack.append(toks[0])
+
+
+def push_unary_minus(toks):
+ for t in toks:
+ if t == "-":
+ exprStack.append("unary -")
+ else:
+ break
+
+
+bnf = None
+
+
+def BNF():
+ """
+ expop :: '^'
+ multop :: '*' | '/'
+ addop :: '+' | '-'
+ integer :: ['+' | '-'] '0'..'9'+
+ atom :: PI | E | real | fn '(' expr ')' | '(' expr ')'
+ factor :: atom [ expop factor ]*
+ term :: factor [ multop factor ]*
+ expr :: term [ addop term ]*
+ """
+ global bnf
+ if not bnf:
+ # use CaselessKeyword for e and pi, to avoid accidentally matching
+ # functions that start with 'e' or 'pi' (such as 'exp'); Keyword
+ # and CaselessKeyword only match whole words
+ e = CaselessKeyword("E")
+ pi = CaselessKeyword("PI")
+ # fnumber = Combine(Word("+-"+nums, nums) +
+ # Optional("." + Optional(Word(nums))) +
+ # Optional(e + Word("+-"+nums, nums)))
+ # or use provided pyparsing_common.number, but convert back to str:
+ # fnumber = ppc.number().addParseAction(lambda t: str(t[0]))
+ fnumber = Regex(r"[+-]?\d+(?:\.\d*)?(?:[eE][+-]?\d+)?")
+ ident = Word(alphas, alphanums + "_$")
+
+ plus, minus, mult, div = map(Literal, "+-*/")
+ lpar, rpar = map(Suppress, "()")
+ addop = plus | minus
+ multop = mult | div
+ expop = Literal("^")
+
+ expr = Forward()
+ expr_list = delimitedList(Group(expr))
+ # add parse action that replaces the function identifier with a (name, number of args) tuple
+ def insert_fn_argcount_tuple(t):
+ fn = t.pop(0)
+ num_args = len(t[0])
+ t.insert(0, (fn, num_args))
+
+ fn_call = (ident + lpar - Group(expr_list) + rpar).setParseAction(
+ insert_fn_argcount_tuple
+ )
+ atom = (
+ addop[...]
+ + (
+ (fn_call | pi | e | fnumber | ident).setParseAction(push_first)
+ | Group(lpar + expr + rpar)
+ )
+ ).setParseAction(push_unary_minus)
+
+ # by defining exponentiation as "atom [ ^ factor ]..." instead of "atom [ ^ atom ]...", we get right-to-left
+ # exponents, instead of left-to-right that is, 2^3^2 = 2^(3^2), not (2^3)^2.
+ factor = Forward()
+ factor <<= atom + (expop + factor).setParseAction(push_first)[...]
+ term = factor + (multop + factor).setParseAction(push_first)[...]
+ expr <<= term + (addop + term).setParseAction(push_first)[...]
+ bnf = expr
+ return bnf
+
+
+# map operator symbols to corresponding arithmetic operations
+epsilon = 1e-12
+opn = {
+ "+": operator.add,
+ "-": operator.sub,
+ "*": operator.mul,
+ "/": operator.truediv,
+ "^": operator.pow,
+}
+
+fn = {
+ "sin": math.sin,
+ "cos": math.cos,
+ "tan": math.tan,
+ "exp": math.exp,
+ "abs": abs,
+ "trunc": int,
+ "round": round,
+ "sgn": lambda a: -1 if a < -epsilon else 1 if a > epsilon else 0,
+ # functionsl with multiple arguments
+ "multiply": lambda a, b: a * b,
+ "hypot": math.hypot,
+ # functions with a variable number of arguments
+ "all": lambda *a: all(a),
+}
+
+
+def evaluate_stack(s):
+ op, num_args = s.pop(), 0
+ if isinstance(op, tuple):
+ op, num_args = op
+ if op == "unary -":
+ return -evaluate_stack(s)
+ if op in "+-*/^":
+ # note: operands are pushed onto the stack in reverse order
+ op2 = evaluate_stack(s)
+ op1 = evaluate_stack(s)
+ return opn[op](op1, op2)
+ elif op == "PI":
+ return math.pi # 3.1415926535
+ elif op == "E":
+ return math.e # 2.718281828
+ elif op in fn:
+ # note: args are pushed onto the stack in reverse order
+ args = reversed([evaluate_stack(s) for _ in range(num_args)])
+ return fn[op](*args)
+ elif op[0].isalpha():
+ raise Exception("invalid identifier '%s'" % op)
+ else:
+ # try to evaluate as int first, then as float if int fails
+ try:
+ return int(op)
+ except ValueError:
+ return float(op)
+
+
+if __name__ == "__main__":
+
+ def test(s, expected):
+ exprStack[:] = []
+ try:
+ results = BNF().parseString(s, parseAll=True)
+ val = evaluate_stack(exprStack[:])
+ except ParseException as pe:
+ print(s, "failed parse:", str(pe))
+ except Exception as e:
+ print(s, "failed eval:", str(e), exprStack)
+ else:
+ if val == expected:
+ print(s, "=", val, results, "=>", exprStack)
+ else:
+ print(s + "!!!", val, "!=", expected, results, "=>", exprStack)
+
+ test("9", 9)
+ test("-9", -9)
+ test("--9", 9)
+ test("-E", -math.e)
+ test("9 + 3 + 6", 9 + 3 + 6)
+ test("9 + 3 / 11", 9 + 3.0 / 11)
+ test("(9 + 3)", (9 + 3))
+ test("(9+3) / 11", (9 + 3.0) / 11)
+ test("9 - 12 - 6", 9 - 12 - 6)
+ test("9 - (12 - 6)", 9 - (12 - 6))
+ test("2*3.14159", 2 * 3.14159)
+ test("3.1415926535*3.1415926535 / 10", 3.1415926535 * 3.1415926535 / 10)
+ test("PI * PI / 10", math.pi * math.pi / 10)
+ test("PI*PI/10", math.pi * math.pi / 10)
+ test("PI^2", math.pi ** 2)
+ test("round(PI^2)", round(math.pi ** 2))
+ test("6.02E23 * 8.048", 6.02e23 * 8.048)
+ test("e / 3", math.e / 3)
+ test("sin(PI/2)", math.sin(math.pi / 2))
+ test("10+sin(PI/4)^2", 10 + math.sin(math.pi / 4) ** 2)
+ test("trunc(E)", int(math.e))
+ test("trunc(-E)", int(-math.e))
+ test("round(E)", round(math.e))
+ test("round(-E)", round(-math.e))
+ test("E^PI", math.e ** math.pi)
+ test("exp(0)", 1)
+ test("exp(1)", math.e)
+ test("2^3^2", 2 ** 3 ** 2)
+ test("(2^3)^2", (2 ** 3) ** 2)
+ test("2^3+2", 2 ** 3 + 2)
+ test("2^3+5", 2 ** 3 + 5)
+ test("2^9", 2 ** 9)
+ test("sgn(-2)", -1)
+ test("sgn(0)", 0)
+ test("sgn(0.1)", 1)
+ test("foo(0.1)", None)
+ test("round(E, 3)", round(math.e, 3))
+ test("round(PI^2, 3)", round(math.pi ** 2, 3))
+ test("sgn(cos(PI/4))", 1)
+ test("sgn(cos(PI/2))", 0)
+ test("sgn(cos(PI*3/4))", -1)
+ test("+(sgn(cos(PI/4)))", 1)
+ test("-(sgn(cos(PI/4)))", -1)
+ test("hypot(3, 4)", 5)
+ test("multiply(3, 7)", 21)
+ test("all(1,1,1)", True)
+ test("all(1,1,1,1,1,0)", False)
+
+
+"""
+Test output:
+>python fourFn.py
+9 = 9 ['9'] => ['9']
+-9 = -9 ['-', '9'] => ['9', 'unary -']
+--9 = 9 ['-', '-', '9'] => ['9', 'unary -', 'unary -']
+-E = -2.718281828459045 ['-', 'E'] => ['E', 'unary -']
+9 + 3 + 6 = 18 ['9', '+', '3', '+', '6'] => ['9', '3', '+', '6', '+']
+9 + 3 / 11 = 9.272727272727273 ['9', '+', '3', '/', '11'] => ['9', '3', '11', '/', '+']
+(9 + 3) = 12 [['9', '+', '3']] => ['9', '3', '+']
+(9+3) / 11 = 1.0909090909090908 [['9', '+', '3'], '/', '11'] => ['9', '3', '+', '11', '/']
+9 - 12 - 6 = -9 ['9', '-', '12', '-', '6'] => ['9', '12', '-', '6', '-']
+9 - (12 - 6) = 3 ['9', '-', ['12', '-', '6']] => ['9', '12', '6', '-', '-']
+2*3.14159 = 6.28318 ['2', '*', '3.14159'] => ['2', '3.14159', '*']
+3.1415926535*3.1415926535 / 10 = 0.9869604400525172 ['3.1415926535', '*', '3.1415926535', '/', '10'] => ['3.1415926535', '3.1415926535', '*', '10', '/']
+PI * PI / 10 = 0.9869604401089358 ['PI', '*', 'PI', '/', '10'] => ['PI', 'PI', '*', '10', '/']
+PI*PI/10 = 0.9869604401089358 ['PI', '*', 'PI', '/', '10'] => ['PI', 'PI', '*', '10', '/']
+PI^2 = 9.869604401089358 ['PI', '^', '2'] => ['PI', '2', '^']
+round(PI^2) = 10 [('round', 1), [['PI', '^', '2']]] => ['PI', '2', '^', ('round', 1)]
+6.02E23 * 8.048 = 4.844896e+24 ['6.02E23', '*', '8.048'] => ['6.02E23', '8.048', '*']
+e / 3 = 0.9060939428196817 ['E', '/', '3'] => ['E', '3', '/']
+sin(PI/2) = 1.0 [('sin', 1), [['PI', '/', '2']]] => ['PI', '2', '/', ('sin', 1)]
+10+sin(PI/4)^2 = 10.5 ['10', '+', ('sin', 1), [['PI', '/', '4']], '^', '2'] => ['10', 'PI', '4', '/', ('sin', 1), '2', '^', '+']
+trunc(E) = 2 [('trunc', 1), [['E']]] => ['E', ('trunc', 1)]
+trunc(-E) = -2 [('trunc', 1), [['-', 'E']]] => ['E', 'unary -', ('trunc', 1)]
+round(E) = 3 [('round', 1), [['E']]] => ['E', ('round', 1)]
+round(-E) = -3 [('round', 1), [['-', 'E']]] => ['E', 'unary -', ('round', 1)]
+E^PI = 23.140692632779263 ['E', '^', 'PI'] => ['E', 'PI', '^']
+exp(0) = 1.0 [('exp', 1), [['0']]] => ['0', ('exp', 1)]
+exp(1) = 2.718281828459045 [('exp', 1), [['1']]] => ['1', ('exp', 1)]
+2^3^2 = 512 ['2', '^', '3', '^', '2'] => ['2', '3', '2', '^', '^']
+(2^3)^2 = 64 [['2', '^', '3'], '^', '2'] => ['2', '3', '^', '2', '^']
+2^3+2 = 10 ['2', '^', '3', '+', '2'] => ['2', '3', '^', '2', '+']
+2^3+5 = 13 ['2', '^', '3', '+', '5'] => ['2', '3', '^', '5', '+']
+2^9 = 512 ['2', '^', '9'] => ['2', '9', '^']
+sgn(-2) = -1 [('sgn', 1), [['-', '2']]] => ['2', 'unary -', ('sgn', 1)]
+sgn(0) = 0 [('sgn', 1), [['0']]] => ['0', ('sgn', 1)]
+sgn(0.1) = 1 [('sgn', 1), [['0.1']]] => ['0.1', ('sgn', 1)]
+foo(0.1) failed eval: invalid identifier 'foo' ['0.1', ('foo', 1)]
+round(E, 3) = 2.718 [('round', 2), [['E'], ['3']]] => ['E', '3', ('round', 2)]
+round(PI^2, 3) = 9.87 [('round', 2), [['PI', '^', '2'], ['3']]] => ['PI', '2', '^', '3', ('round', 2)]
+sgn(cos(PI/4)) = 1 [('sgn', 1), [[('cos', 1), [['PI', '/', '4']]]]] => ['PI', '4', '/', ('cos', 1), ('sgn', 1)]
+sgn(cos(PI/2)) = 0 [('sgn', 1), [[('cos', 1), [['PI', '/', '2']]]]] => ['PI', '2', '/', ('cos', 1), ('sgn', 1)]
+sgn(cos(PI*3/4)) = -1 [('sgn', 1), [[('cos', 1), [['PI', '*', '3', '/', '4']]]]] => ['PI', '3', '*', '4', '/', ('cos', 1), ('sgn', 1)]
++(sgn(cos(PI/4))) = 1 ['+', [('sgn', 1), [[('cos', 1), [['PI', '/', '4']]]]]] => ['PI', '4', '/', ('cos', 1), ('sgn', 1)]
+-(sgn(cos(PI/4))) = -1 ['-', [('sgn', 1), [[('cos', 1), [['PI', '/', '4']]]]]] => ['PI', '4', '/', ('cos', 1), ('sgn', 1), 'unary -']
+"""
from pyparsing import *\r
\r
typemap = {\r
- "byte" : "c_byte",\r
- "char" : "c_char",\r
- "char *" : "c_char_p",\r
- "double" : "c_double",\r
- "float" : "c_float",\r
- "int" : "c_int",\r
- "int16" : "c_int16",\r
- "int32" : "c_int32",\r
- "int64" : "c_int64",\r
- "int8" : "c_int8",\r
- "long" : "c_long",\r
- "longlong" : "c_longlong",\r
- "short" : "c_short",\r
- "size_t" : "c_size_t",\r
- "ubyte" : "c_ubyte",\r
- "uchar" : "c_ubyte",\r
- "u_char" : "c_ubyte",\r
- "uint" : "c_uint",\r
- "u_int" : "c_uint",\r
- "uint16" : "c_uint16",\r
- "uint32" : "c_uint32",\r
- "uint64" : "c_uint64",\r
- "uint8" : "c_uint8",\r
- "u_long" : "c_ulong",\r
- "ulong" : "c_ulong",\r
- "ulonglong" : "c_ulonglong",\r
- "ushort" : "c_ushort",\r
- "u_short" : "c_ushort",\r
- "void *" : "c_void_p",\r
- "voidp" : "c_voidp",\r
- "wchar" : "c_wchar",\r
- "wchar *" : "c_wchar_p",\r
- "Bool" : "c_bool",\r
- "void" : "None",\r
- }\r
-\r
-LPAR,RPAR,LBRACE,RBRACE,COMMA,SEMI = map(Suppress,"(){},;")\r
+ "byte": "c_byte",\r
+ "char": "c_char",\r
+ "char *": "c_char_p",\r
+ "double": "c_double",\r
+ "float": "c_float",\r
+ "int": "c_int",\r
+ "int16": "c_int16",\r
+ "int32": "c_int32",\r
+ "int64": "c_int64",\r
+ "int8": "c_int8",\r
+ "long": "c_long",\r
+ "longlong": "c_longlong",\r
+ "short": "c_short",\r
+ "size_t": "c_size_t",\r
+ "ubyte": "c_ubyte",\r
+ "uchar": "c_ubyte",\r
+ "u_char": "c_ubyte",\r
+ "uint": "c_uint",\r
+ "u_int": "c_uint",\r
+ "uint16": "c_uint16",\r
+ "uint32": "c_uint32",\r
+ "uint64": "c_uint64",\r
+ "uint8": "c_uint8",\r
+ "u_long": "c_ulong",\r
+ "ulong": "c_ulong",\r
+ "ulonglong": "c_ulonglong",\r
+ "ushort": "c_ushort",\r
+ "u_short": "c_ushort",\r
+ "void *": "c_void_p",\r
+ "voidp": "c_voidp",\r
+ "wchar": "c_wchar",\r
+ "wchar *": "c_wchar_p",\r
+ "Bool": "c_bool",\r
+ "void": "None",\r
+}\r
+\r
+LPAR, RPAR, LBRACE, RBRACE, COMMA, SEMI = map(Suppress, "(){},;")\r
ident = Word(alphas, alphanums + "_")\r
integer = Regex(r"[+-]?\d+")\r
hexinteger = Regex(r"0x[0-9a-fA-F]+")\r
const = Suppress("const")\r
primitiveType = oneOf(t for t in typemap if not t.endswith("*"))\r
structType = Suppress("struct") + ident\r
-vartype = (Optional(const) +\r
- (primitiveType | structType | ident) +\r
- Optional(Word("*")("ptr")))\r
+vartype = (\r
+ Optional(const) + (primitiveType | structType | ident) + Optional(Word("*")("ptr"))\r
+)\r
+\r
+\r
def normalizetype(t):\r
if isinstance(t, ParseResults):\r
- return ' '.join(t)\r
- #~ ret = ParseResults([' '.join(t)])\r
- #~ return ret\r
+ return " ".join(t)\r
+ # ~ ret = ParseResults([' '.join(t)])\r
+ # ~ return ret\r
+\r
\r
vartype.setParseAction(normalizetype)\r
\r
arg = Group(vartype("argtype") + Optional(ident("argname")))\r
-func_def = (vartype("fn_type") + ident("fn_name") +\r
- LPAR + Optional(delimitedList(arg|"..."))("fn_args") + RPAR + SEMI)\r
+func_def = (\r
+ vartype("fn_type")\r
+ + ident("fn_name")\r
+ + LPAR\r
+ + Optional(delimitedList(arg | "..."))("fn_args")\r
+ + RPAR\r
+ + SEMI\r
+)\r
+\r
+\r
def derivefields(t):\r
if t.fn_args and t.fn_args[-1] == "...":\r
- t["varargs"]=True\r
+ t["varargs"] = True\r
+\r
+\r
func_def.setParseAction(derivefields)\r
\r
fn_typedef = "typedef" + func_def\r
var_typedef = "typedef" + primitiveType("primType") + ident("name") + SEMI\r
\r
-enum_def = (Keyword("enum") + LBRACE +\r
- delimitedList(Group(ident("name") + '=' + (hexinteger|integer)("value")))("evalues")\r
- + Optional(COMMA)\r
- + RBRACE)\r
+enum_def = (\r
+ Keyword("enum")\r
+ + LBRACE\r
+ + delimitedList(Group(ident("name") + "=" + (hexinteger | integer)("value")))(\r
+ "evalues"\r
+ )\r
+ + Optional(COMMA)\r
+ + RBRACE\r
+)\r
\r
c_header = open("snmp_api.h").read()\r
\r
enum_constants = []\r
\r
# add structures commonly included from std lib headers\r
-def addStdType(t,namespace=""):\r
- fullname = namespace+'_'+t if namespace else t\r
+def addStdType(t, namespace=""):\r
+ fullname = namespace + "_" + t if namespace else t\r
typemap[t] = fullname\r
user_defined_types.add(t)\r
+\r
+\r
addStdType("fd_set", "sys_select")\r
addStdType("timeval", "sys_time")\r
\r
+\r
def getUDType(typestr):\r
key = typestr.rstrip(" *")\r
if key not in typemap:\r
user_defined_types.add(key)\r
- typemap[key] = "{0}_{1}".format(module, key)\r
+ typemap[key] = "{}_{}".format(module, key)\r
+\r
\r
def typeAsCtypes(typestr):\r
if typestr in typemap:\r
return "POINTER(%s)" % typeAsCtypes(typestr.rstrip(" *"))\r
return typestr\r
\r
+\r
# scan input header text for primitive typedefs\r
-for td,_,_ in var_typedef.scanString(c_header):\r
- typedefs.append( (td.name, td.primType) )\r
+for td, _, _ in var_typedef.scanString(c_header):\r
+ typedefs.append((td.name, td.primType))\r
# add typedef type to typemap to map to itself\r
typemap[td.name] = td.name\r
\r
typemap[fntd.fn_name] = fntd.fn_name\r
\r
# scan input header text, and keep running list of user-defined types\r
-for fn,_,_ in (cStyleComment.suppress() | fn_typedef.suppress() | func_def).scanString(c_header):\r
- if not fn: continue\r
+for fn, _, _ in (\r
+ cStyleComment.suppress() | fn_typedef.suppress() | func_def\r
+).scanString(c_header):\r
+ if not fn:\r
+ continue\r
getUDType(fn.fn_type)\r
for arg in fn.fn_args:\r
if arg != "...":\r
\r
# scan input header text for enums\r
enum_def.ignore(cppStyleComment)\r
-for en_,_,_ in enum_def.scanString(c_header):\r
+for en_, _, _ in enum_def.scanString(c_header):\r
for ev in en_.evalues:\r
- enum_constants.append( (ev.name, ev.value) )\r
+ enum_constants.append((ev.name, ev.value))\r
\r
print("from ctypes import *")\r
-print("{0} = CDLL('{1}.dll')".format(module, module))\r
+print("{} = CDLL('{}.dll')".format(module, module))\r
print()\r
print("# user defined types")\r
-for tdname,tdtyp in typedefs:\r
- print("{0} = {1}".format(tdname, typemap[tdtyp]))\r
+for tdname, tdtyp in typedefs:\r
+ print("{} = {}".format(tdname, typemap[tdtyp]))\r
for fntd in fn_typedefs:\r
- print("{0} = CFUNCTYPE({1})".format(fntd.fn_name,\r
- ',\n '.join(typeAsCtypes(a.argtype) for a in fntd.fn_args)))\r
+ print(\r
+ "{} = CFUNCTYPE({})".format(\r
+ fntd.fn_name, ",\n ".join(typeAsCtypes(a.argtype) for a in fntd.fn_args)\r
+ )\r
+ )\r
for udtype in user_defined_types:\r
print("class %s(Structure): pass" % typemap[udtype])\r
\r
print()\r
print("# constant definitions")\r
-for en,ev in enum_constants:\r
- print("{0} = {1}".format(en,ev))\r
+for en, ev in enum_constants:\r
+ print("{} = {}".format(en, ev))\r
\r
print()\r
print("# functions")\r
for fn in functions:\r
- prefix = "{0}.{1}".format(module, fn.fn_name)\r
+ prefix = "{}.{}".format(module, fn.fn_name)\r
\r
- print("{0}.restype = {1}".format(prefix, typeAsCtypes(fn.fn_type)))\r
+ print("{}.restype = {}".format(prefix, typeAsCtypes(fn.fn_type)))\r
if fn.varargs:\r
print("# warning - %s takes variable argument list" % prefix)\r
del fn.fn_args[-1]\r
\r
- if fn.fn_args.asList() != [['void']]:\r
- print("{0}.argtypes = ({1},)".format(prefix, ','.join(typeAsCtypes(a.argtype) for a in fn.fn_args)))\r
+ if fn.fn_args.asList() != [["void"]]:\r
+ print(\r
+ "{}.argtypes = ({},)".format(\r
+ prefix, ",".join(typeAsCtypes(a.argtype) for a in fn.fn_args)\r
+ )\r
+ )\r
else:\r
print("%s.argtypes = ()" % (prefix))\r
# September, 2010 - updated to more current use of setResultsName, new NIST URL
#
import pyparsing as pp
-ppc = pp.pyparsing_common
-from contextlib import closing
-try:
- import urllib.request
- urlopen = urllib.request.urlopen
-except ImportError:
- import urllib
- urlopen = urllib.urlopen
+ppc = pp.pyparsing_common
+from urllib.request import urlopen
integer = pp.Word(pp.nums)
ipAddress = ppc.ipv4_address()
-hostname = pp.delimitedList(pp.Word(pp.alphas, pp.alphanums+"-_"), ".", combine=True)
+hostname = pp.delimitedList(pp.Word(pp.alphas, pp.alphanums + "-_"), ".", combine=True)
tdStart, tdEnd = pp.makeHTMLTags("td")
-timeServerPattern = (tdStart + hostname("hostname") + tdEnd
- + tdStart + ipAddress("ipAddr") + tdEnd
- + tdStart + tdStart.tag_body("loc") + tdEnd)
+timeServerPattern = (
+ tdStart
+ + hostname("hostname")
+ + tdEnd
+ + tdStart
+ + ipAddress("ipAddr")
+ + tdEnd
+ + tdStart
+ + tdStart.tag_body("loc")
+ + tdEnd
+)
# get list of time servers
nistTimeServerURL = "https://tf.nist.gov/tf-cgi/servers.cgi#"
-with closing(urlopen(nistTimeServerURL)) as serverListPage:
+with urlopen(nistTimeServerURL) as serverListPage:
serverListHTML = serverListPage.read().decode("UTF-8")
addrs = {}
for srvr, startloc, endloc in timeServerPattern.scanString(serverListHTML):
- print("{0} ({1}) - {2}".format(srvr.ipAddr, srvr.hostname.strip(), srvr.loc.strip()))
+ print("{} ({}) - {}".format(srvr.ipAddr, srvr.hostname.strip(), srvr.loc.strip()))
addrs[srvr.ipAddr] = srvr.loc
hello = "Hello, World!"
# parse input string
-print(hello, "->", greet.parseString( hello ))
+print(hello, "->", greet.parseString(hello))
# parse a bunch of input strings
-greet.runTests("""\
+greet.runTests(
+ """\
Hello, World!
Ahoy, Matey!
Howdy, Pardner!
Morning, Neighbor!
- """)
\ No newline at end of file
+ """
+)
-# vim:fileencoding=utf-8\r
#\r
# greetingInGreek.py\r
#\r
\r
# define grammar\r
alphas = ppu.Greek.alphas\r
-greet = Word(alphas) + ',' + Word(alphas) + '!'\r
+greet = Word(alphas) + "," + Word(alphas) + "!"\r
\r
# input string\r
hello = "Καλημέρα, κόσμε!"\r
-# vim:fileencoding=utf-8\r
#\r
# greetingInKorean.py\r
#\r
greet = koreanWord + "," + koreanWord + "!"\r
\r
# input string\r
-hello = '안녕, 여러분!' #"Hello, World!" in Korean\r
+hello = "안녕, 여러분!" # "Hello, World!" in Korean\r
\r
# parse input string\r
print(greet.parseString(hello))\r
-# -*- coding: utf-8 -*-\r
-\r
# escrito por Marco Alfonso, 2004 Noviembre\r
\r
# importamos los símbolos requeridos desde el módulo\r
-from pyparsing import Word, alphas, oneOf, nums, Group, OneOrMore, pyparsing_unicode as ppu\r
+from pyparsing import (\r
+ Word,\r
+ alphas,\r
+ oneOf,\r
+ nums,\r
+ Group,\r
+ OneOrMore,\r
+ pyparsing_unicode as ppu,\r
+)\r
\r
# usamos las letras en latin1, que incluye las como 'ñ', 'á', 'é', etc.\r
alphas = ppu.Latin1.alphas\r
# una palabra compuesta de caracteres alfanumericos\r
# (Word(alphas)) mas una ',' mas otra palabra alfanumerica,\r
# mas '!' y esos seian nuestros tokens\r
-saludo = Word(alphas) + ',' + Word(alphas) + oneOf('! . ?')\r
+saludo = Word(alphas) + "," + Word(alphas) + oneOf("! . ?")\r
tokens = saludo.parseString("Hola, Mundo !")\r
\r
# Ahora parseamos una cadena, "Hola, Mundo!",\r
# el metodo parseString, nos devuelve una lista con los tokens\r
# encontrados, en caso de no haber errores...\r
for i, token in enumerate(tokens):\r
- print ("Token %d -> %s" % (i,token))\r
+ print("Token %d -> %s" % (i, token))\r
\r
-#imprimimos cada uno de los tokens Y listooo!!, he aquí a salida\r
+# imprimimos cada uno de los tokens Y listooo!!, he aquí a salida\r
# Token 0 -> Hola\r
# Token 1 -> ,\r
# Token 2-> Mundo\r
# Token 3 -> !\r
\r
# ahora cambia el parseador, aceptando saludos con mas que una sola palabra antes que ','\r
-saludo = Group(OneOrMore(Word(alphas))) + ',' + Word(alphas) + oneOf('! . ?')\r
+saludo = Group(OneOrMore(Word(alphas))) + "," + Word(alphas) + oneOf("! . ?")\r
tokens = saludo.parseString("Hasta mañana, Mundo !")\r
\r
for i, token in enumerate(tokens):\r
- print ("Token %d -> %s" % (i,token))\r
+ print("Token %d -> %s" % (i, token))\r
\r
# Ahora parseamos algunas cadenas, usando el metodo runTests\r
-saludo.runTests("""\\r
+saludo.runTests(\r
+ """\\r
Hola, Mundo!\r
Hasta mañana, Mundo !\r
-""", fullDump=False)\r
+""",\r
+ fullDump=False,\r
+)\r
\r
# Por supuesto, se pueden "reutilizar" gramáticas, por ejemplo:\r
-numimag = Word(nums) + 'i'\r
+numimag = Word(nums) + "i"\r
numreal = Word(nums)\r
-numcomplex = numreal + '+' + numimag\r
-print (numcomplex.parseString("3+5i"))\r
+numcomplex = numreal + "+" + numimag\r
+print(numcomplex.parseString("3+5i"))\r
+\r
+# Funcion para cambiar a complejo numero durante parsear:\r
+def hace_python_complejo(t):\r
+ valid_python = "".join(t).replace("i", "j")\r
+ return complex(valid_python)\r
+\r
\r
-# Cambiar a complejo numero durante parsear:\r
-numcomplex.setParseAction(lambda t: complex(''.join(t).replace('i','j')))\r
-print (numcomplex.parseString("3+5i"))\r
+numcomplex.setParseAction(hace_python_complejo)\r
+print(numcomplex.parseString("3+5i"))\r
\r
# Excelente!!, bueno, los dejo, me voy a seguir tirando código...\r
#
# Copyright (c) 2006, 2016, Paul McGuire
#
-from contextlib import closing
-import urllib.request, urllib.parse, urllib.error
-from pyparsing import (makeHTMLTags, commonHTMLEntity, replaceHTMLEntity,
- htmlComment, anyOpenTag, anyCloseTag, LineEnd, OneOrMore, replaceWith)
+from urllib.request import urlopen
+from pyparsing import (
+ makeHTMLTags,
+ commonHTMLEntity,
+ replaceHTMLEntity,
+ htmlComment,
+ anyOpenTag,
+ anyCloseTag,
+ LineEnd,
+ replaceWith,
+)
scriptOpen, scriptClose = makeHTMLTags("script")
scriptBody = scriptOpen + scriptOpen.tag_body + scriptClose
# get some HTML
targetURL = "https://wiki.python.org/moin/PythonDecoratorLibrary"
-with closing(urllib.request.urlopen( targetURL )) as targetPage:
+with urlopen(targetURL) as targetPage:
targetHTML = targetPage.read().decode("UTF-8")
# first pass, strip out tags and translate entities
-firstPass = (htmlComment | scriptBody | commonHTMLEntity |
- anyOpenTag | anyCloseTag ).suppress().transformString(targetHTML)
+firstPass = (
+ (htmlComment | scriptBody | commonHTMLEntity | anyOpenTag | anyCloseTag)
+ .suppress()
+ .transformString(targetHTML)
+)
# first pass leaves many blank lines, collapse these down
-repeatedNewlines = LineEnd()*(2,)
+repeatedNewlines = LineEnd() * (2,)
repeatedNewlines.setParseAction(replaceWith("\n\n"))
secondPass = repeatedNewlines.transformString(firstPass)
# define basic HTML tags, and compose into a Table
-table, table_end = pp.makeHTMLTags('table')
-thead, thead_end = pp.makeHTMLTags('thead')
-tbody, tbody_end = pp.makeHTMLTags('tbody')
-tr, tr_end = pp.makeHTMLTags('tr')
-th, th_end = pp.makeHTMLTags('th')
-td, td_end = pp.makeHTMLTags('td')
-a, a_end = pp.makeHTMLTags('a')
+table, table_end = pp.makeHTMLTags("table")
+thead, thead_end = pp.makeHTMLTags("thead")
+tbody, tbody_end = pp.makeHTMLTags("tbody")
+tr, tr_end = pp.makeHTMLTags("tr")
+th, th_end = pp.makeHTMLTags("th")
+td, td_end = pp.makeHTMLTags("td")
+a, a_end = pp.makeHTMLTags("a")
# method to strip HTML tags from a string - will be used to clean up content of table cells
strip_html = (pp.anyOpenTag | pp.anyCloseTag).suppress().transformString
# expression for parsing <a href="url">text</a> links, returning a (text, url) tuple
-link = pp.Group(a + a.tag_body('text') + a_end.suppress())
-link.addParseAction(lambda t: (t[0].text, t[0].href))
+link = pp.Group(a + a.tag_body("text") + a_end.suppress())
+
+
+def extract_text_and_url(t):
+ return (t[0].text, t[0].href)
+
+
+link.addParseAction(extract_text_and_url)
# method to create table rows of header and data tags
def table_row(start_tag, end_tag):
body = start_tag.tag_body
- body.addParseAction(pp.tokenMap(str.strip),
- pp.tokenMap(strip_html))
- row = pp.Group(tr.suppress()
- + pp.ZeroOrMore(start_tag.suppress()
- + body
- + end_tag.suppress())
- + tr_end.suppress())
+ body.addParseAction(pp.tokenMap(str.strip), pp.tokenMap(strip_html))
+ row = pp.Group(
+ tr.suppress()
+ + pp.ZeroOrMore(start_tag.suppress() + body + end_tag.suppress())
+ + tr_end.suppress()
+ )
return row
+
th_row = table_row(th, th_end)
td_row = table_row(td, td_end)
# define expression for overall table - may vary slightly for different pages
-html_table = table + tbody + pp.Optional(th_row('headers')) + pp.ZeroOrMore(td_row)('rows') + tbody_end + table_end
+html_table = (
+ table
+ + tbody
+ + pp.Optional(th_row("headers"))
+ + pp.ZeroOrMore(td_row)("rows")
+ + tbody_end
+ + table_end
+)
# read in a web page containing an interesting HTML table
-with urllib.request.urlopen("https://en.wikipedia.org/wiki/List_of_tz_database_time_zones") as page:
+with urllib.request.urlopen(
+ "https://en.wikipedia.org/wiki/List_of_tz_database_time_zones"
+) as page:
page_html = page.read().decode()
tz_table = html_table.searchString(page_html)[0]
rows = [dict(zip(tz_table.headers, row)) for row in tz_table.rows]
# make a dict keyed by TZ database name
-tz_db = {row['TZ database name']: row for row in rows}
+tz_db = {row["TZ database name"]: row for row in rows}
from pprint import pprint
-pprint(tz_db['America/Chicago'])
+
+pprint(tz_db["America/Chicago"])
Client Software
"""
-from pyparsing import alphas,nums, dblQuotedString, Combine, Word, Group, delimitedList, Suppress, removeQuotes
+from pyparsing import (
+ alphas,
+ nums,
+ dblQuotedString,
+ Combine,
+ Word,
+ Group,
+ delimitedList,
+ Suppress,
+ removeQuotes,
+)
import string
-def getCmdFields( s, l, t ):
- t["method"],t["requestURI"],t["protocolVersion"] = t[0].strip('"').split()
+
+def getCmdFields(s, l, t):
+ t["method"], t["requestURI"], t["protocolVersion"] = t[0].strip('"').split()
+
logLineBNF = None
+
+
def getLogLineBNF():
global logLineBNF
if logLineBNF is None:
- integer = Word( nums )
- ipAddress = delimitedList( integer, ".", combine=True )
+ integer = Word(nums)
+ ipAddress = delimitedList(integer, ".", combine=True)
- timeZoneOffset = Word("+-",nums)
+ timeZoneOffset = Word("+-", nums)
month = Word(string.ascii_uppercase, string.ascii_lowercase, exact=3)
- serverDateTime = Group( Suppress("[") +
- Combine( integer + "/" + month + "/" + integer +
- ":" + integer + ":" + integer + ":" + integer ) +
- timeZoneOffset +
- Suppress("]") )
+ serverDateTime = Group(
+ Suppress("[")
+ + Combine(
+ integer
+ + "/"
+ + month
+ + "/"
+ + integer
+ + ":"
+ + integer
+ + ":"
+ + integer
+ + ":"
+ + integer
+ )
+ + timeZoneOffset
+ + Suppress("]")
+ )
- logLineBNF = ( ipAddress.setResultsName("ipAddr") +
- Suppress("-") +
- ("-" | Word( alphas+nums+"@._" )).setResultsName("auth") +
- serverDateTime.setResultsName("timestamp") +
- dblQuotedString.setResultsName("cmd").setParseAction(getCmdFields) +
- (integer | "-").setResultsName("statusCode") +
- (integer | "-").setResultsName("numBytesSent") +
- dblQuotedString.setResultsName("referrer").setParseAction(removeQuotes) +
- dblQuotedString.setResultsName("clientSfw").setParseAction(removeQuotes) )
+ logLineBNF = (
+ ipAddress.setResultsName("ipAddr")
+ + Suppress("-")
+ + ("-" | Word(alphas + nums + "@._")).setResultsName("auth")
+ + serverDateTime.setResultsName("timestamp")
+ + dblQuotedString.setResultsName("cmd").setParseAction(getCmdFields)
+ + (integer | "-").setResultsName("statusCode")
+ + (integer | "-").setResultsName("numBytesSent")
+ + dblQuotedString.setResultsName("referrer").setParseAction(removeQuotes)
+ + dblQuotedString.setResultsName("clientSfw").setParseAction(removeQuotes)
+ )
return logLineBNF
+
testdata = """
195.146.134.15 - - [20/Jan/2003:08:55:36 -0800] "GET /path/to/page.html HTTP/1.0" 200 4649 "http://www.somedomain.com/020602/page.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
111.111.111.11 - - [16/Feb/2004:04:09:49 -0800] "GET /ads/redirectads/336x280redirect.htm HTTP/1.1" 304 - "http://www.foobarp.org/theme_detail.php?type=vs&cat=0&mid=27512" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
127.0.0.1 - u.surname@domain.com [12/Sep/2006:14:13:53 +0300] "GET /skins/monobook/external.png HTTP/1.0" 304 - "http://wiki.mysite.com/skins/monobook/main.css" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6"
"""
for line in testdata.split("\n"):
- if not line: continue
+ if not line:
+ continue
fields = getLogLineBNF().parseString(line)
print(fields.dump())
- #~ print repr(fields)
- #~ for k in fields.keys():
- #~ print "fields." + k + " =", fields[k]
+ # ~ print repr(fields)
+ # ~ for k in fields.keys():
+ # ~ print "fields." + k + " =", fields[k]
print()
# Copyright (c) 2003, Paul McGuire\r
#\r
\r
-from pyparsing import Literal, Word, OneOrMore, ZeroOrMore, \\r
- Forward, delimitedList, Group, Optional, alphas, restOfLine, cStyleComment, \\r
- alphanums, quotedString, ParseException, Keyword, Regex\r
+from pyparsing import (\r
+ Literal,\r
+ Word,\r
+ OneOrMore,\r
+ ZeroOrMore,\r
+ Forward,\r
+ delimitedList,\r
+ Group,\r
+ Optional,\r
+ alphas,\r
+ restOfLine,\r
+ cStyleComment,\r
+ alphanums,\r
+ quotedString,\r
+ ParseException,\r
+ Keyword,\r
+ Regex,\r
+)\r
import pprint\r
-#~ import tree2image\r
+\r
+# ~ import tree2image\r
\r
bnf = None\r
+\r
+\r
def CORBA_IDL_BNF():\r
global bnf\r
\r
if not bnf:\r
\r
# punctuation\r
- (colon,lbrace,rbrace,lbrack,rbrack,lparen,rparen,\r
- equals,comma,dot,slash,bslash,star,semi,langle,rangle) = map(Literal, r":{}[]()=,./\*;<>")\r
+ (\r
+ colon,\r
+ lbrace,\r
+ rbrace,\r
+ lbrack,\r
+ rbrack,\r
+ lparen,\r
+ rparen,\r
+ equals,\r
+ comma,\r
+ dot,\r
+ slash,\r
+ bslash,\r
+ star,\r
+ semi,\r
+ langle,\r
+ rangle,\r
+ ) = map(Literal, r":{}[]()=,./\*;<>")\r
\r
# keywords\r
- (any_, attribute_, boolean_, case_, char_, const_, context_, default_, double_, enum_, exception_,\r
- FALSE_, fixed_, float_, inout_, interface_, in_, long_, module_, Object_, octet_, oneway_, out_, raises_,\r
- readonly_, sequence_, short_, string_, struct_, switch_, TRUE_, typedef_, unsigned_, union_, void_,\r
- wchar_, wstring_) = map(Keyword, """any attribute boolean case char const context\r
+ (\r
+ any_,\r
+ attribute_,\r
+ boolean_,\r
+ case_,\r
+ char_,\r
+ const_,\r
+ context_,\r
+ default_,\r
+ double_,\r
+ enum_,\r
+ exception_,\r
+ FALSE_,\r
+ fixed_,\r
+ float_,\r
+ inout_,\r
+ interface_,\r
+ in_,\r
+ long_,\r
+ module_,\r
+ Object_,\r
+ octet_,\r
+ oneway_,\r
+ out_,\r
+ raises_,\r
+ readonly_,\r
+ sequence_,\r
+ short_,\r
+ string_,\r
+ struct_,\r
+ switch_,\r
+ TRUE_,\r
+ typedef_,\r
+ unsigned_,\r
+ union_,\r
+ void_,\r
+ wchar_,\r
+ wstring_,\r
+ ) = map(\r
+ Keyword,\r
+ """any attribute boolean case char const context\r
default double enum exception FALSE fixed float inout interface in long module\r
Object octet oneway out raises readonly sequence short string struct switch\r
- TRUE typedef unsigned union void wchar wstring""".split())\r
+ TRUE typedef unsigned union void wchar wstring""".split(),\r
+ )\r
\r
- identifier = Word( alphas, alphanums + "_" ).setName("identifier")\r
+ identifier = Word(alphas, alphanums + "_").setName("identifier")\r
\r
real = Regex(r"[+-]?\d+\.\d*([Ee][+-]?\d+)?").setName("real")\r
integer = Regex(r"0x[0-9a-fA-F]+|[+-]?\d+").setName("int")\r
\r
- udTypeName = delimitedList( identifier, "::", combine=True ).setName("udType")\r
- typeName = ( any_ | boolean_ | char_ | double_ | fixed_ |\r
- float_ | long_ | octet_ | short_ | string_ |\r
- wchar_ | wstring_ | udTypeName ).setName("type")\r
+ udTypeName = delimitedList(identifier, "::", combine=True).setName("udType")\r
+ typeName = (\r
+ any_\r
+ | boolean_\r
+ | char_\r
+ | double_\r
+ | fixed_\r
+ | float_\r
+ | long_\r
+ | octet_\r
+ | short_\r
+ | string_\r
+ | wchar_\r
+ | wstring_\r
+ | udTypeName\r
+ ).setName("type")\r
sequenceDef = Forward().setName("seq")\r
- sequenceDef << Group( sequence_ + langle + ( sequenceDef | typeName ) + rangle )\r
- typeDef = sequenceDef | ( typeName + Optional( lbrack + integer + rbrack ) )\r
- typedefDef = Group( typedef_ + typeDef + identifier + semi ).setName("typedef")\r
-\r
- moduleDef = Forward()\r
- constDef = Group( const_ + typeDef + identifier + equals + ( real | integer | quotedString ) + semi ) #| quotedString )\r
- exceptionItem = Group( typeDef + identifier + semi )\r
- exceptionDef = ( exception_ + identifier + lbrace + ZeroOrMore( exceptionItem ) + rbrace + semi )\r
- attributeDef = Optional( readonly_ ) + attribute_ + typeDef + identifier + semi\r
- paramlist = delimitedList( Group( ( inout_ | in_ | out_ ) + typeName + identifier ) ).setName( "paramlist" )\r
- operationDef = ( ( void_ ^ typeDef ) + identifier + lparen + Optional( paramlist ) + rparen + \\r
- Optional( raises_ + lparen + Group( delimitedList( typeName ) ) + rparen ) + semi )\r
- interfaceItem = ( constDef | exceptionDef | attributeDef | operationDef )\r
- interfaceDef = Group( interface_ + identifier + Optional( colon + delimitedList( typeName ) ) + lbrace + \\r
- ZeroOrMore( interfaceItem ) + rbrace + semi ).setName("opnDef")\r
- moduleItem = ( interfaceDef | exceptionDef | constDef | typedefDef | moduleDef )\r
- moduleDef << module_ + identifier + lbrace + ZeroOrMore( moduleItem ) + rbrace + semi\r
-\r
- bnf = ( moduleDef | OneOrMore( moduleItem ) )\r
+ sequenceDef << Group(sequence_ + langle + (sequenceDef | typeName) + rangle)\r
+ typeDef = sequenceDef | (typeName + Optional(lbrack + integer + rbrack))\r
+ typedefDef = Group(typedef_ + typeDef + identifier + semi).setName("typedef")\r
+\r
+ moduleDef = Forward().setName("moduleDef")\r
+ constDef = Group(\r
+ const_\r
+ + typeDef\r
+ + identifier\r
+ + equals\r
+ + (real | integer | quotedString)\r
+ + semi\r
+ ).setName(\r
+ "constDef"\r
+ ) # | quotedString )\r
+ exceptionItem = Group(typeDef + identifier + semi)\r
+ exceptionDef = (\r
+ exception_ + identifier + lbrace + ZeroOrMore(exceptionItem) + rbrace + semi\r
+ ).setName("exceptionDef")\r
+ attributeDef = Optional(readonly_) + attribute_ + typeDef + identifier + semi\r
+ paramlist = delimitedList(\r
+ Group((inout_ | in_ | out_) + typeName + identifier)\r
+ ).setName("paramlist")\r
+ operationDef = (\r
+ (void_ ^ typeDef)\r
+ + identifier\r
+ + lparen\r
+ + Optional(paramlist)\r
+ + rparen\r
+ + Optional(raises_ + lparen + Group(delimitedList(typeName)) + rparen)\r
+ + semi\r
+ ).setName("operationDef")\r
+ interfaceItem = constDef | exceptionDef | attributeDef | operationDef\r
+ interfaceDef = Group(\r
+ interface_\r
+ + identifier\r
+ + Optional(colon + delimitedList(typeName))\r
+ + lbrace\r
+ + ZeroOrMore(interfaceItem)\r
+ + rbrace\r
+ + semi\r
+ ).setName("interfaceDef")\r
+ moduleItem = (\r
+ interfaceDef | exceptionDef | constDef | typedefDef | moduleDef\r
+ ).setName("moduleItem")\r
+ (\r
+ moduleDef\r
+ << module_ + identifier + lbrace + ZeroOrMore(moduleItem) + rbrace + semi\r
+ )\r
+\r
+ bnf = moduleDef | OneOrMore(moduleItem)\r
\r
singleLineComment = "//" + restOfLine\r
- bnf.ignore( singleLineComment )\r
- bnf.ignore( cStyleComment )\r
+ bnf.ignore(singleLineComment)\r
+ bnf.ignore(cStyleComment)\r
\r
return bnf\r
\r
-testnum = 1\r
-def test( strng ):\r
- global testnum\r
- print(strng)\r
- try:\r
- bnf = CORBA_IDL_BNF()\r
- tokens = bnf.parseString( strng )\r
- print("tokens = ")\r
- pprint.pprint( tokens.asList() )\r
- imgname = "idlParse%02d.bmp" % testnum\r
- testnum += 1\r
- #~ tree2image.str2image( str(tokens.asList()), imgname )\r
- except ParseException as err:\r
- print(err.line)\r
- print(" "*(err.column-1) + "^")\r
- print(err)\r
- print()\r
\r
if __name__ == "__main__":\r
+\r
+ testnum = 1\r
+\r
+ def test(strng):\r
+ global testnum\r
+ print(strng)\r
+ try:\r
+ bnf = CORBA_IDL_BNF()\r
+ tokens = bnf.parseString(strng)\r
+ print("tokens = ")\r
+ pprint.pprint(tokens.asList())\r
+ imgname = "idlParse%02d.bmp" % testnum\r
+ testnum += 1\r
+ # ~ tree2image.str2image( str(tokens.asList()), imgname )\r
+ except ParseException as err:\r
+ print(err.line)\r
+ print(" " * (err.column - 1) + "^")\r
+ print(err)\r
+ print()\r
+\r
test(\r
"""\r
/*\r
string method3();\r
};\r
"""\r
- )\r
+ )\r
test(\r
"""\r
/*\r
string method3();\r
};\r
"""\r
- )\r
+ )\r
test(\r
r"""\r
const string test="Test String\n";\r
void method1( in string arg1, inout long arg2 );\r
};\r
"""\r
- )\r
+ )\r
test(\r
"""\r
module Test1\r
};\r
};\r
"""\r
- )\r
+ )\r
test(\r
"""\r
module Test1\r
\r
};\r
"""\r
- )\r
+ )\r
from pathlib import Path
# parser elements to be used to assemble into #include parser
-SEMI = pp.Suppress(';')
+SEMI = pp.Suppress(";")
INCLUDE = pp.Keyword("#include")
quoted_string = pp.quotedString.addParseAction(pp.removeQuotes)
-file_ref = (quoted_string
- | pp.Word(pp.printables, excludeChars=';'))
+file_ref = quoted_string | pp.Word(pp.printables, excludeChars=";")
# parser for parsing "#include xyz.dat;" directives
-include_directive = (INCLUDE + file_ref("include_file_name") + SEMI)
+include_directive = INCLUDE + file_ref("include_file_name") + SEMI
# add parse action that will recursively pull in included files - when
# using transformString, the value returned from the parse action will replace
# the text matched by the attached expression
seen = set()
+
+
def read_include_contents(s, l, t):
include_file_ref = t.include_file_name
include_echo = "/* {} */".format(pp.line(l, s).strip())
if include_file_ref not in seen:
seen.add(include_file_ref)
included_file_contents = Path(include_file_ref).read_text()
- return (include_echo + '\n'
- + include_directive.transformString(included_file_contents))
+ return (
+ include_echo
+ + "\n"
+ + include_directive.transformString(included_file_contents)
+ )
else:
- lead = ' '*(pp.col(l, s) - 1)
+ lead = " " * (pp.col(l, s) - 1)
return "/* recursive include! */\n{}{}".format(lead, include_echo)
+
# attach include processing method as parse action (parse-time callback)
# to include_directive expression
include_directive.addParseAction(read_include_contents)
-if __name__ == '__main__':
+if __name__ == "__main__":
# demo
# - a.txt includes b.txt
# - b.txt includes c.txt
# - c.txt includes b.txt (must catch infinite recursion)
- Path('a.txt').write_text("""\
+ Path("a.txt").write_text(
+ """\
/* a.txt */
int i;
-
- /* sometimes included files aren't in quotes */
+
+ /* sometimes included files aren't in quotes */
#include b.txt;
- """)
+ """
+ )
- Path('b.txt').write_text("""\
+ Path("b.txt").write_text(
+ """\
i = 100;
#include 'c.txt';
- """)
+ """
+ )
- Path('c.txt').write_text("""\
+ Path("c.txt").write_text(
+ """\
i += 1;
-
+
/* watch out! this might be recursive if this file included by b.txt */
#include b.txt;
- """)
-
+ """
+ )
# use include_directive.transformString to perform includes
# read contents of original file
- initial_file = Path('a.txt').read_text()
+ initial_file = Path("a.txt").read_text()
# print original file
print(initial_file)
- print('-----------------')
+ print("-----------------")
# expand includes in source file (and any included files) and print the result
expanded_source = include_directive.transformString(initial_file)
\r
from pyparsing import *\r
\r
+\r
data = """\\r
def A(z):\r
A1\r
"""\r
\r
\r
-indentStack = [1]\r
stmt = Forward()\r
-suite = indentedBlock(stmt, indentStack)\r
+suite = IndentedBlock(stmt)\r
\r
identifier = Word(alphas, alphanums)\r
-funcDecl = ("def" + identifier + Group( "(" + Optional( delimitedList(identifier) ) + ")" ) + ":")\r
-funcDef = Group( funcDecl + suite )\r
+funcDecl = (\r
+ "def" + identifier + Group("(" + Optional(delimitedList(identifier)) + ")") + ":"\r
+)\r
+funcDef = Group(funcDecl + suite)\r
\r
rvalue = Forward()\r
funcCall = Group(identifier + "(" + Optional(delimitedList(rvalue)) + ")")\r
rvalue << (funcCall | identifier | Word(nums))\r
assignment = Group(identifier + "=" + rvalue)\r
-stmt << ( funcDef | assignment | identifier )\r
+stmt << (funcDef | assignment | identifier)\r
\r
module_body = OneOrMore(stmt)\r
\r
--- /dev/null
+#
+# indented_block_example.py
+#
+
+import pyparsing as pp
+
+ppc = pp.pyparsing_common
+
+data = """\
+
+ A
+ 100
+ 101
+
+ 102
+ B
+ 200
+ 201
+
+ C
+ 300
+
+"""
+
+integer = ppc.integer
+group = pp.Group(pp.Char(pp.alphas) + pp.Group(pp.IndentedBlock(integer)))
+
+print(group[...].parseString(data).dump())
+
+# example of a recursive IndentedBlock
+
+data = """\
+
+ A
+ 100
+ 101
+
+ 102
+ B
+ 200
+ b
+ 210
+ 211
+ 202
+ C
+ 300
+
+"""
+
+group = pp.Forward()
+group <<= pp.Group(pp.Char(pp.alphas) + pp.Group(pp.IndentedBlock(integer | group)))
+
+print("using searchString")
+print(sum(group.searchString(data)).dump())
# - () grouping\r
# - | alternation\r
#\r
-__all__ = ["count","invert"]\r
+__all__ = ["count", "invert"]\r
\r
-from pyparsing import (Literal, oneOf, printables, ParserElement, Combine,\r
- SkipTo, infixNotation, ParseFatalException, Word, nums, opAssoc,\r
- Suppress, ParseResults, srange)\r
+from pyparsing import (\r
+ Literal,\r
+ oneOf,\r
+ printables,\r
+ ParserElement,\r
+ Combine,\r
+ SkipTo,\r
+ infixNotation,\r
+ ParseFatalException,\r
+ Word,\r
+ nums,\r
+ opAssoc,\r
+ Suppress,\r
+ ParseResults,\r
+ srange,\r
+)\r
\r
-class CharacterRangeEmitter(object):\r
- def __init__(self,chars):\r
+ParserElement.enablePackrat()\r
+\r
+\r
+class CharacterRangeEmitter:\r
+ def __init__(self, chars):\r
# remove duplicate chars in character range, but preserve original order\r
seen = set()\r
- self.charset = "".join( seen.add(c) or c for c in chars if c not in seen )\r
+ self.charset = "".join(seen.add(c) or c for c in chars if c not in seen)\r
+\r
def __str__(self):\r
- return '['+self.charset+']'\r
+ return "[" + self.charset + "]"\r
+\r
def __repr__(self):\r
- return '['+self.charset+']'\r
+ return "[" + self.charset + "]"\r
+\r
def makeGenerator(self):\r
def genChars():\r
- for s in self.charset:\r
- yield s\r
+ yield from self.charset\r
+\r
return genChars\r
\r
-class OptionalEmitter(object):\r
- def __init__(self,expr):\r
+\r
+class OptionalEmitter:\r
+ def __init__(self, expr):\r
self.expr = expr\r
+\r
def makeGenerator(self):\r
def optionalGen():\r
yield ""\r
- for s in self.expr.makeGenerator()():\r
- yield s\r
+ yield from self.expr.makeGenerator()()\r
+\r
return optionalGen\r
\r
-class DotEmitter(object):\r
+\r
+class DotEmitter:\r
def makeGenerator(self):\r
def dotGen():\r
- for c in printables:\r
- yield c\r
+ yield from printables\r
+\r
return dotGen\r
\r
-class GroupEmitter(object):\r
- def __init__(self,exprs):\r
+\r
+class GroupEmitter:\r
+ def __init__(self, exprs):\r
self.exprs = ParseResults(exprs)\r
+\r
def makeGenerator(self):\r
def groupGen():\r
def recurseList(elist):\r
- if len(elist)==1:\r
- for s in elist[0].makeGenerator()():\r
- yield s\r
+ if len(elist) == 1:\r
+ yield from elist[0].makeGenerator()()\r
else:\r
for s in elist[0].makeGenerator()():\r
for s2 in recurseList(elist[1:]):\r
yield s + s2\r
+\r
if self.exprs:\r
- for s in recurseList(self.exprs):\r
- yield s\r
+ yield from recurseList(self.exprs)\r
+\r
return groupGen\r
\r
-class AlternativeEmitter(object):\r
- def __init__(self,exprs):\r
+\r
+class AlternativeEmitter:\r
+ def __init__(self, exprs):\r
self.exprs = exprs\r
+\r
def makeGenerator(self):\r
def altGen():\r
for e in self.exprs:\r
- for s in e.makeGenerator()():\r
- yield s\r
+ yield from e.makeGenerator()()\r
+\r
return altGen\r
\r
-class LiteralEmitter(object):\r
- def __init__(self,lit):\r
+\r
+class LiteralEmitter:\r
+ def __init__(self, lit):\r
self.lit = lit\r
+\r
def __str__(self):\r
- return "Lit:"+self.lit\r
+ return "Lit:" + self.lit\r
+\r
def __repr__(self):\r
- return "Lit:"+self.lit\r
+ return "Lit:" + self.lit\r
+\r
def makeGenerator(self):\r
def litGen():\r
yield self.lit\r
+\r
return litGen\r
\r
+\r
def handleRange(toks):\r
return CharacterRangeEmitter(srange(toks[0]))\r
\r
+\r
def handleRepetition(toks):\r
- toks=toks[0]\r
+ toks = toks[0]\r
if toks[1] in "*+":\r
- raise ParseFatalException("",0,"unbounded repetition operators not supported")\r
+ raise ParseFatalException("", 0, "unbounded repetition operators not supported")\r
if toks[1] == "?":\r
return OptionalEmitter(toks[0])\r
if "count" in toks:\r
optcount = maxcount - mincount\r
if optcount:\r
opt = OptionalEmitter(toks[0])\r
- for i in range(1,optcount):\r
- opt = OptionalEmitter(GroupEmitter([toks[0],opt]))\r
+ for i in range(1, optcount):\r
+ opt = OptionalEmitter(GroupEmitter([toks[0], opt]))\r
return GroupEmitter([toks[0]] * mincount + [opt])\r
else:\r
return [toks[0]] * mincount\r
\r
+\r
def handleLiteral(toks):\r
lit = ""\r
for t in toks:\r
if t[0] == "\\":\r
if t[1] == "t":\r
- lit += '\t'\r
+ lit += "\t"\r
else:\r
lit += t[1]\r
else:\r
lit += t\r
return LiteralEmitter(lit)\r
\r
+\r
def handleMacro(toks):\r
macroChar = toks[0][1]\r
if macroChar == "d":\r
elif macroChar == "s":\r
return LiteralEmitter(" ")\r
else:\r
- raise ParseFatalException("",0,"unsupported macro character (" + macroChar + ")")\r
+ raise ParseFatalException(\r
+ "", 0, "unsupported macro character (" + macroChar + ")"\r
+ )\r
+\r
\r
def handleSequence(toks):\r
return GroupEmitter(toks[0])\r
\r
+\r
def handleDot():\r
return CharacterRangeEmitter(printables)\r
\r
+\r
def handleAlternative(toks):\r
return AlternativeEmitter(toks[0])\r
\r
\r
_parser = None\r
+\r
+\r
def parser():\r
global _parser\r
if _parser is None:\r
ParserElement.setDefaultWhitespaceChars("")\r
- lbrack,rbrack,lbrace,rbrace,lparen,rparen,colon,qmark = map(Literal,"[]{}():?")\r
+ lbrack, rbrack, lbrace, rbrace, lparen, rparen, colon, qmark = map(\r
+ Literal, "[]{}():?"\r
+ )\r
\r
reMacro = Combine("\\" + oneOf(list("dws")))\r
escapedChar = ~reMacro + Combine("\\" + oneOf(list(printables)))\r
- reLiteralChar = "".join(c for c in printables if c not in r"\[]{}().*?+|") + " \t"\r
+ reLiteralChar = (\r
+ "".join(c for c in printables if c not in r"\[]{}().*?+|") + " \t"\r
+ )\r
\r
- reRange = Combine(lbrack + SkipTo(rbrack,ignore=escapedChar) + rbrack)\r
- reLiteral = ( escapedChar | oneOf(list(reLiteralChar)) )\r
+ reRange = Combine(lbrack + SkipTo(rbrack, ignore=escapedChar) + rbrack)\r
+ reLiteral = escapedChar | oneOf(list(reLiteralChar))\r
reNonCaptureGroup = Suppress("?:")\r
reDot = Literal(".")\r
repetition = (\r
- ( lbrace + Word(nums)("count") + rbrace ) |\r
- ( lbrace + Word(nums)("minCount")+","+ Word(nums)("maxCount") + rbrace ) |\r
- oneOf(list("*+?"))\r
- )\r
+ (lbrace + Word(nums)("count") + rbrace)\r
+ | (lbrace + Word(nums)("minCount") + "," + Word(nums)("maxCount") + rbrace)\r
+ | oneOf(list("*+?"))\r
+ )\r
\r
reRange.setParseAction(handleRange)\r
reLiteral.setParseAction(handleLiteral)\r
reMacro.setParseAction(handleMacro)\r
reDot.setParseAction(handleDot)\r
\r
- reTerm = ( reLiteral | reRange | reMacro | reDot | reNonCaptureGroup)\r
- reExpr = infixNotation( reTerm,\r
+ reTerm = reLiteral | reRange | reMacro | reDot | reNonCaptureGroup\r
+ reExpr = infixNotation(\r
+ reTerm,\r
[\r
- (repetition, 1, opAssoc.LEFT, handleRepetition),\r
- (None, 2, opAssoc.LEFT, handleSequence),\r
- (Suppress('|'), 2, opAssoc.LEFT, handleAlternative),\r
- ]\r
- )\r
+ (repetition, 1, opAssoc.LEFT, handleRepetition),\r
+ (None, 2, opAssoc.LEFT, handleSequence),\r
+ (Suppress("|"), 2, opAssoc.LEFT, handleAlternative),\r
+ ],\r
+ )\r
_parser = reExpr\r
\r
return _parser\r
\r
+\r
def count(gen):\r
"""Simple function to count the number of elements returned by a generator."""\r
return sum(1 for _ in gen)\r
\r
+\r
def invert(regex):\r
- r"""Call this routine as a generator to return all the strings that\r
- match the input regular expression.\r
- for s in invert(r"[A-Z]{3}\d{3}"):\r
- print s\r
+ r"""\r
+ Call this routine as a generator to return all the strings that\r
+ match the input regular expression.\r
+ for s in invert(r"[A-Z]{3}\d{3}"):\r
+ print s\r
"""\r
invReGenerator = GroupEmitter(parser().parseString(regex)).makeGenerator()\r
return invReGenerator()\r
\r
+\r
def main():\r
tests = r"""\r
[A-EA]\r
[ABCDEFG](?:#|##|b|bb)?(?:maj|min|m|sus|aug|dim)?[0-9]?(?:/[ABCDEFG](?:#|##|b|bb)?)?\r
(Fri|Mon|S(atur|un)|T(hur|ue)s|Wednes)day\r
A(pril|ugust)|((Dec|Nov|Sept)em|Octo)ber|(Febr|Jan)uary|Ju(ly|ne)|Ma(rch|y)\r
- """.split('\n')\r
+ """.splitlines()\r
\r
for t in tests:\r
t = t.strip()\r
- if not t: continue\r
- print('-'*50)\r
+ if not t:\r
+ continue\r
+ print("-" * 50)\r
print(t)\r
try:\r
num = count(invert(t))\r
break\r
except ParseFatalException as pfe:\r
print(pfe.msg)\r
- print('')\r
+ print("")\r
continue\r
- print('')\r
+ print("")\r
+\r
\r
if __name__ == "__main__":\r
main()\r
import pyparsing as pp\r
from pyparsing import pyparsing_common as ppc\r
\r
+\r
def make_keyword(kwd_str, kwd_value):\r
return pp.Keyword(kwd_str).setParseAction(pp.replaceWith(kwd_value))\r
-TRUE = make_keyword("true", True)\r
+\r
+\r
+# set to False to return ParseResults\r
+RETURN_PYTHON_COLLECTIONS = True\r
+\r
+TRUE = make_keyword("true", True)\r
FALSE = make_keyword("false", False)\r
-NULL = make_keyword("null", None)\r
+NULL = make_keyword("null", None)\r
\r
LBRACK, RBRACK, LBRACE, RBRACE, COLON = map(pp.Suppress, "[]{}:")\r
\r
jsonString = pp.dblQuotedString().setParseAction(pp.removeQuotes)\r
-jsonNumber = ppc.number()\r
+jsonNumber = ppc.number().setName("jsonNumber")\r
+\r
+jsonObject = pp.Forward().setName("jsonObject")\r
+jsonValue = pp.Forward().setName("jsonValue")\r
+\r
+jsonElements = pp.delimitedList(jsonValue).setName(None)\r
+# jsonArray = pp.Group(LBRACK + pp.Optional(jsonElements, []) + RBRACK)\r
+# jsonValue << (\r
+# jsonString | jsonNumber | pp.Group(jsonObject) | jsonArray | TRUE | FALSE | NULL\r
+# )\r
+# memberDef = pp.Group(jsonString + COLON + jsonValue).setName("jsonMember")\r
+\r
+jsonArray = pp.Group(\r
+ LBRACK + pp.Optional(jsonElements) + RBRACK, aslist=RETURN_PYTHON_COLLECTIONS\r
+).setName("jsonArray")\r
\r
-jsonObject = pp.Forward()\r
-jsonValue = pp.Forward()\r
-jsonElements = pp.delimitedList( jsonValue )\r
-jsonArray = pp.Group(LBRACK + pp.Optional(jsonElements, []) + RBRACK)\r
-jsonValue << (jsonString | jsonNumber | pp.Group(jsonObject) | jsonArray | TRUE | FALSE | NULL)\r
-memberDef = pp.Group(jsonString + COLON + jsonValue)\r
-jsonMembers = pp.delimitedList(memberDef)\r
-jsonObject << pp.Dict(LBRACE + pp.Optional(jsonMembers) + RBRACE)\r
+jsonValue << (jsonString | jsonNumber | jsonObject | jsonArray | TRUE | FALSE | NULL)\r
+\r
+memberDef = pp.Group(\r
+ jsonString + COLON + jsonValue, aslist=RETURN_PYTHON_COLLECTIONS\r
+).setName("jsonMember")\r
+\r
+jsonMembers = pp.delimitedList(memberDef).setName(None)\r
+# jsonObject << pp.Dict(LBRACE + pp.Optional(jsonMembers) + RBRACE)\r
+jsonObject << pp.Dict(\r
+ LBRACE + pp.Optional(jsonMembers) + RBRACE, asdict=RETURN_PYTHON_COLLECTIONS\r
+)\r
\r
jsonComment = pp.cppStyleComment\r
jsonObject.ignore(jsonComment)\r
"title": "example glossary",\r
"GlossDiv": {\r
"title": "S",\r
- "GlossList":\r
+ "GlossList": [\r
{\r
"ID": "SGML",\r
"SortAs": "SGML",\r
"EmptyDict" : {},\r
"EmptyList" : []\r
}\r
+ ]\r
}\r
}\r
}\r
"""\r
\r
results = jsonObject.parseString(testdata)\r
+\r
results.pprint()\r
+ if RETURN_PYTHON_COLLECTIONS:\r
+ from pprint import pprint\r
+\r
+ pprint(results)\r
+ else:\r
+ results.pprint()\r
print()\r
+\r
def testPrint(x):\r
print(type(x), repr(x))\r
- print(list(results.glossary.GlossDiv.GlossList.keys()))\r
- testPrint( results.glossary.title )\r
- testPrint( results.glossary.GlossDiv.GlossList.ID )\r
- testPrint( results.glossary.GlossDiv.GlossList.FalseValue )\r
- testPrint( results.glossary.GlossDiv.GlossList.Acronym )\r
- testPrint( results.glossary.GlossDiv.GlossList.EvenPrimesGreaterThan2 )\r
- testPrint( results.glossary.GlossDiv.GlossList.PrimesLessThan10 )\r
+\r
+ if RETURN_PYTHON_COLLECTIONS:\r
+ results = results[0]\r
+ print(list(results["glossary"]["GlossDiv"]["GlossList"][0].keys()))\r
+ testPrint(results["glossary"]["title"])\r
+ testPrint(results["glossary"]["GlossDiv"]["GlossList"][0]["ID"])\r
+ testPrint(results["glossary"]["GlossDiv"]["GlossList"][0]["FalseValue"])\r
+ testPrint(results["glossary"]["GlossDiv"]["GlossList"][0]["Acronym"])\r
+ testPrint(\r
+ results["glossary"]["GlossDiv"]["GlossList"][0]["EvenPrimesGreaterThan2"]\r
+ )\r
+ testPrint(results["glossary"]["GlossDiv"]["GlossList"][0]["PrimesLessThan10"])\r
+ else:\r
+ print(list(results.glossary.GlossDiv.GlossList.keys()))\r
+ testPrint(results.glossary.title)\r
+ testPrint(results.glossary.GlossDiv.GlossList.ID)\r
+ testPrint(results.glossary.GlossDiv.GlossList.FalseValue)\r
+ testPrint(results.glossary.GlossDiv.GlossList.Acronym)\r
+ testPrint(results.glossary.GlossDiv.GlossList.EvenPrimesGreaterThan2)\r
+ testPrint(results.glossary.GlossDiv.GlossList.PrimesLessThan10)\r
--- /dev/null
+#
+# left_recursion.py
+#
+# Example code illustrating use of left-recursion in Pyparsing.
+#
+import pyparsing as pp
+
+# comment out this line to see the effects without LR parsing enabled
+pp.ParserElement.enableLeftRecursion()
+
+item_list = pp.Forward()
+
+# a common left-recursion definition
+# define a list of items as 'list + item | item'
+# BNF:
+# item_list := item_list item | item
+# item := word of alphas
+item = pp.Word(pp.alphas)
+item_list <<= item_list + item | item
+
+item_list.runTests(
+ """\
+ To parse or not to parse that is the question
+ """
+)
+
+# Define a parser for an expression that can be an identifier, a quoted string, or a
+# function call that starts with an expression
+# BNF:
+# expr := function_call | name | string | '(' expr ')'
+# function_call := expr '(' expr,... ')'
+# name := Python identifier
+# string := a quoted string
+# from https://stackoverflow.com/questions/32809389/parse-python-code-using-pyparsing/32822575#32822575
+
+LPAR, RPAR = map(pp.Suppress, "()")
+expr = pp.Forward()
+string = pp.quotedString
+function_call = expr + pp.Group(LPAR + pp.Optional(pp.delimitedList(expr)) + RPAR)
+name = pp.Word(pp.alphas + "_", pp.alphanums + "_")
+# left recursion - call starts with an expr
+expr <<= function_call | string | name | pp.Group(LPAR + expr + RPAR)
+
+expr.runTests(
+ """\
+ print("Hello, World!")
+ (lookup_function("fprintf"))(stderr, "Hello, World!")
+ """,
+ fullDump=False,
+)
of their country."""\r
\r
# demonstrate use of lineno, line, and col in a parse action\r
-def reportLongWords(st,locn,toks):\r
+def reportLongWords(st, locn, toks):\r
word = toks[0]\r
if len(word) > 3:\r
- print("Found '%s' on line %d at column %d" % (word, lineno(locn,st), col(locn,st)))\r
+ print(\r
+ "Found '%s' on line %d at column %d"\r
+ % (word, lineno(locn, st), col(locn, st))\r
+ )\r
print("The full line of text was:")\r
- print("'%s'" % line(locn,st))\r
- print((" "*col(locn,st))+" ^")\r
+ print("'%s'" % line(locn, st))\r
+ print((" " * col(locn, st)) + " ^")\r
print()\r
\r
-wd = Word(alphas).setParseAction( reportLongWords )\r
+\r
+wd = Word(alphas).setParseAction(reportLongWords)\r
OneOrMore(wd).parseString(data)\r
\r
\r
# demonstrate returning an object from a parse action, containing more information\r
# than just the matching token text\r
-class Token(object):\r
+class Token:\r
def __init__(self, st, locn, tokString):\r
self.tokenString = tokString\r
self.locn = locn\r
- self.sourceLine = line(locn,st)\r
- self.lineNo = lineno(locn,st)\r
- self.col = col(locn,st)\r
+ self.sourceLine = line(locn, st)\r
+ self.lineNo = lineno(locn, st)\r
+ self.col = col(locn, st)\r
+\r
def __str__(self):\r
return "%(tokenString)s (line: %(lineNo)d, col: %(col)d)" % self.__dict__\r
\r
-def createTokenObject(st,locn,toks):\r
- return Token(st,locn, toks[0])\r
\r
-wd = Word(alphas).setParseAction( createTokenObject )\r
+def createTokenObject(st, locn, toks):\r
+ return Token(st, locn, toks[0])\r
+\r
+\r
+wd = Word(alphas).setParseAction(createTokenObject)\r
\r
for tokenObj in OneOrMore(wd).parseString(data):\r
print(tokenObj)\r
+++ /dev/null
-#\r
-# list1.py\r
-#\r
-# an example of using parse actions to convert type of parsed data.\r
-#\r
-# Copyright (c) 2006-2016, Paul McGuire\r
-#\r
-from pyparsing import *\r
-\r
-# first pass\r
-lbrack = Literal("[")\r
-rbrack = Literal("]")\r
-integer = Word(nums).setName("integer")\r
-real = Combine(Optional(oneOf("+ -")) + Word(nums) + "." +\r
- Optional(Word(nums))).setName("real")\r
-\r
-listItem = real | integer | quotedString\r
-\r
-listStr = lbrack + delimitedList(listItem) + rbrack\r
-\r
-test = "['a', 100, 3.14]"\r
-\r
-print(listStr.parseString(test))\r
-\r
-\r
-# second pass, cleanup and add converters\r
-lbrack = Literal("[").suppress()\r
-rbrack = Literal("]").suppress()\r
-cvtInt = lambda s,l,toks: int(toks[0])\r
-cvtReal = lambda s,l,toks: float(toks[0])\r
-integer = Word(nums).setName("integer").setParseAction( cvtInt )\r
-real = Combine(Optional(oneOf("+ -")) + Word(nums) + "." +\r
- Optional(Word(nums))).setName("real").setParseAction( cvtReal )\r
-listItem = real | integer | quotedString.setParseAction( removeQuotes )\r
-\r
-listStr = lbrack + delimitedList(listItem) + rbrack\r
-\r
-test = "['a', 100, 3.14]"\r
-\r
-print(listStr.parseString(test))\r
-\r
-# third pass, add nested list support\r
-lbrack, rbrack = map(Suppress, "[]")\r
-\r
-cvtInt = tokenMap(int)\r
-cvtReal = tokenMap(float)\r
-\r
-integer = Word(nums).setName("integer").setParseAction( cvtInt )\r
-real = Regex(r"[+-]?\d+\.\d*").setName("real").setParseAction( cvtReal )\r
-\r
-listStr = Forward()\r
-listItem = real | integer | quotedString.setParseAction(removeQuotes) | Group(listStr)\r
-listStr << lbrack + delimitedList(listItem) + rbrack\r
-\r
-test = "['a', 100, 3.14, [ +2.718, 'xyzzy', -1.414] ]"\r
-print(listStr.parseString(test))
\ No newline at end of file
from pyparsing import oneOf, OneOrMore, printables, StringEnd\r
\r
test = "The quick brown fox named 'Aloysius' lives at 123 Main Street (and jumps over lazy dogs in his spare time)."\r
-nonAlphas = [ c for c in printables if not c.isalpha() ]\r
+nonAlphas = [c for c in printables if not c.isalpha()]\r
\r
print("Extract vowels, consonants, and special characters from this test string:")\r
print("'" + test + "'")\r
-print('')\r
+print("")\r
\r
print("Define grammar using normal results names")\r
print("(only last matching symbol is saved)")\r
print(results.vowels)\r
print(results.cons)\r
print(results.others)\r
-print('')\r
+print("")\r
\r
\r
print("Define grammar using results names, with listAllMatches=True")\r
results = letters.parseString(test, parseAll=True)\r
print(results)\r
print(sorted(set(results)))\r
-print('')\r
+print("")\r
print(results.vowels)\r
print(sorted(set(results.vowels)))\r
-print('')\r
+print("")\r
print(results.cons)\r
print(sorted(set(results.cons)))\r
-print('')\r
+print("")\r
print(results.others)\r
print(sorted(set(results.others)))\r
--- /dev/null
+#
+# lua_parser.py
+#
+# A simple parser for the Lua language.
+#
+# Copyright 2020, Paul McGuire
+#
+
+"""
+from https://www.lua.org/manual/5.1/manual.html#8
+
+ chunk ::= {stat [';']} [laststat [';']]
+
+ block ::= chunk
+
+ stat ::= varlist '=' explist |
+ functioncall |
+ do block end |
+ while exp do block end |
+ repeat block until exp |
+ if exp then block {elseif exp then block} [else block] end |
+ for Name '=' exp ',' exp [',' exp] do block end |
+ for namelist in explist do block end |
+ function funcname funcbody |
+ local function Name funcbody |
+ local namelist ['=' explist]
+
+ laststat ::= return [explist] | break
+
+ funcname ::= Name {'.' Name} [':' Name]
+
+ varlist ::= var {',' var}
+
+ var ::= Name | prefixexp '[' exp ']' | prefixexp '.' Name
+
+ namelist ::= Name {',' Name}
+
+ explist ::= {exp ','} exp
+
+ exp ::= nil | false | true | Number | String | '...' | function |
+ prefixexp | tableconstructor | exp binop exp | unop exp
+
+ prefixexp ::= var | functioncall | '(' exp ')'
+
+ functioncall ::= prefixexp args | prefixexp ':' Name args
+
+ args ::= '(' [explist] ')' | tableconstructor | String
+
+ function ::= function funcbody
+
+ funcbody ::= '(' [parlist] ')' block end
+
+ parlist ::= namelist [',' '...'] | '...'
+
+ tableconstructor ::= '{' [fieldlist] '}'
+
+ fieldlist ::= field {fieldsep field} [fieldsep]
+
+ field ::= '[' exp ']' '=' exp | Name '=' exp | exp
+
+ fieldsep ::= ',' | ';'
+
+ binop ::= '+' | '-' | '*' | '/' | '^' | '%' | '..' |
+ '<' | '<=' | '>' | '>=' | '==' | '~=' |
+ and | or
+
+ unop ::= '-' | not | '#'
+
+operator precedence:
+
+ or
+ and
+ < > <= >= ~= ==
+ |
+ ~
+ &
+ << >>
+ ..
+ + -
+ * / // %
+ unary operators (not # - ~)
+ ^
+
+"""
+import pyparsing as pp
+
+ppc = pp.pyparsing_common
+pp.ParserElement.enablePackrat()
+
+LBRACK, RBRACK, LBRACE, RBRACE, LPAR, RPAR, EQ, COMMA, SEMI, COLON = map(
+ pp.Suppress, "[]{}()=,;:"
+)
+OPT_SEMI = pp.Optional(SEMI).suppress()
+ELLIPSIS = pp.Literal("...")
+keywords = {
+ k.upper(): pp.Keyword(k)
+ for k in """\
+ return break do end while if then elseif else for in function local repeat until nil false true and or not
+ """.split()
+}
+vars().update(keywords)
+any_keyword = pp.MatchFirst(keywords.values()).setName("<keyword>")
+
+comment_intro = pp.Literal("--")
+short_comment = comment_intro + pp.restOfLine
+long_comment = comment_intro + LBRACK + ... + RBRACK
+lua_comment = long_comment | short_comment
+
+# must use negative lookahead to ensure we don't parse a keyword as an identifier
+ident = ~any_keyword + ppc.identifier
+
+name = pp.delimitedList(ident, delim=".", combine=True)
+
+namelist = pp.delimitedList(name)
+number = ppc.number
+
+# does not parse levels
+multiline_string = pp.QuotedString("[[", endQuoteChar="]]", multiline=True)
+string = pp.QuotedString("'") | pp.QuotedString('"') | multiline_string
+
+exp = pp.Forward()
+
+# explist1 ::= {exp ','} exp
+explist1 = pp.delimitedList(exp)
+
+stat = pp.Forward()
+
+# laststat ::= return [explist1] | break
+laststat = pp.Group(RETURN + explist1) | BREAK
+
+# block ::= {stat [';']} [laststat[';']]
+block = pp.Group(stat + OPT_SEMI)[1, ...] + pp.Optional(laststat + OPT_SEMI)
+
+# field ::= '[' exp ']' '=' exp | Name '=' exp | exp
+field = pp.Group(
+ LBRACK + exp + RBRACK + EQ + pp.Group(exp) | name + EQ + pp.Group(exp) | exp
+)
+
+# fieldsep ::= ',' | ';'
+fieldsep = COMMA | SEMI
+
+# fieldlist ::= field {fieldsep field} [fieldsep]
+field_list = pp.delimitedList(field, delim=fieldsep) + pp.Optional(fieldsep)
+
+# tableconstructor ::= '{' [fieldlist] '}'
+tableconstructor = pp.Group(LBRACE + pp.Optional(field_list) + RBRACE)
+
+# parlist1 ::= namelist [',' '...'] | '...'
+parlist = namelist + pp.Optional(COMMA + ELLIPSIS) | ELLIPSIS
+
+# funcname ::= Name {'.' Name} [':' Name]
+funcname = pp.Group(name + COLON + name) | name
+
+# function ::= function funcbody
+# funcbody ::= '(' [parlist1] ')' block end
+funcbody = pp.Group(LPAR + parlist + RPAR) + block + END
+function = FUNCTION + funcbody
+
+# args ::= '(' [explist1] ')' | tableconstructor | String
+args = LPAR + pp.Optional(explist1) + RPAR | tableconstructor | string
+
+# this portion of the spec is left-recursive, must break LR loop
+# varlist1 ::= var {',' var}
+# var ::= Name | prefixexp '[' exp ']' | prefixexp '.' Name
+# prefixexp ::= var | functioncall | '(' exp ')'
+# functioncall ::= prefixexp args | prefixexp ':' Name args
+
+prefixexp = name | LPAR + exp + RPAR
+functioncall = prefixexp + args | prefixexp + COLON + name + args
+var = pp.Forward()
+var_atom = functioncall | name | LPAR + exp + RPAR
+index_ref = pp.Group(LBRACK + exp + RBRACK)
+var <<= pp.delimitedList(pp.Group(var_atom + index_ref) | var_atom, delim=".")
+
+varlist1 = pp.delimitedList(var)
+
+# exp ::= nil | false | true | Number | String | '...' |
+# function | prefixexp | tableconstructor
+exp_atom = (
+ NIL
+ | FALSE
+ | TRUE
+ | number
+ | string
+ | ELLIPSIS
+ | functioncall
+ | var # prefixexp
+ | tableconstructor
+)
+
+# precedence of operations from https://www.lua.org/manual/5.3/manual.html#3.4.8
+exp <<= pp.infixNotation(
+ exp_atom,
+ [
+ ("^", 2, pp.opAssoc.LEFT),
+ (NOT | pp.oneOf("# - ~"), 1, pp.opAssoc.RIGHT),
+ (pp.oneOf("* / // %"), 2, pp.opAssoc.LEFT),
+ (pp.oneOf("+ -"), 2, pp.opAssoc.LEFT),
+ ("..", 2, pp.opAssoc.LEFT),
+ (pp.oneOf("<< >>"), 2, pp.opAssoc.LEFT),
+ ("&", 2, pp.opAssoc.LEFT),
+ ("~", 2, pp.opAssoc.LEFT),
+ ("|", 2, pp.opAssoc.LEFT),
+ (pp.oneOf("< > <= >= ~= =="), 2, pp.opAssoc.LEFT),
+ (AND, 2, pp.opAssoc.LEFT),
+ (OR, 2, pp.opAssoc.LEFT),
+ ],
+)
+
+assignment_stat = pp.Optional(LOCAL) + varlist1 + EQ + explist1
+func_call_stat = pp.Optional(LOCAL) + functioncall
+do_stat = DO + block + END
+while_stat = WHILE + exp + block + END
+repeat_stat = REPEAT + block + UNTIL + exp
+for_loop_stat = (
+ FOR + name + EQ + exp + COMMA + exp + pp.Optional(COMMA + exp) + DO + block + END
+)
+for_seq_stat = FOR + namelist + IN + explist1 + DO + block + END
+if_stat = (
+ IF
+ + exp
+ + THEN
+ + block
+ + pp.Group(ELSEIF + exp + THEN + block)[...]
+ + pp.Optional(pp.Group(ELSE + block))
+ + END
+)
+function_def = pp.Optional(LOCAL) + FUNCTION + funcname + funcbody
+
+for var_name in """
+ assignment_stat
+ func_call_stat
+ do_stat
+ while_stat
+ repeat_stat
+ for_loop_stat
+ for_seq_stat
+ if_stat
+ function_def
+ """.split():
+ vars()[var_name].setName(var_name)
+
+# stat ::= varlist1 '=' explist1 |
+# functioncall |
+# do block end |
+# while exp do block end |
+# repeat block until exp |
+# if exp then block {elseif exp then block} [else block] end |
+# for Name '=' exp ',' exp [',' exp] do block end |
+# for namelist in explist1 do block end |
+# function funcname funcbody |
+# local function Name funcbody |
+# local namelist ['=' explist1]
+stat <<= pp.Group(
+ assignment_stat
+ | do_stat
+ | while_stat
+ | repeat_stat
+ | for_loop_stat
+ | for_seq_stat
+ | func_call_stat
+ | if_stat
+ | function_def
+)
+
+lua_script = stat[...]
+
+# ignore comments
+lua_script.ignore(lua_comment)
+
+if __name__ == "__main__":
+
+ sample = r"""
+ function test(x)
+ local t = {foo=1, bar=2, arg=x}
+ n = 0
+ if t['foo'] then
+ n = n + 1
+ end
+ if 10 > 8 then
+ n = n + 2
+ end
+ if (10 > 8) then
+ n = n + 2
+ end
+ end
+ """
+
+ try:
+ result = lua_script.parseString(sample)
+ result.pprint()
+ except pp.ParseException as pe:
+ print(pe.explain())
#
# Copyright 2011, Paul McGuire
#
-# implementation of Lucene grammar, as decribed
+# implementation of Lucene grammar, as described
# at http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/docs/queryparsersyntax.html
#
import pyparsing as pp
from pyparsing import pyparsing_common as ppc
+
pp.ParserElement.enablePackrat()
-COLON,LBRACK,RBRACK,LBRACE,RBRACE,TILDE,CARAT = map(pp.Literal,":[]{}~^")
-LPAR,RPAR = map(pp.Suppress,"()")
+COLON, LBRACK, RBRACK, LBRACE, RBRACE, TILDE, CARAT = map(pp.Literal, ":[]{}~^")
+LPAR, RPAR = map(pp.Suppress, "()")
and_, or_, not_, to_ = map(pp.CaselessKeyword, "AND OR NOT TO".split())
keyword = and_ | or_ | not_ | to_
expression = pp.Forward()
-valid_word = pp.Regex(r'([a-zA-Z0-9*_+.-]|\\\\|\\([+\-!(){}\[\]^"~*?:]|\|\||&&))+').setName("word")
+valid_word = pp.Regex(
+ r'([a-zA-Z0-9*_+.-]|\\\\|\\([+\-!(){}\[\]^"~*?:]|\|\||&&))+'
+).setName("word")
valid_word.setParseAction(
- lambda t : t[0].replace('\\\\',chr(127)).replace('\\','').replace(chr(127),'\\')
- )
+ lambda t: t[0].replace("\\\\", chr(127)).replace("\\", "").replace(chr(127), "\\")
+)
string = pp.QuotedString('"')
number = ppc.fnumber()
fuzzy_modifier = TILDE + pp.Optional(number, default=0.5)("fuzzy")
-term = pp.Forward()
+term = pp.Forward().setName("field")
field_name = valid_word().setName("fieldname")
incl_range_search = pp.Group(LBRACK - term("lower") + to_ + term("upper") + RBRACK)
excl_range_search = pp.Group(LBRACE - term("lower") + to_ + term("upper") + RBRACE)
range_search = incl_range_search("incl_range") | excl_range_search("excl_range")
-boost = (CARAT - number("boost"))
+boost = CARAT - number("boost")
string_expr = pp.Group(string + proximity_modifier) | string
word_expr = pp.Group(valid_word + fuzzy_modifier) | valid_word
-term << (pp.Optional(field_name("field") + COLON)
- + (word_expr | string_expr | range_search | pp.Group(LPAR + expression + RPAR))
- + pp.Optional(boost))
-term.setParseAction(lambda t:[t] if 'field' in t or 'boost' in t else None)
+term << (
+ pp.Optional(field_name("field") + COLON)
+ + (word_expr | string_expr | range_search | pp.Group(LPAR + expression + RPAR))
+ + pp.Optional(boost)
+)
+term.setParseAction(lambda t: [t] if "field" in t or "boost" in t else None)
-expression << pp.infixNotation(term,
+expression << pp.infixNotation(
+ term,
[
- (required_modifier | prohibit_modifier, 1, pp.opAssoc.RIGHT),
- ((not_ | '!').setParseAction(lambda: "NOT"), 1, pp.opAssoc.RIGHT),
- ((and_ | '&&').setParseAction(lambda: "AND"), 2, pp.opAssoc.LEFT),
- (pp.Optional(or_ | '||').setParseAction(lambda: "OR"), 2, pp.opAssoc.LEFT),
- ])
+ (required_modifier | prohibit_modifier, 1, pp.opAssoc.RIGHT),
+ ((not_ | "!").setParseAction(lambda: "NOT"), 1, pp.opAssoc.RIGHT),
+ ((and_ | "&&").setParseAction(lambda: "AND"), 2, pp.opAssoc.LEFT),
+ (
+ pp.Optional(or_ | "||").setName("or").setParseAction(lambda: "OR"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ ],
+)
-if __name__ == '__main__':
+if __name__ == "__main__":
# test strings taken from grammar description doc, and TestQueryParser.java
tests = r"""
if not (success1 and success2):
import sys
+
sys.exit(1)
\r
# define the structure of a macro definition (the empty term is used\r
# to advance to the next non-whitespace character)\r
-identifier = Word(alphas+"_",alphanums+"_")\r
+identifier = Word(alphas + "_", alphanums + "_")\r
macroDef = "#def" + identifier("macro") + empty + restOfLine("value")\r
\r
# define a placeholder for defined macros - initially nothing\r
macros = {}\r
\r
# parse action for macro definitions\r
-def processMacroDefn(s,l,t):\r
+def processMacroDefn(s, l, t):\r
macroVal = macroExpander.transformString(t.value)\r
macros[t.macro] = macroVal\r
macroExpr << MatchFirst(map(Keyword, macros.keys()))\r
return "#def " + t.macro + " " + macroVal\r
\r
+\r
# parse action to replace macro references with their respective definition\r
-def processMacroRef(s,l,t):\r
+def processMacroRef(s, l, t):\r
return macros[t[0]]\r
\r
+\r
# attach parse actions to expressions\r
macroExpr.setParseAction(processMacroRef)\r
macroDef.setParseAction(processMacroDefn)\r
macroExpander = macroExpr | macroDef\r
\r
\r
-\r
# test macro substitution using transformString\r
testString = """\r
#def A 100\r
--- /dev/null
+#
+# make_diagram.py
+#
+# Sample railroad diagrams of selected pyparsing examples.
+#
+# Copyright 2021, Paul McGuire
+
+# Uncomment the related import statement and rerun to construct railroad diagram
+
+from examples.delta_time import time_expression as imported_expr
+
+# from examples.sexpParser import sexp as imported_expr
+# from examples.ebnftest import ebnf_parser as imported_expr
+# from examples.jsonParser import jsonObject as imported_expr
+# from examples.lucene_grammar import expression as imported_expr
+# from examples.invRegex import parser; imported_expr = parser()
+# from examples.oc import program as imported_expr
+# from examples.mozillaCalendarParser import calendars as imported_expr
+# from examples.pgn import pgnGrammar as imported_expr
+# from examples.idlParse import CORBA_IDL_BNF; imported_expr = CORBA_IDL_BNF()
+# from examples.chemicalFormulas import formula as imported_expr
+# from examples.romanNumerals import romanNumeral as imported_expr
+# from examples.protobuf_parser import parser as imported_expr
+# from examples.parsePythonValue import listItem as imported_expr
+# from examples.one_to_ninety_nine import one_to_99 as imported_expr
+# from examples.simpleSQL import simpleSQL as imported_expr
+# from examples.simpleBool import boolExpr as imported_expr
+grammar = imported_expr
+
+# or define a custom grammar here
+# import pyparsing as pp
+# grammar = pp.Or(["foo", "bar"]) + pp.Word(pp.nums) + pp.pyparsing_common.uuid
+
+grammar.create_diagram(output_html="output.html", show_results_names=True)
classIdent = identifier("classname") # note that this also makes a copy of identifier\r
classHead = "class" + classIdent\r
classBody = "..."\r
-classEnd = "end" + matchPreviousLiteral(classIdent) + ';'\r
+classEnd = "end" + matchPreviousLiteral(classIdent) + ";"\r
classDefn = classHead + classBody + classEnd\r
\r
# use this form to catch syntax error\r
-from pyparsing import Optional, oneOf, Literal, Word, printables, Group, OneOrMore, ZeroOrMore
+from pyparsing import (
+ Optional,
+ oneOf,
+ Literal,
+ Word,
+ printables,
+ Group,
+ OneOrMore,
+ ZeroOrMore,
+)
"""
A simple parser for calendar (*.ics) files,
# TERMINALS
-BEGIN = Literal("BEGIN:").suppress()
-END = Literal("END:").suppress()
-valstr = printables + "\xe4\xf6\xe5\xd6\xc4\xc5 "
+BEGIN = Literal("BEGIN:").suppress()
+END = Literal("END:").suppress()
+valstr = printables + "\xe4\xf6\xe5\xd6\xc4\xc5 "
-EQ = Literal("=").suppress()
-SEMI = Literal(";").suppress()
-COLON = Literal(":").suppress()
+EQ = Literal("=").suppress()
+SEMI = Literal(";").suppress()
+COLON = Literal(":").suppress()
-EVENT = Literal("VEVENT").suppress()
-CALENDAR = Literal("VCALENDAR").suppress()
-ALARM = Literal("VALARM").suppress()
+EVENT = Literal("VEVENT").suppress()
+CALENDAR = Literal("VCALENDAR").suppress()
+ALARM = Literal("VALARM").suppress()
# TOKENS
-CALPROP = oneOf("VERSION PRODID METHOD")
-ALMPROP = oneOf("TRIGGER")
-EVTPROP = oneOf("X-MOZILLA-RECUR-DEFAULT-INTERVAL \
- X-MOZILLA-RECUR-DEFAULT-UNITS \
- UID DTSTAMP LAST-MODIFIED X RRULE EXDATE")
+CALPROP = oneOf("VERSION PRODID METHOD", asKeyword=True)
+ALMPROP = oneOf("TRIGGER", asKeyword=True)
+EVTPROP = oneOf(
+ """X-MOZILLA-RECUR-DEFAULT-INTERVAL
+ X-MOZILLA-RECUR-DEFAULT-UNITS
+ UID DTSTAMP LAST-MODIFIED X RRULE EXDATE""", asKeyword=True
+)
-propval = Word(valstr)
-typeval = Word(valstr)
-typename = oneOf("VALUE MEMBER FREQ UNTIL INTERVAL")
+valuestr = Word(valstr).setName("valuestr")
+propval = valuestr
+typeval = valuestr
+typename = oneOf("VALUE MEMBER FREQ UNTIL INTERVAL", asKeyword=True)
-proptype = Group(SEMI + typename + EQ + typeval).suppress()
+proptype = Group(SEMI + typename + EQ + typeval).setName("proptype").suppress()
calprop = Group(CALPROP + ZeroOrMore(proptype) + COLON + propval)
almprop = Group(ALMPROP + ZeroOrMore(proptype) + COLON + propval)
-evtprop = Group(EVTPROP + ZeroOrMore(proptype) + COLON + propval).suppress() \
- | "CATEGORIES" + COLON + propval.setResultsName("categories") \
- | "CLASS" + COLON + propval.setResultsName("class") \
- | "DESCRIPTION" + COLON + propval.setResultsName("description") \
- | "DTSTART" + proptype + COLON + propval.setResultsName("begin") \
- | "DTEND" + proptype + COLON + propval.setResultsName("end") \
- | "LOCATION" + COLON + propval.setResultsName("location") \
- | "PRIORITY" + COLON + propval.setResultsName("priority") \
- | "STATUS" + COLON + propval.setResultsName("status") \
- | "SUMMARY" + COLON + propval.setResultsName("summary") \
- | "URL" + COLON + propval.setResultsName("url") \
-
-calprops = Group(OneOrMore(calprop)).suppress()
+evtprop = (
+ Group(EVTPROP + ZeroOrMore(proptype) + COLON + propval).suppress()
+ | "CATEGORIES" + COLON + propval.setResultsName("categories")
+ | "CLASS" + COLON + propval.setResultsName("class")
+ | "DESCRIPTION" + COLON + propval.setResultsName("description")
+ | "DTSTART" + proptype + COLON + propval.setResultsName("begin")
+ | "DTEND" + proptype + COLON + propval.setResultsName("end")
+ | "LOCATION" + COLON + propval.setResultsName("location")
+ | "PRIORITY" + COLON + propval.setResultsName("priority")
+ | "STATUS" + COLON + propval.setResultsName("status")
+ | "SUMMARY" + COLON + propval.setResultsName("summary")
+ | "URL" + COLON + propval.setResultsName("url")
+).setName("evtprop")
+calprops = Group(OneOrMore(calprop)).setName("calprops").suppress()
evtprops = Group(OneOrMore(evtprop))
-almprops = Group(OneOrMore(almprop)).suppress()
+almprops = Group(OneOrMore(almprop)).setName("almprops").suppress()
-alarm = BEGIN + ALARM + almprops + END + ALARM
-event = BEGIN + EVENT + evtprops + Optional(alarm) + END + EVENT
-events = Group(OneOrMore(event))
-calendar = BEGIN + CALENDAR + calprops + ZeroOrMore(event) + END + CALENDAR
-calendars = OneOrMore(calendar)
+alarm = (BEGIN + ALARM + almprops + END + ALARM).setName("alarm")
+event = (BEGIN + EVENT + evtprops + Optional(alarm) + END + EVENT).setName("event")
+events = Group(OneOrMore(event))
+calendar = (BEGIN + CALENDAR + calprops + ZeroOrMore(event) + END + CALENDAR).setName("calendar")
+calendars = OneOrMore(calendar)
# PARSE ACTIONS
-def gotEvent(s,loc,toks):
- for event in toks:
- print(event.dump())
+
+def gotEvent(s, loc, toks):
+ for event in toks:
+ print(event.dump())
+
event.setParseAction(gotEvent)
# MAIN PROGRAM
-if __name__=="__main__":
+if __name__ == "__main__":
- calendars.parseFile("mozilla.ics")
+ calendars.parseFile("mozilla.ics")
\r
# use {}'s for nested lists\r
nestedItems = nestedExpr("{", "}")\r
-print(( (nestedItems+stringEnd).parseString(data).asList() ))\r
+print((nestedItems + stringEnd).parseString(data).asList())\r
\r
# use default delimiters of ()'s\r
mathExpr = nestedExpr()\r
-print(( mathExpr.parseString( "((( ax + by)*C) *(Z | (E^F) & D))") ))\r
+print(mathExpr.parseString("((( ax + by)*C) *(Z | (E^F) & D))"))\r
# a method that will construct and return a parse action that will
# do the proper wrapping in opening and closing HTML, and recursively call
# wiki_markup.transformString on the markup body text
-def convert_markup_to_html(opening,closing):
+def convert_markup_to_html(opening, closing):
def conversionParseAction(s, l, t):
return opening + wiki_markup.transformString(t[1][1:-1]) + closing
+
return conversionParseAction
+
# use a nestedExpr with originalTextFor to parse nested braces, but return the
# parsed text as a single string containing the outermost nested braces instead
# of a nested list of parsed tokens
-markup_body = pp.originalTextFor(pp.nestedExpr('{', '}'))
-italicized = ('ital' + markup_body).setParseAction(convert_markup_to_html("<I>", "</I>"))
-bolded = ('bold' + markup_body).setParseAction(convert_markup_to_html("<B>", "</B>"))
+markup_body = pp.originalTextFor(pp.nestedExpr("{", "}"))
+italicized = ("ital" + markup_body).setParseAction(
+ convert_markup_to_html("<I>", "</I>")
+)
+bolded = ("bold" + markup_body).setParseAction(convert_markup_to_html("<B>", "</B>"))
# another markup and parse action to parse links - again using transform string
# to recursively parse any markup in the link text
def convert_link_to_html(s, l, t):
link_text, url = t._skipped
- t['link_text'] = wiki_markup.transformString(link_text)
- t['url'] = url
+ t["link_text"] = wiki_markup.transformString(link_text)
+ t["url"] = url
return '<A href="{url}">{link_text}</A>'.format_map(t)
-urlRef = (pp.Keyword('link') + '{' + ... + '->' + ... + '}').setParseAction(convert_link_to_html)
+
+urlRef = (pp.Keyword("link") + "{" + ... + "->" + ... + "}").setParseAction(
+ convert_link_to_html
+)
# now inject all the markup bits as possible markup expressions
wiki_markup <<= urlRef | italicized | bolded
--- /dev/null
+# number_words.py
+#
+# Copyright 2020, Paul McGuire
+#
+# Parser/evaluator for expressions of numbers as written out in words:
+# - one
+# - seven
+# - twelve
+# - twenty six
+# - forty-two
+# - one hundred and seven
+#
+#
+# BNF:
+"""
+ optional_and ::= ["and" | "-"]
+ optional_dash ::= ["-"]
+ units ::= one | two | three | ... | nine
+ teens ::= ten | teens_only
+ tens ::= twenty | thirty | ... | ninety
+ one_to_99 ::= units | teens | (tens [optional_dash units])
+ teens_only ::= eleven | twelve | ... | nineteen
+ hundreds ::= (units | teens_only | tens optional_dash units) "hundred"
+ thousands ::= one_to_99 "thousand"
+
+ # number from 1-999,999
+ number ::= [thousands [optional_and]] [hundreds[optional_and]] one_to_99
+ | [thousands [optional_and]] hundreds
+ | thousands
+"""
+import pyparsing as pp
+from operator import mul
+
+
+def define_numeric_word_range(
+ names: str, from_: int, to_: int = None, step: int = 1
+) -> pp.MatchFirst:
+ """
+ Compose a MatchFirst of CaselessKeywords, given their names and values,
+ which when parsed, are converted to their value
+ """
+
+ def define_numeric_word(nm: str, val: int):
+ return pp.CaselessKeyword(nm).add_parse_action(lambda: val)
+
+ names = names.split()
+ if to_ is None:
+ to_ = from_
+ values = range(from_, to_ + 1, step)
+ ret = pp.MatchFirst(
+ define_numeric_word(name, value) for name, value in zip(names, values)
+ )
+
+ if len(names) == 1:
+ ret.setName(names[0])
+ else:
+ ret.setName("{}-{}".format(names[0], names[-1]))
+
+ return ret
+
+
+def multiply(t):
+ """
+ Parse action for hundreds and thousands.
+ """
+ return mul(*t)
+
+
+opt_dash = pp.Optional(pp.Suppress("-")).setName("'-'")
+opt_and = pp.Optional((pp.CaselessKeyword("and") | "-").suppress()).setName("'and/-'")
+
+units = define_numeric_word_range("one two three four five six seven eight nine", 1, 9)
+teens_only = define_numeric_word_range(
+ "eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen",
+ 11,
+ 19,
+)
+ten = define_numeric_word_range("ten", 10)
+teens = ten | teens_only
+
+tens = define_numeric_word_range(
+ "twenty thirty forty fifty sixty seventy eighty ninety", 20, 90, 10
+)
+one_to_99 = (units | teens | (tens + pp.Optional(opt_dash + units))).setName("1-99")
+one_to_99.addParseAction(sum)
+
+hundred = define_numeric_word_range("hundred", 100)
+thousand = define_numeric_word_range("thousand", 1000)
+
+hundreds = (units | teens_only | (tens + opt_dash + units)) + hundred
+hundreds.setName("100s")
+
+one_to_999 = (
+ (pp.Optional(hundreds + opt_and) + one_to_99 | hundreds).addParseAction(sum)
+).setName("1-999")
+
+thousands = one_to_999 + thousand
+thousands.setName("1000s")
+
+# for hundreds and thousands, must scale up (multiply) accordingly
+hundreds.addParseAction(multiply)
+thousands.addParseAction(multiply)
+
+numeric_expression = (
+ pp.Optional(thousands + opt_and) + pp.Optional(hundreds + opt_and) + one_to_99
+ | pp.Optional(thousands + opt_and) + hundreds
+ | thousands
+).setName("numeric_words")
+
+# sum all sub-results into total
+numeric_expression.addParseAction(sum)
+
+
+if __name__ == "__main__":
+ numeric_expression.runTests(
+ """
+ one
+ seven
+ twelve
+ twenty six
+ forty-two
+ two hundred
+ twelve hundred
+ one hundred and eleven
+ ninety nine thousand nine hundred and ninety nine
+ nine hundred thousand nine hundred and ninety nine
+ nine hundred and ninety nine thousand nine hundred and ninety nine
+ nineteen hundred thousand nineteen hundred and ninety nine
+
+ # invalid
+ twenty hundred
+ """,
+ postParse=lambda _, s: "{:,}".format(s[0]),
+ )
+
+ # create railroad diagram
+ numeric_expression.create_diagram("numeric_words_diagram.html", vertical=5)
\r
from pyparsing import Regex\r
\r
-comma_decimal = Regex(r'\d{1,2}(([ .])\d\d\d(\2\d\d\d)*)?,\d*')\r
-comma_decimal.setParseAction(lambda t: float(t[0].replace(' ','').replace('.','').replace(',','.')))\r
+comma_decimal = Regex(r"\d{1,2}(([ .])\d\d\d(\2\d\d\d)*)?,\d*")\r
+comma_decimal.setParseAction(\r
+ lambda t: float(t[0].replace(" ", "").replace(".", "").replace(",", "."))\r
+)\r
\r
-dot_decimal = Regex(r'\d{1,2}(([ ,])\d\d\d(\2\d\d\d)*)?\.\d*')\r
-dot_decimal.setParseAction(lambda t: float(t[0].replace(' ','').replace(',','')))\r
+dot_decimal = Regex(r"\d{1,2}(([ ,])\d\d\d(\2\d\d\d)*)?\.\d*")\r
+dot_decimal.setParseAction(lambda t: float(t[0].replace(" ", "").replace(",", "")))\r
\r
decimal = comma_decimal ^ dot_decimal\r
decimal.runTests(tests, parseAll=True)\r
\r
-grouped_integer = Regex(r'\d{1,2}(([ .,])\d\d\d(\2\d\d\d)*)?')\r
-grouped_integer.setParseAction(lambda t: int(t[0].replace(' ','').replace(',','').replace('.','')))\r
+grouped_integer = Regex(r"\d{1,2}(([ .,])\d\d\d(\2\d\d\d)*)?")\r
+grouped_integer.setParseAction(\r
+ lambda t: int(t[0].replace(" ", "").replace(",", "").replace(".", ""))\r
+)\r
grouped_integer.runTests(tests, parseAll=False)\r
"""\r
\r
from pyparsing import *\r
+\r
ParserElement.enablePackrat()\r
\r
-LPAR,RPAR,LBRACK,RBRACK,LBRACE,RBRACE,SEMI,COMMA = map(Suppress, "()[]{};,")\r
-INT, CHAR, WHILE, DO, IF, ELSE, RETURN = map(Keyword,\r
- "int char while do if else return".split())\r
+LPAR, RPAR, LBRACK, RBRACK, LBRACE, RBRACE, SEMI, COMMA = map(Suppress, "()[]{};,")\r
+INT, CHAR, WHILE, DO, IF, ELSE, RETURN = map(\r
+ Keyword, "int char while do if else return".split()\r
+)\r
\r
-NAME = Word(alphas+"_", alphanums+"_")\r
+NAME = Word(alphas + "_", alphanums + "_")\r
integer = Regex(r"[+-]?\d+")\r
char = Regex(r"'.'")\r
string_ = dblQuotedString\r
expr = Forward()\r
func_call = Group(NAME + LPAR + Group(Optional(delimitedList(expr))) + RPAR)\r
operand = func_call | NAME | integer | char | string_\r
-expr <<= (infixNotation(operand,\r
+expr <<= infixNotation(\r
+ operand,\r
[\r
- (oneOf('! - *'), 1, opAssoc.RIGHT),\r
- (oneOf('++ --'), 1, opAssoc.RIGHT),\r
- (oneOf('++ --'), 1, opAssoc.LEFT),\r
- (oneOf('* / %'), 2, opAssoc.LEFT),\r
- (oneOf('+ -'), 2, opAssoc.LEFT),\r
- (oneOf('< == > <= >= !='), 2, opAssoc.LEFT),\r
- (Regex(r'(?<!=)=(?!=)'), 2, opAssoc.LEFT),\r
- ]) +\r
- Optional( LBRACK + expr + RBRACK |\r
- LPAR + Group(Optional(delimitedList(expr))) + RPAR )\r
- )\r
+ (oneOf("! - *"), 1, opAssoc.RIGHT),\r
+ (oneOf("++ --"), 1, opAssoc.RIGHT),\r
+ (oneOf("++ --"), 1, opAssoc.LEFT),\r
+ (oneOf("* / %"), 2, opAssoc.LEFT),\r
+ (oneOf("+ -"), 2, opAssoc.LEFT),\r
+ (oneOf("< == > <= >= !="), 2, opAssoc.LEFT),\r
+ (Regex(r"(?<!=)=(?!=)"), 2, opAssoc.LEFT),\r
+ ],\r
+) + Optional(\r
+ LBRACK + expr + RBRACK | LPAR + Group(Optional(delimitedList(expr))) + RPAR\r
+)\r
\r
stmt = Forward()\r
\r
dowhilestmt = DO - stmt + WHILE + LPAR + expr + RPAR + SEMI\r
returnstmt = RETURN - expr + SEMI\r
\r
-stmt << Group( ifstmt |\r
- whilestmt |\r
- dowhilestmt |\r
- returnstmt |\r
- expr + SEMI |\r
- LBRACE + ZeroOrMore(stmt) + RBRACE |\r
- SEMI)\r
+stmt << Group(\r
+ ifstmt\r
+ | whilestmt\r
+ | dowhilestmt\r
+ | returnstmt\r
+ | expr + SEMI\r
+ | LBRACE + ZeroOrMore(stmt) + RBRACE\r
+ | SEMI\r
+)\r
\r
vardecl = Group(TYPE + NAME + Optional(LBRACK + integer + RBRACK)) + SEMI\r
\r
arg = Group(TYPE + NAME)\r
body = ZeroOrMore(vardecl) + ZeroOrMore(stmt)\r
-fundecl = Group(TYPE + NAME + LPAR + Optional(Group(delimitedList(arg))) + RPAR +\r
- LBRACE + Group(body) + RBRACE)\r
+fundecl = Group(\r
+ TYPE\r
+ + NAME\r
+ + LPAR\r
+ + Optional(Group(delimitedList(arg)))\r
+ + RPAR\r
+ + LBRACE\r
+ + Group(body)\r
+ + RBRACE\r
+)\r
decl = fundecl | vardecl\r
program = ZeroOrMore(decl)\r
\r
program.ignore(cStyleComment)\r
\r
# set parser element names\r
-for vname in ("ifstmt whilestmt dowhilestmt returnstmt TYPE "\r
- "NAME fundecl vardecl program arg body stmt".split()):\r
+for vname in (\r
+ "ifstmt whilestmt dowhilestmt returnstmt TYPE "\r
+ "NAME fundecl vardecl program arg body stmt".split()\r
+):\r
v = vars()[vname]\r
v.setName(vname)\r
\r
-#~ for vname in "fundecl stmt".split():\r
- #~ v = vars()[vname]\r
- #~ v.setDebug()\r
-\r
-test = r"""\r
-/* A factorial program */\r
-int\r
-putstr(char *s)\r
-{\r
- while(*s)\r
- putchar(*s++);\r
-}\r
-\r
-int\r
-fac(int n)\r
-{\r
- if (n == 0)\r
- return 1;\r
- else\r
- return n*fac(n-1);\r
-}\r
-\r
-int\r
-putn(int n)\r
-{\r
- if (9 < n)\r
- putn(n / 10);\r
- putchar((n%10) + '0');\r
-}\r
-\r
-int\r
-facpr(int n)\r
-{\r
- putstr("factorial ");\r
- putn(n);\r
- putstr(" = ");\r
- putn(fac(n));\r
- putstr("\n");\r
-}\r
-\r
-int\r
-main()\r
-{\r
- int i;\r
- i = 0;\r
- if(a() == 1){}\r
- while(i < 10)\r
- facpr(i++);\r
- return 0;\r
-}\r
-"""\r
-\r
-ast = program.parseString(test, parseAll=True)\r
-ast.pprint()\r
+# ~ for vname in "fundecl stmt".split():\r
+# ~ v = vars()[vname]\r
+# ~ v.setDebug()\r
+\r
+\r
+def main():\r
+ test = r"""\r
+ /* A factorial program */\r
+ int\r
+ putstr(char *s)\r
+ {\r
+ while(*s)\r
+ putchar(*s++);\r
+ }\r
+ \r
+ int\r
+ fac(int n)\r
+ {\r
+ if (n == 0)\r
+ return 1;\r
+ else\r
+ return n*fac(n-1);\r
+ }\r
+ \r
+ int\r
+ putn(int n)\r
+ {\r
+ if (9 < n)\r
+ putn(n / 10);\r
+ putchar((n%10) + '0');\r
+ }\r
+ \r
+ int\r
+ facpr(int n)\r
+ {\r
+ putstr("factorial ");\r
+ putn(n);\r
+ putstr(" = ");\r
+ putn(fac(n));\r
+ putstr("\n");\r
+ }\r
+ \r
+ int\r
+ main()\r
+ {\r
+ int i;\r
+ i = 0;\r
+ if(a() == 1){}\r
+ while(i < 10)\r
+ facpr(i++);\r
+ return 0;\r
+ }\r
+ """\r
+\r
+ ast = program.parseString(test, parseAll=True)\r
+ ast.pprint()\r
+\r
+\r
+if __name__ == "__main__":\r
+ main()\r
--- /dev/null
+#
+# one_to_ninety_nine.py
+#
+# Copyright 2021, Paul McGuire
+#
+# Parser/evaluator for expressions of numbers as written out in words:
+# - one
+# - seven
+# - twelve
+# - twenty six
+# - forty-two
+#
+# BNF:
+# units ::= one | two | three | ... | nine
+# teens ::= ten | eleven | twelve | ... | nineteen
+# tens ::= twenty | thirty | ... | ninety
+# one_to_99 ::= units | teens | (tens [["-"] units])
+#
+import pyparsing as pp
+
+
+def define_numeric_word_range(
+ names: str, from_: int, to_: int, step: int = 1
+) -> pp.MatchFirst:
+ """
+ Compose a MatchFirst of CaselessKeywords, given their names and values,
+ which when parsed, are converted to their value
+ """
+
+ def define_numeric_word(nm: str, val: int):
+ return pp.CaselessKeyword(nm).add_parse_action(lambda: val)
+
+ names = names.split()
+ values = range(from_, to_ + 1, step)
+ return pp.MatchFirst(
+ define_numeric_word(name, value) for name, value in zip(names, values)
+ )
+
+
+units = define_numeric_word_range(
+ "one two three four five six seven eight nine", 1, 9
+).set_name("units")
+teens = define_numeric_word_range(
+ "ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen",
+ 10,
+ 19,
+).set_name("teens")
+tens = define_numeric_word_range(
+ "twenty thirty forty fifty sixty seventy eighty ninety", 20, 90, step=10
+).set_name("tens")
+
+opt_dash = pp.Opt(pp.Suppress("-"))
+twenty_to_99 = tens + pp.Opt(opt_dash + units)
+
+one_to_99 = (units | teens | twenty_to_99).set_name("1-99")
+
+# for expressions that parse multiple values, add them up
+one_to_99.add_parse_action(sum)
+
+numeric_expression = one_to_99
+
+if __name__ == "__main__":
+ numeric_expression.run_tests(
+ """
+ one
+ seven
+ twelve
+ twenty six
+ forty-two
+ """
+ )
+
+ # create railroad diagram
+ numeric_expression.create_diagram("one_to_99_diagram.html", vertical=5)
+++ /dev/null
-# parseListString.py\r
-#\r
-# Copyright, 2006, by Paul McGuire\r
-#\r
-\r
-from pyparsing import *\r
-\r
-# first pass\r
-lbrack = Literal("[")\r
-rbrack = Literal("]")\r
-integer = Word(nums).setName("integer")\r
-real = Combine(Optional(oneOf("+ -")) + Word(nums) + "." +\r
- Optional(Word(nums))).setName("real")\r
-\r
-listItem = real | integer | quotedString\r
-\r
-listStr = lbrack + delimitedList(listItem) + rbrack\r
-\r
-test = "['a', 100, 3.14]"\r
-\r
-print(listStr.parseString(test))\r
-\r
-\r
-# second pass, cleanup and add converters\r
-lbrack = Literal("[").suppress()\r
-rbrack = Literal("]").suppress()\r
-cvtInt = lambda s,l,toks: int(toks[0])\r
-integer = Word(nums).setName("integer").setParseAction( cvtInt )\r
-cvtReal = lambda s,l,toks: float(toks[0])\r
-real = Regex(r'[+-]?\d+\.\d*').setName("floating-point number").setParseAction( cvtReal )\r
-listItem = real | integer | quotedString.setParseAction( removeQuotes )\r
-\r
-listStr = lbrack + delimitedList(listItem) + rbrack\r
-\r
-test = "['a', 100, 3.14]"\r
-\r
-print(listStr.parseString(test))\r
-\r
-# third pass, add nested list support, and tuples, too!\r
-cvtInt = lambda s,l,toks: int(toks[0])\r
-cvtReal = lambda s,l,toks: float(toks[0])\r
-\r
-lbrack = Literal("[").suppress()\r
-rbrack = Literal("]").suppress()\r
-integer = Word(nums).setName("integer").setParseAction( cvtInt )\r
-real = Regex(r'[+-]?\d+\.\d*').setName("floating-point number").setParseAction( cvtReal )\r
-tupleStr = Forward()\r
-listStr = Forward()\r
-listItem = real | integer | quotedString.setParseAction(removeQuotes) | Group(listStr) | tupleStr\r
-tupleStr << ( Suppress("(") + delimitedList(listItem) + Optional(Suppress(",")) + Suppress(")") )\r
-tupleStr.setParseAction( lambda t:tuple(t.asList()) )\r
-listStr << lbrack + delimitedList(listItem) + Optional(Suppress(",")) + rbrack\r
-\r
-test = "['a', 100, ('A', [101,102]), 3.14, [ +2.718, 'xyzzy', -1.414] ]"\r
-print(listStr.parseString(test))\r
-\r
-# fourth pass, add parsing of dicts\r
-cvtInt = lambda s,l,toks: int(toks[0])\r
-cvtReal = lambda s,l,toks: float(toks[0])\r
-cvtDict = lambda s,l,toks: dict(toks[0])\r
-\r
-lbrack = Literal("[").suppress()\r
-rbrack = Literal("]").suppress()\r
-lbrace = Literal("{").suppress()\r
-rbrace = Literal("}").suppress()\r
-colon = Literal(":").suppress()\r
-integer = Word(nums).setName("integer").setParseAction( cvtInt )\r
-real = Regex(r'[+-]?\d+\.\d*').setName("real").setParseAction( cvtReal )\r
-\r
-tupleStr = Forward()\r
-listStr = Forward()\r
-dictStr = Forward()\r
-listItem = real | integer | quotedString.setParseAction(removeQuotes) | Group(listStr) | tupleStr | dictStr\r
-tupleStr <<= ( Suppress("(") + delimitedList(listItem) + Optional(Suppress(",")) + Suppress(")") )\r
-tupleStr.setParseAction( lambda t:tuple(t.asList()) )\r
-listStr <<= (lbrack + Optional(delimitedList(listItem)) + Optional(Suppress(",")) + rbrack)\r
-dictKeyStr = real | integer | quotedString.setParseAction(removeQuotes)\r
-dictStr <<= lbrace + Optional(delimitedList( Group( dictKeyStr + colon + listItem ))) + Optional(Suppress(",")) + rbrace\r
-dictStr.setParseAction(lambda t: {k_v[0]:(k_v[1].asList() if isinstance(k_v[1],ParseResults) else k_v[1]) for k_v in t})\r
-\r
-test = '[{0: [2], 1: []}, {0: [], 1: [], 2: [,]}, {0: [1, 2,],}]'\r
-print(listStr.parseString(test))\r
#
# Copyright, 2006, by Paul McGuire
#
-from __future__ import print_function
import pyparsing as pp
-cvtBool = lambda t:t[0]=='True'
+cvtBool = lambda t: t[0] == "True"
cvtInt = lambda toks: int(toks[0])
cvtReal = lambda toks: float(toks[0])
-cvtTuple = lambda toks : tuple(toks.asList())
+cvtTuple = lambda toks: tuple(toks.asList())
cvtDict = lambda toks: dict(toks.asList())
cvtList = lambda toks: [toks.asList()]
# define punctuation as suppressed literals
-lparen, rparen, lbrack, rbrack, lbrace, rbrace, colon, comma = map(pp.Suppress,"()[]{}:,")
+lparen, rparen, lbrack, rbrack, lbrace, rbrace, colon, comma = map(
+ pp.Suppress, "()[]{}:,"
+)
-integer = pp.Regex(r"[+-]?\d+").setName("integer").setParseAction(cvtInt )
+integer = pp.Regex(r"[+-]?\d+").setName("integer").setParseAction(cvtInt)
real = pp.Regex(r"[+-]?\d+\.\d*([Ee][+-]?\d+)?").setName("real").setParseAction(cvtReal)
tupleStr = pp.Forward()
listStr = pp.Forward()
dictStr = pp.Forward()
-pp.unicodeString.setParseAction(lambda t:t[0][2:-1])
-pp.quotedString.setParseAction(lambda t:t[0][1:-1])
-boolLiteral = pp.oneOf("True False").setParseAction(cvtBool)
-noneLiteral = pp.Literal("None").setParseAction(pp.replaceWith(None))
+unistr = pp.unicodeString().setParseAction(lambda t: t[0][2:-1])
+quoted_str = pp.quotedString().setParseAction(lambda t: t[0][1:-1])
+boolLiteral = pp.oneOf("True False", asKeyword=True).setParseAction(cvtBool)
+noneLiteral = pp.Keyword("None").setParseAction(pp.replaceWith(None))
-listItem = (real
- | integer
- | pp.quotedString
- | pp.unicodeString
- | boolLiteral
- | noneLiteral
- | pp.Group(listStr)
- | tupleStr
- | dictStr)
+listItem = (
+ real
+ | integer
+ | quoted_str
+ | unistr
+ | boolLiteral
+ | noneLiteral
+ | pp.Group(listStr)
+ | tupleStr
+ | dictStr
+)
-tupleStr << (lparen
- + pp.Optional(pp.delimitedList(listItem))
- + pp.Optional(comma)
- + rparen)
+tupleStr <<= (
+ lparen + pp.Optional(pp.delimitedList(listItem)) + pp.Optional(comma) + rparen
+)
tupleStr.setParseAction(cvtTuple)
-listStr << (lbrack
- + pp.Optional(pp.delimitedList(listItem) + pp.Optional(comma))
- + rbrack)
+listStr <<= (
+ lbrack + pp.Optional(pp.delimitedList(listItem) + pp.Optional(comma)) + rbrack
+)
listStr.setParseAction(cvtList, lambda t: t[0])
dictEntry = pp.Group(listItem + colon + listItem)
-dictStr << (lbrace
- + pp.Optional(pp.delimitedList(dictEntry) + pp.Optional(comma))
- + rbrace)
+dictStr <<= (
+ lbrace + pp.Optional(pp.delimitedList(dictEntry) + pp.Optional(comma)) + rbrace
+)
dictStr.setParseAction(cvtDict)
-tests = """['a', 100, ('A', [101,102]), 3.14, [ +2.718, 'xyzzy', -1.414] ]
- [{0: [2], 1: []}, {0: [], 1: [], 2: []}, {0: [1, 2]}]
- { 'A':1, 'B':2, 'C': {'a': 1.2, 'b': 3.4} }
- 3.14159
- 42
- 6.02E23
- 6.02e+023
- 1.0e-7
- 'a quoted string'"""
+if __name__ == "__main__":
-listItem.runTests(tests)
+ tests = """['a', 100, ('A', [101,102]), 3.14, [ +2.718, 'xyzzy', -1.414] ]
+ [{0: [2], 1: []}, {0: [], 1: [], 2: []}, {0: [1, 2]}]
+ { 'A':1, 'B':2, 'C': {'a': 1.2, 'b': 3.4} }
+ 3.14159
+ 42
+ 6.02E23
+ 6.02e+023
+ 1.0e-7
+ 'a quoted string'"""
+
+ listItem.runTests(tests)
samplestr4 = "garbage;ID PARI12345678;more garbage- I am cool"\r
\r
from pyparsing import *\r
+\r
dob_ref = "DOB" + Regex(r"\d{2}-\d{2}-\d{4}")("dob")\r
-id_ref = "ID" + Word(alphanums,exact=12)("id")\r
+id_ref = "ID" + Word(alphanums, exact=12)("id")\r
info_ref = "-" + restOfLine("info")\r
\r
person_data = dob_ref | id_ref | info_ref\r
\r
-for test in (samplestr1,samplestr2,samplestr3,samplestr4,):\r
+for test in (\r
+ samplestr1,\r
+ samplestr2,\r
+ samplestr3,\r
+ samplestr4,\r
+):\r
person = sum(person_data.searchString(test))\r
print(person.id)\r
print(person.dump())\r
#\r
# Copyright 2015, Paul McGuire\r
#\r
-from pyparsing import col,Word,Optional,alphas,nums\r
+from pyparsing import col, Word, Optional, alphas, nums\r
\r
table = """\\r
1 2\r
PURPLE 8"""\r
\r
# function to create column-specific parse conditions\r
-def mustMatchCols(startloc,endloc):\r
- return lambda s,l,t: startloc <= col(l,s) <= endloc\r
+def mustMatchCols(startloc, endloc):\r
+ return lambda s, l, t: startloc <= col(l, s) <= endloc\r
+\r
\r
# helper to define values in a space-delimited table\r
# (change empty_cell_is_zero to True if a value of 0 is desired for empty cells)\r
def tableValue(expr, colstart, colend):\r
empty_cell_is_zero = False\r
if empty_cell_is_zero:\r
- return Optional(expr.copy().addCondition(mustMatchCols(colstart,colend),\r
- message="text not in expected columns"),\r
- default=0)\r
+ return Optional(\r
+ expr.copy().addCondition(\r
+ mustMatchCols(colstart, colend), message="text not in expected columns"\r
+ ),\r
+ default=0,\r
+ )\r
else:\r
- return Optional(expr.copy().addCondition(mustMatchCols(colstart,colend),\r
- message="text not in expected columns"))\r
+ return Optional(\r
+ expr.copy().addCondition(\r
+ mustMatchCols(colstart, colend), message="text not in expected columns"\r
+ )\r
+ )\r
\r
\r
# define the grammar for this simple table\r
colorname = Word(alphas)\r
integer = Word(nums).setParseAction(lambda t: int(t[0])).setName("integer")\r
-row = (colorname("name") +\r
- tableValue(integer, 11, 12)("S") +\r
- tableValue(integer, 15, 16)("M") +\r
- tableValue(integer, 19, 20)("L"))\r
+row = (\r
+ colorname("name")\r
+ + tableValue(integer, 11, 12)("S")\r
+ + tableValue(integer, 15, 16)("M")\r
+ + tableValue(integer, 19, 20)("L")\r
+)\r
\r
# parse the sample text - skip over the header and counter lines\r
for line in table.splitlines()[3:]:\r
print(line)\r
print(row.parseString(line).dump())\r
- print('')\r
+ print("")\r
-# parital_gene_match.py\r
-#\r
-# Example showing how to create a customized pyparsing Token, in this case,\r
-# one that is very much like Literal, but which tolerates up to 'n' character\r
-# mismatches\r
-from pyparsing import *\r
-\r
-import urllib.request, urllib.parse, urllib.error\r
-\r
-# read in a bunch of genomic data\r
-datafile = urllib.request.urlopen("http://toxodb.org/common/downloads/release-6.0/Tgondii/TgondiiApicoplastORFsNAs_ToxoDB-6.0.fasta")\r
-fastasrc = datafile.read()\r
-datafile.close()\r
-\r
-"""\r
-Sample header:\r
->NC_001799-6-2978-2778 | organism=Toxoplasma_gondii_RH | location=NC_001799:2778-2978(-) | length=201\r
-"""\r
-integer = Word(nums).setParseAction(lambda t:int(t[0]))\r
-genebit = Group(">" + Word(alphanums.upper()+"-_") + "|" +\r
- Word(printables)("id") + SkipTo("length=", include=True) +\r
- integer("genelen") + LineEnd() +\r
- Combine(OneOrMore(Word("ACGTN")),adjacent=False)("gene"))\r
-\r
-# read gene data from .fasta file - takes just a few seconds\r
-genedata = OneOrMore(genebit).parseString(fastasrc)\r
-\r
-\r
-class CloseMatch(Token):\r
- """A special subclass of Token that does *close* matches. For each\r
- close match of the given string, a tuple is returned giving the\r
- found close match, and a list of mismatch positions."""\r
- def __init__(self, seq, maxMismatches=1):\r
- super(CloseMatch,self).__init__()\r
- self.name = seq\r
- self.sequence = seq\r
- self.maxMismatches = maxMismatches\r
- self.errmsg = "Expected " + self.sequence\r
- self.mayIndexError = False\r
- self.mayReturnEmpty = False\r
-\r
- def parseImpl( self, instring, loc, doActions=True ):\r
- start = loc\r
- instrlen = len(instring)\r
- maxloc = start + len(self.sequence)\r
-\r
- if maxloc <= instrlen:\r
- seq = self.sequence\r
- seqloc = 0\r
- mismatches = []\r
- throwException = False\r
- done = False\r
- while loc < maxloc and not done:\r
- if instring[loc] != seq[seqloc]:\r
- mismatches.append(seqloc)\r
- if len(mismatches) > self.maxMismatches:\r
- throwException = True\r
- done = True\r
- loc += 1\r
- seqloc += 1\r
- else:\r
- throwException = True\r
-\r
- if throwException:\r
- exc = self.myException\r
- exc.loc = loc\r
- exc.pstr = instring\r
- raise exc\r
-\r
- return loc, (instring[start:loc],mismatches)\r
-\r
-# using the genedata extracted above, look for close matches of a gene sequence\r
-searchseq = CloseMatch("TTAAATCTAGAAGAT", 3)\r
-for g in genedata:\r
- print("%s (%d)" % (g.id, g.genelen))\r
- print("-"*24)\r
- for t,startLoc,endLoc in searchseq.scanString(g.gene, overlap=True):\r
- matched, mismatches = t[0]\r
- print("MATCH:", searchseq.sequence)\r
- print("FOUND:", matched)\r
- if mismatches:\r
- print(" ", ''.join(' ' if i not in mismatches else '*'\r
- for i,c in enumerate(searchseq.sequence)))\r
- else:\r
- print("<exact match>")\r
- print("at location", startLoc)\r
- print()\r
- print()\r
+# partial_gene_match.py
+#
+# Example showing how to use the CloseMatch class, to find strings in a gene with up to 'n' mismatches
+#
+import pyparsing as pp
+
+from urllib.request import urlopen
+
+# read in a bunch of genomic data
+data_url = "http://toxodb.org/common/downloads/release-6.0/Tgondii/TgondiiApicoplastORFsNAs_ToxoDB-6.0.fasta"
+with urlopen(data_url) as datafile:
+ fastasrc = datafile.read().decode()
+
+# define parser to extract gene definitions
+"""
+Sample header:
+>NC_001799-6-2978-2778 | organism=Toxoplasma_gondii_RH | location=NC_001799:2778-2978(-) | length=201
+"""
+integer = pp.pyparsing_common.integer
+genebit = pp.Group(
+ ">"
+ + pp.Word(pp.alphanums.upper() + "-_")("gene_id")
+ + "|"
+ + pp.Word(pp.printables)("organism")
+ + "|"
+ + pp.Word(pp.printables)("location")
+ + "|"
+ + "length="
+ + integer("gene_len")
+ + pp.LineEnd()
+ + pp.Word("ACGTN")[1, ...].addParseAction("".join)("gene")
+)
+
+# read gene data from .fasta file - takes just a few seconds
+# An important aspect of this parsing process is the reassembly of all the separate lines of the
+# gene into a single scannable string. Just searching the raw .fasta file could overlook matches
+# if the match is broken up across separate lines. The parse action in the genebit parser does
+# this reassembly work.
+genedata = genebit[1, ...].parseString(fastasrc)
+
+# using the genedata extracted above, look for close matches of a gene sequence
+searchseq = pp.CloseMatch("TTAAATCTAGAAGAT", 3)
+
+for g in genedata:
+ show_header = True
+ # scan for close matches, list out found strings, and mark mismatch locations
+ for t, startLoc, endLoc in searchseq.scanString(g.gene, overlap=True):
+ if show_header:
+ # only need to show the header once
+ print("%s/%s/%s (%d)" % (g.gene_id, g.organism, g.location, g.gene_len))
+ print("-" * 24)
+ show_header = False
+
+ matched = t[0]
+ mismatches = t["mismatches"]
+ print("MATCH:", searchseq.match_string)
+ print("FOUND:", matched)
+ if mismatches:
+ print(
+ " ",
+ "".join(
+ "*" if i in mismatches else " "
+ for i, c in enumerate(searchseq.match_string)
+ ),
+ )
+ else:
+ print("<exact match>")
+ print("at location", startLoc)
+ print()
# Copyright 2004, by Alberto Santini http://www.albertosantini.it/chess/\r
#\r
from pyparsing import alphanums, nums, quotedString\r
-from pyparsing import Combine, Forward, Group, Literal, oneOf, OneOrMore, Optional, Suppress, ZeroOrMore, Word\r
+from pyparsing import (\r
+ Combine,\r
+ Forward,\r
+ Group,\r
+ Literal,\r
+ oneOf,\r
+ OneOrMore,\r
+ Optional,\r
+ Suppress,\r
+ ZeroOrMore,\r
+ Word,\r
+)\r
from pyparsing import ParseException\r
\r
#\r
castle_kingside = oneOf("O-O 0-0 o-o")\r
\r
move_number = Optional(comment) + Word(nums) + dot\r
-m1 = file_coord + rank_coord # pawn move e.g. d4\r
-m2 = file_coord + capture + file_coord + rank_coord # pawn capture move e.g. dxe5\r
-m3 = file_coord + "8" + promote + piece # pawn promotion e.g. e8=Q\r
-m4 = piece + file_coord + rank_coord # piece move e.g. Be6\r
-m5 = piece + file_coord + file_coord + rank_coord # piece move e.g. Nbd2\r
-m6 = piece + rank_coord + file_coord + rank_coord # piece move e.g. R4a7\r
-m7 = piece + capture + file_coord + rank_coord # piece capture move e.g. Bxh7\r
-m8 = castle_queenside | castle_kingside # castling e.g. o-o\r
+m1 = file_coord + rank_coord # pawn move e.g. d4\r
+m2 = file_coord + capture + file_coord + rank_coord # pawn capture move e.g. dxe5\r
+m3 = file_coord + "8" + promote + piece # pawn promotion e.g. e8=Q\r
+m4 = piece + file_coord + rank_coord # piece move e.g. Be6\r
+m5 = piece + file_coord + file_coord + rank_coord # piece move e.g. Nbd2\r
+m6 = piece + rank_coord + file_coord + rank_coord # piece move e.g. R4a7\r
+m7 = piece + capture + file_coord + rank_coord # piece capture move e.g. Bxh7\r
+m8 = castle_queenside | castle_kingside # castling e.g. o-o\r
\r
check = oneOf("+ ++")\r
mate = Literal("#")\r
decoration = check | mate | annotation | nag\r
\r
variant = Forward()\r
-half_move = Combine((m3 | m1 | m2 | m4 | m5 | m6 | m7 | m8) + Optional(decoration)) \\r
- + Optional(comment) +Optional(variant)\r
+half_move = (\r
+ Combine((m3 | m1 | m2 | m4 | m5 | m6 | m7 | m8) + Optional(decoration))\r
+ + Optional(comment)\r
+ + Optional(variant)\r
+)\r
move = Suppress(move_number) + half_move + Optional(half_move)\r
variant << "(" + OneOrMore(move) + ")"\r
# grouping the plies (half-moves) for each move: useful to group annotations, variants...\r
variant << Group("(" + OneOrMore(move) + ")")\r
game_terminator = oneOf("1-0 0-1 1/2-1/2 *")\r
\r
-pgnGrammar = Suppress(ZeroOrMore(tag)) + ZeroOrMore(move) + Optional(Suppress(game_terminator))\r
+pgnGrammar = (\r
+ Suppress(ZeroOrMore(tag)) + ZeroOrMore(move) + Optional(Suppress(game_terminator))\r
+)\r
+\r
+\r
+def parsePGN(pgn, bnf=pgnGrammar, fn=None):\r
+ try:\r
+ return bnf.parseString(pgn)\r
+ except ParseException as err:\r
+ print(err.line)\r
+ print(" " * (err.column - 1) + "^")\r
+ print(err)\r
\r
-def parsePGN( pgn, bnf=pgnGrammar, fn=None ):\r
- try:\r
- return bnf.parseString( pgn )\r
- except ParseException as err:\r
- print(err.line)\r
- print(" "*(err.column-1) + "^")\r
- print(err)\r
\r
if __name__ == "__main__":\r
- # input string\r
- pgn = """\r
+ # input string\r
+ pgn = """\r
[Event "ICC 5 0 u"]\r
[Site "Internet Chess Club"]\r
[Date "2004.01.25"]\r
g6 fxg6 15. Bg5 Rf8 16. a3 Bd5 17. Re1+ Nde5 18. Nxe5 Nxe5 19. Bf4 Rf5 20.\r
Bxe5 Rxe5 21. Rg5 Rxe1# {Black wins} 0-1\r
"""\r
- # parse input string\r
- tokens = parsePGN(pgn, pgnGrammar)\r
- print(tokens.dump())\r
+ # parse input string\r
+ tokens = parsePGN(pgn, pgnGrammar)\r
+ print(tokens.dump())\r
\r
# find all words beginning with a vowel\r
vowels = "aeiouAEIOU"\r
-initialVowelWord = Word(vowels,alphas)\r
+initialVowelWord = Word(vowels, alphas)\r
\r
# Unfortunately, searchString will advance character by character through\r
# the input text, so it will detect that the initial "Lorem" is not an\r
# consonants, but we will just throw them away when we match them. The key is\r
# that, in having been matched, the parser will skip over them entirely when\r
# looking for initialVowelWords.\r
-consonants = ''.join(c for c in alphas if c not in vowels)\r
+consonants = "".join(c for c in alphas if c not in vowels)\r
initialConsWord = Word(consonants, alphas).suppress()\r
\r
# using scanString to locate where tokens are matched\r
-for t,start,end in (initialConsWord|initialVowelWord).scanString(text):\r
+for t, start, end in (initialConsWord | initialVowelWord).scanString(text):\r
if t:\r
- print(start,':', t[0])\r
+ print(start, ":", t[0])\r
\r
# add parse action to annotate the parsed tokens with their location in the\r
# input string\r
-def addLocnToTokens(s,l,t):\r
- t['locn'] = l\r
- t['word'] = t[0]\r
+def addLocnToTokens(s, l, t):\r
+ t["locn"] = l\r
+ t["word"] = t[0]\r
+\r
+\r
initialVowelWord.setParseAction(addLocnToTokens)\r
\r
for ivowelInfo in (initialConsWord | initialVowelWord).searchString(text):\r
if not ivowelInfo:\r
continue\r
- print(ivowelInfo.locn, ':', ivowelInfo.word)\r
+ print(ivowelInfo.locn, ":", ivowelInfo.word)\r
\r
\r
# alternative - add an Empty that will save the current location\r
def location(name):\r
- return Empty().setParseAction(lambda s,l,t: t.__setitem__(name,l))\r
+ return Empty().setParseAction(lambda s, l, t: t.__setitem__(name, l))\r
+\r
+\r
locateInitialVowels = location("locn") + initialVowelWord("word")\r
\r
# search through the input text\r
for ivowelInfo in (initialConsWord | locateInitialVowels).searchString(text):\r
if not ivowelInfo:\r
continue\r
- print(ivowelInfo.locn, ':', ivowelInfo.word)\r
+ print(ivowelInfo.locn, ":", ivowelInfo.word)\r
# Copyright 2010, Paul McGuire
#
-from pyparsing import (Word, alphas, alphanums, Regex, Suppress, Forward,
- Keyword, Group, oneOf, ZeroOrMore, Optional, delimitedList,
- restOfLine, quotedString, Dict)
-
-ident = Word(alphas+"_",alphanums+"_").setName("identifier")
+from pyparsing import (
+ Word,
+ alphas,
+ alphanums,
+ Regex,
+ Suppress,
+ Forward,
+ Group,
+ oneOf,
+ ZeroOrMore,
+ Optional,
+ delimitedList,
+ restOfLine,
+ quotedString,
+ Dict,
+ Keyword,
+)
+
+ident = Word(alphas + "_", alphanums + "_").setName("identifier")
integer = Regex(r"[+-]?\d+")
-LBRACE,RBRACE,LBRACK,RBRACK,LPAR,RPAR,EQ,SEMI = map(Suppress,"{}[]()=;")
+LBRACE, RBRACE, LBRACK, RBRACK, LPAR, RPAR, EQ, SEMI = map(Suppress, "{}[]()=;")
kwds = """message required optional repeated enum extensions extends extend
- to package service rpc returns true false option import"""
+ to package service rpc returns true false option import syntax"""
for kw in kwds.split():
- exec("{0}_ = Keyword('{1}')".format(kw.upper(), kw))
+ exec("{}_ = Keyword('{}')".format(kw.upper(), kw))
messageBody = Forward()
messageDefn = MESSAGE_ - ident("messageId") + LBRACE + messageBody("body") + RBRACE
-typespec = oneOf("""double float int32 int64 uint32 uint64 sint32 sint64
- fixed32 fixed64 sfixed32 sfixed64 bool string bytes""") | ident
+typespec = (
+ oneOf(
+ """double float int32 int64 uint32 uint64 sint32 sint64
+ fixed32 fixed64 sfixed32 sfixed64 bool string bytes"""
+ )
+ | ident
+)
rvalue = integer | TRUE_ | FALSE_ | ident
fieldDirective = LBRACK + Group(ident + EQ + rvalue) + RBRACK
-fieldDefn = (( REQUIRED_ | OPTIONAL_ | REPEATED_ )("fieldQualifier") -
- typespec("typespec") + ident("ident") + EQ + integer("fieldint") + ZeroOrMore(fieldDirective) + SEMI)
+fieldDefnPrefix = REQUIRED_ | OPTIONAL_ | REPEATED_
+fieldDefn = (
+ Optional(fieldDefnPrefix)
+ + typespec("typespec")
+ + ident("ident")
+ + EQ
+ + integer("fieldint")
+ + ZeroOrMore(fieldDirective)
+ + SEMI
+)
# enumDefn ::= 'enum' ident '{' { ident '=' integer ';' }* '}'
-enumDefn = ENUM_("typespec") - ident('name') + LBRACE + Dict( ZeroOrMore( Group(ident + EQ + integer + SEMI) ))('values') + RBRACE
+enumDefn = (
+ ENUM_("typespec")
+ - ident("name")
+ + LBRACE
+ + Dict(ZeroOrMore(Group(ident + EQ + integer + SEMI)))("values")
+ + RBRACE
+)
# extensionsDefn ::= 'extensions' integer 'to' integer ';'
extensionsDefn = EXTENSIONS_ - integer + TO_ + integer + SEMI
messageExtension = EXTEND_ - ident + LBRACE + messageBody + RBRACE
# messageBody ::= { fieldDefn | enumDefn | messageDefn | extensionsDefn | messageExtension }*
-messageBody << Group(ZeroOrMore( Group(fieldDefn | enumDefn | messageDefn | extensionsDefn | messageExtension) ))
+messageBody << Group(
+ ZeroOrMore(
+ Group(fieldDefn | enumDefn | messageDefn | extensionsDefn | messageExtension)
+ )
+)
# methodDefn ::= 'rpc' ident '(' [ ident ] ')' 'returns' '(' [ ident ] ')' ';'
-methodDefn = (RPC_ - ident("methodName") +
- LPAR + Optional(ident("methodParam")) + RPAR +
- RETURNS_ + LPAR + Optional(ident("methodReturn")) + RPAR)
+methodDefn = (
+ RPC_
+ - ident("methodName")
+ + LPAR
+ + Optional(ident("methodParam"))
+ + RPAR
+ + RETURNS_
+ + LPAR
+ + Optional(ident("methodReturn"))
+ + RPAR
+)
# serviceDefn ::= 'service' ident '{' methodDefn* '}'
-serviceDefn = SERVICE_ - ident("serviceName") + LBRACE + ZeroOrMore(Group(methodDefn)) + RBRACE
+serviceDefn = (
+ SERVICE_ - ident("serviceName") + LBRACE + ZeroOrMore(Group(methodDefn)) + RBRACE
+)
+
+syntaxDefn = SYNTAX_ + EQ - quotedString("syntaxString") + SEMI
# packageDirective ::= 'package' ident [ '.' ident]* ';'
-packageDirective = Group(PACKAGE_ - delimitedList(ident, '.', combine=True) + SEMI)
+packageDirective = Group(PACKAGE_ - delimitedList(ident, ".", combine=True) + SEMI)
-comment = '//' + restOfLine
+comment = "//" + restOfLine
importDirective = IMPORT_ - quotedString("importFileSpec") + SEMI
-optionDirective = OPTION_ - ident("optionName") + EQ + quotedString("optionValue") + SEMI
+optionDirective = (
+ OPTION_ - ident("optionName") + EQ + quotedString("optionValue") + SEMI
+)
-topLevelStatement = Group(messageDefn | messageExtension | enumDefn | serviceDefn | importDirective | optionDirective)
+topLevelStatement = Group(
+ messageDefn
+ | messageExtension
+ | enumDefn
+ | serviceDefn
+ | importDirective
+ | optionDirective
+ | syntaxDefn
+)
parser = Optional(packageDirective) + ZeroOrMore(topLevelStatement)
parser.ignore(comment)
-
-test1 = """message Person {
- required int32 id = 1;
- required string name = 2;
- optional string email = 3;
-}"""
-
-test2 = """package tutorial;
-
-message Person {
- required string name = 1;
- required int32 id = 2;
- optional string email = 3;
-
- enum PhoneType {
- MOBILE = 0;
- HOME = 1;
- WORK = 2;
- }
-
- message PhoneNumber {
- required string number = 1;
- optional PhoneType type = 2 [default = HOME];
- }
-
- repeated PhoneNumber phone = 4;
-}
-
-message AddressBook {
- repeated Person person = 1;
-}"""
-
-parser.runTests([test1, test2])
+if __name__ == "__main__":
+
+ test1 = """message Person {
+ required int32 id = 1;
+ required string name = 2;
+ optional string email = 3;
+ }"""
+
+ test2 = """package tutorial;
+
+ message Person {
+ required string name = 1;
+ required int32 id = 2;
+ optional string email = 3;
+
+ enum PhoneType {
+ MOBILE = 0;
+ HOME = 1;
+ WORK = 2;
+ }
+
+ message PhoneNumber {
+ required string number = 1;
+ optional PhoneType type = 2 [default = HOME];
+ }
+
+ repeated PhoneNumber phone = 4;
+ }
+
+ message AddressBook {
+ repeated Person person = 1;
+ }"""
+
+ test3 = """syntax = "proto3";
+
+ import "test.proto";
+
+ message SearchRequest {
+ string query = 1;
+ int32 page_number = 2;
+ int32 result_per_page = 3;
+ }
+ """
+
+ parser.runTests([test1, test2, test3])
from pyparsing import *\r
from sys import stdin, argv, exit\r
\r
-#defines debug level\r
+# defines debug level\r
# 0 - no debug\r
# 1 - print parsing results\r
# 2 - print parsing results and symbol table\r
# %14 is the stack frame pointer (x86's ebp) and %15 is the stack pointer (x86's esp). All data-handling instructions can be\r
# unsigned (suffix U), or signed (suffix S). These are ADD, SUB, MUL and DIV. These are three-address instructions,\r
# the first two operands are input, the third one is output. Whether these operands are registers, memory or constant\r
-# is not relevant, all combinations are possible (except that output cannot be a constant). Constants are writen with a $ prefix (10-base only).\r
+# is not relevant, all combinations are possible (except that output cannot be a constant). Constants are written with a $ prefix (10-base only).\r
# Conditional jumps are handled by JXXY instructions, where XX is LT, GT, LE, GE, EQ, NE (less than, greater than, less than or equal, etc.)\r
# and Y is U or S (unsigned or signed, except for JEQ i JNE). Unconditional jump is JMP. The move instruction is MOV.\r
# Function handling is done using CALL, RET, PUSH and POP (C style function calls). Static data is defined using the WORD directive\r
##########################################################################################\r
##########################################################################################\r
\r
+\r
class Enumerate(dict):\r
"""C enum emulation (original by Scott David Daniels)"""\r
+\r
def __init__(self, names):\r
for number, name in enumerate(names.split()):\r
setattr(self, name, number)\r
self[number] = name\r
\r
-class SharedData(object):\r
+\r
+class SharedData:\r
"""Data used in all three main classes"""\r
\r
- #Possible kinds of symbol table entries\r
- KINDS = Enumerate("NO_KIND WORKING_REGISTER GLOBAL_VAR FUNCTION PARAMETER LOCAL_VAR CONSTANT")\r
- #Supported types of functions and variables\r
+ # Possible kinds of symbol table entries\r
+ KINDS = Enumerate(\r
+ "NO_KIND WORKING_REGISTER GLOBAL_VAR FUNCTION PARAMETER LOCAL_VAR CONSTANT"\r
+ )\r
+ # Supported types of functions and variables\r
TYPES = Enumerate("NO_TYPE INT UNSIGNED")\r
\r
- #bit size of variables\r
+ # bit size of variables\r
TYPE_BIT_SIZE = 16\r
- #min/max values of constants\r
- MIN_INT = -2 ** (TYPE_BIT_SIZE - 1)\r
+ # min/max values of constants\r
+ MIN_INT = -(2 ** (TYPE_BIT_SIZE - 1))\r
MAX_INT = 2 ** (TYPE_BIT_SIZE - 1) - 1\r
MAX_UNSIGNED = 2 ** TYPE_BIT_SIZE - 1\r
- #available working registers (the last one is the register for function's return value!)\r
+ # available working registers (the last one is the register for function's return value!)\r
REGISTERS = "%0 %1 %2 %3 %4 %5 %6 %7 %8 %9 %10 %11 %12 %13".split()\r
- #register for function's return value\r
+ # register for function's return value\r
FUNCTION_REGISTER = len(REGISTERS) - 1\r
- #the index of last working register\r
+ # the index of last working register\r
LAST_WORKING_REGISTER = len(REGISTERS) - 2\r
- #list of relational operators\r
+ # list of relational operators\r
RELATIONAL_OPERATORS = "< > <= >= == !=".split()\r
\r
def __init__(self):\r
- #index of the currently parsed function\r
+ # index of the currently parsed function\r
self.functon_index = 0\r
- #name of the currently parsed function\r
+ # name of the currently parsed function\r
self.functon_name = 0\r
- #number of parameters of the currently parsed function\r
+ # number of parameters of the currently parsed function\r
self.function_params = 0\r
- #number of local variables of the currently parsed function\r
+ # number of local variables of the currently parsed function\r
self.function_vars = 0\r
\r
+\r
##########################################################################################\r
##########################################################################################\r
\r
-class ExceptionSharedData(object):\r
+\r
+class ExceptionSharedData:\r
"""Class for exception handling data"""\r
\r
def __init__(self):\r
- #position in currently parsed text\r
+ # position in currently parsed text\r
self.location = 0\r
- #currently parsed text\r
+ # currently parsed text\r
self.text = ""\r
\r
def setpos(self, location, text):\r
- """Helper function for setting curently parsed text and position"""\r
+ """Helper function for setting currently parsed text and position"""\r
self.location = location\r
self.text = text\r
\r
+\r
exshared = ExceptionSharedData()\r
\r
+\r
class SemanticException(Exception):\r
- """Exception for semantic errors found during parsing, similar to ParseException.\r
- Introduced because ParseException is used internally in pyparsing and custom\r
- messages got lost and replaced by pyparsing's generic errors.\r
+ """\r
+ Exception for semantic errors found during parsing, similar to ParseException.\r
+ Introduced because ParseException is used internally in pyparsing and custom\r
+ messages got lost and replaced by pyparsing's generic errors.\r
"""\r
\r
def __init__(self, message, print_location=True):\r
- super(SemanticException,self).__init__()\r
+ super().__init__()\r
self._message = message\r
self.location = exshared.location\r
self.print_location = print_location\r
\r
def _get_message(self):\r
return self._message\r
+\r
def _set_message(self, message):\r
self._message = message\r
+\r
message = property(_get_message, _set_message)\r
\r
def __str__(self):\r
msg += "\n%s" % self.text\r
return msg\r
\r
+\r
##########################################################################################\r
##########################################################################################\r
\r
-class SymbolTableEntry(object):\r
+\r
+class SymbolTableEntry:\r
"""Class which represents one symbol table entry."""\r
\r
- def __init__(self, sname = "", skind = 0, stype = 0, sattr = None, sattr_name = "None"):\r
- """Initialization of symbol table entry.\r
- sname - symbol name\r
- skind - symbol kind\r
- stype - symbol type\r
- sattr - symbol attribute\r
- sattr_name - symbol attribute name (used only for table display)\r
+ def __init__(self, sname="", skind=0, stype=0, sattr=None, sattr_name="None"):\r
+ """\r
+ Initialization of symbol table entry.\r
+ sname - symbol name\r
+ skind - symbol kind\r
+ stype - symbol type\r
+ sattr - symbol attribute\r
+ sattr_name - symbol attribute name (used only for table display)\r
"""\r
self.name = sname\r
self.kind = skind\r
\r
def attribute_str(self):\r
"""Returns attribute string (used only for table display)"""\r
- return "{0}={1}".format(self.attribute_name, self.attribute) if self.attribute != None else "None"\r
+ return (\r
+ "{}={}".format(self.attribute_name, self.attribute)\r
+ if self.attribute != None\r
+ else "None"\r
+ )\r
+\r
\r
-class SymbolTable(object):\r
+class SymbolTable:\r
"""Class for symbol table of microC program"""\r
\r
def __init__(self, shared):\r
"""Initialization of the symbol table"""\r
self.table = []\r
self.lable_len = 0\r
- #put working registers in the symbol table\r
- for reg in range(SharedData.FUNCTION_REGISTER+1):\r
- self.insert_symbol(SharedData.REGISTERS[reg], SharedData.KINDS.WORKING_REGISTER, SharedData.TYPES.NO_TYPE)\r
- #shared data\r
+ # put working registers in the symbol table\r
+ for reg in range(SharedData.FUNCTION_REGISTER + 1):\r
+ self.insert_symbol(\r
+ SharedData.REGISTERS[reg],\r
+ SharedData.KINDS.WORKING_REGISTER,\r
+ SharedData.TYPES.NO_TYPE,\r
+ )\r
+ # shared data\r
self.shared = shared\r
\r
def error(self, text=""):\r
- """Symbol table error exception. It should happen only if index is out of range while accessing symbol table.\r
- This exeption is not handled by the compiler, so as to allow traceback printing\r
+ """\r
+ Symbol table error exception. It should happen only if index is out of range while accessing symbol table.\r
+ This exception is not handled by the compiler, so as to allow traceback printing\r
"""\r
if text == "":\r
raise Exception("Symbol table index out of range")\r
\r
def display(self):\r
"""Displays the symbol table content"""\r
- #Finding the maximum length for each column\r
+ # Finding the maximum length for each column\r
sym_name = "Symbol name"\r
- sym_len = max(max(len(i.name) for i in self.table),len(sym_name))\r
+ sym_len = max(max(len(i.name) for i in self.table), len(sym_name))\r
kind_name = "Kind"\r
- kind_len = max(max(len(SharedData.KINDS[i.kind]) for i in self.table),len(kind_name))\r
+ kind_len = max(\r
+ max(len(SharedData.KINDS[i.kind]) for i in self.table), len(kind_name)\r
+ )\r
type_name = "Type"\r
- type_len = max(max(len(SharedData.TYPES[i.type]) for i in self.table),len(type_name))\r
+ type_len = max(\r
+ max(len(SharedData.TYPES[i.type]) for i in self.table), len(type_name)\r
+ )\r
attr_name = "Attribute"\r
- attr_len = max(max(len(i.attribute_str()) for i in self.table),len(attr_name))\r
- #print table header\r
- print("{0:3s} | {1:^{2}s} | {3:^{4}s} | {5:^{6}s} | {7:^{8}} | {9:s}".format(" No", sym_name, sym_len, kind_name, kind_len, type_name, type_len, attr_name, attr_len, "Parameters"))\r
- print("-----------------------------" + "-" * (sym_len + kind_len + type_len + attr_len))\r
- #print symbol table\r
- for i,sym in enumerate(self.table):\r
+ attr_len = max(max(len(i.attribute_str()) for i in self.table), len(attr_name))\r
+ # print table header\r
+ print(\r
+ "{0:3s} | {1:^{2}s} | {3:^{4}s} | {5:^{6}s} | {7:^{8}} | {9:s}".format(\r
+ " No",\r
+ sym_name,\r
+ sym_len,\r
+ kind_name,\r
+ kind_len,\r
+ type_name,\r
+ type_len,\r
+ attr_name,\r
+ attr_len,\r
+ "Parameters",\r
+ )\r
+ )\r
+ print(\r
+ "-----------------------------"\r
+ + "-" * (sym_len + kind_len + type_len + attr_len)\r
+ )\r
+ # print symbol table\r
+ for i, sym in enumerate(self.table):\r
parameters = ""\r
for p in sym.param_types:\r
if parameters == "":\r
- parameters = "{0}".format(SharedData.TYPES[p])\r
+ parameters = "{}".format(SharedData.TYPES[p])\r
else:\r
- parameters += ", {0}".format(SharedData.TYPES[p])\r
- print("{0:3d} | {1:^{2}s} | {3:^{4}s} | {5:^{6}s} | {7:^{8}} | ({9})".format(i, sym.name, sym_len, SharedData.KINDS[sym.kind], kind_len, SharedData.TYPES[sym.type], type_len, sym.attribute_str(), attr_len, parameters))\r
+ parameters += ", {}".format(SharedData.TYPES[p])\r
+ print(\r
+ "{0:3d} | {1:^{2}s} | {3:^{4}s} | {5:^{6}s} | {7:^{8}} | ({9})".format(\r
+ i,\r
+ sym.name,\r
+ sym_len,\r
+ SharedData.KINDS[sym.kind],\r
+ kind_len,\r
+ SharedData.TYPES[sym.type],\r
+ type_len,\r
+ sym.attribute_str(),\r
+ attr_len,\r
+ parameters,\r
+ )\r
+ )\r
\r
def insert_symbol(self, sname, skind, stype):\r
- """Inserts new symbol at the end of the symbol table.\r
- Returns symbol index\r
- sname - symbol name\r
- skind - symbol kind\r
- stype - symbol type\r
+ """\r
+ Inserts new symbol at the end of the symbol table.\r
+ Returns symbol index\r
+ sname - symbol name\r
+ skind - symbol kind\r
+ stype - symbol type\r
"""\r
self.table.append(SymbolTableEntry(sname, skind, stype))\r
self.table_len = len(self.table)\r
- return self.table_len-1\r
+ return self.table_len - 1\r
\r
def clear_symbols(self, index):\r
- """Clears all symbols begining with the index to the end of table"""\r
+ """Clears all symbols beginning with the index to the end of table"""\r
try:\r
del self.table[index:]\r
except Exception:\r
self.error()\r
self.table_len = len(self.table)\r
\r
- def lookup_symbol(self, sname, skind=list(SharedData.KINDS.keys()), stype=list(SharedData.TYPES.keys())):\r
- """Searches for symbol, from the end to the begining.\r
- Returns symbol index or None\r
- sname - symbol name\r
- skind - symbol kind (one kind, list of kinds, or None) deafult: any kind\r
- stype - symbol type (or None) default: any type\r
+ def lookup_symbol(\r
+ self,\r
+ sname,\r
+ skind=list(SharedData.KINDS.keys()),\r
+ stype=list(SharedData.TYPES.keys()),\r
+ ):\r
+ """\r
+ Searches for symbol, from the end to the beginning.\r
+ Returns symbol index or None\r
+ sname - symbol name\r
+ skind - symbol kind (one kind, list of kinds, or None) default: any kind\r
+ stype - symbol type (or None) default: any type\r
"""\r
skind = skind if isinstance(skind, list) else [skind]\r
stype = stype if isinstance(stype, list) else [stype]\r
- for i, sym in [[x, self.table[x]] for x in range(len(self.table) - 1, SharedData.LAST_WORKING_REGISTER, -1)]:\r
+ for i, sym in [\r
+ [x, self.table[x]]\r
+ for x in range(len(self.table) - 1, SharedData.LAST_WORKING_REGISTER, -1)\r
+ ]:\r
if (sym.name == sname) and (sym.kind in skind) and (sym.type in stype):\r
return i\r
return None\r
\r
def insert_id(self, sname, skind, skinds, stype):\r
- """Inserts a new identifier at the end of the symbol table, if possible.\r
- Returns symbol index, or raises an exception if the symbol alredy exists\r
- sname - symbol name\r
- skind - symbol kind\r
- skinds - symbol kinds to check for\r
- stype - symbol type\r
+ """\r
+ Inserts a new identifier at the end of the symbol table, if possible.\r
+ Returns symbol index, or raises an exception if the symbol already exists\r
+ sname - symbol name\r
+ skind - symbol kind\r
+ skinds - symbol kinds to check for\r
+ stype - symbol type\r
"""\r
index = self.lookup_symbol(sname, skinds)\r
if index == None:\r
\r
def insert_global_var(self, vname, vtype):\r
"Inserts a new global variable"\r
- return self.insert_id(vname, SharedData.KINDS.GLOBAL_VAR, [SharedData.KINDS.GLOBAL_VAR, SharedData.KINDS.FUNCTION], vtype)\r
+ return self.insert_id(\r
+ vname,\r
+ SharedData.KINDS.GLOBAL_VAR,\r
+ [SharedData.KINDS.GLOBAL_VAR, SharedData.KINDS.FUNCTION],\r
+ vtype,\r
+ )\r
\r
def insert_local_var(self, vname, vtype, position):\r
"Inserts a new local variable"\r
- index = self.insert_id(vname, SharedData.KINDS.LOCAL_VAR, [SharedData.KINDS.LOCAL_VAR, SharedData.KINDS.PARAMETER], vtype)\r
+ index = self.insert_id(\r
+ vname,\r
+ SharedData.KINDS.LOCAL_VAR,\r
+ [SharedData.KINDS.LOCAL_VAR, SharedData.KINDS.PARAMETER],\r
+ vtype,\r
+ )\r
self.table[index].attribute = position\r
\r
def insert_parameter(self, pname, ptype):\r
"Inserts a new parameter"\r
- index = self.insert_id(pname, SharedData.KINDS.PARAMETER, SharedData.KINDS.PARAMETER, ptype)\r
- #set parameter's attribute to it's ordinal number\r
+ index = self.insert_id(\r
+ pname, SharedData.KINDS.PARAMETER, SharedData.KINDS.PARAMETER, ptype\r
+ )\r
+ # set parameter's attribute to it's ordinal number\r
self.table[index].set_attribute("Index", self.shared.function_params)\r
- #set parameter's type in param_types list of a function\r
+ # set parameter's type in param_types list of a function\r
self.table[self.shared.function_index].param_types.append(ptype)\r
return index\r
\r
def insert_function(self, fname, ftype):\r
"Inserts a new function"\r
- index = self.insert_id(fname, SharedData.KINDS.FUNCTION, [SharedData.KINDS.GLOBAL_VAR, SharedData.KINDS.FUNCTION], ftype)\r
- self.table[index].set_attribute("Params",0)\r
+ index = self.insert_id(\r
+ fname,\r
+ SharedData.KINDS.FUNCTION,\r
+ [SharedData.KINDS.GLOBAL_VAR, SharedData.KINDS.FUNCTION],\r
+ ftype,\r
+ )\r
+ self.table[index].set_attribute("Params", 0)\r
return index\r
\r
def insert_constant(self, cname, ctype):\r
- """Inserts a constant (or returns index if the constant already exists)\r
- Additionally, checks for range.\r
+ """\r
+ Inserts a constant (or returns index if the constant already exists)\r
+ Additionally, checks for range.\r
"""\r
index = self.lookup_symbol(cname, stype=ctype)\r
if index == None:\r
num = int(cname)\r
if ctype == SharedData.TYPES.INT:\r
if (num < SharedData.MIN_INT) or (num > SharedData.MAX_INT):\r
- raise SemanticException("Integer constant '%s' out of range" % cname)\r
+ raise SemanticException(\r
+ "Integer constant '%s' out of range" % cname\r
+ )\r
elif ctype == SharedData.TYPES.UNSIGNED:\r
if (num < 0) or (num > SharedData.MAX_UNSIGNED):\r
- raise SemanticException("Unsigned constant '%s' out of range" % cname)\r
+ raise SemanticException(\r
+ "Unsigned constant '%s' out of range" % cname\r
+ )\r
index = self.insert_symbol(cname, SharedData.KINDS.CONSTANT, ctype)\r
return index\r
\r
def same_types(self, index1, index2):\r
"""Returns True if both symbol table elements are of the same type"""\r
try:\r
- same = self.table[index1].type == self.table[index2].type != SharedData.TYPES.NO_TYPE\r
+ same = (\r
+ self.table[index1].type\r
+ == self.table[index2].type\r
+ != SharedData.TYPES.NO_TYPE\r
+ )\r
except Exception:\r
self.error()\r
return same\r
\r
def same_type_as_argument(self, index, function_index, argument_number):\r
- """Returns True if index and function's argument are of the same type\r
- index - index in symbol table\r
- function_index - function's index in symbol table\r
- argument_number - # of function's argument\r
+ """\r
+ Returns True if index and function's argument are of the same type\r
+ index - index in symbol table\r
+ function_index - function's index in symbol table\r
+ argument_number - # of function's argument\r
"""\r
try:\r
- same = self.table[function_index].param_types[argument_number] == self.table[index].type\r
+ same = (\r
+ self.table[function_index].param_types[argument_number]\r
+ == self.table[index].type\r
+ )\r
except Exception:\r
self.error()\r
return same\r
except Exception:\r
self.error()\r
\r
+\r
##########################################################################################\r
##########################################################################################\r
\r
-class CodeGenerator(object):\r
+\r
+class CodeGenerator:\r
"""Class for code generation methods."""\r
\r
- #dictionary of relational operators\r
- RELATIONAL_DICT = {op:i for i, op in enumerate(SharedData.RELATIONAL_OPERATORS)}\r
- #conditional jumps for relational operators\r
- CONDITIONAL_JUMPS = ["JLTS", "JGTS", "JLES", "JGES", "JEQ ", "JNE ",\r
- "JLTU", "JGTU", "JLEU", "JGEU", "JEQ ", "JNE "]\r
- #opposite conditional jumps for relational operators\r
- OPPOSITE_JUMPS = ["JGES", "JLES", "JGTS", "JLTS", "JNE ", "JEQ ",\r
- "JGEU", "JLEU", "JGTU", "JLTU", "JNE ", "JEQ "]\r
- #supported operations\r
- OPERATIONS = {"+" : "ADD", "-" : "SUB", "*" : "MUL", "/" : "DIV"}\r
- #suffixes for signed and unsigned operations (if no type is specified, unsigned will be assumed)\r
- OPSIGNS = {SharedData.TYPES.NO_TYPE : "U", SharedData.TYPES.INT : "S", SharedData.TYPES.UNSIGNED : "U"}\r
- #text at start of data segment\r
+ # dictionary of relational operators\r
+ RELATIONAL_DICT = {op: i for i, op in enumerate(SharedData.RELATIONAL_OPERATORS)}\r
+ # conditional jumps for relational operators\r
+ CONDITIONAL_JUMPS = [\r
+ "JLTS",\r
+ "JGTS",\r
+ "JLES",\r
+ "JGES",\r
+ "JEQ ",\r
+ "JNE ",\r
+ "JLTU",\r
+ "JGTU",\r
+ "JLEU",\r
+ "JGEU",\r
+ "JEQ ",\r
+ "JNE ",\r
+ ]\r
+ # opposite conditional jumps for relational operators\r
+ OPPOSITE_JUMPS = [\r
+ "JGES",\r
+ "JLES",\r
+ "JGTS",\r
+ "JLTS",\r
+ "JNE ",\r
+ "JEQ ",\r
+ "JGEU",\r
+ "JLEU",\r
+ "JGTU",\r
+ "JLTU",\r
+ "JNE ",\r
+ "JEQ ",\r
+ ]\r
+ # supported operations\r
+ OPERATIONS = {"+": "ADD", "-": "SUB", "*": "MUL", "/": "DIV"}\r
+ # suffixes for signed and unsigned operations (if no type is specified, unsigned will be assumed)\r
+ OPSIGNS = {\r
+ SharedData.TYPES.NO_TYPE: "U",\r
+ SharedData.TYPES.INT: "S",\r
+ SharedData.TYPES.UNSIGNED: "U",\r
+ }\r
+ # text at start of data segment\r
DATA_START_TEXT = "#DATA"\r
- #text at start of code segment\r
+ # text at start of code segment\r
CODE_START_TEXT = "#CODE"\r
\r
def __init__(self, shared, symtab):\r
- #generated code\r
+ # generated code\r
self.code = ""\r
- #prefix for internal labels\r
+ # prefix for internal labels\r
self.internal = "@"\r
- #suffix for label definition\r
+ # suffix for label definition\r
self.definition = ":"\r
- #list of free working registers\r
+ # list of free working registers\r
self.free_registers = list(range(SharedData.FUNCTION_REGISTER, -1, -1))\r
- #list of used working registers\r
+ # list of used working registers\r
self.used_registers = []\r
- #list of used registers needed when function call is inside of a function call\r
+ # list of used registers needed when function call is inside of a function call\r
self.used_registers_stack = []\r
- #shared data\r
+ # shared data\r
self.shared = shared\r
- #symbol table\r
+ # symbol table\r
self.symtab = symtab\r
\r
def error(self, text):\r
- """Compiler error exception. It should happen only if something is wrong with compiler.\r
- This exeption is not handled by the compiler, so as to allow traceback printing\r
+ """\r
+ Compiler error exception. It should happen only if something is wrong with compiler.\r
+ This exception is not handled by the compiler, so as to allow traceback printing\r
"""\r
raise Exception("Compiler error: %s" % text)\r
\r
- def take_register(self, rtype = SharedData.TYPES.NO_TYPE):\r
+ def take_register(self, rtype=SharedData.TYPES.NO_TYPE):\r
"""Reserves one working register and sets its type"""\r
if len(self.free_registers) == 0:\r
self.error("no more free registers")\r
self.symtab.set_type(reg, rtype)\r
return reg\r
\r
- def take_function_register(self, rtype = SharedData.TYPES.NO_TYPE):\r
+ def take_function_register(self, rtype=SharedData.TYPES.NO_TYPE):\r
"""Reserves register for function return value and sets its type"""\r
reg = SharedData.FUNCTION_REGISTER\r
if reg not in self.free_registers:\r
self.error("register %s is not taken" % self.REGISTERS[reg])\r
self.used_registers.remove(reg)\r
self.free_registers.append(reg)\r
- self.free_registers.sort(reverse = True)\r
+ self.free_registers.sort(reverse=True)\r
\r
def free_if_register(self, index):\r
"""If index is a working register, free it, otherwise just return (helper function)"""\r
self.free_register(index)\r
\r
def label(self, name, internal=False, definition=False):\r
- """Generates label name (helper function)\r
- name - label name\r
- internal - boolean value, adds "@" prefix to label\r
- definition - boolean value, adds ":" suffix to label\r
"""\r
- return "{0}{1}{2}".format(self.internal if internal else "", name, self.definition if definition else "")\r
+ Generates label name (helper function)\r
+ name - label name\r
+ internal - boolean value, adds "@" prefix to label\r
+ definition - boolean value, adds ":" suffix to label\r
+ """\r
+ return "{}{}{}".format(\r
+ self.internal if internal else "",\r
+ name,\r
+ self.definition if definition else "",\r
+ )\r
\r
def symbol(self, index):\r
"""Generates symbol name from index"""\r
- #if index is actually a string, just return it\r
+ # if index is actually a string, just return it\r
if isinstance(index, str):\r
return index\r
elif (index < 0) or (index >= self.symtab.table_len):\r
self.error("symbol table index out of range")\r
sym = self.symtab.table[index]\r
- #local variables are located at negative offset from frame pointer register\r
+ # local variables are located at negative offset from frame pointer register\r
if sym.kind == SharedData.KINDS.LOCAL_VAR:\r
- return "-{0}(1:%14)".format(sym.attribute * 4 + 4)\r
- #parameters are located at positive offset from frame pointer register\r
+ return "-{}(1:%14)".format(sym.attribute * 4 + 4)\r
+ # parameters are located at positive offset from frame pointer register\r
elif sym.kind == SharedData.KINDS.PARAMETER:\r
- return "{0}(1:%14)".format(8 + sym.attribute * 4)\r
+ return "{}(1:%14)".format(8 + sym.attribute * 4)\r
elif sym.kind == SharedData.KINDS.CONSTANT:\r
- return "${0}".format(sym.name)\r
+ return "${}".format(sym.name)\r
else:\r
- return "{0}".format(sym.name)\r
+ return "{}".format(sym.name)\r
\r
def save_used_registers(self):\r
"""Pushes all used working registers before function call"""\r
for reg in used:\r
self.newline_text("PUSH\t%s" % SharedData.REGISTERS[reg], True)\r
self.free_registers.extend(used)\r
- self.free_registers.sort(reverse = True)\r
+ self.free_registers.sort(reverse=True)\r
\r
def restore_used_registers(self):\r
"""Pops all used working registers after function call"""\r
used = self.used_registers_stack.pop()\r
self.used_registers = used[:]\r
- used.sort(reverse = True)\r
+ used.sort(reverse=True)\r
for reg in used:\r
self.newline_text("POP \t%s" % SharedData.REGISTERS[reg], True)\r
self.free_registers.remove(reg)\r
if indent:\r
self.text("\t\t\t")\r
\r
- def newline_text(self, text, indent = False):\r
- """Inserts a newline and text, optionally with indentation (helper function)"""\r
+ def newline_text(self, text, indent=False):\r
+ """\r
+ Inserts a newline and text, optionally with indentation (helper function)\r
+ """\r
self.newline(indent)\r
self.text(text)\r
\r
def newline_label(self, name, internal=False, definition=False):\r
- """Inserts a newline and a label (helper function)\r
- name - label name\r
- internal - boolean value, adds "@" prefix to label\r
- definition - boolean value, adds ":" suffix to label\r
"""\r
- self.newline_text(self.label("{0}{1}{2}".format("@" if internal else "", name, ":" if definition else "")))\r
+ Inserts a newline and a label (helper function)\r
+ name - label name\r
+ internal - boolean value, adds "@" prefix to label\r
+ definition - boolean value, adds ":" suffix to label\r
+ """\r
+ self.newline_text(\r
+ self.label(\r
+ "{}{}{}".format(\r
+ "@" if internal else "", name, ":" if definition else ""\r
+ )\r
+ )\r
+ )\r
\r
def global_var(self, name):\r
"""Inserts a new static (global) variable definition"""\r
self.newline_text("WORD\t1", True)\r
\r
def arithmetic_mnemonic(self, op_name, op_type):\r
- """Generates an arithmetic instruction mnemonic"""\r
+ """\r
+ Generates an arithmetic instruction mnemonic\r
+ """\r
return self.OPERATIONS[op_name] + self.OPSIGNS[op_type]\r
\r
- def arithmetic(self, operation, operand1, operand2, operand3 = None):\r
- """Generates an arithmetic instruction\r
- operation - one of supporetd operations\r
- operandX - index in symbol table or text representation of operand\r
- First two operands are input, third one is output\r
+ def arithmetic(self, operation, operand1, operand2, operand3=None):\r
+ """\r
+ Generates an arithmetic instruction\r
+ operation - one of supporetd operations\r
+ operandX - index in symbol table or text representation of operand\r
+ First two operands are input, third one is output\r
"""\r
if isinstance(operand1, int):\r
output_type = self.symtab.get_type(operand1)\r
else:\r
output_type = None\r
if isinstance(operand2, int):\r
- output_type = self.symtab.get_type(operand2) if output_type == None else output_type\r
+ output_type = (\r
+ self.symtab.get_type(operand2) if output_type == None else output_type\r
+ )\r
self.free_if_register(operand2)\r
else:\r
- output_type = SharedData.TYPES.NO_TYPE if output_type == None else output_type\r
- #if operand3 is not defined, reserve one free register for it\r
+ output_type = (\r
+ SharedData.TYPES.NO_TYPE if output_type == None else output_type\r
+ )\r
+ # if operand3 is not defined, reserve one free register for it\r
output = self.take_register(output_type) if operand3 == None else operand3\r
mnemonic = self.arithmetic_mnemonic(operation, output_type)\r
- self.newline_text("{0}\t{1},{2},{3}".format(mnemonic, self.symbol(operand1), self.symbol(operand2), self.symbol(output)), True)\r
+ self.newline_text(\r
+ "{}\t{},{},{}".format(\r
+ mnemonic,\r
+ self.symbol(operand1),\r
+ self.symbol(operand2),\r
+ self.symbol(output),\r
+ ),\r
+ True,\r
+ )\r
return output\r
\r
def relop_code(self, relop, operands_type):\r
- """Returns code for relational operator\r
- relop - relational operator\r
- operands_type - int or unsigned\r
+ """\r
+ Returns code for relational operator\r
+ relop - relational operator\r
+ operands_type - int or unsigned\r
"""\r
code = self.RELATIONAL_DICT[relop]\r
- offset = 0 if operands_type == SharedData.TYPES.INT else len(SharedData.RELATIONAL_OPERATORS)\r
+ offset = (\r
+ 0\r
+ if operands_type == SharedData.TYPES.INT\r
+ else len(SharedData.RELATIONAL_OPERATORS)\r
+ )\r
return code + offset\r
\r
def jump(self, relcode, opposite, label):\r
- """Generates a jump instruction\r
- relcode - relational operator code\r
- opposite - generate normal or opposite jump\r
- label - jump label\r
"""\r
- jump = self.OPPOSITE_JUMPS[relcode] if opposite else self.CONDITIONAL_JUMPS[relcode]\r
- self.newline_text("{0}\t{1}".format(jump, label), True)\r
+ Generates a jump instruction\r
+ relcode - relational operator code\r
+ opposite - generate normal or opposite jump\r
+ label - jump label\r
+ """\r
+ jump = (\r
+ self.OPPOSITE_JUMPS[relcode]\r
+ if opposite\r
+ else self.CONDITIONAL_JUMPS[relcode]\r
+ )\r
+ self.newline_text("{}\t{}".format(jump, label), True)\r
\r
def unconditional_jump(self, label):\r
- """Generates an unconditional jump instruction\r
- label - jump label\r
"""\r
- self.newline_text("JMP \t{0}".format(label), True)\r
+ Generates an unconditional jump instruction\r
+ label - jump label\r
+ """\r
+ self.newline_text("JMP \t{}".format(label), True)\r
\r
- def move(self,operand1, operand2):\r
- """Generates a move instruction\r
- If the output operand (opernad2) is a working register, sets it's type\r
- operandX - index in symbol table or text representation of operand\r
+ def move(self, operand1, operand2):\r
+ """\r
+ Generates a move instruction\r
+ If the output operand (opernad2) is a working register, sets it's type\r
+ operandX - index in symbol table or text representation of operand\r
"""\r
if isinstance(operand1, int):\r
output_type = self.symtab.get_type(operand1)\r
self.free_if_register(operand1)\r
else:\r
output_type = SharedData.TYPES.NO_TYPE\r
- self.newline_text("MOV \t{0},{1}".format(self.symbol(operand1), self.symbol(operand2)), True)\r
+ self.newline_text(\r
+ "MOV \t{},{}".format(self.symbol(operand1), self.symbol(operand2)), True\r
+ )\r
if isinstance(operand2, int):\r
if self.symtab.get_kind(operand2) == SharedData.KINDS.WORKING_REGISTER:\r
self.symtab.set_type(operand2, output_type)\r
self.newline_text("POP \t%s" % self.symbol(operand), True)\r
\r
def compare(self, operand1, operand2):\r
- """Generates a compare instruction\r
- operandX - index in symbol table\r
+ """\r
+ Generates a compare instruction\r
+ operandX - index in symbol table\r
"""\r
typ = self.symtab.get_type(operand1)\r
self.free_if_register(operand1)\r
self.free_if_register(operand2)\r
- self.newline_text("CMP{0}\t{1},{2}".format(self.OPSIGNS[typ], self.symbol(operand1), self.symbol(operand2)), True)\r
+ self.newline_text(\r
+ "CMP{}\t{},{}".format(\r
+ self.OPSIGNS[typ], self.symbol(operand1), self.symbol(operand2)\r
+ ),\r
+ True,\r
+ )\r
\r
def function_begin(self):\r
"""Inserts function name label and function frame initialization"""\r
def function_body(self):\r
"""Inserts a local variable initialization and body label"""\r
if self.shared.function_vars > 0:\r
- const = self.symtab.insert_constant("0{}".format(self.shared.function_vars * 4), SharedData.TYPES.UNSIGNED)\r
+ const = self.symtab.insert_constant(\r
+ "0{}".format(self.shared.function_vars * 4), SharedData.TYPES.UNSIGNED\r
+ )\r
self.arithmetic("-", "%15", const, "%15")\r
self.newline_label(self.shared.function_name + "_body", True, True)\r
\r
self.newline_text("RET", True)\r
\r
def function_call(self, function, arguments):\r
- """Generates code for a function call\r
- function - function index in symbol table\r
- arguments - list of arguments (indexes in symbol table)\r
"""\r
- #push each argument to stack\r
+ Generates code for a function call\r
+ function - function index in symbol table\r
+ arguments - list of arguments (indexes in symbol table)\r
+ """\r
+ # push each argument to stack\r
for arg in arguments:\r
self.push(self.symbol(arg))\r
self.free_if_register(arg)\r
- self.newline_text("CALL\t"+self.symtab.get_name(function), True)\r
+ self.newline_text("CALL\t" + self.symtab.get_name(function), True)\r
args = self.symtab.get_attribute(function)\r
- #generates stack cleanup if function has arguments\r
+ # generates stack cleanup if function has arguments\r
if args > 0:\r
- args_space = self.symtab.insert_constant("{0}".format(args * 4), SharedData.TYPES.UNSIGNED)\r
+ args_space = self.symtab.insert_constant(\r
+ "{}".format(args * 4), SharedData.TYPES.UNSIGNED\r
+ )\r
self.arithmetic("+", "%15", args_space, "%15")\r
\r
+\r
##########################################################################################\r
##########################################################################################\r
\r
-class MicroC(object):\r
+\r
+class MicroC:\r
"""Class for microC parser/compiler"""\r
\r
def __init__(self):\r
- #Definitions of terminal symbols for microC programming language\r
- self.tId = Word(alphas+"_",alphanums+"_")\r
- self.tInteger = Word(nums).setParseAction(lambda x : [x[0], SharedData.TYPES.INT])\r
- self.tUnsigned = Regex(r"[0-9]+[uU]").setParseAction(lambda x : [x[0][:-1], SharedData.TYPES.UNSIGNED])\r
- self.tConstant = (self.tUnsigned | self.tInteger).setParseAction(self.constant_action)\r
- self.tType = Keyword("int").setParseAction(lambda x : SharedData.TYPES.INT) | \\r
- Keyword("unsigned").setParseAction(lambda x : SharedData.TYPES.UNSIGNED)\r
+ # Definitions of terminal symbols for microC programming language\r
+ self.tId = Word(alphas + "_", alphanums + "_")\r
+ self.tInteger = Word(nums).setParseAction(\r
+ lambda x: [x[0], SharedData.TYPES.INT]\r
+ )\r
+ self.tUnsigned = Regex(r"[0-9]+[uU]").setParseAction(\r
+ lambda x: [x[0][:-1], SharedData.TYPES.UNSIGNED]\r
+ )\r
+ self.tConstant = (self.tUnsigned | self.tInteger).setParseAction(\r
+ self.constant_action\r
+ )\r
+ self.tType = Keyword("int").setParseAction(\r
+ lambda x: SharedData.TYPES.INT\r
+ ) | Keyword("unsigned").setParseAction(lambda x: SharedData.TYPES.UNSIGNED)\r
self.tRelOp = oneOf(SharedData.RELATIONAL_OPERATORS)\r
self.tMulOp = oneOf("* /")\r
self.tAddOp = oneOf("+ -")\r
\r
- #Definitions of rules for global variables\r
- self.rGlobalVariable = (self.tType("type") + self.tId("name") +\r
- FollowedBy(";")).setParseAction(self.global_variable_action)\r
+ # Definitions of rules for global variables\r
+ self.rGlobalVariable = (\r
+ self.tType("type") + self.tId("name") + FollowedBy(";")\r
+ ).setParseAction(self.global_variable_action)\r
self.rGlobalVariableList = ZeroOrMore(self.rGlobalVariable + Suppress(";"))\r
\r
- #Definitions of rules for numeric expressions\r
+ # Definitions of rules for numeric expressions\r
self.rExp = Forward()\r
self.rMulExp = Forward()\r
self.rNumExp = Forward()\r
- self.rArguments = delimitedList(self.rNumExp("exp").setParseAction(self.argument_action))\r
- self.rFunctionCall = ((self.tId("name") + FollowedBy("(")).setParseAction(self.function_call_prepare_action) +\r
- Suppress("(") + Optional(self.rArguments)("args") + Suppress(")")).setParseAction(self.function_call_action)\r
- self.rExp << (self.rFunctionCall |\r
- self.tConstant |\r
- self.tId("name").setParseAction(self.lookup_id_action) |\r
- Group(Suppress("(") + self.rNumExp + Suppress(")")) |\r
- Group("+" + self.rExp) |\r
- Group("-" + self.rExp)).setParseAction(lambda x : x[0])\r
- self.rMulExp << ((self.rExp + ZeroOrMore(self.tMulOp + self.rExp))).setParseAction(self.mulexp_action)\r
- self.rNumExp << (self.rMulExp + ZeroOrMore(self.tAddOp + self.rMulExp)).setParseAction(self.numexp_action)\r
-\r
- #Definitions of rules for logical expressions (these are without parenthesis support)\r
+ self.rArguments = delimitedList(\r
+ self.rNumExp("exp").setParseAction(self.argument_action)\r
+ )\r
+ self.rFunctionCall = (\r
+ (self.tId("name") + FollowedBy("(")).setParseAction(\r
+ self.function_call_prepare_action\r
+ )\r
+ + Suppress("(")\r
+ + Optional(self.rArguments)("args")\r
+ + Suppress(")")\r
+ ).setParseAction(self.function_call_action)\r
+ self.rExp << (\r
+ self.rFunctionCall\r
+ | self.tConstant\r
+ | self.tId("name").setParseAction(self.lookup_id_action)\r
+ | Group(Suppress("(") + self.rNumExp + Suppress(")"))\r
+ | Group("+" + self.rExp)\r
+ | Group("-" + self.rExp)\r
+ ).setParseAction(lambda x: x[0])\r
+ self.rMulExp << (\r
+ self.rExp + ZeroOrMore(self.tMulOp + self.rExp)\r
+ ).setParseAction(self.mulexp_action)\r
+ self.rNumExp << (\r
+ self.rMulExp + ZeroOrMore(self.tAddOp + self.rMulExp)\r
+ ).setParseAction(self.numexp_action)\r
+\r
+ # Definitions of rules for logical expressions (these are without parenthesis support)\r
self.rAndExp = Forward()\r
self.rLogExp = Forward()\r
- self.rRelExp = (self.rNumExp + self.tRelOp + self.rNumExp).setParseAction(self.relexp_action)\r
- self.rAndExp << (self.rRelExp("exp") + ZeroOrMore(Literal("&&").setParseAction(self.andexp_action) +\r
- self.rRelExp("exp")).setParseAction(lambda x : self.relexp_code))\r
- self.rLogExp << (self.rAndExp("exp") + ZeroOrMore(Literal("||").setParseAction(self.logexp_action) +\r
- self.rAndExp("exp")).setParseAction(lambda x : self.andexp_code))\r
-\r
- #Definitions of rules for statements\r
+ self.rRelExp = (self.rNumExp + self.tRelOp + self.rNumExp).setParseAction(\r
+ self.relexp_action\r
+ )\r
+ self.rAndExp << (\r
+ self.rRelExp("exp")\r
+ + ZeroOrMore(\r
+ Literal("&&").setParseAction(self.andexp_action) + self.rRelExp("exp")\r
+ ).setParseAction(lambda x: self.relexp_code)\r
+ )\r
+ self.rLogExp << (\r
+ self.rAndExp("exp")\r
+ + ZeroOrMore(\r
+ Literal("||").setParseAction(self.logexp_action) + self.rAndExp("exp")\r
+ ).setParseAction(lambda x: self.andexp_code)\r
+ )\r
+\r
+ # Definitions of rules for statements\r
self.rStatement = Forward()\r
self.rStatementList = Forward()\r
- self.rReturnStatement = (Keyword("return") + self.rNumExp("exp") +\r
- Suppress(";")).setParseAction(self.return_action)\r
- self.rAssignmentStatement = (self.tId("var") + Suppress("=") + self.rNumExp("exp") +\r
- Suppress(";")).setParseAction(self.assignment_action)\r
+ self.rReturnStatement = (\r
+ Keyword("return") + self.rNumExp("exp") + Suppress(";")\r
+ ).setParseAction(self.return_action)\r
+ self.rAssignmentStatement = (\r
+ self.tId("var") + Suppress("=") + self.rNumExp("exp") + Suppress(";")\r
+ ).setParseAction(self.assignment_action)\r
self.rFunctionCallStatement = self.rFunctionCall + Suppress(";")\r
- self.rIfStatement = ( (Keyword("if") + FollowedBy("(")).setParseAction(self.if_begin_action) +\r
- (Suppress("(") + self.rLogExp + Suppress(")")).setParseAction(self.if_body_action) +\r
- (self.rStatement + Empty()).setParseAction(self.if_else_action) +\r
- Optional(Keyword("else") + self.rStatement)).setParseAction(self.if_end_action)\r
- self.rWhileStatement = ( (Keyword("while") + FollowedBy("(")).setParseAction(self.while_begin_action) +\r
- (Suppress("(") + self.rLogExp + Suppress(")")).setParseAction(self.while_body_action) +\r
- self.rStatement).setParseAction(self.while_end_action)\r
- self.rCompoundStatement = Group(Suppress("{") + self.rStatementList + Suppress("}"))\r
- self.rStatement << (self.rReturnStatement | self.rIfStatement | self.rWhileStatement |\r
- self.rFunctionCallStatement | self.rAssignmentStatement | self.rCompoundStatement)\r
+ self.rIfStatement = (\r
+ (Keyword("if") + FollowedBy("(")).setParseAction(self.if_begin_action)\r
+ + (Suppress("(") + self.rLogExp + Suppress(")")).setParseAction(\r
+ self.if_body_action\r
+ )\r
+ + (self.rStatement + Empty()).setParseAction(self.if_else_action)\r
+ + Optional(Keyword("else") + self.rStatement)\r
+ ).setParseAction(self.if_end_action)\r
+ self.rWhileStatement = (\r
+ (Keyword("while") + FollowedBy("(")).setParseAction(self.while_begin_action)\r
+ + (Suppress("(") + self.rLogExp + Suppress(")")).setParseAction(\r
+ self.while_body_action\r
+ )\r
+ + self.rStatement\r
+ ).setParseAction(self.while_end_action)\r
+ self.rCompoundStatement = Group(\r
+ Suppress("{") + self.rStatementList + Suppress("}")\r
+ )\r
+ self.rStatement << (\r
+ self.rReturnStatement\r
+ | self.rIfStatement\r
+ | self.rWhileStatement\r
+ | self.rFunctionCallStatement\r
+ | self.rAssignmentStatement\r
+ | self.rCompoundStatement\r
+ )\r
self.rStatementList << ZeroOrMore(self.rStatement)\r
\r
- self.rLocalVariable = (self.tType("type") + self.tId("name") + FollowedBy(";")).setParseAction(self.local_variable_action)\r
+ self.rLocalVariable = (\r
+ self.tType("type") + self.tId("name") + FollowedBy(";")\r
+ ).setParseAction(self.local_variable_action)\r
self.rLocalVariableList = ZeroOrMore(self.rLocalVariable + Suppress(";"))\r
- self.rFunctionBody = Suppress("{") + Optional(self.rLocalVariableList).setParseAction(self.function_body_action) + \\r
- self.rStatementList + Suppress("}")\r
- self.rParameter = (self.tType("type") + self.tId("name")).setParseAction(self.parameter_action)\r
+ self.rFunctionBody = (\r
+ Suppress("{")\r
+ + Optional(self.rLocalVariableList).setParseAction(\r
+ self.function_body_action\r
+ )\r
+ + self.rStatementList\r
+ + Suppress("}")\r
+ )\r
+ self.rParameter = (self.tType("type") + self.tId("name")).setParseAction(\r
+ self.parameter_action\r
+ )\r
self.rParameterList = delimitedList(self.rParameter)\r
- self.rFunction = ( (self.tType("type") + self.tId("name")).setParseAction(self.function_begin_action) +\r
- Group(Suppress("(") + Optional(self.rParameterList)("params") + Suppress(")") +\r
- self.rFunctionBody)).setParseAction(self.function_end_action)\r
+ self.rFunction = (\r
+ (self.tType("type") + self.tId("name")).setParseAction(\r
+ self.function_begin_action\r
+ )\r
+ + Group(\r
+ Suppress("(")\r
+ + Optional(self.rParameterList)("params")\r
+ + Suppress(")")\r
+ + self.rFunctionBody\r
+ )\r
+ ).setParseAction(self.function_end_action)\r
\r
self.rFunctionList = OneOrMore(self.rFunction)\r
- self.rProgram = (Empty().setParseAction(self.data_begin_action) + self.rGlobalVariableList +\r
- Empty().setParseAction(self.code_begin_action) + self.rFunctionList).setParseAction(self.program_end_action)\r
-\r
- #shared data\r
+ self.rProgram = (\r
+ Empty().setParseAction(self.data_begin_action)\r
+ + self.rGlobalVariableList\r
+ + Empty().setParseAction(self.code_begin_action)\r
+ + self.rFunctionList\r
+ ).setParseAction(self.program_end_action)\r
+\r
+ # shared data\r
self.shared = SharedData()\r
- #symbol table\r
+ # symbol table\r
self.symtab = SymbolTable(self.shared)\r
- #code generator\r
+ # code generator\r
self.codegen = CodeGenerator(self.shared, self.symtab)\r
\r
- #index of the current function call\r
+ # index of the current function call\r
self.function_call_index = -1\r
- #stack for the nested function calls\r
+ # stack for the nested function calls\r
self.function_call_stack = []\r
- #arguments of the current function call\r
+ # arguments of the current function call\r
self.function_arguments = []\r
- #stack for arguments of the nested function calls\r
+ # stack for arguments of the nested function calls\r
self.function_arguments_stack = []\r
- #number of arguments for the curent function call\r
+ # number of arguments for the current function call\r
self.function_arguments_number = -1\r
- #stack for the number of arguments for the nested function calls\r
+ # stack for the number of arguments for the nested function calls\r
self.function_arguments_number_stack = []\r
\r
- #last relational expression\r
+ # last relational expression\r
self.relexp_code = None\r
- #last and expression\r
+ # last and expression\r
self.andexp_code = None\r
- #label number for "false" internal labels\r
+ # label number for "false" internal labels\r
self.false_label_number = -1\r
- #label number for all other internal labels\r
+ # label number for all other internal labels\r
self.label_number = None\r
- #label stack for nested statements\r
+ # label stack for nested statements\r
self.label_stack = []\r
\r
def warning(self, message, print_location=True):\r
msg += "\n%s" % wtext\r
print(msg)\r
\r
-\r
def data_begin_action(self):\r
"""Inserts text at start of data segment"""\r
self.codegen.prepare_data_segment()\r
"""Code executed after recognising a global variable"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("GLOBAL_VAR:",var)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("GLOBAL_VAR:", var)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
index = self.symtab.insert_global_var(var.name, var.type)\r
self.codegen.global_var(var.name)\r
return index\r
"""Code executed after recognising a local variable"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("LOCAL_VAR:",var, var.name, var.type)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- index = self.symtab.insert_local_var(var.name, var.type, self.shared.function_vars)\r
+ print("LOCAL_VAR:", var, var.name, var.type)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ index = self.symtab.insert_local_var(\r
+ var.name, var.type, self.shared.function_vars\r
+ )\r
self.shared.function_vars += 1\r
return index\r
\r
"""Code executed after recognising a parameter"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("PARAM:",par)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("PARAM:", par)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
index = self.symtab.insert_parameter(par.name, par.type)\r
self.shared.function_params += 1\r
return index\r
"""Code executed after recognising a constant"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("CONST:",const)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("CONST:", const)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
return self.symtab.insert_constant(const[0], const[1])\r
\r
def function_begin_action(self, text, loc, fun):\r
"""Code executed after recognising a function definition (type and function name)"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("FUN_BEGIN:",fun)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("FUN_BEGIN:", fun)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
self.shared.function_index = self.symtab.insert_function(fun.name, fun.type)\r
self.shared.function_name = fun.name\r
self.shared.function_params = 0\r
self.shared.function_vars = 0\r
- self.codegen.function_begin();\r
+ self.codegen.function_begin()\r
\r
def function_body_action(self, text, loc, fun):\r
"""Code executed after recognising the beginning of function's body"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("FUN_BODY:",fun)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("FUN_BODY:", fun)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
self.codegen.function_body()\r
\r
def function_end_action(self, text, loc, fun):\r
"""Code executed at the end of function definition"""\r
if DEBUG > 0:\r
- print("FUN_END:",fun)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- #set function's attribute to number of function parameters\r
- self.symtab.set_attribute(self.shared.function_index, self.shared.function_params)\r
- #clear local function symbols (but leave function name)\r
+ print("FUN_END:", fun)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ # set function's attribute to number of function parameters\r
+ self.symtab.set_attribute(\r
+ self.shared.function_index, self.shared.function_params\r
+ )\r
+ # clear local function symbols (but leave function name)\r
self.symtab.clear_symbols(self.shared.function_index + 1)\r
self.codegen.function_end()\r
\r
"""Code executed after recognising a return statement"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("RETURN:",ret)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("RETURN:", ret)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
if not self.symtab.same_types(self.shared.function_index, ret.exp[0]):\r
raise SemanticException("Incompatible type in return")\r
- #set register for function's return value to expression value\r
+ # set register for function's return value to expression value\r
reg = self.codegen.take_function_register()\r
self.codegen.move(ret.exp[0], reg)\r
- #after return statement, register for function's return value is available again\r
+ # after return statement, register for function's return value is available again\r
self.codegen.free_register(reg)\r
- #jump to function's exit\r
- self.codegen.unconditional_jump(self.codegen.label(self.shared.function_name+"_exit", True))\r
+ # jump to function's exit\r
+ self.codegen.unconditional_jump(\r
+ self.codegen.label(self.shared.function_name + "_exit", True)\r
+ )\r
\r
def lookup_id_action(self, text, loc, var):\r
"""Code executed after recognising an identificator in expression"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("EXP_VAR:",var)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- var_index = self.symtab.lookup_symbol(var.name, [SharedData.KINDS.GLOBAL_VAR, SharedData.KINDS.PARAMETER, SharedData.KINDS.LOCAL_VAR])\r
+ print("EXP_VAR:", var)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ var_index = self.symtab.lookup_symbol(\r
+ var.name,\r
+ [\r
+ SharedData.KINDS.GLOBAL_VAR,\r
+ SharedData.KINDS.PARAMETER,\r
+ SharedData.KINDS.LOCAL_VAR,\r
+ ],\r
+ )\r
if var_index == None:\r
raise SemanticException("'%s' undefined" % var.name)\r
return var_index\r
"""Code executed after recognising an assignment statement"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("ASSIGN:",assign)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- var_index = self.symtab.lookup_symbol(assign.var, [SharedData.KINDS.GLOBAL_VAR, SharedData.KINDS.PARAMETER, SharedData.KINDS.LOCAL_VAR])\r
+ print("ASSIGN:", assign)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ var_index = self.symtab.lookup_symbol(\r
+ assign.var,\r
+ [\r
+ SharedData.KINDS.GLOBAL_VAR,\r
+ SharedData.KINDS.PARAMETER,\r
+ SharedData.KINDS.LOCAL_VAR,\r
+ ],\r
+ )\r
if var_index == None:\r
raise SemanticException("Undefined lvalue '%s' in assignment" % assign.var)\r
if not self.symtab.same_types(var_index, assign.exp[0]):\r
"""Code executed after recognising a mulexp expression (something *|/ something)"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("MUL_EXP:",mul)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- #iterate through all multiplications/divisions\r
+ print("MUL_EXP:", mul)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ # iterate through all multiplications/divisions\r
m = list(mul)\r
while len(m) > 1:\r
if not self.symtab.same_types(m[0], m[2]):\r
raise SemanticException("Invalid opernads to binary '%s'" % m[1])\r
reg = self.codegen.arithmetic(m[1], m[0], m[2])\r
- #replace first calculation with it's result\r
+ # replace first calculation with it's result\r
m[0:3] = [reg]\r
return m[0]\r
\r
"""Code executed after recognising a numexp expression (something +|- something)"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("NUM_EXP:",num)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- #iterate through all additions/substractions\r
+ print("NUM_EXP:", num)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ # iterate through all additions/substractions\r
n = list(num)\r
while len(n) > 1:\r
if not self.symtab.same_types(n[0], n[2]):\r
raise SemanticException("Invalid opernads to binary '%s'" % n[1])\r
reg = self.codegen.arithmetic(n[1], n[0], n[2])\r
- #replace first calculation with it's result\r
+ # replace first calculation with it's result\r
n[0:3] = [reg]\r
return n[0]\r
\r
"""Code executed after recognising a function call (type and function name)"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("FUN_PREP:",fun)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("FUN_PREP:", fun)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
index = self.symtab.lookup_symbol(fun.name, SharedData.KINDS.FUNCTION)\r
if index == None:\r
raise SemanticException("'%s' is not a function" % fun.name)\r
- #save any previous function call data (for nested function calls)\r
+ # save any previous function call data (for nested function calls)\r
self.function_call_stack.append(self.function_call_index)\r
self.function_call_index = index\r
self.function_arguments_stack.append(self.function_arguments[:])\r
"""Code executed after recognising each of function's arguments"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("ARGUMENT:",arg.exp)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("ARGUMENT:", arg.exp)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
arg_ordinal = len(self.function_arguments)\r
- #check argument's type\r
- if not self.symtab.same_type_as_argument(arg.exp, self.function_call_index, arg_ordinal):\r
- raise SemanticException("Incompatible type for argument %d in '%s'" % (arg_ordinal + 1, self.symtab.get_name(self.function_call_index)))\r
+ # check argument's type\r
+ if not self.symtab.same_type_as_argument(\r
+ arg.exp, self.function_call_index, arg_ordinal\r
+ ):\r
+ raise SemanticException(\r
+ "Incompatible type for argument %d in '%s'"\r
+ % (arg_ordinal + 1, self.symtab.get_name(self.function_call_index))\r
+ )\r
self.function_arguments.append(arg.exp)\r
\r
def function_call_action(self, text, loc, fun):\r
"""Code executed after recognising the whole function call"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("FUN_CALL:",fun)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- #check number of arguments\r
- if len(self.function_arguments) != self.symtab.get_attribute(self.function_call_index):\r
- raise SemanticException("Wrong number of arguments for function '%s'" % fun.name)\r
- #arguments should be pushed to stack in reverse order\r
+ print("FUN_CALL:", fun)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ # check number of arguments\r
+ if len(self.function_arguments) != self.symtab.get_attribute(\r
+ self.function_call_index\r
+ ):\r
+ raise SemanticException(\r
+ "Wrong number of arguments for function '%s'" % fun.name\r
+ )\r
+ # arguments should be pushed to stack in reverse order\r
self.function_arguments.reverse()\r
self.codegen.function_call(self.function_call_index, self.function_arguments)\r
self.codegen.restore_used_registers()\r
return_type = self.symtab.get_type(self.function_call_index)\r
- #restore previous function call data\r
+ # restore previous function call data\r
self.function_call_index = self.function_call_stack.pop()\r
self.function_arguments = self.function_arguments_stack.pop()\r
register = self.codegen.take_register(return_type)\r
- #move result to a new free register, to allow the next function call\r
+ # move result to a new free register, to allow the next function call\r
self.codegen.move(self.codegen.take_function_register(return_type), register)\r
return register\r
\r
def relexp_action(self, text, loc, arg):\r
"""Code executed after recognising a relexp expression (something relop something)"""\r
if DEBUG > 0:\r
- print("REL_EXP:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("REL_EXP:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
exshared.setpos(loc, text)\r
if not self.symtab.same_types(arg[0], arg[2]):\r
- raise SemanticException("Invalid operands for operator '{0}'".format(arg[1]))\r
+ raise SemanticException("Invalid operands for operator '{}'".format(arg[1]))\r
self.codegen.compare(arg[0], arg[2])\r
- #return relational operator's code\r
+ # return relational operator's code\r
self.relexp_code = self.codegen.relop_code(arg[1], self.symtab.get_type(arg[0]))\r
return self.relexp_code\r
\r
"""Code executed after recognising a andexp expression (something and something)"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("AND+EXP:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- label = self.codegen.label("false{0}".format(self.false_label_number), True, False)\r
+ print("AND+EXP:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ label = self.codegen.label(\r
+ "false{}".format(self.false_label_number), True, False\r
+ )\r
self.codegen.jump(self.relexp_code, True, label)\r
self.andexp_code = self.relexp_code\r
return self.andexp_code\r
"""Code executed after recognising logexp expression (something or something)"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("LOG_EXP:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- label = self.codegen.label("true{0}".format(self.label_number), True, False)\r
+ print("LOG_EXP:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ label = self.codegen.label("true{}".format(self.label_number), True, False)\r
self.codegen.jump(self.relexp_code, False, label)\r
- self.codegen.newline_label("false{0}".format(self.false_label_number), True, True)\r
+ self.codegen.newline_label(\r
+ "false{}".format(self.false_label_number), True, True\r
+ )\r
self.false_label_number += 1\r
\r
def if_begin_action(self, text, loc, arg):\r
"""Code executed after recognising an if statement (if keyword)"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("IF_BEGIN:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("IF_BEGIN:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
self.false_label_number += 1\r
self.label_number = self.false_label_number\r
- self.codegen.newline_label("if{0}".format(self.label_number), True, True)\r
+ self.codegen.newline_label("if{}".format(self.label_number), True, True)\r
\r
def if_body_action(self, text, loc, arg):\r
"""Code executed after recognising if statement's body"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("IF_BODY:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- #generate conditional jump (based on last compare)\r
- label = self.codegen.label("false{0}".format(self.false_label_number), True, False)\r
+ print("IF_BODY:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ # generate conditional jump (based on last compare)\r
+ label = self.codegen.label(\r
+ "false{}".format(self.false_label_number), True, False\r
+ )\r
self.codegen.jump(self.relexp_code, True, label)\r
- #generate 'true' label (executes if condition is satisfied)\r
- self.codegen.newline_label("true{0}".format(self.label_number), True, True)\r
- #save label numbers (needed for nested if/while statements)\r
+ # generate 'true' label (executes if condition is satisfied)\r
+ self.codegen.newline_label("true{}".format(self.label_number), True, True)\r
+ # save label numbers (needed for nested if/while statements)\r
self.label_stack.append(self.false_label_number)\r
self.label_stack.append(self.label_number)\r
\r
"""Code executed after recognising if statement's else body"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("IF_ELSE:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- #jump to exit after all statements for true condition are executed\r
+ print("IF_ELSE:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ # jump to exit after all statements for true condition are executed\r
self.label_number = self.label_stack.pop()\r
- label = self.codegen.label("exit{0}".format(self.label_number), True, False)\r
+ label = self.codegen.label("exit{}".format(self.label_number), True, False)\r
self.codegen.unconditional_jump(label)\r
- #generate final 'false' label (executes if condition isn't satisfied)\r
- self.codegen.newline_label("false{0}".format(self.label_stack.pop()), True, True)\r
+ # generate final 'false' label (executes if condition isn't satisfied)\r
+ self.codegen.newline_label("false{}".format(self.label_stack.pop()), True, True)\r
self.label_stack.append(self.label_number)\r
\r
def if_end_action(self, text, loc, arg):\r
"""Code executed after recognising a whole if statement"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("IF_END:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- self.codegen.newline_label("exit{0}".format(self.label_stack.pop()), True, True)\r
+ print("IF_END:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ self.codegen.newline_label("exit{}".format(self.label_stack.pop()), True, True)\r
\r
def while_begin_action(self, text, loc, arg):\r
"""Code executed after recognising a while statement (while keyword)"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("WHILE_BEGIN:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
+ print("WHILE_BEGIN:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
self.false_label_number += 1\r
self.label_number = self.false_label_number\r
- self.codegen.newline_label("while{0}".format(self.label_number), True, True)\r
+ self.codegen.newline_label("while{}".format(self.label_number), True, True)\r
\r
def while_body_action(self, text, loc, arg):\r
"""Code executed after recognising while statement's body"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("WHILE_BODY:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- #generate conditional jump (based on last compare)\r
- label = self.codegen.label("false{0}".format(self.false_label_number), True, False)\r
+ print("WHILE_BODY:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ # generate conditional jump (based on last compare)\r
+ label = self.codegen.label(\r
+ "false{}".format(self.false_label_number), True, False\r
+ )\r
self.codegen.jump(self.relexp_code, True, label)\r
- #generate 'true' label (executes if condition is satisfied)\r
- self.codegen.newline_label("true{0}".format(self.label_number), True, True)\r
+ # generate 'true' label (executes if condition is satisfied)\r
+ self.codegen.newline_label("true{}".format(self.label_number), True, True)\r
self.label_stack.append(self.false_label_number)\r
self.label_stack.append(self.label_number)\r
\r
"""Code executed after recognising a whole while statement"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("WHILE_END:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- #jump to condition checking after while statement body\r
+ print("WHILE_END:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ # jump to condition checking after while statement body\r
self.label_number = self.label_stack.pop()\r
- label = self.codegen.label("while{0}".format(self.label_number), True, False)\r
+ label = self.codegen.label("while{}".format(self.label_number), True, False)\r
self.codegen.unconditional_jump(label)\r
- #generate final 'false' label and exit label\r
- self.codegen.newline_label("false{0}".format(self.label_stack.pop()), True, True)\r
- self.codegen.newline_label("exit{0}".format(self.label_number), True, True)\r
+ # generate final 'false' label and exit label\r
+ self.codegen.newline_label("false{}".format(self.label_stack.pop()), True, True)\r
+ self.codegen.newline_label("exit{}".format(self.label_number), True, True)\r
\r
def program_end_action(self, text, loc, arg):\r
"""Checks if there is a 'main' function and the type of 'main' function"""\r
exshared.setpos(loc, text)\r
if DEBUG > 0:\r
- print("PROGRAM_END:",arg)\r
- if DEBUG == 2: self.symtab.display()\r
- if DEBUG > 2: return\r
- index = self.symtab.lookup_symbol("main",SharedData.KINDS.FUNCTION)\r
+ print("PROGRAM_END:", arg)\r
+ if DEBUG == 2:\r
+ self.symtab.display()\r
+ if DEBUG > 2:\r
+ return\r
+ index = self.symtab.lookup_symbol("main", SharedData.KINDS.FUNCTION)\r
if index == None:\r
raise SemanticException("Undefined reference to 'main'", False)\r
elif self.symtab.get_type(index) != SharedData.TYPES.INT:\r
self.warning("Return type of 'main' is not int", False)\r
\r
- def parse_text(self,text):\r
+ def parse_text(self, text):\r
"""Parse string (helper function)"""\r
try:\r
return self.rProgram.ignore(cStyleComment).parseString(text, parseAll=True)\r
print(err)\r
exit(3)\r
\r
- def parse_file(self,filename):\r
+ def parse_file(self, filename):\r
"""Parse file (helper function)"""\r
try:\r
- return self.rProgram.ignore(cStyleComment).parseFile(filename, parseAll=True)\r
+ return self.rProgram.ignore(cStyleComment).parseFile(\r
+ filename, parseAll=True\r
+ )\r
except SemanticException as err:\r
print(err)\r
exit(3)\r
print(err)\r
exit(3)\r
\r
+\r
##########################################################################################\r
##########################################################################################\r
if 0:\r
- #main program\r
+ # main program\r
mc = MicroC()\r
output_file = "output.asm"\r
\r
input_file = argv[1]\r
output_file = argv[2]\r
else:\r
- usage = """Usage: {0} [input_file [output_file]]\r
+ usage = """Usage: {} [input_file [output_file]]\r
If output file is omitted, output.asm is used\r
- If input file is omitted, stdin is used""".format(argv[0])\r
+ If input file is omitted, stdin is used""".format(\r
+ argv[0]\r
+ )\r
print(usage)\r
exit(1)\r
try:\r
- parse = stdin if input_file == stdin else open(input_file,'r')\r
+ parse = stdin if input_file == stdin else open(input_file, "r")\r
except Exception:\r
print("Input file '%s' open error" % input_file)\r
exit(2)\r
mc.parse_file(parse)\r
- #if you want to see the final symbol table, uncomment next line\r
- #mc.symtab.display()\r
+ # if you want to see the final symbol table, uncomment next line\r
+ # mc.symtab.display()\r
try:\r
- out = open(output_file, 'w')\r
+ out = open(output_file, "w")\r
out.write(mc.codegen.code)\r
out.close\r
except Exception:\r
# Note: Changing the grammar specified in this file will most likely\r
# require corresponding changes in the parser module\r
# (../Modules/parsermodule.c). If you can't make the changes to\r
-# that module yourself, please co-ordinate the required changes\r
+# that module yourself, please coordinate the required changes\r
# with someone who can; ask around on python-dev for help. Fred\r
# Drake <fdrake@acm.org> will probably be listening there.\r
\r
encoding_decl: NAME\r
"""\r
\r
-class SemanticGroup(object):\r
- def __init__(self,contents):\r
+\r
+class SemanticGroup:\r
+ def __init__(self, contents):\r
self.contents = contents\r
while self.contents[-1].__class__ == self.__class__:\r
self.contents = self.contents[:-1] + self.contents[-1].contents\r
\r
def __str__(self):\r
- return "{0}({1})".format(self.label,\r
- " ".join([isinstance(c,str) and c or str(c) for c in self.contents]) )\r
+ return "{}({})".format(\r
+ self.label,\r
+ " ".join([isinstance(c, str) and c or str(c) for c in self.contents]),\r
+ )\r
+\r
\r
class OrList(SemanticGroup):\r
label = "OR"\r
pass\r
\r
+\r
class AndList(SemanticGroup):\r
label = "AND"\r
pass\r
\r
+\r
class OptionalGroup(SemanticGroup):\r
label = "OPT"\r
pass\r
\r
+\r
class Atom(SemanticGroup):\r
- def __init__(self,contents):\r
+ def __init__(self, contents):\r
if len(contents) > 1:\r
self.rep = contents[1]\r
else:\r
self.rep = ""\r
- if isinstance(contents,str):\r
+ if isinstance(contents, str):\r
self.contents = contents\r
else:\r
self.contents = contents[0]\r
\r
def __str__(self):\r
- return "{0}{1}".format(self.rep, self.contents)\r
+ return "{}{}".format(self.rep, self.contents)\r
+\r
\r
def makeGroupObject(cls):\r
- def groupAction(s,l,t):\r
+ def groupAction(s, l, t):\r
try:\r
return cls(t[0].asList())\r
except Exception:\r
return cls(t)\r
+\r
return groupAction\r
\r
\r
RPAREN = Suppress(")")\r
LBRACK = Suppress("[")\r
RBRACK = Suppress("]")\r
-COLON = Suppress(":")\r
+COLON = Suppress(":")\r
ALT_OP = Suppress("|")\r
\r
# bnf grammar\r
-ident = Word(alphanums+"_")\r
-bnfToken = Word(alphanums+"_") + ~FollowedBy(":")\r
+ident = Word(alphanums + "_")\r
+bnfToken = Word(alphanums + "_") + ~FollowedBy(":")\r
repSymbol = oneOf("* +")\r
bnfExpr = Forward()\r
-optionalTerm = Group(LBRACK + bnfExpr + RBRACK).setParseAction(makeGroupObject(OptionalGroup))\r
-bnfTerm = ( (bnfToken | quotedString | optionalTerm | ( LPAREN + bnfExpr + RPAREN )) + Optional(repSymbol) ).setParseAction(makeGroupObject(Atom))\r
+optionalTerm = Group(LBRACK + bnfExpr + RBRACK).setParseAction(\r
+ makeGroupObject(OptionalGroup)\r
+)\r
+bnfTerm = (\r
+ (bnfToken | quotedString | optionalTerm | (LPAREN + bnfExpr + RPAREN))\r
+ + Optional(repSymbol)\r
+).setParseAction(makeGroupObject(Atom))\r
andList = Group(bnfTerm + OneOrMore(bnfTerm)).setParseAction(makeGroupObject(AndList))\r
bnfFactor = andList | bnfTerm\r
-orList = Group( bnfFactor + OneOrMore( ALT_OP + bnfFactor ) ).setParseAction(makeGroupObject(OrList))\r
-bnfExpr << ( orList | bnfFactor )\r
+orList = Group(bnfFactor + OneOrMore(ALT_OP + bnfFactor)).setParseAction(\r
+ makeGroupObject(OrList)\r
+)\r
+bnfExpr << (orList | bnfFactor)\r
bnfLine = ident + COLON + bnfExpr\r
\r
bnfComment = "#" + restOfLine\r
\r
# correct answer is 78\r
expected = 78\r
-assert len(bnfDefs) == expected, \\r
- "Error, found %d BNF defns, expected %d" % (len(bnfDefs), expected)\r
+assert len(bnfDefs) == expected, "Error, found %d BNF defns, expected %d" % (\r
+ len(bnfDefs),\r
+ expected,\r
+)\r
\r
# list out defns in order they were parsed (to verify accuracy of parsing)\r
-for k,v in bnfDefs:\r
- print(k,"=",v)\r
+for k, v in bnfDefs:\r
+ print(k, "=", v)\r
print()\r
\r
# list out parsed grammar defns (demonstrates dictionary access to parsed tokens)\r
for k in list(bnfDefs.keys()):\r
- print(k,"=",bnfDefs[k])\r
+ print(k, "=", bnfDefs[k])\r
--- /dev/null
+import pyparsing as pp
+
+ppc = pp.pyparsing_common
+
+# fmt: off
+word = pp.Word(pp.alphas).setName("word")
+integer = pp.Word(pp.nums).setName("integer")
+plus_minus = pp.Char("+-").set_name("add_sub")
+mult_div = pp.Char("*/").set_name("mult_div")
+street_address = pp.Group(integer("house_number")
+ + word[1, ...]("street_name")
+ ).setName("street_address")
+time = pp.Regex(r"\d\d:\d\d")
+
+grammar = (pp.Group(integer[1, ...])
+ + (ppc.ipv4_address
+ & word("header_word")
+ & pp.Optional(time)
+ ).setName("header with various elements")("header")
+ + street_address("address")
+ + pp.Group(pp.counted_array(word))
+ + pp.Group(integer * 8)("data")
+ + pp.Group(pp.Word("abc") + pp.Word("def")*3)
+ + pp.infix_notation(integer,
+ [
+ (plus_minus().setName("pos_neg"), 1, pp.opAssoc.RIGHT),
+ (mult_div, 2, pp.opAssoc.LEFT),
+ (plus_minus, 2, pp.opAssoc.LEFT),
+ ]).setName("simple_arithmetic")
+ + ...
+ + pp.Group(ppc.ipv4_address)("ip_address")
+ ).setName("grammar")
+
+
+grammar.create_diagram("railroad_diagram_demo.html", vertical=6, show_results_names=True)
+
+test = """\
+ 1 2 3
+ ABC 1.2.3.4 12:45
+ 123 Main St
+ 4
+ abc def ghi jkl
+ 5 5 5 5 5 5 5 5
+ a d d d
+ 2+2
+ alice bob charlie dave 5.6.7.8"""
+result = grammar.runTests([test])
from pyparsing import Word, nums, Suppress, Optional\r
from datetime import datetime\r
\r
+\r
def ranged_value(expr, minval=None, maxval=None):\r
# have to specify at least one range boundary\r
if minval is None and maxval is None:\r
# set range testing function and error message depending on\r
# whether either or both min and max values are given\r
inRangeCondition = {\r
- (True, False) : lambda s,l,t : t[0] <= maxval,\r
- (False, True) : lambda s,l,t : minval <= t[0],\r
- (False, False) : lambda s,l,t : minval <= t[0] <= maxval,\r
- }[minval is None, maxval is None]\r
+ (True, False): lambda s, l, t: t[0] <= maxval,\r
+ (False, True): lambda s, l, t: minval <= t[0],\r
+ (False, False): lambda s, l, t: minval <= t[0] <= maxval,\r
+ }[minval is None, maxval is None]\r
outOfRangeMessage = {\r
- (True, False) : "value is greater than %s" % maxval,\r
- (False, True) : "value is less than %s" % minval,\r
- (False, False) : "value is not in the range ({0} to {1})".format(minval,maxval),\r
- }[minval is None, maxval is None]\r
+ (True, False): "value is greater than %s" % maxval,\r
+ (False, True): "value is less than %s" % minval,\r
+ (False, False): "value is not in the range ({} to {})".format(minval, maxval),\r
+ }[minval is None, maxval is None]\r
\r
return expr().addCondition(inRangeCondition, message=outOfRangeMessage)\r
\r
+\r
# define the expressions for a date of the form YYYY/MM/DD or YYYY/MM (assumes YYYY/MM/01)\r
integer = Word(nums).setName("integer")\r
-integer.setParseAction(lambda t:int(t[0]))\r
+integer.setParseAction(lambda t: int(t[0]))\r
\r
month = ranged_value(integer, 1, 12)\r
day = ranged_value(integer, 1, 31)\r
year = ranged_value(integer, 2000, None)\r
\r
-SLASH = Suppress('/')\r
+SLASH = Suppress("/")\r
dateExpr = year("year") + SLASH + month("month") + Optional(SLASH + day("day"))\r
dateExpr.setName("date")\r
\r
dateExpr.setParseAction(lambda t: datetime(t.year, t.month, t.day or 1).date())\r
\r
# add range checking on dates\r
-mindate = datetime(2002,1,1).date()\r
+mindate = datetime(2002, 1, 1).date()\r
maxdate = datetime.now().date()\r
dateExpr = ranged_value(dateExpr, mindate, maxdate)\r
\r
\r
-dateExpr.runTests("""\r
+dateExpr.runTests(\r
+ """\r
2011/5/8\r
2001/1/1\r
2004/2/29\r
2004/2\r
- 1999/12/31""")\r
+ 1999/12/31"""\r
+)\r
-#~ url = "http://cmsdoc.cern.ch/cms/test/aprom/phedex/dev/gowri/datasvc/tbedi/requestDetails"\r
-#~ params = {'format':'json'}\r
-#~ import urllib\r
-#~ eparams = urllib.urlencode(params)\r
-#~ import urllib2\r
-#~ request = urllib2.Request(url,eparams)\r
-#~ response = urllib2.urlopen(request)\r
-#~ s = response.read()\r
-#~ response.close()\r
+# ~ url = "http://cmsdoc.cern.ch/cms/test/aprom/phedex/dev/gowri/datasvc/tbedi/requestDetails"\r
+# ~ params = {'format':'json'}\r
+# ~ import urllib\r
+# ~ eparams = urllib.urlencode(params)\r
+# ~ import urllib2\r
+# ~ request = urllib2.Request(url,eparams)\r
+# ~ response = urllib2.urlopen(request)\r
+# ~ s = response.read()\r
+# ~ response.close()\r
\r
-#~ print s\r
+# ~ print s\r
\r
s = """\r
{"phedex":{"request":[{"last_update":"1188037561", "numofapproved":"1",\r
\r
data = jsonObject.parseString(s)\r
\r
-#~ from pprint import pprint\r
-#~ pprint( data[0].asList() )\r
-#~ print\r
-#~ print data.dump()\r
+# ~ from pprint import pprint\r
+# ~ pprint( data[0].asList() )\r
+# ~ print\r
+# ~ print data.dump()\r
print(data.phedex.call_time)\r
print(data.phedex.instance)\r
print(data.phedex.request_call)\r
print(len(data.phedex.request))\r
for req in data.phedex.request[:10]:\r
- #~ print req.dump()\r
+ # ~ print req.dump()\r
print("-", req.id, req.last_update)\r
# define an expression for the body of a line of text - use a predicate condition to
# accept only lines with some content.
def mustBeNonBlank(t):
- return t[0] != ''
+ return t[0] != ""
# could also be written as
# return bool(t[0])
-lineBody = pp.SkipTo(line_end).addCondition(mustBeNonBlank, message="line body can't be empty")
+
+lineBody = pp.SkipTo(line_end).addCondition(
+ mustBeNonBlank, message="line body can't be empty"
+)
# now define a line with a trailing lineEnd, to be replaced with a space character
textLine = lineBody + line_end().setParseAction(pp.replaceWith(" "))
-# romanNumerals.py\r
-#\r
-# Copyright (c) 2006, Paul McGuire\r
-#\r
-\r
-from pyparsing import *\r
-\r
-def romanNumeralLiteral(numeralString, value):\r
- return Literal(numeralString).setParseAction(replaceWith(value))\r
-\r
-one = romanNumeralLiteral("I",1)\r
-four = romanNumeralLiteral("IV",4)\r
-five = romanNumeralLiteral("V",5)\r
-nine = romanNumeralLiteral("IX",9)\r
-ten = romanNumeralLiteral("X",10)\r
-forty = romanNumeralLiteral("XL",40)\r
-fifty = romanNumeralLiteral("L",50)\r
-ninety = romanNumeralLiteral("XC",90)\r
-onehundred = romanNumeralLiteral("C",100)\r
-fourhundred = romanNumeralLiteral("CD",400)\r
-fivehundred = romanNumeralLiteral("D",500)\r
-ninehundred = romanNumeralLiteral("CM",900)\r
-onethousand = romanNumeralLiteral("M",1000)\r
-\r
-numeral = ( onethousand | ninehundred | fivehundred | fourhundred |\r
- onehundred | ninety | fifty | forty | ten | nine | five |\r
- four | one ).leaveWhitespace()\r
-\r
-romanNumeral = OneOrMore(numeral).setParseAction( lambda s,l,t : sum(t) )\r
-\r
-# unit tests\r
-def makeRomanNumeral(n):\r
- def addDigit(n,limit,c,s):\r
- n -= limit\r
- s += c\r
- return n,s\r
-\r
- ret = ""\r
- while n >= 1000: n,ret = addDigit(n,1000,"M",ret)\r
- while n >= 900: n,ret = addDigit(n, 900,"CM",ret)\r
- while n >= 500: n,ret = addDigit(n, 500,"D",ret)\r
- while n >= 400: n,ret = addDigit(n, 400,"CD",ret)\r
- while n >= 100: n,ret = addDigit(n, 100,"C",ret)\r
- while n >= 90: n,ret = addDigit(n, 90,"XC",ret)\r
- while n >= 50: n,ret = addDigit(n, 50,"L",ret)\r
- while n >= 40: n,ret = addDigit(n, 40,"XL",ret)\r
- while n >= 10: n,ret = addDigit(n, 10,"X",ret)\r
- while n >= 9: n,ret = addDigit(n, 9,"IX",ret)\r
- while n >= 5: n,ret = addDigit(n, 5,"V",ret)\r
- while n >= 4: n,ret = addDigit(n, 4,"IV",ret)\r
- while n >= 1: n,ret = addDigit(n, 1,"I",ret)\r
- return ret\r
-tests = " ".join(makeRomanNumeral(i) for i in range(1,5000+1))\r
-\r
-roman_int_map = {}\r
-expected = 1\r
-for t,s,e in romanNumeral.scanString(tests):\r
- orig = tests[s:e]\r
- if t[0] != expected:\r
- print("{0} {1} {2}".format("==>", t, orig))\r
- roman_int_map[orig] = t[0]\r
- expected += 1\r
-\r
-def verify_value(s, tokens):\r
- expected = roman_int_map[s]\r
- if tokens[0] != expected:\r
- raise Exception("incorrect value for {0} ({1}), expected {2}".format(s, tokens[0], expected ))\r
-\r
-romanNumeral.runTests("""\\r
- XVI\r
- XXXIX\r
- XIV\r
- XIX\r
- MCMLXXX\r
- MMVI\r
- """, fullDump=False,\r
- postParse=verify_value)
\ No newline at end of file
+# romanNumerals.py
+#
+# Copyright (c) 2006, 2019, Paul McGuire
+#
+
+import pyparsing as pp
+
+
+def romanNumeralLiteral(numeralString, value):
+ return pp.Literal(numeralString).setParseAction(pp.replaceWith(value))
+
+
+one = romanNumeralLiteral("I", 1)
+four = romanNumeralLiteral("IV", 4)
+five = romanNumeralLiteral("V", 5)
+nine = romanNumeralLiteral("IX", 9)
+ten = romanNumeralLiteral("X", 10)
+forty = romanNumeralLiteral("XL", 40)
+fifty = romanNumeralLiteral("L", 50)
+ninety = romanNumeralLiteral("XC", 90)
+onehundred = romanNumeralLiteral("C", 100)
+fourhundred = romanNumeralLiteral("CD", 400)
+fivehundred = romanNumeralLiteral("D", 500)
+ninehundred = romanNumeralLiteral("CM", 900)
+onethousand = romanNumeralLiteral("M", 1000)
+
+numeral = (
+ onethousand
+ | ninehundred
+ | fivehundred
+ | fourhundred
+ | onehundred
+ | ninety
+ | fifty
+ | forty
+ | ten
+ | nine
+ | five
+ | four
+ | one
+).leaveWhitespace()
+
+romanNumeral = numeral[1, ...].setParseAction(sum)
+
+# unit tests
+def makeRomanNumeral(n):
+ def addDigits(n, limit, c, s):
+ while n >= limit:
+ n -= limit
+ s += c
+ return n, s
+
+ ret = ""
+ n, ret = addDigits(n, 1000, "M", ret)
+ n, ret = addDigits(n, 900, "CM", ret)
+ n, ret = addDigits(n, 500, "D", ret)
+ n, ret = addDigits(n, 400, "CD", ret)
+ n, ret = addDigits(n, 100, "C", ret)
+ n, ret = addDigits(n, 90, "XC", ret)
+ n, ret = addDigits(n, 50, "L", ret)
+ n, ret = addDigits(n, 40, "XL", ret)
+ n, ret = addDigits(n, 10, "X", ret)
+ n, ret = addDigits(n, 9, "IX", ret)
+ n, ret = addDigits(n, 5, "V", ret)
+ n, ret = addDigits(n, 4, "IV", ret)
+ n, ret = addDigits(n, 1, "I", ret)
+ return ret
+
+
+# make a string of all roman numerals from I to MMMMM
+tests = " ".join(makeRomanNumeral(i) for i in range(1, 5000 + 1))
+
+# parse each roman numeral, and populate map for validation below
+roman_int_map = {}
+for expected, (t, s, e) in enumerate(romanNumeral.scanString(tests), start=1):
+ orig = tests[s:e]
+ if t[0] != expected:
+ print("{} {} {}".format("==>", t, orig))
+ roman_int_map[orig] = t[0]
+
+
+def verify_value(s, tokens):
+ expected = roman_int_map[s]
+ if tokens[0] != expected:
+ raise Exception(
+ "incorrect value for {} ({}), expected {}".format(s, tokens[0], expected)
+ )
+
+
+romanNumeral.runTests(
+ """\
+ XVI
+ XXXIX
+ XIV
+ XIX
+ MCMLXXX
+ MMVI
+ MMMMM
+ """,
+ fullDump=False,
+ postParse=verify_value,
+)
#
BNF = """
stmt_list = {stmt} ;
-
+
stmt = ';'
| Identifier '=' expr ';'
| 'while' paren_expr stmt
| 'putc' paren_expr ';'
| '{' stmt_list '}'
;
-
+
paren_expr = '(' expr ')' ;
-
+
prt_list = string | expr {',' String | expr} ;
-
+
expr = and_expr {'||' and_expr} ;
and_expr = equality_expr {'&&' equality_expr} ;
equality_expr = relational_expr [('==' | '!=') relational_expr] ;
"""
import pyparsing as pp
+
pp.ParserElement.enablePackrat()
LBRACE, RBRACE, LPAR, RPAR, SEMI = map(pp.Suppress, "{}();")
-EQ = pp.Literal('=')
+EQ = pp.Literal("=")
-keywords = (WHILE, IF, PRINT, PUTC, ELSE) = map(pp.Keyword, "while if print putc else".split())
-identifier = ~(pp.MatchFirst(keywords)) + pp.pyparsing_common.identifier
+keywords = (WHILE, IF, PRINT, PUTC, ELSE) = map(
+ pp.Keyword, "while if print putc else".split()
+)
+any_keyword = pp.MatchFirst(keywords)
+identifier = ~any_keyword + pp.pyparsing_common.identifier
integer = pp.pyparsing_common.integer
-string = pp.QuotedString('"', convertWhitespaceEscapes=False).setName("quoted string")
+string = pp.QuotedString('"', convertWhitespaceEscapes=False).setName("quoted string")
char = pp.Regex(r"'\\?.'")
-expr = pp.infixNotation(identifier | integer | char,
- [
- (pp.oneOf("+ - !"), 1, pp.opAssoc.RIGHT,),
- (pp.oneOf("* / %"), 2, pp.opAssoc.LEFT, ),
- (pp.oneOf("+ -"), 2, pp.opAssoc.LEFT,),
- (pp.oneOf("< <= > >="), 2, pp.opAssoc.LEFT,),
- (pp.oneOf("== !="), 2, pp.opAssoc.LEFT,),
- (pp.oneOf("&&"), 2, pp.opAssoc.LEFT,),
- (pp.oneOf("||"), 2, pp.opAssoc.LEFT,),
- ])
+expr = pp.infixNotation(
+ identifier | integer | char,
+ [
+ (
+ pp.oneOf("+ - !"),
+ 1,
+ pp.opAssoc.RIGHT,
+ ),
+ (
+ pp.oneOf("* / %"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("+ -"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("< <= > >="),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("== !="),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("&&"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ (
+ pp.oneOf("||"),
+ 2,
+ pp.opAssoc.LEFT,
+ ),
+ ],
+)
prt_list = pp.Group(pp.delimitedList(string | expr))
paren_expr = pp.Group(LPAR + expr + RPAR)
if_stmt = pp.Group(IF - paren_expr + stmt + pp.Optional(ELSE + stmt))
print_stmt = pp.Group(PRINT - pp.Group(LPAR + prt_list + RPAR) + SEMI)
putc_stmt = pp.Group(PUTC - paren_expr + SEMI)
-stmt_list = pp.Group(LBRACE + pp.ZeroOrMore(stmt) + RBRACE)
-stmt <<= (pp.Group(SEMI)
- | assignment_stmt
- | while_stmt
- | if_stmt
- | print_stmt
- | putc_stmt
- | stmt_list
- ).setName("statement")
+stmt_list = pp.Group(LBRACE + stmt[...] + RBRACE)
+stmt <<= (
+ pp.Group(SEMI)
+ | assignment_stmt
+ | while_stmt
+ | if_stmt
+ | print_stmt
+ | putc_stmt
+ | stmt_list
+).setName("statement")
-code = pp.ZeroOrMore(stmt)
+code = stmt[...]
code.ignore(pp.cppStyleComment)
tests = [
- r'''
+ r"""
count = 1;
while (count < 10) {
print("count is: ", count, "\n");
count = count + 1;
}
- ''',
- r'''
+ """,
+ r"""
/*
Simple prime number generator
*/
}
}
print("Total primes found: ", count, "\n");
- ''',
- r'''
+ """,
+ r"""
/*
Hello world
*/
- print("Hello, World!\n");
- ''',
- r'''
+ print("Hello, World!\n");
+ """,
+ r"""
/*
Show Ident and Integers
*/
phoenix_number = 142857;
print(phoenix_number, "\n");
- ''',
- r'''
+ """,
+ r"""
/*** test printing, embedded \n and comments with lots of '*' ***/
print(42);
print("\nHello World\nGood Bye\nok\n");
print("Print a slash n - \\n.\n");
- ''',
- r'''
+ """,
+ r"""
/* 100 Doors */
i = 1;
while (i * i <= 100) {
print("door ", i * i, " is open\n");
i = i + 1;
}
- ''',
- r'''
+ """,
+ r"""
a = (-1 * ((-1 * (5 * 15)) / 10));
print(a, "\n");
b = -a;
print(b, "\n");
print(-b, "\n");
print(-(1), "\n");
- ''',
- r'''
+ """,
+ r"""
print(---------------------------------+++5, "\n");
print(((((((((3 + 2) * ((((((2))))))))))))), "\n");
-
+
if (1) { if (1) { if (1) { if (1) { if (1) { print(15, "\n"); } } } } }
- ''',
- r'''
+ """,
+ r"""
/* Compute the gcd of 1071, 1029: 21 */
-
+
a = 1071;
b = 1029;
-
+
while (b != 0) {
new_a = b;
b = a % b;
a = new_a;
}
print(a);
- ''',
- r'''
+ """,
+ r"""
/* 12 factorial is 479001600 */
-
+
n = 12;
result = 1;
i = 1;
i = i + 1;
}
print(result);
- ''',
- r'''
+ """,
+ r"""
/* fibonacci of 44 is 701408733 */
-
+
n = 44;
i = 1;
a = 0;
i = i + 1;
}
print(w, "\n");
- ''',
- r'''
+ """,
+ r"""
/* FizzBuzz */
i = 1;
while (i <= 100) {
print("Buzz");
else
print(i);
-
+
print("\n");
i = i + 1;
}
- ''',
- r'''
+ """,
+ r"""
/* 99 bottles */
bottles = 99;
while (bottles > 0) {
bottles = bottles - 1;
print(bottles, " bottles of beer on the wall\n\n");
}
- ''',
- r'''
+ """,
+ r"""
{
/*
This is an integer ascii Mandelbrot generator
bottom_edge = -300;
x_step = 7;
y_step = 15;
-
+
max_iter = 200;
-
+
y0 = top_edge;
while (y0 > bottom_edge) {
x0 = left_edge;
y0 = y0 - y_step;
}
}
- ''',
+ """,
]
import sys
+
sys.setrecursionlimit(2000)
for test in tests:
#\r
# Copyright (c) 2004, 2006 Paul McGuire\r
#\r
-from pyparsing import Word, alphas, alphanums, Literal, restOfLine, OneOrMore, \\r
- empty, Suppress, replaceWith\r
+from pyparsing import (\r
+ Word,\r
+ alphas,\r
+ alphanums,\r
+ Literal,\r
+ restOfLine,\r
+ OneOrMore,\r
+ empty,\r
+ Suppress,\r
+ replaceWith,\r
+)\r
\r
# simulate some C++ code\r
testData = """\r
print("----------------------")\r
\r
# simple grammar to match #define's\r
-ident = Word(alphas, alphanums+"_")\r
-macroDef = Literal("#define") + ident.setResultsName("name") + "=" + restOfLine.setResultsName("value")\r
-for t,s,e in macroDef.scanString( testData ):\r
- print(t.name,":", t.value)\r
+ident = Word(alphas, alphanums + "_")\r
+macroDef = (\r
+ Literal("#define")\r
+ + ident.setResultsName("name")\r
+ + "="\r
+ + restOfLine.setResultsName("value")\r
+)\r
+for t, s, e in macroDef.scanString(testData):\r
+ print(t.name, ":", t.value)\r
\r
# or a quick way to make a dictionary of the names and values\r
# (return only key and value tokens, and construct dict from key-value pairs)\r
print("----------------------")\r
\r
# convert C++ namespaces to mangled C-compatible names\r
-scopedIdent = ident + OneOrMore( Literal("::").suppress() + ident )\r
+scopedIdent = ident + OneOrMore(Literal("::").suppress() + ident)\r
scopedIdent.setParseAction(lambda t: "_".join(t))\r
\r
print("(replace namespace-scoped names with C-compatible names)")\r
-print(scopedIdent.transformString( testData ))\r
+print(scopedIdent.transformString(testData))\r
\r
\r
# or a crude pre-processor (use parse actions to replace matching text)\r
-def substituteMacro(s,l,t):\r
+def substituteMacro(s, l, t):\r
if t[0] in macros:\r
return macros[t[0]]\r
-ident.setParseAction( substituteMacro )\r
+\r
+\r
+ident.setParseAction(substituteMacro)\r
ident.ignore(macroDef)\r
\r
print("(simulate #define pre-processor)")\r
-print(ident.transformString( testData ))\r
-\r
+print(ident.transformString(testData))\r
\r
\r
#################\r
\r
# remove all string macro definitions (after extracting to a string resource table?)\r
stringMacroDef = Literal("#define") + ident + "=" + dblQuotedString + LineStart()\r
-stringMacroDef.setParseAction( replaceWith("") )\r
+stringMacroDef.setParseAction(replaceWith(""))\r
\r
-print(stringMacroDef.transformString( testData ))\r
+print(stringMacroDef.transformString(testData))\r
from searchparser import SearchQueryParser\r
\r
-products = [ "grape juice", "grape jelly", "orange juice", "orange jujubees",\r
- "strawberry jam", "prune juice", "prune butter", "orange marmalade",\r
- "grapefruit juice" ]\r
+products = [\r
+ "grape juice",\r
+ "grape jelly",\r
+ "orange juice",\r
+ "orange jujubees",\r
+ "strawberry jam",\r
+ "prune juice",\r
+ "prune butter",\r
+ "orange marmalade",\r
+ "grapefruit juice",\r
+]\r
+\r
\r
class FruitSearchParser(SearchQueryParser):\r
def GetWord(self, word):\r
- return { p for p in products if p.startswith(word + " ") }\r
+ return {p for p in products if p.startswith(word + " ")}\r
\r
def GetWordWildcard(self, word):\r
- return { p for p in products if p.startswith(word[:-1]) }\r
+ return {p for p in products if p.startswith(word[:-1])}\r
\r
def GetQuotes(self, search_string, tmp_result):\r
result = set()\r
return result\r
\r
def GetNot(self, not_set):\r
- return set( products ) - not_set\r
+ return set(products) - not_set\r
\r
\r
parser = FruitSearchParser()\r
for t in tests:\r
print(t.strip())\r
print(parser.Parse(t))\r
- print('')\r
+ print("")\r
- ask someone to check my English texts\r
- add more kinds of wildcards ('*' at the beginning and '*' inside a word)?\r
"""\r
-from pyparsing import Word, alphanums, Keyword, Group, Combine, Forward, Suppress, OneOrMore, oneOf\r
+from pyparsing import (\r
+ Word,\r
+ alphanums,\r
+ Keyword,\r
+ Group,\r
+ Combine,\r
+ Forward,\r
+ Suppress,\r
+ OneOrMore,\r
+ oneOf,\r
+)\r
\r
-class SearchQueryParser:\r
\r
+class SearchQueryParser:\r
def __init__(self):\r
self._methods = {\r
- 'and': self.evaluateAnd,\r
- 'or': self.evaluateOr,\r
- 'not': self.evaluateNot,\r
- 'parenthesis': self.evaluateParenthesis,\r
- 'quotes': self.evaluateQuotes,\r
- 'word': self.evaluateWord,\r
- 'wordwildcard': self.evaluateWordWildcard,\r
+ "and": self.evaluateAnd,\r
+ "or": self.evaluateOr,\r
+ "not": self.evaluateNot,\r
+ "parenthesis": self.evaluateParenthesis,\r
+ "quotes": self.evaluateQuotes,\r
+ "word": self.evaluateWord,\r
+ "wordwildcard": self.evaluateWordWildcard,\r
}\r
self._parser = self.parser()\r
\r
"""\r
operatorOr = Forward()\r
\r
- operatorWord = Group(Combine(Word(alphanums) + Suppress('*'))).setResultsName('wordwildcard') | \\r
- Group(Word(alphanums)).setResultsName('word')\r
+ operatorWord = Group(Combine(Word(alphanums) + Suppress("*"))).setResultsName(\r
+ "wordwildcard"\r
+ ) | Group(Word(alphanums)).setResultsName("word")\r
\r
operatorQuotesContent = Forward()\r
- operatorQuotesContent << (\r
- (operatorWord + operatorQuotesContent) | operatorWord\r
- )\r
+ operatorQuotesContent << ((operatorWord + operatorQuotesContent) | operatorWord)\r
\r
- operatorQuotes = Group(\r
- Suppress('"') + operatorQuotesContent + Suppress('"')\r
- ).setResultsName("quotes") | operatorWord\r
+ operatorQuotes = (\r
+ Group(Suppress('"') + operatorQuotesContent + Suppress('"')).setResultsName(\r
+ "quotes"\r
+ )\r
+ | operatorWord\r
+ )\r
\r
- operatorParenthesis = Group(\r
- (Suppress("(") + operatorOr + Suppress(")"))\r
- ).setResultsName("parenthesis") | operatorQuotes\r
+ operatorParenthesis = (\r
+ Group(Suppress("(") + operatorOr + Suppress(")")).setResultsName(\r
+ "parenthesis"\r
+ )\r
+ | operatorQuotes\r
+ )\r
\r
operatorNot = Forward()\r
- operatorNot << (Group(\r
- Suppress(Keyword("not", caseless=True)) + operatorNot\r
- ).setResultsName("not") | operatorParenthesis)\r
+ operatorNot << (\r
+ Group(Suppress(Keyword("not", caseless=True)) + operatorNot).setResultsName(\r
+ "not"\r
+ )\r
+ | operatorParenthesis\r
+ )\r
\r
operatorAnd = Forward()\r
- operatorAnd << (Group(\r
- operatorNot + Suppress(Keyword("and", caseless=True)) + operatorAnd\r
- ).setResultsName("and") | Group(\r
- operatorNot + OneOrMore(~oneOf("and or") + operatorAnd)\r
- ).setResultsName("and") | operatorNot)\r
+ operatorAnd << (\r
+ Group(\r
+ operatorNot + Suppress(Keyword("and", caseless=True)) + operatorAnd\r
+ ).setResultsName("and")\r
+ | Group(\r
+ operatorNot + OneOrMore(~oneOf("and or") + operatorAnd)\r
+ ).setResultsName("and")\r
+ | operatorNot\r
+ )\r
\r
- operatorOr << (Group(\r
- operatorAnd + Suppress(Keyword("or", caseless=True)) + operatorOr\r
- ).setResultsName("or") | operatorAnd)\r
+ operatorOr << (\r
+ Group(\r
+ operatorAnd + Suppress(Keyword("or", caseless=True)) + operatorOr\r
+ ).setResultsName("or")\r
+ | operatorAnd\r
+ )\r
\r
return operatorOr.parseString\r
\r
r = self.evaluate(item)\r
else:\r
r = r.intersection(self.evaluate(item))\r
- return self.GetQuotes(' '.join(search_terms), r)\r
+ return self.GetQuotes(" ".join(search_terms), r)\r
\r
def evaluateWord(self, argument):\r
return self.GetWord(argument[0])\r
return self._methods[argument.getName()](argument)\r
\r
def Parse(self, query):\r
- #print self._parser(query)[0]\r
+ # print self._parser(query)[0]\r
return self.evaluate(self._parser(query)[0])\r
\r
def GetWord(self, word):\r
\r
class ParserTest(SearchQueryParser):\r
"""Tests the parser with some search queries\r
- tests containts a dictionary with tests and expected results.\r
+ tests contains a dictionary with tests and expected results.\r
"""\r
+\r
tests = {\r
- 'help': {1, 2, 4, 5},\r
- 'help or hulp': {1, 2, 3, 4, 5},\r
- 'help and hulp': {2},\r
- 'help hulp': {2},\r
- 'help and hulp or hilp': {2, 3, 4},\r
- 'help or hulp and hilp': {1, 2, 3, 4, 5},\r
- 'help or hulp or hilp or halp': {1, 2, 3, 4, 5, 6},\r
- '(help or hulp) and (hilp or halp)': {3, 4, 5},\r
- 'help and (hilp or halp)': {4, 5},\r
- '(help and (hilp or halp)) or hulp': {2, 3, 4, 5},\r
- 'not help': {3, 6, 7, 8},\r
- 'not hulp and halp': {5, 6},\r
- 'not (help and halp)': {1, 2, 3, 4, 6, 7, 8},\r
+ "help": {1, 2, 4, 5},\r
+ "help or hulp": {1, 2, 3, 4, 5},\r
+ "help and hulp": {2},\r
+ "help hulp": {2},\r
+ "help and hulp or hilp": {2, 3, 4},\r
+ "help or hulp and hilp": {1, 2, 3, 4, 5},\r
+ "help or hulp or hilp or halp": {1, 2, 3, 4, 5, 6},\r
+ "(help or hulp) and (hilp or halp)": {3, 4, 5},\r
+ "help and (hilp or halp)": {4, 5},\r
+ "(help and (hilp or halp)) or hulp": {2, 3, 4, 5},\r
+ "not help": {3, 6, 7, 8},\r
+ "not hulp and halp": {5, 6},\r
+ "not (help and halp)": {1, 2, 3, 4, 6, 7, 8},\r
'"help me please"': {2},\r
'"help me please" or hulp': {2, 3},\r
'"help me please" or (hulp and halp)': {2},\r
- 'help*': {1, 2, 4, 5, 8},\r
- 'help or hulp*': {1, 2, 3, 4, 5},\r
- 'help* and hulp': {2},\r
- 'help and hulp* or hilp': {2, 3, 4},\r
- 'help* or hulp or hilp or halp': {1, 2, 3, 4, 5, 6, 8},\r
- '(help or hulp*) and (hilp* or halp)': {3, 4, 5},\r
- 'help* and (hilp* or halp*)': {4, 5},\r
- '(help and (hilp* or halp)) or hulp*': {2, 3, 4, 5},\r
- 'not help* and halp': {6},\r
- 'not (help* and helpe*)': {1, 2, 3, 4, 5, 6, 7},\r
+ "help*": {1, 2, 4, 5, 8},\r
+ "help or hulp*": {1, 2, 3, 4, 5},\r
+ "help* and hulp": {2},\r
+ "help and hulp* or hilp": {2, 3, 4},\r
+ "help* or hulp or hilp or halp": {1, 2, 3, 4, 5, 6, 8},\r
+ "(help or hulp*) and (hilp* or halp)": {3, 4, 5},\r
+ "help* and (hilp* or halp*)": {4, 5},\r
+ "(help and (hilp* or halp)) or hulp*": {2, 3, 4, 5},\r
+ "not help* and halp": {6},\r
+ "not (help* and helpe*)": {1, 2, 3, 4, 5, 6, 7},\r
'"help* me please"': {2},\r
'"help* me* please" or hulp*': {2, 3},\r
'"help me please*" or (hulp and halp)': {2},\r
'"help me please" not (hulp and halp)': {2},\r
'"help me please" hulp': {2},\r
- 'help and hilp and not holp': {4},\r
- 'help hilp not holp': {4},\r
- 'help hilp and not holp': {4},\r
+ "help and hilp and not holp": {4},\r
+ "help hilp not holp": {4},\r
+ "help hilp and not holp": {4},\r
}\r
\r
docs = {\r
- 1: 'help',\r
- 2: 'help me please hulp',\r
- 3: 'hulp hilp',\r
- 4: 'help hilp',\r
- 5: 'halp thinks he needs help',\r
- 6: 'he needs halp',\r
- 7: 'nothing',\r
- 8: 'helper',\r
+ 1: "help",\r
+ 2: "help me please hulp",\r
+ 3: "hulp hilp",\r
+ 4: "help hilp",\r
+ 5: "halp thinks he needs help",\r
+ 6: "he needs halp",\r
+ 7: "nothing",\r
+ 8: "helper",\r
}\r
\r
index = {\r
- 'help': {1, 2, 4, 5},\r
- 'me': {2},\r
- 'please': {2},\r
- 'hulp': {2, 3},\r
- 'hilp': {3, 4},\r
- 'halp': {5, 6},\r
- 'thinks': {5},\r
- 'he': {5, 6},\r
- 'needs': {5, 6},\r
- 'nothing': {7},\r
- 'helper': {8},\r
+ "help": {1, 2, 4, 5},\r
+ "me": {2},\r
+ "please": {2},\r
+ "hulp": {2, 3},\r
+ "hilp": {3, 4},\r
+ "halp": {5, 6},\r
+ "thinks": {5},\r
+ "he": {5, 6},\r
+ "needs": {5, 6},\r
+ "nothing": {7},\r
+ "helper": {8},\r
}\r
\r
def GetWord(self, word):\r
- if (word in self.index):\r
+ if word in self.index:\r
return self.index[word]\r
else:\r
return set()\r
def GetWordWildcard(self, word):\r
result = set()\r
for item in list(self.index.keys()):\r
- if word == item[0:len(word)]:\r
+ if word == item[0 : len(word)]:\r
result = result.union(self.index[item])\r
return result\r
\r
print(item)\r
r = self.Parse(item)\r
e = self.tests[item]\r
- print('Result: %s' % r)\r
- print('Expect: %s' % e)\r
+ print("Result: %s" % r)\r
+ print("Expect: %s" % e)\r
if e == r:\r
- print('Test OK')\r
+ print("Test OK")\r
else:\r
all_ok = False\r
- print('>>>>>>>>>>>>>>>>>>>>>>Test ERROR<<<<<<<<<<<<<<<<<<<<<')\r
- print('')\r
+ print(">>>>>>>>>>>>>>>>>>>>>>Test ERROR<<<<<<<<<<<<<<<<<<<<<")\r
+ print("")\r
return all_ok\r
\r
-if __name__=='__main__':\r
+\r
+if __name__ == "__main__":\r
if ParserTest().Test():\r
- print('All tests OK')\r
+ print("All tests OK")\r
else:\r
- print('One or more tests FAILED')\r
+ print("One or more tests FAILED")\r
# a simple SELECT statement parser, taken from SQLite's SELECT statement
# definition at https://www.sqlite.org/lang_select.html
#
+import sys
from pyparsing import *
+
ParserElement.enablePackrat()
-LPAR,RPAR,COMMA = map(Suppress,"(),")
-DOT,STAR = map(Literal, ".*")
+LPAR, RPAR, COMMA = map(Suppress, "(),")
+DOT, STAR = map(Literal, ".*")
select_stmt = Forward().setName("select statement")
# keywords
-(UNION, ALL, AND, INTERSECT, EXCEPT, COLLATE, ASC, DESC, ON, USING, NATURAL, INNER,
- CROSS, LEFT, OUTER, JOIN, AS, INDEXED, NOT, SELECT, DISTINCT, FROM, WHERE, GROUP, BY,
- HAVING, ORDER, BY, LIMIT, OFFSET, OR) = map(CaselessKeyword, """UNION, ALL, AND, INTERSECT,
- EXCEPT, COLLATE, ASC, DESC, ON, USING, NATURAL, INNER, CROSS, LEFT, OUTER, JOIN, AS, INDEXED, NOT, SELECT,
- DISTINCT, FROM, WHERE, GROUP, BY, HAVING, ORDER, BY, LIMIT, OFFSET, OR""".replace(",","").split())
-(CAST, ISNULL, NOTNULL, NULL, IS, BETWEEN, ELSE, END, CASE, WHEN, THEN, EXISTS, IN, LIKE, GLOB, REGEXP,
- MATCH, ESCAPE, CURRENT_TIME, CURRENT_DATE, CURRENT_TIMESTAMP) = map(CaselessKeyword,
- """CAST, ISNULL, NOTNULL, NULL, IS, BETWEEN, ELSE, END, CASE, WHEN, THEN, EXISTS, IN, LIKE, GLOB,
- REGEXP, MATCH, ESCAPE, CURRENT_TIME, CURRENT_DATE, CURRENT_TIMESTAMP""".replace(",","").split())
-keyword = MatchFirst((UNION, ALL, INTERSECT, EXCEPT, COLLATE, ASC, DESC, ON, USING, NATURAL, INNER,
- CROSS, LEFT, OUTER, JOIN, AS, INDEXED, NOT, SELECT, DISTINCT, FROM, WHERE, GROUP, BY,
- HAVING, ORDER, BY, LIMIT, OFFSET, CAST, ISNULL, NOTNULL, NULL, IS, BETWEEN, ELSE, END,
- CASE, WHEN, THEN, EXISTS, COLLATE, IN, LIKE, GLOB, REGEXP, MATCH, ESCAPE,
- CURRENT_TIME, CURRENT_DATE, CURRENT_TIMESTAMP))
-
-identifier = ~keyword + Word(alphas, alphanums+"_")
+keywords = {
+ k: CaselessKeyword(k)
+ for k in """\
+ UNION ALL AND INTERSECT EXCEPT COLLATE ASC DESC ON USING NATURAL INNER CROSS LEFT OUTER JOIN AS INDEXED NOT
+ SELECT DISTINCT FROM WHERE GROUP BY HAVING ORDER LIMIT OFFSET OR CAST ISNULL NOTNULL NULL IS BETWEEN ELSE END
+ CASE WHEN THEN EXISTS IN LIKE GLOB REGEXP MATCH ESCAPE CURRENT_TIME CURRENT_DATE CURRENT_TIMESTAMP TRUE FALSE
+ """.split()
+}
+vars().update(keywords)
+
+any_keyword = MatchFirst(keywords.values())
+
+quoted_identifier = QuotedString('"', escQuote='""')
+identifier = (~any_keyword + Word(alphas, alphanums + "_")).setParseAction(
+ pyparsing_common.downcaseTokens
+) | quoted_identifier
collation_name = identifier.copy()
column_name = identifier.copy()
column_alias = identifier.copy()
parameter_name = identifier.copy()
database_name = identifier.copy()
+comment = "--" + restOfLine
+
# expression
expr = Forward().setName("expression")
-integer = Regex(r"[+-]?\d+")
-numeric_literal = Regex(r"\d+(\.\d*)?([eE][+-]?\d+)?")
-string_literal = QuotedString("'")
+numeric_literal = pyparsing_common.number
+string_literal = QuotedString("'", escQuote="''")
blob_literal = Regex(r"[xX]'[0-9A-Fa-f]+'")
literal_value = (
- numeric_literal
- | string_literal
- | blob_literal
- | NULL
- | CURRENT_TIME
- | CURRENT_DATE
- | CURRENT_TIMESTAMP
- )
-bind_parameter = (
- Word("?",nums)
- | Combine(oneOf(": @ $") + parameter_name)
- )
+ numeric_literal
+ | string_literal
+ | blob_literal
+ | TRUE
+ | FALSE
+ | NULL
+ | CURRENT_TIME
+ | CURRENT_DATE
+ | CURRENT_TIMESTAMP
+)
+bind_parameter = Word("?", nums) | Combine(oneOf(": @ $") + parameter_name)
type_name = oneOf("TEXT REAL INTEGER BLOB NULL")
expr_term = (
CAST + LPAR + expr + AS + type_name + RPAR
| EXISTS + LPAR + select_stmt + RPAR
- | function_name.setName("function_name") + LPAR + Optional(STAR | delimitedList(expr)) + RPAR
+ | function_name.setName("function_name")
+ + LPAR
+ + Optional(STAR | delimitedList(expr))
+ + RPAR
| literal_value
| bind_parameter
- | Group(identifier('col_db') + DOT + identifier('col_tab') + DOT + identifier('col'))
- | Group(identifier('col_tab') + DOT + identifier('col'))
- | Group(identifier('col'))
+ | Group(
+ identifier("col_db") + DOT + identifier("col_tab") + DOT + identifier("col")
)
-
-UNARY,BINARY,TERNARY=1,2,3
-expr << infixNotation(expr_term,
+ | Group(identifier("col_tab") + DOT + identifier("col"))
+ | Group(identifier("col"))
+)
+
+NOT_NULL = Group(NOT + NULL)
+NOT_BETWEEN = Group(NOT + BETWEEN)
+NOT_IN = Group(NOT + IN)
+NOT_LIKE = Group(NOT + LIKE)
+NOT_MATCH = Group(NOT + MATCH)
+NOT_GLOB = Group(NOT + GLOB)
+NOT_REGEXP = Group(NOT + REGEXP)
+
+UNARY, BINARY, TERNARY = 1, 2, 3
+expr << infixNotation(
+ expr_term,
[
- (oneOf('- + ~') | NOT, UNARY, opAssoc.RIGHT),
- (ISNULL | NOTNULL | NOT + NULL, UNARY, opAssoc.LEFT),
- ('||', BINARY, opAssoc.LEFT),
- (oneOf('* / %'), BINARY, opAssoc.LEFT),
- (oneOf('+ -'), BINARY, opAssoc.LEFT),
- (oneOf('<< >> & |'), BINARY, opAssoc.LEFT),
- (oneOf('< <= > >='), BINARY, opAssoc.LEFT),
- (oneOf('= == != <>') | IS | IN | LIKE | GLOB | MATCH | REGEXP, BINARY, opAssoc.LEFT),
- ((BETWEEN,AND), TERNARY, opAssoc.LEFT),
- (IN + LPAR + Group(select_stmt | delimitedList(expr)) + RPAR, UNARY, opAssoc.LEFT),
- (AND, BINARY, opAssoc.LEFT),
- (OR, BINARY, opAssoc.LEFT),
- ])
-
-compound_operator = (UNION + Optional(ALL) | INTERSECT | EXCEPT)
-
-ordering_term = Group(expr('order_key')
- + Optional(COLLATE + collation_name('collate'))
- + Optional(ASC | DESC)('direction'))
-
-join_constraint = Group(Optional(ON + expr | USING + LPAR + Group(delimitedList(column_name)) + RPAR))
-
-join_op = COMMA | Group(Optional(NATURAL) + Optional(INNER | CROSS | LEFT + OUTER | LEFT | OUTER) + JOIN)
+ (oneOf("- + ~") | NOT, UNARY, opAssoc.RIGHT),
+ (ISNULL | NOTNULL | NOT_NULL, UNARY, opAssoc.LEFT),
+ ("||", BINARY, opAssoc.LEFT),
+ (oneOf("* / %"), BINARY, opAssoc.LEFT),
+ (oneOf("+ -"), BINARY, opAssoc.LEFT),
+ (oneOf("<< >> & |"), BINARY, opAssoc.LEFT),
+ (oneOf("< <= > >="), BINARY, opAssoc.LEFT),
+ (
+ oneOf("= == != <>")
+ | IS
+ | IN
+ | LIKE
+ | GLOB
+ | MATCH
+ | REGEXP
+ | NOT_IN
+ | NOT_LIKE
+ | NOT_GLOB
+ | NOT_MATCH
+ | NOT_REGEXP,
+ BINARY,
+ opAssoc.LEFT,
+ ),
+ ((BETWEEN | NOT_BETWEEN, AND), TERNARY, opAssoc.LEFT),
+ (
+ (IN | NOT_IN) + LPAR + Group(select_stmt | delimitedList(expr)) + RPAR,
+ UNARY,
+ opAssoc.LEFT,
+ ),
+ (AND, BINARY, opAssoc.LEFT),
+ (OR, BINARY, opAssoc.LEFT),
+ ],
+)
+
+compound_operator = UNION + Optional(ALL) | INTERSECT | EXCEPT
+
+ordering_term = Group(
+ expr("order_key")
+ + Optional(COLLATE + collation_name("collate"))
+ + Optional(ASC | DESC)("direction")
+)
+
+join_constraint = Group(
+ Optional(ON + expr | USING + LPAR + Group(delimitedList(column_name)) + RPAR)
+)
+
+join_op = COMMA | Group(
+ Optional(NATURAL) + Optional(INNER | CROSS | LEFT + OUTER | LEFT | OUTER) + JOIN
+)
join_source = Forward()
single_source = (
- Group(database_name("database") + DOT + table_name("table*")
- | table_name("table*"))
- + Optional(Optional(AS) + table_alias("table_alias*"))
- + Optional(INDEXED + BY + index_name("name") | NOT + INDEXED)("index")
- | (LPAR + select_stmt + RPAR + Optional(Optional(AS) + table_alias))
- | (LPAR + join_source + RPAR)
- )
-
-join_source <<= (Group(single_source + OneOrMore(join_op + single_source + join_constraint))
- | single_source)
+ Group(database_name("database") + DOT + table_name("table*") | table_name("table*"))
+ + Optional(Optional(AS) + table_alias("table_alias*"))
+ + Optional(INDEXED + BY + index_name("name") | NOT + INDEXED)("index")
+ | (LPAR + select_stmt + RPAR + Optional(Optional(AS) + table_alias))
+ | (LPAR + join_source + RPAR)
+)
+
+join_source <<= (
+ Group(single_source + OneOrMore(join_op + single_source + join_constraint))
+ | single_source
+)
# result_column = "*" | table_name + "." + "*" | Group(expr + Optional(Optional(AS) + column_alias))
-result_column = Group(STAR('col')
- | table_name('col_table') + DOT + STAR('col')
- | expr('col') + Optional(Optional(AS) + column_alias('alias'))
- )
-
-select_core = (SELECT + Optional(DISTINCT | ALL) + Group(delimitedList(result_column))("columns")
- + Optional(FROM + join_source("from*"))
- + Optional(WHERE + expr("where_expr"))
- + Optional(GROUP + BY + Group(delimitedList(ordering_term))("group_by_terms")
- + Optional(HAVING + expr("having_expr"))))
-
-select_stmt << (select_core + ZeroOrMore(compound_operator + select_core)
- + Optional(ORDER + BY + Group(delimitedList(ordering_term))("order_by_terms"))
- + Optional(LIMIT + (Group(expr + OFFSET + expr) | Group(expr + COMMA + expr) | expr)("limit"))
- )
-
-tests = """\
- select * from xyzzy where z > 100
- select * from xyzzy where z > 100 order by zz
- select * from xyzzy
- select z.* from xyzzy
- select a, b from test_table where 1=1 and b='yes'
- select a, b from test_table where 1=1 and b in (select bb from foo)
- select z.a, b from test_table where 1=1 and b in (select bb from foo)
- select z.a, b from test_table where 1=1 and b in (select bb from foo) order by b,c desc,d
- select z.a, b from test_table left join test2_table where 1=1 and b in (select bb from foo)
- select a, db.table.b as BBB from db.table where 1=1 and BBB='yes'
- select a, db.table.b as BBB from test_table,db.table where 1=1 and BBB='yes'
- select a, db.table.b as BBB from test_table,db.table where 1=1 and BBB='yes' limit 50
- select a, b from test_table where (1=1 or 2=3) and b='yes' group by zx having b=2 order by 1
- SELECT emp.ename as e FROM scott.employee as emp
- SELECT ename as e, fname as f FROM scott.employee as emp
- SELECT emp.eid, fname,lname FROM scott.employee as emp
- SELECT ename, lname, emp.eid FROM scott.employee as emp
- select emp.salary * (1.0 + emp.bonus) as salary_plus_bonus from scott.employee as emp
-"""
-select_stmt.runTests(tests)
+result_column = Group(
+ STAR("col")
+ | table_name("col_table") + DOT + STAR("col")
+ | expr("col") + Optional(Optional(AS) + column_alias("alias"))
+)
+
+select_core = (
+ SELECT
+ + Optional(DISTINCT | ALL)
+ + Group(delimitedList(result_column))("columns")
+ + Optional(FROM + join_source("from*"))
+ + Optional(WHERE + expr("where_expr"))
+ + Optional(
+ GROUP
+ + BY
+ + Group(delimitedList(ordering_term))("group_by_terms")
+ + Optional(HAVING + expr("having_expr"))
+ )
+)
+
+select_stmt << (
+ select_core
+ + ZeroOrMore(compound_operator + select_core)
+ + Optional(ORDER + BY + Group(delimitedList(ordering_term))("order_by_terms"))
+ + Optional(
+ LIMIT
+ + (Group(expr + OFFSET + expr) | Group(expr + COMMA + expr) | expr)("limit")
+ )
+)
+
+select_stmt.ignore(comment)
+
+
+def main():
+ tests = """\
+ select * from xyzzy where z > 100
+ select * from xyzzy where z > 100 order by zz
+ select * from xyzzy
+ select z.* from xyzzy
+ select a, b from test_table where 1=1 and b='yes'
+ select a, b from test_table where 1=1 and b in (select bb from foo)
+ select z.a, b from test_table where 1=1 and b in (select bb from foo)
+ select z.a, b from test_table where 1=1 and b in (select bb from foo) order by b,c desc,d
+ select z.a, b from test_table left join test2_table where 1=1 and b in (select bb from foo)
+ select a, db.table.b as BBB from db.table where 1=1 and BBB='yes'
+ select a, db.table.b as BBB from test_table,db.table where 1=1 and BBB='yes'
+ select a, db.table.b as BBB from test_table,db.table where 1=1 and BBB='yes' limit 50
+ select a, b from test_table where (1=1 or 2=3) and b='yes' group by zx having b=2 order by 1
+ SELECT emp.ename as e FROM scott.employee as emp
+ SELECT ename as e, fname as f FROM scott.employee as emp
+ SELECT emp.eid, fname,lname FROM scott.employee as emp
+ SELECT ename, lname, emp.eid FROM scott.employee as emp
+ select emp.salary * (1.0 + emp.bonus) as salary_plus_bonus from scott.employee as emp
+ SELECT * FROM abcd WHERE (ST_Overlaps("GEOM", 'POINT(0 0)'))
+ SELECT * FROM abcd WHERE CAST(foo AS REAL) > -999.123
+ SELECT * FROM abcd WHERE bar BETWEEN +180 AND +10E9
+ SELECT * FROM abcd WHERE CAST(foo AS REAL) < (4 + -9.876E-4)
+ SELECT SomeFunc(99)
+ SELECT * FROM abcd WHERE ST_X(ST_Centroid(geom)) BETWEEN (-180*2) AND (180*2)
+ SELECT * FROM abcd WHERE a
+ SELECT * FROM abcd WHERE snowy_things REGEXP '[⛄️☃️☃🎿🏂🌨❄️⛷🏔🗻❄︎❆❅]'
+ SELECT * FROM abcd WHERE a."b" IN 4
+ SELECT * FROM abcd WHERE a."b" In ('4')
+ SELECT * FROM "a".b AS "E" WHERE "E"."C" >= CURRENT_Time
+ SELECT * FROM abcd WHERE "dave" != "Dave" -- names & things ☃️
+ SELECT * FROM a WHERE a.dave is not null
+ SELECT * FROM abcd WHERE pete == FALSE or peter is true
+ SELECT * FROM abcd WHERE a >= 10 * (2 + 3)
+ SELECT * FROM abcd WHERE frank = 'is ''scary'''
+ SELECT * FROM abcd WHERE "identifier with ""quotes"" and a trailing space " IS NOT FALSE
+ SELECT * FROM abcd WHERE blobby == x'C0FFEE' -- hex
+ SELECT * FROM abcd WHERE ff NOT IN (1,2,4,5)
+ SELECT * FROM abcd WHERE ff not between 3 and 9
+ SELECT * FROM abcd WHERE ff not like 'bob%'
+ """
+
+ success, _ = select_stmt.runTests(tests)
+ print("\n{}".format("OK" if success else "FAIL"))
+ return 0 if success else 1
+
+
+if __name__ == "__main__":
+ main()
import pyparsing as pp
from base64 import b64decode
-import pprint
def verify_length(s, l, t):
if t.len is not None:
t1len = len(t[1])
if t1len != t.len:
- raise pp.ParseFatalException(s, l, "invalid data of length {0}, expected {1}".format(t1len, t.len))
+ raise pp.ParseFatalException(
+ s, l, "invalid data of length {}, expected {}".format(t1len, t.len)
+ )
return t[1]
# define punctuation literals
-LPAR, RPAR, LBRK, RBRK, LBRC, RBRC, VBAR, COLON = (pp.Suppress(c).setName(c) for c in "()[]{}|:")
+LPAR, RPAR, LBRK, RBRK, LBRC, RBRC, VBAR, COLON = (
+ pp.Suppress(c).setName(c) for c in "()[]{}|:"
+)
-decimal = pp.Regex(r'-?0|[1-9]\d*').setParseAction(lambda t: int(t[0]))
-hexadecimal = ("#" + pp.Word(pp.hexnums)[1, ...] + "#").setParseAction(lambda t: int("".join(t[1:-1]), 16))
+decimal = pp.Regex(r"-?0|[1-9]\d*").setParseAction(lambda t: int(t[0]))
+hexadecimal = ("#" + pp.Word(pp.hexnums)[1, ...] + "#").setParseAction(
+ lambda t: int("".join(t[1:-1]), 16)
+)
bytes = pp.Word(pp.printables)
raw = pp.Group(decimal("len") + COLON + bytes).setParseAction(verify_length)
-base64_ = pp.Group(pp.Optional(decimal | hexadecimal, default=None)("len")
- + VBAR
- + pp.Word(pp.alphanums + "+/=")[1, ...].setParseAction(lambda t: b64decode("".join(t)))
- + VBAR
- ).setParseAction(verify_length)
+base64_ = pp.Group(
+ pp.Optional(decimal | hexadecimal, default=None)("len")
+ + VBAR
+ + pp.Word(pp.alphanums + "+/=")[1, ...].setParseAction(
+ lambda t: b64decode("".join(t))
+ )
+ + VBAR
+).setParseAction(verify_length)
-real = pp.Regex(r"[+-]?\d+\.\d*([eE][+-]?\d+)?").setParseAction(lambda tokens: float(tokens[0]))
+real = pp.Regex(r"[+-]?\d+\.\d*([eE][+-]?\d+)?").setParseAction(
+ lambda tokens: float(tokens[0])
+)
token = pp.Word(pp.alphanums + "-./_:*+=!<>")
-qString = pp.Group(pp.Optional(decimal, default=None)("len")
- + pp.dblQuotedString.setParseAction(pp.removeQuotes)
- ).setParseAction(verify_length)
+qString = pp.Group(
+ pp.Optional(decimal, default=None)("len")
+ + pp.dblQuotedString.setParseAction(pp.removeQuotes)
+).setParseAction(verify_length)
simpleString = real | base64_ | raw | decimal | token | hexadecimal | qString
sexp <<= string_ | sexpList
-# Test data
-
-test00 = """(snicker "abc" (#03# |YWJj|))"""
-test01 = """(certificate
- (issuer
- (name
- (public-key
- rsa-with-md5
- (e 15 |NFGq/E3wh9f4rJIQVXhS|)
- (n |d738/4ghP9rFZ0gAIYZ5q9y6iskDJwASi5rEQpEQq8ZyMZeIZzIAR2I5iGE=|))
- aid-committee))
- (subject
- (ref
- (public-key
- rsa-with-md5
- (e |NFGq/E3wh9f4rJIQVXhS|)
- (n |d738/4ghP9rFZ0gAIYZ5q9y6iskDJwASi5rEQpEQq8ZyMZeIZzIAR2I5iGE=|))
- tom
- mother))
- (not-before "1997-01-01_09:00:00")
- (not-after "1998-01-01_09:00:00")
- (tag
- (spend (account "12345678") (* numeric range "1" "1000"))))
-"""
-test02 = """(lambda (x) (* x x))"""
-test03 = """(def length
- (lambda (x)
- (cond
- ((not x) 0)
- ( t (+ 1 (length (cdr x))))
- )
- )
-)
-"""
-test04 = """(2:XX "abc" (#03# |YWJj|))"""
-test05 = """(if (is (window_name) "XMMS") (set_workspace 2))"""
-test06 = """(if
- (and
- (is (application_name) "Firefox")
- (or
- (contains (window_name) "Enter name of file to save to")
- (contains (window_name) "Save As")
- (contains (window_name) "Save Image")
- ()
+def main():
+ # Test data
+
+ test00 = """(snicker "abc" (#03# |YWJj|))"""
+ test01 = """(certificate
+ (issuer
+ (name
+ (public-key
+ rsa-with-md5
+ (e 15 |NFGq/E3wh9f4rJIQVXhS|)
+ (n |d738/4ghP9rFZ0gAIYZ5q9y6iskDJwASi5rEQpEQq8ZyMZeIZzIAR2I5iGE=|))
+ aid-committee))
+ (subject
+ (ref
+ (public-key
+ rsa-with-md5
+ (e |NFGq/E3wh9f4rJIQVXhS|)
+ (n |d738/4ghP9rFZ0gAIYZ5q9y6iskDJwASi5rEQpEQq8ZyMZeIZzIAR2I5iGE=|))
+ tom
+ mother))
+ (not-before "1997-01-01_09:00:00")
+ (not-after "1998-01-01_09:00:00")
+ (tag
+ (spend (account "12345678") (* numeric range "1" "1000"))))
+ """
+ test02 = """(lambda (x) (* x x))"""
+ test03 = """(def length
+ (lambda (x)
+ (cond
+ ((not x) 0)
+ ( t (+ 1 (length (cdr x))))
+ )
+ )
)
- )
- (geometry "+140+122")
-)
-"""
-test07 = """(defun factorial (x)
- (if (zerop x) 1
- (* x (factorial (- x 1)))))
- """
-test51 = """(2:XX "abc" (#03# |YWJj|))"""
-test51error = """(3:XX "abc" (#03# |YWJj|))"""
-
-test52 = """
- (and
- (or (> uid 1000)
- (!= gid 20)
+ """
+ test04 = """(2:XX "abc" (#03# |YWJj|))"""
+ test05 = """(if (is (window_name) "XMMS") (set_workspace 2))"""
+ test06 = """(if
+ (and
+ (is (application_name) "Firefox")
+ (or
+ (contains (window_name) "Enter name of file to save to")
+ (contains (window_name) "Save As")
+ (contains (window_name) "Save Image")
+ ()
+ )
)
- (> quota 5.0e+03)
+ (geometry "+140+122")
)
"""
-
-# Run tests
-alltests = [globals()[testname] for testname in sorted(locals()) if testname.startswith("test")]
-
-sexp.runTests(alltests, fullDump=False)
+ test07 = """(defun factorial (x)
+ (if (zerop x) 1
+ (* x (factorial (- x 1)))))
+ """
+ test51 = """(2:XX "abc" (#03# |YWJj|))"""
+ test51error = """(3:XX "abc" (#03# |YWJj|))"""
+
+ test52 = """
+ (and
+ (or (> uid 1000)
+ (!= gid 20)
+ )
+ (> quota 5.0e+03)
+ )
+ """
+
+ # Run tests
+ local_vars = sorted(locals().items())
+ alltests = [
+ test_fn for testname, test_fn in local_vars if testname.startswith("test")
+ ]
+
+ sexp.runTests(alltests, fullDump=False)
+
+
+if __name__ == "__main__":
+ main()
-# shapes.py\r
-#\r
-# A sample program showing how parse actions can convert parsed\r
-# strings into a data type or object.\r
-#\r
-# Copyright 2012, Paul T. McGuire\r
-#\r
-\r
-# define class hierarchy of Shape classes, with polymorphic area method\r
-class Shape(object):\r
- def __init__(self, tokens):\r
- self.__dict__.update(tokens.asDict())\r
-\r
- def area(self):\r
- raise NotImplementedException()\r
-\r
- def __str__(self):\r
- return "<{0}>: {1}".format(self.__class__.__name__, self.__dict__)\r
-\r
-class Square(Shape):\r
- def area(self):\r
- return self.side**2\r
-\r
-class Rectangle(Shape):\r
- def area(self):\r
- return self.width * self.height\r
-\r
-class Circle(Shape):\r
- def area(self):\r
- return 3.14159 * self.radius**2\r
-\r
-\r
-from pyparsing import *\r
-\r
-number = Regex(r'-?\d+(\.\d*)?').setParseAction(lambda t:float(t[0]))\r
-\r
-# Shape expressions:\r
-# square : S <centerx> <centery> <side>\r
-# rectangle: R <centerx> <centery> <width> <height>\r
-# circle : C <centerx> <centery> <diameter>\r
-\r
-squareDefn = "S" + number('centerx') + number('centery') + number('side')\r
-rectDefn = "R" + number('centerx') + number('centery') + number('width') + number('height')\r
-circleDefn = "C" + number('centerx') + number('centery') + number('diameter')\r
-\r
-squareDefn.setParseAction(Square)\r
-rectDefn.setParseAction(Rectangle)\r
-\r
-def computeRadius(tokens):\r
- tokens['radius'] = tokens.diameter/2.0\r
-circleDefn.setParseAction(computeRadius, Circle)\r
-\r
-shapeExpr = squareDefn | rectDefn | circleDefn\r
-\r
-tests = """\\r
-C 0 0 100\r
-R 10 10 20 50\r
-S -1 5 10""".splitlines()\r
-\r
-for t in tests:\r
- shape = shapeExpr.parseString(t)[0]\r
- print(shape)\r
- print("Area:", shape.area())\r
- print()\r
+# shapes.py
+#
+# A sample program showing how parse actions can convert parsed
+# strings into a data type or object.
+#
+# Copyright 2012, 2019 Paul T. McGuire
+#
+
+# define class hierarchy of Shape classes, with polymorphic area method
+class Shape:
+ def __init__(self, tokens):
+ self.__dict__.update(tokens.asDict())
+
+ def area(self):
+ raise NotImplemented()
+
+ def __str__(self):
+ return "<{}>: {}".format(self.__class__.__name__, vars(self))
+
+
+class Square(Shape):
+ def area(self):
+ return self.side ** 2
+
+
+class Rectangle(Shape):
+ def area(self):
+ return self.width * self.height
+
+
+class Circle(Shape):
+ def area(self):
+ return 3.14159 * self.radius ** 2
+
+
+import pyparsing as pp
+
+ppc = pp.pyparsing_common
+
+# use pyparsing-defined numeric expression that converts all parsed
+# numeric values as floats
+number = ppc.fnumber()
+
+# Shape expressions:
+# square : S <centerx> <centery> <side>
+# rectangle: R <centerx> <centery> <width> <height>
+# circle : C <centerx> <centery> <diameter>
+
+squareDefn = "S" + number("centerx") + number("centery") + number("side")
+rectDefn = (
+ "R" + number("centerx") + number("centery") + number("width") + number("height")
+)
+circleDefn = "C" + number("centerx") + number("centery") + number("diameter")
+
+squareDefn.setParseAction(Square)
+rectDefn.setParseAction(Rectangle)
+
+
+def computeRadius(tokens):
+ tokens["radius"] = tokens.diameter / 2.0
+
+
+circleDefn.setParseAction(computeRadius, Circle)
+
+shapeExpr = squareDefn | rectDefn | circleDefn
+
+tests = """\
+C 0 0 100
+R 10 10 20 50
+S -1 5 10""".splitlines()
+
+for t in tests:
+ shape = shapeExpr.parseString(t)[0]
+ print(shape)
+ print("Area:", shape.area())
+ print()
#\r
# Copyright 2006, by Paul McGuire\r
#\r
-\r
+import sys\r
from pyparsing import *\r
\r
-integer = Word(nums).setParseAction(lambda t:int(t[0]))\r
-variable = Word(alphas,exact=1)\r
+ppc = pyparsing_common\r
+\r
+ParserElement.enablePackrat()\r
+sys.setrecursionlimit(3000)\r
+\r
+integer = ppc.integer\r
+variable = Word(alphas, exact=1)\r
operand = integer | variable\r
\r
-expop = Literal('^')\r
-signop = oneOf('+ -')\r
-multop = oneOf('* /')\r
-plusop = oneOf('+ -')\r
-factop = Literal('!')\r
+expop = Literal("^")\r
+signop = oneOf("+ -")\r
+multop = oneOf("* /")\r
+plusop = oneOf("+ -")\r
+factop = Literal("!")\r
\r
# To use the infixNotation helper:\r
# 1. Define the "atom" operand term of the grammar.\r
# and integer or a variable. This will be the first argument\r
# to the infixNotation method.\r
# 2. Define a list of tuples for each level of operator\r
-# precendence. Each tuple is of the form\r
+# precedence. Each tuple is of the form\r
# (opExpr, numTerms, rightLeftAssoc, parseAction), where\r
# - opExpr is the pyparsing expression for the operator;\r
# may also be a string, which will be converted to a Literal\r
# this expression to parse input strings, or incorporate it\r
# into a larger, more complex grammar.\r
#\r
-expr = infixNotation( operand,\r
- [("!", 1, opAssoc.LEFT),\r
- ("^", 2, opAssoc.RIGHT),\r
- (signop, 1, opAssoc.RIGHT),\r
- (multop, 2, opAssoc.LEFT),\r
- (plusop, 2, opAssoc.LEFT),]\r
- )\r
+expr = infixNotation(\r
+ operand,\r
+ [\r
+ ("!", 1, opAssoc.LEFT),\r
+ ("^", 2, opAssoc.RIGHT),\r
+ (signop, 1, opAssoc.RIGHT),\r
+ (multop, 2, opAssoc.LEFT),\r
+ (plusop, 2, opAssoc.LEFT),\r
+ ],\r
+)\r
\r
-test = ["9 + 2 + 3",\r
- "9 + 2 * 3",\r
- "(9 + 2) * 3",\r
- "(9 + -2) * 3",\r
- "(9 + -2) * 3^2^2",\r
- "(9! + -2) * 3^2^2",\r
- "M*X + B",\r
- "M*(X + B)",\r
- "1+2*-3^4*5+-+-6",]\r
+test = [\r
+ "9 + 2 + 3",\r
+ "9 + 2 * 3",\r
+ "(9 + 2) * 3",\r
+ "(9 + -2) * 3",\r
+ "(9 + -2) * 3^2^2",\r
+ "(9! + -2) * 3^2^2",\r
+ "M*X + B",\r
+ "M*(X + B)",\r
+ "1+2*-3^4*5+-+-6",\r
+ "(a + b)",\r
+ "((a + b))",\r
+ "(((a + b)))",\r
+ "((((a + b))))",\r
+ "((((((((((((((a + b))))))))))))))",\r
+]\r
for t in test:\r
print(t)\r
print(expr.parseString(t))\r
- print('')\r
+ print("")\r
#\r
# Copyright 2006, by Paul McGuire\r
# Updated 2013-Sep-14 - improved Python 2/3 cross-compatibility\r
+# Updated 2021-Sep-27 - removed Py2 compat; added type annotations\r
#\r
-from pyparsing import infixNotation, opAssoc, Keyword, Word, alphas\r
+from typing import Callable, Iterable\r
+\r
+from pyparsing import infixNotation, opAssoc, Keyword, Word, alphas, ParserElement\r
+\r
+ParserElement.enablePackrat()\r
+\r
\r
# define classes to be built at parse time, as each matching\r
# expression type is parsed\r
-class BoolOperand(object):\r
- def __init__(self,t):\r
+class BoolOperand:\r
+ def __init__(self, t):\r
self.label = t[0]\r
self.value = eval(t[0])\r
- def __bool__(self):\r
+\r
+ def __bool__(self) -> bool:\r
return self.value\r
- def __str__(self):\r
+\r
+ def __str__(self) -> str:\r
return self.label\r
- __repr__ = __str__\r
- __nonzero__ = __bool__\r
\r
-class BoolBinOp(object):\r
- def __init__(self,t):\r
- self.args = t[0][0::2]\r
- def __str__(self):\r
- sep = " %s " % self.reprsymbol\r
- return "(" + sep.join(map(str,self.args)) + ")"\r
- def __bool__(self):\r
- return self.evalop(bool(a) for a in self.args)\r
- __nonzero__ = __bool__\r
__repr__ = __str__\r
\r
-class BoolAnd(BoolBinOp):\r
- reprsymbol = '&'\r
- evalop = all\r
-\r
-class BoolOr(BoolBinOp):\r
- reprsymbol = '|'\r
- evalop = any\r
\r
-class BoolNot(object):\r
- def __init__(self,t):\r
+class BoolNot:\r
+ def __init__(self, t):\r
self.arg = t[0][1]\r
- def __bool__(self):\r
+\r
+ def __bool__(self) -> bool:\r
v = bool(self.arg)\r
return not v\r
- def __str__(self):\r
+\r
+ def __str__(self) -> str:\r
return "~" + str(self.arg)\r
+\r
__repr__ = __str__\r
- __nonzero__ = __bool__\r
\r
+\r
+class BoolBinOp:\r
+ repr_symbol: str = ""\r
+ eval_fn: Callable[\r
+ [Iterable[bool]], bool\r
+ ] = lambda _: False\r
+\r
+ def __init__(self, t):\r
+ self.args = t[0][0::2]\r
+\r
+ def __str__(self) -> str:\r
+ sep = " %s " % self.repr_symbol\r
+ return "(" + sep.join(map(str, self.args)) + ")"\r
+\r
+ def __bool__(self) -> bool:\r
+ return self.eval_fn(bool(a) for a in self.args)\r
+\r
+\r
+class BoolAnd(BoolBinOp):\r
+ repr_symbol = "&"\r
+ eval_fn = all\r
+\r
+\r
+class BoolOr(BoolBinOp):\r
+ repr_symbol = "|"\r
+ eval_fn = any\r
+\r
+\r
+# define keywords and simple infix notation grammar for boolean\r
+# expressions\r
TRUE = Keyword("True")\r
FALSE = Keyword("False")\r
-boolOperand = TRUE | FALSE | Word(alphas,max=1)\r
-boolOperand.setParseAction(BoolOperand)\r
+NOT = Keyword("not")\r
+AND = Keyword("and")\r
+OR = Keyword("or")\r
+boolOperand = TRUE | FALSE | Word(alphas, max=1)\r
+boolOperand.setParseAction(BoolOperand).setName("bool_operand")\r
\r
# define expression, based on expression operand and\r
# list of operations in precedence order\r
-boolExpr = infixNotation( boolOperand,\r
+boolExpr = infixNotation(\r
+ boolOperand,\r
[\r
- ("not", 1, opAssoc.RIGHT, BoolNot),\r
- ("and", 2, opAssoc.LEFT, BoolAnd),\r
- ("or", 2, opAssoc.LEFT, BoolOr),\r
- ])\r
+ (NOT, 1, opAssoc.RIGHT, BoolNot),\r
+ (AND, 2, opAssoc.LEFT, BoolAnd),\r
+ (OR, 2, opAssoc.LEFT, BoolOr),\r
+ ],\r
+).setName("boolean_expression")\r
\r
\r
if __name__ == "__main__":\r
p = True\r
q = False\r
r = True\r
- tests = [("p", True),\r
- ("q", False),\r
- ("p and q", False),\r
- ("p and not q", True),\r
- ("not not p", True),\r
- ("not(p and q)", True),\r
- ("q or not p and r", False),\r
- ("q or not p or not r", False),\r
- ("q or not (p and r)", False),\r
- ("p or q or r", True),\r
- ("p or q or r and False", True),\r
- ("(p or q or r) and False", False),\r
- ]\r
+ tests = [\r
+ ("p", True),\r
+ ("q", False),\r
+ ("p and q", False),\r
+ ("p and not q", True),\r
+ ("not not p", True),\r
+ ("not(p and q)", True),\r
+ ("q or not p and r", False),\r
+ ("q or not p or not r", False),\r
+ ("q or not (p and r)", False),\r
+ ("p or q or r", True),\r
+ ("p or q or r and False", True),\r
+ ("(p or q or r) and False", False),\r
+ ]\r
\r
print("p =", p)\r
print("q =", q)\r
print("r =", r)\r
print()\r
- for t,expected in tests:\r
- res = boolExpr.parseString(t)[0]\r
+ for test_string, expected in tests:\r
+ res = boolExpr.parseString(test_string)[0]\r
success = "PASS" if bool(res) == expected else "FAIL"\r
- print (t,'\n', res, '=', bool(res),'\n', success, '\n')\r
+ print(test_string, "\n", res, "=", bool(res), "\n", success, "\n")\r
#
# Copyright (c) 2003,2016, Paul McGuire
#
-from pyparsing import Word, delimitedList, Optional, \
- Group, alphas, alphanums, Forward, oneOf, quotedString, \
- infixNotation, opAssoc, \
- ZeroOrMore, restOfLine, CaselessKeyword, pyparsing_common as ppc
+from pyparsing import (
+ Word,
+ delimitedList,
+ Optional,
+ Group,
+ alphas,
+ alphanums,
+ Forward,
+ oneOf,
+ quotedString,
+ infixNotation,
+ opAssoc,
+ restOfLine,
+ CaselessKeyword,
+ ParserElement,
+ pyparsing_common as ppc,
+)
+
+ParserElement.enablePackrat()
# define SQL tokens
selectStmt = Forward()
-SELECT, FROM, WHERE, AND, OR, IN, IS, NOT, NULL = map(CaselessKeyword,
- "select from where and or in is not null".split())
+SELECT, FROM, WHERE, AND, OR, IN, IS, NOT, NULL = map(
+ CaselessKeyword, "select from where and or in is not null".split()
+)
NOT_NULL = NOT + NULL
-ident = Word( alphas, alphanums + "_$" ).setName("identifier")
-columnName = delimitedList(ident, ".", combine=True).setName("column name")
+ident = Word(alphas, alphanums + "_$").setName("identifier")
+columnName = delimitedList(ident, ".", combine=True).setName("column name")
columnName.addParseAction(ppc.upcaseTokens)
-columnNameList = Group( delimitedList(columnName))
-tableName = delimitedList(ident, ".", combine=True).setName("table name")
+columnNameList = Group(delimitedList(columnName).setName("column_list"))
+tableName = delimitedList(ident, ".", combine=True).setName("table name")
tableName.addParseAction(ppc.upcaseTokens)
-tableNameList = Group(delimitedList(tableName))
+tableNameList = Group(delimitedList(tableName).setName("table_list"))
-binop = oneOf("= != < > >= <= eq ne lt le gt ge", caseless=True)
-realNum = ppc.real()
+binop = oneOf("= != < > >= <= eq ne lt le gt ge", caseless=True).setName("binop")
+realNum = ppc.real().setName("real number")
intNum = ppc.signed_integer()
-columnRval = realNum | intNum | quotedString | columnName # need to add support for alg expressions
+columnRval = (
+ realNum | intNum | quotedString | columnName
+).setName("column_rvalue") # need to add support for alg expressions
whereCondition = Group(
- ( columnName + binop + columnRval ) |
- ( columnName + IN + Group("(" + delimitedList( columnRval ) + ")" )) |
- ( columnName + IN + Group("(" + selectStmt + ")" )) |
- ( columnName + IS + (NULL | NOT_NULL))
- )
-
-whereExpression = infixNotation(whereCondition,
+ (columnName + binop + columnRval)
+ | (columnName + IN + Group("(" + delimitedList(columnRval).setName("in_values_list") + ")"))
+ | (columnName + IN + Group("(" + selectStmt + ")"))
+ | (columnName + IS + (NULL | NOT_NULL))
+).setName("where_condition")
+
+whereExpression = infixNotation(
+ whereCondition,
[
(NOT, 1, opAssoc.RIGHT),
(AND, 2, opAssoc.LEFT),
(OR, 2, opAssoc.LEFT),
- ])
+ ],
+).setName("where_expression")
# define the grammar
-selectStmt <<= (SELECT + ('*' | columnNameList)("columns") +
- FROM + tableNameList( "tables" ) +
- Optional(Group(WHERE + whereExpression), "")("where"))
+selectStmt <<= (
+ SELECT
+ + ("*" | columnNameList)("columns")
+ + FROM
+ + tableNameList("tables")
+ + Optional(Group(WHERE + whereExpression), "")("where")
+).setName("select_statement")
simpleSQL = selectStmt
# define Oracle comment format, and ignore them
oracleSqlComment = "--" + restOfLine
-simpleSQL.ignore( oracleSqlComment )
+simpleSQL.ignore(oracleSqlComment)
if __name__ == "__main__":
- simpleSQL.runTests("""\
+ simpleSQL.runTests(
+ """\
# multiple tables
SELECT * from XYZZY, ABC
# where clause with comparison operator
Select A,b from table1,table2 where table1.id eq table2.id
- """)
+ """
+ )
Here's a URL to {{Pyparsing's Wiki Page->https://site-closed.wikispaces.com}}\r
"""\r
\r
-def convertToHTML(opening,closing):\r
- def conversionParseAction(s,l,t):\r
+\r
+def convertToHTML(opening, closing):\r
+ def conversionParseAction(s, l, t):\r
return opening + t[0] + closing\r
+\r
return conversionParseAction\r
\r
-italicized = QuotedString("*").setParseAction(convertToHTML("<I>","</I>"))\r
-bolded = QuotedString("**").setParseAction(convertToHTML("<B>","</B>"))\r
-boldItalicized = QuotedString("***").setParseAction(convertToHTML("<B><I>","</I></B>"))\r
-def convertToHTML_A(s,l,t):\r
+\r
+italicized = QuotedString("*").setParseAction(convertToHTML("<I>", "</I>"))\r
+bolded = QuotedString("**").setParseAction(convertToHTML("<B>", "</B>"))\r
+boldItalicized = QuotedString("***").setParseAction(convertToHTML("<B><I>", "</I></B>"))\r
+\r
+\r
+def convertToHTML_A(s, l, t):\r
try:\r
- text,url=t[0].split("->")\r
+ text, url = t[0].split("->")\r
except ValueError:\r
- raise ParseFatalException(s,l,"invalid URL link reference: " + t[0])\r
- return '<A href="{0}">{1}</A>'.format(url, text)\r
+ raise ParseFatalException(s, l, "invalid URL link reference: " + t[0])\r
+ return '<A href="{}">{}</A>'.format(url, text)\r
+\r
\r
-urlRef = QuotedString("{{",endQuoteChar="}}").setParseAction(convertToHTML_A)\r
+urlRef = QuotedString("{{", endQuoteChar="}}").setParseAction(convertToHTML_A)\r
\r
wikiMarkup = urlRef | boldItalicized | bolded | italicized\r
\r
* of outstanding requests on this session, then send the pdu.\r
* Returns the request id of the generated packet if applicable, otherwise 1.\r
* On any error, 0 is returned.\r
- * The pdu is freed by snmp_send() unless a failure occured.\r
+ * The pdu is freed by snmp_send() unless a failure occurred.\r
*/\r
int snmp_send (struct snmp_session *, struct snmp_pdu *);\r
\r
* then send the pdu.\r
* Returns the request id of the generated packet if applicable, otherwise 1.\r
* On any error, 0 is returned.\r
- * The pdu is freed by snmp_send() unless a failure occured.\r
+ * The pdu is freed by snmp_send() unless a failure occurred.\r
*/\r
int snmp_async_send (struct snmp_session *, struct snmp_pdu *,\r
snmp_callback, void *);\r
int (*fcheck) (u_char *, size_t)\r
);\r
\r
-/* provided for backwards compatability. Don't use these functions.\r
+/* provided for backwards compatibility. Don't use these functions.\r
See snmp_debug.h and snmp_debug.c instead.\r
*/\r
#if HAVE_STDARG_H\r
# 675 Mass Ave, Cambridge, MA 02139, USA.\r
"""\r
\r
-#===imports======================\r
+# ===imports======================\r
import sys\r
import os\r
import getopt\r
from pyparsing import *\r
\r
\r
-#===globals======================\r
+# ===globals======================\r
modname = "sparser"\r
__version__ = "0.1"\r
\r
\r
-#--option args--\r
+# --option args--\r
debug_p = 0\r
-#opt_b=None #string arg, default is undefined\r
+# opt_b=None #string arg, default is undefined\r
\r
\r
-#---positional args, default is empty---\r
+# ---positional args, default is empty---\r
pargs = []\r
\r
\r
-#---other---\r
+# ---other---\r
\r
\r
-#===utilities====================\r
+# ===utilities====================\r
def msg(txt):\r
"""Send message to stdout."""\r
sys.stdout.write(txt)\r
sys.stdout.flush()\r
\r
+\r
def debug(ftn, txt):\r
"""Used for debugging."""\r
if debug_p:\r
- sys.stdout.write("{0}.{1}:{2}\n".format(modname, ftn, txt))\r
+ sys.stdout.write("{}.{}:{}\n".format(modname, ftn, txt))\r
sys.stdout.flush()\r
\r
+\r
def fatal(ftn, txt):\r
"""If can't continue."""\r
- msg = "{0}.{1}:FATAL:{2}\n".format(modname, ftn, txt)\r
+ msg = "{}.{}:FATAL:{}\n".format(modname, ftn, txt)\r
raise SystemExit(msg)\r
\r
+\r
def usage():\r
"""Prints the docstring."""\r
print(__doc__)\r
\r
\r
-\r
-#====================================\r
+# ====================================\r
class ToInteger(TokenConverter):\r
"""Converter to make token into an integer."""\r
- def postParse( self, instring, loc, tokenlist ):\r
+\r
+ def postParse(self, instring, loc, tokenlist):\r
return int(tokenlist[0])\r
\r
+\r
class ToFloat(TokenConverter):\r
"""Converter to make token into a float."""\r
- def postParse( self, instring, loc, tokenlist ):\r
+\r
+ def postParse(self, instring, loc, tokenlist):\r
return float(tokenlist[0])\r
\r
+\r
class ParseFileLineByLine:\r
"""\r
Bring data from text files into a program, optionally parsing each line\r
'"a"' (append, not supported for .Z files).\r
"""\r
\r
- def __init__(self, filename, mode = 'r'):\r
- """Opens input file, and if available the definition file. If the\r
+ def __init__(self, filename, mode="r"):\r
+ """\r
+ Opens input file, and if available the definition file. If the\r
definition file is available __init__ will then create some pyparsing\r
- helper variables. """\r
- if mode not in ['r', 'w', 'a']:\r
- raise IOError(0, 'Illegal mode: ' + repr(mode))\r
+ helper variables.\r
+ """\r
+ if mode not in ["r", "w", "a"]:\r
+ raise OSError(0, "Illegal mode: " + repr(mode))\r
\r
- if string.find(filename, ':/') > 1: # URL\r
- if mode == 'w':\r
- raise IOError("can't write to a URL")\r
+ if string.find(filename, ":/") > 1: # URL\r
+ if mode == "w":\r
+ raise OSError("can't write to a URL")\r
import urllib.request, urllib.parse, urllib.error\r
+\r
self.file = urllib.request.urlopen(filename)\r
else:\r
filename = os.path.expanduser(filename)\r
- if mode == 'r' or mode == 'a':\r
+ if mode == "r" or mode == "a":\r
if not os.path.exists(filename):\r
- raise IOError(2, 'No such file or directory: ' + filename)\r
+ raise OSError(2, "No such file or directory: " + filename)\r
filen, file_extension = os.path.splitext(filename)\r
command_dict = {\r
- ('.Z', 'r'):\r
- "self.file = os.popen('uncompress -c ' + filename, mode)",\r
- ('.gz', 'r'):\r
- "self.file = gzip.GzipFile(filename, 'rb')",\r
- ('.bz2', 'r'):\r
- "self.file = os.popen('bzip2 -dc ' + filename, mode)",\r
- ('.Z', 'w'):\r
- "self.file = os.popen('compress > ' + filename, mode)",\r
- ('.gz', 'w'):\r
- "self.file = gzip.GzipFile(filename, 'wb')",\r
- ('.bz2', 'w'):\r
- "self.file = os.popen('bzip2 > ' + filename, mode)",\r
- ('.Z', 'a'):\r
- "raise IOError, (0, 'Can\'t append to .Z files')",\r
- ('.gz', 'a'):\r
- "self.file = gzip.GzipFile(filename, 'ab')",\r
- ('.bz2', 'a'):\r
- "raise IOError, (0, 'Can\'t append to .bz2 files')",\r
- }\r
-\r
- exec(command_dict.get((file_extension, mode),\r
- 'self.file = open(filename, mode)'))\r
+ (".Z", "r"): "self.file = os.popen('uncompress -c ' + filename, mode)",\r
+ (".gz", "r"): "self.file = gzip.GzipFile(filename, 'rb')",\r
+ (".bz2", "r"): "self.file = os.popen('bzip2 -dc ' + filename, mode)",\r
+ (".Z", "w"): "self.file = os.popen('compress > ' + filename, mode)",\r
+ (".gz", "w"): "self.file = gzip.GzipFile(filename, 'wb')",\r
+ (".bz2", "w"): "self.file = os.popen('bzip2 > ' + filename, mode)",\r
+ (".Z", "a"): "raise IOError, (0, 'Can't append to .Z files')",\r
+ (".gz", "a"): "self.file = gzip.GzipFile(filename, 'ab')",\r
+ (".bz2", "a"): "raise IOError, (0, 'Can't append to .bz2 files')",\r
+ }\r
+\r
+ exec(\r
+ command_dict.get(\r
+ (file_extension, mode), "self.file = open(filename, mode)"\r
+ )\r
+ )\r
\r
self.grammar = None\r
\r
decimal_sep = "."\r
sign = oneOf("+ -")\r
# part of printables without decimal_sep, +, -\r
- special_chars = string.replace('!"#$%&\'()*,./:;<=>?@[\\]^_`{|}~',\r
- decimal_sep, "")\r
- integer = ToInteger(\r
- Combine(Optional(sign) +\r
- Word(nums))).setName("integer")\r
- positive_integer = ToInteger(\r
- Combine(Optional("+") +\r
- Word(nums))).setName("integer")\r
- negative_integer = ToInteger(\r
- Combine("-" +\r
- Word(nums))).setName("integer")\r
+ special_chars = string.replace(\r
+ "!\"#$%&'()*,./:;<=>?@[\\]^_`{|}~", decimal_sep, ""\r
+ )\r
+ integer = ToInteger(Combine(Optional(sign) + Word(nums))).setName("integer")\r
+ positive_integer = ToInteger(Combine(Optional("+") + Word(nums))).setName(\r
+ "integer"\r
+ )\r
+ negative_integer = ToInteger(Combine("-" + Word(nums))).setName("integer")\r
real = ToFloat(\r
- Combine(Optional(sign) +\r
- Word(nums) +\r
- decimal_sep +\r
- Optional(Word(nums)) +\r
- Optional(oneOf("E e") +\r
- Word(nums)))).setName("real")\r
+ Combine(\r
+ Optional(sign)\r
+ + Word(nums)\r
+ + decimal_sep\r
+ + Optional(Word(nums))\r
+ + Optional(oneOf("E e") + Word(nums))\r
+ )\r
+ ).setName("real")\r
positive_real = ToFloat(\r
- Combine(Optional("+") +\r
- Word(nums) +\r
- decimal_sep +\r
- Optional(Word(nums)) +\r
- Optional(oneOf("E e") +\r
- Word(nums)))).setName("real")\r
+ Combine(\r
+ Optional("+")\r
+ + Word(nums)\r
+ + decimal_sep\r
+ + Optional(Word(nums))\r
+ + Optional(oneOf("E e") + Word(nums))\r
+ )\r
+ ).setName("real")\r
negative_real = ToFloat(\r
- Combine("-" +\r
- Word(nums) +\r
- decimal_sep +\r
- Optional(Word(nums)) +\r
- Optional(oneOf("E e") +\r
- Word(nums)))).setName("real")\r
- qString = ( sglQuotedString | dblQuotedString ).setName("qString")\r
+ Combine(\r
+ "-"\r
+ + Word(nums)\r
+ + decimal_sep\r
+ + Optional(Word(nums))\r
+ + Optional(oneOf("E e") + Word(nums))\r
+ )\r
+ ).setName("real")\r
+ qString = (sglQuotedString | dblQuotedString).setName("qString")\r
\r
# add other characters we should skip over between interesting fields\r
integer_junk = Optional(\r
- Suppress(\r
- Word(alphas +\r
- special_chars +\r
- decimal_sep))).setName("integer_junk")\r
- real_junk = Optional(\r
- Suppress(\r
- Word(alphas +\r
- special_chars))).setName("real_junk")\r
+ Suppress(Word(alphas + special_chars + decimal_sep))\r
+ ).setName("integer_junk")\r
+ real_junk = Optional(Suppress(Word(alphas + special_chars))).setName(\r
+ "real_junk"\r
+ )\r
qString_junk = SkipTo(qString).setName("qString_junk")\r
\r
# Now that 'integer', 'real', and 'qString' have been assigned I can\r
# execute the definition file.\r
- exec(compile(open(self.parsedef).read(), self.parsedef, 'exec'))\r
+ exec(compile(open(self.parsedef).read(), self.parsedef, "exec"))\r
\r
# Build the grammar, combination of the 'integer', 'real, 'qString',\r
# and '*_junk' variables assigned above in the order specified in the\r
# definition file.\r
grammar = []\r
for nam, expr in parse:\r
- grammar.append( eval(expr.name + "_junk"))\r
- grammar.append( expr.setResultsName(nam) )\r
- self.grammar = And( grammar[1:] + [restOfLine] )\r
+ grammar.append(eval(expr.name + "_junk"))\r
+ grammar.append(expr.setResultsName(nam))\r
+ self.grammar = And(grammar[1:] + [restOfLine])\r
\r
def __del__(self):\r
"""Delete (close) the file wrapper."""\r
self.file.flush()\r
\r
\r
-#=============================\r
+# =============================\r
def main(pargs):\r
"""This should only be used for testing. The primary mode of operation is\r
as an imported library.\r
print(i)\r
\r
\r
-#-------------------------\r
-if __name__ == '__main__':\r
+# -------------------------\r
+if __name__ == "__main__":\r
ftn = "main"\r
- opts, pargs = getopt.getopt(sys.argv[1:], 'hvd',\r
- ['help', 'version', 'debug', 'bb='])\r
+ opts, pargs = getopt.getopt(\r
+ sys.argv[1:], "hvd", ["help", "version", "debug", "bb="]\r
+ )\r
for opt in opts:\r
- if opt[0] == '-h' or opt[0] == '--help':\r
- print(modname+": version="+__version__)\r
+ if opt[0] == "-h" or opt[0] == "--help":\r
+ print(modname + ": version=" + __version__)\r
usage()\r
sys.exit(0)\r
- elif opt[0] == '-v' or opt[0] == '--version':\r
- print(modname+": version="+__version__)\r
+ elif opt[0] == "-v" or opt[0] == "--version":\r
+ print(modname + ": version=" + __version__)\r
sys.exit(0)\r
- elif opt[0] == '-d' or opt[0] == '--debug':\r
+ elif opt[0] == "-d" or opt[0] == "--debug":\r
debug_p = 1\r
- elif opt[0] == '--bb':\r
+ elif opt[0] == "--bb":\r
opt_b = opt[1]\r
\r
- #---make the object and run it---\r
+ # ---make the object and run it---\r
main(pargs)\r
\r
-#===Revision Log===\r
-#Created by mkpythonproj:\r
-#2006-02-06 Tim Cera\r
+# ===Revision Log===\r
+# Created by mkpythonproj:\r
+# 2006-02-06 Tim Cera\r
#\r
# Adapted from a post at https://energyblog.blogspot.com/2006/04/blog-post_20.html.\r
#\r
sampleSQL = """\r
-create table student\r
+create table students\r
(\r
student_id integer primary key,\r
firstname varchar(20),\r
(class_id) references classes(class_id);\r
""".upper()\r
\r
-from pyparsing import Literal, Word, delimitedList \\r
- , alphas, alphanums \\r
- , OneOrMore, ZeroOrMore, CharsNotIn \\r
- , replaceWith\r
+from pyparsing import (\r
+ Literal,\r
+ Word,\r
+ delimitedList,\r
+ alphas,\r
+ alphanums,\r
+ OneOrMore,\r
+ ZeroOrMore,\r
+ CharsNotIn,\r
+ replaceWith,\r
+)\r
\r
skobki = "(" + ZeroOrMore(CharsNotIn(")")) + ")"\r
-field_def = OneOrMore(Word(alphas,alphanums+"_\"':-") | skobki)\r
+field_def = OneOrMore(Word(alphas, alphanums + "_\"':-") | skobki)\r
+\r
+\r
+def field_act(s, loc, tok):\r
+ return ("<" + tok[0] + "> " + " ".join(tok)).replace('"', '\\"')\r
\r
-def field_act(s,loc,tok):\r
- return ("<"+tok[0]+"> " + " ".join(tok)).replace("\"","\\\"")\r
\r
field_def.setParseAction(field_act)\r
\r
-field_list_def = delimitedList( field_def )\r
+field_list_def = delimitedList(field_def)\r
+\r
+\r
def field_list_act(toks):\r
return " | ".join(toks)\r
\r
+\r
field_list_def.setParseAction(field_list_act)\r
\r
-create_table_def = Literal("CREATE") + "TABLE" + Word(alphas,alphanums+"_").setResultsName("tablename") + \\r
- "("+field_list_def.setResultsName("columns")+")"+ ";"\r
+create_table_def = (\r
+ Literal("CREATE")\r
+ + "TABLE"\r
+ + Word(alphas, alphanums + "_").setResultsName("tablename")\r
+ + "("\r
+ + field_list_def.setResultsName("columns")\r
+ + ")"\r
+ + ";"\r
+)\r
+\r
\r
def create_table_act(toks):\r
- return """"%(tablename)s" [\n\t label="<%(tablename)s> %(tablename)s | %(columns)s"\n\t shape="record"\n];""" % toks\r
+ return (\r
+ """"%(tablename)s" [\n\t label="<%(tablename)s> %(tablename)s | %(columns)s"\n\t shape="record"\n];"""\r
+ % toks\r
+ )\r
+\r
+\r
create_table_def.setParseAction(create_table_act)\r
\r
-add_fkey_def=Literal("ALTER")+"TABLE"+"ONLY" + Word(alphanums+"_").setResultsName("fromtable") + "ADD" \\r
- + "CONSTRAINT" + Word(alphanums+"_") + "FOREIGN"+"KEY"+"("+Word(alphanums+"_").setResultsName("fromcolumn")+")" \\r
- +"REFERENCES"+Word(alphanums+"_").setResultsName("totable")+"("+Word(alphanums+"_").setResultsName("tocolumn")+")"+";"\r
+add_fkey_def = (\r
+ Literal("ALTER")\r
+ + "TABLE"\r
+ + "ONLY"\r
+ + Word(alphanums + "_").setResultsName("fromtable")\r
+ + "ADD"\r
+ + "CONSTRAINT"\r
+ + Word(alphanums + "_")\r
+ + "FOREIGN"\r
+ + "KEY"\r
+ + "("\r
+ + Word(alphanums + "_").setResultsName("fromcolumn")\r
+ + ")"\r
+ + "REFERENCES"\r
+ + Word(alphanums + "_").setResultsName("totable")\r
+ + "("\r
+ + Word(alphanums + "_").setResultsName("tocolumn")\r
+ + ")"\r
+ + ";"\r
+)\r
+\r
\r
def add_fkey_act(toks):\r
return """ "%(fromtable)s":%(fromcolumn)s -> "%(totable)s":%(tocolumn)s """ % toks\r
+\r
+\r
add_fkey_def.setParseAction(add_fkey_act)\r
\r
-other_statement_def = ( OneOrMore(CharsNotIn(";") ) + ";")\r
-other_statement_def.setParseAction( replaceWith("") )\r
+other_statement_def = OneOrMore(CharsNotIn(";")) + ";"\r
+other_statement_def.setParseAction(replaceWith(""))\r
comment_def = "--" + ZeroOrMore(CharsNotIn("\n"))\r
-comment_def.setParseAction( replaceWith("") )\r
+comment_def.setParseAction(replaceWith(""))\r
\r
-statement_def = comment_def | create_table_def | add_fkey_def | other_statement_def\r
-defs = OneOrMore(statement_def)\r
+statement_def = comment_def | create_table_def | add_fkey_def | other_statement_def\r
+defs = OneOrMore(statement_def)\r
\r
print("""digraph g { graph [ rankdir = "LR" ]; """)\r
for i in defs.parseString(sampleSQL):\r
- if i!="":\r
+ if i != "":\r
print(i)\r
print("}")\r
separation character and perform reasonable diffs on two structures.\r
"""\r
\r
-from pyparsing import Suppress,Word,nums,alphas,alphanums,Combine,oneOf,\\r
- Optional,QuotedString,Forward,Group,ZeroOrMore,srange\r
+from pyparsing import (\r
+ Suppress,\r
+ Word,\r
+ nums,\r
+ alphas,\r
+ alphanums,\r
+ Combine,\r
+ oneOf,\r
+ Optional,\r
+ QuotedString,\r
+ Forward,\r
+ Group,\r
+ ZeroOrMore,\r
+ srange,\r
+ pyparsing_common as ppc,\r
+)\r
\r
-MARK,UNMARK,AT,COLON,QUOTE = map(Suppress,"[]@:'")\r
+MARK, UNMARK, AT, COLON, QUOTE = map(Suppress, "[]@:'")\r
\r
-NUMBER = Word(nums)\r
-NUMBER.setParseAction(lambda t:int(t[0]))\r
-FLOAT = Combine(oneOf("+ -") + Word(nums) + "." + Optional(Word(nums)))\r
-FLOAT.setParseAction(lambda t:float(t[0]))\r
-STRING = QuotedString('"', multiline=True)\r
-WORD = Word(alphas,alphanums+"_:")\r
+NUMBER = ppc.integer()\r
+FLOAT = ppc.real()\r
+STRING = QuotedString('"', multiline=True) | QuotedString("'", multiline=True)\r
+WORD = Word(alphas, alphanums + "_:")\r
ATTRIBUTE = Combine(AT + WORD)\r
\r
strBody = Forward()\r
+\r
+\r
def setBodyLength(tokens):\r
- strBody << Word(srange(r'[\0x00-\0xffff]'), exact=int(tokens[0]))\r
+ strBody << Word(srange(r"[\0x00-\0xffff]"), exact=int(tokens[0]))\r
return ""\r
-BLOB = Combine(QUOTE + Word(nums).setParseAction(setBodyLength) +\r
- COLON + strBody + QUOTE)\r
+\r
+\r
+BLOB = Combine(\r
+ QUOTE + Word(nums).setParseAction(setBodyLength) + COLON + strBody + QUOTE\r
+)\r
\r
item = Forward()\r
+\r
+\r
def assignUsing(s):\r
def assignPA(tokens):\r
if s in tokens:\r
tokens[tokens[s]] = tokens[0]\r
del tokens[s]\r
+\r
return assignPA\r
-GROUP = (MARK +\r
- Group( ZeroOrMore(\r
- (item +\r
- Optional(ATTRIBUTE)("attr")\r
- ).setParseAction(assignUsing("attr"))\r
- )\r
- ) +\r
- ( WORD("name") | UNMARK )\r
- ).setParseAction(assignUsing("name"))\r
-item << (NUMBER | FLOAT | STRING | BLOB | GROUP )\r
-\r
-tests = """\\r
-[ '10:1234567890' @name 25 @age +0.45 @percentage person:zed\r
-[ [ "hello" 1 child root\r
-[ "child" [ 200 '4:like' "I" "hello" things root\r
-[ [ "data" [ 2 1 ] @numbers child root\r
-[ [ 1 2 3 ] @test 4 5 6 root\r
-""".splitlines()\r
-\r
-for test in tests:\r
- if test:\r
- print(test)\r
- print(item.parseString(test).dump())\r
- print()\r
+\r
+\r
+GROUP = (\r
+ MARK\r
+ + Group(\r
+ ZeroOrMore(\r
+ (item + Optional(ATTRIBUTE)("attr")).setParseAction(assignUsing("attr"))\r
+ )\r
+ )\r
+ + (WORD("name") | UNMARK)\r
+).setParseAction(assignUsing("name"))\r
+item <<= FLOAT | NUMBER | STRING | BLOB | GROUP\r
+\r
+item.runTests(\r
+ """\\r
+ [ '10:1234567890' @name 25 @age +0.45 @percentage person:zed\r
+ [ [ "hello" 1 child root\r
+ [ "child" [ 200 '4:like' "I" "hello" things root\r
+ [ [ "data" [ 2 1 ] @numbers child root\r
+ [ [ 1 2 3 ] @test 4 5 6 root\r
+ """\r
+)\r
import statemachine
import documentsignoffstate
-print('\n'.join(t.__name__ for t in documentsignoffstate.DocumentRevisionState.transitions()))
+print(
+ "\n".join(
+ t.__name__ for t in documentsignoffstate.DocumentRevisionState.transitions()
+ )
+)
+
class Document(documentsignoffstate.DocumentRevisionStateMixin):
def __init__(self):
while not isinstance(doc._state, documentsignoffstate.Approved):
- print('...submit')
+ print("...submit")
doc.submit()
print(doc)
print(doc.state.description)
- if random.randint(1,10) > 3:
- print('...reject')
+ if random.randint(1, 10) > 3:
+ print("...reject")
doc.reject()
else:
- print('...approve')
+ print("...approve")
doc.approve()
print(doc)
print(doc)
print(doc.state.description)
-if __name__ == '__main__':
+
+if __name__ == "__main__":
run_demo()
# example using named state transitions
# This implements a state model for submitting,
-# approving, activating, and purging document
+# approving, activating, and purging document
# revisions in a document management system.
#
# The state model looks like:
# |
# | (create)
# |
-# v
+# v
# Editing ----------------------------------------------+
# | ^ |
# | | |
# just an example of a state machine with named transitions.
#
-
+
statemachine DocumentRevisionState:
New -( create )-> Editing
Editing -( cancel )-> Deleted
class RestrictedBook(Book):
def __init__(self):
- super(RestrictedBook, self).__init__()
+ super().__init__()
self._authorized_users = []
def authorize(self, name):
if user in self._authorized_users:
super().checkout()
else:
- raise Exception("{0} could not check out restricted book".format(user if user is not None else "anonymous"))
+ raise Exception(
+ "{} could not check out restricted book".format(
+ user if user is not None else "anonymous"
+ )
+ )
def run_demo():
print(book)
try:
book.checkout()
- except Exception as e: # statemachine.InvalidTransitionException:
+ except librarybookstate.BookState.InvalidTransitionException as e:
print(e)
- print('..cannot check out reserved book')
+ print("..cannot check out reserved book")
book.release()
print(book)
book.checkout()
try:
restricted_book.checkout(name)
except Exception as e:
- print('..' + str(e))
+ print(".." + str(e))
else:
- print('checkout to', name)
+ print("checkout to", name)
print(restricted_book)
restricted_book.checkin()
print(restricted_book)
-if __name__ == '__main__':
+if __name__ == "__main__":
run_demo()
import os
import types
import importlib
-try:
- import urllib.parse
- url_parse = urllib.parse.urlparse
-except ImportError:
- print("import error, Python 2 not supported")
- raise
- import urllib
- url_parse = urllib.parse
+import importlib.machinery
+from urllib.parse import urlparse
DEBUG = False
# define basic exception for invalid state transitions - state machine classes will subclass to
# define their own specific exception type
-class InvalidTransitionException(Exception): pass
+class InvalidTransitionException(Exception):
+ pass
ident = pp.Word(pp.alphas + "_", pp.alphanums + "_$")
def no_keywords_allowed(s, l, t):
wd = t[0]
return not keyword.iskeyword(wd)
-ident.addCondition(no_keywords_allowed, message="cannot use a Python keyword for state or transition identifier")
-stateTransition = ident("from_state") + "->" + ident("to_state")
-stateMachine = (pp.Keyword("statemachine") + ident("name") + ":"
- + pp.OneOrMore(pp.Group(stateTransition))("transitions"))
-namedStateTransition = (ident("from_state")
- + "-(" + ident("transition") + ")->"
- + ident("to_state"))
-namedStateMachine = (pp.Keyword("statemachine") + ident("name") + ":"
- + pp.OneOrMore(pp.Group(namedStateTransition))("transitions"))
+ident.addCondition(
+ no_keywords_allowed,
+ message="cannot use a Python keyword for state or transition identifier",
+)
+
+stateTransition = ident("from_state") + "->" + ident("to_state")
+stateMachine = (
+ pp.Keyword("statemachine")
+ + ident("name")
+ + ":"
+ + pp.OneOrMore(pp.Group(stateTransition))("transitions")
+)
+
+namedStateTransition = (
+ ident("from_state") + "-(" + ident("transition") + ")->" + ident("to_state")
+)
+namedStateMachine = (
+ pp.Keyword("statemachine")
+ + ident("name")
+ + ":"
+ + pp.OneOrMore(pp.Group(namedStateTransition))("transitions")
+)
def expand_state_definition(source, loc, tokens):
# define base class for state classes
baseStateClass = tokens.name
- statedef.extend([
- "class %s(object):" % baseStateClass,
- " def __str__(self):",
- " return self.__class__.__name__",
-
- " @classmethod",
- " def states(cls):",
- " return list(cls.__subclasses__())",
-
- " def next_state(self):",
- " return self._next_state_class()",
- ])
+ statedef.extend(
+ [
+ "class %s(object):" % baseStateClass,
+ " def __str__(self):",
+ " return self.__class__.__name__",
+ " @classmethod",
+ " def states(cls):",
+ " return list(cls.__subclasses__())",
+ " def next_state(self):",
+ " return self._next_state_class()",
+ ]
+ )
# define all state classes
- statedef.extend("class {0}({1}): pass".format(s, baseStateClass) for s in states)
+ statedef.extend("class {}({}): pass".format(s, baseStateClass) for s in states)
# define state->state transitions
- statedef.extend("{0}._next_state_class = {1}".format(s, fromTo[s]) for s in states if s in fromTo)
-
- statedef.extend([
- "class {baseStateClass}Mixin:".format(baseStateClass=baseStateClass),
- " def __init__(self):",
- " self._state = None",
-
- " def initialize_state(self, init_state):",
- " if issubclass(init_state, {baseStateClass}):".format(baseStateClass=baseStateClass),
- " init_state = init_state()",
- " self._state = init_state",
-
- " @property",
- " def state(self):",
- " return self._state",
-
- " # get behavior/properties from current state",
- " def __getattr__(self, attrname):",
- " attr = getattr(self._state, attrname)",
- " return attr",
+ statedef.extend(
+ "{}._next_state_class = {}".format(s, fromTo[s]) for s in states if s in fromTo
+ )
- " def __str__(self):",
- " return '{0}: {1}'.format(self.__class__.__name__, self._state)",
- ])
+ statedef.extend(
+ [
+ "class {baseStateClass}Mixin:".format(baseStateClass=baseStateClass),
+ " def __init__(self):",
+ " self._state = None",
+ " def initialize_state(self, init_state):",
+ " if issubclass(init_state, {baseStateClass}):".format(
+ baseStateClass=baseStateClass
+ ),
+ " init_state = init_state()",
+ " self._state = init_state",
+ " @property",
+ " def state(self):",
+ " return self._state",
+ " # get behavior/properties from current state",
+ " def __getattr__(self, attrname):",
+ " attr = getattr(self._state, attrname)",
+ " return attr",
+ " def __str__(self):",
+ " return '{0}: {1}'.format(self.__class__.__name__, self._state)",
+ ]
+ )
return ("\n" + indent).join(statedef) + "\n"
+
stateMachine.setParseAction(expand_state_definition)
fromTo[s] = {}
# define state transition class
- statedef.extend([
- "class {baseStateClass}Transition:".format(baseStateClass=baseStateClass),
- " def __str__(self):",
- " return self.transitionName",
- ])
statedef.extend(
- "{tn_name} = {baseStateClass}Transition()".format(tn_name=tn,
- baseStateClass=baseStateClass)
- for tn in transitions)
- statedef.extend("{tn_name}.transitionName = '{tn_name}'".format(tn_name=tn)
- for tn in transitions)
+ [
+ "class {baseStateClass}Transition:".format(baseStateClass=baseStateClass),
+ " def __str__(self):",
+ " return self.transitionName",
+ ]
+ )
+ statedef.extend(
+ "{tn_name} = {baseStateClass}Transition()".format(
+ tn_name=tn, baseStateClass=baseStateClass
+ )
+ for tn in transitions
+ )
+ statedef.extend(
+ "{tn_name}.transitionName = '{tn_name}'".format(tn_name=tn)
+ for tn in transitions
+ )
# define base class for state classes
- statedef.extend([
- "class %s(object):" % baseStateClass,
- " from statemachine import InvalidTransitionException as BaseTransitionException",
- " class InvalidTransitionException(BaseTransitionException): pass",
- " def __str__(self):",
- " return self.__class__.__name__",
-
- " @classmethod",
- " def states(cls):",
- " return list(cls.__subclasses__())",
-
- " @classmethod",
- " def next_state(cls, name):",
- " try:",
- " return cls.tnmap[name]()",
- " except KeyError:",
- " raise cls.InvalidTransitionException('%s does not support transition %r'% (cls.__name__, name))",
-
- " def __bad_tn(name):",
- " def _fn(cls):",
- " raise cls.InvalidTransitionException('%s does not support transition %r'% (cls.__name__, name))",
- " _fn.__name__ = name",
- " return _fn",
- ])
+ statedef.extend(
+ [
+ "class %s(object):" % baseStateClass,
+ " from statemachine import InvalidTransitionException as BaseTransitionException",
+ " class InvalidTransitionException(BaseTransitionException): pass",
+ " def __str__(self):",
+ " return self.__class__.__name__",
+ " @classmethod",
+ " def states(cls):",
+ " return list(cls.__subclasses__())",
+ " @classmethod",
+ " def next_state(cls, name):",
+ " try:",
+ " return cls.tnmap[name]()",
+ " except KeyError:",
+ " raise cls.InvalidTransitionException('%s does not support transition %r'% (cls.__name__, name))",
+ " def __bad_tn(name):",
+ " def _fn(cls):",
+ " raise cls.InvalidTransitionException('%s does not support transition %r'% (cls.__name__, name))",
+ " _fn.__name__ = name",
+ " return _fn",
+ ]
+ )
# define default 'invalid transition' methods in base class, valid transitions will be implemented in subclasses
statedef.extend(
" {tn_name} = classmethod(__bad_tn({tn_name!r}))".format(tn_name=tn)
- for tn in transitions)
+ for tn in transitions
+ )
# define all state classes
- statedef.extend("class %s(%s): pass" % (s, baseStateClass)
- for s in states)
+ statedef.extend("class {}({}): pass".format(s, baseStateClass) for s in states)
# define state transition methods for valid transitions from each state
for s in states:
trns = list(fromTo[s].items())
# statedef.append("%s.tnmap = {%s}" % (s, ", ".join("%s:%s" % tn for tn in trns)))
- statedef.extend("%s.%s = classmethod(lambda cls: %s())" % (s, tn_, to_)
- for tn_, to_ in trns)
-
- statedef.extend([
- "{baseStateClass}.transitions = classmethod(lambda cls: [{transition_class_list}])".format(
- baseStateClass=baseStateClass,
- transition_class_list = ', '.join("cls.{0}".format(tn) for tn in transitions)
- ),
- "{baseStateClass}.transition_names = [tn.__name__ for tn in {baseStateClass}.transitions()]".format(
- baseStateClass=baseStateClass
+ statedef.extend(
+ "{}.{} = classmethod(lambda cls: {}())".format(s, tn_, to_)
+ for tn_, to_ in trns
)
- ])
-
- # define <state>Mixin class for application classes that delegate to the state
- statedef.extend([
- "class {baseStateClass}Mixin:".format(baseStateClass=baseStateClass),
- " def __init__(self):",
- " self._state = None",
-
- " def initialize_state(self, init_state):",
- " if issubclass(init_state, {baseStateClass}):".format(baseStateClass=baseStateClass),
- " init_state = init_state()",
- " self._state = init_state",
- " @property",
- " def state(self):",
- " return self._state",
-
- " # get behavior/properties from current state",
- " def __getattr__(self, attrname):",
- " attr = getattr(self._state, attrname)",
- " return attr",
-
- " def __str__(self):",
- " return '{0}: {1}'.format(self.__class__.__name__, self._state)",
+ statedef.extend(
+ [
+ "{baseStateClass}.transitions = classmethod(lambda cls: [{transition_class_list}])".format(
+ baseStateClass=baseStateClass,
+ transition_class_list=", ".join(
+ "cls.{}".format(tn) for tn in transitions
+ ),
+ ),
+ "{baseStateClass}.transition_names = [tn.__name__ for tn in {baseStateClass}.transitions()]".format(
+ baseStateClass=baseStateClass
+ ),
+ ]
+ )
- ])
+ # define <state>Mixin class for application classes that delegate to the state
+ statedef.extend(
+ [
+ "class {baseStateClass}Mixin:".format(baseStateClass=baseStateClass),
+ " def __init__(self):",
+ " self._state = None",
+ " def initialize_state(self, init_state):",
+ " if issubclass(init_state, {baseStateClass}):".format(
+ baseStateClass=baseStateClass
+ ),
+ " init_state = init_state()",
+ " self._state = init_state",
+ " @property",
+ " def state(self):",
+ " return self._state",
+ " # get behavior/properties from current state",
+ " def __getattr__(self, attrname):",
+ " attr = getattr(self._state, attrname)",
+ " return attr",
+ " def __str__(self):",
+ " return '{0}: {1}'.format(self.__class__.__name__, self._state)",
+ ]
+ )
# define transition methods to be delegated to the _state instance variable
statedef.extend(
- " def {tn_name}(self): self._state = self._state.{tn_name}()".format(tn_name=tn)
+ " def {tn_name}(self): self._state = self._state.{tn_name}()".format(
+ tn_name=tn
+ )
for tn in transitions
)
return ("\n" + indent).join(statedef) + "\n"
+
namedStateMachine.setParseAction(expand_named_state_definition)
# ======================================================================
# NEW STUFF - Matt Anderson, 2009-11-26
# ======================================================================
-class SuffixImporter(object):
+class SuffixImporter:
"""An importer designed using the mechanism defined in :pep:`302`. I read
the PEP, and also used Doug Hellmann's PyMOTW article `Modules and
Imports`_, as a pattern.
Define a subclass that specifies a :attr:`suffix` attribute, and
implements a :meth:`process_filedata` method. Then call the classmethod
:meth:`register` on your class to actually install it in the appropriate
- places in :mod:`sys`. """
+ places in :mod:`sys`."""
- scheme = 'suffix'
+ scheme = "suffix"
suffix = None
path_entry = None
@classmethod
def trigger_url(cls):
if cls.suffix is None:
- raise ValueError('%s.suffix is not set' % cls.__name__)
- return 'suffix:%s' % cls.suffix
+ raise ValueError("%s.suffix is not set" % cls.__name__)
+ return "suffix:%s" % cls.suffix
@classmethod
def register(cls):
sys.path.append(cls.trigger_url())
def __init__(self, path_entry):
- pr = url_parse(str(path_entry))
+ pr = urlparse(str(path_entry))
if pr.scheme != self.scheme or pr.path != self.suffix:
raise ImportError()
self.path_entry = path_entry
# it probably isn't even a filesystem path
finder = sys.path_importer_cache.get(dirpath)
if isinstance(finder, (type(None), importlib.machinery.FileFinder)):
- checkpath = os.path.join(dirpath, '{0}.{1}'.format(fullname, self.suffix))
+ checkpath = os.path.join(dirpath, "{}.{}".format(fullname, self.suffix))
yield checkpath
def find_module(self, fullname, path=None):
class PystateImporter(SuffixImporter):
- suffix = 'pystate'
+ suffix = "pystate"
def process_filedata(self, module, data):
# MATT-NOTE: re-worked :func:`get_state_machine`
# convert any statemachine expressions
- stateMachineExpr = (stateMachine | namedStateMachine).ignore(pp.pythonStyleComment)
+ stateMachineExpr = (stateMachine | namedStateMachine).ignore(
+ pp.pythonStyleComment
+ )
generated_code = stateMachineExpr.transformString(data)
- if DEBUG: print(generated_code)
+ if DEBUG:
+ print(generated_code)
# compile code object from generated code
# (strip trailing spaces and tabs, compile doesn't like
# dangling whitespace)
- COMPILE_MODE = 'exec'
+ COMPILE_MODE = "exec"
- codeobj = compile(generated_code.rstrip(" \t"),
- module.__file__,
- COMPILE_MODE)
+ codeobj = compile(generated_code.rstrip(" \t"), module.__file__, COMPILE_MODE)
exec(codeobj, module.__dict__)
PystateImporter.register()
if DEBUG:
- print("registered {0!r} importer".format(PystateImporter.suffix))
+ print("registered {!r} importer".format(PystateImporter.suffix))
light = TrafficLight()
for i in range(10):
- print("{0} {1}".format(light, ("STOP", "GO")[light.cars_can_go]))
+ print("{} {}".format(light, ("STOP", "GO")[light.cars_can_go]))
light.crossing_signal()
light.delay()
print()
"""
# convert state machine text to state classes
-generated = statemachine.namedStateMachine.transformString(vending_machine_state_description)
+generated = statemachine.namedStateMachine.transformString(
+ vending_machine_state_description
+)
# print(generated)
# exec generated code to define state classes and state mixin
exec(generated)
+
class VendingMachine(VendingMachineStateMixin):
def __init__(self):
self.initialize_state(Idle)
self._pressed = button
self.press_digit_button()
else:
- print('Did not recognize button {!r}'.format(str(button)))
+ print("Did not recognize button {!r}".format(str(button)))
def press_alpha_button(self):
try:
- super(VendingMachine, self).press_alpha_button()
+ super().press_alpha_button()
except VendingMachineState.InvalidTransitionException as ite:
print(ite)
else:
def press_digit_button(self):
try:
- super(VendingMachine, self).press_digit_button()
+ super().press_digit_button()
except VendingMachineState.InvalidTransitionException as ite:
print(ite)
else:
def dispense(self):
try:
- super(VendingMachine, self).dispense()
+ super().dispense()
except VendingMachineState.InvalidTransitionException as ite:
print(ite)
else:
while True:
print(v.state)
- cmd = input("Command ({})> ".format('/'.join(videostate.VideoState.transition_names))).lower().strip()
+ cmd = (
+ input("Command ({})> ".format("/".join(videostate.VideoState.transition_names)))
+ .lower()
+ .strip()
+ )
if not cmd:
continue
- if cmd in ('?', 'h', 'help'):
- print('enter a transition {!r}'.format(videostate.VideoState.transition_names))
- print(' q - quit')
- print(' ?, h, help - this message')
+ if cmd in ("?", "h", "help"):
+ print("enter a transition {!r}".format(videostate.VideoState.transition_names))
+ print(" q - quit")
+ print(" ?, h, help - this message")
continue
# quitting out
- if cmd.startswith('q'):
+ if cmd.startswith("q"):
break
# get transition function for given command
state_transition_fn = getattr(v, cmd, None)
if state_transition_fn is None:
- print('???')
+ print("???")
continue
# invoke the input transition, handle invalid commands
from .btpyparse import Macro
from . import btpyparse as bp
+
class TestBibparse(unittest.TestCase):
def test_names(self):
# check various types of names
# All names can contains alphas, but not some special chars
- bad_chars = '"#%\'(),={}'
- for name_type, dig1f in ((bp.macro_def, False),
- (bp.field_name, False),
- (bp.entry_type, False),
- (bp.cite_key, True)):
- if dig1f: # can start with digit
- self.assertEqual(name_type.parseString('2t')[0], '2t')
+ bad_chars = "\"#%'(),={}"
+ for name_type, dig1f in (
+ (bp.macro_def, False),
+ (bp.field_name, False),
+ (bp.entry_type, False),
+ (bp.cite_key, True),
+ ):
+ if dig1f: # can start with digit
+ self.assertEqual("2t", name_type.parseString("2t")[0])
else:
- self.assertRaises(ParseException, name_type.parseString, '2t')
+ self.assertRaises(ParseException, name_type.parseString, "2t")
# All of the names cannot contain some characters
for char in bad_chars:
self.assertRaises(ParseException, name_type.parseString, char)
# standard strings all OK
- self.assertEqual(name_type.parseString('simple_test')[0], 'simple_test')
+ self.assertEqual("simple_test", name_type.parseString("simple_test")[0])
# Test macro ref
mr = bp.macro_ref
# can't start with digit
- self.assertRaises(ParseException, mr.parseString, '2t')
+ self.assertRaises(ParseException, mr.parseString, "2t")
for char in bad_chars:
self.assertRaises(ParseException, mr.parseString, char)
- self.assertEqual(mr.parseString('simple_test')[0].name, 'simple_test')
+ self.assertEqual("simple_test", mr.parseString("simple_test")[0].name)
def test_numbers(self):
- self.assertEqual(bp.number.parseString('1066')[0], '1066')
- self.assertEqual(bp.number.parseString('0')[0], '0')
- self.assertRaises(ParseException, bp.number.parseString, '-4')
- self.assertRaises(ParseException, bp.number.parseString, '+4')
- self.assertRaises(ParseException, bp.number.parseString, '.4')
+ self.assertEqual("1066", bp.number.parseString("1066")[0])
+ self.assertEqual("0", bp.number.parseString("0")[0])
+ self.assertRaises(ParseException, bp.number.parseString, "-4")
+ self.assertRaises(ParseException, bp.number.parseString, "+4")
+ self.assertRaises(ParseException, bp.number.parseString, ".4")
# something point something leaves a trailing .4 unmatched
- self.assertEqual(bp.number.parseString('0.4')[0], '0')
-
+ self.assertEqual("0", bp.number.parseString("0.4")[0])
def test_parse_string(self):
# test string building blocks
- self.assertEqual(bp.chars_no_quotecurly.parseString('x')[0], 'x')
- self.assertEqual(bp.chars_no_quotecurly.parseString("a string")[0], 'a string')
- self.assertEqual(bp.chars_no_quotecurly.parseString('a "string')[0], 'a ')
- self.assertEqual(bp.chars_no_curly.parseString('x')[0], 'x')
- self.assertEqual(bp.chars_no_curly.parseString("a string")[0], 'a string')
- self.assertEqual(bp.chars_no_curly.parseString('a {string')[0], 'a ')
- self.assertEqual(bp.chars_no_curly.parseString('a }string')[0], 'a ')
+ self.assertEqual(bp.chars_no_quotecurly.parseString("x")[0], "x")
+ self.assertEqual(bp.chars_no_quotecurly.parseString("a string")[0], "a string")
+ self.assertEqual(bp.chars_no_quotecurly.parseString('a "string')[0], "a ")
+ self.assertEqual(bp.chars_no_curly.parseString("x")[0], "x")
+ self.assertEqual(bp.chars_no_curly.parseString("a string")[0], "a string")
+ self.assertEqual(bp.chars_no_curly.parseString("a {string")[0], "a ")
+ self.assertEqual(bp.chars_no_curly.parseString("a }string")[0], "a ")
# test more general strings together
for obj in (bp.curly_string, bp.string, bp.field_value):
- self.assertEqual(obj.parseString('{}').asList(), [])
+ self.assertEqual(obj.parseString("{}").asList(), [])
self.assertEqual(obj.parseString('{a "string}')[0], 'a "string')
- self.assertEqual(obj.parseString('{a {nested} string}').asList(),
- ['a ', ['nested'], ' string'])
- self.assertEqual(obj.parseString('{a {double {nested}} string}').asList(),
- ['a ', ['double ', ['nested']], ' string'])
+ self.assertEqual(
+ ["a ", ["nested"], "string"],
+ obj.parseString("{a {nested} string}").asList(),
+ )
+ self.assertEqual(
+ ["a ", ["double ", ["nested"]], "string"],
+ obj.parseString("{a {double {nested}} string}").asList(),
+ )
for obj in (bp.quoted_string, bp.string, bp.field_value):
- self.assertEqual(obj.parseString('""').asList(), [])
- self.assertEqual(obj.parseString('"a string"')[0], 'a string')
- self.assertEqual(obj.parseString('"a {nested} string"').asList(),
- ['a ', ['nested'], ' string'])
- self.assertEqual(obj.parseString('"a {double {nested}} string"').asList(),
- ['a ', ['double ', ['nested']], ' string'])
+ self.assertEqual([], obj.parseString('""').asList())
+ self.assertEqual("a string", obj.parseString('"a string"')[0])
+ self.assertEqual(
+ ["a ", ["nested"], "string"],
+ obj.parseString('"a {nested} string"').asList(),
+ )
+ self.assertEqual(
+ ["a ", ["double ", ["nested"]], "string"],
+ obj.parseString('"a {double {nested}} string"').asList(),
+ )
+
# check macro def in string
- self.assertEqual(bp.string.parseString('someascii')[0], Macro('someascii'))
- self.assertRaises(ParseException, bp.string.parseString, '%#= validstring')
+ self.assertEqual(Macro("someascii"), bp.string.parseString("someascii")[0])
+ self.assertRaises(ParseException, bp.string.parseString, "%#= validstring")
# check number in string
- self.assertEqual(bp.string.parseString('1994')[0], '1994')
-
+ self.assertEqual(bp.string.parseString("1994")[0], "1994")
def test_parse_field(self):
# test field value - hashes included
fv = bp.field_value
# Macro
- self.assertEqual(fv.parseString('aname')[0], Macro('aname'))
- self.assertEqual(fv.parseString('ANAME')[0], Macro('aname'))
+ self.assertEqual(Macro("aname"), fv.parseString("aname")[0])
+ self.assertEqual(Macro("aname"), fv.parseString("ANAME")[0])
# String and macro
- self.assertEqual(fv.parseString('aname # "some string"').asList(),
- [Macro('aname'), 'some string'])
+ self.assertEqual(
+ [Macro("aname"), "some string"],
+ fv.parseString('aname # "some string"').asList(),
+ )
# Nested string
- self.assertEqual(fv.parseString('aname # {some {string}}').asList(),
- [Macro('aname'), 'some ', ['string']])
+ self.assertEqual(
+ [Macro("aname"), "some ", ["string"]],
+ fv.parseString("aname # {some {string}}").asList(),
+ )
# String and number
- self.assertEqual(fv.parseString('"a string" # 1994').asList(),
- ['a string', '1994'])
+ self.assertEqual(
+ ["a string", "1994"], fv.parseString('"a string" # 1994').asList()
+ )
# String and number and macro
- self.assertEqual(fv.parseString('"a string" # 1994 # a_macro').asList(),
- ['a string', '1994', Macro('a_macro')])
-
+ self.assertEqual(
+ ["a string", "1994", Macro("a_macro")],
+ fv.parseString('"a string" # 1994 # a_macro').asList(),
+ )
def test_comments(self):
- res = bp.comment.parseString('@Comment{about something}')
- self.assertEqual(res.asList(), ['comment', '{about something}'])
+ res = bp.comment.parseString("@Comment{about something}")
+ self.assertEqual(res.asList(), ["comment", "{about something}"])
self.assertEqual(
- bp.comment.parseString('@COMMENT{about something').asList(),
- ['comment', '{about something'])
+ ["comment", "{about something"],
+ bp.comment.parseString("@COMMENT{about something").asList(),
+ )
self.assertEqual(
- bp.comment.parseString('@comment(about something').asList(),
- ['comment', '(about something'])
+ ["comment", "(about something"],
+ bp.comment.parseString("@comment(about something").asList(),
+ )
self.assertEqual(
- bp.comment.parseString('@COMment about something').asList(),
- ['comment', ' about something'])
- self.assertRaises(ParseException, bp.comment.parseString,
- '@commentabout something')
- self.assertRaises(ParseException, bp.comment.parseString,
- '@comment+about something')
- self.assertRaises(ParseException, bp.comment.parseString,
- '@comment"about something')
-
+ ["comment", " about something"],
+ bp.comment.parseString("@COMment about something").asList(),
+ )
+ self.assertRaises(
+ ParseException, bp.comment.parseString, "@commentabout something"
+ )
+ self.assertRaises(
+ ParseException, bp.comment.parseString, "@comment+about something"
+ )
+ self.assertRaises(
+ ParseException, bp.comment.parseString, '@comment"about something'
+ )
def test_preamble(self):
res = bp.preamble.parseString('@preamble{"about something"}')
- self.assertEqual(res.asList(), ['preamble', 'about something'])
- self.assertEqual(bp.preamble.parseString(
- '@PREamble{{about something}}').asList(),
- ['preamble', 'about something'])
- self.assertEqual(bp.preamble.parseString("""@PREamble{
+ self.assertEqual(res.asList(), ["preamble", "about something"])
+ self.assertEqual(
+ ["preamble", "about something"],
+ bp.preamble.parseString("@PREamble{{about something}}").asList(),
+ )
+ self.assertEqual(
+ ["preamble", "about something"],
+ bp.preamble.parseString(
+ """@PREamble{
{about something}
- }""").asList(),
- ['preamble', 'about something'])
-
+ }"""
+ ).asList(),
+ )
def test_macro(self):
res = bp.macro.parseString('@string{ANAME = "about something"}')
- self.assertEqual(res.asList(), ['string', 'aname', 'about something'])
+ self.assertEqual(res.asList(), ["string", "aname", "about something"])
self.assertEqual(
- bp.macro.parseString('@string{aname = {about something}}').asList(),
- ['string', 'aname', 'about something'])
-
+ ["string", "aname", "about something"],
+ bp.macro.parseString("@string{aname = {about something}}").asList(),
+ )
def test_entry(self):
txt = """@some_entry{akey, aname = "about something",
another={something else}}"""
res = bp.entry.parseString(txt)
- self.assertEqual(res.asList(),
- ['some_entry', 'akey',
- ['aname', 'about something'], ['another', 'something else']])
+ self.assertEqual(
+ [
+ "some_entry",
+ "akey",
+ ["aname", "about something"],
+ ["another", "something else"],
+ ],
+ res.asList(),
+ )
# Case conversion
txt = """@SOME_ENTRY{akey, ANAME = "about something",
another={something else}}"""
res = bp.entry.parseString(txt)
- self.assertEqual(res.asList(),
- ['some_entry', 'akey',
- ['aname', 'about something'], ['another', 'something else']])
-
+ self.assertEqual(
+ [
+ "some_entry",
+ "akey",
+ ["aname", "about something"],
+ ["another", "something else"],
+ ],
+ res.asList(),
+ )
def test_bibfile(self):
txt = """@some_entry{akey, aname = "about something",
another={something else}}"""
res = bp.bibfile.parseString(txt)
- self.assertEqual(res.asList(),
- [['some_entry', 'akey',
- ['aname', 'about something'],
- ['another', 'something else']]])
-
+ self.assertEqual(
+ [
+ [
+ "some_entry",
+ "akey",
+ ["aname", "about something"],
+ ["another", "something else"],
+ ]
+ ],
+ res.asList(),
+ )
def test_bib1(self):
# First pass whole bib-like tests
self.assertEqual(res.asList(), res3)
-if __name__ == '__main__':
+if __name__ == "__main__":
unittest.main()
# URL extractor
# Copyright 2004, Paul McGuire
from pyparsing import makeHTMLTags, pyparsing_common as ppc
-import urllib.request
-from contextlib import closing
+from urllib.request import urlopen
import pprint
-linkOpenTag, linkCloseTag = makeHTMLTags('a')
+linkOpenTag, linkCloseTag = makeHTMLTags("a")
linkBody = linkOpenTag.tag_body
linkBody.setParseAction(ppc.stripHTMLTags)
-linkBody.addParseAction(lambda toks: ' '.join(toks[0].strip().split()))
+linkBody.addParseAction(lambda toks: " ".join(toks[0].strip().split()))
link = linkOpenTag + linkBody("body") + linkCloseTag.suppress()
# Go get some HTML with some links in it.
-with closing(urllib.request.urlopen("https://www.cnn.com/")) as serverListPage:
+with urlopen("https://www.cnn.com/") as serverListPage:
htmlText = serverListPage.read().decode("UTF-8")
# scanString is a generator that loops through the input htmlText, and for each
# Create dictionary from list comprehension, assembled from each pair of tokens returned
# from a matched URL.
-pprint.pprint(
- {toks.body: toks.href for toks, strt, end in link.scanString(htmlText)}
- )
+pprint.pprint({toks.body: toks.href for toks, strt, end in link.scanString(htmlText)})
# URL extractor
# Copyright 2004, Paul McGuire
-from pyparsing import makeHTMLTags
-from contextlib import closing
-import urllib.request, urllib.parse, urllib.error
+from collections import Counter
import pprint
+from urllib.request import urlopen
+
+from pyparsing import makeHTMLTags, pyparsing_common as ppc, FollowedBy, trace_parse_action
# Define the pyparsing grammar for a URL, that is:
# URLlink ::= <a href= URL>linkText</a>
linkOpenTag, linkCloseTag = makeHTMLTags("a")
link = linkOpenTag + linkOpenTag.tag_body("body") + linkCloseTag.suppress()
+
+# Add a parse action to expand relative URLs
+def expand_relative_url(t):
+ url = t.href
+ if url.startswith("//"):
+ url = "https:" + url
+ elif url.startswith(("/", "?", "#")):
+ url = "https://www.cnn.com" + url
+
+ # Put modified URL back into input tokens
+ t["href"] = url
+
+
+link.add_parse_action(expand_relative_url)
+
# Go get some HTML with some links in it.
-with closing(urllib.request.urlopen("https://www.cnn.com/")) as serverListPage:
- htmlText = serverListPage.read()
+with urlopen("https://www.cnn.com/") as serverListPage:
+ htmlText = serverListPage.read().decode()
# scanString is a generator that loops through the input htmlText, and for each
# match yields the tokens and start and end locations (for this application, we are
for toks, strt, end in link.scanString(htmlText):
print(toks.startA.href, "->", toks.body)
-# Create dictionary from list comprehension, assembled from each pair of tokens returned
+# Create dictionary with a dict comprehension, assembled from each pair of tokens returned
# from a matched URL.
-pprint.pprint(
- {toks.body: toks.startA.href for toks, strt, end in link.scanString(htmlText)}
- )
+links = {toks.body: toks.href for toks, _, _ in link.scanString(htmlText)}
+pprint.pprint(links)
+
+# Parse the urls in the links using pyparsing_common.url, and tally up all
+# the different domains in a Counter.
+domains = Counter()
+for url in links.values():
+
+ print(url)
+ parsed = ppc.url.parseString(url)
+
+ # print parsed fields for each new url
+ if parsed.host not in domains:
+ print(parsed.dump())
+ print()
+
+ # update domain counter
+ domains[parsed.host] += 1
+
+
+# Print out a little table of all the domains in the urls
+max_domain_len = max(len(d) for d in domains)
+print()
+print("{:{}s} {}".format("Domain", max_domain_len, "Count"))
+print("{:=<{}} {:=<5}".format("", max_domain_len, ""))
+
+for domain, count in domains.most_common():
+ print("{:{}s} {:5d}".format(domain, max_domain_len, count))
__version__ = "1.0.11"
-from pyparsing import Literal, Keyword, Word, OneOrMore, ZeroOrMore, \
- Forward, delimitedList, Group, Optional, Combine, alphas, nums, restOfLine, \
- alphanums, dblQuotedString, empty, ParseException, oneOf, \
- StringEnd, FollowedBy, ParserElement, Regex, cppStyleComment
+from pyparsing import (
+ Literal,
+ Keyword,
+ Word,
+ OneOrMore,
+ ZeroOrMore,
+ Forward,
+ delimitedList,
+ Group,
+ Optional,
+ Combine,
+ alphas,
+ nums,
+ restOfLine,
+ alphanums,
+ dblQuotedString,
+ empty,
+ ParseException,
+ oneOf,
+ StringEnd,
+ FollowedBy,
+ ParserElement,
+ Regex,
+ cppStyleComment,
+)
import pyparsing
+
usePackrat = False
packratOn = False
else:
packratOn = True
-def dumpTokens(s,l,t):
+
+def dumpTokens(s, l, t):
import pprint
- pprint.pprint( t.asList() )
+
+ pprint.pprint(t.asList())
+
verilogbnf = None
+
+
def Verilog_BNF():
global verilogbnf
if verilogbnf is None:
# compiler directives
- compilerDirective = Combine( "`" + \
- oneOf("define undef ifdef else endif default_nettype "
- "include resetall timescale unconnected_drive "
- "nounconnected_drive celldefine endcelldefine") + \
- restOfLine ).setName("compilerDirective")
+ compilerDirective = Combine(
+ "`"
+ + oneOf(
+ "define undef ifdef else endif default_nettype "
+ "include resetall timescale unconnected_drive "
+ "nounconnected_drive celldefine endcelldefine"
+ )
+ + restOfLine
+ ).setName("compilerDirective")
# primitives
- SEMI,COLON,LPAR,RPAR,LBRACE,RBRACE,LBRACK,RBRACK,DOT,COMMA,EQ = map(Literal,";:(){}[].,=")
-
- identLead = alphas+"$_"
- identBody = alphanums+"$_"
- identifier1 = Regex( r"\.?["+identLead+"]["+identBody+r"]*(\.["+identLead+"]["+identBody+"]*)*"
- ).setName("baseIdent")
- identifier2 = Regex(r"\\\S+").setParseAction(lambda t:t[0][1:]).setName("escapedIdent")#.setDebug()
+ SEMI, COLON, LPAR, RPAR, LBRACE, RBRACE, LBRACK, RBRACK, DOT, COMMA, EQ = map(
+ Literal, ";:(){}[].,="
+ )
+
+ identLead = alphas + "$_"
+ identBody = alphanums + "$_"
+ identifier1 = Regex(
+ r"\.?["
+ + identLead
+ + "]["
+ + identBody
+ + r"]*(\.["
+ + identLead
+ + "]["
+ + identBody
+ + "]*)*"
+ ).setName("baseIdent")
+ identifier2 = (
+ Regex(r"\\\S+").setParseAction(lambda t: t[0][1:]).setName("escapedIdent")
+ ) # .setDebug()
identifier = identifier1 | identifier2
- assert(identifier2 == r'\abc')
+ assert identifier2 == r"\abc"
hexnums = nums + "abcdefABCDEF" + "_?"
base = Regex("'[bBoOdDhH]").setName("base")
- basedNumber = Combine( Optional( Word(nums + "_") ) + base + Word(hexnums+"xXzZ"),
- joinString=" ", adjacent=False ).setName("basedNumber")
- #~ number = ( basedNumber | Combine( Word( "+-"+spacedNums, spacedNums ) +
- #~ Optional( DOT + Optional( Word( spacedNums ) ) ) +
- #~ Optional( e + Word( "+-"+spacedNums, spacedNums ) ) ).setName("numeric") )
- number = ( basedNumber | \
- Regex(r"[+-]?[0-9_]+(\.[0-9_]*)?([Ee][+-]?[0-9_]+)?") \
- ).setName("numeric")
- #~ decnums = nums + "_"
- #~ octnums = "01234567" + "_"
+ basedNumber = Combine(
+ Optional(Word(nums + "_")) + base + Word(hexnums + "xXzZ"),
+ joinString=" ",
+ adjacent=False,
+ ).setName("basedNumber")
+ # ~ number = ( basedNumber | Combine( Word( "+-"+spacedNums, spacedNums ) +
+ # ~ Optional( DOT + Optional( Word( spacedNums ) ) ) +
+ # ~ Optional( e + Word( "+-"+spacedNums, spacedNums ) ) ).setName("numeric") )
+ number = (
+ basedNumber | Regex(r"[+-]?[0-9_]+(\.[0-9_]*)?([Ee][+-]?[0-9_]+)?")
+ ).setName("numeric")
+ # ~ decnums = nums + "_"
+ # ~ octnums = "01234567" + "_"
expr = Forward().setName("expr")
- concat = Group( LBRACE + delimitedList( expr ) + RBRACE )
+ concat = Group(LBRACE + delimitedList(expr) + RBRACE)
multiConcat = Group("{" + expr + concat + "}").setName("multiConcat")
- funcCall = Group(identifier + LPAR + Optional( delimitedList( expr ) ) + RPAR).setName("funcCall")
+ funcCall = Group(
+ identifier + LPAR + Optional(delimitedList(expr)) + RPAR
+ ).setName("funcCall")
- subscrRef = Group(LBRACK + delimitedList( expr, COLON ) + RBRACK)
- subscrIdentifier = Group( identifier + Optional( subscrRef ) )
- #~ scalarConst = "0" | (( FollowedBy('1') + oneOf("1'b0 1'b1 1'bx 1'bX 1'B0 1'B1 1'Bx 1'BX 1") ))
+ subscrRef = Group(LBRACK + delimitedList(expr, COLON) + RBRACK)
+ subscrIdentifier = Group(identifier + Optional(subscrRef))
+ # ~ scalarConst = "0" | (( FollowedBy('1') + oneOf("1'b0 1'b1 1'bx 1'bX 1'B0 1'B1 1'Bx 1'BX 1") ))
scalarConst = Regex("0|1('[Bb][01xX])?")
- mintypmaxExpr = Group( expr + COLON + expr + COLON + expr ).setName("mintypmax")
+ mintypmaxExpr = Group(expr + COLON + expr + COLON + expr).setName("mintypmax")
primary = (
- number |
- (LPAR + mintypmaxExpr + RPAR ) |
- ( LPAR + Group(expr) + RPAR ).setName("nestedExpr") |
- multiConcat |
- concat |
- dblQuotedString |
- funcCall |
- subscrIdentifier
- )
-
- unop = oneOf( "+ - ! ~ & ~& | ^| ^ ~^" ).setName("unop")
- binop = oneOf( "+ - * / % == != === !== && "
- "|| < <= > >= & | ^ ^~ >> << ** <<< >>>" ).setName("binop")
+ number
+ | (LPAR + mintypmaxExpr + RPAR)
+ | (LPAR + Group(expr) + RPAR).setName("nestedExpr")
+ | multiConcat
+ | concat
+ | dblQuotedString
+ | funcCall
+ | subscrIdentifier
+ )
+
+ unop = oneOf("+ - ! ~ & ~& | ^| ^ ~^").setName("unop")
+ binop = oneOf(
+ "+ - * / % == != === !== && "
+ "|| < <= > >= & | ^ ^~ >> << ** <<< >>>"
+ ).setName("binop")
expr << (
- ( unop + expr ) | # must be first!
- ( primary + "?" + expr + COLON + expr ) |
- ( primary + Optional( binop + expr ) )
- )
+ (unop + expr)
+ | (primary + "?" + expr + COLON + expr) # must be first!
+ | (primary + Optional(binop + expr))
+ )
lvalue = subscrIdentifier | concat
# keywords
- if_ = Keyword("if")
- else_ = Keyword("else")
- edge = Keyword("edge")
- posedge = Keyword("posedge")
- negedge = Keyword("negedge")
- specify = Keyword("specify")
+ if_ = Keyword("if")
+ else_ = Keyword("else")
+ edge = Keyword("edge")
+ posedge = Keyword("posedge")
+ negedge = Keyword("negedge")
+ specify = Keyword("specify")
endspecify = Keyword("endspecify")
- fork = Keyword("fork")
- join = Keyword("join")
- begin = Keyword("begin")
- end = Keyword("end")
- default = Keyword("default")
- forever = Keyword("forever")
- repeat = Keyword("repeat")
- while_ = Keyword("while")
- for_ = Keyword("for")
- case = oneOf( "case casez casex" )
- endcase = Keyword("endcase")
- wait = Keyword("wait")
- disable = Keyword("disable")
- deassign = Keyword("deassign")
- force = Keyword("force")
- release = Keyword("release")
- assign = Keyword("assign")
+ fork = Keyword("fork")
+ join = Keyword("join")
+ begin = Keyword("begin")
+ end = Keyword("end")
+ default = Keyword("default")
+ forever = Keyword("forever")
+ repeat = Keyword("repeat")
+ while_ = Keyword("while")
+ for_ = Keyword("for")
+ case = oneOf("case casez casex")
+ endcase = Keyword("endcase")
+ wait = Keyword("wait")
+ disable = Keyword("disable")
+ deassign = Keyword("deassign")
+ force = Keyword("force")
+ release = Keyword("release")
+ assign = Keyword("assign")
eventExpr = Forward()
- eventTerm = ( posedge + expr ) | ( negedge + expr ) | expr | ( LPAR + eventExpr + RPAR )
- eventExpr << (
- Group( delimitedList( eventTerm, Keyword("or") ) )
- )
- eventControl = Group( "@" + ( ( LPAR + eventExpr + RPAR ) | identifier | "*" ) ).setName("eventCtrl")
-
- delayArg = ( number |
- Word(alphanums+"$_") | #identifier |
- ( LPAR + Group( delimitedList( mintypmaxExpr | expr ) ) + RPAR )
- ).setName("delayArg")#.setDebug()
- delay = Group( "#" + delayArg ).setName("delay")#.setDebug()
+ eventTerm = (
+ (posedge + expr) | (negedge + expr) | expr | (LPAR + eventExpr + RPAR)
+ )
+ eventExpr << (Group(delimitedList(eventTerm, Keyword("or"))))
+ eventControl = Group(
+ "@" + ((LPAR + eventExpr + RPAR) | identifier | "*")
+ ).setName("eventCtrl")
+
+ delayArg = (
+ number
+ | Word(alphanums + "$_")
+ | (LPAR + Group(delimitedList(mintypmaxExpr | expr)) + RPAR) # identifier |
+ ).setName(
+ "delayArg"
+ ) # .setDebug()
+ delay = Group("#" + delayArg).setName("delay") # .setDebug()
delayOrEventControl = delay | eventControl
- assgnmt = Group( lvalue + EQ + Optional( delayOrEventControl ) + expr ).setName( "assgnmt" )
- nbAssgnmt = Group(( lvalue + "<=" + Optional( delay ) + expr ) |
- ( lvalue + "<=" + Optional( eventControl ) + expr )).setName( "nbassgnmt" )
+ assgnmt = Group(lvalue + EQ + Optional(delayOrEventControl) + expr).setName(
+ "assgnmt"
+ )
+ nbAssgnmt = Group(
+ (lvalue + "<=" + Optional(delay) + expr)
+ | (lvalue + "<=" + Optional(eventControl) + expr)
+ ).setName("nbassgnmt")
range = LBRACK + expr + COLON + expr + RBRACK
- paramAssgnmt = Group( identifier + EQ + expr ).setName("paramAssgnmt")
- parameterDecl = Group( "parameter" + Optional( range ) + delimitedList( paramAssgnmt ) + SEMI).setName("paramDecl")
-
- inputDecl = Group( "input" + Optional( range ) + delimitedList( identifier ) + SEMI )
- outputDecl = Group( "output" + Optional( range ) + delimitedList( identifier ) + SEMI )
- inoutDecl = Group( "inout" + Optional( range ) + delimitedList( identifier ) + SEMI )
-
- regIdentifier = Group( identifier + Optional( LBRACK + expr + COLON + expr + RBRACK ) )
- regDecl = Group( "reg" + Optional("signed") + Optional( range ) + delimitedList( regIdentifier ) + SEMI ).setName("regDecl")
- timeDecl = Group( "time" + delimitedList( regIdentifier ) + SEMI )
- integerDecl = Group( "integer" + delimitedList( regIdentifier ) + SEMI )
+ paramAssgnmt = Group(identifier + EQ + expr).setName("paramAssgnmt")
+ parameterDecl = Group(
+ "parameter" + Optional(range) + delimitedList(paramAssgnmt) + SEMI
+ ).setName("paramDecl")
+
+ inputDecl = Group("input" + Optional(range) + delimitedList(identifier) + SEMI)
+ outputDecl = Group(
+ "output" + Optional(range) + delimitedList(identifier) + SEMI
+ )
+ inoutDecl = Group("inout" + Optional(range) + delimitedList(identifier) + SEMI)
+
+ regIdentifier = Group(
+ identifier + Optional(LBRACK + expr + COLON + expr + RBRACK)
+ )
+ regDecl = Group(
+ "reg"
+ + Optional("signed")
+ + Optional(range)
+ + delimitedList(regIdentifier)
+ + SEMI
+ ).setName("regDecl")
+ timeDecl = Group("time" + delimitedList(regIdentifier) + SEMI)
+ integerDecl = Group("integer" + delimitedList(regIdentifier) + SEMI)
strength0 = oneOf("supply0 strong0 pull0 weak0 highz0")
strength1 = oneOf("supply1 strong1 pull1 weak1 highz1")
- driveStrength = Group( LPAR + ( ( strength0 + COMMA + strength1 ) |
- ( strength1 + COMMA + strength0 ) ) + RPAR ).setName("driveStrength")
- nettype = oneOf("wire tri tri1 supply0 wand triand tri0 supply1 wor trior trireg")
- expandRange = Optional( oneOf("scalared vectored") ) + range
- realDecl = Group( "real" + delimitedList( identifier ) + SEMI )
-
- eventDecl = Group( "event" + delimitedList( identifier ) + SEMI )
+ driveStrength = Group(
+ LPAR
+ + ((strength0 + COMMA + strength1) | (strength1 + COMMA + strength0))
+ + RPAR
+ ).setName("driveStrength")
+ nettype = oneOf(
+ "wire tri tri1 supply0 wand triand tri0 supply1 wor trior trireg"
+ )
+ expandRange = Optional(oneOf("scalared vectored")) + range
+ realDecl = Group("real" + delimitedList(identifier) + SEMI)
+
+ eventDecl = Group("event" + delimitedList(identifier) + SEMI)
blockDecl = (
- parameterDecl |
- regDecl |
- integerDecl |
- realDecl |
- timeDecl |
- eventDecl
- )
+ parameterDecl | regDecl | integerDecl | realDecl | timeDecl | eventDecl
+ )
- stmt = Forward().setName("stmt")#.setDebug()
+ stmt = Forward().setName("stmt") # .setDebug()
stmtOrNull = stmt | SEMI
- caseItem = ( delimitedList( expr ) + COLON + stmtOrNull ) | \
- ( default + Optional(":") + stmtOrNull )
+ caseItem = (delimitedList(expr) + COLON + stmtOrNull) | (
+ default + Optional(":") + stmtOrNull
+ )
stmt << Group(
- ( begin + Group( ZeroOrMore( stmt ) ) + end ).setName("begin-end") |
- ( if_ + Group(LPAR + expr + RPAR) + stmtOrNull + Optional( else_ + stmtOrNull ) ).setName("if") |
- ( delayOrEventControl + stmtOrNull ) |
- ( case + LPAR + expr + RPAR + OneOrMore( caseItem ) + endcase ) |
- ( forever + stmt ) |
- ( repeat + LPAR + expr + RPAR + stmt ) |
- ( while_ + LPAR + expr + RPAR + stmt ) |
- ( for_ + LPAR + assgnmt + SEMI + Group( expr ) + SEMI + assgnmt + RPAR + stmt ) |
- ( fork + ZeroOrMore( stmt ) + join ) |
- ( fork + COLON + identifier + ZeroOrMore( blockDecl ) + ZeroOrMore( stmt ) + end ) |
- ( wait + LPAR + expr + RPAR + stmtOrNull ) |
- ( "->" + identifier + SEMI ) |
- ( disable + identifier + SEMI ) |
- ( assign + assgnmt + SEMI ) |
- ( deassign + lvalue + SEMI ) |
- ( force + assgnmt + SEMI ) |
- ( release + lvalue + SEMI ) |
- ( begin + COLON + identifier + ZeroOrMore( blockDecl ) + ZeroOrMore( stmt ) + end ).setName("begin:label-end") |
+ (begin + Group(ZeroOrMore(stmt)) + end).setName("begin-end")
+ | (
+ if_
+ + Group(LPAR + expr + RPAR)
+ + stmtOrNull
+ + Optional(else_ + stmtOrNull)
+ ).setName("if")
+ | (delayOrEventControl + stmtOrNull)
+ | (case + LPAR + expr + RPAR + OneOrMore(caseItem) + endcase)
+ | (forever + stmt)
+ | (repeat + LPAR + expr + RPAR + stmt)
+ | (while_ + LPAR + expr + RPAR + stmt)
+ | (
+ for_
+ + LPAR
+ + assgnmt
+ + SEMI
+ + Group(expr)
+ + SEMI
+ + assgnmt
+ + RPAR
+ + stmt
+ )
+ | (fork + ZeroOrMore(stmt) + join)
+ | (
+ fork
+ + COLON
+ + identifier
+ + ZeroOrMore(blockDecl)
+ + ZeroOrMore(stmt)
+ + end
+ )
+ | (wait + LPAR + expr + RPAR + stmtOrNull)
+ | ("->" + identifier + SEMI)
+ | (disable + identifier + SEMI)
+ | (assign + assgnmt + SEMI)
+ | (deassign + lvalue + SEMI)
+ | (force + assgnmt + SEMI)
+ | (release + lvalue + SEMI)
+ | (
+ begin
+ + COLON
+ + identifier
+ + ZeroOrMore(blockDecl)
+ + ZeroOrMore(stmt)
+ + end
+ ).setName("begin:label-end")
+ |
# these *have* to go at the end of the list!!!
- ( assgnmt + SEMI ) |
- ( nbAssgnmt + SEMI ) |
- ( Combine( Optional("$") + identifier ) + Optional( LPAR + delimitedList(expr|empty) + RPAR ) + SEMI )
- ).setName("stmtBody")
+ (assgnmt + SEMI)
+ | (nbAssgnmt + SEMI)
+ | (
+ Combine(Optional("$") + identifier)
+ + Optional(LPAR + delimitedList(expr | empty) + RPAR)
+ + SEMI
+ )
+ ).setName("stmtBody")
"""
x::=<blocking_assignment> ;
x||= <non_blocking_assignment> ;
x||= force <assignment> ;
x||= release <lvalue> ;
"""
- alwaysStmt = Group( "always" + Optional(eventControl) + stmt ).setName("alwaysStmt")
- initialStmt = Group( "initial" + stmt ).setName("initialStmt")
+ alwaysStmt = Group("always" + Optional(eventControl) + stmt).setName(
+ "alwaysStmt"
+ )
+ initialStmt = Group("initial" + stmt).setName("initialStmt")
- chargeStrength = Group( LPAR + oneOf( "small medium large" ) + RPAR ).setName("chargeStrength")
+ chargeStrength = Group(LPAR + oneOf("small medium large") + RPAR).setName(
+ "chargeStrength"
+ )
continuousAssign = Group(
- assign + Optional( driveStrength ) + Optional( delay ) + delimitedList( assgnmt ) + SEMI
- ).setName("continuousAssign")
-
+ assign
+ + Optional(driveStrength)
+ + Optional(delay)
+ + delimitedList(assgnmt)
+ + SEMI
+ ).setName("continuousAssign")
tfDecl = (
- parameterDecl |
- inputDecl |
- outputDecl |
- inoutDecl |
- regDecl |
- timeDecl |
- integerDecl |
- realDecl
- )
+ parameterDecl
+ | inputDecl
+ | outputDecl
+ | inoutDecl
+ | regDecl
+ | timeDecl
+ | integerDecl
+ | realDecl
+ )
functionDecl = Group(
- "function" + Optional( range | "integer" | "real" ) + identifier + SEMI +
- Group( OneOrMore( tfDecl ) ) +
- Group( ZeroOrMore( stmt ) ) +
- "endfunction"
- )
+ "function"
+ + Optional(range | "integer" | "real")
+ + identifier
+ + SEMI
+ + Group(OneOrMore(tfDecl))
+ + Group(ZeroOrMore(stmt))
+ + "endfunction"
+ )
inputOutput = oneOf("input output")
- netDecl1Arg = ( nettype +
- Optional( expandRange ) +
- Optional( delay ) +
- Group( delimitedList( ~inputOutput + identifier ) ) )
- netDecl2Arg = ( "trireg" +
- Optional( chargeStrength ) +
- Optional( expandRange ) +
- Optional( delay ) +
- Group( delimitedList( ~inputOutput + identifier ) ) )
- netDecl3Arg = ( nettype +
- Optional( driveStrength ) +
- Optional( expandRange ) +
- Optional( delay ) +
- Group( delimitedList( assgnmt ) ) )
+ netDecl1Arg = (
+ nettype
+ + Optional(expandRange)
+ + Optional(delay)
+ + Group(delimitedList(~inputOutput + identifier))
+ )
+ netDecl2Arg = (
+ "trireg"
+ + Optional(chargeStrength)
+ + Optional(expandRange)
+ + Optional(delay)
+ + Group(delimitedList(~inputOutput + identifier))
+ )
+ netDecl3Arg = (
+ nettype
+ + Optional(driveStrength)
+ + Optional(expandRange)
+ + Optional(delay)
+ + Group(delimitedList(assgnmt))
+ )
netDecl1 = Group(netDecl1Arg + SEMI).setName("netDecl1")
netDecl2 = Group(netDecl2Arg + SEMI).setName("netDecl2")
netDecl3 = Group(netDecl3Arg + SEMI).setName("netDecl3")
- gateType = oneOf("and nand or nor xor xnor buf bufif0 bufif1 "
- "not notif0 notif1 pulldown pullup nmos rnmos "
- "pmos rpmos cmos rcmos tran rtran tranif0 "
- "rtranif0 tranif1 rtranif1" )
- gateInstance = Optional( Group( identifier + Optional( range ) ) ) + \
- LPAR + Group( delimitedList( expr ) ) + RPAR
- gateDecl = Group( gateType +
- Optional( driveStrength ) +
- Optional( delay ) +
- delimitedList( gateInstance) +
- SEMI )
-
- udpInstance = Group( Group( identifier + Optional(range | subscrRef) ) +
- LPAR + Group( delimitedList( expr ) ) + RPAR )
- udpInstantiation = Group( identifier -
- Optional( driveStrength ) +
- Optional( delay ) +
- delimitedList( udpInstance ) +
- SEMI ).setName("udpInstantiation")
-
- parameterValueAssignment = Group( Literal("#") + LPAR + Group( delimitedList( expr ) ) + RPAR )
- namedPortConnection = Group( DOT + identifier + LPAR + expr + RPAR ).setName("namedPortConnection")#.setDebug()
- assert(r'.\abc (abc )' == namedPortConnection)
+ gateType = oneOf(
+ "and nand or nor xor xnor buf bufif0 bufif1 "
+ "not notif0 notif1 pulldown pullup nmos rnmos "
+ "pmos rpmos cmos rcmos tran rtran tranif0 "
+ "rtranif0 tranif1 rtranif1"
+ )
+ gateInstance = (
+ Optional(Group(identifier + Optional(range)))
+ + LPAR
+ + Group(delimitedList(expr))
+ + RPAR
+ )
+ gateDecl = Group(
+ gateType
+ + Optional(driveStrength)
+ + Optional(delay)
+ + delimitedList(gateInstance)
+ + SEMI
+ )
+
+ udpInstance = Group(
+ Group(identifier + Optional(range | subscrRef))
+ + LPAR
+ + Group(delimitedList(expr))
+ + RPAR
+ )
+ udpInstantiation = Group(
+ identifier
+ - Optional(driveStrength)
+ + Optional(delay)
+ + delimitedList(udpInstance)
+ + SEMI
+ ).setName("udpInstantiation")
+
+ parameterValueAssignment = Group(
+ Literal("#") + LPAR + Group(delimitedList(expr)) + RPAR
+ )
+ namedPortConnection = Group(DOT + identifier + LPAR + expr + RPAR).setName(
+ "namedPortConnection"
+ ) # .setDebug()
+ assert r".\abc (abc )" == namedPortConnection
modulePortConnection = expr | empty
- #~ moduleInstance = Group( Group ( identifier + Optional(range) ) +
- #~ ( delimitedList( modulePortConnection ) |
- #~ delimitedList( namedPortConnection ) ) )
- inst_args = Group( LPAR + (delimitedList( namedPortConnection ) |
- delimitedList( modulePortConnection )) + RPAR).setName("inst_args")
- moduleInstance = Group( Group ( identifier + Optional(range) ) + inst_args ).setName("moduleInstance")#.setDebug()
-
- moduleInstantiation = Group( identifier +
- Optional( parameterValueAssignment ) +
- delimitedList( moduleInstance ).setName("moduleInstanceList") +
- SEMI ).setName("moduleInstantiation")
-
- parameterOverride = Group( "defparam" + delimitedList( paramAssgnmt ) + SEMI )
- task = Group( "task" + identifier + SEMI +
- ZeroOrMore( tfDecl ) +
- stmtOrNull +
- "endtask" )
-
- specparamDecl = Group( "specparam" + delimitedList( paramAssgnmt ) + SEMI )
-
- pathDescr1 = Group( LPAR + subscrIdentifier + "=>" + subscrIdentifier + RPAR )
- pathDescr2 = Group( LPAR + Group( delimitedList( subscrIdentifier ) ) + "*>" +
- Group( delimitedList( subscrIdentifier ) ) + RPAR )
- pathDescr3 = Group( LPAR + Group( delimitedList( subscrIdentifier ) ) + "=>" +
- Group( delimitedList( subscrIdentifier ) ) + RPAR )
- pathDelayValue = Group( ( LPAR + Group( delimitedList( mintypmaxExpr | expr ) ) + RPAR ) |
- mintypmaxExpr |
- expr )
- pathDecl = Group( ( pathDescr1 | pathDescr2 | pathDescr3 ) + EQ + pathDelayValue + SEMI ).setName("pathDecl")
+ # ~ moduleInstance = Group( Group ( identifier + Optional(range) ) +
+ # ~ ( delimitedList( modulePortConnection ) |
+ # ~ delimitedList( namedPortConnection ) ) )
+ inst_args = Group(
+ LPAR
+ + (delimitedList(namedPortConnection) | delimitedList(modulePortConnection))
+ + RPAR
+ ).setName("inst_args")
+ moduleInstance = Group(Group(identifier + Optional(range)) + inst_args).setName(
+ "moduleInstance"
+ ) # .setDebug()
+
+ moduleInstantiation = Group(
+ identifier
+ + Optional(parameterValueAssignment)
+ + delimitedList(moduleInstance).setName("moduleInstanceList")
+ + SEMI
+ ).setName("moduleInstantiation")
+
+ parameterOverride = Group("defparam" + delimitedList(paramAssgnmt) + SEMI)
+ task = Group(
+ "task" + identifier + SEMI + ZeroOrMore(tfDecl) + stmtOrNull + "endtask"
+ )
+
+ specparamDecl = Group("specparam" + delimitedList(paramAssgnmt) + SEMI)
+
+ pathDescr1 = Group(LPAR + subscrIdentifier + "=>" + subscrIdentifier + RPAR)
+ pathDescr2 = Group(
+ LPAR
+ + Group(delimitedList(subscrIdentifier))
+ + "*>"
+ + Group(delimitedList(subscrIdentifier))
+ + RPAR
+ )
+ pathDescr3 = Group(
+ LPAR
+ + Group(delimitedList(subscrIdentifier))
+ + "=>"
+ + Group(delimitedList(subscrIdentifier))
+ + RPAR
+ )
+ pathDelayValue = Group(
+ (LPAR + Group(delimitedList(mintypmaxExpr | expr)) + RPAR)
+ | mintypmaxExpr
+ | expr
+ )
+ pathDecl = Group(
+ (pathDescr1 | pathDescr2 | pathDescr3) + EQ + pathDelayValue + SEMI
+ ).setName("pathDecl")
portConditionExpr = Forward()
portConditionTerm = Optional(unop) + subscrIdentifier
- portConditionExpr << portConditionTerm + Optional( binop + portConditionExpr )
+ portConditionExpr << portConditionTerm + Optional(binop + portConditionExpr)
polarityOp = oneOf("+ -")
levelSensitivePathDecl1 = Group(
- if_ + Group(LPAR + portConditionExpr + RPAR) +
- subscrIdentifier + Optional( polarityOp ) + "=>" + subscrIdentifier + EQ +
- pathDelayValue +
- SEMI )
+ if_
+ + Group(LPAR + portConditionExpr + RPAR)
+ + subscrIdentifier
+ + Optional(polarityOp)
+ + "=>"
+ + subscrIdentifier
+ + EQ
+ + pathDelayValue
+ + SEMI
+ )
levelSensitivePathDecl2 = Group(
- if_ + Group(LPAR + portConditionExpr + RPAR) +
- LPAR + Group( delimitedList( subscrIdentifier ) ) + Optional( polarityOp ) + "*>" +
- Group( delimitedList( subscrIdentifier ) ) + RPAR + EQ +
- pathDelayValue +
- SEMI )
+ if_
+ + Group(LPAR + portConditionExpr + RPAR)
+ + LPAR
+ + Group(delimitedList(subscrIdentifier))
+ + Optional(polarityOp)
+ + "*>"
+ + Group(delimitedList(subscrIdentifier))
+ + RPAR
+ + EQ
+ + pathDelayValue
+ + SEMI
+ )
levelSensitivePathDecl = levelSensitivePathDecl1 | levelSensitivePathDecl2
edgeIdentifier = posedge | negedge
edgeSensitivePathDecl1 = Group(
- Optional( if_ + Group(LPAR + expr + RPAR) ) +
- LPAR + Optional( edgeIdentifier ) +
- subscrIdentifier + "=>" +
- LPAR + subscrIdentifier + Optional( polarityOp ) + COLON + expr + RPAR + RPAR +
- EQ +
- pathDelayValue +
- SEMI )
+ Optional(if_ + Group(LPAR + expr + RPAR))
+ + LPAR
+ + Optional(edgeIdentifier)
+ + subscrIdentifier
+ + "=>"
+ + LPAR
+ + subscrIdentifier
+ + Optional(polarityOp)
+ + COLON
+ + expr
+ + RPAR
+ + RPAR
+ + EQ
+ + pathDelayValue
+ + SEMI
+ )
edgeSensitivePathDecl2 = Group(
- Optional( if_ + Group(LPAR + expr + RPAR) ) +
- LPAR + Optional( edgeIdentifier ) +
- subscrIdentifier + "*>" +
- LPAR + delimitedList( subscrIdentifier ) + Optional( polarityOp ) + COLON + expr + RPAR + RPAR +
- EQ +
- pathDelayValue +
- SEMI )
+ Optional(if_ + Group(LPAR + expr + RPAR))
+ + LPAR
+ + Optional(edgeIdentifier)
+ + subscrIdentifier
+ + "*>"
+ + LPAR
+ + delimitedList(subscrIdentifier)
+ + Optional(polarityOp)
+ + COLON
+ + expr
+ + RPAR
+ + RPAR
+ + EQ
+ + pathDelayValue
+ + SEMI
+ )
edgeSensitivePathDecl = edgeSensitivePathDecl1 | edgeSensitivePathDecl2
edgeDescr = oneOf("01 10 0x x1 1x x0").setName("edgeDescr")
- timCheckEventControl = Group( posedge | negedge | (edge + LBRACK + delimitedList( edgeDescr ) + RBRACK ))
+ timCheckEventControl = Group(
+ posedge | negedge | (edge + LBRACK + delimitedList(edgeDescr) + RBRACK)
+ )
timCheckCond = Forward()
timCondBinop = oneOf("== === != !==")
- timCheckCondTerm = ( expr + timCondBinop + scalarConst ) | ( Optional("~") + expr )
- timCheckCond << ( ( LPAR + timCheckCond + RPAR ) | timCheckCondTerm )
- timCheckEvent = Group( Optional( timCheckEventControl ) +
- subscrIdentifier +
- Optional( "&&&" + timCheckCond ) )
+ timCheckCondTerm = (expr + timCondBinop + scalarConst) | (Optional("~") + expr)
+ timCheckCond << ((LPAR + timCheckCond + RPAR) | timCheckCondTerm)
+ timCheckEvent = Group(
+ Optional(timCheckEventControl)
+ + subscrIdentifier
+ + Optional("&&&" + timCheckCond)
+ )
timCheckLimit = expr
- controlledTimingCheckEvent = Group( timCheckEventControl + subscrIdentifier +
- Optional( "&&&" + timCheckCond ) )
+ controlledTimingCheckEvent = Group(
+ timCheckEventControl + subscrIdentifier + Optional("&&&" + timCheckCond)
+ )
notifyRegister = identifier
- systemTimingCheck1 = Group( "$setup" +
- LPAR + timCheckEvent + COMMA + timCheckEvent + COMMA + timCheckLimit +
- Optional( COMMA + notifyRegister ) + RPAR +
- SEMI )
- systemTimingCheck2 = Group( "$hold" +
- LPAR + timCheckEvent + COMMA + timCheckEvent + COMMA + timCheckLimit +
- Optional( COMMA + notifyRegister ) + RPAR +
- SEMI )
- systemTimingCheck3 = Group( "$period" +
- LPAR + controlledTimingCheckEvent + COMMA + timCheckLimit +
- Optional( COMMA + notifyRegister ) + RPAR +
- SEMI )
- systemTimingCheck4 = Group( "$width" +
- LPAR + controlledTimingCheckEvent + COMMA + timCheckLimit +
- Optional( COMMA + expr + COMMA + notifyRegister ) + RPAR +
- SEMI )
- systemTimingCheck5 = Group( "$skew" +
- LPAR + timCheckEvent + COMMA + timCheckEvent + COMMA + timCheckLimit +
- Optional( COMMA + notifyRegister ) + RPAR +
- SEMI )
- systemTimingCheck6 = Group( "$recovery" +
- LPAR + controlledTimingCheckEvent + COMMA + timCheckEvent + COMMA + timCheckLimit +
- Optional( COMMA + notifyRegister ) + RPAR +
- SEMI )
- systemTimingCheck7 = Group( "$setuphold" +
- LPAR + timCheckEvent + COMMA + timCheckEvent + COMMA + timCheckLimit + COMMA + timCheckLimit +
- Optional( COMMA + notifyRegister ) + RPAR +
- SEMI )
- systemTimingCheck = (FollowedBy('$') + ( systemTimingCheck1 | systemTimingCheck2 | systemTimingCheck3 |
- systemTimingCheck4 | systemTimingCheck5 | systemTimingCheck6 | systemTimingCheck7 )).setName("systemTimingCheck")
- sdpd = if_ + Group(LPAR + expr + RPAR) + \
- ( pathDescr1 | pathDescr2 ) + EQ + pathDelayValue + SEMI
-
- specifyItem = ~Keyword("endspecify") +(
- specparamDecl |
- pathDecl |
- levelSensitivePathDecl |
- edgeSensitivePathDecl |
- systemTimingCheck |
- sdpd
+ systemTimingCheck1 = Group(
+ "$setup"
+ + LPAR
+ + timCheckEvent
+ + COMMA
+ + timCheckEvent
+ + COMMA
+ + timCheckLimit
+ + Optional(COMMA + notifyRegister)
+ + RPAR
+ + SEMI
+ )
+ systemTimingCheck2 = Group(
+ "$hold"
+ + LPAR
+ + timCheckEvent
+ + COMMA
+ + timCheckEvent
+ + COMMA
+ + timCheckLimit
+ + Optional(COMMA + notifyRegister)
+ + RPAR
+ + SEMI
+ )
+ systemTimingCheck3 = Group(
+ "$period"
+ + LPAR
+ + controlledTimingCheckEvent
+ + COMMA
+ + timCheckLimit
+ + Optional(COMMA + notifyRegister)
+ + RPAR
+ + SEMI
+ )
+ systemTimingCheck4 = Group(
+ "$width"
+ + LPAR
+ + controlledTimingCheckEvent
+ + COMMA
+ + timCheckLimit
+ + Optional(COMMA + expr + COMMA + notifyRegister)
+ + RPAR
+ + SEMI
+ )
+ systemTimingCheck5 = Group(
+ "$skew"
+ + LPAR
+ + timCheckEvent
+ + COMMA
+ + timCheckEvent
+ + COMMA
+ + timCheckLimit
+ + Optional(COMMA + notifyRegister)
+ + RPAR
+ + SEMI
+ )
+ systemTimingCheck6 = Group(
+ "$recovery"
+ + LPAR
+ + controlledTimingCheckEvent
+ + COMMA
+ + timCheckEvent
+ + COMMA
+ + timCheckLimit
+ + Optional(COMMA + notifyRegister)
+ + RPAR
+ + SEMI
+ )
+ systemTimingCheck7 = Group(
+ "$setuphold"
+ + LPAR
+ + timCheckEvent
+ + COMMA
+ + timCheckEvent
+ + COMMA
+ + timCheckLimit
+ + COMMA
+ + timCheckLimit
+ + Optional(COMMA + notifyRegister)
+ + RPAR
+ + SEMI
+ )
+ systemTimingCheck = (
+ FollowedBy("$")
+ + (
+ systemTimingCheck1
+ | systemTimingCheck2
+ | systemTimingCheck3
+ | systemTimingCheck4
+ | systemTimingCheck5
+ | systemTimingCheck6
+ | systemTimingCheck7
)
+ ).setName("systemTimingCheck")
+ sdpd = (
+ if_
+ + Group(LPAR + expr + RPAR)
+ + (pathDescr1 | pathDescr2)
+ + EQ
+ + pathDelayValue
+ + SEMI
+ )
+
+ specifyItem = ~Keyword("endspecify") + (
+ specparamDecl
+ | pathDecl
+ | levelSensitivePathDecl
+ | edgeSensitivePathDecl
+ | systemTimingCheck
+ | sdpd
+ )
"""
x::= <specparam_declaration>
x||= <path_declaration>
x||= <system_timing_check>
x||= <sdpd>
"""
- specifyBlock = Group( "specify" + ZeroOrMore( specifyItem ) + "endspecify" ).setName("specifyBlock")
+ specifyBlock = Group(
+ "specify" + ZeroOrMore(specifyItem) + "endspecify"
+ ).setName("specifyBlock")
moduleItem = ~Keyword("endmodule") + (
- parameterDecl |
- inputDecl |
- outputDecl |
- inoutDecl |
- regDecl |
- netDecl3 |
- netDecl1 |
- netDecl2 |
- timeDecl |
- integerDecl |
- realDecl |
- eventDecl |
- gateDecl |
- parameterOverride |
- continuousAssign |
- specifyBlock |
- initialStmt |
- alwaysStmt |
- task |
- functionDecl |
+ parameterDecl
+ | inputDecl
+ | outputDecl
+ | inoutDecl
+ | regDecl
+ | netDecl3
+ | netDecl1
+ | netDecl2
+ | timeDecl
+ | integerDecl
+ | realDecl
+ | eventDecl
+ | gateDecl
+ | parameterOverride
+ | continuousAssign
+ | specifyBlock
+ | initialStmt
+ | alwaysStmt
+ | task
+ | functionDecl
+ |
# these have to be at the end - they start with identifiers
- moduleInstantiation |
- udpInstantiation
- )
+ moduleInstantiation
+ | udpInstantiation
+ )
""" All possible moduleItems, from Verilog grammar spec
x::= <parameter_declaration>
x||= <input_declaration>
x||= <function>
"""
portRef = subscrIdentifier
- portExpr = portRef | Group( LBRACE + delimitedList( portRef ) + RBRACE )
- port = portExpr | Group( ( DOT + identifier + LPAR + portExpr + RPAR ) )
-
- moduleHdr = Group ( oneOf("module macromodule") + identifier +
- Optional( LPAR + Group( Optional( delimitedList(
- Group(oneOf("input output") +
- (netDecl1Arg | netDecl2Arg | netDecl3Arg) ) |
- port ) ) ) +
- RPAR ) + SEMI ).setName("moduleHdr")
+ portExpr = portRef | Group(LBRACE + delimitedList(portRef) + RBRACE)
+ port = portExpr | Group(DOT + identifier + LPAR + portExpr + RPAR)
+
+ moduleHdr = Group(
+ oneOf("module macromodule")
+ + identifier
+ + Optional(
+ LPAR
+ + Group(
+ Optional(
+ delimitedList(
+ Group(
+ oneOf("input output")
+ + (netDecl1Arg | netDecl2Arg | netDecl3Arg)
+ )
+ | port
+ )
+ )
+ )
+ + RPAR
+ )
+ + SEMI
+ ).setName("moduleHdr")
- module = Group( moduleHdr +
- Group( ZeroOrMore( moduleItem ) ) +
- "endmodule" ).setName("module")#.setDebug()
+ module = Group(moduleHdr + Group(ZeroOrMore(moduleItem)) + "endmodule").setName(
+ "module"
+ ) # .setDebug()
udpDecl = outputDecl | inputDecl | regDecl
- #~ udpInitVal = oneOf("1'b0 1'b1 1'bx 1'bX 1'B0 1'B1 1'Bx 1'BX 1 0 x X")
+ # ~ udpInitVal = oneOf("1'b0 1'b1 1'bx 1'bX 1'B0 1'B1 1'Bx 1'BX 1 0 x X")
udpInitVal = (Regex("1'[bB][01xX]") | Regex("[01xX]")).setName("udpInitVal")
- udpInitialStmt = Group( "initial" +
- identifier + EQ + udpInitVal + SEMI ).setName("udpInitialStmt")
+ udpInitialStmt = Group("initial" + identifier + EQ + udpInitVal + SEMI).setName(
+ "udpInitialStmt"
+ )
levelSymbol = oneOf("0 1 x X ? b B")
- levelInputList = Group( OneOrMore( levelSymbol ).setName("levelInpList") )
+ levelInputList = Group(OneOrMore(levelSymbol).setName("levelInpList"))
outputSymbol = oneOf("0 1 x X")
- combEntry = Group( levelInputList + COLON + outputSymbol + SEMI )
+ combEntry = Group(levelInputList + COLON + outputSymbol + SEMI)
edgeSymbol = oneOf("r R f F p P n N *")
- edge = Group( LPAR + levelSymbol + levelSymbol + RPAR ) | \
- Group( edgeSymbol )
- edgeInputList = Group( ZeroOrMore( levelSymbol ) + edge + ZeroOrMore( levelSymbol ) )
+ edge = Group(LPAR + levelSymbol + levelSymbol + RPAR) | Group(edgeSymbol)
+ edgeInputList = Group(ZeroOrMore(levelSymbol) + edge + ZeroOrMore(levelSymbol))
inputList = levelInputList | edgeInputList
- seqEntry = Group( inputList + COLON + levelSymbol + COLON + ( outputSymbol | "-" ) + SEMI ).setName("seqEntry")
- udpTableDefn = Group( "table" +
- OneOrMore( combEntry | seqEntry ) +
- "endtable" ).setName("table")
+ seqEntry = Group(
+ inputList + COLON + levelSymbol + COLON + (outputSymbol | "-") + SEMI
+ ).setName("seqEntry")
+ udpTableDefn = Group(
+ "table" + OneOrMore(combEntry | seqEntry) + "endtable"
+ ).setName("table")
"""
<UDP>
<table_definition>
endprimitive
"""
- udp = Group( "primitive" + identifier +
- LPAR + Group( delimitedList( identifier ) ) + RPAR + SEMI +
- OneOrMore( udpDecl ) +
- Optional( udpInitialStmt ) +
- udpTableDefn +
- "endprimitive" )
-
- verilogbnf = OneOrMore( module | udp ) + StringEnd()
-
- verilogbnf.ignore( cppStyleComment )
- verilogbnf.ignore( compilerDirective )
+ udp = Group(
+ "primitive"
+ + identifier
+ + LPAR
+ + Group(delimitedList(identifier))
+ + RPAR
+ + SEMI
+ + OneOrMore(udpDecl)
+ + Optional(udpInitialStmt)
+ + udpTableDefn
+ + "endprimitive"
+ )
+
+ verilogbnf = OneOrMore(module | udp) + StringEnd()
+
+ verilogbnf.ignore(cppStyleComment)
+ verilogbnf.ignore(compilerDirective)
return verilogbnf
-def test( strng ):
+def test(strng):
tokens = []
try:
- tokens = Verilog_BNF().parseString( strng )
+ tokens = Verilog_BNF().parseString(strng)
except ParseException as err:
print(err.line)
- print(" "*(err.column-1) + "^")
+ print(" " * (err.column - 1) + "^")
print(err)
return tokens
-#~ if __name__ == "__main__":
-if 0:
- import pprint
- toptest = """
- module TOP( in, out );
- input [7:0] in;
- output [5:0] out;
- COUNT_BITS8 count_bits( .IN( in ), .C( out ) );
- endmodule"""
- pprint.pprint( test(toptest).asList() )
-
-else:
+if __name__ == "__main__":
+
def main():
+ import sys
+
+ sys.setrecursionlimit(5000)
print("Verilog parser test (V %s)" % __version__)
print(" - using pyparsing version", pyparsing.__version__)
print(" - using Python version", sys.version)
- if packratOn: print(" - using packrat parsing")
+ if packratOn:
+ print(" - using packrat parsing")
print()
import os
failCount = 0
Verilog_BNF()
numlines = 0
- startTime = time.clock()
+ startTime = time.time()
fileDir = "verilog"
- #~ fileDir = "verilog/new"
- #~ fileDir = "verilog/new2"
- #~ fileDir = "verilog/new3"
+ # ~ fileDir = "verilog/new2"
+ # ~ fileDir = "verilog/new3"
allFiles = [f for f in os.listdir(fileDir) if f.endswith(".v")]
- #~ allFiles = [ "list_path_delays_test.v" ]
- #~ allFiles = [ "escapedIdent.v" ]
- #~ allFiles = filter( lambda f : f.startswith("a") and f.endswith(".v"), os.listdir(fileDir) )
- #~ allFiles = filter( lambda f : f.startswith("c") and f.endswith(".v"), os.listdir(fileDir) )
- #~ allFiles = [ "ff.v" ]
+ # ~ allFiles = [ "list_path_delays_test.v" ]
+ # ~ allFiles = [ "escapedIdent.v" ]
+ # ~ allFiles = filter( lambda f : f.startswith("a") and f.endswith(".v"), os.listdir(fileDir) )
+ # ~ allFiles = filter( lambda f : f.startswith("c") and f.endswith(".v"), os.listdir(fileDir) )
+ # ~ allFiles = [ "ff.v" ]
- pp = pprint.PrettyPrinter( indent=2 )
+ pp = pprint.PrettyPrinter(indent=2)
totalTime = 0
for vfile in allFiles:
gc.collect()
- fnam = fileDir + "/"+vfile
+ fnam = fileDir + "/" + vfile
infile = open(fnam)
filelines = infile.readlines()
infile.close()
- print(fnam, len(filelines), end=' ')
+ print(fnam, len(filelines), end=" ")
numlines += len(filelines)
teststr = "".join(filelines)
- time1 = time.clock()
- tokens = test( teststr )
- time2 = time.clock()
- elapsed = time2-time1
+ time1 = time.time()
+ tokens = test(teststr)
+ time2 = time.time()
+ elapsed = time2 - time1
totalTime += elapsed
- if ( len( tokens ) ):
+ if len(tokens):
print("OK", elapsed)
- #~ print "tokens="
- #~ pp.pprint( tokens.asList() )
- #~ print
ofnam = fileDir + "/parseOutput/" + vfile + ".parsed.txt"
- outfile = open(ofnam,"w")
- outfile.write( teststr )
- outfile.write("\n")
- outfile.write("\n")
- outfile.write(pp.pformat(tokens.asList()))
- outfile.write("\n")
- outfile.close()
+ with open(ofnam, "w") as outfile:
+ outfile.write(teststr)
+ outfile.write("\n\n")
+ outfile.write(pp.pformat(tokens.asList()))
+ outfile.write("\n")
else:
print("failed", elapsed)
failCount += 1
- for i,line in enumerate(filelines,1):
- print("%4d: %s" % (i,line.rstrip()))
- endTime = time.clock()
+ for i, line in enumerate(filelines, 1):
+ print("%4d: %s" % (i, line.rstrip()))
+ endTime = time.time()
print("Total parse time:", totalTime)
print("Total source lines:", numlines)
- print("Average lines/sec:", ( "%.1f" % (float(numlines)/(totalTime+.05 ) ) ))
+ print("Average lines/sec:", ("%.1f" % (float(numlines) / (totalTime + 0.05))))
if failCount:
print("FAIL - %d files failed to parse" % failCount)
else:
return 0
- #~ from line_profiler import LineProfiler
- #~ from pyparsing import ParseResults
- #~ lp = LineProfiler(ParseResults.__init__)
-
main()
-
- #~ lp.print_stats()
- #~ import hotshot
- #~ p = hotshot.Profile("vparse.prof",1,1)
- #~ p.start()
- #~ main()
- #~ p.stop()
- #~ p.close()
from operator import mul
from functools import reduce
+
def makeLit(s, val):
ret = pp.CaselessLiteral(s)
return ret.setParseAction(pp.replaceWith(val))
+
unitDefinitions = [
- ("zero", 0),
- ("oh", 0),
- ("zip", 0),
- ("zilch", 0),
- ("nada", 0),
- ("bupkis", 0),
- ("one", 1),
- ("two", 2),
- ("three", 3),
- ("four", 4),
- ("five", 5),
- ("six", 6),
- ("seven", 7),
- ("eight", 8),
- ("nine", 9),
- ("ten", 10),
- ("eleven", 11),
- ("twelve", 12),
- ("thirteen", 13),
- ("fourteen", 14),
- ("fifteen", 15),
- ("sixteen", 16),
+ ("zero", 0),
+ ("oh", 0),
+ ("zip", 0),
+ ("zilch", 0),
+ ("nada", 0),
+ ("bupkis", 0),
+ ("one", 1),
+ ("two", 2),
+ ("three", 3),
+ ("four", 4),
+ ("five", 5),
+ ("six", 6),
+ ("seven", 7),
+ ("eight", 8),
+ ("nine", 9),
+ ("ten", 10),
+ ("eleven", 11),
+ ("twelve", 12),
+ ("thirteen", 13),
+ ("fourteen", 14),
+ ("fifteen", 15),
+ ("sixteen", 16),
("seventeen", 17),
- ("eighteen", 18),
- ("nineteen", 19),
- ]
-units = pp.MatchFirst(makeLit(s,v) for s,v in sorted(unitDefinitions, key=lambda d: -len(d[0])))
+ ("eighteen", 18),
+ ("nineteen", 19),
+]
+units = pp.MatchFirst(
+ makeLit(s, v) for s, v in sorted(unitDefinitions, key=lambda d: -len(d[0]))
+)
tensDefinitions = [
- ("ten", 10),
- ("twenty", 20),
- ("thirty", 30),
- ("forty", 40),
- ("fourty", 40), # for the spelling-challenged...
- ("fifty", 50),
- ("sixty", 60),
+ ("ten", 10),
+ ("twenty", 20),
+ ("thirty", 30),
+ ("forty", 40),
+ ("fourty", 40), # for the spelling-challenged...
+ ("fifty", 50),
+ ("sixty", 60),
("seventy", 70),
- ("eighty", 80),
- ("ninety", 90),
- ]
-tens = pp.MatchFirst(makeLit(s,v) for s,v in tensDefinitions)
+ ("eighty", 80),
+ ("ninety", 90),
+]
+tens = pp.MatchFirst(makeLit(s, v) for s, v in tensDefinitions)
hundreds = makeLit("hundred", 100)
majorDefinitions = [
- ("thousand", int(1e3)),
- ("million", int(1e6)),
- ("billion", int(1e9)),
- ("trillion", int(1e12)),
+ ("thousand", int(1e3)),
+ ("million", int(1e6)),
+ ("billion", int(1e9)),
+ ("trillion", int(1e12)),
("quadrillion", int(1e15)),
("quintillion", int(1e18)),
- ]
-mag = pp.MatchFirst(makeLit(s,v) for s,v in majorDefinitions)
-
-wordprod = lambda t: reduce(mul,t)
-numPart = ((((units + pp.Optional(hundreds)).setParseAction(wordprod)
- + pp.Optional(tens)
- ).setParseAction(sum)
- ^ tens)
- + pp.Optional(units)
- ).setParseAction(sum)
-numWords = ((numPart + pp.Optional(mag)).setParseAction(wordprod)[1, ...]).setParseAction(sum)
+]
+mag = pp.MatchFirst(makeLit(s, v) for s, v in majorDefinitions)
+
+wordprod = lambda t: reduce(mul, t)
+numPart = (
+ (
+ (
+ (units + pp.Optional(hundreds)).setParseAction(wordprod) + pp.Optional(tens)
+ ).setParseAction(sum)
+ ^ tens
+ )
+ + pp.Optional(units)
+).setParseAction(sum)
+numWords = (
+ (numPart + pp.Optional(mag)).setParseAction(wordprod)[1, ...]
+).setParseAction(sum)
numWords.setName("num word parser")
numWords.ignore(pp.Literal("-"))
"""
# use '| ...' to indicate "if omitted, skip to next" logic
-test_expr = (numWords('result') | ...) + ',' + (pp.pyparsing_common.integer('expected') | 'None')
+test_expr = (
+ (numWords("result") | ...)
+ + ","
+ + (pp.pyparsing_common.integer("expected") | "None")
+)
+
def verify_result(t):
- if '_skipped' in t:
- t['pass'] = False
- elif 'expected' in t:
- t['pass'] = t.result == t.expected
+ if "_skipped" in t:
+ t["pass"] = False
+ elif "expected" in t:
+ t["pass"] = t.result == t.expected
+
+
test_expr.addParseAction(verify_result)
test_expr.runTests(tests)
-Metadata-Version: 1.2
+Metadata-Version: 2.1
Name: pyparsing
-Version: 2.4.7
+Version: 3.0.0
Summary: Python parsing module
Home-page: https://github.com/pyparsing/pyparsing/
Author: Paul McGuire
-Author-email: ptmcg@users.sourceforge.net
+Author-email: ptmcg.gm+pyparsing@gmail.com
License: MIT License
Download-URL: https://pypi.org/project/pyparsing/
Description: PyParsing -- A Python Parsing Module
====================================
- |Build Status|
+ |Build Status| |Coverage|
Introduction
============
*[Since first writing this description of pyparsing in late 2003, this
technique for developing parsers has become more widespread, under the
- name Parsing Expression Grammars - PEGs. See more information on PEGs at*
- https://en.wikipedia.org/wiki/Parsing_expression_grammar *.]*
+ name Parsing Expression Grammars - PEGs. See more information on PEGs*
+ `here <https://en.wikipedia.org/wiki/Parsing_expression_grammar>`__
+ *.]*
Here is a program to parse ``"Hello, World!"`` (or any greeting of the form
``"salutation, addressee!"``):
self-explanatory class names, and the use of '+', '|' and '^' operator
definitions.
- The parsed results returned from ``parseString()`` can be accessed as a
+ The parsed results returned from ``parseString()`` is a collection of type
+ ``ParseResults``, which can be accessed as a
nested list, a dictionary, or an object with named attributes.
The pyparsing module handles some of the problems that are typically
=============
There are many examples in the online docstrings of the classes
- and methods in pyparsing. You can find them compiled into online docs
- at https://pyparsing-docs.readthedocs.io/en/latest/. Additional
+ and methods in pyparsing. You can find them compiled into `online docs <https://pyparsing-docs.readthedocs.io/en/latest/>`__. Additional
documentation resources and project info are listed in the online
- GitHub wiki, at https://github.com/pyparsing/pyparsing/wiki. An
- entire directory of examples is at
- https://github.com/pyparsing/pyparsing/tree/master/examples.
+ `GitHub wiki <https://github.com/pyparsing/pyparsing/wiki>`__. An
+ entire directory of examples can be found `here <https://github.com/pyparsing/pyparsing/tree/master/examples>`__.
License
=======
- MIT License. See header of pyparsing.py
+ MIT License. See header of the `pyparsing.py <https://github.com/pyparsing/pyparsing/blob/master/pyparsing/__init__.py#L1-L23>`__ file.
History
=======
- See CHANGES file.
+ See `CHANGES <https://github.com/pyparsing/pyparsing/blob/master/CHANGES>`__ file.
- .. |Build Status| image:: https://travis-ci.org/pyparsing/pyparsing.svg?branch=master
- :target: https://travis-ci.org/pyparsing/pyparsing
+ .. |Build Status| image:: https://travis-ci.com/pyparsing/pyparsing.svg?branch=master
+ :target: https://travis-ci.com/pyparsing/pyparsing
+ .. |Coverage| image:: https://codecov.io/gh/pyparsing/pyparsing/branch/master/graph/badge.svg
+ :target: https://codecov.io/gh/pyparsing/pyparsing
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 2
-Classifier: Programming Language :: Python :: 2.6
-Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.3
-Classifier: Programming Language :: Python :: 3.4
-Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
-Requires-Python: >=2.6, !=3.0.*, !=3.1.*, !=3.2.*
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Requires-Python: >=3.6
+Description-Content-Type: text/x-rst
+Provides-Extra: diagrams
LICENSE
MANIFEST.in
README.rst
-pyparsing.py
-setup.cfg
setup.py
-simple_unit_tests.py
-unitTests.py
+tox.ini
docs/CODE_OF_CONDUCT.rst
docs/HowToUsePyparsing.rst
docs/Makefile
docs/index.rst
docs/modules.rst
docs/pyparsing.rst
-docs/_static/pyparsingClassDiagram.jpg
-docs/_static/pyparsingClassDiagram.png
+docs/pyparsing_class_diagrm.puml
+docs/whats_new_in_3_0_0.rst
+docs/_static/pyparsingClassDiagram_1.5.2.jpg
+docs/_static/pyparsingClassDiagram_3.0.0.jpg
+docs/_static/sql_railroad.html
examples/0README.html
examples/AcManForm.dfm
examples/LAparser.py
examples/antlr_grammar.py
examples/antlr_grammar_tests.py
examples/apicheck.py
+examples/bigquery_view_parser.py
+examples/booleansearchparser.py
examples/btpyparse.py
examples/builtin_parse_action_demo.py
examples/cLibHeader.py
examples/commasep.py
examples/configParse.py
examples/cpp_enum_parser.py
+examples/cuneiform_python.py
examples/datetimeParseActions.py
examples/decaf_parser.py
examples/delta_time.py
examples/idlParse.py
examples/include_preprocessor.py
examples/indentedGrammarExample.py
+examples/indented_block_example.py
examples/invRegex.py
examples/javascript_grammar.g
examples/jsonParser.py
+examples/left_recursion.py
examples/linenoExample.py
-examples/list1.py
examples/listAllMatches.py
+examples/lua_parser.py
examples/lucene_grammar.py
examples/macroExpander.py
+examples/make_diagram.py
examples/matchPreviousDemo.py
examples/mozilla.ics
examples/mozillaCalendarParser.py
examples/nested.py
examples/nested_markup.py
+examples/number_words.py
examples/numerics.py
examples/oc.py
-examples/parseListString.py
+examples/one_to_ninety_nine.py
examples/parsePythonValue.py
examples/parseResultsSumExample.py
examples/parseTabularData.py
examples/protobuf_parser.py
examples/pymicko.py
examples/pythonGrammarParser.py
+examples/railroad_diagram_demo.py
examples/rangeCheck.py
examples/readJson.py
examples/removeLineBreaks.py
examples/statemachine/vending_machine.py
examples/statemachine/video_demo.py
examples/statemachine/videostate.pystate
+pyparsing/__init__.py
+pyparsing/actions.py
+pyparsing/common.py
+pyparsing/core.py
+pyparsing/exceptions.py
+pyparsing/helpers.py
+pyparsing/results.py
+pyparsing/testing.py
+pyparsing/unicode.py
+pyparsing/util.py
pyparsing.egg-info/PKG-INFO
pyparsing.egg-info/SOURCES.txt
pyparsing.egg-info/dependency_links.txt
+pyparsing.egg-info/requires.txt
pyparsing.egg-info/top_level.txt
-test/__init__.py
-test/__init__.pyc
-test/jsonParserTests.py
-test/jsonParserTests.pyc
-test/karthik.ini
-test/parsefiletest_input_file.txt
-test/__pycache__/__init__.cpython-35.pyc
-test/__pycache__/__init__.cpython-38.pyc
-test/__pycache__/jsonParserTests.cpython-35.pyc
-test/__pycache__/jsonParserTests.cpython-38.pyc
\ No newline at end of file
+pyparsing/diagram/__init__.py
+pyparsing/diagram/template.jinja2
+tests/README.md
+tests/__init__.py
+tests/json_parser_tests.py
+tests/karthik.ini
+tests/parsefiletest_input_file.txt
+tests/requirements.txt
+tests/test_diagram.py
+tests/test_examples.py
+tests/test_simple_unit.py
+tests/test_unit.py
\ No newline at end of file
--- /dev/null
+
+[diagrams]
+jinja2
+railroad-diagrams
+++ /dev/null
-# -*- coding: utf-8 -*-
-# module pyparsing.py
-#
-# Copyright (c) 2003-2019 Paul T. McGuire
-#
-# Permission is hereby granted, free of charge, to any person obtaining
-# a copy of this software and associated documentation files (the
-# "Software"), to deal in the Software without restriction, including
-# without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and to
-# permit persons to whom the Software is furnished to do so, subject to
-# the following conditions:
-#
-# The above copyright notice and this permission notice shall be
-# included in all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-#
-
-__doc__ = \
-"""
-pyparsing module - Classes and methods to define and execute parsing grammars
-=============================================================================
-
-The pyparsing module is an alternative approach to creating and
-executing simple grammars, vs. the traditional lex/yacc approach, or the
-use of regular expressions. With pyparsing, you don't need to learn
-a new syntax for defining grammars or matching expressions - the parsing
-module provides a library of classes that you use to construct the
-grammar directly in Python.
-
-Here is a program to parse "Hello, World!" (or any greeting of the form
-``"<salutation>, <addressee>!"``), built up using :class:`Word`,
-:class:`Literal`, and :class:`And` elements
-(the :class:`'+'<ParserElement.__add__>` operators create :class:`And` expressions,
-and the strings are auto-converted to :class:`Literal` expressions)::
-
- from pyparsing import Word, alphas
-
- # define grammar of a greeting
- greet = Word(alphas) + "," + Word(alphas) + "!"
-
- hello = "Hello, World!"
- print (hello, "->", greet.parseString(hello))
-
-The program outputs the following::
-
- Hello, World! -> ['Hello', ',', 'World', '!']
-
-The Python representation of the grammar is quite readable, owing to the
-self-explanatory class names, and the use of '+', '|' and '^' operators.
-
-The :class:`ParseResults` object returned from
-:class:`ParserElement.parseString` can be
-accessed as a nested list, a dictionary, or an object with named
-attributes.
-
-The pyparsing module handles some of the problems that are typically
-vexing when writing text parsers:
-
- - extra or missing whitespace (the above program will also handle
- "Hello,World!", "Hello , World !", etc.)
- - quoted strings
- - embedded comments
-
-
-Getting Started -
------------------
-Visit the classes :class:`ParserElement` and :class:`ParseResults` to
-see the base classes that most other pyparsing
-classes inherit from. Use the docstrings for examples of how to:
-
- - construct literal match expressions from :class:`Literal` and
- :class:`CaselessLiteral` classes
- - construct character word-group expressions using the :class:`Word`
- class
- - see how to create repetitive expressions using :class:`ZeroOrMore`
- and :class:`OneOrMore` classes
- - use :class:`'+'<And>`, :class:`'|'<MatchFirst>`, :class:`'^'<Or>`,
- and :class:`'&'<Each>` operators to combine simple expressions into
- more complex ones
- - associate names with your parsed results using
- :class:`ParserElement.setResultsName`
- - access the parsed data, which is returned as a :class:`ParseResults`
- object
- - find some helpful expression short-cuts like :class:`delimitedList`
- and :class:`oneOf`
- - find more useful common expressions in the :class:`pyparsing_common`
- namespace class
-"""
-
-__version__ = "2.4.7"
-__versionTime__ = "30 Mar 2020 00:43 UTC"
-__author__ = "Paul McGuire <ptmcg@users.sourceforge.net>"
-
-import string
-from weakref import ref as wkref
-import copy
-import sys
-import warnings
-import re
-import sre_constants
-import collections
-import pprint
-import traceback
-import types
-from datetime import datetime
-from operator import itemgetter
-import itertools
-from functools import wraps
-from contextlib import contextmanager
-
-try:
- # Python 3
- from itertools import filterfalse
-except ImportError:
- from itertools import ifilterfalse as filterfalse
-
-try:
- from _thread import RLock
-except ImportError:
- from threading import RLock
-
-try:
- # Python 3
- from collections.abc import Iterable
- from collections.abc import MutableMapping, Mapping
-except ImportError:
- # Python 2.7
- from collections import Iterable
- from collections import MutableMapping, Mapping
-
-try:
- from collections import OrderedDict as _OrderedDict
-except ImportError:
- try:
- from ordereddict import OrderedDict as _OrderedDict
- except ImportError:
- _OrderedDict = None
-
-try:
- from types import SimpleNamespace
-except ImportError:
- class SimpleNamespace: pass
-
-# version compatibility configuration
-__compat__ = SimpleNamespace()
-__compat__.__doc__ = """
- A cross-version compatibility configuration for pyparsing features that will be
- released in a future version. By setting values in this configuration to True,
- those features can be enabled in prior versions for compatibility development
- and testing.
-
- - collect_all_And_tokens - flag to enable fix for Issue #63 that fixes erroneous grouping
- of results names when an And expression is nested within an Or or MatchFirst; set to
- True to enable bugfix released in pyparsing 2.3.0, or False to preserve
- pre-2.3.0 handling of named results
-"""
-__compat__.collect_all_And_tokens = True
-
-__diag__ = SimpleNamespace()
-__diag__.__doc__ = """
-Diagnostic configuration (all default to False)
- - warn_multiple_tokens_in_named_alternation - flag to enable warnings when a results
- name is defined on a MatchFirst or Or expression with one or more And subexpressions
- (only warns if __compat__.collect_all_And_tokens is False)
- - warn_ungrouped_named_tokens_in_collection - flag to enable warnings when a results
- name is defined on a containing expression with ungrouped subexpressions that also
- have results names
- - warn_name_set_on_empty_Forward - flag to enable warnings whan a Forward is defined
- with a results name, but has no contents defined
- - warn_on_multiple_string_args_to_oneof - flag to enable warnings whan oneOf is
- incorrectly called with multiple str arguments
- - enable_debug_on_named_expressions - flag to auto-enable debug on all subsequent
- calls to ParserElement.setName()
-"""
-__diag__.warn_multiple_tokens_in_named_alternation = False
-__diag__.warn_ungrouped_named_tokens_in_collection = False
-__diag__.warn_name_set_on_empty_Forward = False
-__diag__.warn_on_multiple_string_args_to_oneof = False
-__diag__.enable_debug_on_named_expressions = False
-__diag__._all_names = [nm for nm in vars(__diag__) if nm.startswith("enable_") or nm.startswith("warn_")]
-
-def _enable_all_warnings():
- __diag__.warn_multiple_tokens_in_named_alternation = True
- __diag__.warn_ungrouped_named_tokens_in_collection = True
- __diag__.warn_name_set_on_empty_Forward = True
- __diag__.warn_on_multiple_string_args_to_oneof = True
-__diag__.enable_all_warnings = _enable_all_warnings
-
-
-__all__ = ['__version__', '__versionTime__', '__author__', '__compat__', '__diag__',
- 'And', 'CaselessKeyword', 'CaselessLiteral', 'CharsNotIn', 'Combine', 'Dict', 'Each', 'Empty',
- 'FollowedBy', 'Forward', 'GoToColumn', 'Group', 'Keyword', 'LineEnd', 'LineStart', 'Literal',
- 'PrecededBy', 'MatchFirst', 'NoMatch', 'NotAny', 'OneOrMore', 'OnlyOnce', 'Optional', 'Or',
- 'ParseBaseException', 'ParseElementEnhance', 'ParseException', 'ParseExpression', 'ParseFatalException',
- 'ParseResults', 'ParseSyntaxException', 'ParserElement', 'QuotedString', 'RecursiveGrammarException',
- 'Regex', 'SkipTo', 'StringEnd', 'StringStart', 'Suppress', 'Token', 'TokenConverter',
- 'White', 'Word', 'WordEnd', 'WordStart', 'ZeroOrMore', 'Char',
- 'alphanums', 'alphas', 'alphas8bit', 'anyCloseTag', 'anyOpenTag', 'cStyleComment', 'col',
- 'commaSeparatedList', 'commonHTMLEntity', 'countedArray', 'cppStyleComment', 'dblQuotedString',
- 'dblSlashComment', 'delimitedList', 'dictOf', 'downcaseTokens', 'empty', 'hexnums',
- 'htmlComment', 'javaStyleComment', 'line', 'lineEnd', 'lineStart', 'lineno',
- 'makeHTMLTags', 'makeXMLTags', 'matchOnlyAtCol', 'matchPreviousExpr', 'matchPreviousLiteral',
- 'nestedExpr', 'nullDebugAction', 'nums', 'oneOf', 'opAssoc', 'operatorPrecedence', 'printables',
- 'punc8bit', 'pythonStyleComment', 'quotedString', 'removeQuotes', 'replaceHTMLEntity',
- 'replaceWith', 'restOfLine', 'sglQuotedString', 'srange', 'stringEnd',
- 'stringStart', 'traceParseAction', 'unicodeString', 'upcaseTokens', 'withAttribute',
- 'indentedBlock', 'originalTextFor', 'ungroup', 'infixNotation', 'locatedExpr', 'withClass',
- 'CloseMatch', 'tokenMap', 'pyparsing_common', 'pyparsing_unicode', 'unicode_set',
- 'conditionAsParseAction', 're',
- ]
-
-system_version = tuple(sys.version_info)[:3]
-PY_3 = system_version[0] == 3
-if PY_3:
- _MAX_INT = sys.maxsize
- basestring = str
- unichr = chr
- unicode = str
- _ustr = str
-
- # build list of single arg builtins, that can be used as parse actions
- singleArgBuiltins = [sum, len, sorted, reversed, list, tuple, set, any, all, min, max]
-
-else:
- _MAX_INT = sys.maxint
- range = xrange
-
- def _ustr(obj):
- """Drop-in replacement for str(obj) that tries to be Unicode
- friendly. It first tries str(obj). If that fails with
- a UnicodeEncodeError, then it tries unicode(obj). It then
- < returns the unicode object | encodes it with the default
- encoding | ... >.
- """
- if isinstance(obj, unicode):
- return obj
-
- try:
- # If this works, then _ustr(obj) has the same behaviour as str(obj), so
- # it won't break any existing code.
- return str(obj)
-
- except UnicodeEncodeError:
- # Else encode it
- ret = unicode(obj).encode(sys.getdefaultencoding(), 'xmlcharrefreplace')
- xmlcharref = Regex(r'&#\d+;')
- xmlcharref.setParseAction(lambda t: '\\u' + hex(int(t[0][2:-1]))[2:])
- return xmlcharref.transformString(ret)
-
- # build list of single arg builtins, tolerant of Python version, that can be used as parse actions
- singleArgBuiltins = []
- import __builtin__
-
- for fname in "sum len sorted reversed list tuple set any all min max".split():
- try:
- singleArgBuiltins.append(getattr(__builtin__, fname))
- except AttributeError:
- continue
-
-_generatorType = type((y for y in range(1)))
-
-def _xml_escape(data):
- """Escape &, <, >, ", ', etc. in a string of data."""
-
- # ampersand must be replaced first
- from_symbols = '&><"\''
- to_symbols = ('&' + s + ';' for s in "amp gt lt quot apos".split())
- for from_, to_ in zip(from_symbols, to_symbols):
- data = data.replace(from_, to_)
- return data
-
-alphas = string.ascii_uppercase + string.ascii_lowercase
-nums = "0123456789"
-hexnums = nums + "ABCDEFabcdef"
-alphanums = alphas + nums
-_bslash = chr(92)
-printables = "".join(c for c in string.printable if c not in string.whitespace)
-
-
-def conditionAsParseAction(fn, message=None, fatal=False):
- msg = message if message is not None else "failed user-defined condition"
- exc_type = ParseFatalException if fatal else ParseException
- fn = _trim_arity(fn)
-
- @wraps(fn)
- def pa(s, l, t):
- if not bool(fn(s, l, t)):
- raise exc_type(s, l, msg)
-
- return pa
-
-class ParseBaseException(Exception):
- """base exception class for all parsing runtime exceptions"""
- # Performance tuning: we construct a *lot* of these, so keep this
- # constructor as small and fast as possible
- def __init__(self, pstr, loc=0, msg=None, elem=None):
- self.loc = loc
- if msg is None:
- self.msg = pstr
- self.pstr = ""
- else:
- self.msg = msg
- self.pstr = pstr
- self.parserElement = elem
- self.args = (pstr, loc, msg)
-
- @classmethod
- def _from_exception(cls, pe):
- """
- internal factory method to simplify creating one type of ParseException
- from another - avoids having __init__ signature conflicts among subclasses
- """
- return cls(pe.pstr, pe.loc, pe.msg, pe.parserElement)
-
- def __getattr__(self, aname):
- """supported attributes by name are:
- - lineno - returns the line number of the exception text
- - col - returns the column number of the exception text
- - line - returns the line containing the exception text
- """
- if aname == "lineno":
- return lineno(self.loc, self.pstr)
- elif aname in ("col", "column"):
- return col(self.loc, self.pstr)
- elif aname == "line":
- return line(self.loc, self.pstr)
- else:
- raise AttributeError(aname)
-
- def __str__(self):
- if self.pstr:
- if self.loc >= len(self.pstr):
- foundstr = ', found end of text'
- else:
- foundstr = (', found %r' % self.pstr[self.loc:self.loc + 1]).replace(r'\\', '\\')
- else:
- foundstr = ''
- return ("%s%s (at char %d), (line:%d, col:%d)" %
- (self.msg, foundstr, self.loc, self.lineno, self.column))
- def __repr__(self):
- return _ustr(self)
- def markInputline(self, markerString=">!<"):
- """Extracts the exception line from the input string, and marks
- the location of the exception with a special symbol.
- """
- line_str = self.line
- line_column = self.column - 1
- if markerString:
- line_str = "".join((line_str[:line_column],
- markerString, line_str[line_column:]))
- return line_str.strip()
- def __dir__(self):
- return "lineno col line".split() + dir(type(self))
-
-class ParseException(ParseBaseException):
- """
- Exception thrown when parse expressions don't match class;
- supported attributes by name are:
- - lineno - returns the line number of the exception text
- - col - returns the column number of the exception text
- - line - returns the line containing the exception text
-
- Example::
-
- try:
- Word(nums).setName("integer").parseString("ABC")
- except ParseException as pe:
- print(pe)
- print("column: {}".format(pe.col))
-
- prints::
-
- Expected integer (at char 0), (line:1, col:1)
- column: 1
-
- """
-
- @staticmethod
- def explain(exc, depth=16):
- """
- Method to take an exception and translate the Python internal traceback into a list
- of the pyparsing expressions that caused the exception to be raised.
-
- Parameters:
-
- - exc - exception raised during parsing (need not be a ParseException, in support
- of Python exceptions that might be raised in a parse action)
- - depth (default=16) - number of levels back in the stack trace to list expression
- and function names; if None, the full stack trace names will be listed; if 0, only
- the failing input line, marker, and exception string will be shown
-
- Returns a multi-line string listing the ParserElements and/or function names in the
- exception's stack trace.
-
- Note: the diagnostic output will include string representations of the expressions
- that failed to parse. These representations will be more helpful if you use `setName` to
- give identifiable names to your expressions. Otherwise they will use the default string
- forms, which may be cryptic to read.
-
- explain() is only supported under Python 3.
- """
- import inspect
-
- if depth is None:
- depth = sys.getrecursionlimit()
- ret = []
- if isinstance(exc, ParseBaseException):
- ret.append(exc.line)
- ret.append(' ' * (exc.col - 1) + '^')
- ret.append("{0}: {1}".format(type(exc).__name__, exc))
-
- if depth > 0:
- callers = inspect.getinnerframes(exc.__traceback__, context=depth)
- seen = set()
- for i, ff in enumerate(callers[-depth:]):
- frm = ff[0]
-
- f_self = frm.f_locals.get('self', None)
- if isinstance(f_self, ParserElement):
- if frm.f_code.co_name not in ('parseImpl', '_parseNoCache'):
- continue
- if f_self in seen:
- continue
- seen.add(f_self)
-
- self_type = type(f_self)
- ret.append("{0}.{1} - {2}".format(self_type.__module__,
- self_type.__name__,
- f_self))
- elif f_self is not None:
- self_type = type(f_self)
- ret.append("{0}.{1}".format(self_type.__module__,
- self_type.__name__))
- else:
- code = frm.f_code
- if code.co_name in ('wrapper', '<module>'):
- continue
-
- ret.append("{0}".format(code.co_name))
-
- depth -= 1
- if not depth:
- break
-
- return '\n'.join(ret)
-
-
-class ParseFatalException(ParseBaseException):
- """user-throwable exception thrown when inconsistent parse content
- is found; stops all parsing immediately"""
- pass
-
-class ParseSyntaxException(ParseFatalException):
- """just like :class:`ParseFatalException`, but thrown internally
- when an :class:`ErrorStop<And._ErrorStop>` ('-' operator) indicates
- that parsing is to stop immediately because an unbacktrackable
- syntax error has been found.
- """
- pass
-
-#~ class ReparseException(ParseBaseException):
- #~ """Experimental class - parse actions can raise this exception to cause
- #~ pyparsing to reparse the input string:
- #~ - with a modified input string, and/or
- #~ - with a modified start location
- #~ Set the values of the ReparseException in the constructor, and raise the
- #~ exception in a parse action to cause pyparsing to use the new string/location.
- #~ Setting the values as None causes no change to be made.
- #~ """
- #~ def __init_( self, newstring, restartLoc ):
- #~ self.newParseText = newstring
- #~ self.reparseLoc = restartLoc
-
-class RecursiveGrammarException(Exception):
- """exception thrown by :class:`ParserElement.validate` if the
- grammar could be improperly recursive
- """
- def __init__(self, parseElementList):
- self.parseElementTrace = parseElementList
-
- def __str__(self):
- return "RecursiveGrammarException: %s" % self.parseElementTrace
-
-class _ParseResultsWithOffset(object):
- def __init__(self, p1, p2):
- self.tup = (p1, p2)
- def __getitem__(self, i):
- return self.tup[i]
- def __repr__(self):
- return repr(self.tup[0])
- def setOffset(self, i):
- self.tup = (self.tup[0], i)
-
-class ParseResults(object):
- """Structured parse results, to provide multiple means of access to
- the parsed data:
-
- - as a list (``len(results)``)
- - by list index (``results[0], results[1]``, etc.)
- - by attribute (``results.<resultsName>`` - see :class:`ParserElement.setResultsName`)
-
- Example::
-
- integer = Word(nums)
- date_str = (integer.setResultsName("year") + '/'
- + integer.setResultsName("month") + '/'
- + integer.setResultsName("day"))
- # equivalent form:
- # date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
- # parseString returns a ParseResults object
- result = date_str.parseString("1999/12/31")
-
- def test(s, fn=repr):
- print("%s -> %s" % (s, fn(eval(s))))
- test("list(result)")
- test("result[0]")
- test("result['month']")
- test("result.day")
- test("'month' in result")
- test("'minutes' in result")
- test("result.dump()", str)
-
- prints::
-
- list(result) -> ['1999', '/', '12', '/', '31']
- result[0] -> '1999'
- result['month'] -> '12'
- result.day -> '31'
- 'month' in result -> True
- 'minutes' in result -> False
- result.dump() -> ['1999', '/', '12', '/', '31']
- - day: 31
- - month: 12
- - year: 1999
- """
- def __new__(cls, toklist=None, name=None, asList=True, modal=True):
- if isinstance(toklist, cls):
- return toklist
- retobj = object.__new__(cls)
- retobj.__doinit = True
- return retobj
-
- # Performance tuning: we construct a *lot* of these, so keep this
- # constructor as small and fast as possible
- def __init__(self, toklist=None, name=None, asList=True, modal=True, isinstance=isinstance):
- if self.__doinit:
- self.__doinit = False
- self.__name = None
- self.__parent = None
- self.__accumNames = {}
- self.__asList = asList
- self.__modal = modal
- if toklist is None:
- toklist = []
- if isinstance(toklist, list):
- self.__toklist = toklist[:]
- elif isinstance(toklist, _generatorType):
- self.__toklist = list(toklist)
- else:
- self.__toklist = [toklist]
- self.__tokdict = dict()
-
- if name is not None and name:
- if not modal:
- self.__accumNames[name] = 0
- if isinstance(name, int):
- name = _ustr(name) # will always return a str, but use _ustr for consistency
- self.__name = name
- if not (isinstance(toklist, (type(None), basestring, list)) and toklist in (None, '', [])):
- if isinstance(toklist, basestring):
- toklist = [toklist]
- if asList:
- if isinstance(toklist, ParseResults):
- self[name] = _ParseResultsWithOffset(ParseResults(toklist.__toklist), 0)
- else:
- self[name] = _ParseResultsWithOffset(ParseResults(toklist[0]), 0)
- self[name].__name = name
- else:
- try:
- self[name] = toklist[0]
- except (KeyError, TypeError, IndexError):
- self[name] = toklist
-
- def __getitem__(self, i):
- if isinstance(i, (int, slice)):
- return self.__toklist[i]
- else:
- if i not in self.__accumNames:
- return self.__tokdict[i][-1][0]
- else:
- return ParseResults([v[0] for v in self.__tokdict[i]])
-
- def __setitem__(self, k, v, isinstance=isinstance):
- if isinstance(v, _ParseResultsWithOffset):
- self.__tokdict[k] = self.__tokdict.get(k, list()) + [v]
- sub = v[0]
- elif isinstance(k, (int, slice)):
- self.__toklist[k] = v
- sub = v
- else:
- self.__tokdict[k] = self.__tokdict.get(k, list()) + [_ParseResultsWithOffset(v, 0)]
- sub = v
- if isinstance(sub, ParseResults):
- sub.__parent = wkref(self)
-
- def __delitem__(self, i):
- if isinstance(i, (int, slice)):
- mylen = len(self.__toklist)
- del self.__toklist[i]
-
- # convert int to slice
- if isinstance(i, int):
- if i < 0:
- i += mylen
- i = slice(i, i + 1)
- # get removed indices
- removed = list(range(*i.indices(mylen)))
- removed.reverse()
- # fixup indices in token dictionary
- for name, occurrences in self.__tokdict.items():
- for j in removed:
- for k, (value, position) in enumerate(occurrences):
- occurrences[k] = _ParseResultsWithOffset(value, position - (position > j))
- else:
- del self.__tokdict[i]
-
- def __contains__(self, k):
- return k in self.__tokdict
-
- def __len__(self):
- return len(self.__toklist)
-
- def __bool__(self):
- return (not not self.__toklist)
- __nonzero__ = __bool__
-
- def __iter__(self):
- return iter(self.__toklist)
-
- def __reversed__(self):
- return iter(self.__toklist[::-1])
-
- def _iterkeys(self):
- if hasattr(self.__tokdict, "iterkeys"):
- return self.__tokdict.iterkeys()
- else:
- return iter(self.__tokdict)
-
- def _itervalues(self):
- return (self[k] for k in self._iterkeys())
-
- def _iteritems(self):
- return ((k, self[k]) for k in self._iterkeys())
-
- if PY_3:
- keys = _iterkeys
- """Returns an iterator of all named result keys."""
-
- values = _itervalues
- """Returns an iterator of all named result values."""
-
- items = _iteritems
- """Returns an iterator of all named result key-value tuples."""
-
- else:
- iterkeys = _iterkeys
- """Returns an iterator of all named result keys (Python 2.x only)."""
-
- itervalues = _itervalues
- """Returns an iterator of all named result values (Python 2.x only)."""
-
- iteritems = _iteritems
- """Returns an iterator of all named result key-value tuples (Python 2.x only)."""
-
- def keys(self):
- """Returns all named result keys (as a list in Python 2.x, as an iterator in Python 3.x)."""
- return list(self.iterkeys())
-
- def values(self):
- """Returns all named result values (as a list in Python 2.x, as an iterator in Python 3.x)."""
- return list(self.itervalues())
-
- def items(self):
- """Returns all named result key-values (as a list of tuples in Python 2.x, as an iterator in Python 3.x)."""
- return list(self.iteritems())
-
- def haskeys(self):
- """Since keys() returns an iterator, this method is helpful in bypassing
- code that looks for the existence of any defined results names."""
- return bool(self.__tokdict)
-
- def pop(self, *args, **kwargs):
- """
- Removes and returns item at specified index (default= ``last``).
- Supports both ``list`` and ``dict`` semantics for ``pop()``. If
- passed no argument or an integer argument, it will use ``list``
- semantics and pop tokens from the list of parsed tokens. If passed
- a non-integer argument (most likely a string), it will use ``dict``
- semantics and pop the corresponding value from any defined results
- names. A second default return value argument is supported, just as in
- ``dict.pop()``.
-
- Example::
-
- def remove_first(tokens):
- tokens.pop(0)
- print(OneOrMore(Word(nums)).parseString("0 123 321")) # -> ['0', '123', '321']
- print(OneOrMore(Word(nums)).addParseAction(remove_first).parseString("0 123 321")) # -> ['123', '321']
-
- label = Word(alphas)
- patt = label("LABEL") + OneOrMore(Word(nums))
- print(patt.parseString("AAB 123 321").dump())
-
- # Use pop() in a parse action to remove named result (note that corresponding value is not
- # removed from list form of results)
- def remove_LABEL(tokens):
- tokens.pop("LABEL")
- return tokens
- patt.addParseAction(remove_LABEL)
- print(patt.parseString("AAB 123 321").dump())
-
- prints::
-
- ['AAB', '123', '321']
- - LABEL: AAB
-
- ['AAB', '123', '321']
- """
- if not args:
- args = [-1]
- for k, v in kwargs.items():
- if k == 'default':
- args = (args[0], v)
- else:
- raise TypeError("pop() got an unexpected keyword argument '%s'" % k)
- if (isinstance(args[0], int)
- or len(args) == 1
- or args[0] in self):
- index = args[0]
- ret = self[index]
- del self[index]
- return ret
- else:
- defaultvalue = args[1]
- return defaultvalue
-
- def get(self, key, defaultValue=None):
- """
- Returns named result matching the given key, or if there is no
- such name, then returns the given ``defaultValue`` or ``None`` if no
- ``defaultValue`` is specified.
-
- Similar to ``dict.get()``.
-
- Example::
-
- integer = Word(nums)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
- result = date_str.parseString("1999/12/31")
- print(result.get("year")) # -> '1999'
- print(result.get("hour", "not specified")) # -> 'not specified'
- print(result.get("hour")) # -> None
- """
- if key in self:
- return self[key]
- else:
- return defaultValue
-
- def insert(self, index, insStr):
- """
- Inserts new element at location index in the list of parsed tokens.
-
- Similar to ``list.insert()``.
-
- Example::
-
- print(OneOrMore(Word(nums)).parseString("0 123 321")) # -> ['0', '123', '321']
-
- # use a parse action to insert the parse location in the front of the parsed results
- def insert_locn(locn, tokens):
- tokens.insert(0, locn)
- print(OneOrMore(Word(nums)).addParseAction(insert_locn).parseString("0 123 321")) # -> [0, '0', '123', '321']
- """
- self.__toklist.insert(index, insStr)
- # fixup indices in token dictionary
- for name, occurrences in self.__tokdict.items():
- for k, (value, position) in enumerate(occurrences):
- occurrences[k] = _ParseResultsWithOffset(value, position + (position > index))
-
- def append(self, item):
- """
- Add single element to end of ParseResults list of elements.
-
- Example::
-
- print(OneOrMore(Word(nums)).parseString("0 123 321")) # -> ['0', '123', '321']
-
- # use a parse action to compute the sum of the parsed integers, and add it to the end
- def append_sum(tokens):
- tokens.append(sum(map(int, tokens)))
- print(OneOrMore(Word(nums)).addParseAction(append_sum).parseString("0 123 321")) # -> ['0', '123', '321', 444]
- """
- self.__toklist.append(item)
-
- def extend(self, itemseq):
- """
- Add sequence of elements to end of ParseResults list of elements.
-
- Example::
-
- patt = OneOrMore(Word(alphas))
-
- # use a parse action to append the reverse of the matched strings, to make a palindrome
- def make_palindrome(tokens):
- tokens.extend(reversed([t[::-1] for t in tokens]))
- return ''.join(tokens)
- print(patt.addParseAction(make_palindrome).parseString("lskdj sdlkjf lksd")) # -> 'lskdjsdlkjflksddsklfjkldsjdksl'
- """
- if isinstance(itemseq, ParseResults):
- self.__iadd__(itemseq)
- else:
- self.__toklist.extend(itemseq)
-
- def clear(self):
- """
- Clear all elements and results names.
- """
- del self.__toklist[:]
- self.__tokdict.clear()
-
- def __getattr__(self, name):
- try:
- return self[name]
- except KeyError:
- return ""
-
- def __add__(self, other):
- ret = self.copy()
- ret += other
- return ret
-
- def __iadd__(self, other):
- if other.__tokdict:
- offset = len(self.__toklist)
- addoffset = lambda a: offset if a < 0 else a + offset
- otheritems = other.__tokdict.items()
- otherdictitems = [(k, _ParseResultsWithOffset(v[0], addoffset(v[1])))
- for k, vlist in otheritems for v in vlist]
- for k, v in otherdictitems:
- self[k] = v
- if isinstance(v[0], ParseResults):
- v[0].__parent = wkref(self)
-
- self.__toklist += other.__toklist
- self.__accumNames.update(other.__accumNames)
- return self
-
- def __radd__(self, other):
- if isinstance(other, int) and other == 0:
- # useful for merging many ParseResults using sum() builtin
- return self.copy()
- else:
- # this may raise a TypeError - so be it
- return other + self
-
- def __repr__(self):
- return "(%s, %s)" % (repr(self.__toklist), repr(self.__tokdict))
-
- def __str__(self):
- return '[' + ', '.join(_ustr(i) if isinstance(i, ParseResults) else repr(i) for i in self.__toklist) + ']'
-
- def _asStringList(self, sep=''):
- out = []
- for item in self.__toklist:
- if out and sep:
- out.append(sep)
- if isinstance(item, ParseResults):
- out += item._asStringList()
- else:
- out.append(_ustr(item))
- return out
-
- def asList(self):
- """
- Returns the parse results as a nested list of matching tokens, all converted to strings.
-
- Example::
-
- patt = OneOrMore(Word(alphas))
- result = patt.parseString("sldkj lsdkj sldkj")
- # even though the result prints in string-like form, it is actually a pyparsing ParseResults
- print(type(result), result) # -> <class 'pyparsing.ParseResults'> ['sldkj', 'lsdkj', 'sldkj']
-
- # Use asList() to create an actual list
- result_list = result.asList()
- print(type(result_list), result_list) # -> <class 'list'> ['sldkj', 'lsdkj', 'sldkj']
- """
- return [res.asList() if isinstance(res, ParseResults) else res for res in self.__toklist]
-
- def asDict(self):
- """
- Returns the named parse results as a nested dictionary.
-
- Example::
-
- integer = Word(nums)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
- result = date_str.parseString('12/31/1999')
- print(type(result), repr(result)) # -> <class 'pyparsing.ParseResults'> (['12', '/', '31', '/', '1999'], {'day': [('1999', 4)], 'year': [('12', 0)], 'month': [('31', 2)]})
-
- result_dict = result.asDict()
- print(type(result_dict), repr(result_dict)) # -> <class 'dict'> {'day': '1999', 'year': '12', 'month': '31'}
-
- # even though a ParseResults supports dict-like access, sometime you just need to have a dict
- import json
- print(json.dumps(result)) # -> Exception: TypeError: ... is not JSON serializable
- print(json.dumps(result.asDict())) # -> {"month": "31", "day": "1999", "year": "12"}
- """
- if PY_3:
- item_fn = self.items
- else:
- item_fn = self.iteritems
-
- def toItem(obj):
- if isinstance(obj, ParseResults):
- if obj.haskeys():
- return obj.asDict()
- else:
- return [toItem(v) for v in obj]
- else:
- return obj
-
- return dict((k, toItem(v)) for k, v in item_fn())
-
- def copy(self):
- """
- Returns a new copy of a :class:`ParseResults` object.
- """
- ret = ParseResults(self.__toklist)
- ret.__tokdict = dict(self.__tokdict.items())
- ret.__parent = self.__parent
- ret.__accumNames.update(self.__accumNames)
- ret.__name = self.__name
- return ret
-
- def asXML(self, doctag=None, namedItemsOnly=False, indent="", formatted=True):
- """
- (Deprecated) Returns the parse results as XML. Tags are created for tokens and lists that have defined results names.
- """
- nl = "\n"
- out = []
- namedItems = dict((v[1], k) for (k, vlist) in self.__tokdict.items()
- for v in vlist)
- nextLevelIndent = indent + " "
-
- # collapse out indents if formatting is not desired
- if not formatted:
- indent = ""
- nextLevelIndent = ""
- nl = ""
-
- selfTag = None
- if doctag is not None:
- selfTag = doctag
- else:
- if self.__name:
- selfTag = self.__name
-
- if not selfTag:
- if namedItemsOnly:
- return ""
- else:
- selfTag = "ITEM"
-
- out += [nl, indent, "<", selfTag, ">"]
-
- for i, res in enumerate(self.__toklist):
- if isinstance(res, ParseResults):
- if i in namedItems:
- out += [res.asXML(namedItems[i],
- namedItemsOnly and doctag is None,
- nextLevelIndent,
- formatted)]
- else:
- out += [res.asXML(None,
- namedItemsOnly and doctag is None,
- nextLevelIndent,
- formatted)]
- else:
- # individual token, see if there is a name for it
- resTag = None
- if i in namedItems:
- resTag = namedItems[i]
- if not resTag:
- if namedItemsOnly:
- continue
- else:
- resTag = "ITEM"
- xmlBodyText = _xml_escape(_ustr(res))
- out += [nl, nextLevelIndent, "<", resTag, ">",
- xmlBodyText,
- "</", resTag, ">"]
-
- out += [nl, indent, "</", selfTag, ">"]
- return "".join(out)
-
- def __lookup(self, sub):
- for k, vlist in self.__tokdict.items():
- for v, loc in vlist:
- if sub is v:
- return k
- return None
-
- def getName(self):
- r"""
- Returns the results name for this token expression. Useful when several
- different expressions might match at a particular location.
-
- Example::
-
- integer = Word(nums)
- ssn_expr = Regex(r"\d\d\d-\d\d-\d\d\d\d")
- house_number_expr = Suppress('#') + Word(nums, alphanums)
- user_data = (Group(house_number_expr)("house_number")
- | Group(ssn_expr)("ssn")
- | Group(integer)("age"))
- user_info = OneOrMore(user_data)
-
- result = user_info.parseString("22 111-22-3333 #221B")
- for item in result:
- print(item.getName(), ':', item[0])
-
- prints::
-
- age : 22
- ssn : 111-22-3333
- house_number : 221B
- """
- if self.__name:
- return self.__name
- elif self.__parent:
- par = self.__parent()
- if par:
- return par.__lookup(self)
- else:
- return None
- elif (len(self) == 1
- and len(self.__tokdict) == 1
- and next(iter(self.__tokdict.values()))[0][1] in (0, -1)):
- return next(iter(self.__tokdict.keys()))
- else:
- return None
-
- def dump(self, indent='', full=True, include_list=True, _depth=0):
- """
- Diagnostic method for listing out the contents of
- a :class:`ParseResults`. Accepts an optional ``indent`` argument so
- that this string can be embedded in a nested display of other data.
-
- Example::
-
- integer = Word(nums)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
- result = date_str.parseString('12/31/1999')
- print(result.dump())
-
- prints::
-
- ['12', '/', '31', '/', '1999']
- - day: 1999
- - month: 31
- - year: 12
- """
- out = []
- NL = '\n'
- if include_list:
- out.append(indent + _ustr(self.asList()))
- else:
- out.append('')
-
- if full:
- if self.haskeys():
- items = sorted((str(k), v) for k, v in self.items())
- for k, v in items:
- if out:
- out.append(NL)
- out.append("%s%s- %s: " % (indent, (' ' * _depth), k))
- if isinstance(v, ParseResults):
- if v:
- out.append(v.dump(indent=indent, full=full, include_list=include_list, _depth=_depth + 1))
- else:
- out.append(_ustr(v))
- else:
- out.append(repr(v))
- elif any(isinstance(vv, ParseResults) for vv in self):
- v = self
- for i, vv in enumerate(v):
- if isinstance(vv, ParseResults):
- out.append("\n%s%s[%d]:\n%s%s%s" % (indent,
- (' ' * (_depth)),
- i,
- indent,
- (' ' * (_depth + 1)),
- vv.dump(indent=indent,
- full=full,
- include_list=include_list,
- _depth=_depth + 1)))
- else:
- out.append("\n%s%s[%d]:\n%s%s%s" % (indent,
- (' ' * (_depth)),
- i,
- indent,
- (' ' * (_depth + 1)),
- _ustr(vv)))
-
- return "".join(out)
-
- def pprint(self, *args, **kwargs):
- """
- Pretty-printer for parsed results as a list, using the
- `pprint <https://docs.python.org/3/library/pprint.html>`_ module.
- Accepts additional positional or keyword args as defined for
- `pprint.pprint <https://docs.python.org/3/library/pprint.html#pprint.pprint>`_ .
-
- Example::
-
- ident = Word(alphas, alphanums)
- num = Word(nums)
- func = Forward()
- term = ident | num | Group('(' + func + ')')
- func <<= ident + Group(Optional(delimitedList(term)))
- result = func.parseString("fna a,b,(fnb c,d,200),100")
- result.pprint(width=40)
-
- prints::
-
- ['fna',
- ['a',
- 'b',
- ['(', 'fnb', ['c', 'd', '200'], ')'],
- '100']]
- """
- pprint.pprint(self.asList(), *args, **kwargs)
-
- # add support for pickle protocol
- def __getstate__(self):
- return (self.__toklist,
- (self.__tokdict.copy(),
- self.__parent is not None and self.__parent() or None,
- self.__accumNames,
- self.__name))
-
- def __setstate__(self, state):
- self.__toklist = state[0]
- self.__tokdict, par, inAccumNames, self.__name = state[1]
- self.__accumNames = {}
- self.__accumNames.update(inAccumNames)
- if par is not None:
- self.__parent = wkref(par)
- else:
- self.__parent = None
-
- def __getnewargs__(self):
- return self.__toklist, self.__name, self.__asList, self.__modal
-
- def __dir__(self):
- return dir(type(self)) + list(self.keys())
-
- @classmethod
- def from_dict(cls, other, name=None):
- """
- Helper classmethod to construct a ParseResults from a dict, preserving the
- name-value relations as results names. If an optional 'name' argument is
- given, a nested ParseResults will be returned
- """
- def is_iterable(obj):
- try:
- iter(obj)
- except Exception:
- return False
- else:
- if PY_3:
- return not isinstance(obj, (str, bytes))
- else:
- return not isinstance(obj, basestring)
-
- ret = cls([])
- for k, v in other.items():
- if isinstance(v, Mapping):
- ret += cls.from_dict(v, name=k)
- else:
- ret += cls([v], name=k, asList=is_iterable(v))
- if name is not None:
- ret = cls([ret], name=name)
- return ret
-
-MutableMapping.register(ParseResults)
-
-def col (loc, strg):
- """Returns current column within a string, counting newlines as line separators.
- The first column is number 1.
-
- Note: the default parsing behavior is to expand tabs in the input string
- before starting the parsing process. See
- :class:`ParserElement.parseString` for more
- information on parsing strings containing ``<TAB>`` s, and suggested
- methods to maintain a consistent view of the parsed string, the parse
- location, and line and column positions within the parsed string.
- """
- s = strg
- return 1 if 0 < loc < len(s) and s[loc-1] == '\n' else loc - s.rfind("\n", 0, loc)
-
-def lineno(loc, strg):
- """Returns current line number within a string, counting newlines as line separators.
- The first line is number 1.
-
- Note - the default parsing behavior is to expand tabs in the input string
- before starting the parsing process. See :class:`ParserElement.parseString`
- for more information on parsing strings containing ``<TAB>`` s, and
- suggested methods to maintain a consistent view of the parsed string, the
- parse location, and line and column positions within the parsed string.
- """
- return strg.count("\n", 0, loc) + 1
-
-def line(loc, strg):
- """Returns the line of text containing loc within a string, counting newlines as line separators.
- """
- lastCR = strg.rfind("\n", 0, loc)
- nextCR = strg.find("\n", loc)
- if nextCR >= 0:
- return strg[lastCR + 1:nextCR]
- else:
- return strg[lastCR + 1:]
-
-def _defaultStartDebugAction(instring, loc, expr):
- print(("Match " + _ustr(expr) + " at loc " + _ustr(loc) + "(%d,%d)" % (lineno(loc, instring), col(loc, instring))))
-
-def _defaultSuccessDebugAction(instring, startloc, endloc, expr, toks):
- print("Matched " + _ustr(expr) + " -> " + str(toks.asList()))
-
-def _defaultExceptionDebugAction(instring, loc, expr, exc):
- print("Exception raised:" + _ustr(exc))
-
-def nullDebugAction(*args):
- """'Do-nothing' debug action, to suppress debugging output during parsing."""
- pass
-
-# Only works on Python 3.x - nonlocal is toxic to Python 2 installs
-#~ 'decorator to trim function calls to match the arity of the target'
-#~ def _trim_arity(func, maxargs=3):
- #~ if func in singleArgBuiltins:
- #~ return lambda s,l,t: func(t)
- #~ limit = 0
- #~ foundArity = False
- #~ def wrapper(*args):
- #~ nonlocal limit,foundArity
- #~ while 1:
- #~ try:
- #~ ret = func(*args[limit:])
- #~ foundArity = True
- #~ return ret
- #~ except TypeError:
- #~ if limit == maxargs or foundArity:
- #~ raise
- #~ limit += 1
- #~ continue
- #~ return wrapper
-
-# this version is Python 2.x-3.x cross-compatible
-'decorator to trim function calls to match the arity of the target'
-def _trim_arity(func, maxargs=2):
- if func in singleArgBuiltins:
- return lambda s, l, t: func(t)
- limit = [0]
- foundArity = [False]
-
- # traceback return data structure changed in Py3.5 - normalize back to plain tuples
- if system_version[:2] >= (3, 5):
- def extract_stack(limit=0):
- # special handling for Python 3.5.0 - extra deep call stack by 1
- offset = -3 if system_version == (3, 5, 0) else -2
- frame_summary = traceback.extract_stack(limit=-offset + limit - 1)[offset]
- return [frame_summary[:2]]
- def extract_tb(tb, limit=0):
- frames = traceback.extract_tb(tb, limit=limit)
- frame_summary = frames[-1]
- return [frame_summary[:2]]
- else:
- extract_stack = traceback.extract_stack
- extract_tb = traceback.extract_tb
-
- # synthesize what would be returned by traceback.extract_stack at the call to
- # user's parse action 'func', so that we don't incur call penalty at parse time
-
- LINE_DIFF = 6
- # IF ANY CODE CHANGES, EVEN JUST COMMENTS OR BLANK LINES, BETWEEN THE NEXT LINE AND
- # THE CALL TO FUNC INSIDE WRAPPER, LINE_DIFF MUST BE MODIFIED!!!!
- this_line = extract_stack(limit=2)[-1]
- pa_call_line_synth = (this_line[0], this_line[1] + LINE_DIFF)
-
- def wrapper(*args):
- while 1:
- try:
- ret = func(*args[limit[0]:])
- foundArity[0] = True
- return ret
- except TypeError:
- # re-raise TypeErrors if they did not come from our arity testing
- if foundArity[0]:
- raise
- else:
- try:
- tb = sys.exc_info()[-1]
- if not extract_tb(tb, limit=2)[-1][:2] == pa_call_line_synth:
- raise
- finally:
- try:
- del tb
- except NameError:
- pass
-
- if limit[0] <= maxargs:
- limit[0] += 1
- continue
- raise
-
- # copy func name to wrapper for sensible debug output
- func_name = "<parse action>"
- try:
- func_name = getattr(func, '__name__',
- getattr(func, '__class__').__name__)
- except Exception:
- func_name = str(func)
- wrapper.__name__ = func_name
-
- return wrapper
-
-
-class ParserElement(object):
- """Abstract base level parser element class."""
- DEFAULT_WHITE_CHARS = " \n\t\r"
- verbose_stacktrace = False
-
- @staticmethod
- def setDefaultWhitespaceChars(chars):
- r"""
- Overrides the default whitespace chars
-
- Example::
-
- # default whitespace chars are space, <TAB> and newline
- OneOrMore(Word(alphas)).parseString("abc def\nghi jkl") # -> ['abc', 'def', 'ghi', 'jkl']
-
- # change to just treat newline as significant
- ParserElement.setDefaultWhitespaceChars(" \t")
- OneOrMore(Word(alphas)).parseString("abc def\nghi jkl") # -> ['abc', 'def']
- """
- ParserElement.DEFAULT_WHITE_CHARS = chars
-
- @staticmethod
- def inlineLiteralsUsing(cls):
- """
- Set class to be used for inclusion of string literals into a parser.
-
- Example::
-
- # default literal class used is Literal
- integer = Word(nums)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
- date_str.parseString("1999/12/31") # -> ['1999', '/', '12', '/', '31']
-
-
- # change to Suppress
- ParserElement.inlineLiteralsUsing(Suppress)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
- date_str.parseString("1999/12/31") # -> ['1999', '12', '31']
- """
- ParserElement._literalStringClass = cls
-
- @classmethod
- def _trim_traceback(cls, tb):
- while tb.tb_next:
- tb = tb.tb_next
- return tb
-
- def __init__(self, savelist=False):
- self.parseAction = list()
- self.failAction = None
- # ~ self.name = "<unknown>" # don't define self.name, let subclasses try/except upcall
- self.strRepr = None
- self.resultsName = None
- self.saveAsList = savelist
- self.skipWhitespace = True
- self.whiteChars = set(ParserElement.DEFAULT_WHITE_CHARS)
- self.copyDefaultWhiteChars = True
- self.mayReturnEmpty = False # used when checking for left-recursion
- self.keepTabs = False
- self.ignoreExprs = list()
- self.debug = False
- self.streamlined = False
- self.mayIndexError = True # used to optimize exception handling for subclasses that don't advance parse index
- self.errmsg = ""
- self.modalResults = True # used to mark results names as modal (report only last) or cumulative (list all)
- self.debugActions = (None, None, None) # custom debug actions
- self.re = None
- self.callPreparse = True # used to avoid redundant calls to preParse
- self.callDuringTry = False
-
- def copy(self):
- """
- Make a copy of this :class:`ParserElement`. Useful for defining
- different parse actions for the same parsing pattern, using copies of
- the original parse element.
-
- Example::
-
- integer = Word(nums).setParseAction(lambda toks: int(toks[0]))
- integerK = integer.copy().addParseAction(lambda toks: toks[0] * 1024) + Suppress("K")
- integerM = integer.copy().addParseAction(lambda toks: toks[0] * 1024 * 1024) + Suppress("M")
-
- print(OneOrMore(integerK | integerM | integer).parseString("5K 100 640K 256M"))
-
- prints::
-
- [5120, 100, 655360, 268435456]
-
- Equivalent form of ``expr.copy()`` is just ``expr()``::
-
- integerM = integer().addParseAction(lambda toks: toks[0] * 1024 * 1024) + Suppress("M")
- """
- cpy = copy.copy(self)
- cpy.parseAction = self.parseAction[:]
- cpy.ignoreExprs = self.ignoreExprs[:]
- if self.copyDefaultWhiteChars:
- cpy.whiteChars = ParserElement.DEFAULT_WHITE_CHARS
- return cpy
-
- def setName(self, name):
- """
- Define name for this expression, makes debugging and exception messages clearer.
-
- Example::
-
- Word(nums).parseString("ABC") # -> Exception: Expected W:(0123...) (at char 0), (line:1, col:1)
- Word(nums).setName("integer").parseString("ABC") # -> Exception: Expected integer (at char 0), (line:1, col:1)
- """
- self.name = name
- self.errmsg = "Expected " + self.name
- if __diag__.enable_debug_on_named_expressions:
- self.setDebug()
- return self
-
- def setResultsName(self, name, listAllMatches=False):
- """
- Define name for referencing matching tokens as a nested attribute
- of the returned parse results.
- NOTE: this returns a *copy* of the original :class:`ParserElement` object;
- this is so that the client can define a basic element, such as an
- integer, and reference it in multiple places with different names.
-
- You can also set results names using the abbreviated syntax,
- ``expr("name")`` in place of ``expr.setResultsName("name")``
- - see :class:`__call__`.
-
- Example::
-
- date_str = (integer.setResultsName("year") + '/'
- + integer.setResultsName("month") + '/'
- + integer.setResultsName("day"))
-
- # equivalent form:
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
- """
- return self._setResultsName(name, listAllMatches)
-
- def _setResultsName(self, name, listAllMatches=False):
- newself = self.copy()
- if name.endswith("*"):
- name = name[:-1]
- listAllMatches = True
- newself.resultsName = name
- newself.modalResults = not listAllMatches
- return newself
-
- def setBreak(self, breakFlag=True):
- """Method to invoke the Python pdb debugger when this element is
- about to be parsed. Set ``breakFlag`` to True to enable, False to
- disable.
- """
- if breakFlag:
- _parseMethod = self._parse
- def breaker(instring, loc, doActions=True, callPreParse=True):
- import pdb
- # this call to pdb.set_trace() is intentional, not a checkin error
- pdb.set_trace()
- return _parseMethod(instring, loc, doActions, callPreParse)
- breaker._originalParseMethod = _parseMethod
- self._parse = breaker
- else:
- if hasattr(self._parse, "_originalParseMethod"):
- self._parse = self._parse._originalParseMethod
- return self
-
- def setParseAction(self, *fns, **kwargs):
- """
- Define one or more actions to perform when successfully matching parse element definition.
- Parse action fn is a callable method with 0-3 arguments, called as ``fn(s, loc, toks)`` ,
- ``fn(loc, toks)`` , ``fn(toks)`` , or just ``fn()`` , where:
-
- - s = the original string being parsed (see note below)
- - loc = the location of the matching substring
- - toks = a list of the matched tokens, packaged as a :class:`ParseResults` object
-
- If the functions in fns modify the tokens, they can return them as the return
- value from fn, and the modified list of tokens will replace the original.
- Otherwise, fn does not need to return any value.
-
- If None is passed as the parse action, all previously added parse actions for this
- expression are cleared.
-
- Optional keyword arguments:
- - callDuringTry = (default= ``False``) indicate if parse action should be run during lookaheads and alternate testing
-
- Note: the default parsing behavior is to expand tabs in the input string
- before starting the parsing process. See :class:`parseString for more
- information on parsing strings containing ``<TAB>`` s, and suggested
- methods to maintain a consistent view of the parsed string, the parse
- location, and line and column positions within the parsed string.
-
- Example::
-
- integer = Word(nums)
- date_str = integer + '/' + integer + '/' + integer
-
- date_str.parseString("1999/12/31") # -> ['1999', '/', '12', '/', '31']
-
- # use parse action to convert to ints at parse time
- integer = Word(nums).setParseAction(lambda toks: int(toks[0]))
- date_str = integer + '/' + integer + '/' + integer
-
- # note that integer fields are now ints, not strings
- date_str.parseString("1999/12/31") # -> [1999, '/', 12, '/', 31]
- """
- if list(fns) == [None,]:
- self.parseAction = []
- else:
- if not all(callable(fn) for fn in fns):
- raise TypeError("parse actions must be callable")
- self.parseAction = list(map(_trim_arity, list(fns)))
- self.callDuringTry = kwargs.get("callDuringTry", False)
- return self
-
- def addParseAction(self, *fns, **kwargs):
- """
- Add one or more parse actions to expression's list of parse actions. See :class:`setParseAction`.
-
- See examples in :class:`copy`.
- """
- self.parseAction += list(map(_trim_arity, list(fns)))
- self.callDuringTry = self.callDuringTry or kwargs.get("callDuringTry", False)
- return self
-
- def addCondition(self, *fns, **kwargs):
- """Add a boolean predicate function to expression's list of parse actions. See
- :class:`setParseAction` for function call signatures. Unlike ``setParseAction``,
- functions passed to ``addCondition`` need to return boolean success/fail of the condition.
-
- Optional keyword arguments:
- - message = define a custom message to be used in the raised exception
- - fatal = if True, will raise ParseFatalException to stop parsing immediately; otherwise will raise ParseException
-
- Example::
-
- integer = Word(nums).setParseAction(lambda toks: int(toks[0]))
- year_int = integer.copy()
- year_int.addCondition(lambda toks: toks[0] >= 2000, message="Only support years 2000 and later")
- date_str = year_int + '/' + integer + '/' + integer
-
- result = date_str.parseString("1999/12/31") # -> Exception: Only support years 2000 and later (at char 0), (line:1, col:1)
- """
- for fn in fns:
- self.parseAction.append(conditionAsParseAction(fn, message=kwargs.get('message'),
- fatal=kwargs.get('fatal', False)))
-
- self.callDuringTry = self.callDuringTry or kwargs.get("callDuringTry", False)
- return self
-
- def setFailAction(self, fn):
- """Define action to perform if parsing fails at this expression.
- Fail acton fn is a callable function that takes the arguments
- ``fn(s, loc, expr, err)`` where:
- - s = string being parsed
- - loc = location where expression match was attempted and failed
- - expr = the parse expression that failed
- - err = the exception thrown
- The function returns no value. It may throw :class:`ParseFatalException`
- if it is desired to stop parsing immediately."""
- self.failAction = fn
- return self
-
- def _skipIgnorables(self, instring, loc):
- exprsFound = True
- while exprsFound:
- exprsFound = False
- for e in self.ignoreExprs:
- try:
- while 1:
- loc, dummy = e._parse(instring, loc)
- exprsFound = True
- except ParseException:
- pass
- return loc
-
- def preParse(self, instring, loc):
- if self.ignoreExprs:
- loc = self._skipIgnorables(instring, loc)
-
- if self.skipWhitespace:
- wt = self.whiteChars
- instrlen = len(instring)
- while loc < instrlen and instring[loc] in wt:
- loc += 1
-
- return loc
-
- def parseImpl(self, instring, loc, doActions=True):
- return loc, []
-
- def postParse(self, instring, loc, tokenlist):
- return tokenlist
-
- # ~ @profile
- def _parseNoCache(self, instring, loc, doActions=True, callPreParse=True):
- TRY, MATCH, FAIL = 0, 1, 2
- debugging = (self.debug) # and doActions)
-
- if debugging or self.failAction:
- # ~ print ("Match", self, "at loc", loc, "(%d, %d)" % (lineno(loc, instring), col(loc, instring)))
- if self.debugActions[TRY]:
- self.debugActions[TRY](instring, loc, self)
- try:
- if callPreParse and self.callPreparse:
- preloc = self.preParse(instring, loc)
- else:
- preloc = loc
- tokensStart = preloc
- if self.mayIndexError or preloc >= len(instring):
- try:
- loc, tokens = self.parseImpl(instring, preloc, doActions)
- except IndexError:
- raise ParseException(instring, len(instring), self.errmsg, self)
- else:
- loc, tokens = self.parseImpl(instring, preloc, doActions)
- except Exception as err:
- # ~ print ("Exception raised:", err)
- if self.debugActions[FAIL]:
- self.debugActions[FAIL](instring, tokensStart, self, err)
- if self.failAction:
- self.failAction(instring, tokensStart, self, err)
- raise
- else:
- if callPreParse and self.callPreparse:
- preloc = self.preParse(instring, loc)
- else:
- preloc = loc
- tokensStart = preloc
- if self.mayIndexError or preloc >= len(instring):
- try:
- loc, tokens = self.parseImpl(instring, preloc, doActions)
- except IndexError:
- raise ParseException(instring, len(instring), self.errmsg, self)
- else:
- loc, tokens = self.parseImpl(instring, preloc, doActions)
-
- tokens = self.postParse(instring, loc, tokens)
-
- retTokens = ParseResults(tokens, self.resultsName, asList=self.saveAsList, modal=self.modalResults)
- if self.parseAction and (doActions or self.callDuringTry):
- if debugging:
- try:
- for fn in self.parseAction:
- try:
- tokens = fn(instring, tokensStart, retTokens)
- except IndexError as parse_action_exc:
- exc = ParseException("exception raised in parse action")
- exc.__cause__ = parse_action_exc
- raise exc
-
- if tokens is not None and tokens is not retTokens:
- retTokens = ParseResults(tokens,
- self.resultsName,
- asList=self.saveAsList and isinstance(tokens, (ParseResults, list)),
- modal=self.modalResults)
- except Exception as err:
- # ~ print "Exception raised in user parse action:", err
- if self.debugActions[FAIL]:
- self.debugActions[FAIL](instring, tokensStart, self, err)
- raise
- else:
- for fn in self.parseAction:
- try:
- tokens = fn(instring, tokensStart, retTokens)
- except IndexError as parse_action_exc:
- exc = ParseException("exception raised in parse action")
- exc.__cause__ = parse_action_exc
- raise exc
-
- if tokens is not None and tokens is not retTokens:
- retTokens = ParseResults(tokens,
- self.resultsName,
- asList=self.saveAsList and isinstance(tokens, (ParseResults, list)),
- modal=self.modalResults)
- if debugging:
- # ~ print ("Matched", self, "->", retTokens.asList())
- if self.debugActions[MATCH]:
- self.debugActions[MATCH](instring, tokensStart, loc, self, retTokens)
-
- return loc, retTokens
-
- def tryParse(self, instring, loc):
- try:
- return self._parse(instring, loc, doActions=False)[0]
- except ParseFatalException:
- raise ParseException(instring, loc, self.errmsg, self)
-
- def canParseNext(self, instring, loc):
- try:
- self.tryParse(instring, loc)
- except (ParseException, IndexError):
- return False
- else:
- return True
-
- class _UnboundedCache(object):
- def __init__(self):
- cache = {}
- self.not_in_cache = not_in_cache = object()
-
- def get(self, key):
- return cache.get(key, not_in_cache)
-
- def set(self, key, value):
- cache[key] = value
-
- def clear(self):
- cache.clear()
-
- def cache_len(self):
- return len(cache)
-
- self.get = types.MethodType(get, self)
- self.set = types.MethodType(set, self)
- self.clear = types.MethodType(clear, self)
- self.__len__ = types.MethodType(cache_len, self)
-
- if _OrderedDict is not None:
- class _FifoCache(object):
- def __init__(self, size):
- self.not_in_cache = not_in_cache = object()
-
- cache = _OrderedDict()
-
- def get(self, key):
- return cache.get(key, not_in_cache)
-
- def set(self, key, value):
- cache[key] = value
- while len(cache) > size:
- try:
- cache.popitem(False)
- except KeyError:
- pass
-
- def clear(self):
- cache.clear()
-
- def cache_len(self):
- return len(cache)
-
- self.get = types.MethodType(get, self)
- self.set = types.MethodType(set, self)
- self.clear = types.MethodType(clear, self)
- self.__len__ = types.MethodType(cache_len, self)
-
- else:
- class _FifoCache(object):
- def __init__(self, size):
- self.not_in_cache = not_in_cache = object()
-
- cache = {}
- key_fifo = collections.deque([], size)
-
- def get(self, key):
- return cache.get(key, not_in_cache)
-
- def set(self, key, value):
- cache[key] = value
- while len(key_fifo) > size:
- cache.pop(key_fifo.popleft(), None)
- key_fifo.append(key)
-
- def clear(self):
- cache.clear()
- key_fifo.clear()
-
- def cache_len(self):
- return len(cache)
-
- self.get = types.MethodType(get, self)
- self.set = types.MethodType(set, self)
- self.clear = types.MethodType(clear, self)
- self.__len__ = types.MethodType(cache_len, self)
-
- # argument cache for optimizing repeated calls when backtracking through recursive expressions
- packrat_cache = {} # this is set later by enabledPackrat(); this is here so that resetCache() doesn't fail
- packrat_cache_lock = RLock()
- packrat_cache_stats = [0, 0]
-
- # this method gets repeatedly called during backtracking with the same arguments -
- # we can cache these arguments and save ourselves the trouble of re-parsing the contained expression
- def _parseCache(self, instring, loc, doActions=True, callPreParse=True):
- HIT, MISS = 0, 1
- lookup = (self, instring, loc, callPreParse, doActions)
- with ParserElement.packrat_cache_lock:
- cache = ParserElement.packrat_cache
- value = cache.get(lookup)
- if value is cache.not_in_cache:
- ParserElement.packrat_cache_stats[MISS] += 1
- try:
- value = self._parseNoCache(instring, loc, doActions, callPreParse)
- except ParseBaseException as pe:
- # cache a copy of the exception, without the traceback
- cache.set(lookup, pe.__class__(*pe.args))
- raise
- else:
- cache.set(lookup, (value[0], value[1].copy()))
- return value
- else:
- ParserElement.packrat_cache_stats[HIT] += 1
- if isinstance(value, Exception):
- raise value
- return value[0], value[1].copy()
-
- _parse = _parseNoCache
-
- @staticmethod
- def resetCache():
- ParserElement.packrat_cache.clear()
- ParserElement.packrat_cache_stats[:] = [0] * len(ParserElement.packrat_cache_stats)
-
- _packratEnabled = False
- @staticmethod
- def enablePackrat(cache_size_limit=128):
- """Enables "packrat" parsing, which adds memoizing to the parsing logic.
- Repeated parse attempts at the same string location (which happens
- often in many complex grammars) can immediately return a cached value,
- instead of re-executing parsing/validating code. Memoizing is done of
- both valid results and parsing exceptions.
-
- Parameters:
-
- - cache_size_limit - (default= ``128``) - if an integer value is provided
- will limit the size of the packrat cache; if None is passed, then
- the cache size will be unbounded; if 0 is passed, the cache will
- be effectively disabled.
-
- This speedup may break existing programs that use parse actions that
- have side-effects. For this reason, packrat parsing is disabled when
- you first import pyparsing. To activate the packrat feature, your
- program must call the class method :class:`ParserElement.enablePackrat`.
- For best results, call ``enablePackrat()`` immediately after
- importing pyparsing.
-
- Example::
-
- import pyparsing
- pyparsing.ParserElement.enablePackrat()
- """
- if not ParserElement._packratEnabled:
- ParserElement._packratEnabled = True
- if cache_size_limit is None:
- ParserElement.packrat_cache = ParserElement._UnboundedCache()
- else:
- ParserElement.packrat_cache = ParserElement._FifoCache(cache_size_limit)
- ParserElement._parse = ParserElement._parseCache
-
- def parseString(self, instring, parseAll=False):
- """
- Execute the parse expression with the given string.
- This is the main interface to the client code, once the complete
- expression has been built.
-
- Returns the parsed data as a :class:`ParseResults` object, which may be
- accessed as a list, or as a dict or object with attributes if the given parser
- includes results names.
-
- If you want the grammar to require that the entire input string be
- successfully parsed, then set ``parseAll`` to True (equivalent to ending
- the grammar with ``StringEnd()``).
-
- Note: ``parseString`` implicitly calls ``expandtabs()`` on the input string,
- in order to report proper column numbers in parse actions.
- If the input string contains tabs and
- the grammar uses parse actions that use the ``loc`` argument to index into the
- string being parsed, you can ensure you have a consistent view of the input
- string by:
-
- - calling ``parseWithTabs`` on your grammar before calling ``parseString``
- (see :class:`parseWithTabs`)
- - define your parse action using the full ``(s, loc, toks)`` signature, and
- reference the input string using the parse action's ``s`` argument
- - explictly expand the tabs in your input string before calling
- ``parseString``
-
- Example::
-
- Word('a').parseString('aaaaabaaa') # -> ['aaaaa']
- Word('a').parseString('aaaaabaaa', parseAll=True) # -> Exception: Expected end of text
- """
- ParserElement.resetCache()
- if not self.streamlined:
- self.streamline()
- # ~ self.saveAsList = True
- for e in self.ignoreExprs:
- e.streamline()
- if not self.keepTabs:
- instring = instring.expandtabs()
- try:
- loc, tokens = self._parse(instring, 0)
- if parseAll:
- loc = self.preParse(instring, loc)
- se = Empty() + StringEnd()
- se._parse(instring, loc)
- except ParseBaseException as exc:
- if ParserElement.verbose_stacktrace:
- raise
- else:
- # catch and re-raise exception from here, clearing out pyparsing internal stack trace
- if getattr(exc, '__traceback__', None) is not None:
- exc.__traceback__ = self._trim_traceback(exc.__traceback__)
- raise exc
- else:
- return tokens
-
- def scanString(self, instring, maxMatches=_MAX_INT, overlap=False):
- """
- Scan the input string for expression matches. Each match will return the
- matching tokens, start location, and end location. May be called with optional
- ``maxMatches`` argument, to clip scanning after 'n' matches are found. If
- ``overlap`` is specified, then overlapping matches will be reported.
-
- Note that the start and end locations are reported relative to the string
- being parsed. See :class:`parseString` for more information on parsing
- strings with embedded tabs.
-
- Example::
-
- source = "sldjf123lsdjjkf345sldkjf879lkjsfd987"
- print(source)
- for tokens, start, end in Word(alphas).scanString(source):
- print(' '*start + '^'*(end-start))
- print(' '*start + tokens[0])
-
- prints::
-
- sldjf123lsdjjkf345sldkjf879lkjsfd987
- ^^^^^
- sldjf
- ^^^^^^^
- lsdjjkf
- ^^^^^^
- sldkjf
- ^^^^^^
- lkjsfd
- """
- if not self.streamlined:
- self.streamline()
- for e in self.ignoreExprs:
- e.streamline()
-
- if not self.keepTabs:
- instring = _ustr(instring).expandtabs()
- instrlen = len(instring)
- loc = 0
- preparseFn = self.preParse
- parseFn = self._parse
- ParserElement.resetCache()
- matches = 0
- try:
- while loc <= instrlen and matches < maxMatches:
- try:
- preloc = preparseFn(instring, loc)
- nextLoc, tokens = parseFn(instring, preloc, callPreParse=False)
- except ParseException:
- loc = preloc + 1
- else:
- if nextLoc > loc:
- matches += 1
- yield tokens, preloc, nextLoc
- if overlap:
- nextloc = preparseFn(instring, loc)
- if nextloc > loc:
- loc = nextLoc
- else:
- loc += 1
- else:
- loc = nextLoc
- else:
- loc = preloc + 1
- except ParseBaseException as exc:
- if ParserElement.verbose_stacktrace:
- raise
- else:
- # catch and re-raise exception from here, clearing out pyparsing internal stack trace
- if getattr(exc, '__traceback__', None) is not None:
- exc.__traceback__ = self._trim_traceback(exc.__traceback__)
- raise exc
-
- def transformString(self, instring):
- """
- Extension to :class:`scanString`, to modify matching text with modified tokens that may
- be returned from a parse action. To use ``transformString``, define a grammar and
- attach a parse action to it that modifies the returned token list.
- Invoking ``transformString()`` on a target string will then scan for matches,
- and replace the matched text patterns according to the logic in the parse
- action. ``transformString()`` returns the resulting transformed string.
-
- Example::
-
- wd = Word(alphas)
- wd.setParseAction(lambda toks: toks[0].title())
-
- print(wd.transformString("now is the winter of our discontent made glorious summer by this sun of york."))
-
- prints::
-
- Now Is The Winter Of Our Discontent Made Glorious Summer By This Sun Of York.
- """
- out = []
- lastE = 0
- # force preservation of <TAB>s, to minimize unwanted transformation of string, and to
- # keep string locs straight between transformString and scanString
- self.keepTabs = True
- try:
- for t, s, e in self.scanString(instring):
- out.append(instring[lastE:s])
- if t:
- if isinstance(t, ParseResults):
- out += t.asList()
- elif isinstance(t, list):
- out += t
- else:
- out.append(t)
- lastE = e
- out.append(instring[lastE:])
- out = [o for o in out if o]
- return "".join(map(_ustr, _flatten(out)))
- except ParseBaseException as exc:
- if ParserElement.verbose_stacktrace:
- raise
- else:
- # catch and re-raise exception from here, clearing out pyparsing internal stack trace
- if getattr(exc, '__traceback__', None) is not None:
- exc.__traceback__ = self._trim_traceback(exc.__traceback__)
- raise exc
-
- def searchString(self, instring, maxMatches=_MAX_INT):
- """
- Another extension to :class:`scanString`, simplifying the access to the tokens found
- to match the given parse expression. May be called with optional
- ``maxMatches`` argument, to clip searching after 'n' matches are found.
-
- Example::
-
- # a capitalized word starts with an uppercase letter, followed by zero or more lowercase letters
- cap_word = Word(alphas.upper(), alphas.lower())
-
- print(cap_word.searchString("More than Iron, more than Lead, more than Gold I need Electricity"))
-
- # the sum() builtin can be used to merge results into a single ParseResults object
- print(sum(cap_word.searchString("More than Iron, more than Lead, more than Gold I need Electricity")))
-
- prints::
-
- [['More'], ['Iron'], ['Lead'], ['Gold'], ['I'], ['Electricity']]
- ['More', 'Iron', 'Lead', 'Gold', 'I', 'Electricity']
- """
- try:
- return ParseResults([t for t, s, e in self.scanString(instring, maxMatches)])
- except ParseBaseException as exc:
- if ParserElement.verbose_stacktrace:
- raise
- else:
- # catch and re-raise exception from here, clearing out pyparsing internal stack trace
- if getattr(exc, '__traceback__', None) is not None:
- exc.__traceback__ = self._trim_traceback(exc.__traceback__)
- raise exc
-
- def split(self, instring, maxsplit=_MAX_INT, includeSeparators=False):
- """
- Generator method to split a string using the given expression as a separator.
- May be called with optional ``maxsplit`` argument, to limit the number of splits;
- and the optional ``includeSeparators`` argument (default= ``False``), if the separating
- matching text should be included in the split results.
-
- Example::
-
- punc = oneOf(list(".,;:/-!?"))
- print(list(punc.split("This, this?, this sentence, is badly punctuated!")))
-
- prints::
-
- ['This', ' this', '', ' this sentence', ' is badly punctuated', '']
- """
- splits = 0
- last = 0
- for t, s, e in self.scanString(instring, maxMatches=maxsplit):
- yield instring[last:s]
- if includeSeparators:
- yield t[0]
- last = e
- yield instring[last:]
-
- def __add__(self, other):
- """
- Implementation of + operator - returns :class:`And`. Adding strings to a ParserElement
- converts them to :class:`Literal`s by default.
-
- Example::
-
- greet = Word(alphas) + "," + Word(alphas) + "!"
- hello = "Hello, World!"
- print (hello, "->", greet.parseString(hello))
-
- prints::
-
- Hello, World! -> ['Hello', ',', 'World', '!']
-
- ``...`` may be used as a parse expression as a short form of :class:`SkipTo`.
-
- Literal('start') + ... + Literal('end')
-
- is equivalent to:
-
- Literal('start') + SkipTo('end')("_skipped*") + Literal('end')
-
- Note that the skipped text is returned with '_skipped' as a results name,
- and to support having multiple skips in the same parser, the value returned is
- a list of all skipped text.
- """
- if other is Ellipsis:
- return _PendingSkip(self)
-
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return And([self, other])
-
- def __radd__(self, other):
- """
- Implementation of + operator when left operand is not a :class:`ParserElement`
- """
- if other is Ellipsis:
- return SkipTo(self)("_skipped*") + self
-
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return other + self
-
- def __sub__(self, other):
- """
- Implementation of - operator, returns :class:`And` with error stop
- """
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return self + And._ErrorStop() + other
-
- def __rsub__(self, other):
- """
- Implementation of - operator when left operand is not a :class:`ParserElement`
- """
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return other - self
-
- def __mul__(self, other):
- """
- Implementation of * operator, allows use of ``expr * 3`` in place of
- ``expr + expr + expr``. Expressions may also me multiplied by a 2-integer
- tuple, similar to ``{min, max}`` multipliers in regular expressions. Tuples
- may also include ``None`` as in:
- - ``expr*(n, None)`` or ``expr*(n, )`` is equivalent
- to ``expr*n + ZeroOrMore(expr)``
- (read as "at least n instances of ``expr``")
- - ``expr*(None, n)`` is equivalent to ``expr*(0, n)``
- (read as "0 to n instances of ``expr``")
- - ``expr*(None, None)`` is equivalent to ``ZeroOrMore(expr)``
- - ``expr*(1, None)`` is equivalent to ``OneOrMore(expr)``
-
- Note that ``expr*(None, n)`` does not raise an exception if
- more than n exprs exist in the input stream; that is,
- ``expr*(None, n)`` does not enforce a maximum number of expr
- occurrences. If this behavior is desired, then write
- ``expr*(None, n) + ~expr``
- """
- if other is Ellipsis:
- other = (0, None)
- elif isinstance(other, tuple) and other[:1] == (Ellipsis,):
- other = ((0, ) + other[1:] + (None,))[:2]
-
- if isinstance(other, int):
- minElements, optElements = other, 0
- elif isinstance(other, tuple):
- other = tuple(o if o is not Ellipsis else None for o in other)
- other = (other + (None, None))[:2]
- if other[0] is None:
- other = (0, other[1])
- if isinstance(other[0], int) and other[1] is None:
- if other[0] == 0:
- return ZeroOrMore(self)
- if other[0] == 1:
- return OneOrMore(self)
- else:
- return self * other[0] + ZeroOrMore(self)
- elif isinstance(other[0], int) and isinstance(other[1], int):
- minElements, optElements = other
- optElements -= minElements
- else:
- raise TypeError("cannot multiply 'ParserElement' and ('%s', '%s') objects", type(other[0]), type(other[1]))
- else:
- raise TypeError("cannot multiply 'ParserElement' and '%s' objects", type(other))
-
- if minElements < 0:
- raise ValueError("cannot multiply ParserElement by negative value")
- if optElements < 0:
- raise ValueError("second tuple value must be greater or equal to first tuple value")
- if minElements == optElements == 0:
- raise ValueError("cannot multiply ParserElement by 0 or (0, 0)")
-
- if optElements:
- def makeOptionalList(n):
- if n > 1:
- return Optional(self + makeOptionalList(n - 1))
- else:
- return Optional(self)
- if minElements:
- if minElements == 1:
- ret = self + makeOptionalList(optElements)
- else:
- ret = And([self] * minElements) + makeOptionalList(optElements)
- else:
- ret = makeOptionalList(optElements)
- else:
- if minElements == 1:
- ret = self
- else:
- ret = And([self] * minElements)
- return ret
-
- def __rmul__(self, other):
- return self.__mul__(other)
-
- def __or__(self, other):
- """
- Implementation of | operator - returns :class:`MatchFirst`
- """
- if other is Ellipsis:
- return _PendingSkip(self, must_skip=True)
-
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return MatchFirst([self, other])
-
- def __ror__(self, other):
- """
- Implementation of | operator when left operand is not a :class:`ParserElement`
- """
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return other | self
-
- def __xor__(self, other):
- """
- Implementation of ^ operator - returns :class:`Or`
- """
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return Or([self, other])
-
- def __rxor__(self, other):
- """
- Implementation of ^ operator when left operand is not a :class:`ParserElement`
- """
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return other ^ self
-
- def __and__(self, other):
- """
- Implementation of & operator - returns :class:`Each`
- """
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return Each([self, other])
-
- def __rand__(self, other):
- """
- Implementation of & operator when left operand is not a :class:`ParserElement`
- """
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- if not isinstance(other, ParserElement):
- warnings.warn("Cannot combine element of type %s with ParserElement" % type(other),
- SyntaxWarning, stacklevel=2)
- return None
- return other & self
-
- def __invert__(self):
- """
- Implementation of ~ operator - returns :class:`NotAny`
- """
- return NotAny(self)
-
- def __iter__(self):
- # must implement __iter__ to override legacy use of sequential access to __getitem__ to
- # iterate over a sequence
- raise TypeError('%r object is not iterable' % self.__class__.__name__)
-
- def __getitem__(self, key):
- """
- use ``[]`` indexing notation as a short form for expression repetition:
- - ``expr[n]`` is equivalent to ``expr*n``
- - ``expr[m, n]`` is equivalent to ``expr*(m, n)``
- - ``expr[n, ...]`` or ``expr[n,]`` is equivalent
- to ``expr*n + ZeroOrMore(expr)``
- (read as "at least n instances of ``expr``")
- - ``expr[..., n]`` is equivalent to ``expr*(0, n)``
- (read as "0 to n instances of ``expr``")
- - ``expr[...]`` and ``expr[0, ...]`` are equivalent to ``ZeroOrMore(expr)``
- - ``expr[1, ...]`` is equivalent to ``OneOrMore(expr)``
- ``None`` may be used in place of ``...``.
-
- Note that ``expr[..., n]`` and ``expr[m, n]``do not raise an exception
- if more than ``n`` ``expr``s exist in the input stream. If this behavior is
- desired, then write ``expr[..., n] + ~expr``.
- """
-
- # convert single arg keys to tuples
- try:
- if isinstance(key, str):
- key = (key,)
- iter(key)
- except TypeError:
- key = (key, key)
-
- if len(key) > 2:
- warnings.warn("only 1 or 2 index arguments supported ({0}{1})".format(key[:5],
- '... [{0}]'.format(len(key))
- if len(key) > 5 else ''))
-
- # clip to 2 elements
- ret = self * tuple(key[:2])
- return ret
-
- def __call__(self, name=None):
- """
- Shortcut for :class:`setResultsName`, with ``listAllMatches=False``.
-
- If ``name`` is given with a trailing ``'*'`` character, then ``listAllMatches`` will be
- passed as ``True``.
-
- If ``name` is omitted, same as calling :class:`copy`.
-
- Example::
-
- # these are equivalent
- userdata = Word(alphas).setResultsName("name") + Word(nums + "-").setResultsName("socsecno")
- userdata = Word(alphas)("name") + Word(nums + "-")("socsecno")
- """
- if name is not None:
- return self._setResultsName(name)
- else:
- return self.copy()
-
- def suppress(self):
- """
- Suppresses the output of this :class:`ParserElement`; useful to keep punctuation from
- cluttering up returned output.
- """
- return Suppress(self)
-
- def leaveWhitespace(self):
- """
- Disables the skipping of whitespace before matching the characters in the
- :class:`ParserElement`'s defined pattern. This is normally only used internally by
- the pyparsing module, but may be needed in some whitespace-sensitive grammars.
- """
- self.skipWhitespace = False
- return self
-
- def setWhitespaceChars(self, chars):
- """
- Overrides the default whitespace chars
- """
- self.skipWhitespace = True
- self.whiteChars = chars
- self.copyDefaultWhiteChars = False
- return self
-
- def parseWithTabs(self):
- """
- Overrides default behavior to expand ``<TAB>``s to spaces before parsing the input string.
- Must be called before ``parseString`` when the input grammar contains elements that
- match ``<TAB>`` characters.
- """
- self.keepTabs = True
- return self
-
- def ignore(self, other):
- """
- Define expression to be ignored (e.g., comments) while doing pattern
- matching; may be called repeatedly, to define multiple comment or other
- ignorable patterns.
-
- Example::
-
- patt = OneOrMore(Word(alphas))
- patt.parseString('ablaj /* comment */ lskjd') # -> ['ablaj']
-
- patt.ignore(cStyleComment)
- patt.parseString('ablaj /* comment */ lskjd') # -> ['ablaj', 'lskjd']
- """
- if isinstance(other, basestring):
- other = Suppress(other)
-
- if isinstance(other, Suppress):
- if other not in self.ignoreExprs:
- self.ignoreExprs.append(other)
- else:
- self.ignoreExprs.append(Suppress(other.copy()))
- return self
-
- def setDebugActions(self, startAction, successAction, exceptionAction):
- """
- Enable display of debugging messages while doing pattern matching.
- """
- self.debugActions = (startAction or _defaultStartDebugAction,
- successAction or _defaultSuccessDebugAction,
- exceptionAction or _defaultExceptionDebugAction)
- self.debug = True
- return self
-
- def setDebug(self, flag=True):
- """
- Enable display of debugging messages while doing pattern matching.
- Set ``flag`` to True to enable, False to disable.
-
- Example::
-
- wd = Word(alphas).setName("alphaword")
- integer = Word(nums).setName("numword")
- term = wd | integer
-
- # turn on debugging for wd
- wd.setDebug()
-
- OneOrMore(term).parseString("abc 123 xyz 890")
-
- prints::
-
- Match alphaword at loc 0(1,1)
- Matched alphaword -> ['abc']
- Match alphaword at loc 3(1,4)
- Exception raised:Expected alphaword (at char 4), (line:1, col:5)
- Match alphaword at loc 7(1,8)
- Matched alphaword -> ['xyz']
- Match alphaword at loc 11(1,12)
- Exception raised:Expected alphaword (at char 12), (line:1, col:13)
- Match alphaword at loc 15(1,16)
- Exception raised:Expected alphaword (at char 15), (line:1, col:16)
-
- The output shown is that produced by the default debug actions - custom debug actions can be
- specified using :class:`setDebugActions`. Prior to attempting
- to match the ``wd`` expression, the debugging message ``"Match <exprname> at loc <n>(<line>,<col>)"``
- is shown. Then if the parse succeeds, a ``"Matched"`` message is shown, or an ``"Exception raised"``
- message is shown. Also note the use of :class:`setName` to assign a human-readable name to the expression,
- which makes debugging and exception messages easier to understand - for instance, the default
- name created for the :class:`Word` expression without calling ``setName`` is ``"W:(ABCD...)"``.
- """
- if flag:
- self.setDebugActions(_defaultStartDebugAction, _defaultSuccessDebugAction, _defaultExceptionDebugAction)
- else:
- self.debug = False
- return self
-
- def __str__(self):
- return self.name
-
- def __repr__(self):
- return _ustr(self)
-
- def streamline(self):
- self.streamlined = True
- self.strRepr = None
- return self
-
- def checkRecursion(self, parseElementList):
- pass
-
- def validate(self, validateTrace=None):
- """
- Check defined expressions for valid structure, check for infinite recursive definitions.
- """
- self.checkRecursion([])
-
- def parseFile(self, file_or_filename, parseAll=False):
- """
- Execute the parse expression on the given file or filename.
- If a filename is specified (instead of a file object),
- the entire file is opened, read, and closed before parsing.
- """
- try:
- file_contents = file_or_filename.read()
- except AttributeError:
- with open(file_or_filename, "r") as f:
- file_contents = f.read()
- try:
- return self.parseString(file_contents, parseAll)
- except ParseBaseException as exc:
- if ParserElement.verbose_stacktrace:
- raise
- else:
- # catch and re-raise exception from here, clearing out pyparsing internal stack trace
- if getattr(exc, '__traceback__', None) is not None:
- exc.__traceback__ = self._trim_traceback(exc.__traceback__)
- raise exc
-
- def __eq__(self, other):
- if self is other:
- return True
- elif isinstance(other, basestring):
- return self.matches(other)
- elif isinstance(other, ParserElement):
- return vars(self) == vars(other)
- return False
-
- def __ne__(self, other):
- return not (self == other)
-
- def __hash__(self):
- return id(self)
-
- def __req__(self, other):
- return self == other
-
- def __rne__(self, other):
- return not (self == other)
-
- def matches(self, testString, parseAll=True):
- """
- Method for quick testing of a parser against a test string. Good for simple
- inline microtests of sub expressions while building up larger parser.
-
- Parameters:
- - testString - to test against this expression for a match
- - parseAll - (default= ``True``) - flag to pass to :class:`parseString` when running tests
-
- Example::
-
- expr = Word(nums)
- assert expr.matches("100")
- """
- try:
- self.parseString(_ustr(testString), parseAll=parseAll)
- return True
- except ParseBaseException:
- return False
-
- def runTests(self, tests, parseAll=True, comment='#',
- fullDump=True, printResults=True, failureTests=False, postParse=None,
- file=None):
- """
- Execute the parse expression on a series of test strings, showing each
- test, the parsed results or where the parse failed. Quick and easy way to
- run a parse expression against a list of sample strings.
-
- Parameters:
- - tests - a list of separate test strings, or a multiline string of test strings
- - parseAll - (default= ``True``) - flag to pass to :class:`parseString` when running tests
- - comment - (default= ``'#'``) - expression for indicating embedded comments in the test
- string; pass None to disable comment filtering
- - fullDump - (default= ``True``) - dump results as list followed by results names in nested outline;
- if False, only dump nested list
- - printResults - (default= ``True``) prints test output to stdout
- - failureTests - (default= ``False``) indicates if these tests are expected to fail parsing
- - postParse - (default= ``None``) optional callback for successful parse results; called as
- `fn(test_string, parse_results)` and returns a string to be added to the test output
- - file - (default=``None``) optional file-like object to which test output will be written;
- if None, will default to ``sys.stdout``
-
- Returns: a (success, results) tuple, where success indicates that all tests succeeded
- (or failed if ``failureTests`` is True), and the results contain a list of lines of each
- test's output
-
- Example::
-
- number_expr = pyparsing_common.number.copy()
-
- result = number_expr.runTests('''
- # unsigned integer
- 100
- # negative integer
- -100
- # float with scientific notation
- 6.02e23
- # integer with scientific notation
- 1e-12
- ''')
- print("Success" if result[0] else "Failed!")
-
- result = number_expr.runTests('''
- # stray character
- 100Z
- # missing leading digit before '.'
- -.100
- # too many '.'
- 3.14.159
- ''', failureTests=True)
- print("Success" if result[0] else "Failed!")
-
- prints::
-
- # unsigned integer
- 100
- [100]
-
- # negative integer
- -100
- [-100]
-
- # float with scientific notation
- 6.02e23
- [6.02e+23]
-
- # integer with scientific notation
- 1e-12
- [1e-12]
-
- Success
-
- # stray character
- 100Z
- ^
- FAIL: Expected end of text (at char 3), (line:1, col:4)
-
- # missing leading digit before '.'
- -.100
- ^
- FAIL: Expected {real number with scientific notation | real number | signed integer} (at char 0), (line:1, col:1)
-
- # too many '.'
- 3.14.159
- ^
- FAIL: Expected end of text (at char 4), (line:1, col:5)
-
- Success
-
- Each test string must be on a single line. If you want to test a string that spans multiple
- lines, create a test like this::
-
- expr.runTest(r"this is a test\\n of strings that spans \\n 3 lines")
-
- (Note that this is a raw string literal, you must include the leading 'r'.)
- """
- if isinstance(tests, basestring):
- tests = list(map(str.strip, tests.rstrip().splitlines()))
- if isinstance(comment, basestring):
- comment = Literal(comment)
- if file is None:
- file = sys.stdout
- print_ = file.write
-
- allResults = []
- comments = []
- success = True
- NL = Literal(r'\n').addParseAction(replaceWith('\n')).ignore(quotedString)
- BOM = u'\ufeff'
- for t in tests:
- if comment is not None and comment.matches(t, False) or comments and not t:
- comments.append(t)
- continue
- if not t:
- continue
- out = ['\n' + '\n'.join(comments) if comments else '', t]
- comments = []
- try:
- # convert newline marks to actual newlines, and strip leading BOM if present
- t = NL.transformString(t.lstrip(BOM))
- result = self.parseString(t, parseAll=parseAll)
- except ParseBaseException as pe:
- fatal = "(FATAL)" if isinstance(pe, ParseFatalException) else ""
- if '\n' in t:
- out.append(line(pe.loc, t))
- out.append(' ' * (col(pe.loc, t) - 1) + '^' + fatal)
- else:
- out.append(' ' * pe.loc + '^' + fatal)
- out.append("FAIL: " + str(pe))
- success = success and failureTests
- result = pe
- except Exception as exc:
- out.append("FAIL-EXCEPTION: " + str(exc))
- success = success and failureTests
- result = exc
- else:
- success = success and not failureTests
- if postParse is not None:
- try:
- pp_value = postParse(t, result)
- if pp_value is not None:
- if isinstance(pp_value, ParseResults):
- out.append(pp_value.dump())
- else:
- out.append(str(pp_value))
- else:
- out.append(result.dump())
- except Exception as e:
- out.append(result.dump(full=fullDump))
- out.append("{0} failed: {1}: {2}".format(postParse.__name__, type(e).__name__, e))
- else:
- out.append(result.dump(full=fullDump))
-
- if printResults:
- if fullDump:
- out.append('')
- print_('\n'.join(out))
-
- allResults.append((t, result))
-
- return success, allResults
-
-
-class _PendingSkip(ParserElement):
- # internal placeholder class to hold a place were '...' is added to a parser element,
- # once another ParserElement is added, this placeholder will be replaced with a SkipTo
- def __init__(self, expr, must_skip=False):
- super(_PendingSkip, self).__init__()
- self.strRepr = str(expr + Empty()).replace('Empty', '...')
- self.name = self.strRepr
- self.anchor = expr
- self.must_skip = must_skip
-
- def __add__(self, other):
- skipper = SkipTo(other).setName("...")("_skipped*")
- if self.must_skip:
- def must_skip(t):
- if not t._skipped or t._skipped.asList() == ['']:
- del t[0]
- t.pop("_skipped", None)
- def show_skip(t):
- if t._skipped.asList()[-1:] == ['']:
- skipped = t.pop('_skipped')
- t['_skipped'] = 'missing <' + repr(self.anchor) + '>'
- return (self.anchor + skipper().addParseAction(must_skip)
- | skipper().addParseAction(show_skip)) + other
-
- return self.anchor + skipper + other
-
- def __repr__(self):
- return self.strRepr
-
- def parseImpl(self, *args):
- raise Exception("use of `...` expression without following SkipTo target expression")
-
-
-class Token(ParserElement):
- """Abstract :class:`ParserElement` subclass, for defining atomic
- matching patterns.
- """
- def __init__(self):
- super(Token, self).__init__(savelist=False)
-
-
-class Empty(Token):
- """An empty token, will always match.
- """
- def __init__(self):
- super(Empty, self).__init__()
- self.name = "Empty"
- self.mayReturnEmpty = True
- self.mayIndexError = False
-
-
-class NoMatch(Token):
- """A token that will never match.
- """
- def __init__(self):
- super(NoMatch, self).__init__()
- self.name = "NoMatch"
- self.mayReturnEmpty = True
- self.mayIndexError = False
- self.errmsg = "Unmatchable token"
-
- def parseImpl(self, instring, loc, doActions=True):
- raise ParseException(instring, loc, self.errmsg, self)
-
-
-class Literal(Token):
- """Token to exactly match a specified string.
-
- Example::
-
- Literal('blah').parseString('blah') # -> ['blah']
- Literal('blah').parseString('blahfooblah') # -> ['blah']
- Literal('blah').parseString('bla') # -> Exception: Expected "blah"
-
- For case-insensitive matching, use :class:`CaselessLiteral`.
-
- For keyword matching (force word break before and after the matched string),
- use :class:`Keyword` or :class:`CaselessKeyword`.
- """
- def __init__(self, matchString):
- super(Literal, self).__init__()
- self.match = matchString
- self.matchLen = len(matchString)
- try:
- self.firstMatchChar = matchString[0]
- except IndexError:
- warnings.warn("null string passed to Literal; use Empty() instead",
- SyntaxWarning, stacklevel=2)
- self.__class__ = Empty
- self.name = '"%s"' % _ustr(self.match)
- self.errmsg = "Expected " + self.name
- self.mayReturnEmpty = False
- self.mayIndexError = False
-
- # Performance tuning: modify __class__ to select
- # a parseImpl optimized for single-character check
- if self.matchLen == 1 and type(self) is Literal:
- self.__class__ = _SingleCharLiteral
-
- def parseImpl(self, instring, loc, doActions=True):
- if instring[loc] == self.firstMatchChar and instring.startswith(self.match, loc):
- return loc + self.matchLen, self.match
- raise ParseException(instring, loc, self.errmsg, self)
-
-class _SingleCharLiteral(Literal):
- def parseImpl(self, instring, loc, doActions=True):
- if instring[loc] == self.firstMatchChar:
- return loc + 1, self.match
- raise ParseException(instring, loc, self.errmsg, self)
-
-_L = Literal
-ParserElement._literalStringClass = Literal
-
-class Keyword(Token):
- """Token to exactly match a specified string as a keyword, that is,
- it must be immediately followed by a non-keyword character. Compare
- with :class:`Literal`:
-
- - ``Literal("if")`` will match the leading ``'if'`` in
- ``'ifAndOnlyIf'``.
- - ``Keyword("if")`` will not; it will only match the leading
- ``'if'`` in ``'if x=1'``, or ``'if(y==2)'``
-
- Accepts two optional constructor arguments in addition to the
- keyword string:
-
- - ``identChars`` is a string of characters that would be valid
- identifier characters, defaulting to all alphanumerics + "_" and
- "$"
- - ``caseless`` allows case-insensitive matching, default is ``False``.
-
- Example::
-
- Keyword("start").parseString("start") # -> ['start']
- Keyword("start").parseString("starting") # -> Exception
-
- For case-insensitive matching, use :class:`CaselessKeyword`.
- """
- DEFAULT_KEYWORD_CHARS = alphanums + "_$"
-
- def __init__(self, matchString, identChars=None, caseless=False):
- super(Keyword, self).__init__()
- if identChars is None:
- identChars = Keyword.DEFAULT_KEYWORD_CHARS
- self.match = matchString
- self.matchLen = len(matchString)
- try:
- self.firstMatchChar = matchString[0]
- except IndexError:
- warnings.warn("null string passed to Keyword; use Empty() instead",
- SyntaxWarning, stacklevel=2)
- self.name = '"%s"' % self.match
- self.errmsg = "Expected " + self.name
- self.mayReturnEmpty = False
- self.mayIndexError = False
- self.caseless = caseless
- if caseless:
- self.caselessmatch = matchString.upper()
- identChars = identChars.upper()
- self.identChars = set(identChars)
-
- def parseImpl(self, instring, loc, doActions=True):
- if self.caseless:
- if ((instring[loc:loc + self.matchLen].upper() == self.caselessmatch)
- and (loc >= len(instring) - self.matchLen
- or instring[loc + self.matchLen].upper() not in self.identChars)
- and (loc == 0
- or instring[loc - 1].upper() not in self.identChars)):
- return loc + self.matchLen, self.match
-
- else:
- if instring[loc] == self.firstMatchChar:
- if ((self.matchLen == 1 or instring.startswith(self.match, loc))
- and (loc >= len(instring) - self.matchLen
- or instring[loc + self.matchLen] not in self.identChars)
- and (loc == 0 or instring[loc - 1] not in self.identChars)):
- return loc + self.matchLen, self.match
-
- raise ParseException(instring, loc, self.errmsg, self)
-
- def copy(self):
- c = super(Keyword, self).copy()
- c.identChars = Keyword.DEFAULT_KEYWORD_CHARS
- return c
-
- @staticmethod
- def setDefaultKeywordChars(chars):
- """Overrides the default Keyword chars
- """
- Keyword.DEFAULT_KEYWORD_CHARS = chars
-
-class CaselessLiteral(Literal):
- """Token to match a specified string, ignoring case of letters.
- Note: the matched results will always be in the case of the given
- match string, NOT the case of the input text.
-
- Example::
-
- OneOrMore(CaselessLiteral("CMD")).parseString("cmd CMD Cmd10") # -> ['CMD', 'CMD', 'CMD']
-
- (Contrast with example for :class:`CaselessKeyword`.)
- """
- def __init__(self, matchString):
- super(CaselessLiteral, self).__init__(matchString.upper())
- # Preserve the defining literal.
- self.returnString = matchString
- self.name = "'%s'" % self.returnString
- self.errmsg = "Expected " + self.name
-
- def parseImpl(self, instring, loc, doActions=True):
- if instring[loc:loc + self.matchLen].upper() == self.match:
- return loc + self.matchLen, self.returnString
- raise ParseException(instring, loc, self.errmsg, self)
-
-class CaselessKeyword(Keyword):
- """
- Caseless version of :class:`Keyword`.
-
- Example::
-
- OneOrMore(CaselessKeyword("CMD")).parseString("cmd CMD Cmd10") # -> ['CMD', 'CMD']
-
- (Contrast with example for :class:`CaselessLiteral`.)
- """
- def __init__(self, matchString, identChars=None):
- super(CaselessKeyword, self).__init__(matchString, identChars, caseless=True)
-
-class CloseMatch(Token):
- """A variation on :class:`Literal` which matches "close" matches,
- that is, strings with at most 'n' mismatching characters.
- :class:`CloseMatch` takes parameters:
-
- - ``match_string`` - string to be matched
- - ``maxMismatches`` - (``default=1``) maximum number of
- mismatches allowed to count as a match
-
- The results from a successful parse will contain the matched text
- from the input string and the following named results:
-
- - ``mismatches`` - a list of the positions within the
- match_string where mismatches were found
- - ``original`` - the original match_string used to compare
- against the input string
-
- If ``mismatches`` is an empty list, then the match was an exact
- match.
-
- Example::
-
- patt = CloseMatch("ATCATCGAATGGA")
- patt.parseString("ATCATCGAAXGGA") # -> (['ATCATCGAAXGGA'], {'mismatches': [[9]], 'original': ['ATCATCGAATGGA']})
- patt.parseString("ATCAXCGAAXGGA") # -> Exception: Expected 'ATCATCGAATGGA' (with up to 1 mismatches) (at char 0), (line:1, col:1)
-
- # exact match
- patt.parseString("ATCATCGAATGGA") # -> (['ATCATCGAATGGA'], {'mismatches': [[]], 'original': ['ATCATCGAATGGA']})
-
- # close match allowing up to 2 mismatches
- patt = CloseMatch("ATCATCGAATGGA", maxMismatches=2)
- patt.parseString("ATCAXCGAAXGGA") # -> (['ATCAXCGAAXGGA'], {'mismatches': [[4, 9]], 'original': ['ATCATCGAATGGA']})
- """
- def __init__(self, match_string, maxMismatches=1):
- super(CloseMatch, self).__init__()
- self.name = match_string
- self.match_string = match_string
- self.maxMismatches = maxMismatches
- self.errmsg = "Expected %r (with up to %d mismatches)" % (self.match_string, self.maxMismatches)
- self.mayIndexError = False
- self.mayReturnEmpty = False
-
- def parseImpl(self, instring, loc, doActions=True):
- start = loc
- instrlen = len(instring)
- maxloc = start + len(self.match_string)
-
- if maxloc <= instrlen:
- match_string = self.match_string
- match_stringloc = 0
- mismatches = []
- maxMismatches = self.maxMismatches
-
- for match_stringloc, s_m in enumerate(zip(instring[loc:maxloc], match_string)):
- src, mat = s_m
- if src != mat:
- mismatches.append(match_stringloc)
- if len(mismatches) > maxMismatches:
- break
- else:
- loc = match_stringloc + 1
- results = ParseResults([instring[start:loc]])
- results['original'] = match_string
- results['mismatches'] = mismatches
- return loc, results
-
- raise ParseException(instring, loc, self.errmsg, self)
-
-
-class Word(Token):
- """Token for matching words composed of allowed character sets.
- Defined with string containing all allowed initial characters, an
- optional string containing allowed body characters (if omitted,
- defaults to the initial character set), and an optional minimum,
- maximum, and/or exact length. The default value for ``min`` is
- 1 (a minimum value < 1 is not valid); the default values for
- ``max`` and ``exact`` are 0, meaning no maximum or exact
- length restriction. An optional ``excludeChars`` parameter can
- list characters that might be found in the input ``bodyChars``
- string; useful to define a word of all printables except for one or
- two characters, for instance.
-
- :class:`srange` is useful for defining custom character set strings
- for defining ``Word`` expressions, using range notation from
- regular expression character sets.
-
- A common mistake is to use :class:`Word` to match a specific literal
- string, as in ``Word("Address")``. Remember that :class:`Word`
- uses the string argument to define *sets* of matchable characters.
- This expression would match "Add", "AAA", "dAred", or any other word
- made up of the characters 'A', 'd', 'r', 'e', and 's'. To match an
- exact literal string, use :class:`Literal` or :class:`Keyword`.
-
- pyparsing includes helper strings for building Words:
-
- - :class:`alphas`
- - :class:`nums`
- - :class:`alphanums`
- - :class:`hexnums`
- - :class:`alphas8bit` (alphabetic characters in ASCII range 128-255
- - accented, tilded, umlauted, etc.)
- - :class:`punc8bit` (non-alphabetic characters in ASCII range
- 128-255 - currency, symbols, superscripts, diacriticals, etc.)
- - :class:`printables` (any non-whitespace character)
-
- Example::
-
- # a word composed of digits
- integer = Word(nums) # equivalent to Word("0123456789") or Word(srange("0-9"))
-
- # a word with a leading capital, and zero or more lowercase
- capital_word = Word(alphas.upper(), alphas.lower())
-
- # hostnames are alphanumeric, with leading alpha, and '-'
- hostname = Word(alphas, alphanums + '-')
-
- # roman numeral (not a strict parser, accepts invalid mix of characters)
- roman = Word("IVXLCDM")
-
- # any string of non-whitespace characters, except for ','
- csv_value = Word(printables, excludeChars=",")
- """
- def __init__(self, initChars, bodyChars=None, min=1, max=0, exact=0, asKeyword=False, excludeChars=None):
- super(Word, self).__init__()
- if excludeChars:
- excludeChars = set(excludeChars)
- initChars = ''.join(c for c in initChars if c not in excludeChars)
- if bodyChars:
- bodyChars = ''.join(c for c in bodyChars if c not in excludeChars)
- self.initCharsOrig = initChars
- self.initChars = set(initChars)
- if bodyChars:
- self.bodyCharsOrig = bodyChars
- self.bodyChars = set(bodyChars)
- else:
- self.bodyCharsOrig = initChars
- self.bodyChars = set(initChars)
-
- self.maxSpecified = max > 0
-
- if min < 1:
- raise ValueError("cannot specify a minimum length < 1; use Optional(Word()) if zero-length word is permitted")
-
- self.minLen = min
-
- if max > 0:
- self.maxLen = max
- else:
- self.maxLen = _MAX_INT
-
- if exact > 0:
- self.maxLen = exact
- self.minLen = exact
-
- self.name = _ustr(self)
- self.errmsg = "Expected " + self.name
- self.mayIndexError = False
- self.asKeyword = asKeyword
-
- if ' ' not in self.initCharsOrig + self.bodyCharsOrig and (min == 1 and max == 0 and exact == 0):
- if self.bodyCharsOrig == self.initCharsOrig:
- self.reString = "[%s]+" % _escapeRegexRangeChars(self.initCharsOrig)
- elif len(self.initCharsOrig) == 1:
- self.reString = "%s[%s]*" % (re.escape(self.initCharsOrig),
- _escapeRegexRangeChars(self.bodyCharsOrig),)
- else:
- self.reString = "[%s][%s]*" % (_escapeRegexRangeChars(self.initCharsOrig),
- _escapeRegexRangeChars(self.bodyCharsOrig),)
- if self.asKeyword:
- self.reString = r"\b" + self.reString + r"\b"
-
- try:
- self.re = re.compile(self.reString)
- except Exception:
- self.re = None
- else:
- self.re_match = self.re.match
- self.__class__ = _WordRegex
-
- def parseImpl(self, instring, loc, doActions=True):
- if instring[loc] not in self.initChars:
- raise ParseException(instring, loc, self.errmsg, self)
-
- start = loc
- loc += 1
- instrlen = len(instring)
- bodychars = self.bodyChars
- maxloc = start + self.maxLen
- maxloc = min(maxloc, instrlen)
- while loc < maxloc and instring[loc] in bodychars:
- loc += 1
-
- throwException = False
- if loc - start < self.minLen:
- throwException = True
- elif self.maxSpecified and loc < instrlen and instring[loc] in bodychars:
- throwException = True
- elif self.asKeyword:
- if (start > 0 and instring[start - 1] in bodychars
- or loc < instrlen and instring[loc] in bodychars):
- throwException = True
-
- if throwException:
- raise ParseException(instring, loc, self.errmsg, self)
-
- return loc, instring[start:loc]
-
- def __str__(self):
- try:
- return super(Word, self).__str__()
- except Exception:
- pass
-
- if self.strRepr is None:
-
- def charsAsStr(s):
- if len(s) > 4:
- return s[:4] + "..."
- else:
- return s
-
- if self.initCharsOrig != self.bodyCharsOrig:
- self.strRepr = "W:(%s, %s)" % (charsAsStr(self.initCharsOrig), charsAsStr(self.bodyCharsOrig))
- else:
- self.strRepr = "W:(%s)" % charsAsStr(self.initCharsOrig)
-
- return self.strRepr
-
-class _WordRegex(Word):
- def parseImpl(self, instring, loc, doActions=True):
- result = self.re_match(instring, loc)
- if not result:
- raise ParseException(instring, loc, self.errmsg, self)
-
- loc = result.end()
- return loc, result.group()
-
-
-class Char(_WordRegex):
- """A short-cut class for defining ``Word(characters, exact=1)``,
- when defining a match of any single character in a string of
- characters.
- """
- def __init__(self, charset, asKeyword=False, excludeChars=None):
- super(Char, self).__init__(charset, exact=1, asKeyword=asKeyword, excludeChars=excludeChars)
- self.reString = "[%s]" % _escapeRegexRangeChars(''.join(self.initChars))
- if asKeyword:
- self.reString = r"\b%s\b" % self.reString
- self.re = re.compile(self.reString)
- self.re_match = self.re.match
-
-
-class Regex(Token):
- r"""Token for matching strings that match a given regular
- expression. Defined with string specifying the regular expression in
- a form recognized by the stdlib Python `re module <https://docs.python.org/3/library/re.html>`_.
- If the given regex contains named groups (defined using ``(?P<name>...)``),
- these will be preserved as named parse results.
-
- If instead of the Python stdlib re module you wish to use a different RE module
- (such as the `regex` module), you can replace it by either building your
- Regex object with a compiled RE that was compiled using regex:
-
- Example::
-
- realnum = Regex(r"[+-]?\d+\.\d*")
- date = Regex(r'(?P<year>\d{4})-(?P<month>\d\d?)-(?P<day>\d\d?)')
- # ref: https://stackoverflow.com/questions/267399/how-do-you-match-only-valid-roman-numerals-with-a-regular-expression
- roman = Regex(r"M{0,4}(CM|CD|D?{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})")
-
- # use regex module instead of stdlib re module to construct a Regex using
- # a compiled regular expression
- import regex
- parser = pp.Regex(regex.compile(r'[0-9]'))
-
- """
- def __init__(self, pattern, flags=0, asGroupList=False, asMatch=False):
- """The parameters ``pattern`` and ``flags`` are passed
- to the ``re.compile()`` function as-is. See the Python
- `re module <https://docs.python.org/3/library/re.html>`_ module for an
- explanation of the acceptable patterns and flags.
- """
- super(Regex, self).__init__()
-
- if isinstance(pattern, basestring):
- if not pattern:
- warnings.warn("null string passed to Regex; use Empty() instead",
- SyntaxWarning, stacklevel=2)
-
- self.pattern = pattern
- self.flags = flags
-
- try:
- self.re = re.compile(self.pattern, self.flags)
- self.reString = self.pattern
- except sre_constants.error:
- warnings.warn("invalid pattern (%s) passed to Regex" % pattern,
- SyntaxWarning, stacklevel=2)
- raise
-
- elif hasattr(pattern, 'pattern') and hasattr(pattern, 'match'):
- self.re = pattern
- self.pattern = self.reString = pattern.pattern
- self.flags = flags
-
- else:
- raise TypeError("Regex may only be constructed with a string or a compiled RE object")
-
- self.re_match = self.re.match
-
- self.name = _ustr(self)
- self.errmsg = "Expected " + self.name
- self.mayIndexError = False
- self.mayReturnEmpty = self.re_match("") is not None
- self.asGroupList = asGroupList
- self.asMatch = asMatch
- if self.asGroupList:
- self.parseImpl = self.parseImplAsGroupList
- if self.asMatch:
- self.parseImpl = self.parseImplAsMatch
-
- def parseImpl(self, instring, loc, doActions=True):
- result = self.re_match(instring, loc)
- if not result:
- raise ParseException(instring, loc, self.errmsg, self)
-
- loc = result.end()
- ret = ParseResults(result.group())
- d = result.groupdict()
- if d:
- for k, v in d.items():
- ret[k] = v
- return loc, ret
-
- def parseImplAsGroupList(self, instring, loc, doActions=True):
- result = self.re_match(instring, loc)
- if not result:
- raise ParseException(instring, loc, self.errmsg, self)
-
- loc = result.end()
- ret = result.groups()
- return loc, ret
-
- def parseImplAsMatch(self, instring, loc, doActions=True):
- result = self.re_match(instring, loc)
- if not result:
- raise ParseException(instring, loc, self.errmsg, self)
-
- loc = result.end()
- ret = result
- return loc, ret
-
- def __str__(self):
- try:
- return super(Regex, self).__str__()
- except Exception:
- pass
-
- if self.strRepr is None:
- self.strRepr = "Re:(%s)" % repr(self.pattern)
-
- return self.strRepr
-
- def sub(self, repl):
- r"""
- Return Regex with an attached parse action to transform the parsed
- result as if called using `re.sub(expr, repl, string) <https://docs.python.org/3/library/re.html#re.sub>`_.
-
- Example::
-
- make_html = Regex(r"(\w+):(.*?):").sub(r"<\1>\2</\1>")
- print(make_html.transformString("h1:main title:"))
- # prints "<h1>main title</h1>"
- """
- if self.asGroupList:
- warnings.warn("cannot use sub() with Regex(asGroupList=True)",
- SyntaxWarning, stacklevel=2)
- raise SyntaxError()
-
- if self.asMatch and callable(repl):
- warnings.warn("cannot use sub() with a callable with Regex(asMatch=True)",
- SyntaxWarning, stacklevel=2)
- raise SyntaxError()
-
- if self.asMatch:
- def pa(tokens):
- return tokens[0].expand(repl)
- else:
- def pa(tokens):
- return self.re.sub(repl, tokens[0])
- return self.addParseAction(pa)
-
-class QuotedString(Token):
- r"""
- Token for matching strings that are delimited by quoting characters.
-
- Defined with the following parameters:
-
- - quoteChar - string of one or more characters defining the
- quote delimiting string
- - escChar - character to escape quotes, typically backslash
- (default= ``None``)
- - escQuote - special quote sequence to escape an embedded quote
- string (such as SQL's ``""`` to escape an embedded ``"``)
- (default= ``None``)
- - multiline - boolean indicating whether quotes can span
- multiple lines (default= ``False``)
- - unquoteResults - boolean indicating whether the matched text
- should be unquoted (default= ``True``)
- - endQuoteChar - string of one or more characters defining the
- end of the quote delimited string (default= ``None`` => same as
- quoteChar)
- - convertWhitespaceEscapes - convert escaped whitespace
- (``'\t'``, ``'\n'``, etc.) to actual whitespace
- (default= ``True``)
-
- Example::
-
- qs = QuotedString('"')
- print(qs.searchString('lsjdf "This is the quote" sldjf'))
- complex_qs = QuotedString('{{', endQuoteChar='}}')
- print(complex_qs.searchString('lsjdf {{This is the "quote"}} sldjf'))
- sql_qs = QuotedString('"', escQuote='""')
- print(sql_qs.searchString('lsjdf "This is the quote with ""embedded"" quotes" sldjf'))
-
- prints::
-
- [['This is the quote']]
- [['This is the "quote"']]
- [['This is the quote with "embedded" quotes']]
- """
- def __init__(self, quoteChar, escChar=None, escQuote=None, multiline=False,
- unquoteResults=True, endQuoteChar=None, convertWhitespaceEscapes=True):
- super(QuotedString, self).__init__()
-
- # remove white space from quote chars - wont work anyway
- quoteChar = quoteChar.strip()
- if not quoteChar:
- warnings.warn("quoteChar cannot be the empty string", SyntaxWarning, stacklevel=2)
- raise SyntaxError()
-
- if endQuoteChar is None:
- endQuoteChar = quoteChar
- else:
- endQuoteChar = endQuoteChar.strip()
- if not endQuoteChar:
- warnings.warn("endQuoteChar cannot be the empty string", SyntaxWarning, stacklevel=2)
- raise SyntaxError()
-
- self.quoteChar = quoteChar
- self.quoteCharLen = len(quoteChar)
- self.firstQuoteChar = quoteChar[0]
- self.endQuoteChar = endQuoteChar
- self.endQuoteCharLen = len(endQuoteChar)
- self.escChar = escChar
- self.escQuote = escQuote
- self.unquoteResults = unquoteResults
- self.convertWhitespaceEscapes = convertWhitespaceEscapes
-
- if multiline:
- self.flags = re.MULTILINE | re.DOTALL
- self.pattern = r'%s(?:[^%s%s]' % (re.escape(self.quoteChar),
- _escapeRegexRangeChars(self.endQuoteChar[0]),
- (escChar is not None and _escapeRegexRangeChars(escChar) or ''))
- else:
- self.flags = 0
- self.pattern = r'%s(?:[^%s\n\r%s]' % (re.escape(self.quoteChar),
- _escapeRegexRangeChars(self.endQuoteChar[0]),
- (escChar is not None and _escapeRegexRangeChars(escChar) or ''))
- if len(self.endQuoteChar) > 1:
- self.pattern += (
- '|(?:' + ')|(?:'.join("%s[^%s]" % (re.escape(self.endQuoteChar[:i]),
- _escapeRegexRangeChars(self.endQuoteChar[i]))
- for i in range(len(self.endQuoteChar) - 1, 0, -1)) + ')')
-
- if escQuote:
- self.pattern += (r'|(?:%s)' % re.escape(escQuote))
- if escChar:
- self.pattern += (r'|(?:%s.)' % re.escape(escChar))
- self.escCharReplacePattern = re.escape(self.escChar) + "(.)"
- self.pattern += (r')*%s' % re.escape(self.endQuoteChar))
-
- try:
- self.re = re.compile(self.pattern, self.flags)
- self.reString = self.pattern
- self.re_match = self.re.match
- except sre_constants.error:
- warnings.warn("invalid pattern (%s) passed to Regex" % self.pattern,
- SyntaxWarning, stacklevel=2)
- raise
-
- self.name = _ustr(self)
- self.errmsg = "Expected " + self.name
- self.mayIndexError = False
- self.mayReturnEmpty = True
-
- def parseImpl(self, instring, loc, doActions=True):
- result = instring[loc] == self.firstQuoteChar and self.re_match(instring, loc) or None
- if not result:
- raise ParseException(instring, loc, self.errmsg, self)
-
- loc = result.end()
- ret = result.group()
-
- if self.unquoteResults:
-
- # strip off quotes
- ret = ret[self.quoteCharLen: -self.endQuoteCharLen]
-
- if isinstance(ret, basestring):
- # replace escaped whitespace
- if '\\' in ret and self.convertWhitespaceEscapes:
- ws_map = {
- r'\t': '\t',
- r'\n': '\n',
- r'\f': '\f',
- r'\r': '\r',
- }
- for wslit, wschar in ws_map.items():
- ret = ret.replace(wslit, wschar)
-
- # replace escaped characters
- if self.escChar:
- ret = re.sub(self.escCharReplacePattern, r"\g<1>", ret)
-
- # replace escaped quotes
- if self.escQuote:
- ret = ret.replace(self.escQuote, self.endQuoteChar)
-
- return loc, ret
-
- def __str__(self):
- try:
- return super(QuotedString, self).__str__()
- except Exception:
- pass
-
- if self.strRepr is None:
- self.strRepr = "quoted string, starting with %s ending with %s" % (self.quoteChar, self.endQuoteChar)
-
- return self.strRepr
-
-
-class CharsNotIn(Token):
- """Token for matching words composed of characters *not* in a given
- set (will include whitespace in matched characters if not listed in
- the provided exclusion set - see example). Defined with string
- containing all disallowed characters, and an optional minimum,
- maximum, and/or exact length. The default value for ``min`` is
- 1 (a minimum value < 1 is not valid); the default values for
- ``max`` and ``exact`` are 0, meaning no maximum or exact
- length restriction.
-
- Example::
-
- # define a comma-separated-value as anything that is not a ','
- csv_value = CharsNotIn(',')
- print(delimitedList(csv_value).parseString("dkls,lsdkjf,s12 34,@!#,213"))
-
- prints::
-
- ['dkls', 'lsdkjf', 's12 34', '@!#', '213']
- """
- def __init__(self, notChars, min=1, max=0, exact=0):
- super(CharsNotIn, self).__init__()
- self.skipWhitespace = False
- self.notChars = notChars
-
- if min < 1:
- raise ValueError("cannot specify a minimum length < 1; use "
- "Optional(CharsNotIn()) if zero-length char group is permitted")
-
- self.minLen = min
-
- if max > 0:
- self.maxLen = max
- else:
- self.maxLen = _MAX_INT
-
- if exact > 0:
- self.maxLen = exact
- self.minLen = exact
-
- self.name = _ustr(self)
- self.errmsg = "Expected " + self.name
- self.mayReturnEmpty = (self.minLen == 0)
- self.mayIndexError = False
-
- def parseImpl(self, instring, loc, doActions=True):
- if instring[loc] in self.notChars:
- raise ParseException(instring, loc, self.errmsg, self)
-
- start = loc
- loc += 1
- notchars = self.notChars
- maxlen = min(start + self.maxLen, len(instring))
- while loc < maxlen and instring[loc] not in notchars:
- loc += 1
-
- if loc - start < self.minLen:
- raise ParseException(instring, loc, self.errmsg, self)
-
- return loc, instring[start:loc]
-
- def __str__(self):
- try:
- return super(CharsNotIn, self).__str__()
- except Exception:
- pass
-
- if self.strRepr is None:
- if len(self.notChars) > 4:
- self.strRepr = "!W:(%s...)" % self.notChars[:4]
- else:
- self.strRepr = "!W:(%s)" % self.notChars
-
- return self.strRepr
-
-class White(Token):
- """Special matching class for matching whitespace. Normally,
- whitespace is ignored by pyparsing grammars. This class is included
- when some whitespace structures are significant. Define with
- a string containing the whitespace characters to be matched; default
- is ``" \\t\\r\\n"``. Also takes optional ``min``,
- ``max``, and ``exact`` arguments, as defined for the
- :class:`Word` class.
- """
- whiteStrs = {
- ' ' : '<SP>',
- '\t': '<TAB>',
- '\n': '<LF>',
- '\r': '<CR>',
- '\f': '<FF>',
- u'\u00A0': '<NBSP>',
- u'\u1680': '<OGHAM_SPACE_MARK>',
- u'\u180E': '<MONGOLIAN_VOWEL_SEPARATOR>',
- u'\u2000': '<EN_QUAD>',
- u'\u2001': '<EM_QUAD>',
- u'\u2002': '<EN_SPACE>',
- u'\u2003': '<EM_SPACE>',
- u'\u2004': '<THREE-PER-EM_SPACE>',
- u'\u2005': '<FOUR-PER-EM_SPACE>',
- u'\u2006': '<SIX-PER-EM_SPACE>',
- u'\u2007': '<FIGURE_SPACE>',
- u'\u2008': '<PUNCTUATION_SPACE>',
- u'\u2009': '<THIN_SPACE>',
- u'\u200A': '<HAIR_SPACE>',
- u'\u200B': '<ZERO_WIDTH_SPACE>',
- u'\u202F': '<NNBSP>',
- u'\u205F': '<MMSP>',
- u'\u3000': '<IDEOGRAPHIC_SPACE>',
- }
- def __init__(self, ws=" \t\r\n", min=1, max=0, exact=0):
- super(White, self).__init__()
- self.matchWhite = ws
- self.setWhitespaceChars("".join(c for c in self.whiteChars if c not in self.matchWhite))
- # ~ self.leaveWhitespace()
- self.name = ("".join(White.whiteStrs[c] for c in self.matchWhite))
- self.mayReturnEmpty = True
- self.errmsg = "Expected " + self.name
-
- self.minLen = min
-
- if max > 0:
- self.maxLen = max
- else:
- self.maxLen = _MAX_INT
-
- if exact > 0:
- self.maxLen = exact
- self.minLen = exact
-
- def parseImpl(self, instring, loc, doActions=True):
- if instring[loc] not in self.matchWhite:
- raise ParseException(instring, loc, self.errmsg, self)
- start = loc
- loc += 1
- maxloc = start + self.maxLen
- maxloc = min(maxloc, len(instring))
- while loc < maxloc and instring[loc] in self.matchWhite:
- loc += 1
-
- if loc - start < self.minLen:
- raise ParseException(instring, loc, self.errmsg, self)
-
- return loc, instring[start:loc]
-
-
-class _PositionToken(Token):
- def __init__(self):
- super(_PositionToken, self).__init__()
- self.name = self.__class__.__name__
- self.mayReturnEmpty = True
- self.mayIndexError = False
-
-class GoToColumn(_PositionToken):
- """Token to advance to a specific column of input text; useful for
- tabular report scraping.
- """
- def __init__(self, colno):
- super(GoToColumn, self).__init__()
- self.col = colno
-
- def preParse(self, instring, loc):
- if col(loc, instring) != self.col:
- instrlen = len(instring)
- if self.ignoreExprs:
- loc = self._skipIgnorables(instring, loc)
- while loc < instrlen and instring[loc].isspace() and col(loc, instring) != self.col:
- loc += 1
- return loc
-
- def parseImpl(self, instring, loc, doActions=True):
- thiscol = col(loc, instring)
- if thiscol > self.col:
- raise ParseException(instring, loc, "Text not in expected column", self)
- newloc = loc + self.col - thiscol
- ret = instring[loc: newloc]
- return newloc, ret
-
-
-class LineStart(_PositionToken):
- r"""Matches if current position is at the beginning of a line within
- the parse string
-
- Example::
-
- test = '''\
- AAA this line
- AAA and this line
- AAA but not this one
- B AAA and definitely not this one
- '''
-
- for t in (LineStart() + 'AAA' + restOfLine).searchString(test):
- print(t)
-
- prints::
-
- ['AAA', ' this line']
- ['AAA', ' and this line']
-
- """
- def __init__(self):
- super(LineStart, self).__init__()
- self.errmsg = "Expected start of line"
-
- def parseImpl(self, instring, loc, doActions=True):
- if col(loc, instring) == 1:
- return loc, []
- raise ParseException(instring, loc, self.errmsg, self)
-
-class LineEnd(_PositionToken):
- """Matches if current position is at the end of a line within the
- parse string
- """
- def __init__(self):
- super(LineEnd, self).__init__()
- self.setWhitespaceChars(ParserElement.DEFAULT_WHITE_CHARS.replace("\n", ""))
- self.errmsg = "Expected end of line"
-
- def parseImpl(self, instring, loc, doActions=True):
- if loc < len(instring):
- if instring[loc] == "\n":
- return loc + 1, "\n"
- else:
- raise ParseException(instring, loc, self.errmsg, self)
- elif loc == len(instring):
- return loc + 1, []
- else:
- raise ParseException(instring, loc, self.errmsg, self)
-
-class StringStart(_PositionToken):
- """Matches if current position is at the beginning of the parse
- string
- """
- def __init__(self):
- super(StringStart, self).__init__()
- self.errmsg = "Expected start of text"
-
- def parseImpl(self, instring, loc, doActions=True):
- if loc != 0:
- # see if entire string up to here is just whitespace and ignoreables
- if loc != self.preParse(instring, 0):
- raise ParseException(instring, loc, self.errmsg, self)
- return loc, []
-
-class StringEnd(_PositionToken):
- """Matches if current position is at the end of the parse string
- """
- def __init__(self):
- super(StringEnd, self).__init__()
- self.errmsg = "Expected end of text"
-
- def parseImpl(self, instring, loc, doActions=True):
- if loc < len(instring):
- raise ParseException(instring, loc, self.errmsg, self)
- elif loc == len(instring):
- return loc + 1, []
- elif loc > len(instring):
- return loc, []
- else:
- raise ParseException(instring, loc, self.errmsg, self)
-
-class WordStart(_PositionToken):
- """Matches if the current position is at the beginning of a Word,
- and is not preceded by any character in a given set of
- ``wordChars`` (default= ``printables``). To emulate the
- ``\b`` behavior of regular expressions, use
- ``WordStart(alphanums)``. ``WordStart`` will also match at
- the beginning of the string being parsed, or at the beginning of
- a line.
- """
- def __init__(self, wordChars=printables):
- super(WordStart, self).__init__()
- self.wordChars = set(wordChars)
- self.errmsg = "Not at the start of a word"
-
- def parseImpl(self, instring, loc, doActions=True):
- if loc != 0:
- if (instring[loc - 1] in self.wordChars
- or instring[loc] not in self.wordChars):
- raise ParseException(instring, loc, self.errmsg, self)
- return loc, []
-
-class WordEnd(_PositionToken):
- """Matches if the current position is at the end of a Word, and is
- not followed by any character in a given set of ``wordChars``
- (default= ``printables``). To emulate the ``\b`` behavior of
- regular expressions, use ``WordEnd(alphanums)``. ``WordEnd``
- will also match at the end of the string being parsed, or at the end
- of a line.
- """
- def __init__(self, wordChars=printables):
- super(WordEnd, self).__init__()
- self.wordChars = set(wordChars)
- self.skipWhitespace = False
- self.errmsg = "Not at the end of a word"
-
- def parseImpl(self, instring, loc, doActions=True):
- instrlen = len(instring)
- if instrlen > 0 and loc < instrlen:
- if (instring[loc] in self.wordChars or
- instring[loc - 1] not in self.wordChars):
- raise ParseException(instring, loc, self.errmsg, self)
- return loc, []
-
-
-class ParseExpression(ParserElement):
- """Abstract subclass of ParserElement, for combining and
- post-processing parsed tokens.
- """
- def __init__(self, exprs, savelist=False):
- super(ParseExpression, self).__init__(savelist)
- if isinstance(exprs, _generatorType):
- exprs = list(exprs)
-
- if isinstance(exprs, basestring):
- self.exprs = [self._literalStringClass(exprs)]
- elif isinstance(exprs, ParserElement):
- self.exprs = [exprs]
- elif isinstance(exprs, Iterable):
- exprs = list(exprs)
- # if sequence of strings provided, wrap with Literal
- if any(isinstance(expr, basestring) for expr in exprs):
- exprs = (self._literalStringClass(e) if isinstance(e, basestring) else e for e in exprs)
- self.exprs = list(exprs)
- else:
- try:
- self.exprs = list(exprs)
- except TypeError:
- self.exprs = [exprs]
- self.callPreparse = False
-
- def append(self, other):
- self.exprs.append(other)
- self.strRepr = None
- return self
-
- def leaveWhitespace(self):
- """Extends ``leaveWhitespace`` defined in base class, and also invokes ``leaveWhitespace`` on
- all contained expressions."""
- self.skipWhitespace = False
- self.exprs = [e.copy() for e in self.exprs]
- for e in self.exprs:
- e.leaveWhitespace()
- return self
-
- def ignore(self, other):
- if isinstance(other, Suppress):
- if other not in self.ignoreExprs:
- super(ParseExpression, self).ignore(other)
- for e in self.exprs:
- e.ignore(self.ignoreExprs[-1])
- else:
- super(ParseExpression, self).ignore(other)
- for e in self.exprs:
- e.ignore(self.ignoreExprs[-1])
- return self
-
- def __str__(self):
- try:
- return super(ParseExpression, self).__str__()
- except Exception:
- pass
-
- if self.strRepr is None:
- self.strRepr = "%s:(%s)" % (self.__class__.__name__, _ustr(self.exprs))
- return self.strRepr
-
- def streamline(self):
- super(ParseExpression, self).streamline()
-
- for e in self.exprs:
- e.streamline()
-
- # collapse nested And's of the form And(And(And(a, b), c), d) to And(a, b, c, d)
- # but only if there are no parse actions or resultsNames on the nested And's
- # (likewise for Or's and MatchFirst's)
- if len(self.exprs) == 2:
- other = self.exprs[0]
- if (isinstance(other, self.__class__)
- and not other.parseAction
- and other.resultsName is None
- and not other.debug):
- self.exprs = other.exprs[:] + [self.exprs[1]]
- self.strRepr = None
- self.mayReturnEmpty |= other.mayReturnEmpty
- self.mayIndexError |= other.mayIndexError
-
- other = self.exprs[-1]
- if (isinstance(other, self.__class__)
- and not other.parseAction
- and other.resultsName is None
- and not other.debug):
- self.exprs = self.exprs[:-1] + other.exprs[:]
- self.strRepr = None
- self.mayReturnEmpty |= other.mayReturnEmpty
- self.mayIndexError |= other.mayIndexError
-
- self.errmsg = "Expected " + _ustr(self)
-
- return self
-
- def validate(self, validateTrace=None):
- tmp = (validateTrace if validateTrace is not None else [])[:] + [self]
- for e in self.exprs:
- e.validate(tmp)
- self.checkRecursion([])
-
- def copy(self):
- ret = super(ParseExpression, self).copy()
- ret.exprs = [e.copy() for e in self.exprs]
- return ret
-
- def _setResultsName(self, name, listAllMatches=False):
- if __diag__.warn_ungrouped_named_tokens_in_collection:
- for e in self.exprs:
- if isinstance(e, ParserElement) and e.resultsName:
- warnings.warn("{0}: setting results name {1!r} on {2} expression "
- "collides with {3!r} on contained expression".format("warn_ungrouped_named_tokens_in_collection",
- name,
- type(self).__name__,
- e.resultsName),
- stacklevel=3)
-
- return super(ParseExpression, self)._setResultsName(name, listAllMatches)
-
-
-class And(ParseExpression):
- """
- Requires all given :class:`ParseExpression` s to be found in the given order.
- Expressions may be separated by whitespace.
- May be constructed using the ``'+'`` operator.
- May also be constructed using the ``'-'`` operator, which will
- suppress backtracking.
-
- Example::
-
- integer = Word(nums)
- name_expr = OneOrMore(Word(alphas))
-
- expr = And([integer("id"), name_expr("name"), integer("age")])
- # more easily written as:
- expr = integer("id") + name_expr("name") + integer("age")
- """
-
- class _ErrorStop(Empty):
- def __init__(self, *args, **kwargs):
- super(And._ErrorStop, self).__init__(*args, **kwargs)
- self.name = '-'
- self.leaveWhitespace()
-
- def __init__(self, exprs, savelist=True):
- exprs = list(exprs)
- if exprs and Ellipsis in exprs:
- tmp = []
- for i, expr in enumerate(exprs):
- if expr is Ellipsis:
- if i < len(exprs) - 1:
- skipto_arg = (Empty() + exprs[i + 1]).exprs[-1]
- tmp.append(SkipTo(skipto_arg)("_skipped*"))
- else:
- raise Exception("cannot construct And with sequence ending in ...")
- else:
- tmp.append(expr)
- exprs[:] = tmp
- super(And, self).__init__(exprs, savelist)
- self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
- self.setWhitespaceChars(self.exprs[0].whiteChars)
- self.skipWhitespace = self.exprs[0].skipWhitespace
- self.callPreparse = True
-
- def streamline(self):
- # collapse any _PendingSkip's
- if self.exprs:
- if any(isinstance(e, ParseExpression) and e.exprs and isinstance(e.exprs[-1], _PendingSkip)
- for e in self.exprs[:-1]):
- for i, e in enumerate(self.exprs[:-1]):
- if e is None:
- continue
- if (isinstance(e, ParseExpression)
- and e.exprs and isinstance(e.exprs[-1], _PendingSkip)):
- e.exprs[-1] = e.exprs[-1] + self.exprs[i + 1]
- self.exprs[i + 1] = None
- self.exprs = [e for e in self.exprs if e is not None]
-
- super(And, self).streamline()
- self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
- return self
-
- def parseImpl(self, instring, loc, doActions=True):
- # pass False as last arg to _parse for first element, since we already
- # pre-parsed the string as part of our And pre-parsing
- loc, resultlist = self.exprs[0]._parse(instring, loc, doActions, callPreParse=False)
- errorStop = False
- for e in self.exprs[1:]:
- if isinstance(e, And._ErrorStop):
- errorStop = True
- continue
- if errorStop:
- try:
- loc, exprtokens = e._parse(instring, loc, doActions)
- except ParseSyntaxException:
- raise
- except ParseBaseException as pe:
- pe.__traceback__ = None
- raise ParseSyntaxException._from_exception(pe)
- except IndexError:
- raise ParseSyntaxException(instring, len(instring), self.errmsg, self)
- else:
- loc, exprtokens = e._parse(instring, loc, doActions)
- if exprtokens or exprtokens.haskeys():
- resultlist += exprtokens
- return loc, resultlist
-
- def __iadd__(self, other):
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- return self.append(other) # And([self, other])
-
- def checkRecursion(self, parseElementList):
- subRecCheckList = parseElementList[:] + [self]
- for e in self.exprs:
- e.checkRecursion(subRecCheckList)
- if not e.mayReturnEmpty:
- break
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
-
- if self.strRepr is None:
- self.strRepr = "{" + " ".join(_ustr(e) for e in self.exprs) + "}"
-
- return self.strRepr
-
-
-class Or(ParseExpression):
- """Requires that at least one :class:`ParseExpression` is found. If
- two expressions match, the expression that matches the longest
- string will be used. May be constructed using the ``'^'``
- operator.
-
- Example::
-
- # construct Or using '^' operator
-
- number = Word(nums) ^ Combine(Word(nums) + '.' + Word(nums))
- print(number.searchString("123 3.1416 789"))
-
- prints::
-
- [['123'], ['3.1416'], ['789']]
- """
- def __init__(self, exprs, savelist=False):
- super(Or, self).__init__(exprs, savelist)
- if self.exprs:
- self.mayReturnEmpty = any(e.mayReturnEmpty for e in self.exprs)
- else:
- self.mayReturnEmpty = True
-
- def streamline(self):
- super(Or, self).streamline()
- if __compat__.collect_all_And_tokens:
- self.saveAsList = any(e.saveAsList for e in self.exprs)
- return self
-
- def parseImpl(self, instring, loc, doActions=True):
- maxExcLoc = -1
- maxException = None
- matches = []
- for e in self.exprs:
- try:
- loc2 = e.tryParse(instring, loc)
- except ParseException as err:
- err.__traceback__ = None
- if err.loc > maxExcLoc:
- maxException = err
- maxExcLoc = err.loc
- except IndexError:
- if len(instring) > maxExcLoc:
- maxException = ParseException(instring, len(instring), e.errmsg, self)
- maxExcLoc = len(instring)
- else:
- # save match among all matches, to retry longest to shortest
- matches.append((loc2, e))
-
- if matches:
- # re-evaluate all matches in descending order of length of match, in case attached actions
- # might change whether or how much they match of the input.
- matches.sort(key=itemgetter(0), reverse=True)
-
- if not doActions:
- # no further conditions or parse actions to change the selection of
- # alternative, so the first match will be the best match
- best_expr = matches[0][1]
- return best_expr._parse(instring, loc, doActions)
-
- longest = -1, None
- for loc1, expr1 in matches:
- if loc1 <= longest[0]:
- # already have a longer match than this one will deliver, we are done
- return longest
-
- try:
- loc2, toks = expr1._parse(instring, loc, doActions)
- except ParseException as err:
- err.__traceback__ = None
- if err.loc > maxExcLoc:
- maxException = err
- maxExcLoc = err.loc
- else:
- if loc2 >= loc1:
- return loc2, toks
- # didn't match as much as before
- elif loc2 > longest[0]:
- longest = loc2, toks
-
- if longest != (-1, None):
- return longest
-
- if maxException is not None:
- maxException.msg = self.errmsg
- raise maxException
- else:
- raise ParseException(instring, loc, "no defined alternatives to match", self)
-
-
- def __ixor__(self, other):
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- return self.append(other) # Or([self, other])
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
-
- if self.strRepr is None:
- self.strRepr = "{" + " ^ ".join(_ustr(e) for e in self.exprs) + "}"
-
- return self.strRepr
-
- def checkRecursion(self, parseElementList):
- subRecCheckList = parseElementList[:] + [self]
- for e in self.exprs:
- e.checkRecursion(subRecCheckList)
-
- def _setResultsName(self, name, listAllMatches=False):
- if (not __compat__.collect_all_And_tokens
- and __diag__.warn_multiple_tokens_in_named_alternation):
- if any(isinstance(e, And) for e in self.exprs):
- warnings.warn("{0}: setting results name {1!r} on {2} expression "
- "may only return a single token for an And alternative, "
- "in future will return the full list of tokens".format(
- "warn_multiple_tokens_in_named_alternation", name, type(self).__name__),
- stacklevel=3)
-
- return super(Or, self)._setResultsName(name, listAllMatches)
-
-
-class MatchFirst(ParseExpression):
- """Requires that at least one :class:`ParseExpression` is found. If
- two expressions match, the first one listed is the one that will
- match. May be constructed using the ``'|'`` operator.
-
- Example::
-
- # construct MatchFirst using '|' operator
-
- # watch the order of expressions to match
- number = Word(nums) | Combine(Word(nums) + '.' + Word(nums))
- print(number.searchString("123 3.1416 789")) # Fail! -> [['123'], ['3'], ['1416'], ['789']]
-
- # put more selective expression first
- number = Combine(Word(nums) + '.' + Word(nums)) | Word(nums)
- print(number.searchString("123 3.1416 789")) # Better -> [['123'], ['3.1416'], ['789']]
- """
- def __init__(self, exprs, savelist=False):
- super(MatchFirst, self).__init__(exprs, savelist)
- if self.exprs:
- self.mayReturnEmpty = any(e.mayReturnEmpty for e in self.exprs)
- else:
- self.mayReturnEmpty = True
-
- def streamline(self):
- super(MatchFirst, self).streamline()
- if __compat__.collect_all_And_tokens:
- self.saveAsList = any(e.saveAsList for e in self.exprs)
- return self
-
- def parseImpl(self, instring, loc, doActions=True):
- maxExcLoc = -1
- maxException = None
- for e in self.exprs:
- try:
- ret = e._parse(instring, loc, doActions)
- return ret
- except ParseException as err:
- if err.loc > maxExcLoc:
- maxException = err
- maxExcLoc = err.loc
- except IndexError:
- if len(instring) > maxExcLoc:
- maxException = ParseException(instring, len(instring), e.errmsg, self)
- maxExcLoc = len(instring)
-
- # only got here if no expression matched, raise exception for match that made it the furthest
- else:
- if maxException is not None:
- maxException.msg = self.errmsg
- raise maxException
- else:
- raise ParseException(instring, loc, "no defined alternatives to match", self)
-
- def __ior__(self, other):
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- return self.append(other) # MatchFirst([self, other])
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
-
- if self.strRepr is None:
- self.strRepr = "{" + " | ".join(_ustr(e) for e in self.exprs) + "}"
-
- return self.strRepr
-
- def checkRecursion(self, parseElementList):
- subRecCheckList = parseElementList[:] + [self]
- for e in self.exprs:
- e.checkRecursion(subRecCheckList)
-
- def _setResultsName(self, name, listAllMatches=False):
- if (not __compat__.collect_all_And_tokens
- and __diag__.warn_multiple_tokens_in_named_alternation):
- if any(isinstance(e, And) for e in self.exprs):
- warnings.warn("{0}: setting results name {1!r} on {2} expression "
- "may only return a single token for an And alternative, "
- "in future will return the full list of tokens".format(
- "warn_multiple_tokens_in_named_alternation", name, type(self).__name__),
- stacklevel=3)
-
- return super(MatchFirst, self)._setResultsName(name, listAllMatches)
-
-
-class Each(ParseExpression):
- """Requires all given :class:`ParseExpression` s to be found, but in
- any order. Expressions may be separated by whitespace.
-
- May be constructed using the ``'&'`` operator.
-
- Example::
-
- color = oneOf("RED ORANGE YELLOW GREEN BLUE PURPLE BLACK WHITE BROWN")
- shape_type = oneOf("SQUARE CIRCLE TRIANGLE STAR HEXAGON OCTAGON")
- integer = Word(nums)
- shape_attr = "shape:" + shape_type("shape")
- posn_attr = "posn:" + Group(integer("x") + ',' + integer("y"))("posn")
- color_attr = "color:" + color("color")
- size_attr = "size:" + integer("size")
-
- # use Each (using operator '&') to accept attributes in any order
- # (shape and posn are required, color and size are optional)
- shape_spec = shape_attr & posn_attr & Optional(color_attr) & Optional(size_attr)
-
- shape_spec.runTests('''
- shape: SQUARE color: BLACK posn: 100, 120
- shape: CIRCLE size: 50 color: BLUE posn: 50,80
- color:GREEN size:20 shape:TRIANGLE posn:20,40
- '''
- )
-
- prints::
-
- shape: SQUARE color: BLACK posn: 100, 120
- ['shape:', 'SQUARE', 'color:', 'BLACK', 'posn:', ['100', ',', '120']]
- - color: BLACK
- - posn: ['100', ',', '120']
- - x: 100
- - y: 120
- - shape: SQUARE
-
-
- shape: CIRCLE size: 50 color: BLUE posn: 50,80
- ['shape:', 'CIRCLE', 'size:', '50', 'color:', 'BLUE', 'posn:', ['50', ',', '80']]
- - color: BLUE
- - posn: ['50', ',', '80']
- - x: 50
- - y: 80
- - shape: CIRCLE
- - size: 50
-
-
- color: GREEN size: 20 shape: TRIANGLE posn: 20,40
- ['color:', 'GREEN', 'size:', '20', 'shape:', 'TRIANGLE', 'posn:', ['20', ',', '40']]
- - color: GREEN
- - posn: ['20', ',', '40']
- - x: 20
- - y: 40
- - shape: TRIANGLE
- - size: 20
- """
- def __init__(self, exprs, savelist=True):
- super(Each, self).__init__(exprs, savelist)
- self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
- self.skipWhitespace = True
- self.initExprGroups = True
- self.saveAsList = True
-
- def streamline(self):
- super(Each, self).streamline()
- self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
- return self
-
- def parseImpl(self, instring, loc, doActions=True):
- if self.initExprGroups:
- self.opt1map = dict((id(e.expr), e) for e in self.exprs if isinstance(e, Optional))
- opt1 = [e.expr for e in self.exprs if isinstance(e, Optional)]
- opt2 = [e for e in self.exprs if e.mayReturnEmpty and not isinstance(e, (Optional, Regex))]
- self.optionals = opt1 + opt2
- self.multioptionals = [e.expr for e in self.exprs if isinstance(e, ZeroOrMore)]
- self.multirequired = [e.expr for e in self.exprs if isinstance(e, OneOrMore)]
- self.required = [e for e in self.exprs if not isinstance(e, (Optional, ZeroOrMore, OneOrMore))]
- self.required += self.multirequired
- self.initExprGroups = False
- tmpLoc = loc
- tmpReqd = self.required[:]
- tmpOpt = self.optionals[:]
- matchOrder = []
-
- keepMatching = True
- while keepMatching:
- tmpExprs = tmpReqd + tmpOpt + self.multioptionals + self.multirequired
- failed = []
- for e in tmpExprs:
- try:
- tmpLoc = e.tryParse(instring, tmpLoc)
- except ParseException:
- failed.append(e)
- else:
- matchOrder.append(self.opt1map.get(id(e), e))
- if e in tmpReqd:
- tmpReqd.remove(e)
- elif e in tmpOpt:
- tmpOpt.remove(e)
- if len(failed) == len(tmpExprs):
- keepMatching = False
-
- if tmpReqd:
- missing = ", ".join(_ustr(e) for e in tmpReqd)
- raise ParseException(instring, loc, "Missing one or more required elements (%s)" % missing)
-
- # add any unmatched Optionals, in case they have default values defined
- matchOrder += [e for e in self.exprs if isinstance(e, Optional) and e.expr in tmpOpt]
-
- resultlist = []
- for e in matchOrder:
- loc, results = e._parse(instring, loc, doActions)
- resultlist.append(results)
-
- finalResults = sum(resultlist, ParseResults([]))
- return loc, finalResults
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
-
- if self.strRepr is None:
- self.strRepr = "{" + " & ".join(_ustr(e) for e in self.exprs) + "}"
-
- return self.strRepr
-
- def checkRecursion(self, parseElementList):
- subRecCheckList = parseElementList[:] + [self]
- for e in self.exprs:
- e.checkRecursion(subRecCheckList)
-
-
-class ParseElementEnhance(ParserElement):
- """Abstract subclass of :class:`ParserElement`, for combining and
- post-processing parsed tokens.
- """
- def __init__(self, expr, savelist=False):
- super(ParseElementEnhance, self).__init__(savelist)
- if isinstance(expr, basestring):
- if issubclass(self._literalStringClass, Token):
- expr = self._literalStringClass(expr)
- else:
- expr = self._literalStringClass(Literal(expr))
- self.expr = expr
- self.strRepr = None
- if expr is not None:
- self.mayIndexError = expr.mayIndexError
- self.mayReturnEmpty = expr.mayReturnEmpty
- self.setWhitespaceChars(expr.whiteChars)
- self.skipWhitespace = expr.skipWhitespace
- self.saveAsList = expr.saveAsList
- self.callPreparse = expr.callPreparse
- self.ignoreExprs.extend(expr.ignoreExprs)
-
- def parseImpl(self, instring, loc, doActions=True):
- if self.expr is not None:
- return self.expr._parse(instring, loc, doActions, callPreParse=False)
- else:
- raise ParseException("", loc, self.errmsg, self)
-
- def leaveWhitespace(self):
- self.skipWhitespace = False
- self.expr = self.expr.copy()
- if self.expr is not None:
- self.expr.leaveWhitespace()
- return self
-
- def ignore(self, other):
- if isinstance(other, Suppress):
- if other not in self.ignoreExprs:
- super(ParseElementEnhance, self).ignore(other)
- if self.expr is not None:
- self.expr.ignore(self.ignoreExprs[-1])
- else:
- super(ParseElementEnhance, self).ignore(other)
- if self.expr is not None:
- self.expr.ignore(self.ignoreExprs[-1])
- return self
-
- def streamline(self):
- super(ParseElementEnhance, self).streamline()
- if self.expr is not None:
- self.expr.streamline()
- return self
-
- def checkRecursion(self, parseElementList):
- if self in parseElementList:
- raise RecursiveGrammarException(parseElementList + [self])
- subRecCheckList = parseElementList[:] + [self]
- if self.expr is not None:
- self.expr.checkRecursion(subRecCheckList)
-
- def validate(self, validateTrace=None):
- if validateTrace is None:
- validateTrace = []
- tmp = validateTrace[:] + [self]
- if self.expr is not None:
- self.expr.validate(tmp)
- self.checkRecursion([])
-
- def __str__(self):
- try:
- return super(ParseElementEnhance, self).__str__()
- except Exception:
- pass
-
- if self.strRepr is None and self.expr is not None:
- self.strRepr = "%s:(%s)" % (self.__class__.__name__, _ustr(self.expr))
- return self.strRepr
-
-
-class FollowedBy(ParseElementEnhance):
- """Lookahead matching of the given parse expression.
- ``FollowedBy`` does *not* advance the parsing position within
- the input string, it only verifies that the specified parse
- expression matches at the current position. ``FollowedBy``
- always returns a null token list. If any results names are defined
- in the lookahead expression, those *will* be returned for access by
- name.
-
- Example::
-
- # use FollowedBy to match a label only if it is followed by a ':'
- data_word = Word(alphas)
- label = data_word + FollowedBy(':')
- attr_expr = Group(label + Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join))
-
- OneOrMore(attr_expr).parseString("shape: SQUARE color: BLACK posn: upper left").pprint()
-
- prints::
-
- [['shape', 'SQUARE'], ['color', 'BLACK'], ['posn', 'upper left']]
- """
- def __init__(self, expr):
- super(FollowedBy, self).__init__(expr)
- self.mayReturnEmpty = True
-
- def parseImpl(self, instring, loc, doActions=True):
- # by using self._expr.parse and deleting the contents of the returned ParseResults list
- # we keep any named results that were defined in the FollowedBy expression
- _, ret = self.expr._parse(instring, loc, doActions=doActions)
- del ret[:]
-
- return loc, ret
-
-
-class PrecededBy(ParseElementEnhance):
- """Lookbehind matching of the given parse expression.
- ``PrecededBy`` does not advance the parsing position within the
- input string, it only verifies that the specified parse expression
- matches prior to the current position. ``PrecededBy`` always
- returns a null token list, but if a results name is defined on the
- given expression, it is returned.
-
- Parameters:
-
- - expr - expression that must match prior to the current parse
- location
- - retreat - (default= ``None``) - (int) maximum number of characters
- to lookbehind prior to the current parse location
-
- If the lookbehind expression is a string, Literal, Keyword, or
- a Word or CharsNotIn with a specified exact or maximum length, then
- the retreat parameter is not required. Otherwise, retreat must be
- specified to give a maximum number of characters to look back from
- the current parse position for a lookbehind match.
-
- Example::
-
- # VB-style variable names with type prefixes
- int_var = PrecededBy("#") + pyparsing_common.identifier
- str_var = PrecededBy("$") + pyparsing_common.identifier
-
- """
- def __init__(self, expr, retreat=None):
- super(PrecededBy, self).__init__(expr)
- self.expr = self.expr().leaveWhitespace()
- self.mayReturnEmpty = True
- self.mayIndexError = False
- self.exact = False
- if isinstance(expr, str):
- retreat = len(expr)
- self.exact = True
- elif isinstance(expr, (Literal, Keyword)):
- retreat = expr.matchLen
- self.exact = True
- elif isinstance(expr, (Word, CharsNotIn)) and expr.maxLen != _MAX_INT:
- retreat = expr.maxLen
- self.exact = True
- elif isinstance(expr, _PositionToken):
- retreat = 0
- self.exact = True
- self.retreat = retreat
- self.errmsg = "not preceded by " + str(expr)
- self.skipWhitespace = False
- self.parseAction.append(lambda s, l, t: t.__delitem__(slice(None, None)))
-
- def parseImpl(self, instring, loc=0, doActions=True):
- if self.exact:
- if loc < self.retreat:
- raise ParseException(instring, loc, self.errmsg)
- start = loc - self.retreat
- _, ret = self.expr._parse(instring, start)
- else:
- # retreat specified a maximum lookbehind window, iterate
- test_expr = self.expr + StringEnd()
- instring_slice = instring[max(0, loc - self.retreat):loc]
- last_expr = ParseException(instring, loc, self.errmsg)
- for offset in range(1, min(loc, self.retreat + 1)+1):
- try:
- # print('trying', offset, instring_slice, repr(instring_slice[loc - offset:]))
- _, ret = test_expr._parse(instring_slice, len(instring_slice) - offset)
- except ParseBaseException as pbe:
- last_expr = pbe
- else:
- break
- else:
- raise last_expr
- return loc, ret
-
-
-class NotAny(ParseElementEnhance):
- """Lookahead to disallow matching with the given parse expression.
- ``NotAny`` does *not* advance the parsing position within the
- input string, it only verifies that the specified parse expression
- does *not* match at the current position. Also, ``NotAny`` does
- *not* skip over leading whitespace. ``NotAny`` always returns
- a null token list. May be constructed using the '~' operator.
-
- Example::
-
- AND, OR, NOT = map(CaselessKeyword, "AND OR NOT".split())
-
- # take care not to mistake keywords for identifiers
- ident = ~(AND | OR | NOT) + Word(alphas)
- boolean_term = Optional(NOT) + ident
-
- # very crude boolean expression - to support parenthesis groups and
- # operation hierarchy, use infixNotation
- boolean_expr = boolean_term + ZeroOrMore((AND | OR) + boolean_term)
-
- # integers that are followed by "." are actually floats
- integer = Word(nums) + ~Char(".")
- """
- def __init__(self, expr):
- super(NotAny, self).__init__(expr)
- # ~ self.leaveWhitespace()
- self.skipWhitespace = False # do NOT use self.leaveWhitespace(), don't want to propagate to exprs
- self.mayReturnEmpty = True
- self.errmsg = "Found unwanted token, " + _ustr(self.expr)
-
- def parseImpl(self, instring, loc, doActions=True):
- if self.expr.canParseNext(instring, loc):
- raise ParseException(instring, loc, self.errmsg, self)
- return loc, []
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
-
- if self.strRepr is None:
- self.strRepr = "~{" + _ustr(self.expr) + "}"
-
- return self.strRepr
-
-class _MultipleMatch(ParseElementEnhance):
- def __init__(self, expr, stopOn=None):
- super(_MultipleMatch, self).__init__(expr)
- self.saveAsList = True
- ender = stopOn
- if isinstance(ender, basestring):
- ender = self._literalStringClass(ender)
- self.stopOn(ender)
-
- def stopOn(self, ender):
- if isinstance(ender, basestring):
- ender = self._literalStringClass(ender)
- self.not_ender = ~ender if ender is not None else None
- return self
-
- def parseImpl(self, instring, loc, doActions=True):
- self_expr_parse = self.expr._parse
- self_skip_ignorables = self._skipIgnorables
- check_ender = self.not_ender is not None
- if check_ender:
- try_not_ender = self.not_ender.tryParse
-
- # must be at least one (but first see if we are the stopOn sentinel;
- # if so, fail)
- if check_ender:
- try_not_ender(instring, loc)
- loc, tokens = self_expr_parse(instring, loc, doActions, callPreParse=False)
- try:
- hasIgnoreExprs = (not not self.ignoreExprs)
- while 1:
- if check_ender:
- try_not_ender(instring, loc)
- if hasIgnoreExprs:
- preloc = self_skip_ignorables(instring, loc)
- else:
- preloc = loc
- loc, tmptokens = self_expr_parse(instring, preloc, doActions)
- if tmptokens or tmptokens.haskeys():
- tokens += tmptokens
- except (ParseException, IndexError):
- pass
-
- return loc, tokens
-
- def _setResultsName(self, name, listAllMatches=False):
- if __diag__.warn_ungrouped_named_tokens_in_collection:
- for e in [self.expr] + getattr(self.expr, 'exprs', []):
- if isinstance(e, ParserElement) and e.resultsName:
- warnings.warn("{0}: setting results name {1!r} on {2} expression "
- "collides with {3!r} on contained expression".format("warn_ungrouped_named_tokens_in_collection",
- name,
- type(self).__name__,
- e.resultsName),
- stacklevel=3)
-
- return super(_MultipleMatch, self)._setResultsName(name, listAllMatches)
-
-
-class OneOrMore(_MultipleMatch):
- """Repetition of one or more of the given expression.
-
- Parameters:
- - expr - expression that must match one or more times
- - stopOn - (default= ``None``) - expression for a terminating sentinel
- (only required if the sentinel would ordinarily match the repetition
- expression)
-
- Example::
-
- data_word = Word(alphas)
- label = data_word + FollowedBy(':')
- attr_expr = Group(label + Suppress(':') + OneOrMore(data_word).setParseAction(' '.join))
-
- text = "shape: SQUARE posn: upper left color: BLACK"
- OneOrMore(attr_expr).parseString(text).pprint() # Fail! read 'color' as data instead of next label -> [['shape', 'SQUARE color']]
-
- # use stopOn attribute for OneOrMore to avoid reading label string as part of the data
- attr_expr = Group(label + Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join))
- OneOrMore(attr_expr).parseString(text).pprint() # Better -> [['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'BLACK']]
-
- # could also be written as
- (attr_expr * (1,)).parseString(text).pprint()
- """
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
-
- if self.strRepr is None:
- self.strRepr = "{" + _ustr(self.expr) + "}..."
-
- return self.strRepr
-
-class ZeroOrMore(_MultipleMatch):
- """Optional repetition of zero or more of the given expression.
-
- Parameters:
- - expr - expression that must match zero or more times
- - stopOn - (default= ``None``) - expression for a terminating sentinel
- (only required if the sentinel would ordinarily match the repetition
- expression)
-
- Example: similar to :class:`OneOrMore`
- """
- def __init__(self, expr, stopOn=None):
- super(ZeroOrMore, self).__init__(expr, stopOn=stopOn)
- self.mayReturnEmpty = True
-
- def parseImpl(self, instring, loc, doActions=True):
- try:
- return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
- except (ParseException, IndexError):
- return loc, []
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
-
- if self.strRepr is None:
- self.strRepr = "[" + _ustr(self.expr) + "]..."
-
- return self.strRepr
-
-
-class _NullToken(object):
- def __bool__(self):
- return False
- __nonzero__ = __bool__
- def __str__(self):
- return ""
-
-class Optional(ParseElementEnhance):
- """Optional matching of the given expression.
-
- Parameters:
- - expr - expression that must match zero or more times
- - default (optional) - value to be returned if the optional expression is not found.
-
- Example::
-
- # US postal code can be a 5-digit zip, plus optional 4-digit qualifier
- zip = Combine(Word(nums, exact=5) + Optional('-' + Word(nums, exact=4)))
- zip.runTests('''
- # traditional ZIP code
- 12345
-
- # ZIP+4 form
- 12101-0001
-
- # invalid ZIP
- 98765-
- ''')
-
- prints::
-
- # traditional ZIP code
- 12345
- ['12345']
-
- # ZIP+4 form
- 12101-0001
- ['12101-0001']
-
- # invalid ZIP
- 98765-
- ^
- FAIL: Expected end of text (at char 5), (line:1, col:6)
- """
- __optionalNotMatched = _NullToken()
-
- def __init__(self, expr, default=__optionalNotMatched):
- super(Optional, self).__init__(expr, savelist=False)
- self.saveAsList = self.expr.saveAsList
- self.defaultValue = default
- self.mayReturnEmpty = True
-
- def parseImpl(self, instring, loc, doActions=True):
- try:
- loc, tokens = self.expr._parse(instring, loc, doActions, callPreParse=False)
- except (ParseException, IndexError):
- if self.defaultValue is not self.__optionalNotMatched:
- if self.expr.resultsName:
- tokens = ParseResults([self.defaultValue])
- tokens[self.expr.resultsName] = self.defaultValue
- else:
- tokens = [self.defaultValue]
- else:
- tokens = []
- return loc, tokens
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
-
- if self.strRepr is None:
- self.strRepr = "[" + _ustr(self.expr) + "]"
-
- return self.strRepr
-
-class SkipTo(ParseElementEnhance):
- """Token for skipping over all undefined text until the matched
- expression is found.
-
- Parameters:
- - expr - target expression marking the end of the data to be skipped
- - include - (default= ``False``) if True, the target expression is also parsed
- (the skipped text and target expression are returned as a 2-element list).
- - ignore - (default= ``None``) used to define grammars (typically quoted strings and
- comments) that might contain false matches to the target expression
- - failOn - (default= ``None``) define expressions that are not allowed to be
- included in the skipped test; if found before the target expression is found,
- the SkipTo is not a match
-
- Example::
-
- report = '''
- Outstanding Issues Report - 1 Jan 2000
-
- # | Severity | Description | Days Open
- -----+----------+-------------------------------------------+-----------
- 101 | Critical | Intermittent system crash | 6
- 94 | Cosmetic | Spelling error on Login ('log|n') | 14
- 79 | Minor | System slow when running too many reports | 47
- '''
- integer = Word(nums)
- SEP = Suppress('|')
- # use SkipTo to simply match everything up until the next SEP
- # - ignore quoted strings, so that a '|' character inside a quoted string does not match
- # - parse action will call token.strip() for each matched token, i.e., the description body
- string_data = SkipTo(SEP, ignore=quotedString)
- string_data.setParseAction(tokenMap(str.strip))
- ticket_expr = (integer("issue_num") + SEP
- + string_data("sev") + SEP
- + string_data("desc") + SEP
- + integer("days_open"))
-
- for tkt in ticket_expr.searchString(report):
- print tkt.dump()
-
- prints::
-
- ['101', 'Critical', 'Intermittent system crash', '6']
- - days_open: 6
- - desc: Intermittent system crash
- - issue_num: 101
- - sev: Critical
- ['94', 'Cosmetic', "Spelling error on Login ('log|n')", '14']
- - days_open: 14
- - desc: Spelling error on Login ('log|n')
- - issue_num: 94
- - sev: Cosmetic
- ['79', 'Minor', 'System slow when running too many reports', '47']
- - days_open: 47
- - desc: System slow when running too many reports
- - issue_num: 79
- - sev: Minor
- """
- def __init__(self, other, include=False, ignore=None, failOn=None):
- super(SkipTo, self).__init__(other)
- self.ignoreExpr = ignore
- self.mayReturnEmpty = True
- self.mayIndexError = False
- self.includeMatch = include
- self.saveAsList = False
- if isinstance(failOn, basestring):
- self.failOn = self._literalStringClass(failOn)
- else:
- self.failOn = failOn
- self.errmsg = "No match found for " + _ustr(self.expr)
-
- def parseImpl(self, instring, loc, doActions=True):
- startloc = loc
- instrlen = len(instring)
- expr = self.expr
- expr_parse = self.expr._parse
- self_failOn_canParseNext = self.failOn.canParseNext if self.failOn is not None else None
- self_ignoreExpr_tryParse = self.ignoreExpr.tryParse if self.ignoreExpr is not None else None
-
- tmploc = loc
- while tmploc <= instrlen:
- if self_failOn_canParseNext is not None:
- # break if failOn expression matches
- if self_failOn_canParseNext(instring, tmploc):
- break
-
- if self_ignoreExpr_tryParse is not None:
- # advance past ignore expressions
- while 1:
- try:
- tmploc = self_ignoreExpr_tryParse(instring, tmploc)
- except ParseBaseException:
- break
-
- try:
- expr_parse(instring, tmploc, doActions=False, callPreParse=False)
- except (ParseException, IndexError):
- # no match, advance loc in string
- tmploc += 1
- else:
- # matched skipto expr, done
- break
-
- else:
- # ran off the end of the input string without matching skipto expr, fail
- raise ParseException(instring, loc, self.errmsg, self)
-
- # build up return values
- loc = tmploc
- skiptext = instring[startloc:loc]
- skipresult = ParseResults(skiptext)
-
- if self.includeMatch:
- loc, mat = expr_parse(instring, loc, doActions, callPreParse=False)
- skipresult += mat
-
- return loc, skipresult
-
-class Forward(ParseElementEnhance):
- """Forward declaration of an expression to be defined later -
- used for recursive grammars, such as algebraic infix notation.
- When the expression is known, it is assigned to the ``Forward``
- variable using the '<<' operator.
-
- Note: take care when assigning to ``Forward`` not to overlook
- precedence of operators.
-
- Specifically, '|' has a lower precedence than '<<', so that::
-
- fwdExpr << a | b | c
-
- will actually be evaluated as::
-
- (fwdExpr << a) | b | c
-
- thereby leaving b and c out as parseable alternatives. It is recommended that you
- explicitly group the values inserted into the ``Forward``::
-
- fwdExpr << (a | b | c)
-
- Converting to use the '<<=' operator instead will avoid this problem.
-
- See :class:`ParseResults.pprint` for an example of a recursive
- parser created using ``Forward``.
- """
- def __init__(self, other=None):
- super(Forward, self).__init__(other, savelist=False)
-
- def __lshift__(self, other):
- if isinstance(other, basestring):
- other = self._literalStringClass(other)
- self.expr = other
- self.strRepr = None
- self.mayIndexError = self.expr.mayIndexError
- self.mayReturnEmpty = self.expr.mayReturnEmpty
- self.setWhitespaceChars(self.expr.whiteChars)
- self.skipWhitespace = self.expr.skipWhitespace
- self.saveAsList = self.expr.saveAsList
- self.ignoreExprs.extend(self.expr.ignoreExprs)
- return self
-
- def __ilshift__(self, other):
- return self << other
-
- def leaveWhitespace(self):
- self.skipWhitespace = False
- return self
-
- def streamline(self):
- if not self.streamlined:
- self.streamlined = True
- if self.expr is not None:
- self.expr.streamline()
- return self
-
- def validate(self, validateTrace=None):
- if validateTrace is None:
- validateTrace = []
-
- if self not in validateTrace:
- tmp = validateTrace[:] + [self]
- if self.expr is not None:
- self.expr.validate(tmp)
- self.checkRecursion([])
-
- def __str__(self):
- if hasattr(self, "name"):
- return self.name
- if self.strRepr is not None:
- return self.strRepr
-
- # Avoid infinite recursion by setting a temporary strRepr
- self.strRepr = ": ..."
-
- # Use the string representation of main expression.
- retString = '...'
- try:
- if self.expr is not None:
- retString = _ustr(self.expr)[:1000]
- else:
- retString = "None"
- finally:
- self.strRepr = self.__class__.__name__ + ": " + retString
- return self.strRepr
-
- def copy(self):
- if self.expr is not None:
- return super(Forward, self).copy()
- else:
- ret = Forward()
- ret <<= self
- return ret
-
- def _setResultsName(self, name, listAllMatches=False):
- if __diag__.warn_name_set_on_empty_Forward:
- if self.expr is None:
- warnings.warn("{0}: setting results name {0!r} on {1} expression "
- "that has no contained expression".format("warn_name_set_on_empty_Forward",
- name,
- type(self).__name__),
- stacklevel=3)
-
- return super(Forward, self)._setResultsName(name, listAllMatches)
-
-class TokenConverter(ParseElementEnhance):
- """
- Abstract subclass of :class:`ParseExpression`, for converting parsed results.
- """
- def __init__(self, expr, savelist=False):
- super(TokenConverter, self).__init__(expr) # , savelist)
- self.saveAsList = False
-
-class Combine(TokenConverter):
- """Converter to concatenate all matching tokens to a single string.
- By default, the matching patterns must also be contiguous in the
- input string; this can be disabled by specifying
- ``'adjacent=False'`` in the constructor.
-
- Example::
-
- real = Word(nums) + '.' + Word(nums)
- print(real.parseString('3.1416')) # -> ['3', '.', '1416']
- # will also erroneously match the following
- print(real.parseString('3. 1416')) # -> ['3', '.', '1416']
-
- real = Combine(Word(nums) + '.' + Word(nums))
- print(real.parseString('3.1416')) # -> ['3.1416']
- # no match when there are internal spaces
- print(real.parseString('3. 1416')) # -> Exception: Expected W:(0123...)
- """
- def __init__(self, expr, joinString="", adjacent=True):
- super(Combine, self).__init__(expr)
- # suppress whitespace-stripping in contained parse expressions, but re-enable it on the Combine itself
- if adjacent:
- self.leaveWhitespace()
- self.adjacent = adjacent
- self.skipWhitespace = True
- self.joinString = joinString
- self.callPreparse = True
-
- def ignore(self, other):
- if self.adjacent:
- ParserElement.ignore(self, other)
- else:
- super(Combine, self).ignore(other)
- return self
-
- def postParse(self, instring, loc, tokenlist):
- retToks = tokenlist.copy()
- del retToks[:]
- retToks += ParseResults(["".join(tokenlist._asStringList(self.joinString))], modal=self.modalResults)
-
- if self.resultsName and retToks.haskeys():
- return [retToks]
- else:
- return retToks
-
-class Group(TokenConverter):
- """Converter to return the matched tokens as a list - useful for
- returning tokens of :class:`ZeroOrMore` and :class:`OneOrMore` expressions.
-
- Example::
-
- ident = Word(alphas)
- num = Word(nums)
- term = ident | num
- func = ident + Optional(delimitedList(term))
- print(func.parseString("fn a, b, 100")) # -> ['fn', 'a', 'b', '100']
-
- func = ident + Group(Optional(delimitedList(term)))
- print(func.parseString("fn a, b, 100")) # -> ['fn', ['a', 'b', '100']]
- """
- def __init__(self, expr):
- super(Group, self).__init__(expr)
- self.saveAsList = True
-
- def postParse(self, instring, loc, tokenlist):
- return [tokenlist]
-
-class Dict(TokenConverter):
- """Converter to return a repetitive expression as a list, but also
- as a dictionary. Each element can also be referenced using the first
- token in the expression as its key. Useful for tabular report
- scraping when the first column can be used as a item key.
-
- Example::
-
- data_word = Word(alphas)
- label = data_word + FollowedBy(':')
- attr_expr = Group(label + Suppress(':') + OneOrMore(data_word).setParseAction(' '.join))
-
- text = "shape: SQUARE posn: upper left color: light blue texture: burlap"
- attr_expr = (label + Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join))
-
- # print attributes as plain groups
- print(OneOrMore(attr_expr).parseString(text).dump())
-
- # instead of OneOrMore(expr), parse using Dict(OneOrMore(Group(expr))) - Dict will auto-assign names
- result = Dict(OneOrMore(Group(attr_expr))).parseString(text)
- print(result.dump())
-
- # access named fields as dict entries, or output as dict
- print(result['shape'])
- print(result.asDict())
-
- prints::
-
- ['shape', 'SQUARE', 'posn', 'upper left', 'color', 'light blue', 'texture', 'burlap']
- [['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'light blue'], ['texture', 'burlap']]
- - color: light blue
- - posn: upper left
- - shape: SQUARE
- - texture: burlap
- SQUARE
- {'color': 'light blue', 'posn': 'upper left', 'texture': 'burlap', 'shape': 'SQUARE'}
-
- See more examples at :class:`ParseResults` of accessing fields by results name.
- """
- def __init__(self, expr):
- super(Dict, self).__init__(expr)
- self.saveAsList = True
-
- def postParse(self, instring, loc, tokenlist):
- for i, tok in enumerate(tokenlist):
- if len(tok) == 0:
- continue
- ikey = tok[0]
- if isinstance(ikey, int):
- ikey = _ustr(tok[0]).strip()
- if len(tok) == 1:
- tokenlist[ikey] = _ParseResultsWithOffset("", i)
- elif len(tok) == 2 and not isinstance(tok[1], ParseResults):
- tokenlist[ikey] = _ParseResultsWithOffset(tok[1], i)
- else:
- dictvalue = tok.copy() # ParseResults(i)
- del dictvalue[0]
- if len(dictvalue) != 1 or (isinstance(dictvalue, ParseResults) and dictvalue.haskeys()):
- tokenlist[ikey] = _ParseResultsWithOffset(dictvalue, i)
- else:
- tokenlist[ikey] = _ParseResultsWithOffset(dictvalue[0], i)
-
- if self.resultsName:
- return [tokenlist]
- else:
- return tokenlist
-
-
-class Suppress(TokenConverter):
- """Converter for ignoring the results of a parsed expression.
-
- Example::
-
- source = "a, b, c,d"
- wd = Word(alphas)
- wd_list1 = wd + ZeroOrMore(',' + wd)
- print(wd_list1.parseString(source))
-
- # often, delimiters that are useful during parsing are just in the
- # way afterward - use Suppress to keep them out of the parsed output
- wd_list2 = wd + ZeroOrMore(Suppress(',') + wd)
- print(wd_list2.parseString(source))
-
- prints::
-
- ['a', ',', 'b', ',', 'c', ',', 'd']
- ['a', 'b', 'c', 'd']
-
- (See also :class:`delimitedList`.)
- """
- def postParse(self, instring, loc, tokenlist):
- return []
-
- def suppress(self):
- return self
-
-
-class OnlyOnce(object):
- """Wrapper for parse actions, to ensure they are only called once.
- """
- def __init__(self, methodCall):
- self.callable = _trim_arity(methodCall)
- self.called = False
- def __call__(self, s, l, t):
- if not self.called:
- results = self.callable(s, l, t)
- self.called = True
- return results
- raise ParseException(s, l, "")
- def reset(self):
- self.called = False
-
-def traceParseAction(f):
- """Decorator for debugging parse actions.
-
- When the parse action is called, this decorator will print
- ``">> entering method-name(line:<current_source_line>, <parse_location>, <matched_tokens>)"``.
- When the parse action completes, the decorator will print
- ``"<<"`` followed by the returned value, or any exception that the parse action raised.
-
- Example::
-
- wd = Word(alphas)
-
- @traceParseAction
- def remove_duplicate_chars(tokens):
- return ''.join(sorted(set(''.join(tokens))))
-
- wds = OneOrMore(wd).setParseAction(remove_duplicate_chars)
- print(wds.parseString("slkdjs sld sldd sdlf sdljf"))
-
- prints::
-
- >>entering remove_duplicate_chars(line: 'slkdjs sld sldd sdlf sdljf', 0, (['slkdjs', 'sld', 'sldd', 'sdlf', 'sdljf'], {}))
- <<leaving remove_duplicate_chars (ret: 'dfjkls')
- ['dfjkls']
- """
- f = _trim_arity(f)
- def z(*paArgs):
- thisFunc = f.__name__
- s, l, t = paArgs[-3:]
- if len(paArgs) > 3:
- thisFunc = paArgs[0].__class__.__name__ + '.' + thisFunc
- sys.stderr.write(">>entering %s(line: '%s', %d, %r)\n" % (thisFunc, line(l, s), l, t))
- try:
- ret = f(*paArgs)
- except Exception as exc:
- sys.stderr.write("<<leaving %s (exception: %s)\n" % (thisFunc, exc))
- raise
- sys.stderr.write("<<leaving %s (ret: %r)\n" % (thisFunc, ret))
- return ret
- try:
- z.__name__ = f.__name__
- except AttributeError:
- pass
- return z
-
-#
-# global helpers
-#
-def delimitedList(expr, delim=",", combine=False):
- """Helper to define a delimited list of expressions - the delimiter
- defaults to ','. By default, the list elements and delimiters can
- have intervening whitespace, and comments, but this can be
- overridden by passing ``combine=True`` in the constructor. If
- ``combine`` is set to ``True``, the matching tokens are
- returned as a single token string, with the delimiters included;
- otherwise, the matching tokens are returned as a list of tokens,
- with the delimiters suppressed.
-
- Example::
-
- delimitedList(Word(alphas)).parseString("aa,bb,cc") # -> ['aa', 'bb', 'cc']
- delimitedList(Word(hexnums), delim=':', combine=True).parseString("AA:BB:CC:DD:EE") # -> ['AA:BB:CC:DD:EE']
- """
- dlName = _ustr(expr) + " [" + _ustr(delim) + " " + _ustr(expr) + "]..."
- if combine:
- return Combine(expr + ZeroOrMore(delim + expr)).setName(dlName)
- else:
- return (expr + ZeroOrMore(Suppress(delim) + expr)).setName(dlName)
-
-def countedArray(expr, intExpr=None):
- """Helper to define a counted list of expressions.
-
- This helper defines a pattern of the form::
-
- integer expr expr expr...
-
- where the leading integer tells how many expr expressions follow.
- The matched tokens returns the array of expr tokens as a list - the
- leading count token is suppressed.
-
- If ``intExpr`` is specified, it should be a pyparsing expression
- that produces an integer value.
-
- Example::
-
- countedArray(Word(alphas)).parseString('2 ab cd ef') # -> ['ab', 'cd']
-
- # in this parser, the leading integer value is given in binary,
- # '10' indicating that 2 values are in the array
- binaryConstant = Word('01').setParseAction(lambda t: int(t[0], 2))
- countedArray(Word(alphas), intExpr=binaryConstant).parseString('10 ab cd ef') # -> ['ab', 'cd']
- """
- arrayExpr = Forward()
- def countFieldParseAction(s, l, t):
- n = t[0]
- arrayExpr << (n and Group(And([expr] * n)) or Group(empty))
- return []
- if intExpr is None:
- intExpr = Word(nums).setParseAction(lambda t: int(t[0]))
- else:
- intExpr = intExpr.copy()
- intExpr.setName("arrayLen")
- intExpr.addParseAction(countFieldParseAction, callDuringTry=True)
- return (intExpr + arrayExpr).setName('(len) ' + _ustr(expr) + '...')
-
-def _flatten(L):
- ret = []
- for i in L:
- if isinstance(i, list):
- ret.extend(_flatten(i))
- else:
- ret.append(i)
- return ret
-
-def matchPreviousLiteral(expr):
- """Helper to define an expression that is indirectly defined from
- the tokens matched in a previous expression, that is, it looks for
- a 'repeat' of a previous expression. For example::
-
- first = Word(nums)
- second = matchPreviousLiteral(first)
- matchExpr = first + ":" + second
-
- will match ``"1:1"``, but not ``"1:2"``. Because this
- matches a previous literal, will also match the leading
- ``"1:1"`` in ``"1:10"``. If this is not desired, use
- :class:`matchPreviousExpr`. Do *not* use with packrat parsing
- enabled.
- """
- rep = Forward()
- def copyTokenToRepeater(s, l, t):
- if t:
- if len(t) == 1:
- rep << t[0]
- else:
- # flatten t tokens
- tflat = _flatten(t.asList())
- rep << And(Literal(tt) for tt in tflat)
- else:
- rep << Empty()
- expr.addParseAction(copyTokenToRepeater, callDuringTry=True)
- rep.setName('(prev) ' + _ustr(expr))
- return rep
-
-def matchPreviousExpr(expr):
- """Helper to define an expression that is indirectly defined from
- the tokens matched in a previous expression, that is, it looks for
- a 'repeat' of a previous expression. For example::
-
- first = Word(nums)
- second = matchPreviousExpr(first)
- matchExpr = first + ":" + second
-
- will match ``"1:1"``, but not ``"1:2"``. Because this
- matches by expressions, will *not* match the leading ``"1:1"``
- in ``"1:10"``; the expressions are evaluated first, and then
- compared, so ``"1"`` is compared with ``"10"``. Do *not* use
- with packrat parsing enabled.
- """
- rep = Forward()
- e2 = expr.copy()
- rep <<= e2
- def copyTokenToRepeater(s, l, t):
- matchTokens = _flatten(t.asList())
- def mustMatchTheseTokens(s, l, t):
- theseTokens = _flatten(t.asList())
- if theseTokens != matchTokens:
- raise ParseException('', 0, '')
- rep.setParseAction(mustMatchTheseTokens, callDuringTry=True)
- expr.addParseAction(copyTokenToRepeater, callDuringTry=True)
- rep.setName('(prev) ' + _ustr(expr))
- return rep
-
-def _escapeRegexRangeChars(s):
- # ~ escape these chars: ^-[]
- for c in r"\^-[]":
- s = s.replace(c, _bslash + c)
- s = s.replace("\n", r"\n")
- s = s.replace("\t", r"\t")
- return _ustr(s)
-
-def oneOf(strs, caseless=False, useRegex=True, asKeyword=False):
- """Helper to quickly define a set of alternative Literals, and makes
- sure to do longest-first testing when there is a conflict,
- regardless of the input order, but returns
- a :class:`MatchFirst` for best performance.
-
- Parameters:
-
- - strs - a string of space-delimited literals, or a collection of
- string literals
- - caseless - (default= ``False``) - treat all literals as
- caseless
- - useRegex - (default= ``True``) - as an optimization, will
- generate a Regex object; otherwise, will generate
- a :class:`MatchFirst` object (if ``caseless=True`` or ``asKeyword=True``, or if
- creating a :class:`Regex` raises an exception)
- - asKeyword - (default=``False``) - enforce Keyword-style matching on the
- generated expressions
-
- Example::
-
- comp_oper = oneOf("< = > <= >= !=")
- var = Word(alphas)
- number = Word(nums)
- term = var | number
- comparison_expr = term + comp_oper + term
- print(comparison_expr.searchString("B = 12 AA=23 B<=AA AA>12"))
-
- prints::
-
- [['B', '=', '12'], ['AA', '=', '23'], ['B', '<=', 'AA'], ['AA', '>', '12']]
- """
- if isinstance(caseless, basestring):
- warnings.warn("More than one string argument passed to oneOf, pass "
- "choices as a list or space-delimited string", stacklevel=2)
-
- if caseless:
- isequal = (lambda a, b: a.upper() == b.upper())
- masks = (lambda a, b: b.upper().startswith(a.upper()))
- parseElementClass = CaselessKeyword if asKeyword else CaselessLiteral
- else:
- isequal = (lambda a, b: a == b)
- masks = (lambda a, b: b.startswith(a))
- parseElementClass = Keyword if asKeyword else Literal
-
- symbols = []
- if isinstance(strs, basestring):
- symbols = strs.split()
- elif isinstance(strs, Iterable):
- symbols = list(strs)
- else:
- warnings.warn("Invalid argument to oneOf, expected string or iterable",
- SyntaxWarning, stacklevel=2)
- if not symbols:
- return NoMatch()
-
- if not asKeyword:
- # if not producing keywords, need to reorder to take care to avoid masking
- # longer choices with shorter ones
- i = 0
- while i < len(symbols) - 1:
- cur = symbols[i]
- for j, other in enumerate(symbols[i + 1:]):
- if isequal(other, cur):
- del symbols[i + j + 1]
- break
- elif masks(cur, other):
- del symbols[i + j + 1]
- symbols.insert(i, other)
- break
- else:
- i += 1
-
- if not (caseless or asKeyword) and useRegex:
- # ~ print (strs, "->", "|".join([_escapeRegexChars(sym) for sym in symbols]))
- try:
- if len(symbols) == len("".join(symbols)):
- return Regex("[%s]" % "".join(_escapeRegexRangeChars(sym) for sym in symbols)).setName(' | '.join(symbols))
- else:
- return Regex("|".join(re.escape(sym) for sym in symbols)).setName(' | '.join(symbols))
- except Exception:
- warnings.warn("Exception creating Regex for oneOf, building MatchFirst",
- SyntaxWarning, stacklevel=2)
-
- # last resort, just use MatchFirst
- return MatchFirst(parseElementClass(sym) for sym in symbols).setName(' | '.join(symbols))
-
-def dictOf(key, value):
- """Helper to easily and clearly define a dictionary by specifying
- the respective patterns for the key and value. Takes care of
- defining the :class:`Dict`, :class:`ZeroOrMore`, and
- :class:`Group` tokens in the proper order. The key pattern
- can include delimiting markers or punctuation, as long as they are
- suppressed, thereby leaving the significant key text. The value
- pattern can include named results, so that the :class:`Dict` results
- can include named token fields.
-
- Example::
-
- text = "shape: SQUARE posn: upper left color: light blue texture: burlap"
- attr_expr = (label + Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join))
- print(OneOrMore(attr_expr).parseString(text).dump())
-
- attr_label = label
- attr_value = Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join)
-
- # similar to Dict, but simpler call format
- result = dictOf(attr_label, attr_value).parseString(text)
- print(result.dump())
- print(result['shape'])
- print(result.shape) # object attribute access works too
- print(result.asDict())
-
- prints::
-
- [['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'light blue'], ['texture', 'burlap']]
- - color: light blue
- - posn: upper left
- - shape: SQUARE
- - texture: burlap
- SQUARE
- SQUARE
- {'color': 'light blue', 'shape': 'SQUARE', 'posn': 'upper left', 'texture': 'burlap'}
- """
- return Dict(OneOrMore(Group(key + value)))
-
-def originalTextFor(expr, asString=True):
- """Helper to return the original, untokenized text for a given
- expression. Useful to restore the parsed fields of an HTML start
- tag into the raw tag text itself, or to revert separate tokens with
- intervening whitespace back to the original matching input text. By
- default, returns astring containing the original parsed text.
-
- If the optional ``asString`` argument is passed as
- ``False``, then the return value is
- a :class:`ParseResults` containing any results names that
- were originally matched, and a single token containing the original
- matched text from the input string. So if the expression passed to
- :class:`originalTextFor` contains expressions with defined
- results names, you must set ``asString`` to ``False`` if you
- want to preserve those results name values.
-
- Example::
-
- src = "this is test <b> bold <i>text</i> </b> normal text "
- for tag in ("b", "i"):
- opener, closer = makeHTMLTags(tag)
- patt = originalTextFor(opener + SkipTo(closer) + closer)
- print(patt.searchString(src)[0])
-
- prints::
-
- ['<b> bold <i>text</i> </b>']
- ['<i>text</i>']
- """
- locMarker = Empty().setParseAction(lambda s, loc, t: loc)
- endlocMarker = locMarker.copy()
- endlocMarker.callPreparse = False
- matchExpr = locMarker("_original_start") + expr + endlocMarker("_original_end")
- if asString:
- extractText = lambda s, l, t: s[t._original_start: t._original_end]
- else:
- def extractText(s, l, t):
- t[:] = [s[t.pop('_original_start'):t.pop('_original_end')]]
- matchExpr.setParseAction(extractText)
- matchExpr.ignoreExprs = expr.ignoreExprs
- return matchExpr
-
-def ungroup(expr):
- """Helper to undo pyparsing's default grouping of And expressions,
- even if all but one are non-empty.
- """
- return TokenConverter(expr).addParseAction(lambda t: t[0])
-
-def locatedExpr(expr):
- """Helper to decorate a returned token with its starting and ending
- locations in the input string.
-
- This helper adds the following results names:
-
- - locn_start = location where matched expression begins
- - locn_end = location where matched expression ends
- - value = the actual parsed results
-
- Be careful if the input text contains ``<TAB>`` characters, you
- may want to call :class:`ParserElement.parseWithTabs`
-
- Example::
-
- wd = Word(alphas)
- for match in locatedExpr(wd).searchString("ljsdf123lksdjjf123lkkjj1222"):
- print(match)
-
- prints::
-
- [[0, 'ljsdf', 5]]
- [[8, 'lksdjjf', 15]]
- [[18, 'lkkjj', 23]]
- """
- locator = Empty().setParseAction(lambda s, l, t: l)
- return Group(locator("locn_start") + expr("value") + locator.copy().leaveWhitespace()("locn_end"))
-
-
-# convenience constants for positional expressions
-empty = Empty().setName("empty")
-lineStart = LineStart().setName("lineStart")
-lineEnd = LineEnd().setName("lineEnd")
-stringStart = StringStart().setName("stringStart")
-stringEnd = StringEnd().setName("stringEnd")
-
-_escapedPunc = Word(_bslash, r"\[]-*.$+^?()~ ", exact=2).setParseAction(lambda s, l, t: t[0][1])
-_escapedHexChar = Regex(r"\\0?[xX][0-9a-fA-F]+").setParseAction(lambda s, l, t: unichr(int(t[0].lstrip(r'\0x'), 16)))
-_escapedOctChar = Regex(r"\\0[0-7]+").setParseAction(lambda s, l, t: unichr(int(t[0][1:], 8)))
-_singleChar = _escapedPunc | _escapedHexChar | _escapedOctChar | CharsNotIn(r'\]', exact=1)
-_charRange = Group(_singleChar + Suppress("-") + _singleChar)
-_reBracketExpr = Literal("[") + Optional("^").setResultsName("negate") + Group(OneOrMore(_charRange | _singleChar)).setResultsName("body") + "]"
-
-def srange(s):
- r"""Helper to easily define string ranges for use in Word
- construction. Borrows syntax from regexp '[]' string range
- definitions::
-
- srange("[0-9]") -> "0123456789"
- srange("[a-z]") -> "abcdefghijklmnopqrstuvwxyz"
- srange("[a-z$_]") -> "abcdefghijklmnopqrstuvwxyz$_"
-
- The input string must be enclosed in []'s, and the returned string
- is the expanded character set joined into a single string. The
- values enclosed in the []'s may be:
-
- - a single character
- - an escaped character with a leading backslash (such as ``\-``
- or ``\]``)
- - an escaped hex character with a leading ``'\x'``
- (``\x21``, which is a ``'!'`` character) (``\0x##``
- is also supported for backwards compatibility)
- - an escaped octal character with a leading ``'\0'``
- (``\041``, which is a ``'!'`` character)
- - a range of any of the above, separated by a dash (``'a-z'``,
- etc.)
- - any combination of the above (``'aeiouy'``,
- ``'a-zA-Z0-9_$'``, etc.)
- """
- _expanded = lambda p: p if not isinstance(p, ParseResults) else ''.join(unichr(c) for c in range(ord(p[0]), ord(p[1]) + 1))
- try:
- return "".join(_expanded(part) for part in _reBracketExpr.parseString(s).body)
- except Exception:
- return ""
-
-def matchOnlyAtCol(n):
- """Helper method for defining parse actions that require matching at
- a specific column in the input text.
- """
- def verifyCol(strg, locn, toks):
- if col(locn, strg) != n:
- raise ParseException(strg, locn, "matched token not at column %d" % n)
- return verifyCol
-
-def replaceWith(replStr):
- """Helper method for common parse actions that simply return
- a literal value. Especially useful when used with
- :class:`transformString<ParserElement.transformString>` ().
-
- Example::
-
- num = Word(nums).setParseAction(lambda toks: int(toks[0]))
- na = oneOf("N/A NA").setParseAction(replaceWith(math.nan))
- term = na | num
-
- OneOrMore(term).parseString("324 234 N/A 234") # -> [324, 234, nan, 234]
- """
- return lambda s, l, t: [replStr]
-
-def removeQuotes(s, l, t):
- """Helper parse action for removing quotation marks from parsed
- quoted strings.
-
- Example::
-
- # by default, quotation marks are included in parsed results
- quotedString.parseString("'Now is the Winter of our Discontent'") # -> ["'Now is the Winter of our Discontent'"]
-
- # use removeQuotes to strip quotation marks from parsed results
- quotedString.setParseAction(removeQuotes)
- quotedString.parseString("'Now is the Winter of our Discontent'") # -> ["Now is the Winter of our Discontent"]
- """
- return t[0][1:-1]
-
-def tokenMap(func, *args):
- """Helper to define a parse action by mapping a function to all
- elements of a ParseResults list. If any additional args are passed,
- they are forwarded to the given function as additional arguments
- after the token, as in
- ``hex_integer = Word(hexnums).setParseAction(tokenMap(int, 16))``,
- which will convert the parsed data to an integer using base 16.
-
- Example (compare the last to example in :class:`ParserElement.transformString`::
-
- hex_ints = OneOrMore(Word(hexnums)).setParseAction(tokenMap(int, 16))
- hex_ints.runTests('''
- 00 11 22 aa FF 0a 0d 1a
- ''')
-
- upperword = Word(alphas).setParseAction(tokenMap(str.upper))
- OneOrMore(upperword).runTests('''
- my kingdom for a horse
- ''')
-
- wd = Word(alphas).setParseAction(tokenMap(str.title))
- OneOrMore(wd).setParseAction(' '.join).runTests('''
- now is the winter of our discontent made glorious summer by this sun of york
- ''')
-
- prints::
-
- 00 11 22 aa FF 0a 0d 1a
- [0, 17, 34, 170, 255, 10, 13, 26]
-
- my kingdom for a horse
- ['MY', 'KINGDOM', 'FOR', 'A', 'HORSE']
-
- now is the winter of our discontent made glorious summer by this sun of york
- ['Now Is The Winter Of Our Discontent Made Glorious Summer By This Sun Of York']
- """
- def pa(s, l, t):
- return [func(tokn, *args) for tokn in t]
-
- try:
- func_name = getattr(func, '__name__',
- getattr(func, '__class__').__name__)
- except Exception:
- func_name = str(func)
- pa.__name__ = func_name
-
- return pa
-
-upcaseTokens = tokenMap(lambda t: _ustr(t).upper())
-"""(Deprecated) Helper parse action to convert tokens to upper case.
-Deprecated in favor of :class:`pyparsing_common.upcaseTokens`"""
-
-downcaseTokens = tokenMap(lambda t: _ustr(t).lower())
-"""(Deprecated) Helper parse action to convert tokens to lower case.
-Deprecated in favor of :class:`pyparsing_common.downcaseTokens`"""
-
-def _makeTags(tagStr, xml,
- suppress_LT=Suppress("<"),
- suppress_GT=Suppress(">")):
- """Internal helper to construct opening and closing tag expressions, given a tag name"""
- if isinstance(tagStr, basestring):
- resname = tagStr
- tagStr = Keyword(tagStr, caseless=not xml)
- else:
- resname = tagStr.name
-
- tagAttrName = Word(alphas, alphanums + "_-:")
- if xml:
- tagAttrValue = dblQuotedString.copy().setParseAction(removeQuotes)
- openTag = (suppress_LT
- + tagStr("tag")
- + Dict(ZeroOrMore(Group(tagAttrName + Suppress("=") + tagAttrValue)))
- + Optional("/", default=[False])("empty").setParseAction(lambda s, l, t: t[0] == '/')
- + suppress_GT)
- else:
- tagAttrValue = quotedString.copy().setParseAction(removeQuotes) | Word(printables, excludeChars=">")
- openTag = (suppress_LT
- + tagStr("tag")
- + Dict(ZeroOrMore(Group(tagAttrName.setParseAction(downcaseTokens)
- + Optional(Suppress("=") + tagAttrValue))))
- + Optional("/", default=[False])("empty").setParseAction(lambda s, l, t: t[0] == '/')
- + suppress_GT)
- closeTag = Combine(_L("</") + tagStr + ">", adjacent=False)
-
- openTag.setName("<%s>" % resname)
- # add start<tagname> results name in parse action now that ungrouped names are not reported at two levels
- openTag.addParseAction(lambda t: t.__setitem__("start" + "".join(resname.replace(":", " ").title().split()), t.copy()))
- closeTag = closeTag("end" + "".join(resname.replace(":", " ").title().split())).setName("</%s>" % resname)
- openTag.tag = resname
- closeTag.tag = resname
- openTag.tag_body = SkipTo(closeTag())
- return openTag, closeTag
-
-def makeHTMLTags(tagStr):
- """Helper to construct opening and closing tag expressions for HTML,
- given a tag name. Matches tags in either upper or lower case,
- attributes with namespaces and with quoted or unquoted values.
-
- Example::
-
- text = '<td>More info at the <a href="https://github.com/pyparsing/pyparsing/wiki">pyparsing</a> wiki page</td>'
- # makeHTMLTags returns pyparsing expressions for the opening and
- # closing tags as a 2-tuple
- a, a_end = makeHTMLTags("A")
- link_expr = a + SkipTo(a_end)("link_text") + a_end
-
- for link in link_expr.searchString(text):
- # attributes in the <A> tag (like "href" shown here) are
- # also accessible as named results
- print(link.link_text, '->', link.href)
-
- prints::
-
- pyparsing -> https://github.com/pyparsing/pyparsing/wiki
- """
- return _makeTags(tagStr, False)
-
-def makeXMLTags(tagStr):
- """Helper to construct opening and closing tag expressions for XML,
- given a tag name. Matches tags only in the given upper/lower case.
-
- Example: similar to :class:`makeHTMLTags`
- """
- return _makeTags(tagStr, True)
-
-def withAttribute(*args, **attrDict):
- """Helper to create a validating parse action to be used with start
- tags created with :class:`makeXMLTags` or
- :class:`makeHTMLTags`. Use ``withAttribute`` to qualify
- a starting tag with a required attribute value, to avoid false
- matches on common tags such as ``<TD>`` or ``<DIV>``.
-
- Call ``withAttribute`` with a series of attribute names and
- values. Specify the list of filter attributes names and values as:
-
- - keyword arguments, as in ``(align="right")``, or
- - as an explicit dict with ``**`` operator, when an attribute
- name is also a Python reserved word, as in ``**{"class":"Customer", "align":"right"}``
- - a list of name-value tuples, as in ``(("ns1:class", "Customer"), ("ns2:align", "right"))``
-
- For attribute names with a namespace prefix, you must use the second
- form. Attribute names are matched insensitive to upper/lower case.
-
- If just testing for ``class`` (with or without a namespace), use
- :class:`withClass`.
-
- To verify that the attribute exists, but without specifying a value,
- pass ``withAttribute.ANY_VALUE`` as the value.
-
- Example::
-
- html = '''
- <div>
- Some text
- <div type="grid">1 4 0 1 0</div>
- <div type="graph">1,3 2,3 1,1</div>
- <div>this has no type</div>
- </div>
-
- '''
- div,div_end = makeHTMLTags("div")
-
- # only match div tag having a type attribute with value "grid"
- div_grid = div().setParseAction(withAttribute(type="grid"))
- grid_expr = div_grid + SkipTo(div | div_end)("body")
- for grid_header in grid_expr.searchString(html):
- print(grid_header.body)
-
- # construct a match with any div tag having a type attribute, regardless of the value
- div_any_type = div().setParseAction(withAttribute(type=withAttribute.ANY_VALUE))
- div_expr = div_any_type + SkipTo(div | div_end)("body")
- for div_header in div_expr.searchString(html):
- print(div_header.body)
-
- prints::
-
- 1 4 0 1 0
-
- 1 4 0 1 0
- 1,3 2,3 1,1
- """
- if args:
- attrs = args[:]
- else:
- attrs = attrDict.items()
- attrs = [(k, v) for k, v in attrs]
- def pa(s, l, tokens):
- for attrName, attrValue in attrs:
- if attrName not in tokens:
- raise ParseException(s, l, "no matching attribute " + attrName)
- if attrValue != withAttribute.ANY_VALUE and tokens[attrName] != attrValue:
- raise ParseException(s, l, "attribute '%s' has value '%s', must be '%s'" %
- (attrName, tokens[attrName], attrValue))
- return pa
-withAttribute.ANY_VALUE = object()
-
-def withClass(classname, namespace=''):
- """Simplified version of :class:`withAttribute` when
- matching on a div class - made difficult because ``class`` is
- a reserved word in Python.
-
- Example::
-
- html = '''
- <div>
- Some text
- <div class="grid">1 4 0 1 0</div>
- <div class="graph">1,3 2,3 1,1</div>
- <div>this <div> has no class</div>
- </div>
-
- '''
- div,div_end = makeHTMLTags("div")
- div_grid = div().setParseAction(withClass("grid"))
-
- grid_expr = div_grid + SkipTo(div | div_end)("body")
- for grid_header in grid_expr.searchString(html):
- print(grid_header.body)
-
- div_any_type = div().setParseAction(withClass(withAttribute.ANY_VALUE))
- div_expr = div_any_type + SkipTo(div | div_end)("body")
- for div_header in div_expr.searchString(html):
- print(div_header.body)
-
- prints::
-
- 1 4 0 1 0
-
- 1 4 0 1 0
- 1,3 2,3 1,1
- """
- classattr = "%s:class" % namespace if namespace else "class"
- return withAttribute(**{classattr: classname})
-
-opAssoc = SimpleNamespace()
-opAssoc.LEFT = object()
-opAssoc.RIGHT = object()
-
-def infixNotation(baseExpr, opList, lpar=Suppress('('), rpar=Suppress(')')):
- """Helper method for constructing grammars of expressions made up of
- operators working in a precedence hierarchy. Operators may be unary
- or binary, left- or right-associative. Parse actions can also be
- attached to operator expressions. The generated parser will also
- recognize the use of parentheses to override operator precedences
- (see example below).
-
- Note: if you define a deep operator list, you may see performance
- issues when using infixNotation. See
- :class:`ParserElement.enablePackrat` for a mechanism to potentially
- improve your parser performance.
-
- Parameters:
- - baseExpr - expression representing the most basic element for the
- nested
- - opList - list of tuples, one for each operator precedence level
- in the expression grammar; each tuple is of the form ``(opExpr,
- numTerms, rightLeftAssoc, parseAction)``, where:
-
- - opExpr is the pyparsing expression for the operator; may also
- be a string, which will be converted to a Literal; if numTerms
- is 3, opExpr is a tuple of two expressions, for the two
- operators separating the 3 terms
- - numTerms is the number of terms for this operator (must be 1,
- 2, or 3)
- - rightLeftAssoc is the indicator whether the operator is right
- or left associative, using the pyparsing-defined constants
- ``opAssoc.RIGHT`` and ``opAssoc.LEFT``.
- - parseAction is the parse action to be associated with
- expressions matching this operator expression (the parse action
- tuple member may be omitted); if the parse action is passed
- a tuple or list of functions, this is equivalent to calling
- ``setParseAction(*fn)``
- (:class:`ParserElement.setParseAction`)
- - lpar - expression for matching left-parentheses
- (default= ``Suppress('(')``)
- - rpar - expression for matching right-parentheses
- (default= ``Suppress(')')``)
-
- Example::
-
- # simple example of four-function arithmetic with ints and
- # variable names
- integer = pyparsing_common.signed_integer
- varname = pyparsing_common.identifier
-
- arith_expr = infixNotation(integer | varname,
- [
- ('-', 1, opAssoc.RIGHT),
- (oneOf('* /'), 2, opAssoc.LEFT),
- (oneOf('+ -'), 2, opAssoc.LEFT),
- ])
-
- arith_expr.runTests('''
- 5+3*6
- (5+3)*6
- -2--11
- ''', fullDump=False)
-
- prints::
-
- 5+3*6
- [[5, '+', [3, '*', 6]]]
-
- (5+3)*6
- [[[5, '+', 3], '*', 6]]
-
- -2--11
- [[['-', 2], '-', ['-', 11]]]
- """
- # captive version of FollowedBy that does not do parse actions or capture results names
- class _FB(FollowedBy):
- def parseImpl(self, instring, loc, doActions=True):
- self.expr.tryParse(instring, loc)
- return loc, []
-
- ret = Forward()
- lastExpr = baseExpr | (lpar + ret + rpar)
- for i, operDef in enumerate(opList):
- opExpr, arity, rightLeftAssoc, pa = (operDef + (None, ))[:4]
- termName = "%s term" % opExpr if arity < 3 else "%s%s term" % opExpr
- if arity == 3:
- if opExpr is None or len(opExpr) != 2:
- raise ValueError(
- "if numterms=3, opExpr must be a tuple or list of two expressions")
- opExpr1, opExpr2 = opExpr
- thisExpr = Forward().setName(termName)
- if rightLeftAssoc == opAssoc.LEFT:
- if arity == 1:
- matchExpr = _FB(lastExpr + opExpr) + Group(lastExpr + OneOrMore(opExpr))
- elif arity == 2:
- if opExpr is not None:
- matchExpr = _FB(lastExpr + opExpr + lastExpr) + Group(lastExpr + OneOrMore(opExpr + lastExpr))
- else:
- matchExpr = _FB(lastExpr + lastExpr) + Group(lastExpr + OneOrMore(lastExpr))
- elif arity == 3:
- matchExpr = (_FB(lastExpr + opExpr1 + lastExpr + opExpr2 + lastExpr)
- + Group(lastExpr + OneOrMore(opExpr1 + lastExpr + opExpr2 + lastExpr)))
- else:
- raise ValueError("operator must be unary (1), binary (2), or ternary (3)")
- elif rightLeftAssoc == opAssoc.RIGHT:
- if arity == 1:
- # try to avoid LR with this extra test
- if not isinstance(opExpr, Optional):
- opExpr = Optional(opExpr)
- matchExpr = _FB(opExpr.expr + thisExpr) + Group(opExpr + thisExpr)
- elif arity == 2:
- if opExpr is not None:
- matchExpr = _FB(lastExpr + opExpr + thisExpr) + Group(lastExpr + OneOrMore(opExpr + thisExpr))
- else:
- matchExpr = _FB(lastExpr + thisExpr) + Group(lastExpr + OneOrMore(thisExpr))
- elif arity == 3:
- matchExpr = (_FB(lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr)
- + Group(lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr))
- else:
- raise ValueError("operator must be unary (1), binary (2), or ternary (3)")
- else:
- raise ValueError("operator must indicate right or left associativity")
- if pa:
- if isinstance(pa, (tuple, list)):
- matchExpr.setParseAction(*pa)
- else:
- matchExpr.setParseAction(pa)
- thisExpr <<= (matchExpr.setName(termName) | lastExpr)
- lastExpr = thisExpr
- ret <<= lastExpr
- return ret
-
-operatorPrecedence = infixNotation
-"""(Deprecated) Former name of :class:`infixNotation`, will be
-dropped in a future release."""
-
-dblQuotedString = Combine(Regex(r'"(?:[^"\n\r\\]|(?:"")|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*') + '"').setName("string enclosed in double quotes")
-sglQuotedString = Combine(Regex(r"'(?:[^'\n\r\\]|(?:'')|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*") + "'").setName("string enclosed in single quotes")
-quotedString = Combine(Regex(r'"(?:[^"\n\r\\]|(?:"")|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*') + '"'
- | Regex(r"'(?:[^'\n\r\\]|(?:'')|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*") + "'").setName("quotedString using single or double quotes")
-unicodeString = Combine(_L('u') + quotedString.copy()).setName("unicode string literal")
-
-def nestedExpr(opener="(", closer=")", content=None, ignoreExpr=quotedString.copy()):
- """Helper method for defining nested lists enclosed in opening and
- closing delimiters ("(" and ")" are the default).
-
- Parameters:
- - opener - opening character for a nested list
- (default= ``"("``); can also be a pyparsing expression
- - closer - closing character for a nested list
- (default= ``")"``); can also be a pyparsing expression
- - content - expression for items within the nested lists
- (default= ``None``)
- - ignoreExpr - expression for ignoring opening and closing
- delimiters (default= :class:`quotedString`)
-
- If an expression is not provided for the content argument, the
- nested expression will capture all whitespace-delimited content
- between delimiters as a list of separate values.
-
- Use the ``ignoreExpr`` argument to define expressions that may
- contain opening or closing characters that should not be treated as
- opening or closing characters for nesting, such as quotedString or
- a comment expression. Specify multiple expressions using an
- :class:`Or` or :class:`MatchFirst`. The default is
- :class:`quotedString`, but if no expressions are to be ignored, then
- pass ``None`` for this argument.
-
- Example::
-
- data_type = oneOf("void int short long char float double")
- decl_data_type = Combine(data_type + Optional(Word('*')))
- ident = Word(alphas+'_', alphanums+'_')
- number = pyparsing_common.number
- arg = Group(decl_data_type + ident)
- LPAR, RPAR = map(Suppress, "()")
-
- code_body = nestedExpr('{', '}', ignoreExpr=(quotedString | cStyleComment))
-
- c_function = (decl_data_type("type")
- + ident("name")
- + LPAR + Optional(delimitedList(arg), [])("args") + RPAR
- + code_body("body"))
- c_function.ignore(cStyleComment)
-
- source_code = '''
- int is_odd(int x) {
- return (x%2);
- }
-
- int dec_to_hex(char hchar) {
- if (hchar >= '0' && hchar <= '9') {
- return (ord(hchar)-ord('0'));
- } else {
- return (10+ord(hchar)-ord('A'));
- }
- }
- '''
- for func in c_function.searchString(source_code):
- print("%(name)s (%(type)s) args: %(args)s" % func)
-
-
- prints::
-
- is_odd (int) args: [['int', 'x']]
- dec_to_hex (int) args: [['char', 'hchar']]
- """
- if opener == closer:
- raise ValueError("opening and closing strings cannot be the same")
- if content is None:
- if isinstance(opener, basestring) and isinstance(closer, basestring):
- if len(opener) == 1 and len(closer) == 1:
- if ignoreExpr is not None:
- content = (Combine(OneOrMore(~ignoreExpr
- + CharsNotIn(opener
- + closer
- + ParserElement.DEFAULT_WHITE_CHARS, exact=1)
- )
- ).setParseAction(lambda t: t[0].strip()))
- else:
- content = (empty.copy() + CharsNotIn(opener
- + closer
- + ParserElement.DEFAULT_WHITE_CHARS
- ).setParseAction(lambda t: t[0].strip()))
- else:
- if ignoreExpr is not None:
- content = (Combine(OneOrMore(~ignoreExpr
- + ~Literal(opener)
- + ~Literal(closer)
- + CharsNotIn(ParserElement.DEFAULT_WHITE_CHARS, exact=1))
- ).setParseAction(lambda t: t[0].strip()))
- else:
- content = (Combine(OneOrMore(~Literal(opener)
- + ~Literal(closer)
- + CharsNotIn(ParserElement.DEFAULT_WHITE_CHARS, exact=1))
- ).setParseAction(lambda t: t[0].strip()))
- else:
- raise ValueError("opening and closing arguments must be strings if no content expression is given")
- ret = Forward()
- if ignoreExpr is not None:
- ret <<= Group(Suppress(opener) + ZeroOrMore(ignoreExpr | ret | content) + Suppress(closer))
- else:
- ret <<= Group(Suppress(opener) + ZeroOrMore(ret | content) + Suppress(closer))
- ret.setName('nested %s%s expression' % (opener, closer))
- return ret
-
-def indentedBlock(blockStatementExpr, indentStack, indent=True):
- """Helper method for defining space-delimited indentation blocks,
- such as those used to define block statements in Python source code.
-
- Parameters:
-
- - blockStatementExpr - expression defining syntax of statement that
- is repeated within the indented block
- - indentStack - list created by caller to manage indentation stack
- (multiple statementWithIndentedBlock expressions within a single
- grammar should share a common indentStack)
- - indent - boolean indicating whether block must be indented beyond
- the current level; set to False for block of left-most
- statements (default= ``True``)
-
- A valid block must contain at least one ``blockStatement``.
-
- Example::
-
- data = '''
- def A(z):
- A1
- B = 100
- G = A2
- A2
- A3
- B
- def BB(a,b,c):
- BB1
- def BBA():
- bba1
- bba2
- bba3
- C
- D
- def spam(x,y):
- def eggs(z):
- pass
- '''
-
-
- indentStack = [1]
- stmt = Forward()
-
- identifier = Word(alphas, alphanums)
- funcDecl = ("def" + identifier + Group("(" + Optional(delimitedList(identifier)) + ")") + ":")
- func_body = indentedBlock(stmt, indentStack)
- funcDef = Group(funcDecl + func_body)
-
- rvalue = Forward()
- funcCall = Group(identifier + "(" + Optional(delimitedList(rvalue)) + ")")
- rvalue << (funcCall | identifier | Word(nums))
- assignment = Group(identifier + "=" + rvalue)
- stmt << (funcDef | assignment | identifier)
-
- module_body = OneOrMore(stmt)
-
- parseTree = module_body.parseString(data)
- parseTree.pprint()
-
- prints::
-
- [['def',
- 'A',
- ['(', 'z', ')'],
- ':',
- [['A1'], [['B', '=', '100']], [['G', '=', 'A2']], ['A2'], ['A3']]],
- 'B',
- ['def',
- 'BB',
- ['(', 'a', 'b', 'c', ')'],
- ':',
- [['BB1'], [['def', 'BBA', ['(', ')'], ':', [['bba1'], ['bba2'], ['bba3']]]]]],
- 'C',
- 'D',
- ['def',
- 'spam',
- ['(', 'x', 'y', ')'],
- ':',
- [[['def', 'eggs', ['(', 'z', ')'], ':', [['pass']]]]]]]
- """
- backup_stack = indentStack[:]
-
- def reset_stack():
- indentStack[:] = backup_stack
-
- def checkPeerIndent(s, l, t):
- if l >= len(s): return
- curCol = col(l, s)
- if curCol != indentStack[-1]:
- if curCol > indentStack[-1]:
- raise ParseException(s, l, "illegal nesting")
- raise ParseException(s, l, "not a peer entry")
-
- def checkSubIndent(s, l, t):
- curCol = col(l, s)
- if curCol > indentStack[-1]:
- indentStack.append(curCol)
- else:
- raise ParseException(s, l, "not a subentry")
-
- def checkUnindent(s, l, t):
- if l >= len(s): return
- curCol = col(l, s)
- if not(indentStack and curCol in indentStack):
- raise ParseException(s, l, "not an unindent")
- if curCol < indentStack[-1]:
- indentStack.pop()
-
- NL = OneOrMore(LineEnd().setWhitespaceChars("\t ").suppress(), stopOn=StringEnd())
- INDENT = (Empty() + Empty().setParseAction(checkSubIndent)).setName('INDENT')
- PEER = Empty().setParseAction(checkPeerIndent).setName('')
- UNDENT = Empty().setParseAction(checkUnindent).setName('UNINDENT')
- if indent:
- smExpr = Group(Optional(NL)
- + INDENT
- + OneOrMore(PEER + Group(blockStatementExpr) + Optional(NL), stopOn=StringEnd())
- + UNDENT)
- else:
- smExpr = Group(Optional(NL)
- + OneOrMore(PEER + Group(blockStatementExpr) + Optional(NL), stopOn=StringEnd())
- + UNDENT)
- smExpr.setFailAction(lambda a, b, c, d: reset_stack())
- blockStatementExpr.ignore(_bslash + LineEnd())
- return smExpr.setName('indented block')
-
-alphas8bit = srange(r"[\0xc0-\0xd6\0xd8-\0xf6\0xf8-\0xff]")
-punc8bit = srange(r"[\0xa1-\0xbf\0xd7\0xf7]")
-
-anyOpenTag, anyCloseTag = makeHTMLTags(Word(alphas, alphanums + "_:").setName('any tag'))
-_htmlEntityMap = dict(zip("gt lt amp nbsp quot apos".split(), '><& "\''))
-commonHTMLEntity = Regex('&(?P<entity>' + '|'.join(_htmlEntityMap.keys()) +");").setName("common HTML entity")
-def replaceHTMLEntity(t):
- """Helper parser action to replace common HTML entities with their special characters"""
- return _htmlEntityMap.get(t.entity)
-
-# it's easy to get these comment structures wrong - they're very common, so may as well make them available
-cStyleComment = Combine(Regex(r"/\*(?:[^*]|\*(?!/))*") + '*/').setName("C style comment")
-"Comment of the form ``/* ... */``"
-
-htmlComment = Regex(r"<!--[\s\S]*?-->").setName("HTML comment")
-"Comment of the form ``<!-- ... -->``"
-
-restOfLine = Regex(r".*").leaveWhitespace().setName("rest of line")
-dblSlashComment = Regex(r"//(?:\\\n|[^\n])*").setName("// comment")
-"Comment of the form ``// ... (to end of line)``"
-
-cppStyleComment = Combine(Regex(r"/\*(?:[^*]|\*(?!/))*") + '*/' | dblSlashComment).setName("C++ style comment")
-"Comment of either form :class:`cStyleComment` or :class:`dblSlashComment`"
-
-javaStyleComment = cppStyleComment
-"Same as :class:`cppStyleComment`"
-
-pythonStyleComment = Regex(r"#.*").setName("Python style comment")
-"Comment of the form ``# ... (to end of line)``"
-
-_commasepitem = Combine(OneOrMore(Word(printables, excludeChars=',')
- + Optional(Word(" \t")
- + ~Literal(",") + ~LineEnd()))).streamline().setName("commaItem")
-commaSeparatedList = delimitedList(Optional(quotedString.copy() | _commasepitem, default="")).setName("commaSeparatedList")
-"""(Deprecated) Predefined expression of 1 or more printable words or
-quoted strings, separated by commas.
-
-This expression is deprecated in favor of :class:`pyparsing_common.comma_separated_list`.
-"""
-
-# some other useful expressions - using lower-case class name since we are really using this as a namespace
-class pyparsing_common:
- """Here are some common low-level expressions that may be useful in
- jump-starting parser development:
-
- - numeric forms (:class:`integers<integer>`, :class:`reals<real>`,
- :class:`scientific notation<sci_real>`)
- - common :class:`programming identifiers<identifier>`
- - network addresses (:class:`MAC<mac_address>`,
- :class:`IPv4<ipv4_address>`, :class:`IPv6<ipv6_address>`)
- - ISO8601 :class:`dates<iso8601_date>` and
- :class:`datetime<iso8601_datetime>`
- - :class:`UUID<uuid>`
- - :class:`comma-separated list<comma_separated_list>`
-
- Parse actions:
-
- - :class:`convertToInteger`
- - :class:`convertToFloat`
- - :class:`convertToDate`
- - :class:`convertToDatetime`
- - :class:`stripHTMLTags`
- - :class:`upcaseTokens`
- - :class:`downcaseTokens`
-
- Example::
-
- pyparsing_common.number.runTests('''
- # any int or real number, returned as the appropriate type
- 100
- -100
- +100
- 3.14159
- 6.02e23
- 1e-12
- ''')
-
- pyparsing_common.fnumber.runTests('''
- # any int or real number, returned as float
- 100
- -100
- +100
- 3.14159
- 6.02e23
- 1e-12
- ''')
-
- pyparsing_common.hex_integer.runTests('''
- # hex numbers
- 100
- FF
- ''')
-
- pyparsing_common.fraction.runTests('''
- # fractions
- 1/2
- -3/4
- ''')
-
- pyparsing_common.mixed_integer.runTests('''
- # mixed fractions
- 1
- 1/2
- -3/4
- 1-3/4
- ''')
-
- import uuid
- pyparsing_common.uuid.setParseAction(tokenMap(uuid.UUID))
- pyparsing_common.uuid.runTests('''
- # uuid
- 12345678-1234-5678-1234-567812345678
- ''')
-
- prints::
-
- # any int or real number, returned as the appropriate type
- 100
- [100]
-
- -100
- [-100]
-
- +100
- [100]
-
- 3.14159
- [3.14159]
-
- 6.02e23
- [6.02e+23]
-
- 1e-12
- [1e-12]
-
- # any int or real number, returned as float
- 100
- [100.0]
-
- -100
- [-100.0]
-
- +100
- [100.0]
-
- 3.14159
- [3.14159]
-
- 6.02e23
- [6.02e+23]
-
- 1e-12
- [1e-12]
-
- # hex numbers
- 100
- [256]
-
- FF
- [255]
-
- # fractions
- 1/2
- [0.5]
-
- -3/4
- [-0.75]
-
- # mixed fractions
- 1
- [1]
-
- 1/2
- [0.5]
-
- -3/4
- [-0.75]
-
- 1-3/4
- [1.75]
-
- # uuid
- 12345678-1234-5678-1234-567812345678
- [UUID('12345678-1234-5678-1234-567812345678')]
- """
-
- convertToInteger = tokenMap(int)
- """
- Parse action for converting parsed integers to Python int
- """
-
- convertToFloat = tokenMap(float)
- """
- Parse action for converting parsed numbers to Python float
- """
-
- integer = Word(nums).setName("integer").setParseAction(convertToInteger)
- """expression that parses an unsigned integer, returns an int"""
-
- hex_integer = Word(hexnums).setName("hex integer").setParseAction(tokenMap(int, 16))
- """expression that parses a hexadecimal integer, returns an int"""
-
- signed_integer = Regex(r'[+-]?\d+').setName("signed integer").setParseAction(convertToInteger)
- """expression that parses an integer with optional leading sign, returns an int"""
-
- fraction = (signed_integer().setParseAction(convertToFloat) + '/' + signed_integer().setParseAction(convertToFloat)).setName("fraction")
- """fractional expression of an integer divided by an integer, returns a float"""
- fraction.addParseAction(lambda t: t[0]/t[-1])
-
- mixed_integer = (fraction | signed_integer + Optional(Optional('-').suppress() + fraction)).setName("fraction or mixed integer-fraction")
- """mixed integer of the form 'integer - fraction', with optional leading integer, returns float"""
- mixed_integer.addParseAction(sum)
-
- real = Regex(r'[+-]?(?:\d+\.\d*|\.\d+)').setName("real number").setParseAction(convertToFloat)
- """expression that parses a floating point number and returns a float"""
-
- sci_real = Regex(r'[+-]?(?:\d+(?:[eE][+-]?\d+)|(?:\d+\.\d*|\.\d+)(?:[eE][+-]?\d+)?)').setName("real number with scientific notation").setParseAction(convertToFloat)
- """expression that parses a floating point number with optional
- scientific notation and returns a float"""
-
- # streamlining this expression makes the docs nicer-looking
- number = (sci_real | real | signed_integer).streamline()
- """any numeric expression, returns the corresponding Python type"""
-
- fnumber = Regex(r'[+-]?\d+\.?\d*([eE][+-]?\d+)?').setName("fnumber").setParseAction(convertToFloat)
- """any int or real number, returned as float"""
-
- identifier = Word(alphas + '_', alphanums + '_').setName("identifier")
- """typical code identifier (leading alpha or '_', followed by 0 or more alphas, nums, or '_')"""
-
- ipv4_address = Regex(r'(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})(\.(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})){3}').setName("IPv4 address")
- "IPv4 address (``0.0.0.0 - 255.255.255.255``)"
-
- _ipv6_part = Regex(r'[0-9a-fA-F]{1,4}').setName("hex_integer")
- _full_ipv6_address = (_ipv6_part + (':' + _ipv6_part) * 7).setName("full IPv6 address")
- _short_ipv6_address = (Optional(_ipv6_part + (':' + _ipv6_part) * (0, 6))
- + "::"
- + Optional(_ipv6_part + (':' + _ipv6_part) * (0, 6))
- ).setName("short IPv6 address")
- _short_ipv6_address.addCondition(lambda t: sum(1 for tt in t if pyparsing_common._ipv6_part.matches(tt)) < 8)
- _mixed_ipv6_address = ("::ffff:" + ipv4_address).setName("mixed IPv6 address")
- ipv6_address = Combine((_full_ipv6_address | _mixed_ipv6_address | _short_ipv6_address).setName("IPv6 address")).setName("IPv6 address")
- "IPv6 address (long, short, or mixed form)"
-
- mac_address = Regex(r'[0-9a-fA-F]{2}([:.-])[0-9a-fA-F]{2}(?:\1[0-9a-fA-F]{2}){4}').setName("MAC address")
- "MAC address xx:xx:xx:xx:xx (may also have '-' or '.' delimiters)"
-
- @staticmethod
- def convertToDate(fmt="%Y-%m-%d"):
- """
- Helper to create a parse action for converting parsed date string to Python datetime.date
-
- Params -
- - fmt - format to be passed to datetime.strptime (default= ``"%Y-%m-%d"``)
-
- Example::
-
- date_expr = pyparsing_common.iso8601_date.copy()
- date_expr.setParseAction(pyparsing_common.convertToDate())
- print(date_expr.parseString("1999-12-31"))
-
- prints::
-
- [datetime.date(1999, 12, 31)]
- """
- def cvt_fn(s, l, t):
- try:
- return datetime.strptime(t[0], fmt).date()
- except ValueError as ve:
- raise ParseException(s, l, str(ve))
- return cvt_fn
-
- @staticmethod
- def convertToDatetime(fmt="%Y-%m-%dT%H:%M:%S.%f"):
- """Helper to create a parse action for converting parsed
- datetime string to Python datetime.datetime
-
- Params -
- - fmt - format to be passed to datetime.strptime (default= ``"%Y-%m-%dT%H:%M:%S.%f"``)
-
- Example::
-
- dt_expr = pyparsing_common.iso8601_datetime.copy()
- dt_expr.setParseAction(pyparsing_common.convertToDatetime())
- print(dt_expr.parseString("1999-12-31T23:59:59.999"))
-
- prints::
-
- [datetime.datetime(1999, 12, 31, 23, 59, 59, 999000)]
- """
- def cvt_fn(s, l, t):
- try:
- return datetime.strptime(t[0], fmt)
- except ValueError as ve:
- raise ParseException(s, l, str(ve))
- return cvt_fn
-
- iso8601_date = Regex(r'(?P<year>\d{4})(?:-(?P<month>\d\d)(?:-(?P<day>\d\d))?)?').setName("ISO8601 date")
- "ISO8601 date (``yyyy-mm-dd``)"
-
- iso8601_datetime = Regex(r'(?P<year>\d{4})-(?P<month>\d\d)-(?P<day>\d\d)[T ](?P<hour>\d\d):(?P<minute>\d\d)(:(?P<second>\d\d(\.\d*)?)?)?(?P<tz>Z|[+-]\d\d:?\d\d)?').setName("ISO8601 datetime")
- "ISO8601 datetime (``yyyy-mm-ddThh:mm:ss.s(Z|+-00:00)``) - trailing seconds, milliseconds, and timezone optional; accepts separating ``'T'`` or ``' '``"
-
- uuid = Regex(r'[0-9a-fA-F]{8}(-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}').setName("UUID")
- "UUID (``xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx``)"
-
- _html_stripper = anyOpenTag.suppress() | anyCloseTag.suppress()
- @staticmethod
- def stripHTMLTags(s, l, tokens):
- """Parse action to remove HTML tags from web page HTML source
-
- Example::
-
- # strip HTML links from normal text
- text = '<td>More info at the <a href="https://github.com/pyparsing/pyparsing/wiki">pyparsing</a> wiki page</td>'
- td, td_end = makeHTMLTags("TD")
- table_text = td + SkipTo(td_end).setParseAction(pyparsing_common.stripHTMLTags)("body") + td_end
- print(table_text.parseString(text).body)
-
- Prints::
-
- More info at the pyparsing wiki page
- """
- return pyparsing_common._html_stripper.transformString(tokens[0])
-
- _commasepitem = Combine(OneOrMore(~Literal(",")
- + ~LineEnd()
- + Word(printables, excludeChars=',')
- + Optional(White(" \t")))).streamline().setName("commaItem")
- comma_separated_list = delimitedList(Optional(quotedString.copy()
- | _commasepitem, default='')
- ).setName("comma separated list")
- """Predefined expression of 1 or more printable words or quoted strings, separated by commas."""
-
- upcaseTokens = staticmethod(tokenMap(lambda t: _ustr(t).upper()))
- """Parse action to convert tokens to upper case."""
-
- downcaseTokens = staticmethod(tokenMap(lambda t: _ustr(t).lower()))
- """Parse action to convert tokens to lower case."""
-
-
-class _lazyclassproperty(object):
- def __init__(self, fn):
- self.fn = fn
- self.__doc__ = fn.__doc__
- self.__name__ = fn.__name__
-
- def __get__(self, obj, cls):
- if cls is None:
- cls = type(obj)
- if not hasattr(cls, '_intern') or any(cls._intern is getattr(superclass, '_intern', [])
- for superclass in cls.__mro__[1:]):
- cls._intern = {}
- attrname = self.fn.__name__
- if attrname not in cls._intern:
- cls._intern[attrname] = self.fn(cls)
- return cls._intern[attrname]
-
-
-class unicode_set(object):
- """
- A set of Unicode characters, for language-specific strings for
- ``alphas``, ``nums``, ``alphanums``, and ``printables``.
- A unicode_set is defined by a list of ranges in the Unicode character
- set, in a class attribute ``_ranges``, such as::
-
- _ranges = [(0x0020, 0x007e), (0x00a0, 0x00ff),]
-
- A unicode set can also be defined using multiple inheritance of other unicode sets::
-
- class CJK(Chinese, Japanese, Korean):
- pass
- """
- _ranges = []
-
- @classmethod
- def _get_chars_for_ranges(cls):
- ret = []
- for cc in cls.__mro__:
- if cc is unicode_set:
- break
- for rr in cc._ranges:
- ret.extend(range(rr[0], rr[-1] + 1))
- return [unichr(c) for c in sorted(set(ret))]
-
- @_lazyclassproperty
- def printables(cls):
- "all non-whitespace characters in this range"
- return u''.join(filterfalse(unicode.isspace, cls._get_chars_for_ranges()))
-
- @_lazyclassproperty
- def alphas(cls):
- "all alphabetic characters in this range"
- return u''.join(filter(unicode.isalpha, cls._get_chars_for_ranges()))
-
- @_lazyclassproperty
- def nums(cls):
- "all numeric digit characters in this range"
- return u''.join(filter(unicode.isdigit, cls._get_chars_for_ranges()))
-
- @_lazyclassproperty
- def alphanums(cls):
- "all alphanumeric characters in this range"
- return cls.alphas + cls.nums
-
-
-class pyparsing_unicode(unicode_set):
- """
- A namespace class for defining common language unicode_sets.
- """
- _ranges = [(32, sys.maxunicode)]
-
- class Latin1(unicode_set):
- "Unicode set for Latin-1 Unicode Character Range"
- _ranges = [(0x0020, 0x007e), (0x00a0, 0x00ff),]
-
- class LatinA(unicode_set):
- "Unicode set for Latin-A Unicode Character Range"
- _ranges = [(0x0100, 0x017f),]
-
- class LatinB(unicode_set):
- "Unicode set for Latin-B Unicode Character Range"
- _ranges = [(0x0180, 0x024f),]
-
- class Greek(unicode_set):
- "Unicode set for Greek Unicode Character Ranges"
- _ranges = [
- (0x0370, 0x03ff), (0x1f00, 0x1f15), (0x1f18, 0x1f1d), (0x1f20, 0x1f45), (0x1f48, 0x1f4d),
- (0x1f50, 0x1f57), (0x1f59,), (0x1f5b,), (0x1f5d,), (0x1f5f, 0x1f7d), (0x1f80, 0x1fb4), (0x1fb6, 0x1fc4),
- (0x1fc6, 0x1fd3), (0x1fd6, 0x1fdb), (0x1fdd, 0x1fef), (0x1ff2, 0x1ff4), (0x1ff6, 0x1ffe),
- ]
-
- class Cyrillic(unicode_set):
- "Unicode set for Cyrillic Unicode Character Range"
- _ranges = [(0x0400, 0x04ff)]
-
- class Chinese(unicode_set):
- "Unicode set for Chinese Unicode Character Range"
- _ranges = [(0x4e00, 0x9fff), (0x3000, 0x303f),]
-
- class Japanese(unicode_set):
- "Unicode set for Japanese Unicode Character Range, combining Kanji, Hiragana, and Katakana ranges"
- _ranges = []
-
- class Kanji(unicode_set):
- "Unicode set for Kanji Unicode Character Range"
- _ranges = [(0x4E00, 0x9Fbf), (0x3000, 0x303f),]
-
- class Hiragana(unicode_set):
- "Unicode set for Hiragana Unicode Character Range"
- _ranges = [(0x3040, 0x309f),]
-
- class Katakana(unicode_set):
- "Unicode set for Katakana Unicode Character Range"
- _ranges = [(0x30a0, 0x30ff),]
-
- class Korean(unicode_set):
- "Unicode set for Korean Unicode Character Range"
- _ranges = [(0xac00, 0xd7af), (0x1100, 0x11ff), (0x3130, 0x318f), (0xa960, 0xa97f), (0xd7b0, 0xd7ff), (0x3000, 0x303f),]
-
- class CJK(Chinese, Japanese, Korean):
- "Unicode set for combined Chinese, Japanese, and Korean (CJK) Unicode Character Range"
- pass
-
- class Thai(unicode_set):
- "Unicode set for Thai Unicode Character Range"
- _ranges = [(0x0e01, 0x0e3a), (0x0e3f, 0x0e5b),]
-
- class Arabic(unicode_set):
- "Unicode set for Arabic Unicode Character Range"
- _ranges = [(0x0600, 0x061b), (0x061e, 0x06ff), (0x0700, 0x077f),]
-
- class Hebrew(unicode_set):
- "Unicode set for Hebrew Unicode Character Range"
- _ranges = [(0x0590, 0x05ff),]
-
- class Devanagari(unicode_set):
- "Unicode set for Devanagari Unicode Character Range"
- _ranges = [(0x0900, 0x097f), (0xa8e0, 0xa8ff)]
-
-pyparsing_unicode.Japanese._ranges = (pyparsing_unicode.Japanese.Kanji._ranges
- + pyparsing_unicode.Japanese.Hiragana._ranges
- + pyparsing_unicode.Japanese.Katakana._ranges)
-
-# define ranges in language character sets
-if PY_3:
- setattr(pyparsing_unicode, u"العربية", pyparsing_unicode.Arabic)
- setattr(pyparsing_unicode, u"中文", pyparsing_unicode.Chinese)
- setattr(pyparsing_unicode, u"кириллица", pyparsing_unicode.Cyrillic)
- setattr(pyparsing_unicode, u"Ελληνικά", pyparsing_unicode.Greek)
- setattr(pyparsing_unicode, u"עִברִית", pyparsing_unicode.Hebrew)
- setattr(pyparsing_unicode, u"日本語", pyparsing_unicode.Japanese)
- setattr(pyparsing_unicode.Japanese, u"漢字", pyparsing_unicode.Japanese.Kanji)
- setattr(pyparsing_unicode.Japanese, u"カタカナ", pyparsing_unicode.Japanese.Katakana)
- setattr(pyparsing_unicode.Japanese, u"ひらがな", pyparsing_unicode.Japanese.Hiragana)
- setattr(pyparsing_unicode, u"한국어", pyparsing_unicode.Korean)
- setattr(pyparsing_unicode, u"ไทย", pyparsing_unicode.Thai)
- setattr(pyparsing_unicode, u"देवनागरी", pyparsing_unicode.Devanagari)
-
-
-class pyparsing_test:
- """
- namespace class for classes useful in writing unit tests
- """
-
- class reset_pyparsing_context:
- """
- Context manager to be used when writing unit tests that modify pyparsing config values:
- - packrat parsing
- - default whitespace characters.
- - default keyword characters
- - literal string auto-conversion class
- - __diag__ settings
-
- Example:
- with reset_pyparsing_context():
- # test that literals used to construct a grammar are automatically suppressed
- ParserElement.inlineLiteralsUsing(Suppress)
-
- term = Word(alphas) | Word(nums)
- group = Group('(' + term[...] + ')')
-
- # assert that the '()' characters are not included in the parsed tokens
- self.assertParseAndCheckLisst(group, "(abc 123 def)", ['abc', '123', 'def'])
-
- # after exiting context manager, literals are converted to Literal expressions again
- """
-
- def __init__(self):
- self._save_context = {}
-
- def save(self):
- self._save_context["default_whitespace"] = ParserElement.DEFAULT_WHITE_CHARS
- self._save_context["default_keyword_chars"] = Keyword.DEFAULT_KEYWORD_CHARS
- self._save_context[
- "literal_string_class"
- ] = ParserElement._literalStringClass
- self._save_context["packrat_enabled"] = ParserElement._packratEnabled
- self._save_context["packrat_parse"] = ParserElement._parse
- self._save_context["__diag__"] = {
- name: getattr(__diag__, name) for name in __diag__._all_names
- }
- self._save_context["__compat__"] = {
- "collect_all_And_tokens": __compat__.collect_all_And_tokens
- }
- return self
-
- def restore(self):
- # reset pyparsing global state
- if (
- ParserElement.DEFAULT_WHITE_CHARS
- != self._save_context["default_whitespace"]
- ):
- ParserElement.setDefaultWhitespaceChars(
- self._save_context["default_whitespace"]
- )
- Keyword.DEFAULT_KEYWORD_CHARS = self._save_context["default_keyword_chars"]
- ParserElement.inlineLiteralsUsing(
- self._save_context["literal_string_class"]
- )
- for name, value in self._save_context["__diag__"].items():
- setattr(__diag__, name, value)
- ParserElement._packratEnabled = self._save_context["packrat_enabled"]
- ParserElement._parse = self._save_context["packrat_parse"]
- __compat__.collect_all_And_tokens = self._save_context["__compat__"]
-
- def __enter__(self):
- return self.save()
-
- def __exit__(self, *args):
- return self.restore()
-
- class TestParseResultsAsserts:
- """
- A mixin class to add parse results assertion methods to normal unittest.TestCase classes.
- """
- def assertParseResultsEquals(
- self, result, expected_list=None, expected_dict=None, msg=None
- ):
- """
- Unit test assertion to compare a ParseResults object with an optional expected_list,
- and compare any defined results names with an optional expected_dict.
- """
- if expected_list is not None:
- self.assertEqual(expected_list, result.asList(), msg=msg)
- if expected_dict is not None:
- self.assertEqual(expected_dict, result.asDict(), msg=msg)
-
- def assertParseAndCheckList(
- self, expr, test_string, expected_list, msg=None, verbose=True
- ):
- """
- Convenience wrapper assert to test a parser element and input string, and assert that
- the resulting ParseResults.asList() is equal to the expected_list.
- """
- result = expr.parseString(test_string, parseAll=True)
- if verbose:
- print(result.dump())
- self.assertParseResultsEquals(result, expected_list=expected_list, msg=msg)
-
- def assertParseAndCheckDict(
- self, expr, test_string, expected_dict, msg=None, verbose=True
- ):
- """
- Convenience wrapper assert to test a parser element and input string, and assert that
- the resulting ParseResults.asDict() is equal to the expected_dict.
- """
- result = expr.parseString(test_string, parseAll=True)
- if verbose:
- print(result.dump())
- self.assertParseResultsEquals(result, expected_dict=expected_dict, msg=msg)
-
- def assertRunTestResults(
- self, run_tests_report, expected_parse_results=None, msg=None
- ):
- """
- Unit test assertion to evaluate output of ParserElement.runTests(). If a list of
- list-dict tuples is given as the expected_parse_results argument, then these are zipped
- with the report tuples returned by runTests and evaluated using assertParseResultsEquals.
- Finally, asserts that the overall runTests() success value is True.
-
- :param run_tests_report: tuple(bool, [tuple(str, ParseResults or Exception)]) returned from runTests
- :param expected_parse_results (optional): [tuple(str, list, dict, Exception)]
- """
- run_test_success, run_test_results = run_tests_report
-
- if expected_parse_results is not None:
- merged = [
- (rpt[0], rpt[1], expected)
- for rpt, expected in zip(run_test_results, expected_parse_results)
- ]
- for test_string, result, expected in merged:
- # expected should be a tuple containing a list and/or a dict or an exception,
- # and optional failure message string
- # an empty tuple will skip any result validation
- fail_msg = next(
- (exp for exp in expected if isinstance(exp, str)), None
- )
- expected_exception = next(
- (
- exp
- for exp in expected
- if isinstance(exp, type) and issubclass(exp, Exception)
- ),
- None,
- )
- if expected_exception is not None:
- with self.assertRaises(
- expected_exception=expected_exception, msg=fail_msg or msg
- ):
- if isinstance(result, Exception):
- raise result
- else:
- expected_list = next(
- (exp for exp in expected if isinstance(exp, list)), None
- )
- expected_dict = next(
- (exp for exp in expected if isinstance(exp, dict)), None
- )
- if (expected_list, expected_dict) != (None, None):
- self.assertParseResultsEquals(
- result,
- expected_list=expected_list,
- expected_dict=expected_dict,
- msg=fail_msg or msg,
- )
- else:
- # warning here maybe?
- print("no validation for {!r}".format(test_string))
-
- # do this last, in case some specific test results can be reported instead
- self.assertTrue(
- run_test_success, msg=msg if msg is not None else "failed runTests"
- )
-
- @contextmanager
- def assertRaisesParseException(self, exc_type=ParseException, msg=None):
- with self.assertRaises(exc_type, msg=msg):
- yield
-
-
-if __name__ == "__main__":
-
- selectToken = CaselessLiteral("select")
- fromToken = CaselessLiteral("from")
-
- ident = Word(alphas, alphanums + "_$")
-
- columnName = delimitedList(ident, ".", combine=True).setParseAction(upcaseTokens)
- columnNameList = Group(delimitedList(columnName)).setName("columns")
- columnSpec = ('*' | columnNameList)
-
- tableName = delimitedList(ident, ".", combine=True).setParseAction(upcaseTokens)
- tableNameList = Group(delimitedList(tableName)).setName("tables")
-
- simpleSQL = selectToken("command") + columnSpec("columns") + fromToken + tableNameList("tables")
-
- # demo runTests method, including embedded comments in test string
- simpleSQL.runTests("""
- # '*' as column list and dotted table name
- select * from SYS.XYZZY
-
- # caseless match on "SELECT", and casts back to "select"
- SELECT * from XYZZY, ABC
-
- # list of column names, and mixed case SELECT keyword
- Select AA,BB,CC from Sys.dual
-
- # multiple tables
- Select A, B, C from Sys.dual, Table2
-
- # invalid SELECT keyword - should fail
- Xelect A, B, C from Sys.dual
-
- # incomplete command - should fail
- Select
-
- # invalid column name - should fail
- Select ^^^ frox Sys.dual
-
- """)
-
- pyparsing_common.number.runTests("""
- 100
- -100
- +100
- 3.14159
- 6.02e23
- 1e-12
- """)
-
- # any int or real number, returned as float
- pyparsing_common.fnumber.runTests("""
- 100
- -100
- +100
- 3.14159
- 6.02e23
- 1e-12
- """)
-
- pyparsing_common.hex_integer.runTests("""
- 100
- FF
- """)
-
- import uuid
- pyparsing_common.uuid.setParseAction(tokenMap(uuid.UUID))
- pyparsing_common.uuid.runTests("""
- 12345678-1234-5678-1234-567812345678
- """)
--- /dev/null
+# module pyparsing.py
+#
+# Copyright (c) 2003-2021 Paul T. McGuire
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+
+__doc__ = """
+pyparsing module - Classes and methods to define and execute parsing grammars
+=============================================================================
+
+The pyparsing module is an alternative approach to creating and
+executing simple grammars, vs. the traditional lex/yacc approach, or the
+use of regular expressions. With pyparsing, you don't need to learn
+a new syntax for defining grammars or matching expressions - the parsing
+module provides a library of classes that you use to construct the
+grammar directly in Python.
+
+Here is a program to parse "Hello, World!" (or any greeting of the form
+``"<salutation>, <addressee>!"``), built up using :class:`Word`,
+:class:`Literal`, and :class:`And` elements
+(the :meth:`'+'<ParserElement.__add__>` operators create :class:`And` expressions,
+and the strings are auto-converted to :class:`Literal` expressions)::
+
+ from pyparsing import Word, alphas
+
+ # define grammar of a greeting
+ greet = Word(alphas) + "," + Word(alphas) + "!"
+
+ hello = "Hello, World!"
+ print(hello, "->", greet.parse_string(hello))
+
+The program outputs the following::
+
+ Hello, World! -> ['Hello', ',', 'World', '!']
+
+The Python representation of the grammar is quite readable, owing to the
+self-explanatory class names, and the use of :class:`'+'<And>`,
+:class:`'|'<MatchFirst>`, :class:`'^'<Or>` and :class:`'&'<Each>` operators.
+
+The :class:`ParseResults` object returned from
+:class:`ParserElement.parseString` can be
+accessed as a nested list, a dictionary, or an object with named
+attributes.
+
+The pyparsing module handles some of the problems that are typically
+vexing when writing text parsers:
+
+ - extra or missing whitespace (the above program will also handle
+ "Hello,World!", "Hello , World !", etc.)
+ - quoted strings
+ - embedded comments
+
+
+Getting Started -
+-----------------
+Visit the classes :class:`ParserElement` and :class:`ParseResults` to
+see the base classes that most other pyparsing
+classes inherit from. Use the docstrings for examples of how to:
+
+ - construct literal match expressions from :class:`Literal` and
+ :class:`CaselessLiteral` classes
+ - construct character word-group expressions using the :class:`Word`
+ class
+ - see how to create repetitive expressions using :class:`ZeroOrMore`
+ and :class:`OneOrMore` classes
+ - use :class:`'+'<And>`, :class:`'|'<MatchFirst>`, :class:`'^'<Or>`,
+ and :class:`'&'<Each>` operators to combine simple expressions into
+ more complex ones
+ - associate names with your parsed results using
+ :class:`ParserElement.setResultsName`
+ - access the parsed data, which is returned as a :class:`ParseResults`
+ object
+ - find some helpful expression short-cuts like :class:`delimitedList`
+ and :class:`oneOf`
+ - find more useful common expressions in the :class:`pyparsing_common`
+ namespace class
+"""
+from collections import namedtuple
+
+version_info = namedtuple("version_info", "major minor micro release_level serial")
+__version_info__ = version_info(3, 0, 0, "final", 0)
+__version__ = "{}.{}.{}".format(*__version_info__[:3]) + (
+ "{}{}{}".format(
+ "r" if __version_info__.release_level[0] == "c" else "",
+ __version_info__.release_level[0],
+ __version_info__.serial,
+ ),
+ "",
+)[__version_info__.release_level == "final"]
+__version_time__ = "23 October 2021 17:05 UTC"
+__versionTime__ = __version_time__
+__author__ = "Paul McGuire <ptmcg.gm+pyparsing@gmail.com>"
+
+from .util import *
+from .exceptions import *
+from .actions import *
+from .core import __diag__, __compat__
+from .results import *
+from .core import *
+from .core import _builtin_exprs as core_builtin_exprs
+from .helpers import *
+from .helpers import _builtin_exprs as helper_builtin_exprs
+
+from .unicode import unicode_set, UnicodeRangeList, pyparsing_unicode as unicode
+from .testing import pyparsing_test as testing
+from .common import (
+ pyparsing_common as common,
+ _builtin_exprs as common_builtin_exprs,
+)
+
+# define backward compat synonyms
+if "pyparsing_unicode" not in globals():
+ pyparsing_unicode = unicode
+if "pyparsing_common" not in globals():
+ pyparsing_common = common
+if "pyparsing_test" not in globals():
+ pyparsing_test = testing
+
+core_builtin_exprs += common_builtin_exprs + helper_builtin_exprs
+
+
+__all__ = [
+ "__version__",
+ "__version_time__",
+ "__author__",
+ "__compat__",
+ "__diag__",
+ "And",
+ "AtLineStart",
+ "AtStringStart",
+ "CaselessKeyword",
+ "CaselessLiteral",
+ "CharsNotIn",
+ "Combine",
+ "Dict",
+ "Each",
+ "Empty",
+ "FollowedBy",
+ "Forward",
+ "GoToColumn",
+ "Group",
+ "IndentedBlock",
+ "Keyword",
+ "LineEnd",
+ "LineStart",
+ "Literal",
+ "Located",
+ "PrecededBy",
+ "MatchFirst",
+ "NoMatch",
+ "NotAny",
+ "OneOrMore",
+ "OnlyOnce",
+ "OpAssoc",
+ "Opt",
+ "Optional",
+ "Or",
+ "ParseBaseException",
+ "ParseElementEnhance",
+ "ParseException",
+ "ParseExpression",
+ "ParseFatalException",
+ "ParseResults",
+ "ParseSyntaxException",
+ "ParserElement",
+ "QuotedString",
+ "RecursiveGrammarException",
+ "Regex",
+ "SkipTo",
+ "StringEnd",
+ "StringStart",
+ "Suppress",
+ "Token",
+ "TokenConverter",
+ "White",
+ "Word",
+ "WordEnd",
+ "WordStart",
+ "ZeroOrMore",
+ "Char",
+ "alphanums",
+ "alphas",
+ "alphas8bit",
+ "any_close_tag",
+ "any_open_tag",
+ "c_style_comment",
+ "col",
+ "common_html_entity",
+ "counted_array",
+ "cpp_style_comment",
+ "dbl_quoted_string",
+ "dbl_slash_comment",
+ "delimited_list",
+ "dict_of",
+ "empty",
+ "hexnums",
+ "html_comment",
+ "identchars",
+ "identbodychars",
+ "java_style_comment",
+ "line",
+ "line_end",
+ "line_start",
+ "lineno",
+ "make_html_tags",
+ "make_xml_tags",
+ "match_only_at_col",
+ "match_previous_expr",
+ "match_previous_literal",
+ "nested_expr",
+ "null_debug_action",
+ "nums",
+ "one_of",
+ "printables",
+ "punc8bit",
+ "python_style_comment",
+ "quoted_string",
+ "remove_quotes",
+ "replace_with",
+ "replace_html_entity",
+ "rest_of_line",
+ "sgl_quoted_string",
+ "srange",
+ "string_end",
+ "string_start",
+ "trace_parse_action",
+ "unicode_string",
+ "with_attribute",
+ "indentedBlock",
+ "original_text_for",
+ "ungroup",
+ "infix_notation",
+ "locatedExpr",
+ "with_class",
+ "CloseMatch",
+ "token_map",
+ "pyparsing_common",
+ "pyparsing_unicode",
+ "unicode_set",
+ "condition_as_parse_action",
+ "pyparsing_test",
+ # pre-PEP8 compatibility names
+ "__versionTime__",
+ "anyCloseTag",
+ "anyOpenTag",
+ "cStyleComment",
+ "commonHTMLEntity",
+ "countedArray",
+ "cppStyleComment",
+ "dblQuotedString",
+ "dblSlashComment",
+ "delimitedList",
+ "dictOf",
+ "htmlComment",
+ "javaStyleComment",
+ "lineEnd",
+ "lineStart",
+ "makeHTMLTags",
+ "makeXMLTags",
+ "matchOnlyAtCol",
+ "matchPreviousExpr",
+ "matchPreviousLiteral",
+ "nestedExpr",
+ "nullDebugAction",
+ "oneOf",
+ "opAssoc",
+ "pythonStyleComment",
+ "quotedString",
+ "removeQuotes",
+ "replaceHTMLEntity",
+ "replaceWith",
+ "restOfLine",
+ "sglQuotedString",
+ "stringEnd",
+ "stringStart",
+ "traceParseAction",
+ "unicodeString",
+ "withAttribute",
+ "indentedBlock",
+ "originalTextFor",
+ "infixNotation",
+ "locatedExpr",
+ "withClass",
+ "tokenMap",
+ "conditionAsParseAction",
+ "autoname_elements",
+]
--- /dev/null
+# actions.py
+
+from .exceptions import ParseException
+from .util import col
+
+
+class OnlyOnce:
+ """
+ Wrapper for parse actions, to ensure they are only called once.
+ """
+
+ def __init__(self, method_call):
+ from .core import _trim_arity
+
+ self.callable = _trim_arity(method_call)
+ self.called = False
+
+ def __call__(self, s, l, t):
+ if not self.called:
+ results = self.callable(s, l, t)
+ self.called = True
+ return results
+ raise ParseException(s, l, "OnlyOnce obj called multiple times w/out reset")
+
+ def reset(self):
+ """
+ Allow the associated parse action to be called once more.
+ """
+
+ self.called = False
+
+
+def match_only_at_col(n):
+ """
+ Helper method for defining parse actions that require matching at
+ a specific column in the input text.
+ """
+
+ def verify_col(strg, locn, toks):
+ if col(locn, strg) != n:
+ raise ParseException(strg, locn, "matched token not at column {}".format(n))
+
+ return verify_col
+
+
+def replace_with(repl_str):
+ """
+ Helper method for common parse actions that simply return
+ a literal value. Especially useful when used with
+ :class:`transform_string<ParserElement.transform_string>` ().
+
+ Example::
+
+ num = Word(nums).set_parse_action(lambda toks: int(toks[0]))
+ na = one_of("N/A NA").set_parse_action(replace_with(math.nan))
+ term = na | num
+
+ OneOrMore(term).parse_string("324 234 N/A 234") # -> [324, 234, nan, 234]
+ """
+ return lambda s, l, t: [repl_str]
+
+
+def remove_quotes(s, l, t):
+ """
+ Helper parse action for removing quotation marks from parsed
+ quoted strings.
+
+ Example::
+
+ # by default, quotation marks are included in parsed results
+ quoted_string.parse_string("'Now is the Winter of our Discontent'") # -> ["'Now is the Winter of our Discontent'"]
+
+ # use remove_quotes to strip quotation marks from parsed results
+ quoted_string.set_parse_action(remove_quotes)
+ quoted_string.parse_string("'Now is the Winter of our Discontent'") # -> ["Now is the Winter of our Discontent"]
+ """
+ return t[0][1:-1]
+
+
+def with_attribute(*args, **attr_dict):
+ """
+ Helper to create a validating parse action to be used with start
+ tags created with :class:`make_xml_tags` or
+ :class:`make_html_tags`. Use ``with_attribute`` to qualify
+ a starting tag with a required attribute value, to avoid false
+ matches on common tags such as ``<TD>`` or ``<DIV>``.
+
+ Call ``with_attribute`` with a series of attribute names and
+ values. Specify the list of filter attributes names and values as:
+
+ - keyword arguments, as in ``(align="right")``, or
+ - as an explicit dict with ``**`` operator, when an attribute
+ name is also a Python reserved word, as in ``**{"class":"Customer", "align":"right"}``
+ - a list of name-value tuples, as in ``(("ns1:class", "Customer"), ("ns2:align", "right"))``
+
+ For attribute names with a namespace prefix, you must use the second
+ form. Attribute names are matched insensitive to upper/lower case.
+
+ If just testing for ``class`` (with or without a namespace), use
+ :class:`with_class`.
+
+ To verify that the attribute exists, but without specifying a value,
+ pass ``with_attribute.ANY_VALUE`` as the value.
+
+ Example::
+
+ html = '''
+ <div>
+ Some text
+ <div type="grid">1 4 0 1 0</div>
+ <div type="graph">1,3 2,3 1,1</div>
+ <div>this has no type</div>
+ </div>
+
+ '''
+ div,div_end = make_html_tags("div")
+
+ # only match div tag having a type attribute with value "grid"
+ div_grid = div().set_parse_action(with_attribute(type="grid"))
+ grid_expr = div_grid + SkipTo(div | div_end)("body")
+ for grid_header in grid_expr.search_string(html):
+ print(grid_header.body)
+
+ # construct a match with any div tag having a type attribute, regardless of the value
+ div_any_type = div().set_parse_action(with_attribute(type=with_attribute.ANY_VALUE))
+ div_expr = div_any_type + SkipTo(div | div_end)("body")
+ for div_header in div_expr.search_string(html):
+ print(div_header.body)
+
+ prints::
+
+ 1 4 0 1 0
+
+ 1 4 0 1 0
+ 1,3 2,3 1,1
+ """
+ if args:
+ attrs = args[:]
+ else:
+ attrs = attr_dict.items()
+ attrs = [(k, v) for k, v in attrs]
+
+ def pa(s, l, tokens):
+ for attrName, attrValue in attrs:
+ if attrName not in tokens:
+ raise ParseException(s, l, "no matching attribute " + attrName)
+ if attrValue != with_attribute.ANY_VALUE and tokens[attrName] != attrValue:
+ raise ParseException(
+ s,
+ l,
+ "attribute {!r} has value {!r}, must be {!r}".format(
+ attrName, tokens[attrName], attrValue
+ ),
+ )
+
+ return pa
+
+
+with_attribute.ANY_VALUE = object()
+
+
+def with_class(classname, namespace=""):
+ """
+ Simplified version of :class:`with_attribute` when
+ matching on a div class - made difficult because ``class`` is
+ a reserved word in Python.
+
+ Example::
+
+ html = '''
+ <div>
+ Some text
+ <div class="grid">1 4 0 1 0</div>
+ <div class="graph">1,3 2,3 1,1</div>
+ <div>this <div> has no class</div>
+ </div>
+
+ '''
+ div,div_end = make_html_tags("div")
+ div_grid = div().set_parse_action(with_class("grid"))
+
+ grid_expr = div_grid + SkipTo(div | div_end)("body")
+ for grid_header in grid_expr.search_string(html):
+ print(grid_header.body)
+
+ div_any_type = div().set_parse_action(with_class(withAttribute.ANY_VALUE))
+ div_expr = div_any_type + SkipTo(div | div_end)("body")
+ for div_header in div_expr.search_string(html):
+ print(div_header.body)
+
+ prints::
+
+ 1 4 0 1 0
+
+ 1 4 0 1 0
+ 1,3 2,3 1,1
+ """
+ classattr = "{}:class".format(namespace) if namespace else "class"
+ return with_attribute(**{classattr: classname})
+
+
+# pre-PEP8 compatibility symbols
+replaceWith = replace_with
+removeQuotes = remove_quotes
+withAttribute = with_attribute
+withClass = with_class
+matchOnlyAtCol = match_only_at_col
--- /dev/null
+# common.py
+from .core import *
+from .helpers import delimited_list, any_open_tag, any_close_tag
+from datetime import datetime
+
+
+# some other useful expressions - using lower-case class name since we are really using this as a namespace
+class pyparsing_common:
+ """Here are some common low-level expressions that may be useful in
+ jump-starting parser development:
+
+ - numeric forms (:class:`integers<integer>`, :class:`reals<real>`,
+ :class:`scientific notation<sci_real>`)
+ - common :class:`programming identifiers<identifier>`
+ - network addresses (:class:`MAC<mac_address>`,
+ :class:`IPv4<ipv4_address>`, :class:`IPv6<ipv6_address>`)
+ - ISO8601 :class:`dates<iso8601_date>` and
+ :class:`datetime<iso8601_datetime>`
+ - :class:`UUID<uuid>`
+ - :class:`comma-separated list<comma_separated_list>`
+ - :class:`url`
+
+ Parse actions:
+
+ - :class:`convertToInteger`
+ - :class:`convertToFloat`
+ - :class:`convertToDate`
+ - :class:`convertToDatetime`
+ - :class:`stripHTMLTags`
+ - :class:`upcaseTokens`
+ - :class:`downcaseTokens`
+
+ Example::
+
+ pyparsing_common.number.runTests('''
+ # any int or real number, returned as the appropriate type
+ 100
+ -100
+ +100
+ 3.14159
+ 6.02e23
+ 1e-12
+ ''')
+
+ pyparsing_common.fnumber.runTests('''
+ # any int or real number, returned as float
+ 100
+ -100
+ +100
+ 3.14159
+ 6.02e23
+ 1e-12
+ ''')
+
+ pyparsing_common.hex_integer.runTests('''
+ # hex numbers
+ 100
+ FF
+ ''')
+
+ pyparsing_common.fraction.runTests('''
+ # fractions
+ 1/2
+ -3/4
+ ''')
+
+ pyparsing_common.mixed_integer.runTests('''
+ # mixed fractions
+ 1
+ 1/2
+ -3/4
+ 1-3/4
+ ''')
+
+ import uuid
+ pyparsing_common.uuid.setParseAction(tokenMap(uuid.UUID))
+ pyparsing_common.uuid.runTests('''
+ # uuid
+ 12345678-1234-5678-1234-567812345678
+ ''')
+
+ prints::
+
+ # any int or real number, returned as the appropriate type
+ 100
+ [100]
+
+ -100
+ [-100]
+
+ +100
+ [100]
+
+ 3.14159
+ [3.14159]
+
+ 6.02e23
+ [6.02e+23]
+
+ 1e-12
+ [1e-12]
+
+ # any int or real number, returned as float
+ 100
+ [100.0]
+
+ -100
+ [-100.0]
+
+ +100
+ [100.0]
+
+ 3.14159
+ [3.14159]
+
+ 6.02e23
+ [6.02e+23]
+
+ 1e-12
+ [1e-12]
+
+ # hex numbers
+ 100
+ [256]
+
+ FF
+ [255]
+
+ # fractions
+ 1/2
+ [0.5]
+
+ -3/4
+ [-0.75]
+
+ # mixed fractions
+ 1
+ [1]
+
+ 1/2
+ [0.5]
+
+ -3/4
+ [-0.75]
+
+ 1-3/4
+ [1.75]
+
+ # uuid
+ 12345678-1234-5678-1234-567812345678
+ [UUID('12345678-1234-5678-1234-567812345678')]
+ """
+
+ convert_to_integer = token_map(int)
+ """
+ Parse action for converting parsed integers to Python int
+ """
+
+ convert_to_float = token_map(float)
+ """
+ Parse action for converting parsed numbers to Python float
+ """
+
+ integer = Word(nums).set_name("integer").set_parse_action(convert_to_integer)
+ """expression that parses an unsigned integer, returns an int"""
+
+ hex_integer = (
+ Word(hexnums).set_name("hex integer").set_parse_action(token_map(int, 16))
+ )
+ """expression that parses a hexadecimal integer, returns an int"""
+
+ signed_integer = (
+ Regex(r"[+-]?\d+")
+ .set_name("signed integer")
+ .set_parse_action(convert_to_integer)
+ )
+ """expression that parses an integer with optional leading sign, returns an int"""
+
+ fraction = (
+ signed_integer().set_parse_action(convert_to_float)
+ + "/"
+ + signed_integer().set_parse_action(convert_to_float)
+ ).set_name("fraction")
+ """fractional expression of an integer divided by an integer, returns a float"""
+ fraction.add_parse_action(lambda tt: tt[0] / tt[-1])
+
+ mixed_integer = (
+ fraction | signed_integer + Opt(Opt("-").suppress() + fraction)
+ ).set_name("fraction or mixed integer-fraction")
+ """mixed integer of the form 'integer - fraction', with optional leading integer, returns float"""
+ mixed_integer.add_parse_action(sum)
+
+ real = (
+ Regex(r"[+-]?(?:\d+\.\d*|\.\d+)")
+ .set_name("real number")
+ .set_parse_action(convert_to_float)
+ )
+ """expression that parses a floating point number and returns a float"""
+
+ sci_real = (
+ Regex(r"[+-]?(?:\d+(?:[eE][+-]?\d+)|(?:\d+\.\d*|\.\d+)(?:[eE][+-]?\d+)?)")
+ .set_name("real number with scientific notation")
+ .set_parse_action(convert_to_float)
+ )
+ """expression that parses a floating point number with optional
+ scientific notation and returns a float"""
+
+ # streamlining this expression makes the docs nicer-looking
+ number = (sci_real | real | signed_integer).setName("number").streamline()
+ """any numeric expression, returns the corresponding Python type"""
+
+ fnumber = (
+ Regex(r"[+-]?\d+\.?\d*([eE][+-]?\d+)?")
+ .set_name("fnumber")
+ .set_parse_action(convert_to_float)
+ )
+ """any int or real number, returned as float"""
+
+ identifier = Word(identchars, identbodychars).set_name("identifier")
+ """typical code identifier (leading alpha or '_', followed by 0 or more alphas, nums, or '_')"""
+
+ ipv4_address = Regex(
+ r"(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})(\.(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})){3}"
+ ).set_name("IPv4 address")
+ "IPv4 address (``0.0.0.0 - 255.255.255.255``)"
+
+ _ipv6_part = Regex(r"[0-9a-fA-F]{1,4}").set_name("hex_integer")
+ _full_ipv6_address = (_ipv6_part + (":" + _ipv6_part) * 7).set_name(
+ "full IPv6 address"
+ )
+ _short_ipv6_address = (
+ Opt(_ipv6_part + (":" + _ipv6_part) * (0, 6))
+ + "::"
+ + Opt(_ipv6_part + (":" + _ipv6_part) * (0, 6))
+ ).set_name("short IPv6 address")
+ _short_ipv6_address.add_condition(
+ lambda t: sum(1 for tt in t if pyparsing_common._ipv6_part.matches(tt)) < 8
+ )
+ _mixed_ipv6_address = ("::ffff:" + ipv4_address).set_name("mixed IPv6 address")
+ ipv6_address = Combine(
+ (_full_ipv6_address | _mixed_ipv6_address | _short_ipv6_address).set_name(
+ "IPv6 address"
+ )
+ ).set_name("IPv6 address")
+ "IPv6 address (long, short, or mixed form)"
+
+ mac_address = Regex(
+ r"[0-9a-fA-F]{2}([:.-])[0-9a-fA-F]{2}(?:\1[0-9a-fA-F]{2}){4}"
+ ).set_name("MAC address")
+ "MAC address xx:xx:xx:xx:xx (may also have '-' or '.' delimiters)"
+
+ @staticmethod
+ def convert_to_date(fmt: str = "%Y-%m-%d"):
+ """
+ Helper to create a parse action for converting parsed date string to Python datetime.date
+
+ Params -
+ - fmt - format to be passed to datetime.strptime (default= ``"%Y-%m-%d"``)
+
+ Example::
+
+ date_expr = pyparsing_common.iso8601_date.copy()
+ date_expr.setParseAction(pyparsing_common.convertToDate())
+ print(date_expr.parseString("1999-12-31"))
+
+ prints::
+
+ [datetime.date(1999, 12, 31)]
+ """
+
+ def cvt_fn(ss, ll, tt):
+ try:
+ return datetime.strptime(tt[0], fmt).date()
+ except ValueError as ve:
+ raise ParseException(ss, ll, str(ve))
+
+ return cvt_fn
+
+ @staticmethod
+ def convert_to_datetime(fmt: str = "%Y-%m-%dT%H:%M:%S.%f"):
+ """Helper to create a parse action for converting parsed
+ datetime string to Python datetime.datetime
+
+ Params -
+ - fmt - format to be passed to datetime.strptime (default= ``"%Y-%m-%dT%H:%M:%S.%f"``)
+
+ Example::
+
+ dt_expr = pyparsing_common.iso8601_datetime.copy()
+ dt_expr.setParseAction(pyparsing_common.convertToDatetime())
+ print(dt_expr.parseString("1999-12-31T23:59:59.999"))
+
+ prints::
+
+ [datetime.datetime(1999, 12, 31, 23, 59, 59, 999000)]
+ """
+
+ def cvt_fn(s, l, t):
+ try:
+ return datetime.strptime(t[0], fmt)
+ except ValueError as ve:
+ raise ParseException(s, l, str(ve))
+
+ return cvt_fn
+
+ iso8601_date = Regex(
+ r"(?P<year>\d{4})(?:-(?P<month>\d\d)(?:-(?P<day>\d\d))?)?"
+ ).set_name("ISO8601 date")
+ "ISO8601 date (``yyyy-mm-dd``)"
+
+ iso8601_datetime = Regex(
+ r"(?P<year>\d{4})-(?P<month>\d\d)-(?P<day>\d\d)[T ](?P<hour>\d\d):(?P<minute>\d\d)(:(?P<second>\d\d(\.\d*)?)?)?(?P<tz>Z|[+-]\d\d:?\d\d)?"
+ ).set_name("ISO8601 datetime")
+ "ISO8601 datetime (``yyyy-mm-ddThh:mm:ss.s(Z|+-00:00)``) - trailing seconds, milliseconds, and timezone optional; accepts separating ``'T'`` or ``' '``"
+
+ uuid = Regex(r"[0-9a-fA-F]{8}(-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}").set_name("UUID")
+ "UUID (``xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx``)"
+
+ _html_stripper = any_open_tag.suppress() | any_close_tag.suppress()
+
+ @staticmethod
+ def strip_html_tags(s: str, l: int, tokens: ParseResults):
+ """Parse action to remove HTML tags from web page HTML source
+
+ Example::
+
+ # strip HTML links from normal text
+ text = '<td>More info at the <a href="https://github.com/pyparsing/pyparsing/wiki">pyparsing</a> wiki page</td>'
+ td, td_end = makeHTMLTags("TD")
+ table_text = td + SkipTo(td_end).setParseAction(pyparsing_common.stripHTMLTags)("body") + td_end
+ print(table_text.parseString(text).body)
+
+ Prints::
+
+ More info at the pyparsing wiki page
+ """
+ return pyparsing_common._html_stripper.transform_string(tokens[0])
+
+ _commasepitem = (
+ Combine(
+ OneOrMore(
+ ~Literal(",")
+ + ~LineEnd()
+ + Word(printables, exclude_chars=",")
+ + Opt(White(" \t") + ~FollowedBy(LineEnd() | ","))
+ )
+ )
+ .streamline()
+ .set_name("commaItem")
+ )
+ comma_separated_list = delimited_list(
+ Opt(quoted_string.copy() | _commasepitem, default="")
+ ).set_name("comma separated list")
+ """Predefined expression of 1 or more printable words or quoted strings, separated by commas."""
+
+ upcase_tokens = staticmethod(token_map(lambda t: t.upper()))
+ """Parse action to convert tokens to upper case."""
+
+ downcase_tokens = staticmethod(token_map(lambda t: t.lower()))
+ """Parse action to convert tokens to lower case."""
+
+ # fmt: off
+ url = Regex(
+ # https://mathiasbynens.be/demo/url-regex
+ # https://gist.github.com/dperini/729294
+ r"^" +
+ # protocol identifier (optional)
+ # short syntax // still required
+ r"(?:(?:(?P<scheme>https?|ftp):)?\/\/)" +
+ # user:pass BasicAuth (optional)
+ r"(?:(?P<auth>\S+(?::\S*)?)@)?" +
+ r"(?P<host>" +
+ # IP address exclusion
+ # private & local networks
+ r"(?!(?:10|127)(?:\.\d{1,3}){3})" +
+ r"(?!(?:169\.254|192\.168)(?:\.\d{1,3}){2})" +
+ r"(?!172\.(?:1[6-9]|2\d|3[0-1])(?:\.\d{1,3}){2})" +
+ # IP address dotted notation octets
+ # excludes loopback network 0.0.0.0
+ # excludes reserved space >= 224.0.0.0
+ # excludes network & broadcast addresses
+ # (first & last IP address of each class)
+ r"(?:[1-9]\d?|1\d\d|2[01]\d|22[0-3])" +
+ r"(?:\.(?:1?\d{1,2}|2[0-4]\d|25[0-5])){2}" +
+ r"(?:\.(?:[1-9]\d?|1\d\d|2[0-4]\d|25[0-4]))" +
+ r"|" +
+ # host & domain names, may end with dot
+ # can be replaced by a shortest alternative
+ # (?![-_])(?:[-\w\u00a1-\uffff]{0,63}[^-_]\.)+
+ r"(?:" +
+ r"(?:" +
+ r"[a-z0-9\u00a1-\uffff]" +
+ r"[a-z0-9\u00a1-\uffff_-]{0,62}" +
+ r")?" +
+ r"[a-z0-9\u00a1-\uffff]\." +
+ r")+" +
+ # TLD identifier name, may end with dot
+ r"(?:[a-z\u00a1-\uffff]{2,}\.?)" +
+ r")" +
+ # port number (optional)
+ r"(:(?P<port>\d{2,5}))?" +
+ # resource path (optional)
+ r"(?P<path>\/[^?# ]*)?" +
+ # query string (optional)
+ r"(\?(?P<query>[^#]*))?" +
+ # fragment (optional)
+ r"(#(?P<fragment>\S*))?" +
+ r"$"
+ ).set_name("url")
+ # fmt: on
+
+ # pre-PEP8 compatibility names
+ convertToInteger = convert_to_integer
+ convertToFloat = convert_to_float
+ convertToDate = convert_to_date
+ convertToDatetime = convert_to_datetime
+ stripHTMLTags = strip_html_tags
+ upcaseTokens = upcase_tokens
+ downcaseTokens = downcase_tokens
+
+
+_builtin_exprs = [
+ v for v in vars(pyparsing_common).values() if isinstance(v, ParserElement)
+]
--- /dev/null
+#
+# core.py
+#
+import os
+from typing import (
+ Optional as OptionalType,
+ Iterable as IterableType,
+ Union,
+ Callable,
+ Any,
+ Generator,
+ Tuple,
+ List,
+ TextIO,
+ Set,
+)
+from abc import ABC, abstractmethod
+from enum import Enum
+import string
+import copy
+import warnings
+import re
+import sre_constants
+import sys
+from collections.abc import Iterable
+import traceback
+import types
+from operator import itemgetter
+from functools import wraps
+from threading import RLock
+from pathlib import Path
+
+from .util import (
+ _FifoCache,
+ _UnboundedCache,
+ __config_flags,
+ _collapseStringToRanges,
+ _escapeRegexRangeChars,
+ _bslash,
+ _flatten,
+ LRUMemo as _LRUMemo,
+ UnboundedMemo as _UnboundedMemo,
+)
+from .exceptions import *
+from .actions import *
+from .results import ParseResults, _ParseResultsWithOffset
+from .unicode import pyparsing_unicode
+
+_MAX_INT = sys.maxsize
+str_type = (str, bytes)
+
+#
+# Copyright (c) 2003-2021 Paul T. McGuire
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+
+
+class __compat__(__config_flags):
+ """
+ A cross-version compatibility configuration for pyparsing features that will be
+ released in a future version. By setting values in this configuration to True,
+ those features can be enabled in prior versions for compatibility development
+ and testing.
+
+ - ``collect_all_And_tokens`` - flag to enable fix for Issue #63 that fixes erroneous grouping
+ of results names when an :class:`And` expression is nested within an :class:`Or` or :class:`MatchFirst`;
+ maintained for compatibility, but setting to ``False`` no longer restores pre-2.3.1
+ behavior
+ """
+
+ _type_desc = "compatibility"
+
+ collect_all_And_tokens = True
+
+ _all_names = [__ for __ in locals() if not __.startswith("_")]
+ _fixed_names = """
+ collect_all_And_tokens
+ """.split()
+
+
+class __diag__(__config_flags):
+ _type_desc = "diagnostic"
+
+ warn_multiple_tokens_in_named_alternation = False
+ warn_ungrouped_named_tokens_in_collection = False
+ warn_name_set_on_empty_Forward = False
+ warn_on_parse_using_empty_Forward = False
+ warn_on_assignment_to_Forward = False
+ warn_on_multiple_string_args_to_oneof = False
+ warn_on_match_first_with_lshift_operator = False
+ enable_debug_on_named_expressions = False
+
+ _all_names = [__ for __ in locals() if not __.startswith("_")]
+ _warning_names = [name for name in _all_names if name.startswith("warn")]
+ _debug_names = [name for name in _all_names if name.startswith("enable_debug")]
+
+ @classmethod
+ def enable_all_warnings(cls):
+ for name in cls._warning_names:
+ cls.enable(name)
+
+
+class Diagnostics(Enum):
+ """
+ Diagnostic configuration (all default to disabled)
+ - ``warn_multiple_tokens_in_named_alternation`` - flag to enable warnings when a results
+ name is defined on a :class:`MatchFirst` or :class:`Or` expression with one or more :class:`And` subexpressions
+ - ``warn_ungrouped_named_tokens_in_collection`` - flag to enable warnings when a results
+ name is defined on a containing expression with ungrouped subexpressions that also
+ have results names
+ - ``warn_name_set_on_empty_Forward`` - flag to enable warnings when a :class:`Forward` is defined
+ with a results name, but has no contents defined
+ - ``warn_on_parse_using_empty_Forward`` - flag to enable warnings when a :class:`Forward` is
+ defined in a grammar but has never had an expression attached to it
+ - ``warn_on_assignment_to_Forward`` - flag to enable warnings when a :class:`Forward` is defined
+ but is overwritten by assigning using ``'='`` instead of ``'<<='`` or ``'<<'``
+ - ``warn_on_multiple_string_args_to_oneof`` - flag to enable warnings when :class:`one_of` is
+ incorrectly called with multiple str arguments
+ - ``enable_debug_on_named_expressions`` - flag to auto-enable debug on all subsequent
+ calls to :class:`ParserElement.set_name`
+
+ Diagnostics are enabled/disabled by calling :class:`enable_diag` and :class:`disable_diag`.
+ All warnings can be enabled by calling :class:`enable_all_warnings`.
+ """
+
+ warn_multiple_tokens_in_named_alternation = 0
+ warn_ungrouped_named_tokens_in_collection = 1
+ warn_name_set_on_empty_Forward = 2
+ warn_on_parse_using_empty_Forward = 3
+ warn_on_assignment_to_Forward = 4
+ warn_on_multiple_string_args_to_oneof = 5
+ warn_on_match_first_with_lshift_operator = 6
+ enable_debug_on_named_expressions = 7
+
+
+def enable_diag(diag_enum):
+ """
+ Enable a global pyparsing diagnostic flag (see :class:`Diagnostics`).
+ """
+ __diag__.enable(diag_enum.name)
+
+
+def disable_diag(diag_enum):
+ """
+ Disable a global pyparsing diagnostic flag (see :class:`Diagnostics`).
+ """
+ __diag__.disable(diag_enum.name)
+
+
+def enable_all_warnings():
+ """
+ Enable all global pyparsing diagnostic warnings (see :class:`Diagnostics`).
+ """
+ __diag__.enable_all_warnings()
+
+
+# hide abstract class
+del __config_flags
+
+
+def _should_enable_warnings(cmd_line_warn_options: List[str], warn_env_var: str) -> bool:
+ enable = bool(warn_env_var)
+ for warn_opt in cmd_line_warn_options:
+ w_action, w_message, w_category, w_module, w_line = (warn_opt + "::::").split(
+ ":"
+ )[:5]
+ if not w_action.lower().startswith("i") and (
+ not (w_message or w_category or w_module) or w_module == "pyparsing"
+ ):
+ enable = True
+ elif w_action.lower().startswith("i") and w_module in ("pyparsing", ""):
+ enable = False
+ return enable
+
+
+if _should_enable_warnings(sys.warnoptions, os.environ.get("PYPARSINGENABLEALLWARNINGS")):
+ enable_all_warnings()
+
+
+# build list of single arg builtins, that can be used as parse actions
+_single_arg_builtins = {
+ sum,
+ len,
+ sorted,
+ reversed,
+ list,
+ tuple,
+ set,
+ any,
+ all,
+ min,
+ max,
+}
+
+_generatorType = types.GeneratorType
+ParseAction = Union[
+ Callable[[], Any],
+ Callable[[ParseResults], Any],
+ Callable[[int, ParseResults], Any],
+ Callable[[str, int, ParseResults], Any],
+]
+ParseCondition = Union[
+ Callable[[], bool],
+ Callable[[ParseResults], bool],
+ Callable[[int, ParseResults], bool],
+ Callable[[str, int, ParseResults], bool],
+]
+ParseFailAction = Callable[[str, int, "ParserElement", Exception], None]
+DebugStartAction = Callable[[str, int, "ParserElement", bool], None]
+DebugSuccessAction = Callable[
+ [str, int, int, "ParserElement", ParseResults, bool], None
+]
+DebugExceptionAction = Callable[[str, int, "ParserElement", Exception, bool], None]
+
+
+alphas = string.ascii_uppercase + string.ascii_lowercase
+identchars = pyparsing_unicode.Latin1.identchars
+identbodychars = pyparsing_unicode.Latin1.identbodychars
+nums = "0123456789"
+hexnums = nums + "ABCDEFabcdef"
+alphanums = alphas + nums
+printables = "".join(c for c in string.printable if c not in string.whitespace)
+
+_trim_arity_call_line = None
+
+
+def _trim_arity(func, maxargs=2):
+ """decorator to trim function calls to match the arity of the target"""
+ global _trim_arity_call_line
+
+ if func in _single_arg_builtins:
+ return lambda s, l, t: func(t)
+
+ limit = 0
+ found_arity = False
+
+ def extract_tb(tb, limit=0):
+ frames = traceback.extract_tb(tb, limit=limit)
+ frame_summary = frames[-1]
+ return [frame_summary[:2]]
+
+ # synthesize what would be returned by traceback.extract_stack at the call to
+ # user's parse action 'func', so that we don't incur call penalty at parse time
+
+ LINE_DIFF = 11
+ # IF ANY CODE CHANGES, EVEN JUST COMMENTS OR BLANK LINES, BETWEEN THE NEXT LINE AND
+ # THE CALL TO FUNC INSIDE WRAPPER, LINE_DIFF MUST BE MODIFIED!!!!
+ _trim_arity_call_line = (
+ _trim_arity_call_line or traceback.extract_stack(limit=2)[-1]
+ )
+ pa_call_line_synth = (
+ _trim_arity_call_line[0],
+ _trim_arity_call_line[1] + LINE_DIFF,
+ )
+
+ def wrapper(*args):
+ nonlocal found_arity, limit
+ while 1:
+ try:
+ ret = func(*args[limit:])
+ found_arity = True
+ return ret
+ except TypeError as te:
+ # re-raise TypeErrors if they did not come from our arity testing
+ if found_arity:
+ raise
+ else:
+ tb = te.__traceback__
+ trim_arity_type_error = (
+ extract_tb(tb, limit=2)[-1][:2] == pa_call_line_synth
+ )
+ del tb
+
+ if trim_arity_type_error:
+ if limit <= maxargs:
+ limit += 1
+ continue
+
+ raise
+
+ # copy func name to wrapper for sensible debug output
+ # (can't use functools.wraps, since that messes with function signature)
+ func_name = getattr(func, "__name__", getattr(func, "__class__").__name__)
+ wrapper.__name__ = func_name
+
+ return wrapper
+
+
+def condition_as_parse_action(
+ fn: ParseCondition, message: str = None, fatal: bool = False
+):
+ """
+ Function to convert a simple predicate function that returns ``True`` or ``False``
+ into a parse action. Can be used in places when a parse action is required
+ and :class:`ParserElement.add_condition` cannot be used (such as when adding a condition
+ to an operator level in :class:`infix_notation`).
+
+ Optional keyword arguments:
+
+ - ``message`` - define a custom message to be used in the raised exception
+ - ``fatal`` - if True, will raise :class:`ParseFatalException` to stop parsing immediately;
+ otherwise will raise :class:`ParseException`
+
+ """
+ msg = message if message is not None else "failed user-defined condition"
+ exc_type = ParseFatalException if fatal else ParseException
+ fn = _trim_arity(fn)
+
+ @wraps(fn)
+ def pa(s, l, t):
+ if not bool(fn(s, l, t)):
+ raise exc_type(s, l, msg)
+
+ return pa
+
+
+def _default_start_debug_action(
+ instring: str, loc: int, expr: "ParserElement", cache_hit: bool = False
+):
+ cache_hit_str = "*" if cache_hit else ""
+ print(
+ (
+ "{}Match {} at loc {}({},{})\n {}\n {}^".format(
+ cache_hit_str,
+ expr,
+ loc,
+ lineno(loc, instring),
+ col(loc, instring),
+ line(loc, instring),
+ " " * (col(loc, instring) - 1),
+ )
+ )
+ )
+
+
+def _default_success_debug_action(
+ instring: str,
+ startloc: int,
+ endloc: int,
+ expr: "ParserElement",
+ toks: ParseResults,
+ cache_hit: bool = False,
+):
+ cache_hit_str = "*" if cache_hit else ""
+ print("{}Matched {} -> {}".format(cache_hit_str, expr, toks.as_list()))
+
+
+def _default_exception_debug_action(
+ instring: str,
+ loc: int,
+ expr: "ParserElement",
+ exc: Exception,
+ cache_hit: bool = False,
+):
+ cache_hit_str = "*" if cache_hit else ""
+ print(
+ "{}Match {} failed, {} raised: {}".format(
+ cache_hit_str, expr, type(exc).__name__, exc
+ )
+ )
+
+
+def null_debug_action(*args):
+ """'Do-nothing' debug action, to suppress debugging output during parsing."""
+
+
+class ParserElement(ABC):
+ """Abstract base level parser element class."""
+
+ DEFAULT_WHITE_CHARS: str = " \n\t\r"
+ verbose_stacktrace: bool = False
+ _literalStringClass: OptionalType[type] = None
+
+ @staticmethod
+ def set_default_whitespace_chars(chars: str):
+ r"""
+ Overrides the default whitespace chars
+
+ Example::
+
+ # default whitespace chars are space, <TAB> and newline
+ OneOrMore(Word(alphas)).parse_string("abc def\nghi jkl") # -> ['abc', 'def', 'ghi', 'jkl']
+
+ # change to just treat newline as significant
+ ParserElement.set_default_whitespace_chars(" \t")
+ OneOrMore(Word(alphas)).parse_string("abc def\nghi jkl") # -> ['abc', 'def']
+ """
+ ParserElement.DEFAULT_WHITE_CHARS = chars
+
+ # update whitespace all parse expressions defined in this module
+ for expr in _builtin_exprs:
+ if expr.copyDefaultWhiteChars:
+ expr.whiteChars = chars
+
+ @staticmethod
+ def inline_literals_using(cls: type):
+ """
+ Set class to be used for inclusion of string literals into a parser.
+
+ Example::
+
+ # default literal class used is Literal
+ integer = Word(nums)
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+
+ date_str.parse_string("1999/12/31") # -> ['1999', '/', '12', '/', '31']
+
+
+ # change to Suppress
+ ParserElement.inline_literals_using(Suppress)
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+
+ date_str.parse_string("1999/12/31") # -> ['1999', '12', '31']
+ """
+ ParserElement._literalStringClass = cls
+
+ def __init__(self, savelist: bool = False):
+ self.parseAction: List[ParseAction] = list()
+ self.failAction: OptionalType[ParseFailAction] = None
+ self.customName = None
+ self._defaultName = None
+ self.resultsName = None
+ self.saveAsList = savelist
+ self.skipWhitespace = True
+ self.whiteChars = set(ParserElement.DEFAULT_WHITE_CHARS)
+ self.copyDefaultWhiteChars = True
+ # used when checking for left-recursion
+ self.mayReturnEmpty = False
+ self.keepTabs = False
+ self.ignoreExprs: List["ParserElement"] = list()
+ self.debug = False
+ self.streamlined = False
+ # optimize exception handling for subclasses that don't advance parse index
+ self.mayIndexError = True
+ self.errmsg = ""
+ # mark results names as modal (report only last) or cumulative (list all)
+ self.modalResults = True
+ # custom debug actions
+ self.debugActions: Tuple[
+ OptionalType[DebugStartAction],
+ OptionalType[DebugSuccessAction],
+ OptionalType[DebugExceptionAction],
+ ] = (None, None, None)
+ self.re = None
+ # avoid redundant calls to preParse
+ self.callPreparse = True
+ self.callDuringTry = False
+
+ def copy(self) -> "ParserElement":
+ """
+ Make a copy of this :class:`ParserElement`. Useful for defining
+ different parse actions for the same parsing pattern, using copies of
+ the original parse element.
+
+ Example::
+
+ integer = Word(nums).set_parse_action(lambda toks: int(toks[0]))
+ integerK = integer.copy().add_parse_action(lambda toks: toks[0] * 1024) + Suppress("K")
+ integerM = integer.copy().add_parse_action(lambda toks: toks[0] * 1024 * 1024) + Suppress("M")
+
+ print(OneOrMore(integerK | integerM | integer).parse_string("5K 100 640K 256M"))
+
+ prints::
+
+ [5120, 100, 655360, 268435456]
+
+ Equivalent form of ``expr.copy()`` is just ``expr()``::
+
+ integerM = integer().add_parse_action(lambda toks: toks[0] * 1024 * 1024) + Suppress("M")
+ """
+ cpy = copy.copy(self)
+ cpy.parseAction = self.parseAction[:]
+ cpy.ignoreExprs = self.ignoreExprs[:]
+ if self.copyDefaultWhiteChars:
+ cpy.whiteChars = set(ParserElement.DEFAULT_WHITE_CHARS)
+ return cpy
+
+ def set_results_name(
+ self, name: str, list_all_matches: bool = False, *, listAllMatches: bool = False
+ ) -> "ParserElement":
+ """
+ Define name for referencing matching tokens as a nested attribute
+ of the returned parse results.
+
+ Normally, results names are assigned as you would assign keys in a dict:
+ any existing value is overwritten by later values. If it is necessary to
+ keep all values captured for a particular results name, call ``set_results_name``
+ with ``list_all_matches`` = True.
+
+ NOTE: ``set_results_name`` returns a *copy* of the original :class:`ParserElement` object;
+ this is so that the client can define a basic element, such as an
+ integer, and reference it in multiple places with different names.
+
+ You can also set results names using the abbreviated syntax,
+ ``expr("name")`` in place of ``expr.set_results_name("name")``
+ - see :class:`__call__`. If ``list_all_matches`` is required, use
+ ``expr("name*")``.
+
+ Example::
+
+ date_str = (integer.set_results_name("year") + '/'
+ + integer.set_results_name("month") + '/'
+ + integer.set_results_name("day"))
+
+ # equivalent form:
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+ """
+ listAllMatches = listAllMatches or list_all_matches
+ return self._setResultsName(name, listAllMatches)
+
+ def _setResultsName(self, name, listAllMatches=False):
+ if name is None:
+ return self
+ newself = self.copy()
+ if name.endswith("*"):
+ name = name[:-1]
+ listAllMatches = True
+ newself.resultsName = name
+ newself.modalResults = not listAllMatches
+ return newself
+
+ def set_break(self, break_flag: bool = True) -> "ParserElement":
+ """
+ Method to invoke the Python pdb debugger when this element is
+ about to be parsed. Set ``break_flag`` to ``True`` to enable, ``False`` to
+ disable.
+ """
+ if break_flag:
+ _parseMethod = self._parse
+
+ def breaker(instring, loc, doActions=True, callPreParse=True):
+ import pdb
+
+ # this call to pdb.set_trace() is intentional, not a checkin error
+ pdb.set_trace()
+ return _parseMethod(instring, loc, doActions, callPreParse)
+
+ breaker._originalParseMethod = _parseMethod
+ self._parse = breaker
+ else:
+ if hasattr(self._parse, "_originalParseMethod"):
+ self._parse = self._parse._originalParseMethod
+ return self
+
+ def set_parse_action(self, *fns: ParseAction, **kwargs) -> OptionalType["ParserElement"]:
+ """
+ Define one or more actions to perform when successfully matching parse element definition.
+
+ Parse actions can be called to perform data conversions, do extra validation,
+ update external data structures, or enhance or replace the parsed tokens.
+ Each parse action ``fn`` is a callable method with 0-3 arguments, called as
+ ``fn(s, loc, toks)`` , ``fn(loc, toks)`` , ``fn(toks)`` , or just ``fn()`` , where:
+
+ - s = the original string being parsed (see note below)
+ - loc = the location of the matching substring
+ - toks = a list of the matched tokens, packaged as a :class:`ParseResults` object
+
+ The parsed tokens are passed to the parse action as ParseResults. They can be
+ modified in place using list-style append, extend, and pop operations to update
+ the parsed list elements; and with dictionary-style item set and del operations
+ to add, update, or remove any named results. If the tokens are modified in place,
+ it is not necessary to return them with a return statement.
+
+ Parse actions can also completely replace the given tokens, with another ``ParseResults``
+ object, or with some entirely different object (common for parse actions that perform data
+ conversions). A convenient way to build a new parse result is to define the values
+ using a dict, and then create the return value using :class:`ParseResults.from_dict`.
+
+ If None is passed as the ``fn`` parse action, all previously added parse actions for this
+ expression are cleared.
+
+ Optional keyword arguments:
+
+ - call_during_try = (default= ``False``) indicate if parse action should be run during
+ lookaheads and alternate testing. For parse actions that have side effects, it is
+ important to only call the parse action once it is determined that it is being
+ called as part of a successful parse. For parse actions that perform additional
+ validation, then call_during_try should be passed as True, so that the validation
+ code is included in the preliminary "try" parses.
+
+ Note: the default parsing behavior is to expand tabs in the input string
+ before starting the parsing process. See :class:`parse_string` for more
+ information on parsing strings containing ``<TAB>`` s, and suggested
+ methods to maintain a consistent view of the parsed string, the parse
+ location, and line and column positions within the parsed string.
+
+ Example::
+
+ # parse dates in the form YYYY/MM/DD
+
+ # use parse action to convert toks from str to int at parse time
+ def convert_to_int(toks):
+ return int(toks[0])
+
+ # use a parse action to verify that the date is a valid date
+ def is_valid_date(toks):
+ from datetime import date
+ year, month, day = toks[::2]
+ try:
+ date(year, month, day)
+ except ValueError:
+ raise ParseException("invalid date given")
+
+ integer = Word(nums)
+ date_str = integer + '/' + integer + '/' + integer
+
+ # add parse actions
+ integer.set_parse_action(convert_to_int)
+ date_str.set_parse_action(is_valid_date)
+
+ # note that integer fields are now ints, not strings
+ date_str.run_tests('''
+ # successful parse - note that integer fields were converted to ints
+ 1999/12/31
+
+ # fail - invalid date
+ 1999/13/31
+ ''')
+ """
+ if list(fns) == [None]:
+ self.parseAction = []
+ else:
+ if not all(callable(fn) for fn in fns):
+ raise TypeError("parse actions must be callable")
+ self.parseAction = list(map(_trim_arity, list(fns)))
+ self.callDuringTry = kwargs.get(
+ "call_during_try", kwargs.get("callDuringTry", False)
+ )
+ return self
+
+ def add_parse_action(self, *fns: ParseAction, **kwargs) -> "ParserElement":
+ """
+ Add one or more parse actions to expression's list of parse actions. See :class:`set_parse_action`.
+
+ See examples in :class:`copy`.
+ """
+ self.parseAction += list(map(_trim_arity, list(fns)))
+ self.callDuringTry = self.callDuringTry or kwargs.get(
+ "call_during_try", kwargs.get("callDuringTry", False)
+ )
+ return self
+
+ def add_condition(self, *fns: ParseCondition, **kwargs) -> "ParserElement":
+ """Add a boolean predicate function to expression's list of parse actions. See
+ :class:`set_parse_action` for function call signatures. Unlike ``set_parse_action``,
+ functions passed to ``add_condition`` need to return boolean success/fail of the condition.
+
+ Optional keyword arguments:
+
+ - message = define a custom message to be used in the raised exception
+ - fatal = if True, will raise ParseFatalException to stop parsing immediately; otherwise will raise
+ ParseException
+ - call_during_try = boolean to indicate if this method should be called during internal tryParse calls,
+ default=False
+
+ Example::
+
+ integer = Word(nums).set_parse_action(lambda toks: int(toks[0]))
+ year_int = integer.copy()
+ year_int.add_condition(lambda toks: toks[0] >= 2000, message="Only support years 2000 and later")
+ date_str = year_int + '/' + integer + '/' + integer
+
+ result = date_str.parse_string("1999/12/31") # -> Exception: Only support years 2000 and later (at char 0),
+ (line:1, col:1)
+ """
+ for fn in fns:
+ self.parseAction.append(
+ condition_as_parse_action(
+ fn, message=kwargs.get("message"), fatal=kwargs.get("fatal", False)
+ )
+ )
+
+ self.callDuringTry = self.callDuringTry or kwargs.get(
+ "call_during_try", kwargs.get("callDuringTry", False)
+ )
+ return self
+
+ def set_fail_action(self, fn: ParseFailAction) -> "ParserElement":
+ """
+ Define action to perform if parsing fails at this expression.
+ Fail acton fn is a callable function that takes the arguments
+ ``fn(s, loc, expr, err)`` where:
+
+ - s = string being parsed
+ - loc = location where expression match was attempted and failed
+ - expr = the parse expression that failed
+ - err = the exception thrown
+
+ The function returns no value. It may throw :class:`ParseFatalException`
+ if it is desired to stop parsing immediately."""
+ self.failAction = fn
+ return self
+
+ def _skipIgnorables(self, instring, loc):
+ exprsFound = True
+ while exprsFound:
+ exprsFound = False
+ for e in self.ignoreExprs:
+ try:
+ while 1:
+ loc, dummy = e._parse(instring, loc)
+ exprsFound = True
+ except ParseException:
+ pass
+ return loc
+
+ def preParse(self, instring, loc):
+ if self.ignoreExprs:
+ loc = self._skipIgnorables(instring, loc)
+
+ if self.skipWhitespace:
+ instrlen = len(instring)
+ white_chars = self.whiteChars
+ while loc < instrlen and instring[loc] in white_chars:
+ loc += 1
+
+ return loc
+
+ def parseImpl(self, instring, loc, doActions=True):
+ return loc, []
+
+ def postParse(self, instring, loc, tokenlist):
+ return tokenlist
+
+ # @profile
+ def _parseNoCache(self, instring, loc, doActions=True, callPreParse=True):
+ TRY, MATCH, FAIL = 0, 1, 2
+ debugging = self.debug # and doActions)
+ len_instring = len(instring)
+
+ if debugging or self.failAction:
+ # print("Match {} at loc {}({}, {})".format(self, loc, lineno(loc, instring), col(loc, instring)))
+ try:
+ if callPreParse and self.callPreparse:
+ preloc = self.preParse(instring, loc)
+ else:
+ preloc = loc
+ tokensStart = preloc
+ if self.debugActions[TRY]:
+ self.debugActions[TRY](instring, tokensStart, self)
+ if self.mayIndexError or preloc >= len_instring:
+ try:
+ loc, tokens = self.parseImpl(instring, preloc, doActions)
+ except IndexError:
+ raise ParseException(instring, len_instring, self.errmsg, self)
+ else:
+ loc, tokens = self.parseImpl(instring, preloc, doActions)
+ except Exception as err:
+ # print("Exception raised:", err)
+ if self.debugActions[FAIL]:
+ self.debugActions[FAIL](instring, tokensStart, self, err)
+ if self.failAction:
+ self.failAction(instring, tokensStart, self, err)
+ raise
+ else:
+ if callPreParse and self.callPreparse:
+ preloc = self.preParse(instring, loc)
+ else:
+ preloc = loc
+ tokensStart = preloc
+ if self.mayIndexError or preloc >= len_instring:
+ try:
+ loc, tokens = self.parseImpl(instring, preloc, doActions)
+ except IndexError:
+ raise ParseException(instring, len_instring, self.errmsg, self)
+ else:
+ loc, tokens = self.parseImpl(instring, preloc, doActions)
+
+ tokens = self.postParse(instring, loc, tokens)
+
+ retTokens = ParseResults(
+ tokens, self.resultsName, asList=self.saveAsList, modal=self.modalResults
+ )
+ if self.parseAction and (doActions or self.callDuringTry):
+ if debugging:
+ try:
+ for fn in self.parseAction:
+ try:
+ tokens = fn(instring, tokensStart, retTokens)
+ except IndexError as parse_action_exc:
+ exc = ParseException("exception raised in parse action")
+ raise exc from parse_action_exc
+
+ if tokens is not None and tokens is not retTokens:
+ retTokens = ParseResults(
+ tokens,
+ self.resultsName,
+ asList=self.saveAsList
+ and isinstance(tokens, (ParseResults, list)),
+ modal=self.modalResults,
+ )
+ except Exception as err:
+ # print "Exception raised in user parse action:", err
+ if self.debugActions[FAIL]:
+ self.debugActions[FAIL](instring, tokensStart, self, err)
+ raise
+ else:
+ for fn in self.parseAction:
+ try:
+ tokens = fn(instring, tokensStart, retTokens)
+ except IndexError as parse_action_exc:
+ exc = ParseException("exception raised in parse action")
+ raise exc from parse_action_exc
+
+ if tokens is not None and tokens is not retTokens:
+ retTokens = ParseResults(
+ tokens,
+ self.resultsName,
+ asList=self.saveAsList
+ and isinstance(tokens, (ParseResults, list)),
+ modal=self.modalResults,
+ )
+ if debugging:
+ # print("Matched", self, "->", retTokens.as_list())
+ if self.debugActions[MATCH]:
+ self.debugActions[MATCH](instring, tokensStart, loc, self, retTokens)
+
+ return loc, retTokens
+
+ def try_parse(self, instring: str, loc: int, raise_fatal: bool = False) -> int:
+ try:
+ return self._parse(instring, loc, doActions=False)[0]
+ except ParseFatalException:
+ if raise_fatal:
+ raise
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ def can_parse_next(self, instring: str, loc: int) -> bool:
+ try:
+ self.try_parse(instring, loc)
+ except (ParseException, IndexError):
+ return False
+ else:
+ return True
+
+ # cache for left-recursion in Forward references
+ recursion_lock = RLock()
+ recursion_memos = (
+ {}
+ ) # type: dict[tuple[int, Forward, bool], tuple[int, ParseResults | Exception]]
+
+ # argument cache for optimizing repeated calls when backtracking through recursive expressions
+ packrat_cache = (
+ {}
+ ) # this is set later by enabled_packrat(); this is here so that reset_cache() doesn't fail
+ packrat_cache_lock = RLock()
+ packrat_cache_stats = [0, 0]
+
+ # this method gets repeatedly called during backtracking with the same arguments -
+ # we can cache these arguments and save ourselves the trouble of re-parsing the contained expression
+ def _parseCache(
+ self, instring, loc, doActions=True, callPreParse=True
+ ) -> Tuple[int, ParseResults]:
+ HIT, MISS = 0, 1
+ TRY, MATCH, FAIL = 0, 1, 2
+ lookup = (self, instring, loc, callPreParse, doActions)
+ with ParserElement.packrat_cache_lock:
+ cache = ParserElement.packrat_cache
+ value = cache.get(lookup)
+ if value is cache.not_in_cache:
+ ParserElement.packrat_cache_stats[MISS] += 1
+ try:
+ value = self._parseNoCache(instring, loc, doActions, callPreParse)
+ except ParseBaseException as pe:
+ # cache a copy of the exception, without the traceback
+ cache.set(lookup, pe.__class__(*pe.args))
+ raise
+ else:
+ cache.set(lookup, (value[0], value[1].copy(), loc))
+ return value
+ else:
+ ParserElement.packrat_cache_stats[HIT] += 1
+ if self.debug and self.debugActions[TRY]:
+ try:
+ self.debugActions[TRY](instring, loc, self, cache_hit=True)
+ except TypeError:
+ pass
+ if isinstance(value, Exception):
+ if self.debug and self.debugActions[FAIL]:
+ try:
+ self.debugActions[FAIL](
+ instring, loc, self, value, cache_hit=True
+ )
+ except TypeError:
+ pass
+ raise value
+
+ loc_, result, endloc = value[0], value[1].copy(), value[2]
+ if self.debug and self.debugActions[MATCH]:
+ try:
+ self.debugActions[MATCH](
+ instring, loc_, endloc, self, result, cache_hit=True
+ )
+ except TypeError:
+ pass
+
+ return loc_, result
+
+ _parse = _parseNoCache
+
+ @staticmethod
+ def reset_cache() -> None:
+ ParserElement.packrat_cache.clear()
+ ParserElement.packrat_cache_stats[:] = [0] * len(
+ ParserElement.packrat_cache_stats
+ )
+ ParserElement.recursion_memos.clear()
+
+ _packratEnabled = False
+ _left_recursion_enabled = False
+
+ @staticmethod
+ def disable_memoization() -> None:
+ """
+ Disables active Packrat or Left Recursion parsing and their memoization
+
+ This method also works if neither Packrat nor Left Recursion are enabled.
+ This makes it safe to call before activating Packrat nor Left Recursion
+ to clear any previous settings.
+ """
+ ParserElement.reset_cache()
+ ParserElement._left_recursion_enabled = False
+ ParserElement._packratEnabled = False
+ ParserElement._parse = ParserElement._parseNoCache
+
+ @staticmethod
+ def enable_left_recursion(
+ cache_size_limit: OptionalType[int] = None, *, force=False
+ ) -> None:
+ """
+ Enables "bounded recursion" parsing, which allows for both direct and indirect
+ left-recursion. During parsing, left-recursive :class:`Forward` elements are
+ repeatedly matched with a fixed recursion depth that is gradually increased
+ until finding the longest match.
+
+ Example::
+
+ import pyparsing as pp
+ pp.ParserElement.enable_left_recursion()
+
+ E = pp.Forward("E")
+ num = pp.Word(pp.nums)
+ # match `num`, or `num '+' num`, or `num '+' num '+' num`, ...
+ E <<= E + '+' - num | num
+
+ print(E.parse_string("1+2+3"))
+
+ Recursion search naturally memoizes matches of ``Forward`` elements and may
+ thus skip reevaluation of parse actions during backtracking. This may break
+ programs with parse actions which rely on strict ordering of side-effects.
+
+ Parameters:
+
+ - cache_size_limit - (default=``None``) - memoize at most this many
+ ``Forward`` elements during matching; if ``None`` (the default),
+ memoize all ``Forward`` elements.
+
+ Bounded Recursion parsing works similar but not identical to Packrat parsing,
+ thus the two cannot be used together. Use ``force=True`` to disable any
+ previous, conflicting settings.
+ """
+ if force:
+ ParserElement.disable_memoization()
+ elif ParserElement._packratEnabled:
+ raise RuntimeError("Packrat and Bounded Recursion are not compatible")
+ if cache_size_limit is None:
+ ParserElement.recursion_memos = _UnboundedMemo()
+ elif cache_size_limit > 0:
+ ParserElement.recursion_memos = _LRUMemo(capacity=cache_size_limit)
+ else:
+ raise NotImplementedError("Memo size of %s" % cache_size_limit)
+ ParserElement._left_recursion_enabled = True
+
+ @staticmethod
+ def enable_packrat(cache_size_limit: int = 128, *, force: bool = False) -> None:
+ """
+ Enables "packrat" parsing, which adds memoizing to the parsing logic.
+ Repeated parse attempts at the same string location (which happens
+ often in many complex grammars) can immediately return a cached value,
+ instead of re-executing parsing/validating code. Memoizing is done of
+ both valid results and parsing exceptions.
+
+ Parameters:
+
+ - cache_size_limit - (default= ``128``) - if an integer value is provided
+ will limit the size of the packrat cache; if None is passed, then
+ the cache size will be unbounded; if 0 is passed, the cache will
+ be effectively disabled.
+
+ This speedup may break existing programs that use parse actions that
+ have side-effects. For this reason, packrat parsing is disabled when
+ you first import pyparsing. To activate the packrat feature, your
+ program must call the class method :class:`ParserElement.enable_packrat`.
+ For best results, call ``enable_packrat()`` immediately after
+ importing pyparsing.
+
+ Example::
+
+ import pyparsing
+ pyparsing.ParserElement.enable_packrat()
+
+ Packrat parsing works similar but not identical to Bounded Recursion parsing,
+ thus the two cannot be used together. Use ``force=True`` to disable any
+ previous, conflicting settings.
+ """
+ if force:
+ ParserElement.disable_memoization()
+ elif ParserElement._left_recursion_enabled:
+ raise RuntimeError("Packrat and Bounded Recursion are not compatible")
+ if not ParserElement._packratEnabled:
+ ParserElement._packratEnabled = True
+ if cache_size_limit is None:
+ ParserElement.packrat_cache = _UnboundedCache()
+ else:
+ ParserElement.packrat_cache = _FifoCache(cache_size_limit)
+ ParserElement._parse = ParserElement._parseCache
+
+ def parse_string(
+ self, instring: str, parse_all: bool = False, *, parseAll: bool = False
+ ) -> ParseResults:
+ """
+ Parse a string with respect to the parser definition. This function is intended as the primary interface to the
+ client code.
+
+ :param instring: The input string to be parsed.
+ :param parse_all: If set, the entire input string must match the grammar.
+ :param parseAll: retained for pre-PEP8 compatibility, will be removed in a future release.
+ :raises ParseException: Raised if ``parse_all`` is set and the input string does not match the whole grammar.
+ :returns: the parsed data as a :class:`ParseResults` object, which may be accessed as a `list`, a `dict`, or
+ an object with attributes if the given parser includes results names.
+
+ If the input string is required to match the entire grammar, ``parse_all`` flag must be set to ``True``. This
+ is also equivalent to ending the grammar with :class:`StringEnd`().
+
+ To report proper column numbers, ``parse_string`` operates on a copy of the input string where all tabs are
+ converted to spaces (8 spaces per tab, as per the default in ``string.expandtabs``). If the input string
+ contains tabs and the grammar uses parse actions that use the ``loc`` argument to index into the string
+ being parsed, one can ensure a consistent view of the input string by doing one of the following:
+
+ - calling ``parse_with_tabs`` on your grammar before calling ``parse_string`` (see :class:`parse_with_tabs`),
+ - define your parse action using the full ``(s,loc,toks)`` signature, and reference the input string using the
+ parse action's ``s`` argument, or
+ - explicitly expand the tabs in your input string before calling ``parse_string``.
+
+ Examples:
+
+ By default, partial matches are OK.
+
+ >>> res = Word('a').parse_string('aaaaabaaa')
+ >>> print(res)
+ ['aaaaa']
+
+ The parsing behavior varies by the inheriting class of this abstract class. Please refer to the children
+ directly to see more examples.
+
+ It raises an exception if parse_all flag is set and instring does not match the whole grammar.
+
+ >>> res = Word('a').parse_string('aaaaabaaa', parse_all=True)
+ Traceback (most recent call last):
+ ...
+ pyparsing.ParseException: Expected end of text, found 'b' (at char 5), (line:1, col:6)
+ """
+ parseAll = parse_all or parseAll
+
+ ParserElement.reset_cache()
+ if not self.streamlined:
+ self.streamline()
+ for e in self.ignoreExprs:
+ e.streamline()
+ if not self.keepTabs:
+ instring = instring.expandtabs()
+ try:
+ loc, tokens = self._parse(instring, 0)
+ if parseAll:
+ loc = self.preParse(instring, loc)
+ se = Empty() + StringEnd()
+ se._parse(instring, loc)
+ except ParseBaseException as exc:
+ if ParserElement.verbose_stacktrace:
+ raise
+ else:
+ # catch and re-raise exception from here, clearing out pyparsing internal stack trace
+ raise exc.with_traceback(None)
+ else:
+ return tokens
+
+ def scan_string(
+ self,
+ instring: str,
+ max_matches: int = _MAX_INT,
+ overlap: bool = False,
+ *,
+ maxMatches: int = _MAX_INT,
+ ) -> Generator[Tuple[ParseResults, int, int], None, None]:
+ """
+ Scan the input string for expression matches. Each match will return the
+ matching tokens, start location, and end location. May be called with optional
+ ``max_matches`` argument, to clip scanning after 'n' matches are found. If
+ ``overlap`` is specified, then overlapping matches will be reported.
+
+ Note that the start and end locations are reported relative to the string
+ being parsed. See :class:`parse_string` for more information on parsing
+ strings with embedded tabs.
+
+ Example::
+
+ source = "sldjf123lsdjjkf345sldkjf879lkjsfd987"
+ print(source)
+ for tokens, start, end in Word(alphas).scan_string(source):
+ print(' '*start + '^'*(end-start))
+ print(' '*start + tokens[0])
+
+ prints::
+
+ sldjf123lsdjjkf345sldkjf879lkjsfd987
+ ^^^^^
+ sldjf
+ ^^^^^^^
+ lsdjjkf
+ ^^^^^^
+ sldkjf
+ ^^^^^^
+ lkjsfd
+ """
+ maxMatches = min(maxMatches, max_matches)
+ if not self.streamlined:
+ self.streamline()
+ for e in self.ignoreExprs:
+ e.streamline()
+
+ if not self.keepTabs:
+ instring = str(instring).expandtabs()
+ instrlen = len(instring)
+ loc = 0
+ preparseFn = self.preParse
+ parseFn = self._parse
+ ParserElement.resetCache()
+ matches = 0
+ try:
+ while loc <= instrlen and matches < maxMatches:
+ try:
+ preloc = preparseFn(instring, loc)
+ nextLoc, tokens = parseFn(instring, preloc, callPreParse=False)
+ except ParseException:
+ loc = preloc + 1
+ else:
+ if nextLoc > loc:
+ matches += 1
+ yield tokens, preloc, nextLoc
+ if overlap:
+ nextloc = preparseFn(instring, loc)
+ if nextloc > loc:
+ loc = nextLoc
+ else:
+ loc += 1
+ else:
+ loc = nextLoc
+ else:
+ loc = preloc + 1
+ except ParseBaseException as exc:
+ if ParserElement.verbose_stacktrace:
+ raise
+ else:
+ # catch and re-raise exception from here, clears out pyparsing internal stack trace
+ raise exc.with_traceback(None)
+
+ def transform_string(self, instring: str) -> str:
+ """
+ Extension to :class:`scan_string`, to modify matching text with modified tokens that may
+ be returned from a parse action. To use ``transform_string``, define a grammar and
+ attach a parse action to it that modifies the returned token list.
+ Invoking ``transform_string()`` on a target string will then scan for matches,
+ and replace the matched text patterns according to the logic in the parse
+ action. ``transform_string()`` returns the resulting transformed string.
+
+ Example::
+
+ wd = Word(alphas)
+ wd.set_parse_action(lambda toks: toks[0].title())
+
+ print(wd.transform_string("now is the winter of our discontent made glorious summer by this sun of york."))
+
+ prints::
+
+ Now Is The Winter Of Our Discontent Made Glorious Summer By This Sun Of York.
+ """
+ out = []
+ lastE = 0
+ # force preservation of <TAB>s, to minimize unwanted transformation of string, and to
+ # keep string locs straight between transform_string and scan_string
+ self.keepTabs = True
+ try:
+ for t, s, e in self.scan_string(instring):
+ out.append(instring[lastE:s])
+ if t:
+ if isinstance(t, ParseResults):
+ out += t.as_list()
+ elif isinstance(t, Iterable) and not isinstance(t, str_type):
+ out += list(t)
+ else:
+ out.append(t)
+ lastE = e
+ out.append(instring[lastE:])
+ out = [o for o in out if o]
+ return "".join(map(str, _flatten(out)))
+ except ParseBaseException as exc:
+ if ParserElement.verbose_stacktrace:
+ raise
+ else:
+ # catch and re-raise exception from here, clears out pyparsing internal stack trace
+ raise exc.with_traceback(None)
+
+ def search_string(
+ self, instring: str, max_matches: int = _MAX_INT, *, maxMatches: int = _MAX_INT
+ ) -> ParseResults:
+ """
+ Another extension to :class:`scan_string`, simplifying the access to the tokens found
+ to match the given parse expression. May be called with optional
+ ``max_matches`` argument, to clip searching after 'n' matches are found.
+
+ Example::
+
+ # a capitalized word starts with an uppercase letter, followed by zero or more lowercase letters
+ cap_word = Word(alphas.upper(), alphas.lower())
+
+ print(cap_word.search_string("More than Iron, more than Lead, more than Gold I need Electricity"))
+
+ # the sum() builtin can be used to merge results into a single ParseResults object
+ print(sum(cap_word.search_string("More than Iron, more than Lead, more than Gold I need Electricity")))
+
+ prints::
+
+ [['More'], ['Iron'], ['Lead'], ['Gold'], ['I'], ['Electricity']]
+ ['More', 'Iron', 'Lead', 'Gold', 'I', 'Electricity']
+ """
+ maxMatches = min(maxMatches, max_matches)
+ try:
+ return ParseResults(
+ [t for t, s, e in self.scan_string(instring, maxMatches)]
+ )
+ except ParseBaseException as exc:
+ if ParserElement.verbose_stacktrace:
+ raise
+ else:
+ # catch and re-raise exception from here, clears out pyparsing internal stack trace
+ raise exc.with_traceback(None)
+
+ def split(
+ self,
+ instring: str,
+ maxsplit: int = _MAX_INT,
+ include_separators: bool = False,
+ *,
+ includeSeparators=False,
+ ) -> Generator[str, None, None]:
+ """
+ Generator method to split a string using the given expression as a separator.
+ May be called with optional ``maxsplit`` argument, to limit the number of splits;
+ and the optional ``include_separators`` argument (default= ``False``), if the separating
+ matching text should be included in the split results.
+
+ Example::
+
+ punc = one_of(list(".,;:/-!?"))
+ print(list(punc.split("This, this?, this sentence, is badly punctuated!")))
+
+ prints::
+
+ ['This', ' this', '', ' this sentence', ' is badly punctuated', '']
+ """
+ includeSeparators = includeSeparators or include_separators
+ last = 0
+ for t, s, e in self.scan_string(instring, max_matches=maxsplit):
+ yield instring[last:s]
+ if includeSeparators:
+ yield t[0]
+ last = e
+ yield instring[last:]
+
+ def __add__(self, other):
+ """
+ Implementation of ``+`` operator - returns :class:`And`. Adding strings to a :class:`ParserElement`
+ converts them to :class:`Literal`s by default.
+
+ Example::
+
+ greet = Word(alphas) + "," + Word(alphas) + "!"
+ hello = "Hello, World!"
+ print(hello, "->", greet.parse_string(hello))
+
+ prints::
+
+ Hello, World! -> ['Hello', ',', 'World', '!']
+
+ ``...`` may be used as a parse expression as a short form of :class:`SkipTo`.
+
+ Literal('start') + ... + Literal('end')
+
+ is equivalent to:
+
+ Literal('start') + SkipTo('end')("_skipped*") + Literal('end')
+
+ Note that the skipped text is returned with '_skipped' as a results name,
+ and to support having multiple skips in the same parser, the value returned is
+ a list of all skipped text.
+ """
+ if other is Ellipsis:
+ return _PendingSkip(self)
+
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return And([self, other])
+
+ def __radd__(self, other):
+ """
+ Implementation of ``+`` operator when left operand is not a :class:`ParserElement`
+ """
+ if other is Ellipsis:
+ return SkipTo(self)("_skipped*") + self
+
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return other + self
+
+ def __sub__(self, other):
+ """
+ Implementation of ``-`` operator, returns :class:`And` with error stop
+ """
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return self + And._ErrorStop() + other
+
+ def __rsub__(self, other):
+ """
+ Implementation of ``-`` operator when left operand is not a :class:`ParserElement`
+ """
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return other - self
+
+ def __mul__(self, other):
+ """
+ Implementation of ``*`` operator, allows use of ``expr * 3`` in place of
+ ``expr + expr + expr``. Expressions may also be multiplied by a 2-integer
+ tuple, similar to ``{min, max}`` multipliers in regular expressions. Tuples
+ may also include ``None`` as in:
+ - ``expr*(n, None)`` or ``expr*(n, )`` is equivalent
+ to ``expr*n + ZeroOrMore(expr)``
+ (read as "at least n instances of ``expr``")
+ - ``expr*(None, n)`` is equivalent to ``expr*(0, n)``
+ (read as "0 to n instances of ``expr``")
+ - ``expr*(None, None)`` is equivalent to ``ZeroOrMore(expr)``
+ - ``expr*(1, None)`` is equivalent to ``OneOrMore(expr)``
+
+ Note that ``expr*(None, n)`` does not raise an exception if
+ more than n exprs exist in the input stream; that is,
+ ``expr*(None, n)`` does not enforce a maximum number of expr
+ occurrences. If this behavior is desired, then write
+ ``expr*(None, n) + ~expr``
+ """
+ if other is Ellipsis:
+ other = (0, None)
+ elif isinstance(other, tuple) and other[:1] == (Ellipsis,):
+ other = ((0,) + other[1:] + (None,))[:2]
+
+ if isinstance(other, int):
+ minElements, optElements = other, 0
+ elif isinstance(other, tuple):
+ other = tuple(o if o is not Ellipsis else None for o in other)
+ other = (other + (None, None))[:2]
+ if other[0] is None:
+ other = (0, other[1])
+ if isinstance(other[0], int) and other[1] is None:
+ if other[0] == 0:
+ return ZeroOrMore(self)
+ if other[0] == 1:
+ return OneOrMore(self)
+ else:
+ return self * other[0] + ZeroOrMore(self)
+ elif isinstance(other[0], int) and isinstance(other[1], int):
+ minElements, optElements = other
+ optElements -= minElements
+ else:
+ raise TypeError(
+ "cannot multiply ParserElement and ({}) objects".format(
+ ",".join(type(item).__name__ for item in other)
+ )
+ )
+ else:
+ raise TypeError(
+ "cannot multiply ParserElement and {} objects".format(
+ type(other).__name__
+ )
+ )
+
+ if minElements < 0:
+ raise ValueError("cannot multiply ParserElement by negative value")
+ if optElements < 0:
+ raise ValueError(
+ "second tuple value must be greater or equal to first tuple value"
+ )
+ if minElements == optElements == 0:
+ return And([])
+
+ if optElements:
+
+ def makeOptionalList(n):
+ if n > 1:
+ return Opt(self + makeOptionalList(n - 1))
+ else:
+ return Opt(self)
+
+ if minElements:
+ if minElements == 1:
+ ret = self + makeOptionalList(optElements)
+ else:
+ ret = And([self] * minElements) + makeOptionalList(optElements)
+ else:
+ ret = makeOptionalList(optElements)
+ else:
+ if minElements == 1:
+ ret = self
+ else:
+ ret = And([self] * minElements)
+ return ret
+
+ def __rmul__(self, other):
+ return self.__mul__(other)
+
+ def __or__(self, other):
+ """
+ Implementation of ``|`` operator - returns :class:`MatchFirst`
+ """
+ if other is Ellipsis:
+ return _PendingSkip(self, must_skip=True)
+
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return MatchFirst([self, other])
+
+ def __ror__(self, other):
+ """
+ Implementation of ``|`` operator when left operand is not a :class:`ParserElement`
+ """
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return other | self
+
+ def __xor__(self, other):
+ """
+ Implementation of ``^`` operator - returns :class:`Or`
+ """
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return Or([self, other])
+
+ def __rxor__(self, other):
+ """
+ Implementation of ``^`` operator when left operand is not a :class:`ParserElement`
+ """
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return other ^ self
+
+ def __and__(self, other):
+ """
+ Implementation of ``&`` operator - returns :class:`Each`
+ """
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return Each([self, other])
+
+ def __rand__(self, other):
+ """
+ Implementation of ``&`` operator when left operand is not a :class:`ParserElement`
+ """
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ if not isinstance(other, ParserElement):
+ raise TypeError(
+ "Cannot combine element of type {} with ParserElement".format(
+ type(other).__name__
+ )
+ )
+ return other & self
+
+ def __invert__(self):
+ """
+ Implementation of ``~`` operator - returns :class:`NotAny`
+ """
+ return NotAny(self)
+
+ # disable __iter__ to override legacy use of sequential access to __getitem__ to
+ # iterate over a sequence
+ __iter__ = None
+
+ def __getitem__(self, key):
+ """
+ use ``[]`` indexing notation as a short form for expression repetition:
+
+ - ``expr[n]`` is equivalent to ``expr*n``
+ - ``expr[m, n]`` is equivalent to ``expr*(m, n)``
+ - ``expr[n, ...]`` or ``expr[n,]`` is equivalent
+ to ``expr*n + ZeroOrMore(expr)``
+ (read as "at least n instances of ``expr``")
+ - ``expr[..., n]`` is equivalent to ``expr*(0, n)``
+ (read as "0 to n instances of ``expr``")
+ - ``expr[...]`` and ``expr[0, ...]`` are equivalent to ``ZeroOrMore(expr)``
+ - ``expr[1, ...]`` is equivalent to ``OneOrMore(expr)``
+
+ ``None`` may be used in place of ``...``.
+
+ Note that ``expr[..., n]`` and ``expr[m, n]``do not raise an exception
+ if more than ``n`` ``expr``s exist in the input stream. If this behavior is
+ desired, then write ``expr[..., n] + ~expr``.
+ """
+
+ # convert single arg keys to tuples
+ try:
+ if isinstance(key, str_type):
+ key = (key,)
+ iter(key)
+ except TypeError:
+ key = (key, key)
+
+ if len(key) > 2:
+ raise TypeError(
+ "only 1 or 2 index arguments supported ({}{})".format(
+ key[:5], "... [{}]".format(len(key)) if len(key) > 5 else ""
+ )
+ )
+
+ # clip to 2 elements
+ ret = self * tuple(key[:2])
+ return ret
+
+ def __call__(self, name: str = None):
+ """
+ Shortcut for :class:`set_results_name`, with ``list_all_matches=False``.
+
+ If ``name`` is given with a trailing ``'*'`` character, then ``list_all_matches`` will be
+ passed as ``True``.
+
+ If ``name` is omitted, same as calling :class:`copy`.
+
+ Example::
+
+ # these are equivalent
+ userdata = Word(alphas).set_results_name("name") + Word(nums + "-").set_results_name("socsecno")
+ userdata = Word(alphas)("name") + Word(nums + "-")("socsecno")
+ """
+ if name is not None:
+ return self._setResultsName(name)
+ else:
+ return self.copy()
+
+ def suppress(self) -> "ParserElement":
+ """
+ Suppresses the output of this :class:`ParserElement`; useful to keep punctuation from
+ cluttering up returned output.
+ """
+ return Suppress(self)
+
+ def ignore_whitespace(self, recursive: bool = True) -> "ParserElement":
+ """
+ Enables the skipping of whitespace before matching the characters in the
+ :class:`ParserElement`'s defined pattern.
+
+ :param recursive: If ``True`` (the default), also enable whitespace skipping in child elements (if any)
+ """
+ self.skipWhitespace = True
+ return self
+
+ def leave_whitespace(self, recursive: bool = True) -> "ParserElement":
+ """
+ Disables the skipping of whitespace before matching the characters in the
+ :class:`ParserElement`'s defined pattern. This is normally only used internally by
+ the pyparsing module, but may be needed in some whitespace-sensitive grammars.
+
+ :param recursive: If true (the default), also disable whitespace skipping in child elements (if any)
+ """
+ self.skipWhitespace = False
+ return self
+
+ def set_whitespace_chars(
+ self, chars: Union[Set[str], str], copy_defaults: bool = False
+ ) -> "ParserElement":
+ """
+ Overrides the default whitespace chars
+ """
+ self.skipWhitespace = True
+ self.whiteChars = set(chars)
+ self.copyDefaultWhiteChars = copy_defaults
+ return self
+
+ def parse_with_tabs(self) -> "ParserElement":
+ """
+ Overrides default behavior to expand ``<TAB>`` s to spaces before parsing the input string.
+ Must be called before ``parse_string`` when the input grammar contains elements that
+ match ``<TAB>`` characters.
+ """
+ self.keepTabs = True
+ return self
+
+ def ignore(self, other: "ParserElement") -> "ParserElement":
+ """
+ Define expression to be ignored (e.g., comments) while doing pattern
+ matching; may be called repeatedly, to define multiple comment or other
+ ignorable patterns.
+
+ Example::
+
+ patt = OneOrMore(Word(alphas))
+ patt.parse_string('ablaj /* comment */ lskjd')
+ # -> ['ablaj']
+
+ patt.ignore(c_style_comment)
+ patt.parse_string('ablaj /* comment */ lskjd')
+ # -> ['ablaj', 'lskjd']
+ """
+ import typing
+ if isinstance(other, str_type):
+ other = Suppress(other)
+
+ if isinstance(other, Suppress):
+ if other not in self.ignoreExprs:
+ self.ignoreExprs.append(other)
+ else:
+ self.ignoreExprs.append(Suppress(other.copy()))
+ return self
+
+ def set_debug_actions(
+ self,
+ start_action: DebugStartAction,
+ success_action: DebugSuccessAction,
+ exception_action: DebugExceptionAction,
+ ) -> "ParserElement":
+ """
+ Customize display of debugging messages while doing pattern matching:
+
+ - ``start_action`` - method to be called when an expression is about to be parsed;
+ should have the signature ``fn(input_string: str, location: int, expression: ParserElement, cache_hit: bool)``
+
+ - ``success_action`` - method to be called when an expression has successfully parsed;
+ should have the signature ``fn(input_string: str, start_location: int, end_location: int, expression: ParserELement, parsed_tokens: ParseResults, cache_hit: bool)``
+
+ - ``exception_action`` - method to be called when expression fails to parse;
+ should have the signature ``fn(input_string: str, location: int, expression: ParserElement, exception: Exception, cache_hit: bool)``
+ """
+ self.debugActions = (
+ start_action or _default_start_debug_action,
+ success_action or _default_success_debug_action,
+ exception_action or _default_exception_debug_action,
+ )
+ self.debug = True
+ return self
+
+ def set_debug(self, flag=True) -> "ParserElement":
+ """
+ Enable display of debugging messages while doing pattern matching.
+ Set ``flag`` to ``True`` to enable, ``False`` to disable.
+
+ Example::
+
+ wd = Word(alphas).set_name("alphaword")
+ integer = Word(nums).set_name("numword")
+ term = wd | integer
+
+ # turn on debugging for wd
+ wd.set_debug()
+
+ OneOrMore(term).parse_string("abc 123 xyz 890")
+
+ prints::
+
+ Match alphaword at loc 0(1,1)
+ Matched alphaword -> ['abc']
+ Match alphaword at loc 3(1,4)
+ Exception raised:Expected alphaword (at char 4), (line:1, col:5)
+ Match alphaword at loc 7(1,8)
+ Matched alphaword -> ['xyz']
+ Match alphaword at loc 11(1,12)
+ Exception raised:Expected alphaword (at char 12), (line:1, col:13)
+ Match alphaword at loc 15(1,16)
+ Exception raised:Expected alphaword (at char 15), (line:1, col:16)
+
+ The output shown is that produced by the default debug actions - custom debug actions can be
+ specified using :class:`set_debug_actions`. Prior to attempting
+ to match the ``wd`` expression, the debugging message ``"Match <exprname> at loc <n>(<line>,<col>)"``
+ is shown. Then if the parse succeeds, a ``"Matched"`` message is shown, or an ``"Exception raised"``
+ message is shown. Also note the use of :class:`set_name` to assign a human-readable name to the expression,
+ which makes debugging and exception messages easier to understand - for instance, the default
+ name created for the :class:`Word` expression without calling ``set_name`` is ``"W:(A-Za-z)"``.
+ """
+ if flag:
+ self.set_debug_actions(
+ _default_start_debug_action,
+ _default_success_debug_action,
+ _default_exception_debug_action,
+ )
+ else:
+ self.debug = False
+ return self
+
+ @property
+ def default_name(self) -> str:
+ if self._defaultName is None:
+ self._defaultName = self._generateDefaultName()
+ return self._defaultName
+
+ @abstractmethod
+ def _generateDefaultName(self):
+ """
+ Child classes must define this method, which defines how the ``default_name`` is set.
+ """
+
+ def set_name(self, name: str) -> "ParserElement":
+ """
+ Define name for this expression, makes debugging and exception messages clearer.
+ Example::
+ Word(nums).parse_string("ABC") # -> Exception: Expected W:(0-9) (at char 0), (line:1, col:1)
+ Word(nums).set_name("integer").parse_string("ABC") # -> Exception: Expected integer (at char 0), (line:1, col:1)
+ """
+ self.customName = name
+ self.errmsg = "Expected " + self.name
+ if __diag__.enable_debug_on_named_expressions:
+ self.set_debug()
+ return self
+
+ @property
+ def name(self) -> str:
+ # This will use a user-defined name if available, but otherwise defaults back to the auto-generated name
+ return self.customName if self.customName is not None else self.default_name
+
+ def __str__(self) -> str:
+ return self.name
+
+ def __repr__(self) -> str:
+ return str(self)
+
+ def streamline(self) -> "ParserElement":
+ self.streamlined = True
+ self._defaultName = None
+ return self
+
+ def recurse(self):
+ return []
+
+ def _checkRecursion(self, parseElementList):
+ subRecCheckList = parseElementList[:] + [self]
+ for e in self.recurse():
+ e._checkRecursion(subRecCheckList)
+
+ def validate(self, validateTrace=None):
+ """
+ Check defined expressions for valid structure, check for infinite recursive definitions.
+ """
+ self._checkRecursion([])
+
+ def parse_file(
+ self,
+ file_or_filename: Union[str, Path, TextIO],
+ encoding: str = "utf-8",
+ parse_all: bool = False,
+ *,
+ parseAll: bool = False,
+ ) -> ParseResults:
+ """
+ Execute the parse expression on the given file or filename.
+ If a filename is specified (instead of a file object),
+ the entire file is opened, read, and closed before parsing.
+ """
+ parseAll = parseAll or parse_all
+ try:
+ file_contents = file_or_filename.read()
+ except AttributeError:
+ with open(file_or_filename, "r", encoding=encoding) as f:
+ file_contents = f.read()
+ try:
+ return self.parse_string(file_contents, parseAll)
+ except ParseBaseException as exc:
+ if ParserElement.verbose_stacktrace:
+ raise
+ else:
+ # catch and re-raise exception from here, clears out pyparsing internal stack trace
+ raise exc.with_traceback(None)
+
+ def __eq__(self, other):
+ if self is other:
+ return True
+ elif isinstance(other, str_type):
+ return self.matches(other, parse_all=True)
+ elif isinstance(other, ParserElement):
+ return vars(self) == vars(other)
+ return False
+
+ def __hash__(self):
+ return id(self)
+
+ def matches(
+ self, test_string: str, parse_all: bool = True, *, parseAll: bool = True
+ ) -> bool:
+ """
+ Method for quick testing of a parser against a test string. Good for simple
+ inline microtests of sub expressions while building up larger parser.
+
+ Parameters:
+ - ``test_string`` - to test against this expression for a match
+ - ``parse_all`` - (default= ``True``) - flag to pass to :class:`parse_string` when running tests
+
+ Example::
+
+ expr = Word(nums)
+ assert expr.matches("100")
+ """
+ parseAll = parseAll and parse_all
+ try:
+ self.parse_string(str(test_string), parse_all=parseAll)
+ return True
+ except ParseBaseException:
+ return False
+
+ def run_tests(
+ self,
+ tests: Union[str, List[str]],
+ parse_all: bool = True,
+ comment: OptionalType[Union["ParserElement", str]] = "#",
+ full_dump: bool = True,
+ print_results: bool = True,
+ failure_tests: bool = False,
+ post_parse: Callable[[str, ParseResults], str] = None,
+ file: OptionalType[TextIO] = None,
+ *,
+ parseAll: bool = True,
+ fullDump: bool = True,
+ printResults: bool = True,
+ failureTests: bool = False,
+ postParse: Callable[[str, ParseResults], str] = None,
+ ):
+ """
+ Execute the parse expression on a series of test strings, showing each
+ test, the parsed results or where the parse failed. Quick and easy way to
+ run a parse expression against a list of sample strings.
+
+ Parameters:
+ - ``tests`` - a list of separate test strings, or a multiline string of test strings
+ - ``parse_all`` - (default= ``True``) - flag to pass to :class:`parse_string` when running tests
+ - ``comment`` - (default= ``'#'``) - expression for indicating embedded comments in the test
+ string; pass None to disable comment filtering
+ - ``full_dump`` - (default= ``True``) - dump results as list followed by results names in nested outline;
+ if False, only dump nested list
+ - ``print_results`` - (default= ``True``) prints test output to stdout
+ - ``failure_tests`` - (default= ``False``) indicates if these tests are expected to fail parsing
+ - ``post_parse`` - (default= ``None``) optional callback for successful parse results; called as
+ `fn(test_string, parse_results)` and returns a string to be added to the test output
+ - ``file`` - (default= ``None``) optional file-like object to which test output will be written;
+ if None, will default to ``sys.stdout``
+
+ Returns: a (success, results) tuple, where success indicates that all tests succeeded
+ (or failed if ``failure_tests`` is True), and the results contain a list of lines of each
+ test's output
+
+ Example::
+
+ number_expr = pyparsing_common.number.copy()
+
+ result = number_expr.run_tests('''
+ # unsigned integer
+ 100
+ # negative integer
+ -100
+ # float with scientific notation
+ 6.02e23
+ # integer with scientific notation
+ 1e-12
+ ''')
+ print("Success" if result[0] else "Failed!")
+
+ result = number_expr.run_tests('''
+ # stray character
+ 100Z
+ # missing leading digit before '.'
+ -.100
+ # too many '.'
+ 3.14.159
+ ''', failure_tests=True)
+ print("Success" if result[0] else "Failed!")
+
+ prints::
+
+ # unsigned integer
+ 100
+ [100]
+
+ # negative integer
+ -100
+ [-100]
+
+ # float with scientific notation
+ 6.02e23
+ [6.02e+23]
+
+ # integer with scientific notation
+ 1e-12
+ [1e-12]
+
+ Success
+
+ # stray character
+ 100Z
+ ^
+ FAIL: Expected end of text (at char 3), (line:1, col:4)
+
+ # missing leading digit before '.'
+ -.100
+ ^
+ FAIL: Expected {real number with scientific notation | real number | signed integer} (at char 0), (line:1, col:1)
+
+ # too many '.'
+ 3.14.159
+ ^
+ FAIL: Expected end of text (at char 4), (line:1, col:5)
+
+ Success
+
+ Each test string must be on a single line. If you want to test a string that spans multiple
+ lines, create a test like this::
+
+ expr.run_tests(r"this is a test\\n of strings that spans \\n 3 lines")
+
+ (Note that this is a raw string literal, you must include the leading ``'r'``.)
+ """
+ parseAll = parseAll and parse_all
+ fullDump = fullDump and full_dump
+ printResults = printResults and print_results
+ failureTests = failureTests or failure_tests
+ postParse = postParse or post_parse
+ if isinstance(tests, str_type):
+ tests = list(map(type(tests).strip, tests.rstrip().splitlines()))
+ if isinstance(comment, str_type):
+ comment = Literal(comment)
+ if file is None:
+ file = sys.stdout
+ print_ = file.write
+
+ result: Union[ParseResults, Exception]
+ allResults = []
+ comments = []
+ success = True
+ NL = Literal(r"\n").add_parse_action(replace_with("\n")).ignore(quoted_string)
+ BOM = "\ufeff"
+ for t in tests:
+ if comment is not None and comment.matches(t, False) or comments and not t:
+ comments.append(t)
+ continue
+ if not t:
+ continue
+ out = ["\n" + "\n".join(comments) if comments else "", t]
+ comments = []
+ try:
+ # convert newline marks to actual newlines, and strip leading BOM if present
+ t = NL.transform_string(t.lstrip(BOM))
+ result = self.parse_string(t, parse_all=parseAll)
+ except ParseBaseException as pe:
+ fatal = "(FATAL)" if isinstance(pe, ParseFatalException) else ""
+ if "\n" in t:
+ out.append(line(pe.loc, t))
+ out.append(" " * (col(pe.loc, t) - 1) + "^" + fatal)
+ else:
+ out.append(" " * pe.loc + "^" + fatal)
+ out.append("FAIL: " + str(pe))
+ success = success and failureTests
+ result = pe
+ except Exception as exc:
+ out.append("FAIL-EXCEPTION: " + str(exc))
+ success = success and failureTests
+ result = exc
+ else:
+ success = success and not failureTests
+ if postParse is not None:
+ try:
+ pp_value = postParse(t, result)
+ if pp_value is not None:
+ if isinstance(pp_value, ParseResults):
+ out.append(pp_value.dump())
+ else:
+ out.append(str(pp_value))
+ else:
+ out.append(result.dump())
+ except Exception as e:
+ out.append(result.dump(full=fullDump))
+ out.append(
+ "{} failed: {}: {}".format(
+ postParse.__name__, type(e).__name__, e
+ )
+ )
+ else:
+ out.append(result.dump(full=fullDump))
+ out.append("")
+
+ if printResults:
+ print_("\n".join(out))
+
+ allResults.append((t, result))
+
+ return success, allResults
+
+ def create_diagram(
+ self,
+ output_html: Union[TextIO, Path, str],
+ vertical: int = 3,
+ show_results_names: bool = False,
+ **kwargs,
+ ) -> None:
+ """
+ Create a railroad diagram for the parser.
+
+ Parameters:
+ - output_html (str or file-like object) - output target for generated
+ diagram HTML
+ - vertical (int) - threshold for formatting multiple alternatives vertically
+ instead of horizontally (default=3)
+ - show_results_names - bool flag whether diagram should show annotations for
+ defined results names
+
+ Additional diagram-formatting keyword arguments can also be included;
+ see railroad.Diagram class.
+ """
+
+ try:
+ from .diagram import to_railroad, railroad_to_html
+ except ImportError as ie:
+ raise Exception(
+ "must ``pip install pyparsing[diagrams]`` to generate parser railroad diagrams"
+ ) from ie
+
+ self.streamline()
+
+ railroad = to_railroad(
+ self,
+ vertical=vertical,
+ show_results_names=show_results_names,
+ diagram_kwargs=kwargs,
+ )
+ if isinstance(output_html, (str, Path)):
+ with open(output_html, "w", encoding="utf-8") as diag_file:
+ diag_file.write(railroad_to_html(railroad))
+ else:
+ # we were passed a file-like object, just write to it
+ output_html.write(railroad_to_html(railroad))
+
+ setDefaultWhitespaceChars = set_default_whitespace_chars
+ inlineLiteralsUsing = inline_literals_using
+ setResultsName = set_results_name
+ setBreak = set_break
+ setParseAction = set_parse_action
+ addParseAction = add_parse_action
+ addCondition = add_condition
+ setFailAction = set_fail_action
+ tryParse = try_parse
+ canParseNext = can_parse_next
+ resetCache = reset_cache
+ enableLeftRecursion = enable_left_recursion
+ enablePackrat = enable_packrat
+ parseString = parse_string
+ scanString = scan_string
+ searchString = search_string
+ transformString = transform_string
+ setWhitespaceChars = set_whitespace_chars
+ parseWithTabs = parse_with_tabs
+ setDebugActions = set_debug_actions
+ setDebug = set_debug
+ defaultName = default_name
+ setName = set_name
+ parseFile = parse_file
+ runTests = run_tests
+ ignoreWhitespace = ignore_whitespace
+ leaveWhitespace = leave_whitespace
+
+
+class _PendingSkip(ParserElement):
+ # internal placeholder class to hold a place were '...' is added to a parser element,
+ # once another ParserElement is added, this placeholder will be replaced with a SkipTo
+ def __init__(self, expr: ParserElement, must_skip: bool = False):
+ super().__init__()
+ self.anchor = expr
+ self.must_skip = must_skip
+
+ def _generateDefaultName(self):
+ return str(self.anchor + Empty()).replace("Empty", "...")
+
+ def __add__(self, other):
+ skipper = SkipTo(other).set_name("...")("_skipped*")
+ if self.must_skip:
+
+ def must_skip(t):
+ if not t._skipped or t._skipped.as_list() == [""]:
+ del t[0]
+ t.pop("_skipped", None)
+
+ def show_skip(t):
+ if t._skipped.as_list()[-1:] == [""]:
+ t.pop("_skipped")
+ t["_skipped"] = "missing <" + repr(self.anchor) + ">"
+
+ return (
+ self.anchor + skipper().add_parse_action(must_skip)
+ | skipper().add_parse_action(show_skip)
+ ) + other
+
+ return self.anchor + skipper + other
+
+ def __repr__(self):
+ return self.defaultName
+
+ def parseImpl(self, *args):
+ raise Exception(
+ "use of `...` expression without following SkipTo target expression"
+ )
+
+
+class Token(ParserElement):
+ """Abstract :class:`ParserElement` subclass, for defining atomic
+ matching patterns.
+ """
+
+ def __init__(self):
+ super().__init__(savelist=False)
+
+ def _generateDefaultName(self):
+ return type(self).__name__
+
+
+class Empty(Token):
+ """
+ An empty token, will always match.
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.mayReturnEmpty = True
+ self.mayIndexError = False
+
+
+class NoMatch(Token):
+ """
+ A token that will never match.
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.mayReturnEmpty = True
+ self.mayIndexError = False
+ self.errmsg = "Unmatchable token"
+
+ def parseImpl(self, instring, loc, doActions=True):
+ raise ParseException(instring, loc, self.errmsg, self)
+
+
+class Literal(Token):
+ """
+ Token to exactly match a specified string.
+
+ Example::
+
+ Literal('blah').parse_string('blah') # -> ['blah']
+ Literal('blah').parse_string('blahfooblah') # -> ['blah']
+ Literal('blah').parse_string('bla') # -> Exception: Expected "blah"
+
+ For case-insensitive matching, use :class:`CaselessLiteral`.
+
+ For keyword matching (force word break before and after the matched string),
+ use :class:`Keyword` or :class:`CaselessKeyword`.
+ """
+
+ def __init__(self, match_string: str = "", *, matchString: str = ""):
+ super().__init__()
+ match_string = matchString or match_string
+ self.match = match_string
+ self.matchLen = len(match_string)
+ try:
+ self.firstMatchChar = match_string[0]
+ except IndexError:
+ raise ValueError("null string passed to Literal; use Empty() instead")
+ self.errmsg = "Expected " + self.name
+ self.mayReturnEmpty = False
+ self.mayIndexError = False
+
+ # Performance tuning: modify __class__ to select
+ # a parseImpl optimized for single-character check
+ if self.matchLen == 1 and type(self) is Literal:
+ self.__class__ = _SingleCharLiteral
+
+ def _generateDefaultName(self):
+ return repr(self.match)
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if instring[loc] == self.firstMatchChar and instring.startswith(
+ self.match, loc
+ ):
+ return loc + self.matchLen, self.match
+ raise ParseException(instring, loc, self.errmsg, self)
+
+
+class _SingleCharLiteral(Literal):
+ def parseImpl(self, instring, loc, doActions=True):
+ if instring[loc] == self.firstMatchChar:
+ return loc + 1, self.match
+ raise ParseException(instring, loc, self.errmsg, self)
+
+
+ParserElement._literalStringClass = Literal
+
+
+class Keyword(Token):
+ """
+ Token to exactly match a specified string as a keyword, that is,
+ it must be immediately followed by a non-keyword character. Compare
+ with :class:`Literal`:
+
+ - ``Literal("if")`` will match the leading ``'if'`` in
+ ``'ifAndOnlyIf'``.
+ - ``Keyword("if")`` will not; it will only match the leading
+ ``'if'`` in ``'if x=1'``, or ``'if(y==2)'``
+
+ Accepts two optional constructor arguments in addition to the
+ keyword string:
+
+ - ``identChars`` is a string of characters that would be valid
+ identifier characters, defaulting to all alphanumerics + "_" and
+ "$"
+ - ``caseless`` allows case-insensitive matching, default is ``False``.
+
+ Example::
+
+ Keyword("start").parse_string("start") # -> ['start']
+ Keyword("start").parse_string("starting") # -> Exception
+
+ For case-insensitive matching, use :class:`CaselessKeyword`.
+ """
+
+ DEFAULT_KEYWORD_CHARS = alphanums + "_$"
+
+ def __init__(
+ self,
+ match_string: str = "",
+ ident_chars: OptionalType[str] = None,
+ caseless: bool = False,
+ *,
+ matchString: str = "",
+ identChars: OptionalType[str] = None,
+ ):
+ super().__init__()
+ identChars = identChars or ident_chars
+ if identChars is None:
+ identChars = Keyword.DEFAULT_KEYWORD_CHARS
+ match_string = matchString or match_string
+ self.match = match_string
+ self.matchLen = len(match_string)
+ try:
+ self.firstMatchChar = match_string[0]
+ except IndexError:
+ raise ValueError("null string passed to Keyword; use Empty() instead")
+ self.errmsg = "Expected {} {}".format(type(self).__name__, self.name)
+ self.mayReturnEmpty = False
+ self.mayIndexError = False
+ self.caseless = caseless
+ if caseless:
+ self.caselessmatch = match_string.upper()
+ identChars = identChars.upper()
+ self.identChars = set(identChars)
+
+ def _generateDefaultName(self):
+ return repr(self.match)
+
+ def parseImpl(self, instring, loc, doActions=True):
+ errmsg = self.errmsg
+ errloc = loc
+ if self.caseless:
+ if instring[loc : loc + self.matchLen].upper() == self.caselessmatch:
+ if loc == 0 or instring[loc - 1].upper() not in self.identChars:
+ if (
+ loc >= len(instring) - self.matchLen
+ or instring[loc + self.matchLen].upper() not in self.identChars
+ ):
+ return loc + self.matchLen, self.match
+ else:
+ # followed by keyword char
+ errmsg += ", was immediately followed by keyword character"
+ errloc = loc + self.matchLen
+ else:
+ # preceded by keyword char
+ errmsg += ", keyword was immediately preceded by keyword character"
+ errloc = loc - 1
+ # else no match just raise plain exception
+
+ else:
+ if (
+ instring[loc] == self.firstMatchChar
+ and self.matchLen == 1
+ or instring.startswith(self.match, loc)
+ ):
+ if loc == 0 or instring[loc - 1] not in self.identChars:
+ if (
+ loc >= len(instring) - self.matchLen
+ or instring[loc + self.matchLen] not in self.identChars
+ ):
+ return loc + self.matchLen, self.match
+ else:
+ # followed by keyword char
+ errmsg += (
+ ", keyword was immediately followed by keyword character"
+ )
+ errloc = loc + self.matchLen
+ else:
+ # preceded by keyword char
+ errmsg += ", keyword was immediately preceded by keyword character"
+ errloc = loc - 1
+ # else no match just raise plain exception
+
+ raise ParseException(instring, errloc, errmsg, self)
+
+ @staticmethod
+ def set_default_keyword_chars(chars):
+ """
+ Overrides the default characters used by :class:`Keyword` expressions.
+ """
+ Keyword.DEFAULT_KEYWORD_CHARS = chars
+
+ setDefaultKeywordChars = set_default_keyword_chars
+
+
+class CaselessLiteral(Literal):
+ """
+ Token to match a specified string, ignoring case of letters.
+ Note: the matched results will always be in the case of the given
+ match string, NOT the case of the input text.
+
+ Example::
+
+ OneOrMore(CaselessLiteral("CMD")).parse_string("cmd CMD Cmd10")
+ # -> ['CMD', 'CMD', 'CMD']
+
+ (Contrast with example for :class:`CaselessKeyword`.)
+ """
+
+ def __init__(self, match_string: str = "", *, matchString: str = ""):
+ match_string = matchString or match_string
+ super().__init__(match_string.upper())
+ # Preserve the defining literal.
+ self.returnString = match_string
+ self.errmsg = "Expected " + self.name
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if instring[loc : loc + self.matchLen].upper() == self.match:
+ return loc + self.matchLen, self.returnString
+ raise ParseException(instring, loc, self.errmsg, self)
+
+
+class CaselessKeyword(Keyword):
+ """
+ Caseless version of :class:`Keyword`.
+
+ Example::
+
+ OneOrMore(CaselessKeyword("CMD")).parse_string("cmd CMD Cmd10")
+ # -> ['CMD', 'CMD']
+
+ (Contrast with example for :class:`CaselessLiteral`.)
+ """
+
+ def __init__(
+ self,
+ match_string: str = "",
+ ident_chars: OptionalType[str] = None,
+ *,
+ matchString: str = "",
+ identChars: OptionalType[str] = None,
+ ):
+ identChars = identChars or ident_chars
+ match_string = matchString or match_string
+ super().__init__(match_string, identChars, caseless=True)
+
+
+class CloseMatch(Token):
+ """A variation on :class:`Literal` which matches "close" matches,
+ that is, strings with at most 'n' mismatching characters.
+ :class:`CloseMatch` takes parameters:
+
+ - ``match_string`` - string to be matched
+ - ``caseless`` - a boolean indicating whether to ignore casing when comparing characters
+ - ``max_mismatches`` - (``default=1``) maximum number of
+ mismatches allowed to count as a match
+
+ The results from a successful parse will contain the matched text
+ from the input string and the following named results:
+
+ - ``mismatches`` - a list of the positions within the
+ match_string where mismatches were found
+ - ``original`` - the original match_string used to compare
+ against the input string
+
+ If ``mismatches`` is an empty list, then the match was an exact
+ match.
+
+ Example::
+
+ patt = CloseMatch("ATCATCGAATGGA")
+ patt.parse_string("ATCATCGAAXGGA") # -> (['ATCATCGAAXGGA'], {'mismatches': [[9]], 'original': ['ATCATCGAATGGA']})
+ patt.parse_string("ATCAXCGAAXGGA") # -> Exception: Expected 'ATCATCGAATGGA' (with up to 1 mismatches) (at char 0), (line:1, col:1)
+
+ # exact match
+ patt.parse_string("ATCATCGAATGGA") # -> (['ATCATCGAATGGA'], {'mismatches': [[]], 'original': ['ATCATCGAATGGA']})
+
+ # close match allowing up to 2 mismatches
+ patt = CloseMatch("ATCATCGAATGGA", max_mismatches=2)
+ patt.parse_string("ATCAXCGAAXGGA") # -> (['ATCAXCGAAXGGA'], {'mismatches': [[4, 9]], 'original': ['ATCATCGAATGGA']})
+ """
+
+ def __init__(
+ self,
+ match_string: str,
+ max_mismatches: int = None,
+ *,
+ maxMismatches: int = 1,
+ caseless=False,
+ ):
+ maxMismatches = max_mismatches if max_mismatches is not None else maxMismatches
+ super().__init__()
+ self.match_string = match_string
+ self.maxMismatches = maxMismatches
+ self.errmsg = "Expected {!r} (with up to {} mismatches)".format(
+ self.match_string, self.maxMismatches
+ )
+ self.caseless = caseless
+ self.mayIndexError = False
+ self.mayReturnEmpty = False
+
+ def _generateDefaultName(self):
+ return "{}:{!r}".format(type(self).__name__, self.match_string)
+
+ def parseImpl(self, instring, loc, doActions=True):
+ start = loc
+ instrlen = len(instring)
+ maxloc = start + len(self.match_string)
+
+ if maxloc <= instrlen:
+ match_string = self.match_string
+ match_stringloc = 0
+ mismatches = []
+ maxMismatches = self.maxMismatches
+
+ for match_stringloc, s_m in enumerate(
+ zip(instring[loc:maxloc], match_string)
+ ):
+ src, mat = s_m
+ if self.caseless:
+ src, mat = src.lower(), mat.lower()
+
+ if src != mat:
+ mismatches.append(match_stringloc)
+ if len(mismatches) > maxMismatches:
+ break
+ else:
+ loc = start + match_stringloc + 1
+ results = ParseResults([instring[start:loc]])
+ results["original"] = match_string
+ results["mismatches"] = mismatches
+ return loc, results
+
+ raise ParseException(instring, loc, self.errmsg, self)
+
+
+class Word(Token):
+ """Token for matching words composed of allowed character sets.
+ Parameters:
+ - ``init_chars`` - string of all characters that should be used to
+ match as a word; "ABC" will match "AAA", "ABAB", "CBAC", etc.;
+ if ``body_chars`` is also specified, then this is the string of
+ initial characters
+ - ``body_chars`` - string of characters that
+ can be used for matching after a matched initial character as
+ given in ``init_chars``; if omitted, same as the initial characters
+ (default=``None``)
+ - ``min`` - minimum number of characters to match (default=1)
+ - ``max`` - maximum number of characters to match (default=0)
+ - ``exact`` - exact number of characters to match (default=0)
+ - ``as_keyword`` - match as a keyword (default=``False``)
+ - ``exclude_chars`` - characters that might be
+ found in the input ``body_chars`` string but which should not be
+ accepted for matching ;useful to define a word of all
+ printables except for one or two characters, for instance
+ (default=``None``)
+
+ :class:`srange` is useful for defining custom character set strings
+ for defining :class:`Word` expressions, using range notation from
+ regular expression character sets.
+
+ A common mistake is to use :class:`Word` to match a specific literal
+ string, as in ``Word("Address")``. Remember that :class:`Word`
+ uses the string argument to define *sets* of matchable characters.
+ This expression would match "Add", "AAA", "dAred", or any other word
+ made up of the characters 'A', 'd', 'r', 'e', and 's'. To match an
+ exact literal string, use :class:`Literal` or :class:`Keyword`.
+
+ pyparsing includes helper strings for building Words:
+
+ - :class:`alphas`
+ - :class:`nums`
+ - :class:`alphanums`
+ - :class:`hexnums`
+ - :class:`alphas8bit` (alphabetic characters in ASCII range 128-255
+ - accented, tilded, umlauted, etc.)
+ - :class:`punc8bit` (non-alphabetic characters in ASCII range
+ 128-255 - currency, symbols, superscripts, diacriticals, etc.)
+ - :class:`printables` (any non-whitespace character)
+
+ ``alphas``, ``nums``, and ``printables`` are also defined in several
+ Unicode sets - see :class:`pyparsing_unicode``.
+
+ Example::
+
+ # a word composed of digits
+ integer = Word(nums) # equivalent to Word("0123456789") or Word(srange("0-9"))
+
+ # a word with a leading capital, and zero or more lowercase
+ capital_word = Word(alphas.upper(), alphas.lower())
+
+ # hostnames are alphanumeric, with leading alpha, and '-'
+ hostname = Word(alphas, alphanums + '-')
+
+ # roman numeral (not a strict parser, accepts invalid mix of characters)
+ roman = Word("IVXLCDM")
+
+ # any string of non-whitespace characters, except for ','
+ csv_value = Word(printables, exclude_chars=",")
+ """
+
+ def __init__(
+ self,
+ init_chars: str = "",
+ body_chars: OptionalType[str] = None,
+ min: int = 1,
+ max: int = 0,
+ exact: int = 0,
+ as_keyword: bool = False,
+ exclude_chars: OptionalType[str] = None,
+ *,
+ initChars: OptionalType[str] = None,
+ bodyChars: OptionalType[str] = None,
+ asKeyword: bool = False,
+ excludeChars: OptionalType[str] = None,
+ ):
+ initChars = initChars or init_chars
+ bodyChars = bodyChars or body_chars
+ asKeyword = asKeyword or as_keyword
+ excludeChars = excludeChars or exclude_chars
+ super().__init__()
+ if not initChars:
+ raise ValueError(
+ "invalid {}, initChars cannot be empty string".format(
+ type(self).__name__
+ )
+ )
+
+ initChars = set(initChars)
+ self.initChars = initChars
+ if excludeChars:
+ excludeChars = set(excludeChars)
+ initChars -= excludeChars
+ if bodyChars:
+ bodyChars = set(bodyChars) - excludeChars
+ self.initCharsOrig = "".join(sorted(initChars))
+
+ if bodyChars:
+ self.bodyCharsOrig = "".join(sorted(bodyChars))
+ self.bodyChars = set(bodyChars)
+ else:
+ self.bodyCharsOrig = "".join(sorted(initChars))
+ self.bodyChars = set(initChars)
+
+ self.maxSpecified = max > 0
+
+ if min < 1:
+ raise ValueError(
+ "cannot specify a minimum length < 1; use Opt(Word()) if zero-length word is permitted"
+ )
+
+ self.minLen = min
+
+ if max > 0:
+ self.maxLen = max
+ else:
+ self.maxLen = _MAX_INT
+
+ if exact > 0:
+ self.maxLen = exact
+ self.minLen = exact
+
+ self.errmsg = "Expected " + self.name
+ self.mayIndexError = False
+ self.asKeyword = asKeyword
+
+ if " " not in self.initChars | self.bodyChars and (min == 1 and exact == 0):
+ if self.bodyChars == self.initChars:
+ if max == 0:
+ repeat = "+"
+ elif max == 1:
+ repeat = ""
+ else:
+ repeat = "{{{}}}".format(max)
+ self.reString = "[{}]{}".format(
+ _collapseStringToRanges(self.initChars),
+ repeat,
+ )
+ elif len(self.initChars) == 1:
+ if max == 0:
+ repeat = "*"
+ else:
+ repeat = "{{0,{}}}".format(max - 1)
+ self.reString = "{}[{}]{}".format(
+ re.escape(self.initCharsOrig),
+ _collapseStringToRanges(self.bodyChars),
+ repeat,
+ )
+ else:
+ if max == 0:
+ repeat = "*"
+ elif max == 2:
+ repeat = ""
+ else:
+ repeat = "{{0,{}}}".format(max - 1)
+ self.reString = "[{}][{}]{}".format(
+ _collapseStringToRanges(self.initChars),
+ _collapseStringToRanges(self.bodyChars),
+ repeat,
+ )
+ if self.asKeyword:
+ self.reString = r"\b" + self.reString + r"\b"
+
+ try:
+ self.re = re.compile(self.reString)
+ except sre_constants.error:
+ self.re = None
+ else:
+ self.re_match = self.re.match
+ self.__class__ = _WordRegex
+
+ def _generateDefaultName(self):
+ def charsAsStr(s):
+ max_repr_len = 16
+ s = _collapseStringToRanges(s, re_escape=False)
+ if len(s) > max_repr_len:
+ return s[: max_repr_len - 3] + "..."
+ else:
+ return s
+
+ if self.initChars != self.bodyChars:
+ base = "W:({}, {})".format(
+ charsAsStr(self.initChars), charsAsStr(self.bodyChars)
+ )
+ else:
+ base = "W:({})".format(charsAsStr(self.initChars))
+
+ # add length specification
+ if self.minLen > 1 or self.maxLen != _MAX_INT:
+ if self.minLen == self.maxLen:
+ if self.minLen == 1:
+ return base[2:]
+ else:
+ return base + "{{{}}}".format(self.minLen)
+ elif self.maxLen == _MAX_INT:
+ return base + "{{{},...}}".format(self.minLen)
+ else:
+ return base + "{{{},{}}}".format(self.minLen, self.maxLen)
+ return base
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if instring[loc] not in self.initChars:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ start = loc
+ loc += 1
+ instrlen = len(instring)
+ bodychars = self.bodyChars
+ maxloc = start + self.maxLen
+ maxloc = min(maxloc, instrlen)
+ while loc < maxloc and instring[loc] in bodychars:
+ loc += 1
+
+ throwException = False
+ if loc - start < self.minLen:
+ throwException = True
+ elif self.maxSpecified and loc < instrlen and instring[loc] in bodychars:
+ throwException = True
+ elif self.asKeyword:
+ if (
+ start > 0
+ and instring[start - 1] in bodychars
+ or loc < instrlen
+ and instring[loc] in bodychars
+ ):
+ throwException = True
+
+ if throwException:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ return loc, instring[start:loc]
+
+
+class _WordRegex(Word):
+ def parseImpl(self, instring, loc, doActions=True):
+ result = self.re_match(instring, loc)
+ if not result:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ loc = result.end()
+ return loc, result.group()
+
+
+class Char(_WordRegex):
+ """A short-cut class for defining :class:`Word` ``(characters, exact=1)``,
+ when defining a match of any single character in a string of
+ characters.
+ """
+
+ def __init__(
+ self,
+ charset: str,
+ as_keyword: bool = False,
+ exclude_chars: OptionalType[str] = None,
+ *,
+ asKeyword: bool = False,
+ excludeChars: OptionalType[str] = None,
+ ):
+ asKeyword = asKeyword or as_keyword
+ excludeChars = excludeChars or exclude_chars
+ super().__init__(
+ charset, exact=1, asKeyword=asKeyword, excludeChars=excludeChars
+ )
+ self.reString = "[{}]".format(_collapseStringToRanges(self.initChars))
+ if asKeyword:
+ self.reString = r"\b{}\b".format(self.reString)
+ self.re = re.compile(self.reString)
+ self.re_match = self.re.match
+
+
+class Regex(Token):
+ r"""Token for matching strings that match a given regular
+ expression. Defined with string specifying the regular expression in
+ a form recognized by the stdlib Python `re module <https://docs.python.org/3/library/re.html>`_.
+ If the given regex contains named groups (defined using ``(?P<name>...)``),
+ these will be preserved as named :class:`ParseResults`.
+
+ If instead of the Python stdlib ``re`` module you wish to use a different RE module
+ (such as the ``regex`` module), you can do so by building your ``Regex`` object with
+ a compiled RE that was compiled using ``regex``.
+
+ Example::
+
+ realnum = Regex(r"[+-]?\d+\.\d*")
+ # ref: https://stackoverflow.com/questions/267399/how-do-you-match-only-valid-roman-numerals-with-a-regular-expression
+ roman = Regex(r"M{0,4}(CM|CD|D?{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})")
+
+ # named fields in a regex will be returned as named results
+ date = Regex(r'(?P<year>\d{4})-(?P<month>\d\d?)-(?P<day>\d\d?)')
+
+ # the Regex class will accept re's compiled using the regex module
+ import regex
+ parser = pp.Regex(regex.compile(r'[0-9]'))
+ """
+
+ def __init__(
+ self,
+ pattern: Any,
+ flags: Union[re.RegexFlag, int] = 0,
+ as_group_list: bool = False,
+ as_match: bool = False,
+ *,
+ asGroupList: bool = False,
+ asMatch: bool = False,
+ ):
+ """The parameters ``pattern`` and ``flags`` are passed
+ to the ``re.compile()`` function as-is. See the Python
+ `re module <https://docs.python.org/3/library/re.html>`_ module for an
+ explanation of the acceptable patterns and flags.
+ """
+ super().__init__()
+ asGroupList = asGroupList or as_group_list
+ asMatch = asMatch or as_match
+
+ if isinstance(pattern, str_type):
+ if not pattern:
+ raise ValueError("null string passed to Regex; use Empty() instead")
+
+ self.pattern = pattern
+ self.flags = flags
+
+ try:
+ self.re = re.compile(self.pattern, self.flags)
+ self.reString = self.pattern
+ except sre_constants.error:
+ raise ValueError(
+ "invalid pattern ({!r}) passed to Regex".format(pattern)
+ )
+
+ elif hasattr(pattern, "pattern") and hasattr(pattern, "match"):
+ self.re = pattern
+ self.pattern = self.reString = pattern.pattern
+ self.flags = flags
+
+ else:
+ raise TypeError(
+ "Regex may only be constructed with a string or a compiled RE object"
+ )
+
+ self.re_match = self.re.match
+
+ self.errmsg = "Expected " + self.name
+ self.mayIndexError = False
+ self.mayReturnEmpty = self.re_match("") is not None
+ self.asGroupList = asGroupList
+ self.asMatch = asMatch
+ if self.asGroupList:
+ self.parseImpl = self.parseImplAsGroupList
+ if self.asMatch:
+ self.parseImpl = self.parseImplAsMatch
+
+ def _generateDefaultName(self):
+ return "Re:({})".format(repr(self.pattern).replace("\\\\", "\\"))
+
+ def parseImpl(self, instring, loc, doActions=True):
+ result = self.re_match(instring, loc)
+ if not result:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ loc = result.end()
+ ret = ParseResults(result.group())
+ d = result.groupdict()
+ if d:
+ for k, v in d.items():
+ ret[k] = v
+ return loc, ret
+
+ def parseImplAsGroupList(self, instring, loc, doActions=True):
+ result = self.re_match(instring, loc)
+ if not result:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ loc = result.end()
+ ret = result.groups()
+ return loc, ret
+
+ def parseImplAsMatch(self, instring, loc, doActions=True):
+ result = self.re_match(instring, loc)
+ if not result:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ loc = result.end()
+ ret = result
+ return loc, ret
+
+ def sub(self, repl):
+ r"""
+ Return :class:`Regex` with an attached parse action to transform the parsed
+ result as if called using `re.sub(expr, repl, string) <https://docs.python.org/3/library/re.html#re.sub>`_.
+
+ Example::
+
+ make_html = Regex(r"(\w+):(.*?):").sub(r"<\1>\2</\1>")
+ print(make_html.transform_string("h1:main title:"))
+ # prints "<h1>main title</h1>"
+ """
+ if self.asGroupList:
+ raise TypeError("cannot use sub() with Regex(asGroupList=True)")
+
+ if self.asMatch and callable(repl):
+ raise TypeError("cannot use sub() with a callable with Regex(asMatch=True)")
+
+ if self.asMatch:
+
+ def pa(tokens):
+ return tokens[0].expand(repl)
+
+ else:
+
+ def pa(tokens):
+ return self.re.sub(repl, tokens[0])
+
+ return self.add_parse_action(pa)
+
+
+class QuotedString(Token):
+ r"""
+ Token for matching strings that are delimited by quoting characters.
+
+ Defined with the following parameters:
+
+ - ``quote_char`` - string of one or more characters defining the
+ quote delimiting string
+ - ``esc_char`` - character to re_escape quotes, typically backslash
+ (default= ``None``)
+ - ``esc_quote`` - special quote sequence to re_escape an embedded quote
+ string (such as SQL's ``""`` to re_escape an embedded ``"``)
+ (default= ``None``)
+ - ``multiline`` - boolean indicating whether quotes can span
+ multiple lines (default= ``False``)
+ - ``unquote_results`` - boolean indicating whether the matched text
+ should be unquoted (default= ``True``)
+ - ``end_quote_char`` - string of one or more characters defining the
+ end of the quote delimited string (default= ``None`` => same as
+ quote_char)
+ - ``convert_whitespace_escapes`` - convert escaped whitespace
+ (``'\t'``, ``'\n'``, etc.) to actual whitespace
+ (default= ``True``)
+
+ Example::
+
+ qs = QuotedString('"')
+ print(qs.search_string('lsjdf "This is the quote" sldjf'))
+ complex_qs = QuotedString('{{', end_quote_char='}}')
+ print(complex_qs.search_string('lsjdf {{This is the "quote"}} sldjf'))
+ sql_qs = QuotedString('"', esc_quote='""')
+ print(sql_qs.search_string('lsjdf "This is the quote with ""embedded"" quotes" sldjf'))
+
+ prints::
+
+ [['This is the quote']]
+ [['This is the "quote"']]
+ [['This is the quote with "embedded" quotes']]
+ """
+ ws_map = ((r"\t", "\t"), (r"\n", "\n"), (r"\f", "\f"), (r"\r", "\r"))
+
+ def __init__(
+ self,
+ quote_char: str = "",
+ esc_char: OptionalType[str] = None,
+ esc_quote: OptionalType[str] = None,
+ multiline: bool = False,
+ unquote_results: bool = True,
+ end_quote_char: OptionalType[str] = None,
+ convert_whitespace_escapes: bool = True,
+ *,
+ quoteChar: str = "",
+ escChar: OptionalType[str] = None,
+ escQuote: OptionalType[str] = None,
+ unquoteResults: bool = True,
+ endQuoteChar: OptionalType[str] = None,
+ convertWhitespaceEscapes: bool = True,
+ ):
+ super().__init__()
+ escChar = escChar or esc_char
+ escQuote = escQuote or esc_quote
+ unquoteResults = unquoteResults and unquote_results
+ endQuoteChar = endQuoteChar or end_quote_char
+ convertWhitespaceEscapes = (
+ convertWhitespaceEscapes and convert_whitespace_escapes
+ )
+ quote_char = quoteChar or quote_char
+
+ # remove white space from quote chars - wont work anyway
+ quote_char = quote_char.strip()
+ if not quote_char:
+ raise ValueError("quote_char cannot be the empty string")
+
+ if endQuoteChar is None:
+ endQuoteChar = quote_char
+ else:
+ endQuoteChar = endQuoteChar.strip()
+ if not endQuoteChar:
+ raise ValueError("endQuoteChar cannot be the empty string")
+
+ self.quoteChar = quote_char
+ self.quoteCharLen = len(quote_char)
+ self.firstQuoteChar = quote_char[0]
+ self.endQuoteChar = endQuoteChar
+ self.endQuoteCharLen = len(endQuoteChar)
+ self.escChar = escChar
+ self.escQuote = escQuote
+ self.unquoteResults = unquoteResults
+ self.convertWhitespaceEscapes = convertWhitespaceEscapes
+
+ sep = ""
+ inner_pattern = ""
+
+ if escQuote:
+ inner_pattern += r"{}(?:{})".format(sep, re.escape(escQuote))
+ sep = "|"
+
+ if escChar:
+ inner_pattern += r"{}(?:{}.)".format(sep, re.escape(escChar))
+ sep = "|"
+ self.escCharReplacePattern = re.escape(self.escChar) + "(.)"
+
+ if len(self.endQuoteChar) > 1:
+ inner_pattern += (
+ "{}(?:".format(sep)
+ + "|".join(
+ "(?:{}(?!{}))".format(
+ re.escape(self.endQuoteChar[:i]),
+ _escapeRegexRangeChars(self.endQuoteChar[i:]),
+ )
+ for i in range(len(self.endQuoteChar) - 1, 0, -1)
+ )
+ + ")"
+ )
+ sep = "|"
+
+ if multiline:
+ self.flags = re.MULTILINE | re.DOTALL
+ inner_pattern += r"{}(?:[^{}{}])".format(
+ sep,
+ _escapeRegexRangeChars(self.endQuoteChar[0]),
+ (_escapeRegexRangeChars(escChar) if escChar is not None else ""),
+ )
+ else:
+ self.flags = 0
+ inner_pattern += r"{}(?:[^{}\n\r{}])".format(
+ sep,
+ _escapeRegexRangeChars(self.endQuoteChar[0]),
+ (_escapeRegexRangeChars(escChar) if escChar is not None else ""),
+ )
+
+ self.pattern = "".join(
+ [
+ re.escape(self.quoteChar),
+ "(?:",
+ inner_pattern,
+ ")*",
+ re.escape(self.endQuoteChar),
+ ]
+ )
+
+ try:
+ self.re = re.compile(self.pattern, self.flags)
+ self.reString = self.pattern
+ self.re_match = self.re.match
+ except sre_constants.error:
+ raise ValueError(
+ "invalid pattern {!r} passed to Regex".format(self.pattern)
+ )
+
+ self.errmsg = "Expected " + self.name
+ self.mayIndexError = False
+ self.mayReturnEmpty = True
+
+ def _generateDefaultName(self):
+ if self.quoteChar == self.endQuoteChar and isinstance(self.quoteChar, str_type):
+ return "string enclosed in {!r}".format(self.quoteChar)
+
+ return "quoted string, starting with {} ending with {}".format(
+ self.quoteChar, self.endQuoteChar
+ )
+
+ def parseImpl(self, instring, loc, doActions=True):
+ result = (
+ instring[loc] == self.firstQuoteChar
+ and self.re_match(instring, loc)
+ or None
+ )
+ if not result:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ loc = result.end()
+ ret = result.group()
+
+ if self.unquoteResults:
+
+ # strip off quotes
+ ret = ret[self.quoteCharLen : -self.endQuoteCharLen]
+
+ if isinstance(ret, str_type):
+ # replace escaped whitespace
+ if "\\" in ret and self.convertWhitespaceEscapes:
+ for wslit, wschar in self.ws_map:
+ ret = ret.replace(wslit, wschar)
+
+ # replace escaped characters
+ if self.escChar:
+ ret = re.sub(self.escCharReplacePattern, r"\g<1>", ret)
+
+ # replace escaped quotes
+ if self.escQuote:
+ ret = ret.replace(self.escQuote, self.endQuoteChar)
+
+ return loc, ret
+
+
+class CharsNotIn(Token):
+ """Token for matching words composed of characters *not* in a given
+ set (will include whitespace in matched characters if not listed in
+ the provided exclusion set - see example). Defined with string
+ containing all disallowed characters, and an optional minimum,
+ maximum, and/or exact length. The default value for ``min`` is
+ 1 (a minimum value < 1 is not valid); the default values for
+ ``max`` and ``exact`` are 0, meaning no maximum or exact
+ length restriction.
+
+ Example::
+
+ # define a comma-separated-value as anything that is not a ','
+ csv_value = CharsNotIn(',')
+ print(delimited_list(csv_value).parse_string("dkls,lsdkjf,s12 34,@!#,213"))
+
+ prints::
+
+ ['dkls', 'lsdkjf', 's12 34', '@!#', '213']
+ """
+
+ def __init__(
+ self,
+ not_chars: str = "",
+ min: int = 1,
+ max: int = 0,
+ exact: int = 0,
+ *,
+ notChars: str = "",
+ ):
+ super().__init__()
+ self.skipWhitespace = False
+ self.notChars = not_chars or notChars
+ self.notCharsSet = set(self.notChars)
+
+ if min < 1:
+ raise ValueError(
+ "cannot specify a minimum length < 1; use "
+ "Opt(CharsNotIn()) if zero-length char group is permitted"
+ )
+
+ self.minLen = min
+
+ if max > 0:
+ self.maxLen = max
+ else:
+ self.maxLen = _MAX_INT
+
+ if exact > 0:
+ self.maxLen = exact
+ self.minLen = exact
+
+ self.errmsg = "Expected " + self.name
+ self.mayReturnEmpty = self.minLen == 0
+ self.mayIndexError = False
+
+ def _generateDefaultName(self):
+ not_chars_str = _collapseStringToRanges(self.notChars)
+ if len(not_chars_str) > 16:
+ return "!W:({}...)".format(self.notChars[: 16 - 3])
+ else:
+ return "!W:({})".format(self.notChars)
+
+ def parseImpl(self, instring, loc, doActions=True):
+ notchars = self.notCharsSet
+ if instring[loc] in notchars:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ start = loc
+ loc += 1
+ maxlen = min(start + self.maxLen, len(instring))
+ while loc < maxlen and instring[loc] not in notchars:
+ loc += 1
+
+ if loc - start < self.minLen:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ return loc, instring[start:loc]
+
+
+class White(Token):
+ """Special matching class for matching whitespace. Normally,
+ whitespace is ignored by pyparsing grammars. This class is included
+ when some whitespace structures are significant. Define with
+ a string containing the whitespace characters to be matched; default
+ is ``" \\t\\r\\n"``. Also takes optional ``min``,
+ ``max``, and ``exact`` arguments, as defined for the
+ :class:`Word` class.
+ """
+
+ whiteStrs = {
+ " ": "<SP>",
+ "\t": "<TAB>",
+ "\n": "<LF>",
+ "\r": "<CR>",
+ "\f": "<FF>",
+ "\u00A0": "<NBSP>",
+ "\u1680": "<OGHAM_SPACE_MARK>",
+ "\u180E": "<MONGOLIAN_VOWEL_SEPARATOR>",
+ "\u2000": "<EN_QUAD>",
+ "\u2001": "<EM_QUAD>",
+ "\u2002": "<EN_SPACE>",
+ "\u2003": "<EM_SPACE>",
+ "\u2004": "<THREE-PER-EM_SPACE>",
+ "\u2005": "<FOUR-PER-EM_SPACE>",
+ "\u2006": "<SIX-PER-EM_SPACE>",
+ "\u2007": "<FIGURE_SPACE>",
+ "\u2008": "<PUNCTUATION_SPACE>",
+ "\u2009": "<THIN_SPACE>",
+ "\u200A": "<HAIR_SPACE>",
+ "\u200B": "<ZERO_WIDTH_SPACE>",
+ "\u202F": "<NNBSP>",
+ "\u205F": "<MMSP>",
+ "\u3000": "<IDEOGRAPHIC_SPACE>",
+ }
+
+ def __init__(self, ws: str = " \t\r\n", min: int = 1, max: int = 0, exact: int = 0):
+ super().__init__()
+ self.matchWhite = ws
+ self.set_whitespace_chars(
+ "".join(c for c in self.whiteChars if c not in self.matchWhite),
+ copy_defaults=True,
+ )
+ # self.leave_whitespace()
+ self.mayReturnEmpty = True
+ self.errmsg = "Expected " + self.name
+
+ self.minLen = min
+
+ if max > 0:
+ self.maxLen = max
+ else:
+ self.maxLen = _MAX_INT
+
+ if exact > 0:
+ self.maxLen = exact
+ self.minLen = exact
+
+ def _generateDefaultName(self):
+ return "".join(White.whiteStrs[c] for c in self.matchWhite)
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if instring[loc] not in self.matchWhite:
+ raise ParseException(instring, loc, self.errmsg, self)
+ start = loc
+ loc += 1
+ maxloc = start + self.maxLen
+ maxloc = min(maxloc, len(instring))
+ while loc < maxloc and instring[loc] in self.matchWhite:
+ loc += 1
+
+ if loc - start < self.minLen:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ return loc, instring[start:loc]
+
+
+class _PositionToken(Token):
+ def __init__(self):
+ super().__init__()
+ self.mayReturnEmpty = True
+ self.mayIndexError = False
+
+
+class GoToColumn(_PositionToken):
+ """Token to advance to a specific column of input text; useful for
+ tabular report scraping.
+ """
+
+ def __init__(self, colno: int):
+ super().__init__()
+ self.col = colno
+
+ def preParse(self, instring, loc):
+ if col(loc, instring) != self.col:
+ instrlen = len(instring)
+ if self.ignoreExprs:
+ loc = self._skipIgnorables(instring, loc)
+ while (
+ loc < instrlen
+ and instring[loc].isspace()
+ and col(loc, instring) != self.col
+ ):
+ loc += 1
+ return loc
+
+ def parseImpl(self, instring, loc, doActions=True):
+ thiscol = col(loc, instring)
+ if thiscol > self.col:
+ raise ParseException(instring, loc, "Text not in expected column", self)
+ newloc = loc + self.col - thiscol
+ ret = instring[loc:newloc]
+ return newloc, ret
+
+
+class LineStart(_PositionToken):
+ r"""Matches if current position is at the beginning of a line within
+ the parse string
+
+ Example::
+
+ test = '''\
+ AAA this line
+ AAA and this line
+ AAA but not this one
+ B AAA and definitely not this one
+ '''
+
+ for t in (LineStart() + 'AAA' + restOfLine).search_string(test):
+ print(t)
+
+ prints::
+
+ ['AAA', ' this line']
+ ['AAA', ' and this line']
+
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.errmsg = "Expected start of line"
+
+ def __add__(self, other):
+ return AtLineStart(other)
+
+ def __sub__(self, other):
+ return AtLineStart(other) - Empty()
+
+ def preParse(self, instring, loc):
+ if loc == 0:
+ return loc
+ else:
+ if instring[loc : loc + 1] == "\n" and "\n" in self.whiteChars:
+ ret = loc + 1
+ else:
+ ret = super().preParse(instring, loc)
+ return ret
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if col(loc, instring) == 1:
+ return loc, []
+ raise ParseException(instring, loc, self.errmsg, self)
+
+
+class LineEnd(_PositionToken):
+ """Matches if current position is at the end of a line within the
+ parse string
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.whiteChars.discard("\n")
+ self.set_whitespace_chars(self.whiteChars, copy_defaults=False)
+ self.errmsg = "Expected end of line"
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if loc < len(instring):
+ if instring[loc] == "\n":
+ return loc + 1, "\n"
+ else:
+ raise ParseException(instring, loc, self.errmsg, self)
+ elif loc == len(instring):
+ return loc + 1, []
+ else:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+
+class StringStart(_PositionToken):
+ """Matches if current position is at the beginning of the parse
+ string
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.errmsg = "Expected start of text"
+
+ def __add__(self, other):
+ return AtStringStart(other)
+
+ def __sub__(self, other):
+ return AtStringStart(other) - Empty()
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if loc != 0:
+ # see if entire string up to here is just whitespace and ignoreables
+ if loc != self.preParse(instring, 0):
+ raise ParseException(instring, loc, self.errmsg, self)
+ return loc, []
+
+
+class StringEnd(_PositionToken):
+ """
+ Matches if current position is at the end of the parse string
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.errmsg = "Expected end of text"
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if loc < len(instring):
+ raise ParseException(instring, loc, self.errmsg, self)
+ elif loc == len(instring):
+ return loc + 1, []
+ elif loc > len(instring):
+ return loc, []
+ else:
+ raise ParseException(instring, loc, self.errmsg, self)
+
+
+class WordStart(_PositionToken):
+ """Matches if the current position is at the beginning of a
+ :class:`Word`, and is not preceded by any character in a given
+ set of ``word_chars`` (default= ``printables``). To emulate the
+ ``\b`` behavior of regular expressions, use
+ ``WordStart(alphanums)``. ``WordStart`` will also match at
+ the beginning of the string being parsed, or at the beginning of
+ a line.
+ """
+
+ def __init__(self, word_chars: str = printables, *, wordChars: str = printables):
+ wordChars = word_chars if wordChars != printables else wordChars
+ super().__init__()
+ self.wordChars = set(wordChars)
+ self.errmsg = "Not at the start of a word"
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if loc != 0:
+ if (
+ instring[loc - 1] in self.wordChars
+ or instring[loc] not in self.wordChars
+ ):
+ raise ParseException(instring, loc, self.errmsg, self)
+ return loc, []
+
+
+class WordEnd(_PositionToken):
+ """Matches if the current position is at the end of a :class:`Word`,
+ and is not followed by any character in a given set of ``word_chars``
+ (default= ``printables``). To emulate the ``\b`` behavior of
+ regular expressions, use ``WordEnd(alphanums)``. ``WordEnd``
+ will also match at the end of the string being parsed, or at the end
+ of a line.
+ """
+
+ def __init__(self, word_chars: str = printables, *, wordChars: str = printables):
+ wordChars = word_chars if wordChars != printables else wordChars
+ super().__init__()
+ self.wordChars = set(wordChars)
+ self.skipWhitespace = False
+ self.errmsg = "Not at the end of a word"
+
+ def parseImpl(self, instring, loc, doActions=True):
+ instrlen = len(instring)
+ if instrlen > 0 and loc < instrlen:
+ if (
+ instring[loc] in self.wordChars
+ or instring[loc - 1] not in self.wordChars
+ ):
+ raise ParseException(instring, loc, self.errmsg, self)
+ return loc, []
+
+
+class ParseExpression(ParserElement):
+ """Abstract subclass of ParserElement, for combining and
+ post-processing parsed tokens.
+ """
+
+ def __init__(self, exprs: IterableType[ParserElement], savelist: bool = False):
+ super().__init__(savelist)
+ self.exprs : List[ParserElement]
+ exprs : Iterable[ParserElement]
+ if isinstance(exprs, _generatorType):
+ exprs = list(exprs)
+
+ if isinstance(exprs, str_type):
+ self.exprs = [self._literalStringClass(exprs)]
+ elif isinstance(exprs, ParserElement):
+ self.exprs = [exprs]
+ elif isinstance(exprs, Iterable):
+ exprs = list(exprs)
+ # if sequence of strings provided, wrap with Literal
+ if any(isinstance(expr, str_type) for expr in exprs):
+ exprs = (
+ self._literalStringClass(e) if isinstance(e, str_type) else e
+ for e in exprs
+ )
+ self.exprs = list(exprs)
+ else:
+ try:
+ self.exprs = list(exprs)
+ except TypeError:
+ self.exprs = [exprs]
+ self.callPreparse = False
+
+ def recurse(self):
+ return self.exprs[:]
+
+ def append(self, other):
+ self.exprs.append(other)
+ self._defaultName = None
+ return self
+
+ def leave_whitespace(self, recursive=True):
+ """
+ Extends ``leave_whitespace`` defined in base class, and also invokes ``leave_whitespace`` on
+ all contained expressions.
+ """
+ super().leave_whitespace(recursive)
+
+ if recursive:
+ self.exprs = [e.copy() for e in self.exprs]
+ for e in self.exprs:
+ e.leave_whitespace(recursive)
+ return self
+
+ def ignore_whitespace(self, recursive=True):
+ """
+ Extends ``ignore_whitespace`` defined in base class, and also invokes ``leave_whitespace`` on
+ all contained expressions.
+ """
+ super().ignore_whitespace(recursive)
+ if recursive:
+ self.exprs = [e.copy() for e in self.exprs]
+ for e in self.exprs:
+ e.ignore_whitespace(recursive)
+ return self
+
+ def ignore(self, other):
+ if isinstance(other, Suppress):
+ if other not in self.ignoreExprs:
+ super().ignore(other)
+ for e in self.exprs:
+ e.ignore(self.ignoreExprs[-1])
+ else:
+ super().ignore(other)
+ for e in self.exprs:
+ e.ignore(self.ignoreExprs[-1])
+ return self
+
+ def _generateDefaultName(self):
+ return "{}:({})".format(self.__class__.__name__, str(self.exprs))
+
+ def streamline(self):
+ if self.streamlined:
+ return
+
+ super().streamline()
+
+ for e in self.exprs:
+ e.streamline()
+
+ # collapse nested :class:`And`'s of the form ``And(And(And(a, b), c), d)`` to ``And(a, b, c, d)``
+ # but only if there are no parse actions or resultsNames on the nested And's
+ # (likewise for :class:`Or`'s and :class:`MatchFirst`'s)
+ if len(self.exprs) == 2:
+ other = self.exprs[0]
+ if (
+ isinstance(other, self.__class__)
+ and not other.parseAction
+ and other.resultsName is None
+ and not other.debug
+ ):
+ self.exprs = other.exprs[:] + [self.exprs[1]]
+ self._defaultName = None
+ self.mayReturnEmpty |= other.mayReturnEmpty
+ self.mayIndexError |= other.mayIndexError
+
+ other = self.exprs[-1]
+ if (
+ isinstance(other, self.__class__)
+ and not other.parseAction
+ and other.resultsName is None
+ and not other.debug
+ ):
+ self.exprs = self.exprs[:-1] + other.exprs[:]
+ self._defaultName = None
+ self.mayReturnEmpty |= other.mayReturnEmpty
+ self.mayIndexError |= other.mayIndexError
+
+ self.errmsg = "Expected " + str(self)
+
+ return self
+
+ def validate(self, validateTrace=None):
+ tmp = (validateTrace if validateTrace is not None else [])[:] + [self]
+ for e in self.exprs:
+ e.validate(tmp)
+ self._checkRecursion([])
+
+ def copy(self):
+ ret = super().copy()
+ ret.exprs = [e.copy() for e in self.exprs]
+ return ret
+
+ def _setResultsName(self, name, listAllMatches=False):
+ if __diag__.warn_ungrouped_named_tokens_in_collection:
+ for e in self.exprs:
+ if isinstance(e, ParserElement) and e.resultsName:
+ warnings.warn(
+ "{}: setting results name {!r} on {} expression "
+ "collides with {!r} on contained expression".format(
+ "warn_ungrouped_named_tokens_in_collection",
+ name,
+ type(self).__name__,
+ e.resultsName,
+ ),
+ stacklevel=3,
+ )
+
+ return super()._setResultsName(name, listAllMatches)
+
+ ignoreWhitespace = ignore_whitespace
+ leaveWhitespace = leave_whitespace
+
+
+class And(ParseExpression):
+ """
+ Requires all given :class:`ParseExpression` s to be found in the given order.
+ Expressions may be separated by whitespace.
+ May be constructed using the ``'+'`` operator.
+ May also be constructed using the ``'-'`` operator, which will
+ suppress backtracking.
+
+ Example::
+
+ integer = Word(nums)
+ name_expr = OneOrMore(Word(alphas))
+
+ expr = And([integer("id"), name_expr("name"), integer("age")])
+ # more easily written as:
+ expr = integer("id") + name_expr("name") + integer("age")
+ """
+
+ class _ErrorStop(Empty):
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.leave_whitespace()
+
+ def _generateDefaultName(self):
+ return "-"
+
+ def __init__(self, exprs_arg: IterableType[ParserElement], savelist: bool = True):
+ exprs: List[ParserElement] = list(exprs_arg)
+ if exprs and Ellipsis in exprs:
+ tmp = []
+ for i, expr in enumerate(exprs):
+ if expr is Ellipsis:
+ if i < len(exprs) - 1:
+ skipto_arg: ParserElement = (Empty() + exprs[i + 1]).exprs[-1]
+ tmp.append(SkipTo(skipto_arg)("_skipped*"))
+ else:
+ raise Exception(
+ "cannot construct And with sequence ending in ..."
+ )
+ else:
+ tmp.append(expr)
+ exprs[:] = tmp
+ super().__init__(exprs, savelist)
+ if self.exprs:
+ self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
+ self.set_whitespace_chars(
+ self.exprs[0].whiteChars,
+ copy_defaults=self.exprs[0].copyDefaultWhiteChars,
+ )
+ self.skipWhitespace = self.exprs[0].skipWhitespace
+ else:
+ self.mayReturnEmpty = True
+ self.callPreparse = True
+
+ def streamline(self):
+ # collapse any _PendingSkip's
+ if self.exprs:
+ if any(
+ isinstance(e, ParseExpression)
+ and e.exprs
+ and isinstance(e.exprs[-1], _PendingSkip)
+ for e in self.exprs[:-1]
+ ):
+ for i, e in enumerate(self.exprs[:-1]):
+ if e is None:
+ continue
+ if (
+ isinstance(e, ParseExpression)
+ and e.exprs
+ and isinstance(e.exprs[-1], _PendingSkip)
+ ):
+ e.exprs[-1] = e.exprs[-1] + self.exprs[i + 1]
+ self.exprs[i + 1] = None
+ self.exprs = [e for e in self.exprs if e is not None]
+
+ super().streamline()
+ self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
+ return self
+
+ def parseImpl(self, instring, loc, doActions=True):
+ # pass False as callPreParse arg to _parse for first element, since we already
+ # pre-parsed the string as part of our And pre-parsing
+ loc, resultlist = self.exprs[0]._parse(
+ instring, loc, doActions, callPreParse=False
+ )
+ errorStop = False
+ for e in self.exprs[1:]:
+ # if isinstance(e, And._ErrorStop):
+ if type(e) is And._ErrorStop:
+ errorStop = True
+ continue
+ if errorStop:
+ try:
+ loc, exprtokens = e._parse(instring, loc, doActions)
+ except ParseSyntaxException:
+ raise
+ except ParseBaseException as pe:
+ pe.__traceback__ = None
+ raise ParseSyntaxException._from_exception(pe)
+ except IndexError:
+ raise ParseSyntaxException(
+ instring, len(instring), self.errmsg, self
+ )
+ else:
+ loc, exprtokens = e._parse(instring, loc, doActions)
+ if exprtokens or exprtokens.haskeys():
+ resultlist += exprtokens
+ return loc, resultlist
+
+ def __iadd__(self, other):
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ return self.append(other) # And([self, other])
+
+ def _checkRecursion(self, parseElementList):
+ subRecCheckList = parseElementList[:] + [self]
+ for e in self.exprs:
+ e._checkRecursion(subRecCheckList)
+ if not e.mayReturnEmpty:
+ break
+
+ def _generateDefaultName(self):
+ inner = " ".join(str(e) for e in self.exprs)
+ # strip off redundant inner {}'s
+ while len(inner) > 1 and inner[0 :: len(inner) - 1] == "{}":
+ inner = inner[1:-1]
+ return "{" + inner + "}"
+
+
+class Or(ParseExpression):
+ """Requires that at least one :class:`ParseExpression` is found. If
+ two expressions match, the expression that matches the longest
+ string will be used. May be constructed using the ``'^'``
+ operator.
+
+ Example::
+
+ # construct Or using '^' operator
+
+ number = Word(nums) ^ Combine(Word(nums) + '.' + Word(nums))
+ print(number.search_string("123 3.1416 789"))
+
+ prints::
+
+ [['123'], ['3.1416'], ['789']]
+ """
+
+ def __init__(self, exprs: IterableType[ParserElement], savelist: bool = False):
+ super().__init__(exprs, savelist)
+ if self.exprs:
+ self.mayReturnEmpty = any(e.mayReturnEmpty for e in self.exprs)
+ else:
+ self.mayReturnEmpty = True
+
+ def streamline(self):
+ super().streamline()
+ if self.exprs:
+ self.saveAsList = any(e.saveAsList for e in self.exprs)
+ else:
+ self.saveAsList = False
+ return self
+
+ def parseImpl(self, instring, loc, doActions=True):
+ maxExcLoc = -1
+ maxException = None
+ matches = []
+ fatals = []
+ if all(e.callPreparse for e in self.exprs):
+ loc = self.preParse(instring, loc)
+ for e in self.exprs:
+ try:
+ loc2 = e.try_parse(instring, loc, raise_fatal=True)
+ except ParseFatalException as pfe:
+ pfe.__traceback__ = None
+ pfe.parserElement = e
+ fatals.append(pfe)
+ maxException = None
+ maxExcLoc = -1
+ except ParseException as err:
+ if not fatals:
+ err.__traceback__ = None
+ if err.loc > maxExcLoc:
+ maxException = err
+ maxExcLoc = err.loc
+ except IndexError:
+ if len(instring) > maxExcLoc:
+ maxException = ParseException(
+ instring, len(instring), e.errmsg, self
+ )
+ maxExcLoc = len(instring)
+ else:
+ # save match among all matches, to retry longest to shortest
+ matches.append((loc2, e))
+
+ if matches:
+ # re-evaluate all matches in descending order of length of match, in case attached actions
+ # might change whether or how much they match of the input.
+ matches.sort(key=itemgetter(0), reverse=True)
+
+ if not doActions:
+ # no further conditions or parse actions to change the selection of
+ # alternative, so the first match will be the best match
+ best_expr = matches[0][1]
+ return best_expr._parse(instring, loc, doActions)
+
+ longest = -1, None
+ for loc1, expr1 in matches:
+ if loc1 <= longest[0]:
+ # already have a longer match than this one will deliver, we are done
+ return longest
+
+ try:
+ loc2, toks = expr1._parse(instring, loc, doActions)
+ except ParseException as err:
+ err.__traceback__ = None
+ if err.loc > maxExcLoc:
+ maxException = err
+ maxExcLoc = err.loc
+ else:
+ if loc2 >= loc1:
+ return loc2, toks
+ # didn't match as much as before
+ elif loc2 > longest[0]:
+ longest = loc2, toks
+
+ if longest != (-1, None):
+ return longest
+
+ if fatals:
+ if len(fatals) > 1:
+ fatals.sort(key=lambda e: -e.loc)
+ if fatals[0].loc == fatals[1].loc:
+ fatals.sort(key=lambda e: (-e.loc, -len(str(e.parserElement))))
+ max_fatal = fatals[0]
+ raise max_fatal
+
+ if maxException is not None:
+ maxException.msg = self.errmsg
+ raise maxException
+ else:
+ raise ParseException(
+ instring, loc, "no defined alternatives to match", self
+ )
+
+ def __ixor__(self, other):
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ return self.append(other) # Or([self, other])
+
+ def _generateDefaultName(self):
+ return "{" + " ^ ".join(str(e) for e in self.exprs) + "}"
+
+ def _setResultsName(self, name, listAllMatches=False):
+ if __diag__.warn_multiple_tokens_in_named_alternation:
+ if any(isinstance(e, And) for e in self.exprs):
+ warnings.warn(
+ "{}: setting results name {!r} on {} expression "
+ "will return a list of all parsed tokens in an And alternative, "
+ "in prior versions only the first token was returned".format(
+ "warn_multiple_tokens_in_named_alternation",
+ name,
+ type(self).__name__,
+ ),
+ stacklevel=3,
+ )
+
+ return super()._setResultsName(name, listAllMatches)
+
+
+class MatchFirst(ParseExpression):
+ """Requires that at least one :class:`ParseExpression` is found. If
+ two expressions match, the first one listed is the one that will
+ match. May be constructed using the ``'|'`` operator.
+
+ Example::
+
+ # construct MatchFirst using '|' operator
+
+ # watch the order of expressions to match
+ number = Word(nums) | Combine(Word(nums) + '.' + Word(nums))
+ print(number.search_string("123 3.1416 789")) # Fail! -> [['123'], ['3'], ['1416'], ['789']]
+
+ # put more selective expression first
+ number = Combine(Word(nums) + '.' + Word(nums)) | Word(nums)
+ print(number.search_string("123 3.1416 789")) # Better -> [['123'], ['3.1416'], ['789']]
+ """
+
+ def __init__(self, exprs: IterableType[ParserElement], savelist: bool = False):
+ super().__init__(exprs, savelist)
+ if self.exprs:
+ self.mayReturnEmpty = any(e.mayReturnEmpty for e in self.exprs)
+ self.callPreparse = all(e.callPreparse for e in self.exprs)
+ else:
+ self.mayReturnEmpty = True
+
+ def streamline(self):
+ if self.streamlined:
+ return
+
+ super().streamline()
+ if self.exprs:
+ self.saveAsList = any(e.saveAsList for e in self.exprs)
+ self.mayReturnEmpty = any(e.mayReturnEmpty for e in self.exprs)
+ self.callPreparse = all(e.callPreparse for e in self.exprs)
+ else:
+ self.saveAsList = False
+ self.mayReturnEmpty = True
+ return self
+
+ def parseImpl(self, instring, loc, doActions=True):
+ maxExcLoc = -1
+ maxException = None
+
+ for e in self.exprs:
+ try:
+ return e._parse(
+ instring, loc, doActions, callPreParse=not self.callPreparse
+ )
+ except ParseFatalException as pfe:
+ pfe.__traceback__ = None
+ pfe.parserElement = e
+ raise
+ except ParseException as err:
+ if err.loc > maxExcLoc:
+ maxException = err
+ maxExcLoc = err.loc
+ except IndexError:
+ if len(instring) > maxExcLoc:
+ maxException = ParseException(
+ instring, len(instring), e.errmsg, self
+ )
+ maxExcLoc = len(instring)
+
+ if maxException is not None:
+ maxException.msg = self.errmsg
+ raise maxException
+ else:
+ raise ParseException(
+ instring, loc, "no defined alternatives to match", self
+ )
+
+ def __ior__(self, other):
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ return self.append(other) # MatchFirst([self, other])
+
+ def _generateDefaultName(self):
+ return "{" + " | ".join(str(e) for e in self.exprs) + "}"
+
+ def _setResultsName(self, name, listAllMatches=False):
+ if __diag__.warn_multiple_tokens_in_named_alternation:
+ if any(isinstance(e, And) for e in self.exprs):
+ warnings.warn(
+ "{}: setting results name {!r} on {} expression "
+ "may only return a single token for an And alternative, "
+ "in future will return the full list of tokens".format(
+ "warn_multiple_tokens_in_named_alternation",
+ name,
+ type(self).__name__,
+ ),
+ stacklevel=3,
+ )
+
+ return super()._setResultsName(name, listAllMatches)
+
+
+class Each(ParseExpression):
+ """Requires all given :class:`ParseExpression` s to be found, but in
+ any order. Expressions may be separated by whitespace.
+
+ May be constructed using the ``'&'`` operator.
+
+ Example::
+
+ color = one_of("RED ORANGE YELLOW GREEN BLUE PURPLE BLACK WHITE BROWN")
+ shape_type = one_of("SQUARE CIRCLE TRIANGLE STAR HEXAGON OCTAGON")
+ integer = Word(nums)
+ shape_attr = "shape:" + shape_type("shape")
+ posn_attr = "posn:" + Group(integer("x") + ',' + integer("y"))("posn")
+ color_attr = "color:" + color("color")
+ size_attr = "size:" + integer("size")
+
+ # use Each (using operator '&') to accept attributes in any order
+ # (shape and posn are required, color and size are optional)
+ shape_spec = shape_attr & posn_attr & Opt(color_attr) & Opt(size_attr)
+
+ shape_spec.run_tests('''
+ shape: SQUARE color: BLACK posn: 100, 120
+ shape: CIRCLE size: 50 color: BLUE posn: 50,80
+ color:GREEN size:20 shape:TRIANGLE posn:20,40
+ '''
+ )
+
+ prints::
+
+ shape: SQUARE color: BLACK posn: 100, 120
+ ['shape:', 'SQUARE', 'color:', 'BLACK', 'posn:', ['100', ',', '120']]
+ - color: BLACK
+ - posn: ['100', ',', '120']
+ - x: 100
+ - y: 120
+ - shape: SQUARE
+
+
+ shape: CIRCLE size: 50 color: BLUE posn: 50,80
+ ['shape:', 'CIRCLE', 'size:', '50', 'color:', 'BLUE', 'posn:', ['50', ',', '80']]
+ - color: BLUE
+ - posn: ['50', ',', '80']
+ - x: 50
+ - y: 80
+ - shape: CIRCLE
+ - size: 50
+
+
+ color: GREEN size: 20 shape: TRIANGLE posn: 20,40
+ ['color:', 'GREEN', 'size:', '20', 'shape:', 'TRIANGLE', 'posn:', ['20', ',', '40']]
+ - color: GREEN
+ - posn: ['20', ',', '40']
+ - x: 20
+ - y: 40
+ - shape: TRIANGLE
+ - size: 20
+ """
+
+ def __init__(self, exprs: IterableType[ParserElement], savelist: bool = True):
+ super().__init__(exprs, savelist)
+ if self.exprs:
+ self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
+ else:
+ self.mayReturnEmpty = True
+ self.skipWhitespace = True
+ self.initExprGroups = True
+ self.saveAsList = True
+
+ def streamline(self):
+ super().streamline()
+ if self.exprs:
+ self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
+ else:
+ self.mayReturnEmpty = True
+ return self
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if self.initExprGroups:
+ self.opt1map = dict(
+ (id(e.expr), e) for e in self.exprs if isinstance(e, Opt)
+ )
+ opt1 = [e.expr for e in self.exprs if isinstance(e, Opt)]
+ opt2 = [
+ e
+ for e in self.exprs
+ if e.mayReturnEmpty and not isinstance(e, (Opt, Regex, ZeroOrMore))
+ ]
+ self.optionals = opt1 + opt2
+ self.multioptionals = [
+ e.expr.set_results_name(e.resultsName, list_all_matches=True)
+ for e in self.exprs
+ if isinstance(e, _MultipleMatch)
+ ]
+ self.multirequired = [
+ e.expr.set_results_name(e.resultsName, list_all_matches=True)
+ for e in self.exprs
+ if isinstance(e, OneOrMore)
+ ]
+ self.required = [
+ e for e in self.exprs if not isinstance(e, (Opt, ZeroOrMore, OneOrMore))
+ ]
+ self.required += self.multirequired
+ self.initExprGroups = False
+
+ tmpLoc = loc
+ tmpReqd = self.required[:]
+ tmpOpt = self.optionals[:]
+ multis = self.multioptionals[:]
+ matchOrder = []
+
+ keepMatching = True
+ failed = []
+ fatals = []
+ while keepMatching:
+ tmpExprs = tmpReqd + tmpOpt + multis
+ failed.clear()
+ fatals.clear()
+ for e in tmpExprs:
+ try:
+ tmpLoc = e.try_parse(instring, tmpLoc, raise_fatal=True)
+ except ParseFatalException as pfe:
+ pfe.__traceback__ = None
+ pfe.parserElement = e
+ fatals.append(pfe)
+ failed.append(e)
+ except ParseException:
+ failed.append(e)
+ else:
+ matchOrder.append(self.opt1map.get(id(e), e))
+ if e in tmpReqd:
+ tmpReqd.remove(e)
+ elif e in tmpOpt:
+ tmpOpt.remove(e)
+ if len(failed) == len(tmpExprs):
+ keepMatching = False
+
+ # look for any ParseFatalExceptions
+ if fatals:
+ if len(fatals) > 1:
+ fatals.sort(key=lambda e: -e.loc)
+ if fatals[0].loc == fatals[1].loc:
+ fatals.sort(key=lambda e: (-e.loc, -len(str(e.parserElement))))
+ max_fatal = fatals[0]
+ raise max_fatal
+
+ if tmpReqd:
+ missing = ", ".join(str(e) for e in tmpReqd)
+ raise ParseException(
+ instring,
+ loc,
+ "Missing one or more required elements ({})".format(missing),
+ )
+
+ # add any unmatched Opts, in case they have default values defined
+ matchOrder += [e for e in self.exprs if isinstance(e, Opt) and e.expr in tmpOpt]
+
+ total_results = ParseResults([])
+ for e in matchOrder:
+ loc, results = e._parse(instring, loc, doActions)
+ total_results += results
+
+ return loc, total_results
+
+ def _generateDefaultName(self):
+ return "{" + " & ".join(str(e) for e in self.exprs) + "}"
+
+
+class ParseElementEnhance(ParserElement):
+ """Abstract subclass of :class:`ParserElement`, for combining and
+ post-processing parsed tokens.
+ """
+
+ def __init__(self, expr: Union[ParserElement, str], savelist: bool = False):
+ super().__init__(savelist)
+ if isinstance(expr, str_type):
+ if issubclass(self._literalStringClass, Token):
+ expr = self._literalStringClass(expr)
+ elif issubclass(type(self), self._literalStringClass):
+ expr = Literal(expr)
+ else:
+ expr = self._literalStringClass(Literal(expr))
+ self.expr = expr
+ if expr is not None:
+ self.mayIndexError = expr.mayIndexError
+ self.mayReturnEmpty = expr.mayReturnEmpty
+ self.set_whitespace_chars(
+ expr.whiteChars, copy_defaults=expr.copyDefaultWhiteChars
+ )
+ self.skipWhitespace = expr.skipWhitespace
+ self.saveAsList = expr.saveAsList
+ self.callPreparse = expr.callPreparse
+ self.ignoreExprs.extend(expr.ignoreExprs)
+
+ def recurse(self):
+ return [self.expr] if self.expr is not None else []
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if self.expr is not None:
+ return self.expr._parse(instring, loc, doActions, callPreParse=False)
+ else:
+ raise ParseException("", loc, self.errmsg, self)
+
+ def leave_whitespace(self, recursive=True):
+ super().leave_whitespace(recursive)
+
+ if recursive:
+ self.expr = self.expr.copy()
+ if self.expr is not None:
+ self.expr.leave_whitespace(recursive)
+ return self
+
+ def ignore_whitespace(self, recursive=True):
+ super().ignore_whitespace(recursive)
+
+ if recursive:
+ self.expr = self.expr.copy()
+ if self.expr is not None:
+ self.expr.ignore_whitespace(recursive)
+ return self
+
+ def ignore(self, other):
+ if isinstance(other, Suppress):
+ if other not in self.ignoreExprs:
+ super().ignore(other)
+ if self.expr is not None:
+ self.expr.ignore(self.ignoreExprs[-1])
+ else:
+ super().ignore(other)
+ if self.expr is not None:
+ self.expr.ignore(self.ignoreExprs[-1])
+ return self
+
+ def streamline(self):
+ super().streamline()
+ if self.expr is not None:
+ self.expr.streamline()
+ return self
+
+ def _checkRecursion(self, parseElementList):
+ if self in parseElementList:
+ raise RecursiveGrammarException(parseElementList + [self])
+ subRecCheckList = parseElementList[:] + [self]
+ if self.expr is not None:
+ self.expr._checkRecursion(subRecCheckList)
+
+ def validate(self, validateTrace=None):
+ if validateTrace is None:
+ validateTrace = []
+ tmp = validateTrace[:] + [self]
+ if self.expr is not None:
+ self.expr.validate(tmp)
+ self._checkRecursion([])
+
+ def _generateDefaultName(self):
+ return "{}:({})".format(self.__class__.__name__, str(self.expr))
+
+ ignoreWhitespace = ignore_whitespace
+ leaveWhitespace = leave_whitespace
+
+
+class AtStringStart(ParseElementEnhance):
+ """Matches if expression matches at the beginning of the parse
+ string::
+
+ AtStringStart(Word(nums)).parse_string("123")
+ # prints ["123"]
+
+ AtStringStart(Word(nums)).parse_string(" 123")
+ # raises ParseException
+ """
+
+ def __init__(self, expr: Union[ParserElement, str]):
+ super().__init__(expr)
+ self.callPreparse = False
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if loc != 0:
+ raise ParseException(instring, loc, "not found at string start")
+ return super().parseImpl(instring, loc, doActions)
+
+
+class AtLineStart(ParseElementEnhance):
+ r"""Matches if an expression matches at the beginning of a line within
+ the parse string
+
+ Example::
+
+ test = '''\
+ AAA this line
+ AAA and this line
+ AAA but not this one
+ B AAA and definitely not this one
+ '''
+
+ for t in (AtLineStart('AAA') + restOfLine).search_string(test):
+ print(t)
+
+ prints::
+
+ ['AAA', ' this line']
+ ['AAA', ' and this line']
+
+ """
+
+ def __init__(self, expr: Union[ParserElement, str]):
+ super().__init__(expr)
+ self.callPreparse = False
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if col(loc, instring) != 1:
+ raise ParseException(instring, loc, "not found at line start")
+ return super().parseImpl(instring, loc, doActions)
+
+
+class FollowedBy(ParseElementEnhance):
+ """Lookahead matching of the given parse expression.
+ ``FollowedBy`` does *not* advance the parsing position within
+ the input string, it only verifies that the specified parse
+ expression matches at the current position. ``FollowedBy``
+ always returns a null token list. If any results names are defined
+ in the lookahead expression, those *will* be returned for access by
+ name.
+
+ Example::
+
+ # use FollowedBy to match a label only if it is followed by a ':'
+ data_word = Word(alphas)
+ label = data_word + FollowedBy(':')
+ attr_expr = Group(label + Suppress(':') + OneOrMore(data_word, stop_on=label).set_parse_action(' '.join))
+
+ OneOrMore(attr_expr).parse_string("shape: SQUARE color: BLACK posn: upper left").pprint()
+
+ prints::
+
+ [['shape', 'SQUARE'], ['color', 'BLACK'], ['posn', 'upper left']]
+ """
+
+ def __init__(self, expr: Union[ParserElement, str]):
+ super().__init__(expr)
+ self.mayReturnEmpty = True
+
+ def parseImpl(self, instring, loc, doActions=True):
+ # by using self._expr.parse and deleting the contents of the returned ParseResults list
+ # we keep any named results that were defined in the FollowedBy expression
+ _, ret = self.expr._parse(instring, loc, doActions=doActions)
+ del ret[:]
+
+ return loc, ret
+
+
+class PrecededBy(ParseElementEnhance):
+ """Lookbehind matching of the given parse expression.
+ ``PrecededBy`` does not advance the parsing position within the
+ input string, it only verifies that the specified parse expression
+ matches prior to the current position. ``PrecededBy`` always
+ returns a null token list, but if a results name is defined on the
+ given expression, it is returned.
+
+ Parameters:
+
+ - expr - expression that must match prior to the current parse
+ location
+ - retreat - (default= ``None``) - (int) maximum number of characters
+ to lookbehind prior to the current parse location
+
+ If the lookbehind expression is a string, :class:`Literal`,
+ :class:`Keyword`, or a :class:`Word` or :class:`CharsNotIn`
+ with a specified exact or maximum length, then the retreat
+ parameter is not required. Otherwise, retreat must be specified to
+ give a maximum number of characters to look back from
+ the current parse position for a lookbehind match.
+
+ Example::
+
+ # VB-style variable names with type prefixes
+ int_var = PrecededBy("#") + pyparsing_common.identifier
+ str_var = PrecededBy("$") + pyparsing_common.identifier
+
+ """
+
+ def __init__(
+ self, expr: Union[ParserElement, str], retreat: OptionalType[int] = None
+ ):
+ super().__init__(expr)
+ self.expr = self.expr().leave_whitespace()
+ self.mayReturnEmpty = True
+ self.mayIndexError = False
+ self.exact = False
+ if isinstance(expr, str_type):
+ retreat = len(expr)
+ self.exact = True
+ elif isinstance(expr, (Literal, Keyword)):
+ retreat = expr.matchLen
+ self.exact = True
+ elif isinstance(expr, (Word, CharsNotIn)) and expr.maxLen != _MAX_INT:
+ retreat = expr.maxLen
+ self.exact = True
+ elif isinstance(expr, _PositionToken):
+ retreat = 0
+ self.exact = True
+ self.retreat = retreat
+ self.errmsg = "not preceded by " + str(expr)
+ self.skipWhitespace = False
+ self.parseAction.append(lambda s, l, t: t.__delitem__(slice(None, None)))
+
+ def parseImpl(self, instring, loc=0, doActions=True):
+ if self.exact:
+ if loc < self.retreat:
+ raise ParseException(instring, loc, self.errmsg)
+ start = loc - self.retreat
+ _, ret = self.expr._parse(instring, start)
+ else:
+ # retreat specified a maximum lookbehind window, iterate
+ test_expr = self.expr + StringEnd()
+ instring_slice = instring[max(0, loc - self.retreat) : loc]
+ last_expr = ParseException(instring, loc, self.errmsg)
+ for offset in range(1, min(loc, self.retreat + 1) + 1):
+ try:
+ # print('trying', offset, instring_slice, repr(instring_slice[loc - offset:]))
+ _, ret = test_expr._parse(
+ instring_slice, len(instring_slice) - offset
+ )
+ except ParseBaseException as pbe:
+ last_expr = pbe
+ else:
+ break
+ else:
+ raise last_expr
+ return loc, ret
+
+
+class Located(ParseElementEnhance):
+ """
+ Decorates a returned token with its starting and ending
+ locations in the input string.
+
+ This helper adds the following results names:
+
+ - ``locn_start`` - location where matched expression begins
+ - ``locn_end`` - location where matched expression ends
+ - ``value`` - the actual parsed results
+
+ Be careful if the input text contains ``<TAB>`` characters, you
+ may want to call :class:`ParserElement.parse_with_tabs`
+
+ Example::
+
+ wd = Word(alphas)
+ for match in Located(wd).search_string("ljsdf123lksdjjf123lkkjj1222"):
+ print(match)
+
+ prints::
+
+ [0, ['ljsdf'], 5]
+ [8, ['lksdjjf'], 15]
+ [18, ['lkkjj'], 23]
+
+ """
+
+ def parseImpl(self, instring, loc, doActions=True):
+ start = loc
+ loc, tokens = self.expr._parse(instring, start, doActions, callPreParse=False)
+ ret_tokens = ParseResults([start, tokens, loc])
+ ret_tokens["locn_start"] = start
+ ret_tokens["value"] = tokens
+ ret_tokens["locn_end"] = loc
+ if self.resultsName:
+ # must return as a list, so that the name will be attached to the complete group
+ return loc, [ret_tokens]
+ else:
+ return loc, ret_tokens
+
+
+class NotAny(ParseElementEnhance):
+ """
+ Lookahead to disallow matching with the given parse expression.
+ ``NotAny`` does *not* advance the parsing position within the
+ input string, it only verifies that the specified parse expression
+ does *not* match at the current position. Also, ``NotAny`` does
+ *not* skip over leading whitespace. ``NotAny`` always returns
+ a null token list. May be constructed using the ``'~'`` operator.
+
+ Example::
+
+ AND, OR, NOT = map(CaselessKeyword, "AND OR NOT".split())
+
+ # take care not to mistake keywords for identifiers
+ ident = ~(AND | OR | NOT) + Word(alphas)
+ boolean_term = Opt(NOT) + ident
+
+ # very crude boolean expression - to support parenthesis groups and
+ # operation hierarchy, use infix_notation
+ boolean_expr = boolean_term + ZeroOrMore((AND | OR) + boolean_term)
+
+ # integers that are followed by "." are actually floats
+ integer = Word(nums) + ~Char(".")
+ """
+
+ def __init__(self, expr: Union[ParserElement, str]):
+ super().__init__(expr)
+ # do NOT use self.leave_whitespace(), don't want to propagate to exprs
+ # self.leave_whitespace()
+ self.skipWhitespace = False
+
+ self.mayReturnEmpty = True
+ self.errmsg = "Found unwanted token, " + str(self.expr)
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if self.expr.can_parse_next(instring, loc):
+ raise ParseException(instring, loc, self.errmsg, self)
+ return loc, []
+
+ def _generateDefaultName(self):
+ return "~{" + str(self.expr) + "}"
+
+
+class _MultipleMatch(ParseElementEnhance):
+ def __init__(
+ self,
+ expr: ParserElement,
+ stop_on: OptionalType[Union[ParserElement, str]] = None,
+ *,
+ stopOn: OptionalType[Union[ParserElement, str]] = None,
+ ):
+ super().__init__(expr)
+ stopOn = stopOn or stop_on
+ self.saveAsList = True
+ ender = stopOn
+ if isinstance(ender, str_type):
+ ender = self._literalStringClass(ender)
+ self.stopOn(ender)
+
+ def stopOn(self, ender):
+ if isinstance(ender, str_type):
+ ender = self._literalStringClass(ender)
+ self.not_ender = ~ender if ender is not None else None
+ return self
+
+ def parseImpl(self, instring, loc, doActions=True):
+ self_expr_parse = self.expr._parse
+ self_skip_ignorables = self._skipIgnorables
+ check_ender = self.not_ender is not None
+ if check_ender:
+ try_not_ender = self.not_ender.tryParse
+
+ # must be at least one (but first see if we are the stopOn sentinel;
+ # if so, fail)
+ if check_ender:
+ try_not_ender(instring, loc)
+ loc, tokens = self_expr_parse(instring, loc, doActions)
+ try:
+ hasIgnoreExprs = not not self.ignoreExprs
+ while 1:
+ if check_ender:
+ try_not_ender(instring, loc)
+ if hasIgnoreExprs:
+ preloc = self_skip_ignorables(instring, loc)
+ else:
+ preloc = loc
+ loc, tmptokens = self_expr_parse(instring, preloc, doActions)
+ if tmptokens or tmptokens.haskeys():
+ tokens += tmptokens
+ except (ParseException, IndexError):
+ pass
+
+ return loc, tokens
+
+ def _setResultsName(self, name, listAllMatches=False):
+ if __diag__.warn_ungrouped_named_tokens_in_collection:
+ for e in [self.expr] + self.expr.recurse():
+ if isinstance(e, ParserElement) and e.resultsName:
+ warnings.warn(
+ "{}: setting results name {!r} on {} expression "
+ "collides with {!r} on contained expression".format(
+ "warn_ungrouped_named_tokens_in_collection",
+ name,
+ type(self).__name__,
+ e.resultsName,
+ ),
+ stacklevel=3,
+ )
+
+ return super()._setResultsName(name, listAllMatches)
+
+
+class OneOrMore(_MultipleMatch):
+ """
+ Repetition of one or more of the given expression.
+
+ Parameters:
+ - expr - expression that must match one or more times
+ - stop_on - (default= ``None``) - expression for a terminating sentinel
+ (only required if the sentinel would ordinarily match the repetition
+ expression)
+
+ Example::
+
+ data_word = Word(alphas)
+ label = data_word + FollowedBy(':')
+ attr_expr = Group(label + Suppress(':') + OneOrMore(data_word).set_parse_action(' '.join))
+
+ text = "shape: SQUARE posn: upper left color: BLACK"
+ OneOrMore(attr_expr).parse_string(text).pprint() # Fail! read 'color' as data instead of next label -> [['shape', 'SQUARE color']]
+
+ # use stop_on attribute for OneOrMore to avoid reading label string as part of the data
+ attr_expr = Group(label + Suppress(':') + OneOrMore(data_word, stop_on=label).set_parse_action(' '.join))
+ OneOrMore(attr_expr).parse_string(text).pprint() # Better -> [['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'BLACK']]
+
+ # could also be written as
+ (attr_expr * (1,)).parse_string(text).pprint()
+ """
+
+ def _generateDefaultName(self):
+ return "{" + str(self.expr) + "}..."
+
+
+class ZeroOrMore(_MultipleMatch):
+ """
+ Optional repetition of zero or more of the given expression.
+
+ Parameters:
+ - ``expr`` - expression that must match zero or more times
+ - ``stop_on`` - expression for a terminating sentinel
+ (only required if the sentinel would ordinarily match the repetition
+ expression) - (default= ``None``)
+
+ Example: similar to :class:`OneOrMore`
+ """
+
+ def __init__(
+ self,
+ expr: ParserElement,
+ stop_on: OptionalType[Union[ParserElement, str]] = None,
+ *,
+ stopOn: OptionalType[Union[ParserElement, str]] = None,
+ ):
+ super().__init__(expr, stopOn=stopOn or stop_on)
+ self.mayReturnEmpty = True
+
+ def parseImpl(self, instring, loc, doActions=True):
+ try:
+ return super().parseImpl(instring, loc, doActions)
+ except (ParseException, IndexError):
+ return loc, ParseResults([], name=self.resultsName)
+
+ def _generateDefaultName(self):
+ return "[" + str(self.expr) + "]..."
+
+
+class _NullToken:
+ def __bool__(self):
+ return False
+
+ def __str__(self):
+ return ""
+
+
+class Opt(ParseElementEnhance):
+ """
+ Optional matching of the given expression.
+
+ Parameters:
+ - ``expr`` - expression that must match zero or more times
+ - ``default`` (optional) - value to be returned if the optional expression is not found.
+
+ Example::
+
+ # US postal code can be a 5-digit zip, plus optional 4-digit qualifier
+ zip = Combine(Word(nums, exact=5) + Opt('-' + Word(nums, exact=4)))
+ zip.run_tests('''
+ # traditional ZIP code
+ 12345
+
+ # ZIP+4 form
+ 12101-0001
+
+ # invalid ZIP
+ 98765-
+ ''')
+
+ prints::
+
+ # traditional ZIP code
+ 12345
+ ['12345']
+
+ # ZIP+4 form
+ 12101-0001
+ ['12101-0001']
+
+ # invalid ZIP
+ 98765-
+ ^
+ FAIL: Expected end of text (at char 5), (line:1, col:6)
+ """
+
+ __optionalNotMatched = _NullToken()
+
+ def __init__(
+ self, expr: Union[ParserElement, str], default: Any = __optionalNotMatched
+ ):
+ super().__init__(expr, savelist=False)
+ self.saveAsList = self.expr.saveAsList
+ self.defaultValue = default
+ self.mayReturnEmpty = True
+
+ def parseImpl(self, instring, loc, doActions=True):
+ self_expr = self.expr
+ try:
+ loc, tokens = self_expr._parse(instring, loc, doActions, callPreParse=False)
+ except (ParseException, IndexError):
+ default_value = self.defaultValue
+ if default_value is not self.__optionalNotMatched:
+ if self_expr.resultsName:
+ tokens = ParseResults([default_value])
+ tokens[self_expr.resultsName] = default_value
+ else:
+ tokens = [default_value]
+ else:
+ tokens = []
+ return loc, tokens
+
+ def _generateDefaultName(self):
+ inner = str(self.expr)
+ # strip off redundant inner {}'s
+ while len(inner) > 1 and inner[0 :: len(inner) - 1] == "{}":
+ inner = inner[1:-1]
+ return "[" + inner + "]"
+
+
+Optional = Opt
+
+
+class SkipTo(ParseElementEnhance):
+ """
+ Token for skipping over all undefined text until the matched
+ expression is found.
+
+ Parameters:
+ - ``expr`` - target expression marking the end of the data to be skipped
+ - ``include`` - if ``True``, the target expression is also parsed
+ (the skipped text and target expression are returned as a 2-element
+ list) (default= ``False``).
+ - ``ignore`` - (default= ``None``) used to define grammars (typically quoted strings and
+ comments) that might contain false matches to the target expression
+ - ``fail_on`` - (default= ``None``) define expressions that are not allowed to be
+ included in the skipped test; if found before the target expression is found,
+ the :class:`SkipTo` is not a match
+
+ Example::
+
+ report = '''
+ Outstanding Issues Report - 1 Jan 2000
+
+ # | Severity | Description | Days Open
+ -----+----------+-------------------------------------------+-----------
+ 101 | Critical | Intermittent system crash | 6
+ 94 | Cosmetic | Spelling error on Login ('log|n') | 14
+ 79 | Minor | System slow when running too many reports | 47
+ '''
+ integer = Word(nums)
+ SEP = Suppress('|')
+ # use SkipTo to simply match everything up until the next SEP
+ # - ignore quoted strings, so that a '|' character inside a quoted string does not match
+ # - parse action will call token.strip() for each matched token, i.e., the description body
+ string_data = SkipTo(SEP, ignore=quoted_string)
+ string_data.set_parse_action(token_map(str.strip))
+ ticket_expr = (integer("issue_num") + SEP
+ + string_data("sev") + SEP
+ + string_data("desc") + SEP
+ + integer("days_open"))
+
+ for tkt in ticket_expr.search_string(report):
+ print tkt.dump()
+
+ prints::
+
+ ['101', 'Critical', 'Intermittent system crash', '6']
+ - days_open: 6
+ - desc: Intermittent system crash
+ - issue_num: 101
+ - sev: Critical
+ ['94', 'Cosmetic', "Spelling error on Login ('log|n')", '14']
+ - days_open: 14
+ - desc: Spelling error on Login ('log|n')
+ - issue_num: 94
+ - sev: Cosmetic
+ ['79', 'Minor', 'System slow when running too many reports', '47']
+ - days_open: 47
+ - desc: System slow when running too many reports
+ - issue_num: 79
+ - sev: Minor
+ """
+
+ def __init__(
+ self,
+ other: Union[ParserElement, str],
+ include: bool = False,
+ ignore: bool = None,
+ fail_on: OptionalType[Union[ParserElement, str]] = None,
+ *,
+ failOn: Union[ParserElement, str] = None,
+ ):
+ super().__init__(other)
+ failOn = failOn or fail_on
+ self.ignoreExpr = ignore
+ self.mayReturnEmpty = True
+ self.mayIndexError = False
+ self.includeMatch = include
+ self.saveAsList = False
+ if isinstance(failOn, str_type):
+ self.failOn = self._literalStringClass(failOn)
+ else:
+ self.failOn = failOn
+ self.errmsg = "No match found for " + str(self.expr)
+
+ def parseImpl(self, instring, loc, doActions=True):
+ startloc = loc
+ instrlen = len(instring)
+ self_expr_parse = self.expr._parse
+ self_failOn_canParseNext = (
+ self.failOn.canParseNext if self.failOn is not None else None
+ )
+ self_ignoreExpr_tryParse = (
+ self.ignoreExpr.tryParse if self.ignoreExpr is not None else None
+ )
+
+ tmploc = loc
+ while tmploc <= instrlen:
+ if self_failOn_canParseNext is not None:
+ # break if failOn expression matches
+ if self_failOn_canParseNext(instring, tmploc):
+ break
+
+ if self_ignoreExpr_tryParse is not None:
+ # advance past ignore expressions
+ while 1:
+ try:
+ tmploc = self_ignoreExpr_tryParse(instring, tmploc)
+ except ParseBaseException:
+ break
+
+ try:
+ self_expr_parse(instring, tmploc, doActions=False, callPreParse=False)
+ except (ParseException, IndexError):
+ # no match, advance loc in string
+ tmploc += 1
+ else:
+ # matched skipto expr, done
+ break
+
+ else:
+ # ran off the end of the input string without matching skipto expr, fail
+ raise ParseException(instring, loc, self.errmsg, self)
+
+ # build up return values
+ loc = tmploc
+ skiptext = instring[startloc:loc]
+ skipresult = ParseResults(skiptext)
+
+ if self.includeMatch:
+ loc, mat = self_expr_parse(instring, loc, doActions, callPreParse=False)
+ skipresult += mat
+
+ return loc, skipresult
+
+
+class Forward(ParseElementEnhance):
+ """
+ Forward declaration of an expression to be defined later -
+ used for recursive grammars, such as algebraic infix notation.
+ When the expression is known, it is assigned to the ``Forward``
+ variable using the ``'<<'`` operator.
+
+ Note: take care when assigning to ``Forward`` not to overlook
+ precedence of operators.
+
+ Specifically, ``'|'`` has a lower precedence than ``'<<'``, so that::
+
+ fwd_expr << a | b | c
+
+ will actually be evaluated as::
+
+ (fwd_expr << a) | b | c
+
+ thereby leaving b and c out as parseable alternatives. It is recommended that you
+ explicitly group the values inserted into the ``Forward``::
+
+ fwd_expr << (a | b | c)
+
+ Converting to use the ``'<<='`` operator instead will avoid this problem.
+
+ See :class:`ParseResults.pprint` for an example of a recursive
+ parser created using ``Forward``.
+ """
+
+ def __init__(self, other: OptionalType[Union[ParserElement, str]] = None):
+ self.caller_frame = traceback.extract_stack(limit=2)[0]
+ super().__init__(other, savelist=False)
+ self.lshift_line = None
+
+ def __lshift__(self, other):
+ if hasattr(self, "caller_frame"):
+ del self.caller_frame
+ if isinstance(other, str_type):
+ other = self._literalStringClass(other)
+ self.expr = other
+ self.mayIndexError = self.expr.mayIndexError
+ self.mayReturnEmpty = self.expr.mayReturnEmpty
+ self.set_whitespace_chars(
+ self.expr.whiteChars, copy_defaults=self.expr.copyDefaultWhiteChars
+ )
+ self.skipWhitespace = self.expr.skipWhitespace
+ self.saveAsList = self.expr.saveAsList
+ self.ignoreExprs.extend(self.expr.ignoreExprs)
+ self.lshift_line = traceback.extract_stack(limit=2)[-2]
+ return self
+
+ def __ilshift__(self, other):
+ return self << other
+
+ def __or__(self, other):
+ caller_line = traceback.extract_stack(limit=2)[-2]
+ if (
+ __diag__.warn_on_match_first_with_lshift_operator
+ and caller_line == self.lshift_line
+ ):
+ warnings.warn(
+ "using '<<' operator with '|' is probably an error, use '<<='",
+ stacklevel=2,
+ )
+ ret = super().__or__(other)
+ return ret
+
+ def __del__(self):
+ # see if we are getting dropped because of '=' reassignment of var instead of '<<=' or '<<'
+ if self.expr is None and __diag__.warn_on_assignment_to_Forward:
+ warnings.warn_explicit(
+ "Forward defined here but no expression attached later using '<<=' or '<<'",
+ UserWarning,
+ filename=self.caller_frame.filename,
+ lineno=self.caller_frame.lineno,
+ )
+
+ def parseImpl(self, instring, loc, doActions=True):
+ if self.expr is None and __diag__.warn_on_parse_using_empty_Forward:
+ # walk stack until parse_string, scan_string, search_string, or transform_string is found
+ parse_fns = [
+ "parse_string",
+ "scan_string",
+ "search_string",
+ "transform_string",
+ ]
+ tb = traceback.extract_stack(limit=200)
+ for i, frm in enumerate(reversed(tb), start=1):
+ if frm.name in parse_fns:
+ stacklevel = i + 1
+ break
+ else:
+ stacklevel = 2
+ warnings.warn(
+ "Forward expression was never assigned a value, will not parse any input",
+ stacklevel=stacklevel,
+ )
+ if not ParserElement._left_recursion_enabled:
+ return super().parseImpl(instring, loc, doActions)
+ # ## Bounded Recursion algorithm ##
+ # Recursion only needs to be processed at ``Forward`` elements, since they are
+ # the only ones that can actually refer to themselves. The general idea is
+ # to handle recursion stepwise: We start at no recursion, then recurse once,
+ # recurse twice, ..., until more recursion offers no benefit (we hit the bound).
+ #
+ # The "trick" here is that each ``Forward`` gets evaluated in two contexts
+ # - to *match* a specific recursion level, and
+ # - to *search* the bounded recursion level
+ # and the two run concurrently. The *search* must *match* each recursion level
+ # to find the best possible match. This is handled by a memo table, which
+ # provides the previous match to the next level match attempt.
+ #
+ # See also "Left Recursion in Parsing Expression Grammars", Medeiros et al.
+ #
+ # There is a complication since we not only *parse* but also *transform* via
+ # actions: We do not want to run the actions too often while expanding. Thus,
+ # we expand using `doActions=False` and only run `doActions=True` if the next
+ # recursion level is acceptable.
+ with ParserElement.recursion_lock:
+ memo = ParserElement.recursion_memos
+ try:
+ # we are parsing at a specific recursion expansion - use it as-is
+ prev_loc, prev_result = memo[loc, self, doActions]
+ if isinstance(prev_result, Exception):
+ raise prev_result
+ return prev_loc, prev_result.copy()
+ except KeyError:
+ act_key = (loc, self, True)
+ peek_key = (loc, self, False)
+ # we are searching for the best recursion expansion - keep on improving
+ # both `doActions` cases must be tracked separately here!
+ prev_loc, prev_peek = memo[peek_key] = (
+ loc - 1,
+ ParseException(
+ instring, loc, "Forward recursion without base case", self
+ ),
+ )
+ if doActions:
+ memo[act_key] = memo[peek_key]
+ while True:
+ try:
+ new_loc, new_peek = super().parseImpl(instring, loc, False)
+ except ParseException:
+ # we failed before getting any match – do not hide the error
+ if isinstance(prev_peek, Exception):
+ raise
+ new_loc, new_peek = prev_loc, prev_peek
+ # the match did not get better: we are done
+ if new_loc <= prev_loc:
+ if doActions:
+ # replace the match for doActions=False as well,
+ # in case the action did backtrack
+ prev_loc, prev_result = memo[peek_key] = memo[act_key]
+ del memo[peek_key], memo[act_key]
+ return prev_loc, prev_result.copy()
+ del memo[peek_key]
+ return prev_loc, prev_peek.copy()
+ # the match did get better: see if we can improve further
+ else:
+ if doActions:
+ try:
+ memo[act_key] = super().parseImpl(instring, loc, True)
+ except ParseException as e:
+ memo[peek_key] = memo[act_key] = (new_loc, e)
+ raise
+ prev_loc, prev_peek = memo[peek_key] = new_loc, new_peek
+
+ def leave_whitespace(self, recursive=True):
+ self.skipWhitespace = False
+ return self
+
+ def ignore_whitespace(self, recursive=True):
+ self.skipWhitespace = True
+ return self
+
+ def streamline(self):
+ if not self.streamlined:
+ self.streamlined = True
+ if self.expr is not None:
+ self.expr.streamline()
+ return self
+
+ def validate(self, validateTrace=None):
+ if validateTrace is None:
+ validateTrace = []
+
+ if self not in validateTrace:
+ tmp = validateTrace[:] + [self]
+ if self.expr is not None:
+ self.expr.validate(tmp)
+ self._checkRecursion([])
+
+ def _generateDefaultName(self):
+ # Avoid infinite recursion by setting a temporary _defaultName
+ self._defaultName = ": ..."
+
+ # Use the string representation of main expression.
+ retString = "..."
+ try:
+ if self.expr is not None:
+ retString = str(self.expr)[:1000]
+ else:
+ retString = "None"
+ finally:
+ return self.__class__.__name__ + ": " + retString
+
+ def copy(self):
+ if self.expr is not None:
+ return super().copy()
+ else:
+ ret = Forward()
+ ret <<= self
+ return ret
+
+ def _setResultsName(self, name, list_all_matches=False):
+ if __diag__.warn_name_set_on_empty_Forward:
+ if self.expr is None:
+ warnings.warn(
+ "{}: setting results name {!r} on {} expression "
+ "that has no contained expression".format(
+ "warn_name_set_on_empty_Forward", name, type(self).__name__
+ ),
+ stacklevel=3,
+ )
+
+ return super()._setResultsName(name, list_all_matches)
+
+ ignoreWhitespace = ignore_whitespace
+ leaveWhitespace = leave_whitespace
+
+
+class TokenConverter(ParseElementEnhance):
+ """
+ Abstract subclass of :class:`ParseExpression`, for converting parsed results.
+ """
+
+ def __init__(self, expr: Union[ParserElement, str], savelist=False):
+ super().__init__(expr) # , savelist)
+ self.saveAsList = False
+
+
+class Combine(TokenConverter):
+ """Converter to concatenate all matching tokens to a single string.
+ By default, the matching patterns must also be contiguous in the
+ input string; this can be disabled by specifying
+ ``'adjacent=False'`` in the constructor.
+
+ Example::
+
+ real = Word(nums) + '.' + Word(nums)
+ print(real.parse_string('3.1416')) # -> ['3', '.', '1416']
+ # will also erroneously match the following
+ print(real.parse_string('3. 1416')) # -> ['3', '.', '1416']
+
+ real = Combine(Word(nums) + '.' + Word(nums))
+ print(real.parse_string('3.1416')) # -> ['3.1416']
+ # no match when there are internal spaces
+ print(real.parse_string('3. 1416')) # -> Exception: Expected W:(0123...)
+ """
+
+ def __init__(
+ self,
+ expr: ParserElement,
+ join_string: str = "",
+ adjacent: bool = True,
+ *,
+ joinString: OptionalType[str] = None,
+ ):
+ super().__init__(expr)
+ joinString = joinString if joinString is not None else join_string
+ # suppress whitespace-stripping in contained parse expressions, but re-enable it on the Combine itself
+ if adjacent:
+ self.leave_whitespace()
+ self.adjacent = adjacent
+ self.skipWhitespace = True
+ self.joinString = joinString
+ self.callPreparse = True
+
+ def ignore(self, other):
+ if self.adjacent:
+ ParserElement.ignore(self, other)
+ else:
+ super().ignore(other)
+ return self
+
+ def postParse(self, instring, loc, tokenlist):
+ retToks = tokenlist.copy()
+ del retToks[:]
+ retToks += ParseResults(
+ ["".join(tokenlist._asStringList(self.joinString))], modal=self.modalResults
+ )
+
+ if self.resultsName and retToks.haskeys():
+ return [retToks]
+ else:
+ return retToks
+
+
+class Group(TokenConverter):
+ """Converter to return the matched tokens as a list - useful for
+ returning tokens of :class:`ZeroOrMore` and :class:`OneOrMore` expressions.
+
+ The optional ``aslist`` argument when set to True will return the
+ parsed tokens as a Python list instead of a pyparsing ParseResults.
+
+ Example::
+
+ ident = Word(alphas)
+ num = Word(nums)
+ term = ident | num
+ func = ident + Opt(delimited_list(term))
+ print(func.parse_string("fn a, b, 100"))
+ # -> ['fn', 'a', 'b', '100']
+
+ func = ident + Group(Opt(delimited_list(term)))
+ print(func.parse_string("fn a, b, 100"))
+ # -> ['fn', ['a', 'b', '100']]
+ """
+
+ def __init__(self, expr: ParserElement, aslist: bool = False):
+ super().__init__(expr)
+ self.saveAsList = True
+ self._asPythonList = aslist
+
+ def postParse(self, instring, loc, tokenlist):
+ if self._asPythonList:
+ return ParseResults.List(
+ tokenlist.asList()
+ if isinstance(tokenlist, ParseResults)
+ else list(tokenlist)
+ )
+ else:
+ return [tokenlist]
+
+
+class Dict(TokenConverter):
+ """Converter to return a repetitive expression as a list, but also
+ as a dictionary. Each element can also be referenced using the first
+ token in the expression as its key. Useful for tabular report
+ scraping when the first column can be used as a item key.
+
+ The optional ``asdict`` argument when set to True will return the
+ parsed tokens as a Python dict instead of a pyparsing ParseResults.
+
+ Example::
+
+ data_word = Word(alphas)
+ label = data_word + FollowedBy(':')
+
+ text = "shape: SQUARE posn: upper left color: light blue texture: burlap"
+ attr_expr = (label + Suppress(':') + OneOrMore(data_word, stop_on=label).set_parse_action(' '.join))
+
+ # print attributes as plain groups
+ print(OneOrMore(attr_expr).parse_string(text).dump())
+
+ # instead of OneOrMore(expr), parse using Dict(OneOrMore(Group(expr))) - Dict will auto-assign names
+ result = Dict(OneOrMore(Group(attr_expr))).parse_string(text)
+ print(result.dump())
+
+ # access named fields as dict entries, or output as dict
+ print(result['shape'])
+ print(result.as_dict())
+
+ prints::
+
+ ['shape', 'SQUARE', 'posn', 'upper left', 'color', 'light blue', 'texture', 'burlap']
+ [['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'light blue'], ['texture', 'burlap']]
+ - color: light blue
+ - posn: upper left
+ - shape: SQUARE
+ - texture: burlap
+ SQUARE
+ {'color': 'light blue', 'posn': 'upper left', 'texture': 'burlap', 'shape': 'SQUARE'}
+
+ See more examples at :class:`ParseResults` of accessing fields by results name.
+ """
+
+ def __init__(self, expr: ParserElement, asdict: bool = False):
+ super().__init__(expr)
+ self.saveAsList = True
+ self._asPythonDict = asdict
+
+ def postParse(self, instring, loc, tokenlist):
+ for i, tok in enumerate(tokenlist):
+ if len(tok) == 0:
+ continue
+
+ ikey = tok[0]
+ if isinstance(ikey, int):
+ ikey = str(ikey).strip()
+
+ if len(tok) == 1:
+ tokenlist[ikey] = _ParseResultsWithOffset("", i)
+
+ elif len(tok) == 2 and not isinstance(tok[1], ParseResults):
+ tokenlist[ikey] = _ParseResultsWithOffset(tok[1], i)
+
+ else:
+ try:
+ dictvalue = tok.copy() # ParseResults(i)
+ except Exception:
+ exc = TypeError(
+ "could not extract dict values from parsed results"
+ " - Dict expression must contain Grouped expressions"
+ )
+ raise exc from None
+
+ del dictvalue[0]
+
+ if len(dictvalue) != 1 or (
+ isinstance(dictvalue, ParseResults) and dictvalue.haskeys()
+ ):
+ tokenlist[ikey] = _ParseResultsWithOffset(dictvalue, i)
+ else:
+ tokenlist[ikey] = _ParseResultsWithOffset(dictvalue[0], i)
+
+ return tokenlist if not self._asPythonDict else tokenlist.as_dict()
+
+
+class Suppress(TokenConverter):
+ """Converter for ignoring the results of a parsed expression.
+
+ Example::
+
+ source = "a, b, c,d"
+ wd = Word(alphas)
+ wd_list1 = wd + ZeroOrMore(',' + wd)
+ print(wd_list1.parse_string(source))
+
+ # often, delimiters that are useful during parsing are just in the
+ # way afterward - use Suppress to keep them out of the parsed output
+ wd_list2 = wd + ZeroOrMore(Suppress(',') + wd)
+ print(wd_list2.parse_string(source))
+
+ # Skipped text (using '...') can be suppressed as well
+ source = "lead in START relevant text END trailing text"
+ start_marker = Keyword("START")
+ end_marker = Keyword("END")
+ find_body = Suppress(...) + start_marker + ... + end_marker
+ print(find_body.parse_string(source)
+
+ prints::
+
+ ['a', ',', 'b', ',', 'c', ',', 'd']
+ ['a', 'b', 'c', 'd']
+ ['START', 'relevant text ', 'END']
+
+ (See also :class:`delimited_list`.)
+ """
+
+ def __init__(self, expr: Union[ParserElement, str], savelist: bool = False):
+ if expr is ...:
+ expr = _PendingSkip(NoMatch())
+ super().__init__(expr)
+
+ def __add__(self, other):
+ if isinstance(self.expr, _PendingSkip):
+ return Suppress(SkipTo(other)) + other
+ else:
+ return super().__add__(other)
+
+ def __sub__(self, other):
+ if isinstance(self.expr, _PendingSkip):
+ return Suppress(SkipTo(other)) - other
+ else:
+ return super().__sub__(other)
+
+ def postParse(self, instring, loc, tokenlist):
+ return []
+
+ def suppress(self):
+ return self
+
+
+def trace_parse_action(f: ParseAction):
+ """Decorator for debugging parse actions.
+
+ When the parse action is called, this decorator will print
+ ``">> entering method-name(line:<current_source_line>, <parse_location>, <matched_tokens>)"``.
+ When the parse action completes, the decorator will print
+ ``"<<"`` followed by the returned value, or any exception that the parse action raised.
+
+ Example::
+
+ wd = Word(alphas)
+
+ @trace_parse_action
+ def remove_duplicate_chars(tokens):
+ return ''.join(sorted(set(''.join(tokens))))
+
+ wds = OneOrMore(wd).set_parse_action(remove_duplicate_chars)
+ print(wds.parse_string("slkdjs sld sldd sdlf sdljf"))
+
+ prints::
+
+ >>entering remove_duplicate_chars(line: 'slkdjs sld sldd sdlf sdljf', 0, (['slkdjs', 'sld', 'sldd', 'sdlf', 'sdljf'], {}))
+ <<leaving remove_duplicate_chars (ret: 'dfjkls')
+ ['dfjkls']
+ """
+ f = _trim_arity(f)
+
+ def z(*paArgs):
+ thisFunc = f.__name__
+ s, l, t = paArgs[-3:]
+ if len(paArgs) > 3:
+ thisFunc = paArgs[0].__class__.__name__ + "." + thisFunc
+ sys.stderr.write(
+ ">>entering {}(line: {!r}, {}, {!r})\n".format(thisFunc, line(l, s), l, t)
+ )
+ try:
+ ret = f(*paArgs)
+ except Exception as exc:
+ sys.stderr.write("<<leaving {} (exception: {})\n".format(thisFunc, exc))
+ raise
+ sys.stderr.write("<<leaving {} (ret: {!r})\n".format(thisFunc, ret))
+ return ret
+
+ z.__name__ = f.__name__
+ return z
+
+
+# convenience constants for positional expressions
+empty = Empty().set_name("empty")
+line_start = LineStart().set_name("line_start")
+line_end = LineEnd().set_name("line_end")
+string_start = StringStart().set_name("string_start")
+string_end = StringEnd().set_name("string_end")
+
+_escapedPunc = Word(_bslash, r"\[]-*.$+^?()~ ", exact=2).set_parse_action(
+ lambda s, l, t: t[0][1]
+)
+_escapedHexChar = Regex(r"\\0?[xX][0-9a-fA-F]+").set_parse_action(
+ lambda s, l, t: chr(int(t[0].lstrip(r"\0x"), 16))
+)
+_escapedOctChar = Regex(r"\\0[0-7]+").set_parse_action(
+ lambda s, l, t: chr(int(t[0][1:], 8))
+)
+_singleChar = (
+ _escapedPunc | _escapedHexChar | _escapedOctChar | CharsNotIn(r"\]", exact=1)
+)
+_charRange = Group(_singleChar + Suppress("-") + _singleChar)
+_reBracketExpr = (
+ Literal("[")
+ + Opt("^").set_results_name("negate")
+ + Group(OneOrMore(_charRange | _singleChar)).set_results_name("body")
+ + "]"
+)
+
+
+def srange(s):
+ r"""Helper to easily define string ranges for use in :class:`Word`
+ construction. Borrows syntax from regexp ``'[]'`` string range
+ definitions::
+
+ srange("[0-9]") -> "0123456789"
+ srange("[a-z]") -> "abcdefghijklmnopqrstuvwxyz"
+ srange("[a-z$_]") -> "abcdefghijklmnopqrstuvwxyz$_"
+
+ The input string must be enclosed in []'s, and the returned string
+ is the expanded character set joined into a single string. The
+ values enclosed in the []'s may be:
+
+ - a single character
+ - an escaped character with a leading backslash (such as ``\-``
+ or ``\]``)
+ - an escaped hex character with a leading ``'\x'``
+ (``\x21``, which is a ``'!'`` character) (``\0x##``
+ is also supported for backwards compatibility)
+ - an escaped octal character with a leading ``'\0'``
+ (``\041``, which is a ``'!'`` character)
+ - a range of any of the above, separated by a dash (``'a-z'``,
+ etc.)
+ - any combination of the above (``'aeiouy'``,
+ ``'a-zA-Z0-9_$'``, etc.)
+ """
+ _expanded = (
+ lambda p: p
+ if not isinstance(p, ParseResults)
+ else "".join(chr(c) for c in range(ord(p[0]), ord(p[1]) + 1))
+ )
+ try:
+ return "".join(_expanded(part) for part in _reBracketExpr.parse_string(s).body)
+ except Exception:
+ return ""
+
+
+def token_map(func, *args):
+ """Helper to define a parse action by mapping a function to all
+ elements of a :class:`ParseResults` list. If any additional args are passed,
+ they are forwarded to the given function as additional arguments
+ after the token, as in
+ ``hex_integer = Word(hexnums).set_parse_action(token_map(int, 16))``,
+ which will convert the parsed data to an integer using base 16.
+
+ Example (compare the last to example in :class:`ParserElement.transform_string`::
+
+ hex_ints = OneOrMore(Word(hexnums)).set_parse_action(token_map(int, 16))
+ hex_ints.run_tests('''
+ 00 11 22 aa FF 0a 0d 1a
+ ''')
+
+ upperword = Word(alphas).set_parse_action(token_map(str.upper))
+ OneOrMore(upperword).run_tests('''
+ my kingdom for a horse
+ ''')
+
+ wd = Word(alphas).set_parse_action(token_map(str.title))
+ OneOrMore(wd).set_parse_action(' '.join).run_tests('''
+ now is the winter of our discontent made glorious summer by this sun of york
+ ''')
+
+ prints::
+
+ 00 11 22 aa FF 0a 0d 1a
+ [0, 17, 34, 170, 255, 10, 13, 26]
+
+ my kingdom for a horse
+ ['MY', 'KINGDOM', 'FOR', 'A', 'HORSE']
+
+ now is the winter of our discontent made glorious summer by this sun of york
+ ['Now Is The Winter Of Our Discontent Made Glorious Summer By This Sun Of York']
+ """
+
+ def pa(s, l, t):
+ return [func(tokn, *args) for tokn in t]
+
+ func_name = getattr(func, "__name__", getattr(func, "__class__").__name__)
+ pa.__name__ = func_name
+
+ return pa
+
+
+def autoname_elements():
+ """
+ Utility to simplify mass-naming of parser elements, for
+ generating railroad diagram with named subdiagrams.
+ """
+ for name, var in sys._getframe().f_back.f_locals.items():
+ if isinstance(var, ParserElement) and not var.customName:
+ var.set_name(name)
+
+
+dbl_quoted_string = Combine(
+ Regex(r'"(?:[^"\n\r\\]|(?:"")|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*') + '"'
+).set_name("string enclosed in double quotes")
+
+sgl_quoted_string = Combine(
+ Regex(r"'(?:[^'\n\r\\]|(?:'')|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*") + "'"
+).set_name("string enclosed in single quotes")
+
+quoted_string = Combine(
+ Regex(r'"(?:[^"\n\r\\]|(?:"")|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*') + '"'
+ | Regex(r"'(?:[^'\n\r\\]|(?:'')|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*") + "'"
+).set_name("quotedString using single or double quotes")
+
+unicode_string = Combine("u" + quoted_string.copy()).set_name("unicode string literal")
+
+
+alphas8bit = srange(r"[\0xc0-\0xd6\0xd8-\0xf6\0xf8-\0xff]")
+punc8bit = srange(r"[\0xa1-\0xbf\0xd7\0xf7]")
+
+# build list of built-in expressions, for future reference if a global default value
+# gets updated
+_builtin_exprs = [v for v in vars().values() if isinstance(v, ParserElement)]
+
+# backward compatibility names
+tokenMap = token_map
+conditionAsParseAction = condition_as_parse_action
+nullDebugAction = null_debug_action
+sglQuotedString = sgl_quoted_string
+dblQuotedString = dbl_quoted_string
+quotedString = quoted_string
+unicodeString = unicode_string
+lineStart = line_start
+lineEnd = line_end
+stringStart = string_start
+stringEnd = string_end
+traceParseAction = trace_parse_action
--- /dev/null
+import railroad
+import pyparsing
+from pkg_resources import resource_filename
+from typing import (
+ List,
+ Optional,
+ NamedTuple,
+ Generic,
+ TypeVar,
+ Dict,
+ Callable,
+ Set,
+)
+from jinja2 import Template
+from io import StringIO
+import inspect
+
+with open(resource_filename(__name__, "template.jinja2"), encoding="utf-8") as fp:
+ template = Template(fp.read())
+
+# Note: ideally this would be a dataclass, but we're supporting Python 3.5+ so we can't do this yet
+NamedDiagram = NamedTuple(
+ "NamedDiagram",
+ [("name", str), ("diagram", Optional[railroad.DiagramItem]), ("index", int)],
+)
+"""
+A simple structure for associating a name with a railroad diagram
+"""
+
+T = TypeVar("T")
+
+
+class EachItem(railroad.Group):
+ """
+ Custom railroad item to compose a:
+ - Group containing a
+ - OneOrMore containing a
+ - Choice of the elements in the Each
+ with the group label indicating that all must be matched
+ """
+
+ all_label = "[ALL]"
+
+ def __init__(self, *items):
+ choice_item = railroad.Choice(len(items) - 1, *items)
+ one_or_more_item = railroad.OneOrMore(item=choice_item)
+ super().__init__(one_or_more_item, label=self.all_label)
+
+
+class AnnotatedItem(railroad.Group):
+ """
+ Simple subclass of Group that creates an annotation label
+ """
+
+ def __init__(self, label: str, item):
+ super().__init__(item=item, label="[{}]".format(label))
+
+
+class EditablePartial(Generic[T]):
+ """
+ Acts like a functools.partial, but can be edited. In other words, it represents a type that hasn't yet been
+ constructed.
+ """
+
+ # We need this here because the railroad constructors actually transform the data, so can't be called until the
+ # entire tree is assembled
+
+ def __init__(self, func: Callable[..., T], args: list, kwargs: dict):
+ self.func = func
+ self.args = args
+ self.kwargs = kwargs
+
+ @classmethod
+ def from_call(cls, func: Callable[..., T], *args, **kwargs) -> "EditablePartial[T]":
+ """
+ If you call this function in the same way that you would call the constructor, it will store the arguments
+ as you expect. For example EditablePartial.from_call(Fraction, 1, 3)() == Fraction(1, 3)
+ """
+ return EditablePartial(func=func, args=list(args), kwargs=kwargs)
+
+ @property
+ def name(self):
+ return self.kwargs["name"]
+
+ def __call__(self) -> T:
+ """
+ Evaluate the partial and return the result
+ """
+ args = self.args.copy()
+ kwargs = self.kwargs.copy()
+
+ # This is a helpful hack to allow you to specify varargs parameters (e.g. *args) as keyword args (e.g.
+ # args=['list', 'of', 'things'])
+ arg_spec = inspect.getfullargspec(self.func)
+ if arg_spec.varargs in self.kwargs:
+ args += kwargs.pop(arg_spec.varargs)
+
+ return self.func(*args, **kwargs)
+
+
+def railroad_to_html(diagrams: List[NamedDiagram], **kwargs) -> str:
+ """
+ Given a list of NamedDiagram, produce a single HTML string that visualises those diagrams
+ :params kwargs: kwargs to be passed in to the template
+ """
+ data = []
+ for diagram in diagrams:
+ io = StringIO()
+ diagram.diagram.writeSvg(io.write)
+ title = diagram.name
+ if diagram.index == 0:
+ title += " (root)"
+ data.append({"title": title, "text": "", "svg": io.getvalue()})
+
+ return template.render(diagrams=data, **kwargs)
+
+
+def resolve_partial(partial: "EditablePartial[T]") -> T:
+ """
+ Recursively resolves a collection of Partials into whatever type they are
+ """
+ if isinstance(partial, EditablePartial):
+ partial.args = resolve_partial(partial.args)
+ partial.kwargs = resolve_partial(partial.kwargs)
+ return partial()
+ elif isinstance(partial, list):
+ return [resolve_partial(x) for x in partial]
+ elif isinstance(partial, dict):
+ return {key: resolve_partial(x) for key, x in partial.items()}
+ else:
+ return partial
+
+
+def to_railroad(
+ element: pyparsing.ParserElement,
+ diagram_kwargs: Optional[dict] = None,
+ vertical: int = 3,
+ show_results_names: bool = False,
+) -> List[NamedDiagram]:
+ """
+ Convert a pyparsing element tree into a list of diagrams. This is the recommended entrypoint to diagram
+ creation if you want to access the Railroad tree before it is converted to HTML
+ :param element: base element of the parser being diagrammed
+ :param diagram_kwargs: kwargs to pass to the Diagram() constructor
+ :param vertical: (optional) - int - limit at which number of alternatives should be
+ shown vertically instead of horizontally
+ :param show_results_names - bool to indicate whether results name annotations should be
+ included in the diagram
+ """
+ # Convert the whole tree underneath the root
+ lookup = ConverterState(diagram_kwargs=diagram_kwargs or {})
+ _to_diagram_element(
+ element,
+ lookup=lookup,
+ parent=None,
+ vertical=vertical,
+ show_results_names=show_results_names,
+ )
+
+ root_id = id(element)
+ # Convert the root if it hasn't been already
+ if root_id in lookup:
+ if not element.customName:
+ lookup[root_id].name = ""
+ lookup[root_id].mark_for_extraction(root_id, lookup, force=True)
+
+ # Now that we're finished, we can convert from intermediate structures into Railroad elements
+ diags = list(lookup.diagrams.values())
+ if len(diags) > 1:
+ # collapse out duplicate diags with the same name
+ seen = set()
+ deduped_diags = []
+ for d in diags:
+ # don't extract SkipTo elements, they are uninformative as subdiagrams
+ if d.name == "...":
+ continue
+ if d.name is not None and d.name not in seen:
+ seen.add(d.name)
+ deduped_diags.append(d)
+ resolved = [resolve_partial(partial) for partial in deduped_diags]
+ else:
+ # special case - if just one diagram, always display it, even if
+ # it has no name
+ resolved = [resolve_partial(partial) for partial in diags]
+ return sorted(resolved, key=lambda diag: diag.index)
+
+
+def _should_vertical(specification: int, count: int) -> bool:
+ """
+ Returns true if we should return a vertical list of elements
+ """
+ if specification is None:
+ return False
+ else:
+ return count >= specification
+
+
+class ElementState:
+ """
+ State recorded for an individual pyparsing Element
+ """
+
+ # Note: this should be a dataclass, but we have to support Python 3.5
+ def __init__(
+ self,
+ element: pyparsing.ParserElement,
+ converted: EditablePartial,
+ parent: EditablePartial,
+ number: int,
+ name: str = None,
+ parent_index: Optional[int] = None,
+ ):
+ #: The pyparsing element that this represents
+ self.element: pyparsing.ParserElement = element
+ #: The name of the element
+ self.name: str = name
+ #: The output Railroad element in an unconverted state
+ self.converted: EditablePartial = converted
+ #: The parent Railroad element, which we store so that we can extract this if it's duplicated
+ self.parent: EditablePartial = parent
+ #: The order in which we found this element, used for sorting diagrams if this is extracted into a diagram
+ self.number: int = number
+ #: The index of this inside its parent
+ self.parent_index: Optional[int] = parent_index
+ #: If true, we should extract this out into a subdiagram
+ self.extract: bool = False
+ #: If true, all of this element's children have been filled out
+ self.complete: bool = False
+
+ def mark_for_extraction(
+ self, el_id: int, state: "ConverterState", name: str = None, force: bool = False
+ ):
+ """
+ Called when this instance has been seen twice, and thus should eventually be extracted into a sub-diagram
+ :param el_id: id of the element
+ :param state: element/diagram state tracker
+ :param name: name to use for this element's text
+ :param force: If true, force extraction now, regardless of the state of this. Only useful for extracting the
+ root element when we know we're finished
+ """
+ self.extract = True
+
+ # Set the name
+ if not self.name:
+ if name:
+ # Allow forcing a custom name
+ self.name = name
+ elif self.element.customName:
+ self.name = self.element.customName
+ else:
+ self.name = ""
+
+ # Just because this is marked for extraction doesn't mean we can do it yet. We may have to wait for children
+ # to be added
+ # Also, if this is just a string literal etc, don't bother extracting it
+ if force or (self.complete and _worth_extracting(self.element)):
+ state.extract_into_diagram(el_id)
+
+
+class ConverterState:
+ """
+ Stores some state that persists between recursions into the element tree
+ """
+
+ def __init__(self, diagram_kwargs: Optional[dict] = None):
+ #: A dictionary mapping ParserElements to state relating to them
+ self._element_diagram_states: Dict[int, ElementState] = {}
+ #: A dictionary mapping ParserElement IDs to subdiagrams generated from them
+ self.diagrams: Dict[int, EditablePartial[NamedDiagram]] = {}
+ #: The index of the next unnamed element
+ self.unnamed_index: int = 1
+ #: The index of the next element. This is used for sorting
+ self.index: int = 0
+ #: Shared kwargs that are used to customize the construction of diagrams
+ self.diagram_kwargs: dict = diagram_kwargs or {}
+ self.extracted_diagram_names: Set[str] = set()
+
+ def __setitem__(self, key: int, value: ElementState):
+ self._element_diagram_states[key] = value
+
+ def __getitem__(self, key: int) -> ElementState:
+ return self._element_diagram_states[key]
+
+ def __delitem__(self, key: int):
+ del self._element_diagram_states[key]
+
+ def __contains__(self, key: int):
+ return key in self._element_diagram_states
+
+ def generate_unnamed(self) -> int:
+ """
+ Generate a number used in the name of an otherwise unnamed diagram
+ """
+ self.unnamed_index += 1
+ return self.unnamed_index
+
+ def generate_index(self) -> int:
+ """
+ Generate a number used to index a diagram
+ """
+ self.index += 1
+ return self.index
+
+ def extract_into_diagram(self, el_id: int):
+ """
+ Used when we encounter the same token twice in the same tree. When this
+ happens, we replace all instances of that token with a terminal, and
+ create a new subdiagram for the token
+ """
+ position = self[el_id]
+
+ # Replace the original definition of this element with a regular block
+ if position.parent:
+ ret = EditablePartial.from_call(railroad.NonTerminal, text=position.name)
+ if "item" in position.parent.kwargs:
+ position.parent.kwargs["item"] = ret
+ elif "items" in position.parent.kwargs:
+ position.parent.kwargs["items"][position.parent_index] = ret
+
+ # If the element we're extracting is a group, skip to its content but keep the title
+ if position.converted.func == railroad.Group:
+ content = position.converted.kwargs["item"]
+ else:
+ content = position.converted
+
+ self.diagrams[el_id] = EditablePartial.from_call(
+ NamedDiagram,
+ name=position.name,
+ diagram=EditablePartial.from_call(
+ railroad.Diagram, content, **self.diagram_kwargs
+ ),
+ index=position.number,
+ )
+
+ del self[el_id]
+
+
+def _worth_extracting(element: pyparsing.ParserElement) -> bool:
+ """
+ Returns true if this element is worth having its own sub-diagram. Simply, if any of its children
+ themselves have children, then its complex enough to extract
+ """
+ children = element.recurse()
+ return any(child.recurse() for child in children)
+
+
+def _apply_diagram_item_enhancements(fn):
+ """
+ decorator to ensure enhancements to a diagram item (such as results name annotations)
+ get applied on return from _to_diagram_element (we do this since there are several
+ returns in _to_diagram_element)
+ """
+
+ def _inner(
+ element: pyparsing.ParserElement,
+ parent: Optional[EditablePartial],
+ lookup: ConverterState = None,
+ vertical: int = None,
+ index: int = 0,
+ name_hint: str = None,
+ show_results_names: bool = False,
+ ) -> Optional[EditablePartial]:
+
+ ret = fn(
+ element,
+ parent,
+ lookup,
+ vertical,
+ index,
+ name_hint,
+ show_results_names,
+ )
+
+ # apply annotation for results name, if present
+ if show_results_names and ret is not None:
+ element_results_name = element.resultsName
+ if element_results_name:
+ # add "*" to indicate if this is a "list all results" name
+ element_results_name += "" if element.modalResults else "*"
+ ret = EditablePartial.from_call(
+ railroad.Group, item=ret, label=element_results_name
+ )
+
+ return ret
+
+ return _inner
+
+
+@_apply_diagram_item_enhancements
+def _to_diagram_element(
+ element: pyparsing.ParserElement,
+ parent: Optional[EditablePartial],
+ lookup: ConverterState = None,
+ vertical: int = None,
+ index: int = 0,
+ name_hint: str = None,
+ show_results_names: bool = False,
+) -> Optional[EditablePartial]:
+ """
+ Recursively converts a PyParsing Element to a railroad Element
+ :param lookup: The shared converter state that keeps track of useful things
+ :param index: The index of this element within the parent
+ :param parent: The parent of this element in the output tree
+ :param vertical: Controls at what point we make a list of elements vertical. If this is an integer (the default),
+ it sets the threshold of the number of items before we go vertical. If True, always go vertical, if False, never
+ do so
+ :param name_hint: If provided, this will override the generated name
+ :param show_results_names: bool flag indicating whether to add annotations for results names
+ :returns: The converted version of the input element, but as a Partial that hasn't yet been constructed
+ """
+ exprs = element.recurse()
+ name = name_hint or element.customName or element.__class__.__name__
+
+ # Python's id() is used to provide a unique identifier for elements
+ el_id = id(element)
+
+ element_results_name = element.resultsName
+
+ # Here we basically bypass processing certain wrapper elements if they contribute nothing to the diagram
+ if not element.customName:
+ if isinstance(
+ element,
+ (
+ pyparsing.TokenConverter,
+ # pyparsing.Forward,
+ pyparsing.Located,
+ ),
+ ):
+ # However, if this element has a useful custom name, and its child does not, we can pass it on to the child
+ if exprs:
+ if not exprs[0].customName:
+ propagated_name = name
+ else:
+ propagated_name = None
+
+ return _to_diagram_element(
+ element.expr,
+ parent=parent,
+ lookup=lookup,
+ vertical=vertical,
+ index=index,
+ name_hint=propagated_name,
+ show_results_names=show_results_names,
+ )
+
+ # If the element isn't worth extracting, we always treat it as the first time we say it
+ if _worth_extracting(element):
+ if el_id in lookup:
+ # If we've seen this element exactly once before, we are only just now finding out that it's a duplicate,
+ # so we have to extract it into a new diagram.
+ looked_up = lookup[el_id]
+ looked_up.mark_for_extraction(el_id, lookup, name=name_hint)
+ ret = EditablePartial.from_call(railroad.NonTerminal, text=looked_up.name)
+ return ret
+
+ elif el_id in lookup.diagrams:
+ # If we have seen the element at least twice before, and have already extracted it into a subdiagram, we
+ # just put in a marker element that refers to the sub-diagram
+ ret = EditablePartial.from_call(
+ railroad.NonTerminal, text=lookup.diagrams[el_id].kwargs["name"]
+ )
+ return ret
+
+ # Recursively convert child elements
+ # Here we find the most relevant Railroad element for matching pyparsing Element
+ # We use ``items=[]`` here to hold the place for where the child elements will go once created
+ if isinstance(element, pyparsing.And):
+ # detect And's created with ``expr*N`` notation - for these use a OneOrMore with a repeat
+ # (all will have the same name, and resultsName)
+ if not exprs:
+ return None
+ if len(set((e.name, e.resultsName) for e in exprs)) == 1:
+ ret = EditablePartial.from_call(
+ railroad.OneOrMore, item="", repeat=str(len(exprs))
+ )
+ elif _should_vertical(vertical, len(exprs)):
+ ret = EditablePartial.from_call(railroad.Stack, items=[])
+ else:
+ ret = EditablePartial.from_call(railroad.Sequence, items=[])
+ elif isinstance(element, (pyparsing.Or, pyparsing.MatchFirst)):
+ if not exprs:
+ return None
+ if _should_vertical(vertical, len(exprs)):
+ ret = EditablePartial.from_call(railroad.Choice, 0, items=[])
+ else:
+ ret = EditablePartial.from_call(railroad.HorizontalChoice, items=[])
+ elif isinstance(element, pyparsing.Each):
+ if not exprs:
+ return None
+ ret = EditablePartial.from_call(EachItem, items=[])
+ elif isinstance(element, pyparsing.NotAny):
+ ret = EditablePartial.from_call(AnnotatedItem, label="NOT", item="")
+ elif isinstance(element, pyparsing.FollowedBy):
+ ret = EditablePartial.from_call(AnnotatedItem, label="LOOKAHEAD", item="")
+ elif isinstance(element, pyparsing.PrecededBy):
+ ret = EditablePartial.from_call(AnnotatedItem, label="LOOKBEHIND", item="")
+ elif isinstance(element, pyparsing.Opt):
+ ret = EditablePartial.from_call(railroad.Optional, item="")
+ elif isinstance(element, pyparsing.OneOrMore):
+ ret = EditablePartial.from_call(railroad.OneOrMore, item="")
+ elif isinstance(element, pyparsing.ZeroOrMore):
+ ret = EditablePartial.from_call(railroad.ZeroOrMore, item="")
+ elif isinstance(element, pyparsing.Group):
+ ret = EditablePartial.from_call(
+ railroad.Group, item=None, label=element_results_name
+ )
+ elif isinstance(element, pyparsing.Empty) and not element.customName:
+ # Skip unnamed "Empty" elements
+ ret = None
+ elif len(exprs) > 1:
+ ret = EditablePartial.from_call(railroad.Sequence, items=[])
+ elif len(exprs) > 0 and not element_results_name:
+ ret = EditablePartial.from_call(railroad.Group, item="", label=name)
+ else:
+ terminal = EditablePartial.from_call(railroad.Terminal, element.defaultName)
+ ret = terminal
+
+ if ret is None:
+ return
+
+ # Indicate this element's position in the tree so we can extract it if necessary
+ lookup[el_id] = ElementState(
+ element=element,
+ converted=ret,
+ parent=parent,
+ parent_index=index,
+ number=lookup.generate_index(),
+ )
+ if element.customName:
+ lookup[el_id].mark_for_extraction(el_id, lookup, element.customName)
+
+ i = 0
+ for expr in exprs:
+ # Add a placeholder index in case we have to extract the child before we even add it to the parent
+ if "items" in ret.kwargs:
+ ret.kwargs["items"].insert(i, None)
+
+ item = _to_diagram_element(
+ expr,
+ parent=ret,
+ lookup=lookup,
+ vertical=vertical,
+ index=i,
+ show_results_names=show_results_names,
+ )
+
+ # Some elements don't need to be shown in the diagram
+ if item is not None:
+ if "item" in ret.kwargs:
+ ret.kwargs["item"] = item
+ elif "items" in ret.kwargs:
+ # If we've already extracted the child, don't touch this index, since it's occupied by a nonterminal
+ ret.kwargs["items"][i] = item
+ i += 1
+ elif "items" in ret.kwargs:
+ # If we're supposed to skip this element, remove it from the parent
+ del ret.kwargs["items"][i]
+
+ # If all this items children are none, skip this item
+ if ret and (
+ ("items" in ret.kwargs and len(ret.kwargs["items"]) == 0)
+ or ("item" in ret.kwargs and ret.kwargs["item"] is None)
+ ):
+ ret = EditablePartial.from_call(railroad.Terminal, name)
+
+ # Mark this element as "complete", ie it has all of its children
+ if el_id in lookup:
+ lookup[el_id].complete = True
+
+ if el_id in lookup and lookup[el_id].extract and lookup[el_id].complete:
+ lookup.extract_into_diagram(el_id)
+ if ret is not None:
+ ret = EditablePartial.from_call(
+ railroad.NonTerminal, text=lookup.diagrams[el_id].kwargs["name"]
+ )
+
+ return ret
--- /dev/null
+<!DOCTYPE html>
+<html>
+<head>
+ {% if not head %}
+ <style type="text/css">
+ .railroad-heading {
+ font-family: monospace;
+ }
+ </style>
+ {% else %}
+ {{ hear | safe }}
+ {% endif %}
+</head>
+<body>
+{{ body | safe }}
+{% for diagram in diagrams %}
+ <div class="railroad-group">
+ <h1 class="railroad-heading">{{ diagram.title }}</h1>
+ <div class="railroad-description">{{ diagram.text }}</div>
+ <div class="railroad-svg">
+ {{ diagram.svg }}
+ </div>
+ </div>
+{% endfor %}
+</body>
+</html>
--- /dev/null
+# exceptions.py
+
+import re
+import sys
+from typing import Optional
+
+from .util import col, line, lineno, _collapseStringToRanges
+from .unicode import pyparsing_unicode as ppu
+
+
+class ExceptionWordUnicode(ppu.Latin1, ppu.LatinA, ppu.LatinB, ppu.Greek, ppu.Cyrillic):
+ pass
+
+
+_extract_alphanums = _collapseStringToRanges(ExceptionWordUnicode.alphanums)
+_exception_word_extractor = re.compile("([" + _extract_alphanums + "]{1,16})|.")
+
+
+class ParseBaseException(Exception):
+ """base exception class for all parsing runtime exceptions"""
+
+ # Performance tuning: we construct a *lot* of these, so keep this
+ # constructor as small and fast as possible
+ def __init__(
+ self,
+ pstr: str,
+ loc: int = 0,
+ msg: Optional[str] = None,
+ elem=None,
+ ):
+ self.loc = loc
+ if msg is None:
+ self.msg = pstr
+ self.pstr = ""
+ else:
+ self.msg = msg
+ self.pstr = pstr
+ self.parser_element = self.parserElement = elem
+ self.args = (pstr, loc, msg)
+
+ @staticmethod
+ def explain_exception(exc, depth=16):
+ """
+ Method to take an exception and translate the Python internal traceback into a list
+ of the pyparsing expressions that caused the exception to be raised.
+
+ Parameters:
+
+ - exc - exception raised during parsing (need not be a ParseException, in support
+ of Python exceptions that might be raised in a parse action)
+ - depth (default=16) - number of levels back in the stack trace to list expression
+ and function names; if None, the full stack trace names will be listed; if 0, only
+ the failing input line, marker, and exception string will be shown
+
+ Returns a multi-line string listing the ParserElements and/or function names in the
+ exception's stack trace.
+ """
+ import inspect
+ from .core import ParserElement
+
+ if depth is None:
+ depth = sys.getrecursionlimit()
+ ret = []
+ if isinstance(exc, ParseBaseException):
+ ret.append(exc.line)
+ ret.append(" " * (exc.column - 1) + "^")
+ ret.append("{}: {}".format(type(exc).__name__, exc))
+
+ if depth > 0:
+ callers = inspect.getinnerframes(exc.__traceback__, context=depth)
+ seen = set()
+ for i, ff in enumerate(callers[-depth:]):
+ frm = ff[0]
+
+ f_self = frm.f_locals.get("self", None)
+ if isinstance(f_self, ParserElement):
+ if frm.f_code.co_name not in ("parseImpl", "_parseNoCache"):
+ continue
+ if id(f_self) in seen:
+ continue
+ seen.add(id(f_self))
+
+ self_type = type(f_self)
+ ret.append(
+ "{}.{} - {}".format(
+ self_type.__module__, self_type.__name__, f_self
+ )
+ )
+
+ elif f_self is not None:
+ self_type = type(f_self)
+ ret.append("{}.{}".format(self_type.__module__, self_type.__name__))
+
+ else:
+ code = frm.f_code
+ if code.co_name in ("wrapper", "<module>"):
+ continue
+
+ ret.append("{}".format(code.co_name))
+
+ depth -= 1
+ if not depth:
+ break
+
+ return "\n".join(ret)
+
+ @classmethod
+ def _from_exception(cls, pe):
+ """
+ internal factory method to simplify creating one type of ParseException
+ from another - avoids having __init__ signature conflicts among subclasses
+ """
+ return cls(pe.pstr, pe.loc, pe.msg, pe.parserElement)
+
+ @property
+ def line(self) -> str:
+ """
+ Return the line of text where the exception occurred.
+ """
+ return line(self.loc, self.pstr)
+
+ @property
+ def lineno(self) -> int:
+ """
+ Return the 1-based line number of text where the exception occurred.
+ """
+ return lineno(self.loc, self.pstr)
+
+ @property
+ def col(self) -> int:
+ """
+ Return the 1-based column on the line of text where the exception occurred.
+ """
+ return col(self.loc, self.pstr)
+
+ @property
+ def column(self) -> int:
+ """
+ Return the 1-based column on the line of text where the exception occurred.
+ """
+ return col(self.loc, self.pstr)
+
+ def __str__(self) -> str:
+ if self.pstr:
+ if self.loc >= len(self.pstr):
+ foundstr = ", found end of text"
+ else:
+ # pull out next word at error location
+ found_match = _exception_word_extractor.match(self.pstr, self.loc)
+ if found_match is not None:
+ found = found_match.group(0)
+ else:
+ found = self.pstr[self.loc : self.loc + 1]
+ foundstr = (", found %r" % found).replace(r"\\", "\\")
+ else:
+ foundstr = ""
+ return "{}{} (at char {}), (line:{}, col:{})".format(
+ self.msg, foundstr, self.loc, self.lineno, self.column
+ )
+
+ def __repr__(self):
+ return str(self)
+
+ def mark_input_line(self, marker_string: str = None, *, markerString=">!<") -> str:
+ """
+ Extracts the exception line from the input string, and marks
+ the location of the exception with a special symbol.
+ """
+ markerString = marker_string if marker_string is not None else markerString
+ line_str = self.line
+ line_column = self.column - 1
+ if markerString:
+ line_str = "".join(
+ (line_str[:line_column], markerString, line_str[line_column:])
+ )
+ return line_str.strip()
+
+ def explain(self, depth=16) -> str:
+ """
+ Method to translate the Python internal traceback into a list
+ of the pyparsing expressions that caused the exception to be raised.
+
+ Parameters:
+
+ - depth (default=16) - number of levels back in the stack trace to list expression
+ and function names; if None, the full stack trace names will be listed; if 0, only
+ the failing input line, marker, and exception string will be shown
+
+ Returns a multi-line string listing the ParserElements and/or function names in the
+ exception's stack trace.
+
+ Example::
+
+ expr = pp.Word(pp.nums) * 3
+ try:
+ expr.parse_string("123 456 A789")
+ except pp.ParseException as pe:
+ print(pe.explain(depth=0))
+
+ prints::
+
+ 123 456 A789
+ ^
+ ParseException: Expected W:(0-9), found 'A' (at char 8), (line:1, col:9)
+
+ Note: the diagnostic output will include string representations of the expressions
+ that failed to parse. These representations will be more helpful if you use `set_name` to
+ give identifiable names to your expressions. Otherwise they will use the default string
+ forms, which may be cryptic to read.
+
+ Note: pyparsing's default truncation of exception tracebacks may also truncate the
+ stack of expressions that are displayed in the ``explain`` output. To get the full listing
+ of parser expressions, you may have to set ``ParserElement.verbose_stacktrace = True``
+ """
+ return self.explain_exception(self, depth)
+
+ markInputline = mark_input_line
+
+
+class ParseException(ParseBaseException):
+ """
+ Exception thrown when a parse expression doesn't match the input string
+
+ Example::
+
+ try:
+ Word(nums).set_name("integer").parse_string("ABC")
+ except ParseException as pe:
+ print(pe)
+ print("column: {}".format(pe.column))
+
+ prints::
+
+ Expected integer (at char 0), (line:1, col:1)
+ column: 1
+
+ """
+
+
+class ParseFatalException(ParseBaseException):
+ """
+ User-throwable exception thrown when inconsistent parse content
+ is found; stops all parsing immediately
+ """
+
+
+class ParseSyntaxException(ParseFatalException):
+ """
+ Just like :class:`ParseFatalException`, but thrown internally
+ when an :class:`ErrorStop<And._ErrorStop>` ('-' operator) indicates
+ that parsing is to stop immediately because an unbacktrackable
+ syntax error has been found.
+ """
+
+
+class RecursiveGrammarException(Exception):
+ """
+ Exception thrown by :class:`ParserElement.validate` if the
+ grammar could be left-recursive; parser may need to enable
+ left recursion using :class:`ParserElement.enable_left_recursion<ParserElement.enable_left_recursion>`
+ """
+
+ def __init__(self, parseElementList):
+ self.parseElementTrace = parseElementList
+
+ def __str__(self) -> str:
+ return "RecursiveGrammarException: {}".format(self.parseElementTrace)
--- /dev/null
+# helpers.py
+import html.entities
+
+from .core import *
+from .util import _bslash, _flatten, _escapeRegexRangeChars
+
+
+#
+# global helpers
+#
+def delimited_list(
+ expr: ParserElement,
+ delim: Union[str, ParserElement] = ",",
+ combine: bool = False,
+ *,
+ allow_trailing_delim: bool = False,
+) -> ParserElement:
+ """Helper to define a delimited list of expressions - the delimiter
+ defaults to ','. By default, the list elements and delimiters can
+ have intervening whitespace, and comments, but this can be
+ overridden by passing ``combine=True`` in the constructor. If
+ ``combine`` is set to ``True``, the matching tokens are
+ returned as a single token string, with the delimiters included;
+ otherwise, the matching tokens are returned as a list of tokens,
+ with the delimiters suppressed.
+
+ If ``allow_trailing_delim`` is set to True, then the list may end with
+ a delimiter.
+
+ Example::
+
+ delimited_list(Word(alphas)).parse_string("aa,bb,cc") # -> ['aa', 'bb', 'cc']
+ delimited_list(Word(hexnums), delim=':', combine=True).parse_string("AA:BB:CC:DD:EE") # -> ['AA:BB:CC:DD:EE']
+ """
+ dlName = "{expr} [{delim} {expr}]...{end}".format(
+ expr=str(expr),
+ delim=str(delim),
+ end=" [{}]".format(str(delim)) if allow_trailing_delim else "",
+ )
+
+ if not combine:
+ delim = Suppress(delim)
+
+ delimited_list_expr = expr + ZeroOrMore(delim + expr)
+
+ if allow_trailing_delim:
+ delimited_list_expr += Opt(delim)
+
+ if combine:
+ return Combine(delimited_list_expr).set_name(dlName)
+ else:
+ return delimited_list_expr.set_name(dlName)
+
+
+def counted_array(
+ expr: ParserElement,
+ int_expr: OptionalType[ParserElement] = None,
+ *,
+ intExpr: OptionalType[ParserElement] = None,
+) -> ParserElement:
+ """Helper to define a counted list of expressions.
+
+ This helper defines a pattern of the form::
+
+ integer expr expr expr...
+
+ where the leading integer tells how many expr expressions follow.
+ The matched tokens returns the array of expr tokens as a list - the
+ leading count token is suppressed.
+
+ If ``int_expr`` is specified, it should be a pyparsing expression
+ that produces an integer value.
+
+ Example::
+
+ counted_array(Word(alphas)).parse_string('2 ab cd ef') # -> ['ab', 'cd']
+
+ # in this parser, the leading integer value is given in binary,
+ # '10' indicating that 2 values are in the array
+ binary_constant = Word('01').set_parse_action(lambda t: int(t[0], 2))
+ counted_array(Word(alphas), int_expr=binary_constant).parse_string('10 ab cd ef') # -> ['ab', 'cd']
+
+ # if other fields must be parsed after the count but before the
+ # list items, give the fields results names and they will
+ # be preserved in the returned ParseResults:
+ count_with_metadata = integer + Word(alphas)("type")
+ typed_array = counted_array(Word(alphanums), int_expr=count_with_metadata)("items")
+ result = typed_array.parse_string("3 bool True True False")
+ print(result.dump())
+
+ # prints
+ # ['True', 'True', 'False']
+ # - items: ['True', 'True', 'False']
+ # - type: 'bool'
+ """
+ intExpr = intExpr or int_expr
+ array_expr = Forward()
+
+ def count_field_parse_action(s, l, t):
+ nonlocal array_expr
+ n = t[0]
+ array_expr <<= (expr * n) if n else Empty()
+ # clear list contents, but keep any named results
+ del t[:]
+
+ if intExpr is None:
+ intExpr = Word(nums).set_parse_action(lambda t: int(t[0]))
+ else:
+ intExpr = intExpr.copy()
+ intExpr.set_name("arrayLen")
+ intExpr.add_parse_action(count_field_parse_action, call_during_try=True)
+ return (intExpr + array_expr).set_name("(len) " + str(expr) + "...")
+
+
+def match_previous_literal(expr: ParserElement) -> ParserElement:
+ """Helper to define an expression that is indirectly defined from
+ the tokens matched in a previous expression, that is, it looks for
+ a 'repeat' of a previous expression. For example::
+
+ first = Word(nums)
+ second = match_previous_literal(first)
+ match_expr = first + ":" + second
+
+ will match ``"1:1"``, but not ``"1:2"``. Because this
+ matches a previous literal, will also match the leading
+ ``"1:1"`` in ``"1:10"``. If this is not desired, use
+ :class:`match_previous_expr`. Do *not* use with packrat parsing
+ enabled.
+ """
+ rep = Forward()
+
+ def copy_token_to_repeater(s, l, t):
+ if t:
+ if len(t) == 1:
+ rep << t[0]
+ else:
+ # flatten t tokens
+ tflat = _flatten(t.as_list())
+ rep << And(Literal(tt) for tt in tflat)
+ else:
+ rep << Empty()
+
+ expr.add_parse_action(copy_token_to_repeater, callDuringTry=True)
+ rep.set_name("(prev) " + str(expr))
+ return rep
+
+
+def match_previous_expr(expr: ParserElement) -> ParserElement:
+ """Helper to define an expression that is indirectly defined from
+ the tokens matched in a previous expression, that is, it looks for
+ a 'repeat' of a previous expression. For example::
+
+ first = Word(nums)
+ second = match_previous_expr(first)
+ match_expr = first + ":" + second
+
+ will match ``"1:1"``, but not ``"1:2"``. Because this
+ matches by expressions, will *not* match the leading ``"1:1"``
+ in ``"1:10"``; the expressions are evaluated first, and then
+ compared, so ``"1"`` is compared with ``"10"``. Do *not* use
+ with packrat parsing enabled.
+ """
+ rep = Forward()
+ e2 = expr.copy()
+ rep <<= e2
+
+ def copy_token_to_repeater(s, l, t):
+ matchTokens = _flatten(t.as_list())
+
+ def must_match_these_tokens(s, l, t):
+ theseTokens = _flatten(t.as_list())
+ if theseTokens != matchTokens:
+ raise ParseException("", 0, "")
+
+ rep.set_parse_action(must_match_these_tokens, callDuringTry=True)
+
+ expr.add_parse_action(copy_token_to_repeater, callDuringTry=True)
+ rep.set_name("(prev) " + str(expr))
+ return rep
+
+
+def one_of(
+ strs: Union[IterableType[str], str],
+ caseless: bool = False,
+ use_regex: bool = True,
+ as_keyword: bool = False,
+ *,
+ useRegex: bool = True,
+ asKeyword: bool = False,
+) -> ParserElement:
+ """Helper to quickly define a set of alternative :class:`Literal` s,
+ and makes sure to do longest-first testing when there is a conflict,
+ regardless of the input order, but returns
+ a :class:`MatchFirst` for best performance.
+
+ Parameters:
+
+ - ``strs`` - a string of space-delimited literals, or a collection of
+ string literals
+ - ``caseless`` - treat all literals as caseless - (default= ``False``)
+ - ``use_regex`` - as an optimization, will
+ generate a :class:`Regex` object; otherwise, will generate
+ a :class:`MatchFirst` object (if ``caseless=True`` or ``asKeyword=True``, or if
+ creating a :class:`Regex` raises an exception) - (default= ``True``)
+ - ``as_keyword`` - enforce :class:`Keyword`-style matching on the
+ generated expressions - (default= ``False``)
+ - ``asKeyword`` and ``useRegex`` are retained for pre-PEP8 compatibility,
+ but will be removed in a future release
+
+ Example::
+
+ comp_oper = one_of("< = > <= >= !=")
+ var = Word(alphas)
+ number = Word(nums)
+ term = var | number
+ comparison_expr = term + comp_oper + term
+ print(comparison_expr.search_string("B = 12 AA=23 B<=AA AA>12"))
+
+ prints::
+
+ [['B', '=', '12'], ['AA', '=', '23'], ['B', '<=', 'AA'], ['AA', '>', '12']]
+ """
+ asKeyword = asKeyword or as_keyword
+ useRegex = useRegex and use_regex
+
+ if isinstance(caseless, str_type):
+ warnings.warn(
+ "More than one string argument passed to one_of, pass"
+ " choices as a list or space-delimited string",
+ stacklevel=2,
+ )
+
+ if caseless:
+ isequal = lambda a, b: a.upper() == b.upper()
+ masks = lambda a, b: b.upper().startswith(a.upper())
+ parseElementClass = CaselessKeyword if asKeyword else CaselessLiteral
+ else:
+ isequal = lambda a, b: a == b
+ masks = lambda a, b: b.startswith(a)
+ parseElementClass = Keyword if asKeyword else Literal
+
+ symbols = []
+ if isinstance(strs, str_type):
+ symbols = strs.split()
+ elif isinstance(strs, Iterable):
+ symbols = list(strs)
+ else:
+ raise TypeError("Invalid argument to one_of, expected string or iterable")
+ if not symbols:
+ return NoMatch()
+
+ if not asKeyword:
+ # if not producing keywords, need to reorder to take care to avoid masking
+ # longer choices with shorter ones
+ i = 0
+ while i < len(symbols) - 1:
+ cur = symbols[i]
+ for j, other in enumerate(symbols[i + 1 :]):
+ if isequal(other, cur):
+ del symbols[i + j + 1]
+ break
+ elif masks(cur, other):
+ del symbols[i + j + 1]
+ symbols.insert(i, other)
+ break
+ else:
+ i += 1
+
+ if not (caseless or asKeyword) and useRegex:
+ # ~ print(strs, "->", "|".join([_escapeRegexChars(sym) for sym in symbols]))
+ try:
+ if len(symbols) == len("".join(symbols)):
+ return Regex(
+ "[%s]" % "".join(_escapeRegexRangeChars(sym) for sym in symbols)
+ ).set_name(" | ".join(symbols))
+ else:
+ return Regex("|".join(re.escape(sym) for sym in symbols)).set_name(
+ " | ".join(symbols)
+ )
+ except sre_constants.error:
+ warnings.warn(
+ "Exception creating Regex for one_of, building MatchFirst", stacklevel=2
+ )
+
+ # last resort, just use MatchFirst
+ return MatchFirst(parseElementClass(sym) for sym in symbols).set_name(
+ " | ".join(symbols)
+ )
+
+
+def dict_of(key: ParserElement, value: ParserElement) -> ParserElement:
+ """Helper to easily and clearly define a dictionary by specifying
+ the respective patterns for the key and value. Takes care of
+ defining the :class:`Dict`, :class:`ZeroOrMore`, and
+ :class:`Group` tokens in the proper order. The key pattern
+ can include delimiting markers or punctuation, as long as they are
+ suppressed, thereby leaving the significant key text. The value
+ pattern can include named results, so that the :class:`Dict` results
+ can include named token fields.
+
+ Example::
+
+ text = "shape: SQUARE posn: upper left color: light blue texture: burlap"
+ attr_expr = (label + Suppress(':') + OneOrMore(data_word, stop_on=label).set_parse_action(' '.join))
+ print(OneOrMore(attr_expr).parse_string(text).dump())
+
+ attr_label = label
+ attr_value = Suppress(':') + OneOrMore(data_word, stop_on=label).set_parse_action(' '.join)
+
+ # similar to Dict, but simpler call format
+ result = dict_of(attr_label, attr_value).parse_string(text)
+ print(result.dump())
+ print(result['shape'])
+ print(result.shape) # object attribute access works too
+ print(result.as_dict())
+
+ prints::
+
+ [['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'light blue'], ['texture', 'burlap']]
+ - color: light blue
+ - posn: upper left
+ - shape: SQUARE
+ - texture: burlap
+ SQUARE
+ SQUARE
+ {'color': 'light blue', 'shape': 'SQUARE', 'posn': 'upper left', 'texture': 'burlap'}
+ """
+ return Dict(OneOrMore(Group(key + value)))
+
+
+def original_text_for(
+ expr: ParserElement, as_string: bool = True, *, asString: bool = True
+) -> ParserElement:
+ """Helper to return the original, untokenized text for a given
+ expression. Useful to restore the parsed fields of an HTML start
+ tag into the raw tag text itself, or to revert separate tokens with
+ intervening whitespace back to the original matching input text. By
+ default, returns astring containing the original parsed text.
+
+ If the optional ``as_string`` argument is passed as
+ ``False``, then the return value is
+ a :class:`ParseResults` containing any results names that
+ were originally matched, and a single token containing the original
+ matched text from the input string. So if the expression passed to
+ :class:`original_text_for` contains expressions with defined
+ results names, you must set ``as_string`` to ``False`` if you
+ want to preserve those results name values.
+
+ The ``asString`` pre-PEP8 argument is retained for compatibility,
+ but will be removed in a future release.
+
+ Example::
+
+ src = "this is test <b> bold <i>text</i> </b> normal text "
+ for tag in ("b", "i"):
+ opener, closer = make_html_tags(tag)
+ patt = original_text_for(opener + SkipTo(closer) + closer)
+ print(patt.search_string(src)[0])
+
+ prints::
+
+ ['<b> bold <i>text</i> </b>']
+ ['<i>text</i>']
+ """
+ asString = asString and as_string
+
+ locMarker = Empty().set_parse_action(lambda s, loc, t: loc)
+ endlocMarker = locMarker.copy()
+ endlocMarker.callPreparse = False
+ matchExpr = locMarker("_original_start") + expr + endlocMarker("_original_end")
+ if asString:
+ extractText = lambda s, l, t: s[t._original_start : t._original_end]
+ else:
+
+ def extractText(s, l, t):
+ t[:] = [s[t.pop("_original_start") : t.pop("_original_end")]]
+
+ matchExpr.set_parse_action(extractText)
+ matchExpr.ignoreExprs = expr.ignoreExprs
+ return matchExpr
+
+
+def ungroup(expr: ParserElement) -> ParserElement:
+ """Helper to undo pyparsing's default grouping of And expressions,
+ even if all but one are non-empty.
+ """
+ return TokenConverter(expr).add_parse_action(lambda t: t[0])
+
+
+def locatedExpr(expr: ParserElement) -> ParserElement:
+ """
+ (DEPRECATED - future code should use the Located class)
+ Helper to decorate a returned token with its starting and ending
+ locations in the input string.
+
+ This helper adds the following results names:
+
+ - ``locn_start`` - location where matched expression begins
+ - ``locn_end`` - location where matched expression ends
+ - ``value`` - the actual parsed results
+
+ Be careful if the input text contains ``<TAB>`` characters, you
+ may want to call :class:`ParserElement.parseWithTabs`
+
+ Example::
+
+ wd = Word(alphas)
+ for match in locatedExpr(wd).searchString("ljsdf123lksdjjf123lkkjj1222"):
+ print(match)
+
+ prints::
+
+ [[0, 'ljsdf', 5]]
+ [[8, 'lksdjjf', 15]]
+ [[18, 'lkkjj', 23]]
+ """
+ locator = Empty().set_parse_action(lambda ss, ll, tt: ll)
+ return Group(
+ locator("locn_start")
+ + expr("value")
+ + locator.copy().leaveWhitespace()("locn_end")
+ )
+
+
+def nested_expr(
+ opener: Union[str, ParserElement] = "(",
+ closer: Union[str, ParserElement] = ")",
+ content: OptionalType[ParserElement] = None,
+ ignore_expr: ParserElement = quoted_string(),
+ *,
+ ignoreExpr: ParserElement = quoted_string(),
+) -> ParserElement:
+ """Helper method for defining nested lists enclosed in opening and
+ closing delimiters (``"("`` and ``")"`` are the default).
+
+ Parameters:
+ - ``opener`` - opening character for a nested list
+ (default= ``"("``); can also be a pyparsing expression
+ - ``closer`` - closing character for a nested list
+ (default= ``")"``); can also be a pyparsing expression
+ - ``content`` - expression for items within the nested lists
+ (default= ``None``)
+ - ``ignore_expr`` - expression for ignoring opening and closing delimiters
+ (default= :class:`quoted_string`)
+ - ``ignoreExpr`` - this pre-PEP8 argument is retained for compatibility
+ but will be removed in a future release
+
+ If an expression is not provided for the content argument, the
+ nested expression will capture all whitespace-delimited content
+ between delimiters as a list of separate values.
+
+ Use the ``ignore_expr`` argument to define expressions that may
+ contain opening or closing characters that should not be treated as
+ opening or closing characters for nesting, such as quoted_string or
+ a comment expression. Specify multiple expressions using an
+ :class:`Or` or :class:`MatchFirst`. The default is
+ :class:`quoted_string`, but if no expressions are to be ignored, then
+ pass ``None`` for this argument.
+
+ Example::
+
+ data_type = one_of("void int short long char float double")
+ decl_data_type = Combine(data_type + Opt(Word('*')))
+ ident = Word(alphas+'_', alphanums+'_')
+ number = pyparsing_common.number
+ arg = Group(decl_data_type + ident)
+ LPAR, RPAR = map(Suppress, "()")
+
+ code_body = nested_expr('{', '}', ignore_expr=(quoted_string | c_style_comment))
+
+ c_function = (decl_data_type("type")
+ + ident("name")
+ + LPAR + Opt(delimited_list(arg), [])("args") + RPAR
+ + code_body("body"))
+ c_function.ignore(c_style_comment)
+
+ source_code = '''
+ int is_odd(int x) {
+ return (x%2);
+ }
+
+ int dec_to_hex(char hchar) {
+ if (hchar >= '0' && hchar <= '9') {
+ return (ord(hchar)-ord('0'));
+ } else {
+ return (10+ord(hchar)-ord('A'));
+ }
+ }
+ '''
+ for func in c_function.search_string(source_code):
+ print("%(name)s (%(type)s) args: %(args)s" % func)
+
+
+ prints::
+
+ is_odd (int) args: [['int', 'x']]
+ dec_to_hex (int) args: [['char', 'hchar']]
+ """
+ if ignoreExpr != ignore_expr:
+ ignoreExpr = ignore_expr if ignoreExpr == quoted_string() else ignoreExpr
+ if opener == closer:
+ raise ValueError("opening and closing strings cannot be the same")
+ if content is None:
+ if isinstance(opener, str_type) and isinstance(closer, str_type):
+ if len(opener) == 1 and len(closer) == 1:
+ if ignoreExpr is not None:
+ content = Combine(
+ OneOrMore(
+ ~ignoreExpr
+ + CharsNotIn(
+ opener + closer + ParserElement.DEFAULT_WHITE_CHARS,
+ exact=1,
+ )
+ )
+ ).set_parse_action(lambda t: t[0].strip())
+ else:
+ content = empty.copy() + CharsNotIn(
+ opener + closer + ParserElement.DEFAULT_WHITE_CHARS
+ ).set_parse_action(lambda t: t[0].strip())
+ else:
+ if ignoreExpr is not None:
+ content = Combine(
+ OneOrMore(
+ ~ignoreExpr
+ + ~Literal(opener)
+ + ~Literal(closer)
+ + CharsNotIn(ParserElement.DEFAULT_WHITE_CHARS, exact=1)
+ )
+ ).set_parse_action(lambda t: t[0].strip())
+ else:
+ content = Combine(
+ OneOrMore(
+ ~Literal(opener)
+ + ~Literal(closer)
+ + CharsNotIn(ParserElement.DEFAULT_WHITE_CHARS, exact=1)
+ )
+ ).set_parse_action(lambda t: t[0].strip())
+ else:
+ raise ValueError(
+ "opening and closing arguments must be strings if no content expression is given"
+ )
+ ret = Forward()
+ if ignoreExpr is not None:
+ ret <<= Group(
+ Suppress(opener) + ZeroOrMore(ignoreExpr | ret | content) + Suppress(closer)
+ )
+ else:
+ ret <<= Group(Suppress(opener) + ZeroOrMore(ret | content) + Suppress(closer))
+ ret.set_name("nested %s%s expression" % (opener, closer))
+ return ret
+
+
+def _makeTags(tagStr, xml, suppress_LT=Suppress("<"), suppress_GT=Suppress(">")):
+ """Internal helper to construct opening and closing tag expressions, given a tag name"""
+ if isinstance(tagStr, str_type):
+ resname = tagStr
+ tagStr = Keyword(tagStr, caseless=not xml)
+ else:
+ resname = tagStr.name
+
+ tagAttrName = Word(alphas, alphanums + "_-:")
+ if xml:
+ tagAttrValue = dbl_quoted_string.copy().set_parse_action(remove_quotes)
+ openTag = (
+ suppress_LT
+ + tagStr("tag")
+ + Dict(ZeroOrMore(Group(tagAttrName + Suppress("=") + tagAttrValue)))
+ + Opt("/", default=[False])("empty").set_parse_action(
+ lambda s, l, t: t[0] == "/"
+ )
+ + suppress_GT
+ )
+ else:
+ tagAttrValue = quoted_string.copy().set_parse_action(remove_quotes) | Word(
+ printables, exclude_chars=">"
+ )
+ openTag = (
+ suppress_LT
+ + tagStr("tag")
+ + Dict(
+ ZeroOrMore(
+ Group(
+ tagAttrName.set_parse_action(lambda t: t[0].lower())
+ + Opt(Suppress("=") + tagAttrValue)
+ )
+ )
+ )
+ + Opt("/", default=[False])("empty").set_parse_action(
+ lambda s, l, t: t[0] == "/"
+ )
+ + suppress_GT
+ )
+ closeTag = Combine(Literal("</") + tagStr + ">", adjacent=False)
+
+ openTag.set_name("<%s>" % resname)
+ # add start<tagname> results name in parse action now that ungrouped names are not reported at two levels
+ openTag.add_parse_action(
+ lambda t: t.__setitem__(
+ "start" + "".join(resname.replace(":", " ").title().split()), t.copy()
+ )
+ )
+ closeTag = closeTag(
+ "end" + "".join(resname.replace(":", " ").title().split())
+ ).set_name("</%s>" % resname)
+ openTag.tag = resname
+ closeTag.tag = resname
+ openTag.tag_body = SkipTo(closeTag())
+ return openTag, closeTag
+
+
+def make_html_tags(
+ tag_str: Union[str, ParserElement]
+) -> Tuple[ParserElement, ParserElement]:
+ """Helper to construct opening and closing tag expressions for HTML,
+ given a tag name. Matches tags in either upper or lower case,
+ attributes with namespaces and with quoted or unquoted values.
+
+ Example::
+
+ text = '<td>More info at the <a href="https://github.com/pyparsing/pyparsing/wiki">pyparsing</a> wiki page</td>'
+ # make_html_tags returns pyparsing expressions for the opening and
+ # closing tags as a 2-tuple
+ a, a_end = make_html_tags("A")
+ link_expr = a + SkipTo(a_end)("link_text") + a_end
+
+ for link in link_expr.search_string(text):
+ # attributes in the <A> tag (like "href" shown here) are
+ # also accessible as named results
+ print(link.link_text, '->', link.href)
+
+ prints::
+
+ pyparsing -> https://github.com/pyparsing/pyparsing/wiki
+ """
+ return _makeTags(tag_str, False)
+
+
+def make_xml_tags(
+ tag_str: Union[str, ParserElement]
+) -> Tuple[ParserElement, ParserElement]:
+ """Helper to construct opening and closing tag expressions for XML,
+ given a tag name. Matches tags only in the given upper/lower case.
+
+ Example: similar to :class:`make_html_tags`
+ """
+ return _makeTags(tag_str, True)
+
+
+any_open_tag, any_close_tag = make_html_tags(
+ Word(alphas, alphanums + "_:").set_name("any tag")
+)
+
+_htmlEntityMap = {k.rstrip(";"): v for k, v in html.entities.html5.items()}
+common_html_entity = Regex(
+ "&(?P<entity>" + "|".join(_htmlEntityMap) + ");"
+).set_name("common HTML entity")
+
+
+def replace_html_entity(t):
+ """Helper parser action to replace common HTML entities with their special characters"""
+ return _htmlEntityMap.get(t.entity)
+
+
+class OpAssoc(Enum):
+ LEFT = 1
+ RIGHT = 2
+
+
+InfixNotationOperatorArgType = Union[
+ ParserElement, str, Tuple[Union[ParserElement, str], Union[ParserElement, str]]
+]
+InfixNotationOperatorSpec = Union[
+ Tuple[
+ InfixNotationOperatorArgType,
+ int,
+ OpAssoc,
+ OptionalType[ParseAction],
+ ],
+ Tuple[
+ InfixNotationOperatorArgType,
+ int,
+ OpAssoc,
+ ],
+]
+
+
+def infix_notation(
+ base_expr: ParserElement,
+ op_list: List[InfixNotationOperatorSpec],
+ lpar: Union[str, ParserElement] = Suppress("("),
+ rpar: Union[str, ParserElement] = Suppress(")"),
+) -> ParserElement:
+ """Helper method for constructing grammars of expressions made up of
+ operators working in a precedence hierarchy. Operators may be unary
+ or binary, left- or right-associative. Parse actions can also be
+ attached to operator expressions. The generated parser will also
+ recognize the use of parentheses to override operator precedences
+ (see example below).
+
+ Note: if you define a deep operator list, you may see performance
+ issues when using infix_notation. See
+ :class:`ParserElement.enable_packrat` for a mechanism to potentially
+ improve your parser performance.
+
+ Parameters:
+ - ``base_expr`` - expression representing the most basic operand to
+ be used in the expression
+ - ``op_list`` - list of tuples, one for each operator precedence level
+ in the expression grammar; each tuple is of the form ``(op_expr,
+ num_operands, right_left_assoc, (optional)parse_action)``, where:
+
+ - ``op_expr`` is the pyparsing expression for the operator; may also
+ be a string, which will be converted to a Literal; if ``num_operands``
+ is 3, ``op_expr`` is a tuple of two expressions, for the two
+ operators separating the 3 terms
+ - ``num_operands`` is the number of terms for this operator (must be 1,
+ 2, or 3)
+ - ``right_left_assoc`` is the indicator whether the operator is right
+ or left associative, using the pyparsing-defined constants
+ ``OpAssoc.RIGHT`` and ``OpAssoc.LEFT``.
+ - ``parse_action`` is the parse action to be associated with
+ expressions matching this operator expression (the parse action
+ tuple member may be omitted); if the parse action is passed
+ a tuple or list of functions, this is equivalent to calling
+ ``set_parse_action(*fn)``
+ (:class:`ParserElement.set_parse_action`)
+ - ``lpar`` - expression for matching left-parentheses
+ (default= ``Suppress('(')``)
+ - ``rpar`` - expression for matching right-parentheses
+ (default= ``Suppress(')')``)
+
+ Example::
+
+ # simple example of four-function arithmetic with ints and
+ # variable names
+ integer = pyparsing_common.signed_integer
+ varname = pyparsing_common.identifier
+
+ arith_expr = infix_notation(integer | varname,
+ [
+ ('-', 1, OpAssoc.RIGHT),
+ (one_of('* /'), 2, OpAssoc.LEFT),
+ (one_of('+ -'), 2, OpAssoc.LEFT),
+ ])
+
+ arith_expr.run_tests('''
+ 5+3*6
+ (5+3)*6
+ -2--11
+ ''', full_dump=False)
+
+ prints::
+
+ 5+3*6
+ [[5, '+', [3, '*', 6]]]
+
+ (5+3)*6
+ [[[5, '+', 3], '*', 6]]
+
+ -2--11
+ [[['-', 2], '-', ['-', 11]]]
+ """
+ # captive version of FollowedBy that does not do parse actions or capture results names
+ class _FB(FollowedBy):
+ def parseImpl(self, instring, loc, doActions=True):
+ self.expr.try_parse(instring, loc)
+ return loc, []
+
+ _FB.__name__ = "FollowedBy>"
+
+ ret = Forward()
+ lpar = Suppress(lpar)
+ rpar = Suppress(rpar)
+ lastExpr = base_expr | (lpar + ret + rpar)
+ for i, operDef in enumerate(op_list):
+ opExpr, arity, rightLeftAssoc, pa = (operDef + (None,))[:4]
+ if isinstance(opExpr, str_type):
+ opExpr = ParserElement._literalStringClass(opExpr)
+ if arity == 3:
+ if not isinstance(opExpr, (tuple, list)) or len(opExpr) != 2:
+ raise ValueError(
+ "if numterms=3, opExpr must be a tuple or list of two expressions"
+ )
+ opExpr1, opExpr2 = opExpr
+ term_name = "{}{} term".format(opExpr1, opExpr2)
+ else:
+ term_name = "{} term".format(opExpr)
+
+ if not 1 <= arity <= 3:
+ raise ValueError("operator must be unary (1), binary (2), or ternary (3)")
+
+ if rightLeftAssoc not in (OpAssoc.LEFT, OpAssoc.RIGHT):
+ raise ValueError("operator must indicate right or left associativity")
+
+ thisExpr = Forward().set_name(term_name)
+ if rightLeftAssoc is OpAssoc.LEFT:
+ if arity == 1:
+ matchExpr = _FB(lastExpr + opExpr) + Group(lastExpr + opExpr[1, ...])
+ elif arity == 2:
+ if opExpr is not None:
+ matchExpr = _FB(lastExpr + opExpr + lastExpr) + Group(
+ lastExpr + (opExpr + lastExpr)[1, ...]
+ )
+ else:
+ matchExpr = _FB(lastExpr + lastExpr) + Group(lastExpr[2, ...])
+ elif arity == 3:
+ matchExpr = _FB(
+ lastExpr + opExpr1 + lastExpr + opExpr2 + lastExpr
+ ) + Group(lastExpr + OneOrMore(opExpr1 + lastExpr + opExpr2 + lastExpr))
+ elif rightLeftAssoc is OpAssoc.RIGHT:
+ if arity == 1:
+ # try to avoid LR with this extra test
+ if not isinstance(opExpr, Opt):
+ opExpr = Opt(opExpr)
+ matchExpr = _FB(opExpr.expr + thisExpr) + Group(opExpr + thisExpr)
+ elif arity == 2:
+ if opExpr is not None:
+ matchExpr = _FB(lastExpr + opExpr + thisExpr) + Group(
+ lastExpr + (opExpr + thisExpr)[1, ...]
+ )
+ else:
+ matchExpr = _FB(lastExpr + thisExpr) + Group(
+ lastExpr + thisExpr[1, ...]
+ )
+ elif arity == 3:
+ matchExpr = _FB(
+ lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr
+ ) + Group(lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr)
+ if pa:
+ if isinstance(pa, (tuple, list)):
+ matchExpr.set_parse_action(*pa)
+ else:
+ matchExpr.set_parse_action(pa)
+ thisExpr <<= (matchExpr | lastExpr).setName(term_name)
+ lastExpr = thisExpr
+ ret <<= lastExpr
+ return ret
+
+
+def indentedBlock(blockStatementExpr, indentStack, indent=True, backup_stacks=[]):
+ """
+ (DEPRECATED - use IndentedBlock class instead)
+ Helper method for defining space-delimited indentation blocks,
+ such as those used to define block statements in Python source code.
+
+ Parameters:
+
+ - ``blockStatementExpr`` - expression defining syntax of statement that
+ is repeated within the indented block
+ - ``indentStack`` - list created by caller to manage indentation stack
+ (multiple ``statementWithIndentedBlock`` expressions within a single
+ grammar should share a common ``indentStack``)
+ - ``indent`` - boolean indicating whether block must be indented beyond
+ the current level; set to ``False`` for block of left-most statements
+ (default= ``True``)
+
+ A valid block must contain at least one ``blockStatement``.
+
+ (Note that indentedBlock uses internal parse actions which make it
+ incompatible with packrat parsing.)
+
+ Example::
+
+ data = '''
+ def A(z):
+ A1
+ B = 100
+ G = A2
+ A2
+ A3
+ B
+ def BB(a,b,c):
+ BB1
+ def BBA():
+ bba1
+ bba2
+ bba3
+ C
+ D
+ def spam(x,y):
+ def eggs(z):
+ pass
+ '''
+
+
+ indentStack = [1]
+ stmt = Forward()
+
+ identifier = Word(alphas, alphanums)
+ funcDecl = ("def" + identifier + Group("(" + Opt(delimitedList(identifier)) + ")") + ":")
+ func_body = indentedBlock(stmt, indentStack)
+ funcDef = Group(funcDecl + func_body)
+
+ rvalue = Forward()
+ funcCall = Group(identifier + "(" + Opt(delimitedList(rvalue)) + ")")
+ rvalue << (funcCall | identifier | Word(nums))
+ assignment = Group(identifier + "=" + rvalue)
+ stmt << (funcDef | assignment | identifier)
+
+ module_body = OneOrMore(stmt)
+
+ parseTree = module_body.parseString(data)
+ parseTree.pprint()
+
+ prints::
+
+ [['def',
+ 'A',
+ ['(', 'z', ')'],
+ ':',
+ [['A1'], [['B', '=', '100']], [['G', '=', 'A2']], ['A2'], ['A3']]],
+ 'B',
+ ['def',
+ 'BB',
+ ['(', 'a', 'b', 'c', ')'],
+ ':',
+ [['BB1'], [['def', 'BBA', ['(', ')'], ':', [['bba1'], ['bba2'], ['bba3']]]]]],
+ 'C',
+ 'D',
+ ['def',
+ 'spam',
+ ['(', 'x', 'y', ')'],
+ ':',
+ [[['def', 'eggs', ['(', 'z', ')'], ':', [['pass']]]]]]]
+ """
+ backup_stacks.append(indentStack[:])
+
+ def reset_stack():
+ indentStack[:] = backup_stacks[-1]
+
+ def checkPeerIndent(s, l, t):
+ if l >= len(s):
+ return
+ curCol = col(l, s)
+ if curCol != indentStack[-1]:
+ if curCol > indentStack[-1]:
+ raise ParseException(s, l, "illegal nesting")
+ raise ParseException(s, l, "not a peer entry")
+
+ def checkSubIndent(s, l, t):
+ curCol = col(l, s)
+ if curCol > indentStack[-1]:
+ indentStack.append(curCol)
+ else:
+ raise ParseException(s, l, "not a subentry")
+
+ def checkUnindent(s, l, t):
+ if l >= len(s):
+ return
+ curCol = col(l, s)
+ if not (indentStack and curCol in indentStack):
+ raise ParseException(s, l, "not an unindent")
+ if curCol < indentStack[-1]:
+ indentStack.pop()
+
+ NL = OneOrMore(LineEnd().set_whitespace_chars("\t ").suppress())
+ INDENT = (Empty() + Empty().set_parse_action(checkSubIndent)).set_name("INDENT")
+ PEER = Empty().set_parse_action(checkPeerIndent).set_name("")
+ UNDENT = Empty().set_parse_action(checkUnindent).set_name("UNINDENT")
+ if indent:
+ smExpr = Group(
+ Opt(NL)
+ + INDENT
+ + OneOrMore(PEER + Group(blockStatementExpr) + Opt(NL))
+ + UNDENT
+ )
+ else:
+ smExpr = Group(
+ Opt(NL)
+ + OneOrMore(PEER + Group(blockStatementExpr) + Opt(NL))
+ + Opt(UNDENT)
+ )
+
+ # add a parse action to remove backup_stack from list of backups
+ smExpr.add_parse_action(
+ lambda: backup_stacks.pop(-1) and None if backup_stacks else None
+ )
+ smExpr.set_fail_action(lambda a, b, c, d: reset_stack())
+ blockStatementExpr.ignore(_bslash + LineEnd())
+ return smExpr.set_name("indented block")
+
+
+class IndentedBlock(ParseElementEnhance):
+ """
+ Expression to match one or more expressions at a given indentation level.
+ Useful for parsing text where structure is implied by indentation (like Python source code).
+ """
+
+ def __init__(self, expr: ParserElement, recursive: bool = True):
+ super().__init__(expr, savelist=True)
+ self._recursive = recursive
+
+ def parseImpl(self, instring, loc, doActions=True):
+ # advance parse position to non-whitespace by using an Empty()
+ # this should be the column to be used for all subsequent indented lines
+ anchor_loc = Empty().preParse(instring, loc)
+
+ # see if self.expr matches at the current location - if not it will raise an exception
+ # and no further work is necessary
+ self.expr.try_parse(instring, anchor_loc, doActions)
+
+ indent_col = col(anchor_loc, instring)
+ peer_parse_action = match_only_at_col(indent_col)
+ peer_detect_expr = Empty().add_parse_action(peer_parse_action)
+ inner_expr = Empty() + peer_detect_expr + self.expr
+ inner_expr.set_name(f"inner {hex(id(inner_expr))[-4:].upper()}@{indent_col}")
+
+ if self._recursive:
+ indent_parse_action = condition_as_parse_action(
+ lambda s, l, t, relative_to_col=indent_col: col(l, s) > relative_to_col
+ )
+ indent_expr = FollowedBy(self.expr).add_parse_action(indent_parse_action)
+ inner_expr += Opt(Group(indent_expr + self.copy()))
+
+ return OneOrMore(inner_expr).parseImpl(instring, loc, doActions)
+
+
+# it's easy to get these comment structures wrong - they're very common, so may as well make them available
+c_style_comment = Combine(Regex(r"/\*(?:[^*]|\*(?!/))*") + "*/").set_name(
+ "C style comment"
+)
+"Comment of the form ``/* ... */``"
+
+html_comment = Regex(r"<!--[\s\S]*?-->").set_name("HTML comment")
+"Comment of the form ``<!-- ... -->``"
+
+rest_of_line = Regex(r".*").leave_whitespace().set_name("rest of line")
+dbl_slash_comment = Regex(r"//(?:\\\n|[^\n])*").set_name("// comment")
+"Comment of the form ``// ... (to end of line)``"
+
+cpp_style_comment = Combine(
+ Regex(r"/\*(?:[^*]|\*(?!/))*") + "*/" | dbl_slash_comment
+).set_name("C++ style comment")
+"Comment of either form :class:`c_style_comment` or :class:`dbl_slash_comment`"
+
+java_style_comment = cpp_style_comment
+"Same as :class:`cpp_style_comment`"
+
+python_style_comment = Regex(r"#.*").set_name("Python style comment")
+"Comment of the form ``# ... (to end of line)``"
+
+
+# build list of built-in expressions, for future reference if a global default value
+# gets updated
+_builtin_exprs = [v for v in vars().values() if isinstance(v, ParserElement)]
+
+
+# pre-PEP8 compatible names
+delimitedList = delimited_list
+countedArray = counted_array
+matchPreviousLiteral = match_previous_literal
+matchPreviousExpr = match_previous_expr
+oneOf = one_of
+dictOf = dict_of
+originalTextFor = original_text_for
+nestedExpr = nested_expr
+makeHTMLTags = make_html_tags
+makeXMLTags = make_xml_tags
+anyOpenTag, anyCloseTag = any_open_tag, any_close_tag
+commonHTMLEntity = common_html_entity
+replaceHTMLEntity = replace_html_entity
+opAssoc = OpAssoc
+infixNotation = infix_notation
+cStyleComment = c_style_comment
+htmlComment = html_comment
+restOfLine = rest_of_line
+dblSlashComment = dbl_slash_comment
+cppStyleComment = cpp_style_comment
+javaStyleComment = java_style_comment
+pythonStyleComment = python_style_comment
--- /dev/null
+# results.py
+from collections.abc import MutableMapping, Mapping, MutableSequence, Iterator
+import pprint
+from weakref import ref as wkref
+from typing import Tuple, Any
+
+str_type = (str, bytes)
+_generator_type = type((_ for _ in ()))
+
+
+class _ParseResultsWithOffset:
+ __slots__ = ["tup"]
+
+ def __init__(self, p1, p2):
+ self.tup = (p1, p2)
+
+ def __getitem__(self, i):
+ return self.tup[i]
+
+ def __getstate__(self):
+ return self.tup
+
+ def __setstate__(self, *args):
+ self.tup = args[0]
+
+
+class ParseResults:
+ """Structured parse results, to provide multiple means of access to
+ the parsed data:
+
+ - as a list (``len(results)``)
+ - by list index (``results[0], results[1]``, etc.)
+ - by attribute (``results.<results_name>`` - see :class:`ParserElement.set_results_name`)
+
+ Example::
+
+ integer = Word(nums)
+ date_str = (integer.set_results_name("year") + '/'
+ + integer.set_results_name("month") + '/'
+ + integer.set_results_name("day"))
+ # equivalent form:
+ # date_str = (integer("year") + '/'
+ # + integer("month") + '/'
+ # + integer("day"))
+
+ # parse_string returns a ParseResults object
+ result = date_str.parse_string("1999/12/31")
+
+ def test(s, fn=repr):
+ print("{} -> {}".format(s, fn(eval(s))))
+ test("list(result)")
+ test("result[0]")
+ test("result['month']")
+ test("result.day")
+ test("'month' in result")
+ test("'minutes' in result")
+ test("result.dump()", str)
+
+ prints::
+
+ list(result) -> ['1999', '/', '12', '/', '31']
+ result[0] -> '1999'
+ result['month'] -> '12'
+ result.day -> '31'
+ 'month' in result -> True
+ 'minutes' in result -> False
+ result.dump() -> ['1999', '/', '12', '/', '31']
+ - day: 31
+ - month: 12
+ - year: 1999
+ """
+
+ _null_values: Tuple[Any, ...] = (None, [], "", ())
+
+ __slots__ = [
+ "_name",
+ "_parent",
+ "_all_names",
+ "_modal",
+ "_toklist",
+ "_tokdict",
+ "__weakref__",
+ ]
+
+ class List(list):
+ """
+ Simple wrapper class to distinguish parsed list results that should be preserved
+ as actual Python lists, instead of being converted to :class:`ParseResults`:
+
+ LBRACK, RBRACK = map(pp.Suppress, "[]")
+ element = pp.Forward()
+ item = ppc.integer
+ element_list = LBRACK + pp.delimited_list(element) + RBRACK
+
+ # add parse actions to convert from ParseResults to actual Python collection types
+ def as_python_list(t):
+ return pp.ParseResults.List(t.as_list())
+ element_list.add_parse_action(as_python_list)
+
+ element <<= item | element_list
+
+ element.run_tests('''
+ 100
+ [2,3,4]
+ [[2, 1],3,4]
+ [(2, 1),3,4]
+ (2,3,4)
+ ''', post_parse=lambda s, r: (r[0], type(r[0])))
+
+ prints:
+
+ 100
+ (100, <class 'int'>)
+
+ [2,3,4]
+ ([2, 3, 4], <class 'list'>)
+
+ [[2, 1],3,4]
+ ([[2, 1], 3, 4], <class 'list'>)
+
+ (Used internally by :class:`Group` when `aslist=True`.)
+ """
+
+ def __new__(cls, contained=None):
+ if contained is None:
+ contained = []
+
+ if not isinstance(contained, list):
+ raise TypeError(
+ "{} may only be constructed with a list,"
+ " not {}".format(cls.__name__, type(contained).__name__)
+ )
+
+ return list.__new__(cls)
+
+ def __new__(cls, toklist=None, name=None, **kwargs):
+ if isinstance(toklist, ParseResults):
+ return toklist
+ self = object.__new__(cls)
+ self._name = None
+ self._parent = None
+ self._all_names = set()
+
+ if toklist is None:
+ self._toklist = []
+ elif isinstance(toklist, (list, _generator_type)):
+ self._toklist = (
+ [toklist[:]]
+ if isinstance(toklist, ParseResults.List)
+ else list(toklist)
+ )
+ else:
+ self._toklist = [toklist]
+ self._tokdict = dict()
+ return self
+
+ # Performance tuning: we construct a *lot* of these, so keep this
+ # constructor as small and fast as possible
+ def __init__(
+ self, toklist=None, name=None, asList=True, modal=True, isinstance=isinstance
+ ):
+ self._modal = modal
+ if name is not None and name != "":
+ if isinstance(name, int):
+ name = str(name)
+ if not modal:
+ self._all_names = {name}
+ self._name = name
+ if toklist not in self._null_values:
+ if isinstance(toklist, (str_type, type)):
+ toklist = [toklist]
+ if asList:
+ if isinstance(toklist, ParseResults):
+ self[name] = _ParseResultsWithOffset(
+ ParseResults(toklist._toklist), 0
+ )
+ else:
+ self[name] = _ParseResultsWithOffset(
+ ParseResults(toklist[0]), 0
+ )
+ self[name]._name = name
+ else:
+ try:
+ self[name] = toklist[0]
+ except (KeyError, TypeError, IndexError):
+ self[name] = toklist
+
+ def __getitem__(self, i):
+ if isinstance(i, (int, slice)):
+ return self._toklist[i]
+ else:
+ if i not in self._all_names:
+ return self._tokdict[i][-1][0]
+ else:
+ return ParseResults([v[0] for v in self._tokdict[i]])
+
+ def __setitem__(self, k, v, isinstance=isinstance):
+ if isinstance(v, _ParseResultsWithOffset):
+ self._tokdict[k] = self._tokdict.get(k, list()) + [v]
+ sub = v[0]
+ elif isinstance(k, (int, slice)):
+ self._toklist[k] = v
+ sub = v
+ else:
+ self._tokdict[k] = self._tokdict.get(k, list()) + [
+ _ParseResultsWithOffset(v, 0)
+ ]
+ sub = v
+ if isinstance(sub, ParseResults):
+ sub._parent = wkref(self)
+
+ def __delitem__(self, i):
+ if isinstance(i, (int, slice)):
+ mylen = len(self._toklist)
+ del self._toklist[i]
+
+ # convert int to slice
+ if isinstance(i, int):
+ if i < 0:
+ i += mylen
+ i = slice(i, i + 1)
+ # get removed indices
+ removed = list(range(*i.indices(mylen)))
+ removed.reverse()
+ # fixup indices in token dictionary
+ for name, occurrences in self._tokdict.items():
+ for j in removed:
+ for k, (value, position) in enumerate(occurrences):
+ occurrences[k] = _ParseResultsWithOffset(
+ value, position - (position > j)
+ )
+ else:
+ del self._tokdict[i]
+
+ def __contains__(self, k) -> bool:
+ return k in self._tokdict
+
+ def __len__(self) -> int:
+ return len(self._toklist)
+
+ def __bool__(self) -> bool:
+ return not not (self._toklist or self._tokdict)
+
+ def __iter__(self) -> Iterator:
+ return iter(self._toklist)
+
+ def __reversed__(self) -> Iterator:
+ return iter(self._toklist[::-1])
+
+ def keys(self):
+ return iter(self._tokdict)
+
+ def values(self):
+ return (self[k] for k in self.keys())
+
+ def items(self):
+ return ((k, self[k]) for k in self.keys())
+
+ def haskeys(self) -> bool:
+ """
+ Since ``keys()`` returns an iterator, this method is helpful in bypassing
+ code that looks for the existence of any defined results names."""
+ return bool(self._tokdict)
+
+ def pop(self, *args, **kwargs):
+ """
+ Removes and returns item at specified index (default= ``last``).
+ Supports both ``list`` and ``dict`` semantics for ``pop()``. If
+ passed no argument or an integer argument, it will use ``list``
+ semantics and pop tokens from the list of parsed tokens. If passed
+ a non-integer argument (most likely a string), it will use ``dict``
+ semantics and pop the corresponding value from any defined results
+ names. A second default return value argument is supported, just as in
+ ``dict.pop()``.
+
+ Example::
+
+ numlist = Word(nums)[...]
+ print(numlist.parse_string("0 123 321")) # -> ['0', '123', '321']
+
+ def remove_first(tokens):
+ tokens.pop(0)
+ numlist.add_parse_action(remove_first)
+ print(numlist.parse_string("0 123 321")) # -> ['123', '321']
+
+ label = Word(alphas)
+ patt = label("LABEL") + OneOrMore(Word(nums))
+ print(patt.parse_string("AAB 123 321").dump())
+
+ # Use pop() in a parse action to remove named result (note that corresponding value is not
+ # removed from list form of results)
+ def remove_LABEL(tokens):
+ tokens.pop("LABEL")
+ return tokens
+ patt.add_parse_action(remove_LABEL)
+ print(patt.parse_string("AAB 123 321").dump())
+
+ prints::
+
+ ['AAB', '123', '321']
+ - LABEL: AAB
+
+ ['AAB', '123', '321']
+ """
+ if not args:
+ args = [-1]
+ for k, v in kwargs.items():
+ if k == "default":
+ args = (args[0], v)
+ else:
+ raise TypeError(
+ "pop() got an unexpected keyword argument {!r}".format(k)
+ )
+ if isinstance(args[0], int) or len(args) == 1 or args[0] in self:
+ index = args[0]
+ ret = self[index]
+ del self[index]
+ return ret
+ else:
+ defaultvalue = args[1]
+ return defaultvalue
+
+ def get(self, key, default_value=None):
+ """
+ Returns named result matching the given key, or if there is no
+ such name, then returns the given ``default_value`` or ``None`` if no
+ ``default_value`` is specified.
+
+ Similar to ``dict.get()``.
+
+ Example::
+
+ integer = Word(nums)
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+
+ result = date_str.parse_string("1999/12/31")
+ print(result.get("year")) # -> '1999'
+ print(result.get("hour", "not specified")) # -> 'not specified'
+ print(result.get("hour")) # -> None
+ """
+ if key in self:
+ return self[key]
+ else:
+ return default_value
+
+ def insert(self, index, ins_string):
+ """
+ Inserts new element at location index in the list of parsed tokens.
+
+ Similar to ``list.insert()``.
+
+ Example::
+
+ numlist = Word(nums)[...]
+ print(numlist.parse_string("0 123 321")) # -> ['0', '123', '321']
+
+ # use a parse action to insert the parse location in the front of the parsed results
+ def insert_locn(locn, tokens):
+ tokens.insert(0, locn)
+ numlist.add_parse_action(insert_locn)
+ print(numlist.parse_string("0 123 321")) # -> [0, '0', '123', '321']
+ """
+ self._toklist.insert(index, ins_string)
+ # fixup indices in token dictionary
+ for name, occurrences in self._tokdict.items():
+ for k, (value, position) in enumerate(occurrences):
+ occurrences[k] = _ParseResultsWithOffset(
+ value, position + (position > index)
+ )
+
+ def append(self, item):
+ """
+ Add single element to end of ``ParseResults`` list of elements.
+
+ Example::
+
+ numlist = Word(nums)[...]
+ print(numlist.parse_string("0 123 321")) # -> ['0', '123', '321']
+
+ # use a parse action to compute the sum of the parsed integers, and add it to the end
+ def append_sum(tokens):
+ tokens.append(sum(map(int, tokens)))
+ numlist.add_parse_action(append_sum)
+ print(numlist.parse_string("0 123 321")) # -> ['0', '123', '321', 444]
+ """
+ self._toklist.append(item)
+
+ def extend(self, itemseq):
+ """
+ Add sequence of elements to end of ``ParseResults`` list of elements.
+
+ Example::
+
+ patt = OneOrMore(Word(alphas))
+
+ # use a parse action to append the reverse of the matched strings, to make a palindrome
+ def make_palindrome(tokens):
+ tokens.extend(reversed([t[::-1] for t in tokens]))
+ return ''.join(tokens)
+ patt.add_parse_action(make_palindrome)
+ print(patt.parse_string("lskdj sdlkjf lksd")) # -> 'lskdjsdlkjflksddsklfjkldsjdksl'
+ """
+ if isinstance(itemseq, ParseResults):
+ self.__iadd__(itemseq)
+ else:
+ self._toklist.extend(itemseq)
+
+ def clear(self):
+ """
+ Clear all elements and results names.
+ """
+ del self._toklist[:]
+ self._tokdict.clear()
+
+ def __getattr__(self, name):
+ try:
+ return self[name]
+ except KeyError:
+ if name.startswith("__"):
+ raise AttributeError(name)
+ return ""
+
+ def __add__(self, other) -> "ParseResults":
+ ret = self.copy()
+ ret += other
+ return ret
+
+ def __iadd__(self, other) -> "ParseResults":
+ if other._tokdict:
+ offset = len(self._toklist)
+ addoffset = lambda a: offset if a < 0 else a + offset
+ otheritems = other._tokdict.items()
+ otherdictitems = [
+ (k, _ParseResultsWithOffset(v[0], addoffset(v[1])))
+ for k, vlist in otheritems
+ for v in vlist
+ ]
+ for k, v in otherdictitems:
+ self[k] = v
+ if isinstance(v[0], ParseResults):
+ v[0]._parent = wkref(self)
+
+ self._toklist += other._toklist
+ self._all_names |= other._all_names
+ return self
+
+ def __radd__(self, other) -> "ParseResults":
+ if isinstance(other, int) and other == 0:
+ # useful for merging many ParseResults using sum() builtin
+ return self.copy()
+ else:
+ # this may raise a TypeError - so be it
+ return other + self
+
+ def __repr__(self) -> str:
+ return "{}({!r}, {})".format(type(self).__name__, self._toklist, self.as_dict())
+
+ def __str__(self) -> str:
+ return (
+ "["
+ + ", ".join(
+ str(i) if isinstance(i, ParseResults) else repr(i)
+ for i in self._toklist
+ )
+ + "]"
+ )
+
+ def _asStringList(self, sep=""):
+ out = []
+ for item in self._toklist:
+ if out and sep:
+ out.append(sep)
+ if isinstance(item, ParseResults):
+ out += item._asStringList()
+ else:
+ out.append(str(item))
+ return out
+
+ def as_list(self) -> list:
+ """
+ Returns the parse results as a nested list of matching tokens, all converted to strings.
+
+ Example::
+
+ patt = OneOrMore(Word(alphas))
+ result = patt.parse_string("sldkj lsdkj sldkj")
+ # even though the result prints in string-like form, it is actually a pyparsing ParseResults
+ print(type(result), result) # -> <class 'pyparsing.ParseResults'> ['sldkj', 'lsdkj', 'sldkj']
+
+ # Use as_list() to create an actual list
+ result_list = result.as_list()
+ print(type(result_list), result_list) # -> <class 'list'> ['sldkj', 'lsdkj', 'sldkj']
+ """
+ return [
+ res.as_list() if isinstance(res, ParseResults) else res
+ for res in self._toklist
+ ]
+
+ def as_dict(self) -> dict:
+ """
+ Returns the named parse results as a nested dictionary.
+
+ Example::
+
+ integer = Word(nums)
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+
+ result = date_str.parse_string('12/31/1999')
+ print(type(result), repr(result)) # -> <class 'pyparsing.ParseResults'> (['12', '/', '31', '/', '1999'], {'day': [('1999', 4)], 'year': [('12', 0)], 'month': [('31', 2)]})
+
+ result_dict = result.as_dict()
+ print(type(result_dict), repr(result_dict)) # -> <class 'dict'> {'day': '1999', 'year': '12', 'month': '31'}
+
+ # even though a ParseResults supports dict-like access, sometime you just need to have a dict
+ import json
+ print(json.dumps(result)) # -> Exception: TypeError: ... is not JSON serializable
+ print(json.dumps(result.as_dict())) # -> {"month": "31", "day": "1999", "year": "12"}
+ """
+
+ def to_item(obj):
+ if isinstance(obj, ParseResults):
+ return obj.as_dict() if obj.haskeys() else [to_item(v) for v in obj]
+ else:
+ return obj
+
+ return dict((k, to_item(v)) for k, v in self.items())
+
+ def copy(self) -> "ParseResults":
+ """
+ Returns a new copy of a :class:`ParseResults` object.
+ """
+ ret = ParseResults(self._toklist)
+ ret._tokdict = self._tokdict.copy()
+ ret._parent = self._parent
+ ret._all_names |= self._all_names
+ ret._name = self._name
+ return ret
+
+ def get_name(self):
+ r"""
+ Returns the results name for this token expression. Useful when several
+ different expressions might match at a particular location.
+
+ Example::
+
+ integer = Word(nums)
+ ssn_expr = Regex(r"\d\d\d-\d\d-\d\d\d\d")
+ house_number_expr = Suppress('#') + Word(nums, alphanums)
+ user_data = (Group(house_number_expr)("house_number")
+ | Group(ssn_expr)("ssn")
+ | Group(integer)("age"))
+ user_info = OneOrMore(user_data)
+
+ result = user_info.parse_string("22 111-22-3333 #221B")
+ for item in result:
+ print(item.get_name(), ':', item[0])
+
+ prints::
+
+ age : 22
+ ssn : 111-22-3333
+ house_number : 221B
+ """
+ if self._name:
+ return self._name
+ elif self._parent:
+ par = self._parent()
+
+ def find_in_parent(sub):
+ return next(
+ (
+ k
+ for k, vlist in par._tokdict.items()
+ for v, loc in vlist
+ if sub is v
+ ),
+ None,
+ )
+
+ return find_in_parent(self) if par else None
+ elif (
+ len(self) == 1
+ and len(self._tokdict) == 1
+ and next(iter(self._tokdict.values()))[0][1] in (0, -1)
+ ):
+ return next(iter(self._tokdict.keys()))
+ else:
+ return None
+
+ def dump(self, indent="", full=True, include_list=True, _depth=0) -> str:
+ """
+ Diagnostic method for listing out the contents of
+ a :class:`ParseResults`. Accepts an optional ``indent`` argument so
+ that this string can be embedded in a nested display of other data.
+
+ Example::
+
+ integer = Word(nums)
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+
+ result = date_str.parse_string('12/31/1999')
+ print(result.dump())
+
+ prints::
+
+ ['12', '/', '31', '/', '1999']
+ - day: 1999
+ - month: 31
+ - year: 12
+ """
+ out = []
+ NL = "\n"
+ out.append(indent + str(self.as_list()) if include_list else "")
+
+ if full:
+ if self.haskeys():
+ items = sorted((str(k), v) for k, v in self.items())
+ for k, v in items:
+ if out:
+ out.append(NL)
+ out.append("{}{}- {}: ".format(indent, (" " * _depth), k))
+ if isinstance(v, ParseResults):
+ if v:
+ out.append(
+ v.dump(
+ indent=indent,
+ full=full,
+ include_list=include_list,
+ _depth=_depth + 1,
+ )
+ )
+ else:
+ out.append(str(v))
+ else:
+ out.append(repr(v))
+ if any(isinstance(vv, ParseResults) for vv in self):
+ v = self
+ for i, vv in enumerate(v):
+ if isinstance(vv, ParseResults):
+ out.append(
+ "\n{}{}[{}]:\n{}{}{}".format(
+ indent,
+ (" " * (_depth)),
+ i,
+ indent,
+ (" " * (_depth + 1)),
+ vv.dump(
+ indent=indent,
+ full=full,
+ include_list=include_list,
+ _depth=_depth + 1,
+ ),
+ )
+ )
+ else:
+ out.append(
+ "\n%s%s[%d]:\n%s%s%s"
+ % (
+ indent,
+ (" " * (_depth)),
+ i,
+ indent,
+ (" " * (_depth + 1)),
+ str(vv),
+ )
+ )
+
+ return "".join(out)
+
+ def pprint(self, *args, **kwargs):
+ """
+ Pretty-printer for parsed results as a list, using the
+ `pprint <https://docs.python.org/3/library/pprint.html>`_ module.
+ Accepts additional positional or keyword args as defined for
+ `pprint.pprint <https://docs.python.org/3/library/pprint.html#pprint.pprint>`_ .
+
+ Example::
+
+ ident = Word(alphas, alphanums)
+ num = Word(nums)
+ func = Forward()
+ term = ident | num | Group('(' + func + ')')
+ func <<= ident + Group(Optional(delimited_list(term)))
+ result = func.parse_string("fna a,b,(fnb c,d,200),100")
+ result.pprint(width=40)
+
+ prints::
+
+ ['fna',
+ ['a',
+ 'b',
+ ['(', 'fnb', ['c', 'd', '200'], ')'],
+ '100']]
+ """
+ pprint.pprint(self.as_list(), *args, **kwargs)
+
+ # add support for pickle protocol
+ def __getstate__(self):
+ return (
+ self._toklist,
+ (
+ self._tokdict.copy(),
+ self._parent is not None and self._parent() or None,
+ self._all_names,
+ self._name,
+ ),
+ )
+
+ def __setstate__(self, state):
+ self._toklist, (self._tokdict, par, inAccumNames, self._name) = state
+ self._all_names = set(inAccumNames)
+ if par is not None:
+ self._parent = wkref(par)
+ else:
+ self._parent = None
+
+ def __getnewargs__(self):
+ return self._toklist, self._name
+
+ def __dir__(self):
+ return dir(type(self)) + list(self.keys())
+
+ @classmethod
+ def from_dict(cls, other, name=None) -> "ParseResults":
+ """
+ Helper classmethod to construct a ``ParseResults`` from a ``dict``, preserving the
+ name-value relations as results names. If an optional ``name`` argument is
+ given, a nested ``ParseResults`` will be returned.
+ """
+
+ def is_iterable(obj):
+ try:
+ iter(obj)
+ except Exception:
+ return False
+ else:
+ return not isinstance(obj, str_type)
+
+ ret = cls([])
+ for k, v in other.items():
+ if isinstance(v, Mapping):
+ ret += cls.from_dict(v, name=k)
+ else:
+ ret += cls([v], name=k, asList=is_iterable(v))
+ if name is not None:
+ ret = cls([ret], name=name)
+ return ret
+
+ asList = as_list
+ asDict = as_dict
+ getName = get_name
+
+
+MutableMapping.register(ParseResults)
+MutableSequence.register(ParseResults)
--- /dev/null
+# testing.py
+
+from contextlib import contextmanager
+from typing import Optional
+
+from .core import (
+ ParserElement,
+ ParseException,
+ Keyword,
+ __diag__,
+ __compat__,
+)
+
+
+class pyparsing_test:
+ """
+ namespace class for classes useful in writing unit tests
+ """
+
+ class reset_pyparsing_context:
+ """
+ Context manager to be used when writing unit tests that modify pyparsing config values:
+ - packrat parsing
+ - bounded recursion parsing
+ - default whitespace characters.
+ - default keyword characters
+ - literal string auto-conversion class
+ - __diag__ settings
+
+ Example::
+
+ with reset_pyparsing_context():
+ # test that literals used to construct a grammar are automatically suppressed
+ ParserElement.inlineLiteralsUsing(Suppress)
+
+ term = Word(alphas) | Word(nums)
+ group = Group('(' + term[...] + ')')
+
+ # assert that the '()' characters are not included in the parsed tokens
+ self.assertParseAndCheckList(group, "(abc 123 def)", ['abc', '123', 'def'])
+
+ # after exiting context manager, literals are converted to Literal expressions again
+ """
+
+ def __init__(self):
+ self._save_context = {}
+
+ def save(self):
+ self._save_context["default_whitespace"] = ParserElement.DEFAULT_WHITE_CHARS
+ self._save_context["default_keyword_chars"] = Keyword.DEFAULT_KEYWORD_CHARS
+
+ self._save_context[
+ "literal_string_class"
+ ] = ParserElement._literalStringClass
+
+ self._save_context["verbose_stacktrace"] = ParserElement.verbose_stacktrace
+
+ self._save_context["packrat_enabled"] = ParserElement._packratEnabled
+ if ParserElement._packratEnabled:
+ self._save_context[
+ "packrat_cache_size"
+ ] = ParserElement.packrat_cache.size
+ else:
+ self._save_context["packrat_cache_size"] = None
+ self._save_context["packrat_parse"] = ParserElement._parse
+ self._save_context[
+ "recursion_enabled"
+ ] = ParserElement._left_recursion_enabled
+
+ self._save_context["__diag__"] = {
+ name: getattr(__diag__, name) for name in __diag__._all_names
+ }
+
+ self._save_context["__compat__"] = {
+ "collect_all_And_tokens": __compat__.collect_all_And_tokens
+ }
+
+ return self
+
+ def restore(self):
+ # reset pyparsing global state
+ if (
+ ParserElement.DEFAULT_WHITE_CHARS
+ != self._save_context["default_whitespace"]
+ ):
+ ParserElement.set_default_whitespace_chars(
+ self._save_context["default_whitespace"]
+ )
+
+ ParserElement.verbose_stacktrace = self._save_context["verbose_stacktrace"]
+
+ Keyword.DEFAULT_KEYWORD_CHARS = self._save_context["default_keyword_chars"]
+ ParserElement.inlineLiteralsUsing(
+ self._save_context["literal_string_class"]
+ )
+
+ for name, value in self._save_context["__diag__"].items():
+ (__diag__.enable if value else __diag__.disable)(name)
+
+ ParserElement._packratEnabled = False
+ if self._save_context["packrat_enabled"]:
+ ParserElement.enable_packrat(self._save_context["packrat_cache_size"])
+ else:
+ ParserElement._parse = self._save_context["packrat_parse"]
+ ParserElement._left_recursion_enabled = self._save_context[
+ "recursion_enabled"
+ ]
+
+ __compat__.collect_all_And_tokens = self._save_context["__compat__"]
+
+ return self
+
+ def copy(self):
+ ret = type(self)()
+ ret._save_context.update(self._save_context)
+ return ret
+
+ def __enter__(self):
+ return self.save()
+
+ def __exit__(self, *args):
+ self.restore()
+
+ class TestParseResultsAsserts:
+ """
+ A mixin class to add parse results assertion methods to normal unittest.TestCase classes.
+ """
+
+ def assertParseResultsEquals(
+ self, result, expected_list=None, expected_dict=None, msg=None
+ ):
+ """
+ Unit test assertion to compare a :class:`ParseResults` object with an optional ``expected_list``,
+ and compare any defined results names with an optional ``expected_dict``.
+ """
+ if expected_list is not None:
+ self.assertEqual(expected_list, result.as_list(), msg=msg)
+ if expected_dict is not None:
+ self.assertEqual(expected_dict, result.as_dict(), msg=msg)
+
+ def assertParseAndCheckList(
+ self, expr, test_string, expected_list, msg=None, verbose=True
+ ):
+ """
+ Convenience wrapper assert to test a parser element and input string, and assert that
+ the resulting ``ParseResults.asList()`` is equal to the ``expected_list``.
+ """
+ result = expr.parse_string(test_string, parse_all=True)
+ if verbose:
+ print(result.dump())
+ else:
+ print(result.as_list())
+ self.assertParseResultsEquals(result, expected_list=expected_list, msg=msg)
+
+ def assertParseAndCheckDict(
+ self, expr, test_string, expected_dict, msg=None, verbose=True
+ ):
+ """
+ Convenience wrapper assert to test a parser element and input string, and assert that
+ the resulting ``ParseResults.asDict()`` is equal to the ``expected_dict``.
+ """
+ result = expr.parse_string(test_string, parseAll=True)
+ if verbose:
+ print(result.dump())
+ else:
+ print(result.as_list())
+ self.assertParseResultsEquals(result, expected_dict=expected_dict, msg=msg)
+
+ def assertRunTestResults(
+ self, run_tests_report, expected_parse_results=None, msg=None
+ ):
+ """
+ Unit test assertion to evaluate output of ``ParserElement.runTests()``. If a list of
+ list-dict tuples is given as the ``expected_parse_results`` argument, then these are zipped
+ with the report tuples returned by ``runTests`` and evaluated using ``assertParseResultsEquals``.
+ Finally, asserts that the overall ``runTests()`` success value is ``True``.
+
+ :param run_tests_report: tuple(bool, [tuple(str, ParseResults or Exception)]) returned from runTests
+ :param expected_parse_results (optional): [tuple(str, list, dict, Exception)]
+ """
+ run_test_success, run_test_results = run_tests_report
+
+ if expected_parse_results is not None:
+ merged = [
+ (*rpt, expected)
+ for rpt, expected in zip(run_test_results, expected_parse_results)
+ ]
+ for test_string, result, expected in merged:
+ # expected should be a tuple containing a list and/or a dict or an exception,
+ # and optional failure message string
+ # an empty tuple will skip any result validation
+ fail_msg = next(
+ (exp for exp in expected if isinstance(exp, str)), None
+ )
+ expected_exception = next(
+ (
+ exp
+ for exp in expected
+ if isinstance(exp, type) and issubclass(exp, Exception)
+ ),
+ None,
+ )
+ if expected_exception is not None:
+ with self.assertRaises(
+ expected_exception=expected_exception, msg=fail_msg or msg
+ ):
+ if isinstance(result, Exception):
+ raise result
+ else:
+ expected_list = next(
+ (exp for exp in expected if isinstance(exp, list)), None
+ )
+ expected_dict = next(
+ (exp for exp in expected if isinstance(exp, dict)), None
+ )
+ if (expected_list, expected_dict) != (None, None):
+ self.assertParseResultsEquals(
+ result,
+ expected_list=expected_list,
+ expected_dict=expected_dict,
+ msg=fail_msg or msg,
+ )
+ else:
+ # warning here maybe?
+ print("no validation for {!r}".format(test_string))
+
+ # do this last, in case some specific test results can be reported instead
+ self.assertTrue(
+ run_test_success, msg=msg if msg is not None else "failed runTests"
+ )
+
+ @contextmanager
+ def assertRaisesParseException(self, exc_type=ParseException, msg=None):
+ with self.assertRaises(exc_type, msg=msg):
+ yield
+
+ @staticmethod
+ def with_line_numbers(
+ s: str,
+ start_line: Optional[int] = None,
+ end_line: Optional[int] = None,
+ expand_tabs: bool = True,
+ eol_mark: str = "|",
+ mark_spaces: Optional[str] = None,
+ mark_control: Optional[str] = None,
+ ) -> str:
+ """
+ Helpful method for debugging a parser - prints a string with line and column numbers.
+ (Line and column numbers are 1-based.)
+
+ :param s: tuple(bool, str - string to be printed with line and column numbers
+ :param start_line: int - (optional) starting line number in s to print (default=1)
+ :param end_line: int - (optional) ending line number in s to print (default=len(s))
+ :param expand_tabs: bool - (optional) expand tabs to spaces, to match the pyparsing default
+ :param eol_mark: str - (optional) string to mark the end of lines, helps visualize trailing spaces (default="|")
+ :param mark_spaces: str - (optional) special character to display in place of spaces
+ :param mark_control: str - (optional) convert non-printing control characters to a placeholding
+ character; valid values:
+ - "unicode" - replaces control chars with Unicode symbols, such as "␍" and "␊"
+ - any single character string - replace control characters with given string
+ - None (default) - string is displayed as-is
+
+ :return: str - input string with leading line numbers and column number headers
+ """
+ if expand_tabs:
+ s = s.expandtabs()
+ if mark_control is not None:
+ if mark_control == "unicode":
+ tbl = str.maketrans(
+ {c: u for c, u in zip(range(0, 33), range(0x2400, 0x2433))}
+ | {127: 0x2421}
+ )
+ eol_mark = ""
+ else:
+ tbl = str.maketrans(
+ {c: mark_control for c in list(range(0, 32)) + [127]}
+ )
+ s = s.translate(tbl)
+ if mark_spaces is not None and mark_spaces != " ":
+ if mark_spaces == "unicode":
+ tbl = str.maketrans({9: 0x2409, 32: 0x2423})
+ s = s.translate(tbl)
+ else:
+ s = s.replace(" ", mark_spaces)
+ if start_line is None:
+ start_line = 1
+ if end_line is None:
+ end_line = len(s)
+ end_line = min(end_line, len(s))
+ start_line = min(max(1, start_line), end_line)
+
+ if mark_control != "unicode":
+ s_lines = s.splitlines()[start_line - 1 : end_line]
+ else:
+ s_lines = [line+"␊" for line in s.split("␊")[start_line - 1 : end_line]]
+ if not s_lines:
+ return ""
+
+ lineno_width = len(str(end_line))
+ max_line_len = max(len(line) for line in s_lines)
+ lead = " " * (lineno_width + 1)
+ header1 = (
+ lead
+ + "".join(
+ " {}".format(i + 1) for i in range(-(-max_line_len // 10))
+ )
+ + "\n"
+ )
+ header2 = lead + "1234567890" * (-(-max_line_len // 10)) + "\n"
+ return (
+ header1
+ + header2
+ + "\n".join(
+ "{:{}d}:{}{}".format(i, lineno_width, line, eol_mark)
+ for i, line in enumerate(s_lines, start=start_line)
+ )
+ + "\n"
+ )
--- /dev/null
+# unicode.py
+
+import sys
+from itertools import filterfalse
+from typing import List, Tuple, Union
+
+
+class _lazyclassproperty:
+ def __init__(self, fn):
+ self.fn = fn
+ self.__doc__ = fn.__doc__
+ self.__name__ = fn.__name__
+
+ def __get__(self, obj, cls):
+ if cls is None:
+ cls = type(obj)
+ if not hasattr(cls, "_intern") or any(
+ cls._intern is getattr(superclass, "_intern", [])
+ for superclass in cls.__mro__[1:]
+ ):
+ cls._intern = {}
+ attrname = self.fn.__name__
+ if attrname not in cls._intern:
+ cls._intern[attrname] = self.fn(cls)
+ return cls._intern[attrname]
+
+
+UnicodeRangeList = List[Union[Tuple[int, int], Tuple[int]]]
+
+
+class unicode_set:
+ """
+ A set of Unicode characters, for language-specific strings for
+ ``alphas``, ``nums``, ``alphanums``, and ``printables``.
+ A unicode_set is defined by a list of ranges in the Unicode character
+ set, in a class attribute ``_ranges``. Ranges can be specified using
+ 2-tuples or a 1-tuple, such as::
+
+ _ranges = [
+ (0x0020, 0x007e),
+ (0x00a0, 0x00ff),
+ (0x0100,),
+ ]
+
+ Ranges are left- and right-inclusive. A 1-tuple of (x,) is treated as (x, x).
+
+ A unicode set can also be defined using multiple inheritance of other unicode sets::
+
+ class CJK(Chinese, Japanese, Korean):
+ pass
+ """
+
+ _ranges: UnicodeRangeList = []
+
+ @_lazyclassproperty
+ def _chars_for_ranges(cls):
+ ret = []
+ for cc in cls.__mro__:
+ if cc is unicode_set:
+ break
+ for rr in getattr(cc, "_ranges", ()):
+ ret.extend(range(rr[0], rr[-1] + 1))
+ return [chr(c) for c in sorted(set(ret))]
+
+ @_lazyclassproperty
+ def printables(cls):
+ "all non-whitespace characters in this range"
+ return "".join(filterfalse(str.isspace, cls._chars_for_ranges))
+
+ @_lazyclassproperty
+ def alphas(cls):
+ "all alphabetic characters in this range"
+ return "".join(filter(str.isalpha, cls._chars_for_ranges))
+
+ @_lazyclassproperty
+ def nums(cls):
+ "all numeric digit characters in this range"
+ return "".join(filter(str.isdigit, cls._chars_for_ranges))
+
+ @_lazyclassproperty
+ def alphanums(cls):
+ "all alphanumeric characters in this range"
+ return cls.alphas + cls.nums
+
+ @_lazyclassproperty
+ def identchars(cls):
+ "all characters in this range that are valid identifier characters, plus underscore '_'"
+ return "".join(
+ sorted(
+ set(
+ "".join(filter(str.isidentifier, cls._chars_for_ranges))
+ + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzªµº"
+ + "ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ"
+ + "_"
+ )
+ )
+ )
+
+ @_lazyclassproperty
+ def identbodychars(cls):
+ """
+ all characters in this range that are valid identifier body characters,
+ plus the digits 0-9
+ """
+ return "".join(
+ sorted(
+ set(
+ cls.identchars
+ + "0123456789"
+ + "".join(
+ c for c in cls._chars_for_ranges if ("_" + c).isidentifier()
+ )
+ )
+ )
+ )
+
+
+class pyparsing_unicode(unicode_set):
+ """
+ A namespace class for defining common language unicode_sets.
+ """
+
+ _ranges: UnicodeRangeList = [(32, sys.maxunicode)]
+
+ class Latin1(unicode_set):
+ "Unicode set for Latin-1 Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x0020, 0x007E),
+ (0x00A0, 0x00FF),
+ ]
+
+ class LatinA(unicode_set):
+ "Unicode set for Latin-A Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x0100, 0x017F),
+ ]
+
+ class LatinB(unicode_set):
+ "Unicode set for Latin-B Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x0180, 0x024F),
+ ]
+
+ class Greek(unicode_set):
+ "Unicode set for Greek Unicode Character Ranges"
+ _ranges: UnicodeRangeList = [
+ (0x0342, 0x0345),
+ (0x0370, 0x0377),
+ (0x037A, 0x037F),
+ (0x0384, 0x038A),
+ (0x038C,),
+ (0x038E, 0x03A1),
+ (0x03A3, 0x03E1),
+ (0x03F0, 0x03FF),
+ (0x1D26, 0x1D2A),
+ (0x1D5E,),
+ (0x1D60,),
+ (0x1D66, 0x1D6A),
+ (0x1F00, 0x1F15),
+ (0x1F18, 0x1F1D),
+ (0x1F20, 0x1F45),
+ (0x1F48, 0x1F4D),
+ (0x1F50, 0x1F57),
+ (0x1F59,),
+ (0x1F5B,),
+ (0x1F5D,),
+ (0x1F5F, 0x1F7D),
+ (0x1F80, 0x1FB4),
+ (0x1FB6, 0x1FC4),
+ (0x1FC6, 0x1FD3),
+ (0x1FD6, 0x1FDB),
+ (0x1FDD, 0x1FEF),
+ (0x1FF2, 0x1FF4),
+ (0x1FF6, 0x1FFE),
+ (0x2129,),
+ (0x2719, 0x271A),
+ (0xAB65,),
+ (0x10140, 0x1018D),
+ (0x101A0,),
+ (0x1D200, 0x1D245),
+ (0x1F7A1, 0x1F7A7),
+ ]
+
+ class Cyrillic(unicode_set):
+ "Unicode set for Cyrillic Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x0400, 0x052F),
+ (0x1C80, 0x1C88),
+ (0x1D2B,),
+ (0x1D78,),
+ (0x2DE0, 0x2DFF),
+ (0xA640, 0xA672),
+ (0xA674, 0xA69F),
+ (0xFE2E, 0xFE2F),
+ ]
+
+ class Chinese(unicode_set):
+ "Unicode set for Chinese Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x2E80, 0x2E99),
+ (0x2E9B, 0x2EF3),
+ (0x31C0, 0x31E3),
+ (0x3400, 0x4DB5),
+ (0x4E00, 0x9FEF),
+ (0xA700, 0xA707),
+ (0xF900, 0xFA6D),
+ (0xFA70, 0xFAD9),
+ (0x16FE2, 0x16FE3),
+ (0x1F210, 0x1F212),
+ (0x1F214, 0x1F23B),
+ (0x1F240, 0x1F248),
+ (0x20000, 0x2A6D6),
+ (0x2A700, 0x2B734),
+ (0x2B740, 0x2B81D),
+ (0x2B820, 0x2CEA1),
+ (0x2CEB0, 0x2EBE0),
+ (0x2F800, 0x2FA1D),
+ ]
+
+ class Japanese(unicode_set):
+ "Unicode set for Japanese Unicode Character Range, combining Kanji, Hiragana, and Katakana ranges"
+ _ranges: UnicodeRangeList = []
+
+ class Kanji(unicode_set):
+ "Unicode set for Kanji Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x4E00, 0x9FBF),
+ (0x3000, 0x303F),
+ ]
+
+ class Hiragana(unicode_set):
+ "Unicode set for Hiragana Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x3041, 0x3096),
+ (0x3099, 0x30A0),
+ (0x30FC,),
+ (0xFF70,),
+ (0x1B001,),
+ (0x1B150, 0x1B152),
+ (0x1F200,),
+ ]
+
+ class Katakana(unicode_set):
+ "Unicode set for Katakana Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x3099, 0x309C),
+ (0x30A0, 0x30FF),
+ (0x31F0, 0x31FF),
+ (0x32D0, 0x32FE),
+ (0xFF65, 0xFF9F),
+ (0x1B000,),
+ (0x1B164, 0x1B167),
+ (0x1F201, 0x1F202),
+ (0x1F213,),
+ ]
+
+ class Hangul(unicode_set):
+ "Unicode set for Hangul (Korean) Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x1100, 0x11FF),
+ (0x302E, 0x302F),
+ (0x3131, 0x318E),
+ (0x3200, 0x321C),
+ (0x3260, 0x327B),
+ (0x327E,),
+ (0xA960, 0xA97C),
+ (0xAC00, 0xD7A3),
+ (0xD7B0, 0xD7C6),
+ (0xD7CB, 0xD7FB),
+ (0xFFA0, 0xFFBE),
+ (0xFFC2, 0xFFC7),
+ (0xFFCA, 0xFFCF),
+ (0xFFD2, 0xFFD7),
+ (0xFFDA, 0xFFDC),
+ ]
+
+ Korean = Hangul
+
+ class CJK(Chinese, Japanese, Hangul):
+ "Unicode set for combined Chinese, Japanese, and Korean (CJK) Unicode Character Range"
+ pass
+
+ class Thai(unicode_set):
+ "Unicode set for Thai Unicode Character Range"
+ _ranges: UnicodeRangeList = [(0x0E01, 0x0E3A), (0x0E3F, 0x0E5B)]
+
+ class Arabic(unicode_set):
+ "Unicode set for Arabic Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x0600, 0x061B),
+ (0x061E, 0x06FF),
+ (0x0700, 0x077F),
+ ]
+
+ class Hebrew(unicode_set):
+ "Unicode set for Hebrew Unicode Character Range"
+ _ranges: UnicodeRangeList = [
+ (0x0591, 0x05C7),
+ (0x05D0, 0x05EA),
+ (0x05EF, 0x05F4),
+ (0xFB1D, 0xFB36),
+ (0xFB38, 0xFB3C),
+ (0xFB3E,),
+ (0xFB40, 0xFB41),
+ (0xFB43, 0xFB44),
+ (0xFB46, 0xFB4F),
+ ]
+
+ class Devanagari(unicode_set):
+ "Unicode set for Devanagari Unicode Character Range"
+ _ranges: UnicodeRangeList = [(0x0900, 0x097F), (0xA8E0, 0xA8FF)]
+
+
+pyparsing_unicode.Japanese._ranges = (
+ pyparsing_unicode.Japanese.Kanji._ranges
+ + pyparsing_unicode.Japanese.Hiragana._ranges
+ + pyparsing_unicode.Japanese.Katakana._ranges
+)
+
+# define ranges in language character sets
+pyparsing_unicode.العربية = pyparsing_unicode.Arabic
+pyparsing_unicode.中文 = pyparsing_unicode.Chinese
+pyparsing_unicode.кириллица = pyparsing_unicode.Cyrillic
+pyparsing_unicode.Ελληνικά = pyparsing_unicode.Greek
+pyparsing_unicode.עִברִית = pyparsing_unicode.Hebrew
+pyparsing_unicode.日本語 = pyparsing_unicode.Japanese
+pyparsing_unicode.Japanese.漢字 = pyparsing_unicode.Japanese.Kanji
+pyparsing_unicode.Japanese.カタカナ = pyparsing_unicode.Japanese.Katakana
+pyparsing_unicode.Japanese.ひらがな = pyparsing_unicode.Japanese.Hiragana
+pyparsing_unicode.한국어 = pyparsing_unicode.Korean
+pyparsing_unicode.ไทย = pyparsing_unicode.Thai
+pyparsing_unicode.देवनागरी = pyparsing_unicode.Devanagari
--- /dev/null
+# util.py
+import warnings
+import types
+import collections
+import itertools
+from functools import lru_cache
+from typing import List
+
+
+_bslash = chr(92)
+
+
+class __config_flags:
+ """Internal class for defining compatibility and debugging flags"""
+
+ _all_names: List[str] = []
+ _fixed_names: List[str] = []
+ _type_desc = "configuration"
+
+ @classmethod
+ def _set(cls, dname, value):
+ if dname in cls._fixed_names:
+ warnings.warn(
+ "{}.{} {} is {} and cannot be overridden".format(
+ cls.__name__,
+ dname,
+ cls._type_desc,
+ str(getattr(cls, dname)).upper(),
+ )
+ )
+ return
+ if dname in cls._all_names:
+ setattr(cls, dname, value)
+ else:
+ raise ValueError("no such {} {!r}".format(cls._type_desc, dname))
+
+ enable = classmethod(lambda cls, name: cls._set(name, True))
+ disable = classmethod(lambda cls, name: cls._set(name, False))
+
+
+@lru_cache(maxsize=128)
+def col(loc: int, strg: str):
+ """
+ Returns current column within a string, counting newlines as line separators.
+ The first column is number 1.
+
+ Note: the default parsing behavior is to expand tabs in the input string
+ before starting the parsing process. See
+ :class:`ParserElement.parseString` for more
+ information on parsing strings containing ``<TAB>`` s, and suggested
+ methods to maintain a consistent view of the parsed string, the parse
+ location, and line and column positions within the parsed string.
+ """
+ s = strg
+ return 1 if 0 < loc < len(s) and s[loc - 1] == "\n" else loc - s.rfind("\n", 0, loc)
+
+
+@lru_cache(maxsize=128)
+def lineno(loc: int, strg: str):
+ """Returns current line number within a string, counting newlines as line separators.
+ The first line is number 1.
+
+ Note - the default parsing behavior is to expand tabs in the input string
+ before starting the parsing process. See :class:`ParserElement.parseString`
+ for more information on parsing strings containing ``<TAB>`` s, and
+ suggested methods to maintain a consistent view of the parsed string, the
+ parse location, and line and column positions within the parsed string.
+ """
+ return strg.count("\n", 0, loc) + 1
+
+
+@lru_cache(maxsize=128)
+def line(loc: int, strg: str):
+ """
+ Returns the line of text containing loc within a string, counting newlines as line separators.
+ """
+ lastCR = strg.rfind("\n", 0, loc)
+ nextCR = strg.find("\n", loc)
+ return strg[lastCR + 1 : nextCR] if nextCR >= 0 else strg[lastCR + 1 :]
+
+
+class _UnboundedCache:
+ def __init__(self):
+ cache = {}
+ cache_get = cache.get
+ self.not_in_cache = not_in_cache = object()
+
+ def get(self, key):
+ return cache_get(key, not_in_cache)
+
+ def set(self, key, value):
+ cache[key] = value
+
+ def clear(self):
+ cache.clear()
+
+ self.size = None
+ self.get = types.MethodType(get, self)
+ self.set = types.MethodType(set, self)
+ self.clear = types.MethodType(clear, self)
+
+
+class _FifoCache:
+ def __init__(self, size):
+ self.not_in_cache = not_in_cache = object()
+ cache = collections.OrderedDict()
+ cache_get = cache.get
+
+ def get(self, key):
+ return cache_get(key, not_in_cache)
+
+ def set(self, key, value):
+ cache[key] = value
+ while len(cache) > size:
+ cache.popitem(last=False)
+
+ def clear(self):
+ cache.clear()
+
+ self.size = size
+ self.get = types.MethodType(get, self)
+ self.set = types.MethodType(set, self)
+ self.clear = types.MethodType(clear, self)
+
+
+class LRUMemo:
+ """
+ A memoizing mapping that retains `capacity` deleted items
+
+ The memo tracks retained items by their access order; once `capacity` items
+ are retained, the least recently used item is discarded.
+ """
+
+ def __init__(self, capacity):
+ self._capacity = capacity
+ self._active = {}
+ self._memory = collections.OrderedDict()
+
+ def __getitem__(self, key):
+ try:
+ return self._active[key]
+ except KeyError:
+ self._memory.move_to_end(key)
+ return self._memory[key]
+
+ def __setitem__(self, key, value):
+ self._memory.pop(key, None)
+ self._active[key] = value
+
+ def __delitem__(self, key):
+ try:
+ value = self._active.pop(key)
+ except KeyError:
+ pass
+ else:
+ while len(self._memory) >= self._capacity:
+ self._memory.popitem(last=False)
+ self._memory[key] = value
+
+ def clear(self):
+ self._active.clear()
+ self._memory.clear()
+
+
+class UnboundedMemo(dict):
+ """
+ A memoizing mapping that retains all deleted items
+ """
+
+ def __delitem__(self, key):
+ pass
+
+
+def _escapeRegexRangeChars(s: str):
+ # escape these chars: ^-[]
+ for c in r"\^-[]":
+ s = s.replace(c, _bslash + c)
+ s = s.replace("\n", r"\n")
+ s = s.replace("\t", r"\t")
+ return str(s)
+
+
+def _collapseStringToRanges(s: str, re_escape: bool = True):
+ def is_consecutive(c):
+ c_int = ord(c)
+ is_consecutive.prev, prev = c_int, is_consecutive.prev
+ if c_int - prev > 1:
+ is_consecutive.value = next(is_consecutive.counter)
+ return is_consecutive.value
+
+ is_consecutive.prev = 0
+ is_consecutive.counter = itertools.count()
+ is_consecutive.value = -1
+
+ def escape_re_range_char(c):
+ return "\\" + c if c in r"\^-][" else c
+
+ def no_escape_re_range_char(c):
+ return c
+
+ if not re_escape:
+ escape_re_range_char = no_escape_re_range_char
+
+ ret = []
+ s = "".join(sorted(set(s)))
+ if len(s) > 3:
+ for _, chars in itertools.groupby(s, key=is_consecutive):
+ first = last = next(chars)
+ last = collections.deque(
+ itertools.chain(iter([last]), chars), maxlen=1
+ ).pop()
+ if first == last:
+ ret.append(escape_re_range_char(first))
+ else:
+ ret.append(
+ "{}-{}".format(
+ escape_re_range_char(first), escape_re_range_char(last)
+ )
+ )
+ else:
+ ret = [escape_re_range_char(c) for c in s]
+
+ return "".join(ret)
+
+
+def _flatten(ll: List):
+ ret = []
+ for i in ll:
+ if isinstance(i, list):
+ ret.extend(_flatten(i))
+ else:
+ ret.append(i)
+ return ret
-[bdist_wheel]
-universal = 1
-
-[metadata]
-license_file = LICENSE
-
[egg_info]
tag_build =
tag_date = 0
"""Setup script for the pyparsing module distribution."""
from setuptools import setup
+import io
+import sys
from pyparsing import __version__ as pyparsing_version
-from io import open
-# The directory containing this file
-README_name = __file__.replace("setup.py", "README.rst")
+# guard against manual invocation of setup.py (when using pip, we shouldn't even get this far)
+if sys.version_info[:2] < (3, 6):
+ sys.exit(
+ "Python < 3.6 is not supported in this version of pyparsing; use latest pyparsing 2.4.x release"
+ )
-# The text of the README file
-with open(README_name, encoding='utf8') as README:
+# get the text of the README file
+README_name = __file__.replace("setup.py", "README.rst")
+with io.open(README_name, encoding="utf8") as README:
pyparsing_main_doc = README.read()
-modules = ["pyparsing",]
+packages = ["pyparsing", "pyparsing.diagram"]
-setup(# Distribution meta-data
- name = "pyparsing",
- version = pyparsing_version,
- description = "Python parsing module",
- long_description = pyparsing_main_doc,
- author = "Paul McGuire",
- author_email = "ptmcg@users.sourceforge.net",
- url = "https://github.com/pyparsing/pyparsing/",
- download_url = "https://pypi.org/project/pyparsing/",
- license = "MIT License",
- py_modules = modules,
- python_requires='>=2.6, !=3.0.*, !=3.1.*, !=3.2.*',
- test_suite="unitTests.suite",
+setup( # Distribution meta-data
+ name="pyparsing",
+ version=pyparsing_version,
+ description="Python parsing module",
+ long_description=pyparsing_main_doc,
+ long_description_content_type="text/x-rst",
+ author="Paul McGuire",
+ author_email="ptmcg.gm+pyparsing@gmail.com",
+ url="https://github.com/pyparsing/pyparsing/",
+ download_url="https://pypi.org/project/pyparsing/",
+ license="MIT License",
+ packages=packages,
+ python_requires=">=3.6",
+ extras_require={
+ "diagrams": ["railroad-diagrams", "jinja2"],
+ },
+ package_data={"pyparsing.diagram": ["*.jinja2"]},
classifiers=[
- 'Development Status :: 5 - Production/Stable',
- 'Intended Audience :: Developers',
- 'Intended Audience :: Information Technology',
- 'License :: OSI Approved :: MIT License',
- 'Operating System :: OS Independent',
- 'Programming Language :: Python',
- 'Programming Language :: Python :: 2',
- 'Programming Language :: Python :: 2.6',
- 'Programming Language :: Python :: 2.7',
- 'Programming Language :: Python :: 3',
- 'Programming Language :: Python :: 3.3',
- 'Programming Language :: Python :: 3.4',
- 'Programming Language :: Python :: 3.5',
- 'Programming Language :: Python :: 3.6',
- 'Programming Language :: Python :: 3.7',
- 'Programming Language :: Python :: 3.8',
- ]
- )
+ "Development Status :: 5 - Production/Stable",
+ "Intended Audience :: Developers",
+ "Intended Audience :: Information Technology",
+ "License :: OSI Approved :: MIT License",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Programming Language :: Python :: 3.8",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3 :: Only",
+ "Programming Language :: Python :: Implementation :: CPython",
+ "Programming Language :: Python :: Implementation :: PyPy",
+ ],
+)
+++ /dev/null
-#
-# simple_unit_tests.py
-#
-# While these unit tests *do* perform low-level unit testing of the classes in pyparsing,
-# this testing module should also serve an instructional purpose, to clearly show simple passing
-# and failing parse cases of some basic pyparsing expressions.
-#
-# Copyright (c) 2018 Paul T. McGuire
-#
-from __future__ import division
-
-try:
- import unittest2 as unittest
-except ImportError:
- import unittest
-import pyparsing as pp
-from collections import namedtuple
-from datetime import datetime
-
-# Test spec data class for specifying simple pyparsing test cases
-PpTestSpec = namedtuple("PpTestSpec", "desc expr text parse_fn "
- "expected_list expected_dict expected_fail_locn")
-PpTestSpec.__new__.__defaults__ = ('', pp.Empty(), '', 'parseString', None, None, None)
-
-
-class PyparsingExpressionTestCase(unittest.TestCase):
- """
- Base pyparsing testing class to parse various pyparsing expressions against
- given text strings. Subclasses must define a class attribute 'tests' which
- is a list of PpTestSpec instances.
- """
-
- if not hasattr(unittest.TestCase, 'subTest'):
- # Python 2 compatibility
- from contextlib import contextmanager
- @contextmanager
- def subTest(self, **params):
- print('subTest:', params)
- yield
-
- tests = []
- def runTest(self):
- if self.__class__ is PyparsingExpressionTestCase:
- return
-
- for test_spec in self.tests:
- # for each spec in the class's tests list, create a subtest
- # that will either:
- # - parse the string with expected success, display the
- # results, and validate the returned ParseResults
- # - or parse the string with expected failure, display the
- # error message and mark the error location, and validate
- # the location against an expected value
- with self.subTest(test_spec=test_spec):
- test_spec.expr.streamline()
- print("\n{0} - {1}({2})".format(test_spec.desc,
- type(test_spec.expr).__name__,
- test_spec.expr))
-
- parsefn = getattr(test_spec.expr, test_spec.parse_fn)
- if test_spec.expected_fail_locn is None:
- # expect success
- result = parsefn(test_spec.text)
- if test_spec.parse_fn == 'parseString':
- print(result.dump())
- # compare results against given list and/or dict
- if test_spec.expected_list is not None:
- self.assertEqual(result.asList(), test_spec.expected_list)
- if test_spec.expected_dict is not None:
- self.assertEqual(result.asDict(), test_spec.expected_dict)
- elif test_spec.parse_fn == 'transformString':
- print(result)
- # compare results against given list and/or dict
- if test_spec.expected_list is not None:
- self.assertEqual([result], test_spec.expected_list)
- elif test_spec.parse_fn == 'searchString':
- print(result)
- # compare results against given list and/or dict
- if test_spec.expected_list is not None:
- self.assertEqual([result], test_spec.expected_list)
- else:
- # expect fail
- try:
- parsefn(test_spec.text)
- except Exception as exc:
- if not hasattr(exc, '__traceback__'):
- # Python 2 compatibility
- from sys import exc_info
- etype, value, traceback = exc_info()
- exc.__traceback__ = traceback
- print(pp.ParseException.explain(exc))
- self.assertEqual(exc.loc, test_spec.expected_fail_locn)
- else:
- self.assertTrue(False, "failed to raise expected exception")
-
-
-# =========== TEST DEFINITIONS START HERE ==============
-
-class TestLiteral(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc = "Simple match",
- expr = pp.Literal("xyz"),
- text = "xyz",
- expected_list = ["xyz"],
- ),
- PpTestSpec(
- desc = "Simple match after skipping whitespace",
- expr = pp.Literal("xyz"),
- text = " xyz",
- expected_list = ["xyz"],
- ),
- PpTestSpec(
- desc = "Simple fail - parse an empty string",
- expr = pp.Literal("xyz"),
- text = "",
- expected_fail_locn = 0,
- ),
- PpTestSpec(
- desc = "Simple fail - parse a mismatching string",
- expr = pp.Literal("xyz"),
- text = "xyu",
- expected_fail_locn = 0,
- ),
- PpTestSpec(
- desc = "Simple fail - parse a partially matching string",
- expr = pp.Literal("xyz"),
- text = "xy",
- expected_fail_locn = 0,
- ),
- PpTestSpec(
- desc = "Fail - parse a partially matching string by matching individual letters",
- expr = pp.Literal("x") + pp.Literal("y") + pp.Literal("z"),
- text = "xy",
- expected_fail_locn = 2,
- ),
- ]
-
-class TestCaselessLiteral(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc = "Match colors, converting to consistent case",
- expr = (pp.CaselessLiteral("RED")
- | pp.CaselessLiteral("GREEN")
- | pp.CaselessLiteral("BLUE"))[...],
- text = "red Green BluE blue GREEN green rEd",
- expected_list = ['RED', 'GREEN', 'BLUE', 'BLUE', 'GREEN', 'GREEN', 'RED'],
- ),
- ]
-
-class TestWord(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc = "Simple Word match",
- expr = pp.Word("xy"),
- text = "xxyxxyy",
- expected_list = ["xxyxxyy"],
- ),
- PpTestSpec(
- desc = "Simple Word match of two separate Words",
- expr = pp.Word("x") + pp.Word("y"),
- text = "xxxxxyy",
- expected_list = ["xxxxx", "yy"],
- ),
- PpTestSpec(
- desc = "Simple Word match of two separate Words - implicitly skips whitespace",
- expr = pp.Word("x") + pp.Word("y"),
- text = "xxxxx yy",
- expected_list = ["xxxxx", "yy"],
- ),
- ]
-
-class TestCombine(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc="Parsing real numbers - fail, parsed numbers are in pieces",
- expr=(pp.Word(pp.nums) + '.' + pp.Word(pp.nums))[...],
- text="1.2 2.3 3.1416 98.6",
- expected_list=['1', '.', '2', '2', '.', '3', '3', '.', '1416', '98', '.', '6'],
- ),
- PpTestSpec(
- desc="Parsing real numbers - better, use Combine to combine multiple tokens into one",
- expr=pp.Combine(pp.Word(pp.nums) + '.' + pp.Word(pp.nums))[...],
- text="1.2 2.3 3.1416 98.6",
- expected_list=['1.2', '2.3', '3.1416', '98.6'],
- ),
- ]
-
-class TestRepetition(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc = "Match several words",
- expr = (pp.Word("x") | pp.Word("y"))[...],
- text = "xxyxxyyxxyxyxxxy",
- expected_list = ['xx', 'y', 'xx', 'yy', 'xx', 'y', 'x', 'y', 'xxx', 'y'],
- ),
- PpTestSpec(
- desc = "Match several words, skipping whitespace",
- expr = (pp.Word("x") | pp.Word("y"))[...],
- text = "x x y xxy yxx y xyx xxy",
- expected_list = ['x', 'x', 'y', 'xx', 'y', 'y', 'xx', 'y', 'x', 'y', 'x', 'xx', 'y'],
- ),
- PpTestSpec(
- desc = "Match several words, skipping whitespace (old style)",
- expr = pp.OneOrMore(pp.Word("x") | pp.Word("y")),
- text = "x x y xxy yxx y xyx xxy",
- expected_list = ['x', 'x', 'y', 'xx', 'y', 'y', 'xx', 'y', 'x', 'y', 'x', 'xx', 'y'],
- ),
- PpTestSpec(
- desc = "Match words and numbers - show use of results names to collect types of tokens",
- expr = (pp.Word(pp.alphas)("alpha*")
- | pp.pyparsing_common.integer("int*"))[...],
- text = "sdlfj23084ksdfs08234kjsdlfkjd0934",
- expected_list = ['sdlfj', 23084, 'ksdfs', 8234, 'kjsdlfkjd', 934],
- expected_dict = { 'alpha': ['sdlfj', 'ksdfs', 'kjsdlfkjd'], 'int': [23084, 8234, 934] }
- ),
- PpTestSpec(
- desc = "Using delimitedList (comma is the default delimiter)",
- expr = pp.delimitedList(pp.Word(pp.alphas)),
- text = "xxyx,xy,y,xxyx,yxx, xy",
- expected_list = ['xxyx', 'xy', 'y', 'xxyx', 'yxx', 'xy'],
- ),
- PpTestSpec(
- desc = "Using delimitedList, with ':' delimiter",
- expr = pp.delimitedList(pp.Word(pp.hexnums, exact=2), delim=':', combine=True),
- text = "0A:4B:73:21:FE:76",
- expected_list = ['0A:4B:73:21:FE:76'],
- ),
- ]
-
-class TestResultsName(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc = "Match with results name",
- expr = pp.Literal("xyz").setResultsName("value"),
- text = "xyz",
- expected_dict = {'value': 'xyz'},
- expected_list = ['xyz'],
- ),
- PpTestSpec(
- desc = "Match with results name - using naming short-cut",
- expr = pp.Literal("xyz")("value"),
- text = "xyz",
- expected_dict = {'value': 'xyz'},
- expected_list = ['xyz'],
- ),
- PpTestSpec(
- desc = "Define multiple results names",
- expr = pp.Word(pp.alphas, pp.alphanums)("key") + '=' + pp.pyparsing_common.integer("value"),
- text = "range=5280",
- expected_dict = {'key': 'range', 'value': 5280},
- expected_list = ['range', '=', 5280],
- ),
- ]
-
-class TestGroups(PyparsingExpressionTestCase):
- EQ = pp.Suppress('=')
- tests = [
- PpTestSpec(
- desc = "Define multiple results names in groups",
- expr = pp.Group(pp.Word(pp.alphas)("key")
- + EQ
- + pp.pyparsing_common.number("value"))[...],
- text = "range=5280 long=-138.52 lat=46.91",
- expected_list = [['range', 5280], ['long', -138.52], ['lat', 46.91]],
- ),
- PpTestSpec(
- desc = "Define multiple results names in groups - use Dict to define results names using parsed keys",
- expr = pp.Dict(pp.Group(pp.Word(pp.alphas)
- + EQ
- + pp.pyparsing_common.number)[...]),
- text = "range=5280 long=-138.52 lat=46.91",
- expected_list = [['range', 5280], ['long', -138.52], ['lat', 46.91]],
- expected_dict = {'lat': 46.91, 'long': -138.52, 'range': 5280}
- ),
- PpTestSpec(
- desc = "Define multiple value types",
- expr = pp.Dict(pp.Group(pp.Word(pp.alphas)
- + EQ
- + (pp.pyparsing_common.number | pp.oneOf("True False") | pp.QuotedString("'"))
- )[...]
- ),
- text = "long=-122.47 lat=37.82 public=True name='Golden Gate Bridge'",
- expected_list = [['long', -122.47], ['lat', 37.82], ['public', 'True'], ['name', 'Golden Gate Bridge']],
- expected_dict = {'long': -122.47, 'lat': 37.82, 'public': 'True', 'name': 'Golden Gate Bridge'}
- ),
- ]
-
-class TestParseAction(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc="Parsing real numbers - use parse action to convert to float at parse time",
- expr=pp.Combine(pp.Word(pp.nums) + '.' + pp.Word(pp.nums)).addParseAction(lambda t: float(t[0]))[...],
- text="1.2 2.3 3.1416 98.6",
- expected_list= [1.2, 2.3, 3.1416, 98.6], # note, these are now floats, not strs
- ),
- PpTestSpec(
- desc = "Match with numeric string converted to int",
- expr = pp.Word("0123456789").addParseAction(lambda t: int(t[0])),
- text = "12345",
- expected_list = [12345], # note - result is type int, not str
- ),
- PpTestSpec(
- desc = "Use two parse actions to convert numeric string, then convert to datetime",
- expr = pp.Word(pp.nums).addParseAction(lambda t: int(t[0]),
- lambda t: datetime.utcfromtimestamp(t[0])),
- text = "1537415628",
- expected_list = [datetime(2018, 9, 20, 3, 53, 48)],
- ),
- PpTestSpec(
- desc = "Use tokenMap for parse actions that operate on a single-length token",
- expr = pp.Word(pp.nums).addParseAction(pp.tokenMap(int),
- pp.tokenMap(datetime.utcfromtimestamp)),
- text = "1537415628",
- expected_list = [datetime(2018, 9, 20, 3, 53, 48)],
- ),
- PpTestSpec(
- desc = "Using a built-in function that takes a sequence of strs as a parse action",
- expr = pp.Word(pp.hexnums, exact=2)[...].addParseAction(':'.join),
- text = "0A4B7321FE76",
- expected_list = ['0A:4B:73:21:FE:76'],
- ),
- PpTestSpec(
- desc = "Using a built-in function that takes a sequence of strs as a parse action",
- expr = pp.Word(pp.hexnums, exact=2)[...].addParseAction(sorted),
- text = "0A4B7321FE76",
- expected_list = ['0A', '21', '4B', '73', '76', 'FE'],
- ),
- ]
-
-class TestResultsModifyingParseAction(PyparsingExpressionTestCase):
- def compute_stats_parse_action(t):
- # by the time this parse action is called, parsed numeric words
- # have been converted to ints by a previous parse action, so
- # they can be treated as ints
- t['sum'] = sum(t)
- t['ave'] = sum(t) / len(t)
- t['min'] = min(t)
- t['max'] = max(t)
-
- tests = [
- PpTestSpec(
- desc = "A parse action that adds new key-values",
- expr = pp.pyparsing_common.integer[...].addParseAction(compute_stats_parse_action),
- text = "27 1 14 22 89",
- expected_list = [27, 1, 14, 22, 89],
- expected_dict = {'ave': 30.6, 'max': 89, 'min': 1, 'sum': 153}
- ),
- ]
-
-class TestRegex(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc="Parsing real numbers - using Regex instead of Combine",
- expr=pp.Regex(r'\d+\.\d+').addParseAction(lambda t: float(t[0]))[...],
- text="1.2 2.3 3.1416 98.6",
- expected_list=[1.2, 2.3, 3.1416, 98.6], # note, these are now floats, not strs
- ),
- ]
-
-class TestParseCondition(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc = "Define a condition to only match numeric values that are multiples of 7",
- expr = pp.Word(pp.nums).addCondition(lambda t: int(t[0]) % 7 == 0)[...],
- text = "14 35 77 12 28",
- expected_list = ['14', '35', '77'],
- ),
- PpTestSpec(
- desc = "Separate conversion to int and condition into separate parse action/conditions",
- expr = pp.Word(pp.nums).addParseAction(lambda t: int(t[0]))
- .addCondition(lambda t: t[0] % 7 == 0)[...],
- text = "14 35 77 12 28",
- expected_list = [14, 35, 77],
- ),
- ]
-
-class TestTransformStringUsingParseActions(PyparsingExpressionTestCase):
- markup_convert_map = {
- '*' : 'B',
- '_' : 'U',
- '/' : 'I',
- }
- def markup_convert(t):
- htmltag = TestTransformStringUsingParseActions.markup_convert_map[t.markup_symbol]
- return "<{0}>{1}</{2}>".format(htmltag, t.body, htmltag)
-
- tests = [
- PpTestSpec(
- desc = "Use transformString to convert simple markup to HTML",
- expr = (pp.oneOf(markup_convert_map)('markup_symbol')
- + "(" + pp.CharsNotIn(")")('body') + ")").addParseAction(markup_convert),
- text = "Show in *(bold), _(underscore), or /(italic) type",
- expected_list = ['Show in <B>bold</B>, <U>underscore</U>, or <I>italic</I> type'],
- parse_fn = 'transformString',
- ),
- ]
-
-class TestCommonHelperExpressions(PyparsingExpressionTestCase):
- tests = [
- PpTestSpec(
- desc = "A comma-delimited list of words",
- expr = pp.delimitedList(pp.Word(pp.alphas)),
- text = "this, that, blah,foo, bar",
- expected_list = ['this', 'that', 'blah', 'foo', 'bar'],
- ),
- PpTestSpec(
- desc = "A counted array of words",
- expr = pp.countedArray(pp.Word('ab'))[...],
- text = "2 aaa bbb 0 3 abab bbaa abbab",
- expected_list = [['aaa', 'bbb'], [], ['abab', 'bbaa', 'abbab']],
- ),
- PpTestSpec(
- desc = "skipping comments with ignore",
- expr = (pp.pyparsing_common.identifier('lhs')
- + '='
- + pp.pyparsing_common.fnumber('rhs')).ignore(pp.cppStyleComment),
- text = "abc_100 = /* value to be tested */ 3.1416",
- expected_list = ['abc_100', '=', 3.1416],
- expected_dict = {'lhs': 'abc_100', 'rhs': 3.1416},
- ),
- PpTestSpec(
- desc = "some pre-defined expressions in pyparsing_common, and building a dotted identifier with delimted_list",
- expr = (pp.pyparsing_common.number("id_num")
- + pp.delimitedList(pp.pyparsing_common.identifier, '.', combine=True)("name")
- + pp.pyparsing_common.ipv4_address("ip_address")
- ),
- text = "1001 www.google.com 192.168.10.199",
- expected_list = [1001, 'www.google.com', '192.168.10.199'],
- expected_dict = {'id_num': 1001, 'name': 'www.google.com', 'ip_address': '192.168.10.199'},
- ),
- PpTestSpec(
- desc = "using oneOf (shortcut for Literal('a') | Literal('b') | Literal('c'))",
- expr = pp.oneOf("a b c")[...],
- text = "a b a b b a c c a b b",
- expected_list = ['a', 'b', 'a', 'b', 'b', 'a', 'c', 'c', 'a', 'b', 'b'],
- ),
- PpTestSpec(
- desc = "parsing nested parentheses",
- expr = pp.nestedExpr(),
- text = "(a b (c) d (e f g ()))",
- expected_list = [['a', 'b', ['c'], 'd', ['e', 'f', 'g', []]]],
- ),
- PpTestSpec(
- desc = "parsing nested braces",
- expr = (pp.Keyword('if')
- + pp.nestedExpr()('condition')
- + pp.nestedExpr('{', '}')('body')),
- text = 'if ((x == y) || !z) {printf("{}");}',
- expected_list = ['if', [['x', '==', 'y'], '||', '!z'], ['printf(', '"{}"', ');']],
- expected_dict = {'condition': [[['x', '==', 'y'], '||', '!z']],
- 'body': [['printf(', '"{}"', ');']]},
- ),
- ]
-
-
-def _get_decl_line_no(cls):
- import inspect
- return inspect.getsourcelines(cls)[1]
-
-
-# get all test case classes defined in this module and sort them by decl line no
-test_case_classes = list(PyparsingExpressionTestCase.__subclasses__())
-test_case_classes.sort(key=_get_decl_line_no)
-
-# make into a suite and run it - this will run the tests in the same order
-# they are declared in this module
-#
-# runnable from setup.py using "python setup.py test -s simple_unit_tests.suite"
-#
-suite = unittest.TestSuite(cls() for cls in test_case_classes)
-
-
-# ============ MAIN ================
-
-if __name__ == '__main__':
- import sys
- if sys.version_info[0] < 3:
- print("simple_unit_tests.py requires Python 3.x - exiting...")
- exit(0)
-
- result = unittest.TextTestRunner().run(suite)
-
- exit(0 if result.wasSuccessful() else 1)
+++ /dev/null
-# jsonParser.py\r
-#\r
-# Copyright (c) 2006, Paul McGuire\r
-#\r
-\r
-test1 = """\r
-{\r
- "glossary": {\r
- "title": "example glossary",\r
- "GlossDiv": {\r
- "title": "S",\r
- "GlossList": [{\r
- "ID": "SGML",\r
- "SortAs": "SGML",\r
- "GlossTerm": "Standard Generalized Markup Language",\r
- "Acronym": "SGML",\r
- "LargestPrimeLessThan100": 97,\r
- "AvogadroNumber": 6.02E23,\r
- "EvenPrimesGreaterThan2": null,\r
- "PrimesLessThan10" : [2,3,5,7],\r
- "WMDsFound" : false,\r
- "IraqAlQaedaConnections" : null,\r
- "Abbrev": "ISO 8879:1986",\r
- "GlossDef":\r
-"A meta-markup language, used to create markup languages such as DocBook.",\r
- "GlossSeeAlso": ["GML", "XML", "markup"],\r
- "EmptyDict" : {},\r
- "EmptyList" : []\r
- }]\r
- }\r
- }\r
-}\r
-"""\r
-\r
-test2 = """\r
-{"menu": {\r
- "id": "file",\r
- "value": "File:",\r
- "popup": {\r
- "menuitem": [\r
- {"value": "New", "onclick": "CreateNewDoc()"},\r
- {"value": "Open", "onclick": "OpenDoc()"},\r
- {"value": "Close", "onclick": "CloseDoc()"}\r
- ]\r
- }\r
-}}\r
-"""\r
-test3 = """\r
-{"widget": {\r
- "debug": "on",\r
- "window": {\r
- "title": "Sample Konfabulator Widget", "name": "main_window", "width": 500, "height": 500\r
- }, "image": {\r
- "src": "Images/Sun.png",\r
- "name": "sun1", "hOffset": 250, "vOffset": 250, "alignment": "center"\r
- }, "text": {\r
- "data": "Click Here",\r
- "size": 36,\r
- "style": "bold", "name": "text1", "hOffset": 250, "vOffset": 100, "alignment": "center",\r
- "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"\r
- }\r
-}}\r
-"""\r
-test4 = """\r
-{"web-app": {\r
- "servlet": [ // Defines the CDSServlet\r
- {\r
- "servlet-name": "cofaxCDS",\r
- "servlet-class": "org.cofax.cds.CDSServlet",\r
-/*\r
- Defines glossary variables that template designers\r
- can use across the site. You can add new\r
- variables to this set by creating a new init-param, with\r
- the param-name prefixed with "configGlossary:".\r
-*/\r
- "init-param": {\r
- "configGlossary:installationAt": "Philadelphia, PA",\r
- "configGlossary:adminEmail": "ksm@pobox.com",\r
- "configGlossary:poweredBy": "Cofax",\r
- "configGlossary:poweredByIcon": "/images/cofax.gif",\r
- "configGlossary:staticPath": "/content/static",\r
-/*\r
- Defines the template loader and template processor\r
- classes. These are implementations of org.cofax.TemplateProcessor\r
- and org.cofax.TemplateLoader respectively. Simply create new\r
- implementation of these classes and set them here if the default\r
- implementations do not suit your needs. Leave these alone\r
- for the defaults.\r
-*/\r
- "templateProcessorClass": "org.cofax.WysiwygTemplate",\r
- "templateLoaderClass": "org.cofax.FilesTemplateLoader",\r
- "templatePath": "templates",\r
- "templateOverridePath": "",\r
-/*\r
- Defines the names of the default templates to look for\r
- when acquiring WYSIWYG templates. Leave these at their\r
- defaults for most usage.\r
-*/\r
- "defaultListTemplate": "listTemplate.htm",\r
- "defaultFileTemplate": "articleTemplate.htm",\r
-/*\r
- New! useJSP switches on JSP template processing.\r
- jspListTemplate and jspFileTemplate are the names\r
- of the default templates to look for when aquiring JSP\r
- templates. Cofax currently in production at KR has useJSP\r
- set to false, since our sites currently use WYSIWYG\r
- templating exclusively.\r
-*/\r
- "useJSP": false,\r
- "jspListTemplate": "listTemplate.jsp",\r
- "jspFileTemplate": "articleTemplate.jsp",\r
-/*\r
- Defines the packageTag cache. This cache keeps\r
- Cofax from needing to interact with the database\r
- to look up packageTag commands.\r
-*/\r
- "cachePackageTagsTrack": 200,\r
- "cachePackageTagsStore": 200,\r
- "cachePackageTagsRefresh": 60,\r
-/*\r
- Defines the template cache. Keeps Cofax from needing\r
- to go to the file system to load a raw template from\r
- the file system.\r
-*/\r
- "cacheTemplatesTrack": 100,\r
- "cacheTemplatesStore": 50,\r
- "cacheTemplatesRefresh": 15,\r
-/*\r
- Defines the page cache. Keeps Cofax from processing\r
- templates to deliver to users.\r
-*/\r
- "cachePagesTrack": 200,\r
- "cachePagesStore": 100,\r
- "cachePagesRefresh": 10,\r
- "cachePagesDirtyRead": 10,\r
-/*\r
- Defines the templates Cofax will use when\r
- being browsed by a search engine identified in\r
- searchEngineRobotsDb\r
-*/\r
- "searchEngineListTemplate": "forSearchEnginesList.htm",\r
- "searchEngineFileTemplate": "forSearchEngines.htm",\r
- "searchEngineRobotsDb": "WEB-INF/robots.db",\r
-/*\r
- New! useDataStore enables/disables the Cofax database pool\r
-*/\r
- "useDataStore": true,\r
-/*\r
- Defines the implementation of org.cofax.DataStore that Cofax\r
- will use. If this DataStore class does not suit your needs\r
- simply implement a new DataStore class and set here.\r
-*/\r
- "dataStoreClass": "org.cofax.SqlDataStore",\r
-/*\r
- Defines the implementation of org.cofax.Redirection that\r
- Cofax will use. If this Redirection class does not suit\r
- your needs simply implenet a new Redirection class\r
- and set here.\r
-*/\r
- "redirectionClass": "org.cofax.SqlRedirection",\r
-/*\r
- Defines the data store name. Keep this at the default\r
-*/\r
- "dataStoreName": "cofax",\r
-/*\r
- Defines the JDBC driver that Cofax's database pool will use\r
-*/\r
- "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver",\r
-/*\r
- Defines the JDBC connection URL to connect to the database\r
-*/\r
- "dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon",\r
-/*\r
- Defines the user name to connect to the database\r
-*/\r
- "dataStoreUser": "sa",\r
-/*\r
- Defines the password to connect to the database\r
-*/\r
- "dataStorePassword": "dataStoreTestQuery",\r
-/*\r
- A query that will run to test the validity of the\r
- connection in the pool.\r
-*/\r
- "dataStoreTestQuery": "SET NOCOUNT ON;select test='test';",\r
-/*\r
- A log file to print out database information\r
-*/\r
- "dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log",\r
-/*\r
- The number of connection to initialize on startup\r
-*/\r
- "dataStoreInitConns": 10,\r
-/*\r
- The maximum number of connection to use in the pool\r
-*/\r
- "dataStoreMaxConns": 100,\r
-/*\r
- The number of times a connection will be utilized from the\r
- pool before disconnect\r
-*/\r
- "dataStoreConnUsageLimit": 100,\r
-/*\r
- The level of information to print to the log\r
-*/\r
- "dataStoreLogLevel": "debug",\r
-/*\r
- The maximum URL length allowable by the CDS Servlet\r
- Helps to prevent hacking\r
-*/\r
- "maxUrlLength": 500}},\r
-/*\r
- Defines the Email Servlet\r
-*/\r
- {\r
- "servlet-name": "cofaxEmail",\r
- "servlet-class": "org.cofax.cds.EmailServlet",\r
- "init-param": {\r
-/*\r
- The mail host to be used by the mail servlet\r
-*/\r
- "mailHost": "mail1",\r
-/*\r
- An override\r
-*/\r
- "mailHostOverride": "mail2"}},\r
-/*\r
- Defines the Admin Servlet - used to refresh cache on\r
- demand and see statistics\r
-*/\r
- {\r
- "servlet-name": "cofaxAdmin",\r
- "servlet-class": "org.cofax.cds.AdminServlet"},\r
-/*\r
- Defines the File Servlet - used to display files like Apache\r
-*/\r
- {\r
- "servlet-name": "fileServlet",\r
- "servlet-class": "org.cofax.cds.FileServlet"},\r
- {\r
- "servlet-name": "cofaxTools",\r
- "servlet-class": "org.cofax.cms.CofaxToolsServlet",\r
- "init-param": {\r
-/*\r
- Path to the template folder relative to the tools tomcat installation.\r
-*/\r
- "templatePath": "toolstemplates/",\r
-/*\r
- Logging boolean 1 = on, 0 = off\r
-*/\r
- "log": 1,\r
-/*\r
- Location of log. If empty, log will be written System.out\r
-*/\r
- "logLocation": "/usr/local/tomcat/logs/CofaxTools.log",\r
-/*\r
- Max size of log in BITS. If size is empty, no limit to log.\r
- If size is defined, log will be overwritten upon reaching defined size.\r
-*/\r
- "logMaxSize": "",\r
-/*\r
- DataStore logging boolean 1 = on, 0 = off\r
-*/\r
- "dataLog": 1,\r
-/*\r
- DataStore location of log. If empty, log will be written System.out\r
-*/\r
- "dataLogLocation": "/usr/local/tomcat/logs/dataLog.log",\r
-/*\r
- Max size of log in BITS. If size is empty, no limit to log.\r
- If size is defined, log will be overwritten upon reaching defined size.\r
-*/\r
- "dataLogMaxSize": "",\r
-/*\r
- Http string relative to server root to call for page cache\r
- removal to Cofax Servlet.\r
-*/\r
- "removePageCache": "/content/admin/remove?cache=pages&id=",\r
-/*\r
- Http string relative to server root to call for template\r
- cache removal to Cofax Servlet.\r
-*/\r
- "removeTemplateCache": "/content/admin/remove?cache=templates&id=",\r
-/*\r
- Location of folder from root of drive that will be used for\r
- ftp transfer from beta server or user hard drive to live servers.\r
- Note that Edit Article will not function without this variable\r
- set correctly. MultiPart request relies upon access to this folder.\r
-*/\r
- "fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder",\r
-/*\r
- Defines whether the Server should look in another path for\r
- config files or variables.\r
-*/\r
- "lookInContext": 1,\r
-/*\r
- Number of the ID of the top level administration group in tblPermGroups.\r
-*/\r
- "adminGroupID": 4,\r
-/*\r
- Is the tools app running on the 'beta server'.\r
-*/\r
- "betaServer": true}}],\r
- "servlet-mapping": {\r
-/*\r
- URL mapping for the CDS Servlet\r
-*/\r
- "cofaxCDS": "/",\r
-/*\r
- URL mapping for the Email Servlet\r
-*/\r
- "cofaxEmail": "/cofaxutil/aemail/*",\r
-/*\r
- URL mapping for the Admin servlet\r
-*/\r
- "cofaxAdmin": "/admin/*",\r
-/*\r
- URL mapping for the Files servlet\r
-*/\r
- "fileServlet": "/static/*",\r
- "cofaxTools": "/tools/*"},\r
-/*\r
- New! The cofax taglib descriptor file\r
-*/\r
- "taglib": {\r
- "taglib-uri": "cofax.tld",\r
- "taglib-location": "/WEB-INF/tlds/cofax.tld"}}}\r
-\r
-"""\r
-\r
-test5 = """\r
-{"menu": {\r
- "header": "SVG Viewer",\r
- "items": [\r
- {"id": "Open"},\r
- {"id": "OpenNew", "label": "Open New"},\r
- null,\r
- {"id": "ZoomIn", "label": "Zoom In"},\r
- {"id": "ZoomOut", "label": "Zoom Out"},\r
- {"id": "OriginalView", "label": "Original View"},\r
- null,\r
- {"id": "Quality"},\r
- {"id": "Pause"},\r
- {"id": "Mute"},\r
- null,\r
- {"id": "Find", "label": "Find..."},\r
- {"id": "FindAgain", "label": "Find Again"},\r
- {"id": "Copy"},\r
- {"id": "CopyAgain", "label": "Copy Again"},\r
- {"id": "CopySVG", "label": "Copy SVG"},\r
- {"id": "ViewSVG", "label": "View SVG"},\r
- {"id": "ViewSource", "label": "View Source"},\r
- {"id": "SaveAs", "label": "Save As"},\r
- null,\r
- {"id": "Help"},\r
- {"id": "About", "label": "About Adobe CVG Viewer..."}\r
- ]\r
-}}\r
-"""\r
+++ /dev/null
-[users]\r
-source_dir = '/home/karthik/Projects/python'\r
-data_dir = '/home/karthik/Projects/data'\r
-result_dir = '/home/karthik/Projects/Results'\r
-param_file = $result_dir/param_file\r
-res_file = $result_dir/result_file\r
-comment = 'this is a comment'\r
-; a line starting with ';' is a comment\r
-K = 8\r
-simulate_K = 0\r
-N = 4000\r
-mod_scheme = 'QPSK'\r
-\r
-Na = K+2
\ No newline at end of file
+++ /dev/null
-123 456 789
\ No newline at end of file
--- /dev/null
+## Development
+
+After forking the pyparsing repo, and cloning your fork locally, install the libraries needed to run tests
+
+ pip install -r tests/requirements.txt
+ pre-commit install
+
+Run the tests to ensure your environment is setup
+
+ python -m unittest discover tests
--- /dev/null
+# jsonParser.py
+#
+# Copyright (c) 2006, Paul McGuire
+#
+
+test1 = """
+{
+ "glossary": {
+ "title": "example glossary",
+ "GlossDiv": {
+ "title": "S",
+ "GlossList": [
+ {
+ "ID": "SGML",
+ "SortAs": "SGML",
+ "GlossDef": "A meta-markup language, used to create markup languages such as DocBook.",
+ "GlossSeeAlso": ["GML", "XML", "markup"],
+ "GlossTerm": "Standard Generalized Markup Language",
+ "Acronym": "SGML",
+ "LargestPrimeLessThan100": 97,
+ "AvogadroNumber": 6.02E23,
+ "EvenPrimesGreaterThan2": [],
+ "PrimesLessThan10" : [2,3,5,7],
+ "FermatTheoremInMargin" : false,
+ "MapRequiringFiveColors" : null,
+ "Abbrev": "ISO 8879:1986",
+ "EmptyDict" : {},
+ "EmptyList" : []
+ }
+ ]
+ }
+ }
+}
+"""
+
+test2 = """
+{"menu": {
+ "id": "file",
+ "value": "File:",
+ "popup": {
+ "menuitem": [
+ {"value": "New", "onclick": "CreateNewDoc()"},
+ {"value": "Open", "onclick": "OpenDoc()"},
+ {"value": "Close", "onclick": "CloseDoc()"}
+ ]
+ }
+}}
+"""
+
+test3 = """
+{"widget": {
+ "debug": "on",
+ "window": {
+ "title": "Sample Konfabulator Widget", "name": "main_window", "width": 500, "height": 500
+ }, "image": {
+ "src": "Images/Sun.png",
+ "name": "sun1", "hOffset": 250, "vOffset": 250, "alignment": "center"
+ }, "text": {
+ "data": "Click Here",
+ "size": 36,
+ "style": "bold", "name": "text1", "hOffset": 250, "vOffset": 100, "alignment": "center",
+ "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
+ }
+}}
+"""
+test4 = """
+{"web-app": {
+ "servlet": [ // Defines the CDSServlet
+ {
+ "servlet-name": "cofaxCDS",
+ "servlet-class": "org.cofax.cds.CDSServlet",
+ /*
+ Defines glossary variables that template designers
+ can use across the site. You can add new
+ variables to this set by creating a new init-param, with
+ the param-name prefixed with "configGlossary:".
+ */
+ "init-param": {
+ "configGlossary:installationAt": "Philadelphia, PA",
+ "configGlossary:adminEmail": "ksm@pobox.com",
+ "configGlossary:poweredBy": "Cofax",
+ "configGlossary:poweredByIcon": "/images/cofax.gif",
+ "configGlossary:staticPath": "/content/static",
+ /*
+ Defines the template loader and template processor
+ classes. These are implementations of org.cofax.TemplateProcessor
+ and org.cofax.TemplateLoader respectively. Simply create new
+ implementation of these classes and set them here if the default
+ implementations do not suit your needs. Leave these alone
+ for the defaults.
+ */
+ "templateProcessorClass": "org.cofax.WysiwygTemplate",
+ "templateLoaderClass": "org.cofax.FilesTemplateLoader",
+ "templatePath": "templates",
+ "templateOverridePath": "",
+ /*
+ Defines the names of the default templates to look for
+ when acquiring WYSIWYG templates. Leave these at their
+ defaults for most usage.
+ */
+ "defaultListTemplate": "listTemplate.htm",
+ "defaultFileTemplate": "articleTemplate.htm",
+ /*
+ New! useJSP switches on JSP template processing.
+ jspListTemplate and jspFileTemplate are the names
+ of the default templates to look for when acquiring JSP
+ templates. Cofax currently in production at KR has useJSP
+ set to false, since our sites currently use WYSIWYG
+ templating exclusively.
+ */
+ "useJSP": false,
+ "jspListTemplate": "listTemplate.jsp",
+ "jspFileTemplate": "articleTemplate.jsp",
+ /*
+ Defines the packageTag cache. This cache keeps
+ Cofax from needing to interact with the database
+ to look up packageTag commands.
+ */
+ "cachePackageTagsTrack": 200,
+ "cachePackageTagsStore": 200,
+ "cachePackageTagsRefresh": 60,
+ /*
+ Defines the template cache. Keeps Cofax from needing
+ to go to the file system to load a raw template from
+ the file system.
+ */
+ "cacheTemplatesTrack": 100,
+ "cacheTemplatesStore": 50,
+ "cacheTemplatesRefresh": 15,
+ /*
+ Defines the page cache. Keeps Cofax from processing
+ templates to deliver to users.
+ */
+ "cachePagesTrack": 200,
+ "cachePagesStore": 100,
+ "cachePagesRefresh": 10,
+ "cachePagesDirtyRead": 10,
+ /*
+ Defines the templates Cofax will use when
+ being browsed by a search engine identified in
+ searchEngineRobotsDb
+ */
+ "searchEngineListTemplate": "forSearchEnginesList.htm",
+ "searchEngineFileTemplate": "forSearchEngines.htm",
+ "searchEngineRobotsDb": "WEB-INF/robots.db",
+ /*
+ New! useDataStore enables/disables the Cofax database pool
+ */
+ "useDataStore": true,
+ /*
+ Defines the implementation of org.cofax.DataStore that Cofax
+ will use. If this DataStore class does not suit your needs
+ simply implement a new DataStore class and set here.
+ */
+ "dataStoreClass": "org.cofax.SqlDataStore",
+ /*
+ Defines the implementation of org.cofax.Redirection that
+ Cofax will use. If this Redirection class does not suit
+ your needs simply implenet a new Redirection class
+ and set here.
+ */
+ "redirectionClass": "org.cofax.SqlRedirection",
+ /*
+ Defines the data store name. Keep this at the default
+ */
+ "dataStoreName": "cofax",
+ /*
+ Defines the JDBC driver that Cofax's database pool will use
+ */
+ "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver",
+ /*
+ Defines the JDBC connection URL to connect to the database
+ */
+ "dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon",
+ /*
+ Defines the user name to connect to the database
+ */
+ "dataStoreUser": "sa",
+ /*
+ Defines the password to connect to the database
+ */
+ "dataStorePassword": "dataStoreTestQuery",
+ /*
+ A query that will run to test the validity of the
+ connection in the pool.
+ */
+ "dataStoreTestQuery": "SET NOCOUNT ON;select test='test';",
+ /*
+ A log file to print out database information
+ */
+ "dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log",
+ /*
+ The number of connection to initialize on startup
+ */
+ "dataStoreInitConns": 10,
+ /*
+ The maximum number of connection to use in the pool
+ */
+ "dataStoreMaxConns": 100,
+ /*
+ The number of times a connection will be utilized from the
+ pool before disconnect
+ */
+ "dataStoreConnUsageLimit": 100,
+ /*
+ The level of information to print to the log
+ */
+ "dataStoreLogLevel": "debug",
+ /*
+ The maximum URL length allowable by the CDS Servlet
+ Helps to prevent hacking
+ */
+ "maxUrlLength": 500}},
+ /*
+ Defines the Email Servlet
+ */
+ {
+ "servlet-name": "cofaxEmail",
+ "servlet-class": "org.cofax.cds.EmailServlet",
+ "init-param": {
+ /*
+ The mail host to be used by the mail servlet
+ */
+ "mailHost": "mail1",
+ /*
+ An override
+ */
+ "mailHostOverride": "mail2"}},
+ /*
+ Defines the Admin Servlet - used to refresh cache on
+ demand and see statistics
+ */
+ {
+ "servlet-name": "cofaxAdmin",
+ "servlet-class": "org.cofax.cds.AdminServlet"},
+ /*
+ Defines the File Servlet - used to display files like Apache
+ */
+ {
+ "servlet-name": "fileServlet",
+ "servlet-class": "org.cofax.cds.FileServlet"},
+ {
+ "servlet-name": "cofaxTools",
+ "servlet-class": "org.cofax.cms.CofaxToolsServlet",
+ "init-param": {
+ /*
+ Path to the template folder relative to the tools tomcat installation.
+ */
+ "templatePath": "toolstemplates/",
+ /*
+ Logging boolean 1 = on, 0 = off
+ */
+ "log": 1,
+ /*
+ Location of log. If empty, log will be written System.out
+ */
+ "logLocation": "/usr/local/tomcat/logs/CofaxTools.log",
+ /*
+ Max size of log in BITS. If size is empty, no limit to log.
+ If size is defined, log will be overwritten upon reaching defined size.
+ */
+ "logMaxSize": "",
+ /*
+ DataStore logging boolean 1 = on, 0 = off
+ */
+ "dataLog": 1,
+ /*
+ DataStore location of log. If empty, log will be written System.out
+ */
+ "dataLogLocation": "/usr/local/tomcat/logs/dataLog.log",
+ /*
+ Max size of log in BITS. If size is empty, no limit to log.
+ If size is defined, log will be overwritten upon reaching defined size.
+ */
+ "dataLogMaxSize": "",
+ /*
+ Http string relative to server root to call for page cache
+ removal to Cofax Servlet.
+ */
+ "removePageCache": "/content/admin/remove?cache=pages&id=",
+ /*
+ Http string relative to server root to call for template
+ cache removal to Cofax Servlet.
+ */
+ "removeTemplateCache": "/content/admin/remove?cache=templates&id=",
+ /*
+ Location of folder from root of drive that will be used for
+ ftp transfer from beta server or user hard drive to live servers.
+ Note that Edit Article will not function without this variable
+ set correctly. MultiPart request relies upon access to this folder.
+ */
+ "fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder",
+ /*
+ Defines whether the Server should look in another path for
+ config files or variables.
+ */
+ "lookInContext": 1,
+ /*
+ Number of the ID of the top level administration group in tblPermGroups.
+ */
+ "adminGroupID": 4,
+ /*
+ Is the tools app running on the 'beta server'.
+ */
+ "betaServer": true}}],
+ "servlet-mapping": {
+ /*
+ URL mapping for the CDS Servlet
+ */
+ "cofaxCDS": "/",
+ /*
+ URL mapping for the Email Servlet
+ */
+ "cofaxEmail": "/cofaxutil/aemail/*",
+ /*
+ URL mapping for the Admin servlet
+ */
+ "cofaxAdmin": "/admin/*",
+ /*
+ URL mapping for the Files servlet
+ */
+ "fileServlet": "/static/*",
+ "cofaxTools": "/tools/*"},
+ /*
+ New! The cofax taglib descriptor file
+ */
+ "taglib": {
+ "taglib-uri": "cofax.tld",
+ "taglib-location": "/WEB-INF/tlds/cofax.tld"}}}
+
+"""
+
+test5 = """
+{"menu": {
+ "header": "SVG Viewer",
+ "items": [
+ {"id": "Open"},
+ {"id": "OpenNew", "label": "Open New"},
+ null,
+ {"id": "ZoomIn", "label": "Zoom In"},
+ {"id": "ZoomOut", "label": "Zoom Out"},
+ {"id": "OriginalView", "label": "Original View"},
+ null,
+ {"id": "Quality"},
+ {"id": "Pause"},
+ {"id": "Mute"},
+ null,
+ {"id": "Find", "label": "Find..."},
+ {"id": "FindAgain", "label": "Find Again"},
+ {"id": "Copy"},
+ {"id": "CopyAgain", "label": "Copy Again"},
+ {"id": "CopySVG", "label": "Copy SVG"},
+ {"id": "ViewSVG", "label": "View SVG"},
+ {"id": "ViewSource", "label": "View Source"},
+ {"id": "SaveAs", "label": "Save As"},
+ null,
+ {"id": "Help"},
+ {"id": "About", "label": "About Adobe CVG Viewer..."}
+ ]
+}}
+"""
--- /dev/null
+[users]
+source_dir = '/home/karthik/Projects/python'
+data_dir = '/home/karthik/Projects/data'
+result_dir = '/home/karthik/Projects/Results'
+param_file = $result_dir/param_file
+res_file = $result_dir/result_file
+comment = 'this is a comment'
+; a line starting with ';' is a comment
+K = 8
+simulate_K = 0
+N = 4000
+mod_scheme = 'QPSK'
+
+Na = K+2
\ No newline at end of file
--- /dev/null
+123 456 789
\ No newline at end of file
--- /dev/null
+coverage==4.4.2
+tox==3.5.2
+pre-commit
--- /dev/null
+import unittest
+from typing import List
+
+from examples.jsonParser import jsonObject
+from examples.simpleBool import boolExpr
+from examples.simpleSQL import simpleSQL
+from examples.mozillaCalendarParser import calendars
+from pyparsing.diagram import to_railroad, railroad_to_html, NamedDiagram
+import pyparsing as pp
+import tempfile
+import os
+import sys
+
+
+class TestRailroadDiagrams(unittest.TestCase):
+ def railroad_debug(self) -> bool:
+ """
+ Returns True if we're in debug mode (determined by either setting
+ environment var, or running in a debugger which sets sys.settrace)
+ """
+ return os.environ.get("RAILROAD_DEBUG", False) or sys.gettrace()
+
+ def get_temp(self):
+ """
+ Returns an appropriate temporary file for writing a railroad diagram
+ """
+ return tempfile.NamedTemporaryFile(
+ dir=".",
+ delete=not self.railroad_debug(),
+ mode="w",
+ encoding="utf-8",
+ suffix=".html",
+ )
+
+ def generate_railroad(
+ self, expr: pp.ParserElement, label: str, show_results_names: bool = False
+ ) -> List[NamedDiagram]:
+ """
+ Generate an intermediate list of NamedDiagrams from a pyparsing expression.
+ """
+ with self.get_temp() as temp:
+ railroad = to_railroad(expr, show_results_names=show_results_names)
+ temp.write(railroad_to_html(railroad))
+
+ if self.railroad_debug():
+ print(f"{label}: {temp.name}")
+
+ return railroad
+
+ def test_bool_expr(self):
+ railroad = self.generate_railroad(boolExpr, "boolExpr")
+ assert len(railroad) == 5
+
+ def test_json(self):
+ railroad = self.generate_railroad(jsonObject, "jsonObject")
+ assert len(railroad) == 9
+
+ def test_sql(self):
+ railroad = self.generate_railroad(simpleSQL, "simpleSQL")
+ assert len(railroad) == 18
+
+ def test_calendars(self):
+ railroad = self.generate_railroad(calendars, "calendars")
+ assert len(railroad) == 13
+
+ def test_nested_forward_with_inner_and_outer_names(self):
+ outer = pp.Forward().setName("outer")
+ inner = pp.Word(pp.alphas)[...].setName("inner")
+ outer <<= inner
+
+ railroad = self.generate_railroad(outer, "inner_outer_names")
+ assert len(railroad) == 2
+
+ def test_nested_forward_with_inner_name_only(self):
+ outer = pp.Forward()
+ inner = pp.Word(pp.alphas)[...].setName("inner")
+ outer <<= inner
+
+ railroad = self.generate_railroad(outer, "inner_only")
+ assert len(railroad) == 2
+
+ def test_each_grammar(self):
+
+ grammar = pp.Each(
+ [
+ pp.Word(pp.nums),
+ pp.Word(pp.alphas),
+ pp.pyparsing_common.uuid,
+ ]
+ ).setName("int-word-uuid in any order")
+ railroad = self.generate_railroad(grammar, "each_expression")
+ assert len(railroad) == 2
+
+ def test_none_name(self):
+ grammar = pp.Or(["foo", "bar"])
+ railroad = to_railroad(grammar)
+ assert len(railroad) == 1
+ assert railroad[0].name is not None
+
+ def test_none_name2(self):
+ grammar = pp.Or(["foo", "bar"]) + pp.Word(pp.nums).setName("integer")
+ railroad = to_railroad(grammar)
+ assert len(railroad) == 2
+ assert railroad[0].name is not None
--- /dev/null
+#
+# test_examples.py
+#
+from importlib import import_module
+import unittest
+
+
+class TestExamples(unittest.TestCase):
+ def _run(self, name):
+ mod = import_module("examples." + name)
+ getattr(mod, "main", lambda *args, **kwargs: None)()
+
+ def test_numerics(self):
+ self._run("numerics")
+
+ def test_tap(self):
+ self._run("TAP")
+
+ def test_roman_numerals(self):
+ self._run("romanNumerals")
+
+ def test_sexp_parser(self):
+ self._run("sexpParser")
+
+ def test_oc(self):
+ self._run("oc")
+
+ def test_delta_time(self):
+ self._run("delta_time")
+
+ def test_eval_arith(self):
+ self._run("eval_arith")
+
+ def test_select_parser(self):
+ self._run("select_parser")
--- /dev/null
+#
+# test_simple_unit.py
+#
+# While these unit tests *do* perform low-level unit testing of the classes in pyparsing,
+# this testing module should also serve an instructional purpose, to clearly show simple passing
+# and failing parse cases of some basic pyparsing expressions.
+#
+# Copyright (c) 2018 Paul T. McGuire
+#
+import unittest
+import pyparsing as pp
+from collections import namedtuple
+from datetime import datetime
+
+ppt = pp.pyparsing_test
+TestParseResultsAsserts = ppt.TestParseResultsAsserts
+
+# Test spec data class for specifying simple pyparsing test cases
+PpTestSpec = namedtuple(
+ "PpTestSpec",
+ "desc expr text parse_fn " "expected_list expected_dict expected_fail_locn",
+)
+PpTestSpec.__new__.__defaults__ = ("", pp.Empty(), "", "parseString", None, None, None)
+
+
+class PyparsingExpressionTestCase(ppt.TestParseResultsAsserts, unittest.TestCase):
+ """
+ Base pyparsing testing class to parse various pyparsing expressions against
+ given text strings. Subclasses must define a class attribute 'tests' which
+ is a list of PpTestSpec instances.
+ """
+
+ tests = []
+
+ def runTest(self):
+ if self.__class__ is PyparsingExpressionTestCase:
+ return
+
+ for test_spec in self.tests:
+ # for each spec in the class's tests list, create a subtest
+ # that will either:
+ # - parse the string with expected success, display the
+ # results, and validate the returned ParseResults
+ # - or parse the string with expected failure, display the
+ # error message and mark the error location, and validate
+ # the location against an expected value
+ with self.subTest(test_spec=test_spec):
+ test_spec.expr.streamline()
+ print(
+ "\n{} - {}({})".format(
+ test_spec.desc, type(test_spec.expr).__name__, test_spec.expr
+ )
+ )
+
+ parsefn = getattr(test_spec.expr, test_spec.parse_fn)
+ if test_spec.expected_fail_locn is None:
+ # expect success
+ result = parsefn(test_spec.text)
+ if test_spec.parse_fn == "parseString":
+ print(result.dump())
+ # compare results against given list and/or dict
+ self.assertParseResultsEquals(
+ result,
+ expected_list=test_spec.expected_list,
+ expected_dict=test_spec.expected_dict,
+ )
+ elif test_spec.parse_fn == "transformString":
+ print(result)
+ # compare results against given list and/or dict
+ if test_spec.expected_list is not None:
+ self.assertEqual([result], test_spec.expected_list)
+ elif test_spec.parse_fn == "searchString":
+ print(result)
+ # compare results against given list and/or dict
+ if test_spec.expected_list is not None:
+ self.assertEqual([result], test_spec.expected_list)
+ else:
+ # expect fail
+ with self.assertRaisesParseException():
+ try:
+ parsefn(test_spec.text)
+ except Exception as exc:
+ print(pp.ParseException.explain(exc))
+ self.assertEqual(exc.loc, test_spec.expected_fail_locn)
+ raise
+
+
+# =========== TEST DEFINITIONS START HERE ==============
+
+
+class TestLiteral(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Simple match",
+ expr=pp.Literal("xyz"),
+ text="xyz",
+ expected_list=["xyz"],
+ ),
+ PpTestSpec(
+ desc="Simple match after skipping whitespace",
+ expr=pp.Literal("xyz"),
+ text=" xyz",
+ expected_list=["xyz"],
+ ),
+ PpTestSpec(
+ desc="Simple fail - parse an empty string",
+ expr=pp.Literal("xyz"),
+ text="",
+ expected_fail_locn=0,
+ ),
+ PpTestSpec(
+ desc="Simple fail - parse a mismatching string",
+ expr=pp.Literal("xyz"),
+ text="xyu",
+ expected_fail_locn=0,
+ ),
+ PpTestSpec(
+ desc="Simple fail - parse a partially matching string",
+ expr=pp.Literal("xyz"),
+ text="xy",
+ expected_fail_locn=0,
+ ),
+ PpTestSpec(
+ desc="Fail - parse a partially matching string by matching individual letters",
+ expr=pp.Literal("x") + pp.Literal("y") + pp.Literal("z"),
+ text="xy",
+ expected_fail_locn=2,
+ ),
+ ]
+
+
+class TestCaselessLiteral(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Match colors, converting to consistent case",
+ expr=(
+ pp.CaselessLiteral("RED")
+ | pp.CaselessLiteral("GREEN")
+ | pp.CaselessLiteral("BLUE")
+ )[...],
+ text="red Green BluE blue GREEN green rEd",
+ expected_list=["RED", "GREEN", "BLUE", "BLUE", "GREEN", "GREEN", "RED"],
+ ),
+ ]
+
+
+class TestWord(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Simple Word match",
+ expr=pp.Word("xy"),
+ text="xxyxxyy",
+ expected_list=["xxyxxyy"],
+ ),
+ PpTestSpec(
+ desc="Simple Word match of two separate Words",
+ expr=pp.Word("x") + pp.Word("y"),
+ text="xxxxxyy",
+ expected_list=["xxxxx", "yy"],
+ ),
+ PpTestSpec(
+ desc="Simple Word match of two separate Words - implicitly skips whitespace",
+ expr=pp.Word("x") + pp.Word("y"),
+ text="xxxxx yy",
+ expected_list=["xxxxx", "yy"],
+ ),
+ ]
+
+
+class TestCombine(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Parsing real numbers - fail, parsed numbers are in pieces",
+ expr=(pp.Word(pp.nums) + "." + pp.Word(pp.nums))[...],
+ text="1.2 2.3 3.1416 98.6",
+ expected_list=[
+ "1",
+ ".",
+ "2",
+ "2",
+ ".",
+ "3",
+ "3",
+ ".",
+ "1416",
+ "98",
+ ".",
+ "6",
+ ],
+ ),
+ PpTestSpec(
+ desc="Parsing real numbers - better, use Combine to combine multiple tokens into one",
+ expr=pp.Combine(pp.Word(pp.nums) + "." + pp.Word(pp.nums))[...],
+ text="1.2 2.3 3.1416 98.6",
+ expected_list=["1.2", "2.3", "3.1416", "98.6"],
+ ),
+ ]
+
+
+class TestRepetition(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Match several words",
+ expr=(pp.Word("x") | pp.Word("y"))[...],
+ text="xxyxxyyxxyxyxxxy",
+ expected_list=["xx", "y", "xx", "yy", "xx", "y", "x", "y", "xxx", "y"],
+ ),
+ PpTestSpec(
+ desc="Match several words, skipping whitespace",
+ expr=(pp.Word("x") | pp.Word("y"))[...],
+ text="x x y xxy yxx y xyx xxy",
+ expected_list=[
+ "x",
+ "x",
+ "y",
+ "xx",
+ "y",
+ "y",
+ "xx",
+ "y",
+ "x",
+ "y",
+ "x",
+ "xx",
+ "y",
+ ],
+ ),
+ PpTestSpec(
+ desc="Match several words, skipping whitespace (old style)",
+ expr=pp.OneOrMore(pp.Word("x") | pp.Word("y")),
+ text="x x y xxy yxx y xyx xxy",
+ expected_list=[
+ "x",
+ "x",
+ "y",
+ "xx",
+ "y",
+ "y",
+ "xx",
+ "y",
+ "x",
+ "y",
+ "x",
+ "xx",
+ "y",
+ ],
+ ),
+ PpTestSpec(
+ desc="Match words and numbers - show use of results names to collect types of tokens",
+ expr=(pp.Word(pp.alphas)("alpha*") | pp.pyparsing_common.integer("int*"))[
+ ...
+ ],
+ text="sdlfj23084ksdfs08234kjsdlfkjd0934",
+ expected_list=["sdlfj", 23084, "ksdfs", 8234, "kjsdlfkjd", 934],
+ expected_dict={
+ "alpha": ["sdlfj", "ksdfs", "kjsdlfkjd"],
+ "int": [23084, 8234, 934],
+ },
+ ),
+ PpTestSpec(
+ desc="Using delimited_list (comma is the default delimiter)",
+ expr=pp.delimited_list(pp.Word(pp.alphas)),
+ text="xxyx,xy,y,xxyx,yxx, xy",
+ expected_list=["xxyx", "xy", "y", "xxyx", "yxx", "xy"],
+ ),
+ PpTestSpec(
+ desc="Using delimited_list (comma is the default delimiter) with trailing delimiter",
+ expr=pp.delimited_list(pp.Word(pp.alphas), allow_trailing_delim=True),
+ text="xxyx,xy,y,xxyx,yxx, xy,",
+ expected_list=["xxyx", "xy", "y", "xxyx", "yxx", "xy"],
+ ),
+ PpTestSpec(
+ desc="Using delimited_list, with ':' delimiter",
+ expr=pp.delimited_list(
+ pp.Word(pp.hexnums, exact=2), delim=":", combine=True
+ ),
+ text="0A:4B:73:21:FE:76",
+ expected_list=["0A:4B:73:21:FE:76"],
+ ),
+ PpTestSpec(
+ desc="Using delimited_list, with ':' delimiter",
+ expr=pp.delimited_list(
+ pp.Word(pp.hexnums, exact=2),
+ delim=":",
+ combine=True,
+ allow_trailing_delim=True,
+ ),
+ text="0A:4B:73:21:FE:76:",
+ expected_list=["0A:4B:73:21:FE:76:"],
+ ),
+ ]
+
+
+class TestResultsName(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Match with results name",
+ expr=pp.Literal("xyz").set_results_name("value"),
+ text="xyz",
+ expected_dict={"value": "xyz"},
+ expected_list=["xyz"],
+ ),
+ PpTestSpec(
+ desc="Match with results name - using naming short-cut",
+ expr=pp.Literal("xyz")("value"),
+ text="xyz",
+ expected_dict={"value": "xyz"},
+ expected_list=["xyz"],
+ ),
+ PpTestSpec(
+ desc="Define multiple results names",
+ expr=pp.Word(pp.alphas, pp.alphanums)("key")
+ + "="
+ + pp.pyparsing_common.integer("value"),
+ text="range=5280",
+ expected_dict={"key": "range", "value": 5280},
+ expected_list=["range", "=", 5280],
+ ),
+ ]
+
+
+class TestGroups(PyparsingExpressionTestCase):
+ EQ = pp.Suppress("=")
+ tests = [
+ PpTestSpec(
+ desc="Define multiple results names in groups",
+ expr=pp.Group(
+ pp.Word(pp.alphas)("key") + EQ + pp.pyparsing_common.number("value")
+ )[...],
+ text="range=5280 long=-138.52 lat=46.91",
+ expected_list=[["range", 5280], ["long", -138.52], ["lat", 46.91]],
+ ),
+ PpTestSpec(
+ desc="Define multiple results names in groups - use Dict to define results names using parsed keys",
+ expr=pp.Dict(
+ pp.Group(pp.Word(pp.alphas) + EQ + pp.pyparsing_common.number)[...]
+ ),
+ text="range=5280 long=-138.52 lat=46.91",
+ expected_list=[["range", 5280], ["long", -138.52], ["lat", 46.91]],
+ expected_dict={"lat": 46.91, "long": -138.52, "range": 5280},
+ ),
+ PpTestSpec(
+ desc="Define multiple value types",
+ expr=pp.Dict(
+ pp.Group(
+ pp.Word(pp.alphas)
+ + EQ
+ + (
+ pp.pyparsing_common.number
+ | pp.oneOf("True False")
+ | pp.QuotedString("'")
+ )
+ )[...]
+ ),
+ text="long=-122.47 lat=37.82 public=True name='Golden Gate Bridge'",
+ expected_list=[
+ ["long", -122.47],
+ ["lat", 37.82],
+ ["public", "True"],
+ ["name", "Golden Gate Bridge"],
+ ],
+ expected_dict={
+ "long": -122.47,
+ "lat": 37.82,
+ "public": "True",
+ "name": "Golden Gate Bridge",
+ },
+ ),
+ ]
+
+
+class TestParseAction(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Parsing real numbers - use parse action to convert to float at parse time",
+ expr=pp.Combine(pp.Word(pp.nums) + "." + pp.Word(pp.nums)).add_parse_action(
+ lambda t: float(t[0])
+ )[...],
+ text="1.2 2.3 3.1416 98.6",
+ expected_list=[
+ 1.2,
+ 2.3,
+ 3.1416,
+ 98.6,
+ ], # note, these are now floats, not strs
+ ),
+ PpTestSpec(
+ desc="Match with numeric string converted to int",
+ expr=pp.Word("0123456789").addParseAction(lambda t: int(t[0])),
+ text="12345",
+ expected_list=[12345], # note - result is type int, not str
+ ),
+ PpTestSpec(
+ desc="Use two parse actions to convert numeric string, then convert to datetime",
+ expr=pp.Word(pp.nums).add_parse_action(
+ lambda t: int(t[0]), lambda t: datetime.utcfromtimestamp(t[0])
+ ),
+ text="1537415628",
+ expected_list=[datetime(2018, 9, 20, 3, 53, 48)],
+ ),
+ PpTestSpec(
+ desc="Use tokenMap for parse actions that operate on a single-length token",
+ expr=pp.Word(pp.nums).add_parse_action(
+ pp.token_map(int), pp.token_map(datetime.utcfromtimestamp)
+ ),
+ text="1537415628",
+ expected_list=[datetime(2018, 9, 20, 3, 53, 48)],
+ ),
+ PpTestSpec(
+ desc="Using a built-in function that takes a sequence of strs as a parse action",
+ expr=pp.Word(pp.hexnums, exact=2)[...].add_parse_action(":".join),
+ text="0A4B7321FE76",
+ expected_list=["0A:4B:73:21:FE:76"],
+ ),
+ PpTestSpec(
+ desc="Using a built-in function that takes a sequence of strs as a parse action",
+ expr=pp.Word(pp.hexnums, exact=2)[...].add_parse_action(sorted),
+ text="0A4B7321FE76",
+ expected_list=["0A", "21", "4B", "73", "76", "FE"],
+ ),
+ ]
+
+
+class TestResultsModifyingParseAction(PyparsingExpressionTestCase):
+ # do not make staticmethod
+ # @staticmethod
+ def compute_stats_parse_action(t):
+ # by the time this parse action is called, parsed numeric words
+ # have been converted to ints by a previous parse action, so
+ # they can be treated as ints
+ t["sum"] = sum(t)
+ t["ave"] = sum(t) / len(t)
+ t["min"] = min(t)
+ t["max"] = max(t)
+
+ tests = [
+ PpTestSpec(
+ desc="A parse action that adds new key-values",
+ expr=pp.pyparsing_common.integer[...].addParseAction(
+ compute_stats_parse_action
+ ),
+ text="27 1 14 22 89",
+ expected_list=[27, 1, 14, 22, 89],
+ expected_dict={"ave": 30.6, "max": 89, "min": 1, "sum": 153},
+ ),
+ ]
+
+
+class TestRegex(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Parsing real numbers - using Regex instead of Combine",
+ expr=pp.Regex(r"\d+\.\d+").add_parse_action(lambda t: float(t[0]))[...],
+ text="1.2 2.3 3.1416 98.6",
+ expected_list=[
+ 1.2,
+ 2.3,
+ 3.1416,
+ 98.6,
+ ], # note, these are now floats, not strs
+ ),
+ ]
+
+
+class TestParseCondition(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="Define a condition to only match numeric values that are multiples of 7",
+ expr=pp.Word(pp.nums).addCondition(lambda t: int(t[0]) % 7 == 0)[...],
+ text="14 35 77 12 28",
+ expected_list=["14", "35", "77"],
+ ),
+ PpTestSpec(
+ desc="Separate conversion to int and condition into separate parse action/conditions",
+ expr=pp.Word(pp.nums)
+ .add_parse_action(lambda t: int(t[0]))
+ .add_condition(lambda t: t[0] % 7 == 0)[...],
+ text="14 35 77 12 28",
+ expected_list=[14, 35, 77],
+ ),
+ ]
+
+
+class TestTransformStringUsingParseActions(PyparsingExpressionTestCase):
+ markup_convert_map = {
+ "*": "B",
+ "_": "U",
+ "/": "I",
+ }
+
+ # do not make staticmethod
+ # @staticmethod
+ def markup_convert(t):
+ htmltag = TestTransformStringUsingParseActions.markup_convert_map[
+ t.markup_symbol
+ ]
+ return "<{}>{}</{}>".format(htmltag, t.body, htmltag)
+
+ tests = [
+ PpTestSpec(
+ desc="Use transformString to convert simple markup to HTML",
+ expr=(
+ pp.one_of(markup_convert_map)("markup_symbol")
+ + "("
+ + pp.CharsNotIn(")")("body")
+ + ")"
+ ).add_parse_action(markup_convert),
+ text="Show in *(bold), _(underscore), or /(italic) type",
+ expected_list=[
+ "Show in <B>bold</B>, <U>underscore</U>, or <I>italic</I> type"
+ ],
+ parse_fn="transformString",
+ ),
+ ]
+
+
+class TestCommonHelperExpressions(PyparsingExpressionTestCase):
+ tests = [
+ PpTestSpec(
+ desc="A comma-delimited list of words",
+ expr=pp.delimited_list(pp.Word(pp.alphas)),
+ text="this, that, blah,foo, bar",
+ expected_list=["this", "that", "blah", "foo", "bar"],
+ ),
+ PpTestSpec(
+ desc="A counted array of words",
+ expr=pp.Group(pp.counted_array(pp.Word("ab")))[...],
+ text="2 aaa bbb 0 3 abab bbaa abbab",
+ expected_list=[["aaa", "bbb"], [], ["abab", "bbaa", "abbab"]],
+ ),
+ PpTestSpec(
+ desc="skipping comments with ignore",
+ expr=(
+ pp.pyparsing_common.identifier("lhs")
+ + "="
+ + pp.pyparsing_common.fnumber("rhs")
+ ).ignore(pp.cpp_style_comment),
+ text="abc_100 = /* value to be tested */ 3.1416",
+ expected_list=["abc_100", "=", 3.1416],
+ expected_dict={"lhs": "abc_100", "rhs": 3.1416},
+ ),
+ PpTestSpec(
+ desc="some pre-defined expressions in pyparsing_common, and building a dotted identifier with delimted_list",
+ expr=(
+ pp.pyparsing_common.number("id_num")
+ + pp.delimitedList(pp.pyparsing_common.identifier, ".", combine=True)(
+ "name"
+ )
+ + pp.pyparsing_common.ipv4_address("ip_address")
+ ),
+ text="1001 www.google.com 192.168.10.199",
+ expected_list=[1001, "www.google.com", "192.168.10.199"],
+ expected_dict={
+ "id_num": 1001,
+ "name": "www.google.com",
+ "ip_address": "192.168.10.199",
+ },
+ ),
+ PpTestSpec(
+ desc="using one_of (shortcut for Literal('a') | Literal('b') | Literal('c'))",
+ expr=pp.one_of("a b c")[...],
+ text="a b a b b a c c a b b",
+ expected_list=["a", "b", "a", "b", "b", "a", "c", "c", "a", "b", "b"],
+ ),
+ PpTestSpec(
+ desc="parsing nested parentheses",
+ expr=pp.nested_expr(),
+ text="(a b (c) d (e f g ()))",
+ expected_list=[["a", "b", ["c"], "d", ["e", "f", "g", []]]],
+ ),
+ PpTestSpec(
+ desc="parsing nested braces",
+ expr=(
+ pp.Keyword("if")
+ + pp.nested_expr()("condition")
+ + pp.nested_expr("{", "}")("body")
+ ),
+ text='if ((x == y) || !z) {printf("{}");}',
+ expected_list=[
+ "if",
+ [["x", "==", "y"], "||", "!z"],
+ ["printf(", '"{}"', ");"],
+ ],
+ expected_dict={
+ "condition": [[["x", "==", "y"], "||", "!z"]],
+ "body": [["printf(", '"{}"', ");"]],
+ },
+ ),
+ ]
+
+
+class TestWhitespaceMethods(PyparsingExpressionTestCase):
+ tests = [
+ # These test the single-element versions
+ PpTestSpec(
+ desc="The word foo",
+ expr=pp.Literal("foo").ignore_whitespace(),
+ text=" foo ",
+ expected_list=["foo"],
+ ),
+ PpTestSpec(
+ desc="The word foo",
+ expr=pp.Literal("foo").leave_whitespace(),
+ text=" foo ",
+ expected_fail_locn=0,
+ ),
+ PpTestSpec(
+ desc="The word foo",
+ expr=pp.Literal("foo").ignore_whitespace(),
+ text="foo",
+ expected_list=["foo"],
+ ),
+ PpTestSpec(
+ desc="The word foo",
+ expr=pp.Literal("foo").leave_whitespace(),
+ text="foo",
+ expected_list=["foo"],
+ ),
+ # These test the composite elements
+ PpTestSpec(
+ desc="If we recursively leave whitespace on the parent, this whitespace-dependent grammar will succeed, even if the children themselves skip whitespace",
+ expr=pp.And(
+ [
+ pp.Literal(" foo").ignore_whitespace(),
+ pp.Literal(" bar").ignore_whitespace(),
+ ]
+ ).leave_whitespace(recursive=True),
+ text=" foo bar",
+ expected_list=[" foo", " bar"],
+ ),
+ #
+ PpTestSpec(
+ desc="If we recursively ignore whitespace in our parsing, this whitespace-dependent grammar will fail, even if the children themselves keep whitespace",
+ expr=pp.And(
+ [
+ pp.Literal(" foo").leave_whitespace(),
+ pp.Literal(" bar").leave_whitespace(),
+ ]
+ ).ignore_whitespace(recursive=True),
+ text=" foo bar",
+ expected_fail_locn=1,
+ ),
+ PpTestSpec(
+ desc="If we leave whitespace on the parent, but it isn't recursive, this whitespace-dependent grammar will fail",
+ expr=pp.And(
+ [
+ pp.Literal(" foo").ignore_whitespace(),
+ pp.Literal(" bar").ignore_whitespace(),
+ ]
+ ).leave_whitespace(recursive=False),
+ text=" foo bar",
+ expected_fail_locn=5,
+ ),
+ # These test the Enhance classes
+ PpTestSpec(
+ desc="If we recursively leave whitespace on the parent, this whitespace-dependent grammar will succeed, even if the children themselves skip whitespace",
+ expr=pp.Optional(pp.Literal(" foo").ignore_whitespace()).leave_whitespace(
+ recursive=True
+ ),
+ text=" foo",
+ expected_list=[" foo"],
+ ),
+ #
+ PpTestSpec(
+ desc="If we ignore whitespace on the parent, but it isn't recursive, parsing will fail because we skip to the first character 'f' before the internal expr can see it",
+ expr=pp.Optional(pp.Literal(" foo").leave_whitespace()).ignore_whitespace(
+ recursive=True
+ ),
+ text=" foo",
+ expected_list=[],
+ ),
+ # PpTestSpec(
+ # desc="If we leave whitespace on the parent, this whitespace-dependent grammar will succeed, even if the children themselves skip whitespace",
+ # expr=pp.Optional(pp.Literal(" foo").ignoreWhitespace()).leaveWhitespace(
+ # recursive=False
+ # ),
+ # text=" foo",
+ # expected_list=[]
+ # ),
+ ]
+
+
+def _get_decl_line_no(cls):
+ import inspect
+
+ return inspect.getsourcelines(cls)[1]
+
+
+# get all test case classes defined in this module and sort them by decl line no
+test_case_classes = list(PyparsingExpressionTestCase.__subclasses__())
+test_case_classes.sort(key=_get_decl_line_no)
+
+# make into a suite and run it - this will run the tests in the same order
+# they are declared in this module
+#
+# runnable from setup.py using "python setup.py test -s simple_unit_tests.suite"
+#
+suite = unittest.TestSuite(cls() for cls in test_case_classes)
+
+
+# ============ MAIN ================
+
+if __name__ == "__main__":
+
+ result = unittest.TextTestRunner().run(suite)
+
+ exit(0 if result.wasSuccessful() else 1)
--- /dev/null
+#
+# test_unit.py
+#
+# Unit tests for pyparsing module
+#
+# Copyright 2002-2021, Paul McGuire
+#
+#
+
+import contextlib
+import datetime
+import re
+import sys
+from types import SimpleNamespace
+from io import StringIO
+from textwrap import dedent
+from unittest import TestCase
+
+import pyparsing as pp
+from examples.jsonParser import jsonObject
+from pyparsing import ParserElement, ParseException, ParseFatalException
+from tests.json_parser_tests import test1, test2, test3, test4, test5
+import platform
+
+ppc = pp.pyparsing_common
+ppt = pp.pyparsing_test
+
+# see which Python implementation we are running
+python_impl = platform.python_implementation()
+CPYTHON_ENV = python_impl == "CPython"
+IRON_PYTHON_ENV = python_impl == "IronPython"
+JYTHON_ENV = python_impl == "Jython"
+PYPY_ENV = python_impl == "PyPy"
+
+# get full stack traces during testing
+pp.ParserElement.verbose_stacktrace = True
+
+
+# simple utility for flattening nested lists
+def flatten(nested_list):
+ if not isinstance(nested_list, list):
+ return [nested_list]
+ if not nested_list:
+ return nested_list
+ return flatten(nested_list[0]) + flatten(nested_list[1:])
+
+
+class resetting:
+ def __init__(self, *args):
+ ob = args[0]
+ attrnames = args[1:]
+ self.ob = ob
+ self.save_attrs = attrnames
+ self.save_values = [getattr(ob, attrname) for attrname in attrnames]
+
+ def __enter__(self):
+ pass
+
+ def __exit__(self, *args):
+ for attr, value in zip(self.save_attrs, self.save_values):
+ setattr(self.ob, attr, value)
+
+
+def find_all_re_matches(patt, s):
+ ret = []
+ start = 0
+ if isinstance(patt, str):
+ patt = re.compile(patt)
+ while True:
+ found = patt.search(s, pos=start)
+ if found:
+ ret.append(found)
+ start = found.end()
+ else:
+ break
+ return ret
+
+
+def current_method_name(level=2):
+ import traceback
+
+ stack = traceback.extract_stack(limit=level)
+ return stack[0].name
+
+
+def __():
+ return current_method_name(3) + ": "
+
+
+class Test01_PyparsingTestInit(TestCase):
+ def runTest(self):
+ from pyparsing import (
+ __version__ as pyparsing_version,
+ __version_time__ as pyparsing_version_time,
+ )
+
+ print(
+ "Beginning test of pyparsing, version",
+ pyparsing_version,
+ pyparsing_version_time,
+ )
+ print("Python version", sys.version)
+
+
+class Test01a_PyparsingEnvironmentTests(TestCase):
+ def runTest(self):
+ # test warnings enable detection
+ tests = [
+ (([], "",), False),
+ ((["d", ], "",), True),
+ ((["d", "i:::pyparsing", ], "",), False),
+ ((["d:::pyparsing", ], "",), True),
+ ((["d:::pyparsing", "i", ], "",), False),
+ ((["d:::blah", ], "",), False),
+ ((["i", ], "",), False),
+ (([], "1",), True),
+ ((["d", ], "1",), True),
+ ((["d", "i:::pyparsing", ], "1",), False),
+ ((["d:::pyparsing", ], "1",), True),
+ ((["d:::pyparsing", "i", ], "1",), False),
+ ((["d:::blah", ], "1",), True),
+ ((["i", ], "1",), False),
+ ]
+
+ all_success = True
+ for args, expected in tests:
+ message = "{} should be {}".format(args, expected)
+ print(message, end=" -> ")
+ actual = pp.core._should_enable_warnings(*args)
+ print("PASS" if actual == expected else "FAIL")
+ if actual != expected:
+ all_success = False
+ self.assertTrue(all_success, "failed warnings enable test")
+
+
+class Test02_WithoutPackrat(ppt.TestParseResultsAsserts, TestCase):
+ suite_context = None
+ save_suite_context = None
+
+ def setUp(self):
+ self.suite_context.restore()
+
+ @contextlib.contextmanager
+ def assertRaises(self, expected_exception_type, msg=None):
+ """
+ Simple wrapper to print out the exceptions raised after assertRaises
+ """
+ try:
+ with super().assertRaises(expected_exception_type, msg=msg) as ar:
+ yield
+ finally:
+ if getattr(ar, "exception", None) is not None:
+ print(
+ "Raised expected exception: {}: {}".format(
+ type(ar.exception).__name__, str(ar.exception)
+ )
+ )
+ else:
+ print(
+ "Expected {} exception not raised".format(
+ expected_exception_type.__name__
+ )
+ )
+ return ar
+
+ def test000_assert_packrat_status(self):
+ print("Packrat enabled:", ParserElement._packratEnabled)
+ self.assertFalse(ParserElement._packratEnabled, "packrat enabled")
+
+ def testScanStringWithOverlap(self):
+ parser = pp.Word(pp.alphas, exact=3)
+ without_overlaps = sum(t for t, s, e in parser.scanString("ABCDEFGHI")).asList()
+ self.assertEqual(
+ ["ABC", "DEF", "GHI"],
+ without_overlaps,
+ msg="scanString without overlaps failed",
+ )
+ with_overlaps = sum(
+ t for t, s, e in parser.scanString("ABCDEFGHI", overlap=True)
+ ).asList()
+ self.assertEqual(
+ ["ABC", "BCD", "CDE", "DEF", "EFG", "FGH", "GHI"],
+ with_overlaps,
+ msg="scanString with overlaps failed",
+ )
+
+ def testTransformString(self):
+ make_int_with_commas = ppc.integer().addParseAction(
+ lambda t: "{:,}".format(t[0])
+ )
+ lower_case_words = pp.Word(pp.alphas.lower(), asKeyword=True) + pp.Optional(
+ pp.White()
+ )
+ nested_list = pp.nestedExpr().addParseAction(pp.ParseResults.asList)
+ transformer = make_int_with_commas | nested_list | lower_case_words.suppress()
+
+ in_string = (
+ "I wish to buy 12345 shares of Acme Industries (as a gift to my (ex)wife)"
+ )
+ print(in_string)
+ out_string = transformer.transformString(in_string)
+ print(out_string)
+ self.assertEqual(
+ "I 12,345 Acme Industries asagifttomyexwife",
+ out_string,
+ msg="failure in transformString",
+ )
+
+ def testUpdateDefaultWhitespace(self):
+ prev_default_whitespace_chars = pp.ParserElement.DEFAULT_WHITE_CHARS
+ try:
+ pp.dblQuotedString.copyDefaultWhiteChars = False
+ pp.ParserElement.setDefaultWhitespaceChars(" \t")
+ self.assertEqual(
+ set(" \t"),
+ set(pp.sglQuotedString.whiteChars),
+ "setDefaultWhitespaceChars did not update sglQuotedString",
+ )
+ self.assertEqual(
+ set(prev_default_whitespace_chars),
+ set(pp.dblQuotedString.whiteChars),
+ "setDefaultWhitespaceChars updated dblQuotedString but should not",
+ )
+ finally:
+ pp.dblQuotedString.copyDefaultWhiteChars = True
+ pp.ParserElement.setDefaultWhitespaceChars(prev_default_whitespace_chars)
+
+ self.assertEqual(
+ set(prev_default_whitespace_chars),
+ set(pp.dblQuotedString.whiteChars),
+ "setDefaultWhitespaceChars updated dblQuotedString",
+ )
+
+ with ppt.reset_pyparsing_context():
+ pp.ParserElement.setDefaultWhitespaceChars(" \t")
+ self.assertNotEqual(
+ set(prev_default_whitespace_chars),
+ set(pp.dblQuotedString.whiteChars),
+ "setDefaultWhitespaceChars updated dblQuotedString but should not",
+ )
+
+ EOL = pp.LineEnd().suppress().setName("EOL")
+
+ # Identifiers is a string + optional $
+ identifier = pp.Combine(pp.Word(pp.alphas) + pp.Optional("$"))
+
+ # Literals (number or double quoted string)
+ literal = ppc.number | pp.dblQuotedString
+ expression = literal | identifier
+ # expression.setName("expression").setDebug()
+ # ppc.number.setDebug()
+ # ppc.integer.setDebug()
+
+ line_number = ppc.integer
+
+ # Keywords
+ PRINT = pp.CaselessKeyword("print")
+ print_stmt = PRINT - pp.ZeroOrMore(expression | ";")
+ statement = print_stmt
+ code_line = pp.Group(line_number + statement + EOL)
+ program = pp.ZeroOrMore(code_line)
+
+ test = """\
+ 10 print 123;
+ 20 print 234; 567;
+ 30 print 890
+ """
+
+ parsed_program = program.parseString(test)
+ print(parsed_program.dump())
+ self.assertEqual(
+ 3,
+ len(parsed_program),
+ "failed to apply new whitespace chars to existing builtins",
+ )
+
+ def testUpdateDefaultWhitespace2(self):
+ with ppt.reset_pyparsing_context():
+ expr_tests = [
+ (pp.dblQuotedString, '"abc"'),
+ (pp.sglQuotedString, "'def'"),
+ (ppc.integer, "123"),
+ (ppc.number, "4.56"),
+ (ppc.identifier, "a_bc"),
+ ]
+ NL = pp.LineEnd()
+
+ for expr, test_str in expr_tests:
+ parser = pp.Group(expr[1, ...] + pp.Optional(NL))[1, ...]
+ test_string = "\n".join([test_str] * 3)
+ result = parser.parseString(test_string, parseAll=True)
+ print(result.dump())
+ self.assertEqual(1, len(result), "failed {!r}".format(test_string))
+
+ pp.ParserElement.setDefaultWhitespaceChars(" \t")
+
+ for expr, test_str in expr_tests:
+ parser = pp.Group(expr[1, ...] + pp.Optional(NL))[1, ...]
+ test_string = "\n".join([test_str] * 3)
+ result = parser.parseString(test_string, parseAll=True)
+ print(result.dump())
+ self.assertEqual(3, len(result), "failed {!r}".format(test_string))
+
+ pp.ParserElement.setDefaultWhitespaceChars(" \n\t")
+
+ for expr, test_str in expr_tests:
+ parser = pp.Group(expr[1, ...] + pp.Optional(NL))[1, ...]
+ test_string = "\n".join([test_str] * 3)
+ result = parser.parseString(test_string, parseAll=True)
+ print(result.dump())
+ self.assertEqual(1, len(result), "failed {!r}".format(test_string))
+
+ def testParseFourFn(self):
+ import examples.fourFn as fourFn
+ import math
+
+ def test(s, ans):
+ fourFn.exprStack[:] = []
+ results = fourFn.BNF().parseString(s)
+ try:
+ resultValue = fourFn.evaluate_stack(fourFn.exprStack)
+ except Exception:
+ self.assertIsNone(ans, "exception raised for expression {!r}".format(s))
+ else:
+ self.assertEqual(
+ ans,
+ resultValue,
+ "failed to evaluate {}, got {:f}".format(s, resultValue),
+ )
+ print(s, "->", resultValue)
+
+ test("9", 9)
+ test("-9", -9)
+ test("--9", 9)
+ test("-E", -math.e)
+ test("9 + 3 + 5", 9 + 3 + 5)
+ test("9 + 3 / 11", 9 + 3.0 / 11)
+ test("(9 + 3)", (9 + 3))
+ test("(9+3) / 11", (9 + 3.0) / 11)
+ test("9 - 12 - 6", 9 - 12 - 6)
+ test("9 - (12 - 6)", 9 - (12 - 6))
+ test("2*3.14159", 2 * 3.14159)
+ test("3.1415926535*3.1415926535 / 10", 3.1415926535 * 3.1415926535 / 10)
+ test("PI * PI / 10", math.pi * math.pi / 10)
+ test("PI*PI/10", math.pi * math.pi / 10)
+ test("PI^2", math.pi ** 2)
+ test("round(PI^2)", round(math.pi ** 2))
+ test("6.02E23 * 8.048", 6.02e23 * 8.048)
+ test("e / 3", math.e / 3)
+ test("sin(PI/2)", math.sin(math.pi / 2))
+ test("10+sin(PI/4)^2", 10 + math.sin(math.pi / 4) ** 2)
+ test("trunc(E)", int(math.e))
+ test("trunc(-E)", int(-math.e))
+ test("round(E)", round(math.e))
+ test("round(-E)", round(-math.e))
+ test("E^PI", math.e ** math.pi)
+ test("exp(0)", 1)
+ test("exp(1)", math.e)
+ test("2^3^2", 2 ** 3 ** 2)
+ test("(2^3)^2", (2 ** 3) ** 2)
+ test("2^3+2", 2 ** 3 + 2)
+ test("2^3+5", 2 ** 3 + 5)
+ test("2^9", 2 ** 9)
+ test("sgn(-2)", -1)
+ test("sgn(0)", 0)
+ test("sgn(0.1)", 1)
+ test("foo(0.1)", None)
+ test("round(E, 3)", round(math.e, 3))
+ test("round(PI^2, 3)", round(math.pi ** 2, 3))
+ test("sgn(cos(PI/4))", 1)
+ test("sgn(cos(PI/2))", 0)
+ test("sgn(cos(PI*3/4))", -1)
+ test("+(sgn(cos(PI/4)))", 1)
+ test("-(sgn(cos(PI/4)))", -1)
+
+ def testParseSQL(self):
+ # SQL parser uses packrat parsing, not compatible with LR
+ if ParserElement._left_recursion_enabled:
+ return
+
+ import examples.simpleSQL as simpleSQL
+
+ def test(s, num_expected_toks, expected_errloc=-1):
+ try:
+ sqlToks = flatten(simpleSQL.simpleSQL.parseString(s).asList())
+ print(s, sqlToks, len(sqlToks))
+ self.assertEqual(
+ num_expected_toks,
+ len(sqlToks),
+ "invalid parsed tokens, expected {}, found {} ({})".format(
+ num_expected_toks, len(sqlToks), sqlToks
+ ),
+ )
+ except ParseException as e:
+ if expected_errloc >= 0:
+ self.assertEqual(
+ expected_errloc,
+ e.loc,
+ "expected error at {}, found at {}".format(
+ expected_errloc, e.loc
+ ),
+ )
+
+ test("SELECT * from XYZZY, ABC", 6)
+ test("select * from SYS.XYZZY", 5)
+ test("Select A from Sys.dual", 5)
+ test("Select A,B,C from Sys.dual", 7)
+ test("Select A, B, C from Sys.dual", 7)
+ test("Select A, B, C from Sys.dual, Table2 ", 8)
+ test("Xelect A, B, C from Sys.dual", 0, 0)
+ test("Select A, B, C frox Sys.dual", 0, 15)
+ test("Select", 0, 6)
+ test("Select &&& frox Sys.dual", 0, 7)
+ test("Select A from Sys.dual where a in ('RED','GREEN','BLUE')", 12)
+ test(
+ "Select A from Sys.dual where a in ('RED','GREEN','BLUE') and b in (10,20,30)",
+ 20,
+ )
+ test(
+ "Select A,b from table1,table2 where table1.id eq table2.id -- test out comparison operators",
+ 10,
+ )
+
+ def testParseConfigFile(self):
+ from examples import configParse
+
+ def test(fnam, num_expected_toks, resCheckList):
+ print("Parsing", fnam, "...", end=" ")
+ with open(fnam) as infile:
+ iniFileLines = "\n".join(infile.read().splitlines())
+ iniData = configParse.inifile_BNF().parseString(iniFileLines)
+ print(len(flatten(iniData.asList())))
+ print(list(iniData.keys()))
+ self.assertEqual(
+ num_expected_toks,
+ len(flatten(iniData.asList())),
+ "file %s not parsed correctly" % fnam,
+ )
+ for chkkey, chkexpect in resCheckList:
+ var = iniData
+ for attr in chkkey.split("."):
+ var = getattr(var, attr)
+ print(chkkey, var, chkexpect)
+ self.assertEqual(
+ chkexpect,
+ var,
+ "ParseConfigFileTest: failed to parse ini {!r} as expected {!r}, found {}".format(
+ chkkey, chkexpect, var
+ ),
+ )
+ print("OK")
+
+ test(
+ "tests/karthik.ini",
+ 23,
+ [("users.K", "8"), ("users.mod_scheme", "'QPSK'"), ("users.Na", "K+2")],
+ )
+ test(
+ "examples/Setup.ini",
+ 125,
+ [
+ ("Startup.audioinf", "M3i"),
+ ("Languages.key1", "0x0003"),
+ ("test.foo", "bar"),
+ ],
+ )
+
+ def testParseJSONData(self):
+ expected = [
+ {
+ "glossary": {
+ "GlossDiv": {
+ "GlossList": [
+ {
+ "Abbrev": "ISO 8879:1986",
+ "Acronym": "SGML",
+ "AvogadroNumber": 6.02e23,
+ "EmptyDict": {},
+ "EmptyList": [],
+ "EvenPrimesGreaterThan2": [],
+ "FermatTheoremInMargin": False,
+ "GlossDef": "A meta-markup language, "
+ "used to create markup "
+ "languages such as "
+ "DocBook.",
+ "GlossSeeAlso": ["GML", "XML", "markup"],
+ "GlossTerm": "Standard Generalized " "Markup Language",
+ "ID": "SGML",
+ "LargestPrimeLessThan100": 97,
+ "MapRequiringFiveColors": None,
+ "PrimesLessThan10": [2, 3, 5, 7],
+ "SortAs": "SGML",
+ }
+ ],
+ "title": "S",
+ },
+ "title": "example glossary",
+ }
+ },
+ {
+ "menu": {
+ "id": "file",
+ "popup": {
+ "menuitem": [
+ {"onclick": "CreateNewDoc()", "value": "New"},
+ {"onclick": "OpenDoc()", "value": "Open"},
+ {"onclick": "CloseDoc()", "value": "Close"},
+ ]
+ },
+ "value": "File:",
+ }
+ },
+ {
+ "widget": {
+ "debug": "on",
+ "image": {
+ "alignment": "center",
+ "hOffset": 250,
+ "name": "sun1",
+ "src": "Images/Sun.png",
+ "vOffset": 250,
+ },
+ "text": {
+ "alignment": "center",
+ "data": "Click Here",
+ "hOffset": 250,
+ "name": "text1",
+ "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;",
+ "size": 36,
+ "style": "bold",
+ "vOffset": 100,
+ },
+ "window": {
+ "height": 500,
+ "name": "main_window",
+ "title": "Sample Konfabulator Widget",
+ "width": 500,
+ },
+ }
+ },
+ {
+ "web-app": {
+ "servlet": [
+ {
+ "init-param": {
+ "cachePackageTagsRefresh": 60,
+ "cachePackageTagsStore": 200,
+ "cachePackageTagsTrack": 200,
+ "cachePagesDirtyRead": 10,
+ "cachePagesRefresh": 10,
+ "cachePagesStore": 100,
+ "cachePagesTrack": 200,
+ "cacheTemplatesRefresh": 15,
+ "cacheTemplatesStore": 50,
+ "cacheTemplatesTrack": 100,
+ "configGlossary:adminEmail": "ksm@pobox.com",
+ "configGlossary:installationAt": "Philadelphia, " "PA",
+ "configGlossary:poweredBy": "Cofax",
+ "configGlossary:poweredByIcon": "/images/cofax.gif",
+ "configGlossary:staticPath": "/content/static",
+ "dataStoreClass": "org.cofax.SqlDataStore",
+ "dataStoreConnUsageLimit": 100,
+ "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver",
+ "dataStoreInitConns": 10,
+ "dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log",
+ "dataStoreLogLevel": "debug",
+ "dataStoreMaxConns": 100,
+ "dataStoreName": "cofax",
+ "dataStorePassword": "dataStoreTestQuery",
+ "dataStoreTestQuery": "SET NOCOUNT "
+ "ON;select "
+ "test='test';",
+ "dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon",
+ "dataStoreUser": "sa",
+ "defaultFileTemplate": "articleTemplate.htm",
+ "defaultListTemplate": "listTemplate.htm",
+ "jspFileTemplate": "articleTemplate.jsp",
+ "jspListTemplate": "listTemplate.jsp",
+ "maxUrlLength": 500,
+ "redirectionClass": "org.cofax.SqlRedirection",
+ "searchEngineFileTemplate": "forSearchEngines.htm",
+ "searchEngineListTemplate": "forSearchEnginesList.htm",
+ "searchEngineRobotsDb": "WEB-INF/robots.db",
+ "templateLoaderClass": "org.cofax.FilesTemplateLoader",
+ "templateOverridePath": "",
+ "templatePath": "templates",
+ "templateProcessorClass": "org.cofax.WysiwygTemplate",
+ "useDataStore": True,
+ "useJSP": False,
+ },
+ "servlet-class": "org.cofax.cds.CDSServlet",
+ "servlet-name": "cofaxCDS",
+ },
+ {
+ "init-param": {
+ "mailHost": "mail1",
+ "mailHostOverride": "mail2",
+ },
+ "servlet-class": "org.cofax.cds.EmailServlet",
+ "servlet-name": "cofaxEmail",
+ },
+ {
+ "servlet-class": "org.cofax.cds.AdminServlet",
+ "servlet-name": "cofaxAdmin",
+ },
+ {
+ "servlet-class": "org.cofax.cds.FileServlet",
+ "servlet-name": "fileServlet",
+ },
+ {
+ "init-param": {
+ "adminGroupID": 4,
+ "betaServer": True,
+ "dataLog": 1,
+ "dataLogLocation": "/usr/local/tomcat/logs/dataLog.log",
+ "dataLogMaxSize": "",
+ "fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder",
+ "log": 1,
+ "logLocation": "/usr/local/tomcat/logs/CofaxTools.log",
+ "logMaxSize": "",
+ "lookInContext": 1,
+ "removePageCache": "/content/admin/remove?cache=pages&id=",
+ "removeTemplateCache": "/content/admin/remove?cache=templates&id=",
+ "templatePath": "toolstemplates/",
+ },
+ "servlet-class": "org.cofax.cms.CofaxToolsServlet",
+ "servlet-name": "cofaxTools",
+ },
+ ],
+ "servlet-mapping": {
+ "cofaxAdmin": "/admin/*",
+ "cofaxCDS": "/",
+ "cofaxEmail": "/cofaxutil/aemail/*",
+ "cofaxTools": "/tools/*",
+ "fileServlet": "/static/*",
+ },
+ "taglib": {
+ "taglib-location": "/WEB-INF/tlds/cofax.tld",
+ "taglib-uri": "cofax.tld",
+ },
+ }
+ },
+ {
+ "menu": {
+ "header": "SVG Viewer",
+ "items": [
+ {"id": "Open"},
+ {"id": "OpenNew", "label": "Open New"},
+ None,
+ {"id": "ZoomIn", "label": "Zoom In"},
+ {"id": "ZoomOut", "label": "Zoom Out"},
+ {"id": "OriginalView", "label": "Original View"},
+ None,
+ {"id": "Quality"},
+ {"id": "Pause"},
+ {"id": "Mute"},
+ None,
+ {"id": "Find", "label": "Find..."},
+ {"id": "FindAgain", "label": "Find Again"},
+ {"id": "Copy"},
+ {"id": "CopyAgain", "label": "Copy Again"},
+ {"id": "CopySVG", "label": "Copy SVG"},
+ {"id": "ViewSVG", "label": "View SVG"},
+ {"id": "ViewSource", "label": "View Source"},
+ {"id": "SaveAs", "label": "Save As"},
+ None,
+ {"id": "Help"},
+ {"id": "About", "label": "About Adobe CVG Viewer..."},
+ ],
+ }
+ },
+ ]
+
+ for t, exp_result in zip((test1, test2, test3, test4, test5), expected):
+ result = jsonObject.parseString(t)
+ self.assertEqual(exp_result, result[0])
+
+ def testParseCommaSeparatedValues(self):
+ testData = [
+ "a,b,c,100.2,,3",
+ "d, e, j k , m ",
+ "'Hello, World', f, g , , 5.1,x",
+ "John Doe, 123 Main St., Cleveland, Ohio",
+ "Jane Doe, 456 St. James St., Los Angeles , California ",
+ "",
+ ]
+ testVals = [
+ [(3, "100.2"), (4, ""), (5, "3")],
+ [(2, "j k"), (3, "m")],
+ [(0, "'Hello, World'"), (2, "g"), (3, "")],
+ [(0, "John Doe"), (1, "123 Main St."), (2, "Cleveland"), (3, "Ohio")],
+ [
+ (0, "Jane Doe"),
+ (1, "456 St. James St."),
+ (2, "Los Angeles"),
+ (3, "California"),
+ ],
+ ]
+ for line, tests in zip(testData, testVals):
+ print("Parsing: %r ->" % line, end=" ")
+ results = ppc.comma_separated_list.parseString(line)
+ print(results)
+ for t in tests:
+ if not (len(results) > t[0] and results[t[0]] == t[1]):
+ print("$$$", results.dump())
+ print("$$$", results[0])
+ self.assertTrue(
+ len(results) > t[0] and results[t[0]] == t[1],
+ "failed on %s, item %d s/b '%s', got '%s'"
+ % (line, t[0], t[1], str(results.asList())),
+ )
+
+ def testParseEBNF(self):
+ from examples import ebnf
+
+ print("Constructing EBNF parser with pyparsing...")
+
+ grammar = """
+ syntax = (syntax_rule), {(syntax_rule)};
+ syntax_rule = meta_identifier, '=', definitions_list, ';';
+ definitions_list = single_definition, {'|', single_definition};
+ single_definition = syntactic_term, {',', syntactic_term};
+ syntactic_term = syntactic_factor,['-', syntactic_factor];
+ syntactic_factor = [integer, '*'], syntactic_primary;
+ syntactic_primary = optional_sequence | repeated_sequence |
+ grouped_sequence | meta_identifier | terminal_string;
+ optional_sequence = '[', definitions_list, ']';
+ repeated_sequence = '{', definitions_list, '}';
+ grouped_sequence = '(', definitions_list, ')';
+ (*
+ terminal_string = "'", character - "'", {character - "'"}, "'" |
+ '"', character - '"', {character - '"'}, '"';
+ meta_identifier = letter, {letter | digit};
+ integer = digit, {digit};
+ *)
+ """
+
+ table = {}
+ table["terminal_string"] = pp.quotedString
+ table["meta_identifier"] = pp.Word(pp.alphas + "_", pp.alphas + "_" + pp.nums)
+ table["integer"] = pp.Word(pp.nums)
+
+ print("Parsing EBNF grammar with EBNF parser...")
+ parsers = ebnf.parse(grammar, table)
+ ebnf_parser = parsers["syntax"]
+ print("-", "\n- ".join(parsers.keys()))
+ self.assertEqual(
+ 13, len(list(parsers.keys())), "failed to construct syntax grammar"
+ )
+
+ print("Parsing EBNF grammar with generated EBNF parser...")
+ parsed_chars = ebnf_parser.parseString(grammar)
+ parsed_char_len = len(parsed_chars)
+
+ print("],\n".join(str(parsed_chars.asList()).split("],")))
+ self.assertEqual(
+ 98,
+ len(flatten(parsed_chars.asList())),
+ "failed to tokenize grammar correctly",
+ )
+
+ def testParseIDL(self):
+ from examples import idlParse
+
+ def test(strng, numToks, expectedErrloc=0):
+ print(strng)
+ try:
+ bnf = idlParse.CORBA_IDL_BNF()
+ tokens = bnf.parseString(strng)
+ print("tokens = ")
+ tokens.pprint()
+ tokens = flatten(tokens.asList())
+ print(len(tokens))
+ self.assertEqual(
+ numToks,
+ len(tokens),
+ "error matching IDL string, {} -> {}".format(strng, str(tokens)),
+ )
+ except ParseException as err:
+ print(err.line)
+ print(" " * (err.column - 1) + "^")
+ print(err)
+ self.assertEqual(
+ 0,
+ numToks,
+ "unexpected ParseException while parsing {}, {}".format(
+ strng, str(err)
+ ),
+ )
+ self.assertEqual(
+ expectedErrloc,
+ err.loc,
+ "expected ParseException at %d, found exception at %d"
+ % (expectedErrloc, err.loc),
+ )
+
+ test(
+ """
+ /*
+ * a block comment *
+ */
+ typedef string[10] tenStrings;
+ typedef sequence<string> stringSeq;
+ typedef sequence< sequence<string> > stringSeqSeq;
+
+ interface QoSAdmin {
+ stringSeq method1(in string arg1, inout long arg2);
+ stringSeqSeq method2(in string arg1, inout long arg2, inout long arg3);
+ string method3();
+ };
+ """,
+ 59,
+ )
+ test(
+ """
+ /*
+ * a block comment *
+ */
+ typedef string[10] tenStrings;
+ typedef
+ /** ** *** **** *
+ * a block comment *
+ */
+ sequence<string> /*comment inside an And */ stringSeq;
+ /* */ /**/ /***/ /****/
+ typedef sequence< sequence<string> > stringSeqSeq;
+
+ interface QoSAdmin {
+ stringSeq method1(in string arg1, inout long arg2);
+ stringSeqSeq method2(in string arg1, inout long arg2, inout long arg3);
+ string method3();
+ };
+ """,
+ 59,
+ )
+ test(
+ r"""
+ const string test="Test String\n";
+ const long a = 0;
+ const long b = -100;
+ const float c = 3.14159;
+ const long d = 0x007f7f7f;
+ exception TestException
+ {
+ string msg;
+ sequence<string> dataStrings;
+ };
+
+ interface TestInterface
+ {
+ void method1(in string arg1, inout long arg2);
+ };
+ """,
+ 60,
+ )
+ test(
+ """
+ module Test1
+ {
+ exception TestException
+ {
+ string msg;
+ ];
+
+ interface TestInterface
+ {
+ void method1(in string arg1, inout long arg2)
+ raises (TestException);
+ };
+ };
+ """,
+ 0,
+ 56,
+ )
+ test(
+ """
+ module Test1
+ {
+ exception TestException
+ {
+ string msg;
+ };
+
+ };
+ """,
+ 13,
+ )
+
+ def testParseVerilog(self):
+ pass
+
+ def testScanString(self):
+
+ testdata = """
+ <table border="0" cellpadding="3" cellspacing="3" frame="" width="90%">
+ <tr align="left" valign="top">
+ <td><b>Name</b></td>
+ <td><b>IP Address</b></td>
+ <td><b>Location</b></td>
+ </tr>
+ <tr align="left" valign="top" bgcolor="#c7efce">
+ <td>time-a.nist.gov</td>
+ <td>129.6.15.28</td>
+ <td>NIST, Gaithersburg, Maryland</td>
+ </tr>
+ <tr align="left" valign="top">
+ <td>time-b.nist.gov</td>
+ <td>129.6.15.29</td>
+ <td>NIST, Gaithersburg, Maryland</td>
+ </tr>
+ <tr align="left" valign="top" bgcolor="#c7efce">
+ <td>time-a.timefreq.bldrdoc.gov</td>
+ <td>132.163.4.101</td>
+ <td>NIST, Boulder, Colorado</td>
+ </tr>
+ <tr align="left" valign="top">
+ <td>time-b.timefreq.bldrdoc.gov</td>
+ <td>132.163.4.102</td>
+ <td>NIST, Boulder, Colorado</td>
+ </tr>
+ <tr align="left" valign="top" bgcolor="#c7efce">
+ <td>time-c.timefreq.bldrdoc.gov</td>
+ <td>132.163.4.103</td>
+ <td>NIST, Boulder, Colorado</td>
+ </tr>
+ </table>
+ """
+ integer = pp.Word(pp.nums)
+ ipAddress = pp.Combine(integer + "." + integer + "." + integer + "." + integer)
+ tdStart = pp.Suppress("<td>")
+ tdEnd = pp.Suppress("</td>")
+ timeServerPattern = (
+ tdStart
+ + ipAddress("ipAddr")
+ + tdEnd
+ + tdStart
+ + pp.CharsNotIn("<")("loc")
+ + tdEnd
+ )
+ servers = [
+ srvr.ipAddr
+ for srvr, startloc, endloc in timeServerPattern.scanString(testdata)
+ ]
+
+ print(servers)
+ self.assertEqual(
+ [
+ "129.6.15.28",
+ "129.6.15.29",
+ "132.163.4.101",
+ "132.163.4.102",
+ "132.163.4.103",
+ ],
+ servers,
+ "failed scanString()",
+ )
+
+ # test for stringEnd detection in scanString
+ foundStringEnds = [r for r in pp.StringEnd().scanString("xyzzy")]
+ print(foundStringEnds)
+ self.assertTrue(foundStringEnds, "Failed to find StringEnd in scanString")
+
+ def testQuotedStrings(self):
+ testData = """
+ 'a valid single quoted string'
+ 'an invalid single quoted string
+ because it spans lines'
+ "a valid double quoted string"
+ "an invalid double quoted string
+ because it spans lines"
+ """
+ print(testData)
+
+ sglStrings = [
+ (t[0], b, e) for (t, b, e) in pp.sglQuotedString.scanString(testData)
+ ]
+ print(sglStrings)
+ self.assertTrue(
+ len(sglStrings) == 1
+ and (sglStrings[0][1] == 17 and sglStrings[0][2] == 47),
+ "single quoted string failure",
+ )
+
+ dblStrings = [
+ (t[0], b, e) for (t, b, e) in pp.dblQuotedString.scanString(testData)
+ ]
+ print(dblStrings)
+ self.assertTrue(
+ len(dblStrings) == 1
+ and (dblStrings[0][1] == 154 and dblStrings[0][2] == 184),
+ "double quoted string failure",
+ )
+
+ allStrings = [
+ (t[0], b, e) for (t, b, e) in pp.quotedString.scanString(testData)
+ ]
+ print(allStrings)
+ self.assertTrue(
+ len(allStrings) == 2
+ and (allStrings[0][1] == 17 and allStrings[0][2] == 47)
+ and (allStrings[1][1] == 154 and allStrings[1][2] == 184),
+ "quoted string failure",
+ )
+
+ escapedQuoteTest = r"""
+ 'This string has an escaped (\') quote character'
+ "This string has an escaped (\") quote character"
+ """
+
+ sglStrings = [
+ (t[0], b, e)
+ for (t, b, e) in pp.sglQuotedString.scanString(escapedQuoteTest)
+ ]
+ print(sglStrings)
+ self.assertTrue(
+ len(sglStrings) == 1
+ and (sglStrings[0][1] == 17 and sglStrings[0][2] == 66),
+ "single quoted string escaped quote failure (%s)" % str(sglStrings[0]),
+ )
+
+ dblStrings = [
+ (t[0], b, e)
+ for (t, b, e) in pp.dblQuotedString.scanString(escapedQuoteTest)
+ ]
+ print(dblStrings)
+ self.assertTrue(
+ len(dblStrings) == 1
+ and (dblStrings[0][1] == 83 and dblStrings[0][2] == 132),
+ "double quoted string escaped quote failure (%s)" % str(dblStrings[0]),
+ )
+
+ allStrings = [
+ (t[0], b, e) for (t, b, e) in pp.quotedString.scanString(escapedQuoteTest)
+ ]
+ print(allStrings)
+ self.assertTrue(
+ len(allStrings) == 2
+ and (
+ allStrings[0][1] == 17
+ and allStrings[0][2] == 66
+ and allStrings[1][1] == 83
+ and allStrings[1][2] == 132
+ ),
+ "quoted string escaped quote failure (%s)"
+ % ([str(s[0]) for s in allStrings]),
+ )
+
+ dblQuoteTest = r"""
+ 'This string has an doubled ('') quote character'
+ "This string has an doubled ("") quote character"
+ """
+ sglStrings = [
+ (t[0], b, e) for (t, b, e) in pp.sglQuotedString.scanString(dblQuoteTest)
+ ]
+ print(sglStrings)
+ self.assertTrue(
+ len(sglStrings) == 1
+ and (sglStrings[0][1] == 17 and sglStrings[0][2] == 66),
+ "single quoted string escaped quote failure (%s)" % str(sglStrings[0]),
+ )
+ dblStrings = [
+ (t[0], b, e) for (t, b, e) in pp.dblQuotedString.scanString(dblQuoteTest)
+ ]
+ print(dblStrings)
+ self.assertTrue(
+ len(dblStrings) == 1
+ and (dblStrings[0][1] == 83 and dblStrings[0][2] == 132),
+ "double quoted string escaped quote failure (%s)" % str(dblStrings[0]),
+ )
+ allStrings = [
+ (t[0], b, e) for (t, b, e) in pp.quotedString.scanString(dblQuoteTest)
+ ]
+ print(allStrings)
+ self.assertTrue(
+ len(allStrings) == 2
+ and (
+ allStrings[0][1] == 17
+ and allStrings[0][2] == 66
+ and allStrings[1][1] == 83
+ and allStrings[1][2] == 132
+ ),
+ "quoted string escaped quote failure (%s)"
+ % ([str(s[0]) for s in allStrings]),
+ )
+
+ print(
+ "testing catastrophic RE backtracking in implementation of dblQuotedString"
+ )
+ for expr, test_string in [
+ (pp.dblQuotedString, '"' + "\\xff" * 500),
+ (pp.sglQuotedString, "'" + "\\xff" * 500),
+ (pp.quotedString, '"' + "\\xff" * 500),
+ (pp.quotedString, "'" + "\\xff" * 500),
+ (pp.QuotedString('"'), '"' + "\\xff" * 500),
+ (pp.QuotedString("'"), "'" + "\\xff" * 500),
+ ]:
+ expr.parseString(test_string + test_string[0])
+ try:
+ expr.parseString(test_string)
+ except Exception:
+ continue
+
+ # test invalid endQuoteChar
+ with self.assertRaises(
+ ValueError, msg="issue raising error for invalid endQuoteChar"
+ ):
+ expr = pp.QuotedString('"', endQuoteChar=" ")
+
+ def testCaselessOneOf(self):
+ caseless1 = pp.oneOf("d a b c aA B A C", caseless=True)
+ caseless1str = str(caseless1)
+ print(caseless1str)
+ caseless2 = pp.oneOf("d a b c Aa B A C", caseless=True)
+ caseless2str = str(caseless2)
+ print(caseless2str)
+ self.assertEqual(
+ caseless1str.upper(),
+ caseless2str.upper(),
+ "oneOf not handling caseless option properly",
+ )
+ self.assertNotEqual(
+ caseless1str, caseless2str, "Caseless option properly sorted"
+ )
+
+ res = caseless1[...].parseString("AAaaAaaA")
+ print(res)
+ self.assertEqual(4, len(res), "caseless1 oneOf failed")
+ self.assertEqual(
+ "aA" * 4, "".join(res), "caseless1 CaselessLiteral return failed"
+ )
+
+ res = caseless2[...].parseString("AAaaAaaA")
+ print(res)
+ self.assertEqual(4, len(res), "caseless2 oneOf failed")
+ self.assertEqual(
+ "Aa" * 4, "".join(res), "caseless1 CaselessLiteral return failed"
+ )
+
+ def testCommentParser(self):
+ print("verify processing of C and HTML comments")
+ testdata = """
+ /* */
+ /** **/
+ /**/
+ /***/
+ /****/
+ /* /*/
+ /** /*/
+ /*** /*/
+ /*
+ ablsjdflj
+ */
+ """
+ foundLines = [
+ pp.lineno(s, testdata) for t, s, e in pp.cStyleComment.scanString(testdata)
+ ]
+ self.assertEqual(
+ list(range(11))[2:],
+ foundLines,
+ "only found C comments on lines " + str(foundLines),
+ )
+ testdata = """
+ <!-- -->
+ <!--- --->
+ <!---->
+ <!----->
+ <!------>
+ <!-- /-->
+ <!--- /-->
+ <!---- /-->
+ <!---- /- ->
+ <!---- / -- >
+ <!--
+ ablsjdflj
+ -->
+ """
+ foundLines = [
+ pp.lineno(s, testdata) for t, s, e in pp.htmlComment.scanString(testdata)
+ ]
+ self.assertEqual(
+ list(range(11))[2:],
+ foundLines,
+ "only found HTML comments on lines " + str(foundLines),
+ )
+
+ # test C++ single line comments that have line terminated with '\' (should continue comment to following line)
+ testSource = r"""
+ // comment1
+ // comment2 \
+ still comment 2
+ // comment 3
+ """
+ self.assertEqual(
+ 41,
+ len(pp.cppStyleComment.searchString(testSource)[1][0]),
+ r"failed to match single-line comment with '\' at EOL",
+ )
+
+ def testParseExpressionResults(self):
+ a = pp.Word("a", pp.alphas).setName("A")
+ b = pp.Word("b", pp.alphas).setName("B")
+ c = pp.Word("c", pp.alphas).setName("C")
+ ab = (a + b).setName("AB")
+ abc = (ab + c).setName("ABC")
+ word = pp.Word(pp.alphas).setName("word")
+
+ words = pp.Group(pp.OneOrMore(~a + word)).setName("words")
+
+ phrase = (
+ words("Head")
+ + pp.Group(a + pp.Optional(b + pp.Optional(c)))("ABC")
+ + words("Tail")
+ )
+
+ results = phrase.parseString("xavier yeti alpha beta charlie will beaver")
+ print(results, results.Head, results.ABC, results.Tail)
+ for key, ln in [("Head", 2), ("ABC", 3), ("Tail", 2)]:
+ self.assertEqual(
+ ln,
+ len(results[key]),
+ "expected %d elements in %s, found %s" % (ln, key, str(results[key])),
+ )
+
+ def testParseKeyword(self):
+ kw = pp.Keyword("if")
+ lit = pp.Literal("if")
+
+ def test(s, litShouldPass, kwShouldPass):
+ print("Test", s)
+ print("Match Literal", end=" ")
+ try:
+ print(lit.parseString(s))
+ except Exception:
+ print("failed")
+ if litShouldPass:
+ self.fail("Literal failed to match %s, should have" % s)
+ else:
+ if not litShouldPass:
+ self.fail("Literal matched %s, should not have" % s)
+
+ print("Match Keyword", end=" ")
+ try:
+ print(kw.parseString(s))
+ except Exception:
+ print("failed")
+ if kwShouldPass:
+ self.fail("Keyword failed to match %s, should have" % s)
+ else:
+ if not kwShouldPass:
+ self.fail("Keyword matched %s, should not have" % s)
+
+ test("ifOnlyIfOnly", True, False)
+ test("if(OnlyIfOnly)", True, True)
+ test("if (OnlyIf Only)", True, True)
+
+ kw = pp.Keyword("if", caseless=True)
+
+ test("IFOnlyIfOnly", False, False)
+ test("If(OnlyIfOnly)", False, True)
+ test("iF (OnlyIf Only)", False, True)
+
+ with self.assertRaises(
+ ValueError, msg="failed to warn empty string passed to Keyword"
+ ):
+ kw = pp.Keyword("")
+
+ def testParseExpressionResultsAccumulate(self):
+ num = pp.Word(pp.nums).setName("num")("base10*")
+ hexnum = pp.Combine("0x" + pp.Word(pp.nums)).setName("hexnum")("hex*")
+ name = pp.Word(pp.alphas).setName("word")("word*")
+ list_of_num = pp.delimitedList(hexnum | num | name, ",")
+
+ tokens = list_of_num.parseString("1, 0x2, 3, 0x4, aaa")
+ print(tokens.dump())
+ self.assertParseResultsEquals(
+ tokens,
+ expected_list=["1", "0x2", "3", "0x4", "aaa"],
+ expected_dict={
+ "base10": ["1", "3"],
+ "hex": ["0x2", "0x4"],
+ "word": ["aaa"],
+ },
+ )
+
+ lbrack = pp.Literal("(").suppress()
+ rbrack = pp.Literal(")").suppress()
+ integer = pp.Word(pp.nums).setName("int")
+ variable = pp.Word(pp.alphas, max=1).setName("variable")
+ relation_body_item = (
+ variable | integer | pp.quotedString().setParseAction(pp.removeQuotes)
+ )
+ relation_name = pp.Word(pp.alphas + "_", pp.alphanums + "_")
+ relation_body = lbrack + pp.Group(pp.delimitedList(relation_body_item)) + rbrack
+ Goal = pp.Dict(pp.Group(relation_name + relation_body))
+ Comparison_Predicate = pp.Group(variable + pp.oneOf("< >") + integer)("pred*")
+ Query = Goal("head") + ":-" + pp.delimitedList(Goal | Comparison_Predicate)
+
+ test = """Q(x,y,z):-Bloo(x,"Mitsis",y),Foo(y,z,1243),y>28,x<12,x>3"""
+
+ queryRes = Query.parseString(test)
+ print(queryRes.dump())
+ self.assertParseResultsEquals(
+ queryRes.pred,
+ expected_list=[["y", ">", "28"], ["x", "<", "12"], ["x", ">", "3"]],
+ msg="Incorrect list for attribute pred, %s" % str(queryRes.pred.asList()),
+ )
+
+ def testReStringRange(self):
+ testCases = (
+ r"[A-Z]",
+ r"[A-A]",
+ r"[A-Za-z]",
+ r"[A-z]",
+ r"[\ -\~]",
+ r"[\0x20-0]",
+ r"[\0x21-\0x7E]",
+ r"[\0xa1-\0xfe]",
+ r"[\040-0]",
+ r"[A-Za-z0-9]",
+ r"[A-Za-z0-9_]",
+ r"[A-Za-z0-9_$]",
+ r"[A-Za-z0-9_$\-]",
+ r"[^0-9\\]",
+ r"[a-zA-Z]",
+ r"[/\^~]",
+ r"[=\+\-!]",
+ r"[A-]",
+ r"[-A]",
+ r"[\x21]",
+ r"[а-яА-ЯёЁA-Z$_\041α-ω]",
+ )
+ expectedResults = (
+ "ABCDEFGHIJKLMNOPQRSTUVWXYZ",
+ "A",
+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",
+ "ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz",
+ " !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~",
+ " !\"#$%&'()*+,-./0",
+ "!\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~",
+ "¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþ",
+ " !\"#$%&'()*+,-./0",
+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789",
+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_",
+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$",
+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$-",
+ "0123456789\\",
+ "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ",
+ "/^~",
+ "=+-!",
+ "A-",
+ "-A",
+ "!",
+ "абвгдежзийклмнопрстуфхцчшщъыьэюяАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯёЁABCDEFGHIJKLMNOPQRSTUVWXYZ$_!αβγδεζηθικλμνξοπρςστυφχψω",
+ )
+ for test in zip(testCases, expectedResults):
+ t, exp = test
+ res = pp.srange(t)
+ # print(t, "->", res)
+ self.assertEqual(
+ exp,
+ res,
+ "srange error, srange({!r})->'{!r}', expected '{!r}'".format(
+ t, res, exp
+ ),
+ )
+
+ def testSkipToParserTests(self):
+ thingToFind = pp.Literal("working")
+ testExpr = (
+ pp.SkipTo(pp.Literal(";"), include=True, ignore=pp.cStyleComment)
+ + thingToFind
+ )
+
+ def test_parse(someText):
+ print(testExpr.parseString(someText))
+
+ # This first test works, as the SkipTo expression is immediately following the ignore expression (cStyleComment)
+ test_parse("some text /* comment with ; in */; working")
+ # This second test previously failed, as there is text following the ignore expression, and before the SkipTo expression.
+ test_parse("some text /* comment with ; in */some other stuff; working")
+
+ # tests for optional failOn argument
+ testExpr = (
+ pp.SkipTo(
+ pp.Literal(";"), include=True, ignore=pp.cStyleComment, failOn="other"
+ )
+ + thingToFind
+ )
+ test_parse("some text /* comment with ; in */; working")
+
+ with self.assertRaisesParseException():
+ test_parse("some text /* comment with ; in */some other stuff; working")
+
+ # test that we correctly create named results
+ text = "prefixDATAsuffix"
+ data = pp.Literal("DATA")
+ suffix = pp.Literal("suffix")
+ expr = pp.SkipTo(data + suffix)("prefix") + data + suffix
+ result = expr.parseString(text)
+ self.assertTrue(
+ isinstance(result.prefix, str),
+ "SkipTo created with wrong saveAsList attribute",
+ )
+
+ alpha_word = (~pp.Literal("end") + pp.Word(pp.alphas, asKeyword=True)).setName(
+ "alpha"
+ )
+ num_word = pp.Word(pp.nums, asKeyword=True).setName("int")
+
+ def test(expr, test_string, expected_list, expected_dict):
+ if (expected_list, expected_dict) == (None, None):
+ with self.assertRaises(
+ Exception, msg="{} failed to parse {!r}".format(expr, test_string)
+ ):
+ expr.parseString(test_string)
+ else:
+ result = expr.parseString(test_string)
+ self.assertParseResultsEquals(
+ result, expected_list=expected_list, expected_dict=expected_dict
+ )
+
+ # ellipses for SkipTo
+ e = ... + pp.Literal("end")
+ test(e, "start 123 end", ["start 123 ", "end"], {"_skipped": ["start 123 "]})
+
+ e = pp.Suppress(...) + pp.Literal("end")
+ test(e, "start 123 end", ["end"], {})
+
+ e = pp.Literal("start") + ... + pp.Literal("end")
+ test(e, "start 123 end", ["start", "123 ", "end"], {"_skipped": ["123 "]})
+
+ e = ... + pp.Literal("middle") + ... + pp.Literal("end")
+ test(
+ e,
+ "start 123 middle 456 end",
+ ["start 123 ", "middle", "456 ", "end"],
+ {"_skipped": ["start 123 ", "456 "]},
+ )
+
+ e = pp.Suppress(...) + pp.Literal("middle") + ... + pp.Literal("end")
+ test(
+ e,
+ "start 123 middle 456 end",
+ ["middle", "456 ", "end"],
+ {"_skipped": ["456 "]},
+ )
+
+ e = pp.Literal("start") + ...
+ test(e, "start 123 end", None, None)
+
+ e = pp.And(["start", ..., "end"])
+ test(e, "start 123 end", ["start", "123 ", "end"], {"_skipped": ["123 "]})
+
+ e = pp.And([..., "end"])
+ test(e, "start 123 end", ["start 123 ", "end"], {"_skipped": ["start 123 "]})
+
+ e = "start" + (num_word | ...) + "end"
+ test(e, "start 456 end", ["start", "456", "end"], {})
+ test(
+ e,
+ "start 123 456 end",
+ ["start", "123", "456 ", "end"],
+ {"_skipped": ["456 "]},
+ )
+ test(e, "start end", ["start", "", "end"], {"_skipped": ["missing <int>"]})
+
+ # e = define_expr('"start" + (num_word | ...)("inner") + "end"')
+ # test(e, "start 456 end", ['start', '456', 'end'], {'inner': '456'})
+
+ e = "start" + (alpha_word[...] & num_word[...] | ...) + "end"
+ test(e, "start 456 red end", ["start", "456", "red", "end"], {})
+ test(e, "start red 456 end", ["start", "red", "456", "end"], {})
+ test(
+ e,
+ "start 456 red + end",
+ ["start", "456", "red", "+ ", "end"],
+ {"_skipped": ["+ "]},
+ )
+ test(e, "start red end", ["start", "red", "end"], {})
+ test(e, "start 456 end", ["start", "456", "end"], {})
+ test(e, "start end", ["start", "end"], {})
+ test(e, "start 456 + end", ["start", "456", "+ ", "end"], {"_skipped": ["+ "]})
+
+ e = "start" + (alpha_word[1, ...] & num_word[1, ...] | ...) + "end"
+ test(e, "start 456 red end", ["start", "456", "red", "end"], {})
+ test(e, "start red 456 end", ["start", "red", "456", "end"], {})
+ test(
+ e,
+ "start 456 red + end",
+ ["start", "456", "red", "+ ", "end"],
+ {"_skipped": ["+ "]},
+ )
+ test(e, "start red end", ["start", "red ", "end"], {"_skipped": ["red "]})
+ test(e, "start 456 end", ["start", "456 ", "end"], {"_skipped": ["456 "]})
+ test(
+ e,
+ "start end",
+ ["start", "", "end"],
+ {"_skipped": ["missing <{{alpha}... & {int}...}>"]},
+ )
+ test(e, "start 456 + end", ["start", "456 + ", "end"], {"_skipped": ["456 + "]})
+
+ e = "start" + (alpha_word | ...) + (num_word | ...) + "end"
+ test(e, "start red 456 end", ["start", "red", "456", "end"], {})
+ test(
+ e,
+ "start red end",
+ ["start", "red", "", "end"],
+ {"_skipped": ["missing <int>"]},
+ )
+ test(
+ e,
+ "start end",
+ ["start", "", "", "end"],
+ {"_skipped": ["missing <alpha>", "missing <int>"]},
+ )
+
+ e = pp.Literal("start") + ... + "+" + ... + "end"
+ test(
+ e,
+ "start red + 456 end",
+ ["start", "red ", "+", "456 ", "end"],
+ {"_skipped": ["red ", "456 "]},
+ )
+
+ def testEllipsisRepetition(self):
+
+ word = pp.Word(pp.alphas).setName("word")
+ num = pp.Word(pp.nums).setName("num")
+
+ exprs = [
+ word[...] + num,
+ word[0, ...] + num,
+ word[1, ...] + num,
+ word[2, ...] + num,
+ word[..., 3] + num,
+ word[2] + num,
+ ]
+
+ expected_res = [
+ r"([abcd]+ )*\d+",
+ r"([abcd]+ )*\d+",
+ r"([abcd]+ )+\d+",
+ r"([abcd]+ ){2,}\d+",
+ r"([abcd]+ ){0,3}\d+",
+ r"([abcd]+ ){2}\d+",
+ ]
+
+ tests = ["aa bb cc dd 123", "bb cc dd 123", "cc dd 123", "dd 123", "123"]
+
+ all_success = True
+ for expr, expected_re in zip(exprs, expected_res):
+ successful_tests = [t for t in tests if re.match(expected_re, t)]
+ failure_tests = [t for t in tests if not re.match(expected_re, t)]
+ success1, _ = expr.runTests(successful_tests)
+ success2, _ = expr.runTests(failure_tests, failureTests=True)
+ all_success = all_success and success1 and success2
+ if not all_success:
+ print("Failed expression:", expr)
+ break
+
+ self.assertTrue(all_success, "failed getItem_ellipsis test")
+
+ def testEllipsisRepetitionWithResultsNames(self):
+ label = pp.Word(pp.alphas)
+ val = ppc.integer()
+ parser = label("label") + pp.ZeroOrMore(val)("values")
+
+ _, results = parser.runTests(
+ """
+ a 1
+ b 1 2 3
+ c
+ """
+ )
+ expected = [
+ (["a", 1], {"label": "a", "values": [1]}),
+ (["b", 1, 2, 3], {"label": "b", "values": [1, 2, 3]}),
+ (["c"], {"label": "c", "values": []}),
+ ]
+ for obs, exp in zip(results, expected):
+ test, result = obs
+ exp_list, exp_dict = exp
+ self.assertParseResultsEquals(
+ result, expected_list=exp_list, expected_dict=exp_dict
+ )
+
+ parser = label("label") + val[...]("values")
+
+ _, results = parser.runTests(
+ """
+ a 1
+ b 1 2 3
+ c
+ """
+ )
+ expected = [
+ (["a", 1], {"label": "a", "values": [1]}),
+ (["b", 1, 2, 3], {"label": "b", "values": [1, 2, 3]}),
+ (["c"], {"label": "c", "values": []}),
+ ]
+ for obs, exp in zip(results, expected):
+ test, result = obs
+ exp_list, exp_dict = exp
+ self.assertParseResultsEquals(
+ result, expected_list=exp_list, expected_dict=exp_dict
+ )
+
+ pt = pp.Group(val("x") + pp.Suppress(",") + val("y"))
+ parser = label("label") + pt[...]("points")
+ _, results = parser.runTests(
+ """
+ a 1,1
+ b 1,1 2,2 3,3
+ c
+ """
+ )
+ expected = [
+ (["a", [1, 1]], {"label": "a", "points": [{"x": 1, "y": 1}]}),
+ (
+ ["b", [1, 1], [2, 2], [3, 3]],
+ {
+ "label": "b",
+ "points": [{"x": 1, "y": 1}, {"x": 2, "y": 2}, {"x": 3, "y": 3}],
+ },
+ ),
+ (["c"], {"label": "c", "points": []}),
+ ]
+ for obs, exp in zip(results, expected):
+ test, result = obs
+ exp_list, exp_dict = exp
+ self.assertParseResultsEquals(
+ result, expected_list=exp_list, expected_dict=exp_dict
+ )
+
+ def testCustomQuotes(self):
+ testString = r"""
+ sdlfjs :sdf\:jls::djf: sl:kfsjf
+ sdlfjs -sdf\:jls::--djf: sl-kfsjf
+ sdlfjs -sdf\:::jls::--djf: sl:::-kfsjf
+ sdlfjs ^sdf\:jls^^--djf^ sl-kfsjf
+ sdlfjs ^^^==sdf\:j=lz::--djf: sl=^^=kfsjf
+ sdlfjs ==sdf\:j=ls::--djf: sl==kfsjf^^^
+ """
+ print(testString)
+
+ colonQuotes = pp.QuotedString(":", "\\", "::")
+ dashQuotes = pp.QuotedString("-", "\\", "--")
+ hatQuotes = pp.QuotedString("^", "\\")
+ hatQuotes1 = pp.QuotedString("^", "\\", "^^")
+ dblEqQuotes = pp.QuotedString("==", "\\")
+
+ def test(label, quoteExpr, expected):
+ print(label)
+ print(quoteExpr.pattern)
+ print(quoteExpr.searchString(testString))
+ print(quoteExpr.searchString(testString)[0][0])
+ print(f"{expected}")
+ self.assertEqual(
+ expected,
+ quoteExpr.searchString(testString)[0][0],
+ "failed to match {}, expected '{}', got '{}'".format(
+ quoteExpr, expected, quoteExpr.searchString(testString)[0]
+ ),
+ )
+ print()
+
+ test("colonQuotes", colonQuotes, r"sdf:jls:djf")
+ test("dashQuotes", dashQuotes, r"sdf:jls::-djf: sl")
+ test("hatQuotes", hatQuotes, r"sdf:jls")
+ test("hatQuotes1", hatQuotes1, r"sdf:jls^--djf")
+ test("dblEqQuotes", dblEqQuotes, r"sdf:j=ls::--djf: sl")
+ test("::: quotes", pp.QuotedString(":::"), "jls::--djf: sl")
+ test("==-- quotes", pp.QuotedString("==", endQuoteChar="--"), r"sdf\:j=lz::")
+ test(
+ "^^^ multiline quotes",
+ pp.QuotedString("^^^", multiline=True),
+ r"""==sdf\:j=lz::--djf: sl=^^=kfsjf
+ sdlfjs ==sdf\:j=ls::--djf: sl==kfsjf""",
+ )
+ with self.assertRaises(ValueError):
+ pp.QuotedString("", "\\")
+
+ def testRepeater(self):
+ if ParserElement._packratEnabled or ParserElement._left_recursion_enabled:
+ print("skipping this test, not compatible with memoization")
+ return
+
+ first = pp.Word("abcdef").setName("word1")
+ bridge = pp.Word(pp.nums).setName("number")
+ second = pp.matchPreviousLiteral(first).setName("repeat(word1Literal)")
+
+ seq = first + bridge + second
+
+ tests = [
+ ("abc12abc", True),
+ ("abc12aabc", False),
+ ("abc12cba", True),
+ ("abc12bca", True),
+ ]
+
+ for tst, expected in tests:
+ found = False
+ for tokens, start, end in seq.scanString(tst):
+ f, b, s = tokens
+ print(f, b, s)
+ found = True
+ if not found:
+ print("No literal match in", tst)
+ self.assertEqual(
+ expected,
+ found,
+ "Failed repeater for test: {}, matching {}".format(tst, str(seq)),
+ )
+ print()
+
+ # retest using matchPreviousExpr instead of matchPreviousLiteral
+ second = pp.matchPreviousExpr(first).setName("repeat(word1expr)")
+ seq = first + bridge + second
+
+ tests = [("abc12abc", True), ("abc12cba", False), ("abc12abcdef", False)]
+
+ for tst, expected in tests:
+ found = False
+ for tokens, start, end in seq.scanString(tst):
+ print(tokens)
+ found = True
+ if not found:
+ print("No expression match in", tst)
+ self.assertEqual(
+ expected,
+ found,
+ "Failed repeater for test: {}, matching {}".format(tst, str(seq)),
+ )
+
+ print()
+
+ first = pp.Word("abcdef").setName("word1")
+ bridge = pp.Word(pp.nums).setName("number")
+ second = pp.matchPreviousExpr(first).setName("repeat(word1)")
+ seq = first + bridge + second
+ csFirst = seq.setName("word-num-word")
+ csSecond = pp.matchPreviousExpr(csFirst)
+ compoundSeq = csFirst + ":" + csSecond
+ compoundSeq.streamline()
+ print(compoundSeq)
+
+ tests = [
+ ("abc12abc:abc12abc", True),
+ ("abc12cba:abc12abc", False),
+ ("abc12abc:abc12abcdef", False),
+ ]
+
+ for tst, expected in tests:
+ found = False
+ for tokens, start, end in compoundSeq.scanString(tst):
+ print("match:", tokens)
+ found = True
+ break
+ if not found:
+ print("No expression match in", tst)
+ self.assertEqual(
+ expected,
+ found,
+ "Failed repeater for test: {}, matching {}".format(tst, str(seq)),
+ )
+
+ print()
+ eFirst = pp.Word(pp.nums)
+ eSecond = pp.matchPreviousExpr(eFirst)
+ eSeq = eFirst + ":" + eSecond
+
+ tests = [("1:1A", True), ("1:10", False)]
+
+ for tst, expected in tests:
+ found = False
+ for tokens, start, end in eSeq.scanString(tst):
+ print(tokens)
+ found = True
+ if not found:
+ print("No match in", tst)
+ self.assertEqual(
+ expected,
+ found,
+ "Failed repeater for test: {}, matching {}".format(tst, str(seq)),
+ )
+
+ def testRepeater2(self):
+ """test matchPreviousLiteral with empty repeater"""
+
+ if ParserElement._packratEnabled or ParserElement._left_recursion_enabled:
+ print("skipping this test, not compatible with memoization")
+ return
+
+ first = pp.Optional(pp.Word("abcdef").setName("words1"))
+ bridge = pp.Word(pp.nums).setName("number")
+ second = pp.matchPreviousLiteral(first).setName("repeat(word1Literal)")
+
+ seq = first + bridge + second
+
+ tst = "12"
+ expected = ["12"]
+ result = seq.parseString(tst)
+ print(result.dump())
+
+ self.assertParseResultsEquals(result, expected_list=expected)
+
+ def testRepeater3(self):
+ """test matchPreviousLiteral with multiple repeater tokens"""
+
+ if ParserElement._packratEnabled or ParserElement._left_recursion_enabled:
+ print("skipping this test, not compatible with memoization")
+ return
+
+ first = pp.Word("a") + pp.Word("d")
+ bridge = pp.Word(pp.nums).setName("number")
+ second = pp.matchPreviousLiteral(first) # ("second")
+
+ seq = first + bridge + second
+
+ tst = "aaaddd12aaaddd"
+ expected = ["aaa", "ddd", "12", "aaa", "ddd"]
+ result = seq.parseString(tst)
+ print(result.dump())
+
+ self.assertParseResultsEquals(result, expected_list=expected)
+
+ def testRepeater4(self):
+ """test matchPreviousExpr with multiple repeater tokens"""
+
+ if ParserElement._packratEnabled or ParserElement._left_recursion_enabled:
+ print("skipping this test, not compatible with memoization")
+ return
+
+ first = pp.Group(pp.Word(pp.alphas) + pp.Word(pp.alphas))
+ bridge = pp.Word(pp.nums)
+
+ # no matching is used - this is just here for a sanity check
+ # second = pp.Group(pp.Word(pp.alphas) + pp.Word(pp.alphas))("second")
+ # second = pp.Group(pp.Word(pp.alphas) + pp.Word(pp.alphas)).setResultsName("second")
+
+ # ISSUE: when matchPreviousExpr returns multiple tokens the matching tokens are nested an extra level deep.
+ # This behavior is not seen with a single return token (see testRepeater5 directly below.)
+ second = pp.matchPreviousExpr(first)
+
+ expr = first + bridge.suppress() + second
+
+ tst = "aaa ddd 12 aaa ddd"
+ expected = [["aaa", "ddd"], ["aaa", "ddd"]]
+ result = expr.parseString(tst)
+ print(result.dump())
+
+ self.assertParseResultsEquals(result, expected_list=expected)
+
+ def testRepeater5(self):
+ """a simplified testRepeater4 to examine matchPreviousExpr with a single repeater token"""
+
+ if ParserElement._packratEnabled or ParserElement._left_recursion_enabled:
+ print("skipping this test, not compatible with memoization")
+ return
+
+ first = pp.Word(pp.alphas)
+ bridge = pp.Word(pp.nums)
+ second = pp.matchPreviousExpr(first)
+
+ expr = first + bridge.suppress() + second
+
+ tst = "aaa 12 aaa"
+ expected = tst.replace("12", "").split()
+ result = expr.parseString(tst)
+ print(result.dump())
+
+ self.assertParseResultsEquals(result, expected_list=expected)
+
+ def testRecursiveCombine(self):
+ testInput = "myc(114)r(11)dd"
+ stream = pp.Forward()
+ stream <<= pp.Optional(pp.Word(pp.alphas)) + pp.Optional(
+ "(" + pp.Word(pp.nums) + ")" + stream
+ )
+ expected = ["".join(stream.parseString(testInput))]
+ print(expected)
+
+ stream = pp.Forward()
+ stream << pp.Combine(
+ pp.Optional(pp.Word(pp.alphas))
+ + pp.Optional("(" + pp.Word(pp.nums) + ")" + stream)
+ )
+ testVal = stream.parseString(testInput)
+ print(testVal)
+
+ self.assertParseResultsEquals(testVal, expected_list=expected)
+
+ def testHTMLEntities(self):
+ html_source = dedent("""\
+ This & that
+ 2 > 1
+ 0 < 1
+ Don't get excited!
+ I said "Don't get excited!"
+ Copyright © 2021
+ Dot ⟶ ˙
+ """)
+ transformer = pp.common_html_entity.add_parse_action(pp.replace_html_entity)
+ transformed = transformer.transform_string(html_source)
+ print(transformed)
+
+ expected = dedent("""\
+ This & that
+ 2 > 1
+ 0 < 1
+ Don't get excited!
+ I said "Don't get excited!"
+ Copyright © 2021
+ Dot ⟶ ˙
+ """)
+ self.assertEqual(expected, transformed)
+
+ def testInfixNotationBasicArithEval(self):
+ import ast
+
+ integer = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
+ variable = pp.Word(pp.alphas, exact=1)
+ operand = integer | variable
+
+ expop = pp.Literal("^")
+ signop = pp.oneOf("+ -")
+ multop = pp.oneOf("* /")
+ plusop = pp.oneOf("+ -")
+ factop = pp.Literal("!")
+
+ expr = pp.infixNotation(
+ operand,
+ [
+ (factop, 1, pp.opAssoc.LEFT),
+ (expop, 2, pp.opAssoc.RIGHT),
+ (signop, 1, pp.opAssoc.RIGHT),
+ (multop, 2, pp.opAssoc.LEFT),
+ (plusop, 2, pp.opAssoc.LEFT),
+ ],
+ )
+
+ test = [
+ "9 + 2 + 3",
+ "9 + 2 * 3",
+ "(9 + 2) * 3",
+ "(9 + -2) * 3",
+ "(9 + --2) * 3",
+ "(9 + -2) * 3^2^2",
+ "(9! + -2) * 3^2^2",
+ "M*X + B",
+ "M*(X + B)",
+ "1+2*-3^4*5+-+-6",
+ "3!!",
+ ]
+ expected = """[[9, '+', 2, '+', 3]]
+ [[9, '+', [2, '*', 3]]]
+ [[[9, '+', 2], '*', 3]]
+ [[[9, '+', ['-', 2]], '*', 3]]
+ [[[9, '+', ['-', ['-', 2]]], '*', 3]]
+ [[[9, '+', ['-', 2]], '*', [3, '^', [2, '^', 2]]]]
+ [[[[9, '!'], '+', ['-', 2]], '*', [3, '^', [2, '^', 2]]]]
+ [[['M', '*', 'X'], '+', 'B']]
+ [['M', '*', ['X', '+', 'B']]]
+ [[1, '+', [2, '*', ['-', [3, '^', 4]], '*', 5], '+', ['-', ['+', ['-', 6]]]]]
+ [[3, '!', '!']]""".split(
+ "\n"
+ )
+ expected = [ast.literal_eval(x.strip()) for x in expected]
+ for test_str, exp_list in zip(test, expected):
+ self.assertParseAndCheckList(expr, test_str, exp_list, verbose=True)
+
+ def testInfixNotationEvalBoolExprUsingAstClasses(self):
+ boolVars = {"True": True, "False": False}
+
+ class BoolOperand:
+ reprsymbol = ""
+
+ def __init__(self, t):
+ self.args = t[0][0::2]
+
+ def __str__(self):
+ sep = " %s " % self.reprsymbol
+ return "(" + sep.join(map(str, self.args)) + ")"
+
+ class BoolAnd(BoolOperand):
+ reprsymbol = "&"
+
+ def __bool__(self):
+ for a in self.args:
+ if isinstance(a, str):
+ v = boolVars[a]
+ else:
+ v = bool(a)
+ if not v:
+ return False
+ return True
+
+ class BoolOr(BoolOperand):
+ reprsymbol = "|"
+
+ def __bool__(self):
+ for a in self.args:
+ if isinstance(a, str):
+ v = boolVars[a]
+ else:
+ v = bool(a)
+ if v:
+ return True
+ return False
+
+ class BoolNot:
+ def __init__(self, t):
+ self.arg = t[0][1]
+
+ def __str__(self):
+ return "~" + str(self.arg)
+
+ def __bool__(self):
+ if isinstance(self.arg, str):
+ v = boolVars[self.arg]
+ else:
+ v = bool(self.arg)
+ return not v
+
+ boolOperand = pp.Word(pp.alphas, max=1, asKeyword=True) | pp.oneOf("True False")
+ boolExpr = pp.infixNotation(
+ boolOperand,
+ [
+ ("not", 1, pp.opAssoc.RIGHT, BoolNot),
+ ("and", 2, pp.opAssoc.LEFT, BoolAnd),
+ ("or", 2, pp.opAssoc.LEFT, BoolOr),
+ ],
+ )
+ test = [
+ "p and not q",
+ "not not p",
+ "not(p and q)",
+ "q or not p and r",
+ "q or not p or not r",
+ "q or not (p and r)",
+ "p or q or r",
+ "p or q or r and False",
+ "(p or q or r) and False",
+ ]
+
+ boolVars["p"] = True
+ boolVars["q"] = False
+ boolVars["r"] = True
+ print("p =", boolVars["p"])
+ print("q =", boolVars["q"])
+ print("r =", boolVars["r"])
+ print()
+ for t in test:
+ res = boolExpr.parseString(t)
+ print(t, "\n", res[0], "=", bool(res[0]), "\n")
+ expected = eval(t, {}, boolVars)
+ self.assertEqual(
+ expected, bool(res[0]), "failed boolean eval test {}".format(t)
+ )
+
+ def testInfixNotationMinimalParseActionCalls(self):
+ count = 0
+
+ def evaluate_int(t):
+ nonlocal count
+ value = int(t[0])
+ print("evaluate_int", value)
+ count += 1
+ return value
+
+ integer = pp.Word(pp.nums).setParseAction(evaluate_int)
+ variable = pp.Word(pp.alphas, exact=1)
+ operand = integer | variable
+
+ expop = pp.Literal("^")
+ signop = pp.oneOf("+ -")
+ multop = pp.oneOf("* /")
+ plusop = pp.oneOf("+ -")
+ factop = pp.Literal("!")
+
+ expr = pp.infixNotation(
+ operand,
+ [
+ ("!", 1, pp.opAssoc.LEFT),
+ ("^", 2, pp.opAssoc.LEFT),
+ (signop, 1, pp.opAssoc.RIGHT),
+ (multop, 2, pp.opAssoc.LEFT),
+ (plusop, 2, pp.opAssoc.LEFT),
+ ],
+ )
+
+ test = ["9"]
+ for t in test:
+ count = 0
+ print("%r => %s (count=%d)" % (t, expr.parseString(t), count))
+ self.assertEqual(1, count, "count evaluated too many times!")
+
+ def testInfixNotationWithParseActions(self):
+ word = pp.Word(pp.alphas)
+
+ def supLiteral(s):
+ """Returns the suppressed literal s"""
+ return pp.Literal(s).suppress()
+
+ def booleanExpr(atom):
+ ops = [
+ (supLiteral("!"), 1, pp.opAssoc.RIGHT, lambda s, l, t: ["!", t[0][0]]),
+ (pp.oneOf("= !="), 2, pp.opAssoc.LEFT),
+ (supLiteral("&"), 2, pp.opAssoc.LEFT, lambda s, l, t: ["&", t[0]]),
+ (supLiteral("|"), 2, pp.opAssoc.LEFT, lambda s, l, t: ["|", t[0]]),
+ ]
+ return pp.infixNotation(atom, ops)
+
+ f = booleanExpr(word) + pp.StringEnd()
+
+ tests = [
+ ("bar = foo", [["bar", "=", "foo"]]),
+ (
+ "bar = foo & baz = fee",
+ ["&", [["bar", "=", "foo"], ["baz", "=", "fee"]]],
+ ),
+ ]
+ for test, expected in tests:
+ print(test)
+ results = f.parseString(test)
+ print(results)
+ self.assertParseResultsEquals(results, expected_list=expected)
+ print()
+
+ def testInfixNotationGrammarTest5(self):
+ expop = pp.Literal("**")
+ signop = pp.oneOf("+ -")
+ multop = pp.oneOf("* /")
+ plusop = pp.oneOf("+ -")
+
+ class ExprNode:
+ def __init__(self, tokens):
+ self.tokens = tokens[0]
+
+ def eval(self):
+ return None
+
+ class NumberNode(ExprNode):
+ def eval(self):
+ return self.tokens
+
+ class SignOp(ExprNode):
+ def eval(self):
+ mult = {"+": 1, "-": -1}[self.tokens[0]]
+ return mult * self.tokens[1].eval()
+
+ class BinOp(ExprNode):
+ opn_map = {}
+
+ def eval(self):
+ ret = self.tokens[0].eval()
+ for op, operand in zip(self.tokens[1::2], self.tokens[2::2]):
+ ret = self.opn_map[op](ret, operand.eval())
+ return ret
+
+ class ExpOp(BinOp):
+ opn_map = {"**": lambda a, b: b ** a}
+
+ class MultOp(BinOp):
+ import operator
+
+ opn_map = {"*": operator.mul, "/": operator.truediv}
+
+ class AddOp(BinOp):
+ import operator
+
+ opn_map = {"+": operator.add, "-": operator.sub}
+
+ operand = ppc.number().setParseAction(NumberNode)
+ expr = pp.infixNotation(
+ operand,
+ [
+ (expop, 2, pp.opAssoc.LEFT, (lambda pr: [pr[0][::-1]], ExpOp)),
+ (signop, 1, pp.opAssoc.RIGHT, SignOp),
+ (multop, 2, pp.opAssoc.LEFT, MultOp),
+ (plusop, 2, pp.opAssoc.LEFT, AddOp),
+ ],
+ )
+
+ tests = """\
+ 2+7
+ 2**3
+ 2**3**2
+ 3**9
+ 3**3**2
+ """
+
+ for t in tests.splitlines():
+ t = t.strip()
+ if not t:
+ continue
+
+ parsed = expr.parseString(t)
+ eval_value = parsed[0].eval()
+ self.assertEqual(
+ eval(t),
+ eval_value,
+ "Error evaluating {!r}, expected {!r}, got {!r}".format(
+ t, eval(t), eval_value
+ ),
+ )
+
+ def testInfixNotationExceptions(self):
+ num = pp.Word(pp.nums)
+
+ # arity 3 with None opExpr - should raise ValueError
+ with self.assertRaises(ValueError):
+ expr = pp.infixNotation(num, [(None, 3, pp.opAssoc.LEFT)])
+
+ # arity 3 with invalid tuple - should raise ValueError
+ with self.assertRaises(ValueError):
+ expr = pp.infixNotation(num, [(("+", "-", "*"), 3, pp.opAssoc.LEFT)])
+
+ # left arity > 3 - should raise ValueError
+ with self.assertRaises(ValueError):
+ expr = pp.infixNotation(num, [("*", 4, pp.opAssoc.LEFT)])
+
+ # right arity > 3 - should raise ValueError
+ with self.assertRaises(ValueError):
+ expr = pp.infixNotation(num, [("*", 4, pp.opAssoc.RIGHT)])
+
+ # assoc not from opAssoc - should raise ValueError
+ with self.assertRaises(ValueError):
+ expr = pp.infixNotation(num, [("*", 2, "LEFT")])
+
+ def testInfixNotationWithNonOperators(self):
+ # left arity 2 with None expr
+ # right arity 2 with None expr
+ num = pp.Word(pp.nums).addParseAction(pp.tokenMap(int))
+ ident = ppc.identifier()
+ for assoc in (pp.opAssoc.LEFT, pp.opAssoc.RIGHT):
+ expr = pp.infixNotation(
+ num | ident, [(None, 2, assoc), ("+", 2, pp.opAssoc.LEFT)]
+ )
+ self.assertParseAndCheckList(expr, "3x+2", [[[3, "x"], "+", 2]])
+
+ def testInfixNotationTernaryOperator(self):
+ # left arity 3
+ # right arity 3
+ num = pp.Word(pp.nums).addParseAction(pp.tokenMap(int))
+ for assoc in (pp.opAssoc.LEFT, pp.opAssoc.RIGHT):
+ expr = pp.infixNotation(
+ num, [("+", 2, pp.opAssoc.LEFT), (("?", ":"), 3, assoc)]
+ )
+ self.assertParseAndCheckList(
+ expr, "3 + 2? 12: 13", [[[3, "+", 2], "?", 12, ":", 13]]
+ )
+
+ def testParseResultsPickle(self):
+ import pickle
+
+ # test 1
+ body = pp.makeHTMLTags("BODY")[0]
+ result = body.parseString("<BODY BGCOLOR='#00FFBB' FGCOLOR=black>")
+ print(result.dump())
+
+ for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
+ print("Test pickle dump protocol", protocol)
+ try:
+ pickleString = pickle.dumps(result, protocol)
+ except Exception as e:
+ print("dumps exception:", e)
+ newresult = pp.ParseResults()
+ else:
+ newresult = pickle.loads(pickleString)
+ print(newresult.dump())
+
+ self.assertEqual(
+ result.dump(),
+ newresult.dump(),
+ "Error pickling ParseResults object (protocol=%d)" % protocol,
+ )
+
+ def testParseResultsPickle2(self):
+ import pickle
+
+ word = pp.Word(pp.alphas + "'.")
+ salutation = pp.OneOrMore(word)
+ comma = pp.Literal(",")
+ greetee = pp.OneOrMore(word)
+ endpunc = pp.oneOf("! ?")
+ greeting = (
+ salutation("greeting")
+ + pp.Suppress(comma)
+ + greetee("greetee")
+ + endpunc("punc*")[1, ...]
+ )
+
+ string = "Good morning, Miss Crabtree!"
+
+ result = greeting.parseString(string)
+ self.assertParseResultsEquals(
+ result,
+ ["Good", "morning", "Miss", "Crabtree", "!"],
+ {
+ "greeting": ["Good", "morning"],
+ "greetee": ["Miss", "Crabtree"],
+ "punc": ["!"],
+ },
+ )
+ print(result.dump())
+
+ for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
+ print("Test pickle dump protocol", protocol)
+ try:
+ pickleString = pickle.dumps(result, protocol)
+ except Exception as e:
+ print("dumps exception:", e)
+ newresult = pp.ParseResults()
+ else:
+ newresult = pickle.loads(pickleString)
+ print(newresult.dump())
+ self.assertEqual(
+ newresult.dump(),
+ result.dump(),
+ "failed to pickle/unpickle ParseResults: expected {!r}, got {!r}".format(
+ result, newresult
+ ),
+ )
+
+ def testParseResultsPickle3(self):
+ import pickle
+
+ # result with aslist=False
+ res_not_as_list = pp.Word("ABC").parseString("BABBAB")
+
+ # result with aslist=True
+ res_as_list = pp.Group(pp.Word("ABC")).parseString("BABBAB")
+
+ # result with modal=True
+ res_modal = pp.Word("ABC")("name").parseString("BABBAB")
+ # self.assertTrue(res_modal._modal)
+
+ # result with modal=False
+ res_not_modal = pp.Word("ABC")("name*").parseString("BABBAB")
+ # self.assertFalse(res_not_modal._modal)
+
+ for result in (res_as_list, res_not_as_list, res_modal, res_not_modal):
+ for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
+ print("Test pickle dump protocol", protocol)
+ try:
+ pickleString = pickle.dumps(result, protocol)
+ except Exception as e:
+ print("dumps exception:", e)
+ newresult = pp.ParseResults()
+ else:
+ newresult = pickle.loads(pickleString)
+ print(newresult.dump())
+ self.assertEqual(
+ newresult.dump(),
+ result.dump(),
+ "failed to pickle/unpickle ParseResults: expected {!r}, got {!r}".format(
+ result, newresult
+ ),
+ )
+
+ def testParseResultsInsertWithResultsNames(self):
+ test_string = "1 2 3 dice rolled first try"
+
+ wd = pp.Word(pp.alphas)
+ num = ppc.number
+
+ expr = (
+ pp.Group(num[1, ...])("nums")
+ + wd("label")
+ + pp.Group(wd[...])("additional")
+ )
+
+ result = expr.parseString(test_string)
+ print("Pre-insert")
+ print(result.dump())
+
+ result.insert(1, sum(result.nums))
+
+ print("\nPost-insert")
+ print(result.dump())
+
+ self.assertParseResultsEquals(
+ result,
+ expected_list=[[1, 2, 3], 6, "dice", ["rolled", "first", "try"]],
+ expected_dict={
+ "additional": ["rolled", "first", "try"],
+ "label": "dice",
+ "nums": [1, 2, 3],
+ },
+ )
+
+ def testParseResultsStringListUsingCombine(self):
+ test_string = "1 2 3 dice rolled first try"
+
+ wd = pp.Word(pp.alphas)
+ num = ppc.number
+
+ expr = pp.Combine(
+ pp.Group(num[1, ...])("nums")
+ + wd("label")
+ + pp.Group(wd[...])("additional"),
+ joinString="/",
+ adjacent=False,
+ )
+ self.assertEqual("123/dice/rolledfirsttry", expr.parseString(test_string)[0])
+
+ def testParseResultsAcceptingACollectionTypeValue(self):
+ # from Issue #276 - ParseResults parameterizes generic types if passed as the value of toklist parameter
+ # https://github.com/pyparsing/pyparsing/issues/276?notification_referrer_id=MDE4Ok5vdGlmaWNhdGlvblRocmVhZDE4MzU4NDYwNzI6MzgzODc1
+ #
+ # behavior of ParseResults code changed with Python 3.9
+
+ results_with_int = pp.ParseResults(toklist=int, name="type_", asList=False)
+ self.assertEqual(int, results_with_int["type_"])
+
+ results_with_tuple = pp.ParseResults(toklist=tuple, name="type_", asList=False)
+ self.assertEqual(tuple, results_with_tuple["type_"])
+
+ def testParseResultsReturningDunderAttribute(self):
+ # from Issue #208
+ parser = pp.Word(pp.alphas)("A")
+ result = parser.parseString("abc")
+ print(result.dump())
+ self.assertEqual("abc", result.A)
+ self.assertEqual("", result.B)
+ with self.assertRaises(AttributeError):
+ result.__xyz__
+
+ def testMatchOnlyAtCol(self):
+ """successfully use matchOnlyAtCol helper function"""
+
+ expr = pp.Word(pp.nums)
+ expr.setParseAction(pp.matchOnlyAtCol(5))
+ largerExpr = pp.ZeroOrMore(pp.Word("A")) + expr + pp.ZeroOrMore(pp.Word("A"))
+
+ res = largerExpr.parseString("A A 3 A")
+ print(res.dump())
+
+ def testMatchOnlyAtColErr(self):
+ """raise a ParseException in matchOnlyAtCol with incorrect col"""
+
+ expr = pp.Word(pp.nums)
+ expr.setParseAction(pp.matchOnlyAtCol(1))
+ largerExpr = pp.ZeroOrMore(pp.Word("A")) + expr + pp.ZeroOrMore(pp.Word("A"))
+
+ with self.assertRaisesParseException():
+ largerExpr.parseString("A A 3 A")
+
+ def testParseResultsWithNamedTuple(self):
+ expr = pp.Literal("A")("Achar")
+ expr.setParseAction(pp.replaceWith(tuple(["A", "Z"])))
+
+ res = expr.parseString("A")
+ print(repr(res))
+ print(res.Achar)
+ self.assertParseResultsEquals(
+ res,
+ expected_dict={"Achar": ("A", "Z")},
+ msg="Failed accessing named results containing a tuple, "
+ "got {!r}".format(res.Achar),
+ )
+
+ def testParserElementAddOperatorWithOtherTypes(self):
+ """test the overridden "+" operator with other data types"""
+
+ # ParserElement + str
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second") + "suf"
+ result = expr.parseString("spam eggs suf")
+ print(result)
+
+ expected_l = ["spam", "eggs", "suf"]
+ self.assertParseResultsEquals(
+ result, expected_l, msg="issue with ParserElement + str"
+ )
+
+ # str + ParserElement
+ expr = "pre" + pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second")
+ result = expr.parseString("pre spam eggs")
+ print(result)
+
+ expected_l = ["pre", "spam", "eggs"]
+ self.assertParseResultsEquals(
+ result, expected_l, msg="issue with str + ParserElement"
+ )
+
+ # ParserElement + int
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn ParserElement + int"):
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second") + 12
+ self.assertEqual(expr, None)
+
+ # int + ParserElement
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn int + ParserElement"):
+ expr = 12 + pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second")
+ self.assertEqual(expr, None)
+
+ def testParserElementSubOperatorWithOtherTypes(self):
+ """test the overridden "-" operator with other data types"""
+
+ # ParserElement - str
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second") - "suf"
+ result = expr.parseString("spam eggs suf")
+ print(result)
+ expected = ["spam", "eggs", "suf"]
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with ParserElement - str"
+ )
+
+ # str - ParserElement
+ expr = "pre" - pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second")
+ result = expr.parseString("pre spam eggs")
+ print(result)
+ expected = ["pre", "spam", "eggs"]
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with str - ParserElement"
+ )
+
+ # ParserElement - int
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn ParserElement - int"):
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second") - 12
+ self.assertEqual(expr, None)
+
+ # int - ParserElement
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn int - ParserElement"):
+ expr = 12 - pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second")
+ self.assertEqual(expr, None)
+
+ def testParserElementMulOperatorWithTuples(self):
+ """test ParserElement "*" with various tuples"""
+
+ # ParserElement * (None, n)
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.nums)("second*") * (None, 3)
+
+ results1 = expr.parseString("spam")
+ print(results1.dump())
+ expected = ["spam"]
+ self.assertParseResultsEquals(
+ results1, expected, msg="issue with ParserElement * w/ optional matches"
+ )
+
+ results2 = expr.parseString("spam 12 23 34")
+ print(results2.dump())
+ expected = ["spam", "12", "23", "34"]
+ self.assertParseResultsEquals(
+ results2, expected, msg="issue with ParserElement * w/ optional matches"
+ )
+
+ # ParserElement * (1, 1)
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.nums)("second*") * (1, 1)
+ results = expr.parseString("spam 45")
+ print(results.dump())
+
+ expected = ["spam", "45"]
+ self.assertParseResultsEquals(
+ results, expected, msg="issue with ParserElement * (1, 1)"
+ )
+
+ # ParserElement * (1, 1+n)
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.nums)("second*") * (1, 3)
+
+ results1 = expr.parseString("spam 100")
+ print(results1.dump())
+ expected = ["spam", "100"]
+ self.assertParseResultsEquals(
+ results1, expected, msg="issue with ParserElement * (1, 1+n)"
+ )
+
+ results2 = expr.parseString("spam 100 200 300")
+ print(results2.dump())
+ expected = ["spam", "100", "200", "300"]
+ self.assertParseResultsEquals(
+ results2, expected, msg="issue with ParserElement * (1, 1+n)"
+ )
+
+ # ParserElement * (lesser, greater)
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.nums)("second*") * (2, 3)
+
+ results1 = expr.parseString("spam 1 2")
+ print(results1.dump())
+ expected = ["spam", "1", "2"]
+ self.assertParseResultsEquals(
+ results1, expected, msg="issue with ParserElement * (lesser, greater)"
+ )
+
+ results2 = expr.parseString("spam 1 2 3")
+ print(results2.dump())
+ expected = ["spam", "1", "2", "3"]
+ self.assertParseResultsEquals(
+ results2, expected, msg="issue with ParserElement * (lesser, greater)"
+ )
+
+ # ParserElement * (greater, lesser)
+ with self.assertRaises(
+ ValueError, msg="ParserElement * (greater, lesser) should raise error"
+ ):
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.nums)("second") * (3, 2)
+
+ # ParserElement * (str, str)
+ with self.assertRaises(
+ TypeError, msg="ParserElement * (str, str) should raise error"
+ ):
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.nums)("second") * ("2", "3")
+
+ def testParserElementMulByZero(self):
+ alpwd = pp.Word(pp.alphas)
+ numwd = pp.Word(pp.nums)
+
+ test_string = "abd def ghi jkl"
+
+ parser = alpwd * 2 + numwd * 0 + alpwd * 2
+ self.assertParseAndCheckList(
+ parser, test_string, expected_list=test_string.split()
+ )
+
+ parser = alpwd * 2 + numwd * (0, 0) + alpwd * 2
+ self.assertParseAndCheckList(
+ parser, test_string, expected_list=test_string.split()
+ )
+
+ def testParserElementMulOperatorWithOtherTypes(self):
+ """test the overridden "*" operator with other data types"""
+
+ # ParserElement * str
+ with self.assertRaises(TypeError, msg="ParserElement * str should raise error"):
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.nums)("second") * "3"
+
+ # str * ParserElement
+ with self.assertRaises(TypeError, msg="str * ParserElement should raise error"):
+ expr = pp.Word(pp.alphas)("first") + "3" * pp.Word(pp.nums)("second")
+
+ # ParserElement * int
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.nums)("second*") * 2
+ results = expr.parseString("spam 11 22")
+
+ print(results.dump())
+ expected = ["spam", "11", "22"]
+ self.assertParseResultsEquals(
+ results, expected, msg="issue with ParserElement * int"
+ )
+
+ # int * ParserElement
+ expr = pp.Word(pp.alphas)("first") + 2 * pp.Word(pp.nums)("second*")
+ results = expr.parseString("spam 111 222")
+
+ print(results.dump())
+ expected = ["spam", "111", "222"]
+ self.assertParseResultsEquals(
+ results, expected, msg="issue with int * ParserElement"
+ )
+
+ def testParserElementMatchFirstOperatorWithOtherTypes(self):
+ """test the overridden "|" operator with other data types"""
+
+ # ParserElement | int
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn ParserElement | int"):
+ expr = pp.Word(pp.alphas)("first") + (pp.Word(pp.alphas)("second") | 12)
+ self.assertEqual(expr, None)
+
+ # int | ParserElement
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn int | ParserElement"):
+ expr = pp.Word(pp.alphas)("first") + (12 | pp.Word(pp.alphas)("second"))
+ self.assertEqual(expr, None)
+
+ def testParserElementMatchLongestWithOtherTypes(self):
+ """test the overridden "^" operator with other data types"""
+
+ # ParserElement ^ str
+ expr = pp.Word(pp.alphas)("first") + (pp.Word(pp.nums)("second") ^ "eggs")
+ result = expr.parseString("spam eggs")
+ print(result)
+
+ expected = ["spam", "eggs"]
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with ParserElement ^ str"
+ )
+
+ # str ^ ParserElement
+ expr = ("pre" ^ pp.Word("pr")("first")) + pp.Word(pp.alphas)("second")
+ result = expr.parseString("pre eggs")
+ print(result)
+
+ expected = ["pre", "eggs"]
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with str ^ ParserElement"
+ )
+
+ # ParserElement ^ int
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn ParserElement ^ int"):
+ expr = pp.Word(pp.alphas)("first") + (pp.Word(pp.alphas)("second") ^ 54)
+ self.assertEqual(expr, None)
+
+ # int ^ ParserElement
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn int ^ ParserElement"):
+ expr = pp.Word(pp.alphas)("first") + (65 ^ pp.Word(pp.alphas)("second"))
+ self.assertEqual(expr, None)
+
+ def testParserElementEachOperatorWithOtherTypes(self):
+ """test the overridden "&" operator with other data types"""
+
+ # ParserElement & str
+ expr = pp.Word(pp.alphas)("first") + (pp.Word(pp.alphas)("second") & "and")
+ with self.assertRaisesParseException(msg="issue with ParserElement & str"):
+ result = expr.parseString("spam and eggs")
+
+ # str & ParserElement
+ expr = pp.Word(pp.alphas)("first") + ("and" & pp.Word(pp.alphas)("second"))
+ result = expr.parseString("spam and eggs")
+
+ print(result.dump())
+ expected_l = ["spam", "and", "eggs"]
+ expected_d = {"first": "spam", "second": "eggs"}
+ self.assertParseResultsEquals(
+ result,
+ expected_list=expected_l,
+ expected_dict=expected_d,
+ msg="issue with str & ParserElement",
+ )
+
+ # ParserElement & int
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn ParserElement & int"):
+ expr = pp.Word(pp.alphas)("first") + (pp.Word(pp.alphas) & 78)
+ self.assertEqual(expr, None)
+
+ # int & ParserElement
+ expr = None
+ with self.assertRaises(TypeError, msg="failed to warn int & ParserElement"):
+ expr = pp.Word(pp.alphas)("first") + (89 & pp.Word(pp.alphas))
+ self.assertEqual(expr, None)
+
+ def testParserElementPassedThreeArgsToMultiplierShorthand(self):
+ """test the ParserElement form expr[m,n,o]"""
+
+ with self.assertRaises(
+ TypeError, msg="failed to warn three index arguments to expr[m, n, o]"
+ ):
+ expr = pp.Word(pp.alphas)[2, 3, 4]
+
+ def testParserElementPassedStrToMultiplierShorthand(self):
+ """test the ParserElement form expr[str]"""
+
+ with self.assertRaises(
+ TypeError, msg="failed to raise expected error using string multiplier"
+ ):
+ expr2 = pp.Word(pp.alphas)["2"]
+
+ def testParseResultsNewEdgeCases(self):
+ """test less common paths of ParseResults.__new__()"""
+
+ parser = pp.Word(pp.alphas)[...]
+ result = parser.parseString("sldkjf sldkjf")
+
+ # hasattr uses __getattr__, which for ParseResults will return "" if the
+ # results name is not defined. So hasattr() won't work with ParseResults.
+ # Have to use __contains__ instead to test for existence.
+ # self.assertFalse(hasattr(result, "A"))
+ self.assertFalse("A" in result)
+
+ # create new ParseResults w/ None
+ result1 = pp.ParseResults(None)
+ print(result1.dump())
+ self.assertParseResultsEquals(
+ result1, [], msg="ParseResults(None) should return empty ParseResults"
+ )
+
+ # create new ParseResults w/ integer name
+ result2 = pp.ParseResults(name=12)
+ print(result2.dump())
+ self.assertEqual(
+ "12",
+ result2.getName(),
+ "ParseResults int name should be accepted and converted to str",
+ )
+
+ # create new ParseResults w/ generator type
+ gen = (a for a in range(1, 6))
+ result3 = pp.ParseResults(gen)
+ print(result3.dump())
+ expected3 = [1, 2, 3, 4, 5]
+ self.assertParseResultsEquals(
+ result3, expected3, msg="issue initializing ParseResults w/ gen type"
+ )
+
+ def testParseResultsReversed(self):
+ """test simple case of reversed(ParseResults)"""
+
+ tst = "1 2 3 4 5"
+ expr = pp.OneOrMore(pp.Word(pp.nums))
+ result = expr.parseString(tst)
+
+ reversed_list = [ii for ii in reversed(result)]
+ print(reversed_list)
+ expected = ["5", "4", "3", "2", "1"]
+ self.assertEqual(
+ expected, reversed_list, msg="issue calling reversed(ParseResults)"
+ )
+
+ def testParseResultsValues(self):
+ """test simple case of ParseResults.values()"""
+
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second")
+ result = expr.parseString("spam eggs")
+
+ values_set = set(result.values())
+ print(values_set)
+ expected = {"spam", "eggs"}
+ self.assertEqual(
+ expected, values_set, msg="issue calling ParseResults.values()"
+ )
+
+ def testParseResultsAppend(self):
+ """test simple case of ParseResults.append()"""
+
+ # use a parse action to compute the sum of the parsed integers, and add it to the end
+ def append_sum(tokens):
+ tokens.append(sum(map(int, tokens)))
+
+ expr = pp.OneOrMore(pp.Word(pp.nums)).addParseAction(append_sum)
+ result = expr.parseString("0 123 321")
+
+ expected = ["0", "123", "321", 444]
+ print(result.dump())
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with ParseResults.append()"
+ )
+
+ def testParseResultsClear(self):
+ """test simple case of ParseResults.clear()"""
+
+ tst = "spam eggs"
+ expr = pp.Word(pp.alphas)("first") + pp.Word(pp.alphas)("second")
+ result = expr.parseString(tst)
+
+ print(result.dump())
+ self.assertParseResultsEquals(
+ result, ["spam", "eggs"], msg="issue with ParseResults before clear()"
+ )
+
+ result.clear()
+
+ print(result.dump())
+ self.assertParseResultsEquals(
+ result,
+ expected_list=[],
+ expected_dict={},
+ msg="issue with ParseResults.clear()",
+ )
+
+ def testParseResultsExtendWithString(self):
+ """test ParseResults.extend() with input of type str"""
+
+ # use a parse action to append the reverse of the matched strings to make a palindrome
+ def make_palindrome(tokens):
+ tokens.extend(reversed([t[::-1] for t in tokens]))
+
+ tst = "abc def ghi"
+ expr = pp.OneOrMore(pp.Word(pp.alphas))
+ result = expr.addParseAction(make_palindrome).parseString(tst)
+ print(result.dump())
+
+ expected = ["abc", "def", "ghi", "ihg", "fed", "cba"]
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with ParseResults.extend(str)"
+ )
+
+ def testParseResultsExtendWithParseResults(self):
+ """test ParseResults.extend() with input of type ParseResults"""
+
+ expr = pp.OneOrMore(pp.Word(pp.alphas))
+ result1 = expr.parseString("spam eggs")
+ result2 = expr.parseString("foo bar")
+
+ result1.extend(result2)
+ print(result1.dump())
+ expected = ["spam", "eggs", "foo", "bar"]
+ self.assertParseResultsEquals(
+ result1, expected, msg="issue with ParseResults.extend(ParseResults)"
+ )
+
+ def testParseResultsFromDict(self):
+ """test helper classmethod ParseResults.from_dict()"""
+
+ dict = {
+ "first": "123",
+ "second": 456,
+ "third": {"threeStr": "789", "threeInt": 789},
+ }
+ name = "trios"
+ result = pp.ParseResults.from_dict(dict, name=name)
+
+ print(result.dump())
+ expected = {name: dict}
+ self.assertParseResultsEquals(
+ result,
+ expected_dict=expected,
+ msg="issue creating ParseResults.from _dict()",
+ )
+
+ def testParseResultsDir(self):
+ """test dir(ParseResults)"""
+
+ dict = {"first": "123", "second": "456", "third": "789"}
+ name = "trios"
+ result = pp.ParseResults.from_dict(dict, name=name)
+ dir_result = dir(result)
+
+ print(dir_result)
+ self.assertIn(
+ name, dir_result, msg="name value wasn't returned by dir(ParseResults)"
+ )
+ self.assertIn(
+ "asList", dir_result, msg="asList was not returned by dir(ParseResults)"
+ )
+
+ def testParseResultsInsert(self):
+ """test ParseResults.insert() with named tokens"""
+
+ from random import randint
+
+ result = pp.Word(pp.alphas)[...].parseString("A B C D E F G H I J")
+ compare_list = result.asList()
+
+ print(result)
+ print(compare_list)
+
+ for s in "abcdefghij":
+ index = randint(-5, 5)
+ result.insert(index, s)
+ compare_list.insert(index, s)
+
+ print(result)
+ print(compare_list)
+
+ self.assertParseResultsEquals(
+ result, compare_list, msg="issue with ParseResults.insert()"
+ )
+
+ def testIgnoreString(self):
+ """test ParserElement.ignore() passed a string arg"""
+
+ tst = "I like totally like love pickles"
+ expr = pp.Word(pp.alphas)[...].ignore("like")
+ result = expr.parseString(tst)
+
+ print(result)
+ expected = ["I", "totally", "love", "pickles"]
+ self.assertParseResultsEquals(result, expected, msg="issue with ignore(string)")
+
+ def testParseHTMLTags(self):
+ test = """
+ <BODY>
+ <BODY BGCOLOR="#00FFCC">
+ <BODY BGCOLOR="#00FFAA"/>
+ <BODY BGCOLOR='#00FFBB' FGCOLOR=black>
+ <BODY/>
+ </BODY>
+ """
+ results = [
+ ("startBody", False, "", ""),
+ ("startBody", False, "#00FFCC", ""),
+ ("startBody", True, "#00FFAA", ""),
+ ("startBody", False, "#00FFBB", "black"),
+ ("startBody", True, "", ""),
+ ("endBody", False, "", ""),
+ ]
+
+ bodyStart, bodyEnd = pp.makeHTMLTags("BODY")
+ resIter = iter(results)
+ for t, s, e in (bodyStart | bodyEnd).scanString(test):
+ print(test[s:e], "->", t)
+ (expectedType, expectedEmpty, expectedBG, expectedFG) = next(resIter)
+
+ print(t.dump())
+ if "startBody" in t:
+ self.assertEqual(
+ expectedEmpty,
+ bool(t.empty),
+ "expected {} token, got {}".format(
+ expectedEmpty and "empty" or "not empty",
+ t.empty and "empty" or "not empty",
+ ),
+ )
+ self.assertEqual(
+ expectedBG,
+ t.bgcolor,
+ "failed to match BGCOLOR, expected {}, got {}".format(
+ expectedBG, t.bgcolor
+ ),
+ )
+ self.assertEqual(
+ expectedFG,
+ t.fgcolor,
+ "failed to match FGCOLOR, expected {}, got {}".format(
+ expectedFG, t.bgcolor
+ ),
+ )
+ elif "endBody" in t:
+ print("end tag")
+ pass
+ else:
+ print("BAD!!!")
+
+ def testSetParseActionUncallableErr(self):
+ """raise a TypeError in setParseAction() by adding uncallable arg"""
+
+ expr = pp.Literal("A")("Achar")
+ uncallable = 12
+
+ with self.assertRaises(TypeError):
+ expr.setParseAction(uncallable)
+
+ res = expr.parseString("A")
+ print(res.dump())
+
+ def testMulWithNegativeNumber(self):
+ """raise a ValueError in __mul__ by multiplying a negative number"""
+
+ with self.assertRaises(ValueError):
+ pp.Literal("A")("Achar") * (-1)
+
+ def testMulWithEllipsis(self):
+ """multiply an expression with Ellipsis as ``expr * ...`` to match ZeroOrMore"""
+
+ expr = pp.Literal("A")("Achar") * ...
+ res = expr.parseString("A")
+ self.assertEqual(["A"], res.asList(), "expected expr * ... to match ZeroOrMore")
+ print(res.dump())
+
+ def testUpcaseDowncaseUnicode(self):
+ import sys
+
+ ppu = pp.pyparsing_unicode
+
+ a = "\u00bfC\u00f3mo esta usted?"
+ if not JYTHON_ENV:
+ ualphas = ppu.alphas
+ else:
+ ualphas = "".join(
+ chr(i)
+ for i in list(range(0xD800)) + list(range(0xE000, sys.maxunicode))
+ if chr(i).isalpha()
+ )
+ uword = pp.Word(ualphas).setParseAction(ppc.upcaseTokens)
+
+ print = lambda *args: None
+ print(uword.searchString(a))
+
+ uword = pp.Word(ualphas).setParseAction(ppc.downcaseTokens)
+
+ print(uword.searchString(a))
+
+ kw = pp.Keyword("mykey", caseless=True).setParseAction(ppc.upcaseTokens)(
+ "rname"
+ )
+ ret = kw.parseString("mykey")
+ print(ret.rname)
+ self.assertEqual(
+ "MYKEY", ret.rname, "failed to upcase with named result (pyparsing_common)"
+ )
+
+ kw = pp.Keyword("MYKEY", caseless=True).setParseAction(ppc.downcaseTokens)(
+ "rname"
+ )
+ ret = kw.parseString("mykey")
+ print(ret.rname)
+ self.assertEqual("mykey", ret.rname, "failed to upcase with named result")
+
+ if not IRON_PYTHON_ENV:
+ # test html data
+ html = "<TR class=maintxt bgColor=#ffffff> \
+ <TD vAlign=top>Производитель, модель</TD> \
+ <TD vAlign=top><STRONG>BenQ-Siemens CF61</STRONG></TD> \
+ " # .decode('utf-8')
+
+ # 'Manufacturer, model
+ text_manuf = "Производитель, модель"
+ manufacturer = pp.Literal(text_manuf)
+
+ td_start, td_end = pp.makeHTMLTags("td")
+ manuf_body = (
+ td_start.suppress()
+ + manufacturer
+ + pp.SkipTo(td_end)("cells*")
+ + td_end.suppress()
+ )
+
+ def testParseUsingRegex(self):
+
+ signedInt = pp.Regex(r"[-+][0-9]+")
+ unsignedInt = pp.Regex(r"[0-9]+")
+ simpleString = pp.Regex(r'("[^\"]*")|(\'[^\']*\')')
+ namedGrouping = pp.Regex(r'("(?P<content>[^\"]*)")')
+ compiledRE = pp.Regex(re.compile(r"[A-Z]+"))
+
+ def testMatch(expression, instring, shouldPass, expectedString=None):
+ if shouldPass:
+ try:
+ result = expression.parseString(instring)
+ print(
+ "{} correctly matched {}".format(
+ repr(expression), repr(instring)
+ )
+ )
+ if expectedString != result[0]:
+ print("\tbut failed to match the pattern as expected:")
+ print(
+ "\tproduced %s instead of %s"
+ % (repr(result[0]), repr(expectedString))
+ )
+ return True
+ except pp.ParseException:
+ print(
+ "%s incorrectly failed to match %s"
+ % (repr(expression), repr(instring))
+ )
+ else:
+ try:
+ result = expression.parseString(instring)
+ print(
+ "{} incorrectly matched {}".format(
+ repr(expression), repr(instring)
+ )
+ )
+ print("\tproduced %s as a result" % repr(result[0]))
+ except pp.ParseException:
+ print(
+ "%s correctly failed to match %s"
+ % (repr(expression), repr(instring))
+ )
+ return True
+ return False
+
+ # These should fail
+ self.assertTrue(
+ testMatch(signedInt, "1234 foo", False), "Re: (1) passed, expected fail"
+ )
+ self.assertTrue(
+ testMatch(signedInt, " +foo", False), "Re: (2) passed, expected fail"
+ )
+ self.assertTrue(
+ testMatch(unsignedInt, "abc", False), "Re: (3) passed, expected fail"
+ )
+ self.assertTrue(
+ testMatch(unsignedInt, "+123 foo", False), "Re: (4) passed, expected fail"
+ )
+ self.assertTrue(
+ testMatch(simpleString, "foo", False), "Re: (5) passed, expected fail"
+ )
+ self.assertTrue(
+ testMatch(simpleString, "\"foo bar'", False),
+ "Re: (6) passed, expected fail",
+ )
+ self.assertTrue(
+ testMatch(simpleString, "'foo bar\"", False),
+ "Re: (7) passed, expected fail",
+ )
+
+ # These should pass
+ self.assertTrue(
+ testMatch(signedInt, " +123", True, "+123"),
+ "Re: (8) failed, expected pass",
+ )
+ self.assertTrue(
+ testMatch(signedInt, "+123", True, "+123"), "Re: (9) failed, expected pass"
+ )
+ self.assertTrue(
+ testMatch(signedInt, "+123 foo", True, "+123"),
+ "Re: (10) failed, expected pass",
+ )
+ self.assertTrue(
+ testMatch(signedInt, "-0 foo", True, "-0"), "Re: (11) failed, expected pass"
+ )
+ self.assertTrue(
+ testMatch(unsignedInt, "123 foo", True, "123"),
+ "Re: (12) failed, expected pass",
+ )
+ self.assertTrue(
+ testMatch(unsignedInt, "0 foo", True, "0"), "Re: (13) failed, expected pass"
+ )
+ self.assertTrue(
+ testMatch(simpleString, '"foo"', True, '"foo"'),
+ "Re: (14) failed, expected pass",
+ )
+ self.assertTrue(
+ testMatch(simpleString, "'foo bar' baz", True, "'foo bar'"),
+ "Re: (15) failed, expected pass",
+ )
+
+ self.assertTrue(
+ testMatch(compiledRE, "blah", False), "Re: (16) passed, expected fail"
+ )
+ self.assertTrue(
+ testMatch(compiledRE, "BLAH", True, "BLAH"),
+ "Re: (17) failed, expected pass",
+ )
+
+ self.assertTrue(
+ testMatch(namedGrouping, '"foo bar" baz', True, '"foo bar"'),
+ "Re: (16) failed, expected pass",
+ )
+ ret = namedGrouping.parseString('"zork" blah')
+ print(ret)
+ print(list(ret.items()))
+ print(ret.content)
+ self.assertEqual("zork", ret.content, "named group lookup failed")
+ self.assertEqual(
+ simpleString.parseString('"zork" blah')[0],
+ ret[0],
+ "Regex not properly returning ParseResults for named vs. unnamed groups",
+ )
+
+ try:
+ print("lets try an invalid RE")
+ invRe = pp.Regex("(\"[^\"]*\")|('[^']*'")
+ except Exception as e:
+ print("successfully rejected an invalid RE:", end=" ")
+ print(e)
+ else:
+ self.fail("failed to reject invalid RE")
+
+ with self.assertRaises(
+ ValueError, msg="failed to warn empty string passed to Regex"
+ ):
+ invRe = pp.Regex("")
+
+ def testRegexAsType(self):
+
+ test_str = "sldkjfj 123 456 lsdfkj"
+
+ print("return as list of match groups")
+ expr = pp.Regex(r"\w+ (\d+) (\d+) (\w+)", asGroupList=True)
+ expected_group_list = [tuple(test_str.split()[1:])]
+ result = expr.parseString(test_str)
+ print(result.dump())
+ print(expected_group_list)
+ self.assertParseResultsEquals(
+ result,
+ expected_list=expected_group_list,
+ msg="incorrect group list returned by Regex)",
+ )
+
+ print("return as re.match instance")
+ expr = pp.Regex(
+ r"\w+ (?P<num1>\d+) (?P<num2>\d+) (?P<last_word>\w+)", asMatch=True
+ )
+ result = expr.parseString(test_str)
+ print(result.dump())
+ print(result[0].groups())
+ print(expected_group_list)
+ self.assertEqual(
+ {"num1": "123", "num2": "456", "last_word": "lsdfkj"},
+ result[0].groupdict(),
+ "invalid group dict from Regex(asMatch=True)",
+ )
+ self.assertEqual(
+ expected_group_list[0],
+ result[0].groups(),
+ "incorrect group list returned by Regex(asMatch)",
+ )
+
+ def testRegexSub(self):
+
+ print("test sub with string")
+ expr = pp.Regex(r"<title>").sub("'Richard III'")
+ result = expr.transformString("This is the title: <title>")
+ print(result)
+ self.assertEqual(
+ "This is the title: 'Richard III'",
+ result,
+ "incorrect Regex.sub result with simple string",
+ )
+
+ print("test sub with re string")
+ expr = pp.Regex(r"([Hh]\d):\s*(.*)").sub(r"<\1>\2</\1>")
+ result = expr.transformString(
+ "h1: This is the main heading\nh2: This is the sub-heading"
+ )
+ print(result)
+ self.assertEqual(
+ "<h1>This is the main heading</h1>\n<h2>This is the sub-heading</h2>",
+ result,
+ "incorrect Regex.sub result with re string",
+ )
+
+ print("test sub with re string (Regex returns re.match)")
+ expr = pp.Regex(r"([Hh]\d):\s*(.*)", asMatch=True).sub(r"<\1>\2</\1>")
+ result = expr.transformString(
+ "h1: This is the main heading\nh2: This is the sub-heading"
+ )
+ print(result)
+ self.assertEqual(
+ "<h1>This is the main heading</h1>\n<h2>This is the sub-heading</h2>",
+ result,
+ "incorrect Regex.sub result with re string",
+ )
+
+ print("test sub with callable that return str")
+ expr = pp.Regex(r"<(.*?)>").sub(lambda m: m.group(1).upper())
+ result = expr.transformString("I want this in upcase: <what? what?>")
+ print(result)
+ self.assertEqual(
+ "I want this in upcase: WHAT? WHAT?",
+ result,
+ "incorrect Regex.sub result with callable",
+ )
+
+ with self.assertRaises(TypeError):
+ pp.Regex(r"<(.*?)>", asMatch=True).sub(lambda m: m.group(1).upper())
+
+ with self.assertRaises(TypeError):
+ pp.Regex(r"<(.*?)>", asGroupList=True).sub(lambda m: m.group(1).upper())
+
+ with self.assertRaises(TypeError):
+ pp.Regex(r"<(.*?)>", asGroupList=True).sub("")
+
+ def testRegexInvalidType(self):
+ """test Regex of an invalid type"""
+
+ with self.assertRaises(TypeError, msg="issue with Regex of type int"):
+ expr = pp.Regex(12)
+
+ def testPrecededBy(self):
+
+ num = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
+ interesting_num = pp.PrecededBy(pp.Char("abc")("prefix*")) + num
+ semi_interesting_num = pp.PrecededBy("_") + num
+ crazy_num = pp.PrecededBy(pp.Word("^", "$%^")("prefix*"), 10) + num
+ boring_num = ~pp.PrecededBy(pp.Char("abc_$%^" + pp.nums)) + num
+ very_boring_num = pp.PrecededBy(pp.WordStart()) + num
+ finicky_num = pp.PrecededBy(pp.Word("^", "$%^"), retreat=3) + num
+
+ s = "c384 b8324 _9293874 _293 404 $%^$^%$2939"
+ print(s)
+ for expr, expected_list, expected_dict in [
+ (interesting_num, [384, 8324], {"prefix": ["c", "b"]}),
+ (semi_interesting_num, [9293874, 293], {}),
+ (boring_num, [404], {}),
+ (crazy_num, [2939], {"prefix": ["^%$"]}),
+ (finicky_num, [2939], {}),
+ (very_boring_num, [404], {}),
+ ]:
+ # print(expr.searchString(s))
+ result = sum(expr.searchString(s))
+ print(result.dump())
+ self.assertParseResultsEquals(result, expected_list, expected_dict)
+
+ # infinite loop test - from Issue #127
+ string_test = "notworking"
+ # negs = pp.Or(['not', 'un'])('negs')
+ negs_pb = pp.PrecededBy("not", retreat=100)("negs_lb")
+ # negs_pb = pp.PrecededBy(negs, retreat=100)('negs_lb')
+ pattern = (negs_pb + pp.Literal("working"))("main")
+
+ results = pattern.searchString(string_test)
+ try:
+ print(results.dump())
+ except RecursionError:
+ self.fail("got maximum excursion limit exception")
+ else:
+ print("got maximum excursion limit exception")
+
+ def testCountedArray(self):
+
+ testString = "2 5 7 6 0 1 2 3 4 5 0 3 5 4 3"
+
+ integer = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
+ countedField = pp.countedArray(integer)
+
+ r = pp.OneOrMore(pp.Group(countedField)).parseString(testString)
+ print(testString)
+ print(r)
+
+ self.assertParseResultsEquals(
+ r, expected_list=[[5, 7], [0, 1, 2, 3, 4, 5], [], [5, 4, 3]]
+ )
+
+ # addresses bug raised by Ralf Vosseler
+ def testCountedArrayTest2(self):
+
+ testString = "2 5 7 6 0 1 2 3 4 5 0 3 5 4 3"
+
+ integer = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
+ countedField = pp.countedArray(integer)
+
+ dummy = pp.Word("A")
+ r = pp.OneOrMore(pp.Group(dummy ^ countedField)).parseString(testString)
+ print(testString)
+ print(r)
+
+ self.assertParseResultsEquals(
+ r, expected_list=[[5, 7], [0, 1, 2, 3, 4, 5], [], [5, 4, 3]]
+ )
+
+ def testCountedArrayTest3(self):
+
+ int_chars = "_" + pp.alphas
+ array_counter = pp.Word(int_chars).setParseAction(
+ lambda t: int_chars.index(t[0])
+ )
+
+ # 123456789012345678901234567890
+ testString = "B 5 7 F 0 1 2 3 4 5 _ C 5 4 3"
+
+ integer = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
+ countedField = pp.countedArray(integer, intExpr=array_counter)
+
+ r = pp.OneOrMore(pp.Group(countedField)).parseString(testString)
+ print(testString)
+ print(r)
+
+ self.assertParseResultsEquals(
+ r, expected_list=[[5, 7], [0, 1, 2, 3, 4, 5], [], [5, 4, 3]]
+ )
+
+ def testCountedArrayTest4(self):
+ ppc = pp.pyparsing_common
+
+ # array counter contains several fields - first field *must* be the number of
+ # items in the array
+ # - number of elements
+ # - type of elements
+ # - source of elements
+ counter_with_metadata = (
+ ppc.integer("count") + ppc.identifier("type") + ppc.identifier("source")
+ )
+
+ countedField = pp.countedArray(
+ pp.Word(pp.alphanums), intExpr=counter_with_metadata
+ )
+
+ testString = (
+ "5 string input item1 item2 item3 item4 item5 0 int user 2 int file 3 8"
+ )
+ r = pp.Group(countedField("items"))[...].parseString(testString, parseAll=True)
+
+ print(testString)
+ print(r.dump())
+ print("type = {!r}".format(r.type))
+ print("source = {!r}".format(r.source))
+
+ self.assertParseResultsEquals(
+ r,
+ expected_list=[
+ ["item1", "item2", "item3", "item4", "item5"],
+ [],
+ ["3", "8"],
+ ],
+ )
+
+ self.assertParseResultsEquals(
+ r[0],
+ expected_dict={
+ "count": 5,
+ "source": "input",
+ "type": "string",
+ "items": ["item1", "item2", "item3", "item4", "item5"],
+ },
+ )
+
+ # parse with additional fields between the count and the actual list items
+ count_with_metadata = ppc.integer + pp.Word(pp.alphas)("type")
+ typed_array = pp.countedArray(
+ pp.Word(pp.alphanums), intExpr=count_with_metadata
+ )("items")
+ result = typed_array.parseString("3 bool True True False")
+ print(result.dump())
+
+ self.assertParseResultsEquals(
+ result,
+ expected_list=["True", "True", "False"],
+ expected_dict={"type": "bool", "items": ["True", "True", "False"]},
+ )
+
+ def testLineStart(self):
+
+ pass_tests = [
+ """\
+ AAA
+ BBB
+ """,
+ """\
+ AAA...
+ BBB
+ """,
+ ]
+ fail_tests = [
+ """\
+ AAA...
+ ...BBB
+ """,
+ """\
+ AAA BBB
+ """,
+ ]
+
+ # cleanup test strings
+ pass_tests = [
+ "\n".join(s.lstrip() for s in t.splitlines()).replace(".", " ")
+ for t in pass_tests
+ ]
+ fail_tests = [
+ "\n".join(s.lstrip() for s in t.splitlines()).replace(".", " ")
+ for t in fail_tests
+ ]
+
+ test_patt = pp.Word("A") - pp.LineStart() + pp.Word("B")
+ print(test_patt.streamline())
+ success = test_patt.runTests(pass_tests)[0]
+ self.assertTrue(success, "failed LineStart passing tests (1)")
+
+ success = test_patt.runTests(fail_tests, failureTests=True)[0]
+ self.assertTrue(success, "failed LineStart failure mode tests (1)")
+
+ with ppt.reset_pyparsing_context():
+ print(r"no \n in default whitespace chars")
+ pp.ParserElement.setDefaultWhitespaceChars(" ")
+
+ test_patt = pp.Word("A") - pp.LineStart() + pp.Word("B")
+ print(test_patt.streamline())
+ # should fail the pass tests too, since \n is no longer valid whitespace and we aren't parsing for it
+ success = test_patt.runTests(pass_tests, failureTests=True)[0]
+ self.assertTrue(success, "failed LineStart passing tests (2)")
+
+ success = test_patt.runTests(fail_tests, failureTests=True)[0]
+ self.assertTrue(success, "failed LineStart failure mode tests (2)")
+
+ test_patt = (
+ pp.Word("A")
+ - pp.LineEnd().suppress()
+ + pp.LineStart()
+ + pp.Word("B")
+ + pp.LineEnd().suppress()
+ )
+ print(test_patt.streamline())
+ success = test_patt.runTests(pass_tests)[0]
+ self.assertTrue(success, "failed LineStart passing tests (3)")
+
+ success = test_patt.runTests(fail_tests, failureTests=True)[0]
+ self.assertTrue(success, "failed LineStart failure mode tests (3)")
+
+ def testLineStart2(self):
+
+ test = """\
+ AAA 1
+ AAA 2
+
+ AAA
+
+ B AAA
+
+ """
+
+ test = dedent(test)
+ print(test)
+
+ print("normal parsing")
+ for t, s, e in (pp.LineStart() + "AAA").scanString(test):
+ print(s, e, pp.lineno(s, test), pp.line(s, test), repr(test[s]))
+ print()
+ self.assertEqual(
+ "A", test[s], "failed LineStart with insignificant newlines"
+ )
+
+ print(r"parsing without \n in whitespace chars")
+ with ppt.reset_pyparsing_context():
+ pp.ParserElement.setDefaultWhitespaceChars(" ")
+ for t, s, e in (pp.LineStart() + "AAA").scanString(test):
+ print(s, e, pp.lineno(s, test), pp.line(s, test), repr(test[s]))
+ print()
+ self.assertEqual(
+ "A", test[s], "failed LineStart with insignificant newlines"
+ )
+
+ def testLineStart3(self):
+ # testing issue #272
+ instring = dedent(
+ """
+ a
+ b
+ c
+ d
+ e
+ f
+ g
+ """
+ )
+ print(pp.testing.with_line_numbers(instring))
+
+ alpha_line = (
+ pp.LineStart().leaveWhitespace()
+ + pp.Word(pp.alphas)
+ + pp.LineEnd().suppress()
+ )
+
+ tests = [
+ alpha_line,
+ pp.Group(alpha_line),
+ alpha_line | pp.Word("_"),
+ alpha_line | alpha_line,
+ pp.MatchFirst([alpha_line, alpha_line]),
+ pp.LineStart() + pp.Word(pp.alphas) + pp.LineEnd().suppress(),
+ pp.And([pp.LineStart(), pp.Word(pp.alphas), pp.LineEnd().suppress()]),
+ ]
+ for test in tests:
+ print(test.searchString(instring))
+ self.assertEqual(
+ ["a", "d", "e"], flatten(sum(test.search_string(instring)).as_list())
+ )
+
+ def testLineStart4(self):
+ test = dedent(
+ """\
+ AAA this line
+ AAA and this line
+ AAA but not this one
+ B AAA and definitely not this one
+ """
+ )
+
+ expr = pp.AtLineStart("AAA") + pp.restOfLine
+ for t in expr.search_string(test):
+ print(t)
+
+ self.assertEqual(
+ ["AAA", " this line", "AAA", " and this line"],
+ sum(expr.search_string(test)).as_list(),
+ )
+
+ def testStringStart(self):
+ self.assertParseAndCheckList(pp.AtStringStart(pp.Word(pp.nums)), "123", ["123"])
+
+ self.assertParseAndCheckList(pp.AtStringStart("123"), "123", ["123"])
+
+ with self.assertRaisesParseException():
+ pp.AtStringStart(pp.Word(pp.nums)).parse_string(" 123")
+
+ with self.assertRaisesParseException():
+ pp.AtStringStart("123").parse_string(" 123")
+
+ def testLineAndStringEnd(self):
+
+ NLs = pp.OneOrMore(pp.lineEnd)
+ bnf1 = pp.delimitedList(pp.Word(pp.alphanums).leaveWhitespace(), NLs)
+ bnf2 = pp.Word(pp.alphanums) + pp.stringEnd
+ bnf3 = pp.Word(pp.alphanums) + pp.SkipTo(pp.stringEnd)
+ tests = [
+ ("testA\ntestB\ntestC\n", ["testA", "testB", "testC"]),
+ ("testD\ntestE\ntestF", ["testD", "testE", "testF"]),
+ ("a", ["a"]),
+ ]
+
+ for test, expected in tests:
+ res1 = bnf1.parseString(test)
+ print(res1, "=?", expected)
+ self.assertParseResultsEquals(
+ res1,
+ expected_list=expected,
+ msg="Failed lineEnd/stringEnd test (1): "
+ + repr(test)
+ + " -> "
+ + str(res1),
+ )
+
+ res2 = bnf2.searchString(test)[0]
+ print(res2, "=?", expected[-1:])
+ self.assertParseResultsEquals(
+ res2,
+ expected_list=expected[-1:],
+ msg="Failed lineEnd/stringEnd test (2): "
+ + repr(test)
+ + " -> "
+ + str(res2),
+ )
+
+ res3 = bnf3.parseString(test)
+ first = res3[0]
+ rest = res3[1]
+ # ~ print res3.dump()
+ print(repr(rest), "=?", repr(test[len(first) + 1 :]))
+ self.assertEqual(
+ rest,
+ test[len(first) + 1 :],
+ "Failed lineEnd/stringEnd test (3): "
+ + repr(test)
+ + " -> "
+ + str(res3.asList()),
+ )
+ print()
+
+ k = pp.Regex(r"a+", flags=re.S + re.M)
+ k = k.parseWithTabs()
+ k = k.leaveWhitespace()
+
+ tests = [
+ (r"aaa", ["aaa"]),
+ (r"\naaa", None),
+ (r"a\naa", None),
+ (r"aaa\n", None),
+ ]
+ for i, (src, expected) in enumerate(tests):
+ print(i, repr(src).replace("\\\\", "\\"), end=" ")
+ if expected is None:
+ with self.assertRaisesParseException():
+ k.parseString(src, parseAll=True)
+ else:
+ res = k.parseString(src, parseAll=True)
+ self.assertParseResultsEquals(
+ res, expected, msg="Failed on parseAll=True test %d" % i
+ )
+
+ def testVariableParseActionArgs(self):
+ pa3 = lambda s, l, t: t
+ pa2 = lambda l, t: t
+ pa1 = lambda t: t
+ pa0 = lambda: None
+
+ class Callable3:
+ def __call__(self, s, l, t):
+ return t
+
+ class Callable2:
+ def __call__(self, l, t):
+ return t
+
+ class Callable1:
+ def __call__(self, t):
+ return t
+
+ class Callable0:
+ def __call__(self):
+ return
+
+ class CallableS3:
+ @staticmethod
+ def __call__(s, l, t):
+ return t
+
+ class CallableS2:
+ @staticmethod
+ def __call__(l, t):
+ return t
+
+ class CallableS1:
+ @staticmethod
+ def __call__(t):
+ return t
+
+ class CallableS0:
+ @staticmethod
+ def __call__():
+ return
+
+ class CallableC3:
+ @classmethod
+ def __call__(cls, s, l, t):
+ return t
+
+ class CallableC2:
+ @classmethod
+ def __call__(cls, l, t):
+ return t
+
+ class CallableC1:
+ @classmethod
+ def __call__(cls, t):
+ return t
+
+ class CallableC0:
+ @classmethod
+ def __call__(cls):
+ return
+
+ class parseActionHolder:
+ @staticmethod
+ def pa3(s, l, t):
+ return t
+
+ @staticmethod
+ def pa2(l, t):
+ return t
+
+ @staticmethod
+ def pa1(t):
+ return t
+
+ @staticmethod
+ def pa0():
+ return
+
+ def paArgs(*args):
+ print(args)
+ return args[2]
+
+ class ClassAsPA0:
+ def __init__(self):
+ pass
+
+ def __str__(self):
+ return "A"
+
+ class ClassAsPA1:
+ def __init__(self, t):
+ print("making a ClassAsPA1")
+ self.t = t
+
+ def __str__(self):
+ return self.t[0]
+
+ class ClassAsPA2:
+ def __init__(self, l, t):
+ self.t = t
+
+ def __str__(self):
+ return self.t[0]
+
+ class ClassAsPA3:
+ def __init__(self, s, l, t):
+ self.t = t
+
+ def __str__(self):
+ return self.t[0]
+
+ class ClassAsPAStarNew(tuple):
+ def __new__(cls, *args):
+ print("make a ClassAsPAStarNew", args)
+ return tuple.__new__(cls, *args[2].asList())
+
+ def __str__(self):
+ return "".join(self)
+
+ A = pp.Literal("A").setParseAction(pa0)
+ B = pp.Literal("B").setParseAction(pa1)
+ C = pp.Literal("C").setParseAction(pa2)
+ D = pp.Literal("D").setParseAction(pa3)
+ E = pp.Literal("E").setParseAction(Callable0())
+ F = pp.Literal("F").setParseAction(Callable1())
+ G = pp.Literal("G").setParseAction(Callable2())
+ H = pp.Literal("H").setParseAction(Callable3())
+ I = pp.Literal("I").setParseAction(CallableS0())
+ J = pp.Literal("J").setParseAction(CallableS1())
+ K = pp.Literal("K").setParseAction(CallableS2())
+ L = pp.Literal("L").setParseAction(CallableS3())
+ M = pp.Literal("M").setParseAction(CallableC0())
+ N = pp.Literal("N").setParseAction(CallableC1())
+ O = pp.Literal("O").setParseAction(CallableC2())
+ P = pp.Literal("P").setParseAction(CallableC3())
+ Q = pp.Literal("Q").setParseAction(paArgs)
+ R = pp.Literal("R").setParseAction(parseActionHolder.pa3)
+ S = pp.Literal("S").setParseAction(parseActionHolder.pa2)
+ T = pp.Literal("T").setParseAction(parseActionHolder.pa1)
+ U = pp.Literal("U").setParseAction(parseActionHolder.pa0)
+ V = pp.Literal("V")
+
+ gg = pp.OneOrMore(
+ A
+ | B
+ | C
+ | D
+ | E
+ | F
+ | G
+ | H
+ | I
+ | J
+ | K
+ | L
+ | M
+ | N
+ | O
+ | P
+ | Q
+ | R
+ | S
+ | U
+ | V
+ | B
+ | T
+ )
+ testString = "VUTSRQPONMLKJIHGFEDCBA"
+ res = gg.parseString(testString)
+ print(res)
+ self.assertParseResultsEquals(
+ res,
+ expected_list=list(testString),
+ msg="Failed to parse using variable length parse actions",
+ )
+
+ A = pp.Literal("A").setParseAction(ClassAsPA0)
+ B = pp.Literal("B").setParseAction(ClassAsPA1)
+ C = pp.Literal("C").setParseAction(ClassAsPA2)
+ D = pp.Literal("D").setParseAction(ClassAsPA3)
+ E = pp.Literal("E").setParseAction(ClassAsPAStarNew)
+
+ gg = pp.OneOrMore(
+ A
+ | B
+ | C
+ | D
+ | E
+ | F
+ | G
+ | H
+ | I
+ | J
+ | K
+ | L
+ | M
+ | N
+ | O
+ | P
+ | Q
+ | R
+ | S
+ | T
+ | U
+ | V
+ )
+ testString = "VUTSRQPONMLKJIHGFEDCBA"
+ res = gg.parseString(testString)
+ print(list(map(str, res)))
+ self.assertEqual(
+ list(testString),
+ list(map(str, res)),
+ "Failed to parse using variable length parse actions "
+ "using class constructors as parse actions",
+ )
+
+ def testSingleArgException(self):
+
+ testMessage = "just one arg"
+ try:
+ raise pp.ParseFatalException(testMessage)
+ except pp.ParseBaseException as pbe:
+ print("Received expected exception:", pbe)
+ raisedMsg = pbe.msg
+ self.assertEqual(
+ testMessage, raisedMsg, "Failed to get correct exception message"
+ )
+
+ def testOriginalTextFor(self):
+ def rfn(t):
+ return "%s:%d" % (t.src, len("".join(t)))
+
+ makeHTMLStartTag = lambda tag: pp.originalTextFor(
+ pp.makeHTMLTags(tag)[0], asString=False
+ )
+
+ # use the lambda, Luke
+ start = makeHTMLStartTag("IMG")
+
+ # don't replace our fancy parse action with rfn,
+ # append rfn to the list of parse actions
+ start.addParseAction(rfn)
+
+ text = """_<img src="images/cal.png"
+ alt="cal image" width="16" height="15">_"""
+ s = start.transformString(text)
+ print(s)
+ self.assertTrue(
+ s.startswith("_images/cal.png:"), "failed to preserve input s properly"
+ )
+ self.assertTrue(
+ s.endswith("77_"), "failed to return full original text properly"
+ )
+
+ tag_fields = makeHTMLStartTag("IMG").searchString(text)[0]
+ print(sorted(tag_fields.keys()))
+ self.assertEqual(
+ ["alt", "empty", "height", "src", "startImg", "tag", "width"],
+ sorted(tag_fields.keys()),
+ "failed to preserve results names in originalTextFor",
+ )
+
+ def testPackratParsingCacheCopy(self):
+
+ integer = pp.Word(pp.nums).setName("integer")
+ id = pp.Word(pp.alphas + "_", pp.alphanums + "_")
+ simpleType = pp.Literal("int")
+ arrayType = simpleType + ("[" + pp.delimitedList(integer) + "]")[...]
+ varType = arrayType | simpleType
+ varDec = varType + pp.delimitedList(id + pp.Optional("=" + integer)) + ";"
+
+ codeBlock = pp.Literal("{}")
+
+ funcDef = (
+ pp.Optional(varType | "void")
+ + id
+ + "("
+ + (pp.delimitedList(varType + id) | "void" | pp.empty)
+ + ")"
+ + codeBlock
+ )
+
+ program = varDec | funcDef
+ input = "int f(){}"
+ self.assertParseAndCheckList(
+ program,
+ input,
+ ["int", "f", "(", ")", "{}"],
+ msg="Error in packrat parsing",
+ verbose=True,
+ )
+
+ def testPackratParsingCacheCopyTest2(self):
+
+ DO, AA = list(map(pp.Keyword, "DO AA".split()))
+ LPAR, RPAR = list(map(pp.Suppress, "()"))
+ identifier = ~AA + pp.Word("Z")
+
+ function_name = identifier.copy()
+ # ~ function_name = ~AA + Word("Z") #identifier.copy()
+ expr = pp.Forward().setName("expr")
+ expr <<= pp.Group(
+ function_name + LPAR + pp.Optional(pp.delimitedList(expr)) + RPAR
+ ).setName("functionCall") | identifier.setName(
+ "ident"
+ ) # .setDebug()#.setBreak()
+
+ stmt = DO + pp.Group(pp.delimitedList(identifier + ".*" | expr))
+ result = stmt.parseString("DO Z")
+ print(result.asList())
+ self.assertEqual(
+ 1, len(result[1]), "packrat parsing is duplicating And term exprs"
+ )
+
+ def testParseResultsDel(self):
+ grammar = pp.OneOrMore(pp.Word(pp.nums))("ints") + pp.OneOrMore(
+ pp.Word(pp.alphas)
+ )("words")
+ res = grammar.parseString("123 456 ABC DEF")
+ print(res.dump())
+ origInts = res.ints.asList()
+ origWords = res.words.asList()
+ del res[1]
+ del res["words"]
+ print(res.dump())
+ self.assertEqual("ABC", res[1], "failed to delete 0'th element correctly")
+ self.assertEqual(
+ origInts,
+ res.ints.asList(),
+ "updated named attributes, should have updated list only",
+ )
+ self.assertEqual("", res.words, "failed to update named attribute correctly")
+ self.assertEqual(
+ "DEF", res[-1], "updated list, should have updated named attributes only"
+ )
+
+ def testWithAttributeParseAction(self):
+ """
+ This unit test checks withAttribute in these ways:
+
+ * Argument forms as keywords and tuples
+ * Selecting matching tags by attribute
+ * Case-insensitive attribute matching
+ * Correctly matching tags having the attribute, and rejecting tags not having the attribute
+
+ (Unit test written by voigts as part of the Google Highly Open Participation Contest)
+ """
+
+ data = """
+ <a>1</a>
+ <a b="x">2</a>
+ <a B="x">3</a>
+ <a b="X">4</a>
+ <a b="y">5</a>
+ <a class="boo">8</ a>
+ """
+ tagStart, tagEnd = pp.makeHTMLTags("a")
+
+ expr = tagStart + pp.Word(pp.nums)("value") + tagEnd
+
+ expected = (
+ [
+ ["a", ["b", "x"], False, "2", "</a>"],
+ ["a", ["b", "x"], False, "3", "</a>"],
+ ],
+ [
+ ["a", ["b", "x"], False, "2", "</a>"],
+ ["a", ["b", "x"], False, "3", "</a>"],
+ ],
+ [["a", ["class", "boo"], False, "8", "</a>"]],
+ )
+
+ for attrib, exp in zip(
+ [
+ pp.withAttribute(b="x"),
+ # withAttribute(B="x"),
+ pp.withAttribute(("b", "x")),
+ # withAttribute(("B", "x")),
+ pp.withClass("boo"),
+ ],
+ expected,
+ ):
+
+ tagStart.setParseAction(attrib)
+ result = expr.searchString(data)
+
+ print(result.dump())
+ self.assertParseResultsEquals(
+ result,
+ expected_list=exp,
+ msg="Failed test, expected {}, got {}".format(
+ expected, result.asList()
+ ),
+ )
+
+ def testNestedExpressions(self):
+ """
+ This unit test checks nestedExpr in these ways:
+ - use of default arguments
+ - use of non-default arguments (such as a pyparsing-defined comment
+ expression in place of quotedString)
+ - use of a custom content expression
+ - use of a pyparsing expression for opener and closer is *OPTIONAL*
+ - use of input data containing nesting delimiters
+ - correct grouping of parsed tokens according to nesting of opening
+ and closing delimiters in the input string
+
+ (Unit test written by christoph... as part of the Google Highly Open Participation Contest)
+ """
+
+ # All defaults. Straight out of the example script. Also, qualifies for
+ # the bonus: note the fact that (Z | (E^F) & D) is not parsed :-).
+ # Tests for bug fixed in 1.4.10
+ print("Test defaults:")
+ teststring = "((ax + by)*C) (Z | (E^F) & D)"
+
+ expr = pp.nestedExpr()
+
+ expected = [[["ax", "+", "by"], "*C"]]
+ result = expr.parseString(teststring)
+ print(result.dump())
+ self.assertParseResultsEquals(
+ result,
+ expected_list=expected,
+ msg="Defaults didn't work. That's a bad sign. Expected: {}, got: {}".format(
+ expected, result
+ ),
+ )
+
+ # Going through non-defaults, one by one; trying to think of anything
+ # odd that might not be properly handled.
+
+ # Change opener
+ print("\nNon-default opener")
+ teststring = "[[ ax + by)*C)"
+ expected = [[["ax", "+", "by"], "*C"]]
+ expr = pp.nestedExpr("[")
+ self.assertParseAndCheckList(
+ expr,
+ teststring,
+ expected,
+ "Non-default opener didn't work. Expected: {}, got: {}".format(
+ expected, result
+ ),
+ verbose=True,
+ )
+
+ # Change closer
+ print("\nNon-default closer")
+
+ teststring = "((ax + by]*C]"
+ expected = [[["ax", "+", "by"], "*C"]]
+ expr = pp.nestedExpr(closer="]")
+ self.assertParseAndCheckList(
+ expr,
+ teststring,
+ expected,
+ "Non-default closer didn't work. Expected: {}, got: {}".format(
+ expected, result
+ ),
+ verbose=True,
+ )
+
+ # #Multicharacter opener, closer
+ # opener = "bar"
+ # closer = "baz"
+ print("\nLiteral expressions for opener and closer")
+
+ opener, closer = map(pp.Literal, "bar baz".split())
+ expr = pp.nestedExpr(
+ opener, closer, content=pp.Regex(r"([^b ]|b(?!a)|ba(?![rz]))+")
+ )
+
+ teststring = "barbar ax + bybaz*Cbaz"
+ expected = [[["ax", "+", "by"], "*C"]]
+ self.assertParseAndCheckList(
+ expr,
+ teststring,
+ expected,
+ "Multicharacter opener and closer didn't work. Expected: {}, got: {}".format(
+ expected, result
+ ),
+ verbose=True,
+ )
+
+ # Lisp-ish comments
+ print("\nUse ignore expression (1)")
+ comment = pp.Regex(r";;.*")
+ teststring = """
+ (let ((greeting "Hello, world!")) ;;(foo bar
+ (display greeting))
+ """
+
+ expected = [
+ [
+ "let",
+ [["greeting", '"Hello,', 'world!"']],
+ ";;(foo bar",
+ ["display", "greeting"],
+ ]
+ ]
+ expr = pp.nestedExpr(ignoreExpr=comment)
+ self.assertParseAndCheckList(
+ expr,
+ teststring,
+ expected,
+ 'Lisp-ish comments (";; <...> $") didn\'t work. Expected: {}, got: {}'.format(
+ expected, result
+ ),
+ verbose=True,
+ )
+
+ # Lisp-ish comments, using a standard bit of pyparsing, and an Or.
+ print("\nUse ignore expression (2)")
+ comment = ";;" + pp.restOfLine
+
+ teststring = """
+ (let ((greeting "Hello, )world!")) ;;(foo bar
+ (display greeting))
+ """
+
+ expected = [
+ [
+ "let",
+ [["greeting", '"Hello, )world!"']],
+ ";;",
+ "(foo bar",
+ ["display", "greeting"],
+ ]
+ ]
+ expr = pp.nestedExpr(ignoreExpr=(comment ^ pp.quotedString))
+ self.assertParseAndCheckList(
+ expr,
+ teststring,
+ expected,
+ 'Lisp-ish comments (";; <...> $") and quoted strings didn\'t work. Expected: {}, got: {}'.format(
+ expected, result
+ ),
+ verbose=True,
+ )
+
+ def testNestedExpressions2(self):
+ """test nestedExpr with conditions that explore other paths
+
+ identical opener and closer
+ opener and/or closer of type other than string or iterable
+ multi-character opener and/or closer
+ single character opener and closer with ignoreExpr=None
+ multi-character opener and/or closer with ignoreExpr=None
+ """
+
+ name = pp.Word(pp.alphanums + "_")
+
+ # identical opener and closer
+ with self.assertRaises(
+ ValueError, msg="matching opener and closer should raise error"
+ ):
+ expr = name + pp.nestedExpr(opener="{", closer="{")
+
+ # opener and/or closer of type other than string or iterable
+ with self.assertRaises(
+ ValueError, msg="opener and closer as ints should raise error"
+ ):
+ expr = name + pp.nestedExpr(opener=12, closer=18)
+
+ # multi-character opener and/or closer
+ tstMulti = "aName {{ outer {{ 'inner with opener {{ and closer }} in quoted string' }} }}"
+ expr = name + pp.nestedExpr(opener="{{", closer="}}")
+ result = expr.parseString(tstMulti)
+ expected = [
+ "aName",
+ ["outer", ["'inner with opener {{ and closer }} in quoted string'"]],
+ ]
+ print(result.dump())
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with multi-character opener and closer"
+ )
+
+ # single character opener and closer with ignoreExpr=None
+ tst = (
+ "aName { outer { 'inner with opener { and closer } in quoted string' }} }}"
+ )
+ expr = name + pp.nestedExpr(opener="{", closer="}", ignoreExpr=None)
+ singleCharResult = expr.parseString(tst)
+ print(singleCharResult.dump())
+
+ # multi-character opener and/or closer with ignoreExpr=None
+ expr = name + pp.nestedExpr(opener="{{", closer="}}", ignoreExpr=None)
+ multiCharResult = expr.parseString(tstMulti)
+ print(multiCharResult.dump())
+
+ self.assertParseResultsEquals(
+ singleCharResult,
+ multiCharResult.asList(),
+ msg="using different openers and closers shouldn't affect resulting ParseResults",
+ )
+
+ def testWordExclude(self):
+
+ allButPunc = pp.Word(pp.printables, excludeChars=".,:;-_!?")
+
+ test = "Hello, Mr. Ed, it's Wilbur!"
+ result = allButPunc.searchString(test).asList()
+ print(result)
+ self.assertEqual(
+ [["Hello"], ["Mr"], ["Ed"], ["it's"], ["Wilbur"]],
+ result,
+ "failed WordExcludeTest",
+ )
+
+ def testWordMinOfZero(self):
+ """test a Word with min=0"""
+
+ with self.assertRaises(ValueError, msg="expected min 0 to error"):
+ expr = pp.Word(pp.nums, min=0, max=10)
+
+ @staticmethod
+ def setup_testWordMaxGreaterThanZeroAndAsKeyword():
+ # fmt: off
+ bool_operand = (
+ pp.Word(pp.alphas, max=1, asKeyword=True)
+ | pp.one_of("True False")
+ )
+ test_string = "p q r False"
+ return SimpleNamespace(**locals())
+ # fmt: on
+
+ def testWordMaxGreaterThanZeroAndAsKeyword1(self):
+ """test a Word with max>0 and asKeyword=True"""
+ setup = self.setup_testWordMaxGreaterThanZeroAndAsKeyword()
+
+ result = setup.bool_operand[...].parseString(setup.test_string)
+ self.assertParseAndCheckList(
+ setup.bool_operand[...],
+ setup.test_string,
+ setup.test_string.split(),
+ msg=__() + "Failed to parse Word(max=1, asKeyword=True)",
+ verbose=True,
+ )
+
+ def testWordMaxGreaterThanZeroAndAsKeyword2(self):
+ """test a Word with max>0 and asKeyword=True"""
+ setup = self.setup_testWordMaxGreaterThanZeroAndAsKeyword()
+
+ with self.assertRaisesParseException(
+ msg=__() + "failed to detect Word with max > 0 and asKeyword=True"
+ ):
+ setup.bool_operand.parseString("abc")
+
+ def testCharAsKeyword(self):
+ """test a Char with asKeyword=True"""
+
+ grade = pp.OneOrMore(pp.Char("ABCDF", asKeyword=True))
+
+ # all single char words
+ result = grade.parseString("B B C A D")
+
+ print(result)
+ expected = ["B", "B", "C", "A", "D"]
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with Char asKeyword=True"
+ )
+
+ # NOT all single char words
+ test2 = "B BB C A D"
+ result2 = grade.parseString(test2)
+
+ print(result2)
+ expected2 = ["B"]
+ self.assertParseResultsEquals(
+ result2, expected2, msg="issue with Char asKeyword=True parsing 2 chars"
+ )
+
+ def testCharsNotIn(self):
+ """test CharsNotIn initialized with various arguments"""
+
+ vowels = "AEIOU"
+ tst = "bcdfghjklmnpqrstvwxyz"
+
+ # default args
+ consonants = pp.CharsNotIn(vowels)
+ result = consonants.parseString(tst)
+ print(result)
+ self.assertParseResultsEquals(
+ result, [tst], msg="issue with CharsNotIn w/ default args"
+ )
+
+ # min = 0
+ with self.assertRaises(ValueError, msg="issue with CharsNotIn w/ min=0"):
+ consonants = pp.CharsNotIn(vowels, min=0)
+
+ # max > 0
+ consonants = pp.CharsNotIn(vowels, max=5)
+ result = consonants.parseString(tst)
+ print(result)
+ self.assertParseResultsEquals(
+ result, [tst[:5]], msg="issue with CharsNotIn w max > 0"
+ )
+
+ # exact > 0
+ consonants = pp.CharsNotIn(vowels, exact=10)
+ result = consonants.parseString(tst[:10])
+ print(result)
+ self.assertParseResultsEquals(
+ result, [tst[:10]], msg="issue with CharsNotIn w/ exact > 0"
+ )
+
+ # min > length
+ consonants = pp.CharsNotIn(vowels, min=25)
+ with self.assertRaisesParseException(msg="issue with CharsNotIn min > tokens"):
+ result = consonants.parseString(tst)
+
+ def testParseAll(self):
+
+ testExpr = pp.Word("A")
+
+ tests = [
+ ("AAAAA", False, True),
+ ("AAAAA", True, True),
+ ("AAABB", False, True),
+ ("AAABB", True, False),
+ ]
+ for s, parseAllFlag, shouldSucceed in tests:
+ try:
+ print(
+ "'{}' parseAll={} (shouldSucceed={})".format(
+ s, parseAllFlag, shouldSucceed
+ )
+ )
+ testExpr.parseString(s, parseAll=parseAllFlag)
+ self.assertTrue(
+ shouldSucceed, "successfully parsed when should have failed"
+ )
+ except ParseException as pe:
+ print(pe.explain())
+ self.assertFalse(
+ shouldSucceed, "failed to parse when should have succeeded"
+ )
+
+ # add test for trailing comments
+ testExpr.ignore(pp.cppStyleComment)
+
+ tests = [
+ ("AAAAA //blah", False, True),
+ ("AAAAA //blah", True, True),
+ ("AAABB //blah", False, True),
+ ("AAABB //blah", True, False),
+ ]
+ for s, parseAllFlag, shouldSucceed in tests:
+ try:
+ print(
+ "'{}' parseAll={} (shouldSucceed={})".format(
+ s, parseAllFlag, shouldSucceed
+ )
+ )
+ testExpr.parseString(s, parseAll=parseAllFlag)
+ self.assertTrue(
+ shouldSucceed, "successfully parsed when should have failed"
+ )
+ except ParseException as pe:
+ print(pe.explain())
+ self.assertFalse(
+ shouldSucceed, "failed to parse when should have succeeded"
+ )
+
+ # add test with very long expression string
+ # testExpr = pp.MatchFirst([pp.Literal(c) for c in pp.printables if c != 'B'])[1, ...]
+ anything_but_an_f = pp.OneOrMore(
+ pp.MatchFirst([pp.Literal(c) for c in pp.printables if c != "f"])
+ )
+ testExpr = pp.Word("012") + anything_but_an_f
+
+ tests = [
+ ("00aab", False, True),
+ ("00aab", True, True),
+ ("00aaf", False, True),
+ ("00aaf", True, False),
+ ]
+ for s, parseAllFlag, shouldSucceed in tests:
+ try:
+ print(
+ "'{}' parseAll={} (shouldSucceed={})".format(
+ s, parseAllFlag, shouldSucceed
+ )
+ )
+ testExpr.parseString(s, parseAll=parseAllFlag)
+ self.assertTrue(
+ shouldSucceed, "successfully parsed when should have failed"
+ )
+ except ParseException as pe:
+ print(pe.explain())
+ self.assertFalse(
+ shouldSucceed, "failed to parse when should have succeeded"
+ )
+
+ def testGreedyQuotedStrings(self):
+
+ src = """\
+ "string1", "strin""g2"
+ 'string1', 'string2'
+ ^string1^, ^string2^
+ <string1>, <string2>"""
+
+ testExprs = (
+ pp.sglQuotedString,
+ pp.dblQuotedString,
+ pp.quotedString,
+ pp.QuotedString('"', escQuote='""'),
+ pp.QuotedString("'", escQuote="''"),
+ pp.QuotedString("^"),
+ pp.QuotedString("<", endQuoteChar=">"),
+ )
+ for expr in testExprs:
+ strs = pp.delimitedList(expr).searchString(src)
+ print(strs)
+ self.assertTrue(
+ bool(strs), "no matches found for test expression '%s'" % expr
+ )
+ for lst in strs:
+ self.assertEqual(
+ 2, len(lst), "invalid match found for test expression '%s'" % expr
+ )
+
+ src = """'ms1',1,0,'2009-12-22','2009-12-22 10:41:22') ON DUPLICATE KEY UPDATE sent_count = sent_count + 1, mtime = '2009-12-22 10:41:22';"""
+ tok_sql_quoted_value = pp.QuotedString(
+ "'", "\\", "''", True, False
+ ) ^ pp.QuotedString('"', "\\", '""', True, False)
+ tok_sql_computed_value = pp.Word(pp.nums)
+ tok_sql_identifier = pp.Word(pp.alphas)
+
+ val = tok_sql_quoted_value | tok_sql_computed_value | tok_sql_identifier
+ vals = pp.delimitedList(val)
+ print(vals.parseString(src))
+ self.assertEqual(
+ 5, len(vals.parseString(src)), "error in greedy quote escaping"
+ )
+
+ def testQuotedStringEscapedQuotes(self):
+ quoted = pp.QuotedString('"', escQuote='""')
+ res = quoted.parseString('"like ""SQL"""')
+ print(res.asList())
+ self.assertEqual(['like "SQL"'], res.asList())
+
+ # Issue #263 - handle case when the escQuote is not a repeated character
+ quoted = pp.QuotedString("y", escChar=None, escQuote="xy")
+ res = quoted.parseString("yaaay")
+ self.assertEqual(["aaa"], res.asList())
+ res = quoted.parseString("yaaaxyaaay")
+ print(res.asList())
+ self.assertEqual(["aaayaaa"], res.asList())
+
+ def testWordBoundaryExpressions(self):
+
+ ws = pp.WordStart()
+ we = pp.WordEnd()
+ vowel = pp.oneOf(list("AEIOUY"))
+ consonant = pp.oneOf(list("BCDFGHJKLMNPQRSTVWXZ"))
+
+ leadingVowel = ws + vowel
+ trailingVowel = vowel + we
+ leadingConsonant = ws + consonant
+ trailingConsonant = consonant + we
+ internalVowel = ~ws + vowel + ~we
+
+ bnf = leadingVowel | trailingVowel
+
+ tests = """\
+ ABC DEF GHI
+ JKL MNO PQR
+ STU VWX YZ """.splitlines()
+ tests.append("\n".join(tests))
+
+ expectedResult = [
+ [["D", "G"], ["A"], ["C", "F"], ["I"], ["E"], ["A", "I"]],
+ [["J", "M", "P"], [], ["L", "R"], ["O"], [], ["O"]],
+ [["S", "V"], ["Y"], ["X", "Z"], ["U"], [], ["U", "Y"]],
+ [
+ ["D", "G", "J", "M", "P", "S", "V"],
+ ["A", "Y"],
+ ["C", "F", "L", "R", "X", "Z"],
+ ["I", "O", "U"],
+ ["E"],
+ ["A", "I", "O", "U", "Y"],
+ ],
+ ]
+
+ for t, expected in zip(tests, expectedResult):
+ print(t)
+ results = [
+ flatten(e.searchString(t).asList())
+ for e in [
+ leadingConsonant,
+ leadingVowel,
+ trailingConsonant,
+ trailingVowel,
+ internalVowel,
+ bnf,
+ ]
+ ]
+ print(results)
+ print()
+ self.assertEqual(
+ expected,
+ results,
+ "Failed WordBoundaryTest, expected {}, got {}".format(
+ expected, results
+ ),
+ )
+
+ def testRequiredEach(self):
+
+ parser = pp.Keyword("bam") & pp.Keyword("boo")
+ try:
+ res1 = parser.parseString("bam boo")
+ print(res1.asList())
+ res2 = parser.parseString("boo bam")
+ print(res2.asList())
+ except ParseException:
+ failed = True
+ else:
+ failed = False
+ self.assertFalse(failed, "invalid logic in Each")
+
+ self.assertEqual(
+ set(res1),
+ set(res2),
+ "Failed RequiredEachTest, expected "
+ + str(res1.asList())
+ + " and "
+ + str(res2.asList())
+ + "to contain same words in any order",
+ )
+
+ def testOptionalEachTest1(self):
+
+ for the_input in [
+ "Tal Weiss Major",
+ "Tal Major",
+ "Weiss Major",
+ "Major",
+ "Major Tal",
+ "Major Weiss",
+ "Major Tal Weiss",
+ ]:
+ print(the_input)
+ parser1 = (pp.Optional("Tal") + pp.Optional("Weiss")) & pp.Keyword("Major")
+ parser2 = pp.Optional(
+ pp.Optional("Tal") + pp.Optional("Weiss")
+ ) & pp.Keyword("Major")
+ p1res = parser1.parseString(the_input)
+ p2res = parser2.parseString(the_input)
+ self.assertEqual(
+ p1res.asList(),
+ p2res.asList(),
+ "Each failed to match with nested Optionals, "
+ + str(p1res.asList())
+ + " should match "
+ + str(p2res.asList()),
+ )
+
+ def testOptionalEachTest2(self):
+
+ word = pp.Word(pp.alphanums + "_").setName("word")
+ with_stmt = "with" + pp.OneOrMore(pp.Group(word("key") + "=" + word("value")))(
+ "overrides"
+ )
+ using_stmt = "using" + pp.Regex("id-[0-9a-f]{8}")("id")
+ modifiers = pp.Optional(with_stmt("with_stmt")) & pp.Optional(
+ using_stmt("using_stmt")
+ )
+
+ self.assertEqual("with foo=bar bing=baz using id-deadbeef", modifiers)
+ self.assertNotEqual(
+ "with foo=bar bing=baz using id-deadbeef using id-feedfeed", modifiers
+ )
+
+ def testOptionalEachTest3(self):
+
+ foo = pp.Literal("foo")
+ bar = pp.Literal("bar")
+
+ openBrace = pp.Suppress(pp.Literal("{"))
+ closeBrace = pp.Suppress(pp.Literal("}"))
+
+ exp = openBrace + (foo[1, ...]("foo") & bar[...]("bar")) + closeBrace
+
+ tests = """\
+ {foo}
+ {bar foo bar foo bar foo}
+ """.splitlines()
+ for test in tests:
+ test = test.strip()
+ if not test:
+ continue
+ self.assertParseAndCheckList(
+ exp,
+ test,
+ test.strip("{}").split(),
+ "failed to parse Each expression {!r}".format(test),
+ verbose=True,
+ )
+
+ with self.assertRaisesParseException():
+ exp.parseString("{bar}")
+
+ def testOptionalEachTest4(self):
+
+ expr = (~ppc.iso8601_date + ppc.integer("id")) & (
+ pp.Group(ppc.iso8601_date)("date*")[...]
+ )
+
+ expr.runTests(
+ """
+ 1999-12-31 100 2001-01-01
+ 42
+ """
+ )
+
+ def testEachWithParseFatalException(self):
+
+ option_expr = pp.Keyword("options") - "(" + ppc.integer + ")"
+ step_expr1 = pp.Keyword("step") - "(" + ppc.integer + ")"
+ step_expr2 = pp.Keyword("step") - "(" + ppc.integer + "Z" + ")"
+ step_expr = step_expr1 ^ step_expr2
+
+ parser = option_expr & step_expr[...]
+ tests = [
+ (
+ "options(100) step(A)",
+ "Expected integer, found 'A' (at char 18), (line:1, col:19)",
+ ),
+ (
+ "step(A) options(100)",
+ "Expected integer, found 'A' (at char 5), (line:1, col:6)",
+ ),
+ (
+ "options(100) step(100A)",
+ """Expected 'Z', found 'A' (at char 21), (line:1, col:22)""",
+ ),
+ (
+ "options(100) step(22) step(100ZA)",
+ """Expected ')', found 'A' (at char 31), (line:1, col:32)""",
+ ),
+ ]
+ test_lookup = dict(tests)
+
+ success, output = parser.runTests((t[0] for t in tests), failureTests=True)
+ for test_str, result in output:
+ self.assertEqual(
+ test_lookup[test_str],
+ str(result),
+ "incorrect exception raised for test string {!r}".format(test_str),
+ )
+
+ def testEachWithMultipleMatch(self):
+ size = "size" + pp.oneOf("S M L XL")
+ color = pp.Group(
+ "color" + pp.oneOf("red orange yellow green blue purple white black brown")
+ )
+ size.setName("size_spec")
+ color.setName("color_spec")
+
+ spec0 = size("size") & color[...]("colors")
+ spec1 = size("size") & color[1, ...]("colors")
+
+ for spec in (spec0, spec1):
+ for test, expected_dict in [
+ (
+ "size M color red color yellow",
+ {
+ "colors": [["color", "red"], ["color", "yellow"]],
+ "size": ["size", "M"],
+ },
+ ),
+ (
+ "color green size M color red color yellow",
+ {
+ "colors": [
+ ["color", "green"],
+ ["color", "red"],
+ ["color", "yellow"],
+ ],
+ "size": ["size", "M"],
+ },
+ ),
+ ]:
+ result = spec.parseString(test, parseAll=True)
+ self.assertParseResultsEquals(result, expected_dict=expected_dict)
+
+ def testSumParseResults(self):
+
+ samplestr1 = "garbage;DOB 10-10-2010;more garbage\nID PARI12345678;more garbage"
+ samplestr2 = "garbage;ID PARI12345678;more garbage\nDOB 10-10-2010;more garbage"
+ samplestr3 = "garbage;DOB 10-10-2010"
+ samplestr4 = "garbage;ID PARI12345678;more garbage- I am cool"
+
+ res1 = "ID:PARI12345678 DOB:10-10-2010 INFO:"
+ res2 = "ID:PARI12345678 DOB:10-10-2010 INFO:"
+ res3 = "ID: DOB:10-10-2010 INFO:"
+ res4 = "ID:PARI12345678 DOB: INFO: I am cool"
+
+ dob_ref = "DOB" + pp.Regex(r"\d{2}-\d{2}-\d{4}")("dob")
+ id_ref = "ID" + pp.Word(pp.alphanums, exact=12)("id")
+ info_ref = "-" + pp.restOfLine("info")
+
+ person_data = dob_ref | id_ref | info_ref
+
+ tests = (samplestr1, samplestr2, samplestr3, samplestr4)
+ results = (res1, res2, res3, res4)
+ for test, expected in zip(tests, results):
+ person = sum(person_data.searchString(test))
+ result = "ID:{} DOB:{} INFO:{}".format(person.id, person.dob, person.info)
+ print(test)
+ print(expected)
+ print(result)
+ for pd in person_data.searchString(test):
+ print(pd.dump())
+ print()
+ self.assertEqual(
+ expected,
+ result,
+ "Failed to parse '{}' correctly, \nexpected '{}', got '{}'".format(
+ test, expected, result
+ ),
+ )
+
+ def testMarkInputLine(self):
+
+ samplestr1 = "DOB 100-10-2010;more garbage\nID PARI12345678;more garbage"
+
+ dob_ref = "DOB" + pp.Regex(r"\d{2}-\d{2}-\d{4}")("dob")
+
+ try:
+ res = dob_ref.parseString(samplestr1)
+ except ParseException as pe:
+ outstr = pe.markInputline()
+ print(outstr)
+ self.assertEqual(
+ "DOB >!<100-10-2010;more garbage",
+ outstr,
+ "did not properly create marked input line",
+ )
+ else:
+ self.fail("test construction failed - should have raised an exception")
+
+ def testLocatedExpr(self):
+ # 012345678901234567890123456789012345678901234567890
+ samplestr1 = "DOB 10-10-2010;more garbage;ID PARI12345678 ;more garbage"
+
+ id_ref = pp.locatedExpr("ID" + pp.Word(pp.alphanums, exact=12)("id"))
+
+ res = id_ref.searchString(samplestr1)[0][0]
+ print(res.dump())
+ self.assertEqual(
+ "ID PARI12345678",
+ samplestr1[res.locn_start : res.locn_end],
+ "incorrect location calculation",
+ )
+
+ def testLocatedExprUsingLocated(self):
+ # 012345678901234567890123456789012345678901234567890
+ samplestr1 = "DOB 10-10-2010;more garbage;ID PARI12345678 ;more garbage"
+
+ id_ref = pp.Located("ID" + pp.Word(pp.alphanums, exact=12)("id"))
+
+ res = id_ref.searchString(samplestr1)[0]
+ print(res.dump())
+ self.assertEqual(
+ "ID PARI12345678",
+ samplestr1[res.locn_start : res.locn_end],
+ "incorrect location calculation",
+ )
+ self.assertParseResultsEquals(
+ res,
+ [28, ["ID", "PARI12345678"], 43],
+ {"locn_end": 43, "locn_start": 28, "value": {"id": "PARI12345678"}},
+ )
+ self.assertEqual("PARI12345678", res.value.id)
+
+ # if Located has a results name, handle appropriately
+ id_ref = pp.Located("ID" + pp.Word(pp.alphanums, exact=12)("id"))("loc")
+
+ res = id_ref.searchString(samplestr1)[0]
+ print(res.dump())
+ self.assertEqual(
+ "ID PARI12345678",
+ samplestr1[res.loc.locn_start : res.loc.locn_end],
+ "incorrect location calculation",
+ )
+ self.assertParseResultsEquals(
+ res.loc,
+ [28, ["ID", "PARI12345678"], 43],
+ {"locn_end": 43, "locn_start": 28, "value": {"id": "PARI12345678"}},
+ )
+ self.assertEqual("PARI12345678", res.loc.value.id)
+
+ wd = pp.Word(pp.alphas)
+ test_string = "ljsdf123lksdjjf123lkkjj1222"
+ pp_matches = pp.Located(wd).searchString(test_string)
+ re_matches = find_all_re_matches("[a-z]+", test_string)
+ for pp_match, re_match in zip(pp_matches, re_matches):
+ self.assertParseResultsEquals(
+ pp_match, [re_match.start(), [re_match.group(0)], re_match.end()]
+ )
+ print(pp_match)
+ print(re_match)
+ print(pp_match.value)
+
+ def testPop(self):
+ source = "AAA 123 456 789 234"
+ patt = pp.Word(pp.alphas)("name") + pp.Word(pp.nums) * (1,)
+
+ result = patt.parseString(source)
+ tests = [
+ (0, "AAA", ["123", "456", "789", "234"]),
+ (None, "234", ["123", "456", "789"]),
+ ("name", "AAA", ["123", "456", "789"]),
+ (-1, "789", ["123", "456"]),
+ ]
+ for test in tests:
+ idx, val, remaining = test
+ if idx is not None:
+ ret = result.pop(idx)
+ else:
+ ret = result.pop()
+ print("EXP:", val, remaining)
+ print("GOT:", ret, result.asList())
+ print(ret, result.asList())
+ self.assertEqual(
+ val,
+ ret,
+ "wrong value returned, got {!r}, expected {!r}".format(ret, val),
+ )
+ self.assertEqual(
+ remaining,
+ result.asList(),
+ "list is in wrong state after pop, got {!r}, expected {!r}".format(
+ result.asList(), remaining
+ ),
+ )
+ print()
+
+ prevlist = result.asList()
+ ret = result.pop("name", default="noname")
+ print(ret)
+ print(result.asList())
+ self.assertEqual(
+ "noname",
+ ret,
+ "default value not successfully returned, got {!r}, expected {!r}".format(
+ ret, "noname"
+ ),
+ )
+ self.assertEqual(
+ prevlist,
+ result.asList(),
+ "list is in wrong state after pop, got {!r}, expected {!r}".format(
+ result.asList(), remaining
+ ),
+ )
+
+ def testPopKwargsErr(self):
+ """raise a TypeError in pop by adding invalid named args"""
+
+ source = "AAA 123 456 789 234"
+ patt = pp.Word(pp.alphas)("name") + pp.Word(pp.nums) * (1,)
+ result = patt.parseString(source)
+ print(result.dump())
+
+ with self.assertRaises(TypeError):
+ result.pop(notDefault="foo")
+
+ def testAddCondition(self):
+
+ numParser = pp.Word(pp.nums)
+ numParser.addParseAction(lambda s, l, t: int(t[0]))
+ numParser.addCondition(lambda s, l, t: t[0] % 2)
+ numParser.addCondition(lambda s, l, t: t[0] >= 7)
+
+ result = numParser.searchString("1 2 3 4 5 6 7 8 9 10")
+ print(result.asList())
+ self.assertEqual(
+ [[7], [9]], result.asList(), "failed to properly process conditions"
+ )
+
+ numParser = pp.Word(pp.nums)
+ numParser.addParseAction(lambda s, l, t: int(t[0]))
+ rangeParser = numParser("from_") + pp.Suppress("-") + numParser("to")
+
+ result = rangeParser.searchString("1-4 2-4 4-3 5 6 7 8 9 10")
+ print(result.asList())
+ self.assertEqual(
+ [[1, 4], [2, 4], [4, 3]],
+ result.asList(),
+ "failed to properly process conditions",
+ )
+
+ rangeParser.addCondition(
+ lambda t: t.to > t.from_, message="from must be <= to", fatal=False
+ )
+ result = rangeParser.searchString("1-4 2-4 4-3 5 6 7 8 9 10")
+ print(result.asList())
+ self.assertEqual(
+ [[1, 4], [2, 4]], result.asList(), "failed to properly process conditions"
+ )
+
+ rangeParser = numParser("from_") + pp.Suppress("-") + numParser("to")
+ rangeParser.addCondition(
+ lambda t: t.to > t.from_, message="from must be <= to", fatal=True
+ )
+ try:
+ result = rangeParser.searchString("1-4 2-4 4-3 5 6 7 8 9 10")
+ self.fail("failed to interrupt parsing on fatal condition failure")
+ except ParseFatalException:
+ print("detected fatal condition")
+
+ def testPatientOr(self):
+ # Two expressions and a input string which could - syntactically - be matched against
+ # both expressions. The "Literal" expression is considered invalid though, so this PE
+ # should always detect the "Word" expression.
+ def validate(token):
+ if token[0] == "def":
+ raise pp.ParseException("signalling invalid token")
+ return token
+
+ a = pp.Word("de").setName("Word") # .setDebug()
+ b = pp.Literal("def").setName("Literal").setParseAction(validate) # .setDebug()
+ c = pp.Literal("d").setName("d") # .setDebug()
+
+ # The "Literal" expressions's ParseAction is not executed directly after syntactically
+ # detecting the "Literal" Expression but only after the Or-decision has been made
+ # (which is too late)...
+ try:
+ result = (a ^ b ^ c).parseString("def")
+ self.assertEqual(
+ ["de"],
+ result.asList(),
+ "failed to select longest match, chose %s" % result,
+ )
+ except ParseException:
+ failed = True
+ else:
+ failed = False
+
+ if failed:
+ self.fail(
+ "invalid logic in Or, fails on longest match with exception in parse action"
+ )
+
+ # from issue #93
+ word = pp.Word(pp.alphas).setName("word")
+ word_1 = (
+ pp.Word(pp.alphas).setName("word_1").addCondition(lambda t: len(t[0]) == 1)
+ )
+
+ a = word + (word_1 + word ^ word)
+ b = word * 3
+ c = a ^ b
+ c.streamline()
+ print(c)
+ test_string = "foo bar temp"
+ result = c.parseString(test_string)
+ print(test_string, "->", result.asList())
+
+ self.assertEqual(
+ test_string.split(), result.asList(), "failed to match longest choice"
+ )
+
+ def testEachWithOptionalWithResultsName(self):
+
+ result = (pp.Optional("foo")("one") & pp.Optional("bar")("two")).parseString(
+ "bar foo"
+ )
+ print(result.dump())
+ self.assertEqual(sorted(["one", "two"]), sorted(result.keys()))
+
+ def testUnicodeExpression(self):
+
+ z = "a" | pp.Literal("\u1111")
+ z.streamline()
+ try:
+ z.parseString("b")
+ except ParseException as pe:
+ self.assertEqual(
+ r"""Expected {'a' | 'ᄑ'}""",
+ pe.msg,
+ "Invalid error message raised, got %r" % pe.msg,
+ )
+
+ def testSetName(self):
+
+ a = pp.oneOf("a b c")
+ b = pp.oneOf("d e f")
+ arith_expr = pp.infixNotation(
+ pp.Word(pp.nums),
+ [
+ (pp.oneOf("* /"), 2, pp.opAssoc.LEFT),
+ (pp.oneOf("+ -"), 2, pp.opAssoc.LEFT),
+ ],
+ )
+ arith_expr2 = pp.infixNotation(
+ pp.Word(pp.nums), [(("?", ":"), 3, pp.opAssoc.LEFT)]
+ )
+ recursive = pp.Forward()
+ recursive <<= a + (b + recursive)[...]
+
+ tests = [
+ a,
+ b,
+ (a | b),
+ arith_expr,
+ arith_expr.expr,
+ arith_expr2,
+ arith_expr2.expr,
+ recursive,
+ pp.delimitedList(pp.Word(pp.nums).setName("int")),
+ pp.countedArray(pp.Word(pp.nums).setName("int")),
+ pp.nestedExpr(),
+ pp.makeHTMLTags("Z"),
+ (pp.anyOpenTag, pp.anyCloseTag),
+ pp.commonHTMLEntity,
+ pp.commonHTMLEntity.setParseAction(pp.replaceHTMLEntity).transformString(
+ "lsdjkf <lsdjkf>&'"&xyzzy;"
+ ),
+ ]
+
+ expected = map(
+ str.strip,
+ """\
+ a | b | c
+ d | e | f
+ {a | b | c | d | e | f}
+ Forward: + | - term
+ + | - term
+ Forward: ?: term
+ ?: term
+ Forward: {a | b | c [{d | e | f : ...}]...}
+ int [, int]...
+ (len) int...
+ nested () expression
+ (<Z>, </Z>)
+ (<any tag>, </any tag>)
+ common HTML entity
+ lsdjkf <lsdjkf>&'"&xyzzy;""".splitlines(),
+ )
+
+ for t, e in zip(tests, expected):
+ tname = str(t)
+ print(tname)
+ self.assertEqual(
+ e,
+ tname,
+ "expression name mismatch, expected {} got {}".format(e, tname),
+ )
+
+ def testTrimArityExceptionMasking(self):
+
+ invalid_message = "<lambda>() missing 1 required positional argument: 't'"
+ try:
+ pp.Word("a").setParseAction(lambda t: t[0] + 1).parseString("aaa")
+ except Exception as e:
+ exc_msg = str(e)
+ self.assertNotEqual(
+ exc_msg,
+ invalid_message,
+ "failed to catch TypeError thrown in _trim_arity",
+ )
+
+ def testTrimArityExceptionMaskingTest2(self):
+ # construct deep call tree
+ def A():
+ import traceback
+
+ traceback.print_stack(limit=2)
+
+ invalid_message = "<lambda>() missing 1 required positional argument: 't'"
+ try:
+ pp.Word("a").setParseAction(lambda t: t[0] + 1).parseString("aaa")
+ except Exception as e:
+ exc_msg = str(e)
+ self.assertNotEqual(
+ exc_msg,
+ invalid_message,
+ "failed to catch TypeError thrown in _trim_arity",
+ )
+
+ def B():
+ A()
+
+ def C():
+ B()
+
+ def D():
+ C()
+
+ def E():
+ D()
+
+ def F():
+ E()
+
+ def G():
+ F()
+
+ def H():
+ G()
+
+ def J():
+ H()
+
+ def K():
+ J()
+
+ K()
+
+ def testClearParseActions(self):
+
+ realnum = ppc.real()
+ self.assertEqual(
+ 3.14159,
+ realnum.parseString("3.14159")[0],
+ "failed basic real number parsing",
+ )
+
+ # clear parse action that converts to float
+ realnum.setParseAction(None)
+ self.assertEqual(
+ "3.14159", realnum.parseString("3.14159")[0], "failed clearing parse action"
+ )
+
+ # add a new parse action that tests if a '.' is prsent
+ realnum.addParseAction(lambda t: "." in t[0])
+ self.assertEqual(
+ True,
+ realnum.parseString("3.14159")[0],
+ "failed setting new parse action after clearing parse action",
+ )
+
+ def testOneOrMoreStop(self):
+
+ test = "BEGIN aaa bbb ccc END"
+ BEGIN, END = map(pp.Keyword, "BEGIN,END".split(","))
+ body_word = pp.Word(pp.alphas).setName("word")
+ for ender in (END, "END", pp.CaselessKeyword("END")):
+ expr = BEGIN + pp.OneOrMore(body_word, stopOn=ender) + END
+ self.assertEqual(
+ expr, test, "Did not successfully stop on ending expression %r" % ender
+ )
+
+ expr = BEGIN + body_word[...].stopOn(ender) + END
+ self.assertEqual(
+ expr, test, "Did not successfully stop on ending expression %r" % ender
+ )
+
+ number = pp.Word(pp.nums + ",.()").setName("number with optional commas")
+ parser = pp.OneOrMore(pp.Word(pp.alphanums + "-/."), stopOn=number)(
+ "id"
+ ).setParseAction(" ".join) + number("data")
+ self.assertParseAndCheckList(
+ parser,
+ " XXX Y/123 1,234.567890",
+ ["XXX Y/123", "1,234.567890"],
+ "Did not successfully stop on ending expression %r" % number,
+ verbose=True,
+ )
+
+ def testZeroOrMoreStop(self):
+ test = "BEGIN END"
+ BEGIN, END = map(pp.Keyword, "BEGIN,END".split(","))
+ body_word = pp.Word(pp.alphas).setName("word")
+ for ender in (END, "END", pp.CaselessKeyword("END")):
+ expr = BEGIN + pp.ZeroOrMore(body_word, stopOn=ender) + END
+ self.assertEqual(
+ expr, test, "Did not successfully stop on ending expression %r" % ender
+ )
+
+ expr = BEGIN + body_word[0, ...].stopOn(ender) + END
+ self.assertEqual(
+ expr, test, "Did not successfully stop on ending expression %r" % ender
+ )
+
+ def testNestedAsDict(self):
+
+ equals = pp.Literal("=").suppress()
+ lbracket = pp.Literal("[").suppress()
+ rbracket = pp.Literal("]").suppress()
+ lbrace = pp.Literal("{").suppress()
+ rbrace = pp.Literal("}").suppress()
+
+ value_dict = pp.Forward()
+ value_list = pp.Forward()
+ value_string = pp.Word(pp.alphanums + "@. ")
+
+ value = value_list ^ value_dict ^ value_string
+ values = pp.Group(pp.delimitedList(value, ","))
+ # ~ values = delimitedList(value, ",").setParseAction(lambda toks: [toks.asList()])
+
+ value_list <<= lbracket + values + rbracket
+
+ identifier = pp.Word(pp.alphanums + "_.")
+
+ assignment = pp.Group(identifier + equals + pp.Optional(value))
+ assignments = pp.Dict(pp.delimitedList(assignment, ";"))
+ value_dict <<= lbrace + assignments + rbrace
+
+ response = assignments
+
+ rsp = (
+ "username=goat; errors={username=[already taken, too short]}; empty_field="
+ )
+ result_dict = response.parseString(rsp).asDict()
+ print(result_dict)
+ self.assertEqual(
+ "goat",
+ result_dict["username"],
+ "failed to process string in ParseResults correctly",
+ )
+ self.assertEqual(
+ ["already taken", "too short"],
+ result_dict["errors"]["username"],
+ "failed to process nested ParseResults correctly",
+ )
+
+ def testTraceParseActionDecorator(self):
+ @pp.traceParseAction
+ def convert_to_int(t):
+ return int(t[0])
+
+ class Z:
+ def __call__(self, other):
+ return other[0] * 1000
+
+ integer = pp.Word(pp.nums).addParseAction(convert_to_int)
+ integer.addParseAction(pp.traceParseAction(lambda t: t[0] * 10))
+ integer.addParseAction(pp.traceParseAction(Z()))
+ integer.parseString("132")
+
+ def testRunTests(self):
+ integer = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
+ intrange = integer("start") + "-" + integer("end")
+ intrange.addCondition(
+ lambda t: t.end > t.start,
+ message="invalid range, start must be <= end",
+ fatal=True,
+ )
+ intrange.addParseAction(lambda t: list(range(t.start, t.end + 1)))
+
+ indices = pp.delimitedList(intrange | integer)
+ indices.addParseAction(lambda t: sorted(set(t)))
+
+ tests = """\
+ # normal data
+ 1-3,2-4,6,8-10,16
+
+ # lone integer
+ 11"""
+ results = indices.runTests(tests, printResults=False)[1]
+
+ expectedResults = [[1, 2, 3, 4, 6, 8, 9, 10, 16], [11]]
+ for res, expected in zip(results, expectedResults):
+ print(res[1].asList())
+ print(expected)
+ self.assertEqual(expected, res[1].asList(), "failed test: " + str(expected))
+
+ tests = """\
+ # invalid range
+ 1-2, 3-1, 4-6, 7, 12
+ """
+ success = indices.runTests(tests, printResults=False, failureTests=True)[0]
+ self.assertTrue(success, "failed to raise exception on improper range test")
+
+ def testRunTestsPostParse(self):
+ integer = ppc.integer
+ fraction = integer("numerator") + "/" + integer("denominator")
+
+ accum = []
+
+ def eval_fraction(test, result):
+ accum.append((test, result.asList()))
+ return "eval: {}".format(result.numerator / result.denominator)
+
+ success = fraction.runTests(
+ """\
+ 1/2
+ 1/0
+ """,
+ postParse=eval_fraction,
+ )[0]
+ print(success)
+
+ self.assertTrue(success, "failed to parse fractions in RunTestsPostParse")
+
+ expected_accum = [("1/2", [1, "/", 2]), ("1/0", [1, "/", 0])]
+ self.assertEqual(
+ expected_accum, accum, "failed to call postParse method during runTests"
+ )
+
+ def testConvertToDateErr(self):
+ """raise a ParseException in convertToDate with incompatible date str"""
+
+ expr = pp.Word(pp.alphanums + "-")
+ expr.addParseAction(ppc.convertToDate())
+
+ with self.assertRaisesParseException():
+ expr.parseString("1997-07-error")
+
+ def testConvertToDatetimeErr(self):
+ """raise a ParseException in convertToDatetime with incompatible datetime str"""
+
+ expr = pp.Word(pp.alphanums + "-")
+ expr.addParseAction(ppc.convertToDatetime())
+
+ with self.assertRaisesParseException():
+ expr.parseString("1997-07-error")
+
+ def testCommonExpressions(self):
+ import ast
+
+ success = ppc.mac_address.runTests(
+ """
+ AA:BB:CC:DD:EE:FF
+ AA.BB.CC.DD.EE.FF
+ AA-BB-CC-DD-EE-FF
+ """
+ )[0]
+ self.assertTrue(success, "error in parsing valid MAC address")
+
+ success = ppc.mac_address.runTests(
+ """
+ # mixed delimiters
+ AA.BB:CC:DD:EE:FF
+ """,
+ failureTests=True,
+ )[0]
+ self.assertTrue(success, "error in detecting invalid mac address")
+
+ success = ppc.ipv4_address.runTests(
+ """
+ 0.0.0.0
+ 1.1.1.1
+ 127.0.0.1
+ 1.10.100.199
+ 255.255.255.255
+ """
+ )[0]
+ self.assertTrue(success, "error in parsing valid IPv4 address")
+
+ success = ppc.ipv4_address.runTests(
+ """
+ # out of range value
+ 256.255.255.255
+ """,
+ failureTests=True,
+ )[0]
+ self.assertTrue(success, "error in detecting invalid IPv4 address")
+
+ success = ppc.ipv6_address.runTests(
+ """
+ 2001:0db8:85a3:0000:0000:8a2e:0370:7334
+ 2134::1234:4567:2468:1236:2444:2106
+ 0:0:0:0:0:0:A00:1
+ 1080::8:800:200C:417A
+ ::A00:1
+
+ # loopback address
+ ::1
+
+ # the null address
+ ::
+
+ # ipv4 compatibility form
+ ::ffff:192.168.0.1
+ """
+ )[0]
+ self.assertTrue(success, "error in parsing valid IPv6 address")
+
+ success = ppc.ipv6_address.runTests(
+ """
+ # too few values
+ 1080:0:0:0:8:800:200C
+
+ # too many ::'s, only 1 allowed
+ 2134::1234:4567::2444:2106
+ """,
+ failureTests=True,
+ )[0]
+ self.assertTrue(success, "error in detecting invalid IPv6 address")
+
+ success = ppc.number.runTests(
+ """
+ 100
+ -100
+ +100
+ 3.14159
+ 6.02e23
+ 1e-12
+ """
+ )[0]
+ self.assertTrue(success, "error in parsing valid numerics")
+
+ success = ppc.sci_real.runTests(
+ """
+ 1e12
+ -1e12
+ 3.14159
+ 6.02e23
+ """
+ )[0]
+ self.assertTrue(success, "error in parsing valid scientific notation reals")
+
+ # any int or real number, returned as float
+ success = ppc.fnumber.runTests(
+ """
+ 100
+ -100
+ +100
+ 3.14159
+ 6.02e23
+ 1e-12
+ """
+ )[0]
+ self.assertTrue(success, "error in parsing valid numerics")
+
+ success, results = ppc.iso8601_date.runTests(
+ """
+ 1997
+ 1997-07
+ 1997-07-16
+ """
+ )
+ self.assertTrue(success, "error in parsing valid iso8601_date")
+ expected = [("1997", None, None), ("1997", "07", None), ("1997", "07", "16")]
+ for r, exp in zip(results, expected):
+ self.assertEqual(
+ exp,
+ (r[1].year, r[1].month, r[1].day),
+ "failed to parse date into fields",
+ )
+
+ success, results = (
+ ppc.iso8601_date()
+ .addParseAction(ppc.convertToDate())
+ .runTests(
+ """
+ 1997-07-16
+ """
+ )
+ )
+ self.assertTrue(
+ success, "error in parsing valid iso8601_date with parse action"
+ )
+ self.assertEqual(
+ datetime.date(1997, 7, 16),
+ results[0][1][0],
+ "error in parsing valid iso8601_date with parse action - incorrect value",
+ )
+
+ success, results = ppc.iso8601_datetime.runTests(
+ """
+ 1997-07-16T19:20+01:00
+ 1997-07-16T19:20:30+01:00
+ 1997-07-16T19:20:30.45Z
+ 1997-07-16 19:20:30.45
+ """
+ )
+ self.assertTrue(success, "error in parsing valid iso8601_datetime")
+
+ success, results = (
+ ppc.iso8601_datetime()
+ .addParseAction(ppc.convertToDatetime())
+ .runTests(
+ """
+ 1997-07-16T19:20:30.45
+ """
+ )
+ )
+
+ self.assertTrue(success, "error in parsing valid iso8601_datetime")
+ self.assertEqual(
+ datetime.datetime(1997, 7, 16, 19, 20, 30, 450000),
+ results[0][1][0],
+ "error in parsing valid iso8601_datetime - incorrect value",
+ )
+
+ success = ppc.uuid.runTests(
+ """
+ 123e4567-e89b-12d3-a456-426655440000
+ """
+ )[0]
+ self.assertTrue(success, "failed to parse valid uuid")
+
+ success = ppc.fraction.runTests(
+ """
+ 1/2
+ -15/16
+ -3/-4
+ """
+ )[0]
+ self.assertTrue(success, "failed to parse valid fraction")
+
+ success = ppc.mixed_integer.runTests(
+ """
+ 1/2
+ -15/16
+ -3/-4
+ 1 1/2
+ 2 -15/16
+ 0 -3/-4
+ 12
+ """
+ )[0]
+ self.assertTrue(success, "failed to parse valid mixed integer")
+
+ success, results = ppc.number.runTests(
+ """
+ 100
+ -3
+ 1.732
+ -3.14159
+ 6.02e23"""
+ )
+ self.assertTrue(success, "failed to parse numerics")
+
+ for test, result in results:
+ expected = ast.literal_eval(test)
+ self.assertEqual(
+ expected,
+ result[0],
+ "numeric parse failed (wrong value) ({} should be {})".format(
+ result[0], expected
+ ),
+ )
+ self.assertEqual(
+ type(expected),
+ type(result[0]),
+ "numeric parse failed (wrong type) ({} should be {})".format(
+ type(result[0]), type(expected)
+ ),
+ )
+
+ def testCommonUrl(self):
+ url_good_tests = """\
+ http://foo.com/blah_blah
+ http://foo.com/blah_blah/
+ http://foo.com/blah_blah_(wikipedia)
+ http://foo.com/blah_blah_(wikipedia)_(again)
+ http://www.example.com/wpstyle/?p=364
+ https://www.example.com/foo/?bar=baz&inga=42&quux
+ http://✪df.ws/123
+ http://userid:password@example.com:8080
+ http://userid:password@example.com:8080/
+ http://userid@example.com
+ http://userid@example.com/
+ http://userid@example.com:8080
+ http://userid@example.com:8080/
+ http://userid:password@example.com
+ http://userid:password@example.com/
+ http://142.42.1.1/
+ http://142.42.1.1:8080/
+ http://➡.ws/䨹
+ http://⌘.ws
+ http://⌘.ws/
+ http://foo.com/blah_(wikipedia)#cite-1
+ http://foo.com/blah_(wikipedia)_blah#cite-1
+ http://foo.com/unicode_(✪)_in_parens
+ http://foo.com/(something)?after=parens
+ http://☺.damowmow.com/
+ http://code.google.com/events/#&product=browser
+ http://j.mp
+ ftp://foo.bar/baz
+ http://foo.bar/?q=Test%20URL-encoded%20stuff
+ http://مثال.إختبار
+ """
+ success, report = ppc.url.runTests(url_good_tests)
+ self.assertTrue(success)
+
+ url_bad_tests = """\
+ http://
+ http://.
+ http://..
+ http://../
+ http://?
+ http://??
+ http://??/
+ http://#
+ http://##
+ http://##/
+ # skip: http://foo.bar?q=Spaces should be encoded
+ //
+ //a
+ ///a
+ ///
+ http:///a
+ foo.com
+ rdar://1234
+ h://test
+ http:// shouldfail.com
+
+ :// should fail
+ http://foo.bar/foo(bar)baz quux
+ ftps://foo.bar/
+ http://-error-.invalid/
+ # skip: http://a.b--c.de/
+ http://-a.b.co
+ http://a.b-.co
+ http://0.0.0.0
+ http://10.1.1.0
+ http://10.1.1.255
+ http://224.1.1.1
+ http://1.1.1.1.1
+ http://123.123.123
+ http://3628126748
+ http://.www.foo.bar/
+ # skip: http://www.foo.bar./
+ http://.www.foo.bar./
+ http://10.1.1.1
+ """
+ success, report = ppc.url.runTests(url_bad_tests, failure_tests=True)
+ self.assertTrue(success)
+
+ def testCommonUrlParts(self):
+ from urllib.parse import urlparse
+ sample_url = "https://bob:secret@www.example.com:8080/path/to/resource?filter=int#book-mark"
+
+ parts = urlparse(sample_url)
+ expected = {
+ "scheme": parts.scheme,
+ "auth": "{}:{}".format(parts.username, parts.password),
+ "host": parts.hostname,
+ "port": str(parts.port),
+ "path": parts.path,
+ "query": parts.query,
+ "fragment": parts.fragment,
+ }
+
+ self.assertParseAndCheckDict(ppc.url, sample_url, expected, verbose=True)
+
+ def testNumericExpressions(self):
+
+ # disable parse actions that do type conversion so we don't accidentally trigger
+ # conversion exceptions when what we want to check is the parsing expression
+ real = ppc.real().setParseAction(None)
+ sci_real = ppc.sci_real().setParseAction(None)
+ signed_integer = ppc.signed_integer().setParseAction(None)
+
+ from itertools import product
+
+ def make_tests():
+ leading_sign = ["+", "-", ""]
+ leading_digit = ["0", ""]
+ dot = [".", ""]
+ decimal_digit = ["1", ""]
+ e = ["e", "E", ""]
+ e_sign = ["+", "-", ""]
+ e_int = ["22", ""]
+ stray = ["9", ".", ""]
+
+ seen = set()
+ seen.add("")
+ for parts in product(
+ leading_sign,
+ stray,
+ leading_digit,
+ dot,
+ decimal_digit,
+ stray,
+ e,
+ e_sign,
+ e_int,
+ stray,
+ ):
+ parts_str = "".join(parts).strip()
+ if parts_str in seen:
+ continue
+ seen.add(parts_str)
+ yield parts_str
+
+ print(len(seen) - 1, "tests produced")
+
+ # collect tests into valid/invalid sets, depending on whether they evaluate to valid Python floats or ints
+ valid_ints = set()
+ valid_reals = set()
+ valid_sci_reals = set()
+ invalid_ints = set()
+ invalid_reals = set()
+ invalid_sci_reals = set()
+
+ # check which strings parse as valid floats or ints, and store in related valid or invalid test sets
+ for test_str in make_tests():
+ if "." in test_str or "e" in test_str.lower():
+ try:
+ float(test_str)
+ except ValueError:
+ invalid_sci_reals.add(test_str)
+ if "e" not in test_str.lower():
+ invalid_reals.add(test_str)
+ else:
+ valid_sci_reals.add(test_str)
+ if "e" not in test_str.lower():
+ valid_reals.add(test_str)
+
+ try:
+ int(test_str)
+ except ValueError:
+ invalid_ints.add(test_str)
+ else:
+ valid_ints.add(test_str)
+
+ # now try all the test sets against their respective expressions
+ all_pass = True
+ suppress_results = {"printResults": False}
+ for expr, tests, is_fail, fn in zip(
+ [real, sci_real, signed_integer] * 2,
+ [
+ valid_reals,
+ valid_sci_reals,
+ valid_ints,
+ invalid_reals,
+ invalid_sci_reals,
+ invalid_ints,
+ ],
+ [False, False, False, True, True, True],
+ [float, float, int] * 2,
+ ):
+ #
+ # success, test_results = expr.runTests(sorted(tests, key=len), failureTests=is_fail, **suppress_results)
+ # filter_result_fn = (lambda r: isinstance(r, Exception),
+ # lambda r: not isinstance(r, Exception))[is_fail]
+ # print(expr, ('FAIL', 'PASS')[success], "{}valid tests ({})".format(len(tests),
+ # 'in' if is_fail else ''))
+ # if not success:
+ # all_pass = False
+ # for test_string, result in test_results:
+ # if filter_result_fn(result):
+ # try:
+ # test_value = fn(test_string)
+ # except ValueError as ve:
+ # test_value = str(ve)
+ # print("{!r}: {} {} {}".format(test_string, result,
+ # expr.matches(test_string, parseAll=True), test_value))
+
+ success = True
+ for t in tests:
+ if expr.matches(t, parseAll=True):
+ if is_fail:
+ print(t, "should fail but did not")
+ success = False
+ else:
+ if not is_fail:
+ print(t, "should not fail but did")
+ success = False
+ print(
+ expr,
+ ("FAIL", "PASS")[success],
+ "{}valid tests ({})".format("in" if is_fail else "", len(tests)),
+ )
+ all_pass = all_pass and success
+
+ self.assertTrue(all_pass, "failed one or more numeric tests")
+
+ def testTokenMap(self):
+
+ parser = pp.OneOrMore(pp.Word(pp.hexnums)).setParseAction(pp.tokenMap(int, 16))
+ success, report = parser.runTests(
+ """
+ 00 11 22 aa FF 0a 0d 1a
+ """
+ )
+
+ self.assertRunTestResults(
+ (success, report),
+ [([0, 17, 34, 170, 255, 10, 13, 26], "tokenMap parse action failed")],
+ msg="failed to parse hex integers",
+ )
+
+ def testParseFile(self):
+
+ s = """
+ 123 456 789
+ """
+ input_file = StringIO(s)
+ integer = ppc.integer
+
+ results = pp.OneOrMore(integer).parseFile(input_file)
+ print(results)
+
+ results = pp.OneOrMore(integer).parseFile("tests/parsefiletest_input_file.txt")
+ print(results)
+
+ def testHTMLStripper(self):
+
+ sample = """
+ <html>
+ Here is some sample <i>HTML</i> text.
+ </html>
+ """
+ read_everything = pp.originalTextFor(pp.OneOrMore(pp.Word(pp.printables)))
+ read_everything.addParseAction(ppc.stripHTMLTags)
+
+ result = read_everything.parseString(sample)
+ self.assertEqual("Here is some sample HTML text.", result[0].strip())
+
+ def testExprSplitter(self):
+
+ expr = pp.Literal(";") + pp.Empty()
+ expr.ignore(pp.quotedString)
+ expr.ignore(pp.pythonStyleComment)
+
+ sample = """
+ def main():
+ this_semi_does_nothing();
+ neither_does_this_but_there_are_spaces_afterward();
+ a = "a;b"; return a # this is a comment; it has a semicolon!
+
+ def b():
+ if False:
+ z=1000;b("; in quotes"); c=200;return z
+ return ';'
+
+ class Foo(object):
+ def bar(self):
+ '''a docstring; with a semicolon'''
+ a = 10; b = 11; c = 12
+
+ # this comment; has several; semicolons
+ if self.spam:
+ x = 12; return x # so; does; this; one
+ x = 15;;; y += x; return y
+
+ def baz(self):
+ return self.bar
+ """
+ expected = [
+ [" this_semi_does_nothing()", ""],
+ [" neither_does_this_but_there_are_spaces_afterward()", ""],
+ [
+ ' a = "a;b"',
+ "return a # this is a comment; it has a semicolon!",
+ ],
+ [" z=1000", 'b("; in quotes")', "c=200", "return z"],
+ [" return ';'"],
+ [" '''a docstring; with a semicolon'''"],
+ [" a = 10", "b = 11", "c = 12"],
+ [" # this comment; has several; semicolons"],
+ [" x = 12", "return x # so; does; this; one"],
+ [" x = 15", "", "", "y += x", "return y"],
+ ]
+
+ exp_iter = iter(expected)
+ for line in filter(lambda ll: ";" in ll, sample.splitlines()):
+ print(str(list(expr.split(line))) + ",")
+ self.assertEqual(
+ next(exp_iter), list(expr.split(line)), "invalid split on expression"
+ )
+
+ print()
+
+ expected = [
+ [" this_semi_does_nothing()", ";", ""],
+ [" neither_does_this_but_there_are_spaces_afterward()", ";", ""],
+ [
+ ' a = "a;b"',
+ ";",
+ "return a # this is a comment; it has a semicolon!",
+ ],
+ [
+ " z=1000",
+ ";",
+ 'b("; in quotes")',
+ ";",
+ "c=200",
+ ";",
+ "return z",
+ ],
+ [" return ';'"],
+ [" '''a docstring; with a semicolon'''"],
+ [" a = 10", ";", "b = 11", ";", "c = 12"],
+ [" # this comment; has several; semicolons"],
+ [" x = 12", ";", "return x # so; does; this; one"],
+ [
+ " x = 15",
+ ";",
+ "",
+ ";",
+ "",
+ ";",
+ "y += x",
+ ";",
+ "return y",
+ ],
+ ]
+ exp_iter = iter(expected)
+ for line in filter(lambda ll: ";" in ll, sample.splitlines()):
+ print(str(list(expr.split(line, includeSeparators=True))) + ",")
+ self.assertEqual(
+ next(exp_iter),
+ list(expr.split(line, includeSeparators=True)),
+ "invalid split on expression",
+ )
+
+ print()
+
+ expected = [
+ [" this_semi_does_nothing()", ""],
+ [" neither_does_this_but_there_are_spaces_afterward()", ""],
+ [
+ ' a = "a;b"',
+ "return a # this is a comment; it has a semicolon!",
+ ],
+ [" z=1000", 'b("; in quotes"); c=200;return z'],
+ [" a = 10", "b = 11; c = 12"],
+ [" x = 12", "return x # so; does; this; one"],
+ [" x = 15", ";; y += x; return y"],
+ ]
+ exp_iter = iter(expected)
+ for line in sample.splitlines():
+ pieces = list(expr.split(line, maxsplit=1))
+ print(str(pieces) + ",")
+ if len(pieces) == 2:
+ exp = next(exp_iter)
+ self.assertEqual(
+ exp, pieces, "invalid split on expression with maxSplits=1"
+ )
+ elif len(pieces) == 1:
+ self.assertEqual(
+ 0,
+ len(expr.searchString(line)),
+ "invalid split with maxSplits=1 when expr not present",
+ )
+ else:
+ print("\n>>> " + line)
+ self.fail("invalid split on expression with maxSplits=1, corner case")
+
+ def testParseFatalException(self):
+
+ with self.assertRaisesParseException(
+ exc_type=ParseFatalException, msg="failed to raise ErrorStop exception"
+ ):
+ expr = "ZZZ" - pp.Word(pp.nums)
+ expr.parseString("ZZZ bad")
+
+ # WAS:
+ # success = False
+ # try:
+ # expr = "ZZZ" - Word(nums)
+ # expr.parseString("ZZZ bad")
+ # except ParseFatalException as pfe:
+ # print('ParseFatalException raised correctly')
+ # success = True
+ # except Exception as e:
+ # print(type(e))
+ # print(e)
+ #
+ # self.assertTrue(success, "bad handling of syntax error")
+
+ def testParseFatalException2(self):
+ # Fatal exception raised in MatchFirst should not be superseded later non-fatal exceptions
+ # addresses Issue #251
+
+ def raise_exception(tokens):
+ raise pp.ParseSyntaxException("should raise here")
+
+ test = pp.MatchFirst(
+ (
+ pp.pyparsing_common.integer + pp.pyparsing_common.identifier
+ ).setParseAction(raise_exception)
+ | pp.pyparsing_common.number
+ )
+
+ with self.assertRaisesParseException(pp.ParseFatalException):
+ test.parseString("1s")
+
+ def testParseFatalException3(self):
+ # Fatal exception raised in MatchFirst should not be superseded later non-fatal exceptions
+ # addresses Issue #251
+
+ test = pp.MatchFirst(
+ (pp.pyparsing_common.integer - pp.pyparsing_common.identifier)
+ | pp.pyparsing_common.integer
+ )
+
+ with self.assertRaisesParseException(pp.ParseFatalException):
+ test.parseString("1")
+
+ def testInlineLiteralsUsing(self):
+
+ wd = pp.Word(pp.alphas)
+
+ pp.ParserElement.inlineLiteralsUsing(pp.Suppress)
+ result = (wd + "," + wd + pp.oneOf("! . ?")).parseString("Hello, World!")
+ self.assertEqual(3, len(result), "inlineLiteralsUsing(Suppress) failed!")
+
+ pp.ParserElement.inlineLiteralsUsing(pp.Literal)
+ result = (wd + "," + wd + pp.oneOf("! . ?")).parseString("Hello, World!")
+ self.assertEqual(4, len(result), "inlineLiteralsUsing(Literal) failed!")
+
+ pp.ParserElement.inlineLiteralsUsing(pp.CaselessKeyword)
+ self.assertParseAndCheckList(
+ "SELECT" + wd + "FROM" + wd,
+ "select color from colors",
+ expected_list=["SELECT", "color", "FROM", "colors"],
+ msg="inlineLiteralsUsing(CaselessKeyword) failed!",
+ verbose=True,
+ )
+
+ pp.ParserElement.inlineLiteralsUsing(pp.CaselessLiteral)
+ self.assertParseAndCheckList(
+ "SELECT" + wd + "FROM" + wd,
+ "select color from colors",
+ expected_list=["SELECT", "color", "FROM", "colors"],
+ msg="inlineLiteralsUsing(CaselessLiteral) failed!",
+ verbose=True,
+ )
+
+ integer = pp.Word(pp.nums)
+ pp.ParserElement.inlineLiteralsUsing(pp.Literal)
+ date_str = integer("year") + "/" + integer("month") + "/" + integer("day")
+ self.assertParseAndCheckList(
+ date_str,
+ "1999/12/31",
+ expected_list=["1999", "/", "12", "/", "31"],
+ msg="inlineLiteralsUsing(example 1) failed!",
+ verbose=True,
+ )
+
+ # change to Suppress
+ pp.ParserElement.inlineLiteralsUsing(pp.Suppress)
+ date_str = integer("year") + "/" + integer("month") + "/" + integer("day")
+
+ self.assertParseAndCheckList(
+ date_str,
+ "1999/12/31",
+ expected_list=["1999", "12", "31"],
+ msg="inlineLiteralsUsing(example 2) failed!",
+ verbose=True,
+ )
+
+ def testCloseMatch(self):
+
+ searchseq = pp.CloseMatch("ATCATCGAATGGA", 2)
+
+ _, results = searchseq.runTests(
+ """
+ ATCATCGAATGGA
+ XTCATCGAATGGX
+ ATCATCGAAXGGA
+ ATCAXXGAATGGA
+ ATCAXXGAATGXA
+ ATCAXXGAATGG
+ """
+ )
+ expected = ([], [0, 12], [9], [4, 5], None, None)
+
+ for r, exp in zip(results, expected):
+ if exp is not None:
+ self.assertEqual(
+ exp,
+ r[1].mismatches,
+ "fail CloseMatch between {!r} and {!r}".format(
+ searchseq.match_string, r[0]
+ ),
+ )
+ print(
+ r[0],
+ "exc: %s" % r[1]
+ if exp is None and isinstance(r[1], Exception)
+ else ("no match", "match")[r[1].mismatches == exp],
+ )
+
+ def testCloseMatchCaseless(self):
+
+ searchseq = pp.CloseMatch("ATCATCGAATGGA", 2, caseless=True)
+
+ _, results = searchseq.runTests(
+ """
+ atcatcgaatgga
+ xtcatcgaatggx
+ atcatcgaaxgga
+ atcaxxgaatgga
+ atcaxxgaatgxa
+ atcaxxgaatgg
+ """
+ )
+ expected = ([], [0, 12], [9], [4, 5], None, None)
+
+ for r, exp in zip(results, expected):
+ if exp is not None:
+ self.assertEqual(
+ exp,
+ r[1].mismatches,
+ "fail CaselessCloseMatch between {!r} and {!r}".format(
+ searchseq.match_string, r[0]
+ ),
+ )
+ print(
+ r[0],
+ "exc: %s" % r[1]
+ if exp is None and isinstance(r[1], Exception)
+ else ("no match", "match")[r[1].mismatches == exp],
+ )
+
+ def testDefaultKeywordChars(self):
+
+ with self.assertRaisesParseException(
+ msg="failed to fail matching keyword using updated keyword chars"
+ ):
+ pp.Keyword("start").parseString("start1000")
+
+ try:
+ pp.Keyword("start", identChars=pp.alphas).parseString("start1000")
+ except pp.ParseException:
+ self.fail("failed to match keyword using updated keyword chars")
+
+ with ppt.reset_pyparsing_context():
+ pp.Keyword.setDefaultKeywordChars(pp.alphas)
+ try:
+ pp.Keyword("start").parseString("start1000")
+ except pp.ParseException:
+ self.fail("failed to match keyword using updated keyword chars")
+
+ with self.assertRaisesParseException(
+ msg="failed to fail matching keyword using updated keyword chars"
+ ):
+ pp.CaselessKeyword("START").parseString("start1000")
+
+ try:
+ pp.CaselessKeyword("START", identChars=pp.alphas).parseString("start1000")
+ except pp.ParseException:
+ self.fail("failed to match keyword using updated keyword chars")
+
+ with ppt.reset_pyparsing_context():
+ pp.Keyword.setDefaultKeywordChars(pp.alphas)
+ try:
+ pp.CaselessKeyword("START").parseString("start1000")
+ except pp.ParseException:
+ self.assertTrue(
+ False, "failed to match keyword using updated keyword chars"
+ )
+
+ def testKeywordCopyIdentChars(self):
+ a_keyword = pp.Keyword("start", identChars="_")
+ b_keyword = a_keyword.copy()
+ self.assertEqual(a_keyword.identChars, b_keyword.identChars)
+
+ def testLiteralVsKeyword(self):
+
+ integer = ppc.integer
+ literal_expr = integer + pp.Literal("start") + integer
+ keyword_expr = integer + pp.Keyword("start") + integer
+ caseless_keyword_expr = integer + pp.CaselessKeyword("START") + integer
+ word_keyword_expr = (
+ integer + pp.Word(pp.alphas, asKeyword=True).setName("word") + integer
+ )
+
+ print()
+ test_string = "1 start 2"
+ print(test_string)
+ print(literal_expr, literal_expr.parseString(test_string, parseAll=True))
+ print(keyword_expr, keyword_expr.parseString(test_string, parseAll=True))
+ print(
+ caseless_keyword_expr,
+ caseless_keyword_expr.parseString(test_string, parseAll=True),
+ )
+ print(
+ word_keyword_expr, word_keyword_expr.parseString(test_string, parseAll=True)
+ )
+ print()
+
+ test_string = "3 start4"
+ print(test_string)
+ print(literal_expr, literal_expr.parseString(test_string, parseAll=True))
+ with self.assertRaisesParseException(
+ msg="failed to fail matching keyword using updated keyword chars"
+ ):
+ print(keyword_expr.parseString(test_string, parseAll=True))
+
+ with self.assertRaisesParseException(
+ msg="failed to fail matching keyword using updated keyword chars"
+ ):
+ print(caseless_keyword_expr.parseString(test_string, parseAll=True))
+
+ with self.assertRaisesParseException(
+ msg="failed to fail matching keyword using updated keyword chars"
+ ):
+ print(word_keyword_expr.parseString(test_string, parseAll=True))
+ print()
+
+ test_string = "5start 6"
+ print(test_string)
+ print(literal_expr.parseString(test_string, parseAll=True))
+ with self.assertRaisesParseException(
+ msg="failed to fail matching keyword using updated keyword chars"
+ ):
+ print(keyword_expr.parseString(test_string, parseAll=True))
+
+ with self.assertRaisesParseException(
+ msg="failed to fail matching keyword using updated keyword chars"
+ ):
+ print(caseless_keyword_expr.parseString(test_string, parseAll=True))
+
+ with self.assertRaisesParseException(
+ msg="failed to fail matching keyword using updated keyword chars"
+ ):
+ print(word_keyword_expr.parseString(test_string, parseAll=True))
+
+ def testCol(self):
+
+ test = "*\n* \n* ALF\n*\n"
+ initials = [c for i, c in enumerate(test) if pp.col(i, test) == 1]
+ print(initials)
+ self.assertTrue(
+ len(initials) == 4 and all(c == "*" for c in initials), "fail col test"
+ )
+
+ def testLiteralException(self):
+
+ for cls in (
+ pp.Literal,
+ pp.CaselessLiteral,
+ pp.Keyword,
+ pp.CaselessKeyword,
+ pp.Word,
+ pp.Regex,
+ ):
+ expr = cls("xyz") # .setName('{}_expr'.format(cls.__name__.lower()))
+
+ try:
+ expr.parseString(" ")
+ except Exception as e:
+ print(cls.__name__, str(e))
+ self.assertTrue(
+ isinstance(e, pp.ParseBaseException),
+ "class {} raised wrong exception type {}".format(
+ cls.__name__, type(e).__name__
+ ),
+ )
+
+ def testParseActionException(self):
+ import traceback
+
+ number = pp.Word(pp.nums)
+
+ def number_action():
+ raise IndexError # this is the important line!
+
+ number.setParseAction(number_action)
+ symbol = pp.Word("abcd", max=1)
+ expr = number | symbol
+
+ try:
+ expr.parseString("1 + 2")
+ except Exception as e:
+ print_traceback = True
+ try:
+ self.assertTrue(
+ hasattr(e, "__cause__"),
+ "no __cause__ attribute in the raised exception",
+ )
+ self.assertTrue(
+ e.__cause__ is not None,
+ "__cause__ not propagated to outer exception",
+ )
+ self.assertEqual(
+ IndexError,
+ type(e.__cause__),
+ "__cause__ references wrong exception",
+ )
+ print_traceback = False
+ finally:
+ if print_traceback:
+ traceback.print_exc()
+ else:
+ self.fail("Expected ParseException not raised")
+
+ # tests Issue #22
+ def testParseActionNesting(self):
+
+ vals = pp.OneOrMore(ppc.integer)("int_values")
+
+ def add_total(tokens):
+ tokens["total"] = sum(tokens)
+ return tokens
+
+ vals.addParseAction(add_total)
+ results = vals.parseString("244 23 13 2343")
+ print(results.dump())
+ self.assertParseResultsEquals(
+ results,
+ expected_dict={"int_values": [244, 23, 13, 2343], "total": 2623},
+ msg="noop parse action changed ParseResults structure",
+ )
+
+ name = pp.Word(pp.alphas)("name")
+ score = pp.Word(pp.nums + ".")("score")
+ nameScore = pp.Group(name + score)
+ line1 = nameScore("Rider")
+
+ result1 = line1.parseString("Mauney 46.5")
+
+ print("### before parse action is added ###")
+ print("result1.dump():\n" + result1.dump() + "\n")
+ before_pa_dict = result1.asDict()
+
+ line1.setParseAction(lambda t: t)
+
+ result1 = line1.parseString("Mauney 46.5")
+ after_pa_dict = result1.asDict()
+
+ print("### after parse action was added ###")
+ print("result1.dump():\n" + result1.dump() + "\n")
+ self.assertEqual(
+ before_pa_dict,
+ after_pa_dict,
+ "noop parse action changed ParseResults structure",
+ )
+
+ def testParseResultsNameBelowUngroupedName(self):
+
+ rule_num = pp.Regex("[0-9]+")("LIT_NUM*")
+ list_num = pp.Group(
+ pp.Literal("[")("START_LIST")
+ + pp.delimitedList(rule_num)("LIST_VALUES")
+ + pp.Literal("]")("END_LIST")
+ )("LIST")
+
+ test_string = "[ 1,2,3,4,5,6 ]"
+ list_num.runTests(test_string)
+
+ U = list_num.parseString(test_string)
+ self.assertTrue(
+ "LIT_NUM" not in U.LIST.LIST_VALUES,
+ "results name retained as sub in ungrouped named result",
+ )
+
+ def testParseResultsNamesInGroupWithDict(self):
+
+ key = ppc.identifier()
+ value = ppc.integer()
+ lat = ppc.real()
+ long = ppc.real()
+ EQ = pp.Suppress("=")
+
+ data = (
+ lat("lat")
+ + long("long")
+ + pp.Dict(pp.OneOrMore(pp.Group(key + EQ + value)))
+ )
+ site = pp.QuotedString('"')("name") + pp.Group(data)("data")
+
+ test_string = '"Golden Gate Bridge" 37.819722 -122.478611 height=746 span=4200'
+ site.runTests(test_string)
+
+ a, aEnd = pp.makeHTMLTags("a")
+ attrs = a.parseString("<a href='blah'>")
+ print(attrs.dump())
+ self.assertParseResultsEquals(
+ attrs,
+ expected_dict={
+ "startA": {"href": "blah", "tag": "a", "empty": False},
+ "href": "blah",
+ "tag": "a",
+ "empty": False,
+ },
+ )
+
+ def testMakeXMLTags(self):
+ """test helper function makeXMLTags in simple use case"""
+
+ body, bodyEnd = pp.makeXMLTags("body")
+ tst = "<body>Hello</body>"
+ expr = body + pp.Word(pp.alphas)("contents") + bodyEnd
+ result = expr.parseString(tst)
+ print(result.dump())
+ self.assertParseResultsEquals(
+ result, ["body", False, "Hello", "</body>"], msg="issue using makeXMLTags"
+ )
+
+ def testFollowedBy(self):
+
+ expr = pp.Word(pp.alphas)("item") + pp.FollowedBy(ppc.integer("qty"))
+ result = expr.parseString("balloon 99", parseAll=False)
+ print(result.dump())
+ self.assertTrue("qty" in result, "failed to capture results name in FollowedBy")
+ self.assertEqual(
+ {"item": "balloon", "qty": 99},
+ result.asDict(),
+ "invalid results name structure from FollowedBy",
+ )
+
+ def testSetBreak(self):
+ """
+ Test behavior of ParserElement.setBreak(), to invoke the debugger before parsing that element is attempted.
+
+ Temporarily monkeypatches pdb.set_trace.
+ """
+ was_called = False
+
+ def mock_set_trace():
+ nonlocal was_called
+ was_called = True
+
+ wd = pp.Word(pp.alphas)
+ wd.setBreak()
+
+ print("Before parsing with setBreak:", was_called)
+ import pdb
+
+ with ppt.reset_pyparsing_context():
+ pdb.set_trace = mock_set_trace
+ wd.parseString("ABC")
+
+ print("After parsing with setBreak:", was_called)
+ self.assertTrue(was_called, "set_trace wasn't called by setBreak")
+
+ def testUnicodeTests(self):
+
+ ppu = pp.pyparsing_unicode
+
+ # verify proper merging of ranges by addition
+ kanji_printables = ppu.Japanese.Kanji.printables
+ katakana_printables = ppu.Japanese.Katakana.printables
+ hiragana_printables = ppu.Japanese.Hiragana.printables
+ japanese_printables = ppu.Japanese.printables
+ self.assertEqual(
+ set(kanji_printables + katakana_printables + hiragana_printables),
+ set(japanese_printables),
+ "failed to construct ranges by merging Japanese types",
+ )
+
+ # verify proper merging of ranges using multiple inheritance
+ cjk_printables = ppu.CJK.printables
+ self.assertEqual(
+ len(set(cjk_printables)),
+ len(cjk_printables),
+ "CJK contains duplicate characters - all should be unique",
+ )
+
+ chinese_printables = ppu.Chinese.printables
+ korean_printables = ppu.Korean.printables
+ print(
+ len(set(chinese_printables + korean_printables + japanese_printables)),
+ len(cjk_printables),
+ )
+
+ self.assertEqual(
+ len(set(chinese_printables + korean_printables + japanese_printables)),
+ len(cjk_printables),
+ "failed to construct ranges by merging Chinese, Japanese and Korean",
+ )
+
+ alphas = ppu.Greek.alphas
+ greet = pp.Word(alphas) + "," + pp.Word(alphas) + "!"
+
+ # input string
+ hello = "Καλημέρα, κόσμε!"
+ result = greet.parseString(hello)
+ print(result)
+ self.assertParseResultsEquals(
+ result,
+ expected_list=["Καλημέρα", ",", "κόσμε", "!"],
+ msg="Failed to parse Greek 'Hello, World!' using "
+ "pyparsing_unicode.Greek.alphas",
+ )
+
+ # define a custom unicode range using multiple inheritance
+ class Turkish_set(ppu.Latin1, ppu.LatinA):
+ pass
+
+ self.assertEqual(
+ set(ppu.Latin1.printables + ppu.LatinA.printables),
+ set(Turkish_set.printables),
+ "failed to construct ranges by merging Latin1 and LatinA (printables)",
+ )
+
+ self.assertEqual(
+ set(ppu.Latin1.alphas + ppu.LatinA.alphas),
+ set(Turkish_set.alphas),
+ "failed to construct ranges by merging Latin1 and LatinA (alphas)",
+ )
+
+ self.assertEqual(
+ set(ppu.Latin1.nums + ppu.LatinA.nums),
+ set(Turkish_set.nums),
+ "failed to construct ranges by merging Latin1 and LatinA (nums)",
+ )
+
+ key = pp.Word(Turkish_set.alphas)
+ value = ppc.integer | pp.Word(Turkish_set.alphas, Turkish_set.alphanums)
+ EQ = pp.Suppress("=")
+ key_value = key + EQ + value
+
+ sample = """\
+ şehir=İzmir
+ ülke=Türkiye
+ nüfus=4279677"""
+ result = pp.Dict(pp.OneOrMore(pp.Group(key_value))).parseString(sample)
+
+ print(result.dump())
+ self.assertParseResultsEquals(
+ result,
+ expected_dict={"şehir": "İzmir", "ülke": "Türkiye", "nüfus": 4279677},
+ msg="Failed to parse Turkish key-value pairs",
+ )
+
+ # Make sure example in indentedBlock docstring actually works!
+ def testIndentedBlockExample(self):
+
+ data = dedent(
+ """
+ def A(z):
+ A1
+ B = 100
+ G = A2
+ A2
+ A3
+ B
+ def BB(a,b,c):
+ BB1
+ def BBA():
+ bba1
+ bba2
+ bba3
+ C
+ D
+ def spam(x,y):
+ def eggs(z):
+ pass
+ """
+ )
+
+ indentStack = [1]
+ stmt = pp.Forward()
+
+ identifier = pp.Word(pp.alphas, pp.alphanums)
+ funcDecl = (
+ "def"
+ + identifier
+ + pp.Group("(" + pp.Optional(pp.delimitedList(identifier)) + ")")
+ + ":"
+ )
+ func_body = pp.indentedBlock(stmt, indentStack)
+ funcDef = pp.Group(funcDecl + func_body)
+
+ rvalue = pp.Forward()
+ funcCall = pp.Group(
+ identifier + "(" + pp.Optional(pp.delimitedList(rvalue)) + ")"
+ )
+ rvalue << (funcCall | identifier | pp.Word(pp.nums))
+ assignment = pp.Group(identifier + "=" + rvalue)
+ stmt <<= funcDef | assignment | identifier
+
+ module_body = pp.OneOrMore(stmt)
+
+ self.assertParseAndCheckList(
+ module_body,
+ data,
+ [
+ [
+ "def",
+ "A",
+ ["(", "z", ")"],
+ ":",
+ [["A1"], [["B", "=", "100"]], [["G", "=", "A2"]], ["A2"], ["A3"]],
+ ],
+ "B",
+ [
+ "def",
+ "BB",
+ ["(", "a", "b", "c", ")"],
+ ":",
+ [
+ ["BB1"],
+ [
+ [
+ "def",
+ "BBA",
+ ["(", ")"],
+ ":",
+ [["bba1"], ["bba2"], ["bba3"]],
+ ]
+ ],
+ ],
+ ],
+ "C",
+ "D",
+ [
+ "def",
+ "spam",
+ ["(", "x", "y", ")"],
+ ":",
+ [[["def", "eggs", ["(", "z", ")"], ":", [["pass"]]]]],
+ ],
+ ],
+ "Failed indentedBlock example",
+ verbose=True,
+ )
+
+ def testIndentedBlock(self):
+ # parse pseudo-yaml indented text
+
+ EQ = pp.Suppress("=")
+ stack = [1]
+ key = ppc.identifier
+ value = pp.Forward()
+ key_value = key + EQ + value
+ compound_value = pp.Dict(pp.ungroup(pp.indentedBlock(key_value, stack)))
+ value <<= ppc.integer | pp.QuotedString("'") | compound_value
+ parser = pp.Dict(pp.OneOrMore(pp.Group(key_value)))
+
+ text = """
+ a = 100
+ b = 101
+ c =
+ c1 = 200
+ c2 =
+ c21 = 999
+ c3 = 'A horse, a horse, my kingdom for a horse'
+ d = 505
+ """
+ text = dedent(text)
+ print(text)
+
+ result = parser.parseString(text)
+ print(result.dump())
+ self.assertEqual(100, result.a, "invalid indented block result")
+ self.assertEqual(200, result.c.c1, "invalid indented block result")
+ self.assertEqual(999, result.c.c2.c21, "invalid indented block result")
+
+ # exercise indentedBlock with example posted in issue #87
+ def testIndentedBlockTest2(self):
+
+ indent_stack = [1]
+
+ key = pp.Word(pp.alphas, pp.alphanums) + pp.Suppress(":")
+ stmt = pp.Forward()
+
+ suite = pp.indentedBlock(stmt, indent_stack)
+ body = key + suite
+
+ pattern = (
+ pp.Word(pp.alphas)
+ + pp.Suppress("(")
+ + pp.Word(pp.alphas)
+ + pp.Suppress(")")
+ )
+ stmt <<= pattern
+
+ def key_parse_action(toks):
+ print("Parsing '%s'..." % toks[0])
+
+ key.setParseAction(key_parse_action)
+ header = pp.Suppress("[") + pp.Literal("test") + pp.Suppress("]")
+ content = header - pp.OneOrMore(pp.indentedBlock(body, indent_stack, False))
+
+ contents = pp.Forward()
+ suites = pp.indentedBlock(content, indent_stack)
+
+ extra = pp.Literal("extra") + pp.Suppress(":") - suites
+ contents <<= content | extra
+
+ parser = pp.OneOrMore(contents)
+
+ sample = dedent(
+ """
+ extra:
+ [test]
+ one0:
+ two (three)
+ four0:
+ five (seven)
+ extra:
+ [test]
+ one1:
+ two (three)
+ four1:
+ five (seven)
+ """
+ )
+
+ success, _ = parser.runTests([sample])
+ self.assertTrue(success, "Failed indentedBlock test for issue #87")
+
+ sample2 = dedent(
+ """
+ extra:
+ [test]
+ one:
+ two (three)
+ four:
+ five (seven)
+ extra:
+ [test]
+ one:
+ two (three)
+ four:
+ five (seven)
+
+ [test]
+ one:
+ two (three)
+ four:
+ five (seven)
+
+ [test]
+ eight:
+ nine (ten)
+ eleven:
+ twelve (thirteen)
+
+ fourteen:
+ fifteen (sixteen)
+ seventeen:
+ eighteen (nineteen)
+ """
+ )
+
+ del indent_stack[1:]
+ success, _ = parser.runTests([sample2])
+ self.assertTrue(success, "Failed indentedBlock multi-block test for issue #87")
+
+ def testIndentedBlockScan(self):
+ def get_parser():
+ """
+ A valid statement is the word "block:", followed by an indent, followed by the letter A only, or another block
+ """
+ stack = [1]
+ block = pp.Forward()
+ body = pp.indentedBlock(
+ pp.Literal("A") ^ block, indentStack=stack, indent=True
+ )
+ block <<= pp.Literal("block:") + body
+ return block
+
+ # This input string is a perfect match for the parser, so a single match is found
+ p1 = get_parser()
+ r1 = list(
+ p1.scanString(
+ dedent(
+ """\
+ block:
+ A
+ """
+ )
+ )
+ )
+ self.assertEqual(1, len(r1))
+
+ # This input string is a perfect match for the parser, except for the letter B instead of A, so this will fail (and should)
+ p2 = get_parser()
+ r2 = list(
+ p2.scanString(
+ dedent(
+ """\
+ block:
+ B
+ """
+ )
+ )
+ )
+ self.assertEqual(0, len(r2))
+
+ # This input string contains both string A and string B, and it finds one match (as it should)
+ p3 = get_parser()
+ r3 = list(
+ p3.scanString(
+ dedent(
+ """\
+ block:
+ A
+ block:
+ B
+ """
+ )
+ )
+ )
+ self.assertEqual(1, len(r3))
+
+ # This input string contains both string A and string B, but in a different order.
+ p4 = get_parser()
+ r4 = list(
+ p4.scanString(
+ dedent(
+ """\
+ block:
+ B
+ block:
+ A
+ """
+ )
+ )
+ )
+ self.assertEqual(1, len(r4))
+
+ # This is the same as case 3, but with nesting
+ p5 = get_parser()
+ r5 = list(
+ p5.scanString(
+ dedent(
+ """\
+ block:
+ block:
+ A
+ block:
+ block:
+ B
+ """
+ )
+ )
+ )
+ self.assertEqual(1, len(r5))
+
+ # This is the same as case 4, but with nesting
+ p6 = get_parser()
+ r6 = list(
+ p6.scanString(
+ dedent(
+ """\
+ block:
+ block:
+ B
+ block:
+ block:
+ A
+ """
+ )
+ )
+ )
+ self.assertEqual(1, len(r6))
+
+ def testIndentedBlockClass(self):
+ data = """\
+ A
+ 100
+ 101
+
+ 102
+ B
+ 200
+ 201
+
+ C
+ 300
+
+ """
+
+ integer = ppc.integer
+ group = pp.Group(pp.Char(pp.alphas) + pp.Group(pp.IndentedBlock(integer)))
+
+ group[...].parseString(data).pprint()
+
+ self.assertParseAndCheckList(
+ group[...], data, [["A", [100, 101, 102]], ["B", [200, 201]], ["C", [300]]]
+ )
+
+ def testIndentedBlockClass2(self):
+ datas = [
+ """\
+ A
+ 100
+ B
+ 200
+ 201
+ """,
+ """\
+ A
+ 100
+ B
+ 200
+ 201
+ """,
+ """\
+ A
+ 100
+ B
+ 200
+ 201
+ """,
+ ]
+ integer = ppc.integer
+ group = pp.Group(
+ pp.Char(pp.alphas) + pp.Group(pp.IndentedBlock(integer, recursive=False))
+ )
+
+ for data in datas:
+ print()
+ print(ppt.with_line_numbers(data))
+
+ print(group[...].parse_string(data).as_list())
+ self.assertParseAndCheckList(
+ group[...] + integer.suppress(),
+ data,
+ [["A", [100]], ["B", [200]]],
+ verbose=False,
+ )
+
+ def testIndentedBlockClassWithRecursion(self):
+ data = """\
+
+ A
+ 100
+ 101
+
+ 102
+ B
+ b
+ 200
+ 201
+
+ C
+ 300
+
+ """
+
+ integer = ppc.integer
+ group = pp.Forward()
+ group <<= pp.Group(
+ pp.Char(pp.alphas) + pp.Group(pp.IndentedBlock(integer | group))
+ )
+
+ print("using searchString")
+ print(group.searchString(data))
+ # print(sum(group.searchString(data)).dump())
+
+ self.assertParseAndCheckList(
+ group[...],
+ data,
+ [["A", [100, 101, 102]], ["B", [["b", [200, 201]]]], ["C", [300]]],
+ )
+
+ print("using parseString")
+ print(group[...].parseString(data).dump())
+
+ print("test bad indentation")
+ dotted_int = pp.delimited_list(
+ pp.Word(pp.nums), ".", allow_trailing_delim=True, combine=True
+ )
+ indented_expr = pp.IndentedBlock(dotted_int, recursive=True)
+ good_data = """\
+ 1.
+ 1.1
+ 1.1.1
+ 2."""
+ bad_data = """\
+ 1.
+ 1.1
+ 1.1.1
+ 1.2
+ 2."""
+ indented_expr.parseString(good_data, parseAll=True)
+ with self.assertRaisesParseException(
+ msg="Failed to raise exception with bad indentation"
+ ):
+ indented_expr.parseString(bad_data, parseAll=True)
+
+ def testInvalidDiagSetting(self):
+ with self.assertRaises(
+ ValueError,
+ msg="failed to raise exception when setting non-existent __diag__",
+ ):
+ pp.__diag__.enable("xyzzy")
+
+ with self.assertWarns(
+ UserWarning, msg="failed to warn disabling 'collect_all_And_tokens"
+ ):
+ pp.__compat__.disable("collect_all_And_tokens")
+
+ def testParseResultsWithNameMatchFirst(self):
+ expr_a = pp.Literal("not") + pp.Literal("the") + pp.Literal("bird")
+ expr_b = pp.Literal("the") + pp.Literal("bird")
+ expr = (expr_a | expr_b)("rexp")
+
+ success, report = expr.runTests(
+ """\
+ not the bird
+ the bird
+ """
+ )
+ results = [rpt[1] for rpt in report]
+ self.assertParseResultsEquals(
+ results[0], ["not", "the", "bird"], {"rexp": ["not", "the", "bird"]}
+ )
+ self.assertParseResultsEquals(
+ results[1], ["the", "bird"], {"rexp": ["the", "bird"]}
+ )
+
+ # test compatibility mode, no longer restoring pre-2.3.1 behavior
+ with ppt.reset_pyparsing_context():
+ pp.__compat__.collect_all_And_tokens = False
+ pp.enable_diag(pp.Diagnostics.warn_multiple_tokens_in_named_alternation)
+ expr_a = pp.Literal("not") + pp.Literal("the") + pp.Literal("bird")
+ expr_b = pp.Literal("the") + pp.Literal("bird")
+ with self.assertWarns(
+ UserWarning, msg="failed to warn of And within alternation"
+ ):
+ expr = (expr_a | expr_b)("rexp")
+
+ success, report = expr.runTests(
+ """
+ not the bird
+ the bird
+ """
+ )
+ results = [rpt[1] for rpt in report]
+ self.assertParseResultsEquals(
+ results[0], ["not", "the", "bird"], {"rexp": ["not", "the", "bird"]}
+ )
+ self.assertParseResultsEquals(
+ results[1], ["the", "bird"], {"rexp": ["the", "bird"]}
+ )
+
+ def testParseResultsWithNameOr(self):
+ expr_a = pp.Literal("not") + pp.Literal("the") + pp.Literal("bird")
+ expr_b = pp.Literal("the") + pp.Literal("bird")
+ expr = (expr_a ^ expr_b)("rexp")
+ expr.runTests(
+ """\
+ not the bird
+ the bird
+ """
+ )
+ result = expr.parseString("not the bird")
+ self.assertParseResultsEquals(
+ result, ["not", "the", "bird"], {"rexp": ["not", "the", "bird"]}
+ )
+ result = expr.parseString("the bird")
+ self.assertParseResultsEquals(
+ result, ["the", "bird"], {"rexp": ["the", "bird"]}
+ )
+
+ expr = (expr_a | expr_b)("rexp")
+ expr.runTests(
+ """\
+ not the bird
+ the bird
+ """
+ )
+ result = expr.parseString("not the bird")
+ self.assertParseResultsEquals(
+ result, ["not", "the", "bird"], {"rexp": ["not", "the", "bird"]}
+ )
+ result = expr.parseString("the bird")
+ self.assertParseResultsEquals(
+ result, ["the", "bird"], {"rexp": ["the", "bird"]}
+ )
+
+ # test compatibility mode, no longer restoring pre-2.3.1 behavior
+ with ppt.reset_pyparsing_context():
+ pp.__compat__.collect_all_And_tokens = False
+ pp.enable_diag(pp.Diagnostics.warn_multiple_tokens_in_named_alternation)
+ expr_a = pp.Literal("not") + pp.Literal("the") + pp.Literal("bird")
+ expr_b = pp.Literal("the") + pp.Literal("bird")
+
+ with self.assertWarns(
+ UserWarning, msg="failed to warn of And within alternation"
+ ):
+ expr = (expr_a ^ expr_b)("rexp")
+
+ expr.runTests(
+ """\
+ not the bird
+ the bird
+ """
+ )
+ self.assertEqual(
+ "not the bird".split(), list(expr.parseString("not the bird")["rexp"])
+ )
+ self.assertEqual(
+ "the bird".split(), list(expr.parseString("the bird")["rexp"])
+ )
+
+ def testEmptyDictDoesNotRaiseException(self):
+ key = pp.Word(pp.alphas)
+ value = pp.Word(pp.nums)
+ EQ = pp.Suppress("=")
+ key_value_dict = pp.dictOf(key, EQ + value)
+
+ print(
+ key_value_dict.parseString(
+ """\
+ a = 10
+ b = 20
+ """
+ ).dump()
+ )
+
+ try:
+ print(key_value_dict.parseString("").dump())
+ except pp.ParseException as pe:
+ print(pp.ParseException.explain(pe))
+ else:
+ self.fail("failed to raise exception when matching empty string")
+
+ def testExplainException(self):
+ expr = pp.Word(pp.nums).setName("int") + pp.Word(pp.alphas).setName("word")
+ try:
+ expr.parseString("123 355")
+ except pp.ParseException as pe:
+ print(pe.explain(depth=0))
+
+ expr = pp.Word(pp.nums).setName("int") - pp.Word(pp.alphas).setName("word")
+ try:
+ expr.parseString("123 355 (test using ErrorStop)")
+ except pp.ParseSyntaxException as pe:
+ print(pe.explain())
+
+ integer = pp.Word(pp.nums).setName("int").addParseAction(lambda t: int(t[0]))
+ expr = integer + integer
+
+ def divide_args(t):
+ integer.parseString("A")
+ return t[0] / t[1]
+
+ expr.addParseAction(divide_args)
+ try:
+ expr.parseString("123 0")
+ except pp.ParseException as pe:
+ print(pe.explain())
+ except Exception as exc:
+ print(pp.ParseBaseException.explain_exception(exc))
+ raise
+
+ def testExplainExceptionWithMemoizationCheck(self):
+ if pp.ParserElement._left_recursion_enabled or pp.ParserElement._packratEnabled:
+ print("test does local memoization enable/disable during test")
+ return
+
+ pp.ParserElement.disable_memoization()
+
+ integer = pp.Word(pp.nums).setName("int").addParseAction(lambda t: int(t[0]))
+ expr = integer + integer
+
+ def divide_args(t):
+ integer.parseString("A")
+ return t[0] / t[1]
+
+ expr.addParseAction(divide_args)
+ for memo_kind, enable_memo in [
+ ("Packrat", pp.ParserElement.enablePackrat),
+ ("Left Recursion", pp.ParserElement.enable_left_recursion),
+ ]:
+ enable_memo(force=True)
+ print("Explain for", memo_kind)
+
+ try:
+ expr.parseString("123 0")
+ except pp.ParseException as pe:
+ print(pe.explain())
+ except Exception as exc:
+ print(pp.ParseBaseException.explain_exception(exc))
+ raise
+
+ # make sure we leave the state compatible with everything
+ pp.ParserElement.disable_memoization()
+
+ def testCaselessKeywordVsKeywordCaseless(self):
+ frule = pp.Keyword("t", caseless=True) + pp.Keyword("yes", caseless=True)
+ crule = pp.CaselessKeyword("t") + pp.CaselessKeyword("yes")
+
+ flist = frule.searchString("not yes").asList()
+ print(flist)
+ clist = crule.searchString("not yes").asList()
+ print(clist)
+ self.assertEqual(
+ flist,
+ clist,
+ "CaselessKeyword not working the same as Keyword(caseless=True)",
+ )
+
+ def testOneOfKeywords(self):
+ literal_expr = pp.oneOf("a b c")
+ success, _ = literal_expr[...].runTests(
+ """
+ # literal oneOf tests
+ a b c
+ a a a
+ abc
+ """
+ )
+ self.assertTrue(success, "failed literal oneOf matching")
+
+ keyword_expr = pp.oneOf("a b c", asKeyword=True)
+ success, _ = keyword_expr[...].runTests(
+ """
+ # keyword oneOf tests
+ a b c
+ a a a
+ """
+ )
+ self.assertTrue(success, "failed keyword oneOf matching")
+
+ success, _ = keyword_expr[...].runTests(
+ """
+ # keyword oneOf failure tests
+ abc
+ """,
+ failureTests=True,
+ )
+ self.assertTrue(success, "failed keyword oneOf failure tests")
+
+ def testWarnUngroupedNamedTokens(self):
+ """
+ - warn_ungrouped_named_tokens_in_collection - flag to enable warnings when a results
+ name is defined on a containing expression with ungrouped subexpressions that also
+ have results names (default=True)
+ """
+
+ with ppt.reset_pyparsing_context():
+ pp.enable_diag(pp.Diagnostics.warn_ungrouped_named_tokens_in_collection)
+
+ COMMA = pp.Suppress(",").setName("comma")
+ coord = ppc.integer("x") + COMMA + ppc.integer("y")
+
+ # this should emit a warning
+ with self.assertWarns(
+ UserWarning,
+ msg="failed to warn with named repetition of"
+ " ungrouped named expressions",
+ ):
+ path = coord[...].setResultsName("path")
+
+ def testWarnNameSetOnEmptyForward(self):
+ """
+ - warn_name_set_on_empty_Forward - flag to enable warnings when a Forward is defined
+ with a results name, but has no contents defined (default=False)
+ """
+
+ with ppt.reset_pyparsing_context():
+ pp.enable_diag(pp.Diagnostics.warn_name_set_on_empty_Forward)
+
+ base = pp.Forward()
+
+ with self.assertWarns(
+ UserWarning,
+ msg="failed to warn when naming an empty Forward expression",
+ ):
+ base("x")
+
+ def testWarnParsingEmptyForward(self):
+ """
+ - warn_on_parse_using_empty_Forward - flag to enable warnings when a Forward
+ has no contents defined (default=False)
+ """
+
+ with ppt.reset_pyparsing_context():
+ pp.enable_diag(pp.Diagnostics.warn_on_parse_using_empty_Forward)
+
+ base = pp.Forward()
+
+ with self.assertWarns(
+ UserWarning,
+ msg="failed to warn when naming an empty Forward expression",
+ ):
+ try:
+ print(base.parseString("x"))
+ except ParseException as pe:
+ pass
+
+ def testWarnIncorrectAssignmentToForward(self):
+ """
+ - warn_on_parse_using_empty_Forward - flag to enable warnings when a Forward
+ has no contents defined (default=False)
+ """
+ if PYPY_ENV:
+ print("warn_on_assignment_to_Forward not supported on PyPy")
+ return
+
+ with ppt.reset_pyparsing_context():
+ pp.enable_diag(pp.Diagnostics.warn_on_assignment_to_Forward)
+
+ def a_method():
+ base = pp.Forward()
+ base = pp.Word(pp.alphas)[...] | "(" + base + ")"
+
+ with self.assertWarns(
+ UserWarning,
+ msg="failed to warn when using '=' to assign expression to a Forward",
+ ):
+ a_method()
+
+ def testWarnOnMultipleStringArgsToOneOf(self):
+ """
+ - warn_on_multiple_string_args_to_oneof - flag to enable warnings when oneOf is
+ incorrectly called with multiple str arguments (default=True)
+ """
+
+ with ppt.reset_pyparsing_context():
+ pp.enable_diag(pp.Diagnostics.warn_on_multiple_string_args_to_oneof)
+
+ with self.assertWarns(
+ UserWarning,
+ msg="failed to warn when incorrectly calling oneOf(string, string)",
+ ):
+ a = pp.oneOf("A", "B")
+
+ def testAutonameElements(self):
+ with ppt.reset_pyparsing_context():
+ pp.enable_diag(pp.Diagnostics.enable_debug_on_named_expressions)
+
+ a = pp.Literal("a")
+ b = pp.Literal("b").set_name("bbb")
+ z = pp.Literal("z")
+ leading_a = a + pp.FollowedBy(z | a | b)
+
+ grammar = (z | leading_a | b)[...] + "a"
+
+ self.assertFalse(a.debug)
+ self.assertFalse(a.customName)
+ pp.autoname_elements()
+ self.assertTrue(a.debug)
+ self.assertEqual('a', a.name)
+ self.assertEqual('bbb', b.name)
+
+ def testEnableDebugOnNamedExpressions(self):
+ """
+ - enable_debug_on_named_expressions - flag to auto-enable debug on all subsequent
+ calls to ParserElement.setName() (default=False)
+ """
+ with ppt.reset_pyparsing_context():
+ test_stdout = StringIO()
+
+ with resetting(sys, "stdout", "stderr"):
+ sys.stdout = test_stdout
+ sys.stderr = test_stdout
+
+ pp.enable_diag(pp.Diagnostics.enable_debug_on_named_expressions)
+ integer = pp.Word(pp.nums).setName("integer")
+
+ integer[...].parseString("1 2 3")
+
+ expected_debug_output = dedent(
+ """\
+ Match integer at loc 0(1,1)
+ 1 2 3
+ ^
+ Matched integer -> ['1']
+ Match integer at loc 2(1,3)
+ 1 2 3
+ ^
+ Matched integer -> ['2']
+ Match integer at loc 4(1,5)
+ 1 2 3
+ ^
+ Matched integer -> ['3']
+ Match integer at loc 5(1,6)
+ 1 2 3
+ ^
+ Match integer failed, ParseException raised: Expected integer, found end of text (at char 5), (line:1, col:6)
+ """
+ )
+ output = test_stdout.getvalue()
+ print(output)
+ self.assertEqual(
+ expected_debug_output,
+ output,
+ "failed to auto-enable debug on named expressions "
+ "using enable_debug_on_named_expressions",
+ )
+
+ def testEnableDebugOnExpressionWithParseAction(self):
+
+ test_stdout = StringIO()
+ with resetting(sys, "stdout", "stderr"):
+ sys.stdout = test_stdout
+ sys.stderr = test_stdout
+
+ parser = (ppc.integer().setDebug() | pp.Word(pp.alphanums).setDebug())[...]
+ parser.setDebug()
+ parser.parseString("123 A100")
+
+ # now turn off debug - should only get output for components, not overall parser
+ print()
+ parser.setDebug(False)
+ parser.parseString("123 A100")
+
+ expected_debug_output = dedent(
+ """\
+ Match [{integer | W:(0-9A-Za-z)}]... at loc 0(1,1)
+ 123 A100
+ ^
+ Match integer at loc 0(1,1)
+ 123 A100
+ ^
+ Matched integer -> [123]
+ Match integer at loc 4(1,5)
+ 123 A100
+ ^
+ Match integer failed, ParseException raised: Expected integer, found 'A100' (at char 4), (line:1, col:5)
+ Match W:(0-9A-Za-z) at loc 4(1,5)
+ 123 A100
+ ^
+ Matched W:(0-9A-Za-z) -> ['A100']
+ Match integer at loc 8(1,9)
+ 123 A100
+ ^
+ Match integer failed, ParseException raised: Expected integer, found end of text (at char 8), (line:1, col:9)
+ Match W:(0-9A-Za-z) at loc 8(1,9)
+ 123 A100
+ ^
+ Match W:(0-9A-Za-z) failed, ParseException raised: Expected W:(0-9A-Za-z), found end of text (at char 8), (line:1, col:9)
+ Matched [{integer | W:(0-9A-Za-z)}]... -> [123, 'A100']
+
+ Match integer at loc 0(1,1)
+ 123 A100
+ ^
+ Matched integer -> [123]
+ Match integer at loc 4(1,5)
+ 123 A100
+ ^
+ Match integer failed, ParseException raised: Expected integer, found 'A100' (at char 4), (line:1, col:5)
+ Match W:(0-9A-Za-z) at loc 4(1,5)
+ 123 A100
+ ^
+ Matched W:(0-9A-Za-z) -> ['A100']
+ Match integer at loc 8(1,9)
+ 123 A100
+ ^
+ Match integer failed, ParseException raised: Expected integer, found end of text (at char 8), (line:1, col:9)
+ Match W:(0-9A-Za-z) at loc 8(1,9)
+ 123 A100
+ ^
+ Match W:(0-9A-Za-z) failed, ParseException raised: Expected W:(0-9A-Za-z), found end of text (at char 8), (line:1, col:9)
+ """
+ )
+ output = test_stdout.getvalue()
+ print(output)
+ self.assertEqual(
+ expected_debug_output,
+ output,
+ "invalid debug output when using parse action",
+ )
+
+ def testEnableDebugWithCachedExpressionsMarkedWithAsterisk(self):
+
+ a = pp.Literal("a").setName("A").setDebug()
+ b = pp.Literal("b").setName("B").setDebug()
+ z = pp.Literal("z").setName("Z").setDebug()
+ leading_a = a + pp.FollowedBy(z | a | b)
+ leading_a.setName("leading_a").setDebug()
+
+ grammar = (z | leading_a | b)[...] + "a"
+
+ # parse test string and capture debug output
+ test_stdout = StringIO()
+ with resetting(sys, "stdout", "stderr"):
+ sys.stdout = test_stdout
+ sys.stderr = test_stdout
+ grammar.parseString("aba")
+
+ expected_debug_output = dedent(
+ """\
+ Match Z at loc 0(1,1)
+ aba
+ ^
+ Match Z failed, ParseException raised: Expected Z, found 'aba' (at char 0), (line:1, col:1)
+ Match leading_a at loc 0(1,1)
+ aba
+ ^
+ Match A at loc 0(1,1)
+ aba
+ ^
+ Matched A -> ['a']
+ Match Z at loc 1(1,2)
+ aba
+ ^
+ Match Z failed, ParseException raised: Expected Z, found 'ba' (at char 1), (line:1, col:2)
+ Match A at loc 1(1,2)
+ aba
+ ^
+ Match A failed, ParseException raised: Expected A, found 'ba' (at char 1), (line:1, col:2)
+ Match B at loc 1(1,2)
+ aba
+ ^
+ Matched B -> ['b']
+ Matched leading_a -> ['a']
+ *Match Z at loc 1(1,2)
+ aba
+ ^
+ *Match Z failed, ParseException raised: Expected Z, found 'ba' (at char 1), (line:1, col:2)
+ Match leading_a at loc 1(1,2)
+ aba
+ ^
+ *Match A at loc 1(1,2)
+ aba
+ ^
+ *Match A failed, ParseException raised: Expected A, found 'ba' (at char 1), (line:1, col:2)
+ Match leading_a failed, ParseException raised: Expected A, found 'ba' (at char 1), (line:1, col:2)
+ *Match B at loc 1(1,2)
+ aba
+ ^
+ *Matched B -> ['b']
+ Match Z at loc 2(1,3)
+ aba
+ ^
+ Match Z failed, ParseException raised: Expected Z, found 'a' (at char 2), (line:1, col:3)
+ Match leading_a at loc 2(1,3)
+ aba
+ ^
+ Match A at loc 2(1,3)
+ aba
+ ^
+ Matched A -> ['a']
+ Match Z at loc 3(1,4)
+ aba
+ ^
+ Match Z failed, ParseException raised: Expected Z, found end of text (at char 3), (line:1, col:4)
+ Match A at loc 3(1,4)
+ aba
+ ^
+ Match A failed, ParseException raised: Expected A, found end of text (at char 3), (line:1, col:4)
+ Match B at loc 3(1,4)
+ aba
+ ^
+ Match B failed, ParseException raised: Expected B, found end of text (at char 3), (line:1, col:4)
+ Match leading_a failed, ParseException raised: Expected {Z | A | B}, found end of text (at char 3), (line:1, col:4)
+ Match B at loc 2(1,3)
+ aba
+ ^
+ Match B failed, ParseException raised: Expected B, found 'a' (at char 2), (line:1, col:3)
+ """
+ )
+ if pp.ParserElement._packratEnabled:
+ packrat_status = "enabled"
+ else:
+ # remove '*' cache markers from expected output
+ expected_debug_output = expected_debug_output.replace("*", "")
+ packrat_status = "disabled"
+ print("Packrat status:", packrat_status)
+
+ output = test_stdout.getvalue()
+ print(output)
+ self.assertEqual(
+ expected_debug_output,
+ output,
+ (
+ "invalid debug output showing cached results marked with '*',"
+ " and packrat parsing {}".format(packrat_status)
+ ),
+ )
+
+ def testUndesirableButCommonPractices(self):
+ # While these are valid constructs, and they are not encouraged
+ # there is apparently a lot of code out there using these
+ # coding styles.
+ #
+ # Even though they are not encouraged, we shouldn't break them.
+
+ # Create an And using a list of expressions instead of using '+' operator
+ expr = pp.And([pp.Word("abc"), pp.Word("123")])
+ expr.runTests(
+ """
+ aaa 333
+ b 1
+ ababab 32123
+ """
+ )
+
+ # Passing a single expression to a ParseExpression, when it really wants a sequence
+ expr = pp.Or(pp.Or(ppc.integer))
+ expr.runTests(
+ """
+ 123
+ 456
+ abc
+ """
+ )
+
+ def testEnableWarnDiags(self):
+ import pprint
+
+ def filtered_vars(var_dict):
+ dunders = [nm for nm in var_dict if nm.startswith("__")]
+ return {
+ k: v
+ for k, v in var_dict.items()
+ if isinstance(v, bool) and k not in dunders
+ }
+
+ pprint.pprint(filtered_vars(vars(pp.__diag__)), width=30)
+
+ warn_names = pp.__diag__._warning_names
+ other_names = pp.__diag__._debug_names
+
+ # make sure they are off by default
+ for diag_name in warn_names:
+ self.assertFalse(
+ getattr(pp.__diag__, diag_name),
+ "__diag__.{} not set to True".format(diag_name),
+ )
+
+ with ppt.reset_pyparsing_context():
+ # enable all warn_* diag_names
+ pp.enable_all_warnings()
+ pprint.pprint(filtered_vars(vars(pp.__diag__)), width=30)
+
+ # make sure they are on after being enabled
+ for diag_name in warn_names:
+ self.assertTrue(
+ getattr(pp.__diag__, diag_name),
+ "__diag__.{} not set to True".format(diag_name),
+ )
+
+ # non-warn diag_names must be enabled individually
+ for diag_name in other_names:
+ self.assertFalse(
+ getattr(pp.__diag__, diag_name),
+ "__diag__.{} not set to True".format(diag_name),
+ )
+
+ # make sure they are off after AutoReset
+ for diag_name in warn_names:
+ self.assertFalse(
+ getattr(pp.__diag__, diag_name),
+ "__diag__.{} not set to True".format(diag_name),
+ )
+
+ def testWordInternalReRangesKnownSets(self):
+ self.assertEqual(
+ "[!-~]+",
+ pp.Word(pp.printables).reString,
+ "failed to generate correct internal re",
+ )
+ self.assertEqual(
+ "[0-9A-Za-z]+",
+ pp.Word(pp.alphanums).reString,
+ "failed to generate correct internal re",
+ )
+ self.assertEqual(
+ "[!-~¡-ÿ]+",
+ pp.Word(pp.pyparsing_unicode.Latin1.printables).reString,
+ "failed to generate correct internal re",
+ )
+ self.assertEqual(
+ "[À-ÖØ-öø-ÿ]+",
+ pp.Word(pp.alphas8bit).reString,
+ "failed to generate correct internal re",
+ )
+
+ def testWordInternalReRanges(self):
+ import random
+
+ esc_chars = r"\^-]["
+ esc_chars2 = r"*+.?"
+
+ def esc_re_set_char(c):
+ return "\\" + c if c in esc_chars else c
+
+ def esc_re_set2_char(c):
+ return "\\" + c if c in esc_chars + esc_chars2 else c
+
+ for esc_char in esc_chars + esc_chars2:
+ # test escape char as first character in range
+ next_char = chr(ord(esc_char) + 1)
+ prev_char = chr(ord(esc_char) - 1)
+ esc_word = pp.Word(esc_char + next_char)
+ expected = r"[{}{}]+".format(
+ esc_re_set_char(esc_char),
+ esc_re_set_char(next_char),
+ )
+ print(
+ "Testing escape char: {} -> {} re: '{}')".format(
+ esc_char, esc_word, esc_word.reString
+ )
+ )
+ self.assertEqual(
+ expected, esc_word.reString, "failed to generate correct internal re"
+ )
+ test_string = "".join(
+ random.choice([esc_char, next_char]) for __ in range(16)
+ )
+ print(
+ "Match '{}' -> {}".format(
+ test_string, test_string == esc_word.parseString(test_string)[0]
+ )
+ )
+ self.assertEqual(
+ test_string,
+ esc_word.parseString(test_string)[0],
+ "Word using escaped range char failed to parse",
+ )
+
+ # test escape char as last character in range
+ esc_word = pp.Word(prev_char + esc_char)
+ expected = r"[{}{}]+".format(
+ esc_re_set_char(prev_char),
+ esc_re_set_char(esc_char),
+ )
+ print(
+ "Testing escape char: {} -> {} re: '{}')".format(
+ esc_char, esc_word, esc_word.reString
+ )
+ )
+ self.assertEqual(
+ expected, esc_word.reString, "failed to generate correct internal re"
+ )
+ test_string = "".join(
+ random.choice([esc_char, prev_char]) for __ in range(16)
+ )
+ print(
+ "Match '{}' -> {}".format(
+ test_string, test_string == esc_word.parseString(test_string)[0]
+ )
+ )
+ self.assertEqual(
+ test_string,
+ esc_word.parseString(test_string)[0],
+ "Word using escaped range char failed to parse",
+ )
+
+ # test escape char as first character in range
+ next_char = chr(ord(esc_char) + 1)
+ prev_char = chr(ord(esc_char) - 1)
+ esc_word = pp.Word(esc_char + next_char)
+ expected = r"[{}{}]+".format(
+ esc_re_set_char(esc_char),
+ esc_re_set_char(next_char),
+ )
+ print(
+ "Testing escape char: {} -> {} re: '{}')".format(
+ esc_char, esc_word, esc_word.reString
+ )
+ )
+ self.assertEqual(
+ expected, esc_word.reString, "failed to generate correct internal re"
+ )
+ test_string = "".join(
+ random.choice([esc_char, next_char]) for __ in range(16)
+ )
+ print(
+ "Match '{}' -> {}".format(
+ test_string, test_string == esc_word.parseString(test_string)[0]
+ )
+ )
+ self.assertEqual(
+ test_string,
+ esc_word.parseString(test_string)[0],
+ "Word using escaped range char failed to parse",
+ )
+
+ # test escape char as only character in range
+ esc_word = pp.Word(esc_char, pp.alphas.upper())
+ expected = r"{}[A-Z]*".format(esc_re_set2_char(esc_char))
+ print(
+ "Testing escape char: {} -> {} re: '{}')".format(
+ esc_char, esc_word, esc_word.reString
+ )
+ )
+ self.assertEqual(
+ expected, esc_word.reString, "failed to generate correct internal re"
+ )
+ test_string = esc_char + "".join(
+ random.choice(pp.alphas.upper()) for __ in range(16)
+ )
+ print(
+ "Match '{}' -> {}".format(
+ test_string, test_string == esc_word.parseString(test_string)[0]
+ )
+ )
+ self.assertEqual(
+ test_string,
+ esc_word.parseString(test_string)[0],
+ "Word using escaped range char failed to parse",
+ )
+
+ # test escape char as only character
+ esc_word = pp.Word(esc_char, pp.alphas.upper())
+ expected = r"{}[A-Z]*".format(re.escape(esc_char))
+ print(
+ "Testing escape char: {} -> {} re: '{}')".format(
+ esc_char, esc_word, esc_word.reString
+ )
+ )
+ self.assertEqual(
+ expected, esc_word.reString, "failed to generate correct internal re"
+ )
+ test_string = esc_char + "".join(
+ random.choice(pp.alphas.upper()) for __ in range(16)
+ )
+ print(
+ "Match '{}' -> {}".format(
+ test_string, test_string == esc_word.parseString(test_string)[0]
+ )
+ )
+ self.assertEqual(
+ test_string,
+ esc_word.parseString(test_string)[0],
+ "Word using escaped range char failed to parse",
+ )
+ print()
+
+ def testWordWithIdentChars(self):
+ ppu = pp.pyparsing_unicode
+
+ latin_identifier = pp.Word(pp.identchars, pp.identbodychars)("latin*")
+ japanese_identifier = pp.Word(
+ ppu.Japanese.identchars, ppu.Japanese.identbodychars
+ )("japanese*")
+ cjk_identifier = pp.Word(ppu.CJK.identchars, ppu.CJK.identbodychars)("cjk*")
+ greek_identifier = pp.Word(ppu.Greek.identchars, ppu.Greek.identbodychars)(
+ "greek*"
+ )
+ cyrillic_identifier = pp.Word(
+ ppu.Cyrillic.identchars, ppu.Cyrillic.identbodychars
+ )("cyrillic*")
+ thai_identifier = pp.Word(ppu.Thai.identchars, ppu.Thai.identbodychars)("thai*")
+ idents = (
+ latin_identifier
+ | japanese_identifier
+ | cjk_identifier # must follow japanese_identifier, since CJK is superset
+ | thai_identifier
+ | greek_identifier
+ | cyrillic_identifier
+ )
+
+ result = idents[...].parseString(
+ "abc_100 кириллицаx_10 日本語f_300 ไทยg_600 def_200 漢字y_300 한국어_中文c_400 Ελληνικάb_500"
+ )
+ self.assertParseResultsEquals(
+ result,
+ [
+ "abc_100",
+ "кириллицаx_10",
+ "日本語f_300",
+ "ไทยg_600",
+ "def_200",
+ "漢字y_300",
+ "한국어_中文c_400",
+ "Ελληνικάb_500",
+ ],
+ {
+ "cjk": ["한국어_中文c_400"],
+ "cyrillic": ["кириллицаx_10"],
+ "greek": ["Ελληνικάb_500"],
+ "japanese": ["日本語f_300", "漢字y_300"],
+ "latin": ["abc_100", "def_200"],
+ "thai": ["ไทยg_600"],
+ },
+ )
+
+ def testChainedTernaryOperator(self):
+ TERNARY_INFIX = pp.infixNotation(
+ ppc.integer, [(("?", ":"), 3, pp.opAssoc.LEFT)]
+ )
+ self.assertParseAndCheckList(
+ TERNARY_INFIX, "1?1:0?1:0", [[1, "?", 1, ":", 0, "?", 1, ":", 0]]
+ )
+
+ TERNARY_INFIX = pp.infixNotation(
+ ppc.integer, [(("?", ":"), 3, pp.opAssoc.RIGHT)]
+ )
+ self.assertParseAndCheckList(
+ TERNARY_INFIX, "1?1:0?1:0", [[1, "?", 1, ":", [0, "?", 1, ":", 0]]]
+ )
+
+ def testOneOfWithDuplicateSymbols(self):
+ # test making oneOf with duplicate symbols
+ print("verify oneOf handles duplicate symbols")
+ try:
+ test1 = pp.oneOf("a b c d a")
+ except RuntimeError:
+ self.fail(
+ "still have infinite loop in oneOf with duplicate symbols (string input)"
+ )
+
+ print("verify oneOf handles generator input")
+ try:
+ test1 = pp.oneOf(c for c in "a b c d a" if not c.isspace())
+ except RuntimeError:
+ self.fail(
+ "still have infinite loop in oneOf with duplicate symbols (generator input)"
+ )
+
+ print("verify oneOf handles list input")
+ try:
+ test1 = pp.oneOf("a b c d a".split())
+ except RuntimeError:
+ self.fail(
+ "still have infinite loop in oneOf with duplicate symbols (list input)"
+ )
+
+ print("verify oneOf handles set input")
+ try:
+ test1 = pp.oneOf(set("a b c d a"))
+ except RuntimeError:
+ self.fail(
+ "still have infinite loop in oneOf with duplicate symbols (set input)"
+ )
+
+ def testOneOfWithEmptyList(self):
+ """test oneOf helper function with an empty list as input"""
+
+ tst = []
+ result = pp.oneOf(tst)
+
+ expected = True
+ found = isinstance(result, pp.NoMatch)
+ self.assertEqual(expected, found)
+
+ def testOneOfWithUnexpectedInput(self):
+ """test oneOf with an input that isn't a string or iterable"""
+
+ with self.assertRaises(
+ TypeError, msg="failed to warn use of integer for oneOf"
+ ):
+ expr = pp.oneOf(6)
+
+ def testMatchFirstIteratesOverAllChoices(self):
+ # test MatchFirst bugfix
+ print("verify MatchFirst iterates properly")
+ results = pp.quotedString.parseString("'this is a single quoted string'")
+ self.assertTrue(
+ len(results) > 0, "MatchFirst error - not iterating over all choices"
+ )
+
+ def testStreamlineOfSubexpressions(self):
+ # verify streamline of subexpressions
+ print("verify proper streamline logic")
+ compound = pp.Literal("A") + "B" + "C" + "D"
+ self.assertEqual(2, len(compound.exprs), "bad test setup")
+ print(compound)
+ compound.streamline()
+ print(compound)
+ self.assertEqual(4, len(compound.exprs), "streamline not working")
+
+ def testOptionalWithResultsNameAndNoMatch(self):
+ # test for Optional with results name and no match
+ print("verify Optional's do not cause match failure if have results name")
+ testGrammar = pp.Literal("A") + pp.Optional("B")("gotB") + pp.Literal("C")
+ try:
+ testGrammar.parseString("ABC")
+ testGrammar.parseString("AC")
+ except pp.ParseException as pe:
+ print(pe.pstr, "->", pe)
+ self.fail("error in Optional matching of string %s" % pe.pstr)
+
+ def testReturnOfFurthestException(self):
+ # test return of furthest exception
+ testGrammar = (
+ pp.Literal("A") | (pp.Literal("B") + pp.Literal("C")) | pp.Literal("E")
+ )
+ try:
+ testGrammar.parseString("BC")
+ testGrammar.parseString("BD")
+ except pp.ParseException as pe:
+ print(pe.pstr, "->", pe)
+ self.assertEqual("BD", pe.pstr, "wrong test string failed to parse")
+ self.assertEqual(
+ 1, pe.loc, "error in Optional matching, pe.loc=" + str(pe.loc)
+ )
+ self.assertTrue(
+ "found 'D'" in str(pe), "wrong alternative raised exception"
+ )
+
+ def testValidateCorrectlyDetectsInvalidLeftRecursion(self):
+ # test validate
+ print("verify behavior of validate()")
+ if IRON_PYTHON_ENV:
+ print("disable this test under IronPython")
+ return
+
+ def testValidation(grmr, gnam, isValid):
+ try:
+ grmr.streamline()
+ grmr.validate()
+ self.assertTrue(isValid, "validate() accepted invalid grammar " + gnam)
+ except pp.RecursiveGrammarException as rge:
+ print(grmr)
+ print(rge)
+ self.assertFalse(isValid, "validate() rejected valid grammar " + gnam)
+
+ fwd = pp.Forward()
+ g1 = pp.OneOrMore((pp.Literal("A") + "B" + "C") | fwd)
+ g2 = ("C" + g1)[...]
+ fwd <<= pp.Group(g2)
+ testValidation(fwd, "fwd", isValid=True)
+
+ fwd2 = pp.Forward()
+ fwd2 <<= pp.Group("A" | fwd2)
+ testValidation(fwd2, "fwd2", isValid=False)
+
+ fwd3 = pp.Forward()
+ fwd3 <<= pp.Optional("A") + fwd3
+ testValidation(fwd3, "fwd3", isValid=False)
+
+ def testGetNameBehavior(self):
+ # test getName
+ print("verify behavior of getName()")
+ aaa = pp.Group(pp.Word("a")("A"))
+ bbb = pp.Group(pp.Word("b")("B"))
+ ccc = pp.Group(":" + pp.Word("c")("C"))
+ g1 = "XXX" + (aaa | bbb | ccc)[...]
+ teststring = "XXX b bb a bbb bbbb aa bbbbb :c bbbbbb aaa"
+ names = []
+ print(g1.parseString(teststring).dump())
+ for t in g1.parseString(teststring):
+ print(t, repr(t))
+ try:
+ names.append(t[0].getName())
+ except Exception:
+ try:
+ names.append(t.getName())
+ except Exception:
+ names.append(None)
+ print(teststring)
+ print(names)
+ self.assertEqual(
+ [None, "B", "B", "A", "B", "B", "A", "B", None, "B", "A"],
+ names,
+ "failure in getting names for tokens",
+ )
+
+ IF, AND, BUT = map(pp.Keyword, "if and but".split())
+ ident = ~(IF | AND | BUT) + pp.Word(pp.alphas)("non-key")
+ scanner = pp.OneOrMore(IF | AND | BUT | ident)
+
+ def getNameTester(s, l, t):
+ print(t, t.getName())
+
+ ident.addParseAction(getNameTester)
+ scanner.parseString("lsjd sldkjf IF Saslkj AND lsdjf")
+
+ # test ParseResults.get() method
+ print("verify behavior of ParseResults.get()")
+ # use sum() to merge separate groups into single ParseResults
+ res = sum(g1.parseString(teststring)[1:])
+ print(res.dump())
+ print(res.get("A", "A not found"))
+ print(res.get("D", "!D"))
+ self.assertEqual(
+ "aaa", res.get("A", "A not found"), "get on existing key failed"
+ )
+ self.assertEqual("!D", res.get("D", "!D"), "get on missing key failed")
+
+ def testOptionalBeyondEndOfString(self):
+ print("verify handling of Optional's beyond the end of string")
+ testGrammar = "A" + pp.Optional("B") + pp.Optional("C") + pp.Optional("D")
+ testGrammar.parseString("A")
+ testGrammar.parseString("AB")
+
+ def testCreateLiteralWithEmptyString(self):
+ # test creating Literal with empty string
+ print('verify non-fatal usage of Literal("")')
+ with self.assertRaises(
+ ValueError, msg="failed to warn use of empty string for Literal"
+ ):
+ e = pp.Literal("")
+
+ def testLineMethodSpecialCaseAtStart(self):
+ # test line() behavior when starting at 0 and the opening line is an \n
+ print("verify correct line() behavior when first line is empty string")
+ self.assertEqual(
+ "",
+ pp.line(0, "\nabc\ndef\n"),
+ "Error in line() with empty first line in text",
+ )
+ txt = "\nabc\ndef\n"
+ results = [pp.line(i, txt) for i in range(len(txt))]
+ self.assertEqual(
+ ["", "abc", "abc", "abc", "abc", "def", "def", "def", "def"],
+ results,
+ "Error in line() with empty first line in text",
+ )
+ txt = "abc\ndef\n"
+ results = [pp.line(i, txt) for i in range(len(txt))]
+ self.assertEqual(
+ ["abc", "abc", "abc", "abc", "def", "def", "def", "def"],
+ results,
+ "Error in line() with non-empty first line in text",
+ )
+
+ def testRepeatedTokensWhenPackratting(self):
+ # test bugfix with repeated tokens when packrat parsing enabled
+ print("verify behavior with repeated tokens when packrat parsing is enabled")
+ a = pp.Literal("a")
+ b = pp.Literal("b")
+ c = pp.Literal("c")
+
+ abb = a + b + b
+ abc = a + b + c
+ aba = a + b + a
+ grammar = abb | abc | aba
+
+ self.assertEqual(
+ "aba", "".join(grammar.parseString("aba")), "Packrat ABA failure!"
+ )
+
+ def testSetResultsNameWithOneOrMoreAndZeroOrMore(self):
+ print("verify behavior of setResultsName with OneOrMore and ZeroOrMore")
+ stmt = pp.Keyword("test")
+ print(stmt[...]("tests").parseString("test test").tests)
+ print(stmt[1, ...]("tests").parseString("test test").tests)
+ print(pp.Optional(stmt[1, ...]("tests")).parseString("test test").tests)
+ print(pp.Optional(stmt[1, ...])("tests").parseString("test test").tests)
+ print(
+ pp.Optional(pp.delimitedList(stmt))("tests").parseString("test,test").tests
+ )
+ self.assertEqual(
+ 2,
+ len(stmt[...]("tests").parseString("test test").tests),
+ "ZeroOrMore failure with setResultsName",
+ )
+ self.assertEqual(
+ 2,
+ len(stmt[1, ...]("tests").parseString("test test").tests),
+ "OneOrMore failure with setResultsName",
+ )
+ self.assertEqual(
+ 2,
+ len(pp.Optional(stmt[1, ...]("tests")).parseString("test test").tests),
+ "OneOrMore failure with setResultsName",
+ )
+ self.assertEqual(
+ 2,
+ len(
+ pp.Optional(pp.delimitedList(stmt))("tests")
+ .parseString("test,test")
+ .tests
+ ),
+ "delimitedList failure with setResultsName",
+ )
+ self.assertEqual(
+ 2,
+ len((stmt * 2)("tests").parseString("test test").tests),
+ "multiplied(1) failure with setResultsName",
+ )
+ self.assertEqual(
+ 2,
+ len(stmt[..., 2]("tests").parseString("test test").tests),
+ "multiplied(2) failure with setResultsName",
+ )
+ self.assertEqual(
+ 2,
+ len(stmt[1, ...]("tests").parseString("test test").tests),
+ "multiplied(3) failure with setResultsName",
+ )
+ self.assertEqual(
+ 2,
+ len(stmt[2, ...]("tests").parseString("test test").tests),
+ "multiplied(3) failure with setResultsName",
+ )
+
+ def testParseResultsReprWithResultsNames(self):
+ word = pp.Word(pp.printables)("word")
+ res = word[...].parseString("test blub")
+
+ print(repr(res))
+ print(res["word"])
+ print(res.asDict())
+
+ self.assertEqual(
+ "ParseResults(['test', 'blub'], {'word': 'blub'})",
+ repr(res),
+ "incorrect repr for ParseResults with listAllMatches=False",
+ )
+
+ word = pp.Word(pp.printables)("word*")
+ res = word[...].parseString("test blub")
+
+ print(repr(res))
+ print(res["word"])
+ print(res.asDict())
+
+ self.assertEqual(
+ "ParseResults(['test', 'blub'], {'word': ['test', 'blub']})",
+ repr(res),
+ "incorrect repr for ParseResults with listAllMatches=True",
+ )
+
+ def testWarnUsingLshiftForward(self):
+ import warnings
+
+ print(
+ "verify that using '<<' operator with a Forward raises a warning if there is a dangling '|' operator"
+ )
+
+ fwd = pp.Forward()
+ print("unsafe << and |, but diag not enabled, should not warn")
+ fwd << pp.Word("a") | pp.Word("b")
+
+ pp.enable_diag(pp.Diagnostics.warn_on_match_first_with_lshift_operator)
+ with self.assertWarns(
+ UserWarning, msg="failed to warn of using << and | operators"
+ ):
+ fwd = pp.Forward()
+ print("unsafe << and |, should warn")
+ fwd << pp.Word("a") | pp.Word("b")
+
+ with self.assertWarns(
+ UserWarning,
+ msg="failed to warn of using << and | operators (within lambda)",
+ ):
+ fwd = pp.Forward()
+ print("unsafe << and |, should warn")
+ fwd_fn = lambda expr1, expr2: fwd << expr1 | expr2
+ fwd_fn(pp.Word("a"), pp.Word("b"))
+
+ fwd = pp.Forward()
+ print("safe <<= and |, should not warn")
+ fwd <<= pp.Word("a") | pp.Word("b")
+ c = fwd | pp.Word("c")
+
+ print("safe << and (|), should not warn")
+ with warnings.catch_warnings(record=True) as w:
+ warnings.simplefilter("error")
+
+ fwd = pp.Forward()
+ fwd << (pp.Word("a") | pp.Word("b"))
+ try:
+ c = fwd | pp.Word("c")
+ except Exception as e:
+ self.fail("raised warning when it should not have")
+
+ def testParseExpressionsWithRegex(self):
+ from itertools import product
+
+ match_empty_regex = pp.Regex(r"[a-z]*")
+ match_nonempty_regex = pp.Regex(r"[a-z]+")
+
+ parser_classes = pp.ParseExpression.__subclasses__()
+ test_string = "abc def"
+ expected = ["abc"]
+ for expr, cls in product(
+ (match_nonempty_regex, match_empty_regex), parser_classes
+ ):
+ print(expr, cls)
+ parser = cls([expr])
+ parsed_result = parser.parseString(test_string)
+ print(parsed_result.dump())
+ self.assertParseResultsEquals(parsed_result, expected)
+
+ for expr, cls in product(
+ (match_nonempty_regex, match_empty_regex), (pp.MatchFirst, pp.Or)
+ ):
+ parser = cls([expr, expr])
+ print(parser)
+ parsed_result = parser.parseString(test_string)
+ print(parsed_result.dump())
+ self.assertParseResultsEquals(parsed_result, expected)
+
+ def testAssertParseAndCheckDict(self):
+ """test assertParseAndCheckDict in test framework"""
+
+ expr = pp.Word(pp.alphas)("item") + pp.Word(pp.nums)("qty")
+ self.assertParseAndCheckDict(
+ expr, "balloon 25", {"item": "balloon", "qty": "25"}
+ )
+
+ exprWithInt = pp.Word(pp.alphas)("item") + ppc.integer("qty")
+ self.assertParseAndCheckDict(
+ exprWithInt, "rucksack 49", {"item": "rucksack", "qty": 49}
+ )
+
+ def testOnlyOnce(self):
+ """test class OnlyOnce and its reset method"""
+
+ # use a parse action to compute the sum of the parsed integers,
+ # and add it to the end
+ def append_sum(tokens):
+ tokens.append(sum(map(int, tokens)))
+
+ pa = pp.OnlyOnce(append_sum)
+ expr = pp.OneOrMore(pp.Word(pp.nums)).addParseAction(pa)
+
+ result = expr.parseString("0 123 321")
+ print(result.dump())
+ expected = ["0", "123", "321", 444]
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with OnlyOnce first call"
+ )
+
+ with self.assertRaisesParseException(
+ msg="failed to raise exception calling OnlyOnce more than once"
+ ):
+ result2 = expr.parseString("1 2 3 4 5")
+
+ pa.reset()
+ result = expr.parseString("100 200 300")
+ print(result.dump())
+ expected = ["100", "200", "300", 600]
+ self.assertParseResultsEquals(
+ result, expected, msg="issue with OnlyOnce after reset"
+ )
+
+ def testGoToColumn(self):
+ """tests for GoToColumn class"""
+
+ dateExpr = pp.Regex(r"\d\d(\.\d\d){2}")("date")
+ numExpr = ppc.number("num")
+
+ sample = """\
+ date Not Important value NotImportant2
+ 11.11.13 | useless . useless,21 useless 2 | 14.21 | asmdakldm
+ 21.12.12 | fmpaosmfpoamsp 4 | 41 | ajfa9si90""".splitlines()
+
+ # Column number finds match
+ patt = dateExpr + pp.GoToColumn(70).ignore("|") + numExpr + pp.restOfLine
+
+ infile = iter(sample)
+ next(infile)
+
+ expecteds = [["11.11.13", 14.21], ["21.12.12", 41]]
+ for line, expected in zip(infile, expecteds):
+ result = patt.parseString(line)
+ print(result)
+
+ self.assertEqual(
+ expected, [result.date, result.num], msg="issue with GoToColumn"
+ )
+
+ # Column number does NOT match
+ patt = dateExpr("date") + pp.GoToColumn(30) + numExpr + pp.restOfLine
+
+ infile = iter(sample)
+ next(infile)
+
+ for line in infile:
+ with self.assertRaisesParseException(
+ msg="issue with GoToColumn not finding match"
+ ):
+ result = patt.parseString(line)
+
+ def testExceptionExplainVariations(self):
+ class Modifier:
+ def modify_upper(self, tokens):
+ tokens[:] = map(str.upper, tokens)
+
+ modder = Modifier()
+
+ # force an exception in the attached parse action
+ # integer has a parse action to convert to an int;
+ # this parse action should fail with a TypeError, since
+ # str.upper expects a str argument, not an int
+ grammar = ppc.integer().addParseAction(modder.modify_upper)
+
+ self_testcase_name = "tests.test_unit." + type(self).__name__
+
+ try:
+ grammar.parseString("1000")
+ except Exception as e:
+ # extract the exception explanation
+ explain_str = ParseException.explain_exception(e)
+ print(explain_str)
+ explain_str_lines = explain_str.splitlines()
+
+ expected = [
+ self_testcase_name,
+ "pyparsing.core._WordRegex - integer",
+ "tests.test_unit.Modifier",
+ "pyparsing.results.ParseResults",
+ ]
+
+ # verify the list of names shown in the explain "stack"
+ self.assertEqual(
+ expected, explain_str_lines[-len(expected) :], msg="invalid explain str"
+ )
+
+ # check type of raised exception matches explain output
+ # (actual exception text varies by Python version, and even
+ # by how the exception is raised, so we can only check the
+ # type name)
+ exception_line = explain_str_lines[-(len(expected) + 1)]
+ self.assertTrue(
+ exception_line.startswith("TypeError:"),
+ msg="unexpected exception line ({!r})".format(exception_line),
+ )
+
+ def testMiscellaneousExceptionBits(self):
+ pp.ParserElement.verbose_stacktrace = True
+
+ self_testcase_name = "tests.test_unit." + type(self).__name__
+
+ # force a parsing exception - match an integer against "ABC"
+ try:
+ pp.Word(pp.nums).parseString("ABC")
+ except pp.ParseException as pe:
+ expected_str = "Expected W:(0-9), found 'ABC' (at char 0), (line:1, col:1)"
+ self.assertEqual(expected_str, str(pe), "invalid ParseException str")
+ self.assertEqual(expected_str, repr(pe), "invalid ParseException repr")
+
+ self.assertEqual(
+ ">!<ABC", pe.markInputline(), "invalid default mark input line"
+ )
+ self.assertEqual(
+ "ABC", pe.markInputline(""), "invalid mark input line with '' marker"
+ )
+
+ # test explain using depth=None, 0, 1
+ depth_none_explain_str = pe.explain(depth=None)
+ depth_0_explain_str = pe.explain(depth=0)
+ depth_1_explain_str = pe.explain(depth=1)
+ print(depth_none_explain_str)
+
+ expr_name = "pyparsing.core._WordRegex - W:(0-9)"
+ for expected_function in [self_testcase_name, expr_name]:
+ self.assertTrue(
+ expected_function in depth_none_explain_str,
+ "{!r} not found in ParseException.explain()".format(
+ expected_function
+ ),
+ )
+ self.assertFalse(
+ expected_function in depth_0_explain_str,
+ "{!r} found in ParseException.explain(depth=0)".format(
+ expected_function
+ ),
+ )
+
+ self.assertTrue(
+ expr_name in depth_1_explain_str,
+ "{!r} not found in ParseException.explain()".format(expected_function),
+ )
+ self.assertFalse(
+ self_testcase_name in depth_1_explain_str,
+ "{!r} not found in ParseException.explain()".format(expected_function),
+ )
+
+ def testExpressionDefaultStrings(self):
+ expr = pp.Word(pp.nums)
+ print(expr)
+ self.assertEqual("W:(0-9)", repr(expr))
+
+ expr = pp.Word(pp.nums, exact=3)
+ print(expr)
+ self.assertEqual("W:(0-9){3}", repr(expr))
+
+ expr = pp.Word(pp.nums, min=2)
+ print(expr)
+ self.assertEqual("W:(0-9){2,...}", repr(expr))
+
+ expr = pp.Word(pp.nums, max=3)
+ print(expr)
+ self.assertEqual("W:(0-9){1,3}", repr(expr))
+
+ expr = pp.Word(pp.nums, min=2, max=3)
+ print(expr)
+ self.assertEqual("W:(0-9){2,3}", repr(expr))
+
+ expr = pp.Char(pp.nums)
+ print(expr)
+ self.assertEqual("(0-9)", repr(expr))
+
+ def testEmptyExpressionsAreHandledProperly(self):
+ from pyparsing.diagram import to_railroad
+ for cls in (pp.And, pp.Or, pp.MatchFirst, pp.Each):
+ print("testing empty", cls.__name__)
+ expr = cls([])
+ expr.streamline()
+ to_railroad(expr)
+
+
+class Test03_EnablePackratParsing(TestCase):
+ def runTest(self):
+ Test02_WithoutPackrat.suite_context.restore()
+
+ ParserElement.enablePackrat()
+
+ # SAVE A NEW SUITE CONTEXT
+ Test02_WithoutPackrat.suite_context = ppt.reset_pyparsing_context().save()
+
+
+class Test04_WithPackrat(Test02_WithoutPackrat):
+ """
+ rerun Test2 tests, now that packrat is enabled
+ """
+
+ def test000_assert_packrat_status(self):
+ print("Packrat enabled:", ParserElement._packratEnabled)
+ print(
+ "Packrat cache:",
+ type(ParserElement.packrat_cache).__name__,
+ getattr(ParserElement.packrat_cache, "size", "- no size attribute -"),
+ )
+ self.assertTrue(ParserElement._packratEnabled, "packrat not enabled")
+ self.assertEqual(
+ "_FifoCache",
+ type(ParserElement.packrat_cache).__name__,
+ msg="incorrect cache type",
+ )
+
+
+class Test05_EnableBoundedPackratParsing(TestCase):
+ def runTest(self):
+ Test02_WithoutPackrat.suite_context = Test02_WithoutPackrat.save_suite_context
+ Test02_WithoutPackrat.suite_context.restore()
+
+ ParserElement.enablePackrat(cache_size_limit=16)
+
+ # SAVE A NEW SUITE CONTEXT
+ Test02_WithoutPackrat.suite_context = ppt.reset_pyparsing_context().save()
+
+
+class Test06_WithBoundedPackrat(Test02_WithoutPackrat):
+ """
+ rerun Test2 tests, now with bounded packrat cache
+ """
+
+ def test000_assert_packrat_status(self):
+ print("Packrat enabled:", ParserElement._packratEnabled)
+ print(
+ "Packrat cache:",
+ type(ParserElement.packrat_cache).__name__,
+ getattr(ParserElement.packrat_cache, "size", "- no size attribute -"),
+ )
+ self.assertTrue(ParserElement._packratEnabled, "packrat not enabled")
+ self.assertEqual(
+ "_FifoCache",
+ type(ParserElement.packrat_cache).__name__,
+ msg="incorrect cache type",
+ )
+
+
+class Test07_EnableUnboundedPackratParsing(TestCase):
+ def runTest(self):
+ Test02_WithoutPackrat.suite_context = Test02_WithoutPackrat.save_suite_context
+ Test02_WithoutPackrat.suite_context.restore()
+
+ ParserElement.enablePackrat(cache_size_limit=None)
+
+ # SAVE A NEW SUITE CONTEXT
+ Test02_WithoutPackrat.suite_context = ppt.reset_pyparsing_context().save()
+
+
+class Test08_WithUnboundedPackrat(Test02_WithoutPackrat):
+ """
+ rerun Test2 tests, now with unbounded packrat cache
+ """
+
+ def test000_assert_packrat_status(self):
+ print("Packrat enabled:", ParserElement._packratEnabled)
+ print(
+ "Packrat cache:",
+ type(ParserElement.packrat_cache).__name__,
+ getattr(ParserElement.packrat_cache, "size", "- no size attribute -"),
+ )
+ self.assertTrue(ParserElement._packratEnabled, "packrat not enabled")
+ self.assertEqual(
+ "_UnboundedCache",
+ type(ParserElement.packrat_cache).__name__,
+ msg="incorrect cache type",
+ )
+
+
+class Test09_WithLeftRecursionParsing(Test02_WithoutPackrat):
+ """
+ rerun Test2 tests, now with unbounded left recursion cache
+ """
+
+ def setUp(self):
+ ParserElement.enable_left_recursion(force=True)
+
+ def tearDown(self):
+ default_suite_context.restore()
+
+ def test000_assert_packrat_status(self):
+ print("Left-Recursion enabled:", ParserElement._left_recursion_enabled)
+ self.assertTrue(
+ ParserElement._left_recursion_enabled, "left recursion not enabled"
+ )
+ self.assertIsInstance(ParserElement.recursion_memos, pp.util.UnboundedMemo)
+
+
+class Test10_WithLeftRecursionParsingBoundedMemo(Test02_WithoutPackrat):
+ """
+ rerun Test2 tests, now with bounded left recursion cache
+ """
+
+ def setUp(self):
+ ParserElement.enable_left_recursion(cache_size_limit=4, force=True)
+
+ def tearDown(self):
+ default_suite_context.restore()
+
+ def test000_assert_packrat_status(self):
+ print("Left-Recursion enabled:", ParserElement._left_recursion_enabled)
+ self.assertTrue(
+ ParserElement._left_recursion_enabled, "left recursion not enabled"
+ )
+ self.assertIsInstance(ParserElement.recursion_memos, pp.util.LRUMemo)
+ # check that the cache matches roughly what we expect
+ # – it may be larger due to action handling
+ self.assertLessEqual(ParserElement.recursion_memos._capacity, 4)
+ self.assertGreater(ParserElement.recursion_memos._capacity * 3, 4)
+
+
+class Test11_LR1_Recursion(ppt.TestParseResultsAsserts, TestCase):
+ """
+ Tests for recursive parsing
+ """
+
+ suite_context = None
+ save_suite_context = None
+
+ def setUp(self):
+ recursion_suite_context.restore()
+
+ def tearDown(self):
+ default_suite_context.restore()
+
+ def test_repeat_as_recurse(self):
+ """repetition rules formulated with recursion"""
+ one_or_more = pp.Forward().setName("one_or_more")
+ one_or_more <<= one_or_more + "a" | "a"
+ self.assertParseResultsEquals(one_or_more.parseString("a"), expected_list=["a"])
+ self.assertParseResultsEquals(
+ one_or_more.parseString("aaa aa"), expected_list=["a", "a", "a", "a", "a"]
+ )
+ delimited_list = pp.Forward().setName("delimited_list")
+ delimited_list <<= delimited_list + pp.Suppress(",") + "b" | "b"
+ self.assertParseResultsEquals(
+ delimited_list.parseString("b"), expected_list=["b"]
+ )
+ self.assertParseResultsEquals(
+ delimited_list.parseString("b,b"), expected_list=["b", "b"]
+ )
+ self.assertParseResultsEquals(
+ delimited_list.parseString("b,b , b, b,b"),
+ expected_list=["b", "b", "b", "b", "b"],
+ )
+
+ def test_binary_recursive(self):
+ """parsing of single left-recursive binary operator"""
+ expr = pp.Forward().setName("expr")
+ num = pp.Word(pp.nums)
+ expr <<= expr + "+" - num | num
+ self.assertParseResultsEquals(
+ expr.parseString("1+2"), expected_list=["1", "+", "2"]
+ )
+ self.assertParseResultsEquals(
+ expr.parseString("1+2+3+4"),
+ expected_list=["1", "+", "2", "+", "3", "+", "4"],
+ )
+
+ def test_binary_associative(self):
+ """associative is preserved for single left-recursive binary operator"""
+ expr = pp.Forward().setName("expr")
+ num = pp.Word(pp.nums)
+ expr <<= pp.Group(expr) + "+" - num | num
+ self.assertParseResultsEquals(
+ expr.parseString("1+2"), expected_list=[["1"], "+", "2"]
+ )
+ self.assertParseResultsEquals(
+ expr.parseString("1+2+3+4"),
+ expected_list=[[[["1"], "+", "2"], "+", "3"], "+", "4"],
+ )
+
+ def test_add_sub(self):
+ """indirectly left-recursive/associative add/sub calculator"""
+ expr = pp.Forward().setName("expr")
+ num = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
+ expr <<= (
+ (expr + "+" - num).setParseAction(lambda t: t[0] + t[2])
+ | (expr + "-" - num).setParseAction(lambda t: t[0] - t[2])
+ | num
+ )
+ self.assertEqual(expr.parseString("1+2")[0], 3)
+ self.assertEqual(expr.parseString("1+2+3")[0], 6)
+ self.assertEqual(expr.parseString("1+2-3")[0], 0)
+ self.assertEqual(expr.parseString("1-2+3")[0], 2)
+ self.assertEqual(expr.parseString("1-2-3")[0], -4)
+
+ def test_math(self):
+ """precedence climbing parser for math"""
+ # named references
+ expr = pp.Forward().setName("expr")
+ add_sub = pp.Forward().setName("add_sub")
+ mul_div = pp.Forward().setName("mul_div")
+ power = pp.Forward().setName("power")
+ terminal = pp.Forward().setName("terminal")
+ # concrete rules
+ number = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
+ signed = ("+" - expr) | ("-" - expr).setParseAction(lambda t: -t[1])
+ group = pp.Suppress("(") - expr - pp.Suppress(")")
+ add_sub <<= (
+ (add_sub + "+" - mul_div).setParseAction(lambda t: t[0] + t[2])
+ | (add_sub + "-" - mul_div).setParseAction(lambda t: t[0] - t[2])
+ | mul_div
+ )
+ mul_div <<= (
+ (mul_div + "*" - power).setParseAction(lambda t: t[0] * t[2])
+ | (mul_div + "/" - power).setParseAction(lambda t: t[0] / t[2])
+ | power
+ )
+ power <<= (terminal + "^" - power).setParseAction(
+ lambda t: t[0] ** t[2]
+ ) | terminal
+ terminal <<= number | signed | group
+ expr <<= add_sub
+ # simple add_sub expressions
+ self.assertEqual(expr.parseString("1+2")[0], 3)
+ self.assertEqual(expr.parseString("1+2+3")[0], 6)
+ self.assertEqual(expr.parseString("1+2-3")[0], 0)
+ self.assertEqual(expr.parseString("1-2+3")[0], 2)
+ self.assertEqual(expr.parseString("1-2-3")[0], -4)
+ # precedence overwriting via parentheses
+ self.assertEqual(expr.parseString("1+(2+3)")[0], 6)
+ self.assertEqual(expr.parseString("1+(2-3)")[0], 0)
+ self.assertEqual(expr.parseString("1-(2+3)")[0], -4)
+ self.assertEqual(expr.parseString("1-(2-3)")[0], 2)
+ # complicated math expressions – same as Python expressions
+ self.assertEqual(expr.parseString("1----3")[0], 1 - ---3)
+ self.assertEqual(expr.parseString("1+2*3")[0], 1 + 2 * 3)
+ self.assertEqual(expr.parseString("1*2+3")[0], 1 * 2 + 3)
+ self.assertEqual(expr.parseString("1*2^3")[0], 1 * 2 ** 3)
+ self.assertEqual(expr.parseString("4^3^2^1")[0], 4 ** 3 ** 2 ** 1)
+
+ def test_terminate_empty(self):
+ """Recursion with ``Empty`` terminates"""
+ empty = pp.Forward().setName("e")
+ empty <<= empty + pp.Empty() | pp.Empty()
+ self.assertParseResultsEquals(empty.parseString(""), expected_list=[])
+
+ def test_non_peg(self):
+ """Recursion works for non-PEG operators"""
+ expr = pp.Forward().setName("expr")
+ expr <<= expr + "a" ^ expr + "ab" ^ expr + "abc" ^ "."
+ self.assertParseResultsEquals(
+ expr.parseString(".abcabaabc"), expected_list=[".", "abc", "ab", "a", "abc"]
+ )
+
+
+# force clear of packrat parsing flags before saving contexts
+pp.ParserElement._packratEnabled = False
+pp.ParserElement._parse = pp.ParserElement._parseNoCache
+
+Test02_WithoutPackrat.suite_context = ppt.reset_pyparsing_context().save()
+Test02_WithoutPackrat.save_suite_context = ppt.reset_pyparsing_context().save()
+
+default_suite_context = ppt.reset_pyparsing_context().save()
+pp.ParserElement.enable_left_recursion()
+recursion_suite_context = ppt.reset_pyparsing_context().save()
+default_suite_context.restore()
--- /dev/null
+[tox]
+envlist =
+ py{36,37,38,39,310,py3}
+
+[testenv]
+deps=coverage
+extras=diagrams
+commands=
+ coverage run --parallel --branch -m unittest
+
+++ /dev/null
-# -*- coding: utf-8 -*-
-#
-# unitTests.py
-#
-# Unit tests for pyparsing module
-#
-# Copyright 2002-2018, Paul McGuire
-#
-#
-from __future__ import division
-
-from unittest import TestCase, TestSuite, TextTestRunner
-import datetime
-from pyparsing import ParseException, pyparsing_test as ppt
-import pyparsing as pp
-
-import sys
-
-PY_3 = sys.version.startswith('3')
-if PY_3:
- import builtins
- print_ = getattr(builtins, "print")
-
- # catch calls to builtin print(), should be print_
- def printX(*args, **kwargs):
- raise Exception("Test coding error: using print() directly, should use print_()")
- globals()['print'] = printX
-
- from io import StringIO
-else:
- def _print(*args, **kwargs):
- if 'end' in kwargs:
- sys.stdout.write(' '.join(map(str,args)) + kwargs['end'])
- else:
- sys.stdout.write(' '.join(map(str,args)) + '\n')
- print_ = _print
- from cStringIO import StringIO
-
-
-# see which Python implementation we are running
-CPYTHON_ENV = (sys.platform == "win32")
-IRON_PYTHON_ENV = (sys.platform == "cli")
-JYTHON_ENV = sys.platform.startswith("java")
-
-TEST_USING_PACKRAT = True
-#~ TEST_USING_PACKRAT = False
-
-VERBOSE = True
-
-# simple utility for flattening nested lists
-def flatten(L):
- if type(L) is not list: return [L]
- if L == []: return L
- return flatten(L[0]) + flatten(L[1:])
-
-"""
-class ParseTest(TestCase):
- def setUp(self):
- pass
-
- def runTest(self):
- self.assertTrue(1==1, "we've got bigger problems...")
-
- def tearDown(self):
- pass
-"""
-
-class AutoReset(object):
- def __init__(self, *args):
- ob = args[0]
- attrnames = args[1:]
- self.ob = ob
- self.save_attrs = attrnames
- self.save_values = [getattr(ob, attrname) for attrname in attrnames]
-
- def __enter__(self):
- pass
-
- def __exit__(self, *args):
- for attr, value in zip(self.save_attrs, self.save_values):
- setattr(self.ob, attr, value)
-
-BUFFER_OUTPUT = True
-
-class ParseTestCase(ppt.TestParseResultsAsserts, TestCase):
- def __init__(self):
- super(ParseTestCase, self).__init__(methodName='_runTest')
- self.expect_traceback = False
- self.expect_warning = False
-
- def _runTest(self):
-
- buffered_stdout = StringIO()
-
- try:
- with AutoReset(sys, 'stdout', 'stderr'):
- try:
- if BUFFER_OUTPUT:
- sys.stdout = buffered_stdout
- sys.stderr = buffered_stdout
- print_(">>>> Starting test",str(self))
- with ppt.reset_pyparsing_context():
- self.runTest()
-
- finally:
- print_("<<<< End of test",str(self))
- print_()
-
- output = buffered_stdout.getvalue()
- if "Traceback" in output and not self.expect_traceback:
- raise Exception("traceback in stdout")
- if "Warning" in output and not self.expect_warning:
- raise Exception("warning in stdout")
-
- except Exception as exc:
- if BUFFER_OUTPUT:
- print_()
- print_(buffered_stdout.getvalue())
- raise
-
- def runTest(self):
- pass
-
- def __str__(self):
- return self.__class__.__name__
-
-class PyparsingTestInit(ParseTestCase):
- def setUp(self):
- from pyparsing import __version__ as pyparsingVersion, __versionTime__ as pyparsingVersionTime
- print_("Beginning test of pyparsing, version", pyparsingVersion, pyparsingVersionTime)
- print_("Python version", sys.version)
- def tearDown(self):
- pass
-
-
-class ParseFourFnTest(ParseTestCase):
- def runTest(self):
- import examples.fourFn as fourFn
- def test(s,ans):
- fourFn.exprStack = []
- results = fourFn.BNF().parseString( s )
- resultValue = fourFn.evaluateStack( fourFn.exprStack )
- self.assertTrue(resultValue == ans, "failed to evaluate %s, got %f" % ( s, resultValue ))
- print_(s, "->", resultValue)
-
- from math import pi,exp
- e = exp(1)
-
- test( "9", 9 )
- test( "9 + 3 + 6", 18 )
- test( "9 + 3 / 11", 9.0+3.0/11.0)
- test( "(9 + 3)", 12 )
- test( "(9+3) / 11", (9.0+3.0)/11.0 )
- test( "9 - (12 - 6)", 3)
- test( "2*3.14159", 6.28318)
- test( "3.1415926535*3.1415926535 / 10", 3.1415926535*3.1415926535/10.0 )
- test( "PI * PI / 10", pi*pi/10.0 )
- test( "PI*PI/10", pi*pi/10.0 )
- test( "6.02E23 * 8.048", 6.02E23 * 8.048 )
- test( "e / 3", e/3.0 )
- test( "sin(PI/2)", 1.0 )
- test( "trunc(E)", 2.0 )
- test( "E^PI", e**pi )
- test( "2^3^2", 2**3**2)
- test( "2^3+2", 2**3+2)
- test( "2^9", 2**9 )
- test( "sgn(-2)", -1 )
- test( "sgn(0)", 0 )
- test( "sgn(0.1)", 1 )
-
-class ParseSQLTest(ParseTestCase):
- def runTest(self):
- import examples.simpleSQL as simpleSQL
-
- def test(s, numToks, errloc=-1):
- try:
- sqlToks = flatten(simpleSQL.simpleSQL.parseString(s).asList())
- print_(s,sqlToks,len(sqlToks))
- self.assertEqual(len(sqlToks), numToks,
- "invalid parsed tokens, expected {0}, found {1} ({2})".format(numToks,
- len(sqlToks),
- sqlToks))
- except ParseException as e:
- if errloc >= 0:
- self.assertEqual(e.loc, errloc, "expected error at {0}, found at {1}".format(errloc, e.loc))
-
- test( "SELECT * from XYZZY, ABC", 6 )
- test( "select * from SYS.XYZZY", 5 )
- test( "Select A from Sys.dual", 5 )
- test( "Select A,B,C from Sys.dual", 7 )
- test( "Select A, B, C from Sys.dual", 7 )
- test( "Select A, B, C from Sys.dual, Table2 ", 8 )
- test( "Xelect A, B, C from Sys.dual", 0, 0 )
- test( "Select A, B, C frox Sys.dual", 0, 15 )
- test( "Select", 0, 6 )
- test( "Select &&& frox Sys.dual", 0, 7 )
- test( "Select A from Sys.dual where a in ('RED','GREEN','BLUE')", 12 )
- test( "Select A from Sys.dual where a in ('RED','GREEN','BLUE') and b in (10,20,30)", 20 )
- test( "Select A,b from table1,table2 where table1.id eq table2.id -- test out comparison operators", 10 )
-
-class ParseConfigFileTest(ParseTestCase):
- def runTest(self):
- from examples import configParse
-
- def test(fnam,numToks,resCheckList):
- print_("Parsing",fnam,"...", end=' ')
- with open(fnam) as infile:
- iniFileLines = "\n".join(infile.read().splitlines())
- iniData = configParse.inifile_BNF().parseString( iniFileLines )
- print_(len(flatten(iniData.asList())))
- print_(list(iniData.keys()))
- self.assertEqual(len(flatten(iniData.asList())), numToks, "file %s not parsed correctly" % fnam)
- for chk in resCheckList:
- var = iniData
- for attr in chk[0].split('.'):
- var = getattr(var, attr)
- print_(chk[0], var, chk[1])
- self.assertEqual(var, chk[1],
- "ParseConfigFileTest: failed to parse ini {0!r} as expected {1}, found {2}".format(chk[0],
- chk[1],
- var))
- print_("OK")
-
- test("test/karthik.ini", 23,
- [ ("users.K","8"),
- ("users.mod_scheme","'QPSK'"),
- ("users.Na", "K+2") ]
- )
- test("examples/Setup.ini", 125,
- [ ("Startup.audioinf", "M3i"),
- ("Languages.key1", "0x0003"),
- ("test.foo","bar") ] )
-
-class ParseJSONDataTest(ParseTestCase):
- def runTest(self):
- from examples.jsonParser import jsonObject
- from test.jsonParserTests import test1,test2,test3,test4,test5
-
- expected = [
- [['glossary',
- [['title', 'example glossary'],
- ['GlossDiv',
- [['title', 'S'],
- ['GlossList',
- [[['ID', 'SGML'],
- ['SortAs', 'SGML'],
- ['GlossTerm', 'Standard Generalized Markup Language'],
- ['Acronym', 'SGML'],
- ['LargestPrimeLessThan100', 97],
- ['AvogadroNumber', 6.02e+23],
- ['EvenPrimesGreaterThan2', None],
- ['PrimesLessThan10', [2, 3, 5, 7]],
- ['WMDsFound', False],
- ['IraqAlQaedaConnections', None],
- ['Abbrev', 'ISO 8879:1986'],
- ['GlossDef',
- 'A meta-markup language, used to create markup languages such as '
- 'DocBook.'],
- ['GlossSeeAlso', ['GML', 'XML', 'markup']],
- ['EmptyDict', []],
- ['EmptyList', [[]]]]]]]]]
- ]]
- ,
- [['menu',
- [['id', 'file'],
- ['value', 'File:'],
- ['popup',
- [['menuitem',
- [[['value', 'New'], ['onclick', 'CreateNewDoc()']],
- [['value', 'Open'], ['onclick', 'OpenDoc()']],
- [['value', 'Close'], ['onclick', 'CloseDoc()']]]]]]]]]
- ,
- [['widget',
- [['debug', 'on'],
- ['window',
- [['title', 'Sample Konfabulator Widget'],
- ['name', 'main_window'],
- ['width', 500],
- ['height', 500]]],
- ['image',
- [['src', 'Images/Sun.png'],
- ['name', 'sun1'],
- ['hOffset', 250],
- ['vOffset', 250],
- ['alignment', 'center']]],
- ['text',
- [['data', 'Click Here'],
- ['size', 36],
- ['style', 'bold'],
- ['name', 'text1'],
- ['hOffset', 250],
- ['vOffset', 100],
- ['alignment', 'center'],
- ['onMouseUp', 'sun1.opacity = (sun1.opacity / 100) * 90;']]]]]]
- ,
- [['web-app',
- [['servlet',
- [[['servlet-name', 'cofaxCDS'],
- ['servlet-class', 'org.cofax.cds.CDSServlet'],
- ['init-param',
- [['configGlossary:installationAt', 'Philadelphia, PA'],
- ['configGlossary:adminEmail', 'ksm@pobox.com'],
- ['configGlossary:poweredBy', 'Cofax'],
- ['configGlossary:poweredByIcon', '/images/cofax.gif'],
- ['configGlossary:staticPath', '/content/static'],
- ['templateProcessorClass', 'org.cofax.WysiwygTemplate'],
- ['templateLoaderClass', 'org.cofax.FilesTemplateLoader'],
- ['templatePath', 'templates'],
- ['templateOverridePath', ''],
- ['defaultListTemplate', 'listTemplate.htm'],
- ['defaultFileTemplate', 'articleTemplate.htm'],
- ['useJSP', False],
- ['jspListTemplate', 'listTemplate.jsp'],
- ['jspFileTemplate', 'articleTemplate.jsp'],
- ['cachePackageTagsTrack', 200],
- ['cachePackageTagsStore', 200],
- ['cachePackageTagsRefresh', 60],
- ['cacheTemplatesTrack', 100],
- ['cacheTemplatesStore', 50],
- ['cacheTemplatesRefresh', 15],
- ['cachePagesTrack', 200],
- ['cachePagesStore', 100],
- ['cachePagesRefresh', 10],
- ['cachePagesDirtyRead', 10],
- ['searchEngineListTemplate', 'forSearchEnginesList.htm'],
- ['searchEngineFileTemplate', 'forSearchEngines.htm'],
- ['searchEngineRobotsDb', 'WEB-INF/robots.db'],
- ['useDataStore', True],
- ['dataStoreClass', 'org.cofax.SqlDataStore'],
- ['redirectionClass', 'org.cofax.SqlRedirection'],
- ['dataStoreName', 'cofax'],
- ['dataStoreDriver', 'com.microsoft.jdbc.sqlserver.SQLServerDriver'],
- ['dataStoreUrl',
- 'jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon'],
- ['dataStoreUser', 'sa'],
- ['dataStorePassword', 'dataStoreTestQuery'],
- ['dataStoreTestQuery', "SET NOCOUNT ON;select test='test';"],
- ['dataStoreLogFile', '/usr/local/tomcat/logs/datastore.log'],
- ['dataStoreInitConns', 10],
- ['dataStoreMaxConns', 100],
- ['dataStoreConnUsageLimit', 100],
- ['dataStoreLogLevel', 'debug'],
- ['maxUrlLength', 500]]]],
- [['servlet-name', 'cofaxEmail'],
- ['servlet-class', 'org.cofax.cds.EmailServlet'],
- ['init-param', [['mailHost', 'mail1'], ['mailHostOverride', 'mail2']]]],
- [['servlet-name', 'cofaxAdmin'],
- ['servlet-class', 'org.cofax.cds.AdminServlet']],
- [['servlet-name', 'fileServlet'],
- ['servlet-class', 'org.cofax.cds.FileServlet']],
- [['servlet-name', 'cofaxTools'],
- ['servlet-class', 'org.cofax.cms.CofaxToolsServlet'],
- ['init-param',
- [['templatePath', 'toolstemplates/'],
- ['log', 1],
- ['logLocation', '/usr/local/tomcat/logs/CofaxTools.log'],
- ['logMaxSize', ''],
- ['dataLog', 1],
- ['dataLogLocation', '/usr/local/tomcat/logs/dataLog.log'],
- ['dataLogMaxSize', ''],
- ['removePageCache', '/content/admin/remove?cache=pages&id='],
- ['removeTemplateCache', '/content/admin/remove?cache=templates&id='],
- ['fileTransferFolder',
- '/usr/local/tomcat/webapps/content/fileTransferFolder'],
- ['lookInContext', 1],
- ['adminGroupID', 4],
- ['betaServer', True]]]]]],
- ['servlet-mapping',
- [['cofaxCDS', '/'],
- ['cofaxEmail', '/cofaxutil/aemail/*'],
- ['cofaxAdmin', '/admin/*'],
- ['fileServlet', '/static/*'],
- ['cofaxTools', '/tools/*']]],
- ['taglib',
- [['taglib-uri', 'cofax.tld'],
- ['taglib-location', '/WEB-INF/tlds/cofax.tld']]]]]]
- ,
- [['menu',
- [['header', 'SVG Viewer'],
- ['items',
- [[['id', 'Open']],
- [['id', 'OpenNew'], ['label', 'Open New']],
- None,
- [['id', 'ZoomIn'], ['label', 'Zoom In']],
- [['id', 'ZoomOut'], ['label', 'Zoom Out']],
- [['id', 'OriginalView'], ['label', 'Original View']],
- None,
- [['id', 'Quality']],
- [['id', 'Pause']],
- [['id', 'Mute']],
- None,
- [['id', 'Find'], ['label', 'Find...']],
- [['id', 'FindAgain'], ['label', 'Find Again']],
- [['id', 'Copy']],
- [['id', 'CopyAgain'], ['label', 'Copy Again']],
- [['id', 'CopySVG'], ['label', 'Copy SVG']],
- [['id', 'ViewSVG'], ['label', 'View SVG']],
- [['id', 'ViewSource'], ['label', 'View Source']],
- [['id', 'SaveAs'], ['label', 'Save As']],
- None,
- [['id', 'Help']],
- [['id', 'About'], ['label', 'About Adobe CVG Viewer...']]]]]]]
- ,
- ]
-
- for t, exp in zip((test1,test2,test3,test4,test5), expected):
- result = jsonObject.parseString(t)
- result.pprint()
- self.assertEqual(result.asList(), exp, "failed test {0}".format(t))
-
-class ParseCommaSeparatedValuesTest(ParseTestCase):
- def runTest(self):
- from pyparsing import commaSeparatedList
-
- testData = [
- "a,b,c,100.2,,3",
- "d, e, j k , m ",
- "'Hello, World', f, g , , 5.1,x",
- "John Doe, 123 Main St., Cleveland, Ohio",
- "Jane Doe, 456 St. James St., Los Angeles , California ",
- "",
- ]
- testVals = [
- [ (3,'100.2'), (4,''), (5, '3') ],
- [ (2, 'j k'), (3, 'm') ],
- [ (0, "'Hello, World'"), (2, 'g'), (3, '') ],
- [ (0,'John Doe'), (1, '123 Main St.'), (2, 'Cleveland'), (3, 'Ohio') ],
- [ (0,'Jane Doe'), (1, '456 St. James St.'), (2, 'Los Angeles'), (3, 'California') ]
- ]
- for line,tests in zip(testData, testVals):
- print_("Parsing: \""+line+"\" ->", end=' ')
- results = commaSeparatedList.parseString(line)
- print_(results.asList())
- for t in tests:
- if not(len(results)>t[0] and results[t[0]] == t[1]):
- print_("$$$", results.dump())
- print_("$$$", results[0])
- self.assertTrue(len(results)>t[0] and results[t[0]] == t[1],
- "failed on %s, item %d s/b '%s', got '%s'" % (line, t[0], t[1], str(results.asList())))
-
-class ParseEBNFTest(ParseTestCase):
- def runTest(self):
- from examples import ebnf
- from pyparsing import Word, quotedString, alphas, nums
-
- print_('Constructing EBNF parser with pyparsing...')
-
- grammar = '''
- syntax = (syntax_rule), {(syntax_rule)};
- syntax_rule = meta_identifier, '=', definitions_list, ';';
- definitions_list = single_definition, {'|', single_definition};
- single_definition = syntactic_term, {',', syntactic_term};
- syntactic_term = syntactic_factor,['-', syntactic_factor];
- syntactic_factor = [integer, '*'], syntactic_primary;
- syntactic_primary = optional_sequence | repeated_sequence |
- grouped_sequence | meta_identifier | terminal_string;
- optional_sequence = '[', definitions_list, ']';
- repeated_sequence = '{', definitions_list, '}';
- grouped_sequence = '(', definitions_list, ')';
- (*
- terminal_string = "'", character - "'", {character - "'"}, "'" |
- '"', character - '"', {character - '"'}, '"';
- meta_identifier = letter, {letter | digit};
- integer = digit, {digit};
- *)
- '''
-
- table = {}
- table['terminal_string'] = quotedString
- table['meta_identifier'] = Word(alphas+"_", alphas+"_"+nums)
- table['integer'] = Word(nums)
-
- print_('Parsing EBNF grammar with EBNF parser...')
- parsers = ebnf.parse(grammar, table)
- ebnf_parser = parsers['syntax']
- print_("-","\n- ".join(parsers.keys()))
- self.assertEqual(len(list(parsers.keys())), 13, "failed to construct syntax grammar")
-
- print_('Parsing EBNF grammar with generated EBNF parser...')
- parsed_chars = ebnf_parser.parseString(grammar)
- parsed_char_len = len(parsed_chars)
-
- print_("],\n".join(str( parsed_chars.asList() ).split("],")))
- self.assertEqual(len(flatten(parsed_chars.asList())), 98, "failed to tokenize grammar correctly")
-
-
-class ParseIDLTest(ParseTestCase):
- def runTest(self):
- from examples import idlParse
-
- def test( strng, numToks, errloc=0 ):
- print_(strng)
- try:
- bnf = idlParse.CORBA_IDL_BNF()
- tokens = bnf.parseString( strng )
- print_("tokens = ")
- tokens.pprint()
- tokens = flatten( tokens.asList() )
- print_(len(tokens))
- self.assertEqual(len(tokens), numToks, "error matching IDL string, %s -> %s" % (strng, str(tokens)))
- except ParseException as err:
- print_(err.line)
- print_(" "*(err.column-1) + "^")
- print_(err)
- self.assertEqual(numToks, 0, "unexpected ParseException while parsing %s, %s" % (strng, str(err)))
- self.assertEqual(err.loc, errloc,
- "expected ParseException at %d, found exception at %d" % (errloc, err.loc))
-
- test(
- """
- /*
- * a block comment *
- */
- typedef string[10] tenStrings;
- typedef sequence<string> stringSeq;
- typedef sequence< sequence<string> > stringSeqSeq;
-
- interface QoSAdmin {
- stringSeq method1( in string arg1, inout long arg2 );
- stringSeqSeq method2( in string arg1, inout long arg2, inout long arg3);
- string method3();
- };
- """, 59
- )
- test(
- """
- /*
- * a block comment *
- */
- typedef string[10] tenStrings;
- typedef
- /** ** *** **** *
- * a block comment *
- */
- sequence<string> /*comment inside an And */ stringSeq;
- /* */ /**/ /***/ /****/
- typedef sequence< sequence<string> > stringSeqSeq;
-
- interface QoSAdmin {
- stringSeq method1( in string arg1, inout long arg2 );
- stringSeqSeq method2( in string arg1, inout long arg2, inout long arg3);
- string method3();
- };
- """, 59
- )
- test(
- r"""
- const string test="Test String\n";
- const long a = 0;
- const long b = -100;
- const float c = 3.14159;
- const long d = 0x007f7f7f;
- exception TestException
- {
- string msg;
- sequence<string> dataStrings;
- };
-
- interface TestInterface
- {
- void method1( in string arg1, inout long arg2 );
- };
- """, 60
- )
- test(
- """
- module Test1
- {
- exception TestException
- {
- string msg;
- ];
-
- interface TestInterface
- {
- void method1( in string arg1, inout long arg2 )
- raises ( TestException );
- };
- };
- """, 0, 56
- )
- test(
- """
- module Test1
- {
- exception TestException
- {
- string msg;
- };
-
- };
- """, 13
- )
-
-class ParseVerilogTest(ParseTestCase):
- def runTest(self):
- pass
-
-class ScanStringTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word, Combine, Suppress, CharsNotIn, nums, StringEnd
- testdata = """
- <table border="0" cellpadding="3" cellspacing="3" frame="" width="90%">
- <tr align="left" valign="top">
- <td><b>Name</b></td>
- <td><b>IP Address</b></td>
- <td><b>Location</b></td>
- </tr>
- <tr align="left" valign="top" bgcolor="#c7efce">
- <td>time-a.nist.gov</td>
- <td>129.6.15.28</td>
- <td>NIST, Gaithersburg, Maryland</td>
- </tr>
- <tr align="left" valign="top">
- <td>time-b.nist.gov</td>
- <td>129.6.15.29</td>
- <td>NIST, Gaithersburg, Maryland</td>
- </tr>
- <tr align="left" valign="top" bgcolor="#c7efce">
- <td>time-a.timefreq.bldrdoc.gov</td>
- <td>132.163.4.101</td>
- <td>NIST, Boulder, Colorado</td>
- </tr>
- <tr align="left" valign="top">
- <td>time-b.timefreq.bldrdoc.gov</td>
- <td>132.163.4.102</td>
- <td>NIST, Boulder, Colorado</td>
- </tr>
- <tr align="left" valign="top" bgcolor="#c7efce">
- <td>time-c.timefreq.bldrdoc.gov</td>
- <td>132.163.4.103</td>
- <td>NIST, Boulder, Colorado</td>
- </tr>
- </table>
- """
- integer = Word(nums)
- ipAddress = Combine( integer + "." + integer + "." + integer + "." + integer )
- tdStart = Suppress("<td>")
- tdEnd = Suppress("</td>")
- timeServerPattern = (tdStart + ipAddress("ipAddr") + tdEnd
- + tdStart + CharsNotIn("<")("loc") + tdEnd)
- servers = [srvr.ipAddr for srvr,startloc,endloc in timeServerPattern.scanString( testdata )]
-
- print_(servers)
- self.assertEqual(servers,
- ['129.6.15.28', '129.6.15.29', '132.163.4.101', '132.163.4.102', '132.163.4.103'],
- "failed scanString()")
-
- # test for stringEnd detection in scanString
- foundStringEnds = [ r for r in StringEnd().scanString("xyzzy") ]
- print_(foundStringEnds)
- self.assertTrue(foundStringEnds, "Failed to find StringEnd in scanString")
-
-class QuotedStringsTest(ParseTestCase):
- def runTest(self):
- from pyparsing import sglQuotedString,dblQuotedString,quotedString,QuotedString
- testData = \
- """
- 'a valid single quoted string'
- 'an invalid single quoted string
- because it spans lines'
- "a valid double quoted string"
- "an invalid double quoted string
- because it spans lines"
- """
- print_(testData)
-
- sglStrings = [(t[0],b,e) for (t,b,e) in sglQuotedString.scanString(testData)]
- print_(sglStrings)
- self.assertTrue(len(sglStrings) == 1 and (sglStrings[0][1] == 17 and sglStrings[0][2] == 47),
- "single quoted string failure")
-
- dblStrings = [(t[0],b,e) for (t,b,e) in dblQuotedString.scanString(testData)]
- print_(dblStrings)
- self.assertTrue(len(dblStrings) == 1 and (dblStrings[0][1] == 154 and dblStrings[0][2] == 184),
- "double quoted string failure")
-
- allStrings = [(t[0],b,e) for (t,b,e) in quotedString.scanString(testData)]
- print_(allStrings)
- self.assertTrue(len(allStrings) == 2
- and (allStrings[0][1] == 17
- and allStrings[0][2] == 47)
- and (allStrings[1][1] == 154
- and allStrings[1][2] == 184),
- "quoted string failure")
-
- escapedQuoteTest = \
- r"""
- 'This string has an escaped (\') quote character'
- "This string has an escaped (\") quote character"
- """
-
- sglStrings = [(t[0],b,e) for (t,b,e) in sglQuotedString.scanString(escapedQuoteTest)]
- print_(sglStrings)
- self.assertTrue(len(sglStrings) == 1 and (sglStrings[0][1]==17 and sglStrings[0][2]==66),
- "single quoted string escaped quote failure (%s)" % str(sglStrings[0]))
-
- dblStrings = [(t[0],b,e) for (t,b,e) in dblQuotedString.scanString(escapedQuoteTest)]
- print_(dblStrings)
- self.assertTrue(len(dblStrings) == 1 and (dblStrings[0][1]==83 and dblStrings[0][2]==132),
- "double quoted string escaped quote failure (%s)" % str(dblStrings[0]))
-
- allStrings = [(t[0],b,e) for (t,b,e) in quotedString.scanString(escapedQuoteTest)]
- print_(allStrings)
- self.assertTrue(len(allStrings) == 2
- and (allStrings[0][1] == 17
- and allStrings[0][2] == 66
- and allStrings[1][1] == 83
- and allStrings[1][2] == 132),
- "quoted string escaped quote failure (%s)" % ([str(s[0]) for s in allStrings]))
-
- dblQuoteTest = \
- r"""
- 'This string has an doubled ('') quote character'
- "This string has an doubled ("") quote character"
- """
- sglStrings = [(t[0],b,e) for (t,b,e) in sglQuotedString.scanString(dblQuoteTest)]
- print_(sglStrings)
- self.assertTrue(len(sglStrings) == 1 and (sglStrings[0][1]==17 and sglStrings[0][2]==66),
- "single quoted string escaped quote failure (%s)" % str(sglStrings[0]))
- dblStrings = [(t[0],b,e) for (t,b,e) in dblQuotedString.scanString(dblQuoteTest)]
- print_(dblStrings)
- self.assertTrue(len(dblStrings) == 1 and (dblStrings[0][1]==83 and dblStrings[0][2]==132),
- "double quoted string escaped quote failure (%s)" % str(dblStrings[0]))
- allStrings = [(t[0],b,e) for (t,b,e) in quotedString.scanString(dblQuoteTest)]
- print_(allStrings)
- self.assertTrue(len(allStrings) == 2
- and (allStrings[0][1] == 17
- and allStrings[0][2] == 66
- and allStrings[1][1] == 83
- and allStrings[1][2] == 132),
- "quoted string escaped quote failure (%s)" % ([str(s[0]) for s in allStrings]))
-
- print_("testing catastrophic RE backtracking in implementation of dblQuotedString")
- for expr, test_string in [
- (dblQuotedString, '"' + '\\xff' * 500),
- (sglQuotedString, "'" + '\\xff' * 500),
- (quotedString, '"' + '\\xff' * 500),
- (quotedString, "'" + '\\xff' * 500),
- (QuotedString('"'), '"' + '\\xff' * 500),
- (QuotedString("'"), "'" + '\\xff' * 500),
- ]:
- expr.parseString(test_string+test_string[0])
- try:
- expr.parseString(test_string)
- except Exception:
- continue
-
-class CaselessOneOfTest(ParseTestCase):
- def runTest(self):
- from pyparsing import oneOf,ZeroOrMore
-
- caseless1 = oneOf("d a b c aA B A C", caseless=True)
- caseless1str = str( caseless1 )
- print_(caseless1str)
- caseless2 = oneOf("d a b c Aa B A C", caseless=True)
- caseless2str = str( caseless2 )
- print_(caseless2str)
- self.assertEqual(caseless1str.upper(), caseless2str.upper(), "oneOf not handling caseless option properly")
- self.assertNotEqual(caseless1str, caseless2str, "Caseless option properly sorted")
-
- res = ZeroOrMore(caseless1).parseString("AAaaAaaA")
- print_(res)
- self.assertEqual(len(res), 4, "caseless1 oneOf failed")
- self.assertEqual("".join(res), "aA"*4,"caseless1 CaselessLiteral return failed")
-
- res = ZeroOrMore(caseless2).parseString("AAaaAaaA")
- print_(res)
- self.assertEqual(len(res), 4, "caseless2 oneOf failed")
- self.assertEqual("".join(res), "Aa"*4,"caseless1 CaselessLiteral return failed")
-
-
-class AsXMLTest(ParseTestCase):
- def runTest(self):
-
- # test asXML()
-
- aaa = pp.Word("a")("A")
- bbb = pp.Group(pp.Word("b"))("B")
- ccc = pp.Combine(":" + pp.Word("c"))("C")
- g1 = "XXX>&<" + pp.ZeroOrMore( aaa | bbb | ccc )
- teststring = "XXX>&< b b a b b a b :c b a"
- #~ print teststring
- print_("test including all items")
- xml = g1.parseString(teststring).asXML("TEST",namedItemsOnly=False)
- assert xml=="\n".join(["",
- "<TEST>",
- " <ITEM>XXX>&<</ITEM>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <A>a</A>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <A>a</A>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <C>:c</C>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <A>a</A>",
- "</TEST>",
- ] ), \
- "failed to generate XML correctly showing all items: \n[" + xml + "]"
- print_("test filtering unnamed items")
- xml = g1.parseString(teststring).asXML("TEST",namedItemsOnly=True)
- assert xml=="\n".join(["",
- "<TEST>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <A>a</A>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <A>a</A>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <C>:c</C>",
- " <B>",
- " <ITEM>b</ITEM>",
- " </B>",
- " <A>a</A>",
- "</TEST>",
- ] ), \
- "failed to generate XML correctly, filtering unnamed items: " + xml
-
-class AsXMLTest2(ParseTestCase):
- def runTest(self):
- from pyparsing import Suppress,Optional,CharsNotIn,Combine,ZeroOrMore,Word,\
- Group,Literal,alphas,alphanums,delimitedList,OneOrMore
-
- EndOfLine = Word("\n").setParseAction(lambda s,l,t: [' '])
- whiteSpace=Word('\t ')
- Mexpr = Suppress(Optional(whiteSpace)) + CharsNotIn('\\"\t \n') + Optional(" ") + \
- Suppress(Optional(whiteSpace))
- reducedString = Combine(Mexpr + ZeroOrMore(EndOfLine + Mexpr))
- _bslash = "\\"
- _escapables = "tnrfbacdeghijklmopqsuvwxyz" + _bslash + "'" + '"'
- _octDigits = "01234567"
- _escapedChar = ( Word( _bslash, _escapables, exact=2 ) |
- Word( _bslash, _octDigits, min=2, max=4 ) )
- _sglQuote = Literal("'")
- _dblQuote = Literal('"')
- QuotedReducedString = Combine( Suppress(_dblQuote) + ZeroOrMore( reducedString |
- _escapedChar ) + \
- Suppress(_dblQuote )).streamline()
-
- Manifest_string = QuotedReducedString('manifest_string')
-
- Identifier = Word( alphas, alphanums+ '_$' )("identifier")
- Index_string = CharsNotIn('\\";\n')
- Index_string.setName('index_string')
- Index_term_list = (
- Group(delimitedList(Manifest_string, delim=',')) | \
- Index_string
- )('value')
-
- IndexKey = Identifier('key')
- IndexKey.setName('key')
- Index_clause = Group(IndexKey + Suppress(':') + Optional(Index_term_list))
- Index_clause.setName('index_clause')
- Index_list = Index_clause('index')
- Index_list.setName('index_list')
- Index_block = Group('indexing' + Group(OneOrMore(Index_list + Suppress(';'))))('indexes')
-
-
-class CommentParserTest(ParseTestCase):
- def runTest(self):
-
- print_("verify processing of C and HTML comments")
- testdata = """
- /* */
- /** **/
- /**/
- /***/
- /****/
- /* /*/
- /** /*/
- /*** /*/
- /*
- ablsjdflj
- */
- """
- foundLines = [ pp.lineno(s,testdata)
- for t,s,e in pp.cStyleComment.scanString(testdata) ]
- self.assertEqual(foundLines, list(range(11))[2:],"only found C comments on lines "+str(foundLines))
- testdata = """
- <!-- -->
- <!--- --->
- <!---->
- <!----->
- <!------>
- <!-- /-->
- <!--- /-->
- <!---- /-->
- <!---- /- ->
- <!---- / -- >
- <!--
- ablsjdflj
- -->
- """
- foundLines = [ pp.lineno(s,testdata)
- for t,s,e in pp.htmlComment.scanString(testdata) ]
- self.assertEqual(foundLines, list(range(11))[2:],"only found HTML comments on lines "+str(foundLines))
-
- # test C++ single line comments that have line terminated with '\' (should continue comment to following line)
- testSource = r"""
- // comment1
- // comment2 \
- still comment 2
- // comment 3
- """
- self.assertEqual(len(pp.cppStyleComment.searchString(testSource)[1][0]), 41,
- r"failed to match single-line comment with '\' at EOL")
-
-class ParseExpressionResultsTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word,alphas,OneOrMore,Optional,Group
-
- a = Word("a", alphas).setName("A")
- b = Word("b", alphas).setName("B")
- c = Word("c", alphas).setName("C")
- ab = (a + b).setName("AB")
- abc = (ab + c).setName("ABC")
- word = Word(alphas).setName("word")
-
- words = Group(OneOrMore(~a + word)).setName("words")
-
- phrase = (words("Head")
- + Group(a + Optional(b + Optional(c)))("ABC")
- + words("Tail"))
-
- results = phrase.parseString("xavier yeti alpha beta charlie will beaver")
- print_(results,results.Head, results.ABC,results.Tail)
- for key,ln in [("Head",2), ("ABC",3), ("Tail",2)]:
- self.assertEqual(len(results[key]), ln,
- "expected %d elements in %s, found %s" % (ln, key, str(results[key])))
-
-
-class ParseKeywordTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Literal,Keyword
-
- kw = Keyword("if")
- lit = Literal("if")
-
- def test(s,litShouldPass,kwShouldPass):
- print_("Test",s)
- print_("Match Literal", end=' ')
- try:
- print_(lit.parseString(s))
- except Exception:
- print_("failed")
- if litShouldPass:
- self.assertTrue(False, "Literal failed to match %s, should have" % s)
- else:
- if not litShouldPass:
- self.assertTrue(False, "Literal matched %s, should not have" % s)
-
- print_("Match Keyword", end=' ')
- try:
- print_(kw.parseString(s))
- except Exception:
- print_("failed")
- if kwShouldPass:
- self.assertTrue(False, "Keyword failed to match %s, should have" % s)
- else:
- if not kwShouldPass:
- self.assertTrue(False, "Keyword matched %s, should not have" % s)
-
- test("ifOnlyIfOnly", True, False)
- test("if(OnlyIfOnly)", True, True)
- test("if (OnlyIf Only)", True, True)
-
- kw = Keyword("if",caseless=True)
-
- test("IFOnlyIfOnly", False, False)
- test("If(OnlyIfOnly)", False, True)
- test("iF (OnlyIf Only)", False, True)
-
-
-
-class ParseExpressionResultsAccumulateTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word,delimitedList,Combine,alphas,nums
-
- num=Word(nums).setName("num")("base10*")
- hexnum=Combine("0x"+ Word(nums)).setName("hexnum")("hex*")
- name = Word(alphas).setName("word")("word*")
- list_of_num=delimitedList( hexnum | num | name, "," )
-
- tokens = list_of_num.parseString('1, 0x2, 3, 0x4, aaa')
- for k,llen,lst in ( ("base10",2,['1','3']),
- ("hex",2,['0x2','0x4']),
- ("word",1,['aaa']) ):
- print_(k,tokens[k])
- self.assertEqual(len(tokens[k]), llen, "Wrong length for key %s, %s" % (k,str(tokens[k].asList())))
- self.assertEqual(lst, tokens[k].asList(),
- "Incorrect list returned for key %s, %s" % (k,str(tokens[k].asList())))
- self.assertEqual(tokens.base10.asList(), ['1','3'],
- "Incorrect list for attribute base10, %s" % str(tokens.base10.asList()))
- self.assertEqual(tokens.hex.asList(), ['0x2','0x4'],
- "Incorrect list for attribute hex, %s" % str(tokens.hex.asList()))
- self.assertEqual(tokens.word.asList(), ['aaa'],
- "Incorrect list for attribute word, %s" % str(tokens.word.asList()))
-
- from pyparsing import Literal, Word, nums, Group, Dict, alphas, \
- quotedString, oneOf, delimitedList, removeQuotes, alphanums
-
- lbrack = Literal("(").suppress()
- rbrack = Literal(")").suppress()
- integer = Word( nums ).setName("int")
- variable = Word( alphas, max=1 ).setName("variable")
- relation_body_item = variable | integer | quotedString.copy().setParseAction(removeQuotes)
- relation_name = Word( alphas+"_", alphanums+"_" )
- relation_body = lbrack + Group(delimitedList(relation_body_item)) + rbrack
- Goal = Dict(Group( relation_name + relation_body ))
- Comparison_Predicate = Group(variable + oneOf("< >") + integer)("pred*")
- Query = Goal("head") + ":-" + delimitedList(Goal | Comparison_Predicate)
-
- test="""Q(x,y,z):-Bloo(x,"Mitsis",y),Foo(y,z,1243),y>28,x<12,x>3"""
-
- queryRes = Query.parseString(test)
- print_("pred",queryRes.pred)
- self.assertEqual(queryRes.pred.asList(), [['y', '>', '28'], ['x', '<', '12'], ['x', '>', '3']],
- "Incorrect list for attribute pred, %s" % str(queryRes.pred.asList()))
- print_(queryRes.dump())
-
-class ReStringRangeTest(ParseTestCase):
- def runTest(self):
- testCases = (
- (r"[A-Z]"),
- (r"[A-A]"),
- (r"[A-Za-z]"),
- (r"[A-z]"),
- (r"[\ -\~]"),
- (r"[\0x20-0]"),
- (r"[\0x21-\0x7E]"),
- (r"[\0xa1-\0xfe]"),
- (r"[\040-0]"),
- (r"[A-Za-z0-9]"),
- (r"[A-Za-z0-9_]"),
- (r"[A-Za-z0-9_$]"),
- (r"[A-Za-z0-9_$\-]"),
- (r"[^0-9\\]"),
- (r"[a-zA-Z]"),
- (r"[/\^~]"),
- (r"[=\+\-!]"),
- (r"[A-]"),
- (r"[-A]"),
- (r"[\x21]"),
- #(r"[а-яА-ЯёЁA-Z$_\041α-ω]".decode('utf-8')),
- (u'[\u0430-\u044f\u0410-\u042f\u0451\u0401ABCDEFGHIJKLMNOPQRSTUVWXYZ$_\041\u03b1-\u03c9]'),
- )
- expectedResults = (
- "ABCDEFGHIJKLMNOPQRSTUVWXYZ",
- "A",
- "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",
- "ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz",
- " !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~",
- " !\"#$%&'()*+,-./0",
- "!\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~",
- #~ "¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþ",
- u'\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe',
- " !\"#$%&'()*+,-./0",
- "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789",
- "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_",
- "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$",
- "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$-",
- "0123456789\\",
- "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ",
- "/^~",
- "=+-!",
- "A-",
- "-A",
- "!",
- u"абвгдежзийклмнопрстуфхцчшщъыьэюяАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯёЁABCDEFGHIJKLMNOPQRSTUVWXYZ$_!αβγδεζηθικλμνξοπρςστυφχψω",
- )
- for test in zip( testCases, expectedResults ):
- t,exp = test
- res = pp.srange(t)
- #print_(t,"->",res)
- self.assertEqual(res, exp, "srange error, srange(%r)->'%r', expected '%r'" % (t, res, exp))
-
-class SkipToParserTests(ParseTestCase):
- def runTest(self):
-
- from pyparsing import Literal, SkipTo, cStyleComment, ParseBaseException, And, Word, alphas, nums, Optional, NotAny
-
- thingToFind = Literal('working')
- testExpr = SkipTo(Literal(';'), include=True, ignore=cStyleComment) + thingToFind
-
- def tryToParse (someText, fail_expected=False):
- try:
- print_(testExpr.parseString(someText))
- self.assertFalse(fail_expected, "expected failure but no exception raised")
- except Exception as e:
- print_("Exception %s while parsing string %s" % (e,repr(someText)))
- self.assertTrue(fail_expected and isinstance(e,ParseBaseException),
- "Exception %s while parsing string %s" % (e,repr(someText)))
-
- # This first test works, as the SkipTo expression is immediately following the ignore expression (cStyleComment)
- tryToParse('some text /* comment with ; in */; working')
- # This second test previously failed, as there is text following the ignore expression, and before the SkipTo expression.
- tryToParse('some text /* comment with ; in */some other stuff; working')
-
- # tests for optional failOn argument
- testExpr = SkipTo(Literal(';'), include=True, ignore=cStyleComment, failOn='other') + thingToFind
- tryToParse('some text /* comment with ; in */; working')
- tryToParse('some text /* comment with ; in */some other stuff; working', fail_expected=True)
-
- # test that we correctly create named results
- text = "prefixDATAsuffix"
- data = Literal("DATA")
- suffix = Literal("suffix")
- expr = SkipTo(data + suffix)('prefix') + data + suffix
- result = expr.parseString(text)
- self.assertTrue(isinstance(result.prefix, str), "SkipTo created with wrong saveAsList attribute")
-
- if PY_3:
- def define_expr(s):
- from pyparsing import Literal, And, Word, alphas, nums, Optional, NotAny
- alpha_word = (~Literal("end") + Word(alphas, asKeyword=True)).setName("alpha")
- num_word = Word(nums, asKeyword=True).setName("int")
-
- ret = eval(s)
- ret.streamline()
- print_(ret)
- return ret
-
- def test(expr, test_string, expected_list, expected_dict):
- try:
- result = expr.parseString(test_string)
- except Exception as pe:
- if any(expected is not None for expected in (expected_list, expected_dict)):
- self.assertTrue(False, "{} failed to parse {!r}".format(expr, test_string))
- else:
- self.assertEqual(result.asList(), expected_list)
- self.assertEqual(result.asDict(), expected_dict)
-
- # ellipses for SkipTo
- # (use eval() to avoid syntax problems when running in Py2)
- e = define_expr('... + Literal("end")')
- test(e, "start 123 end", ['start 123 ', 'end'], {'_skipped': ['start 123 ']})
-
- e = define_expr('Literal("start") + ... + Literal("end")')
- test(e, "start 123 end", ['start', '123 ', 'end'], {'_skipped': ['123 ']})
-
- e = define_expr('Literal("start") + ...')
- test(e, "start 123 end", None, None)
-
- e = define_expr('And(["start", ..., "end"])')
- test(e, "start 123 end", ['start', '123 ', 'end'], {'_skipped': ['123 ']})
-
- e = define_expr('And([..., "end"])')
- test(e, "start 123 end", ['start 123 ', 'end'], {'_skipped': ['start 123 ']})
-
- e = define_expr('"start" + (num_word | ...) + "end"')
- test(e, "start 456 end", ['start', '456', 'end'], {})
- test(e, "start 123 456 end", ['start', '123', '456 ', 'end'], {'_skipped': ['456 ']})
- test(e, "start end", ['start', '', 'end'], {'_skipped': ['missing <int>']})
-
- # e = define_expr('"start" + (num_word | ...)("inner") + "end"')
- # test(e, "start 456 end", ['start', '456', 'end'], {'inner': '456'})
-
- e = define_expr('"start" + (alpha_word[...] & num_word[...] | ...) + "end"')
- test(e, "start 456 red end", ['start', '456', 'red', 'end'], {})
- test(e, "start red 456 end", ['start', 'red', '456', 'end'], {})
- test(e, "start 456 red + end", ['start', '456', 'red', '+ ', 'end'], {'_skipped': ['+ ']})
- test(e, "start red end", ['start', 'red', 'end'], {})
- test(e, "start 456 end", ['start', '456', 'end'], {})
- test(e, "start end", ['start', 'end'], {})
- test(e, "start 456 + end", ['start', '456', '+ ', 'end'], {'_skipped': ['+ ']})
-
- e = define_expr('"start" + (alpha_word[1, ...] & num_word[1, ...] | ...) + "end"')
- test(e, "start 456 red end", ['start', '456', 'red', 'end'], {})
- test(e, "start red 456 end", ['start', 'red', '456', 'end'], {})
- test(e, "start 456 red + end", ['start', '456', 'red', '+ ', 'end'], {'_skipped': ['+ ']})
- test(e, "start red end", ['start', 'red ', 'end'], {'_skipped': ['red ']})
- test(e, "start 456 end", ['start', '456 ', 'end'], {'_skipped': ['456 ']})
- test(e, "start end", ['start', '', 'end'], {'_skipped': ['missing <{{alpha}... & {int}...}>']})
- test(e, "start 456 + end", ['start', '456 + ', 'end'], {'_skipped': ['456 + ']})
-
- e = define_expr('"start" + (alpha_word | ...) + (num_word | ...) + "end"')
- test(e, "start red 456 end", ['start', 'red', '456', 'end'], {})
- test(e, "start red end", ['start', 'red', '', 'end'], {'_skipped': ['missing <int>']})
- test(e, "start end", ['start', '', '', 'end'], {'_skipped': ['missing <alpha>', 'missing <int>']})
-
- e = define_expr('Literal("start") + ... + "+" + ... + "end"')
- test(e, "start red + 456 end", ['start', 'red ', '+', '456 ', 'end'], {'_skipped': ['red ', '456 ']})
-
-class EllipsisRepetionTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- import re
-
- word = pp.Word(pp.alphas).setName("word")
- num = pp.Word(pp.nums).setName("num")
-
- exprs = [
- word[...] + num,
- word[0, ...] + num,
- word[1, ...] + num,
- word[2, ...] + num,
- word[..., 3] + num,
- word[2] + num,
- ]
-
- expected_res = [
- r"([abcd]+ )*\d+",
- r"([abcd]+ )*\d+",
- r"([abcd]+ )+\d+",
- r"([abcd]+ ){2,}\d+",
- r"([abcd]+ ){0,3}\d+",
- r"([abcd]+ ){2}\d+",
- ]
-
- tests = [
- "aa bb cc dd 123",
- "bb cc dd 123",
- "cc dd 123",
- "dd 123",
- "123",
- ]
-
- all_success = True
- for expr, expected_re in zip(exprs, expected_res):
- successful_tests = [t for t in tests if re.match(expected_re, t)]
- failure_tests = [t for t in tests if not re.match(expected_re, t)]
- success1, _ = expr.runTests(successful_tests)
- success2, _ = expr.runTests(failure_tests, failureTests=True)
- all_success = all_success and success1 and success2
- if not all_success:
- print_("Failed expression:", expr)
- break
-
- self.assertTrue(all_success, "failed getItem_ellipsis test")
-
-
-class CustomQuotesTest(ParseTestCase):
- def runTest(self):
- self.expect_warning = True
-
- from pyparsing import QuotedString
-
- testString = r"""
- sdlfjs :sdf\:jls::djf: sl:kfsjf
- sdlfjs -sdf\:jls::--djf: sl-kfsjf
- sdlfjs -sdf\:::jls::--djf: sl:::-kfsjf
- sdlfjs ^sdf\:jls^^--djf^ sl-kfsjf
- sdlfjs ^^^==sdf\:j=lz::--djf: sl=^^=kfsjf
- sdlfjs ==sdf\:j=ls::--djf: sl==kfsjf^^^
- """
- colonQuotes = QuotedString(':','\\','::')
- dashQuotes = QuotedString('-','\\', '--')
- hatQuotes = QuotedString('^','\\')
- hatQuotes1 = QuotedString('^','\\','^^')
- dblEqQuotes = QuotedString('==','\\')
-
- def test(quoteExpr, expected):
- print_(quoteExpr.pattern)
- print_(quoteExpr.searchString(testString))
- print_(quoteExpr.searchString(testString)[0][0])
- print_(expected)
- self.assertEqual(quoteExpr.searchString(testString)[0][0],
- expected,
- "failed to match %s, expected '%s', got '%s'" % (quoteExpr, expected,
- quoteExpr.searchString(testString)[0]))
- print_()
-
- test(colonQuotes, r"sdf:jls:djf")
- test(dashQuotes, r"sdf:jls::-djf: sl")
- test(hatQuotes, r"sdf:jls")
- test(hatQuotes1, r"sdf:jls^--djf")
- test(dblEqQuotes, r"sdf:j=ls::--djf: sl")
- test(QuotedString(':::'), 'jls::--djf: sl')
- test(QuotedString('==',endQuoteChar='--'), r'sdf\:j=lz::')
- test(QuotedString('^^^',multiline=True), r"""==sdf\:j=lz::--djf: sl=^^=kfsjf
- sdlfjs ==sdf\:j=ls::--djf: sl==kfsjf""")
- try:
- bad1 = QuotedString('','\\')
- except SyntaxError as se:
- pass
- else:
- self.assertTrue(False,"failed to raise SyntaxError with empty quote string")
-
-class RepeaterTest(ParseTestCase):
- def runTest(self):
- from pyparsing import matchPreviousLiteral,matchPreviousExpr, Word, nums, ParserElement
-
- if ParserElement._packratEnabled:
- print_("skipping this test, not compatible with packratting")
- return
-
- first = Word("abcdef").setName("word1")
- bridge = Word(nums).setName("number")
- second = matchPreviousLiteral(first).setName("repeat(word1Literal)")
-
- seq = first + bridge + second
-
- tests = [
- ( "abc12abc", True ),
- ( "abc12aabc", False ),
- ( "abc12cba", True ),
- ( "abc12bca", True ),
- ]
-
- for tst,result in tests:
- found = False
- for tokens,start,end in seq.scanString(tst):
- f,b,s = tokens
- print_(f,b,s)
- found = True
- if not found:
- print_("No literal match in", tst)
- self.assertEqual(found, result, "Failed repeater for test: %s, matching %s" % (tst, str(seq)))
- print_()
-
- # retest using matchPreviousExpr instead of matchPreviousLiteral
- second = matchPreviousExpr(first).setName("repeat(word1expr)")
- seq = first + bridge + second
-
- tests = [
- ( "abc12abc", True ),
- ( "abc12cba", False ),
- ( "abc12abcdef", False ),
- ]
-
- for tst,result in tests:
- found = False
- for tokens,start,end in seq.scanString(tst):
- print_(tokens.asList())
- found = True
- if not found:
- print_("No expression match in", tst)
- self.assertEqual(found, result, "Failed repeater for test: %s, matching %s" % (tst, str(seq)))
-
- print_()
-
- first = Word("abcdef").setName("word1")
- bridge = Word(nums).setName("number")
- second = matchPreviousExpr(first).setName("repeat(word1)")
- seq = first + bridge + second
- csFirst = seq.setName("word-num-word")
- csSecond = matchPreviousExpr(csFirst)
- compoundSeq = csFirst + ":" + csSecond
- compoundSeq.streamline()
- print_(compoundSeq)
-
- tests = [
- ( "abc12abc:abc12abc", True ),
- ( "abc12cba:abc12abc", False ),
- ( "abc12abc:abc12abcdef", False ),
- ]
-
- for tst, result in tests:
- found = False
- for tokens, start, end in compoundSeq.scanString(tst):
- print_("match:", tokens.asList())
- found = True
- break
- if not found:
- print_("No expression match in", tst)
- self.assertEqual(found, result, "Failed repeater for test: %s, matching %s" % (tst, str(seq)))
-
- print_()
- eFirst = Word(nums)
- eSecond = matchPreviousExpr(eFirst)
- eSeq = eFirst + ":" + eSecond
-
- tests = [
- ( "1:1A", True ),
- ( "1:10", False ),
- ]
-
- for tst,result in tests:
- found = False
- for tokens,start,end in eSeq.scanString(tst):
- print_(tokens.asList())
- found = True
- if not found:
- print_("No match in", tst)
- self.assertEqual(found, result, "Failed repeater for test: %s, matching %s" % (tst, str(seq)))
-
-class RecursiveCombineTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Forward,Word,alphas,nums,Optional,Combine
-
- testInput = "myc(114)r(11)dd"
- Stream=Forward()
- Stream << Optional(Word(alphas))+Optional("("+Word(nums)+")"+Stream)
- expected = Stream.parseString(testInput).asList()
- print_(["".join(expected)])
-
- Stream=Forward()
- Stream << Combine(Optional(Word(alphas))+Optional("("+Word(nums)+")"+Stream))
- testVal = Stream.parseString(testInput).asList()
- print_(testVal)
-
- self.assertEqual("".join(testVal), "".join(expected), "Failed to process Combine with recursive content")
-
-class InfixNotationGrammarTest1(ParseTestCase):
- def runTest(self):
- from pyparsing import Word,nums,alphas,Literal,oneOf,infixNotation,opAssoc
- import ast
-
- integer = Word(nums).setParseAction(lambda t:int(t[0]))
- variable = Word(alphas,exact=1)
- operand = integer | variable
-
- expop = Literal('^')
- signop = oneOf('+ -')
- multop = oneOf('* /')
- plusop = oneOf('+ -')
- factop = Literal('!')
-
- expr = infixNotation( operand,
- [(factop, 1, opAssoc.LEFT),
- (expop, 2, opAssoc.RIGHT),
- (signop, 1, opAssoc.RIGHT),
- (multop, 2, opAssoc.LEFT),
- (plusop, 2, opAssoc.LEFT),]
- )
-
- test = ["9 + 2 + 3",
- "9 + 2 * 3",
- "(9 + 2) * 3",
- "(9 + -2) * 3",
- "(9 + --2) * 3",
- "(9 + -2) * 3^2^2",
- "(9! + -2) * 3^2^2",
- "M*X + B",
- "M*(X + B)",
- "1+2*-3^4*5+-+-6",
- "3!!"]
- expected = """[[9, '+', 2, '+', 3]]
- [[9, '+', [2, '*', 3]]]
- [[[9, '+', 2], '*', 3]]
- [[[9, '+', ['-', 2]], '*', 3]]
- [[[9, '+', ['-', ['-', 2]]], '*', 3]]
- [[[9, '+', ['-', 2]], '*', [3, '^', [2, '^', 2]]]]
- [[[[9, '!'], '+', ['-', 2]], '*', [3, '^', [2, '^', 2]]]]
- [[['M', '*', 'X'], '+', 'B']]
- [['M', '*', ['X', '+', 'B']]]
- [[1, '+', [2, '*', ['-', [3, '^', 4]], '*', 5], '+', ['-', ['+', ['-', 6]]]]]
- [[3, '!', '!']]""".split('\n')
- expected = [ast.literal_eval(x.strip()) for x in expected]
- for t,e in zip(test,expected):
- print_(t,"->",e, "got", expr.parseString(t).asList())
- self.assertEqual(expr.parseString(t).asList(), e,
- "mismatched results for infixNotation: got %s, expected %s" % (expr.parseString(t).asList(),e))
-
-class InfixNotationGrammarTest2(ParseTestCase):
- def runTest(self):
-
- from pyparsing import infixNotation, Word, alphas, oneOf, opAssoc
-
- boolVars = { "True":True, "False":False }
- class BoolOperand(object):
- reprsymbol = ''
- def __init__(self,t):
- self.args = t[0][0::2]
- def __str__(self):
- sep = " %s " % self.reprsymbol
- return "(" + sep.join(map(str,self.args)) + ")"
-
- class BoolAnd(BoolOperand):
- reprsymbol = '&'
- def __bool__(self):
- for a in self.args:
- if isinstance(a,str):
- v = boolVars[a]
- else:
- v = bool(a)
- if not v:
- return False
- return True
-
- class BoolOr(BoolOperand):
- reprsymbol = '|'
- def __bool__(self):
- for a in self.args:
- if isinstance(a,str):
- v = boolVars[a]
- else:
- v = bool(a)
- if v:
- return True
- return False
-
- class BoolNot(BoolOperand):
- def __init__(self,t):
- self.arg = t[0][1]
- def __str__(self):
- return "~" + str(self.arg)
- def __bool__(self):
- if isinstance(self.arg,str):
- v = boolVars[self.arg]
- else:
- v = bool(self.arg)
- return not v
-
- boolOperand = Word(alphas,max=1) | oneOf("True False")
- boolExpr = infixNotation( boolOperand,
- [
- ("not", 1, opAssoc.RIGHT, BoolNot),
- ("and", 2, opAssoc.LEFT, BoolAnd),
- ("or", 2, opAssoc.LEFT, BoolOr),
- ])
- test = ["p and not q",
- "not not p",
- "not(p and q)",
- "q or not p and r",
- "q or not p or not r",
- "q or not (p and r)",
- "p or q or r",
- "p or q or r and False",
- "(p or q or r) and False",
- ]
-
- boolVars["p"] = True
- boolVars["q"] = False
- boolVars["r"] = True
- print_("p =", boolVars["p"])
- print_("q =", boolVars["q"])
- print_("r =", boolVars["r"])
- print_()
- for t in test:
- res = boolExpr.parseString(t)[0]
- print_(t,'\n', res, '=', bool(res),'\n')
-
-
-class InfixNotationGrammarTest3(ParseTestCase):
- def runTest(self):
-
- from pyparsing import infixNotation, Word, alphas, oneOf, opAssoc, nums, Literal
-
- global count
- count = 0
-
- def evaluate_int(t):
- global count
- value = int(t[0])
- print_("evaluate_int", value)
- count += 1
- return value
-
- integer = Word(nums).setParseAction(evaluate_int)
- variable = Word(alphas,exact=1)
- operand = integer | variable
-
- expop = Literal('^')
- signop = oneOf('+ -')
- multop = oneOf('* /')
- plusop = oneOf('+ -')
- factop = Literal('!')
-
- expr = infixNotation( operand,
- [
- ("!", 1, opAssoc.LEFT),
- ("^", 2, opAssoc.LEFT),
- (signop, 1, opAssoc.RIGHT),
- (multop, 2, opAssoc.LEFT),
- (plusop, 2, opAssoc.LEFT),
- ])
-
- test = ["9"]
- for t in test:
- count = 0
- print_("%r => %s (count=%d)" % (t, expr.parseString(t), count))
- self.assertEqual(count, 1, "count evaluated too many times!")
-
-class InfixNotationGrammarTest4(ParseTestCase):
- def runTest(self):
-
- word = pp.Word(pp.alphas)
-
- def supLiteral(s):
- """Returns the suppressed literal s"""
- return pp.Literal(s).suppress()
-
- def booleanExpr(atom):
- ops = [
- (supLiteral("!"), 1, pp.opAssoc.RIGHT, lambda s, l, t: ["!", t[0][0]]),
- (pp.oneOf("= !="), 2, pp.opAssoc.LEFT, ),
- (supLiteral("&"), 2, pp.opAssoc.LEFT, lambda s, l, t: ["&", t[0]]),
- (supLiteral("|"), 2, pp.opAssoc.LEFT, lambda s, l, t: ["|", t[0]])]
- return pp.infixNotation(atom, ops)
-
- f = booleanExpr(word) + pp.StringEnd()
-
- tests = [
- ("bar = foo", "[['bar', '=', 'foo']]"),
- ("bar = foo & baz = fee", "['&', [['bar', '=', 'foo'], ['baz', '=', 'fee']]]"),
- ]
- for test,expected in tests:
- print_(test)
- results = f.parseString(test)
- print_(results)
- self.assertEqual(str(results), expected, "failed to match expected results, got '%s'" % str(results))
- print_()
-
-class InfixNotationGrammarTest5(ParseTestCase):
-
- def runTest(self):
- from pyparsing import infixNotation, opAssoc, pyparsing_common as ppc, Literal, oneOf
-
- expop = Literal('**')
- signop = oneOf('+ -')
- multop = oneOf('* /')
- plusop = oneOf('+ -')
-
- class ExprNode(object):
- def __init__(self, tokens):
- self.tokens = tokens[0]
-
- def eval(self):
- return None
-
- class NumberNode(ExprNode):
- def eval(self):
- return self.tokens
-
- class SignOp(ExprNode):
- def eval(self):
- mult = {'+': 1, '-': -1}[self.tokens[0]]
- return mult * self.tokens[1].eval()
-
- class BinOp(ExprNode):
- def eval(self):
- ret = self.tokens[0].eval()
- for op, operand in zip(self.tokens[1::2], self.tokens[2::2]):
- ret = self.opn_map[op](ret, operand.eval())
- return ret
-
- class ExpOp(BinOp):
- opn_map = {'**': lambda a, b: b ** a}
-
- class MultOp(BinOp):
- import operator
- opn_map = {'*': operator.mul, '/': operator.truediv}
-
- class AddOp(BinOp):
- import operator
- opn_map = {'+': operator.add, '-': operator.sub}
-
- operand = ppc.number().setParseAction(NumberNode)
- expr = infixNotation(operand,
- [
- (expop, 2, opAssoc.LEFT, (lambda pr: [pr[0][::-1]], ExpOp)),
- (signop, 1, opAssoc.RIGHT, SignOp),
- (multop, 2, opAssoc.LEFT, MultOp),
- (plusop, 2, opAssoc.LEFT, AddOp),
- ])
-
- tests = """\
- 2+7
- 2**3
- 2**3**2
- 3**9
- 3**3**2
- """
-
- for t in tests.splitlines():
- t = t.strip()
- if not t:
- continue
-
- parsed = expr.parseString(t)
- eval_value = parsed[0].eval()
- self.assertEqual(eval_value, eval(t),
- "Error evaluating %r, expected %r, got %r" % (t, eval(t), eval_value))
-
-
-class PickleTest_Greeting():
- def __init__(self, toks):
- self.salutation = toks[0]
- self.greetee = toks[1]
-
- def __repr__(self):
- return "%s: {%s}" % (self.__class__.__name__,
- ', '.join('%r: %r' % (k, getattr(self,k)) for k in sorted(self.__dict__)))
-
-class ParseResultsPickleTest(ParseTestCase):
- def runTest(self):
- from pyparsing import makeHTMLTags, ParseResults
- import pickle
-
- # test 1
- body = makeHTMLTags("BODY")[0]
- result = body.parseString("<BODY BGCOLOR='#00FFBB' FGCOLOR=black>")
- if VERBOSE:
- print_(result.dump())
- print_()
-
- for protocol in range(pickle.HIGHEST_PROTOCOL+1):
- print_("Test pickle dump protocol", protocol)
- try:
- pickleString = pickle.dumps(result, protocol)
- except Exception as e:
- print_("dumps exception:", e)
- newresult = ParseResults()
- else:
- newresult = pickle.loads(pickleString)
- if VERBOSE:
- print_(newresult.dump())
- print_()
-
- self.assertEqual(result.dump(), newresult.dump(),
- "Error pickling ParseResults object (protocol=%d)" % protocol)
-
- # test 2
- import pyparsing as pp
-
- word = pp.Word(pp.alphas+"'.")
- salutation = pp.OneOrMore(word)
- comma = pp.Literal(",")
- greetee = pp.OneOrMore(word)
- endpunc = pp.oneOf("! ?")
- greeting = salutation + pp.Suppress(comma) + greetee + pp.Suppress(endpunc)
- greeting.setParseAction(PickleTest_Greeting)
-
- string = 'Good morning, Miss Crabtree!'
-
- result = greeting.parseString(string)
-
- for protocol in range(pickle.HIGHEST_PROTOCOL+1):
- print_("Test pickle dump protocol", protocol)
- try:
- pickleString = pickle.dumps(result, protocol)
- except Exception as e:
- print_("dumps exception:", e)
- newresult = ParseResults()
- else:
- newresult = pickle.loads(pickleString)
- print_(newresult.dump())
- self.assertEqual(newresult.dump(), result.dump(),
- "failed to pickle/unpickle ParseResults: expected %r, got %r" % (result, newresult))
-
-class ParseResultsWithNamedTupleTest(ParseTestCase):
- def runTest(self):
-
- from pyparsing import Literal,replaceWith
-
- expr = Literal("A")("Achar")
- expr.setParseAction(replaceWith(tuple(["A","Z"])))
-
- res = expr.parseString("A")
- print_(repr(res))
- print_(res.Achar)
- self.assertEqual(res.Achar, ("A","Z"),
- "Failed accessing named results containing a tuple, got {0!r}".format(res.Achar))
-
-
-class ParseHTMLTagsTest(ParseTestCase):
- def runTest(self):
- test = """
- <BODY>
- <BODY BGCOLOR="#00FFCC">
- <BODY BGCOLOR="#00FFAA"/>
- <BODY BGCOLOR='#00FFBB' FGCOLOR=black>
- <BODY/>
- </BODY>
- """
- results = [
- ("startBody", False, "", ""),
- ("startBody", False, "#00FFCC", ""),
- ("startBody", True, "#00FFAA", ""),
- ("startBody", False, "#00FFBB", "black"),
- ("startBody", True, "", ""),
- ("endBody", False, "", ""),
- ]
-
- bodyStart, bodyEnd = pp.makeHTMLTags("BODY")
- resIter = iter(results)
- for t,s,e in (bodyStart | bodyEnd).scanString( test ):
- print_(test[s:e], "->", t.asList())
- (expectedType, expectedEmpty, expectedBG, expectedFG) = next(resIter)
-
- print_(t.dump())
- if "startBody" in t:
- self.assertEqual(bool(t.empty), expectedEmpty,
- "expected %s token, got %s" % (expectedEmpty and "empty" or "not empty",
- t.empty and "empty" or "not empty"))
- self.assertEqual(t.bgcolor, expectedBG,
- "failed to match BGCOLOR, expected %s, got %s" % (expectedBG, t.bgcolor))
- self.assertEqual(t.fgcolor, expectedFG,
- "failed to match FGCOLOR, expected %s, got %s" % (expectedFG, t.bgcolor))
- elif "endBody" in t:
- print_("end tag")
- pass
- else:
- print_("BAD!!!")
-
-
-class UpcaseDowncaseUnicode(ParseTestCase):
- def runTest(self):
-
- import pyparsing as pp
- from pyparsing import pyparsing_unicode as ppu
- import sys
- if PY_3:
- unichr = chr
- else:
- from __builtin__ import unichr
-
- a = u'\u00bfC\u00f3mo esta usted?'
- if not JYTHON_ENV:
- ualphas = ppu.alphas
- else:
- ualphas = "".join( unichr(i) for i in list(range(0xd800)) + list(range(0xe000,sys.maxunicode))
- if unichr(i).isalpha() )
- uword = pp.Word(ualphas).setParseAction(pp.upcaseTokens)
-
- print_ = lambda *args: None
- print_(uword.searchString(a))
-
- uword = pp.Word(ualphas).setParseAction(pp.downcaseTokens)
-
- print_(uword.searchString(a))
-
- kw = pp.Keyword('mykey', caseless=True).setParseAction(pp.upcaseTokens)('rname')
- ret = kw.parseString('mykey')
- print_(ret.rname)
- self.assertEqual(ret.rname, 'MYKEY', "failed to upcase with named result")
-
- kw = pp.Keyword('mykey', caseless=True).setParseAction(pp.pyparsing_common.upcaseTokens)('rname')
- ret = kw.parseString('mykey')
- print_(ret.rname)
- self.assertEqual(ret.rname, 'MYKEY', "failed to upcase with named result (pyparsing_common)")
-
- kw = pp.Keyword('MYKEY', caseless=True).setParseAction(pp.pyparsing_common.downcaseTokens)('rname')
- ret = kw.parseString('mykey')
- print_(ret.rname)
- self.assertEqual(ret.rname, 'mykey', "failed to upcase with named result")
-
- if not IRON_PYTHON_ENV:
- #test html data
- html = u"<TR class=maintxt bgColor=#ffffff> \
- <TD vAlign=top>Производитель, модель</TD> \
- <TD vAlign=top><STRONG>BenQ-Siemens CF61</STRONG></TD> \
- "#.decode('utf-8')
-
- # u'Manufacturer, model
- text_manuf = u'Производитель, модель'
- manufacturer = pp.Literal(text_manuf)
-
- td_start, td_end = pp.makeHTMLTags("td")
- manuf_body = td_start.suppress() + manufacturer + pp.SkipTo(td_end)("cells*") + td_end.suppress()
-
- #~ manuf_body.setDebug()
-
- #~ for tokens in manuf_body.scanString(html):
- #~ print_(tokens)
-
-class ParseUsingRegex(ParseTestCase):
- def runTest(self):
- self.expect_warning = True
-
- import re
-
- signedInt = pp.Regex(r'[-+][0-9]+')
- unsignedInt = pp.Regex(r'[0-9]+')
- simpleString = pp.Regex(r'("[^\"]*")|(\'[^\']*\')')
- namedGrouping = pp.Regex(r'("(?P<content>[^\"]*)")')
- compiledRE = pp.Regex(re.compile(r'[A-Z]+'))
-
- def testMatch (expression, instring, shouldPass, expectedString=None):
- if shouldPass:
- try:
- result = expression.parseString(instring)
- print_('%s correctly matched %s' % (repr(expression), repr(instring)))
- if expectedString != result[0]:
- print_('\tbut failed to match the pattern as expected:')
- print_('\tproduced %s instead of %s' % \
- (repr(result[0]), repr(expectedString)))
- return True
- except pp.ParseException:
- print_('%s incorrectly failed to match %s' % \
- (repr(expression), repr(instring)))
- else:
- try:
- result = expression.parseString(instring)
- print_('%s incorrectly matched %s' % (repr(expression), repr(instring)))
- print_('\tproduced %s as a result' % repr(result[0]))
- except pp.ParseException:
- print_('%s correctly failed to match %s' % \
- (repr(expression), repr(instring)))
- return True
- return False
-
- # These should fail
- self.assertTrue(testMatch(signedInt, '1234 foo', False), "Re: (1) passed, expected fail")
- self.assertTrue(testMatch(signedInt, ' +foo', False), "Re: (2) passed, expected fail")
- self.assertTrue(testMatch(unsignedInt, 'abc', False), "Re: (3) passed, expected fail")
- self.assertTrue(testMatch(unsignedInt, '+123 foo', False), "Re: (4) passed, expected fail")
- self.assertTrue(testMatch(simpleString, 'foo', False), "Re: (5) passed, expected fail")
- self.assertTrue(testMatch(simpleString, '"foo bar\'', False), "Re: (6) passed, expected fail")
- self.assertTrue(testMatch(simpleString, '\'foo bar"', False), "Re: (7) passed, expected fail")
-
- # These should pass
- self.assertTrue(testMatch(signedInt, ' +123', True, '+123'), "Re: (8) failed, expected pass")
- self.assertTrue(testMatch(signedInt, '+123', True, '+123'), "Re: (9) failed, expected pass")
- self.assertTrue(testMatch(signedInt, '+123 foo', True, '+123'), "Re: (10) failed, expected pass")
- self.assertTrue(testMatch(signedInt, '-0 foo', True, '-0'), "Re: (11) failed, expected pass")
- self.assertTrue(testMatch(unsignedInt, '123 foo', True, '123'), "Re: (12) failed, expected pass")
- self.assertTrue(testMatch(unsignedInt, '0 foo', True, '0'), "Re: (13) failed, expected pass")
- self.assertTrue(testMatch(simpleString, '"foo"', True, '"foo"'), "Re: (14) failed, expected pass")
- self.assertTrue(testMatch(simpleString, "'foo bar' baz", True, "'foo bar'"), "Re: (15) failed, expected pass")
-
- self.assertTrue(testMatch(compiledRE, 'blah', False), "Re: (16) passed, expected fail")
- self.assertTrue(testMatch(compiledRE, 'BLAH', True, 'BLAH'), "Re: (17) failed, expected pass")
-
- self.assertTrue(testMatch(namedGrouping, '"foo bar" baz', True, '"foo bar"'), "Re: (16) failed, expected pass")
- ret = namedGrouping.parseString('"zork" blah')
- print_(ret.asList())
- print_(list(ret.items()))
- print_(ret.content)
- self.assertEqual(ret.content, 'zork', "named group lookup failed")
- self.assertEqual(ret[0], simpleString.parseString('"zork" blah')[0],
- "Regex not properly returning ParseResults for named vs. unnamed groups")
-
- try:
- #~ print "lets try an invalid RE"
- invRe = pp.Regex('("[^\"]*")|(\'[^\']*\'')
- except Exception as e:
- print_("successfully rejected an invalid RE:", end=' ')
- print_(e)
- else:
- self.assertTrue(False, "failed to reject invalid RE")
-
- invRe = pp.Regex('')
-
-class RegexAsTypeTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- test_str = "sldkjfj 123 456 lsdfkj"
-
- print_("return as list of match groups")
- expr = pp.Regex(r"\w+ (\d+) (\d+) (\w+)", asGroupList=True)
- expected_group_list = [tuple(test_str.split()[1:])]
- result = expr.parseString(test_str)
- print_(result.dump())
- print_(expected_group_list)
- self.assertEqual(result.asList(), expected_group_list, "incorrect group list returned by Regex)")
-
- print_("return as re.match instance")
- expr = pp.Regex(r"\w+ (?P<num1>\d+) (?P<num2>\d+) (?P<last_word>\w+)", asMatch=True)
- result = expr.parseString(test_str)
- print_(result.dump())
- print_(result[0].groups())
- print_(expected_group_list)
- self.assertEqual(result[0].groupdict(), {'num1': '123', 'num2': '456', 'last_word': 'lsdfkj'},
- 'invalid group dict from Regex(asMatch=True)')
- self.assertEqual(result[0].groups(), expected_group_list[0],
- "incorrect group list returned by Regex(asMatch)")
-
-class RegexSubTest(ParseTestCase):
- def runTest(self):
- self.expect_warning = True
- import pyparsing as pp
-
- print_("test sub with string")
- expr = pp.Regex(r"<title>").sub("'Richard III'")
- result = expr.transformString("This is the title: <title>")
- print_(result)
- self.assertEqual(result, "This is the title: 'Richard III'", "incorrect Regex.sub result with simple string")
-
- print_("test sub with re string")
- expr = pp.Regex(r"([Hh]\d):\s*(.*)").sub(r"<\1>\2</\1>")
- result = expr.transformString("h1: This is the main heading\nh2: This is the sub-heading")
- print_(result)
- self.assertEqual(result, '<h1>This is the main heading</h1>\n<h2>This is the sub-heading</h2>',
- "incorrect Regex.sub result with re string")
-
- print_("test sub with re string (Regex returns re.match)")
- expr = pp.Regex(r"([Hh]\d):\s*(.*)", asMatch=True).sub(r"<\1>\2</\1>")
- result = expr.transformString("h1: This is the main heading\nh2: This is the sub-heading")
- print_(result)
- self.assertEqual(result, '<h1>This is the main heading</h1>\n<h2>This is the sub-heading</h2>',
- "incorrect Regex.sub result with re string")
-
- print_("test sub with callable that return str")
- expr = pp.Regex(r"<(.*?)>").sub(lambda m: m.group(1).upper())
- result = expr.transformString("I want this in upcase: <what? what?>")
- print_(result)
- self.assertEqual(result, 'I want this in upcase: WHAT? WHAT?', "incorrect Regex.sub result with callable")
-
- try:
- expr = pp.Regex(r"<(.*?)>", asMatch=True).sub(lambda m: m.group(1).upper())
- except SyntaxError:
- pass
- else:
- self.assertTrue(False, "failed to warn using a Regex.sub(callable) with asMatch=True")
-
- try:
- expr = pp.Regex(r"<(.*?)>", asGroupList=True).sub(lambda m: m.group(1).upper())
- except SyntaxError:
- pass
- else:
- self.assertTrue(False, "failed to warn using a Regex.sub() with asGroupList=True")
-
- try:
- expr = pp.Regex(r"<(.*?)>", asGroupList=True).sub("")
- except SyntaxError:
- pass
- else:
- self.assertTrue(False, "failed to warn using a Regex.sub() with asGroupList=True")
-
-class PrecededByTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- num = pp.Word(pp.nums).setParseAction(lambda t: int(t[0]))
- interesting_num = pp.PrecededBy(pp.Char("abc")("prefix*")) + num
- semi_interesting_num = pp.PrecededBy('_') + num
- crazy_num = pp.PrecededBy(pp.Word("^", "$%^")("prefix*"), 10) + num
- boring_num = ~pp.PrecededBy(pp.Char("abc_$%^" + pp.nums)) + num
- very_boring_num = pp.PrecededBy(pp.WordStart()) + num
- finicky_num = pp.PrecededBy(pp.Word("^", "$%^"), retreat=3) + num
-
- s = "c384 b8324 _9293874 _293 404 $%^$^%$2939"
- print_(s)
- for expr, expected_list, expected_dict in [
- (interesting_num, [384, 8324], {'prefix': ['c', 'b']}),
- (semi_interesting_num, [9293874, 293], {}),
- (boring_num, [404], {}),
- (crazy_num, [2939], {'prefix': ['^%$']}),
- (finicky_num, [2939], {}),
- (very_boring_num, [404], {}),
- ]:
- print_(expr.searchString(s))
- result = sum(expr.searchString(s))
- print_(result)
-
- self.assertEqual(result.asList(), expected_list,
- "Erroneous tokens for {0}: expected {1}, got {2}".format(expr,
- expected_list,
- result.asList()))
- self.assertEqual(result.asDict(), expected_dict,
- "Erroneous named results for {0}: expected {1}, got {2}".format(expr,
- expected_dict,
- result.asDict()))
- # infinite loop test - from Issue #127
- string_test = 'notworking'
- # negs = pp.Or(['not', 'un'])('negs')
- negs_pb = pp.PrecededBy('not', retreat=100)('negs_lb')
- # negs_pb = pp.PrecededBy(negs, retreat=100)('negs_lb')
- pattern = pp.Group(negs_pb + pp.Literal('working'))('main')
-
- results = pattern.searchString(string_test)
- try:
- print_(results.dump())
- except RecursionError:
- self.assertTrue(False, "got maximum excursion limit exception")
- else:
- self.assertTrue(True, "got maximum excursion limit exception")
-
-
-class CountedArrayTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word,nums,OneOrMore,countedArray
-
- testString = "2 5 7 6 0 1 2 3 4 5 0 3 5 4 3"
-
- integer = Word(nums).setParseAction(lambda t: int(t[0]))
- countedField = countedArray(integer)
-
- r = OneOrMore(countedField).parseString( testString )
- print_(testString)
- print_(r.asList())
-
- self.assertEqual(r.asList(), [[5,7],[0,1,2,3,4,5],[],[5,4,3]],
- "Failed matching countedArray, got " + str(r.asList()))
-
-class CountedArrayTest2(ParseTestCase):
- # addresses bug raised by Ralf Vosseler
- def runTest(self):
- from pyparsing import Word,nums,OneOrMore,countedArray
-
- testString = "2 5 7 6 0 1 2 3 4 5 0 3 5 4 3"
-
- integer = Word(nums).setParseAction(lambda t: int(t[0]))
- countedField = countedArray(integer)
-
- dummy = Word("A")
- r = OneOrMore(dummy ^ countedField).parseString( testString )
- print_(testString)
- print_(r.asList())
-
- self.assertEqual(r.asList(), [[5,7],[0,1,2,3,4,5],[],[5,4,3]],
- "Failed matching countedArray, got " + str(r.asList()))
-
-class CountedArrayTest3(ParseTestCase):
- # test case where counter is not a decimal integer
- def runTest(self):
- from pyparsing import Word,nums,OneOrMore,countedArray,alphas
- int_chars = "_"+alphas
- array_counter = Word(int_chars).setParseAction(lambda t: int_chars.index(t[0]))
-
- # 123456789012345678901234567890
- testString = "B 5 7 F 0 1 2 3 4 5 _ C 5 4 3"
-
- integer = Word(nums).setParseAction(lambda t: int(t[0]))
- countedField = countedArray(integer, intExpr=array_counter)
-
- r = OneOrMore(countedField).parseString( testString )
- print_(testString)
- print_(r.asList())
-
- self.assertEqual(r.asList(), [[5,7],[0,1,2,3,4,5],[],[5,4,3]],
- "Failed matching countedArray, got " + str(r.asList()))
-
-class LineStartTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- pass_tests = [
- """\
- AAA
- BBB
- """,
- """\
- AAA...
- BBB
- """,
- ]
- fail_tests = [
- """\
- AAA...
- ...BBB
- """,
- """\
- AAA BBB
- """,
- ]
-
- # cleanup test strings
- pass_tests = ['\n'.join(s.lstrip() for s in t.splitlines()).replace('.', ' ') for t in pass_tests]
- fail_tests = ['\n'.join(s.lstrip() for s in t.splitlines()).replace('.', ' ') for t in fail_tests]
-
- test_patt = pp.Word('A') - pp.LineStart() + pp.Word('B')
- print_(test_patt.streamline())
- success = test_patt.runTests(pass_tests)[0]
- self.assertTrue(success, "failed LineStart passing tests (1)")
-
- success = test_patt.runTests(fail_tests, failureTests=True)[0]
- self.assertTrue(success, "failed LineStart failure mode tests (1)")
-
- with AutoReset(pp.ParserElement, "DEFAULT_WHITE_CHARS"):
- print_(r'no \n in default whitespace chars')
- pp.ParserElement.setDefaultWhitespaceChars(' ')
-
- test_patt = pp.Word('A') - pp.LineStart() + pp.Word('B')
- print_(test_patt.streamline())
- # should fail the pass tests too, since \n is no longer valid whitespace and we aren't parsing for it
- success = test_patt.runTests(pass_tests, failureTests=True)[0]
- self.assertTrue(success, "failed LineStart passing tests (2)")
-
- success = test_patt.runTests(fail_tests, failureTests=True)[0]
- self.assertTrue(success, "failed LineStart failure mode tests (2)")
-
- test_patt = pp.Word('A') - pp.LineEnd().suppress() + pp.LineStart() + pp.Word('B') + pp.LineEnd().suppress()
- print_(test_patt.streamline())
- success = test_patt.runTests(pass_tests)[0]
- self.assertTrue(success, "failed LineStart passing tests (3)")
-
- success = test_patt.runTests(fail_tests, failureTests=True)[0]
- self.assertTrue(success, "failed LineStart failure mode tests (3)")
-
- test = """\
- AAA 1
- AAA 2
-
- AAA
-
- B AAA
-
- """
-
- from textwrap import dedent
- test = dedent(test)
- print_(test)
-
- for t, s, e in (pp.LineStart() + 'AAA').scanString(test):
- print_(s, e, pp.lineno(s, test), pp.line(s, test), ord(test[s]))
- print_()
- self.assertEqual(test[s], 'A', 'failed LineStart with insignificant newlines')
-
- with AutoReset(pp.ParserElement, "DEFAULT_WHITE_CHARS"):
- pp.ParserElement.setDefaultWhitespaceChars(' ')
- for t, s, e in (pp.LineStart() + 'AAA').scanString(test):
- print_(s, e, pp.lineno(s, test), pp.line(s, test), ord(test[s]))
- print_()
- self.assertEqual(test[s], 'A', 'failed LineStart with insignificant newlines')
-
-
-class LineAndStringEndTest(ParseTestCase):
- def runTest(self):
- from pyparsing import OneOrMore,lineEnd,alphanums,Word,stringEnd,delimitedList,SkipTo
-
- NLs = OneOrMore(lineEnd)
- bnf1 = delimitedList(Word(alphanums).leaveWhitespace(), NLs)
- bnf2 = Word(alphanums) + stringEnd
- bnf3 = Word(alphanums) + SkipTo(stringEnd)
- tests = [
- ("testA\ntestB\ntestC\n", ['testA', 'testB', 'testC']),
- ("testD\ntestE\ntestF", ['testD', 'testE', 'testF']),
- ("a", ['a']),
- ]
-
- for test,expected in tests:
- res1 = bnf1.parseString(test)
- print_(res1,'=?',expected)
- self.assertEqual(res1.asList(), expected,
- "Failed lineEnd/stringEnd test (1): "+repr(test)+ " -> "+str(res1.asList()))
-
- res2 = bnf2.searchString(test)[0]
- print_(res2.asList(),'=?',expected[-1:])
- self.assertEqual(res2.asList(), expected[-1:],
- "Failed lineEnd/stringEnd test (2): "+repr(test)+ " -> "+str(res2.asList()))
-
- res3 = bnf3.parseString(test)
- first = res3[0]
- rest = res3[1]
- #~ print res3.dump()
- print_(repr(rest),'=?',repr(test[len(first)+1:]))
- self.assertEqual(rest, test[len(first)+1:],
- "Failed lineEnd/stringEnd test (3): " +repr(test)+ " -> "+str(res3.asList()))
- print_()
-
- from pyparsing import Regex
- import re
-
- k = Regex(r'a+',flags=re.S+re.M)
- k = k.parseWithTabs()
- k = k.leaveWhitespace()
-
- tests = [
- (r'aaa',['aaa']),
- (r'\naaa',None),
- (r'a\naa',None),
- (r'aaa\n',None),
- ]
- for i,(src,expected) in enumerate(tests):
- print_(i, repr(src).replace('\\\\','\\'), end=' ')
- try:
- res = k.parseString(src, parseAll=True).asList()
- except ParseException as pe:
- res = None
- print_(res)
- self.assertEqual(res, expected, "Failed on parseAll=True test %d" % i)
-
-class VariableParseActionArgsTest(ParseTestCase):
- def runTest(self):
-
- pa3 = lambda s,l,t: t
- pa2 = lambda l,t: t
- pa1 = lambda t: t
- pa0 = lambda : None
- class Callable3(object):
- def __call__(self,s,l,t):
- return t
- class Callable2(object):
- def __call__(self,l,t):
- return t
- class Callable1(object):
- def __call__(self,t):
- return t
- class Callable0(object):
- def __call__(self):
- return
- class CallableS3(object):
- #~ @staticmethod
- def __call__(s,l,t):
- return t
- __call__=staticmethod(__call__)
- class CallableS2(object):
- #~ @staticmethod
- def __call__(l,t):
- return t
- __call__=staticmethod(__call__)
- class CallableS1(object):
- #~ @staticmethod
- def __call__(t):
- return t
- __call__=staticmethod(__call__)
- class CallableS0(object):
- #~ @staticmethod
- def __call__():
- return
- __call__=staticmethod(__call__)
- class CallableC3(object):
- #~ @classmethod
- def __call__(cls,s,l,t):
- return t
- __call__=classmethod(__call__)
- class CallableC2(object):
- #~ @classmethod
- def __call__(cls,l,t):
- return t
- __call__=classmethod(__call__)
- class CallableC1(object):
- #~ @classmethod
- def __call__(cls,t):
- return t
- __call__=classmethod(__call__)
- class CallableC0(object):
- #~ @classmethod
- def __call__(cls):
- return
- __call__=classmethod(__call__)
-
- class parseActionHolder(object):
- #~ @staticmethod
- def pa3(s,l,t):
- return t
- pa3=staticmethod(pa3)
- #~ @staticmethod
- def pa2(l,t):
- return t
- pa2=staticmethod(pa2)
- #~ @staticmethod
- def pa1(t):
- return t
- pa1=staticmethod(pa1)
- #~ @staticmethod
- def pa0():
- return
- pa0=staticmethod(pa0)
-
- def paArgs(*args):
- print_(args)
- return args[2]
-
- class ClassAsPA0(object):
- def __init__(self):
- pass
- def __str__(self):
- return "A"
-
- class ClassAsPA1(object):
- def __init__(self,t):
- print_("making a ClassAsPA1")
- self.t = t
- def __str__(self):
- return self.t[0]
-
- class ClassAsPA2(object):
- def __init__(self,l,t):
- self.t = t
- def __str__(self):
- return self.t[0]
-
- class ClassAsPA3(object):
- def __init__(self,s,l,t):
- self.t = t
- def __str__(self):
- return self.t[0]
-
- class ClassAsPAStarNew(tuple):
- def __new__(cls, *args):
- print_("make a ClassAsPAStarNew", args)
- return tuple.__new__(cls, *args[2].asList())
- def __str__(self):
- return ''.join(self)
-
- from pyparsing import Literal,OneOrMore
-
- A = Literal("A").setParseAction(pa0)
- B = Literal("B").setParseAction(pa1)
- C = Literal("C").setParseAction(pa2)
- D = Literal("D").setParseAction(pa3)
- E = Literal("E").setParseAction(Callable0())
- F = Literal("F").setParseAction(Callable1())
- G = Literal("G").setParseAction(Callable2())
- H = Literal("H").setParseAction(Callable3())
- I = Literal("I").setParseAction(CallableS0())
- J = Literal("J").setParseAction(CallableS1())
- K = Literal("K").setParseAction(CallableS2())
- L = Literal("L").setParseAction(CallableS3())
- M = Literal("M").setParseAction(CallableC0())
- N = Literal("N").setParseAction(CallableC1())
- O = Literal("O").setParseAction(CallableC2())
- P = Literal("P").setParseAction(CallableC3())
- Q = Literal("Q").setParseAction(paArgs)
- R = Literal("R").setParseAction(parseActionHolder.pa3)
- S = Literal("S").setParseAction(parseActionHolder.pa2)
- T = Literal("T").setParseAction(parseActionHolder.pa1)
- U = Literal("U").setParseAction(parseActionHolder.pa0)
- V = Literal("V")
-
- gg = OneOrMore( A | C | D | E | F | G | H |
- I | J | K | L | M | N | O | P | Q | R | S | U | V | B | T)
- testString = "VUTSRQPONMLKJIHGFEDCBA"
- res = gg.parseString(testString)
- print_(res.asList())
- self.assertEqual(res.asList(), list(testString), "Failed to parse using variable length parse actions")
-
- A = Literal("A").setParseAction(ClassAsPA0)
- B = Literal("B").setParseAction(ClassAsPA1)
- C = Literal("C").setParseAction(ClassAsPA2)
- D = Literal("D").setParseAction(ClassAsPA3)
- E = Literal("E").setParseAction(ClassAsPAStarNew)
-
- gg = OneOrMore( A | B | C | D | E | F | G | H |
- I | J | K | L | M | N | O | P | Q | R | S | T | U | V)
- testString = "VUTSRQPONMLKJIHGFEDCBA"
- res = gg.parseString(testString)
- print_(list(map(str,res)))
- self.assertEqual(list(map(str,res)), list(testString),
- "Failed to parse using variable length parse actions "
- "using class constructors as parse actions")
-
-class EnablePackratParsing(ParseTestCase):
- def runTest(self):
- from pyparsing import ParserElement
- ParserElement.enablePackrat()
-
-class SingleArgExceptionTest(ParseTestCase):
- def runTest(self):
- from pyparsing import ParseBaseException,ParseFatalException
-
- msg = ""
- raisedMsg = ""
- testMessage = "just one arg"
- try:
- raise ParseFatalException(testMessage)
- except ParseBaseException as pbe:
- print_("Received expected exception:", pbe)
- raisedMsg = pbe.msg
- self.assertEqual(raisedMsg, testMessage, "Failed to get correct exception message")
-
-
-class OriginalTextForTest(ParseTestCase):
- def runTest(self):
- from pyparsing import makeHTMLTags, originalTextFor
-
- def rfn(t):
- return "%s:%d" % (t.src, len("".join(t)))
-
- makeHTMLStartTag = lambda tag: originalTextFor(makeHTMLTags(tag)[0], asString=False)
-
- # use the lambda, Luke
- start = makeHTMLStartTag('IMG')
-
- # don't replace our fancy parse action with rfn,
- # append rfn to the list of parse actions
- start.addParseAction(rfn)
-
- text = '''_<img src="images/cal.png"
- alt="cal image" width="16" height="15">_'''
- s = start.transformString(text)
- if VERBOSE:
- print_(s)
- self.assertTrue(s.startswith("_images/cal.png:"), "failed to preserve input s properly")
- self.assertTrue(s.endswith("77_"),"failed to return full original text properly")
-
- tag_fields = makeHTMLStartTag("IMG").searchString(text)[0]
- if VERBOSE:
- print_(sorted(tag_fields.keys()))
- self.assertEqual(sorted(tag_fields.keys()),
- ['alt', 'empty', 'height', 'src', 'startImg', 'tag', 'width'],
- 'failed to preserve results names in originalTextFor')
-
-class PackratParsingCacheCopyTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word,nums,delimitedList,Literal,Optional,alphas,alphanums,ZeroOrMore,empty
-
- integer = Word(nums).setName("integer")
- id = Word(alphas+'_',alphanums+'_')
- simpleType = Literal('int');
- arrayType= simpleType+ZeroOrMore('['+delimitedList(integer)+']')
- varType = arrayType | simpleType
- varDec = varType + delimitedList(id + Optional('='+integer))+';'
-
- codeBlock = Literal('{}')
-
- funcDef = Optional(varType | 'void')+id+'('+(delimitedList(varType+id)|'void'|empty)+')'+codeBlock
-
- program = varDec | funcDef
- input = 'int f(){}'
- results = program.parseString(input)
- print_("Parsed '%s' as %s" % (input, results.asList()))
- self.assertEqual(results.asList(), ['int', 'f', '(', ')', '{}'], "Error in packrat parsing")
-
-class PackratParsingCacheCopyTest2(ParseTestCase):
- def runTest(self):
- from pyparsing import Keyword,Word,Suppress,Forward,Optional,delimitedList,Group
-
- DO,AA = list(map(Keyword, "DO AA".split()))
- LPAR,RPAR = list(map(Suppress,"()"))
- identifier = ~AA + Word("Z")
-
- function_name = identifier.copy()
- #~ function_name = ~AA + Word("Z") #identifier.copy()
- expr = Forward().setName("expr")
- expr << (Group(function_name + LPAR + Optional(delimitedList(expr)) + RPAR).setName("functionCall") |
- identifier.setName("ident")#.setDebug()#.setBreak()
- )
-
- stmt = DO + Group(delimitedList(identifier + ".*" | expr))
- result = stmt.parseString("DO Z")
- print_(result.asList())
- self.assertEqual(len(result[1]), 1, "packrat parsing is duplicating And term exprs")
-
-class ParseResultsDelTest(ParseTestCase):
- def runTest(self):
- from pyparsing import OneOrMore, Word, alphas, nums
-
- grammar = OneOrMore(Word(nums))("ints") + OneOrMore(Word(alphas))("words")
- res = grammar.parseString("123 456 ABC DEF")
- print_(res.dump())
- origInts = res.ints.asList()
- origWords = res.words.asList()
- del res[1]
- del res["words"]
- print_(res.dump())
- self.assertEqual(res[1], 'ABC',"failed to delete 0'th element correctly")
- self.assertEqual(res.ints.asList(), origInts, "updated named attributes, should have updated list only")
- self.assertEqual(res.words, "", "failed to update named attribute correctly")
- self.assertEqual(res[-1], 'DEF', "updated list, should have updated named attributes only")
-
-class WithAttributeParseActionTest(ParseTestCase):
- def runTest(self):
- """
- This unit test checks withAttribute in these ways:
-
- * Argument forms as keywords and tuples
- * Selecting matching tags by attribute
- * Case-insensitive attribute matching
- * Correctly matching tags having the attribute, and rejecting tags not having the attribute
-
- (Unit test written by voigts as part of the Google Highly Open Participation Contest)
- """
-
- from pyparsing import makeHTMLTags, Word, withAttribute, withClass, nums
-
- data = """
- <a>1</a>
- <a b="x">2</a>
- <a B="x">3</a>
- <a b="X">4</a>
- <a b="y">5</a>
- <a class="boo">8</ a>
- """
- tagStart, tagEnd = makeHTMLTags("a")
-
- expr = tagStart + Word(nums)("value") + tagEnd
-
- expected = ([['a', ['b', 'x'], False, '2', '</a>'],
- ['a', ['b', 'x'], False, '3', '</a>']],
- [['a', ['b', 'x'], False, '2', '</a>'],
- ['a', ['b', 'x'], False, '3', '</a>']],
- [['a', ['class', 'boo'], False, '8', '</a>']],
- )
-
- for attrib, exp in zip([
- withAttribute(b="x"),
- #withAttribute(B="x"),
- withAttribute(("b","x")),
- #withAttribute(("B","x")),
- withClass("boo"),
- ], expected):
-
- tagStart.setParseAction(attrib)
- result = expr.searchString(data)
-
- print_(result.dump())
- self.assertEqual(result.asList(), exp, "Failed test, expected %s, got %s" % (expected, result.asList()))
-
-class NestedExpressionsTest(ParseTestCase):
- def runTest(self):
- """
- This unit test checks nestedExpr in these ways:
- - use of default arguments
- - use of non-default arguments (such as a pyparsing-defined comment
- expression in place of quotedString)
- - use of a custom content expression
- - use of a pyparsing expression for opener and closer is *OPTIONAL*
- - use of input data containing nesting delimiters
- - correct grouping of parsed tokens according to nesting of opening
- and closing delimiters in the input string
-
- (Unit test written by christoph... as part of the Google Highly Open Participation Contest)
- """
- from pyparsing import nestedExpr, Literal, Regex, restOfLine, quotedString
-
- #All defaults. Straight out of the example script. Also, qualifies for
- #the bonus: note the fact that (Z | (E^F) & D) is not parsed :-).
- # Tests for bug fixed in 1.4.10
- print_("Test defaults:")
- teststring = "(( ax + by)*C) (Z | (E^F) & D)"
-
- expr = nestedExpr()
-
- expected = [[['ax', '+', 'by'], '*C']]
- result = expr.parseString(teststring)
- print_(result.dump())
- self.assertEqual(result.asList(), expected, "Defaults didn't work. That's a bad sign. Expected: %s, got: %s" % (expected, result))
-
- #Going through non-defaults, one by one; trying to think of anything
- #odd that might not be properly handled.
-
- #Change opener
- print_("\nNon-default opener")
- opener = "["
- teststring = test_string = "[[ ax + by)*C)"
- expected = [[['ax', '+', 'by'], '*C']]
- expr = nestedExpr("[")
- result = expr.parseString(teststring)
- print_(result.dump())
- self.assertEqual(result.asList(), expected, "Non-default opener didn't work. Expected: %s, got: %s" % (expected, result))
-
- #Change closer
- print_("\nNon-default closer")
-
- teststring = test_string = "(( ax + by]*C]"
- expected = [[['ax', '+', 'by'], '*C']]
- expr = nestedExpr(closer="]")
- result = expr.parseString(teststring)
- print_(result.dump())
- self.assertEqual(result.asList(), expected, "Non-default closer didn't work. Expected: %s, got: %s" % (expected, result))
-
- # #Multicharacter opener, closer
- # opener = "bar"
- # closer = "baz"
- print_("\nLiteral expressions for opener and closer")
-
- opener,closer = list(map(Literal, "bar baz".split()))
- expr = nestedExpr(opener, closer,
- content=Regex(r"([^b ]|b(?!a)|ba(?![rz]))+"))
-
- teststring = "barbar ax + bybaz*Cbaz"
- expected = [[['ax', '+', 'by'], '*C']]
- # expr = nestedExpr(opener, closer)
- result = expr.parseString(teststring)
- print_(result.dump())
- self.assertEqual(result.asList(), expected, "Multicharacter opener and closer didn't work. Expected: %s, got: %s" % (expected, result))
-
- #Lisp-ish comments
- print_("\nUse ignore expression (1)")
- comment = Regex(r";;.*")
- teststring = \
- """
- (let ((greeting "Hello, world!")) ;;(foo bar
- (display greeting))
- """
-
- expected = [['let', [['greeting', '"Hello,', 'world!"']], ';;(foo bar',\
- ['display', 'greeting']]]
- expr = nestedExpr(ignoreExpr=comment)
- result = expr.parseString(teststring)
- print_(result.dump())
- self.assertEqual(result.asList(), expected , "Lisp-ish comments (\";; <...> $\") didn't work. Expected: %s, got: %s" % (expected, result))
-
-
- #Lisp-ish comments, using a standard bit of pyparsing, and an Or.
- print_("\nUse ignore expression (2)")
- comment = ';;' + restOfLine
-
- teststring = \
- """
- (let ((greeting "Hello, )world!")) ;;(foo bar
- (display greeting))
- """
-
- expected = [['let', [['greeting', '"Hello, )world!"']], ';;', '(foo bar',
- ['display', 'greeting']]]
- expr = nestedExpr(ignoreExpr=(comment ^ quotedString))
- result = expr.parseString(teststring)
- print_(result.dump())
- self.assertEqual(result.asList(), expected ,
- "Lisp-ish comments (\";; <...> $\") and quoted strings didn't work. Expected: %s, got: %s" % (expected, result))
-
-class WordExcludeTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word, printables
- allButPunc = Word(printables, excludeChars=".,:;-_!?")
-
- test = "Hello, Mr. Ed, it's Wilbur!"
- result = allButPunc.searchString(test).asList()
- print_(result)
- self.assertEqual(result, [['Hello'], ['Mr'], ['Ed'], ["it's"], ['Wilbur']], "failed WordExcludeTest")
-
-class ParseAllTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word, cppStyleComment
-
- testExpr = Word("A")
-
- tests = [
- ("AAAAA", False, True),
- ("AAAAA", True, True),
- ("AAABB", False, True),
- ("AAABB", True, False),
- ]
- for s,parseAllFlag,shouldSucceed in tests:
- try:
- print_("'%s' parseAll=%s (shouldSuceed=%s)" % (s, parseAllFlag, shouldSucceed))
- testExpr.parseString(s,parseAllFlag)
- self.assertTrue(shouldSucceed, "successfully parsed when should have failed")
- except ParseException as pe:
- self.assertFalse(shouldSucceed, "failed to parse when should have succeeded")
-
- # add test for trailing comments
- testExpr.ignore(cppStyleComment)
-
- tests = [
- ("AAAAA //blah", False, True),
- ("AAAAA //blah", True, True),
- ("AAABB //blah", False, True),
- ("AAABB //blah", True, False),
- ]
- for s,parseAllFlag,shouldSucceed in tests:
- try:
- print_("'%s' parseAll=%s (shouldSucceed=%s)" % (s, parseAllFlag, shouldSucceed))
- testExpr.parseString(s,parseAllFlag)
- self.assertTrue(shouldSucceed, "successfully parsed when should have failed")
- except ParseException as pe:
- self.assertFalse(shouldSucceed, "failed to parse when should have succeeded")
-
-class GreedyQuotedStringsTest(ParseTestCase):
- def runTest(self):
- from pyparsing import QuotedString, sglQuotedString, dblQuotedString, quotedString, delimitedList
-
- src = """\
- "string1", "strin""g2"
- 'string1', 'string2'
- ^string1^, ^string2^
- <string1>, <string2>"""
-
- testExprs = (sglQuotedString, dblQuotedString, quotedString,
- QuotedString('"', escQuote='""'), QuotedString("'", escQuote="''"),
- QuotedString("^"), QuotedString("<",endQuoteChar=">"))
- for expr in testExprs:
- strs = delimitedList(expr).searchString(src)
- print_(strs)
- self.assertTrue(bool(strs), "no matches found for test expression '%s'" % expr)
- for lst in strs:
- self.assertEqual(len(lst), 2, "invalid match found for test expression '%s'" % expr)
-
- from pyparsing import alphas, nums, Word
- src = """'ms1',1,0,'2009-12-22','2009-12-22 10:41:22') ON DUPLICATE KEY UPDATE sent_count = sent_count + 1, mtime = '2009-12-22 10:41:22';"""
- tok_sql_quoted_value = (
- QuotedString("'", "\\", "''", True, False) ^
- QuotedString('"', "\\", '""', True, False))
- tok_sql_computed_value = Word(nums)
- tok_sql_identifier = Word(alphas)
-
- val = tok_sql_quoted_value | tok_sql_computed_value | tok_sql_identifier
- vals = delimitedList(val)
- print_(vals.parseString(src))
- self.assertEqual(len(vals.parseString(src)), 5, "error in greedy quote escaping")
-
-
-class WordBoundaryExpressionsTest(ParseTestCase):
- def runTest(self):
- from pyparsing import WordEnd, WordStart, oneOf
-
- ws = WordStart()
- we = WordEnd()
- vowel = oneOf(list("AEIOUY"))
- consonant = oneOf(list("BCDFGHJKLMNPQRSTVWXZ"))
-
- leadingVowel = ws + vowel
- trailingVowel = vowel + we
- leadingConsonant = ws + consonant
- trailingConsonant = consonant + we
- internalVowel = ~ws + vowel + ~we
-
- bnf = leadingVowel | trailingVowel
-
- tests = """\
- ABC DEF GHI
- JKL MNO PQR
- STU VWX YZ """.splitlines()
- tests.append( "\n".join(tests) )
-
- expectedResult = [
- [['D', 'G'], ['A'], ['C', 'F'], ['I'], ['E'], ['A', 'I']],
- [['J', 'M', 'P'], [], ['L', 'R'], ['O'], [], ['O']],
- [['S', 'V'], ['Y'], ['X', 'Z'], ['U'], [], ['U', 'Y']],
- [['D', 'G', 'J', 'M', 'P', 'S', 'V'],
- ['A', 'Y'],
- ['C', 'F', 'L', 'R', 'X', 'Z'],
- ['I', 'O', 'U'],
- ['E'],
- ['A', 'I', 'O', 'U', 'Y']],
- ]
-
- for t,expected in zip(tests, expectedResult):
- print_(t)
- results = [flatten(e.searchString(t).asList()) for e in [
- leadingConsonant,
- leadingVowel,
- trailingConsonant,
- trailingVowel,
- internalVowel,
- bnf,
- ]]
- print_(results)
- print_()
- self.assertEqual(results, expected,"Failed WordBoundaryTest, expected %s, got %s" % (expected,results))
-
-class RequiredEachTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Keyword
-
- parser = Keyword('bam') & Keyword('boo')
- try:
- res1 = parser.parseString('bam boo')
- print_(res1.asList())
- res2 = parser.parseString('boo bam')
- print_(res2.asList())
- except ParseException:
- failed = True
- else:
- failed = False
- self.assertFalse(failed, "invalid logic in Each")
-
- self.assertEqual(set(res1), set(res2), "Failed RequiredEachTest, expected "
- + str(res1.asList()) + " and " + str(res2.asList())
- + "to contain same words in any order" )
-
-class OptionalEachTest(ParseTestCase):
- def runTest1(self):
- from pyparsing import Optional, Keyword
-
- for the_input in [
- "Tal Weiss Major",
- "Tal Major",
- "Weiss Major",
- "Major",
- "Major Tal",
- "Major Weiss",
- "Major Tal Weiss",
- ]:
- print_(the_input)
- parser1 = (Optional("Tal") + Optional("Weiss")) & Keyword("Major")
- parser2 = Optional(Optional("Tal") + Optional("Weiss")) & Keyword("Major")
- p1res = parser1.parseString(the_input)
- p2res = parser2.parseString(the_input)
- self.assertEqual(
- p1res.asList(),
- p2res.asList(),
- "Each failed to match with nested Optionals, "
- + str(p1res.asList())
- + " should match "
- + str(p2res.asList()),
- )
-
- def runTest2(self):
- from pyparsing import Word, alphanums, OneOrMore, Group, Regex, Optional
-
- word = Word(alphanums + '_').setName("word")
- with_stmt = 'with' + OneOrMore(Group(word('key') + '=' + word('value')))('overrides')
- using_stmt = 'using' + Regex('id-[0-9a-f]{8}')('id')
- modifiers = Optional(with_stmt('with_stmt')) & Optional(using_stmt('using_stmt'))
-
- self.assertEqual(modifiers, "with foo=bar bing=baz using id-deadbeef")
- self.assertNotEqual(modifiers, "with foo=bar bing=baz using id-deadbeef using id-feedfeed")
-
- def runTest3(self):
- from pyparsing import Literal,Suppress,ZeroOrMore,OneOrMore
-
- foo = Literal('foo')
- bar = Literal('bar')
-
- openBrace = Suppress(Literal("{"))
- closeBrace = Suppress(Literal("}"))
-
- exp = openBrace + (OneOrMore(foo)("foo") & ZeroOrMore(bar)("bar")) + closeBrace
-
- tests = """\
- {foo}
- {bar foo bar foo bar foo}
- """.splitlines()
- for test in tests:
- test = test.strip()
- if not test:
- continue
- result = exp.parseString(test)
- print_(test, '->', result.asList())
- self.assertEqual(result.asList(), test.strip("{}").split(), "failed to parse Each expression %r" % test)
- print_(result.dump())
-
- try:
- result = exp.parseString("{bar}")
- self.assertTrue(False, "failed to raise exception when required element is missing")
- except ParseException as pe:
- pass
-
- def runTest4(self):
- from pyparsing import pyparsing_common, ZeroOrMore, Group
-
- expr = ((~pyparsing_common.iso8601_date + pyparsing_common.integer("id"))
- & ZeroOrMore(Group(pyparsing_common.iso8601_date)("date*")))
-
- expr.runTests("""
- 1999-12-31 100 2001-01-01
- 42
- """)
-
- def testParseExpressionsWithRegex(self):
- from itertools import product
- match_empty_regex = pp.Regex(r"[a-z]*")
- match_nonempty_regex = pp.Regex(r"[a-z]+")
-
- parser_classes = pp.ParseExpression.__subclasses__()
- test_string = "abc def"
- expected = ["abc"]
- for expr, cls in product((match_nonempty_regex, match_empty_regex), parser_classes):
- print_(expr, cls)
- parser = cls([expr])
- parsed_result = parser.parseString(test_string)
- print_(parsed_result.dump())
- self.assertParseResultsEquals(parsed_result, expected)
-
- for expr, cls in product((match_nonempty_regex, match_empty_regex), (pp.MatchFirst, pp.Or)):
- parser = cls([expr, expr])
- print_(parser)
- parsed_result = parser.parseString(test_string)
- print_(parsed_result.dump())
- self.assertParseResultsEquals(parsed_result, expected)
-
- def runTest(self):
- self.runTest1()
- self.runTest2()
- self.runTest3()
- self.runTest4()
- self.testParseExpressionsWithRegex()
-
-class SumParseResultsTest(ParseTestCase):
- def runTest(self):
-
- samplestr1 = "garbage;DOB 10-10-2010;more garbage\nID PARI12345678;more garbage"
- samplestr2 = "garbage;ID PARI12345678;more garbage\nDOB 10-10-2010;more garbage"
- samplestr3 = "garbage;DOB 10-10-2010"
- samplestr4 = "garbage;ID PARI12345678;more garbage- I am cool"
-
- res1 = "ID:PARI12345678 DOB:10-10-2010 INFO:"
- res2 = "ID:PARI12345678 DOB:10-10-2010 INFO:"
- res3 = "ID: DOB:10-10-2010 INFO:"
- res4 = "ID:PARI12345678 DOB: INFO: I am cool"
-
- from pyparsing import Regex, Word, alphanums, restOfLine
- dob_ref = "DOB" + Regex(r"\d{2}-\d{2}-\d{4}")("dob")
- id_ref = "ID" + Word(alphanums,exact=12)("id")
- info_ref = "-" + restOfLine("info")
-
- person_data = dob_ref | id_ref | info_ref
-
- tests = (samplestr1,samplestr2,samplestr3,samplestr4,)
- results = (res1, res2, res3, res4,)
- for test,expected in zip(tests, results):
- person = sum(person_data.searchString(test))
- result = "ID:%s DOB:%s INFO:%s" % (person.id, person.dob, person.info)
- print_(test)
- print_(expected)
- print_(result)
- for pd in person_data.searchString(test):
- print_(pd.dump())
- print_()
- self.assertEqual(expected, result,
- "Failed to parse '%s' correctly, \nexpected '%s', got '%s'" % (test,expected,result))
-
-class MarkInputLineTest(ParseTestCase):
- def runTest(self):
-
- samplestr1 = "DOB 100-10-2010;more garbage\nID PARI12345678;more garbage"
-
- from pyparsing import Regex
- dob_ref = "DOB" + Regex(r"\d{2}-\d{2}-\d{4}")("dob")
-
- try:
- res = dob_ref.parseString(samplestr1)
- except ParseException as pe:
- outstr = pe.markInputline()
- print_(outstr)
- self.assertEqual(outstr, "DOB >!<100-10-2010;more garbage", "did not properly create marked input line")
- else:
- self.assertEqual(False, "test construction failed - should have raised an exception")
-
-class LocatedExprTest(ParseTestCase):
- def runTest(self):
-
- # 012345678901234567890123456789012345678901234567890
- samplestr1 = "DOB 10-10-2010;more garbage;ID PARI12345678 ;more garbage"
-
- from pyparsing import Word, alphanums, locatedExpr
- id_ref = locatedExpr("ID" + Word(alphanums,exact=12)("id"))
-
- res = id_ref.searchString(samplestr1)[0][0]
- print_(res.dump())
- self.assertEqual(samplestr1[res.locn_start:res.locn_end], 'ID PARI12345678', "incorrect location calculation")
-
-
-class PopTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word, alphas, nums
-
- source = "AAA 123 456 789 234"
- patt = Word(alphas)("name") + Word(nums)*(1,)
-
- result = patt.parseString(source)
- tests = [
- (0, 'AAA', ['123', '456', '789', '234']),
- (None, '234', ['123', '456', '789']),
- ('name', 'AAA', ['123', '456', '789']),
- (-1, '789', ['123', '456']),
- ]
- for test in tests:
- idx, val, remaining = test
- if idx is not None:
- ret = result.pop(idx)
- else:
- ret = result.pop()
- print_("EXP:", val, remaining)
- print_("GOT:", ret, result.asList())
- print_(ret, result.asList())
- self.assertEqual(ret, val, "wrong value returned, got %r, expected %r" % (ret, val))
- self.assertEqual(remaining, result.asList(),
- "list is in wrong state after pop, got %r, expected %r" % (result.asList(), remaining))
- print_()
-
- prevlist = result.asList()
- ret = result.pop('name', default="noname")
- print_(ret)
- print_(result.asList())
- self.assertEqual(ret, "noname",
- "default value not successfully returned, got %r, expected %r" % (ret, "noname"))
- self.assertEqual(result.asList(), prevlist,
- "list is in wrong state after pop, got %r, expected %r" % (result.asList(), remaining))
-
-
-class AddConditionTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word, nums, Suppress, ParseFatalException
-
- numParser = Word(nums)
- numParser.addParseAction(lambda s,l,t: int(t[0]))
- numParser.addCondition(lambda s,l,t: t[0] % 2)
- numParser.addCondition(lambda s,l,t: t[0] >= 7)
-
- result = numParser.searchString("1 2 3 4 5 6 7 8 9 10")
- print_(result.asList())
- self.assertEqual(result.asList(), [[7],[9]], "failed to properly process conditions")
-
- numParser = Word(nums)
- numParser.addParseAction(lambda s,l,t: int(t[0]))
- rangeParser = (numParser("from_") + Suppress('-') + numParser("to"))
-
- result = rangeParser.searchString("1-4 2-4 4-3 5 6 7 8 9 10")
- print_(result.asList())
- self.assertEqual(result.asList(), [[1, 4], [2, 4], [4, 3]], "failed to properly process conditions")
-
- rangeParser.addCondition(lambda t: t.to > t.from_, message="from must be <= to", fatal=False)
- result = rangeParser.searchString("1-4 2-4 4-3 5 6 7 8 9 10")
- print_(result.asList())
- self.assertEqual(result.asList(), [[1, 4], [2, 4]], "failed to properly process conditions")
-
- rangeParser = (numParser("from_") + Suppress('-') + numParser("to"))
- rangeParser.addCondition(lambda t: t.to > t.from_, message="from must be <= to", fatal=True)
- try:
- result = rangeParser.searchString("1-4 2-4 4-3 5 6 7 8 9 10")
- self.assertTrue(False, "failed to interrupt parsing on fatal condition failure")
- except ParseFatalException:
- print_("detected fatal condition")
-
-class PatientOrTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- # Two expressions and a input string which could - syntactically - be matched against
- # both expressions. The "Literal" expression is considered invalid though, so this PE
- # should always detect the "Word" expression.
- def validate(token):
- if token[0] == "def":
- raise pp.ParseException("signalling invalid token")
- return token
-
- a = pp.Word("de").setName("Word")#.setDebug()
- b = pp.Literal("def").setName("Literal").setParseAction(validate)#.setDebug()
- c = pp.Literal("d").setName("d")#.setDebug()
-
- # The "Literal" expressions's ParseAction is not executed directly after syntactically
- # detecting the "Literal" Expression but only after the Or-decision has been made
- # (which is too late)...
- try:
- result = (a ^ b ^ c).parseString("def")
- self.assertEqual(result.asList(), ['de'], "failed to select longest match, chose %s" % result)
- except ParseException:
- failed = True
- else:
- failed = False
- self.assertFalse(failed, "invalid logic in Or, fails on longest match with exception in parse action")
-
- # from issue #93
- word = pp.Word(pp.alphas).setName('word')
- word_1 = pp.Word(pp.alphas).setName('word_1').addCondition(lambda t: len(t[0]) == 1)
-
- a = word + (word_1 + word ^ word)
- b = word * 3
- c = a ^ b
- c.streamline()
- print_(c)
- test_string = 'foo bar temp'
- result = c.parseString(test_string)
- print_(test_string, '->', result.asList())
-
- self.assertEqual(result.asList(), test_string.split(), "failed to match longest choice")
-
-
-class EachWithOptionalWithResultsNameTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Optional
-
- result = (Optional('foo')('one') & Optional('bar')('two')).parseString('bar foo')
- print_(result.dump())
- self.assertEqual(sorted(result.keys()), ['one','two'])
-
-class UnicodeExpressionTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Literal, ParseException
-
- z = 'a' | Literal(u'\u1111')
- z.streamline()
- try:
- z.parseString('b')
- except ParseException as pe:
- if not PY_3:
- self.assertEqual(pe.msg, r'''Expected {"a" | "\u1111"}''',
- "Invalid error message raised, got %r" % pe.msg)
- else:
- self.assertEqual(pe.msg, r'''Expected {"a" | "ᄑ"}''',
- "Invalid error message raised, got %r" % pe.msg)
-
-class SetNameTest(ParseTestCase):
- def runTest(self):
- from pyparsing import (oneOf,infixNotation,Word,nums,opAssoc,delimitedList,countedArray,
- nestedExpr,makeHTMLTags,anyOpenTag,anyCloseTag,commonHTMLEntity,replaceHTMLEntity,
- Forward,ZeroOrMore)
-
- a = oneOf("a b c")
- b = oneOf("d e f")
- arith_expr = infixNotation(Word(nums),
- [
- (oneOf('* /'),2,opAssoc.LEFT),
- (oneOf('+ -'),2,opAssoc.LEFT),
- ])
- arith_expr2 = infixNotation(Word(nums),
- [
- (('?',':'),3,opAssoc.LEFT),
- ])
- recursive = Forward()
- recursive <<= a + ZeroOrMore(b + recursive)
-
- tests = [
- a,
- b,
- (a | b),
- arith_expr,
- arith_expr.expr,
- arith_expr2,
- arith_expr2.expr,
- recursive,
- delimitedList(Word(nums).setName("int")),
- countedArray(Word(nums).setName("int")),
- nestedExpr(),
- makeHTMLTags('Z'),
- (anyOpenTag,anyCloseTag),
- commonHTMLEntity,
- commonHTMLEntity.setParseAction(replaceHTMLEntity).transformString("lsdjkf <lsdjkf>&'"&xyzzy;"),
- ]
-
- expected = map(str.strip, """\
- a | b | c
- d | e | f
- {a | b | c | d | e | f}
- Forward: + | - term
- + | - term
- Forward: ?: term
- ?: term
- Forward: {a | b | c [{d | e | f : ...}]...}
- int [, int]...
- (len) int...
- nested () expression
- (<Z>, </Z>)
- (<any tag>, </any tag>)
- common HTML entity
- lsdjkf <lsdjkf>&'"&xyzzy;""".splitlines())
-
- for t,e in zip(tests, expected):
- tname = str(t)
- print_(tname)
- self.assertEqual(tname, e, "expression name mismatch, expected {0} got {1}".format(e, tname))
-
-class TrimArityExceptionMaskingTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word
-
- invalid_message = [
- "<lambda>() takes exactly 1 argument (0 given)",
- "<lambda>() missing 1 required positional argument: 't'"
- ][PY_3]
- try:
- Word('a').setParseAction(lambda t: t[0]+1).parseString('aaa')
- except Exception as e:
- exc_msg = str(e)
- self.assertNotEqual(exc_msg, invalid_message, "failed to catch TypeError thrown in _trim_arity")
-
-class TrimArityExceptionMaskingTest2(ParseTestCase):
- def runTest(self):
- # construct deep call tree
- def A():
- import traceback
-
- traceback.print_stack(limit=2)
-
- from pyparsing import Word
-
- invalid_message = [
- "<lambda>() takes exactly 1 argument (0 given)",
- "<lambda>() missing 1 required positional argument: 't'"
- ][PY_3]
- try:
- Word('a').setParseAction(lambda t: t[0]+1).parseString('aaa')
- except Exception as e:
- exc_msg = str(e)
- self.assertNotEqual(exc_msg, invalid_message, "failed to catch TypeError thrown in _trim_arity")
-
-
- def B():
- A()
-
- def C():
- B()
-
- def D():
- C()
-
- def E():
- D()
-
- def F():
- E()
-
- def G():
- F()
-
- def H():
- G()
-
- def J():
- H()
-
- def K():
- J()
-
- K()
-
-
-class ClearParseActionsTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- ppc = pp.pyparsing_common
-
- realnum = ppc.real()
- self.assertEqual(realnum.parseString("3.14159")[0], 3.14159, "failed basic real number parsing")
-
- # clear parse action that converts to float
- realnum.setParseAction(None)
- self.assertEqual(realnum.parseString("3.14159")[0], "3.14159", "failed clearing parse action")
-
- # add a new parse action that tests if a '.' is prsent
- realnum.addParseAction(lambda t: '.' in t[0])
- self.assertEqual(realnum.parseString("3.14159")[0], True,
- "failed setting new parse action after clearing parse action")
-
-class OneOrMoreStopTest(ParseTestCase):
- def runTest(self):
- from pyparsing import (Word, OneOrMore, alphas, Keyword, CaselessKeyword,
- nums, alphanums)
-
- test = "BEGIN aaa bbb ccc END"
- BEGIN,END = map(Keyword, "BEGIN,END".split(','))
- body_word = Word(alphas).setName("word")
- for ender in (END, "END", CaselessKeyword("END")):
- expr = BEGIN + OneOrMore(body_word, stopOn=ender) + END
- self.assertEqual(test, expr, "Did not successfully stop on ending expression %r" % ender)
-
- if PY_3:
- expr = eval('BEGIN + body_word[...].stopOn(ender) + END')
- self.assertEqual(test, expr, "Did not successfully stop on ending expression %r" % ender)
-
- number = Word(nums+',.()').setName("number with optional commas")
- parser= (OneOrMore(Word(alphanums+'-/.'), stopOn=number)('id').setParseAction(' '.join)
- + number('data'))
- result = parser.parseString(' XXX Y/123 1,234.567890')
- self.assertEqual(result.asList(), ['XXX Y/123', '1,234.567890'],
- "Did not successfully stop on ending expression %r" % number)
-
-class ZeroOrMoreStopTest(ParseTestCase):
- def runTest(self):
- from pyparsing import (Word, ZeroOrMore, alphas, Keyword, CaselessKeyword)
-
- test = "BEGIN END"
- BEGIN,END = map(Keyword, "BEGIN,END".split(','))
- body_word = Word(alphas).setName("word")
- for ender in (END, "END", CaselessKeyword("END")):
- expr = BEGIN + ZeroOrMore(body_word, stopOn=ender) + END
- self.assertEqual(test, expr, "Did not successfully stop on ending expression %r" % ender)
-
- if PY_3:
- expr = eval('BEGIN + body_word[0, ...].stopOn(ender) + END')
- self.assertEqual(test, expr, "Did not successfully stop on ending expression %r" % ender)
-
-class NestedAsDictTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Literal, Forward, alphanums, Group, delimitedList, Dict, Word, Optional
-
- equals = Literal("=").suppress()
- lbracket = Literal("[").suppress()
- rbracket = Literal("]").suppress()
- lbrace = Literal("{").suppress()
- rbrace = Literal("}").suppress()
-
- value_dict = Forward()
- value_list = Forward()
- value_string = Word(alphanums + "@. ")
-
- value = value_list ^ value_dict ^ value_string
- values = Group(delimitedList(value, ","))
- #~ values = delimitedList(value, ",").setParseAction(lambda toks: [toks.asList()])
-
- value_list << lbracket + values + rbracket
-
- identifier = Word(alphanums + "_.")
-
- assignment = Group(identifier + equals + Optional(value))
- assignments = Dict(delimitedList(assignment, ';'))
- value_dict << lbrace + assignments + rbrace
-
- response = assignments
-
- rsp = 'username=goat; errors={username=[already taken, too short]}; empty_field='
- result_dict = response.parseString(rsp).asDict()
- print_(result_dict)
- self.assertEqual(result_dict['username'], 'goat', "failed to process string in ParseResults correctly")
- self.assertEqual(result_dict['errors']['username'], ['already taken', 'too short'],
- "failed to process nested ParseResults correctly")
-
-class TraceParseActionDecoratorTest(ParseTestCase):
- def runTest(self):
- from pyparsing import traceParseAction, Word, nums
-
- @traceParseAction
- def convert_to_int(t):
- return int(t[0])
-
- class Z(object):
- def __call__(self, other):
- return other[0] * 1000
-
- integer = Word(nums).addParseAction(convert_to_int)
- integer.addParseAction(traceParseAction(lambda t: t[0]*10))
- integer.addParseAction(traceParseAction(Z()))
- integer.parseString("132")
-
-class RunTestsTest(ParseTestCase):
- def runTest(self):
- from pyparsing import Word, nums, delimitedList
-
- integer = Word(nums).setParseAction(lambda t : int(t[0]))
- intrange = integer("start") + '-' + integer("end")
- intrange.addCondition(lambda t: t.end > t.start, message="invalid range, start must be <= end", fatal=True)
- intrange.addParseAction(lambda t: list(range(t.start, t.end+1)))
-
- indices = delimitedList(intrange | integer)
- indices.addParseAction(lambda t: sorted(set(t)))
-
- tests = """\
- # normal data
- 1-3,2-4,6,8-10,16
-
- # lone integer
- 11"""
- results = indices.runTests(tests, printResults=False)[1]
-
- expectedResults = [
- [1, 2, 3, 4, 6, 8, 9, 10, 16],
- [11],
- ]
- for res, expected in zip(results, expectedResults):
- print_(res[1].asList())
- print_(expected)
- self.assertEqual(res[1].asList(), expected, "failed test: " + str(expected))
-
- tests = """\
- # invalid range
- 1-2, 3-1, 4-6, 7, 12
- """
- success = indices.runTests(tests, printResults=False, failureTests=True)[0]
- self.assertTrue(success, "failed to raise exception on improper range test")
-
-class RunTestsPostParseTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- integer = pp.pyparsing_common.integer
- fraction = integer('numerator') + '/' + integer('denominator')
-
- accum = []
- def eval_fraction(test, result):
- accum.append((test, result.asList()))
- return "eval: {0}".format(result.numerator / result.denominator)
-
- success = fraction.runTests("""\
- 1/2
- 1/0
- """, postParse=eval_fraction)[0]
- print_(success)
-
- self.assertTrue(success, "failed to parse fractions in RunTestsPostParse")
-
- expected_accum = [('1/2', [1, '/', 2]), ('1/0', [1, '/', 0])]
- self.assertEqual(accum, expected_accum, "failed to call postParse method during runTests")
-
-class CommonExpressionsTest(ParseTestCase):
- def runTest(self):
- from pyparsing import pyparsing_common
- import ast
-
- success = pyparsing_common.mac_address.runTests("""
- AA:BB:CC:DD:EE:FF
- AA.BB.CC.DD.EE.FF
- AA-BB-CC-DD-EE-FF
- """)[0]
- self.assertTrue(success, "error in parsing valid MAC address")
-
- success = pyparsing_common.mac_address.runTests("""
- # mixed delimiters
- AA.BB:CC:DD:EE:FF
- """, failureTests=True)[0]
- self.assertTrue( success, "error in detecting invalid mac address")
-
- success = pyparsing_common.ipv4_address.runTests("""
- 0.0.0.0
- 1.1.1.1
- 127.0.0.1
- 1.10.100.199
- 255.255.255.255
- """)[0]
- self.assertTrue(success, "error in parsing valid IPv4 address")
-
- success = pyparsing_common.ipv4_address.runTests("""
- # out of range value
- 256.255.255.255
- """, failureTests=True)[0]
- self.assertTrue(success, "error in detecting invalid IPv4 address")
-
- success = pyparsing_common.ipv6_address.runTests("""
- 2001:0db8:85a3:0000:0000:8a2e:0370:7334
- 2134::1234:4567:2468:1236:2444:2106
- 0:0:0:0:0:0:A00:1
- 1080::8:800:200C:417A
- ::A00:1
-
- # loopback address
- ::1
-
- # the null address
- ::
-
- # ipv4 compatibility form
- ::ffff:192.168.0.1
- """)[0]
- self.assertTrue(success, "error in parsing valid IPv6 address")
-
- success = pyparsing_common.ipv6_address.runTests("""
- # too few values
- 1080:0:0:0:8:800:200C
-
- # too many ::'s, only 1 allowed
- 2134::1234:4567::2444:2106
- """, failureTests=True)[0]
- self.assertTrue(success, "error in detecting invalid IPv6 address")
-
- success = pyparsing_common.number.runTests("""
- 100
- -100
- +100
- 3.14159
- 6.02e23
- 1e-12
- """)[0]
- self.assertTrue(success, "error in parsing valid numerics")
-
- success = pyparsing_common.sci_real.runTests("""
- 1e12
- -1e12
- 3.14159
- 6.02e23
- """)[0]
- self.assertTrue(success, "error in parsing valid scientific notation reals")
-
- # any int or real number, returned as float
- success = pyparsing_common.fnumber.runTests("""
- 100
- -100
- +100
- 3.14159
- 6.02e23
- 1e-12
- """)[0]
- self.assertTrue(success, "error in parsing valid numerics")
-
- success, results = pyparsing_common.iso8601_date.runTests("""
- 1997
- 1997-07
- 1997-07-16
- """)
- self.assertTrue(success, "error in parsing valid iso8601_date")
- expected = [
- ('1997', None, None),
- ('1997', '07', None),
- ('1997', '07', '16'),
- ]
- for r,exp in zip(results, expected):
- self.assertTrue((r[1].year,r[1].month,r[1].day,) == exp, "failed to parse date into fields")
-
- success, results = pyparsing_common.iso8601_date().addParseAction(pyparsing_common.convertToDate()).runTests("""
- 1997-07-16
- """)
- self.assertTrue(success, "error in parsing valid iso8601_date with parse action")
- self.assertTrue(results[0][1][0] == datetime.date(1997, 7, 16))
-
- success, results = pyparsing_common.iso8601_datetime.runTests("""
- 1997-07-16T19:20+01:00
- 1997-07-16T19:20:30+01:00
- 1997-07-16T19:20:30.45Z
- 1997-07-16 19:20:30.45
- """)
- self.assertTrue(success, "error in parsing valid iso8601_datetime")
-
- success, results = pyparsing_common.iso8601_datetime().addParseAction(pyparsing_common.convertToDatetime()).runTests("""
- 1997-07-16T19:20:30.45
- """)
- self.assertTrue(success, "error in parsing valid iso8601_datetime")
- self.assertTrue(results[0][1][0] == datetime.datetime(1997, 7, 16, 19, 20, 30, 450000))
-
- success = pyparsing_common.uuid.runTests("""
- 123e4567-e89b-12d3-a456-426655440000
- """)[0]
- self.assertTrue(success, "failed to parse valid uuid")
-
- success = pyparsing_common.fraction.runTests("""
- 1/2
- -15/16
- -3/-4
- """)[0]
- self.assertTrue(success, "failed to parse valid fraction")
-
- success = pyparsing_common.mixed_integer.runTests("""
- 1/2
- -15/16
- -3/-4
- 1 1/2
- 2 -15/16
- 0 -3/-4
- 12
- """)[0]
- self.assertTrue(success, "failed to parse valid mixed integer")
-
- success, results = pyparsing_common.number.runTests("""
- 100
- -3
- 1.732
- -3.14159
- 6.02e23""")
- self.assertTrue(success, "failed to parse numerics")
-
- for test,result in results:
- expected = ast.literal_eval(test)
- self.assertEqual(result[0], expected, "numeric parse failed (wrong value) (%s should be %s)" % (result[0], expected))
- self.assertEqual(type(result[0]), type(expected), "numeric parse failed (wrong type) (%s should be %s)" % (type(result[0]), type(expected)))
-
-
-class NumericExpressionsTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- ppc = pp.pyparsing_common
-
- # disable parse actions that do type conversion so we don't accidentally trigger
- # conversion exceptions when what we want to check is the parsing expression
- real = ppc.real().setParseAction(None)
- sci_real = ppc.sci_real().setParseAction(None)
- signed_integer = ppc.signed_integer().setParseAction(None)
-
- from itertools import product
-
- def make_tests():
- leading_sign = ['+', '-', '']
- leading_digit = ['0', '']
- dot = ['.', '']
- decimal_digit = ['1', '']
- e = ['e', 'E', '']
- e_sign = ['+', '-', '']
- e_int = ['22', '']
- stray = ['9', '.', '']
-
- seen = set()
- seen.add('')
- for parts in product(leading_sign, stray, leading_digit, dot, decimal_digit, stray, e, e_sign, e_int,
- stray):
- parts_str = ''.join(parts).strip()
- if parts_str in seen:
- continue
- seen.add(parts_str)
- yield parts_str
-
- print_(len(seen)-1, "tests produced")
-
- # collect tests into valid/invalid sets, depending on whether they evaluate to valid Python floats or ints
- valid_ints = set()
- valid_reals = set()
- valid_sci_reals = set()
- invalid_ints = set()
- invalid_reals = set()
- invalid_sci_reals = set()
-
- # check which strings parse as valid floats or ints, and store in related valid or invalid test sets
- for test_str in make_tests():
- if '.' in test_str or 'e' in test_str.lower():
- try:
- float(test_str)
- except ValueError:
- invalid_sci_reals.add(test_str)
- if 'e' not in test_str.lower():
- invalid_reals.add(test_str)
- else:
- valid_sci_reals.add(test_str)
- if 'e' not in test_str.lower():
- valid_reals.add(test_str)
-
- try:
- int(test_str)
- except ValueError:
- invalid_ints.add(test_str)
- else:
- valid_ints.add(test_str)
-
- # now try all the test sets against their respective expressions
- all_pass = True
- suppress_results = {'printResults': False}
- for expr, tests, is_fail, fn in zip([real, sci_real, signed_integer]*2,
- [valid_reals, valid_sci_reals, valid_ints,
- invalid_reals, invalid_sci_reals, invalid_ints],
- [False, False, False, True, True, True],
- [float, float, int]*2):
- #
- # success, test_results = expr.runTests(sorted(tests, key=len), failureTests=is_fail, **suppress_results)
- # filter_result_fn = (lambda r: isinstance(r, Exception),
- # lambda r: not isinstance(r, Exception))[is_fail]
- # print_(expr, ('FAIL', 'PASS')[success], "{1}valid tests ({0})".format(len(tests),
- # 'in' if is_fail else ''))
- # if not success:
- # all_pass = False
- # for test_string, result in test_results:
- # if filter_result_fn(result):
- # try:
- # test_value = fn(test_string)
- # except ValueError as ve:
- # test_value = str(ve)
- # print_("{0!r}: {1} {2} {3}".format(test_string, result,
- # expr.matches(test_string, parseAll=True), test_value))
-
- success = True
- for t in tests:
- if expr.matches(t, parseAll=True):
- if is_fail:
- print_(t, "should fail but did not")
- success = False
- else:
- if not is_fail:
- print_(t, "should not fail but did")
- success = False
- print_(expr, ('FAIL', 'PASS')[success], "{1}valid tests ({0})".format(len(tests),
- 'in' if is_fail else ''))
- all_pass = all_pass and success
-
- self.assertTrue(all_pass, "failed one or more numeric tests")
-
-class TokenMapTest(ParseTestCase):
- def runTest(self):
- from pyparsing import tokenMap, Word, hexnums, OneOrMore
-
- parser = OneOrMore(Word(hexnums)).setParseAction(tokenMap(int, 16))
- success, results = parser.runTests("""
- 00 11 22 aa FF 0a 0d 1a
- """, printResults=False)
- self.assertTrue(success, "failed to parse hex integers")
- print_(results)
- self.assertEqual(results[0][-1].asList(), [0, 17, 34, 170, 255, 10, 13, 26], "tokenMap parse action failed")
-
-
-class ParseFileTest(ParseTestCase):
- def runTest(self):
- from pyparsing import pyparsing_common, OneOrMore
- s = """
- 123 456 789
- """
- input_file = StringIO(s)
- integer = pyparsing_common.integer
-
- results = OneOrMore(integer).parseFile(input_file)
- print_(results)
-
- results = OneOrMore(integer).parseFile('test/parsefiletest_input_file.txt')
- print_(results)
-
-
-class HTMLStripperTest(ParseTestCase):
- def runTest(self):
- from pyparsing import pyparsing_common, originalTextFor, OneOrMore, Word, printables
-
- sample = """
- <html>
- Here is some sample <i>HTML</i> text.
- </html>
- """
- read_everything = originalTextFor(OneOrMore(Word(printables)))
- read_everything.addParseAction(pyparsing_common.stripHTMLTags)
-
- result = read_everything.parseString(sample)
- self.assertEqual(result[0].strip(), 'Here is some sample HTML text.')
-
-class ExprSplitterTest(ParseTestCase):
- def runTest(self):
-
- from pyparsing import Literal, quotedString, pythonStyleComment, Empty
-
- expr = Literal(';') + Empty()
- expr.ignore(quotedString)
- expr.ignore(pythonStyleComment)
-
-
- sample = """
- def main():
- this_semi_does_nothing();
- neither_does_this_but_there_are_spaces_afterward();
- a = "a;b"; return a # this is a comment; it has a semicolon!
-
- def b():
- if False:
- z=1000;b("; in quotes"); c=200;return z
- return ';'
-
- class Foo(object):
- def bar(self):
- '''a docstring; with a semicolon'''
- a = 10; b = 11; c = 12
-
- # this comment; has several; semicolons
- if self.spam:
- x = 12; return x # so; does; this; one
- x = 15;;; y += x; return y
-
- def baz(self):
- return self.bar
- """
- expected = [
- [' this_semi_does_nothing()', ''],
- [' neither_does_this_but_there_are_spaces_afterward()', ''],
- [' a = "a;b"', 'return a # this is a comment; it has a semicolon!'],
- [' z=1000', 'b("; in quotes")', 'c=200', 'return z'],
- [" return ';'"],
- [" '''a docstring; with a semicolon'''"],
- [' a = 10', 'b = 11', 'c = 12'],
- [' # this comment; has several; semicolons'],
- [' x = 12', 'return x # so; does; this; one'],
- [' x = 15', '', '', 'y += x', 'return y'],
- ]
-
- exp_iter = iter(expected)
- for line in filter(lambda ll: ';' in ll, sample.splitlines()):
- print_(str(list(expr.split(line)))+',')
- self.assertEqual(list(expr.split(line)), next(exp_iter), "invalid split on expression")
-
- print_()
-
- expected = [
- [' this_semi_does_nothing()', ';', ''],
- [' neither_does_this_but_there_are_spaces_afterward()', ';', ''],
- [' a = "a;b"', ';', 'return a # this is a comment; it has a semicolon!'],
- [' z=1000', ';', 'b("; in quotes")', ';', 'c=200', ';', 'return z'],
- [" return ';'"],
- [" '''a docstring; with a semicolon'''"],
- [' a = 10', ';', 'b = 11', ';', 'c = 12'],
- [' # this comment; has several; semicolons'],
- [' x = 12', ';', 'return x # so; does; this; one'],
- [' x = 15', ';', '', ';', '', ';', 'y += x', ';', 'return y'],
- ]
- exp_iter = iter(expected)
- for line in filter(lambda ll: ';' in ll, sample.splitlines()):
- print_(str(list(expr.split(line, includeSeparators=True)))+',')
- self.assertEqual(list(expr.split(line, includeSeparators=True)), next(exp_iter),
- "invalid split on expression")
-
- print_()
-
-
- expected = [
- [' this_semi_does_nothing()', ''],
- [' neither_does_this_but_there_are_spaces_afterward()', ''],
- [' a = "a;b"', 'return a # this is a comment; it has a semicolon!'],
- [' z=1000', 'b("; in quotes"); c=200;return z'],
- [' a = 10', 'b = 11; c = 12'],
- [' x = 12', 'return x # so; does; this; one'],
- [' x = 15', ';; y += x; return y'],
- ]
- exp_iter = iter(expected)
- for line in sample.splitlines():
- pieces = list(expr.split(line, maxsplit=1))
- print_(str(pieces)+',')
- if len(pieces) == 2:
- exp = next(exp_iter)
- self.assertEqual(pieces, exp, "invalid split on expression with maxSplits=1")
- elif len(pieces) == 1:
- self.assertEqual(len(expr.searchString(line)), 0, "invalid split with maxSplits=1 when expr not present")
- else:
- print_("\n>>> " + line)
- self.assertTrue(False, "invalid split on expression with maxSplits=1, corner case")
-
-class ParseFatalExceptionTest(ParseTestCase):
- def runTest(self):
-
- from pyparsing import Word, nums, ParseFatalException
-
- success = False
- try:
- expr = "ZZZ" - Word(nums)
- expr.parseString("ZZZ bad")
- except ParseFatalException as pfe:
- print_('ParseFatalException raised correctly')
- success = True
- except Exception as e:
- print_(type(e))
- print_(e)
-
- self.assertTrue(success, "bad handling of syntax error")
-
-class InlineLiteralsUsingTest(ParseTestCase):
- def runTest(self):
-
- from pyparsing import ParserElement, Suppress, Literal, CaselessLiteral, Word, alphas, oneOf, CaselessKeyword, nums
-
- with AutoReset(ParserElement, "_literalStringClass"):
- ParserElement.inlineLiteralsUsing(Suppress)
- wd = Word(alphas)
- result = (wd + ',' + wd + oneOf("! . ?")).parseString("Hello, World!")
- self.assertEqual(len(result), 3, "inlineLiteralsUsing(Suppress) failed!")
-
- ParserElement.inlineLiteralsUsing(Literal)
- result = (wd + ',' + wd + oneOf("! . ?")).parseString("Hello, World!")
- self.assertEqual(len(result), 4, "inlineLiteralsUsing(Literal) failed!")
-
- ParserElement.inlineLiteralsUsing(CaselessKeyword)
- result = ("SELECT" + wd + "FROM" + wd).parseString("select color from colors")
- self.assertEqual(result.asList(), "SELECT color FROM colors".split(),
- "inlineLiteralsUsing(CaselessKeyword) failed!")
-
- ParserElement.inlineLiteralsUsing(CaselessLiteral)
- result = ("SELECT" + wd + "FROM" + wd).parseString("select color from colors")
- self.assertEqual(result.asList(), "SELECT color FROM colors".split(),
- "inlineLiteralsUsing(CaselessLiteral) failed!")
-
- integer = Word(nums)
- ParserElement.inlineLiteralsUsing(Literal)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
- result = date_str.parseString("1999/12/31")
- self.assertEqual(result.asList(), ['1999', '/', '12', '/', '31'], "inlineLiteralsUsing(example 1) failed!")
-
- # change to Suppress
- ParserElement.inlineLiteralsUsing(Suppress)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
- result = date_str.parseString("1999/12/31") # -> ['1999', '12', '31']
- self.assertEqual(result.asList(), ['1999', '12', '31'], "inlineLiteralsUsing(example 2) failed!")
-
-class CloseMatchTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- searchseq = pp.CloseMatch("ATCATCGAATGGA", 2)
-
- _, results = searchseq.runTests("""
- ATCATCGAATGGA
- XTCATCGAATGGX
- ATCATCGAAXGGA
- ATCAXXGAATGGA
- ATCAXXGAATGXA
- ATCAXXGAATGG
- """)
- expected = (
- [],
- [0,12],
- [9],
- [4,5],
- None,
- None
- )
-
- for r, exp in zip(results, expected):
- if exp is not None:
- self.assertEqual(r[1].mismatches, exp,
- "fail CloseMatch between %r and %r" % (searchseq.match_string, r[0]))
- print_(r[0], 'exc: %s' % r[1] if exp is None and isinstance(r[1], Exception)
- else ("no match", "match")[r[1].mismatches == exp])
-
-class DefaultKeywordCharsTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- try:
- pp.Keyword("start").parseString("start1000")
- except pp.ParseException:
- pass
- else:
- self.assertTrue(False, "failed to fail on default keyword chars")
-
- try:
- pp.Keyword("start", identChars=pp.alphas).parseString("start1000")
- except pp.ParseException:
- self.assertTrue(False, "failed to match keyword using updated keyword chars")
- else:
- pass
-
- with AutoReset(pp.Keyword, "DEFAULT_KEYWORD_CHARS"):
- pp.Keyword.setDefaultKeywordChars(pp.alphas)
- try:
- pp.Keyword("start").parseString("start1000")
- except pp.ParseException:
- self.assertTrue(False, "failed to match keyword using updated keyword chars")
- else:
- pass
-
- try:
- pp.CaselessKeyword("START").parseString("start1000")
- except pp.ParseException:
- pass
- else:
- self.assertTrue(False, "failed to fail on default keyword chars")
-
- try:
- pp.CaselessKeyword("START", identChars=pp.alphas).parseString("start1000")
- except pp.ParseException:
- self.assertTrue(False, "failed to match keyword using updated keyword chars")
- else:
- pass
-
- with AutoReset(pp.Keyword, "DEFAULT_KEYWORD_CHARS"):
- pp.Keyword.setDefaultKeywordChars(pp.alphas)
- try:
- pp.CaselessKeyword("START").parseString("start1000")
- except pp.ParseException:
- self.assertTrue(False, "failed to match keyword using updated keyword chars")
- else:
- pass
-
-class ColTest(ParseTestCase):
- def runTest(self):
-
- test = "*\n* \n* ALF\n*\n"
- initials = [c for i, c in enumerate(test) if pp.col(i, test) == 1]
- print_(initials)
- self.assertTrue(len(initials) == 4 and all(c=='*' for c in initials), 'fail col test')
-
-class LiteralExceptionTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- for cls in (pp.Literal, pp.CaselessLiteral, pp.Keyword, pp.CaselessKeyword,
- pp.Word, pp.Regex):
- expr = cls('xyz')#.setName('{0}_expr'.format(cls.__name__.lower()))
-
- try:
- expr.parseString(' ')
- except Exception as e:
- print_(cls.__name__, str(e))
- self.assertTrue(isinstance(e, pp.ParseBaseException),
- "class {0} raised wrong exception type {1}".format(cls.__name__, type(e).__name__))
-
-class ParseActionExceptionTest(ParseTestCase):
- def runTest(self):
- self.expect_traceback = True
-
- import pyparsing as pp
- import traceback
-
- number = pp.Word(pp.nums)
- def number_action():
- raise IndexError # this is the important line!
-
- number.setParseAction(number_action)
- symbol = pp.Word('abcd', max=1)
- expr = number | symbol
-
- try:
- expr.parseString('1 + 2')
- except Exception as e:
- self.assertTrue(hasattr(e, '__cause__'), "no __cause__ attribute in the raised exception")
- self.assertTrue(e.__cause__ is not None, "__cause__ not propagated to outer exception")
- self.assertTrue(type(e.__cause__) == IndexError, "__cause__ references wrong exception")
- traceback.print_exc()
- else:
- self.assertTrue(False, "Expected ParseException not raised")
-
-class ParseActionNestingTest(ParseTestCase):
- # tests Issue #22
- def runTest(self):
-
- vals = pp.OneOrMore(pp.pyparsing_common.integer)("int_values")
- def add_total(tokens):
- tokens['total'] = sum(tokens)
- return tokens
- vals.addParseAction(add_total)
- results = vals.parseString("244 23 13 2343")
- print_(results.dump())
- self.assertEqual(results.int_values.asDict(), {}, "noop parse action changed ParseResults structure")
-
- name = pp.Word(pp.alphas)('name')
- score = pp.Word(pp.nums + '.')('score')
- nameScore = pp.Group(name + score)
- line1 = nameScore('Rider')
-
- result1 = line1.parseString('Mauney 46.5')
-
- print_("### before parse action is added ###")
- print_("result1.dump():\n" + result1.dump() + "\n")
- before_pa_dict = result1.asDict()
-
- line1.setParseAction(lambda t: t)
-
- result1 = line1.parseString('Mauney 46.5')
- after_pa_dict = result1.asDict()
-
- print_("### after parse action was added ###")
- print_("result1.dump():\n" + result1.dump() + "\n")
- self.assertEqual(before_pa_dict, after_pa_dict, "noop parse action changed ParseResults structure")
-
-class ParseResultsNameBelowUngroupedNameTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- rule_num = pp.Regex("[0-9]+")("LIT_NUM*")
- list_num = pp.Group(pp.Literal("[")("START_LIST")
- + pp.delimitedList(rule_num)("LIST_VALUES")
- + pp.Literal("]")("END_LIST"))("LIST")
-
- test_string = "[ 1,2,3,4,5,6 ]"
- list_num.runTests(test_string)
-
- U = list_num.parseString(test_string)
- self.assertTrue("LIT_NUM" not in U.LIST.LIST_VALUES, "results name retained as sub in ungrouped named result")
-
-class ParseResultsNamesInGroupWithDictTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- from pyparsing import pyparsing_common as ppc
-
- key = ppc.identifier()
- value = ppc.integer()
- lat = ppc.real()
- long = ppc.real()
- EQ = pp.Suppress('=')
-
- data = lat("lat") + long("long") + pp.Dict(pp.OneOrMore(pp.Group(key + EQ + value)))
- site = pp.QuotedString('"')("name") + pp.Group(data)("data")
-
- test_string = '"Golden Gate Bridge" 37.819722 -122.478611 height=746 span=4200'
- site.runTests(test_string)
-
- # U = list_num.parseString(test_string)
- # self.assertTrue("LIT_NUM" not in U.LIST.LIST_VALUES, "results name retained as sub in ungrouped named result")
-
- a, aEnd = pp.makeHTMLTags('a')
- attrs = a.parseString("<a href='blah'>")
- print_(attrs.dump())
- self.assertEqual(attrs.startA.href, 'blah')
- self.assertEqual(attrs.asDict(), {'startA': {'href': 'blah', 'tag': 'a', 'empty': False},
- 'href': 'blah', 'tag': 'a', 'empty': False})
-
-
-class FollowedByTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- from pyparsing import pyparsing_common as ppc
- expr = pp.Word(pp.alphas)("item") + pp.FollowedBy(ppc.integer("qty"))
- result = expr.parseString("balloon 99")
- print_(result.dump())
- self.assertTrue('qty' in result, "failed to capture results name in FollowedBy")
- self.assertEqual(result.asDict(), {'item': 'balloon', 'qty': 99},
- "invalid results name structure from FollowedBy")
-
-class SetBreakTest(ParseTestCase):
- """
- Test behavior of ParserElement.setBreak(), to invoke the debugger before parsing that element is attempted.
-
- Temporarily monkeypatches pdb.set_trace.
- """
- def runTest(self):
- was_called = []
- def mock_set_trace():
- was_called.append(True)
-
- import pyparsing as pp
- wd = pp.Word(pp.alphas)
- wd.setBreak()
-
- print_("Before parsing with setBreak:", was_called)
- import pdb
- with AutoReset(pdb, "set_trace"):
- pdb.set_trace = mock_set_trace
- wd.parseString("ABC")
-
- print_("After parsing with setBreak:", was_called)
- self.assertTrue(bool(was_called), "set_trace wasn't called by setBreak")
-
-class UnicodeTests(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- ppu = pp.pyparsing_unicode
- ppc = pp.pyparsing_common
-
- # verify proper merging of ranges by addition
- kanji_printables = ppu.Japanese.Kanji.printables
- katakana_printables = ppu.Japanese.Katakana.printables
- hiragana_printables = ppu.Japanese.Hiragana.printables
- japanese_printables = ppu.Japanese.printables
- self.assertEqual(set(japanese_printables), set(kanji_printables
- + katakana_printables
- + hiragana_printables),
- "failed to construct ranges by merging Japanese types")
-
- # verify proper merging of ranges using multiple inheritance
- cjk_printables = ppu.CJK.printables
- self.assertEqual(len(cjk_printables), len(set(cjk_printables)),
- "CJK contains duplicate characters - all should be unique")
-
- chinese_printables = ppu.Chinese.printables
- korean_printables = ppu.Korean.printables
- print_(len(cjk_printables), len(set(chinese_printables
- + korean_printables
- + japanese_printables)))
-
- self.assertEqual(len(cjk_printables), len(set(chinese_printables
- + korean_printables
- + japanese_printables)),
- "failed to construct ranges by merging Chinese, Japanese and Korean")
-
- alphas = ppu.Greek.alphas
- greet = pp.Word(alphas) + ',' + pp.Word(alphas) + '!'
-
- # input string
- hello = u"Καλημέρα, κόσμε!"
- result = greet.parseString(hello)
- print_(result)
- self.assertTrue(result.asList() == [u'Καλημέρα', ',', u'κόσμε', '!'],
- "Failed to parse Greek 'Hello, World!' using pyparsing_unicode.Greek.alphas")
-
- # define a custom unicode range using multiple inheritance
- class Turkish_set(ppu.Latin1, ppu.LatinA):
- pass
-
- self.assertEqual(set(Turkish_set.printables),
- set(ppu.Latin1.printables + ppu.LatinA.printables),
- "failed to construct ranges by merging Latin1 and LatinA (printables)")
-
- self.assertEqual(set(Turkish_set.alphas),
- set(ppu.Latin1.alphas + ppu.LatinA.alphas),
- "failed to construct ranges by merging Latin1 and LatinA (alphas)")
-
- self.assertEqual(set(Turkish_set.nums),
- set(ppu.Latin1.nums + ppu.LatinA.nums),
- "failed to construct ranges by merging Latin1 and LatinA (nums)")
-
- key = pp.Word(Turkish_set.alphas)
- value = ppc.integer | pp.Word(Turkish_set.alphas, Turkish_set.alphanums)
- EQ = pp.Suppress('=')
- key_value = key + EQ + value
-
- sample = u"""\
- şehir=İzmir
- ülke=Türkiye
- nüfus=4279677"""
- result = pp.Dict(pp.OneOrMore(pp.Group(key_value))).parseString(sample)
-
- print_(result.asDict())
- self.assertEqual(result.asDict(), {u'şehir': u'İzmir', u'ülke': u'Türkiye', u'nüfus': 4279677},
- "Failed to parse Turkish key-value pairs")
-
-
-class IndentedBlockExampleTest(ParseTestCase):
- # Make sure example in indentedBlock docstring actually works!
- def runTest(self):
- from textwrap import dedent
- from pyparsing import (Word, alphas, alphanums, indentedBlock, Optional, delimitedList, Group, Forward,
- nums, OneOrMore)
- data = dedent('''
- def A(z):
- A1
- B = 100
- G = A2
- A2
- A3
- B
- def BB(a,b,c):
- BB1
- def BBA():
- bba1
- bba2
- bba3
- C
- D
- def spam(x,y):
- def eggs(z):
- pass
- ''')
-
- indentStack = [1]
- stmt = Forward()
-
- identifier = Word(alphas, alphanums)
- funcDecl = ("def" + identifier + Group("(" + Optional(delimitedList(identifier)) + ")") + ":")
- func_body = indentedBlock(stmt, indentStack)
- funcDef = Group(funcDecl + func_body)
-
- rvalue = Forward()
- funcCall = Group(identifier + "(" + Optional(delimitedList(rvalue)) + ")")
- rvalue << (funcCall | identifier | Word(nums))
- assignment = Group(identifier + "=" + rvalue)
- stmt << (funcDef | assignment | identifier)
-
- module_body = OneOrMore(stmt)
-
- parseTree = module_body.parseString(data)
- parseTree.pprint()
- self.assertEqual(parseTree.asList(),
- [['def',
- 'A',
- ['(', 'z', ')'],
- ':',
- [['A1'], [['B', '=', '100']], [['G', '=', 'A2']], ['A2'], ['A3']]],
- 'B',
- ['def',
- 'BB',
- ['(', 'a', 'b', 'c', ')'],
- ':',
- [['BB1'], [['def', 'BBA', ['(', ')'], ':', [['bba1'], ['bba2'], ['bba3']]]]]],
- 'C',
- 'D',
- ['def',
- 'spam',
- ['(', 'x', 'y', ')'],
- ':',
- [[['def', 'eggs', ['(', 'z', ')'], ':', [['pass']]]]]]],
- "Failed indentedBlock example"
- )
-
-
-class IndentedBlockTest(ParseTestCase):
- # parse pseudo-yaml indented text
- def runTest(self):
- import textwrap
-
- EQ = pp.Suppress('=')
- stack = [1]
- key = pp.pyparsing_common.identifier
- value = pp.Forward()
- key_value = key + EQ + value
- compound_value = pp.Dict(pp.ungroup(pp.indentedBlock(key_value, stack)))
- value <<= pp.pyparsing_common.integer | pp.QuotedString("'") | compound_value
- parser = pp.Dict(pp.OneOrMore(pp.Group(key_value)))
-
- text = """
- a = 100
- b = 101
- c =
- c1 = 200
- c2 =
- c21 = 999
- c3 = 'A horse, a horse, my kingdom for a horse'
- d = 505
- """
- text = textwrap.dedent(text)
- print_(text)
-
- result = parser.parseString(text)
- print_(result.dump())
- self.assertEqual(result.a, 100, "invalid indented block result")
- self.assertEqual(result.c.c1, 200, "invalid indented block result")
- self.assertEqual(result.c.c2.c21, 999, "invalid indented block result")
-
-
-class IndentedBlockTest2(ParseTestCase):
- # exercise indentedBlock with example posted in issue #87
- def runTest(self):
- from textwrap import dedent
- from pyparsing import Word, alphas, alphanums, Suppress, Forward, indentedBlock, Literal, OneOrMore
-
- indent_stack = [1]
-
- key = Word(alphas, alphanums) + Suppress(":")
- stmt = Forward()
-
- suite = indentedBlock(stmt, indent_stack)
- body = key + suite
-
- pattern = (Word(alphas) + Suppress("(") + Word(alphas) + Suppress(")"))
- stmt << pattern
-
- def key_parse_action(toks):
- print_("Parsing '%s'..." % toks[0])
-
- key.setParseAction(key_parse_action)
- header = Suppress("[") + Literal("test") + Suppress("]")
- content = (header + OneOrMore(indentedBlock(body, indent_stack, False)))
-
- contents = Forward()
- suites = indentedBlock(content, indent_stack)
-
- extra = Literal("extra") + Suppress(":") + suites
- contents << (content | extra)
-
- parser = OneOrMore(contents)
-
- sample = dedent("""
- extra:
- [test]
- one0:
- two (three)
- four0:
- five (seven)
- extra:
- [test]
- one1:
- two (three)
- four1:
- five (seven)
- """)
-
- success, _ = parser.runTests([sample])
- self.assertTrue(success, "Failed indentedBlock test for issue #87")
-
-
-class IndentedBlockScanTest(ParseTestCase):
- def get_parser(self):
- """
- A valid statement is the word "block:", followed by an indent, followed by the letter A only, or another block
- """
- stack = [1]
- block = pp.Forward()
- body = pp.indentedBlock(pp.Literal('A') ^ block, indentStack=stack, indent=True)
- block <<= pp.Literal('block:') + body
- return block
-
- def runTest(self):
- from textwrap import dedent
-
- # This input string is a perfect match for the parser, so a single match is found
- p1 = self.get_parser()
- r1 = list(p1.scanString(dedent("""\
- block:
- A
- """)))
- self.assertEqual(len(r1), 1)
-
- # This input string is a perfect match for the parser, except for the letter B instead of A, so this will fail (and should)
- p2 = self.get_parser()
- r2 = list(p2.scanString(dedent("""\
- block:
- B
- """)))
- self.assertEqual(len(r2), 0)
-
- # This input string contains both string A and string B, and it finds one match (as it should)
- p3 = self.get_parser()
- r3 = list(p3.scanString(dedent("""\
- block:
- A
- block:
- B
- """)))
- self.assertEqual(len(r3), 1)
-
- # This input string contains both string A and string B, but in a different order.
- p4 = self.get_parser()
- r4 = list(p4.scanString(dedent("""\
- block:
- B
- block:
- A
- """)))
- self.assertEqual(len(r4), 1)
-
- # This is the same as case 3, but with nesting
- p5 = self.get_parser()
- r5 = list(p5.scanString(dedent("""\
- block:
- block:
- A
- block:
- block:
- B
- """)))
- self.assertEqual(len(r5), 1)
-
- # This is the same as case 4, but with nesting
- p6 = self.get_parser()
- r6 = list(p6.scanString(dedent("""\
- block:
- block:
- B
- block:
- block:
- A
- """)))
- self.assertEqual(len(r6), 1)
-
-
-class ParseResultsWithNameMatchFirst(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- expr_a = pp.Literal('not') + pp.Literal('the') + pp.Literal('bird')
- expr_b = pp.Literal('the') + pp.Literal('bird')
- expr = (expr_a | expr_b)('rexp')
- expr.runTests("""\
- not the bird
- the bird
- """)
- self.assertEqual(list(expr.parseString('not the bird')['rexp']), 'not the bird'.split())
- self.assertEqual(list(expr.parseString('the bird')['rexp']), 'the bird'.split())
-
- # test compatibility mode, restoring pre-2.3.1 behavior
- with AutoReset(pp.__compat__, "collect_all_And_tokens"):
- pp.__compat__.collect_all_And_tokens = False
- pp.__diag__.warn_multiple_tokens_in_named_alternation = True
- expr_a = pp.Literal('not') + pp.Literal('the') + pp.Literal('bird')
- expr_b = pp.Literal('the') + pp.Literal('bird')
- if PY_3:
- with self.assertWarns(UserWarning, msg="failed to warn of And within alternation"):
- expr = (expr_a | expr_b)('rexp')
- else:
- self.expect_warning = True
- expr = (expr_a | expr_b)('rexp')
- expr.runTests("""
- not the bird
- the bird
- """)
- self.assertEqual(expr.parseString('not the bird')['rexp'], 'not')
- self.assertEqual(expr.parseString('the bird')['rexp'], 'the')
-
-
-class ParseResultsWithNameOr(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- expr_a = pp.Literal('not') + pp.Literal('the') + pp.Literal('bird')
- expr_b = pp.Literal('the') + pp.Literal('bird')
- expr = (expr_a ^ expr_b)('rexp')
- expr.runTests("""\
- not the bird
- the bird
- """)
- self.assertEqual(list(expr.parseString('not the bird')['rexp']), 'not the bird'.split())
- self.assertEqual(list(expr.parseString('the bird')['rexp']), 'the bird'.split())
-
- expr = (expr_a | expr_b)('rexp')
- expr.runTests("""\
- not the bird
- the bird
- """)
- self.assertEqual(list(expr.parseString('not the bird')['rexp']), 'not the bird'.split())
- self.assertEqual(list(expr.parseString('the bird')['rexp']), 'the bird'.split())
-
- # test compatibility mode, restoring pre-2.3.1 behavior
- with AutoReset(pp.__compat__, "collect_all_And_tokens"):
- pp.__compat__.collect_all_And_tokens = False
- pp.__diag__.warn_multiple_tokens_in_named_alternation = True
- expr_a = pp.Literal('not') + pp.Literal('the') + pp.Literal('bird')
- expr_b = pp.Literal('the') + pp.Literal('bird')
- if PY_3:
- with self.assertWarns(UserWarning, msg="failed to warn of And within alternation"):
- expr = (expr_a ^ expr_b)('rexp')
- else:
- self.expect_warning = True
- expr = (expr_a ^ expr_b)('rexp')
- expr.runTests("""\
- not the bird
- the bird
- """)
- self.assertEqual(expr.parseString('not the bird')['rexp'], 'not')
- self.assertEqual(expr.parseString('the bird')['rexp'], 'the')
-
-
-class EmptyDictDoesNotRaiseException(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- key = pp.Word(pp.alphas)
- value = pp.Word(pp.nums)
- EQ = pp.Suppress('=')
- key_value_dict = pp.dictOf(key, EQ + value)
-
- print_(key_value_dict.parseString("""\
- a = 10
- b = 20
- """).dump())
-
- try:
- print_(key_value_dict.parseString("").dump())
- except pp.ParseException as pe:
- exc = pe
- if not hasattr(exc, '__traceback__'):
- # Python 2 compatibility
- etype, value, traceback = sys.exc_info()
- exc.__traceback__ = traceback
- print_(pp.ParseException.explain(pe))
- else:
- self.assertTrue(False, "failed to raise exception when matching empty string")
-
-class ExplainExceptionTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- expr = pp.Word(pp.nums).setName("int") + pp.Word(pp.alphas).setName("word")
- try:
- expr.parseString("123 355")
- except pp.ParseException as pe:
- exc = pe
- if not hasattr(exc, '__traceback__'):
- # Python 2 compatibility
- etype, value, traceback = sys.exc_info()
- exc.__traceback__ = traceback
- print_(pp.ParseException.explain(pe, depth=0))
-
- expr = pp.Word(pp.nums).setName("int") - pp.Word(pp.alphas).setName("word")
- try:
- expr.parseString("123 355 (test using ErrorStop)")
- except pp.ParseSyntaxException as pe:
- exc = pe
- if not hasattr(exc, '__traceback__'):
- # Python 2 compatibility
- etype, value, traceback = sys.exc_info()
- exc.__traceback__ = traceback
- print_(pp.ParseException.explain(pe))
-
- integer = pp.Word(pp.nums).setName("int").addParseAction(lambda t: int(t[0]))
- expr = integer + integer
-
- def divide_args(t):
- integer.parseString("A")
- return t[0] / t[1]
-
- expr.addParseAction(divide_args)
- pp.ParserElement.enablePackrat()
- print_()
- # ~ print(expr.parseString("125 25"))
-
- try:
- expr.parseString("123 0")
- except pp.ParseException as pe:
- exc = pe
- if not hasattr(exc, '__traceback__'):
- # Python 2 compatibility
- etype, value, traceback = sys.exc_info()
- exc.__traceback__ = traceback
- print_(pp.ParseException.explain(pe))
- except Exception as exc:
- if not hasattr(exc, '__traceback__'):
- # Python 2 compatibility
- etype, value, traceback = sys.exc_info()
- exc.__traceback__ = traceback
- print_(pp.ParseException.explain(exc))
- raise
-
-
-class CaselessKeywordVsKeywordCaselessTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- frule = pp.Keyword('t', caseless=True) + pp.Keyword('yes', caseless=True)
- crule = pp.CaselessKeyword('t') + pp.CaselessKeyword('yes')
-
- flist = frule.searchString('not yes').asList()
- print_(flist)
- clist = crule.searchString('not yes').asList()
- print_(clist)
- self.assertEqual(flist, clist, "CaselessKeyword not working the same as Keyword(caseless=True)")
-
-
-class OneOfKeywordsTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- literal_expr = pp.oneOf("a b c")
- success, _ = literal_expr[...].runTests("""
- # literal oneOf tests
- a b c
- a a a
- abc
- """)
- self.assertTrue(success, "failed literal oneOf matching")
-
- keyword_expr = pp.oneOf("a b c", asKeyword=True)
- success, _ = keyword_expr[...].runTests("""
- # keyword oneOf tests
- a b c
- a a a
- """)
- self.assertTrue(success, "failed keyword oneOf matching")
-
- success, _ = keyword_expr[...].runTests("""
- # keyword oneOf failure tests
- abc
- """, failureTests=True)
- self.assertTrue(success, "failed keyword oneOf failure tests")
-
-
-class WarnUngroupedNamedTokensTest(ParseTestCase):
- """
- - warn_ungrouped_named_tokens_in_collection - flag to enable warnings when a results
- name is defined on a containing expression with ungrouped subexpressions that also
- have results names (default=True)
- """
- def runTest(self):
- import pyparsing as pp
- ppc = pp.pyparsing_common
-
- pp.__diag__.warn_ungrouped_named_tokens_in_collection = True
-
- COMMA = pp.Suppress(',').setName("comma")
- coord = (ppc.integer('x') + COMMA + ppc.integer('y'))
-
- # this should emit a warning
- if PY_3:
- with self.assertWarns(UserWarning, msg="failed to warn with named repetition of"
- " ungrouped named expressions"):
- path = coord[...].setResultsName('path')
-
- pp.__diag__.warn_ungrouped_named_tokens_in_collection = False
-
-
-class WarnNameSetOnEmptyForwardTest(ParseTestCase):
- """
- - warn_name_set_on_empty_Forward - flag to enable warnings whan a Forward is defined
- with a results name, but has no contents defined (default=False)
- """
- def runTest(self):
- import pyparsing as pp
-
- pp.__diag__.warn_name_set_on_empty_Forward = True
-
- base = pp.Forward()
-
- if PY_3:
- with self.assertWarns(UserWarning, msg="failed to warn when naming an empty Forward expression"):
- base("x")
-
-
-class WarnOnMultipleStringArgsToOneOfTest(ParseTestCase):
- """
- - warn_on_multiple_string_args_to_oneof - flag to enable warnings whan oneOf is
- incorrectly called with multiple str arguments (default=True)
- """
- def runTest(self):
- import pyparsing as pp
-
- pp.__diag__.warn_on_multiple_string_args_to_oneof = True
-
- if PY_3:
- with self.assertWarns(UserWarning, msg="failed to warn when incorrectly calling oneOf(string, string)"):
- a = pp.oneOf('A', 'B')
-
-
-class EnableDebugOnNamedExpressionsTest(ParseTestCase):
- """
- - enable_debug_on_named_expressions - flag to auto-enable debug on all subsequent
- calls to ParserElement.setName() (default=False)
- """
- def runTest(self):
- import pyparsing as pp
- import textwrap
-
- test_stdout = StringIO()
-
- with AutoReset(sys, 'stdout', 'stderr'):
- sys.stdout = test_stdout
- sys.stderr = test_stdout
-
- pp.__diag__.enable_debug_on_named_expressions = True
- integer = pp.Word(pp.nums).setName('integer')
-
- integer[...].parseString("1 2 3")
-
- expected_debug_output = textwrap.dedent("""\
- Match integer at loc 0(1,1)
- Matched integer -> ['1']
- Match integer at loc 1(1,2)
- Matched integer -> ['2']
- Match integer at loc 3(1,4)
- Matched integer -> ['3']
- Match integer at loc 5(1,6)
- Exception raised:Expected integer, found end of text (at char 5), (line:1, col:6)
- """)
- output = test_stdout.getvalue()
- print_(output)
- self.assertEqual(output,
- expected_debug_output,
- "failed to auto-enable debug on named expressions "
- "using enable_debug_on_named_expressions")
-
-
-class UndesirableButCommonPracticesTest(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
- ppc = pp.pyparsing_common
-
- # While these are valid constructs, and they are not encouraged
- # there is apparently a lot of code out there using these
- # coding styles.
- #
- # Even though they are not encouraged, we shouldn't break them.
-
- # Create an And using a list of expressions instead of using '+' operator
- expr = pp.And([pp.Word('abc'), pp.Word('123')])
- expr.runTests("""
- aaa 333
- b 1
- ababab 32123
- """)
-
- # Passing a single expression to a ParseExpression, when it really wants a sequence
- expr = pp.Or(pp.Or(ppc.integer))
- expr.runTests("""
- 123
- 456
- abc
- """)
-
-
-class ChainedTernaryOperator(ParseTestCase):
- def runTest(self):
- import pyparsing as pp
-
- TERNARY_INFIX = pp.infixNotation(
- pp.pyparsing_common.integer, [
- (("?", ":"), 3, pp.opAssoc.LEFT),
- ])
- self.assertParseAndCheckList(TERNARY_INFIX,
- "1?1:0?1:0",
- [[1, '?', 1, ':', 0, '?', 1, ':', 0]])
-
- TERNARY_INFIX = pp.infixNotation(
- pp.pyparsing_common.integer, [
- (("?", ":"), 3, pp.opAssoc.RIGHT),
- ])
- self.assertParseAndCheckList(TERNARY_INFIX,
- "1?1:0?1:0",
- [[1, '?', 1, ':', [0, '?', 1, ':', 0]]])
-
-
-class MiscellaneousParserTests(ParseTestCase):
- def runTest(self):
- self.expect_warning = True
-
- runtests = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
- if IRON_PYTHON_ENV:
- runtests = "ABCDEGHIJKLMNOPQRSTUVWXYZ"
-
- # test making oneOf with duplicate symbols
- if "A" in runtests:
- print_("verify oneOf handles duplicate symbols")
- try:
- test1 = pp.oneOf("a b c d a")
- except RuntimeError:
- self.assertTrue(False,"still have infinite loop in oneOf with duplicate symbols (string input)")
-
- print_("verify oneOf handles generator input")
- try:
- test1 = pp.oneOf(c for c in "a b c d a" if not c.isspace())
- except RuntimeError:
- self.assertTrue(False,"still have infinite loop in oneOf with duplicate symbols (generator input)")
-
- print_("verify oneOf handles list input")
- try:
- test1 = pp.oneOf("a b c d a".split())
- except RuntimeError:
- self.assertTrue(False,"still have infinite loop in oneOf with duplicate symbols (list input)")
-
- print_("verify oneOf handles set input")
- try:
- test1 = pp.oneOf(set("a b c d a"))
- except RuntimeError:
- self.assertTrue(False,"still have infinite loop in oneOf with duplicate symbols (set input)")
-
- # test MatchFirst bugfix
- if "B" in runtests:
- print_("verify MatchFirst iterates properly")
- results = pp.quotedString.parseString("'this is a single quoted string'")
- self.assertTrue(len(results) > 0, "MatchFirst error - not iterating over all choices")
-
- # verify streamline of subexpressions
- if "C" in runtests:
- print_("verify proper streamline logic")
- compound = pp.Literal("A") + "B" + "C" + "D"
- self.assertEqual(len(compound.exprs), 2,"bad test setup")
- print_(compound)
- compound.streamline()
- print_(compound)
- self.assertEqual(len(compound.exprs), 4,"streamline not working")
-
- # test for Optional with results name and no match
- if "D" in runtests:
- print_("verify Optional's do not cause match failure if have results name")
- testGrammar = pp.Literal("A") + pp.Optional("B")("gotB") + pp.Literal("C")
- try:
- testGrammar.parseString("ABC")
- testGrammar.parseString("AC")
- except pp.ParseException as pe:
- print_(pe.pstr,"->",pe)
- self.assertTrue(False, "error in Optional matching of string %s" % pe.pstr)
-
- # test return of furthest exception
- if "E" in runtests:
- testGrammar = ( pp.Literal("A") |
- ( pp.Optional("B") + pp.Literal("C") ) |
- pp.Literal("D") )
- try:
- testGrammar.parseString("BC")
- testGrammar.parseString("BD")
- except pp.ParseException as pe:
- print_(pe.pstr,"->",pe)
- self.assertEqual(pe.pstr, "BD", "wrong test string failed to parse")
- self.assertEqual(pe.loc, 1, "error in Optional matching, pe.loc="+str(pe.loc))
-
- # test validate
- if "F" in runtests:
- print_("verify behavior of validate()")
- def testValidation( grmr, gnam, isValid ):
- try:
- grmr.streamline()
- grmr.validate()
- self.assertTrue(isValid,"validate() accepted invalid grammar " + gnam)
- except pp.RecursiveGrammarException as e:
- print_(grmr)
- self.assertFalse(isValid, "validate() rejected valid grammar " + gnam)
-
- fwd = pp.Forward()
- g1 = pp.OneOrMore( ( pp.Literal("A") + "B" + "C" ) | fwd )
- g2 = pp.ZeroOrMore("C" + g1)
- fwd << pp.Group(g2)
- testValidation( fwd, "fwd", isValid=True )
-
- fwd2 = pp.Forward()
- fwd2 << pp.Group("A" | fwd2)
- testValidation( fwd2, "fwd2", isValid=False )
-
- fwd3 = pp.Forward()
- fwd3 << pp.Optional("A") + fwd3
- testValidation( fwd3, "fwd3", isValid=False )
-
- # test getName
- if "G" in runtests:
- print_("verify behavior of getName()")
- aaa = pp.Group(pp.Word("a")("A"))
- bbb = pp.Group(pp.Word("b")("B"))
- ccc = pp.Group(":" + pp.Word("c")("C"))
- g1 = "XXX" + pp.ZeroOrMore( aaa | bbb | ccc )
- teststring = "XXX b bb a bbb bbbb aa bbbbb :c bbbbbb aaa"
- names = []
- print_(g1.parseString(teststring).dump())
- for t in g1.parseString(teststring):
- print_(t, repr(t))
- try:
- names.append( t[0].getName() )
- except Exception:
- try:
- names.append( t.getName() )
- except Exception:
- names.append( None )
- print_(teststring)
- print_(names)
- self.assertEqual(names, [None, 'B', 'B', 'A', 'B', 'B', 'A', 'B', None, 'B', 'A'],
- "failure in getting names for tokens")
-
- from pyparsing import Keyword, Word, alphas, OneOrMore
- IF,AND,BUT = map(Keyword, "if and but".split())
- ident = ~(IF | AND | BUT) + Word(alphas)("non-key")
- scanner = OneOrMore(IF | AND | BUT | ident)
- def getNameTester(s,l,t):
- print_(t, t.getName())
- ident.addParseAction(getNameTester)
- scanner.parseString("lsjd sldkjf IF Saslkj AND lsdjf")
-
- # test ParseResults.get() method
- if "H" in runtests:
- print_("verify behavior of ParseResults.get()")
- # use sum() to merge separate groups into single ParseResults
- res = sum(g1.parseString(teststring)[1:])
- print_(res.dump())
- print_(res.get("A","A not found"))
- print_(res.get("D","!D"))
- self.assertEqual(res.get("A","A not found"), "aaa", "get on existing key failed")
- self.assertEqual(res.get("D","!D"), "!D", "get on missing key failed")
-
- if "I" in runtests:
- print_("verify handling of Optional's beyond the end of string")
- testGrammar = "A" + pp.Optional("B") + pp.Optional("C") + pp.Optional("D")
- testGrammar.parseString("A")
- testGrammar.parseString("AB")
-
- # test creating Literal with empty string
- if "J" in runtests:
- print_('verify non-fatal usage of Literal("")')
- e = pp.Literal("")
- try:
- e.parseString("SLJFD")
- except Exception as e:
- self.assertTrue(False, "Failed to handle empty Literal")
-
- # test line() behavior when starting at 0 and the opening line is an \n
- if "K" in runtests:
- print_('verify correct line() behavior when first line is empty string')
- self.assertEqual(pp.line(0, "\nabc\ndef\n"), '', "Error in line() with empty first line in text")
- txt = "\nabc\ndef\n"
- results = [ pp.line(i,txt) for i in range(len(txt)) ]
- self.assertEqual(results, ['', 'abc', 'abc', 'abc', 'abc', 'def', 'def', 'def', 'def'],
- "Error in line() with empty first line in text")
- txt = "abc\ndef\n"
- results = [ pp.line(i,txt) for i in range(len(txt)) ]
- self.assertEqual(results, ['abc', 'abc', 'abc', 'abc', 'def', 'def', 'def', 'def'],
- "Error in line() with non-empty first line in text")
-
- # test bugfix with repeated tokens when packrat parsing enabled
- if "L" in runtests:
- print_('verify behavior with repeated tokens when packrat parsing is enabled')
- a = pp.Literal("a")
- b = pp.Literal("b")
- c = pp.Literal("c")
-
- abb = a + b + b
- abc = a + b + c
- aba = a + b + a
- grammar = abb | abc | aba
-
- self.assertEqual(''.join(grammar.parseString( "aba" )), 'aba', "Packrat ABA failure!")
-
- if "M" in runtests:
- print_('verify behavior of setResultsName with OneOrMore and ZeroOrMore')
-
- stmt = pp.Keyword('test')
- print_(pp.ZeroOrMore(stmt)('tests').parseString('test test').tests)
- print_(pp.OneOrMore(stmt)('tests').parseString('test test').tests)
- print_(pp.Optional(pp.OneOrMore(stmt)('tests')).parseString('test test').tests)
- print_(pp.Optional(pp.OneOrMore(stmt))('tests').parseString('test test').tests)
- print_(pp.Optional(pp.delimitedList(stmt))('tests').parseString('test,test').tests)
- self.assertEqual(len(pp.ZeroOrMore(stmt)('tests').parseString('test test').tests), 2, "ZeroOrMore failure with setResultsName")
- self.assertEqual(len(pp.OneOrMore(stmt)('tests').parseString('test test').tests), 2, "OneOrMore failure with setResultsName")
- self.assertEqual(len(pp.Optional(pp.OneOrMore(stmt)('tests')).parseString('test test').tests), 2, "OneOrMore failure with setResultsName")
- self.assertEqual(len(pp.Optional(pp.delimitedList(stmt))('tests').parseString('test,test').tests), 2, "delimitedList failure with setResultsName")
- self.assertEqual(len((stmt*2)('tests').parseString('test test').tests), 2, "multiplied(1) failure with setResultsName")
- self.assertEqual(len((stmt*(None,2))('tests').parseString('test test').tests), 2, "multiplied(2) failure with setResultsName")
- self.assertEqual(len((stmt*(1,))('tests').parseString('test test').tests), 2, "multipled(3) failure with setResultsName")
- self.assertEqual(len((stmt*(2,))('tests').parseString('test test').tests), 2, "multipled(3) failure with setResultsName")
-
-def makeTestSuite():
- import inspect
- suite = TestSuite()
- suite.addTest(PyparsingTestInit())
-
- test_case_classes = ParseTestCase.__subclasses__()
- # put classes in order as they are listed in the source code
- test_case_classes.sort(key=lambda cls: inspect.getsourcelines(cls)[1])
-
- test_case_classes.remove(PyparsingTestInit)
- # test_case_classes.remove(ParseASMLTest)
- test_case_classes.remove(EnablePackratParsing)
- if IRON_PYTHON_ENV:
- test_case_classes.remove(OriginalTextForTest)
-
- suite.addTests(T() for T in test_case_classes)
-
- if TEST_USING_PACKRAT:
- # retest using packrat parsing (disable those tests that aren't compatible)
- suite.addTest( EnablePackratParsing() )
-
- unpackrattables = [ PyparsingTestInit, EnablePackratParsing, RepeaterTest, ]
-
- # add tests to test suite a second time, to run with packrat parsing
- # (leaving out those that we know wont work with packrat)
- packratTests = [t.__class__() for t in suite._tests
- if t.__class__ not in unpackrattables]
- suite.addTests( packratTests )
-
- return suite
-
-def makeTestSuiteTemp(classes):
- suite = TestSuite()
- suite.addTest(PyparsingTestInit())
- suite.addTests(cls() for cls in classes)
- return suite
-
-# runnable from setup.py using "python setup.py test -s unitTests.suite"
-suite = makeTestSuite()
-
-
-if __name__ == '__main__':
-
- # run specific tests by including them in this list, otherwise
- # all tests will be run
- testclasses = [
- ]
-
- if not testclasses:
- testRunner = TextTestRunner()
- result = testRunner.run(suite)
- else:
- # disable chaser '.' display
- testRunner = TextTestRunner(verbosity=0)
- BUFFER_OUTPUT = False
- result = testRunner.run(makeTestSuiteTemp(testclasses))
-
- sys.stdout.flush()
- exit(0 if result.wasSuccessful() else 1)
projects / platform / upstream / python3-pyparsing.git / commitdiff
