From: DongHun Kwak Date: Mon, 18 Jul 2022 02:07:34 +0000 (+0900) Subject: Imported Upstream version 2.11.0 X-Git-Tag: upstream/2.11.0^0 X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=d5cf78785202573dbdfed52952484b6b7b444497;p=platform%2Fupstream%2Fpython3-pygments.git Imported Upstream version 2.11.0 --- diff --git a/AUTHORS b/AUTHORS index 9fc0393..d33be64 100644 --- a/AUTHORS +++ b/AUTHORS @@ -6,6 +6,7 @@ Major developers are Tim Hatch and Armin Ronacher Other contributors, listed alphabetically, are: * Sam Aaron -- Ioke lexer +* Jean Abou Samra -- LilyPond lexer * João Abecasis -- JSLT lexer * Ali Afshar -- image formatter * Thomas Aglassinger -- Easytrieve, JCL, Rexx, Transact-SQL and VBScript @@ -117,6 +118,7 @@ Other contributors, listed alphabetically, are: * Gerwin Klein -- Isabelle lexer * Eric Knibbe -- Lasso lexer * Stepan Koltsov -- Clay lexer +* Oliver Kopp - Friendly grayscale style * Adam Koprowski -- Opa lexer * Benjamin Kowarsch -- Modula-2 lexer * Domen Kožar -- Nix lexer @@ -144,6 +146,7 @@ Other contributors, listed alphabetically, are: * Brian McKenna -- F# lexer * Charles McLaughlin -- Puppet lexer * Kurt McKee -- Tera Term macro lexer, PostgreSQL updates, MySQL overhaul +* Joe Eli McIlvain -- Savi lexer * Lukas Meuser -- BBCode formatter, Lua lexer * Cat Miller -- Pig lexer * Paul Miller -- LiveScript lexer @@ -219,6 +222,7 @@ Other contributors, listed alphabetically, are: * Brian Tiffin -- OpenCOBOL lexer * Bob Tolbert -- Hy lexer * Matthias Trute -- Forth lexer +* Tuoa Spi T4 -- Bdd lexer * Erick Tryzelaar -- Felix lexer * Alexander Udalov -- Kotlin lexer improvements * Thomas Van Doren -- Chapel lexer @@ -246,5 +250,8 @@ Other contributors, listed alphabetically, are: * Fabian Neumann -- CDDL lexer * Thomas Duboucher -- CDDL lexer * Philipp Imhof -- Pango Markup formatter +* Thomas Voss -- Sed lexer +* Martin Fischer -- WCAG contrast testing +* Marc Auberer -- Spice lexer Many thanks for all contributions! diff --git a/CHANGES b/CHANGES index 4827f92..ae04ef5 100644 --- a/CHANGES +++ b/CHANGES @@ -1,10 +1,84 @@ Pygments changelog ================== -Since 2.5.0, issue numbers refer to the tracker at -, -pull request numbers to the requests at -. +Pull request numbers before 2.4.2 are not linked as they refer to the now defunct Bitbucket project. + +Version 2.11.0 +-------------- +(released December 30th, 2021) + +- Added lexers: + + * BDD (#1803) + * Elpi (#1894) + * LilyPond (#1845, #1968, #1971, #2001). This comes with a custom style as well. + * Maxima (#1885) + * Rita (#1541, #2003) + * Savi (#1863) + * Sed (#1935) + * Sophia contracts (#1974) + * Spice (#1980) + * ``.SRCINFO`` (#1951) + +- Updated lexers: + + * ABNF: Allow one-character rules (#1804) + * Assembly: Fix incorrect token endings (#1895, #1961) + * Bibtex: Distinguish between ``comment`` and ``commentary`` (#1899, #1806) + * C family: Support unicode identifiers (#1848) + * CDDL: Fix slow lexing speed (#1959) + * Debian control: Add missing fields (#1946) + * Devicetree: Recognize hexadecimal addresses for nodes (#1949) + * GDScript: Add ``void`` data type (#1948) + * GSQL + + - Fix comment handling (#2002) + - Fix catastrophic backtracking (#2006) + + * HTML, XML: Improve comment handling (#1896) + * Java: Add ``yield`` (#1941) and sealed classes/record (#1902) + * Makefiles (#1860, #1898) + * objdump-nasm: Improve handling of ``--no-show-raw-insn`` dumps (#1981) + * Prolog: Support escaped ``\`` inside quoted strings (#1479) + * Python: + + - Support ``~`` in tracebacks (#2004) + - Support the pattern matching keywords (#1797, #1994) + + * RobotFramework: Improve empty brace handling (#1921, #1922) + * Terraform + + - Add the 'set' type (#1909) + - Support heredocs (#1909) + +- Added styles: + + * Dracula (#1796) + * Friendly Grayscale (#1040, #1273) + * LilyPond (#1845) -- to be used for the ``LilyPond`` language. + * One-Dark (#1924, #1979) + +.. note:: + + All of the new styles unfortunately do not conform to WCAG recommendations. + +- There is new infrastructure in place to improve style accessibility. The default style has been updated to conform to WCAG recommendations. All styles are now checked for sufficient contrast by default to prevent regressions. (#1919, #1937, #1938, #1940) +- Clean up unused imports (#1887) +- Fix multiple lexers producing repeated single-character tokens +- Fix multiple lexers marking whitespace as ``Text`` (#1237, #1905, #1908, #1914, #1911, #1923, #1939, #1957, #1978) +- Remove duplicated assignments in the Paraiso style (#1934) +- ``pygmentize`` supports JSON output for the various list functions now, making it easier to consume them from scripts. (#1437, #1890) +- Use the ``shell`` lexer for ``kshrc`` files (#1947) +- Use the ``ruby`` lexer for ``Vagrantfile`` files (#1936) +- Use the C lexer for ``.xbm`` and ``.xpm`` files (#1802) +- Add a ``groff`` formatter (#1873) +- Update documentation (#1928) +- Line anchors now link to themselves (#1973) +- Add official support for Python 3.10 (#1917) +- Fix several missing colors in dark styles: Gruvbox dark, Monokai, Rrt, Sas, Strata dark (#1955) +- Associate more file types with ``man`` pages +- The ``HtmlFormatter`` can now emit tooltips for each token to ease debugging of lexers (#1822) +- Add ``f90`` as an alias for ``fortran`` (#2000) Version 2.10.0 @@ -27,6 +101,7 @@ Version 2.10.0 * Fennel (#1862) * JavaScript (#1741, #1814) * LLVM (#1824) + * Python (#1852) * Rust - Fix lexing of "break" and "continue" (#1843) @@ -35,6 +110,7 @@ Version 2.10.0 * Scala: Add support for the ``\`` operator (#1857) * Swift (#1767, #1842) * Tcl: Allow ``,`` and ``@`` in strings (#1834, #1742) + * TOML (#1870, #1872) - Fix assert statements in TNT lexer. - Token types across all lexers have been unified (using the most common token @@ -70,6 +146,7 @@ Version 2.9.0 * Chapel (#1743) * Coq (#1721) * Cython (#853) + * DeviceTree (#1755) * Groovy (#1765) * Julia (#1715) * Octave: Allow multiline and block-percent comments (#1726) @@ -89,6 +166,7 @@ Version 2.9.0 - Improve line number colors (#1779, #1778) - Fix CTag related issue (#1724) - Recognize ``.leex`` as Elixir templates +- Fix incorrect variable being accessed (#1748) - Updated `filename` handling in HTML formatter if `linenos='table'` (#1757) @@ -139,6 +217,7 @@ Version 2.8.0 - Update builtin functions (#1705) - Various cleanups (#1673) + * OpenEdge (#1696) * Python: Improve handling of raw f-strings (#1681, #1683) * Ruby: Better method name handling (#1531) * Stata: Updated keywords (#1470) @@ -213,9 +292,9 @@ Version 2.7.4 - SQL - Fix keywords (#1668) - Typescript: Fix incorrect punctuation handling (#1510, #1511) -- Fix infinite loop in SML lexer (#1625) +- Fix infinite loop in SML lexer (#1625), `CVE-2021-20270 `_ - Fix backtracking string regexes in JavaScript/TypeScript, Modula2 - and many other lexers (#1637) + and many other lexers (#1637) `CVE-2021-27291 `_ - Limit recursion with nesting Ruby heredocs (#1638) - Fix a few inefficient regexes for guessing lexers - Fix the raw token lexer handling of Unicode (#1616) diff --git a/Makefile b/Makefile index f490b76..404bd6d 100644 --- a/Makefile +++ b/Makefile @@ -20,10 +20,11 @@ all: clean-pyc check test check: @$(PYTHON) scripts/check_crlf.py pygments build external - @$(PYTHON) scripts/detect_missing_analyse_text.py || true + @$(PYTHON) scripts/detect_missing_analyse_text.py --skip-no-aliases @pyflakes pygments | grep -v 'but unused' || true @$(PYTHON) scripts/check_sources.py -i build -i dist -i pygments/lexers/_mapping.py \ - -i docs/build -i pygments/formatters/_mapping.py -i pygments/unistring.py + -i docs/build -i pygments/formatters/_mapping.py -i pygments/unistring.py \ + -i tests/support/empty.py @$(PYTHON) scripts/count_token_references.py --minfiles=1 --maxfiles=1 \ --minlines=1 --maxlines=3 --subtoken @@ -50,10 +51,10 @@ reindent: TEST = tests test: - @$(PYTHON) `which py.test` $(TEST) + @$(PYTHON) `which pytest` $(TEST) test-coverage: - @$(PYTHON) `which py.test` --cov --cov-report=html --cov-report=term $(TEST) + @$(PYTHON) `which pytest` --cov --cov-report=html --cov-report=term $(TEST) tox-test: @tox -- $(TEST) diff --git a/PKG-INFO b/PKG-INFO index ba79439..0c69451 100644 --- a/PKG-INFO +++ b/PKG-INFO @@ -1,29 +1,15 @@ -Metadata-Version: 1.2 +Metadata-Version: 2.1 Name: Pygments -Version: 2.10.0 +Version: 2.11.0 Summary: Pygments is a syntax highlighting package written in Python. Home-page: https://pygments.org/ Author: Georg Brandl Author-email: georg@python.org License: BSD License -Description: Pygments - ~~~~~~~~ - - Pygments is a syntax highlighting package written in Python. - - It is a generic syntax highlighter suitable for use in code hosting, forums, - wikis or other applications that need to prettify source code. Highlights - are: - - * a wide range of over 500 languages and other text formats is supported - * special attention is paid to details, increasing quality by a fair amount - * support for new languages and formats are added easily - * a number of output formats, presently HTML, LaTeX, RTF, SVG, all image - formats that PIL supports and ANSI sequences - * it is usable as a command-line tool and as a library - - Copyright 2006-2021 by the Pygments team, see ``AUTHORS``. - Licensed under the BSD, see ``LICENSE`` for details. +Project-URL: Documentation, https://pygments.org/docs/ +Project-URL: Source, https://github.com/pygments/pygments +Project-URL: Bug Tracker, https://github.com/pygments/pygments/issues +Project-URL: Changelog, https://github.com/pygments/pygments/blob/master/CHANGES Keywords: syntax highlighting Platform: any Classifier: Development Status :: 6 - Mature @@ -39,8 +25,31 @@ Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 +Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Text Processing :: Filters Classifier: Topic :: Utilities Requires-Python: >=3.5 +License-File: LICENSE +License-File: AUTHORS + +Pygments +~~~~~~~~ + +Pygments is a syntax highlighting package written in Python. + +It is a generic syntax highlighter suitable for use in code hosting, forums, +wikis or other applications that need to prettify source code. Highlights +are: + +* a wide range of over 500 languages and other text formats is supported +* special attention is paid to details, increasing quality by a fair amount +* support for new languages and formats are added easily +* a number of output formats, presently HTML, LaTeX, RTF, SVG, all image + formats that PIL supports and ANSI sequences +* it is usable as a command-line tool and as a library + +Copyright 2006-2021 by the Pygments team, see ``AUTHORS``. +Licensed under the BSD, see ``LICENSE`` for details. + diff --git a/Pygments.egg-info/PKG-INFO b/Pygments.egg-info/PKG-INFO index ba79439..0c69451 100644 --- a/Pygments.egg-info/PKG-INFO +++ b/Pygments.egg-info/PKG-INFO @@ -1,29 +1,15 @@ -Metadata-Version: 1.2 +Metadata-Version: 2.1 Name: Pygments -Version: 2.10.0 +Version: 2.11.0 Summary: Pygments is a syntax highlighting package written in Python. Home-page: https://pygments.org/ Author: Georg Brandl Author-email: georg@python.org License: BSD License -Description: Pygments - ~~~~~~~~ - - Pygments is a syntax highlighting package written in Python. - - It is a generic syntax highlighter suitable for use in code hosting, forums, - wikis or other applications that need to prettify source code. Highlights - are: - - * a wide range of over 500 languages and other text formats is supported - * special attention is paid to details, increasing quality by a fair amount - * support for new languages and formats are added easily - * a number of output formats, presently HTML, LaTeX, RTF, SVG, all image - formats that PIL supports and ANSI sequences - * it is usable as a command-line tool and as a library - - Copyright 2006-2021 by the Pygments team, see ``AUTHORS``. - Licensed under the BSD, see ``LICENSE`` for details. +Project-URL: Documentation, https://pygments.org/docs/ +Project-URL: Source, https://github.com/pygments/pygments +Project-URL: Bug Tracker, https://github.com/pygments/pygments/issues +Project-URL: Changelog, https://github.com/pygments/pygments/blob/master/CHANGES Keywords: syntax highlighting Platform: any Classifier: Development Status :: 6 - Mature @@ -39,8 +25,31 @@ Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 +Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Text Processing :: Filters Classifier: Topic :: Utilities Requires-Python: >=3.5 +License-File: LICENSE +License-File: AUTHORS + +Pygments +~~~~~~~~ + +Pygments is a syntax highlighting package written in Python. + +It is a generic syntax highlighter suitable for use in code hosting, forums, +wikis or other applications that need to prettify source code. Highlights +are: + +* a wide range of over 500 languages and other text formats is supported +* special attention is paid to details, increasing quality by a fair amount +* support for new languages and formats are added easily +* a number of output formats, presently HTML, LaTeX, RTF, SVG, all image + formats that PIL supports and ANSI sequences +* it is usable as a command-line tool and as a library + +Copyright 2006-2021 by the Pygments team, see ``AUTHORS``. +Licensed under the BSD, see ``LICENSE`` for details. + diff --git a/Pygments.egg-info/SOURCES.txt b/Pygments.egg-info/SOURCES.txt index 5c185ae..e05bca2 100644 --- a/Pygments.egg-info/SOURCES.txt +++ b/Pygments.egg-info/SOURCES.txt @@ -20,6 +20,8 @@ doc/index.rst doc/languages.rst doc/make.bat doc/pygmentize.1 +doc/styles.rst +doc/_static/demo-worker.js doc/_static/demo.css doc/_static/demo.js doc/_static/favicon.ico @@ -32,7 +34,10 @@ doc/_templates/demo_sidebar.html doc/_templates/docssidebar.html doc/_templates/index_with_try.html doc/_templates/indexsidebar.html +doc/_templates/styles.html doc/_themes/pygments14/layout.html +doc/_themes/pygments14/localtoc.html +doc/_themes/pygments14/relations.html doc/_themes/pygments14/theme.conf doc/_themes/pygments14/static/bodybg.png doc/_themes/pygments14/static/docbg.png @@ -61,8 +66,10 @@ doc/docs/styledevelopment.rst doc/docs/styles.rst doc/docs/tokens.rst doc/docs/unicode.rst +doc/examples/example.py external/autopygmentize external/lasso-builtins-generator-9.lasso +external/lilypond-builtins-generator.ly external/markdown-processor.py external/moin-parser.py external/pygments.bashcomp @@ -87,6 +94,7 @@ pygments/filters/__init__.py pygments/formatters/__init__.py pygments/formatters/_mapping.py pygments/formatters/bbcode.py +pygments/formatters/groff.py pygments/formatters/html.py pygments/formatters/img.py pygments/formatters/irc.py @@ -104,6 +112,7 @@ pygments/lexers/_cocoa_builtins.py pygments/lexers/_csound_builtins.py pygments/lexers/_julia_builtins.py pygments/lexers/_lasso_builtins.py +pygments/lexers/_lilypond_builtins.py pygments/lexers/_lua_builtins.py pygments/lexers/_mapping.py pygments/lexers/_mql_builtins.py @@ -134,6 +143,7 @@ pygments/lexers/asm.py pygments/lexers/automation.py pygments/lexers/bare.py pygments/lexers/basic.py +pygments/lexers/bdd.py pygments/lexers/bibtex.py pygments/lexers/boa.py pygments/lexers/business.py @@ -160,6 +170,7 @@ pygments/lexers/dylan.py pygments/lexers/ecl.py pygments/lexers/eiffel.py pygments/lexers/elm.py +pygments/lexers/elpi.py pygments/lexers/email.py pygments/lexers/erlang.py pygments/lexers/esoteric.py @@ -199,11 +210,13 @@ pygments/lexers/jslt.py pygments/lexers/julia.py pygments/lexers/jvm.py pygments/lexers/kuin.py +pygments/lexers/lilypond.py pygments/lexers/lisp.py pygments/lexers/make.py pygments/lexers/markup.py pygments/lexers/math.py pygments/lexers/matlab.py +pygments/lexers/maxima.py pygments/lexers/meson.py pygments/lexers/mime.py pygments/lexers/ml.py @@ -238,12 +251,14 @@ pygments/lexers/rdf.py pygments/lexers/rebol.py pygments/lexers/resource.py pygments/lexers/ride.py +pygments/lexers/rita.py pygments/lexers/rnc.py pygments/lexers/roboconf.py pygments/lexers/robotframework.py pygments/lexers/ruby.py pygments/lexers/rust.py pygments/lexers/sas.py +pygments/lexers/savi.py pygments/lexers/scdoc.py pygments/lexers/scripting.py pygments/lexers/sgf.py @@ -255,8 +270,11 @@ pygments/lexers/smithy.py pygments/lexers/smv.py pygments/lexers/snobol.py pygments/lexers/solidity.py +pygments/lexers/sophia.py pygments/lexers/special.py +pygments/lexers/spice.py pygments/lexers/sql.py +pygments/lexers/srcinfo.py pygments/lexers/stata.py pygments/lexers/supercollider.py pygments/lexers/tcl.py @@ -296,18 +314,22 @@ pygments/styles/borland.py pygments/styles/bw.py pygments/styles/colorful.py pygments/styles/default.py +pygments/styles/dracula.py pygments/styles/emacs.py pygments/styles/friendly.py +pygments/styles/friendly_grayscale.py pygments/styles/fruity.py pygments/styles/gruvbox.py pygments/styles/igor.py pygments/styles/inkpot.py +pygments/styles/lilypond.py pygments/styles/lovelace.py pygments/styles/manni.py pygments/styles/material.py pygments/styles/monokai.py pygments/styles/murphy.py pygments/styles/native.py +pygments/styles/onedark.py pygments/styles/paraiso_dark.py pygments/styles/paraiso_light.py pygments/styles/pastie.py @@ -325,6 +347,7 @@ pygments/styles/vs.py pygments/styles/xcode.py pygments/styles/zenburn.py scripts/check_crlf.py +scripts/check_repeated_token.py scripts/check_sources.py scripts/count_token_references.py scripts/debug_lexer.py @@ -332,6 +355,7 @@ scripts/detect_missing_analyse_text.py scripts/get_vimkw.py scripts/pylintrc scripts/release-checklist +scripts/update_contrasts.py scripts/vim2pygments.py tests/__init__.py tests/conftest.py @@ -359,6 +383,7 @@ tests/test_procfile.py tests/test_raw_token.py tests/test_regexlexer.py tests/test_regexopt.py +tests/test_robotframework_lexer.py tests/test_rtf_formatter.py tests/test_ruby.py tests/test_sql.py @@ -372,6 +397,8 @@ tests/test_usd.py tests/test_using_api.py tests/test_util.py tests/test_words.py +tests/contrast/min_contrasts.json +tests/contrast/test_contrasts.py tests/dtds/HTML4-f.dtd tests/dtds/HTML4-s.dtd tests/dtds/HTML4.dcl @@ -459,6 +486,8 @@ tests/examplefiles/bbcbasic/example.bbc tests/examplefiles/bbcbasic/example.bbc.output tests/examplefiles/bc/example.bc tests/examplefiles/bc/example.bc.output +tests/examplefiles/bdd/example.feature +tests/examplefiles/bdd/example.feature.output tests/examplefiles/bib/test.bib tests/examplefiles/bib/test.bib.output tests/examplefiles/blitzbasic/test.bb @@ -493,6 +522,8 @@ tests/examplefiles/camkes/simple.camkes tests/examplefiles/camkes/simple.camkes.output tests/examplefiles/capdl/capdl_example.cdl tests/examplefiles/capdl/capdl_example.cdl.output +tests/examplefiles/capnp/pets.capnp +tests/examplefiles/capnp/pets.capnp.output tests/examplefiles/cbmbas/cbmbas_example tests/examplefiles/cbmbas/cbmbas_example.output tests/examplefiles/cddl/example.cddl @@ -541,6 +572,8 @@ tests/examplefiles/console/example.sh-session tests/examplefiles/console/example.sh-session.output tests/examplefiles/console/example.shell-session tests/examplefiles/console/example.shell-session.output +tests/examplefiles/control/control +tests/examplefiles/control/control.output tests/examplefiles/coq/coq_RelationClasses tests/examplefiles/coq/coq_RelationClasses.output tests/examplefiles/cpp/example.cpp @@ -861,6 +894,8 @@ tests/examplefiles/lhs/Sudoku.lhs tests/examplefiles/lhs/Sudoku.lhs.output tests/examplefiles/lighty/lighttpd_config.conf tests/examplefiles/lighty/lighttpd_config.conf.output +tests/examplefiles/lilypond/example.ly +tests/examplefiles/lilypond/example.ly.output tests/examplefiles/limbo/limbo.b tests/examplefiles/limbo/limbo.b.output tests/examplefiles/liquid/example.liquid @@ -901,6 +936,8 @@ tests/examplefiles/matlab/matlab_sample tests/examplefiles/matlab/matlab_sample.output tests/examplefiles/matlabsession/matlabsession_sample.txt tests/examplefiles/matlabsession/matlabsession_sample.txt.output +tests/examplefiles/maxima/foo.mac +tests/examplefiles/maxima/foo.mac.output tests/examplefiles/md/example.md tests/examplefiles/md/example.md.output tests/examplefiles/meson/meson.build @@ -985,6 +1022,8 @@ tests/examplefiles/nt/test.nt tests/examplefiles/nt/test.nt.output tests/examplefiles/nusmv/guidance.smv tests/examplefiles/nusmv/guidance.smv.output +tests/examplefiles/objdump-nasm/no-show-raw-isns.dump +tests/examplefiles/objdump-nasm/no-show-raw-isns.dump.output tests/examplefiles/objective-c/objc_example.m tests/examplefiles/objective-c/objc_example.m.output tests/examplefiles/objective-j/CPDictionary.j @@ -1063,6 +1102,8 @@ tests/examplefiles/prolog/qsort.prolog tests/examplefiles/prolog/qsort.prolog.output tests/examplefiles/prolog/sibling.prolog tests/examplefiles/prolog/sibling.prolog.output +tests/examplefiles/prolog/strings.prolog +tests/examplefiles/prolog/strings.prolog.output tests/examplefiles/promql/example.promql tests/examplefiles/promql/example.promql.output tests/examplefiles/properties/java.properties @@ -1107,6 +1148,8 @@ tests/examplefiles/python/linecontinuation.py tests/examplefiles/python/linecontinuation.py.output tests/examplefiles/python/py3_test.txt tests/examplefiles/python/py3_test.txt.output +tests/examplefiles/python/switch_case.txt +tests/examplefiles/python/switch_case.txt.output tests/examplefiles/python/unicodedoc.py tests/examplefiles/python/unicodedoc.py.output tests/examplefiles/qbasic/qbasic_example @@ -1165,6 +1208,8 @@ tests/examplefiles/rhtml/test.rhtml tests/examplefiles/rhtml/test.rhtml.output tests/examplefiles/ride/auction.ride tests/examplefiles/ride/auction.ride.output +tests/examplefiles/rita/example.rita +tests/examplefiles/rita/example.rita.output tests/examplefiles/rnc/rnc_example.rnc tests/examplefiles/rnc/rnc_example.rnc.output tests/examplefiles/roboconf-graph/roboconf.graph @@ -1185,6 +1230,8 @@ tests/examplefiles/rts/example.rts tests/examplefiles/rts/example.rts.output tests/examplefiles/rust/eval.rs tests/examplefiles/rust/eval.rs.output +tests/examplefiles/savi/example.savi +tests/examplefiles/savi/example.savi.output tests/examplefiles/sc/example.scd tests/examplefiles/sc/example.scd.output tests/examplefiles/scala/char.scala @@ -1251,6 +1298,14 @@ tests/examplefiles/scilab/foo.sce tests/examplefiles/scilab/foo.sce.output tests/examplefiles/scilab/scilab.sci tests/examplefiles/scilab/scilab.sci.output +tests/examplefiles/sed/all_sorts_of_syntax.sed +tests/examplefiles/sed/all_sorts_of_syntax.sed.output +tests/examplefiles/sed/count_words.sed +tests/examplefiles/sed/count_words.sed.output +tests/examplefiles/sed/increment_number.sed +tests/examplefiles/sed/increment_number.sed.output +tests/examplefiles/sed/reverse.sed +tests/examplefiles/sed/reverse.sed.output tests/examplefiles/sgf/example.sgf tests/examplefiles/sgf/example.sgf.output tests/examplefiles/shen/test.shen @@ -1287,6 +1342,8 @@ tests/examplefiles/snowball/example.sbl tests/examplefiles/snowball/example.sbl.output tests/examplefiles/solidity/test.sol tests/examplefiles/solidity/test.sol.output +tests/examplefiles/sophia/test.aes +tests/examplefiles/sophia/test.aes.output tests/examplefiles/sourceslist/sources.list tests/examplefiles/sourceslist/sources.list.output tests/examplefiles/sp/hello.sp @@ -1295,12 +1352,22 @@ tests/examplefiles/sparql/sparql.rq tests/examplefiles/sparql/sparql.rq.output tests/examplefiles/spec/phpMyAdmin.spec tests/examplefiles/spec/phpMyAdmin.spec.output +tests/examplefiles/spice/example.spice +tests/examplefiles/spice/example.spice.output tests/examplefiles/splus/test.R tests/examplefiles/splus/test.R.output tests/examplefiles/sqlite3/sqlite3.sqlite3-console tests/examplefiles/sqlite3/sqlite3.sqlite3-console.output tests/examplefiles/squidconf/squid.conf tests/examplefiles/squidconf/squid.conf.output +tests/examplefiles/srcinfo/c-intercal.SRCINFO +tests/examplefiles/srcinfo/c-intercal.SRCINFO.output +tests/examplefiles/srcinfo/compiler-rt.SRCINFO +tests/examplefiles/srcinfo/compiler-rt.SRCINFO.output +tests/examplefiles/srcinfo/hunspell.SRCINFO +tests/examplefiles/srcinfo/hunspell.SRCINFO.output +tests/examplefiles/srcinfo/thunderbird.SRCINFO +tests/examplefiles/srcinfo/thunderbird.SRCINFO.output tests/examplefiles/ssp/test.ssp tests/examplefiles/ssp/test.ssp.output tests/examplefiles/stan/example.stan @@ -1604,6 +1671,7 @@ tests/snippets/console/test_virtualenv.txt tests/snippets/coq/test_unicode.txt tests/snippets/cpp/test_good_comment.txt tests/snippets/cpp/test_open_comment.txt +tests/snippets/cpp/test_unicode_identifiers.txt tests/snippets/crystal/test_annotation.txt tests/snippets/crystal/test_array_access.txt tests/snippets/crystal/test_chars.txt @@ -1642,6 +1710,11 @@ tests/snippets/csound/test_printks_and_prints_escape_sequences.txt tests/snippets/csound/test_quoted_strings.txt tests/snippets/csound/test_user_defined_opcodes.txt tests/snippets/doscon/test_gt_only.txt +tests/snippets/elpi/test_chr.txt +tests/snippets/elpi/test_clause.txt +tests/snippets/elpi/test_namespace.txt +tests/snippets/elpi/test_pred.txt +tests/snippets/elpi/test_type.txt tests/snippets/ezhil/test_function.txt tests/snippets/ezhil/test_gcd_expr.txt tests/snippets/ezhil/test_if_statement.txt @@ -1671,6 +1744,7 @@ tests/snippets/j/test_deal_operator_fixed_seed.txt tests/snippets/java/test_default.txt tests/snippets/java/test_enhanced_for.txt tests/snippets/java/test_numeric_literals.txt +tests/snippets/java/test_record.txt tests/snippets/jslt/test_sample.txt tests/snippets/json/test_basic.txt tests/snippets/json/test_basic_bare.txt @@ -1730,6 +1804,7 @@ tests/snippets/md/test_reference_style_links.txt tests/snippets/md/test_strikethrough.txt tests/snippets/md/test_task_list.txt tests/snippets/md/test_topics.txt +tests/snippets/nasm/checkid.txt tests/snippets/objectivec/test_literal_number_bool.txt tests/snippets/objectivec/test_literal_number_bool_expression.txt tests/snippets/objectivec/test_literal_number_expression.txt @@ -1911,6 +1986,7 @@ tests/snippets/terraform/test_attributes.txt tests/snippets/terraform/test_backend.txt tests/snippets/terraform/test_comment.txt tests/snippets/terraform/test_functions.txt +tests/snippets/terraform/test_heredoc.txt tests/snippets/terraform/test_module.txt tests/snippets/terraform/test_resource.txt tests/snippets/terraform/test_types.txt @@ -1955,4 +2031,5 @@ tests/snippets/yang/test_string_value.txt tests/support/empty.py tests/support/html_formatter.py tests/support/python_lexer.py +tests/support/structural_diff.py tests/support/tags \ No newline at end of file diff --git a/doc/_static/demo-worker.js b/doc/_static/demo-worker.js new file mode 100644 index 0000000..af8b009 --- /dev/null +++ b/doc/_static/demo-worker.js @@ -0,0 +1,72 @@ +self.languagePluginUrl = '/_static/pyodide/'; +importScripts('/_static/pyodide/pyodide.js'); + +(async function() { + await languagePluginLoader; + await self.pyodide.loadPackage(["Pygments"]); + const styles = self.pyodide.runPython(` + from pygments.formatters.html import HtmlFormatter + from pygments.styles import STYLE_MAP + {s: HtmlFormatter(style=s).get_style_defs('.demo-highlight') for s in STYLE_MAP} + `); + self.postMessage({loaded: {styles}}) +})(); + +self.onmessage = async (event) => { + if (event.data.highlight) { + self.pyodide.globals['code'] = event.data.highlight.code; + self.pyodide.globals['lexer_name'] = event.data.highlight.lexer; + + self.pyodide.runPython(` + import pygments.lexers + + lexer = pygments.lexers.get_lexer_by_name(lexer_name) + if type(code) == memoryview: + code = bytes(code) + tokens = lexer.get_tokens(code) + `); + + const formatter = event.data.highlight.formatter; + if (formatter == 'html') { + + const html = self.pyodide.runPython(` + import io + from pygments.formatters.html import HtmlFormatter + + fmter = HtmlFormatter(cssclass='demo-highlight') + buf = io.StringIO() + fmter.format(tokens, buf) + buf.getvalue() + `); + self.postMessage({html}); + } else if (formatter == 'tokens') { + const tokens = self.pyodide.runPython('list(tokens)'); + self.postMessage({tokens}); + } else { + console.warn('unknown formatter:', formatter); + } + } else if (event.data.guess_lexer) { + self.pyodide.globals['code'] = event.data.guess_lexer.code; + self.pyodide.globals['filename'] = event.data.guess_lexer.filename; + const lexer = self.pyodide.runPython(` + import sys + sys.setrecursionlimit(1000) + # TODO: remove after upgrading to Pyodide 0.19 + + import pygments.lexers + import pygments.util + + if type(code) == memoryview: + code = bytes(code) + + if filename: + lexer = pygments.lexers.guess_lexer_for_filename(filename, code) + else: + lexer = pygments.lexers.guess_lexer(code) + lexer.aliases[0] + `); + self.postMessage({lexer}); + } else { + console.warn('unknown command: expected highlight or guess_lexer but received ', event.data); + } +} diff --git a/doc/_static/demo.css b/doc/_static/demo.css index 90ceee2..e14379a 100644 --- a/doc/_static/demo.css +++ b/doc/_static/demo.css @@ -3,6 +3,7 @@ border-radius: 0; border: 1px solid #ccc; margin-top: 15px; + margin-bottom: 10px; padding: 10px 15px 5px 10px; position: relative; } @@ -19,29 +20,69 @@ } #hlcode { - padding: 10px 0; + margin: 10px 0; + max-height: 500px; + overflow: auto; + border: 1px solid #ccc; +} + +#hlcode:empty { + display: none; } #hlcode pre { background-color: transparent; - border-radius: 0; + border: 0; + margin: 0; +} +#hlcode table { + /* unset negative margin from pygments14.css */ + margin: unset; } -#hlcodedl { - text-align: right; +#code-header:not([hidden]) { + display: flex; + gap: 1em; padding: 0 15px; } +.flex-grow-1 { + flex-grow: 1; +} +#lexer { + margin-right: 0.5em; +} +#guessed-lexer:not(:empty):before { + content: '(guessed '; +} +#guessed-lexer:not(:empty):after { + content: ')'; +} + +#loading[hidden] { + visibility: hidden; + display: flex; +} #loading { - position: absolute; - top: 0; - left: 0; - width: 100%; - height: 100%; - margin: auto auto; - background-color: #cccccccc; display: flex; - flex-direction: column; - justify-content: center; - text-align: center; + align-items: center; + gap: 1em; +} + +#format-settings { + display: flex; + gap: 1em; + border-top: 1px solid #ccc; + padding-top: 0.5em; + margin-top: 0.5em; +} + +.tokens code { + /* make whitespace visible */ + white-space: pre; + background: #d9d9d9; +} + +#contrast-warning { + color: darkred; } diff --git a/doc/_static/demo.js b/doc/_static/demo.js index 1fc7a08..488ded0 100644 --- a/doc/_static/demo.js +++ b/doc/_static/demo.js @@ -1,126 +1,186 @@ -languagePluginLoader.then(() => { - // pyodide is now ready to use... - pyodide.loadPackage('Pygments').then(() => { - pyodide.runPython('import pygments.lexers, pygments.formatters.html, pygments.styles'); - - let qvars = getQueryVariables(); - - var lexerlist = pyodide.runPython('list(pygments.lexers.get_all_lexers())'); - var sel = document.getElementById("lang"); - for (lex of lexerlist) { - if (lex[1][0] === undefined) { - continue; - } - var opt = document.createElement("option"); - opt.text = lex[0]; - opt.value = lex[1][0]; - sel.add(opt); - if (lex[1].indexOf(qvars['lexer']) >= 0) { - opt.selected = true; - } - } +const loadingDiv = document.getElementById("loading"); +const langSelect = document.getElementById("lang"); +const styleSelect = document.getElementById("style"); +const formatterSelect = document.getElementById("formatter"); +const outputDiv = document.getElementById("hlcode"); +const codeHeader = document.getElementById("code-header"); +const copyLink = document.getElementById("copylink"); +const style = document.getElementById("css-style"); +const textarea = document.getElementById("code"); +const uriTooLongMsg = document.getElementById('uri-too-long'); +const contrastWarning = document.getElementById('contrast-warning'); +const fileInput = document.getElementById("file"); +const fileInputResetButton = document.getElementById('reset-file'); + +const qvars = Object.fromEntries(new URLSearchParams(window.location.search)); +if (qvars.lexer) { + langSelect.value = qvars.lexer; +} +if (qvars.code !== undefined) { + textarea.value = qvars.code; + loadingDiv.hidden = false; +} +if (qvars.style !== undefined) { + styleSelect.value = qvars.style; + updateContrastWarning(); +} +if (qvars.formatter !== undefined) { + formatterSelect.value = qvars.formatter; +} - var stylelist = pyodide.runPython('list(pygments.styles.get_all_styles())'); - var sel = document.getElementById("style"); - for (sty of stylelist) { - if (sty != "default") { - var opt = document.createElement("option"); - opt.text = sty; - opt.value = sty; - sel.add(opt); - } - } +styleSelect.addEventListener('change', () => { + style.textContent = styles[styleSelect.value]; + updateCopyLink(); + updateContrastWarning(); +}); - document.getElementById("hlbtn").disabled = false; - document.getElementById("loading").style.display = "none"; +function updateContrastWarning() { + contrastWarning.hidden = styleSelect.selectedOptions[0].dataset.wcag == 'aa'; +} - if (qvars['code'] !== undefined) { - document.getElementById("code").value = qvars['code']; - highlight(); - } - }); -}); +function debounce(func, timeout) { + let timer; + return (...args) => { + clearTimeout(timer); + timer = setTimeout(() => func.apply(this, args), timeout); + }; +} + +const highlightShortDebounce = debounce(highlight, 50); +const highlightLongDebounce = debounce(highlight, 500); + +function debouncedUpdate() { + if (fileInput.files.length > 0) + return; -function getQueryVariables() { - var query = window.location.search.substring(1); - var vars = query.split('&'); - var var_obj = {}; - for (var i = 0; i < vars.length; i++) { - var pair = vars[i].split('='); - var_obj[decodeURIComponent(pair[0])] = decodeURIComponent(pair[1]); + if (textarea.value.length < 1000) { + highlightShortDebounce(); + } else { + highlightLongDebounce(); } - return var_obj; } -function new_file() { - pyodide.globals['fname'] = document.getElementById("file").files[0].name; - var alias = pyodide.runPython('pygments.lexers.find_lexer_class_for_filename(fname).aliases[0]'); - var sel = document.getElementById("lang"); - for (var i = 0; i < sel.length; i++) { - if (sel.options[i].value == alias) { - sel.selectedIndex = i; - reset_err_hl(); - break; +langSelect.addEventListener('change', debouncedUpdate); +textarea.addEventListener('input', debouncedUpdate); +formatterSelect.addEventListener('change', debouncedUpdate); +fileInput.addEventListener('change', () => { + fileInputResetButton.hidden = false; + highlight(); +}); +fileInputResetButton.hidden = fileInput.files.length == 0; +fileInputResetButton.addEventListener('click', () => { + fileInputResetButton.hidden = true; + fileInput.value = ''; + highlight(); +}); + +let styles; + +const highlightWorker = new Worker("/_static/demo-worker.js"); +highlightWorker.onmessage = (msg) => { + if (msg.data.loaded) { + styles = msg.data.loaded.styles; + + if (qvars.code !== undefined || textarea.value) { + loadingDiv.hidden = true; + highlight(); } + } else if (msg.data.html) { + outputDiv.innerHTML = msg.data.html; + codeHeader.hidden = false; + loadingDiv.hidden = true; + style.textContent = styles[styleSelect.value]; + } else if (msg.data.tokens) { + const table = document.createElement('table'); + table.className = 'tokens'; + for (const [tokenType, value] of msg.data.tokens) { + const tr = document.createElement('tr'); + const td1 = document.createElement('td'); + td1.textContent = tokenType.join('.'); + const td2 = document.createElement('td'); + const inlineCode = document.createElement('code'); + inlineCode.textContent = value; + td2.appendChild(inlineCode); + tr.appendChild(td1); + tr.appendChild(td2); + table.appendChild(tr); + } + outputDiv.innerHTML = ''; + outputDiv.appendChild(table); + + codeHeader.hidden = false; + loadingDiv.hidden = true; + } else if (msg.data.lexer) { + highlight(msg.data.lexer); + } else { + console.warn('unexpected message from highlight worker', msg); + } +}; + +function updateCopyLink() { + var url = document.location.origin + document.location.pathname + + "?" + new URLSearchParams({ + lexer: langSelect.value, + style: styleSelect.value, + formatter: formatterSelect.value, + code: textarea.value, + }).toString() + if (url.length > 8201) { + // pygments.org is hosted on GitHub pages which does not support URIs longer than 8201 + copyLink.hidden = true; + uriTooLongMsg.hidden = false; + } else { + copyLink.href = url; + copyLink.textContent = 'Copy link'; + copyLink.hidden = false; + uriTooLongMsg.hidden = true; } } -function reset_err_hl() { - document.getElementById("aroundlang").style.backgroundColor = null; -} +async function highlight(guessedLexer) { + var lexer = langSelect.value || guessedLexer; + var file = fileInput.files[0]; -function highlight() { - var select = document.getElementById("lang"); - var alias = select.options.item(select.selectedIndex).value + let code; + if (file) { + code = await file.arrayBuffer(); + } else { + code = textarea.value; + } + + loadingDiv.hidden = false; - if (alias == "") { - document.getElementById("aroundlang").style.backgroundColor = "#ffcccc"; + if (!lexer) { + const guess_lexer = {code}; + if (file) + guess_lexer.filename = file.name; + highlightWorker.postMessage({guess_lexer}); + document.getElementById('loading-text').textContent = 'guessing lexer...'; return; } - pyodide.globals['alias'] = alias; - var select = document.getElementById("style"); - pyodide.globals['style'] = select.options.item(select.selectedIndex).value; + document.getElementById('loading-text').textContent = 'highlighting code...'; - pyodide.runPython('lexer = pygments.lexers.get_lexer_by_name(alias)'); - pyodide.runPython('fmter = pygments.formatters.html.HtmlFormatter(noclasses=True, style=style)'); + document.getElementById('guessed-lexer').textContent = guessedLexer; - var file = document.getElementById("file").files[0]; - if (file) { - file.arrayBuffer().then(function(buf) { - pyodide.globals['code_mem'] = buf; - pyodide.runPython('code = bytes(code_mem)'); - document.getElementById("copy_btn").style.display = "none"; - highlight_now(); - }); + highlightWorker.postMessage({highlight: {code, lexer, formatter: formatterSelect.value}}); + + if (code instanceof ArrayBuffer) { + copyLink.hidden = true; + uriTooLongMsg.hidden = true; } else { - var code = document.getElementById("code").value; - pyodide.globals['code'] = code; - var link = document.location.origin + document.location.pathname + - "?lexer=" + encodeURIComponent(alias) + "&code=" + encodeURIComponent(code); - document.getElementById("copy_field").value = link; - document.getElementById("copy_btn").style.display = ""; - highlight_now(); + updateCopyLink(); } } -function highlight_now() { - var out = document.getElementById("hlcode"); - out.innerHTML = pyodide.runPython('pygments.highlight(code, lexer, fmter)'); - document.location.hash = "#try"; - document.getElementById("hlcodedl").style.display = "block"; -} - -function copy_link() { - var copy_field = document.getElementById("copy_field"); - copy_field.select(); - copy_field.setSelectionRange(0, 99999); - document.execCommand("copy"); -} +copyLink.addEventListener('click', async (e) => { + e.preventDefault(); + await navigator.clipboard.writeText(e.target.href); +}); function download_code() { var filename = "highlighted.html"; - var hlcode = document.getElementById("hlcode").innerHTML; + var hlcode = document.getElementById("hlcode").innerHTML + style.outerHTML; var blob = new Blob([hlcode], {type: 'text/html'}); if (window.navigator.msSaveOrOpenBlob) { window.navigator.msSaveBlob(blob, filename); diff --git a/doc/_templates/demo.html b/doc/_templates/demo.html index 912ab3d..8e2a7c6 100644 --- a/doc/_templates/demo.html +++ b/doc/_templates/demo.html @@ -4,9 +4,6 @@ {% block extrahead %} {{ super() }} - - - {% endblock %} {% block htmltitle %}Demo{{ titlesuffix }}{% endblock %} @@ -14,43 +11,87 @@ {% block body %} {{ body }} -

Demo - Try it out!

-

The highlighting here is performed in-browser using - a WebAssembly translation of the latest Pygments master branch, courtesy of - Pyodide.

-

Your content is neither sent over the web nor stored anywhere.

+

Try out Pygments!

+ +
-

Enter code and select a language

-
-

-   + + {% for name, info, _, _ in lexers %} + + {% endfor %} -    - ·   -  

-

  -   or enter code below:

-

-

- -    

-
-
-

-

Loading Python...

+ + +

+

+ +

+

+ + +

+
+ + +
+
+ + + +
-
+
{% endblock %} {% block footer %} +
{# closes "flexwrapper" div #}