1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
5 <meta name="generator" content="Docutils 0.14: http://docutils.sourceforge.net/" />
6 <title>Benchmarks and Speed</title>
7 <meta name="author" content="Stefan Behnel" />
8 <meta content="Performance evaluation of lxml and ElementTree: fast operations, common pitfalls and optimisation hints." name="description" />
9 <meta content="Python XML parser performance, XML processing, performance comparison, lxml performance, lxml.etree, lxml.objectify, benchmarks, ElementTree" name="keywords" />
10 <link rel="stylesheet" href="style.css" type="text/css" />
11 <script type="text/javascript">
12 function trigger_menu(event) {
13 var sidemenu = document.getElementById("sidemenu");
14 var classes = sidemenu.getAttribute("class");
15 classes = (classes.indexOf(" visible") === -1) ? classes + " visible" : classes.replace(" visible", "");
16 sidemenu.setAttribute("class", classes);
17 event.preventDefault();
18 event.stopPropagation();
20 function hide_menu() {
21 var sidemenu = document.getElementById("sidemenu");
22 var classes = sidemenu.getAttribute("class");
23 if (classes.indexOf(" visible") !== -1) {
24 sidemenu.setAttribute("class", classes.replace(" visible", ""));
27 </script><meta content="width=device-width, initial-scale=1" name="viewport" /></head>
28 <body onclick="hide_menu()">
29 <div class="document" id="benchmarks-and-speed">
30 <div class="sidemenu" id="sidemenu"><div class="menutrigger" onclick="trigger_menu(event)">Menu</div><div class="menu"><ul id="lxml-section"><li><span class="section title">lxml</span><ul class="menu foreign" id="index-menu"><li class="menu title"><a href="index.html">lxml</a><ul class="submenu"><li class="menu item"><a href="index.html#introduction">Introduction</a></li><li class="menu item"><a href="index.html#support-the-project">Support the project</a></li><li class="menu item"><a href="index.html#documentation">Documentation</a></li><li class="menu item"><a href="index.html#download">Download</a></li><li class="menu item"><a href="index.html#mailing-list">Mailing list</a></li><li class="menu item"><a href="index.html#bug-tracker">Bug tracker</a></li><li class="menu item"><a href="index.html#license">License</a></li><li class="menu item"><a href="index.html#old-versions">Old Versions</a></li><li class="menu item"><a href="index.html#legal-notice-for-donations">Legal Notice for Donations</a></li></ul></li></ul><ul class="menu foreign" id="intro-menu"><li class="menu title"><a href="intro.html">Why lxml?</a><ul class="submenu"><li class="menu item"><a href="intro.html#motto">Motto</a></li><li class="menu item"><a href="intro.html#aims">Aims</a></li></ul></li></ul><ul class="menu foreign" id="installation-menu"><li class="menu title"><a href="installation.html">Installing lxml</a><ul class="submenu"><li class="menu item"><a href="installation.html#where-to-get-it">Where to get it</a></li><li class="menu item"><a href="installation.html#requirements">Requirements</a></li><li class="menu item"><a href="installation.html#installation">Installation</a></li><li class="menu item"><a href="installation.html#building-lxml-from-dev-sources">Building lxml from dev sources</a></li><li class="menu item"><a href="installation.html#using-lxml-with-python-libxml2">Using lxml with python-libxml2</a></li><li class="menu item"><a href="installation.html#source-builds-on-ms-windows">Source builds on MS Windows</a></li><li class="menu item"><a href="installation.html#source-builds-on-macos-x">Source builds on MacOS-X</a></li></ul></li></ul><ul class="menu current" id="performance-menu"><li class="menu title"><a href="performance.html">Benchmarks and Speed</a><ul class="submenu"><li class="menu item"><a href="performance.html#general-notes">General notes</a></li><li class="menu item"><a href="performance.html#how-to-read-the-timings">How to read the timings</a></li><li class="menu item"><a href="performance.html#parsing-and-serialising">Parsing and Serialising</a></li><li class="menu item"><a href="performance.html#the-elementtree-api">The ElementTree API</a></li><li class="menu item"><a href="performance.html#xpath">XPath</a></li><li class="menu item"><a href="performance.html#a-longer-example">A longer example</a></li><li class="menu item"><a href="performance.html#lxml-objectify">lxml.objectify</a></li></ul></li></ul><ul class="menu foreign" id="compatibility-menu"><li class="menu title"><a href="compatibility.html">ElementTree compatibility of lxml.etree</a></li></ul><ul class="menu foreign" id="FAQ-menu"><li class="menu title"><a href="FAQ.html">lxml FAQ - Frequently Asked Questions</a><ul class="submenu"><li class="menu item"><a href="FAQ.html#general-questions">General Questions</a></li><li class="menu item"><a href="FAQ.html#installation">Installation</a></li><li class="menu item"><a href="FAQ.html#contributing">Contributing</a></li><li class="menu item"><a href="FAQ.html#bugs">Bugs</a></li><li class="menu item"><a href="FAQ.html#id1">Threading</a></li><li class="menu item"><a href="FAQ.html#parsing-and-serialisation">Parsing and Serialisation</a></li><li class="menu item"><a href="FAQ.html#xpath-and-document-traversal">XPath and Document Traversal</a></li></ul></li></ul></li></ul><ul id="Developing with lxml-section"><li><span class="section title">Developing with lxml</span><ul class="menu foreign" id="tutorial-menu"><li class="menu title"><a href="tutorial.html">The lxml.etree Tutorial</a><ul class="submenu"><li class="menu item"><a href="tutorial.html#the-element-class">The Element class</a></li><li class="menu item"><a href="tutorial.html#the-elementtree-class">The ElementTree class</a></li><li class="menu item"><a href="tutorial.html#parsing-from-strings-and-files">Parsing from strings and files</a></li><li class="menu item"><a href="tutorial.html#namespaces">Namespaces</a></li><li class="menu item"><a href="tutorial.html#the-e-factory">The E-factory</a></li><li class="menu item"><a href="tutorial.html#elementpath">ElementPath</a></li></ul></li></ul><ul class="menu foreign" id="api index-menu"><li class="menu title"><a href="api/index.html">API reference</a></li></ul><ul class="menu foreign" id="api-menu"><li class="menu title"><a href="api.html">APIs specific to lxml.etree</a><ul class="submenu"><li class="menu item"><a href="api.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="api.html#other-element-apis">Other Element APIs</a></li><li class="menu item"><a href="api.html#trees-and-documents">Trees and Documents</a></li><li class="menu item"><a href="api.html#iteration">Iteration</a></li><li class="menu item"><a href="api.html#error-handling-on-exceptions">Error handling on exceptions</a></li><li class="menu item"><a href="api.html#error-logging">Error logging</a></li><li class="menu item"><a href="api.html#serialisation">Serialisation</a></li><li class="menu item"><a href="api.html#incremental-xml-generation">Incremental XML generation</a></li><li class="menu item"><a href="api.html#cdata">CDATA</a></li><li class="menu item"><a href="api.html#xinclude-and-elementinclude">XInclude and ElementInclude</a></li></ul></li></ul><ul class="menu foreign" id="parsing-menu"><li class="menu title"><a href="parsing.html">Parsing XML and HTML with lxml</a><ul class="submenu"><li class="menu item"><a href="parsing.html#parsers">Parsers</a></li><li class="menu item"><a href="parsing.html#the-target-parser-interface">The target parser interface</a></li><li class="menu item"><a href="parsing.html#the-feed-parser-interface">The feed parser interface</a></li><li class="menu item"><a href="parsing.html#incremental-event-parsing">Incremental event parsing</a></li><li class="menu item"><a href="parsing.html#iterparse-and-iterwalk">iterparse and iterwalk</a></li><li class="menu item"><a href="parsing.html#python-unicode-strings">Python unicode strings</a></li></ul></li></ul><ul class="menu foreign" id="validation-menu"><li class="menu title"><a href="validation.html">Validation with lxml</a><ul class="submenu"><li class="menu item"><a href="validation.html#validation-at-parse-time">Validation at parse time</a></li><li class="menu item"><a href="validation.html#id1">DTD</a></li><li class="menu item"><a href="validation.html#relaxng">RelaxNG</a></li><li class="menu item"><a href="validation.html#xmlschema">XMLSchema</a></li><li class="menu item"><a href="validation.html#id2">Schematron</a></li><li class="menu item"><a href="validation.html#id3">(Pre-ISO-Schematron)</a></li></ul></li></ul><ul class="menu foreign" id="xpathxslt-menu"><li class="menu title"><a href="xpathxslt.html">XPath and XSLT with lxml</a><ul class="submenu"><li class="menu item"><a href="xpathxslt.html#xpath">XPath</a></li><li class="menu item"><a href="xpathxslt.html#xslt">XSLT</a></li></ul></li></ul><ul class="menu foreign" id="objectify-menu"><li class="menu title"><a href="objectify.html">lxml.objectify</a><ul class="submenu"><li class="menu item"><a href="objectify.html#the-lxml-objectify-api">The lxml.objectify API</a></li><li class="menu item"><a href="objectify.html#asserting-a-schema">Asserting a Schema</a></li><li class="menu item"><a href="objectify.html#objectpath">ObjectPath</a></li><li class="menu item"><a href="objectify.html#python-data-types">Python data types</a></li><li class="menu item"><a href="objectify.html#how-data-types-are-matched">How data types are matched</a></li><li class="menu item"><a href="objectify.html#what-is-different-from-lxml-etree">What is different from lxml.etree?</a></li></ul></li></ul><ul class="menu foreign" id="lxmlhtml-menu"><li class="menu title"><a href="lxmlhtml.html">lxml.html</a><ul class="submenu"><li class="menu item"><a href="lxmlhtml.html#parsing-html">Parsing HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-element-methods">HTML Element Methods</a></li><li class="menu item"><a href="lxmlhtml.html#running-html-doctests">Running HTML doctests</a></li><li class="menu item"><a href="lxmlhtml.html#creating-html-with-the-e-factory">Creating HTML with the E-factory</a></li><li class="menu item"><a href="lxmlhtml.html#working-with-links">Working with links</a></li><li class="menu item"><a href="lxmlhtml.html#forms">Forms</a></li><li class="menu item"><a href="lxmlhtml.html#cleaning-up-html">Cleaning up HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-diff">HTML Diff</a></li><li class="menu item"><a href="lxmlhtml.html#examples">Examples</a></li></ul></li></ul><ul class="menu foreign" id="cssselect-menu"><li class="menu title"><a href="cssselect.html">lxml.cssselect</a><ul class="submenu"><li class="menu item"><a href="cssselect.html#the-cssselector-class">The CSSSelector class</a></li><li class="menu item"><a href="cssselect.html#the-cssselect-method">The cssselect method</a></li><li class="menu item"><a href="cssselect.html#supported-selectors">Supported Selectors</a></li><li class="menu item"><a href="cssselect.html#namespaces">Namespaces</a></li></ul></li></ul><ul class="menu foreign" id="elementsoup-menu"><li class="menu title"><a href="elementsoup.html">BeautifulSoup Parser</a><ul class="submenu"><li class="menu item"><a href="elementsoup.html#parsing-with-the-soupparser">Parsing with the soupparser</a></li><li class="menu item"><a href="elementsoup.html#entity-handling">Entity handling</a></li><li class="menu item"><a href="elementsoup.html#using-soupparser-as-a-fallback">Using soupparser as a fallback</a></li><li class="menu item"><a href="elementsoup.html#using-only-the-encoding-detection">Using only the encoding detection</a></li></ul></li></ul><ul class="menu foreign" id="html5parser-menu"><li class="menu title"><a href="html5parser.html">html5lib Parser</a><ul class="submenu"><li class="menu item"><a href="html5parser.html#differences-to-regular-html-parsing">Differences to regular HTML parsing</a></li><li class="menu item"><a href="html5parser.html#function-reference">Function Reference</a></li></ul></li></ul></li></ul><ul id="Extending lxml-section"><li><span class="section title">Extending lxml</span><ul class="menu foreign" id="resolvers-menu"><li class="menu title"><a href="resolvers.html">Document loading and URL resolving</a><ul class="submenu"><li class="menu item"><a href="resolvers.html#xml-catalogs">XML Catalogs</a></li><li class="menu item"><a href="resolvers.html#uri-resolvers">URI Resolvers</a></li><li class="menu item"><a href="resolvers.html#document-loading-in-context">Document loading in context</a></li><li class="menu item"><a href="resolvers.html#i-o-access-control-in-xslt">I/O access control in XSLT</a></li></ul></li></ul><ul class="menu foreign" id="extensions-menu"><li class="menu title"><a href="extensions.html">Python extensions for XPath and XSLT</a><ul class="submenu"><li class="menu item"><a href="extensions.html#xpath-extension-functions">XPath Extension functions</a></li><li class="menu item"><a href="extensions.html#xslt-extension-elements">XSLT extension elements</a></li></ul></li></ul><ul class="menu foreign" id="element classes-menu"><li class="menu title"><a href="element_classes.html">Using custom Element classes in lxml</a><ul class="submenu"><li class="menu item"><a href="element_classes.html#background-on-element-proxies">Background on Element proxies</a></li><li class="menu item"><a href="element_classes.html#element-initialization">Element initialization</a></li><li class="menu item"><a href="element_classes.html#setting-up-a-class-lookup-scheme">Setting up a class lookup scheme</a></li><li class="menu item"><a href="element_classes.html#generating-xml-with-custom-classes">Generating XML with custom classes</a></li><li class="menu item"><a href="element_classes.html#id1">Implementing namespaces</a></li></ul></li></ul><ul class="menu foreign" id="sax-menu"><li class="menu title"><a href="sax.html">Sax support</a><ul class="submenu"><li class="menu item"><a href="sax.html#building-a-tree-from-sax-events">Building a tree from SAX events</a></li><li class="menu item"><a href="sax.html#producing-sax-events-from-an-elementtree-or-element">Producing SAX events from an ElementTree or Element</a></li><li class="menu item"><a href="sax.html#interfacing-with-pulldom-minidom">Interfacing with pulldom/minidom</a></li></ul></li></ul><ul class="menu foreign" id="capi-menu"><li class="menu title"><a href="capi.html">The public C-API of lxml.etree</a><ul class="submenu"><li class="menu item"><a href="capi.html#passing-generated-trees-through-python">Passing generated trees through Python</a></li><li class="menu item"><a href="capi.html#writing-external-modules-in-cython">Writing external modules in Cython</a></li><li class="menu item"><a href="capi.html#writing-external-modules-in-c">Writing external modules in C</a></li></ul></li></ul></li></ul><ul id="Developing lxml-section"><li><span class="section title">Developing lxml</span><ul class="menu foreign" id="build-menu"><li class="menu title"><a href="build.html">How to build lxml from source</a><ul class="submenu"><li class="menu item"><a href="build.html#cython">Cython</a></li><li class="menu item"><a href="build.html#github-git-and-hg">Github, git and hg</a></li><li class="menu item"><a href="build.html#building-the-sources">Building the sources</a></li><li class="menu item"><a href="build.html#running-the-tests-and-reporting-errors">Running the tests and reporting errors</a></li><li class="menu item"><a href="build.html#building-an-egg-or-wheel">Building an egg or wheel</a></li><li class="menu item"><a href="build.html#building-lxml-on-macos-x">Building lxml on MacOS-X</a></li><li class="menu item"><a href="build.html#static-linking-on-windows">Static linking on Windows</a></li><li class="menu item"><a href="build.html#building-debian-packages-from-svn-sources">Building Debian packages from SVN sources</a></li></ul></li></ul><ul class="menu foreign" id="lxml source howto-menu"><li class="menu title"><a href="lxml-source-howto.html">How to read the source of lxml</a><ul class="submenu"><li class="menu item"><a href="lxml-source-howto.html#what-is-cython">What is Cython?</a></li><li class="menu item"><a href="lxml-source-howto.html#where-to-start">Where to start?</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="lxml-source-howto.html#python-modules">Python modules</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-objectify">lxml.objectify</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-html">lxml.html</a></li></ul></li></ul><ul class="menu foreign" id="changes 4 4 3-menu"><li class="menu title"><a href="changes-4.4.3.html">Release Changelog</a></li></ul><ul class="menu foreign" id="credits-menu"><li class="menu title"><a href="credits.html">Credits</a><ul class="submenu"><li class="menu item"><a href="credits.html#main-contributors">Main contributors</a></li><li class="menu item"><a href="credits.html#special-thanks-goes-to">Special thanks goes to:</a></li></ul></li></ul></li><li><a href="/sitemap.html">Sitemap</a></li></ul></div></div><h1 class="title">Benchmarks and Speed</h1>
31 <table class="docinfo" frame="void" rules="none">
32 <col class="docinfo-name" />
33 <col class="docinfo-content" />
35 <tr><th class="docinfo-name">Author:</th>
36 <td>Stefan Behnel</td></tr>
39 <p>lxml.etree is a very fast XML library. Most of this is due to the
40 speed of libxml2, e.g. the parser and serialiser, or the XPath engine.
41 Other areas of lxml were specifically written for high performance in
42 high-level operations, such as the tree iterators.</p>
43 <p>On the other hand, the simplicity of lxml sometimes hides internal
44 operations that are more costly than the API suggests. If you are not
45 aware of these cases, lxml may not always perform as you expect. A
46 common example in the Python world is the Python list type. New users
47 often expect it to be a linked list, while it actually is implemented
48 as an array, which results in a completely different complexity for
49 common operations.</p>
50 <p>Similarly, the tree model of libxml2 is more complex than what lxml's
51 ElementTree API projects into Python space, so some operations may
52 show unexpected performance. Rest assured that most lxml users will
53 not notice this in real life, as lxml is very fast in absolute
54 numbers. It is definitely fast enough for most applications, so lxml
55 is probably somewhere between 'fast enough' and 'the best choice' for
56 yours. Read some <a class="reference external" href="http://permalink.gmane.org/gmane.comp.python.lxml.devel/3250">messages</a> from <a class="reference external" href="http://article.gmane.org/gmane.comp.python.lxml.devel/3246">happy</a> <a class="reference external" href="http://thread.gmane.org/gmane.comp.python.lxml.devel/3244/focus=3244">users</a> to see what we mean.</p>
57 <p>This text describes where lxml.etree (abbreviated to 'lxe') excels, gives
58 hints on some performance traps and compares the overall performance to the
59 original <a class="reference external" href="http://effbot.org/zone/element-index.htm">ElementTree</a> (ET) and <a class="reference external" href="http://effbot.org/zone/celementtree.htm">cElementTree</a> (cET) libraries by Fredrik Lundh.
60 The cElementTree library is a fast C-implementation of the original
62 <div class="contents topic" id="contents">
63 <p class="topic-title first">Contents</p>
65 <li><a class="reference internal" href="#general-notes" id="id1">General notes</a></li>
66 <li><a class="reference internal" href="#how-to-read-the-timings" id="id2">How to read the timings</a></li>
67 <li><a class="reference internal" href="#parsing-and-serialising" id="id3">Parsing and Serialising</a></li>
68 <li><a class="reference internal" href="#the-elementtree-api" id="id4">The ElementTree API</a><ul>
69 <li><a class="reference internal" href="#child-access" id="id5">Child access</a></li>
70 <li><a class="reference internal" href="#element-creation" id="id6">Element creation</a></li>
71 <li><a class="reference internal" href="#merging-different-sources" id="id7">Merging different sources</a></li>
72 <li><a class="reference internal" href="#deepcopy" id="id8">deepcopy</a></li>
73 <li><a class="reference internal" href="#tree-traversal" id="id9">Tree traversal</a></li>
76 <li><a class="reference internal" href="#xpath" id="id10">XPath</a></li>
77 <li><a class="reference internal" href="#a-longer-example" id="id11">A longer example</a></li>
78 <li><a class="reference internal" href="#lxml-objectify" id="id12">lxml.objectify</a><ul>
79 <li><a class="reference internal" href="#objectpath" id="id13">ObjectPath</a></li>
80 <li><a class="reference internal" href="#caching-elements" id="id14">Caching Elements</a></li>
81 <li><a class="reference internal" href="#further-optimisations" id="id15">Further optimisations</a></li>
86 <div class="section" id="general-notes">
87 <h1>General notes</h1>
88 <p>First thing to say: there <em>is</em> an overhead involved in having a DOM-like C
89 library mimic the ElementTree API. As opposed to ElementTree, lxml has to
90 generate Python representations of tree nodes on the fly when asked for them,
91 and the internal tree structure of libxml2 results in a higher maintenance
92 overhead than the simpler top-down structure of ElementTree. What this means
93 is: the more of your code runs in Python, the less you can benefit from the
94 speed of lxml and libxml2. Note, however, that this is true for most
95 performance critical Python applications. No one would implement Fourier
96 transformations in pure Python when you can use NumPy.</p>
97 <p>The up side then is that lxml provides powerful tools like tree iterators,
98 XPath and XSLT, that can handle complex operations at the speed of C. Their
99 pythonic API in lxml makes them so flexible that most applications can easily
100 benefit from them.</p>
102 <div class="section" id="how-to-read-the-timings">
103 <h1>How to read the timings</h1>
104 <p>The statements made here are backed by the (micro-)benchmark scripts
105 <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_etree.py">bench_etree.py</a>, <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_xpath.py">bench_xpath.py</a> and <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_objectify.py">bench_objectify.py</a> that come with
106 the lxml source distribution. They are distributed under the same BSD license
107 as lxml itself, and the lxml project would like to promote them as a general
108 benchmarking suite for all ElementTree implementations. New benchmarks are
109 very easy to add as tiny test methods, so if you write a performance test for
110 a specific part of the API yourself, please consider sending it to the lxml
112 <p>The timings presented below compare lxml 3.1.1 (with libxml2 2.9.0) to the
113 latest released versions of ElementTree (with cElementTree as accelerator
114 module) in the standard library of CPython 3.3.0. They were run
115 single-threaded on a 2.9GHz 64bit double core Intel i7 machine under
116 Ubuntu Linux 12.10 (Quantal). The C libraries were compiled with the
117 same platform specific optimisation flags. The Python interpreter was
118 also manually compiled for the platform. Note that many of the following
119 ElementTree timings are therefore better than what a normal Python
120 installation with the standard library (c)ElementTree modules would yield.
121 Note also that CPython 2.7 and 3.2+ come with a newer ElementTree version,
122 so older Python installations will not perform as good for (c)ElementTree,
123 and sometimes substantially worse.</p>
124 <p>The scripts run a number of simple tests on the different libraries, using
125 different XML tree configurations: different tree sizes (T1-4), with or
126 without attributes (-/A), with or without ASCII string or unicode text
127 (-/S/U), and either against a tree or its serialised XML form (T/X). In the
128 result extracts cited below, T1 refers to a 3-level tree with many children at
129 the third level, T2 is swapped around to have many children below the root
130 element, T3 is a deep tree with few children at each level and T4 is a small
131 tree, slightly broader than deep. If repetition is involved, this usually
132 means running the benchmark in a loop over all children of the tree root,
133 otherwise, the operation is run on the root node (C/R).</p>
134 <p>As an example, the character code <tt class="docutils literal">(SATR T1)</tt> states that the benchmark was
135 running for tree T1, with plain string text (S) and attributes (A). It was
136 run against the root element (R) in the tree structure of the data (T).</p>
137 <p>Note that very small operations are repeated in integer loops to make them
138 measurable. It is therefore not always possible to compare the absolute
139 timings of, say, a single access benchmark (which usually loops) and a 'get
140 all in one step' benchmark, which already takes enough time to be measurable
141 and is therefore measured as is. An example is the index access to a single
142 child, which cannot be compared to the timings for <tt class="docutils literal">getchildren()</tt>. Take a
143 look at the concrete benchmarks in the scripts to understand how the numbers
146 <div class="section" id="parsing-and-serialising">
147 <h1>Parsing and Serialising</h1>
148 <p>Serialisation is an area where lxml excels. The reason is that it
149 executes entirely at the C level, without any interaction with Python
150 code. The results are rather impressive, especially for UTF-8, which
151 is native to libxml2. While 20 to 40 times faster than (c)ElementTree
152 1.2 (which was part of the standard library before Python 2.7/3.2),
153 lxml is still more than 10 times as fast as the much improved
154 ElementTree 1.3 in recent Python versions:</p>
155 <pre class="literal-block">
156 lxe: tostring_utf16 (S-TR T1) 7.9958 msec/pass
157 cET: tostring_utf16 (S-TR T1) 83.1358 msec/pass
159 lxe: tostring_utf16 (UATR T1) 8.3222 msec/pass
160 cET: tostring_utf16 (UATR T1) 84.4688 msec/pass
162 lxe: tostring_utf16 (S-TR T2) 8.2297 msec/pass
163 cET: tostring_utf16 (S-TR T2) 87.3415 msec/pass
165 lxe: tostring_utf8 (S-TR T2) 6.5677 msec/pass
166 cET: tostring_utf8 (S-TR T2) 76.2064 msec/pass
168 lxe: tostring_utf8 (U-TR T3) 1.1952 msec/pass
169 cET: tostring_utf8 (U-TR T3) 22.0058 msec/pass
171 <p>The difference is somewhat smaller for plain text serialisation:</p>
172 <pre class="literal-block">
173 lxe: tostring_text_ascii (S-TR T1) 2.7738 msec/pass
174 cET: tostring_text_ascii (S-TR T1) 4.7629 msec/pass
176 lxe: tostring_text_ascii (S-TR T3) 0.8273 msec/pass
177 cET: tostring_text_ascii (S-TR T3) 1.5273 msec/pass
179 lxe: tostring_text_utf16 (S-TR T1) 2.7659 msec/pass
180 cET: tostring_text_utf16 (S-TR T1) 10.5038 msec/pass
182 lxe: tostring_text_utf16 (U-TR T1) 2.8017 msec/pass
183 cET: tostring_text_utf16 (U-TR T1) 10.5207 msec/pass
185 <p>The <tt class="docutils literal">tostring()</tt> function also supports serialisation to a Python
186 unicode string object, which is currently faster in ElementTree
187 under CPython 3.3:</p>
188 <pre class="literal-block">
189 lxe: tostring_text_unicode (S-TR T1) 2.6896 msec/pass
190 cET: tostring_text_unicode (S-TR T1) 1.0056 msec/pass
192 lxe: tostring_text_unicode (U-TR T1) 2.7366 msec/pass
193 cET: tostring_text_unicode (U-TR T1) 1.0154 msec/pass
195 lxe: tostring_text_unicode (S-TR T3) 0.7997 msec/pass
196 cET: tostring_text_unicode (S-TR T3) 0.3154 msec/pass
198 lxe: tostring_text_unicode (U-TR T4) 0.0048 msec/pass
199 cET: tostring_text_unicode (U-TR T4) 0.0160 msec/pass
201 <p>For parsing, lxml.etree and cElementTree compete for the medal.
202 Depending on the input, either of the two can be faster. The (c)ET
203 libraries use a very thin layer on top of the expat parser, which is
204 known to be very fast. Here are some timings from the benchmarking
206 <pre class="literal-block">
207 lxe: parse_bytesIO (SAXR T1) 13.0246 msec/pass
208 cET: parse_bytesIO (SAXR T1) 8.2929 msec/pass
210 lxe: parse_bytesIO (S-XR T3) 1.3542 msec/pass
211 cET: parse_bytesIO (S-XR T3) 2.4023 msec/pass
213 lxe: parse_bytesIO (UAXR T3) 7.5610 msec/pass
214 cET: parse_bytesIO (UAXR T3) 11.2455 msec/pass
216 <p>And another couple of timings <a class="reference external" href="http://svn.effbot.org/public/elementtree-1.3/benchmark.py">from a benchmark</a> that Fredrik Lundh
217 <a class="reference external" href="http://effbot.org/zone/celementtree.htm#benchmarks">used to promote cElementTree</a>, comparing a number of different
218 parsers. First, parsing a 274KB XML file containing Shakespeare's
220 <pre class="literal-block">
221 xml.etree.ElementTree.parse done in 0.017 seconds
222 xml.etree.cElementTree.parse done in 0.007 seconds
223 xml.etree.cElementTree.XMLParser.feed(): 6636 nodes read in 0.007 seconds
224 lxml.etree.parse done in 0.003 seconds
225 drop_whitespace.parse done in 0.003 seconds
226 lxml.etree.XMLParser.feed(): 6636 nodes read in 0.004 seconds
227 minidom tree read in 0.080 seconds
229 <p>And a 3.4MB XML file containing the Old Testament:</p>
230 <pre class="literal-block">
231 xml.etree.ElementTree.parse done in 0.038 seconds
232 xml.etree.cElementTree.parse done in 0.030 seconds
233 xml.etree.cElementTree.XMLParser.feed(): 25317 nodes read in 0.030 seconds
234 lxml.etree.parse done in 0.016 seconds
235 drop_whitespace.parse done in 0.015 seconds
236 lxml.etree.XMLParser.feed(): 25317 nodes read in 0.022 seconds
237 minidom tree read in 0.288 seconds
239 <p>Here are the same benchmarks again, but including the memory usage
240 of the process in KB before and after parsing (using os.fork() to
241 make sure we start from a clean state each time). For the 274KB
243 <pre class="literal-block">
245 xml.etree.ElementTree.parse done in 0.017 seconds
246 Memory usage: 9432 (+2148)
247 xml.etree.cElementTree.parse done in 0.007 seconds
248 Memory usage: 9432 (+2152)
249 xml.etree.cElementTree.XMLParser.feed(): 6636 nodes read in 0.007 seconds
250 Memory usage: 9448 (+2164)
251 lxml.etree.parse done in 0.003 seconds
252 Memory usage: 11032 (+3748)
253 drop_whitespace.parse done in 0.003 seconds
254 Memory usage: 10224 (+2940)
255 lxml.etree.XMLParser.feed(): 6636 nodes read in 0.004 seconds
256 Memory usage: 11804 (+4520)
257 minidom tree read in 0.080 seconds
258 Memory usage: 12324 (+5040)
260 <p>And for the 3.4MB Old Testament XML file:</p>
261 <pre class="literal-block">
263 xml.etree.ElementTree.parse done in 0.038 seconds
264 Memory usage: 20660 (+10240)
265 xml.etree.cElementTree.parse done in 0.030 seconds
266 Memory usage: 20660 (+10240)
267 xml.etree.cElementTree.XMLParser.feed(): 25317 nodes read in 0.030 seconds
268 Memory usage: 20844 (+10424)
269 lxml.etree.parse done in 0.016 seconds
270 Memory usage: 27624 (+17204)
271 drop_whitespace.parse done in 0.015 seconds
272 Memory usage: 24468 (+14052)
273 lxml.etree.XMLParser.feed(): 25317 nodes read in 0.022 seconds
274 Memory usage: 29844 (+19424)
275 minidom tree read in 0.288 seconds
276 Memory usage: 28788 (+18368)
278 <p>As can be seen from the sizes, both lxml.etree and cElementTree are
279 rather memory friendly compared to the pure Python libraries
280 ElementTree and (especially) minidom. Comparing to older CPython
281 versions, the memory footprint of the minidom library was considerably
282 reduced in CPython 3.3, by about a factor of 4 in this case.</p>
283 <p>For plain parser performance, lxml.etree and cElementTree tend to stay
284 rather close to each other, usually within a factor of two, with
285 winners well distributed over both sides. Similar timings can be
286 observed for the <tt class="docutils literal">iterparse()</tt> function:</p>
287 <pre class="literal-block">
288 lxe: iterparse_bytesIO (SAXR T1) 17.9198 msec/pass
289 cET: iterparse_bytesIO (SAXR T1) 14.4982 msec/pass
291 lxe: iterparse_bytesIO (UAXR T3) 8.8522 msec/pass
292 cET: iterparse_bytesIO (UAXR T3) 12.9857 msec/pass
294 <p>However, if you benchmark the complete round-trip of a serialise-parse
295 cycle, the numbers will look similar to these:</p>
296 <pre class="literal-block">
297 lxe: write_utf8_parse_bytesIO (S-TR T1) 19.8867 msec/pass
298 cET: write_utf8_parse_bytesIO (S-TR T1) 80.7259 msec/pass
300 lxe: write_utf8_parse_bytesIO (UATR T2) 23.7896 msec/pass
301 cET: write_utf8_parse_bytesIO (UATR T2) 98.0766 msec/pass
303 lxe: write_utf8_parse_bytesIO (S-TR T3) 3.0684 msec/pass
304 cET: write_utf8_parse_bytesIO (S-TR T3) 24.6122 msec/pass
306 lxe: write_utf8_parse_bytesIO (SATR T4) 0.3495 msec/pass
307 cET: write_utf8_parse_bytesIO (SATR T4) 1.9610 msec/pass
309 <p>For applications that require a high parser throughput of large files,
310 and that do little to no serialization, both cET and lxml.etree are a
311 good choice. The cET library is particularly fast for iterparse
312 applications that extract small amounts of data or aggregate
313 information from large XML data sets that do not fit into memory. If
314 it comes to round-trip performance, however, lxml is multiple times
315 faster in total. So, whenever the input documents are not
316 considerably larger than the output, lxml is the clear winner.</p>
317 <p>Regarding HTML parsing, Ian Bicking has done some <a class="reference external" href="http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/">benchmarking on
318 lxml's HTML parser</a>, comparing it to a number of other famous HTML
319 parser tools for Python. lxml wins this contest by quite a length.
320 To give an idea, the numbers suggest that lxml.html can run a couple
321 of parse-serialise cycles in the time that other tools need for
322 parsing alone. The comparison even shows some very favourable results
323 regarding memory consumption.</p>
324 <p>Liza Daly has written an article that presents a couple of tweaks to
325 get the most out of lxml's parser for very large XML documents. She
326 quite favourably positions <tt class="docutils literal">lxml.etree</tt> as a tool for
327 <a class="reference external" href="http://www.ibm.com/developerworks/xml/library/x-hiperfparse/">high-performance XML parsing</a>.</p>
328 <p>Finally, <a class="reference external" href="http://www.xml.com/">xml.com</a> has a couple of publications about XML parser
329 performance. Farwick and Hafner have written two interesting articles
330 that compare the parser of libxml2 to some major Java based XML
331 parsers. One deals with <a class="reference external" href="http://www.xml.com/lpt/a/1702">event-driven parser performance</a>, the other
332 one presents <a class="reference external" href="http://www.xml.com/lpt/a/1703">benchmark results comparing DOM parsers</a>. Both
333 comparisons suggest that libxml2's parser performance is largely
334 superior to all commonly used Java parsers in almost all cases. Note
335 that the C parser benchmark results are based on <a class="reference external" href="http://xmlbench.sourceforge.net/">xmlbench</a>, which uses
336 a simpler setup for libxml2 than lxml does.</p>
338 <div class="section" id="the-elementtree-api">
339 <h1>The ElementTree API</h1>
340 <p>Since all three libraries implement the same API, their performance is
341 easy to compare in this area. A major disadvantage for lxml's
342 performance is the different tree model that underlies libxml2. It
343 allows lxml to provide parent pointers for elements and full XPath
344 support, but also increases the overhead of tree building and
345 restructuring. This can be seen from the tree setup times of the
346 benchmark (given in seconds):</p>
347 <pre class="literal-block">
348 lxe: -- S- U- -A SA UA
349 T1: 0.0299 0.0343 0.0344 0.0293 0.0345 0.0342
350 T2: 0.0368 0.0423 0.0418 0.0427 0.0474 0.0459
351 T3: 0.0088 0.0084 0.0086 0.0251 0.0258 0.0261
352 T4: 0.0002 0.0002 0.0002 0.0005 0.0006 0.0006
353 cET: -- S- U- -A SA UA
354 T1: 0.0050 0.0045 0.0093 0.0044 0.0043 0.0043
355 T2: 0.0073 0.0075 0.0074 0.0201 0.0075 0.0074
356 T3: 0.0033 0.0213 0.0032 0.0034 0.0033 0.0035
357 T4: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
359 <p>The timings are somewhat close to each other, although cET can be
360 several times faster than lxml for larger trees. One of the
361 reasons is that lxml must encode incoming string data and tag names
362 into UTF-8, and additionally discard the created Python elements
363 after their use, when they are no longer referenced. ElementTree
364 represents the tree itself through these objects, which reduces
365 the overhead in creating them.</p>
366 <div class="section" id="child-access">
367 <h2>Child access</h2>
368 <p>The same tree overhead makes operations like collecting children as in
369 <tt class="docutils literal">list(element)</tt> more costly in lxml. Where cET can quickly create
370 a shallow copy of their list of children, lxml has to create a Python
371 object for each child and collect them in a list:</p>
372 <pre class="literal-block">
373 lxe: root_list_children (--TR T1) 0.0038 msec/pass
374 cET: root_list_children (--TR T1) 0.0010 msec/pass
376 lxe: root_list_children (--TR T2) 0.0455 msec/pass
377 cET: root_list_children (--TR T2) 0.0050 msec/pass
379 <p>This handicap is also visible when accessing single children:</p>
380 <pre class="literal-block">
381 lxe: first_child (--TR T2) 0.0424 msec/pass
382 cET: first_child (--TR T2) 0.0384 msec/pass
384 lxe: last_child (--TR T1) 0.0477 msec/pass
385 cET: last_child (--TR T1) 0.0467 msec/pass
387 <p>... unless you also add the time to find a child index in a bigger
388 list. ET and cET use Python lists here, which are based on arrays.
389 The data structure used by libxml2 is a linked tree, and thus, a
390 linked list of children:</p>
391 <pre class="literal-block">
392 lxe: middle_child (--TR T1) 0.0710 msec/pass
393 cET: middle_child (--TR T1) 0.0420 msec/pass
395 lxe: middle_child (--TR T2) 1.7393 msec/pass
396 cET: middle_child (--TR T2) 0.0396 msec/pass
399 <div class="section" id="element-creation">
400 <h2>Element creation</h2>
401 <p>As opposed to ET, libxml2 has a notion of documents that each element must be
402 in. This results in a major performance difference for creating independent
403 Elements that end up in independently created documents:</p>
404 <pre class="literal-block">
405 lxe: create_elements (--TC T2) 1.0045 msec/pass
406 cET: create_elements (--TC T2) 0.0753 msec/pass
408 <p>Therefore, it is always preferable to create Elements for the document they
409 are supposed to end up in, either as SubElements of an Element or using the
410 explicit <tt class="docutils literal">Element.makeelement()</tt> call:</p>
411 <pre class="literal-block">
412 lxe: makeelement (--TC T2) 1.0586 msec/pass
413 cET: makeelement (--TC T2) 0.1483 msec/pass
415 lxe: create_subelements (--TC T2) 0.8826 msec/pass
416 cET: create_subelements (--TC T2) 0.0827 msec/pass
418 <p>So, if the main performance bottleneck of an application is creating large XML
419 trees in memory through calls to Element and SubElement, cET is the best
420 choice. Note, however, that the serialisation performance may even out this
421 advantage, especially for smaller trees and trees with many attributes.</p>
423 <div class="section" id="merging-different-sources">
424 <h2>Merging different sources</h2>
425 <p>A critical action for lxml is moving elements between document contexts. It
426 requires lxml to do recursive adaptations throughout the moved tree structure.</p>
427 <p>The following benchmark appends all root children of the second tree to the
428 root of the first tree:</p>
429 <pre class="literal-block">
430 lxe: append_from_document (--TR T1,T2) 1.0812 msec/pass
431 cET: append_from_document (--TR T1,T2) 0.1104 msec/pass
433 lxe: append_from_document (--TR T3,T4) 0.0155 msec/pass
434 cET: append_from_document (--TR T3,T4) 0.0060 msec/pass
436 <p>Although these are fairly small numbers compared to parsing, this easily shows
437 the different performance classes for lxml and (c)ET. Where the latter do not
438 have to care about parent pointers and tree structures, lxml has to deep
439 traverse the appended tree. The performance difference therefore increases
440 with the size of the tree that is moved.</p>
441 <p>This difference is not always as visible, but applies to most parts of the
442 API, like inserting newly created elements:</p>
443 <pre class="literal-block">
444 lxe: insert_from_document (--TR T1,T2) 3.9763 msec/pass
445 cET: insert_from_document (--TR T1,T2) 0.1459 msec/pass
447 <p>or replacing the child slice by a newly created element:</p>
448 <pre class="literal-block">
449 lxe: replace_children_element (--TC T1) 0.0749 msec/pass
450 cET: replace_children_element (--TC T1) 0.0081 msec/pass
452 <p>as opposed to replacing the slice with an existing element from the
454 <pre class="literal-block">
455 lxe: replace_children (--TC T1) 0.0052 msec/pass
456 cET: replace_children (--TC T1) 0.0036 msec/pass
458 <p>While these numbers are too small to provide a major performance
459 impact in practice, you should keep this difference in mind when you
460 merge very large trees. Note that Elements have a <tt class="docutils literal">makeelement()</tt>
461 method that allows to create an Element within the same document,
462 thus avoiding the merge overhead when inserting it into that tree.</p>
464 <div class="section" id="deepcopy">
466 <p>Deep copying a tree is fast in lxml:</p>
467 <pre class="literal-block">
468 lxe: deepcopy_all (--TR T1) 3.1650 msec/pass
469 cET: deepcopy_all (--TR T1) 53.9973 msec/pass
471 lxe: deepcopy_all (-ATR T2) 3.7365 msec/pass
472 cET: deepcopy_all (-ATR T2) 61.6267 msec/pass
474 lxe: deepcopy_all (S-TR T3) 0.7913 msec/pass
475 cET: deepcopy_all (S-TR T3) 13.6220 msec/pass
477 <p>So, for example, if you have a database-like scenario where you parse in a
478 large tree and then search and copy independent subtrees from it for further
479 processing, lxml is by far the best choice here.</p>
481 <div class="section" id="tree-traversal">
482 <h2>Tree traversal</h2>
483 <p>Another important area in XML processing is iteration for tree
484 traversal. If your algorithms can benefit from step-by-step
485 traversal of the XML tree and especially if few elements are of
486 interest or the target element tag name is known, the <tt class="docutils literal">.iter()</tt>
487 method is a good choice:</p>
488 <pre class="literal-block">
489 lxe: iter_all (--TR T1) 1.0529 msec/pass
490 cET: iter_all (--TR T1) 0.2635 msec/pass
492 lxe: iter_islice (--TR T2) 0.0110 msec/pass
493 cET: iter_islice (--TR T2) 0.0050 msec/pass
495 lxe: iter_tag (--TR T2) 0.0079 msec/pass
496 cET: iter_tag (--TR T2) 0.0112 msec/pass
498 lxe: iter_tag_all (--TR T2) 0.1822 msec/pass
499 cET: iter_tag_all (--TR T2) 0.5343 msec/pass
501 <p>This translates directly into similar timings for <tt class="docutils literal">Element.findall()</tt>:</p>
502 <pre class="literal-block">
503 lxe: findall (--TR T2) 1.7176 msec/pass
504 cET: findall (--TR T2) 0.9973 msec/pass
506 lxe: findall (--TR T3) 0.3967 msec/pass
507 cET: findall (--TR T3) 0.2525 msec/pass
509 lxe: findall_tag (--TR T2) 0.2258 msec/pass
510 cET: findall_tag (--TR T2) 0.5770 msec/pass
512 lxe: findall_tag (--TR T3) 0.1085 msec/pass
513 cET: findall_tag (--TR T3) 0.1919 msec/pass
515 <p>Note that all three libraries currently use the same Python
516 implementation for <tt class="docutils literal">.findall()</tt>, except for their native tree
517 iterator (<tt class="docutils literal">element.iter()</tt>). In general, lxml is very fast
518 for iteration, but loses ground against cET when many Elements
519 are found and need to be instantiated. So, the more selective
520 your search is, the faster lxml will run.</p>
523 <div class="section" id="xpath">
525 <p>The following timings are based on the benchmark script <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_xpath.py">bench_xpath.py</a>.</p>
526 <p>This part of lxml does not have an equivalent in ElementTree. However, lxml
527 provides more than one way of accessing it and you should take care which part
528 of the lxml API you use. The most straight forward way is to call the
529 <tt class="docutils literal">xpath()</tt> method on an Element or ElementTree:</p>
530 <pre class="literal-block">
531 lxe: xpath_method (--TC T1) 0.3982 msec/pass
532 lxe: xpath_method (--TC T2) 7.8895 msec/pass
533 lxe: xpath_method (--TC T3) 0.0477 msec/pass
534 lxe: xpath_method (--TC T4) 0.3982 msec/pass
536 <p>This is well suited for testing and when the XPath expressions are as diverse
537 as the trees they are called on. However, if you have a single XPath
538 expression that you want to apply to a larger number of different elements,
539 the <tt class="docutils literal">XPath</tt> class is the most efficient way to do it:</p>
540 <pre class="literal-block">
541 lxe: xpath_class (--TC T1) 0.0713 msec/pass
542 lxe: xpath_class (--TC T2) 1.1325 msec/pass
543 lxe: xpath_class (--TC T3) 0.0215 msec/pass
544 lxe: xpath_class (--TC T4) 0.0722 msec/pass
546 <p>Note that this still allows you to use variables in the expression, so you can
547 parse it once and then adapt it through variables at call time. In other
548 cases, where you have a fixed Element or ElementTree and want to run different
549 expressions on it, you should consider the <tt class="docutils literal">XPathEvaluator</tt>:</p>
550 <pre class="literal-block">
551 lxe: xpath_element (--TR T1) 0.1101 msec/pass
552 lxe: xpath_element (--TR T2) 2.0473 msec/pass
553 lxe: xpath_element (--TR T3) 0.0267 msec/pass
554 lxe: xpath_element (--TR T4) 0.1087 msec/pass
556 <p>While it looks slightly slower, creating an XPath object for each of the
557 expressions generates a much higher overhead here:</p>
558 <pre class="literal-block">
559 lxe: xpath_class_repeat (--TC T1 ) 0.3884 msec/pass
560 lxe: xpath_class_repeat (--TC T2 ) 7.6182 msec/pass
561 lxe: xpath_class_repeat (--TC T3 ) 0.0465 msec/pass
562 lxe: xpath_class_repeat (--TC T4 ) 0.3877 msec/pass
564 <p>Note that tree iteration can be substantially faster than XPath if
565 your code short-circuits after the first couple of elements were
566 found. The XPath engine will always return the complete result set,
567 regardless of how much of it will actually be used.</p>
568 <p>Here is an example where only the first matching element is being
569 searched, a case for which XPath has syntax support as well:</p>
570 <pre class="literal-block">
571 lxe: find_single (--TR T2) 0.0184 msec/pass
572 cET: find_single (--TR T2) 0.0052 msec/pass
574 lxe: iter_single (--TR T2) 0.0024 msec/pass
575 cET: iter_single (--TR T2) 0.0007 msec/pass
577 lxe: xpath_single (--TR T2) 0.0033 msec/pass
579 <p>When looking for the first two elements out of many, the numbers
580 explode for XPath, as restricting the result subset requires a
581 more complex expression:</p>
582 <pre class="literal-block">
583 lxe: iterfind_two (--TR T2) 0.0184 msec/pass
584 cET: iterfind_two (--TR T2) 0.0062 msec/pass
586 lxe: iter_two (--TR T2) 0.0029 msec/pass
587 cET: iter_two (--TR T2) 0.0017 msec/pass
589 lxe: xpath_two (--TR T2) 0.2768 msec/pass
592 <div class="section" id="a-longer-example">
593 <h1>A longer example</h1>
594 <p>... based on lxml 1.3.</p>
595 <p>A while ago, Uche Ogbuji posted a <a class="reference external" href="http://www.onlamp.com/pub/wlg/6291">benchmark proposal</a> that would
596 read in a 3MB XML version of the <a class="reference external" href="http://www.ibiblio.org/bosak/xml/eg/religion.2.00.xml.zip">Old Testament</a> of the Bible and
597 look for the word <em>begat</em> in all verses. Apparently, it is contained
598 in 120 out of almost 24000 verses. This is easy to implement in
599 ElementTree using <tt class="docutils literal">findall()</tt>. However, the fastest and most memory
600 friendly way to do this is obviously <tt class="docutils literal">iterparse()</tt>, as most of the
601 data is not of any interest.</p>
602 <p>Now, Uche's original proposal was more or less the following:</p>
603 <div class="syntax"><pre><span></span><span class="k">def</span> <span class="nf">bench_ET</span><span class="p">():</span>
604 <span class="n">tree</span> <span class="o">=</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">)</span>
605 <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
606 <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s2">"//v"</span><span class="p">):</span>
607 <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
608 <span class="k">if</span> <span class="s1">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
609 <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
610 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
612 <p>which takes about one second on my machine today. The faster <tt class="docutils literal">iterparse()</tt>
613 variant looks like this:</p>
614 <div class="syntax"><pre><span></span><span class="k">def</span> <span class="nf">bench_ET_iterparse</span><span class="p">():</span>
615 <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
616 <span class="k">for</span> <span class="n">event</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">iterparse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">):</span>
617 <span class="k">if</span> <span class="n">v</span><span class="o">.</span><span class="n">tag</span> <span class="o">==</span> <span class="s1">'v'</span><span class="p">:</span>
618 <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
619 <span class="k">if</span> <span class="s1">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
620 <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
621 <span class="n">v</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
622 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
624 <p>The improvement is about 10%. At the time I first tried (early 2006), lxml
625 didn't have <tt class="docutils literal">iterparse()</tt> support, but the <tt class="docutils literal">findall()</tt> variant was already
626 faster than ElementTree. This changes immediately when you switch to
627 cElementTree. The latter only needs 0.17 seconds to do the trick today and
628 only some impressive 0.10 seconds when running the iterparse version. And
629 even back then, it was quite a bit faster than what lxml could achieve.</p>
630 <p>Since then, lxml has matured a lot and has gotten much faster. The iterparse
631 variant now runs in 0.14 seconds, and if you remove the <tt class="docutils literal">v.clear()</tt>, it is
632 even a little faster (which isn't the case for cElementTree).</p>
633 <p>One of the many great tools in lxml is XPath, a Swiss army knife for finding
634 things in XML documents. It is possible to move the whole thing to a pure
635 XPath implementation, which looks like this:</p>
636 <div class="syntax"><pre><span></span><span class="k">def</span> <span class="nf">bench_lxml_xpath_all</span><span class="p">():</span>
637 <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">)</span>
638 <span class="n">result</span> <span class="o">=</span> <span class="n">tree</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">"//v[contains(., 'begat')]/text()"</span><span class="p">)</span>
639 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
641 <p>This runs in about 0.13 seconds and is about the shortest possible
642 implementation (in lines of Python code) that I could come up with. Now, this
643 is already a rather complex XPath expression compared to the simple "//v"
644 ElementPath expression we started with. Since this is also valid XPath, let's
645 try this instead:</p>
646 <div class="syntax"><pre><span></span><span class="k">def</span> <span class="nf">bench_lxml_xpath</span><span class="p">():</span>
647 <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">)</span>
648 <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
649 <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">"//v"</span><span class="p">):</span>
650 <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
651 <span class="k">if</span> <span class="s1">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
652 <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
653 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
655 <p>This gets us down to 0.12 seconds, thus showing that a generic XPath
656 evaluation engine cannot always compete with a simpler, tailored solution.
657 However, since this is not much different from the original findall variant,
658 we can remove the complexity of the XPath call completely and just go with
659 what we had in the beginning. Under lxml, this runs in the same 0.12 seconds.</p>
660 <p>But there is one thing left to try. We can replace the simple ElementPath
661 expression with a native tree iterator:</p>
662 <div class="syntax"><pre><span></span><span class="k">def</span> <span class="nf">bench_lxml_getiterator</span><span class="p">():</span>
663 <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">)</span>
664 <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
665 <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">getiterator</span><span class="p">(</span><span class="s2">"v"</span><span class="p">):</span>
666 <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
667 <span class="k">if</span> <span class="s1">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
668 <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
669 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
671 <p>This implements the same thing, just without the overhead of parsing and
672 evaluating a path expression. And this makes it another bit faster, down to
673 0.11 seconds. For comparison, cElementTree runs this version in 0.17 seconds.</p>
674 <p>So, what have we learned?</p>
676 <li>Python code is not slow. The pure XPath solution was not even as fast as
677 the first shot Python implementation. In general, a few more lines in
678 Python make things more readable, which is much more important than the last
679 5% of performance.</li>
680 <li>It's important to know the available options - and it's worth starting with
681 the most simple one. In this case, a programmer would then probably have
682 started with <tt class="docutils literal"><span class="pre">getiterator("v")</span></tt> or <tt class="docutils literal">iterparse()</tt>. Either of them would
683 already have been the most efficient, depending on which library is used.</li>
684 <li>It's important to know your tool. lxml and cElementTree are both very fast
685 libraries, but they do not have the same performance characteristics. The
686 fastest solution in one library can be comparatively slow in the other. If
687 you optimise, optimise for the specific target platform.</li>
688 <li>It's not always worth optimising. After all that hassle we got from 0.12
689 seconds for the initial implementation to 0.11 seconds. Switching over to
690 cElementTree and writing an <tt class="docutils literal">iterparse()</tt> based version would have given
691 us 0.10 seconds - not a big difference for 3MB of XML.</li>
692 <li>Take care what operation is really dominating in your use case. If we split
693 up the operations, we can see that lxml is slightly slower than cElementTree
694 on <tt class="docutils literal">parse()</tt> (both about 0.06 seconds), but more visibly slower on
695 <tt class="docutils literal">iterparse()</tt>: 0.07 versus 0.10 seconds. However, tree iteration in lxml
696 is incredibly fast, so it can be better to parse the whole tree and then
697 iterate over it rather than using <tt class="docutils literal">iterparse()</tt> to do both in one step.
698 Or, you can just wait for the lxml developers to optimise iterparse in one
699 of the next releases...</li>
702 <div class="section" id="lxml-objectify">
703 <h1>lxml.objectify</h1>
704 <p>The following timings are based on the benchmark script <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_objectify.py">bench_objectify.py</a>.</p>
705 <p>Objectify is a data-binding API for XML based on lxml.etree, that was added in
706 version 1.1. It uses standard Python attribute access to traverse the XML
707 tree. It also features ObjectPath, a fast path language based on the same
709 <p>Just like lxml.etree, lxml.objectify creates Python representations of
710 elements on the fly. To save memory, the normal Python garbage collection
711 mechanisms will discard them when their last reference is gone. In cases
712 where deeply nested elements are frequently accessed through the objectify
713 API, the create-discard cycles can become a bottleneck, as elements have to be
714 instantiated over and over again.</p>
715 <div class="section" id="objectpath">
717 <p>ObjectPath can be used to speed up the access to elements that are deep in the
718 tree. It avoids step-by-step Python element instantiations along the path,
719 which can substantially improve the access time:</p>
720 <pre class="literal-block">
721 lxe: attribute (--TR T1) 4.1828 msec/pass
722 lxe: attribute (--TR T2) 17.3802 msec/pass
723 lxe: attribute (--TR T4) 3.8657 msec/pass
725 lxe: objectpath (--TR T1) 0.9289 msec/pass
726 lxe: objectpath (--TR T2) 13.3109 msec/pass
727 lxe: objectpath (--TR T4) 0.9289 msec/pass
729 lxe: attributes_deep (--TR T1) 6.2900 msec/pass
730 lxe: attributes_deep (--TR T2) 20.4713 msec/pass
731 lxe: attributes_deep (--TR T4) 6.1679 msec/pass
733 lxe: objectpath_deep (--TR T1) 1.3049 msec/pass
734 lxe: objectpath_deep (--TR T2) 14.0815 msec/pass
735 lxe: objectpath_deep (--TR T4) 1.3051 msec/pass
737 <p>Note, however, that parsing ObjectPath expressions is not for free either, so
738 this is most effective for frequently accessing the same element.</p>
740 <div class="section" id="caching-elements">
741 <h2>Caching Elements</h2>
742 <p>A way to improve the normal attribute access time is static instantiation of
743 the Python objects, thus trading memory for speed. Just create a cache
744 dictionary and run:</p>
745 <div class="syntax"><pre><span></span><span class="n">cache</span><span class="p">[</span><span class="n">root</span><span class="p">]</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">root</span><span class="o">.</span><span class="n">iter</span><span class="p">())</span>
747 <p>after parsing and:</p>
748 <div class="syntax"><pre><span></span><span class="k">del</span> <span class="n">cache</span><span class="p">[</span><span class="n">root</span><span class="p">]</span>
750 <p>when you are done with the tree. This will keep the Python element
751 representations of all elements alive and thus avoid the overhead of repeated
752 Python object creation. You can also consider using filters or generator
753 expressions to be more selective. By choosing the right trees (or even
754 subtrees and elements) to cache, you can trade memory usage against access
756 <pre class="literal-block">
757 lxe: attribute_cached (--TR T1) 3.1357 msec/pass
758 lxe: attribute_cached (--TR T2) 15.8911 msec/pass
759 lxe: attribute_cached (--TR T4) 2.9194 msec/pass
761 lxe: attributes_deep_cached (--TR T1) 3.8984 msec/pass
762 lxe: attributes_deep_cached (--TR T2) 16.8300 msec/pass
763 lxe: attributes_deep_cached (--TR T4) 3.6936 msec/pass
765 lxe: objectpath_deep_cached (--TR T1) 0.7496 msec/pass
766 lxe: objectpath_deep_cached (--TR T2) 12.3763 msec/pass
767 lxe: objectpath_deep_cached (--TR T4) 0.7427 msec/pass
769 <p>Things to note: you cannot currently use <tt class="docutils literal">weakref.WeakKeyDictionary</tt> objects
770 for this as lxml's element objects do not support weak references (which are
771 costly in terms of memory). Also note that new element objects that you add
772 to these trees will not turn up in the cache automatically and will therefore
773 still be garbage collected when all their Python references are gone, so this
774 is most effective for largely immutable trees. You should consider using a
775 set instead of a list in this case and add new elements by hand.</p>
777 <div class="section" id="further-optimisations">
778 <h2>Further optimisations</h2>
779 <p>Here are some more things to try if optimisation is required:</p>
781 <li>A lot of time is usually spent in tree traversal to find the addressed
782 elements in the tree. If you often work in subtrees, do what you would also
783 do with deep Python objects: assign the parent of the subtree to a variable
784 or pass it into functions instead of starting at the root. This allows
785 accessing its descendants more directly.</li>
786 <li>Try assigning data values directly to attributes instead of passing them
787 through DataElement.</li>
788 <li>If you use custom data types that are costly to parse, try running
789 <tt class="docutils literal">objectify.annotate()</tt> over read-only trees to speed up the attribute type
790 inference on read access.</li>
792 <p>Note that none of these measures is guaranteed to speed up your application.
793 As usual, you should prefer readable code over premature optimisations and
794 profile your expected use cases before bothering to apply optimisations at
800 <hr class="footer" />
801 Generated on: 2020-01-29.
projects / platform / upstream / python-lxml.git / blob