1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
5 <meta name="generator" content="Docutils 0.8.1: http://docutils.sourceforge.net/" />
6 <title>Benchmarks and Speed</title>
7 <meta name="author" content="Stefan Behnel" />
8 <meta content="Performance evaluation of lxml and ElementTree: fast operations, common pitfalls and optimisation hints." name="description" />
9 <meta content="Python XML parser performance, XML processing, performance comparison, lxml performance, lxml.etree, lxml.objectify, benchmarks, ElementTree" name="keywords" />
10 <link rel="stylesheet" href="style.css" type="text/css" />
13 <div class="document" id="benchmarks-and-speed">
14 <div class="sidemenu"><ul id="lxml-section"><li><span class="section title">lxml</span><ul class="menu foreign" id="index-menu"><li class="menu title"><a href="index.html">lxml</a><ul class="submenu"><li class="menu item"><a href="index.html#introduction">Introduction</a></li><li class="menu item"><a href="index.html#support-the-project">Support the project</a></li><li class="menu item"><a href="index.html#documentation">Documentation</a></li><li class="menu item"><a href="index.html#download">Download</a></li><li class="menu item"><a href="index.html#mailing-list">Mailing list</a></li><li class="menu item"><a href="index.html#bug-tracker">Bug tracker</a></li><li class="menu item"><a href="index.html#license">License</a></li><li class="menu item"><a href="index.html#old-versions">Old Versions</a></li><li class="menu item"><a href="index.html#legal-notice-for-donations">Legal Notice for Donations</a></li></ul></li></ul><ul class="menu foreign" id="intro-menu"><li class="menu title"><a href="intro.html">Why lxml?</a><ul class="submenu"><li class="menu item"><a href="intro.html#motto">Motto</a></li><li class="menu item"><a href="intro.html#aims">Aims</a></li></ul></li></ul><ul class="menu foreign" id="installation-menu"><li class="menu title"><a href="installation.html">Installing lxml</a><ul class="submenu"><li class="menu item"><a href="installation.html#requirements">Requirements</a></li><li class="menu item"><a href="installation.html#installation">Installation</a></li><li class="menu item"><a href="installation.html#building-lxml-from-sources">Building lxml from sources</a></li><li class="menu item"><a href="installation.html#using-lxml-with-python-libxml2">Using lxml with python-libxml2</a></li><li class="menu item"><a href="installation.html#ms-windows">MS Windows</a></li><li class="menu item"><a href="installation.html#macos-x">MacOS-X</a></li></ul></li></ul><ul class="menu current" id="performance-menu"><li class="menu title"><a href="performance.html">Benchmarks and Speed</a><ul class="submenu"><li class="menu item"><a href="performance.html#general-notes">General notes</a></li><li class="menu item"><a href="performance.html#how-to-read-the-timings">How to read the timings</a></li><li class="menu item"><a href="performance.html#parsing-and-serialising">Parsing and Serialising</a></li><li class="menu item"><a href="performance.html#the-elementtree-api">The ElementTree API</a></li><li class="menu item"><a href="performance.html#xpath">XPath</a></li><li class="menu item"><a href="performance.html#a-longer-example">A longer example</a></li><li class="menu item"><a href="performance.html#lxml-objectify">lxml.objectify</a></li></ul></li></ul><ul class="menu foreign" id="compatibility-menu"><li class="menu title"><a href="compatibility.html">ElementTree compatibility of lxml.etree</a></li></ul><ul class="menu foreign" id="FAQ-menu"><li class="menu title"><a href="FAQ.html">lxml FAQ - Frequently Asked Questions</a><ul class="submenu"><li class="menu item"><a href="FAQ.html#general-questions">General Questions</a></li><li class="menu item"><a href="FAQ.html#installation">Installation</a></li><li class="menu item"><a href="FAQ.html#contributing">Contributing</a></li><li class="menu item"><a href="FAQ.html#bugs">Bugs</a></li><li class="menu item"><a href="FAQ.html#id1">Threading</a></li><li class="menu item"><a href="FAQ.html#parsing-and-serialisation">Parsing and Serialisation</a></li><li class="menu item"><a href="FAQ.html#xpath-and-document-traversal">XPath and Document Traversal</a></li></ul></li></ul></li></ul><ul id="Developing with lxml-section"><li><span class="section title">Developing with lxml</span><ul class="menu foreign" id="tutorial-menu"><li class="menu title"><a href="tutorial.html">The lxml.etree Tutorial</a><ul class="submenu"><li class="menu item"><a href="tutorial.html#the-element-class">The Element class</a></li><li class="menu item"><a href="tutorial.html#the-elementtree-class">The ElementTree class</a></li><li class="menu item"><a href="tutorial.html#parsing-from-strings-and-files">Parsing from strings and files</a></li><li class="menu item"><a href="tutorial.html#namespaces">Namespaces</a></li><li class="menu item"><a href="tutorial.html#the-e-factory">The E-factory</a></li><li class="menu item"><a href="tutorial.html#elementpath">ElementPath</a></li></ul></li></ul><ul class="menu foreign" id="api index-menu"><li class="menu title"><a href="api/index.html">API reference</a></li></ul><ul class="menu foreign" id="api-menu"><li class="menu title"><a href="api.html">APIs specific to lxml.etree</a><ul class="submenu"><li class="menu item"><a href="api.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="api.html#other-element-apis">Other Element APIs</a></li><li class="menu item"><a href="api.html#trees-and-documents">Trees and Documents</a></li><li class="menu item"><a href="api.html#iteration">Iteration</a></li><li class="menu item"><a href="api.html#error-handling-on-exceptions">Error handling on exceptions</a></li><li class="menu item"><a href="api.html#error-logging">Error logging</a></li><li class="menu item"><a href="api.html#serialisation">Serialisation</a></li><li class="menu item"><a href="api.html#cdata">CDATA</a></li><li class="menu item"><a href="api.html#xinclude-and-elementinclude">XInclude and ElementInclude</a></li><li class="menu item"><a href="api.html#write-c14n-on-elementtree">write_c14n on ElementTree</a></li></ul></li></ul><ul class="menu foreign" id="parsing-menu"><li class="menu title"><a href="parsing.html">Parsing XML and HTML with lxml</a><ul class="submenu"><li class="menu item"><a href="parsing.html#parsers">Parsers</a></li><li class="menu item"><a href="parsing.html#the-target-parser-interface">The target parser interface</a></li><li class="menu item"><a href="parsing.html#the-feed-parser-interface">The feed parser interface</a></li><li class="menu item"><a href="parsing.html#iterparse-and-iterwalk">iterparse and iterwalk</a></li><li class="menu item"><a href="parsing.html#python-unicode-strings">Python unicode strings</a></li></ul></li></ul><ul class="menu foreign" id="validation-menu"><li class="menu title"><a href="validation.html">Validation with lxml</a><ul class="submenu"><li class="menu item"><a href="validation.html#validation-at-parse-time">Validation at parse time</a></li><li class="menu item"><a href="validation.html#id1">DTD</a></li><li class="menu item"><a href="validation.html#relaxng">RelaxNG</a></li><li class="menu item"><a href="validation.html#xmlschema">XMLSchema</a></li><li class="menu item"><a href="validation.html#id2">Schematron</a></li><li class="menu item"><a href="validation.html#id3">(Pre-ISO-Schematron)</a></li></ul></li></ul><ul class="menu foreign" id="xpathxslt-menu"><li class="menu title"><a href="xpathxslt.html">XPath and XSLT with lxml</a><ul class="submenu"><li class="menu item"><a href="xpathxslt.html#xpath">XPath</a></li><li class="menu item"><a href="xpathxslt.html#xslt">XSLT</a></li></ul></li></ul><ul class="menu foreign" id="objectify-menu"><li class="menu title"><a href="objectify.html">lxml.objectify</a><ul class="submenu"><li class="menu item"><a href="objectify.html#the-lxml-objectify-api">The lxml.objectify API</a></li><li class="menu item"><a href="objectify.html#asserting-a-schema">Asserting a Schema</a></li><li class="menu item"><a href="objectify.html#objectpath">ObjectPath</a></li><li class="menu item"><a href="objectify.html#python-data-types">Python data types</a></li><li class="menu item"><a href="objectify.html#how-data-types-are-matched">How data types are matched</a></li><li class="menu item"><a href="objectify.html#what-is-different-from-lxml-etree">What is different from lxml.etree?</a></li></ul></li></ul><ul class="menu foreign" id="lxmlhtml-menu"><li class="menu title"><a href="lxmlhtml.html">lxml.html</a><ul class="submenu"><li class="menu item"><a href="lxmlhtml.html#parsing-html">Parsing HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-element-methods">HTML Element Methods</a></li><li class="menu item"><a href="lxmlhtml.html#running-html-doctests">Running HTML doctests</a></li><li class="menu item"><a href="lxmlhtml.html#creating-html-with-the-e-factory">Creating HTML with the E-factory</a></li><li class="menu item"><a href="lxmlhtml.html#working-with-links">Working with links</a></li><li class="menu item"><a href="lxmlhtml.html#forms">Forms</a></li><li class="menu item"><a href="lxmlhtml.html#cleaning-up-html">Cleaning up HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-diff">HTML Diff</a></li><li class="menu item"><a href="lxmlhtml.html#examples">Examples</a></li></ul></li></ul><ul class="menu foreign" id="cssselect-menu"><li class="menu title"><a href="cssselect.html">lxml.cssselect</a><ul class="submenu"><li class="menu item"><a href="cssselect.html#the-cssselector-class">The CSSSelector class</a></li><li class="menu item"><a href="cssselect.html#css-selectors">CSS Selectors</a></li><li class="menu item"><a href="cssselect.html#namespaces">Namespaces</a></li><li class="menu item"><a href="cssselect.html#limitations">Limitations</a></li></ul></li></ul><ul class="menu foreign" id="elementsoup-menu"><li class="menu title"><a href="elementsoup.html">BeautifulSoup Parser</a><ul class="submenu"><li class="menu item"><a href="elementsoup.html#parsing-with-the-soupparser">Parsing with the soupparser</a></li><li class="menu item"><a href="elementsoup.html#entity-handling">Entity handling</a></li><li class="menu item"><a href="elementsoup.html#using-soupparser-as-a-fallback">Using soupparser as a fallback</a></li><li class="menu item"><a href="elementsoup.html#using-only-the-encoding-detection">Using only the encoding detection</a></li></ul></li></ul><ul class="menu foreign" id="html5parser-menu"><li class="menu title"><a href="html5parser.html">html5lib Parser</a><ul class="submenu"><li class="menu item"><a href="html5parser.html#differences-to-regular-html-parsing">Differences to regular HTML parsing</a></li><li class="menu item"><a href="html5parser.html#function-reference">Function Reference</a></li></ul></li></ul></li></ul><ul id="Extending lxml-section"><li><span class="section title">Extending lxml</span><ul class="menu foreign" id="resolvers-menu"><li class="menu title"><a href="resolvers.html">Document loading and URL resolving</a><ul class="submenu"><li class="menu item"><a href="resolvers.html#xml-catalogs">XML Catalogs</a></li><li class="menu item"><a href="resolvers.html#uri-resolvers">URI Resolvers</a></li><li class="menu item"><a href="resolvers.html#document-loading-in-context">Document loading in context</a></li><li class="menu item"><a href="resolvers.html#i-o-access-control-in-xslt">I/O access control in XSLT</a></li></ul></li></ul><ul class="menu foreign" id="extensions-menu"><li class="menu title"><a href="extensions.html">Python extensions for XPath and XSLT</a><ul class="submenu"><li class="menu item"><a href="extensions.html#xpath-extension-functions">XPath Extension functions</a></li><li class="menu item"><a href="extensions.html#xslt-extension-elements">XSLT extension elements</a></li></ul></li></ul><ul class="menu foreign" id="element classes-menu"><li class="menu title"><a href="element_classes.html">Using custom Element classes in lxml</a><ul class="submenu"><li class="menu item"><a href="element_classes.html#background-on-element-proxies">Background on Element proxies</a></li><li class="menu item"><a href="element_classes.html#element-initialization">Element initialization</a></li><li class="menu item"><a href="element_classes.html#setting-up-a-class-lookup-scheme">Setting up a class lookup scheme</a></li><li class="menu item"><a href="element_classes.html#generating-xml-with-custom-classes">Generating XML with custom classes</a></li><li class="menu item"><a href="element_classes.html#id1">Implementing namespaces</a></li></ul></li></ul><ul class="menu foreign" id="sax-menu"><li class="menu title"><a href="sax.html">Sax support</a><ul class="submenu"><li class="menu item"><a href="sax.html#building-a-tree-from-sax-events">Building a tree from SAX events</a></li><li class="menu item"><a href="sax.html#producing-sax-events-from-an-elementtree-or-element">Producing SAX events from an ElementTree or Element</a></li><li class="menu item"><a href="sax.html#interfacing-with-pulldom-minidom">Interfacing with pulldom/minidom</a></li></ul></li></ul><ul class="menu foreign" id="capi-menu"><li class="menu title"><a href="capi.html">The public C-API of lxml.etree</a><ul class="submenu"><li class="menu item"><a href="capi.html#writing-external-modules-in-cython">Writing external modules in Cython</a></li><li class="menu item"><a href="capi.html#writing-external-modules-in-c">Writing external modules in C</a></li></ul></li></ul></li></ul><ul id="Developing lxml-section"><li><span class="section title">Developing lxml</span><ul class="menu foreign" id="build-menu"><li class="menu title"><a href="build.html">How to build lxml from source</a><ul class="submenu"><li class="menu item"><a href="build.html#cython">Cython</a></li><li class="menu item"><a href="build.html#github-git-and-hg">Github, git and hg</a></li><li class="menu item"><a href="build.html#id2">Setuptools</a></li><li class="menu item"><a href="build.html#running-the-tests-and-reporting-errors">Running the tests and reporting errors</a></li><li class="menu item"><a href="build.html#building-an-egg">Building an egg</a></li><li class="menu item"><a href="build.html#building-lxml-on-macos-x">Building lxml on MacOS-X</a></li><li class="menu item"><a href="build.html#static-linking-on-windows">Static linking on Windows</a></li><li class="menu item"><a href="build.html#building-debian-packages-from-svn-sources">Building Debian packages from SVN sources</a></li></ul></li></ul><ul class="menu foreign" id="lxml source howto-menu"><li class="menu title"><a href="lxml-source-howto.html">How to read the source of lxml</a><ul class="submenu"><li class="menu item"><a href="lxml-source-howto.html#what-is-cython">What is Cython?</a></li><li class="menu item"><a href="lxml-source-howto.html#where-to-start">Where to start?</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="lxml-source-howto.html#python-modules">Python modules</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-objectify">lxml.objectify</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-html">lxml.html</a></li></ul></li></ul><ul class="menu foreign" id="changes 2 3 5-menu"><li class="menu title"><a href="changes-2.3.5.html">Release Changelog</a></li></ul><ul class="menu foreign" id="credits-menu"><li class="menu title"><a href="credits.html">Credits</a><ul class="submenu"><li class="menu item"><a href="credits.html#main-contributors">Main contributors</a></li><li class="menu item"><a href="credits.html#special-thanks-goes-to">Special thanks goes to:</a></li></ul></li></ul></li><li><a href="http://lxml.de/sitemap.html">Sitemap</a></li></ul></div><h1 class="title">Benchmarks and Speed</h1>
15 <table class="docinfo" frame="void" rules="none">
16 <col class="docinfo-name" />
17 <col class="docinfo-content" />
19 <tr><th class="docinfo-name">Author:</th>
20 <td>Stefan Behnel</td></tr>
23 <p>lxml.etree is a very fast XML library. Most of this is due to the
24 speed of libxml2, e.g. the parser and serialiser, or the XPath engine.
25 Other areas of lxml were specifically written for high performance in
26 high-level operations, such as the tree iterators.</p>
27 <p>On the other hand, the simplicity of lxml sometimes hides internal
28 operations that are more costly than the API suggests. If you are not
29 aware of these cases, lxml may not always perform as you expect. A
30 common example in the Python world is the Python list type. New users
31 often expect it to be a linked list, while it actually is implemented
32 as an array, which results in a completely different complexity for
33 common operations.</p>
34 <p>Similarly, the tree model of libxml2 is more complex than what lxml's
35 ElementTree API projects into Python space, so some operations may
36 show unexpected performance. Rest assured that most lxml users will
37 not notice this in real life, as lxml is very fast in absolute
38 numbers. It is definitely fast enough for most applications, so lxml
39 is probably somewhere between 'fast enough' and 'the best choice' for
40 yours. Read some <a class="reference external" href="http://permalink.gmane.org/gmane.comp.python.lxml.devel/3250">messages</a> from <a class="reference external" href="http://article.gmane.org/gmane.comp.python.lxml.devel/3246">happy</a> <a class="reference external" href="http://thread.gmane.org/gmane.comp.python.lxml.devel/3244/focus=3244">users</a> to see what we mean.</p>
41 <p>This text describes where lxml.etree (abbreviated to 'lxe') excels, gives
42 hints on some performance traps and compares the overall performance to the
43 original <a class="reference external" href="http://effbot.org/zone/element-index.htm">ElementTree</a> (ET) and <a class="reference external" href="http://effbot.org/zone/celementtree.htm">cElementTree</a> (cET) libraries by Fredrik Lundh.
44 The cElementTree library is a fast C-implementation of the original
46 <div class="contents topic" id="contents">
47 <p class="topic-title first">Contents</p>
49 <li><a class="reference internal" href="#general-notes" id="id1">General notes</a></li>
50 <li><a class="reference internal" href="#how-to-read-the-timings" id="id2">How to read the timings</a></li>
51 <li><a class="reference internal" href="#parsing-and-serialising" id="id3">Parsing and Serialising</a></li>
52 <li><a class="reference internal" href="#the-elementtree-api" id="id4">The ElementTree API</a><ul>
53 <li><a class="reference internal" href="#child-access" id="id5">Child access</a></li>
54 <li><a class="reference internal" href="#element-creation" id="id6">Element creation</a></li>
55 <li><a class="reference internal" href="#merging-different-sources" id="id7">Merging different sources</a></li>
56 <li><a class="reference internal" href="#deepcopy" id="id8">deepcopy</a></li>
57 <li><a class="reference internal" href="#tree-traversal" id="id9">Tree traversal</a></li>
60 <li><a class="reference internal" href="#xpath" id="id10">XPath</a></li>
61 <li><a class="reference internal" href="#a-longer-example" id="id11">A longer example</a></li>
62 <li><a class="reference internal" href="#lxml-objectify" id="id12">lxml.objectify</a><ul>
63 <li><a class="reference internal" href="#objectpath" id="id13">ObjectPath</a></li>
64 <li><a class="reference internal" href="#caching-elements" id="id14">Caching Elements</a></li>
65 <li><a class="reference internal" href="#further-optimisations" id="id15">Further optimisations</a></li>
70 <div class="section" id="general-notes">
71 <h1>General notes</h1>
72 <p>First thing to say: there <em>is</em> an overhead involved in having a DOM-like C
73 library mimic the ElementTree API. As opposed to ElementTree, lxml has to
74 generate Python representations of tree nodes on the fly when asked for them,
75 and the internal tree structure of libxml2 results in a higher maintenance
76 overhead than the simpler top-down structure of ElementTree. What this means
77 is: the more of your code runs in Python, the less you can benefit from the
78 speed of lxml and libxml2. Note, however, that this is true for most
79 performance critical Python applications. No one would implement fourier
80 transformations in pure Python when you can use NumPy.</p>
81 <p>The up side then is that lxml provides powerful tools like tree iterators,
82 XPath and XSLT, that can handle complex operations at the speed of C. Their
83 pythonic API in lxml makes them so flexible that most applications can easily
84 benefit from them.</p>
86 <div class="section" id="how-to-read-the-timings">
87 <h1>How to read the timings</h1>
88 <p>The statements made here are backed by the (micro-)benchmark scripts
89 <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_etree.py">bench_etree.py</a>, <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_xpath.py">bench_xpath.py</a> and <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_objectify.py">bench_objectify.py</a> that come with
90 the lxml source distribution. They are distributed under the same BSD license
91 as lxml itself, and the lxml project would like to promote them as a general
92 benchmarking suite for all ElementTree implementations. New benchmarks are
93 very easy to add as tiny test methods, so if you write a performance test for
94 a specific part of the API yourself, please consider sending it to the lxml
96 <p>The timings cited below compare lxml 2.3 (with libxml2 2.7.6) to the
97 latest developer versions of ElementTree (1.3beta2) and cElementTree
98 (1.0.6a3). They were run single-threaded on a 2.5GHz 64bit Intel Core
99 Duo machine under Ubuntu Linux 9.10 (Karmic). The C libraries were
100 compiled with the same platform specific optimisation flags. The
101 Python interpreter (2.6.4) was also manually compiled for the
102 platform. Note that many of the following ElementTree timings are
103 therefore better then what a normal Python installation with the
104 standard library (c)ElementTree modules would yield.</p>
105 <p>The scripts run a number of simple tests on the different libraries, using
106 different XML tree configurations: different tree sizes (T1-4), with or
107 without attributes (-/A), with or without ASCII string or unicode text
108 (-/S/U), and either against a tree or its serialised XML form (T/X). In the
109 result extracts cited below, T1 refers to a 3-level tree with many children at
110 the third level, T2 is swapped around to have many children below the root
111 element, T3 is a deep tree with few children at each level and T4 is a small
112 tree, slightly broader than deep. If repetition is involved, this usually
113 means running the benchmark in a loop over all children of the tree root,
114 otherwise, the operation is run on the root node (C/R).</p>
115 <p>As an example, the character code <tt class="docutils literal">(SATR T1)</tt> states that the benchmark was
116 running for tree T1, with plain string text (S) and attributes (A). It was
117 run against the root element (R) in the tree structure of the data (T).</p>
118 <p>Note that very small operations are repeated in integer loops to make them
119 measurable. It is therefore not always possible to compare the absolute
120 timings of, say, a single access benchmark (which usually loops) and a 'get
121 all in one step' benchmark, which already takes enough time to be measurable
122 and is therefore measured as is. An example is the index access to a single
123 child, which cannot be compared to the timings for <tt class="docutils literal">getchildren()</tt>. Take a
124 look at the concrete benchmarks in the scripts to understand how the numbers
127 <div class="section" id="parsing-and-serialising">
128 <h1>Parsing and Serialising</h1>
129 <p>Serialisation is an area where lxml excels. The reason is that it
130 executes entirely at the C level, without any interaction with Python
131 code. The results are rather impressive, especially for UTF-8, which
132 is native to libxml2. While 20 to 40 times faster than (c)ElementTree
133 1.2 (which is part of the standard library since Python 2.5), lxml is
134 still more than 7 times as fast as the much improved ElementTree 1.3:</p>
135 <pre class="literal-block">
136 lxe: tostring_utf16 (S-TR T1) 9.8219 msec/pass
137 cET: tostring_utf16 (S-TR T1) 88.7740 msec/pass
138 ET : tostring_utf16 (S-TR T1) 99.6690 msec/pass
140 lxe: tostring_utf16 (UATR T1) 10.3750 msec/pass
141 cET: tostring_utf16 (UATR T1) 90.7581 msec/pass
142 ET : tostring_utf16 (UATR T1) 102.3569 msec/pass
144 lxe: tostring_utf16 (S-TR T2) 10.2711 msec/pass
145 cET: tostring_utf16 (S-TR T2) 93.5340 msec/pass
146 ET : tostring_utf16 (S-TR T2) 105.8500 msec/pass
148 lxe: tostring_utf8 (S-TR T2) 7.1261 msec/pass
149 cET: tostring_utf8 (S-TR T2) 93.4091 msec/pass
150 ET : tostring_utf8 (S-TR T2) 105.5419 msec/pass
152 lxe: tostring_utf8 (U-TR T3) 1.4591 msec/pass
153 cET: tostring_utf8 (U-TR T3) 29.6180 msec/pass
154 ET : tostring_utf8 (U-TR T3) 31.9080 msec/pass
156 <p>The same applies to plain text serialisation. Note that the
157 cElementTree version in the standard library does not currently
158 support this, as it is a new feature in ET 1.3 and lxml.etree 2.0:</p>
159 <pre class="literal-block">
160 lxe: tostring_text_ascii (S-TR T1) 1.9400 msec/pass
161 cET: tostring_text_ascii (S-TR T1) 41.6231 msec/pass
162 ET : tostring_text_ascii (S-TR T1) 52.7501 msec/pass
164 lxe: tostring_text_ascii (S-TR T3) 0.5331 msec/pass
165 cET: tostring_text_ascii (S-TR T3) 12.9712 msec/pass
166 ET : tostring_text_ascii (S-TR T3) 15.3620 msec/pass
168 lxe: tostring_text_utf16 (S-TR T1) 3.2430 msec/pass
169 cET: tostring_text_utf16 (S-TR T1) 41.9259 msec/pass
170 ET : tostring_text_utf16 (S-TR T1) 53.4091 msec/pass
172 lxe: tostring_text_utf16 (U-TR T1) 3.6838 msec/pass
173 cET: tostring_text_utf16 (U-TR T1) 38.7859 msec/pass
174 ET : tostring_text_utf16 (U-TR T1) 50.8440 msec/pass
176 <p>Unlike ElementTree, the <tt class="docutils literal">tostring()</tt> function in lxml also supports
177 serialisation to a Python unicode string object:</p>
178 <pre class="literal-block">
179 lxe: tostring_text_unicode (S-TR T1) 2.4869 msec/pass
180 lxe: tostring_text_unicode (U-TR T1) 3.0370 msec/pass
181 lxe: tostring_text_unicode (S-TR T3) 0.6518 msec/pass
182 lxe: tostring_text_unicode (U-TR T3) 0.7300 msec/pass
184 <p>For parsing, lxml.etree and cElementTree compete for the medal.
185 Depending on the input, either of the two can be faster. The (c)ET
186 libraries use a very thin layer on top of the expat parser, which is
187 known to be very fast. Here are some timings from the benchmarking
189 <pre class="literal-block">
190 lxe: parse_stringIO (SAXR T1) 19.9990 msec/pass
191 cET: parse_stringIO (SAXR T1) 8.4970 msec/pass
192 ET : parse_stringIO (SAXR T1) 183.9781 msec/pass
194 lxe: parse_stringIO (S-XR T3) 2.0790 msec/pass
195 cET: parse_stringIO (S-XR T3) 2.7430 msec/pass
196 ET : parse_stringIO (S-XR T3) 47.4229 msec/pass
198 lxe: parse_stringIO (UAXR T3) 11.1630 msec/pass
199 cET: parse_stringIO (UAXR T3) 15.0940 msec/pass
200 ET : parse_stringIO (UAXR T3) 92.6890 msec/pass
202 <p>And another couple of timings <a class="reference external" href="http://svn.effbot.org/public/elementtree-1.3/benchmark.py">from a benchmark</a> that Fredrik Lundh
203 <a class="reference external" href="http://effbot.org/zone/celementtree.htm#benchmarks">used to promote cElementTree</a>, comparing a number of different
204 parsers. First, parsing a 274KB XML file containing Shakespeare's
206 <pre class="literal-block">
207 lxml.etree.parse done in 0.005 seconds
208 cElementTree.parse done in 0.012 seconds
209 elementtree.ElementTree.parse done in 0.136 seconds
210 elementtree.XMLTreeBuilder: 6636 nodes read in 0.243 seconds
211 elementtree.SimpleXMLTreeBuilder: 6636 nodes read in 0.314 seconds
212 elementtree.SgmlopXMLTreeBuilder: 6636 nodes read in 0.104 seconds
213 minidom tree read in 0.137 seconds
215 <p>And a 3.4MB XML file containing the Old Testament:</p>
216 <pre class="literal-block">
217 lxml.etree.parse done in 0.031 seconds
218 cElementTree.parse done in 0.039 seconds
219 elementtree.ElementTree.parse done in 0.537 seconds
220 elementtree.XMLTreeBuilder: 25317 nodes read in 0.577 seconds
221 elementtree.SimpleXMLTreeBuilder: 25317 nodes read in 1.265 seconds
222 elementtree.SgmlopXMLTreeBuilder: 25317 nodes read in 0.331 seconds
223 minidom tree read in 0.643 seconds
225 <p>Here are the same benchmarks run under an early alpha of CPython 3.3,
226 but on a different machine, which makes the absolute numbers
227 uncomparable. This time, however, we include the memory usage of the
228 process in KB before and after parsing (using os.fork() to make sure
229 we start from a clean state each time). For the 274KB hamlet.xml
231 <pre class="literal-block">
232 Memory usage at start: 7288
233 xml.etree.ElementTree.parse done in 0.104 seconds
234 Memory usage: 14252 (+6964)
235 xml.etree.cElementTree.parse done in 0.016 seconds
236 Memory usage: 9748 (+2460)
237 lxml.etree.parse done in 0.017 seconds
238 Memory usage: 11040 (+3752)
239 lxml.etree[remove_blank_space].parse done in 0.015 seconds
240 Memory usage: 10088 (+2800)
241 minidom tree read in 0.152 seconds
242 Memory usage: 30376 (+23088)
244 <p>And for the 3.4MB Old Testament XML file:</p>
245 <pre class="literal-block">
246 Memory usage at start: 20456
247 xml.etree.ElementTree.parse done in 0.419 seconds
248 Memory usage: 46112 (+25656)
249 xml.etree.cElementTree.parse done in 0.054 seconds
250 Memory usage: 32644 (+12188)
251 lxml.etree.parse done in 0.041 seconds
252 Memory usage: 37508 (+17052)
253 lxml.etree[remove_blank_space].parse done in 0.037 seconds
254 Memory usage: 34356 (+13900)
255 minidom tree read in 0.671 seconds
256 Memory usage: 110448 (+89992)
258 <p>As can be seen from the sizes, both lxml.etree and cElementTree are
259 rather memory friendly compared to the pure Python libraries
260 ElementTree and (especially) minidom. And the timings speak for
261 themselves anyway.</p>
262 <p>For plain parser performance, lxml.etree and cElementTree tend to stay
263 rather close to each other, usually within a factor of two, with
264 winners well distributed over both sides. Similar timings can be
265 observed for the <tt class="docutils literal">iterparse()</tt> function:</p>
266 <pre class="literal-block">
267 lxe: iterparse_stringIO (SAXR T1) 24.8621 msec/pass
268 cET: iterparse_stringIO (SAXR T1) 17.3280 msec/pass
269 ET : iterparse_stringIO (SAXR T1) 199.1270 msec/pass
271 lxe: iterparse_stringIO (UAXR T3) 12.3630 msec/pass
272 cET: iterparse_stringIO (UAXR T3) 17.5190 msec/pass
273 ET : iterparse_stringIO (UAXR T3) 95.8610 msec/pass
275 <p>However, if you benchmark the complete round-trip of a serialise-parse
276 cycle, the numbers will look similar to these:</p>
277 <pre class="literal-block">
278 lxe: write_utf8_parse_stringIO (S-TR T1) 27.5791 msec/pass
279 cET: write_utf8_parse_stringIO (S-TR T1) 158.9060 msec/pass
280 ET : write_utf8_parse_stringIO (S-TR T1) 347.8320 msec/pass
282 lxe: write_utf8_parse_stringIO (UATR T2) 34.4141 msec/pass
283 cET: write_utf8_parse_stringIO (UATR T2) 187.7041 msec/pass
284 ET : write_utf8_parse_stringIO (UATR T2) 388.9449 msec/pass
286 lxe: write_utf8_parse_stringIO (S-TR T3) 3.7861 msec/pass
287 cET: write_utf8_parse_stringIO (S-TR T3) 52.4600 msec/pass
288 ET : write_utf8_parse_stringIO (S-TR T3) 101.4550 msec/pass
290 lxe: write_utf8_parse_stringIO (SATR T4) 0.5522 msec/pass
291 cET: write_utf8_parse_stringIO (SATR T4) 3.8941 msec/pass
292 ET : write_utf8_parse_stringIO (SATR T4) 5.9431 msec/pass
294 <p>For applications that require a high parser throughput of large files,
295 and that do little to no serialization, both cET and lxml.etree are a
296 good choice. The cET library is particularly fast for iterparse
297 applications that extract small amounts of data or aggregate
298 information from large XML data sets that do not fit into memory. If
299 it comes to round-trip performance, however, lxml is multiple times
300 faster in total. So, whenever the input documents are not
301 considerably larger than the output, lxml is the clear winner.</p>
302 <p>Regarding HTML parsing, Ian Bicking has done some <a class="reference external" href="http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/">benchmarking on
303 lxml's HTML parser</a>, comparing it to a number of other famous HTML
304 parser tools for Python. lxml wins this contest by quite a length.
305 To give an idea, the numbers suggest that lxml.html can run a couple
306 of parse-serialise cycles in the time that other tools need for
307 parsing alone. The comparison even shows some very favourable results
308 regarding memory consumption.</p>
309 <p>Liza Daly has written an article that presents a couple of tweaks to
310 get the most out of lxml's parser for very large XML documents. She
311 quite favourably positions <tt class="docutils literal">lxml.etree</tt> as a tool for
312 <a class="reference external" href="http://www.ibm.com/developerworks/xml/library/x-hiperfparse/">high-performance XML parsing</a>.</p>
313 <p>Finally, <a class="reference external" href="http://www.xml.com/">xml.com</a> has a couple of publications about XML parser
314 performance. Farwick and Hafner have written two interesting articles
315 that compare the parser of libxml2 to some major Java based XML
316 parsers. One deals with <a class="reference external" href="http://www.xml.com/lpt/a/1702">event-driven parser performance</a>, the other
317 one presents <a class="reference external" href="http://www.xml.com/lpt/a/1703">benchmark results comparing DOM parsers</a>. Both
318 comparisons suggest that libxml2's parser performance is largely
319 superiour to all commonly used Java parsers in almost all cases. Note
320 that the C parser benchmark results are based on <a class="reference external" href="http://xmlbench.sourceforge.net/">xmlbench</a>, which uses
321 a simpler setup for libxml2 than lxml does.</p>
323 <div class="section" id="the-elementtree-api">
324 <h1>The ElementTree API</h1>
325 <p>Since all three libraries implement the same API, their performance is
326 easy to compare in this area. A major disadvantage for lxml's
327 performance is the different tree model that underlies libxml2. It
328 allows lxml to provide parent pointers for elements and full XPath
329 support, but also increases the overhead of tree building and
330 restructuring. This can be seen from the tree setup times of the
331 benchmark (given in seconds):</p>
332 <pre class="literal-block">
333 lxe: -- S- U- -A SA UA
334 T1: 0.0407 0.0470 0.0506 0.0396 0.0464 0.0504
335 T2: 0.0480 0.0557 0.0584 0.0520 0.0608 0.0627
336 T3: 0.0118 0.0132 0.0136 0.0319 0.0322 0.0319
337 T4: 0.0002 0.0002 0.0002 0.0006 0.0006 0.0006
338 cET: -- S- U- -A SA UA
339 T1: 0.0045 0.0043 0.0043 0.0045 0.0043 0.0043
340 T2: 0.0068 0.0069 0.0066 0.0078 0.0070 0.0069
341 T3: 0.0040 0.0040 0.0040 0.0050 0.0052 0.0067
342 T4: 0.0000 0.0000 0.0000 0.0001 0.0001 0.0001
343 ET : -- S- U- -A SA UA
344 T1: 0.0479 0.1051 0.1279 0.0487 0.1597 0.0484
345 T2: 0.1995 0.0553 0.2297 0.2550 0.0550 0.2881
346 T3: 0.0177 0.0169 0.0174 0.0185 0.2895 0.0189
347 T4: 0.0003 0.0002 0.0003 0.0003 0.0014 0.0003
349 <p>While lxml is still a lot faster than ET in most cases, cET can be
350 several times faster than lxml here. One of the reasons is that lxml
351 must encode incoming string data and tag names into UTF-8, and
352 additionally discard the created Python elements after their use, when
353 they are no longer referenced. ET and cET represent the tree itself
354 through these objects, which reduces the overhead in creating them.</p>
355 <div class="section" id="child-access">
356 <h2>Child access</h2>
357 <p>The same reason makes operations like collecting children as in
358 <tt class="docutils literal">list(element)</tt> more costly in lxml. Where ET and cET can quickly
359 create a shallow copy of their list of children, lxml has to create a
360 Python object for each child and collect them in a list:</p>
361 <pre class="literal-block">
362 lxe: root_list_children (--TR T1) 0.0079 msec/pass
363 cET: root_list_children (--TR T1) 0.0029 msec/pass
364 ET : root_list_children (--TR T1) 0.0100 msec/pass
366 lxe: root_list_children (--TR T2) 0.0849 msec/pass
367 cET: root_list_children (--TR T2) 0.0110 msec/pass
368 ET : root_list_children (--TR T2) 0.1481 msec/pass
370 <p>This handicap is also visible when accessing single children:</p>
371 <pre class="literal-block">
372 lxe: first_child (--TR T2) 0.0699 msec/pass
373 cET: first_child (--TR T2) 0.0608 msec/pass
374 ET : first_child (--TR T2) 0.3419 msec/pass
376 lxe: last_child (--TR T1) 0.0710 msec/pass
377 cET: last_child (--TR T1) 0.0648 msec/pass
378 ET : last_child (--TR T1) 0.3309 msec/pass
380 <p>... unless you also add the time to find a child index in a bigger
381 list. ET and cET use Python lists here, which are based on arrays.
382 The data structure used by libxml2 is a linked tree, and thus, a
383 linked list of children:</p>
384 <pre class="literal-block">
385 lxe: middle_child (--TR T1) 0.0989 msec/pass
386 cET: middle_child (--TR T1) 0.0598 msec/pass
387 ET : middle_child (--TR T1) 0.3390 msec/pass
389 lxe: middle_child (--TR T2) 2.7599 msec/pass
390 cET: middle_child (--TR T2) 0.0620 msec/pass
391 ET : middle_child (--TR T2) 0.3610 msec/pass
394 <div class="section" id="element-creation">
395 <h2>Element creation</h2>
396 <p>As opposed to ET, libxml2 has a notion of documents that each element must be
397 in. This results in a major performance difference for creating independent
398 Elements that end up in independently created documents:</p>
399 <pre class="literal-block">
400 lxe: create_elements (--TC T2) 1.1640 msec/pass
401 cET: create_elements (--TC T2) 0.0808 msec/pass
402 ET : create_elements (--TC T2) 0.5801 msec/pass
404 <p>Therefore, it is always preferable to create Elements for the document they
405 are supposed to end up in, either as SubElements of an Element or using the
406 explicit <tt class="docutils literal">Element.makeelement()</tt> call:</p>
407 <pre class="literal-block">
408 lxe: makeelement (--TC T2) 1.2751 msec/pass
409 cET: makeelement (--TC T2) 0.1469 msec/pass
410 ET : makeelement (--TC T2) 0.7451 msec/pass
412 lxe: create_subelements (--TC T2) 1.1470 msec/pass
413 cET: create_subelements (--TC T2) 0.1080 msec/pass
414 ET : create_subelements (--TC T2) 1.4369 msec/pass
416 <p>So, if the main performance bottleneck of an application is creating large XML
417 trees in memory through calls to Element and SubElement, cET is the best
418 choice. Note, however, that the serialisation performance may even out this
419 advantage, especially for smaller trees and trees with many attributes.</p>
421 <div class="section" id="merging-different-sources">
422 <h2>Merging different sources</h2>
423 <p>A critical action for lxml is moving elements between document contexts. It
424 requires lxml to do recursive adaptations throughout the moved tree structure.</p>
425 <p>The following benchmark appends all root children of the second tree to the
426 root of the first tree:</p>
427 <pre class="literal-block">
428 lxe: append_from_document (--TR T1,T2) 2.0740 msec/pass
429 cET: append_from_document (--TR T1,T2) 0.1271 msec/pass
430 ET : append_from_document (--TR T1,T2) 0.4020 msec/pass
432 lxe: append_from_document (--TR T3,T4) 0.0229 msec/pass
433 cET: append_from_document (--TR T3,T4) 0.0088 msec/pass
434 ET : append_from_document (--TR T3,T4) 0.0291 msec/pass
436 <p>Although these are fairly small numbers compared to parsing, this easily shows
437 the different performance classes for lxml and (c)ET. Where the latter do not
438 have to care about parent pointers and tree structures, lxml has to deep
439 traverse the appended tree. The performance difference therefore increases
440 with the size of the tree that is moved.</p>
441 <p>This difference is not always as visible, but applies to most parts of the
442 API, like inserting newly created elements:</p>
443 <pre class="literal-block">
444 lxe: insert_from_document (--TR T1,T2) 7.2598 msec/pass
445 cET: insert_from_document (--TR T1,T2) 0.1578 msec/pass
446 ET : insert_from_document (--TR T1,T2) 0.5150 msec/pass
448 <p>or replacing the child slice by a newly created element:</p>
449 <pre class="literal-block">
450 lxe: replace_children_element (--TC T1) 0.1149 msec/pass
451 cET: replace_children_element (--TC T1) 0.0110 msec/pass
452 ET : replace_children_element (--TC T1) 0.0558 msec/pass
454 <p>as opposed to replacing the slice with an existing element from the
456 <pre class="literal-block">
457 lxe: replace_children (--TC T1) 0.0091 msec/pass
458 cET: replace_children (--TC T1) 0.0060 msec/pass
459 ET : replace_children (--TC T1) 0.0188 msec/pass
461 <p>While these numbers are too small to provide a major performance
462 impact in practice, you should keep this difference in mind when you
463 merge very large trees.</p>
465 <div class="section" id="deepcopy">
467 <p>Deep copying a tree is fast in lxml:</p>
468 <pre class="literal-block">
469 lxe: deepcopy_all (--TR T1) 5.0900 msec/pass
470 cET: deepcopy_all (--TR T1) 57.9181 msec/pass
471 ET : deepcopy_all (--TR T1) 499.1000 msec/pass
473 lxe: deepcopy_all (-ATR T2) 6.3980 msec/pass
474 cET: deepcopy_all (-ATR T2) 65.6390 msec/pass
475 ET : deepcopy_all (-ATR T2) 526.5379 msec/pass
477 lxe: deepcopy_all (S-TR T3) 1.4491 msec/pass
478 cET: deepcopy_all (S-TR T3) 14.7018 msec/pass
479 ET : deepcopy_all (S-TR T3) 123.5120 msec/pass
481 <p>So, for example, if you have a database-like scenario where you parse in a
482 large tree and then search and copy independent subtrees from it for further
483 processing, lxml is by far the best choice here.</p>
485 <div class="section" id="tree-traversal">
486 <h2>Tree traversal</h2>
487 <p>Another area where lxml is very fast is iteration for tree traversal. If your
488 algorithms can benefit from step-by-step traversal of the XML tree and
489 especially if few elements are of interest or the target element tag name is
490 known, lxml is a good choice:</p>
491 <pre class="literal-block">
492 lxe: getiterator_all (--TR T1) 1.6890 msec/pass
493 cET: getiterator_all (--TR T1) 23.8621 msec/pass
494 ET : getiterator_all (--TR T1) 11.1070 msec/pass
496 lxe: getiterator_islice (--TR T2) 0.0188 msec/pass
497 cET: getiterator_islice (--TR T2) 0.1841 msec/pass
498 ET : getiterator_islice (--TR T2) 11.7059 msec/pass
500 lxe: getiterator_tag (--TR T2) 0.0119 msec/pass
501 cET: getiterator_tag (--TR T2) 0.3560 msec/pass
502 ET : getiterator_tag (--TR T2) 10.6668 msec/pass
504 lxe: getiterator_tag_all (--TR T2) 0.2429 msec/pass
505 cET: getiterator_tag_all (--TR T2) 20.3710 msec/pass
506 ET : getiterator_tag_all (--TR T2) 10.6280 msec/pass
508 <p>This translates directly into similar timings for <tt class="docutils literal">Element.findall()</tt>:</p>
509 <pre class="literal-block">
510 lxe: findall (--TR T2) 2.4588 msec/pass
511 cET: findall (--TR T2) 24.1358 msec/pass
512 ET : findall (--TR T2) 13.0949 msec/pass
514 lxe: findall (--TR T3) 0.5939 msec/pass
515 cET: findall (--TR T3) 6.9802 msec/pass
516 ET : findall (--TR T3) 3.8991 msec/pass
518 lxe: findall_tag (--TR T2) 0.2789 msec/pass
519 cET: findall_tag (--TR T2) 20.5719 msec/pass
520 ET : findall_tag (--TR T2) 10.8678 msec/pass
522 lxe: findall_tag (--TR T3) 0.1638 msec/pass
523 cET: findall_tag (--TR T3) 5.0790 msec/pass
524 ET : findall_tag (--TR T3) 2.5120 msec/pass
526 <p>Note that all three libraries currently use the same Python
527 implementation for <tt class="docutils literal">.findall()</tt>, except for their native tree
528 iterator (<tt class="docutils literal">element.iter()</tt>).</p>
531 <div class="section" id="xpath">
533 <p>The following timings are based on the benchmark script <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_xpath.py">bench_xpath.py</a>.</p>
534 <p>This part of lxml does not have an equivalent in ElementTree. However, lxml
535 provides more than one way of accessing it and you should take care which part
536 of the lxml API you use. The most straight forward way is to call the
537 <tt class="docutils literal">xpath()</tt> method on an Element or ElementTree:</p>
538 <pre class="literal-block">
539 lxe: xpath_method (--TC T1) 0.7598 msec/pass
540 lxe: xpath_method (--TC T2) 12.6798 msec/pass
541 lxe: xpath_method (--TC T3) 0.0758 msec/pass
542 lxe: xpath_method (--TC T4) 0.6182 msec/pass
544 <p>This is well suited for testing and when the XPath expressions are as diverse
545 as the trees they are called on. However, if you have a single XPath
546 expression that you want to apply to a larger number of different elements,
547 the <tt class="docutils literal">XPath</tt> class is the most efficient way to do it:</p>
548 <pre class="literal-block">
549 lxe: xpath_class (--TC T1) 0.2189 msec/pass
550 lxe: xpath_class (--TC T2) 1.4110 msec/pass
551 lxe: xpath_class (--TC T3) 0.0319 msec/pass
552 lxe: xpath_class (--TC T4) 0.0880 msec/pass
554 <p>Note that this still allows you to use variables in the expression, so you can
555 parse it once and then adapt it through variables at call time. In other
556 cases, where you have a fixed Element or ElementTree and want to run different
557 expressions on it, you should consider the <tt class="docutils literal">XPathEvaluator</tt>:</p>
558 <pre class="literal-block">
559 lxe: xpath_element (--TR T1) 0.1669 msec/pass
560 lxe: xpath_element (--TR T2) 6.9060 msec/pass
561 lxe: xpath_element (--TR T3) 0.0451 msec/pass
562 lxe: xpath_element (--TR T4) 0.1681 msec/pass
564 <p>While it looks slightly slower, creating an XPath object for each of the
565 expressions generates a much higher overhead here:</p>
566 <pre class="literal-block">
567 lxe: xpath_class_repeat (--TC T1) 0.7451 msec/pass
568 lxe: xpath_class_repeat (--TC T2) 12.2290 msec/pass
569 lxe: xpath_class_repeat (--TC T3) 0.0730 msec/pass
570 lxe: xpath_class_repeat (--TC T4) 0.5970 msec/pass
573 <div class="section" id="a-longer-example">
574 <h1>A longer example</h1>
575 <p>... based on lxml 1.3.</p>
576 <p>A while ago, Uche Ogbuji posted a <a class="reference external" href="http://www.onlamp.com/pub/wlg/6291">benchmark proposal</a> that would
577 read in a 3MB XML version of the <a class="reference external" href="http://www.ibiblio.org/bosak/xml/eg/religion.2.00.xml.zip">Old Testament</a> of the Bible and
578 look for the word <em>begat</em> in all verses. Apparently, it is contained
579 in 120 out of almost 24000 verses. This is easy to implement in
580 ElementTree using <tt class="docutils literal">findall()</tt>. However, the fastest and most memory
581 friendly way to do this is obviously <tt class="docutils literal">iterparse()</tt>, as most of the
582 data is not of any interest.</p>
583 <p>Now, Uche's original proposal was more or less the following:</p>
584 <div class="syntax"><pre><span class="k">def</span> <span class="nf">bench_ET</span><span class="p">():</span>
585 <span class="n">tree</span> <span class="o">=</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s">"ot.xml"</span><span class="p">)</span>
586 <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
587 <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">"//v"</span><span class="p">):</span>
588 <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
589 <span class="k">if</span> <span class="s">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
590 <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
591 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
593 <p>which takes about one second on my machine today. The faster <tt class="docutils literal">iterparse()</tt>
594 variant looks like this:</p>
595 <div class="syntax"><pre><span class="k">def</span> <span class="nf">bench_ET_iterparse</span><span class="p">():</span>
596 <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
597 <span class="k">for</span> <span class="n">event</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">iterparse</span><span class="p">(</span><span class="s">"ot.xml"</span><span class="p">):</span>
598 <span class="k">if</span> <span class="n">v</span><span class="o">.</span><span class="n">tag</span> <span class="o">==</span> <span class="s">'v'</span><span class="p">:</span>
599 <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
600 <span class="k">if</span> <span class="s">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
601 <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
602 <span class="n">v</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
603 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
605 <p>The improvement is about 10%. At the time I first tried (early 2006), lxml
606 didn't have <tt class="docutils literal">iterparse()</tt> support, but the <tt class="docutils literal">findall()</tt> variant was already
607 faster than ElementTree. This changes immediately when you switch to
608 cElementTree. The latter only needs 0.17 seconds to do the trick today and
609 only some impressive 0.10 seconds when running the iterparse version. And
610 even back then, it was quite a bit faster than what lxml could achieve.</p>
611 <p>Since then, lxml has matured a lot and has gotten much faster. The iterparse
612 variant now runs in 0.14 seconds, and if you remove the <tt class="docutils literal">v.clear()</tt>, it is
613 even a little faster (which isn't the case for cElementTree).</p>
614 <p>One of the many great tools in lxml is XPath, a swiss army knife for finding
615 things in XML documents. It is possible to move the whole thing to a pure
616 XPath implementation, which looks like this:</p>
617 <div class="syntax"><pre><span class="k">def</span> <span class="nf">bench_lxml_xpath_all</span><span class="p">():</span>
618 <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s">"ot.xml"</span><span class="p">)</span>
619 <span class="n">result</span> <span class="o">=</span> <span class="n">tree</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s">"//v[contains(., 'begat')]/text()"</span><span class="p">)</span>
620 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
622 <p>This runs in about 0.13 seconds and is about the shortest possible
623 implementation (in lines of Python code) that I could come up with. Now, this
624 is already a rather complex XPath expression compared to the simple "//v"
625 ElementPath expression we started with. Since this is also valid XPath, let's
626 try this instead:</p>
627 <div class="syntax"><pre><span class="k">def</span> <span class="nf">bench_lxml_xpath</span><span class="p">():</span>
628 <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s">"ot.xml"</span><span class="p">)</span>
629 <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
630 <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s">"//v"</span><span class="p">):</span>
631 <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
632 <span class="k">if</span> <span class="s">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
633 <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
634 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
636 <p>This gets us down to 0.12 seconds, thus showing that a generic XPath
637 evaluation engine cannot always compete with a simpler, tailored solution.
638 However, since this is not much different from the original findall variant,
639 we can remove the complexity of the XPath call completely and just go with
640 what we had in the beginning. Under lxml, this runs in the same 0.12 seconds.</p>
641 <p>But there is one thing left to try. We can replace the simple ElementPath
642 expression with a native tree iterator:</p>
643 <div class="syntax"><pre><span class="k">def</span> <span class="nf">bench_lxml_getiterator</span><span class="p">():</span>
644 <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s">"ot.xml"</span><span class="p">)</span>
645 <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
646 <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">getiterator</span><span class="p">(</span><span class="s">"v"</span><span class="p">):</span>
647 <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
648 <span class="k">if</span> <span class="s">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
649 <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
650 <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
652 <p>This implements the same thing, just without the overhead of parsing and
653 evaluating a path expression. And this makes it another bit faster, down to
654 0.11 seconds. For comparison, cElementTree runs this version in 0.17 seconds.</p>
655 <p>So, what have we learned?</p>
657 <li>Python code is not slow. The pure XPath solution was not even as fast as
658 the first shot Python implementation. In general, a few more lines in
659 Python make things more readable, which is much more important than the last
660 5% of performance.</li>
661 <li>It's important to know the available options - and it's worth starting with
662 the most simple one. In this case, a programmer would then probably have
663 started with <tt class="docutils literal"><span class="pre">getiterator("v")</span></tt> or <tt class="docutils literal">iterparse()</tt>. Either of them would
664 already have been the most efficient, depending on which library is used.</li>
665 <li>It's important to know your tool. lxml and cElementTree are both very fast
666 libraries, but they do not have the same performance characteristics. The
667 fastest solution in one library can be comparatively slow in the other. If
668 you optimise, optimise for the specific target platform.</li>
669 <li>It's not always worth optimising. After all that hassle we got from 0.12
670 seconds for the initial implementation to 0.11 seconds. Switching over to
671 cElementTree and writing an <tt class="docutils literal">iterparse()</tt> based version would have given
672 us 0.10 seconds - not a big difference for 3MB of XML.</li>
673 <li>Take care what operation is really dominating in your use case. If we split
674 up the operations, we can see that lxml is slightly slower than cElementTree
675 on <tt class="docutils literal">parse()</tt> (both about 0.06 seconds), but more visibly slower on
676 <tt class="docutils literal">iterparse()</tt>: 0.07 versus 0.10 seconds. However, tree iteration in lxml
677 is increadibly fast, so it can be better to parse the whole tree and then
678 iterate over it rather than using <tt class="docutils literal">iterparse()</tt> to do both in one step.
679 Or, you can just wait for the lxml developers to optimise iterparse in one
680 of the next releases...</li>
683 <div class="section" id="lxml-objectify">
684 <h1>lxml.objectify</h1>
685 <p>The following timings are based on the benchmark script <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_objectify.py">bench_objectify.py</a>.</p>
686 <p>Objectify is a data-binding API for XML based on lxml.etree, that was added in
687 version 1.1. It uses standard Python attribute access to traverse the XML
688 tree. It also features ObjectPath, a fast path language based on the same
690 <p>Just like lxml.etree, lxml.objectify creates Python representations of
691 elements on the fly. To save memory, the normal Python garbage collection
692 mechanisms will discard them when their last reference is gone. In cases
693 where deeply nested elements are frequently accessed through the objectify
694 API, the create-discard cycles can become a bottleneck, as elements have to be
695 instantiated over and over again.</p>
696 <div class="section" id="objectpath">
698 <p>ObjectPath can be used to speed up the access to elements that are deep in the
699 tree. It avoids step-by-step Python element instantiations along the path,
700 which can substantially improve the access time:</p>
701 <pre class="literal-block">
702 lxe: attribute (--TR T1) 4.8928 msec/pass
703 lxe: attribute (--TR T2) 25.5480 msec/pass
704 lxe: attribute (--TR T4) 4.6349 msec/pass
706 lxe: objectpath (--TR T1) 1.4842 msec/pass
707 lxe: objectpath (--TR T2) 21.1990 msec/pass
708 lxe: objectpath (--TR T4) 1.4892 msec/pass
710 lxe: attributes_deep (--TR T1) 11.9710 msec/pass
711 lxe: attributes_deep (--TR T2) 32.4290 msec/pass
712 lxe: attributes_deep (--TR T4) 11.4839 msec/pass
714 lxe: objectpath_deep (--TR T1) 4.8139 msec/pass
715 lxe: objectpath_deep (--TR T2) 24.6511 msec/pass
716 lxe: objectpath_deep (--TR T4) 4.7588 msec/pass
718 <p>Note, however, that parsing ObjectPath expressions is not for free either, so
719 this is most effective for frequently accessing the same element.</p>
721 <div class="section" id="caching-elements">
722 <h2>Caching Elements</h2>
723 <p>A way to improve the normal attribute access time is static instantiation of
724 the Python objects, thus trading memory for speed. Just create a cache
725 dictionary and run:</p>
726 <div class="syntax"><pre><span class="n">cache</span><span class="p">[</span><span class="n">root</span><span class="p">]</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">root</span><span class="o">.</span><span class="n">iter</span><span class="p">())</span>
728 <p>after parsing and:</p>
729 <div class="syntax"><pre><span class="k">del</span> <span class="n">cache</span><span class="p">[</span><span class="n">root</span><span class="p">]</span>
731 <p>when you are done with the tree. This will keep the Python element
732 representations of all elements alive and thus avoid the overhead of repeated
733 Python object creation. You can also consider using filters or generator
734 expressions to be more selective. By choosing the right trees (or even
735 subtrees and elements) to cache, you can trade memory usage against access
737 <pre class="literal-block">
738 lxe: attribute_cached (--TR T1) 3.8228 msec/pass
739 lxe: attribute_cached (--TR T2) 23.7138 msec/pass
740 lxe: attribute_cached (--TR T4) 3.5269 msec/pass
742 lxe: attributes_deep_cached (--TR T1) 4.6771 msec/pass
743 lxe: attributes_deep_cached (--TR T2) 24.8699 msec/pass
744 lxe: attributes_deep_cached (--TR T4) 4.3321 msec/pass
746 lxe: objectpath_deep_cached (--TR T1) 1.1430 msec/pass
747 lxe: objectpath_deep_cached (--TR T2) 19.7470 msec/pass
748 lxe: objectpath_deep_cached (--TR T4) 1.1740 msec/pass
750 <p>Things to note: you cannot currently use <tt class="docutils literal">weakref.WeakKeyDictionary</tt> objects
751 for this as lxml's element objects do not support weak references (which are
752 costly in terms of memory). Also note that new element objects that you add
753 to these trees will not turn up in the cache automatically and will therefore
754 still be garbage collected when all their Python references are gone, so this
755 is most effective for largely immutable trees. You should consider using a
756 set instead of a list in this case and add new elements by hand.</p>
758 <div class="section" id="further-optimisations">
759 <h2>Further optimisations</h2>
760 <p>Here are some more things to try if optimisation is required:</p>
762 <li>A lot of time is usually spent in tree traversal to find the addressed
763 elements in the tree. If you often work in subtrees, do what you would also
764 do with deep Python objects: assign the parent of the subtree to a variable
765 or pass it into functions instead of starting at the root. This allows
766 accessing its descendents more directly.</li>
767 <li>Try assigning data values directly to attributes instead of passing them
768 through DataElement.</li>
769 <li>If you use custom data types that are costly to parse, try running
770 <tt class="docutils literal">objectify.annotate()</tt> over read-only trees to speed up the attribute type
771 inference on read access.</li>
773 <p>Note that none of these measures is guaranteed to speed up your application.
774 As usual, you should prefer readable code over premature optimisations and
775 profile your expected use cases before bothering to apply optimisations at
781 <hr class="footer" />
782 Generated on: 2012-07-31.
projects / platform / upstream / python-lxml.git / blob