1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
2 "http://www.w3.org/TR/html4/loose.dtd">
5 <meta http-equiv="Content-Type" content="text/html">
6 <style type="text/css"></style>
8 TD {font-family: Verdana,Arial,Helvetica}
9 BODY {font-family: Verdana,Arial,Helvetica; margin-top: 2em; margin-left: 0em; margin-right: 0em}
10 H1 {font-family: Verdana,Arial,Helvetica}
11 H2 {font-family: Verdana,Arial,Helvetica}
12 H3 {font-family: Verdana,Arial,Helvetica}
13 A:link, A:visited, A:active { text-decoration: underline }
16 <title>Libxml2 XmlTextReader Interface tutorial</title>
19 <body bgcolor="#fffacd" text="#000000">
20 <h1 align="center">Libxml2 XmlTextReader Interface tutorial</h1>
24 <p>This document describes the use of the XmlTextReader streaming API added
25 to libxml2 in version 2.5.0 . This API is closely modeled after the <a
26 href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">XmlTextReader</a>
28 href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlReader.html">XmlReader</a>
29 classes of the C# language.</p>
31 <p>This tutorial will present the key points of this API, and working
32 examples using both C and the Python bindings:</p>
34 <p>Table of content:</p>
36 <li><a href="#Introducti">Introduction: why a new API</a></li>
37 <li><a href="#Walking">Walking a simple tree</a></li>
38 <li><a href="#Extracting">Extracting informations for the current
40 <li><a href="#Extracting1">Extracting informations for the
42 <li><a href="#Validating">Validating a document</a></li>
43 <li><a href="#Entities">Entities substitution</a></li>
44 <li><a href="#L1142">Relax-NG Validation</a></li>
45 <li><a href="#Mixing">Mixing the reader and tree or XPath
51 <h2><a name="Introducti">Introduction: why a new API</a></h2>
53 <p>Libxml2 <a href="http://xmlsoft.org/html/libxml-tree.html">main API is
54 tree based</a>, where the parsing operation results in a document loaded
55 completely in memory, and expose it as a tree of nodes all availble at the
56 same time. This is very simple and quite powerful, but has the major
57 limitation that the size of the document that can be hamdled is limited by
58 the size of the memory available. Libxml2 also provide a <a
59 href="http://www.saxproject.org/">SAX</a> based API, but that version was
60 designed upon one of the early <a
61 href="http://www.jclark.com/xml/expat.html">expat</a> version of SAX, SAX is
62 also not formally defined for C. SAX basically work by registering callbacks
63 which are called directly by the parser as it progresses through the document
64 streams. The problem is that this programming model is relatively complex,
65 not well standardized, cannot provide validation directly, makes entity,
66 namespace and base processing relatively hard.</p>
69 href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">XmlTextReader
70 API from C#</a> provides a far simpler programming model. The API acts as a
71 cursor going forward on the document stream and stopping at each node in the
72 way. The user's code keeps control of the progress and simply calls a
73 Read() function repeatedly to progress to each node in sequence in document
74 order. There is direct support for namespaces, xml:base, entity handling and
75 adding DTD validation on top of it was relatively simple. This API is really
76 close to the <a href="http://www.w3.org/TR/DOM-Level-2-Core/">DOM Core
77 specification</a> This provides a far more standard, easy to use and powerful
78 API than the existing SAX. Moreover integrating extension features based on
79 the tree seems relatively easy.</p>
81 <p>In a nutshell the XmlTextReader API provides a simpler, more standard and
82 more extensible interface to handle large documents than the existing SAX
85 <h2><a name="Walking">Walking a simple tree</a></h2>
87 <p>Basically the XmlTextReader API is a forward only tree walking interface.
88 The basic steps are:</p>
90 <li>prepare a reader context operating on some input</li>
91 <li>run a loop iterating over all nodes in the document</li>
92 <li>free up the reader context</li>
95 <p>Here is a basic C sample doing this:</p>
96 <pre>#include <libxml/xmlreader.h>
98 void processNode(xmlTextReaderPtr reader) {
99 /* handling of a node in the tree */
102 int streamFile(char *filename) {
103 xmlTextReaderPtr reader;
106 reader = xmlNewTextReaderFilename(filename);
107 if (reader != NULL) {
108 ret = xmlTextReaderRead(reader);
111 ret = xmlTextReaderRead(reader);
113 xmlFreeTextReader(reader);
115 printf("%s : failed to parse\n", filename);
118 printf("Unable to open %s\n", filename);
122 <p>A few things to notice:</p>
124 <li>the include file needed : <code>libxml/xmlreader.h</code></li>
125 <li>the creation of the reader using a filename</li>
126 <li>the repeated call to xmlTextReaderRead() and how any return value
127 different from 1 should stop the loop</li>
128 <li>that a negative return means a parsing error</li>
129 <li>how xmlFreeTextReader() should be used to free up the resources used by
133 <p>Here is similar code in python for exactly the same processing:</p>
136 def processNode(reader):
139 def streamFile(filename):
141 reader = libxml2.newTextReaderFilename(filename)
143 print "unable to open %s" % (filename)
152 print "%s : failed to parse" % (filename)</pre>
154 <p>The only things worth adding are that the <a
155 href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">xmlTextReader
156 is abstracted as a class like in C#</a> with the same method names (but the
157 properties are currently accessed with methods) and that one doesn't need to
158 free the reader at the end of the processing. It will get garbage collected
159 once all references have disapeared.</p>
161 <h2><a name="Extracting">Extracting information for the current node</a></h2>
163 <p>So far the example code did not indicate how information was extracted
164 from the reader. It was abstrated as a call to the processNode() routine,
165 with the reader as the argument. At each invocation, the parser is stopped on
166 a given node and the reader can be used to query those node properties. Each
167 <em>Property</em> is available at the C level as a function taking a single
168 xmlTextReaderPtr argument whose name is
169 <code>xmlTextReader</code><em>Property</em> , if the return type is an
170 <code>xmlChar *</code> string then it must be deallocated with
171 <code>xmlFree()</code> to avoid leaks. For the Python interface, there is a
172 <em>Property</em> method to the reader class that can be called on the
173 instance. The list of the properties is based on the <a
174 href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">C#
175 XmlTextReader class</a> set of properties and methods:</p>
177 <li><em>NodeType</em>: The node type, 1 for start element, 15 for end of
178 element, 2 for attributes, 3 for text nodes, 4 for CData sections, 5 for
179 entity references, 6 for entity declarations, 7 for PIs, 8 for comments,
180 9 for the document nodes, 10 for DTD/Doctype nodes, 11 for document
181 fragment and 12 for notation nodes.</li>
182 <li><em>Name</em>: the <a
183 href="http://www.w3.org/TR/REC-xml-names/#ns-qualnames">qualified
184 name</a> of the node, equal to (<em>Prefix</em>:)<em>LocalName</em>.</li>
185 <li><em>LocalName</em>: the <a
186 href="http://www.w3.org/TR/REC-xml-names/#NT-LocalPart">local name</a> of
188 <li><em>Prefix</em>: a shorthand reference to the <a
189 href="http://www.w3.org/TR/REC-xml-names/">namespace</a> associated with
191 <li><em>NamespaceUri</em>: the URI defining the <a
192 href="http://www.w3.org/TR/REC-xml-names/">namespace</a> associated with
194 <li><em>BaseUri:</em> the base URI of the node. See the <a
195 href="http://www.w3.org/TR/xmlbase/">XML Base W3C specification</a>.</li>
196 <li><em>Depth:</em> the depth of the node in the tree, starts at 0 for the
198 <li><em>HasAttributes</em>: whether the node has attributes.</li>
199 <li><em>HasValue</em>: whether the node can have a text value.</li>
200 <li><em>Value</em>: provides the text value of the node if present.</li>
201 <li><em>IsDefault</em>: whether an Attribute node was generated from the
202 default value defined in the DTD or schema (<em>unsupported
204 <li><em>XmlLang</em>: the <a
205 href="http://www.w3.org/TR/REC-xml#sec-lang-tag">xml:lang</a> scope
206 within which the node resides.</li>
207 <li><em>IsEmptyElement</em>: check if the current node is empty, this is a
208 bit bizarre in the sense that <code><a/></code> will be considered
209 empty while <code><a></a></code> will not.</li>
210 <li><em>AttributeCount</em>: provides the number of attributes of the
214 <p>Let's look first at a small example to get this in practice by redefining
215 the processNode() function in the Python example:</p>
216 <pre>def processNode(reader):
217 print "%d %d %s %d" % (reader.Depth(), reader.NodeType(),
218 reader.Name(), reader.IsEmptyElement())</pre>
220 <p>and look at the result of calling streamFile("tst.xml") for various
221 content of the XML test file.</p>
223 <p>For the minimal document "<code><doc/></code>" we get:</p>
226 <p>Only one node is found, its depth is 0, type 1 indicate an element start,
227 of name "doc" and it is empty. Trying now with
228 "<code><doc></doc></code>" instead leads to:</p>
232 <p>The document root node is not flagged as empty anymore and both a start
233 and an end of element are detected. The following document shows how
234 character data are reported:</p>
235 <pre><doc><a/><b>some text</b>
236 <c/></doc></pre>
238 <p>We modifying the processNode() function to also report the node Value:</p>
239 <pre>def processNode(reader):
240 print "%d %d %s %d %s" % (reader.Depth(), reader.NodeType(),
241 reader.Name(), reader.IsEmptyElement(),
242 reader.Value())</pre>
244 <p>The result of the test is:</p>
248 2 3 #text 0 some text
253 0 15 doc 0 None</pre>
255 <p>There are a few things to note:</p>
257 <li>the increase of the depth value (first row) as children nodes are
259 <li>the text node child of the b element, of type 3 and its content</li>
260 <li>the text node containing the line return between elements b and c</li>
261 <li>that elements have the Value None (or NULL in C)</li>
264 <p>The equivalent routine for <code>processNode()</code> as used by
265 <code>xmllint --stream --debug</code> is the following and can be found in
266 the xmllint.c module in the source distribution:</p>
267 <pre>static void processNode(xmlTextReaderPtr reader) {
268 xmlChar *name, *value;
270 name = xmlTextReaderName(reader);
272 name = xmlStrdup(BAD_CAST "--");
273 value = xmlTextReaderValue(reader);
275 printf("%d %d %s %d",
276 xmlTextReaderDepth(reader),
277 xmlTextReaderNodeType(reader),
279 xmlTextReaderIsEmptyElement(reader));
284 printf(" %s\n", value);
289 <h2><a name="Extracting1">Extracting information for the attributes</a></h2>
291 <p>The previous examples don't indicate how attributes are processed. The
292 simple test "<code><doc a="b"/></code>" provides the following
294 <pre>0 1 doc 1 None</pre>
296 <p>This proves that attribute nodes are not traversed by default. The
297 <em>HasAttributes</em> property allow to detect their presence. To check
298 their content the API has special instructions. Basically two kinds of operations
301 <li>to move the reader to the attribute nodes of the current element, in
302 that case the cursor is positionned on the attribute node</li>
303 <li>to directly query the element node for the attribute value</li>
306 <p>In both case the attribute can be designed either by its position in the
307 list of attribute (<em>MoveToAttributeNo</em> or <em>GetAttributeNo</em>) or
308 by their name (and namespace):</p>
310 <li><em>GetAttributeNo</em>(no): provides the value of the attribute with
311 the specified index no relative to the containing element.</li>
312 <li><em>GetAttribute</em>(name): provides the value of the attribute with
313 the specified qualified name.</li>
314 <li>GetAttributeNs(localName, namespaceURI): provides the value of the
315 attribute with the specified local name and namespace URI.</li>
316 <li><em>MoveToAttributeNo</em>(no): moves the position of the current
317 instance to the attribute with the specified index relative to the
318 containing element.</li>
319 <li><em>MoveToAttribute</em>(name): moves the position of the current
320 instance to the attribute with the specified qualified name.</li>
321 <li><em>MoveToAttributeNs</em>(localName, namespaceURI): moves the position
322 of the current instance to the attribute with the specified local name
323 and namespace URI.</li>
324 <li><em>MoveToFirstAttribute</em>: moves the position of the current
325 instance to the first attribute associated with the current node.</li>
326 <li><em>MoveToNextAttribute</em>: moves the position of the current
327 instance to the next attribute associated with the current node.</li>
328 <li><em>MoveToElement</em>: moves the position of the current instance to
329 the node that contains the current Attribute node.</li>
332 <p>After modifying the processNode() function to show attributes:</p>
333 <pre>def processNode(reader):
334 print "%d %d %s %d %s" % (reader.Depth(), reader.NodeType(),
335 reader.Name(), reader.IsEmptyElement(),
337 if reader.NodeType() == 1: # Element
338 while reader.MoveToNextAttribute():
339 print "-- %d %d (%s) [%s]" % (reader.Depth(), reader.NodeType(),
340 reader.Name(),reader.Value())</pre>
342 <p>The output for the same input document reflects the attribute:</p>
346 <p>There are a couple of things to note on the attribute processing:</p>
348 <li>Their depth is the one of the carrying element plus one.</li>
349 <li>Namespace declarations are seen as attributes, as in DOM.</li>
352 <h2><a name="Validating">Validating a document</a></h2>
354 <p>Libxml2 implementation adds some extra features on top of the XmlTextReader
355 API. The main one is the ability to DTD validate the parsed document
356 progressively. This is simply the activation of the associated feature of the
357 parser used by the reader structure. There are a few options available
358 defined as the enum xmlParserProperties in the libxml/xmlreader.h header
361 <li>XML_PARSER_LOADDTD: force loading the DTD (without validating)</li>
362 <li>XML_PARSER_DEFAULTATTRS: force attribute defaulting (this also imply
363 loading the DTD)</li>
364 <li>XML_PARSER_VALIDATE: activate DTD validation (this also imply loading
366 <li>XML_PARSER_SUBST_ENTITIES: substitute entities on the fly, entity
367 reference nodes are not generated and are replaced by their expanded
369 <li>more settings might be added, those were the one available at the 2.5.0
373 <p>The GetParserProp() and SetParserProp() methods can then be used to get
374 and set the values of those parser properties of the reader. For example</p>
375 <pre>def parseAndValidate(file):
376 reader = libxml2.newTextReaderFilename(file)
377 reader.SetParserProp(libxml2.PARSER_VALIDATE, 1)
382 print "Error parsing and validating %s" % (file)</pre>
384 <p>This routine will parse and validate the file. Error messages can be
385 captured by registering an error handler. See python/tests/reader2.py for
386 more complete Python examples. At the C level the equivalent call to cativate
387 the validation feature is just:</p>
388 <pre>ret = xmlTextReaderSetParserProp(reader, XML_PARSER_VALIDATE, 1)</pre>
390 <p>and a return value of 0 indicates success.</p>
392 <h2><a name="Entities">Entities substitution</a></h2>
394 <p>By default the xmlReader will report entities as such and not replace them
395 with their content. This default behaviour can however be overriden using:</p>
397 <p><code>reader.SetParserProp(libxml2.PARSER_SUBST_ENTITIES,1)</code></p>
399 <h2><a name="L1142">Relax-NG Validation</a></h2>
401 <p style="font-size: 10pt">Introduced in version 2.5.7</p>
403 <p>Libxml2 can now validate the document being read using the xmlReader using
404 Relax-NG schemas. While the Relax NG validator can't always work in a
405 streamable mode, only subsets which cannot be reduced to regular expressions
406 need to have their subtree expanded for validation. In practice it means
407 that, unless the schemas for the top level element content is not expressable
408 as a regexp, only chunk of the document needs to be parsed while
411 <p>The steps to do so are:</p>
413 <li>create a reader working on a document as usual</li>
414 <li>before any call to read associate it to a Relax NG schemas, either the
415 preparsed schemas or the URL to the schemas to use</li>
416 <li>errors will be reported the usual way, and the validity status can be
417 obtained using the IsValid() interface of the reader like for DTDs.</li>
420 <p>Example, assuming the reader has already being created and that the schema
421 string contains the Relax-NG schemas:</p>
422 <pre><code>rngp = libxml2.relaxNGNewMemParserCtxt(schema, len(schema))<br>
423 rngs = rngp.relaxNGParse()<br>
424 reader.RelaxNGSetSchema(rngs)<br>
425 ret = reader.Read()<br>
427 ret = reader.Read()<br>
429 print "Error parsing the document"<br>
430 if reader.IsValid() != 1:<br>
431 print "Document failed to validate"</code><br>
434 <p>See <code>reader6.py</code> in the sources or documentation for a complete
437 <h2><a name="Mixing">Mixing the reader and tree or XPath operations</a></h2>
439 <p style="font-size: 10pt">Introduced in version 2.5.7</p>
441 <p>While the reader is a streaming interface, its underlying implementation
442 is based on the DOM builder of libxml2. As a result it is relatively simple
443 to mix operations based on both models under some constraints. To do so the
444 reader has an Expand() operation allowing to grow the subtree under the
445 current node. It returns a pointer to a standard node which can be
446 manipulated in the usual ways. The node will get all its ancestors and the
447 full subtree available. Usual operations like XPath queries can be used on
448 that reduced view of the document. Here is an example extracted from
449 reader5.py in the sources which extract and prints the bibliography for the
450 "Dragon" compiler book from the XML 1.0 recommendation:</p>
451 <pre>f = open('../../test/valid/REC-xml-19980210.xml')
452 input = libxml2.inputBuffer(f)
453 reader = input.newTextReader("REC")
456 while reader.Name() == 'bibl':
457 node = reader.Expand() # expand the subtree
458 if node.xpathEval("@id = 'Aho'"): # use XPath on it
459 res = res + node.serialize()
460 if reader.Next() != 1: # skip the subtree
463 <p>Note, however that the node instance returned by the Expand() call is only
464 valid until the next Read() operation. The Expand() operation does not
465 affects the Read() ones, however usually once processed the full subtree is
466 not useful anymore, and the Next() operation allows to skip it completely and
467 process to the successor or return 0 if the document end is reached.</p>
469 <p><a href="mailto:xml@gnome.org">Daniel Veillard</a></p>