1 How to build lxml from source
2 =============================
4 To build lxml from source, you need libxml2 and libxslt properly
5 installed, *including the header files*. These are likely shipped in
6 separate ``-dev`` or ``-devel`` packages like ``libxml2-dev``, which
7 you must install before trying to build lxml.
13 3 Building the sources
14 4 Running the tests and reporting errors
16 6 Building lxml on MacOS-X
17 7 Static linking on Windows
18 8 Building Debian packages from SVN sources
24 .. _pip: http://pypi.python.org/pypi/pip
25 .. _Cython: http://cython.org
26 .. _wheel: https://wheel.readthedocs.io/en/latest/
28 The lxml.etree and lxml.objectify modules are written in Cython_.
29 Since we distribute the Cython-generated .c files with lxml releases,
30 however, you do not need Cython to build lxml from the normal release
31 sources. We even encourage you to *not install Cython* for a normal
32 release build, as the generated C code can vary quite heavily between
33 Cython versions, which may or may not generate correct code for lxml.
34 The pre-generated release sources were tested and therefore are known
37 So, if you want a reliable build of lxml, we suggest to a) use a
38 source release of lxml and b) disable or uninstall Cython for the
41 *Only* if you are interested in building lxml from a checkout of the
42 developer sources (e.g. to test a bug fix that has not been release
43 yet) or if you want to be an lxml developer, then you do need a
44 working Cython installation. You can use pip_ to install it::
46 pip install -r requirements.txt
48 https://github.com/lxml/lxml/blob/master/requirements.txt
50 lxml currently requires at least Cython 0.20, later release versions
57 The lxml package is developed in a repository on Github_ using
58 Mercurial_ and the `hg-git`_ plugin. You can retrieve the current
59 developer version using::
61 hg clone git+ssh://git@github.com/lxml/lxml.git lxml
65 git clone ssh://git@github.com/lxml/lxml.git lxml
67 This will create a directory ``lxml`` and download the source into it,
68 including the complete development history. Don't be afraid, the
69 repository download is fairly quick. You can also browse the
70 `lxml repository`_ through the web or download a ZIP archive with the
71 `latest master branch <https://github.com/lxml/lxml/archive/master.zip>`_.
73 .. _Github: https://github.com/lxml/
74 .. _Mercurial: http://mercurial.selenic.com/
75 .. _`hg-git`: http://hg-git.github.com/
76 .. _`lxml repository`: https://github.com/lxml/lxml
77 .. _`source tar-ball`: https://github.com/lxml/lxml/tarball/master
83 Clone the source repository as described above (or download
84 the `source tar-ball`_ and unpack it) and then type::
90 python setup.py bdist_egg # requires 'setuptools' or 'distribute'
92 To (re-)build the C sources with Cython, you must additionally pass the
93 option ``--with-cython``::
95 python setup.py build --with-cython
97 If you want to test lxml from the source directory, it is better to build it
100 python setup.py build_ext -i --with-cython
102 or, in Unix-like environments::
106 To speed up the build in test environments (e.g. on a continuous
107 integration server), set the ``CFLAGS`` environment variable to
108 disable C compiler optimisations (e.g. "-O0" for gcc, that's
109 minus-oh-zero), for example::
111 CFLAGS="-O0" make inplace
113 If you get errors about missing header files (e.g. ``Python.h`` or
114 ``libxml/xmlversion.h``) then you need to make sure the development
115 packages of Python, libxml2 and libxslt are properly installed. On
116 Linux distributions, they are usually called something like
117 ``libxml2-dev`` or ``libxslt-devel``. If these packages were
118 installed in non-standard places, try passing the following option to
119 setup.py to make sure the right config is found::
121 python setup.py build --with-xslt-config=/path/to/xslt-config
123 If this doesn't help, you may have to add the location of the header
124 files to the include path like::
126 python setup.py build_ext -i -I /usr/include/libxml2
128 where the file is in ``/usr/include/libxml2/libxml/xmlversion.h``
130 To use lxml.etree in-place, you can place lxml's ``src`` directory
131 on your Python module search path (PYTHONPATH) and then import
132 ``lxml.etree`` to play with it::
135 # PYTHONPATH=src python
137 Type "help", "copyright", "credits" or "license" for more information.
138 >>> from lxml import etree
141 To make sure everything gets recompiled cleanly after changes, you can
142 run ``make clean`` or delete the file ``src/lxml/etree.c``.
145 Running the tests and reporting errors
146 --------------------------------------
148 The source distribution (tgz) and the source repository contain a test
149 suite for lxml. You can run it from the top-level directory::
153 Note that the test script only tests the in-place build (see distutils
154 building above), as it searches the ``src`` directory. You can use the
155 following one-step command to trigger an in-place build and test it::
159 This also runs the ElementTree and cElementTree compatibility tests. To call
160 them separately, make sure you have lxml on your PYTHONPATH first, then run::
168 If the tests give failures, errors, or worse, segmentation faults, we'd really
169 like to know. Please contact us on the `mailing list`_, and please specify
170 the version of lxml, libxml2, libxslt and Python you were using, as well as
171 your operating system type (Linux, Windows, MacOS-X, ...).
173 .. _`mailing list`: http://lxml.de/mailinglist/
176 Building an egg or wheel
177 ------------------------
179 This is the procedure to make an lxml egg or wheel_ for your platform.
180 It assumes that you have ``setuptools`` or ``distribute`` installed, as well
181 as the ``wheel`` package.
183 First, download the lxml-x.y.tar.gz release. This contains the pregenerated
184 C files so that you can be sure you build exactly from the release sources.
185 Unpack them and ``cd`` into the resulting directory. Then, to build a wheel,
186 simply run the command
190 python setup.py bdist_wheel
192 or, to build a statically linked wheel with all of libxml2, libxslt and
193 friends compiled in, run
195 python setup.py bdist_wheel --static-deps
197 The resulting .whl file will be written into the ``dist`` directory.
199 To build an egg file, run
203 python setup.py build_egg
205 If you are on a Unix-like platform, you can first build the extension modules
210 python setup.py build
212 and then ``cd`` into the directory ``build/lib.your.platform`` to call
213 ``strip`` on any ``.so`` file you find there. This reduces the size of
214 the binary distribution considerably. Then, from the package root directory,
219 python setup.py bdist_egg
221 This will quickly package the pre-built packages into an egg file and
222 drop it into the ``dist`` directory.
225 Building lxml on MacOS-X
226 ------------------------
228 Apple regularly ships new system releases with horribly outdated
229 system libraries. This is specifically the case for libxml2 and
230 libxslt, where the system provided versions used to be too old
231 to even build lxml for a long time.
233 While the Unix environment in MacOS-X makes it relatively easy to
234 install Unix/Linux style package management tools and new software, it
235 actually seems to be hard to get libraries set up for exclusive usage
236 that MacOS-X ships in an older version. Alternative distributions
237 (like macports) install their libraries in addition to the system
238 libraries, but the compiler and the runtime loader on MacOS still sees
239 the system libraries before the new libraries. This can lead to
240 undebuggable crashes where the newer library seems to be loaded but
241 the older system library is used.
243 Apple discourages static building against libraries, which would help
244 working around this problem. Apple does not ship static library
245 binaries with its system and several package management systems follow
246 this decision. Therefore, building static binaries requires building
247 the dependencies first. The ``setup.py`` script does this
248 automatically when you call it like this::
250 python setup.py build --static-deps
252 This will download and build the latest versions of libxml2 and
253 libxslt from the official FTP download site. If you want to use
254 specific versions, or want to prevent any online access, you can
255 download both ``tar.gz`` release files yourself, place them into a
256 subdirectory ``libs`` in the lxml distribution, and call ``setup.py``
257 with the desired target versions like this::
259 python setup.py build --static-deps \
260 --libxml2-version=2.9.1 \
261 --libxslt-version=1.1.28 \
263 sudo python setup.py install
265 Instead of ``build``, you can use any target, like ``bdist_egg``
266 if you want to use setuptools to build an installable egg, or
267 ``bdist_wheel`` for a wheel package.
269 Note that this also works with pip_. Since you can't pass
270 command line options in this case, you have to use an environment
273 STATIC_DEPS=true pip install lxml
275 To install the package into the system Python package directory,
276 run the installation with "sudo"::
278 STATIC_DEPS=true sudo pip install lxml
280 The ``STATICBUILD`` environment variable is handled equivalently to
281 the ``STATIC_DEPS`` variable, but is used by some other extension
284 If you decide to do a non-static build, you may also have to install
285 the command line tools in addition to the XCode build environment.
286 They are available as a restricted download from here:
288 https://developer.apple.com/downloads/index.action?=command%20line%20tools#
290 Without them, the compiler may not find the necessary header files of
291 the XML libraries, according to the second comment in this ticket:
293 https://bugs.launchpad.net/lxml/+bug/1244094
296 Static linking on Windows
297 -------------------------
299 Most operating systems have proper package management that makes installing
300 current versions of libxml2 and libxslt easy. The most famous exception is
301 Microsoft Windows, which entirely lacks these capabilities. To work around
302 the limits of this platform, lxml's installation can download pre-built
303 packages of the dependencies and build statically against them. Assuming
304 you have a proper C compiler setup to build Python extensions, this should
307 python setup.py bdist_wininst --static-deps
309 It should create a windows installer in the ``pkg`` directory.
312 Building Debian packages from SVN sources
313 -----------------------------------------
315 `Andreas Pakulat`_ proposed the following approach.
317 .. _`Andreas Pakulat`: http://thread.gmane.org/gmane.comp.python.lxml.devel/1239/focus=1249
319 * ``apt-get source lxml``
320 * remove the unpacked directory
321 * tar.gz the lxml SVN version and replace the orig.tar.gz that lies in the
323 * check md5sum of created tar.gz file and place new sum and size in dsc file
324 * do ``dpkg-source -x lxml-[VERSION].dsc`` and cd into the newly created directory
325 * run ``dch -i`` and add a comment like "use trunk version", this will
326 increase the debian version number so apt/dpkg won't get confused
327 * run ``dpkg-buildpackage -rfakeroot -us -uc`` to build the package
329 In case ``dpkg-buildpackage`` tells you that some dependencies are missing, you
330 can either install them manually or run ``apt-get build-dep lxml``.
332 That will give you .deb packages in the parent directory which can be
333 installed using ``dpkg -i``.