3 <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
4 <title>Pickle support</title>
5 <link rel="stylesheet" href="../../boostbook.css" type="text/css">
6 <meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
7 <link rel="home" href="../index.html" title="Boost.Python Reference Manual">
8 <link rel="up" href="../topics.html" title="Chapter 8. Topics">
9 <link rel="prev" href="../topics.html" title="Chapter 8. Topics">
10 <link rel="next" href="indexing_support.html" title="Indexing support">
12 <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
13 <table cellpadding="2" width="100%"><tr><td valign="top"><img alt="" width="" height="" src="../../images/boost.png"></td></tr></table>
15 <div class="spirit-nav">
16 <a accesskey="p" href="../topics.html"><img src="../../images/prev.png" alt="Prev"></a><a accesskey="u" href="../topics.html"><img src="../../images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../images/home.png" alt="Home"></a><a accesskey="n" href="indexing_support.html"><img src="../../images/next.png" alt="Next"></a>
19 <div class="titlepage"><div><div><h2 class="title" style="clear: both">
20 <a name="topics.pickle_support"></a><a class="link" href="pickle_support.html" title="Pickle support">Pickle support</a>
21 </h2></div></div></div>
22 <div class="toc"><dl class="toc">
23 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.introduction">Introduction</a></span></dt>
24 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.the_pickle_interface">The Pickle
25 Interface</a></span></dt>
26 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.example">Example</a></span></dt>
27 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.pitfall_and_safety_guard">Pitfall
28 and Safety Guard</a></span></dt>
29 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.practical_advice">Practical Advice</a></span></dt>
30 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.light_weight_alternative_pickle_">Light-weight
31 alternative: pickle support implemented in Python</a></span></dt>
34 <div class="titlepage"><div><div><h3 class="title">
35 <a name="topics.pickle_support.introduction"></a><a class="link" href="pickle_support.html#topics.pickle_support.introduction" title="Introduction">Introduction</a>
36 </h3></div></div></div>
38 Pickle is a Python module for object serialization, also known as persistence,
39 marshalling, or flattening.
42 It is often necessary to save and restore the contents of an object to
43 a file. One approach to this problem is to write a pair of functions that
44 read and write data from a file in a special format. A powerful alternative
45 approach is to use Python's pickle module. Exploiting Python's ability
46 for introspection, the pickle module recursively converts nearly arbitrary
47 Python objects into a stream of bytes that can be written to a file.
50 The Boost Python Library supports the pickle module through the interface
51 as described in detail in the <a href="https://docs.python.org/2/library/pickle.html" target="_top">Python
52 Library Reference for pickle</a>. This interface involves the special
53 methods <code class="computeroutput"><span class="identifier">__getinitargs__</span></code>,
54 <code class="computeroutput"><span class="identifier">__getstate__</span></code> and <code class="computeroutput"><span class="identifier">__setstate__</span></code> as described in the following.
55 Note that <code class="computeroutput"><span class="identifier">Boost</span><span class="special">.</span><span class="identifier">Python</span></code> is also fully compatible with
56 Python's cPickle module.
60 <div class="titlepage"><div><div><h3 class="title">
61 <a name="topics.pickle_support.the_pickle_interface"></a><a class="link" href="pickle_support.html#topics.pickle_support.the_pickle_interface" title="The Pickle Interface">The Pickle
63 </h3></div></div></div>
65 At the user level, the Boost.Python pickle interface involves three special
68 <div class="variablelist">
69 <p class="title"><b></b></p>
70 <dl class="variablelist">
71 <dt><span class="term">__getinitargs__</span></dt>
74 When an instance of a Boost.Python extension class is pickled, the
75 pickler tests if the instance has a <code class="computeroutput"><span class="identifier">__getinitargs__</span></code>
76 method. This method must return a Python <code class="computeroutput"><span class="identifier">tuple</span></code>
77 (it is most convenient to use a <a class="link" href="../object_wrappers/boost_python_tuple_hpp.html#object_wrappers.boost_python_tuple_hpp.class_tuple" title="Class tuple"><code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">tuple</span></code></a>). When the instance
78 is restored by the unpickler, the contents of this tuple are used
79 as the arguments for the class constructor.
82 If <code class="computeroutput"><span class="identifier">__getinitargs__</span></code>
83 is not defined, <code class="computeroutput"><span class="identifier">pickle</span><span class="special">.</span><span class="identifier">load</span></code>
84 will call the constructor (<code class="computeroutput"><span class="identifier">__init__</span></code>)
85 without arguments; i.e., the object must be default-constructible.
88 <dt><span class="term">__getstate__</span></dt>
90 When an instance of a <code class="computeroutput"><span class="identifier">Boost</span><span class="special">.</span><span class="identifier">Python</span></code>
91 extension class is pickled, the pickler tests if the instance has
92 a <code class="computeroutput"><span class="identifier">__getstate__</span></code> method.
93 This method should return a Python object representing the state
96 <dt><span class="term">__setstate__</span></dt>
98 When an instance of a <code class="computeroutput"><span class="identifier">Boost</span><span class="special">.</span><span class="identifier">Python</span></code>
99 extension class is restored by the unpickler (<code class="computeroutput"><span class="identifier">pickle</span><span class="special">.</span><span class="identifier">load</span></code>),
100 it is first constructed using the result of <code class="computeroutput"><span class="identifier">__getinitargs__</span></code>
101 as arguments (see above). Subsequently the unpickler tests if the
102 new instance has a <code class="computeroutput"><span class="identifier">__setstate__</span></code>
103 method. If so, this method is called with the result of <code class="computeroutput"><span class="identifier">__getstate__</span></code> (a Python object)
109 The three special methods described above may be <code class="computeroutput"><span class="special">.</span><span class="identifier">def</span><span class="special">()</span></code>'ed
110 individually by the user. However, <code class="computeroutput"><span class="identifier">Boost</span><span class="special">.</span><span class="identifier">Python</span></code>
111 provides an easy to use high-level interface via the <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">pickle_suite</span></code>
112 class that also enforces consistency: <code class="computeroutput"><span class="identifier">__getstate__</span></code>
113 and <code class="computeroutput"><span class="identifier">__setstate__</span></code> must be
114 defined as pairs. Use of this interface is demonstrated by the following
118 <div class="section">
119 <div class="titlepage"><div><div><h3 class="title">
120 <a name="topics.pickle_support.example"></a><a class="link" href="pickle_support.html#topics.pickle_support.example" title="Example">Example</a>
121 </h3></div></div></div>
122 <div class="toc"><dl class="toc">
123 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.example.pickle1_cpp">pickle1.cpp</a></span></dt>
124 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.example.pickle2_cpp">pickle2.cpp</a></span></dt>
125 <dt><span class="section"><a href="pickle_support.html#topics.pickle_support.example.pickle3_cpp">pickle3.cpp</a></span></dt>
128 There are three files in <code class="computeroutput"><span class="identifier">python</span><span class="special">/</span><span class="identifier">test</span></code>
129 that show how to provide pickle support.
131 <div class="section">
132 <div class="titlepage"><div><div><h4 class="title">
133 <a name="topics.pickle_support.example.pickle1_cpp"></a><a class="link" href="pickle_support.html#topics.pickle_support.example.pickle1_cpp" title="pickle1.cpp">pickle1.cpp</a>
134 </h4></div></div></div>
136 The C++ class in this example can be fully restored by passing the appropriate
137 argument to the constructor. Therefore it is sufficient to define the
138 pickle interface method <code class="computeroutput"><span class="identifier">__getinitargs__</span></code>.
139 This is done in the following way: Definition of the C++ pickle function:
141 <pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">world_pickle_suite</span> <span class="special">:</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">pickle_suite</span>
142 <span class="special">{</span>
143 <span class="keyword">static</span>
144 <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">tuple</span>
145 <span class="identifier">getinitargs</span><span class="special">(</span><span class="identifier">world</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">w</span><span class="special">)</span>
146 <span class="special">{</span>
147 <span class="keyword">return</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">make_tuple</span><span class="special">(</span><span class="identifier">w</span><span class="special">.</span><span class="identifier">get_country</span><span class="special">());</span>
148 <span class="special">}</span>
149 <span class="special">};</span>
152 Establishing the Python binding:
154 <pre class="programlisting"><span class="identifier">class_</span><span class="special"><</span><span class="identifier">world</span><span class="special">>(</span><span class="string">"world"</span><span class="special">,</span> <span class="identifier">args</span><span class="special"><</span><span class="keyword">const</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">&>())</span>
155 <span class="comment">// ...</span>
156 <span class="special">.</span><span class="identifier">def_pickle</span><span class="special">(</span><span class="identifier">world_pickle_suite</span><span class="special">())</span>
157 <span class="comment">// ...</span>
160 <div class="section">
161 <div class="titlepage"><div><div><h4 class="title">
162 <a name="topics.pickle_support.example.pickle2_cpp"></a><a class="link" href="pickle_support.html#topics.pickle_support.example.pickle2_cpp" title="pickle2.cpp">pickle2.cpp</a>
163 </h4></div></div></div>
165 The C++ class in this example contains member data that cannot be restored
166 by any of the constructors. Therefore it is necessary to provide the
167 <code class="computeroutput"><span class="identifier">__getstate__</span></code>/<code class="computeroutput"><span class="identifier">__setstate__</span></code> pair of pickle interface
171 Definition of the C++ pickle functions:
173 <pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">world_pickle_suite</span> <span class="special">:</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">pickle_suite</span>
174 <span class="special">{</span>
175 <span class="keyword">static</span>
176 <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">tuple</span>
177 <span class="identifier">getinitargs</span><span class="special">(</span><span class="keyword">const</span> <span class="identifier">world</span><span class="special">&</span> <span class="identifier">w</span><span class="special">)</span>
178 <span class="special">{</span>
179 <span class="comment">// ...</span>
180 <span class="special">}</span>
182 <span class="keyword">static</span>
183 <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">tuple</span>
184 <span class="identifier">getstate</span><span class="special">(</span><span class="keyword">const</span> <span class="identifier">world</span><span class="special">&</span> <span class="identifier">w</span><span class="special">)</span>
185 <span class="special">{</span>
186 <span class="comment">// ...</span>
187 <span class="special">}</span>
189 <span class="keyword">static</span>
190 <span class="keyword">void</span>
191 <span class="identifier">setstate</span><span class="special">(</span><span class="identifier">world</span><span class="special">&</span> <span class="identifier">w</span><span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">tuple</span> <span class="identifier">state</span><span class="special">)</span>
192 <span class="special">{</span>
193 <span class="comment">// ...</span>
194 <span class="special">}</span>
195 <span class="special">};</span>
198 Establishing the Python bindings for the entire suite:
200 <pre class="programlisting"><span class="identifier">class_</span><span class="special"><</span><span class="identifier">world</span><span class="special">>(</span><span class="string">"world"</span><span class="special">,</span> <span class="identifier">args</span><span class="special"><</span><span class="keyword">const</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">&>())</span>
201 <span class="comment">// ...</span>
202 <span class="special">.</span><span class="identifier">def_pickle</span><span class="special">(</span><span class="identifier">world_pickle_suite</span><span class="special">())</span>
203 <span class="comment">// ...</span>
206 For simplicity, the <code class="computeroutput"><span class="identifier">__dict__</span></code>
207 is not included in the result of <code class="computeroutput"><span class="identifier">__getstate__</span></code>.
208 This is not generally recommended, but a valid approach if it is anticipated
209 that the object's <code class="computeroutput"><span class="identifier">__dict__</span></code>
210 will always be empty. Note that the safety guard described below will
211 catch the cases where this assumption is violated.
214 <div class="section">
215 <div class="titlepage"><div><div><h4 class="title">
216 <a name="topics.pickle_support.example.pickle3_cpp"></a><a class="link" href="pickle_support.html#topics.pickle_support.example.pickle3_cpp" title="pickle3.cpp">pickle3.cpp</a>
217 </h4></div></div></div>
219 This example is similar to pickle2.cpp. However, the object's <code class="computeroutput"><span class="identifier">__dict__</span></code> is included in the result
220 of <code class="computeroutput"><span class="identifier">__getstate__</span></code>. This
221 requires a little more code but is unavoidable if the object's <code class="computeroutput"><span class="identifier">__dict__</span></code> is not always empty.
225 <div class="section">
226 <div class="titlepage"><div><div><h3 class="title">
227 <a name="topics.pickle_support.pitfall_and_safety_guard"></a><a class="link" href="pickle_support.html#topics.pickle_support.pitfall_and_safety_guard" title="Pitfall and Safety Guard">Pitfall
229 </h3></div></div></div>
231 The pickle protocol described above has an important pitfall that the end
232 user of a Boost.Python extension module might not be aware of:
235 <span class="bold"><strong><code class="computeroutput"><span class="identifier">__getstate__</span></code>
236 is defined and the instance's <code class="computeroutput"><span class="identifier">__dict__</span></code>
237 is not empty.</strong></span>
240 The author of a <code class="computeroutput"><span class="identifier">Boost</span><span class="special">.</span><span class="identifier">Python</span></code> extension class might provide
241 a <code class="computeroutput"><span class="identifier">__getstate__</span></code> method without
242 considering the possibilities that: * his class is used in Python as a
243 base class. Most likely the <code class="computeroutput"><span class="identifier">__dict__</span></code>
244 of instances of the derived class needs to be pickled in order to restore
245 the instances correctly. * the user adds items to the instance's <code class="computeroutput"><span class="identifier">__dict__</span></code> directly. Again, the <code class="computeroutput"><span class="identifier">__dict__</span></code> of the instance then needs to
249 To alert the user to this highly unobvious problem, a safety guard is provided.
250 If <code class="computeroutput"><span class="identifier">__getstate__</span></code> is defined
251 and the instance's <code class="computeroutput"><span class="identifier">__dict__</span></code>
252 is not empty, <code class="computeroutput"><span class="identifier">Boost</span><span class="special">.</span><span class="identifier">Python</span></code> tests if the class has an attribute
253 <code class="computeroutput"><span class="identifier">__getstate_manages_dict__</span></code>.
254 An exception is raised if this attribute is not defined:
256 <pre class="programlisting"><span class="identifier">RuntimeError</span><span class="special">:</span> <span class="identifier">Incomplete</span> <span class="identifier">pickle</span> <span class="identifier">support</span> <span class="special">(</span><span class="identifier">__getstate_manages_dict__</span> <span class="keyword">not</span> <span class="identifier">set</span><span class="special">)</span>
259 To resolve this problem, it should first be established that the <code class="computeroutput"><span class="identifier">__getstate__</span></code> and <code class="computeroutput"><span class="identifier">__setstate__</span></code>
260 methods manage the instances's <code class="computeroutput"><span class="identifier">__dict__</span></code>
261 correctly. Note that this can be done either at the C++ or the Python level.
262 Finally, the safety guard should intentionally be overridden. E.g. in C++
265 <pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">world_pickle_suite</span> <span class="special">:</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">pickle_suite</span>
266 <span class="special">{</span>
267 <span class="comment">// ...</span>
269 <span class="keyword">static</span> <span class="keyword">bool</span> <span class="identifier">getstate_manages_dict</span><span class="special">()</span> <span class="special">{</span> <span class="keyword">return</span> <span class="keyword">true</span><span class="special">;</span> <span class="special">}</span>
270 <span class="special">};</span>
273 Alternatively in Python:
275 <pre class="programlisting"><span class="identifier">import</span> <span class="identifier">your_bpl_module</span>
276 <span class="keyword">class</span> <span class="identifier">your_class</span><span class="special">(</span><span class="identifier">your_bpl_module</span><span class="special">.</span><span class="identifier">your_class</span><span class="special">):</span>
277 <span class="identifier">__getstate_manages_dict__</span> <span class="special">=</span> <span class="number">1</span>
278 <span class="identifier">def</span> <span class="identifier">__getstate__</span><span class="special">(</span><span class="identifier">self</span><span class="special">):</span>
279 <span class="preprocessor"># your</span> <span class="identifier">code</span> <span class="identifier">here</span>
280 <span class="identifier">def</span> <span class="identifier">__setstate__</span><span class="special">(</span><span class="identifier">self</span><span class="special">,</span> <span class="identifier">state</span><span class="special">):</span>
281 <span class="preprocessor"># your</span> <span class="identifier">code</span> <span class="identifier">here</span>
284 <div class="section">
285 <div class="titlepage"><div><div><h3 class="title">
286 <a name="topics.pickle_support.practical_advice"></a><a class="link" href="pickle_support.html#topics.pickle_support.practical_advice" title="Practical Advice">Practical Advice</a>
287 </h3></div></div></div>
288 <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
289 <li class="listitem">
290 In <code class="computeroutput"><span class="identifier">Boost</span><span class="special">.</span><span class="identifier">Python</span></code> extension modules with many
291 extension classes, providing complete pickle support for all classes
292 would be a significant overhead. In general complete pickle support
293 should only be implemented for extension classes that will eventually
296 <li class="listitem">
297 Avoid using <code class="computeroutput"><span class="identifier">__getstate__</span></code>
298 if the instance can also be reconstructed by way of <code class="computeroutput"><span class="identifier">__getinitargs__</span></code>.
299 This automatically avoids the pitfall described above.
301 <li class="listitem">
302 If <code class="computeroutput"><span class="identifier">__getstate__</span></code> is
303 required, include the instance's <code class="computeroutput"><span class="identifier">__dict__</span></code>
304 in the Python object that is returned.
308 <div class="section">
309 <div class="titlepage"><div><div><h3 class="title">
310 <a name="topics.pickle_support.light_weight_alternative_pickle_"></a><a class="link" href="pickle_support.html#topics.pickle_support.light_weight_alternative_pickle_" title="Light-weight alternative: pickle support implemented in Python">Light-weight
311 alternative: pickle support implemented in Python</a>
312 </h3></div></div></div>
314 The pickle4.cpp example demonstrates an alternative technique for implementing
315 pickle support. First we direct Boost.Python via the class_::enable_pickling()
316 member function to define only the basic attributes required for pickling:
318 <pre class="programlisting"><span class="identifier">class_</span><span class="special"><</span><span class="identifier">world</span><span class="special">>(</span><span class="string">"world"</span><span class="special">,</span> <span class="identifier">args</span><span class="special"><</span><span class="keyword">const</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">&>())</span>
319 <span class="comment">// ...</span>
320 <span class="special">.</span><span class="identifier">enable_pickling</span><span class="special">()</span>
321 <span class="comment">// ...</span>
324 This enables the standard Python pickle interface as described in the Python
325 documentation. By "injecting" a <code class="computeroutput"><span class="identifier">__getinitargs__</span></code>
326 method into the definition of the wrapped class we make all instances pickleable:
328 <pre class="programlisting"><span class="preprocessor"># import</span> <span class="identifier">the</span> <span class="identifier">wrapped</span> <span class="identifier">world</span> <span class="keyword">class</span>
329 <span class="identifier">from</span> <span class="identifier">pickle4_ext</span> <span class="identifier">import</span> <span class="identifier">world</span>
331 <span class="preprocessor"># definition</span> <span class="identifier">of</span> <span class="identifier">__getinitargs__</span>
332 <span class="identifier">def</span> <span class="identifier">world_getinitargs</span><span class="special">(</span><span class="identifier">self</span><span class="special">):</span>
333 <span class="keyword">return</span> <span class="special">(</span><span class="identifier">self</span><span class="special">.</span><span class="identifier">get_country</span><span class="special">(),)</span>
335 <span class="preprocessor"># now</span> <span class="identifier">inject</span> <span class="identifier">__getinitargs__</span> <span class="special">(</span><span class="identifier">Python</span> <span class="identifier">is</span> <span class="identifier">a</span> <span class="identifier">dynamic</span> <span class="identifier">language</span><span class="special">!)</span>
336 <span class="identifier">world</span><span class="special">.</span><span class="identifier">__getinitargs__</span> <span class="special">=</span> <span class="identifier">world_getinitargs</span>
339 See also the tutorial section on injecting additional methods from Python.
343 <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
344 <td align="left"></td>
345 <td align="right"><div class="copyright-footer">Copyright © 2002-2005, 2015 David Abrahams, Stefan Seefeld<p>
346 Distributed under the Boost Software License, Version 1.0. (See accompanying
347 file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>
352 <div class="spirit-nav">
353 <a accesskey="p" href="../topics.html"><img src="../../images/prev.png" alt="Prev"></a><a accesskey="u" href="../topics.html"><img src="../../images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../images/home.png" alt="Home"></a><a accesskey="n" href="indexing_support.html"><img src="../../images/next.png" alt="Next"></a>