Initial import to Tizen
[profile/ivi/python-twisted.git] / doc / core / howto / defer.html
1 <?xml version="1.0" encoding="utf-8"?><!DOCTYPE html  PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN'  'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html lang="en" xmlns="http://www.w3.org/1999/xhtml">
2   <head>
3 <title>Twisted Documentation: Deferred Reference</title>
4 <link href="stylesheet.css" rel="stylesheet" type="text/css"/>
5   </head>
6
7   <body bgcolor="white">
8     <h1 class="title">Deferred Reference</h1>
9     <div class="toc"><ol><li><a href="#auto0">Deferreds</a></li><li><a href="#auto1">Callbacks</a></li><ul><li><a href="#auto2">Multiple callbacks</a></li><li><a href="#auto3">Visual Explanation</a></li></ul><li><a href="#auto4">Errbacks</a></li><ul><li><a href="#auto5">Unhandled Errors</a></li></ul><li><a href="#auto6">Handling either synchronous or asynchronous results</a></li><ul><li><a href="#auto7">Handling possible Deferreds in the library code</a></li></ul><li><a href="#auto8">DeferredList</a></li><ul><li><a href="#auto9">Other behaviours</a></li><li><a href="#auto10">gatherResults</a></li></ul><li><a href="#auto11">Class Overview</a></li><ul><li><a href="#auto12">Basic Callback Functions</a></li><li><a href="#auto13">Chaining Deferreds</a></li></ul><li><a href="#auto14">See also</a></li></ol></div>
10     <div class="content">
11
12 <span/>
13
14 <p>This document is a guide to the behaviour of the <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferred.html" title="twisted.internet.defer.Deferred">twisted.internet.defer.Deferred</a></code> object, and to various
15 ways you can use them when they are returned by functions.</p>
16
17 <p>This document assumes that you are familiar with the basic principle that
18 the Twisted framework is structured around: asynchronous, callback-based
19 programming, where instead of having blocking code in your program or using
20 threads to run blocking code, you have functions that return immediately and
21 then begin a callback chain when data is available.</p>
22
23 <p>
24 After reading this document, the reader should expect to be able to
25 deal with most simple APIs in Twisted and Twisted-using code that
26 return Deferreds.
27 </p>
28
29 <ul>
30 <li>what sorts of things you can do when you get a Deferred from a
31 function call; and</li>
32 <li>how you can write your code to robustly handle errors in Deferred
33 code.</li>
34 </ul>
35
36 <a name="deferreds" shape="rect"/>
37 <h2>Deferreds<a name="auto0"/></h2>
38
39 <p>Twisted uses the <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferred.html" title="twisted.internet.defer.Deferred">Deferred</a></code> object to manage the callback
40 sequence. The client application attaches a series of functions to the
41 deferred to be called in order when the results of the asychronous request are
42 available (this series of functions is known as a series of 
43 <strong>callbacks</strong>, or a <strong>callback chain</strong>), together
44 with a series of functions to be called if there is an error in the
45 asychronous request (known as a series of <strong>errbacks</strong> or an  <strong>errback chain</strong>). The asychronous library code calls the first
46 callback when the result is available, or the first errback when an error
47 occurs, and the <code>Deferred</code> object then hands the results of each
48 callback or errback function to the next function in the chain.</p> 
49
50 <h2>Callbacks<a name="auto1"/></h2>
51
52 <p>A <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferred.html" title="twisted.internet.defer.Deferred">twisted.internet.defer.Deferred</a></code> is a promise that
53 a function will at some point have a result.  We can attach callback functions
54 to a Deferred, and once it gets a result these callbacks will be called. In
55 addition Deferreds allow the developer to register a callback for an error,
56 with the default behavior of logging the error.  The deferred mechanism 
57 standardizes the application programmer's interface with all sorts of 
58 blocking or delayed operations.</p>
59
60 <pre class="python"><p class="py-linenumber"> 1
61  2
62  3
63  4
64  5
65  6
66  7
67  8
68  9
69 10
70 11
71 12
72 13
73 14
74 15
75 16
76 17
77 18
78 19
79 20
80 21
81 22
82 23
83 24
84 25
85 26
86 27
87 28
88 29
89 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>, <span class="py-src-variable">defer</span>
90
91 <span class="py-src-keyword">def</span> <span class="py-src-identifier">getDummyData</span>(<span class="py-src-parameter">x</span>):
92     <span class="py-src-string">&quot;&quot;&quot;
93     This function is a dummy which simulates a delayed result and
94     returns a Deferred which will fire with that result. Don't try too
95     hard to understand this.
96     &quot;&quot;&quot;</span>
97     <span class="py-src-variable">d</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
98     <span class="py-src-comment"># simulate a delayed result by asking the reactor to fire the</span>
99     <span class="py-src-comment"># Deferred in 2 seconds time with the result x * 3</span>
100     <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callLater</span>(<span class="py-src-number">2</span>, <span class="py-src-variable">d</span>.<span class="py-src-variable">callback</span>, <span class="py-src-variable">x</span> * <span class="py-src-number">3</span>)
101     <span class="py-src-keyword">return</span> <span class="py-src-variable">d</span>
102
103 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printData</span>(<span class="py-src-parameter">d</span>):
104     <span class="py-src-string">&quot;&quot;&quot;
105     Data handling function to be added as a callback: handles the
106     data by printing the result
107     &quot;&quot;&quot;</span>
108     <span class="py-src-keyword">print</span> <span class="py-src-variable">d</span>
109
110 <span class="py-src-variable">d</span> = <span class="py-src-variable">getDummyData</span>(<span class="py-src-number">3</span>)
111 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printData</span>)
112
113 <span class="py-src-comment"># manually set up the end of the process by asking the reactor to</span>
114 <span class="py-src-comment"># stop itself in 4 seconds time</span>
115 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callLater</span>(<span class="py-src-number">4</span>, <span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>)
116 <span class="py-src-comment"># start up the Twisted reactor (event loop handler) manually</span>
117 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
118 </pre>
119
120 <h3>Multiple callbacks<a name="auto2"/></h3>
121
122 <p>Multiple callbacks can be added to a Deferred.  The first callback in the
123 Deferred's callback chain will be called with the result, the second with the
124 result of the first callback, and so on. Why do we need this?  Well, consider
125 a Deferred returned by twisted.enterprise.adbapi - the result of a SQL query.
126 A web widget might add a callback that converts this result into HTML, and
127 pass the Deferred onwards, where the callback will be used by twisted to
128 return the result to the HTTP client. The callback chain will be bypassed in
129 case of errors or exceptions.</p>
130
131 <pre class="python"><p class="py-linenumber"> 1
132  2
133  3
134  4
135  5
136  6
137  7
138  8
139  9
140 10
141 11
142 12
143 13
144 14
145 15
146 16
147 17
148 18
149 19
150 20
151 21
152 22
153 23
154 24
155 25
156 26
157 27
158 28
159 29
160 30
161 31
162 32
163 33
164 34
165 35
166 36
167 37
168 38
169 39
170 40
171 41
172 42
173 43
174 44
175 45
176 46
177 47
178 48
179 49
180 50
181 51
182 52
183 53
184 54
185 55
186 56
187 57
188 58
189 59
190 60
191 61
192 62
193 63
194 64
195 65
196 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>, <span class="py-src-variable">defer</span>
197
198 <span class="py-src-keyword">class</span> <span class="py-src-identifier">Getter</span>:
199     <span class="py-src-keyword">def</span> <span class="py-src-identifier">gotResults</span>(<span class="py-src-parameter">self</span>, <span class="py-src-parameter">x</span>):
200         <span class="py-src-string">&quot;&quot;&quot;
201         The Deferred mechanism provides a mechanism to signal error
202         conditions.  In this case, odd numbers are bad.
203
204         This function demonstrates a more complex way of starting
205         the callback chain by checking for expected results and
206         choosing whether to fire the callback or errback chain
207         &quot;&quot;&quot;</span>
208         <span class="py-src-keyword">if</span> <span class="py-src-variable">x</span> % <span class="py-src-number">2</span> == <span class="py-src-number">0</span>:
209             <span class="py-src-variable">self</span>.<span class="py-src-variable">d</span>.<span class="py-src-variable">callback</span>(<span class="py-src-variable">x</span>*<span class="py-src-number">3</span>)
210         <span class="py-src-keyword">else</span>:
211             <span class="py-src-variable">self</span>.<span class="py-src-variable">d</span>.<span class="py-src-variable">errback</span>(<span class="py-src-variable">ValueError</span>(<span class="py-src-string">&quot;You used an odd number!&quot;</span>))
212
213     <span class="py-src-keyword">def</span> <span class="py-src-identifier">_toHTML</span>(<span class="py-src-parameter">self</span>, <span class="py-src-parameter">r</span>):
214         <span class="py-src-string">&quot;&quot;&quot;
215         This function converts r to HTML.
216
217         It is added to the callback chain by getDummyData in
218         order to demonstrate how a callback passes its own result
219         to the next callback
220         &quot;&quot;&quot;</span>
221         <span class="py-src-keyword">return</span> <span class="py-src-string">&quot;Result: %s&quot;</span> % <span class="py-src-variable">r</span>
222
223     <span class="py-src-keyword">def</span> <span class="py-src-identifier">getDummyData</span>(<span class="py-src-parameter">self</span>, <span class="py-src-parameter">x</span>):
224         <span class="py-src-string">&quot;&quot;&quot;
225         The Deferred mechanism allows for chained callbacks.
226         In this example, the output of gotResults is first
227         passed through _toHTML on its way to printData.
228
229         Again this function is a dummy, simulating a delayed result
230         using callLater, rather than using a real asynchronous
231         setup.
232         &quot;&quot;&quot;</span>
233         <span class="py-src-variable">self</span>.<span class="py-src-variable">d</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
234         <span class="py-src-comment"># simulate a delayed result by asking the reactor to schedule</span>
235         <span class="py-src-comment"># gotResults in 2 seconds time</span>
236         <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callLater</span>(<span class="py-src-number">2</span>, <span class="py-src-variable">self</span>.<span class="py-src-variable">gotResults</span>, <span class="py-src-variable">x</span>)
237         <span class="py-src-variable">self</span>.<span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">self</span>.<span class="py-src-variable">_toHTML</span>)
238         <span class="py-src-keyword">return</span> <span class="py-src-variable">self</span>.<span class="py-src-variable">d</span>
239
240 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printData</span>(<span class="py-src-parameter">d</span>):
241     <span class="py-src-keyword">print</span> <span class="py-src-variable">d</span>
242
243 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printError</span>(<span class="py-src-parameter">failure</span>):
244     <span class="py-src-keyword">import</span> <span class="py-src-variable">sys</span>
245     <span class="py-src-variable">sys</span>.<span class="py-src-variable">stderr</span>.<span class="py-src-variable">write</span>(<span class="py-src-variable">str</span>(<span class="py-src-variable">failure</span>))
246
247 <span class="py-src-comment"># this series of callbacks and errbacks will print an error message</span>
248 <span class="py-src-variable">g</span> = <span class="py-src-variable">Getter</span>()
249 <span class="py-src-variable">d</span> = <span class="py-src-variable">g</span>.<span class="py-src-variable">getDummyData</span>(<span class="py-src-number">3</span>)
250 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printData</span>)
251 <span class="py-src-variable">d</span>.<span class="py-src-variable">addErrback</span>(<span class="py-src-variable">printError</span>)
252
253 <span class="py-src-comment"># this series of callbacks and errbacks will print &quot;Result: 12&quot;</span>
254 <span class="py-src-variable">g</span> = <span class="py-src-variable">Getter</span>()
255 <span class="py-src-variable">d</span> = <span class="py-src-variable">g</span>.<span class="py-src-variable">getDummyData</span>(<span class="py-src-number">4</span>)
256 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printData</span>)
257 <span class="py-src-variable">d</span>.<span class="py-src-variable">addErrback</span>(<span class="py-src-variable">printError</span>)
258
259 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callLater</span>(<span class="py-src-number">4</span>, <span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>)
260 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
261 </pre>
262
263 <h3>Visual Explanation<a name="auto3"/></h3>
264
265 <div align="center" hlint="off">
266 <img src="../img/deferred-attach.png"/>
267 </div>
268
269 <ol>
270   <li>Requesting method (data sink) requests data, gets
271   Deferred object.</li>
272   
273   <li>Requesting method attaches callbacks to Deferred
274   object.</li>
275 </ol>
276 <img align="left" hlint="off" src="../img/deferred-process.png"/> 
277
278 <ol>
279   
280   <li>When the result is ready, give it to the Deferred
281   object. <code>.callback(result)</code> if the operation succeeded,
282   <code>.errback(failure)</code> if it failed. Note that
283   <code>failure</code> is typically an instance of a <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.python.failure.Failure.html" title="twisted.python.failure.Failure">twisted.python.failure.Failure</a></code>
284   instance.</li>
285   
286   <li>Deferred object triggers previously-added (call/err)back
287   with the <code>result</code> or <code>failure</code>.
288   Execution then follows the following rules, going down the
289   chain of callbacks to be processed. 
290   
291   <ul>
292     <li>Result of the callback is always passed as the first
293     argument to the next callback, creating a chain of
294     processors.</li>
295     
296     <li>If a callback raises an exception, switch to
297     errback.</li>
298     
299     <li>An unhandled failure gets passed down the line of
300     errbacks, this creating an asynchronous analog to a
301     series to a series of <code>except:</code>
302     statements.</li>
303     
304     <li>If an errback doesn't raise an exception or return a
305     <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.python.failure.Failure.html" title="twisted.python.failure.Failure">twisted.python.failure.Failure</a></code>
306     instance, switch to callback.</li>
307   </ul> </li>
308 </ol>
309 <br clear="all" hlint="off"/>
310
311 <h2>Errbacks<a name="auto4"/></h2>
312
313 <p>Deferred's error handling is modeled after Python's
314 exception handling. In the case that no errors occur, all the
315 callbacks run, one after the other, as described above.</p>
316
317 <p>If the errback is called instead of the callback (e.g.  because a DB query
318 raised an error), then a <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.python.failure.Failure.html" title="twisted.python.failure.Failure">twisted.python.failure.Failure</a></code> is passed into the first
319 errback (you can add multiple errbacks, just like with callbacks). You can
320 think of your errbacks as being like <code class="python">except</code> blocks
321 of ordinary Python code.</p>
322
323 <p>Unless you explicitly <code class="python">raise</code> an error in except
324 block, the <code class="python">Exception</code> is caught and stops
325 propagating, and normal execution continues. The same thing happens with
326 errbacks: unless you explicitly <code class="python">return</code> a <code class="python">Failure</code> or (re-)raise an exception, the error stops
327 propagating, and normal callbacks continue executing from that point (using the
328 value returned from the errback). If the errback does returns a <code class="python">Failure</code> or raise an exception, then that is passed to the
329 next errback, and so on.</p>
330
331 <p><em>Note:</em> If an errback doesn't return anything, then it effectively
332 returns <code class="python">None</code>, meaning that callbacks will continue
333 to be executed after this errback.  This may not be what you expect to happen,
334 so be careful. Make sure your errbacks return a <code class="python">Failure</code> (probably the one that was passed to it), or a
335 meaningful return value for the next callback.</p>
336
337 <p>Also, <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.python.failure.Failure.html" title="twisted.python.failure.Failure">twisted.python.failure.Failure</a></code> instances have
338 a useful method called trap, allowing you to effectively do the equivalent
339 of:</p>
340
341 <pre class="python"><p class="py-linenumber">1
342 2
343 3
344 4
345 5
346 6
347 </p><span class="py-src-keyword">try</span>:
348     <span class="py-src-comment"># code that may throw an exception</span>
349     <span class="py-src-variable">cookSpamAndEggs</span>()
350 <span class="py-src-keyword">except</span> (<span class="py-src-variable">SpamException</span>, <span class="py-src-variable">EggException</span>):
351     <span class="py-src-comment"># Handle SpamExceptions and EggExceptions</span>
352     ...
353 </pre>
354
355 <p>You do this by:</p>
356 <pre class="python"><p class="py-linenumber">1
357 2
358 3
359 4
360 5
361 6
362 </p><span class="py-src-keyword">def</span> <span class="py-src-identifier">errorHandler</span>(<span class="py-src-parameter">failure</span>):
363     <span class="py-src-variable">failure</span>.<span class="py-src-variable">trap</span>(<span class="py-src-variable">SpamException</span>, <span class="py-src-variable">EggException</span>)
364     <span class="py-src-comment"># Handle SpamExceptions and EggExceptions</span>
365
366 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">cookSpamAndEggs</span>)
367 <span class="py-src-variable">d</span>.<span class="py-src-variable">addErrback</span>(<span class="py-src-variable">errorHandler</span>)
368 </pre>
369
370 <p>If none of arguments passed to <code class="python">failure.trap</code>
371 match the error encapsulated in that <code class="python">Failure</code>, then
372 it re-raises the error.</p>
373
374 <p>There's another potential <q>gotcha</q> here.  There's a
375 method <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferred.addCallbacks.html" title="twisted.internet.defer.Deferred.addCallbacks">twisted.internet.defer.Deferred.addCallbacks</a></code>
376 which is similar to, but not exactly the same as, <code class="python">addCallback</code> followed by <code class="python">addErrback</code>. In particular, consider these two cases:</p>
377
378 <pre class="python"><p class="py-linenumber"> 1
379  2
380  3
381  4
382  5
383  6
384  7
385  8
386  9
387 10
388 11
389 </p><span class="py-src-comment"># Case 1</span>
390 <span class="py-src-variable">d</span> = <span class="py-src-variable">getDeferredFromSomewhere</span>()
391 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">callback1</span>)       <span class="py-src-comment"># A</span>
392 <span class="py-src-variable">d</span>.<span class="py-src-variable">addErrback</span>(<span class="py-src-variable">errback1</span>)         <span class="py-src-comment"># B</span>
393 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">callback2</span>)       
394 <span class="py-src-variable">d</span>.<span class="py-src-variable">addErrback</span>(<span class="py-src-variable">errback2</span>)        
395
396 <span class="py-src-comment"># Case 2</span>
397 <span class="py-src-variable">d</span> = <span class="py-src-variable">getDeferredFromSomewhere</span>()
398 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallbacks</span>(<span class="py-src-variable">callback1</span>, <span class="py-src-variable">errback1</span>)  <span class="py-src-comment"># C</span>
399 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallbacks</span>(<span class="py-src-variable">callback2</span>, <span class="py-src-variable">errback2</span>)
400 </pre>
401
402 <p>If an error occurs in <code class="python">callback1</code>, then for Case 1 
403 <code class="python">errback1</code> will be called with the failure. For Case
404 2, <code class="python">errback2</code> will be called. Be careful with your
405 callbacks and errbacks.</p>
406
407 <p>What this means in a practical sense is in Case 1, the callback in line
408 A will handle a success condition from <code>getDeferredFromSomewhere</code>,
409 and the errback in line B will handle any errors that occur <em>from either the
410 upstream source, or that occur in A</em>.  In Case 2, the errback in line C  <em>will 
411 only handle an error condition raised by</em> <code>getDeferredFromSomewhere</code>, 
412 it will not do any handling of errors
413 raised in <code>callback1</code>.</p>
414
415
416 <h3>Unhandled Errors<a name="auto5"/></h3>
417
418 <p>If a Deferred is garbage-collected with an unhandled error (i.e. it would
419 call the next errback if there was one), then Twisted will write the error's
420 traceback to the log file.  This means that you can typically get away with not
421 adding errbacks and still get errors logged.  Be careful though; if you keep a
422 reference to the Deferred around, preventing it from being garbage-collected,
423 then you may never see the error (and your callbacks will mysteriously seem to
424 have never been called).  If unsure, you should explicitly add an errback after
425 your callbacks, even if all you do is:</p>
426
427 <pre class="python"><p class="py-linenumber">1
428 2
429 3
430 </p><span class="py-src-comment"># Make sure errors get logged</span>
431 <span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">python</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">log</span>
432 <span class="py-src-variable">d</span>.<span class="py-src-variable">addErrback</span>(<span class="py-src-variable">log</span>.<span class="py-src-variable">err</span>)
433 </pre>
434
435 <h2>Handling either synchronous or asynchronous results<a name="auto6"/></h2>
436 <p>
437 In some applications, there are functions that might be either asynchronous or
438 synchronous. For example, a user authentication function might be able to
439 check in memory whether a user is authenticated, allowing the authentication
440 function to return an immediate result, or it may need to wait on
441 network data, in which case it should return a Deferred to be fired
442 when that data arrives. However, a function that wants to check if a user is
443 authenticated will then need to accept both immediate results <em> and</em>
444 Deferreds.
445 </p>
446
447 <p>
448 In this example, the library function <code>authenticateUser</code> uses the
449 application function <code>isValidUser</code> to authenticate a user:
450 </p>
451
452 <pre class="python"><p class="py-linenumber">1
453 2
454 3
455 4
456 5
457 </p><span class="py-src-keyword">def</span> <span class="py-src-identifier">authenticateUser</span>(<span class="py-src-parameter">isValidUser</span>, <span class="py-src-parameter">user</span>):
458     <span class="py-src-keyword">if</span> <span class="py-src-variable">isValidUser</span>(<span class="py-src-variable">user</span>):
459         <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;User is authenticated&quot;</span>
460     <span class="py-src-keyword">else</span>:
461         <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;User is not authenticated&quot;</span>
462 </pre>
463
464 <p>
465 However, it assumes that <code>isValidUser</code> returns immediately,
466 whereas <code>isValidUser</code> may actually authenticate the user
467 asynchronously and return a Deferred. It is possible to adapt this
468 trivial user authentication code to accept either a
469 synchronous <code>isValidUser</code> or an
470 asynchronous <code>isValidUser</code>, allowing the library to handle
471 either type of function. It is, however, also possible to adapt
472 synchronous functions to return Deferreds. This section describes both
473 alternatives: handling functions that might be synchronous or
474 asynchronous in the library function (<code>authenticateUser</code>)
475 or in the application code.
476 </p>
477
478 <h3>Handling possible Deferreds in the library code<a name="auto7"/></h3>
479
480 <p>
481 Here is an example of a synchronous user authentication function that might be
482 passed to <code>authenticateUser</code>:
483 </p>
484
485 <div class="py-listing"><pre><p class="py-linenumber">1
486 2
487 3
488 4
489 5
490 </p><span class="py-src-keyword">def</span> <span class="py-src-identifier">synchronousIsValidUser</span>(<span class="py-src-parameter">user</span>):
491     <span class="py-src-string">'''
492     Return true if user is a valid user, false otherwise
493     '''</span>
494     <span class="py-src-keyword">return</span> <span class="py-src-variable">user</span> <span class="py-src-keyword">in</span> [<span class="py-src-string">&quot;Alice&quot;</span>, <span class="py-src-string">&quot;Angus&quot;</span>, <span class="py-src-string">&quot;Agnes&quot;</span>]
495 </pre><div class="caption">Source listing - <a href="listings/deferred/synch-validation.py"><span class="filename">listings/deferred/synch-validation.py</span></a></div></div>
496
497 <p>
498 However, here's an <code>asynchronousIsValidUser</code> function that returns
499 a Deferred:
500 </p>
501
502 <pre class="python"><p class="py-linenumber">1
503 2
504 3
505 4
506 5
507 6
508 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>, <span class="py-src-variable">defer</span>
509
510 <span class="py-src-keyword">def</span> <span class="py-src-identifier">asynchronousIsValidUser</span>(<span class="py-src-parameter">user</span>):
511     <span class="py-src-variable">d</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
512     <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callLater</span>(<span class="py-src-number">2</span>, <span class="py-src-variable">d</span>.<span class="py-src-variable">callback</span>, <span class="py-src-variable">user</span> <span class="py-src-keyword">in</span> [<span class="py-src-string">&quot;Alice&quot;</span>, <span class="py-src-string">&quot;Angus&quot;</span>, <span class="py-src-string">&quot;Agnes&quot;</span>])
513     <span class="py-src-keyword">return</span> <span class="py-src-variable">d</span>
514 </pre>
515
516 <p> Our original implementation of <code>authenticateUser</code> expected  <code>isValidUser</code> to be synchronous, but now we need to change it to handle both
517 synchronous and asynchronous implementations of <code>isValidUser</code>. For this, we
518 use <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.maybeDeferred.html" title="twisted.internet.defer.maybeDeferred">maybeDeferred</a></code> to
519 call <code>isValidUser</code>, ensuring that the result of <code>isValidUser</code> is a Deferred,
520 even if <code>isValidUser</code> is a synchronous function:
521 </p>
522
523 <pre class="python"><p class="py-linenumber"> 1
524  2
525  3
526  4
527  5
528  6
529  7
530  8
531  9
532 10
533 11
534 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">defer</span>
535
536 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printResult</span>(<span class="py-src-parameter">result</span>):
537     <span class="py-src-keyword">if</span> <span class="py-src-variable">result</span>:
538         <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;User is authenticated&quot;</span>
539     <span class="py-src-keyword">else</span>:
540         <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;User is not authenticated&quot;</span>
541
542 <span class="py-src-keyword">def</span> <span class="py-src-identifier">authenticateUser</span>(<span class="py-src-parameter">isValidUser</span>, <span class="py-src-parameter">user</span>):
543     <span class="py-src-variable">d</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">maybeDeferred</span>(<span class="py-src-variable">isValidUser</span>, <span class="py-src-variable">user</span>)
544     <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printResult</span>)
545 </pre>
546
547 <p>
548 Now <code>isValidUser</code> could be either <code>synchronousIsValidUser</code> or  <code>asynchronousIsValidUser</code>.
549 </p>
550
551 <p>It is also possible to modify <code>synchronousIsValidUser</code> to return
552 a Deferred, see <a href="gendefer.html" shape="rect">Generating Deferreds</a> for more
553 information.</p>
554
555 <a name="deferredlist" shape="rect"/>
556 <h2>DeferredList<a name="auto8"/></h2>
557
558 <p>Sometimes you want to be notified after several different events have all
559 happened, rather than waiting for each one individually.  For example, you may
560 want to wait for all the connections in a list to close.  <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.DeferredList.html" title="twisted.internet.defer.DeferredList">twisted.internet.defer.DeferredList</a></code> is the way to do
561 this.</p>
562
563 <p>To create a DeferredList from multiple Deferreds, you simply pass a list of
564 the Deferreds you want it to wait for:</p>
565 <pre class="python"><p class="py-linenumber">1
566 2
567 </p><span class="py-src-comment"># Creates a DeferredList</span>
568 <span class="py-src-variable">dl</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">DeferredList</span>([<span class="py-src-variable">deferred1</span>, <span class="py-src-variable">deferred2</span>, <span class="py-src-variable">deferred3</span>])
569 </pre>
570
571 <p>You can now treat the DeferredList like an ordinary Deferred; you can call  <code>addCallbacks</code> and so on.  The DeferredList will call its callback
572 when all the deferreds have completed.  The callback will be called with a list
573 of the results of the Deferreds it contains, like so:</p>
574
575 <pre class="python"><p class="py-linenumber"> 1
576  2
577  3
578  4
579  5
580  6
581  7
582  8
583  9
584 10
585 11
586 12
587 13
588 14
589 15
590 16
591 17
592 18
593 19
594 20
595 21
596 22
597 23
598 24
599 25
600 26
601 27
602 28
603 29
604 </p><span class="py-src-comment"># A callback that unpacks and prints the results of a DeferredList</span>
605 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printResult</span>(<span class="py-src-parameter">result</span>):
606     <span class="py-src-keyword">for</span> (<span class="py-src-variable">success</span>, <span class="py-src-variable">value</span>) <span class="py-src-keyword">in</span> <span class="py-src-variable">result</span>:
607         <span class="py-src-keyword">if</span> <span class="py-src-variable">success</span>:
608             <span class="py-src-keyword">print</span> <span class="py-src-string">'Success:'</span>, <span class="py-src-variable">value</span>
609         <span class="py-src-keyword">else</span>:
610             <span class="py-src-keyword">print</span> <span class="py-src-string">'Failure:'</span>, <span class="py-src-variable">value</span>.<span class="py-src-variable">getErrorMessage</span>()
611
612 <span class="py-src-comment"># Create three deferreds.</span>
613 <span class="py-src-variable">deferred1</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
614 <span class="py-src-variable">deferred2</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
615 <span class="py-src-variable">deferred3</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
616
617 <span class="py-src-comment"># Pack them into a DeferredList</span>
618 <span class="py-src-variable">dl</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">DeferredList</span>([<span class="py-src-variable">deferred1</span>, <span class="py-src-variable">deferred2</span>, <span class="py-src-variable">deferred3</span>], <span class="py-src-variable">consumeErrors</span>=<span class="py-src-variable">True</span>)
619
620 <span class="py-src-comment"># Add our callback</span>
621 <span class="py-src-variable">dl</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printResult</span>)
622
623 <span class="py-src-comment"># Fire our three deferreds with various values.</span>
624 <span class="py-src-variable">deferred1</span>.<span class="py-src-variable">callback</span>(<span class="py-src-string">'one'</span>)
625 <span class="py-src-variable">deferred2</span>.<span class="py-src-variable">errback</span>(<span class="py-src-variable">Exception</span>(<span class="py-src-string">'bang!'</span>))
626 <span class="py-src-variable">deferred3</span>.<span class="py-src-variable">callback</span>(<span class="py-src-string">'three'</span>)
627
628 <span class="py-src-comment"># At this point, dl will fire its callback, printing:</span>
629 <span class="py-src-comment">#    Success: one</span>
630 <span class="py-src-comment">#    Failure: bang!</span>
631 <span class="py-src-comment">#    Success: three</span>
632 <span class="py-src-comment"># (note that defer.SUCCESS == True, and defer.FAILURE == False)</span>
633 </pre>
634
635 <p>A standard DeferredList will never call errback, but failures in Deferreds
636 passed to a DeferredList will still errback unless <code>consumeErrors</code>
637 is passed <code>True</code>.  See below for more details about this and other
638 flags which modify the behavior of DeferredList.</p>
639
640 <div class="note"><strong>Note: </strong>
641 <p>If you want to apply callbacks to the individual Deferreds that
642 go into the DeferredList, you should be careful about when those callbacks
643 are added. The act of adding a Deferred to a DeferredList inserts a callback
644 into that Deferred (when that callback is run, it checks to see if the
645 DeferredList has been completed yet). The important thing to remember is
646 that it is <em>this callback</em> which records the value that goes into the
647 result list handed to the DeferredList's callback.</p>
648
649
650
651 <p>Therefore, if you add a callback to the Deferred <em>after</em> adding the
652 Deferred to the DeferredList, the value returned by that callback will not
653 be given to the DeferredList's callback.  To avoid confusion, we recommend not
654 adding callbacks to a Deferred once it has been used in a DeferredList.</p>
655 </div>
656
657 <pre class="python"><p class="py-linenumber"> 1
658  2
659  3
660  4
661  5
662  6
663  7
664  8
665  9
666 10
667 11
668 12
669 13
670 14
671 15
672 16
673 17
674 18
675 19
676 20
677 21
678 22
679 23
680 24
681 25
682 26
683 </p><span class="py-src-keyword">def</span> <span class="py-src-identifier">printResult</span>(<span class="py-src-parameter">result</span>):
684     <span class="py-src-keyword">print</span> <span class="py-src-variable">result</span>
685 <span class="py-src-keyword">def</span> <span class="py-src-identifier">addTen</span>(<span class="py-src-parameter">result</span>):
686     <span class="py-src-keyword">return</span> <span class="py-src-variable">result</span> + <span class="py-src-string">&quot; ten&quot;</span>
687
688 <span class="py-src-comment"># Deferred gets callback before DeferredList is created</span>
689 <span class="py-src-variable">deferred1</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
690 <span class="py-src-variable">deferred2</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
691 <span class="py-src-variable">deferred1</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">addTen</span>)
692 <span class="py-src-variable">dl</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">DeferredList</span>([<span class="py-src-variable">deferred1</span>, <span class="py-src-variable">deferred2</span>])
693 <span class="py-src-variable">dl</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printResult</span>)
694 <span class="py-src-variable">deferred1</span>.<span class="py-src-variable">callback</span>(<span class="py-src-string">&quot;one&quot;</span>) <span class="py-src-comment"># fires addTen, checks DeferredList, stores &quot;one ten&quot;</span>
695 <span class="py-src-variable">deferred2</span>.<span class="py-src-variable">callback</span>(<span class="py-src-string">&quot;two&quot;</span>)
696 <span class="py-src-comment"># At this point, dl will fire its callback, printing:</span>
697 <span class="py-src-comment">#     [(1, 'one ten'), (1, 'two')]</span>
698
699 <span class="py-src-comment"># Deferred gets callback after DeferredList is created</span>
700 <span class="py-src-variable">deferred1</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
701 <span class="py-src-variable">deferred2</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
702 <span class="py-src-variable">dl</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">DeferredList</span>([<span class="py-src-variable">deferred1</span>, <span class="py-src-variable">deferred2</span>])
703 <span class="py-src-variable">deferred1</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">addTen</span>) <span class="py-src-comment"># will run *after* DeferredList gets its value</span>
704 <span class="py-src-variable">dl</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printResult</span>)
705 <span class="py-src-variable">deferred1</span>.<span class="py-src-variable">callback</span>(<span class="py-src-string">&quot;one&quot;</span>) <span class="py-src-comment"># checks DeferredList, stores &quot;one&quot;, fires addTen</span>
706 <span class="py-src-variable">deferred2</span>.<span class="py-src-variable">callback</span>(<span class="py-src-string">&quot;two&quot;</span>)
707 <span class="py-src-comment"># At this point, dl will fire its callback, printing:</span>
708 <span class="py-src-comment">#     [(1, 'one), (1, 'two')]</span>
709 </pre>
710
711 <h3>Other behaviours<a name="auto9"/></h3>
712
713 <p>DeferredList accepts three keyword arguments that modify its behaviour:
714 <code>fireOnOneCallback</code>, <code>fireOnOneErrback</code> and 
715 <code>consumeErrors</code>.  If <code>fireOnOneCallback</code> is set, the
716 DeferredList will immediately call its callback as soon as any of its Deferreds
717 call their callback.  Similarly, <code>fireOnOneErrback</code> will call errback
718 as soon as any of the Deferreds call their errback.  Note that DeferredList is
719 still one-shot, like ordinary Deferreds, so after a callback or errback has been
720 called the DeferredList will do nothing further (it will just silently ignore
721 any other results from its Deferreds).</p>
722
723 <p>The <code>fireOnOneErrback</code> option is particularly useful when you
724 want to wait for all the results if everything succeeds, but also want to know
725 immediately if something fails.</p>
726
727 <p>The <code>consumeErrors</code> argument will stop the DeferredList from
728 propagating any errors along the callback chains of any Deferreds it contains
729 (usually creating a DeferredList has no effect on the results passed along the
730 callbacks and errbacks of their Deferreds).  Stopping errors at the DeferredList
731 with this option will prevent <q>Unhandled error in Deferred</q> warnings from
732 the Deferreds it contains without needing to add extra errbacks<a href="#footnote-1" title="Unless of course a later callback starts a fresh error — but as we've already noted, adding callbacks to a Deferred after its used in a DeferredList is confusing and usually avoided."><super>1</super></a>.  Passing a true value
733 for the <code>consumeErrors</code> parameter will not change the behavior of <code>fireOnOneCallback</code> or <code>fireOnOneErrback</code>.</p>
734
735 <h3>gatherResults<a name="auto10"/></h3>
736
737 <p>A common use for DeferredList is to &quot;join&quot; a number of parallel asynchronous
738 operations, finishing successfully if all of the operations were successful, or
739 failing if any one of the operations fails.  In this case, <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.gatherResults.html" title="twisted.internet.defer.gatherResults">twisted.internet.defer.gatherResults</a></code> is a useful
740 shortcut:</p>
741
742 <pre class="python"><p class="py-linenumber"> 1
743  2
744  3
745  4
746  5
747  6
748  7
749  8
750  9
751 10
752 11
753 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">defer</span>
754 <span class="py-src-variable">d1</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
755 <span class="py-src-variable">d2</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
756 <span class="py-src-variable">d</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">gatherResults</span>([<span class="py-src-variable">d1</span>, <span class="py-src-variable">d2</span>], <span class="py-src-variable">consumeErrors</span>=<span class="py-src-variable">True</span>)
757 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printResult</span>(<span class="py-src-parameter">result</span>):
758     <span class="py-src-keyword">print</span> <span class="py-src-variable">result</span>
759 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printResult</span>)
760 <span class="py-src-variable">d1</span>.<span class="py-src-variable">callback</span>(<span class="py-src-string">&quot;one&quot;</span>)
761 <span class="py-src-comment"># nothing is printed yet; d is still awaiting completion of d2</span>
762 <span class="py-src-variable">d2</span>.<span class="py-src-variable">callback</span>(<span class="py-src-string">&quot;two&quot;</span>)
763 <span class="py-src-comment"># printResult prints [&quot;one&quot;, &quot;two&quot;]</span>
764 </pre>
765
766 <p>The <code>consumeErrors</code> argument has the same meaning as it does
767 for <a href="#deferredlist" shape="rect"><code>DeferredList</code></a>: if true, it causes
768 <code>gatherResults</code> to consume any errors in the passed-in Deferreds.
769 Always use this argument unless you are adding further callbacks or errbacks to
770 the passed-in Deferreds, or unless you know that they will not fail.
771 Otherwise, a failure will result in an unhandled error being logged by Twisted.
772 This argument is available since Twisted 11.1.0.</p>
773
774 <a name="class" shape="rect"/>
775
776 <h2>Class Overview<a name="auto11"/></h2>
777
778 <p>This is an overview API reference for Deferred from the point of using a
779 Deferred returned by a function. It is not meant to be a
780 substitute for the docstrings in the Deferred class, but can provide guidelines
781 for its use.</p>
782
783 <p>There is a parallel overview of functions used by the Deferred's  <em>creator</em> in <a href="gendefer.html#class" shape="rect">Generating Deferreds</a>.</p>
784
785 <h3>Basic Callback Functions<a name="auto12"/></h3>
786
787 <ul>
788   <li>
789   <code class="py-prototype">addCallbacks(self, callback[, errback, callbackArgs,
790   callbackKeywords, errbackArgs, errbackKeywords])</code> 
791   
792   <p>This is the method you will use to interact
793   with Deferred. It adds a pair of callbacks <q>parallel</q> to
794   each other (see diagram above) in the list of callbacks
795   made when the Deferred is called back to. The signature of
796   a method added using addCallbacks should be
797   <code>myMethod(result, *methodArgs,
798   **methodKeywords)</code>. If your method is passed in the
799   callback slot, for example, all arguments in the tuple
800   <code>callbackArgs</code> will be passed as
801   <code>*methodArgs</code> to your method.</p>
802   
803   <p>There are various convenience methods that are
804   derivative of addCallbacks. I will not cover them in detail
805   here, but it is important to know about them in order to
806   create concise code.</p>
807   
808   <ul>
809     <li>
810     <code class="py-prototype">addCallback(callback, *callbackArgs,
811     **callbackKeywords)</code> 
812     
813     <p>Adds your callback at the next point in the
814     processing chain, while adding an errback that will
815     re-raise its first argument, not affecting further
816     processing in the error case.</p>
817
818     <p>Note that, while addCallbacks (plural) requires the arguments to be
819     passed in a tuple, addCallback (singular) takes all its remaining
820     arguments as things to be passed to the callback function. The reason is
821     obvious: addCallbacks (plural) cannot tell whether the arguments are
822     meant for the callback or the errback, so they must be specifically
823     marked by putting them into a tuple. addCallback (singular) knows that
824     everything is destined to go to the callback, so it can use Python's
825     <q>*</q> and <q>**</q> syntax to collect the remaining arguments.</p>
826     
827     </li>
828     
829     <li>
830     <code class="py-prototype">addErrback(errback, *errbackArgs,
831     **errbackKeywords)</code> 
832     
833     <p>Adds your errback at the next point in the
834     processing chain, while adding a callback that will
835     return its first argument, not affecting further
836     processing in the success case.</p>
837     </li>
838     
839     <li>
840     <code class="py-prototype">addBoth(callbackOrErrback,
841     *callbackOrErrbackArgs,
842     **callbackOrErrbackKeywords)</code> 
843     
844     <p>This method adds the same callback into both sides
845     of the processing chain at both points. Keep in mind
846     that the type of the first argument is indeterminate if
847     you use this method! Use it for <code>finally:</code>
848     style blocks.</p>
849     </li>
850   </ul> </li>
851   
852 </ul>
853
854
855 <h3>Chaining Deferreds<a name="auto13"/></h3>
856
857 <p>If you need one Deferred to wait on another, all you need to do is return a
858 Deferred from a method added to addCallbacks.  Specifically, if you return
859 Deferred B from a method added to Deferred A using A.addCallbacks, Deferred A's
860 processing chain will stop until Deferred B's .callback() method is called; at
861 that point, the next callback in A will be passed the result of the last
862 callback in Deferred B's processing chain at the time.</p>
863
864 <p>If this seems confusing, don't worry about it right now -- when you run into
865 a situation where you need this behavior, you will probably recognize it
866 immediately and realize why this happens.  If you want to chain deferreds
867 manually, there is also a convenience method to help you.</p>
868
869 <ul>
870   <li>
871   <code class="py-prototype">chainDeferred(otherDeferred)</code> 
872   
873   <p>Add <code>otherDeferred</code> to the end of this
874   Deferred's processing chain. When self.callback is called,
875   the result of my processing chain up to this point will be
876   passed to <code>otherDeferred.callback</code>. Further
877   additions to my callback chain do not affect
878   <code>otherDeferred</code></p>
879   <p>This is the same as <code class="python">self.addCallbacks(otherDeferred.callback,
880   otherDeferred.errback)</code></p>
881   </li>
882 </ul>
883     
884 <h2>See also<a name="auto14"/></h2>
885
886 <ol>
887 <li><a href="gendefer.html" shape="rect">Generating Deferreds</a>, an introduction to
888 writing asynchronous functions that return Deferreds.</li>
889 </ol>
890
891 <h2>Footnotes</h2><ol><li><a name="footnote-1"><span class="footnote">Unless of course a later callback starts a fresh error —
892 but as we've already noted, adding callbacks to a Deferred after its used in a
893 DeferredList is confusing and usually avoided.</span></a></li></ol></div>
894
895     <p><a href="index.html">Index</a></p>
896     <span class="version">Version: 12.1.0</span>
897   </body>
898 </html>