doc/core/howto/gendefer.html

   1 <?xml version="1.0" encoding="utf-8"?><!DOCTYPE html  PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN'  'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html lang="en" xmlns="http://www.w3.org/1999/xhtml">
   2   <head>
   3 <title>Twisted Documentation: Generating Deferreds</title>
   4 <link href="stylesheet.css" rel="stylesheet" type="text/css"/>
   5   </head>
   6
   7   <body bgcolor="white">
   8     <h1 class="title">Generating Deferreds</h1>
   9     <div class="toc"><ol><li><a href="#auto0">Class overview</a></li><ul><li><a href="#auto1">Basic Callback Functions</a></li></ul><li><a href="#auto2">What Deferreds don't do: make your code asynchronous</a></li><li><a href="#auto3">Advanced Processing Chain Control</a></li><li><a href="#auto4">Returning Deferreds from synchronous functions</a></li><li><a href="#auto5">Integrating blocking code with Twisted</a></li><li><a href="#auto6">Possible sources of error</a></li><ul><li><a href="#auto7">Firing Deferreds more than once is impossible</a></li><li><a href="#auto8">Synchronous callback execution</a></li></ul></ol></div>
  10     <div class="content">
  11
  12 <span/>
  13
  14 <p><code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferred.html" title="twisted.internet.defer.Deferred">Deferred</a></code> objects are
  15 signals that a function you have called does not yet have the data you want
  16 available. When a function returns a Deferred object, your calling function
  17 attaches callbacks to it to handle the data when available.</p>
  18
  19 <p>This document addresses the other half of the question: writing functions
  20 that return Deferreds, that is, constructing Deferred objects, arranging for
  21 them to be returned immediately without blocking until data is available, and
  22 firing their callbacks when the data is available.</p>
  23
  24 <p>This document assumes that you are familiar with the asynchronous model used
  25 by Twisted, and with <a href="defer.html" shape="rect">using deferreds returned by functions</a>
  26 .</p>
  27
  28 <a name="class" shape="rect"/>
  29
  30 <h2>Class overview<a name="auto0"/></h2>
  31
  32 <p>This is an overview API reference for Deferred from the point of creating a
  33 Deferred and firing its callbacks and errbacks.  It is not meant to be a
  34 substitute for the docstrings in the Deferred class, but can provide
  35 guidelines for its use.</p>
  36
  37 <p>There is a parallel overview of functions used by calling function which
  38 the Deferred is returned to at <a href="defer.html#class" shape="rect">Using Deferreds</a>.</p>
  39
  40 <h3>Basic Callback Functions<a name="auto1"/></h3>
  41
  42 <ul>
  43   <li>
  44   <code class="py-prototype">callback(result)</code>
  45
  46   <p>Run success callbacks with the given result. <em>This
  47   can only be run once.</em> Later calls to this or
  48   <code>errback</code> will raise <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.AlreadyCalledError.html" title="twisted.internet.defer.AlreadyCalledError">twisted.internet.defer.AlreadyCalledError</a></code>.
  49   If further callbacks or errbacks are added after this
  50   point, addCallbacks will run the callbacks immediately.</p>
  51   </li>
  52
  53   <li>
  54   <code class="py-prototype">errback(failure)</code>
  55
  56   <p>Run error callbacks with the given failure. <em>This can
  57   only be run once.</em> Later calls to this or
  58   <code>callback</code> will raise <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.AlreadyCalledError.html" title="twisted.internet.defer.AlreadyCalledError">twisted.internet.defer.AlreadyCalledError</a></code>.
  59   If further callbacks or errbacks are added after this
  60   point, addCallbacks will run the callbacks immediately.</p>
  61   </li>
  62 </ul>
  63
  64 <h2>What Deferreds don't do: make your code asynchronous<a name="auto2"/></h2>
  65
  66 <p><em>Deferreds do not make the code magically not block.</em></p>
  67
  68 <p>Let's take this function as an example:</p>
  69
  70 <pre class="python"><p class="py-linenumber"> 1
  71  2
  72  3
  73  4
  74  5
  75  6
  76  7
  77  8
  78  9
  79 10
  80 11
  81 12
  82 13
  83 14
  84 15
  85 16
  86 17
  87 18
  88 19
  89 20
  90 21
  91 22
  92 23
  93 24
  94 25
  95 26
  96 27
  97 28
  98 29
  99 30
 100 31
 101 32
 102 33
 103 34
 104 35
 105 36
 106 37
 107 38
 108 39
 109 40
 110 41
 111 42
 112 43
 113 44
 114 45
 115 46
 116 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">defer</span>
 117
 118 <span class="py-src-variable">TARGET</span> = <span class="py-src-number">10000</span>
 119
 120 <span class="py-src-keyword">def</span> <span class="py-src-identifier">largeFibonnaciNumber</span>():
 121     <span class="py-src-comment"># create a Deferred object to return:</span>
 122     <span class="py-src-variable">d</span> = <span class="py-src-variable">defer</span>.<span class="py-src-variable">Deferred</span>()
 123
 124     <span class="py-src-comment"># calculate the ten thousandth Fibonnaci number</span>
 125
 126     <span class="py-src-variable">first</span> = <span class="py-src-number">0</span>
 127     <span class="py-src-variable">second</span> = <span class="py-src-number">1</span>
 128
 129     <span class="py-src-keyword">for</span> <span class="py-src-variable">i</span> <span class="py-src-keyword">in</span> <span class="py-src-variable">xrange</span>(<span class="py-src-variable">TARGET</span> - <span class="py-src-number">1</span>):
 130         <span class="py-src-variable">new</span> = <span class="py-src-variable">first</span> + <span class="py-src-variable">second</span>
 131         <span class="py-src-variable">first</span> = <span class="py-src-variable">second</span>
 132         <span class="py-src-variable">second</span> = <span class="py-src-variable">new</span>
 133         <span class="py-src-keyword">if</span> <span class="py-src-variable">i</span> % <span class="py-src-number">100</span> == <span class="py-src-number">0</span>:
 134             <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;Progress: calculating the %dth Fibonnaci number&quot;</span> % <span class="py-src-variable">i</span>
 135
 136     <span class="py-src-comment"># give the Deferred the answer to pass to the callbacks:</span>
 137     <span class="py-src-variable">d</span>.<span class="py-src-variable">callback</span>(<span class="py-src-variable">second</span>)
 138
 139     <span class="py-src-comment"># return the Deferred with the answer:</span>
 140     <span class="py-src-keyword">return</span> <span class="py-src-variable">d</span>
 141
 142 <span class="py-src-keyword">import</span> <span class="py-src-variable">time</span>
 143
 144 <span class="py-src-variable">timeBefore</span> = <span class="py-src-variable">time</span>.<span class="py-src-variable">time</span>()
 145
 146 <span class="py-src-comment"># call the function and get our Deferred</span>
 147 <span class="py-src-variable">d</span> = <span class="py-src-variable">largeFibonnaciNumber</span>()
 148
 149 <span class="py-src-variable">timeAfter</span> = <span class="py-src-variable">time</span>.<span class="py-src-variable">time</span>()
 150
 151 <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;Total time taken for largeFibonnaciNumber call: %0.3f seconds&quot;</span> %
 152       (<span class="py-src-variable">timeAfter</span> - <span class="py-src-variable">timeBefore</span>)
 153
 154 <span class="py-src-comment"># add a callback to it to print the number</span>
 155
 156 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printNumber</span>(<span class="py-src-parameter">number</span>):
 157     <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;The %dth Fibonacci number is %d&quot;</span> % (<span class="py-src-variable">TARGET</span>, <span class="py-src-variable">number</span>)
 158
 159 <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;Adding the callback now.&quot;</span>
 160
 161 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printNumber</span>)
 162 </pre>
 163
 164 <p>You will notice that despite creating a Deferred in the  <code>largeFibonnaciNumber</code> function, these things happened:</p>
 165 <ul>
 166 <li>the &quot;Total time taken for largeFibonnaciNumber call&quot; output
 167 shows that the function did not return immediately as asynchronous functions
 168 are expected to do; and</li>
 169 <li>rather than the callback being added before the result was available and
 170 called after the result is available, it isn't even added until after the
 171 calculation has been completed.</li>
 172 </ul>
 173
 174 <p> The function completed its calculation before returning, blocking the
 175 process until it had finished, which is exactly what asynchronous functions
 176 are not meant to do.  Deferreds are not a non-blocking talisman: they are a
 177 signal for asynchronous functions to <em>use</em> to pass results onto
 178 callbacks, but using them does not guarantee that you have an asynchronous
 179 function.</p>
 180
 181
 182 <h2>Advanced Processing Chain Control<a name="auto3"/></h2>
 183
 184 <ul>
 185   <li>
 186   <code class="py-prototype">pause()</code>
 187
 188   <p>Cease calling any methods as they are added, and do not
 189   respond to <code>callback</code>, until
 190   <code>self.unpause()</code> is called.</p>
 191   </li>
 192
 193   <li>
 194   <code class="py-prototype">unpause()</code>
 195
 196   <p>If <code>callback</code> has been called on this
 197   Deferred already, call all the callbacks that have been
 198   added to this Deferred since <code>pause</code> was
 199   called.</p>
 200
 201   <p>Whether it was called or not, this will put this
 202   Deferred in a state where further calls to
 203   <code>addCallbacks</code> or <code>callback</code> will
 204   work as normal.</p>
 205   </li>
 206 </ul>
 207
 208 <h2>Returning Deferreds from synchronous functions<a name="auto4"/></h2>
 209
 210 <p>Sometimes you might wish to return a Deferred from a synchronous function.
 211 There are several reasons why, the major two are maintaining API compatibility
 212 with another version of your function which returns a Deferred, or allowing
 213 for the possiblity that in the future your function might need to be
 214 asynchronous.</p>
 215
 216 <p>In the <a href="defer.html" shape="rect">Using Deferreds</a> reference, we gave the
 217 following example of a synchronous function:</p>
 218
 219 <div class="py-listing"><pre><p class="py-linenumber">1
 220 2
 221 3
 222 4
 223 5
 224 </p><span class="py-src-keyword">def</span> <span class="py-src-identifier">synchronousIsValidUser</span>(<span class="py-src-parameter">user</span>):
 225     <span class="py-src-string">'''
 226     Return true if user is a valid user, false otherwise
 227     '''</span>
 228     <span class="py-src-keyword">return</span> <span class="py-src-variable">user</span> <span class="py-src-keyword">in</span> [<span class="py-src-string">&quot;Alice&quot;</span>, <span class="py-src-string">&quot;Angus&quot;</span>, <span class="py-src-string">&quot;Agnes&quot;</span>]
 229 </pre><div class="caption">Source listing - <a href="listings/deferred/synch-validation.py"><span class="filename">listings/deferred/synch-validation.py</span></a></div></div>
 230
 231 <p>While we can require that callers of our function wrap our synchronous
 232 result in a Deferred using <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.maybeDeferred.html" title="twisted.internet.defer.maybeDeferred">maybeDeferred</a></code>, for the sake of API
 233 compatibility it is better to return a Deferred ourselves using  <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.succeed.html" title="twisted.internet.defer.succeed">defer.succeed</a></code>:</p>
 234
 235 <pre class="python"><p class="py-linenumber"> 1
 236  2
 237  3
 238  4
 239  5
 240  6
 241  7
 242  8
 243  9
 244 10
 245 11
 246 12
 247 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">defer</span>
 248
 249 <span class="py-src-keyword">def</span> <span class="py-src-identifier">immediateIsValidUser</span>(<span class="py-src-parameter">user</span>):
 250     <span class="py-src-string">'''
 251     Returns a Deferred resulting in true if user is a valid user, false
 252     otherwise
 253     '''</span>
 254
 255     <span class="py-src-variable">result</span> = <span class="py-src-variable">user</span> <span class="py-src-keyword">in</span> [<span class="py-src-string">&quot;Alice&quot;</span>, <span class="py-src-string">&quot;Angus&quot;</span>, <span class="py-src-string">&quot;Agnes&quot;</span>]
 256
 257     <span class="py-src-comment"># return a Deferred object already called back with the value of result</span>
 258     <span class="py-src-keyword">return</span> <span class="py-src-variable">defer</span>.<span class="py-src-variable">succeed</span>(<span class="py-src-variable">result</span>)
 259 </pre>
 260
 261 <p>There is an equivalent <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.fail.html" title="twisted.internet.defer.fail">defer.fail</a></code> method to return a Deferred with the
 262 errback chain already fired.</p>
 263
 264 <h2>Integrating blocking code with Twisted<a name="auto5"/></h2>
 265
 266 <p>At some point, you are likely to need to call a blocking function: many
 267 functions in third party libraries will have long running blocking functions.
 268 There is no way to 'force' a function to be asynchronous: it must be written
 269 that way specifically. When using Twisted, your own code should be
 270 asynchronous, but there is no way to make third party functions asynchronous
 271 other than rewriting them.</p>
 272
 273 <p>In this case, Twisted provides the ability to run the blocking code in a
 274 separate thread rather than letting it block your application. The <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.threads.deferToThread.html" title="twisted.internet.threads.deferToThread">twisted.internet.threads.deferToThread</a></code> function will set up
 275 a thread to run your blocking function, return a Deferred and later fire that
 276 Deferred when the thread completes.</p>
 277
 278 <p>Let's assume our <code class="python">largeFibonnaciNumber</code> function
 279 from above is in a third party library (returning the result of the
 280 calculation, not a Deferred) and is not easily modifiable to be finished in
 281 discrete blocks. This example shows it being called in a thread, unlike in the
 282 earlier section we'll see that the operation does not block our entire
 283 program:</p>
 284
 285 <pre class="python"><p class="py-linenumber"> 1
 286  2
 287  3
 288  4
 289  5
 290  6
 291  7
 292  8
 293  9
 294 10
 295 11
 296 12
 297 13
 298 14
 299 15
 300 16
 301 17
 302 18
 303 19
 304 20
 305 21
 306 22
 307 23
 308 24
 309 25
 310 26
 311 27
 312 28
 313 29
 314 30
 315 31
 316 32
 317 33
 318 34
 319 35
 320 36
 321 37
 322 38
 323 39
 324 40
 325 41
 326 42
 327 43
 328 44
 329 45
 330 </p><span class="py-src-keyword">def</span> <span class="py-src-identifier">largeFibonnaciNumber</span>():
 331     <span class="py-src-string">&quot;&quot;&quot;
 332     Represent a long running blocking function by calculating
 333     the TARGETth Fibonnaci number
 334     &quot;&quot;&quot;</span>
 335     <span class="py-src-variable">TARGET</span> = <span class="py-src-number">10000</span>
 336
 337     <span class="py-src-variable">first</span> = <span class="py-src-number">0</span>
 338     <span class="py-src-variable">second</span> = <span class="py-src-number">1</span>
 339
 340     <span class="py-src-keyword">for</span> <span class="py-src-variable">i</span> <span class="py-src-keyword">in</span> <span class="py-src-variable">xrange</span>(<span class="py-src-variable">TARGET</span> - <span class="py-src-number">1</span>):
 341         <span class="py-src-variable">new</span> = <span class="py-src-variable">first</span> + <span class="py-src-variable">second</span>
 342         <span class="py-src-variable">first</span> = <span class="py-src-variable">second</span>
 343         <span class="py-src-variable">second</span> = <span class="py-src-variable">new</span>
 344
 345     <span class="py-src-keyword">return</span> <span class="py-src-variable">second</span>
 346
 347 <span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">threads</span>, <span class="py-src-variable">reactor</span>
 348
 349 <span class="py-src-keyword">def</span> <span class="py-src-identifier">fibonacciCallback</span>(<span class="py-src-parameter">result</span>):
 350     <span class="py-src-string">&quot;&quot;&quot;
 351     Callback which manages the largeFibonnaciNumber result by
 352     printing it out
 353     &quot;&quot;&quot;</span>
 354     <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;largeFibonnaciNumber result =&quot;</span>, <span class="py-src-variable">result</span>
 355     <span class="py-src-comment"># make sure the reactor stops after the callback chain finishes,</span>
 356     <span class="py-src-comment"># just so that this example terminates</span>
 357     <span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>()
 358
 359 <span class="py-src-keyword">def</span> <span class="py-src-identifier">run</span>():
 360     <span class="py-src-string">&quot;&quot;&quot;
 361     Run a series of operations, deferring the largeFibonnaciNumber
 362     operation to a thread and performing some other operations after
 363     adding the callback
 364     &quot;&quot;&quot;</span>
 365     <span class="py-src-comment"># get our Deferred which will be called with the largeFibonnaciNumber result</span>
 366     <span class="py-src-variable">d</span> = <span class="py-src-variable">threads</span>.<span class="py-src-variable">deferToThread</span>(<span class="py-src-variable">largeFibonnaciNumber</span>)
 367     <span class="py-src-comment"># add our callback to print it out</span>
 368     <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">fibonacciCallback</span>)
 369     <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;1st line after the addition of the callback&quot;</span>
 370     <span class="py-src-keyword">print</span> <span class="py-src-string">&quot;2nd line after the addition of the callback&quot;</span>
 371
 372 <span class="py-src-keyword">if</span> <span class="py-src-variable">__name__</span> == <span class="py-src-string">'__main__'</span>:
 373     <span class="py-src-variable">run</span>()
 374     <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
 375 </pre>
 376
 377 <h2>Possible sources of error<a name="auto6"/></h2>
 378
 379 <p>Deferreds greatly simplify the process of writing asynchronous code by
 380 providing a standard for registering callbacks, but there are some subtle and
 381 sometimes confusing rules that you need to follow if you are going to use
 382 them. This mostly applies to people who are writing new systems that use
 383 Deferreds internally, and not writers of applications that just add callbacks
 384 to Deferreds produced and processed by other systems. Nevertheless, it is good
 385 to know.</p>
 386
 387 <h3>Firing Deferreds more than once is impossible<a name="auto7"/></h3>
 388
 389 <p>Deferreds are one-shot. You can only call <code>Deferred.callback</code> or <code>Deferred.errback</code> once. The processing chain continues each time
 390 you add new callbacks to an already-called-back-to Deferred.</p>
 391
 392 <h3>Synchronous callback execution<a name="auto8"/></h3>
 393
 394 <p>If a Deferred already has a result available, <code>addCallback</code> <strong>may</strong> call the callback synchronously: that is, immediately
 395 after it's been added.  In situations where callbacks modify state, it is
 396 might be desirable for the chain of processing to halt until all callbacks are
 397 added. For this, it is possible to <code>pause</code> and <code>unpause</code>
 398 a Deferred's processing chain while you are adding lots of callbacks.</p>
 399
 400 <p>Be careful when you use these methods! If you <code>pause</code> a
 401 Deferred, it is <em>your</em> responsibility to make sure that you unpause it.
 402 The function adding the callbacks must unpause a paused Deferred, it should <em>never</em> be the responsibility of the code that actually fires the
 403 callback chain by calling <code>callback</code> or <code>errback</code> as
 404 this would negate its usefulness!</p>
 405
 406 </div>
 407
 408     <p><a href="index.html">Index</a></p>
 409     <span class="version">Version: 12.1.0</span>
 410   </body>
 411 </html>