doc/core/howto/threading.html

   1 <?xml version="1.0" encoding="utf-8"?><!DOCTYPE html  PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN'  'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html lang="en" xmlns="http://www.w3.org/1999/xhtml">
   2   <head>
   3 <title>Twisted Documentation: Using Threads in Twisted</title>
   4 <link href="stylesheet.css" rel="stylesheet" type="text/css"/>
   5   </head>
   6
   7   <body bgcolor="white">
   8     <h1 class="title">Using Threads in Twisted</h1>
   9     <div class="toc"><ol><li><a href="#auto0">Running code in a thread-safe manner</a></li><li><a href="#auto1">Running code in threads</a></li><li><a href="#auto2">Utility Methods</a></li><li><a href="#auto3">Managing the Thread Pool</a></li></ol></div>
  10     <div class="content">
  11     <span/>
  12
  13     <h2>Running code in a thread-safe manner<a name="auto0"/></h2>
  14
  15     <p>Most code in Twisted is not thread-safe. For example,
  16     writing data to a transport from a protocol is not thread-safe.
  17     Therefore, we want a way to schedule methods to be run in the
  18     main event loop. This can be done using the function <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.interfaces.IReactorThreads.callFromThread.html" title="twisted.internet.interfaces.IReactorThreads.callFromThread">twisted.internet.interfaces.IReactorThreads.callFromThread</a></code>:</p>
  19 <pre class="python"><p class="py-linenumber"> 1
  20  2
  21  3
  22  4
  23  5
  24  6
  25  7
  26  8
  27  9
  28 10
  29 11
  30 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
  31
  32 <span class="py-src-keyword">def</span> <span class="py-src-identifier">notThreadSafe</span>(<span class="py-src-parameter">x</span>):
  33      <span class="py-src-string">&quot;&quot;&quot;do something that isn't thread-safe&quot;&quot;&quot;</span>
  34      <span class="py-src-comment"># ...</span>
  35
  36 <span class="py-src-keyword">def</span> <span class="py-src-identifier">threadSafeScheduler</span>():
  37     <span class="py-src-string">&quot;&quot;&quot;Run in thread-safe manner.&quot;&quot;&quot;</span>
  38     <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callFromThread</span>(<span class="py-src-variable">notThreadSafe</span>, <span class="py-src-number">3</span>) <span class="py-src-comment"># will run 'notThreadSafe(3)'</span>
  39                                              <span class="py-src-comment"># in the event loop</span>
  40 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
  41 </pre>
  42
  43     <h2>Running code in threads<a name="auto1"/></h2>
  44
  45     <p>Sometimes we may want to run methods in threads - for
  46     example, in order to access blocking APIs. Twisted provides
  47     methods for doing so using the IReactorThreads API (<code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.interfaces.IReactorThreads.html" title="twisted.internet.interfaces.IReactorThreads">twisted.internet.interfaces.IReactorThreads</a></code>).
  48     Additional utility functions are provided in <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.threads.html" title="twisted.internet.threads">twisted.internet.threads</a></code>. Basically, these
  49     methods allow us to queue methods to be run by a thread
  50     pool.</p>
  51
  52     <p>For example, to run a method in a thread we can do:</p>
  53 <pre class="python"><p class="py-linenumber"> 1
  54  2
  55  3
  56  4
  57  5
  58  6
  59  7
  60  8
  61  9
  62 10
  63 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
  64
  65 <span class="py-src-keyword">def</span> <span class="py-src-identifier">aSillyBlockingMethod</span>(<span class="py-src-parameter">x</span>):
  66     <span class="py-src-keyword">import</span> <span class="py-src-variable">time</span>
  67     <span class="py-src-variable">time</span>.<span class="py-src-variable">sleep</span>(<span class="py-src-number">2</span>)
  68     <span class="py-src-keyword">print</span> <span class="py-src-variable">x</span>
  69
  70 <span class="py-src-comment"># run method in thread</span>
  71 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callInThread</span>(<span class="py-src-variable">aSillyBlockingMethod</span>, <span class="py-src-string">&quot;2 seconds have passed&quot;</span>)
  72 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
  73 </pre>
  74
  75     <h2>Utility Methods<a name="auto2"/></h2>
  76
  77     <p>The utility methods are not part of the <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.reactor.html" title="twisted.internet.reactor">twisted.internet.reactor</a></code> APIs, but are implemented
  78     in <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.threads.html" title="twisted.internet.threads">twisted.internet.threads</a></code>.</p>
  79
  80     <p>If we have multiple methods to run sequentially within a thread,
  81     we can do:</p>
  82
  83 <pre class="python"><p class="py-linenumber"> 1
  84  2
  85  3
  86  4
  87  5
  88  6
  89  7
  90  8
  91  9
  92 10
  93 11
  94 12
  95 13
  96 14
  97 15
  98 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>, <span class="py-src-variable">threads</span>
  99
 100 <span class="py-src-keyword">def</span> <span class="py-src-identifier">aSillyBlockingMethodOne</span>(<span class="py-src-parameter">x</span>):
 101     <span class="py-src-keyword">import</span> <span class="py-src-variable">time</span>
 102     <span class="py-src-variable">time</span>.<span class="py-src-variable">sleep</span>(<span class="py-src-number">2</span>)
 103     <span class="py-src-keyword">print</span> <span class="py-src-variable">x</span>
 104
 105 <span class="py-src-keyword">def</span> <span class="py-src-identifier">aSillyBlockingMethodTwo</span>(<span class="py-src-parameter">x</span>):
 106     <span class="py-src-keyword">print</span> <span class="py-src-variable">x</span>
 107
 108 <span class="py-src-comment"># run both methods sequentially in a thread</span>
 109 <span class="py-src-variable">commands</span> = [(<span class="py-src-variable">aSillyBlockingMethodOne</span>, [<span class="py-src-string">&quot;Calling First&quot;</span>], {})]
 110 <span class="py-src-variable">commands</span>.<span class="py-src-variable">append</span>((<span class="py-src-variable">aSillyBlockingMethodTwo</span>, [<span class="py-src-string">&quot;And the second&quot;</span>], {}))
 111 <span class="py-src-variable">threads</span>.<span class="py-src-variable">callMultipleInThread</span>(<span class="py-src-variable">commands</span>)
 112 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
 113 </pre>
 114
 115     <p>For functions whose results we wish to get, we can have the
 116     result returned as a Deferred:</p>
 117 <pre class="python"><p class="py-linenumber"> 1
 118  2
 119  3
 120  4
 121  5
 122  6
 123  7
 124  8
 125  9
 126 10
 127 11
 128 12
 129 13
 130 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>, <span class="py-src-variable">threads</span>
 131
 132 <span class="py-src-keyword">def</span> <span class="py-src-identifier">doLongCalculation</span>():
 133     <span class="py-src-comment"># .... do long calculation here ...</span>
 134     <span class="py-src-keyword">return</span> <span class="py-src-number">3</span>
 135
 136 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printResult</span>(<span class="py-src-parameter">x</span>):
 137     <span class="py-src-keyword">print</span> <span class="py-src-variable">x</span>
 138
 139 <span class="py-src-comment"># run method in thread and get result as defer.Deferred</span>
 140 <span class="py-src-variable">d</span> = <span class="py-src-variable">threads</span>.<span class="py-src-variable">deferToThread</span>(<span class="py-src-variable">doLongCalculation</span>)
 141 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printResult</span>)
 142 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
 143 </pre>
 144
 145     <p>If you wish to call a method in the reactor thread and get its result,
 146     you can use <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.threads.blockingCallFromThread.html" title="twisted.internet.threads.blockingCallFromThread">blockingCallFromThread</a></code>:</p>
 147
 148 <pre class="python"><p class="py-linenumber"> 1
 149  2
 150  3
 151  4
 152  5
 153  6
 154  7
 155  8
 156  9
 157 10
 158 11
 159 12
 160 13
 161 14
 162 15
 163 16
 164 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">threads</span>, <span class="py-src-variable">reactor</span>, <span class="py-src-variable">defer</span>
 165 <span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">web</span>.<span class="py-src-variable">client</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">getPage</span>
 166 <span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">web</span>.<span class="py-src-variable">error</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">Error</span>
 167
 168 <span class="py-src-keyword">def</span> <span class="py-src-identifier">inThread</span>():
 169     <span class="py-src-keyword">try</span>:
 170         <span class="py-src-variable">result</span> = <span class="py-src-variable">threads</span>.<span class="py-src-variable">blockingCallFromThread</span>(
 171             <span class="py-src-variable">reactor</span>, <span class="py-src-variable">getPage</span>, <span class="py-src-string">&quot;http://twistedmatrix.com/&quot;</span>)
 172     <span class="py-src-keyword">except</span> <span class="py-src-variable">Error</span>, <span class="py-src-variable">exc</span>:
 173         <span class="py-src-keyword">print</span> <span class="py-src-variable">exc</span>
 174     <span class="py-src-keyword">else</span>:
 175         <span class="py-src-keyword">print</span> <span class="py-src-variable">result</span>
 176     <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callFromThread</span>(<span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>)
 177
 178 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callInThread</span>(<span class="py-src-variable">inThread</span>)
 179 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
 180 </pre>
 181
 182     <p><code>blockingCallFromThread</code> will return the object or raise
 183     the exception returned or raised by the function passed to it. If the
 184     function passed to it returns a Deferred, it will return the value the
 185     Deferred is called back with or raise the exception it is errbacked
 186     with.</p>
 187
 188     <h2>Managing the Thread Pool<a name="auto3"/></h2>
 189
 190     <p>The thread pool is implemented by <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.python.threadpool.ThreadPool.html" title="twisted.python.threadpool.ThreadPool">twisted.python.threadpool.ThreadPool</a></code>.</p>
 191
 192     <p>We may want to modify the size of the threadpool, increasing
 193     or decreasing the number of threads in use.  We can do this
 194     do this quite easily:</p>
 195
 196 <pre class="python"><p class="py-linenumber">1
 197 2
 198 3
 199 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
 200
 201 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">suggestThreadPoolSize</span>(<span class="py-src-number">30</span>)
 202 </pre>
 203
 204   <p>The default size of the thread pool depends on the reactor being used;
 205   the default reactor uses a minimum size of 5 and a maximum size of 10.  Be
 206   careful that you understand threads and their resource usage before
 207   drastically altering the thread pool sizes.</p>
 208   </div>
 209
 210     <p><a href="index.html">Index</a></p>
 211     <span class="version">Version: 12.1.0</span>
 212   </body>
 213 </html>