1 <?xml version="1.0" encoding="utf-8"?><!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html lang="en" xmlns="http://www.w3.org/1999/xhtml">
3 <title>Twisted Documentation: Using Threads in Twisted</title>
4 <link href="stylesheet.css" rel="stylesheet" type="text/css"/>
8 <h1 class="title">Using Threads in Twisted</h1>
9 <div class="toc"><ol><li><a href="#auto0">Running code in a thread-safe manner</a></li><li><a href="#auto1">Running code in threads</a></li><li><a href="#auto2">Utility Methods</a></li><li><a href="#auto3">Managing the Thread Pool</a></li></ol></div>
13 <h2>Running code in a thread-safe manner<a name="auto0"/></h2>
15 <p>Most code in Twisted is not thread-safe. For example,
16 writing data to a transport from a protocol is not thread-safe.
17 Therefore, we want a way to schedule methods to be run in the
18 main event loop. This can be done using the function <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.interfaces.IReactorThreads.callFromThread.html" title="twisted.internet.interfaces.IReactorThreads.callFromThread">twisted.internet.interfaces.IReactorThreads.callFromThread</a></code>:</p>
19 <pre class="python"><p class="py-linenumber"> 1
30 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
32 <span class="py-src-keyword">def</span> <span class="py-src-identifier">notThreadSafe</span>(<span class="py-src-parameter">x</span>):
33 <span class="py-src-string">"""do something that isn't thread-safe"""</span>
34 <span class="py-src-comment"># ...</span>
36 <span class="py-src-keyword">def</span> <span class="py-src-identifier">threadSafeScheduler</span>():
37 <span class="py-src-string">"""Run in thread-safe manner."""</span>
38 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callFromThread</span>(<span class="py-src-variable">notThreadSafe</span>, <span class="py-src-number">3</span>) <span class="py-src-comment"># will run 'notThreadSafe(3)'</span>
39 <span class="py-src-comment"># in the event loop</span>
40 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
43 <h2>Running code in threads<a name="auto1"/></h2>
45 <p>Sometimes we may want to run methods in threads - for
46 example, in order to access blocking APIs. Twisted provides
47 methods for doing so using the IReactorThreads API (<code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.interfaces.IReactorThreads.html" title="twisted.internet.interfaces.IReactorThreads">twisted.internet.interfaces.IReactorThreads</a></code>).
48 Additional utility functions are provided in <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.threads.html" title="twisted.internet.threads">twisted.internet.threads</a></code>. Basically, these
49 methods allow us to queue methods to be run by a thread
52 <p>For example, to run a method in a thread we can do:</p>
53 <pre class="python"><p class="py-linenumber"> 1
63 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
65 <span class="py-src-keyword">def</span> <span class="py-src-identifier">aSillyBlockingMethod</span>(<span class="py-src-parameter">x</span>):
66 <span class="py-src-keyword">import</span> <span class="py-src-variable">time</span>
67 <span class="py-src-variable">time</span>.<span class="py-src-variable">sleep</span>(<span class="py-src-number">2</span>)
68 <span class="py-src-keyword">print</span> <span class="py-src-variable">x</span>
70 <span class="py-src-comment"># run method in thread</span>
71 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callInThread</span>(<span class="py-src-variable">aSillyBlockingMethod</span>, <span class="py-src-string">"2 seconds have passed"</span>)
72 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
75 <h2>Utility Methods<a name="auto2"/></h2>
77 <p>The utility methods are not part of the <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.reactor.html" title="twisted.internet.reactor">twisted.internet.reactor</a></code> APIs, but are implemented
78 in <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.threads.html" title="twisted.internet.threads">twisted.internet.threads</a></code>.</p>
80 <p>If we have multiple methods to run sequentially within a thread,
83 <pre class="python"><p class="py-linenumber"> 1
98 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>, <span class="py-src-variable">threads</span>
100 <span class="py-src-keyword">def</span> <span class="py-src-identifier">aSillyBlockingMethodOne</span>(<span class="py-src-parameter">x</span>):
101 <span class="py-src-keyword">import</span> <span class="py-src-variable">time</span>
102 <span class="py-src-variable">time</span>.<span class="py-src-variable">sleep</span>(<span class="py-src-number">2</span>)
103 <span class="py-src-keyword">print</span> <span class="py-src-variable">x</span>
105 <span class="py-src-keyword">def</span> <span class="py-src-identifier">aSillyBlockingMethodTwo</span>(<span class="py-src-parameter">x</span>):
106 <span class="py-src-keyword">print</span> <span class="py-src-variable">x</span>
108 <span class="py-src-comment"># run both methods sequentially in a thread</span>
109 <span class="py-src-variable">commands</span> = [(<span class="py-src-variable">aSillyBlockingMethodOne</span>, [<span class="py-src-string">"Calling First"</span>], {})]
110 <span class="py-src-variable">commands</span>.<span class="py-src-variable">append</span>((<span class="py-src-variable">aSillyBlockingMethodTwo</span>, [<span class="py-src-string">"And the second"</span>], {}))
111 <span class="py-src-variable">threads</span>.<span class="py-src-variable">callMultipleInThread</span>(<span class="py-src-variable">commands</span>)
112 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
115 <p>For functions whose results we wish to get, we can have the
116 result returned as a Deferred:</p>
117 <pre class="python"><p class="py-linenumber"> 1
130 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>, <span class="py-src-variable">threads</span>
132 <span class="py-src-keyword">def</span> <span class="py-src-identifier">doLongCalculation</span>():
133 <span class="py-src-comment"># .... do long calculation here ...</span>
134 <span class="py-src-keyword">return</span> <span class="py-src-number">3</span>
136 <span class="py-src-keyword">def</span> <span class="py-src-identifier">printResult</span>(<span class="py-src-parameter">x</span>):
137 <span class="py-src-keyword">print</span> <span class="py-src-variable">x</span>
139 <span class="py-src-comment"># run method in thread and get result as defer.Deferred</span>
140 <span class="py-src-variable">d</span> = <span class="py-src-variable">threads</span>.<span class="py-src-variable">deferToThread</span>(<span class="py-src-variable">doLongCalculation</span>)
141 <span class="py-src-variable">d</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printResult</span>)
142 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
145 <p>If you wish to call a method in the reactor thread and get its result,
146 you can use <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.threads.blockingCallFromThread.html" title="twisted.internet.threads.blockingCallFromThread">blockingCallFromThread</a></code>:</p>
148 <pre class="python"><p class="py-linenumber"> 1
164 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">threads</span>, <span class="py-src-variable">reactor</span>, <span class="py-src-variable">defer</span>
165 <span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">web</span>.<span class="py-src-variable">client</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">getPage</span>
166 <span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">web</span>.<span class="py-src-variable">error</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">Error</span>
168 <span class="py-src-keyword">def</span> <span class="py-src-identifier">inThread</span>():
169 <span class="py-src-keyword">try</span>:
170 <span class="py-src-variable">result</span> = <span class="py-src-variable">threads</span>.<span class="py-src-variable">blockingCallFromThread</span>(
171 <span class="py-src-variable">reactor</span>, <span class="py-src-variable">getPage</span>, <span class="py-src-string">"http://twistedmatrix.com/"</span>)
172 <span class="py-src-keyword">except</span> <span class="py-src-variable">Error</span>, <span class="py-src-variable">exc</span>:
173 <span class="py-src-keyword">print</span> <span class="py-src-variable">exc</span>
174 <span class="py-src-keyword">else</span>:
175 <span class="py-src-keyword">print</span> <span class="py-src-variable">result</span>
176 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callFromThread</span>(<span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>)
178 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">callInThread</span>(<span class="py-src-variable">inThread</span>)
179 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
182 <p><code>blockingCallFromThread</code> will return the object or raise
183 the exception returned or raised by the function passed to it. If the
184 function passed to it returns a Deferred, it will return the value the
185 Deferred is called back with or raise the exception it is errbacked
188 <h2>Managing the Thread Pool<a name="auto3"/></h2>
190 <p>The thread pool is implemented by <code class="API"><a href="http://twistedmatrix.com/documents/12.1.0/api/twisted.python.threadpool.ThreadPool.html" title="twisted.python.threadpool.ThreadPool">twisted.python.threadpool.ThreadPool</a></code>.</p>
192 <p>We may want to modify the size of the threadpool, increasing
193 or decreasing the number of threads in use. We can do this
194 do this quite easily:</p>
196 <pre class="python"><p class="py-linenumber">1
199 </p><span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
201 <span class="py-src-variable">reactor</span>.<span class="py-src-variable">suggestThreadPoolSize</span>(<span class="py-src-number">30</span>)
204 <p>The default size of the thread pool depends on the reactor being used;
205 the default reactor uses a minimum size of 5 and a maximum size of 10. Be
206 careful that you understand threads and their resource usage before
207 drastically altering the thread pool sizes.</p>
210 <p><a href="index.html">Index</a></p>
211 <span class="version">Version: 12.1.0</span>