1 # -*- test-case-name: twisted.test.test_amp -*-
2 # Copyright (c) 2005 Divmod, Inc.
3 # Copyright (c) Twisted Matrix Laboratories.
4 # See LICENSE for details.
7 This module implements AMP, the Asynchronous Messaging Protocol.
9 AMP is a protocol for sending multiple asynchronous request/response pairs over
10 the same connection. Requests and responses are both collections of key/value
13 AMP is a very simple protocol which is not an application. This module is a
14 "protocol construction kit" of sorts; it attempts to be the simplest wire-level
15 implementation of Deferreds. AMP provides the following base-level features:
17 - Asynchronous request/response handling (hence the name)
19 - Requests and responses are both key/value pairs
21 - Binary transfer of all data: all data is length-prefixed. Your
22 application will never need to worry about quoting.
24 - Command dispatching (like HTTP Verbs): the protocol is extensible, and
25 multiple AMP sub-protocols can be grouped together easily.
27 The protocol implementation also provides a few additional features which are
28 not part of the core wire protocol, but are nevertheless very useful:
30 - Tight TLS integration, with an included StartTLS command.
32 - Handshaking to other protocols: because AMP has well-defined message
33 boundaries and maintains all incoming and outgoing requests for you, you
34 can start a connection over AMP and then switch to another protocol.
35 This makes it ideal for firewall-traversal applications where you may
36 have only one forwarded port but multiple applications that want to use
39 Using AMP with Twisted is simple. Each message is a command, with a response.
40 You begin by defining a command type. Commands specify their input and output
41 in terms of the types that they expect to see in the request and response
42 key-value pairs. Here's an example of a command that adds two integers, 'a'
45 class Sum(amp.Command):
46 arguments = [('a', amp.Integer()),
48 response = [('total', amp.Integer())]
50 Once you have specified a command, you need to make it part of a protocol, and
51 define a responder for it. Here's a 'JustSum' protocol that includes a
52 responder for our 'Sum' command::
54 class JustSum(amp.AMP):
57 print 'Did a sum: %d + %d = %d' % (a, b, total)
58 return {'total': total}
61 Later, when you want to actually do a sum, the following expression will return
62 a L{Deferred} which will fire with the result::
64 ClientCreator(reactor, amp.AMP).connectTCP(...).addCallback(
65 lambda p: p.callRemote(Sum, a=13, b=81)).addCallback(
66 lambda result: result['total'])
68 Command responders may also return Deferreds, causing the response to be
69 sent only once the Deferred fires::
71 class DelayedSum(amp.AMP):
72 def slowSum(self, a, b):
74 result = defer.Deferred()
75 reactor.callLater(3, result.callback, {'total': total})
77 Sum.responder(slowSum)
79 This is transparent to the caller.
81 You can also define the propagation of specific errors in AMP. For example,
82 for the slightly more complicated case of division, we might have to deal with
85 class Divide(amp.Command):
86 arguments = [('numerator', amp.Integer()),
87 ('denominator', amp.Integer())]
88 response = [('result', amp.Float())]
89 errors = {ZeroDivisionError: 'ZERO_DIVISION'}
91 The 'errors' mapping here tells AMP that if a responder to Divide emits a
92 L{ZeroDivisionError}, then the other side should be informed that an error of
93 the type 'ZERO_DIVISION' has occurred. Writing a responder which takes
94 advantage of this is very simple - just raise your exception normally::
96 class JustDivide(amp.AMP):
97 def divide(self, numerator, denominator):
98 result = numerator / denominator
99 print 'Divided: %d / %d = %d' % (numerator, denominator, total)
100 return {'result': result}
101 Divide.responder(divide)
103 On the client side, the errors mapping will be used to determine what the
104 'ZERO_DIVISION' error means, and translated into an asynchronous exception,
105 which can be handled normally as any L{Deferred} would be::
107 def trapZero(result):
108 result.trap(ZeroDivisionError)
109 print "Divided by zero: returning INF"
111 ClientCreator(reactor, amp.AMP).connectTCP(...).addCallback(
112 lambda p: p.callRemote(Divide, numerator=1234,
114 ).addErrback(trapZero)
116 For a complete, runnable example of both of these commands, see the files in
117 the Twisted repository::
119 doc/core/examples/ampserver.py
120 doc/core/examples/ampclient.py
122 On the wire, AMP is a protocol which uses 2-byte lengths to prefix keys and
123 values, and empty keys to separate messages::
125 <2-byte length><key><2-byte length><value>
126 <2-byte length><key><2-byte length><value>
128 <2-byte length><key><2-byte length><value>
129 <NUL><NUL> # Empty Key == End of Message
131 And so on. Because it's tedious to refer to lengths and NULs constantly, the
132 documentation will refer to packets as if they were newline delimited, like
136 C: _ask: ef639e5c892ccb54
140 S: _answer: ef639e5c892ccb54
145 In general, the order of keys is arbitrary. Specific uses of AMP may impose an
146 ordering requirement, but unless this is specified explicitly, any ordering may
147 be generated and any ordering must be accepted. This applies to the
148 command-related keys I{_command} and I{_ask} as well as any other keys.
150 Values are limited to the maximum encodable size in a 16-bit length, 65535
153 Keys are limited to the maximum encodable size in a 8-bit length, 255 bytes.
154 Note that we still use 2-byte lengths to encode keys. This small redundancy
155 has several features:
157 - If an implementation becomes confused and starts emitting corrupt data,
158 or gets keys confused with values, many common errors will be signalled
159 immediately instead of delivering obviously corrupt packets.
161 - A single NUL will separate every key, and a double NUL separates
162 messages. This provides some redundancy when debugging traffic dumps.
164 - NULs will be present at regular intervals along the protocol, providing
165 some padding for otherwise braindead C implementations of the protocol,
166 so that <stdio.h> string functions will see the NUL and stop.
168 - This makes it possible to run an AMP server on a port also used by a
169 plain-text protocol, and easily distinguish between non-AMP clients (like
170 web browsers) which issue non-NUL as the first byte, and AMP clients,
171 which always issue NUL as the first byte.
176 import types, warnings
178 from cStringIO import StringIO
179 from struct import pack
180 import decimal, datetime
181 from itertools import count
183 from zope.interface import Interface, implements
185 from twisted.python.compat import set
186 from twisted.python.util import unsignedID
187 from twisted.python.reflect import accumulateClassDict
188 from twisted.python.failure import Failure
189 from twisted.python import log, filepath
191 from twisted.internet.interfaces import IFileDescriptorReceiver
192 from twisted.internet.main import CONNECTION_LOST
193 from twisted.internet.error import PeerVerifyError, ConnectionLost
194 from twisted.internet.error import ConnectionClosed
195 from twisted.internet.defer import Deferred, maybeDeferred, fail
196 from twisted.protocols.basic import Int16StringReceiver, StatefulStringProtocol
199 from twisted.internet import ssl
203 if ssl and not ssl.supported:
207 from twisted.internet.ssl import CertificateOptions, Certificate, DN, KeyPair
213 ERROR_CODE = '_error_code'
214 ERROR_DESCRIPTION = '_error_description'
215 UNKNOWN_ERROR_CODE = 'UNKNOWN'
216 UNHANDLED_ERROR_CODE = 'UNHANDLED'
218 MAX_KEY_LENGTH = 0xff
219 MAX_VALUE_LENGTH = 0xffff
222 class IArgumentType(Interface):
224 An L{IArgumentType} can serialize a Python object into an AMP box and
225 deserialize information from an AMP box back into a Python object.
229 def fromBox(name, strings, objects, proto):
231 Given an argument name and an AMP box containing serialized values,
232 extract one or more Python objects and add them to the C{objects}
235 @param name: The name associated with this argument. Most commonly,
236 this is the key which can be used to find a serialized value in
237 C{strings} and which should be used as the key in C{objects} to
238 associate with a structured Python object.
241 @param strings: The AMP box from which to extract one or more
243 @type strings: C{dict}
245 @param objects: The output dictionary to populate with the value for
247 @type objects: C{dict}
249 @param proto: The protocol instance which received the AMP box being
250 interpreted. Most likely this is an instance of L{AMP}, but
251 this is not guaranteed.
257 def toBox(name, strings, objects, proto):
259 Given an argument name and a dictionary containing structured Python
260 objects, serialize values into one or more strings and add them to
261 the C{strings} dictionary.
263 @param name: The name associated with this argument. Most commonly,
264 this is the key which can be used to find an object in
265 C{objects} and which should be used as the key in C{strings} to
266 associate with a C{str} giving the serialized form of that
270 @param strings: The AMP box into which to insert one or more
272 @type strings: C{dict}
274 @param objects: The input dictionary from which to extract Python
275 objects to serialize.
276 @type objects: C{dict}
278 @param proto: The protocol instance which will send the AMP box once
279 it is fully populated. Most likely this is an instance of
280 L{AMP}, but this is not guaranteed.
287 class IBoxSender(Interface):
289 A transport which can send L{AmpBox} objects.
296 @raise ProtocolSwitched: if the underlying protocol has been
299 @raise ConnectionLost: if the underlying connection has already been
303 def unhandledError(failure):
305 An unhandled error occurred in response to a box. Log it
308 @param failure: a L{Failure} describing the error that occurred.
313 class IBoxReceiver(Interface):
315 An application object which can receive L{AmpBox} objects and dispatch them
319 def startReceivingBoxes(boxSender):
321 The L{ampBoxReceived} method will start being called; boxes may be
322 responded to by responding to the given L{IBoxSender}.
324 @param boxSender: an L{IBoxSender} provider.
328 def ampBoxReceived(box):
330 A box was received from the transport; dispatch it appropriately.
334 def stopReceivingBoxes(reason):
336 No further boxes will be received on this connection.
338 @type reason: L{Failure}
343 class IResponderLocator(Interface):
345 An application object which can look up appropriate responder methods for
349 def locateResponder(name):
351 Locate a responder method appropriate for the named command.
353 @param name: the wire-level name (commandName) of the AMP command to be
356 @return: a 1-argument callable that takes an L{AmpBox} with argument
357 values for the given command, and returns an L{AmpBox} containing
358 argument values for the named command, or a L{Deferred} that fires the
364 class AmpError(Exception):
366 Base class of all Amp-related exceptions.
371 class ProtocolSwitched(Exception):
373 Connections which have been switched to other protocols can no longer
374 accept traffic at the AMP level. This is raised when you try to send it.
379 class OnlyOneTLS(AmpError):
381 This is an implementation limitation; TLS may only be started once per
387 class NoEmptyBoxes(AmpError):
389 You can't have empty boxes on the connection. This is raised when you
390 receive or attempt to send one.
395 class InvalidSignature(AmpError):
397 You didn't pass all the required arguments.
402 class TooLong(AmpError):
404 One of the protocol's length limitations was violated.
406 @ivar isKey: true if the string being encoded in a key position, false if
407 it was in a value position.
409 @ivar isLocal: Was the string encoded locally, or received too long from
410 the network? (It's only physically possible to encode "too long" values on
411 the network for keys.)
413 @ivar value: The string that was too long.
415 @ivar keyName: If the string being encoded was in a value position, what
416 key was it being encoded for?
419 def __init__(self, isKey, isLocal, value, keyName=None):
420 AmpError.__init__(self)
422 self.isLocal = isLocal
424 self.keyName = keyName
428 hdr = self.isKey and "key" or "value"
430 hdr += ' ' + repr(self.keyName)
431 lcl = self.isLocal and "local" or "remote"
432 return "%s %s too long: %d" % (lcl, hdr, len(self.value))
436 class BadLocalReturn(AmpError):
438 A bad value was returned from a local command; we were unable to coerce it.
440 def __init__(self, message, enclosed):
441 AmpError.__init__(self)
442 self.message = message
443 self.enclosed = enclosed
447 return self.message + " " + self.enclosed.getBriefTraceback()
453 class RemoteAmpError(AmpError):
455 This error indicates that something went wrong on the remote end of the
456 connection, and the error was serialized and transmitted to you.
458 def __init__(self, errorCode, description, fatal=False, local=None):
459 """Create a remote error with an error code and description.
461 @param errorCode: the AMP error code of this error.
463 @param description: some text to show to the user.
465 @param fatal: a boolean, true if this error should terminate the
468 @param local: a local Failure, if one exists.
471 localwhat = ' (local)'
472 othertb = local.getBriefTraceback()
476 Exception.__init__(self, "Code<%s>%s: %s%s" % (
477 errorCode, localwhat,
478 description, othertb))
480 self.errorCode = errorCode
481 self.description = description
486 class UnknownRemoteError(RemoteAmpError):
488 This means that an error whose type we can't identify was raised from the
491 def __init__(self, description):
492 errorCode = UNKNOWN_ERROR_CODE
493 RemoteAmpError.__init__(self, errorCode, description)
497 class MalformedAmpBox(AmpError):
499 This error indicates that the wire-level protocol was malformed.
504 class UnhandledCommand(AmpError):
506 A command received via amp could not be dispatched.
511 class IncompatibleVersions(AmpError):
513 It was impossible to negotiate a compatible version of the protocol with
514 the other end of the connection.
518 PROTOCOL_ERRORS = {UNHANDLED_ERROR_CODE: UnhandledCommand}
522 I am a packet in the AMP protocol, much like a regular str:str dictionary.
524 __slots__ = [] # be like a regular dictionary, don't magically
525 # acquire a __dict__...
530 Return another AmpBox just like me.
532 newBox = self.__class__()
539 Convert me into a wire-encoded string.
541 @return: a str encoded according to the rules described in the module
549 if type(k) == unicode:
550 raise TypeError("Unicode key not allowed: %r" % k)
551 if type(v) == unicode:
553 "Unicode value for key %r not allowed: %r" % (k, v))
554 if len(k) > MAX_KEY_LENGTH:
555 raise TooLong(True, True, k, None)
556 if len(v) > MAX_VALUE_LENGTH:
557 raise TooLong(False, True, v, k)
559 w(pack("!H", len(kv)))
565 def _sendTo(self, proto):
567 Serialize and send this box to a Amp instance. By the time it is being
568 sent, several keys are required. I must have exactly ONE of::
574 If the '_ask' key is set, then the '_command' key must also be
577 @param proto: an AMP instance.
582 return 'AmpBox(%s)' % (dict.__repr__(self),)
588 class QuitBox(AmpBox):
590 I am an AmpBox that, upon being sent, terminates the connection.
596 return 'QuitBox(**%s)' % (super(QuitBox, self).__repr__(),)
599 def _sendTo(self, proto):
601 Immediately call loseConnection after sending.
603 super(QuitBox, self)._sendTo(proto)
604 proto.transport.loseConnection()
608 class _SwitchBox(AmpBox):
610 Implementation detail of ProtocolSwitchCommand: I am a AmpBox which sets
611 up state for the protocol to switch.
614 # DON'T set __slots__ here; we do have an attribute.
616 def __init__(self, innerProto, **kw):
618 Create a _SwitchBox with the protocol to switch to after being sent.
620 @param innerProto: the protocol instance to switch to.
621 @type innerProto: an IProtocol provider.
623 super(_SwitchBox, self).__init__(**kw)
624 self.innerProto = innerProto
628 return '_SwitchBox(%r, **%s)' % (self.innerProto,
629 dict.__repr__(self),)
632 def _sendTo(self, proto):
634 Send me; I am the last box on the connection. All further traffic will be
635 over the new protocol.
637 super(_SwitchBox, self)._sendTo(proto)
638 proto._lockForSwitch()
639 proto._switchTo(self.innerProto)
645 A L{BoxDispatcher} dispatches '_ask', '_answer', and '_error' L{AmpBox}es,
646 both incoming and outgoing, to their appropriate destinations.
648 Outgoing commands are converted into L{Deferred}s and outgoing boxes, and
649 associated tracking state to fire those L{Deferred} when '_answer' boxes
650 come back. Incoming '_answer' and '_error' boxes are converted into
651 callbacks and errbacks on those L{Deferred}s, respectively.
653 Incoming '_ask' boxes are converted into method calls on a supplied method
656 @ivar _outstandingRequests: a dictionary mapping request IDs to
657 L{Deferred}s which were returned for those requests.
659 @ivar locator: an object with a L{locateResponder} method that locates a
660 responder function that takes a Box and returns a result (either a Box or a
661 Deferred which fires one).
663 @ivar boxSender: an object which can send boxes, via the L{_sendBox}
664 method, such as an L{AMP} instance.
665 @type boxSender: L{IBoxSender}
668 implements(IBoxReceiver)
670 _failAllReason = None
671 _outstandingRequests = None
675 def __init__(self, locator):
676 self._outstandingRequests = {}
677 self.locator = locator
680 def startReceivingBoxes(self, boxSender):
682 The given boxSender is going to start calling boxReceived on this
685 @param boxSender: The L{IBoxSender} to send command responses to.
687 self.boxSender = boxSender
690 def stopReceivingBoxes(self, reason):
692 No further boxes will be received here. Terminate all currently
693 oustanding command deferreds with the given reason.
695 self.failAllOutgoing(reason)
698 def failAllOutgoing(self, reason):
700 Call the errback on all outstanding requests awaiting responses.
702 @param reason: the Failure instance to pass to those errbacks.
704 self._failAllReason = reason
705 OR = self._outstandingRequests.items()
706 self._outstandingRequests = None # we can never send another request
707 for key, value in OR:
708 value.errback(reason)
713 Generate protocol-local serial numbers for _ask keys.
715 @return: a string that has not yet been used on this connection.
718 return '%x' % (self._counter,)
721 def _sendBoxCommand(self, command, box, requiresAnswer=True):
723 Send a command across the wire with the given C{amp.Box}.
725 Mutate the given box to give it any additional keys (_command, _ask)
726 required for the command and request/response machinery, then send it.
728 If requiresAnswer is True, returns a C{Deferred} which fires when a
729 response is received. The C{Deferred} is fired with an C{amp.Box} on
730 success, or with an C{amp.RemoteAmpError} if an error is received.
732 If the Deferred fails and the error is not handled by the caller of
733 this method, the failure will be logged and the connection dropped.
735 @param command: a str, the name of the command to issue.
737 @param box: an AmpBox with the arguments for the command.
739 @param requiresAnswer: a boolean. Defaults to True. If True, return a
740 Deferred which will fire when the other side responds to this command.
741 If False, return None and do not ask the other side for acknowledgement.
743 @return: a Deferred which fires the AmpBox that holds the response to
744 this command, or None, as specified by requiresAnswer.
746 @raise ProtocolSwitched: if the protocol has been switched.
748 if self._failAllReason is not None:
749 return fail(self._failAllReason)
750 box[COMMAND] = command
751 tag = self._nextTag()
754 box._sendTo(self.boxSender)
756 result = self._outstandingRequests[tag] = Deferred()
762 def callRemoteString(self, command, requiresAnswer=True, **kw):
764 This is a low-level API, designed only for optimizing simple messages
765 for which the overhead of parsing is too great.
767 @param command: a str naming the command.
769 @param kw: arguments to the amp box.
771 @param requiresAnswer: a boolean. Defaults to True. If True, return a
772 Deferred which will fire when the other side responds to this command.
773 If False, return None and do not ask the other side for acknowledgement.
775 @return: a Deferred which fires the AmpBox that holds the response to
776 this command, or None, as specified by requiresAnswer.
779 return self._sendBoxCommand(command, box, requiresAnswer)
782 def callRemote(self, commandType, *a, **kw):
784 This is the primary high-level API for sending messages via AMP. Invoke it
785 with a command and appropriate arguments to send a message to this
788 @param commandType: a subclass of Command.
789 @type commandType: L{type}
791 @param a: Positional (special) parameters taken by the command.
792 Positional parameters will typically not be sent over the wire. The
793 only command included with AMP which uses positional parameters is
794 L{ProtocolSwitchCommand}, which takes the protocol that will be
795 switched to as its first argument.
797 @param kw: Keyword arguments taken by the command. These are the
798 arguments declared in the command's 'arguments' attribute. They will
799 be encoded and sent to the peer as arguments for the L{commandType}.
801 @return: If L{commandType} has a C{requiresAnswer} attribute set to
802 L{False}, then return L{None}. Otherwise, return a L{Deferred} which
803 fires with a dictionary of objects representing the result of this
804 call. Additionally, this L{Deferred} may fail with an exception
805 representing a connection failure, with L{UnknownRemoteError} if the
806 other end of the connection fails for an unknown reason, or with any
807 error specified as a key in L{commandType}'s C{errors} dictionary.
810 # XXX this takes command subclasses and not command objects on purpose.
811 # There's really no reason to have all this back-and-forth between
812 # command objects and the protocol, and the extra object being created
813 # (the Command instance) is pointless. Command is kind of like
814 # Interface, and should be more like it.
816 # In other words, the fact that commandType is instantiated here is an
817 # implementation detail. Don't rely on it.
820 co = commandType(*a, **kw)
823 return co._doCommand(self)
826 def unhandledError(self, failure):
828 This is a terminal callback called after application code has had a
829 chance to quash any errors.
831 return self.boxSender.unhandledError(failure)
834 def _answerReceived(self, box):
836 An AMP box was received that answered a command previously sent with
839 @param box: an AmpBox with a value for its L{ANSWER} key.
841 question = self._outstandingRequests.pop(box[ANSWER])
842 question.addErrback(self.unhandledError)
843 question.callback(box)
846 def _errorReceived(self, box):
848 An AMP box was received that answered a command previously sent with
849 L{callRemote}, with an error.
851 @param box: an L{AmpBox} with a value for its L{ERROR}, L{ERROR_CODE},
852 and L{ERROR_DESCRIPTION} keys.
854 question = self._outstandingRequests.pop(box[ERROR])
855 question.addErrback(self.unhandledError)
856 errorCode = box[ERROR_CODE]
857 description = box[ERROR_DESCRIPTION]
858 if errorCode in PROTOCOL_ERRORS:
859 exc = PROTOCOL_ERRORS[errorCode](errorCode, description)
861 exc = RemoteAmpError(errorCode, description)
862 question.errback(Failure(exc))
865 def _commandReceived(self, box):
867 @param box: an L{AmpBox} with a value for its L{COMMAND} and L{ASK}
870 def formatAnswer(answerBox):
871 answerBox[ANSWER] = box[ASK]
873 def formatError(error):
874 if error.check(RemoteAmpError):
875 code = error.value.errorCode
876 desc = error.value.description
877 if error.value.fatal:
883 log.err(error) # here is where server-side logging happens
884 # if the error isn't handled
885 code = UNKNOWN_ERROR_CODE
886 desc = "Unknown Error"
887 errorBox[ERROR] = box[ASK]
888 errorBox[ERROR_DESCRIPTION] = desc
889 errorBox[ERROR_CODE] = code
891 deferred = self.dispatchCommand(box)
893 deferred.addCallbacks(formatAnswer, formatError)
894 deferred.addCallback(self._safeEmit)
895 deferred.addErrback(self.unhandledError)
898 def ampBoxReceived(self, box):
900 An AmpBox was received, representing a command, or an answer to a
901 previously issued command (either successful or erroneous). Respond to
902 it according to its contents.
904 @param box: an AmpBox
906 @raise NoEmptyBoxes: when a box is received that does not contain an
907 '_answer', '_command' / '_ask', or '_error' key; i.e. one which does not
908 fit into the command / response protocol defined by AMP.
911 self._answerReceived(box)
913 self._errorReceived(box)
915 self._commandReceived(box)
917 raise NoEmptyBoxes(box)
920 def _safeEmit(self, aBox):
922 Emit a box, ignoring L{ProtocolSwitched} and L{ConnectionLost} errors
923 which cannot be usefully handled.
926 aBox._sendTo(self.boxSender)
927 except (ProtocolSwitched, ConnectionLost):
931 def dispatchCommand(self, box):
933 A box with a _command key was received.
935 Dispatch it to a local handler call it.
937 @param proto: an AMP instance.
938 @param box: an AmpBox to be dispatched.
941 responder = self.locator.locateResponder(cmd)
942 if responder is None:
943 return fail(RemoteAmpError(
944 UNHANDLED_ERROR_CODE,
945 "Unhandled Command: %r" % (cmd,),
947 local=Failure(UnhandledCommand())))
948 return maybeDeferred(responder, box)
952 class CommandLocator:
954 A L{CommandLocator} is a collection of responders to AMP L{Command}s, with
955 the help of the L{Command.responder} decorator.
958 class __metaclass__(type):
960 This metaclass keeps track of all of the Command.responder-decorated
961 methods defined since the last CommandLocator subclass was defined. It
962 assumes (usually correctly, but unfortunately not necessarily so) that
963 those commands responders were all declared as methods of the class
964 being defined. Note that this list can be incorrect if users use the
965 Command.responder decorator outside the context of a CommandLocator
968 Command responders defined on subclasses are given precedence over
969 those inherited from a base class.
971 The Command.responder decorator explicitly cooperates with this
975 _currentClassCommands = []
976 def __new__(cls, name, bases, attrs):
977 commands = cls._currentClassCommands[:]
978 cls._currentClassCommands[:] = []
979 cd = attrs['_commandDispatch'] = {}
980 subcls = type.__new__(cls, name, bases, attrs)
981 ancestors = list(subcls.__mro__[1:])
983 for ancestor in ancestors:
984 cd.update(getattr(ancestor, '_commandDispatch', {}))
985 for commandClass, responderFunc in commands:
986 cd[commandClass.commandName] = (commandClass, responderFunc)
988 subcls.lookupFunction != CommandLocator.lookupFunction)):
989 def locateResponder(self, name):
991 "Override locateResponder, not lookupFunction.",
992 category=PendingDeprecationWarning,
994 return self.lookupFunction(name)
995 subcls.locateResponder = locateResponder
999 implements(IResponderLocator)
1002 def _wrapWithSerialization(self, aCallable, command):
1004 Wrap aCallable with its command's argument de-serialization
1005 and result serialization logic.
1007 @param aCallable: a callable with a 'command' attribute, designed to be
1008 called with keyword arguments.
1010 @param command: the command class whose serialization to use.
1012 @return: a 1-arg callable which, when invoked with an AmpBox, will
1013 deserialize the argument list and invoke appropriate user code for the
1014 callable's command, returning a Deferred which fires with the result or
1015 fails with an error.
1018 kw = command.parseArguments(box, self)
1019 def checkKnownErrors(error):
1020 key = error.trap(*command.allErrors)
1021 code = command.allErrors[key]
1022 desc = str(error.value)
1023 return Failure(RemoteAmpError(
1024 code, desc, key in command.fatalErrors, local=error))
1025 def makeResponseFor(objects):
1027 return command.makeResponse(objects, self)
1029 # let's helpfully log this.
1030 originalFailure = Failure()
1031 raise BadLocalReturn(
1032 "%r returned %r and %r could not serialize it" % (
1037 return maybeDeferred(aCallable, **kw).addCallback(
1038 makeResponseFor).addErrback(
1043 def lookupFunction(self, name):
1045 Deprecated synonym for L{locateResponder}
1047 if self.__class__.lookupFunction != CommandLocator.lookupFunction:
1048 return CommandLocator.locateResponder(self, name)
1050 warnings.warn("Call locateResponder, not lookupFunction.",
1051 category=PendingDeprecationWarning,
1053 return self.locateResponder(name)
1056 def locateResponder(self, name):
1058 Locate a callable to invoke when executing the named command.
1060 @param name: the normalized name (from the wire) of the command.
1062 @return: a 1-argument function that takes a Box and returns a box or a
1063 Deferred which fires a Box, for handling the command identified by the
1064 given name, or None, if no appropriate responder can be found.
1066 # Try to find a high-level method to invoke, and if we can't find one,
1067 # fall back to a low-level one.
1068 cd = self._commandDispatch
1070 commandClass, responderFunc = cd[name]
1071 responderMethod = types.MethodType(
1072 responderFunc, self, self.__class__)
1073 return self._wrapWithSerialization(responderMethod, commandClass)
1077 class SimpleStringLocator(object):
1079 Implement the L{locateResponder} method to do simple, string-based
1083 implements(IResponderLocator)
1085 baseDispatchPrefix = 'amp_'
1087 def locateResponder(self, name):
1089 Locate a callable to invoke when executing the named command.
1091 @return: a function with the name C{"amp_" + name} on L{self}, or None
1092 if no such function exists. This function will then be called with the
1093 L{AmpBox} itself as an argument.
1095 @param name: the normalized name (from the wire) of the command.
1097 fName = self.baseDispatchPrefix + (name.upper())
1098 return getattr(self, fName, None)
1103 'and', 'del', 'for', 'is', 'raise', 'assert', 'elif', 'from', 'lambda',
1104 'return', 'break', 'else', 'global', 'not', 'try', 'class', 'except',
1105 'if', 'or', 'while', 'continue', 'exec', 'import', 'pass', 'yield',
1106 'def', 'finally', 'in', 'print']
1110 def _wireNameToPythonIdentifier(key):
1112 (Private) Normalize an argument name from the wire for use with Python
1113 code. If the return value is going to be a python keyword it will be
1114 capitalized. If it contains any dashes they will be replaced with
1117 The rationale behind this method is that AMP should be an inherently
1118 multi-language protocol, so message keys may contain all manner of bizarre
1119 bytes. This is not a complete solution; there are still forms of arguments
1120 that this implementation will be unable to parse. However, Python
1121 identifiers share a huge raft of properties with identifiers from many
1122 other languages, so this is a 'good enough' effort for now. We deal
1123 explicitly with dashes because that is the most likely departure: Lisps
1124 commonly use dashes to separate method names, so protocols initially
1125 implemented in a lisp amp dialect may use dashes in argument or command
1128 @param key: a str, looking something like 'foo-bar-baz' or 'from'
1130 @return: a str which is a valid python identifier, looking something like
1131 'foo_bar_baz' or 'From'.
1133 lkey = key.replace("-", "_")
1134 if lkey in PYTHON_KEYWORDS:
1142 Base-class of all objects that take values from Amp packets and convert
1143 them into objects for Python functions.
1145 This implementation of L{IArgumentType} provides several higher-level
1146 hooks for subclasses to override. See L{toString} and L{fromString}
1147 which will be used to define the behavior of L{IArgumentType.toBox} and
1148 L{IArgumentType.fromBox}, respectively.
1150 implements(IArgumentType)
1155 def __init__(self, optional=False):
1159 @param optional: a boolean indicating whether this argument can be
1160 omitted in the protocol.
1162 self.optional = optional
1165 def retrieve(self, d, name, proto):
1167 Retrieve the given key from the given dictionary, removing it if found.
1169 @param d: a dictionary.
1171 @param name: a key in L{d}.
1173 @param proto: an instance of an AMP.
1175 @raise KeyError: if I am not optional and no value was found.
1181 if value is not None:
1188 def fromBox(self, name, strings, objects, proto):
1190 Populate an 'out' dictionary with mapping names to Python values
1191 decoded from an 'in' AmpBox mapping strings to string values.
1193 @param name: the argument name to retrieve
1196 @param strings: The AmpBox to read string(s) from, a mapping of
1197 argument names to string values.
1198 @type strings: AmpBox
1200 @param objects: The dictionary to write object(s) to, a mapping of
1201 names to Python objects.
1204 @param proto: an AMP instance.
1206 st = self.retrieve(strings, name, proto)
1207 nk = _wireNameToPythonIdentifier(name)
1208 if self.optional and st is None:
1211 objects[nk] = self.fromStringProto(st, proto)
1214 def toBox(self, name, strings, objects, proto):
1216 Populate an 'out' AmpBox with strings encoded from an 'in' dictionary
1217 mapping names to Python values.
1219 @param name: the argument name to retrieve
1222 @param strings: The AmpBox to write string(s) to, a mapping of
1223 argument names to string values.
1224 @type strings: AmpBox
1226 @param objects: The dictionary to read object(s) from, a mapping of
1227 names to Python objects.
1231 @param proto: the protocol we are converting for.
1234 obj = self.retrieve(objects, _wireNameToPythonIdentifier(name), proto)
1235 if self.optional and obj is None:
1236 # strings[name] = None
1239 strings[name] = self.toStringProto(obj, proto)
1242 def fromStringProto(self, inString, proto):
1244 Convert a string to a Python value.
1246 @param inString: the string to convert.
1248 @param proto: the protocol we are converting for.
1251 @return: a Python object.
1253 return self.fromString(inString)
1256 def toStringProto(self, inObject, proto):
1258 Convert a Python object to a string.
1260 @param inObject: the object to convert.
1262 @param proto: the protocol we are converting for.
1265 return self.toString(inObject)
1268 def fromString(self, inString):
1270 Convert a string to a Python object. Subclasses must implement this.
1272 @param inString: the string to convert.
1275 @return: the decoded value from inString
1279 def toString(self, inObject):
1281 Convert a Python object into a string for passing over the network.
1283 @param inObject: an object of the type that this Argument is intended
1286 @return: the wire encoding of inObject
1292 class Integer(Argument):
1294 Encode any integer values of any size on the wire as the string
1297 Example: C{123} becomes C{"123"}
1300 def toString(self, inObject):
1301 return str(int(inObject))
1305 class String(Argument):
1307 Don't do any conversion at all; just pass through 'str'.
1309 def toString(self, inObject):
1313 def fromString(self, inString):
1318 class Float(Argument):
1320 Encode floating-point values on the wire as their repr.
1327 class Boolean(Argument):
1329 Encode True or False as "True" or "False" on the wire.
1331 def fromString(self, inString):
1332 if inString == 'True':
1334 elif inString == 'False':
1337 raise TypeError("Bad boolean value: %r" % (inString,))
1340 def toString(self, inObject):
1348 class Unicode(String):
1350 Encode a unicode string on the wire as UTF-8.
1353 def toString(self, inObject):
1354 # assert isinstance(inObject, unicode)
1355 return String.toString(self, inObject.encode('utf-8'))
1358 def fromString(self, inString):
1359 # assert isinstance(inString, str)
1360 return String.fromString(self, inString).decode('utf-8')
1364 class Path(Unicode):
1366 Encode and decode L{filepath.FilePath} instances as paths on the wire.
1368 This is really intended for use with subprocess communication tools:
1369 exchanging pathnames on different machines over a network is not generally
1370 meaningful, but neither is it disallowed; you can use this to communicate
1371 about NFS paths, for example.
1373 def fromString(self, inString):
1374 return filepath.FilePath(Unicode.fromString(self, inString))
1377 def toString(self, inObject):
1378 return Unicode.toString(self, inObject.path)
1382 class ListOf(Argument):
1384 Encode and decode lists of instances of a single other argument type.
1386 For example, if you want to pass::
1390 You can create an argument like this::
1394 The serialized form of the entire list is subject to the limit imposed by
1395 L{MAX_VALUE_LENGTH}. List elements are represented as 16-bit length
1396 prefixed strings. The argument type passed to the L{ListOf} initializer is
1397 responsible for producing the serialized form of each element.
1399 @ivar elementType: The L{Argument} instance used to encode and decode list
1400 elements (note, not an arbitrary L{IArgument} implementation:
1401 arguments must be implemented using only the C{fromString} and
1402 C{toString} methods, not the C{fromBox} and C{toBox} methods).
1404 @param optional: a boolean indicating whether this argument can be
1405 omitted in the protocol.
1409 def __init__(self, elementType, optional=False):
1410 self.elementType = elementType
1411 Argument.__init__(self, optional)
1414 def fromString(self, inString):
1416 Convert the serialized form of a list of instances of some type back
1420 parser = Int16StringReceiver()
1421 parser.stringReceived = strings.append
1422 parser.dataReceived(inString)
1423 return map(self.elementType.fromString, strings)
1426 def toString(self, inObject):
1428 Serialize the given list of objects to a single string.
1431 for obj in inObject:
1432 serialized = self.elementType.toString(obj)
1433 strings.append(pack('!H', len(serialized)))
1434 strings.append(serialized)
1435 return ''.join(strings)
1439 class AmpList(Argument):
1441 Convert a list of dictionaries into a list of AMP boxes on the wire.
1443 For example, if you want to pass::
1445 [{'a': 7, 'b': u'hello'}, {'a': 9, 'b': u'goodbye'}]
1447 You might use an AmpList like this in your arguments or response list::
1449 AmpList([('a', Integer()),
1452 def __init__(self, subargs, optional=False):
1456 @param subargs: a list of 2-tuples of ('name', argument) describing the
1457 schema of the dictionaries in the sequence of amp boxes.
1459 @param optional: a boolean indicating whether this argument can be
1460 omitted in the protocol.
1462 self.subargs = subargs
1463 Argument.__init__(self, optional)
1466 def fromStringProto(self, inString, proto):
1467 boxes = parseString(inString)
1468 values = [_stringsToObjects(box, self.subargs, proto)
1473 def toStringProto(self, inObject, proto):
1474 return ''.join([_objectsToStrings(
1475 objects, self.subargs, Box(), proto
1476 ).serialize() for objects in inObject])
1480 class Descriptor(Integer):
1482 Encode and decode file descriptors for exchange over a UNIX domain socket.
1484 This argument type requires an AMP connection set up over an
1485 L{IUNIXTransport<twisted.internet.interfaces.IUNIXTransport>} provider (for
1486 example, the kind of connection created by
1487 L{IReactorUNIX.connectUNIX<twisted.internet.interfaces.IReactorUNIX.connectUNIX>}
1488 and L{UNIXClientEndpoint<twisted.internet.endpoints.UNIXClientEndpoint>}).
1490 There is no correspondence between the integer value of the file descriptor
1491 on the sending and receiving sides, therefore an alternate approach is taken
1492 to matching up received descriptors with particular L{Descriptor}
1493 parameters. The argument is encoded to an ordinal (unique per connection)
1494 for inclusion in the AMP command or response box. The descriptor itself is
1496 L{IUNIXTransport.sendFileDescriptor<twisted.internet.interfaces.IUNIXTransport.sendFileDescriptor>}.
1497 The receiver uses the order in which file descriptors are received and the
1498 ordinal value to come up with the received copy of the descriptor.
1500 def fromStringProto(self, inString, proto):
1502 Take a unique identifier associated with a file descriptor which must
1503 have been received by now and use it to look up that descriptor in a
1504 dictionary where they are kept.
1506 @param inString: The base representation (as a byte string) of an
1507 ordinal indicating which file descriptor corresponds to this usage
1509 @type inString: C{str}
1511 @param proto: The protocol used to receive this descriptor. This
1512 protocol must be connected via a transport providing
1513 L{IUNIXTransport<twisted.internet.interfaces.IUNIXTransport>}.
1514 @type proto: L{BinaryBoxProtocol}
1516 @return: The file descriptor represented by C{inString}.
1519 return proto._getDescriptor(int(inString))
1522 def toStringProto(self, inObject, proto):
1524 Send C{inObject}, an integer file descriptor, over C{proto}'s connection
1525 and return a unique identifier which will allow the receiver to
1526 associate the file descriptor with this argument.
1528 @param inObject: A file descriptor to duplicate over an AMP connection
1529 as the value for this argument.
1530 @type inObject: C{int}
1532 @param proto: The protocol which will be used to send this descriptor.
1533 This protocol must be connected via a transport providing
1534 L{IUNIXTransport<twisted.internet.interfaces.IUNIXTransport>}.
1536 @return: A byte string which can be used by the receiver to reconstruct
1537 the file descriptor.
1540 identifier = proto._sendFileDescriptor(inObject)
1541 outString = Integer.toStringProto(self, identifier, proto)
1548 Subclass me to specify an AMP Command.
1550 @cvar arguments: A list of 2-tuples of (name, Argument-subclass-instance),
1551 specifying the names and values of the parameters which are required for
1554 @cvar response: A list like L{arguments}, but instead used for the return
1557 @cvar errors: A mapping of subclasses of L{Exception} to wire-protocol tags
1558 for errors represented as L{str}s. Responders which raise keys from this
1559 dictionary will have the error translated to the corresponding tag on the
1560 wire. Invokers which receive Deferreds from invoking this command with
1561 L{AMP.callRemote} will potentially receive Failures with keys from this
1562 mapping as their value. This mapping is inherited; if you declare a
1563 command which handles C{FooError} as 'FOO_ERROR', then subclass it and
1564 specify C{BarError} as 'BAR_ERROR', responders to the subclass may raise
1565 either C{FooError} or C{BarError}, and invokers must be able to deal with
1566 either of those exceptions.
1568 @cvar fatalErrors: like 'errors', but errors in this list will always
1569 terminate the connection, despite being of a recognizable error type.
1571 @cvar commandType: The type of Box used to issue commands; useful only for
1572 protocol-modifying behavior like startTLS or protocol switching. Defaults
1573 to a plain vanilla L{Box}.
1575 @cvar responseType: The type of Box used to respond to this command; only
1576 useful for protocol-modifying behavior like startTLS or protocol switching.
1577 Defaults to a plain vanilla L{Box}.
1579 @ivar requiresAnswer: a boolean; defaults to True. Set it to False on your
1580 subclass if you want callRemote to return None. Note: this is a hint only
1581 to the client side of the protocol. The return-type of a command responder
1582 method must always be a dictionary adhering to the contract specified by
1583 L{response}, because clients are always free to request a response if they
1587 class __metaclass__(type):
1589 Metaclass hack to establish reverse-mappings for 'errors' and
1590 'fatalErrors' as class vars.
1592 def __new__(cls, name, bases, attrs):
1593 reverseErrors = attrs['reverseErrors'] = {}
1594 er = attrs['allErrors'] = {}
1595 if 'commandName' not in attrs:
1596 attrs['commandName'] = name
1597 newtype = type.__new__(cls, name, bases, attrs)
1600 accumulateClassDict(newtype, 'errors', errors)
1601 accumulateClassDict(newtype, 'fatalErrors', fatalErrors)
1602 for v, k in errors.iteritems():
1603 reverseErrors[k] = v
1605 for v, k in fatalErrors.iteritems():
1606 reverseErrors[k] = v
1619 requiresAnswer = True
1622 def __init__(self, **kw):
1624 Create an instance of this command with specified values for its
1627 @param kw: a dict containing an appropriate value for each name
1628 specified in the L{arguments} attribute of my class.
1630 @raise InvalidSignature: if you forgot any required arguments.
1632 self.structured = kw
1633 givenArgs = kw.keys()
1635 for name, arg in self.arguments:
1636 pythonName = _wireNameToPythonIdentifier(name)
1637 if pythonName not in givenArgs and not arg.optional:
1638 forgotten.append(pythonName)
1640 raise InvalidSignature("forgot %s for %s" % (
1641 ', '.join(forgotten), self.commandName))
1645 def makeResponse(cls, objects, proto):
1647 Serialize a mapping of arguments using this L{Command}'s
1650 @param objects: a dict with keys matching the names specified in
1651 self.response, having values of the types that the Argument objects in
1652 self.response can format.
1654 @param proto: an L{AMP}.
1656 @return: an L{AmpBox}.
1659 responseType = cls.responseType()
1662 return _objectsToStrings(objects, cls.response, responseType, proto)
1663 makeResponse = classmethod(makeResponse)
1666 def makeArguments(cls, objects, proto):
1668 Serialize a mapping of arguments using this L{Command}'s
1671 @param objects: a dict with keys similar to the names specified in
1672 self.arguments, having values of the types that the Argument objects in
1673 self.arguments can parse.
1675 @param proto: an L{AMP}.
1677 @return: An instance of this L{Command}'s C{commandType}.
1679 allowedNames = set()
1680 for (argName, ignored) in cls.arguments:
1681 allowedNames.add(_wireNameToPythonIdentifier(argName))
1683 for intendedArg in objects:
1684 if intendedArg not in allowedNames:
1685 raise InvalidSignature(
1686 "%s is not a valid argument" % (intendedArg,))
1687 return _objectsToStrings(objects, cls.arguments, cls.commandType(),
1689 makeArguments = classmethod(makeArguments)
1692 def parseResponse(cls, box, protocol):
1694 Parse a mapping of serialized arguments using this
1695 L{Command}'s response schema.
1697 @param box: A mapping of response-argument names to the
1698 serialized forms of those arguments.
1699 @param protocol: The L{AMP} protocol.
1701 @return: A mapping of response-argument names to the parsed
1704 return _stringsToObjects(box, cls.response, protocol)
1705 parseResponse = classmethod(parseResponse)
1708 def parseArguments(cls, box, protocol):
1710 Parse a mapping of serialized arguments using this
1711 L{Command}'s argument schema.
1713 @param box: A mapping of argument names to the seralized forms
1715 @param protocol: The L{AMP} protocol.
1717 @return: A mapping of argument names to the parsed forms.
1719 return _stringsToObjects(box, cls.arguments, protocol)
1720 parseArguments = classmethod(parseArguments)
1723 def responder(cls, methodfunc):
1725 Declare a method to be a responder for a particular command.
1727 This is a decorator.
1731 class MyCommand(Command):
1732 arguments = [('a', ...), ('b', ...)]
1735 def myFunMethod(self, a, b):
1737 MyCommand.responder(myFunMethod)
1739 Notes: Although decorator syntax is not used within Twisted, this
1740 function returns its argument and is therefore safe to use with
1743 This is not thread safe. Don't declare AMP subclasses in other
1744 threads. Don't declare responders outside the scope of AMP subclasses;
1745 the behavior is undefined.
1747 @param methodfunc: A function which will later become a method, which
1748 has a keyword signature compatible with this command's L{argument} list
1749 and returns a dictionary with a set of keys compatible with this
1750 command's L{response} list.
1752 @return: the methodfunc parameter.
1754 CommandLocator._currentClassCommands.append((cls, methodfunc))
1756 responder = classmethod(responder)
1759 # Our only instance method
1760 def _doCommand(self, proto):
1762 Encode and send this Command to the given protocol.
1764 @param proto: an AMP, representing the connection to send to.
1766 @return: a Deferred which will fire or error appropriately when the
1767 other side responds to the command (or error if the connection is lost
1768 before it is responded to).
1771 def _massageError(error):
1772 error.trap(RemoteAmpError)
1774 errorType = self.reverseErrors.get(rje.errorCode,
1776 return Failure(errorType(rje.description))
1778 d = proto._sendBoxCommand(self.commandName,
1779 self.makeArguments(self.structured, proto),
1780 self.requiresAnswer)
1782 if self.requiresAnswer:
1783 d.addCallback(self.parseResponse, proto)
1784 d.addErrback(_massageError)
1790 class _NoCertificate:
1792 This is for peers which don't want to use a local certificate. Used by
1793 AMP because AMP's internal language is all about certificates and this
1794 duck-types in the appropriate place; this API isn't really stable though,
1795 so it's not exposed anywhere public.
1797 For clients, it will use ephemeral DH keys, or whatever the default is for
1798 certificate-less clients in OpenSSL. For servers, it will generate a
1799 temporary self-signed certificate with garbage values in the DN and use
1803 def __init__(self, client):
1805 Create a _NoCertificate which either is or isn't for the client side of
1808 @param client: True if we are a client and should truly have no
1809 certificate and be anonymous, False if we are a server and actually
1810 have to generate a temporary certificate.
1814 self.client = client
1817 def options(self, *authorities):
1819 Behaves like L{twisted.internet.ssl.PrivateCertificate.options}().
1822 # do some crud with sslverify to generate a temporary self-signed
1823 # certificate. This is SLOOOWWWWW so it is only in the absolute
1824 # worst, most naive case.
1826 # We have to do this because OpenSSL will not let both the server
1827 # and client be anonymous.
1828 sharedDN = DN(CN='TEMPORARY CERTIFICATE')
1829 key = KeyPair.generate()
1830 cr = key.certificateRequest(sharedDN)
1831 sscrd = key.signCertificateRequest(sharedDN, cr, lambda dn: True, 1)
1832 cert = key.newCertificate(sscrd)
1833 return cert.options(*authorities)
1836 options.update(dict(verify=True,
1837 requireCertificate=True,
1838 caCerts=[auth.original for auth in authorities]))
1839 occo = CertificateOptions(**options)
1844 class _TLSBox(AmpBox):
1846 I am an AmpBox that, upon being sent, initiates a TLS connection.
1852 raise RemoteAmpError("TLS_ERROR", "TLS not available")
1853 AmpBox.__init__(self)
1856 def _keyprop(k, default):
1857 return property(lambda self: self.get(k, default))
1860 # These properties are described in startTLS
1861 certificate = _keyprop('tls_localCertificate', _NoCertificate(False))
1862 verify = _keyprop('tls_verifyAuthorities', None)
1864 def _sendTo(self, proto):
1866 Send my encoded value to the protocol, then initiate TLS.
1869 for k in ['tls_localCertificate',
1870 'tls_verifyAuthorities']:
1873 proto._startTLS(self.certificate, self.verify)
1877 class _LocalArgument(String):
1879 Local arguments are never actually relayed across the wire. This is just a
1880 shim so that StartTLS can pretend to have some arguments: if arguments
1881 acquire documentation properties, replace this with something nicer later.
1884 def fromBox(self, name, strings, objects, proto):
1889 class StartTLS(Command):
1891 Use, or subclass, me to implement a command that starts TLS.
1893 Callers of StartTLS may pass several special arguments, which affect the
1896 - tls_localCertificate: This is a
1897 twisted.internet.ssl.PrivateCertificate which will be used to secure
1898 the side of the connection it is returned on.
1900 - tls_verifyAuthorities: This is a list of
1901 twisted.internet.ssl.Certificate objects that will be used as the
1902 certificate authorities to verify our peer's certificate.
1904 Each of those special parameters may also be present as a key in the
1905 response dictionary.
1908 arguments = [("tls_localCertificate", _LocalArgument(optional=True)),
1909 ("tls_verifyAuthorities", _LocalArgument(optional=True))]
1911 response = [("tls_localCertificate", _LocalArgument(optional=True)),
1912 ("tls_verifyAuthorities", _LocalArgument(optional=True))]
1914 responseType = _TLSBox
1916 def __init__(self, **kw):
1918 Create a StartTLS command. (This is private. Use AMP.callRemote.)
1920 @param tls_localCertificate: the PrivateCertificate object to use to
1921 secure the connection. If it's None, or unspecified, an ephemeral DH
1922 key is used instead.
1924 @param tls_verifyAuthorities: a list of Certificate objects which
1925 represent root certificates to verify our peer with.
1928 raise RuntimeError("TLS not available.")
1929 self.certificate = kw.pop('tls_localCertificate', _NoCertificate(True))
1930 self.authorities = kw.pop('tls_verifyAuthorities', None)
1931 Command.__init__(self, **kw)
1934 def _doCommand(self, proto):
1936 When a StartTLS command is sent, prepare to start TLS, but don't actually
1937 do it; wait for the acknowledgement, then initiate the TLS handshake.
1939 d = Command._doCommand(self, proto)
1940 proto._prepareTLS(self.certificate, self.authorities)
1941 # XXX before we get back to user code we are going to start TLS...
1942 def actuallystart(response):
1943 proto._startTLS(self.certificate, self.authorities)
1945 d.addCallback(actuallystart)
1950 class ProtocolSwitchCommand(Command):
1952 Use this command to switch from something Amp-derived to a different
1953 protocol mid-connection. This can be useful to use amp as the
1954 connection-startup negotiation phase. Since TLS is a different layer
1955 entirely, you can use Amp to negotiate the security parameters of your
1956 connection, then switch to a different protocol, and the connection will
1960 def __init__(self, _protoToSwitchToFactory, **kw):
1962 Create a ProtocolSwitchCommand.
1964 @param _protoToSwitchToFactory: a ProtocolFactory which will generate
1965 the Protocol to switch to.
1967 @param kw: Keyword arguments, encoded and handled normally as
1971 self.protoToSwitchToFactory = _protoToSwitchToFactory
1972 super(ProtocolSwitchCommand, self).__init__(**kw)
1975 def makeResponse(cls, innerProto, proto):
1976 return _SwitchBox(innerProto)
1977 makeResponse = classmethod(makeResponse)
1980 def _doCommand(self, proto):
1982 When we emit a ProtocolSwitchCommand, lock the protocol, but don't actually
1983 switch to the new protocol unless an acknowledgement is received. If
1984 an error is received, switch back.
1986 d = super(ProtocolSwitchCommand, self)._doCommand(proto)
1987 proto._lockForSwitch()
1989 innerProto = self.protoToSwitchToFactory.buildProtocol(
1990 proto.transport.getPeer())
1991 proto._switchTo(innerProto, self.protoToSwitchToFactory)
1994 proto._unlockFromSwitch()
1995 self.protoToSwitchToFactory.clientConnectionFailed(
1996 None, Failure(CONNECTION_LOST))
1998 return d.addCallbacks(switchNow, handle)
2002 class _DescriptorExchanger(object):
2004 L{_DescriptorExchanger} is a mixin for L{BinaryBoxProtocol} which adds
2005 support for receiving file descriptors, a feature offered by
2006 L{IUNIXTransport<twisted.internet.interfaces.IUNIXTransport>}.
2008 @ivar _descriptors: Temporary storage for all file descriptors received.
2009 Values in this dictionary are the file descriptors (as integers). Keys
2010 in this dictionary are ordinals giving the order in which each
2011 descriptor was received. The ordering information is used to allow
2012 L{Descriptor} to determine which is the correct descriptor for any
2013 particular usage of that argument type.
2014 @type _descriptors: C{dict}
2016 @ivar _sendingDescriptorCounter: A no-argument callable which returns the
2017 ordinals, starting from 0. This is used to construct values for
2018 C{_sendFileDescriptor}.
2020 @ivar _receivingDescriptorCounter: A no-argument callable which returns the
2021 ordinals, starting from 0. This is used to construct values for
2022 C{fileDescriptorReceived}.
2024 implements(IFileDescriptorReceiver)
2027 self._descriptors = {}
2028 self._getDescriptor = self._descriptors.pop
2029 self._sendingDescriptorCounter = count().next
2030 self._receivingDescriptorCounter = count().next
2033 def _sendFileDescriptor(self, descriptor):
2035 Assign and return the next ordinal to the given descriptor after sending
2036 the descriptor over this protocol's transport.
2038 self.transport.sendFileDescriptor(descriptor)
2039 return self._sendingDescriptorCounter()
2042 def fileDescriptorReceived(self, descriptor):
2044 Collect received file descriptors to be claimed later by L{Descriptor}.
2046 @param descriptor: The received file descriptor.
2047 @type descriptor: C{int}
2049 self._descriptors[self._receivingDescriptorCounter()] = descriptor
2053 class BinaryBoxProtocol(StatefulStringProtocol, Int16StringReceiver,
2054 _DescriptorExchanger):
2056 A protocol for receiving L{AmpBox}es - key/value pairs - via length-prefixed
2057 strings. A box is composed of:
2059 - any number of key-value pairs, described by:
2060 - a 2-byte network-endian packed key length (of which the first
2061 byte must be null, and the second must be non-null: i.e. the
2062 value of the length must be 1-255)
2063 - a key, comprised of that many bytes
2064 - a 2-byte network-endian unsigned value length (up to the maximum
2066 - a value, comprised of that many bytes
2069 In other words, an even number of strings prefixed with packed unsigned
2070 16-bit integers, and then a 0-length string to indicate the end of the box.
2072 This protocol also implements 2 extra private bits of functionality related
2073 to the byte boundaries between messages; it can start TLS between two given
2074 boxes or switch to an entirely different protocol. However, due to some
2075 tricky elements of the implementation, the public interface to this
2076 functionality is L{ProtocolSwitchCommand} and L{StartTLS}.
2078 @ivar _keyLengthLimitExceeded: A flag which is only true when the
2079 connection is being closed because a key length prefix which was longer
2080 than allowed by the protocol was received.
2082 @ivar boxReceiver: an L{IBoxReceiver} provider, whose L{ampBoxReceived}
2083 method will be invoked for each L{AmpBox} that is received.
2086 implements(IBoxSender)
2088 _justStartedTLS = False
2089 _startingTLSBuffer = None
2094 _keyLengthLimitExceeded = False
2096 hostCertificate = None
2097 noPeerCertificate = False # for tests
2098 innerProtocol = None
2099 innerProtocolClientFactory = None
2101 def __init__(self, boxReceiver):
2102 _DescriptorExchanger.__init__(self)
2103 self.boxReceiver = boxReceiver
2106 def _switchTo(self, newProto, clientFactory=None):
2108 Switch this BinaryBoxProtocol's transport to a new protocol. You need
2109 to do this 'simultaneously' on both ends of a connection; the easiest
2110 way to do this is to use a subclass of ProtocolSwitchCommand.
2112 @param newProto: the new protocol instance to switch to.
2114 @param clientFactory: the ClientFactory to send the
2115 L{clientConnectionLost} notification to.
2117 # All the data that Int16Receiver has not yet dealt with belongs to our
2118 # new protocol: luckily it's keeping that in a handy (although
2119 # ostensibly internal) variable for us:
2120 newProtoData = self.recvd
2121 # We're quite possibly in the middle of a 'dataReceived' loop in
2122 # Int16StringReceiver: let's make sure that the next iteration, the
2123 # loop will break and not attempt to look at something that isn't a
2126 # Finally, do the actual work of setting up the protocol and delivering
2127 # its first chunk of data, if one is available.
2128 self.innerProtocol = newProto
2129 self.innerProtocolClientFactory = clientFactory
2130 newProto.makeConnection(self.transport)
2132 newProto.dataReceived(newProtoData)
2135 def sendBox(self, box):
2137 Send a amp.Box to my peer.
2139 Note: transport.write is never called outside of this method.
2141 @param box: an AmpBox.
2143 @raise ProtocolSwitched: if the protocol has previously been switched.
2145 @raise ConnectionLost: if the connection has previously been lost.
2148 raise ProtocolSwitched(
2149 "This connection has switched: no AMP traffic allowed.")
2150 if self.transport is None:
2151 raise ConnectionLost()
2152 if self._startingTLSBuffer is not None:
2153 self._startingTLSBuffer.append(box)
2155 self.transport.write(box.serialize())
2158 def makeConnection(self, transport):
2160 Notify L{boxReceiver} that it is about to receive boxes from this
2161 protocol by invoking L{startReceivingBoxes}.
2163 self.transport = transport
2164 self.boxReceiver.startReceivingBoxes(self)
2165 self.connectionMade()
2168 def dataReceived(self, data):
2170 Either parse incoming data as L{AmpBox}es or relay it to our nested
2173 if self._justStartedTLS:
2174 self._justStartedTLS = False
2175 # If we already have an inner protocol, then we don't deliver data to
2176 # the protocol parser any more; we just hand it off.
2177 if self.innerProtocol is not None:
2178 self.innerProtocol.dataReceived(data)
2180 return Int16StringReceiver.dataReceived(self, data)
2183 def connectionLost(self, reason):
2185 The connection was lost; notify any nested protocol.
2187 if self.innerProtocol is not None:
2188 self.innerProtocol.connectionLost(reason)
2189 if self.innerProtocolClientFactory is not None:
2190 self.innerProtocolClientFactory.clientConnectionLost(None, reason)
2191 if self._keyLengthLimitExceeded:
2192 failReason = Failure(TooLong(True, False, None, None))
2193 elif reason.check(ConnectionClosed) and self._justStartedTLS:
2194 # We just started TLS and haven't received any data. This means
2195 # the other connection didn't like our cert (although they may not
2196 # have told us why - later Twisted should make 'reason' into a TLS
2198 failReason = PeerVerifyError(
2199 "Peer rejected our certificate for an unknown reason.")
2202 self.boxReceiver.stopReceivingBoxes(failReason)
2205 # The longest key allowed
2206 _MAX_KEY_LENGTH = 255
2208 # The longest value allowed (this is somewhat redundant, as longer values
2209 # cannot be encoded - ah well).
2210 _MAX_VALUE_LENGTH = 65535
2212 # The first thing received is a key.
2213 MAX_LENGTH = _MAX_KEY_LENGTH
2215 def proto_init(self, string):
2217 String received in the 'init' state.
2219 self._currentBox = AmpBox()
2220 return self.proto_key(string)
2223 def proto_key(self, string):
2225 String received in the 'key' state. If the key is empty, a complete
2226 box has been received.
2229 self._currentKey = string
2230 self.MAX_LENGTH = self._MAX_VALUE_LENGTH
2233 self.boxReceiver.ampBoxReceived(self._currentBox)
2234 self._currentBox = None
2238 def proto_value(self, string):
2240 String received in the 'value' state.
2242 self._currentBox[self._currentKey] = string
2243 self._currentKey = None
2244 self.MAX_LENGTH = self._MAX_KEY_LENGTH
2248 def lengthLimitExceeded(self, length):
2250 The key length limit was exceeded. Disconnect the transport and make
2251 sure a meaningful exception is reported.
2253 self._keyLengthLimitExceeded = True
2254 self.transport.loseConnection()
2257 def _lockForSwitch(self):
2259 Lock this binary protocol so that no further boxes may be sent. This
2260 is used when sending a request to switch underlying protocols. You
2261 probably want to subclass ProtocolSwitchCommand rather than calling
2267 def _unlockFromSwitch(self):
2269 Unlock this locked binary protocol so that further boxes may be sent
2270 again. This is used after an attempt to switch protocols has failed
2273 if self.innerProtocol is not None:
2274 raise ProtocolSwitched("Protocol already switched. Cannot unlock.")
2275 self._locked = False
2278 def _prepareTLS(self, certificate, verifyAuthorities):
2280 Used by StartTLSCommand to put us into the state where we don't
2281 actually send things that get sent, instead we buffer them. see
2284 self._startingTLSBuffer = []
2285 if self.hostCertificate is not None:
2287 "Previously authenticated connection between %s and %s "
2288 "is trying to re-establish as %s" % (
2289 self.hostCertificate,
2290 self.peerCertificate,
2291 (certificate, verifyAuthorities)))
2294 def _startTLS(self, certificate, verifyAuthorities):
2296 Used by TLSBox to initiate the SSL handshake.
2298 @param certificate: a L{twisted.internet.ssl.PrivateCertificate} for
2301 @param verifyAuthorities: L{twisted.internet.ssl.Certificate} instances
2302 representing certificate authorities which will verify our peer.
2304 self.hostCertificate = certificate
2305 self._justStartedTLS = True
2306 if verifyAuthorities is None:
2307 verifyAuthorities = ()
2308 self.transport.startTLS(certificate.options(*verifyAuthorities))
2309 stlsb = self._startingTLSBuffer
2310 if stlsb is not None:
2311 self._startingTLSBuffer = None
2316 def _getPeerCertificate(self):
2317 if self.noPeerCertificate:
2319 return Certificate.peerFromTransport(self.transport)
2320 peerCertificate = property(_getPeerCertificate)
2323 def unhandledError(self, failure):
2325 The buck stops here. This error was completely unhandled, time to
2326 terminate the connection.
2330 "Amp server or network failure unhandled by client application. "
2331 "Dropping connection! To avoid, add errbacks to ALL remote "
2333 if self.transport is not None:
2334 self.transport.loseConnection()
2337 def _defaultStartTLSResponder(self):
2339 The default TLS responder doesn't specify any certificate or anything.
2341 From a security perspective, it's little better than a plain-text
2342 connection - but it is still a *bit* better, so it's included for
2345 You probably want to override this by providing your own StartTLS.responder.
2348 StartTLS.responder(_defaultStartTLSResponder)
2352 class AMP(BinaryBoxProtocol, BoxDispatcher,
2353 CommandLocator, SimpleStringLocator):
2355 This protocol is an AMP connection. See the module docstring for protocol
2359 _ampInitialized = False
2361 def __init__(self, boxReceiver=None, locator=None):
2362 # For backwards compatibility. When AMP did not separate parsing logic
2363 # (L{BinaryBoxProtocol}), request-response logic (L{BoxDispatcher}) and
2364 # command routing (L{CommandLocator}), it did not have a constructor.
2365 # Now it does, so old subclasses might have defined their own that did
2366 # not upcall. If this flag isn't set, we'll call the constructor in
2367 # makeConnection before anything actually happens.
2368 self._ampInitialized = True
2369 if boxReceiver is None:
2373 BoxDispatcher.__init__(self, locator)
2374 BinaryBoxProtocol.__init__(self, boxReceiver)
2377 def locateResponder(self, name):
2379 Unify the implementations of L{CommandLocator} and
2380 L{SimpleStringLocator} to perform both kinds of dispatch, preferring
2383 firstResponder = CommandLocator.locateResponder(self, name)
2384 if firstResponder is not None:
2385 return firstResponder
2386 secondResponder = SimpleStringLocator.locateResponder(self, name)
2387 return secondResponder
2392 A verbose string representation which gives us information about this
2395 if self.innerProtocol is not None:
2396 innerRepr = ' inner %r' % (self.innerProtocol,)
2399 return '<%s%s at 0x%x>' % (
2400 self.__class__.__name__, innerRepr, unsignedID(self))
2403 def makeConnection(self, transport):
2405 Emit a helpful log message when the connection is made.
2407 if not self._ampInitialized:
2408 # See comment in the constructor re: backward compatibility. I
2409 # should probably emit a deprecation warning here.
2411 # Save these so we can emit a similar log message in L{connectionLost}.
2412 self._transportPeer = transport.getPeer()
2413 self._transportHost = transport.getHost()
2414 log.msg("%s connection established (HOST:%s PEER:%s)" % (
2415 self.__class__.__name__,
2416 self._transportHost,
2417 self._transportPeer))
2418 BinaryBoxProtocol.makeConnection(self, transport)
2421 def connectionLost(self, reason):
2423 Emit a helpful log message when the connection is lost.
2425 log.msg("%s connection lost (HOST:%s PEER:%s)" %
2426 (self.__class__.__name__,
2427 self._transportHost,
2428 self._transportPeer))
2429 BinaryBoxProtocol.connectionLost(self, reason)
2430 self.transport = None
2434 class _ParserHelper:
2436 A box receiver which records all boxes received.
2449 disconnecting = False
2452 def startReceivingBoxes(self, sender):
2454 No initialization is required.
2458 def ampBoxReceived(self, box):
2459 self.boxes.append(box)
2462 # Synchronous helpers
2463 def parse(cls, fileObj):
2465 Parse some amp data stored in a file.
2467 @param fileObj: a file-like object.
2469 @return: a list of AmpBoxes encoded in the given file.
2471 parserHelper = cls()
2472 bbp = BinaryBoxProtocol(boxReceiver=parserHelper)
2473 bbp.makeConnection(parserHelper)
2474 bbp.dataReceived(fileObj.read())
2475 return parserHelper.boxes
2476 parse = classmethod(parse)
2479 def parseString(cls, data):
2481 Parse some amp data stored in a string.
2483 @param data: a str holding some amp-encoded data.
2485 @return: a list of AmpBoxes encoded in the given string.
2487 return cls.parse(StringIO(data))
2488 parseString = classmethod(parseString)
2492 parse = _ParserHelper.parse
2493 parseString = _ParserHelper.parseString
2495 def _stringsToObjects(strings, arglist, proto):
2497 Convert an AmpBox to a dictionary of python objects, converting through a
2500 @param strings: an AmpBox (or dict of strings)
2502 @param arglist: a list of 2-tuples of strings and Argument objects, as
2503 described in L{Command.arguments}.
2505 @param proto: an L{AMP} instance.
2507 @return: the converted dictionary mapping names to argument objects.
2510 myStrings = strings.copy()
2511 for argname, argparser in arglist:
2512 argparser.fromBox(argname, myStrings, objects, proto)
2517 def _objectsToStrings(objects, arglist, strings, proto):
2519 Convert a dictionary of python objects to an AmpBox, converting through a
2522 @param objects: a dict mapping names to python objects
2524 @param arglist: a list of 2-tuples of strings and Argument objects, as
2525 described in L{Command.arguments}.
2527 @param strings: [OUT PARAMETER] An object providing the L{dict}
2528 interface which will be populated with serialized data.
2530 @param proto: an L{AMP} instance.
2532 @return: The converted dictionary mapping names to encoded argument
2533 strings (identical to C{strings}).
2535 myObjects = objects.copy()
2536 for argname, argparser in arglist:
2537 argparser.toBox(argname, strings, myObjects, proto)
2542 class _FixedOffsetTZInfo(datetime.tzinfo):
2544 Represents a fixed timezone offset (without daylight saving time).
2546 @ivar name: A C{str} giving the name of this timezone; the name just
2547 includes how much time this offset represents.
2549 @ivar offset: A C{datetime.timedelta} giving the amount of time this
2553 def __init__(self, sign, hours, minutes):
2554 self.name = '%s%02i:%02i' % (sign, hours, minutes)
2559 raise ValueError('invalid sign for timezone %r' % (sign,))
2560 self.offset = datetime.timedelta(hours=hours, minutes=minutes)
2563 def utcoffset(self, dt):
2565 Return this timezone's offset from UTC.
2572 Return a zero C{datetime.timedelta} for the daylight saving time offset,
2573 since there is never one.
2575 return datetime.timedelta(0)
2578 def tzname(self, dt):
2580 Return a string describing this timezone.
2586 utc = _FixedOffsetTZInfo('+', 0, 0)
2590 class Decimal(Argument):
2592 Encodes C{decimal.Decimal} instances.
2594 There are several ways in which a decimal value might be encoded.
2596 Special values are encoded as special strings::
2598 - Positive infinity is encoded as C{"Infinity"}
2599 - Negative infinity is encoded as C{"-Infinity"}
2600 - Quiet not-a-number is encoded as either C{"NaN"} or C{"-NaN"}
2601 - Signalling not-a-number is encoded as either C{"sNaN"} or C{"-sNaN"}
2603 Normal values are encoded using the base ten string representation, using
2604 engineering notation to indicate magnitude without precision, and "normal"
2605 digits to indicate precision. For example::
2607 - C{"1"} represents the value I{1} with precision to one place.
2608 - C{"-1"} represents the value I{-1} with precision to one place.
2609 - C{"1.0"} represents the value I{1} with precision to two places.
2610 - C{"10"} represents the value I{10} with precision to two places.
2611 - C{"1E+2"} represents the value I{10} with precision to one place.
2612 - C{"1E-1"} represents the value I{0.1} with precision to one place.
2613 - C{"1.5E+2"} represents the value I{15} with precision to two places.
2615 U{http://speleotrove.com/decimal/} should be considered the authoritative
2616 specification for the format.
2618 fromString = decimal.Decimal
2620 def toString(self, inObject):
2622 Serialize a C{decimal.Decimal} instance to the specified wire format.
2624 if isinstance(inObject, decimal.Decimal):
2625 # Hopefully decimal.Decimal.__str__ actually does what we want.
2626 return str(inObject)
2628 "amp.Decimal can only encode instances of decimal.Decimal")
2632 class DateTime(Argument):
2634 Encodes C{datetime.datetime} instances.
2636 Wire format: '%04i-%02i-%02iT%02i:%02i:%02i.%06i%s%02i:%02i'. Fields in
2637 order are: year, month, day, hour, minute, second, microsecond, timezone
2638 direction (+ or -), timezone hour, timezone minute. Encoded string is
2639 always exactly 32 characters long. This format is compatible with ISO 8601,
2640 but that does not mean all ISO 8601 dates can be accepted.
2642 Also, note that the datetime module's notion of a "timezone" can be
2643 complex, but the wire format includes only a fixed offset, so the
2644 conversion is not lossless. A lossless transmission of a C{datetime} instance
2645 is not feasible since the receiving end would require a Python interpreter.
2647 @ivar _positions: A sequence of slices giving the positions of various
2648 interesting parts of the wire format.
2652 slice(0, 4), slice(5, 7), slice(8, 10), # year, month, day
2653 slice(11, 13), slice(14, 16), slice(17, 19), # hour, minute, second
2654 slice(20, 26), # microsecond
2655 # intentionally skip timezone direction, as it is not an integer
2656 slice(27, 29), slice(30, 32) # timezone hour, timezone minute
2659 def fromString(self, s):
2661 Parse a string containing a date and time in the wire format into a
2662 C{datetime.datetime} instance.
2665 raise ValueError('invalid date format %r' % (s,))
2667 values = [int(s[p]) for p in self._positions]
2669 timezone = _FixedOffsetTZInfo(sign, *values[7:])
2670 values[7:] = [timezone]
2671 return datetime.datetime(*values)
2674 def toString(self, i):
2676 Serialize a C{datetime.datetime} instance to a string in the specified
2679 offset = i.utcoffset()
2682 'amp.DateTime cannot serialize naive datetime instances. '
2683 'You may find amp.utc useful.')
2685 minutesOffset = (offset.days * 86400 + offset.seconds) // 60
2687 if minutesOffset > 0:
2692 # strftime has no way to format the microseconds, or put a ':' in the
2693 # timezone. Suprise!
2695 return '%04i-%02i-%02iT%02i:%02i:%02i.%06i%s%02i:%02i' % (
2704 abs(minutesOffset) // 60,
2705 abs(minutesOffset) % 60)