From: DongHun Kwak
Date: Wed, 9 Dec 2020 01:06:55 +0000 (+0900)
Subject: Imported Upstream version 3.9.1
X-Git-Tag: upstream/3.9.1^0
X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=1e5655582c5ecbdf15f6c33507dae77560180029;p=platform%2Fupstream%2Fpython3.git
Imported Upstream version 3.9.1
---
diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst
index ea027ee..ed3735a 100644
--- a/Doc/c-api/file.rst
+++ b/Doc/c-api/file.rst
@@ -82,6 +82,8 @@ the :mod:`io` APIs instead.
This function is safe to call before :c:func:`Py_Initialize`.
+ .. audit-event:: setopencodehook "" c.PyFile_SetOpenCodeHook
+
.. versionadded:: 3.8
diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index 73f2687..822360e 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -155,7 +155,8 @@ The following functions and structs are used to create
If *bases* is a tuple, the created heap type contains all types contained
in it as base types.
- If *bases* is ``NULL``, the *Py_tp_base* slot is used instead.
+ If *bases* is ``NULL``, the *Py_tp_bases* slot is used instead.
+ If that also is ``NULL``, the *Py_tp_base* slot is used instead.
If that also is ``NULL``, the new type derives from :class:`object`.
The *module* argument can be used to record the module in which the new
@@ -247,7 +248,8 @@ The following functions and structs are used to create
* :c:member:`~PyBufferProcs.bf_getbuffer`
* :c:member:`~PyBufferProcs.bf_releasebuffer`
- Setting :c:data:`Py_tp_bases` may be problematic on some platforms.
+ Setting :c:data:`Py_tp_bases` or :c:data:`Py_tp_base` may be
+ problematic on some platforms.
To avoid issues, use the *bases* argument of
:py:func:`PyType_FromSpecWithBases` instead.
diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat
index 4dacbe2..aac1351 100644
--- a/Doc/data/refcounts.dat
+++ b/Doc/data/refcounts.dat
@@ -2306,6 +2306,11 @@ PyType_CheckExact:PyObject*:o:0:
PyType_FromSpec:PyObject*::+1:
PyType_FromSpec:PyType_Spec*:spec::
+PyType_FromModuleAndSpec:PyObject*::+1:
+PyType_FromModuleAndSpec:PyObject*:module:+1:
+PyType_FromModuleAndSpec:PyType_Spec*:spec::
+PyType_FromModuleAndSpec:PyObject*:bases:0:
+
PyType_FromSpecWithBases:PyObject*::+1:
PyType_FromSpecWithBases:PyType_Spec*:spec::
PyType_FromSpecWithBases:PyObject*:bases:0:
diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
index 66d210a..8df62c5 100644
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -942,7 +942,7 @@ There are various techniques.
f()
-* Use :func:`locals` or :func:`eval` to resolve the function name::
+* Use :func:`locals` to resolve the function name::
def myFunc():
print("hello")
@@ -952,12 +952,6 @@ There are various techniques.
f = locals()[fname]
f()
- f = eval(fname)
- f()
-
- Note: Using :func:`eval` is slow and dangerous. If you don't have absolute
- control over the contents of the string, someone could pass a string that
- resulted in an arbitrary function being executed.
Is there an equivalent to Perl's chomp() for removing trailing newlines from strings?
-------------------------------------------------------------------------------------
@@ -1122,7 +1116,7 @@ trailing newline from a string.
How do I iterate over a sequence in reverse order?
--------------------------------------------------
-Use the :func:`reversed` built-in function, which is new in Python 2.4::
+Use the :func:`reversed` built-in function::
for x in reversed(sequence):
... # do something with x ...
@@ -1130,11 +1124,6 @@ Use the :func:`reversed` built-in function, which is new in Python 2.4::
This won't touch your original sequence, but build a new copy with reversed
order to iterate over.
-With Python 2.3, you can use an extended slice syntax::
-
- for x in sequence[::-1]:
- ... # do something with x ...
-
How do you remove duplicates from a list?
-----------------------------------------
@@ -1164,6 +1153,21 @@ This converts the list into a set, thereby removing duplicates, and then back
into a list.
+How do you remove multiple items from a list
+--------------------------------------------
+
+As with removing duplicates, explicitly iterating in reverse with a
+delete condition is one possibility. However, it is easier and faster
+to use slice replacement with an implicit or explicit forward iteration.
+Here are three variations.::
+
+ mylist[:] = filter(keep_function, mylist)
+ mylist[:] = (x for x in mylist if keep_condition)
+ mylist[:] = [x for x in mylist if keep_condition]
+
+The list comprehension may be fastest.
+
+
How do you make an array in Python?
-----------------------------------
@@ -1366,20 +1370,6 @@ out the element you want. ::
['else', 'sort', 'to', 'something']
-An alternative for the last step is::
-
- >>> result = []
- >>> for p in pairs: result.append(p[1])
-
-If you find this more legible, you might prefer to use this instead of the final
-list comprehension. However, it is almost twice as slow for long lists. Why?
-First, the ``append()`` operation has to reallocate memory, and while it uses
-some tricks to avoid doing that each time, it still has to do it occasionally,
-and that costs quite a bit. Second, the expression "result.append" requires an
-extra attribute lookup, and third, there's a speed reduction from having to make
-all those function calls.
-
-
Objects
=======
@@ -1523,18 +1513,18 @@ provide the ``self`` argument.
How can I organize my code to make it easier to change the base class?
----------------------------------------------------------------------
-You could define an alias for the base class, assign the real base class to it
-before your class definition, and use the alias throughout your class. Then all
+You could assign the base class to an alias and derive from the alias. Then all
you have to change is the value assigned to the alias. Incidentally, this trick
is also handy if you want to decide dynamically (e.g. depending on availability
of resources) which base class to use. Example::
- BaseAlias =
+ class Base:
+ ...
+
+ BaseAlias = Base
class Derived(BaseAlias):
- def meth(self):
- BaseAlias.meth(self)
- ...
+ ...
How do I create static class data and static class methods?
diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst
index a181086..c8e9c5f 100644
--- a/Doc/faq/windows.rst
+++ b/Doc/faq/windows.rst
@@ -140,7 +140,7 @@ offender.
How do I make an executable from a Python script?
-------------------------------------------------
-See `cx_Freeze `_ for a distutils extension
+See `cx_Freeze `_ for a distutils extension
that allows you to create console and GUI executables from Python code.
`py2exe `_, the most popular extension for building
Python 2.x-based executables, does not yet support Python 3 but a version that
@@ -279,7 +279,7 @@ in batch mode.
How do I check for a keypress without blocking?
-----------------------------------------------
-Use the msvcrt module. This is a standard Windows-specific extension module.
+Use the :mod:`msvcrt` module. This is a standard Windows-specific extension module.
It defines a function ``kbhit()`` which checks whether a keyboard hit is
present, and ``getch()`` which gets one character without echoing it.
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
index 9fdbdb1..4fd01e0 100644
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -301,13 +301,20 @@ Glossary
including functions, methods, properties, class methods, static methods,
and reference to super classes.
- For more information about descriptors' methods, see :ref:`descriptors`.
+ For more information about descriptors' methods, see :ref:`descriptors`
+ or the :ref:`Descriptor How To Guide `.
dictionary
An associative array, where arbitrary keys are mapped to values. The
keys can be any object with :meth:`__hash__` and :meth:`__eq__` methods.
Called a hash in Perl.
+ dictionary comprehension
+ A compact way to process all or part of the elements in an iterable and
+ return a dictionary with the results. ``results = {n: n ** 2 for n in
+ range(10)}`` generates a dictionary containing key ``n`` mapped to
+ value ``n ** 2``. See :ref:`comprehensions`.
+
dictionary view
The objects returned from :meth:`dict.keys`, :meth:`dict.values`, and
:meth:`dict.items` are called dictionary views. They provide a dynamic
@@ -476,6 +483,13 @@ Glossary
See also the :term:`single dispatch` glossary entry, the
:func:`functools.singledispatch` decorator, and :pep:`443`.
+ generic type
+ A :term:`type` that can be parameterized; typically a container like
+ :class:`list`. Used for :term:`type hints ` and
+ :term:`annotations `.
+
+ See :pep:`483` for more details, and :mod:`typing` or
+ :ref:`generic alias type ` for its uses.
GIL
See :term:`global interpreter lock`.
@@ -1024,7 +1038,13 @@ Glossary
:meth:`index`, :meth:`__contains__`, and
:meth:`__reversed__`. Types that implement this expanded
interface can be registered explicitly using
- :func:`~abc.register`.
+ :func:`~abc.ABCMeta.register`.
+
+ set comprehension
+ A compact way to process all or part of the elements in an iterable and
+ return a set with the results. ``results = {c for c in 'abracadabra' if
+ c not in 'abc'}`` generates the set of strings ``{'r', 'd'}``. See
+ :ref:`comprehensions`.
single dispatch
A form of :term:`generic function` dispatch where the implementation is
diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst
index b792b6c..ab5a573 100644
--- a/Doc/howto/descriptor.rst
+++ b/Doc/howto/descriptor.rst
@@ -1,3 +1,5 @@
+.. _descriptorhowto:
+
======================
Descriptor HowTo Guide
======================
@@ -7,45 +9,514 @@ Descriptor HowTo Guide
.. Contents::
+
+:term:`Descriptors ` let objects customize attribute lookup,
+storage, and deletion.
+
+This guide has four major sections:
+
+1) The "primer" gives a basic overview, moving gently from simple examples,
+ adding one feature at a time. Start here if you're new to descriptors.
+
+2) The second section shows a complete, practical descriptor example. If you
+ already know the basics, start there.
+
+3) The third section provides a more technical tutorial that goes into the
+ detailed mechanics of how descriptors work. Most people don't need this
+ level of detail.
+
+4) The last section has pure Python equivalents for built-in descriptors that
+ are written in C. Read this if you're curious about how functions turn
+ into bound methods or about the implementation of common tools like
+ :func:`classmethod`, :func:`staticmethod`, :func:`property`, and
+ :term:`__slots__`.
+
+
+Primer
+^^^^^^
+
+In this primer, we start with the most basic possible example and then we'll
+add new capabilities one by one.
+
+
+Simple example: A descriptor that returns a constant
+----------------------------------------------------
+
+The :class:`Ten` class is a descriptor that always returns the constant ``10``
+from its :meth:`__get__` method:
+
+.. testcode::
+
+ class Ten:
+ def __get__(self, obj, objtype=None):
+ return 10
+
+To use the descriptor, it must be stored as a class variable in another class:
+
+.. testcode::
+
+ class A:
+ x = 5 # Regular class attribute
+ y = Ten() # Descriptor instance
+
+An interactive session shows the difference between normal attribute lookup
+and descriptor lookup:
+
+.. doctest::
+
+ >>> a = A() # Make an instance of class A
+ >>> a.x # Normal attribute lookup
+ 5
+ >>> a.y # Descriptor lookup
+ 10
+
+In the ``a.x`` attribute lookup, the dot operator finds the key ``x`` and the
+value ``5`` in the class dictionary. In the ``a.y`` lookup, the dot operator
+finds a descriptor instance, recognized by its ``__get__`` method, and calls
+that method which returns ``10``.
+
+Note that the value ``10`` is not stored in either the class dictionary or the
+instance dictionary. Instead, the value ``10`` is computed on demand.
+
+This example shows how a simple descriptor works, but it isn't very useful.
+For retrieving constants, normal attribute lookup would be better.
+
+In the next section, we'll create something more useful, a dynamic lookup.
+
+
+Dynamic lookups
+---------------
+
+Interesting descriptors typically run computations instead of returning
+constants:
+
+.. testcode::
+
+ import os
+
+ class DirectorySize:
+
+ def __get__(self, obj, objtype=None):
+ return len(os.listdir(obj.dirname))
+
+ class Directory:
+
+ size = DirectorySize() # Descriptor instance
+
+ def __init__(self, dirname):
+ self.dirname = dirname # Regular instance attribute
+
+An interactive session shows that the lookup is dynamic â it computes
+different, updated answers each time::
+
+ >>> s = Directory('songs')
+ >>> g = Directory('games')
+ >>> s.size # The songs directory has twenty files
+ 20
+ >>> g.size # The games directory has three files
+ 3
+ >>> open('games/newfile').close() # Add a fourth file to the directory
+ >>> g.size # File count is automatically updated
+ 4
+
+Besides showing how descriptors can run computations, this example also
+reveals the purpose of the parameters to :meth:`__get__`. The *self*
+parameter is *size*, an instance of *DirectorySize*. The *obj* parameter is
+either *g* or *s*, an instance of *Directory*. It is the *obj* parameter that
+lets the :meth:`__get__` method learn the target directory. The *objtype*
+parameter is the class *Directory*.
+
+
+Managed attributes
+------------------
+
+A popular use for descriptors is managing access to instance data. The
+descriptor is assigned to a public attribute in the class dictionary while the
+actual data is stored as a private attribute in the instance dictionary. The
+descriptor's :meth:`__get__` and :meth:`__set__` methods are triggered when
+the public attribute is accessed.
+
+In the following example, *age* is the public attribute and *_age* is the
+private attribute. When the public attribute is accessed, the descriptor logs
+the lookup or update:
+
+.. testcode::
+
+ import logging
+
+ logging.basicConfig(level=logging.INFO)
+
+ class LoggedAgeAccess:
+
+ def __get__(self, obj, objtype=None):
+ value = obj._age
+ logging.info('Accessing %r giving %r', 'age', value)
+ return value
+
+ def __set__(self, obj, value):
+ logging.info('Updating %r to %r', 'age', value)
+ obj._age = value
+
+ class Person:
+
+ age = LoggedAgeAccess() # Descriptor instance
+
+ def __init__(self, name, age):
+ self.name = name # Regular instance attribute
+ self.age = age # Calls __set__()
+
+ def birthday(self):
+ self.age += 1 # Calls both __get__() and __set__()
+
+
+An interactive session shows that all access to the managed attribute *age* is
+logged, but that the regular attribute *name* is not logged:
+
+.. testcode::
+ :hide:
+
+ import logging, sys
+ logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True)
+
+.. doctest::
+
+ >>> mary = Person('Mary M', 30) # The initial age update is logged
+ INFO:root:Updating 'age' to 30
+ >>> dave = Person('David D', 40)
+ INFO:root:Updating 'age' to 40
+
+ >>> vars(mary) # The actual data is in a private attribute
+ {'name': 'Mary M', '_age': 30}
+ >>> vars(dave)
+ {'name': 'David D', '_age': 40}
+
+ >>> mary.age # Access the data and log the lookup
+ INFO:root:Accessing 'age' giving 30
+ 30
+ >>> mary.birthday() # Updates are logged as well
+ INFO:root:Accessing 'age' giving 30
+ INFO:root:Updating 'age' to 31
+
+ >>> dave.name # Regular attribute lookup isn't logged
+ 'David D'
+ >>> dave.age # Only the managed attribute is logged
+ INFO:root:Accessing 'age' giving 40
+ 40
+
+One major issue with this example is that the private name *_age* is hardwired in
+the *LoggedAgeAccess* class. That means that each instance can only have one
+logged attribute and that its name is unchangeable. In the next example,
+we'll fix that problem.
+
+
+Customized names
+----------------
+
+When a class uses descriptors, it can inform each descriptor about which
+variable name was used.
+
+In this example, the :class:`Person` class has two descriptor instances,
+*name* and *age*. When the :class:`Person` class is defined, it makes a
+callback to :meth:`__set_name__` in *LoggedAccess* so that the field names can
+be recorded, giving each descriptor its own *public_name* and *private_name*:
+
+.. testcode::
+
+ import logging
+
+ logging.basicConfig(level=logging.INFO)
+
+ class LoggedAccess:
+
+ def __set_name__(self, owner, name):
+ self.public_name = name
+ self.private_name = '_' + name
+
+ def __get__(self, obj, objtype=None):
+ value = getattr(obj, self.private_name)
+ logging.info('Accessing %r giving %r', self.public_name, value)
+ return value
+
+ def __set__(self, obj, value):
+ logging.info('Updating %r to %r', self.public_name, value)
+ setattr(obj, self.private_name, value)
+
+ class Person:
+
+ name = LoggedAccess() # First descriptor instance
+ age = LoggedAccess() # Second descriptor instance
+
+ def __init__(self, name, age):
+ self.name = name # Calls the first descriptor
+ self.age = age # Calls the second descriptor
+
+ def birthday(self):
+ self.age += 1
+
+An interactive session shows that the :class:`Person` class has called
+:meth:`__set_name__` so that the field names would be recorded. Here
+we call :func:`vars` to look up the descriptor without triggering it:
+
+.. doctest::
+
+ >>> vars(vars(Person)['name'])
+ {'public_name': 'name', 'private_name': '_name'}
+ >>> vars(vars(Person)['age'])
+ {'public_name': 'age', 'private_name': '_age'}
+
+The new class now logs access to both *name* and *age*:
+
+.. testcode::
+ :hide:
+
+ import logging, sys
+ logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True)
+
+.. doctest::
+
+ >>> pete = Person('Peter P', 10)
+ INFO:root:Updating 'name' to 'Peter P'
+ INFO:root:Updating 'age' to 10
+ >>> kate = Person('Catherine C', 20)
+ INFO:root:Updating 'name' to 'Catherine C'
+ INFO:root:Updating 'age' to 20
+
+The two *Person* instances contain only the private names::
+
+ >>> vars(pete)
+ {'_name': 'Peter P', '_age': 10}
+ >>> vars(kate)
+ {'_name': 'Catherine C', '_age': 20}
+
+
+Closing thoughts
+----------------
+
+A :term:`descriptor` is what we call any object that defines :meth:`__get__`,
+:meth:`__set__`, or :meth:`__delete__`.
+
+Optionally, descriptors can have a :meth:`__set_name__` method. This is only
+used in cases where a descriptor needs to know either the class where it was
+created or the name of class variable it was assigned to. (This method, if
+present, is called even if the class is not a descriptor.)
+
+Descriptors get invoked by the dot "operator" during attribute lookup. If a
+descriptor is accessed indirectly with ``vars(some_class)[descriptor_name]``,
+the descriptor instance is returned without invoking it.
+
+Descriptors only work when used as class variables. When put in instances,
+they have no effect.
+
+The main motivation for descriptors is to provide a hook allowing objects
+stored in class variables to control what happens during attribute lookup.
+
+Traditionally, the calling class controls what happens during lookup.
+Descriptors invert that relationship and allow the data being looked-up to
+have a say in the matter.
+
+Descriptors are used throughout the language. It is how functions turn into
+bound methods. Common tools like :func:`classmethod`, :func:`staticmethod`,
+:func:`property`, and :func:`functools.cached_property` are all implemented as
+descriptors.
+
+
+Complete Practical Example
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this example, we create a practical and powerful tool for locating
+notoriously hard to find data corruption bugs.
+
+
+Validator class
+---------------
+
+A validator is a descriptor for managed attribute access. Prior to storing
+any data, it verifies that the new value meets various type and range
+restrictions. If those restrictions aren't met, it raises an exception to
+prevent data corruption at its source.
+
+This :class:`Validator` class is both an :term:`abstract base class` and a
+managed attribute descriptor:
+
+.. testcode::
+
+ from abc import ABC, abstractmethod
+
+ class Validator(ABC):
+
+ def __set_name__(self, owner, name):
+ self.private_name = '_' + name
+
+ def __get__(self, obj, objtype=None):
+ return getattr(obj, self.private_name)
+
+ def __set__(self, obj, value):
+ self.validate(value)
+ setattr(obj, self.private_name, value)
+
+ @abstractmethod
+ def validate(self, value):
+ pass
+
+Custom validators need to inherit from :class:`Validator` and must supply a
+:meth:`validate` method to test various restrictions as needed.
+
+
+Custom validators
+-----------------
+
+Here are three practical data validation utilities:
+
+1) :class:`OneOf` verifies that a value is one of a restricted set of options.
+
+2) :class:`Number` verifies that a value is either an :class:`int` or
+ :class:`float`. Optionally, it verifies that a value is between a given
+ minimum or maximum.
+
+3) :class:`String` verifies that a value is a :class:`str`. Optionally, it
+ validates a given minimum or maximum length. It can validate a
+ user-defined `predicate
+ `_ as well.
+
+.. testcode::
+
+ class OneOf(Validator):
+
+ def __init__(self, *options):
+ self.options = set(options)
+
+ def validate(self, value):
+ if value not in self.options:
+ raise ValueError(f'Expected {value!r} to be one of {self.options!r}')
+
+ class Number(Validator):
+
+ def __init__(self, minvalue=None, maxvalue=None):
+ self.minvalue = minvalue
+ self.maxvalue = maxvalue
+
+ def validate(self, value):
+ if not isinstance(value, (int, float)):
+ raise TypeError(f'Expected {value!r} to be an int or float')
+ if self.minvalue is not None and value < self.minvalue:
+ raise ValueError(
+ f'Expected {value!r} to be at least {self.minvalue!r}'
+ )
+ if self.maxvalue is not None and value > self.maxvalue:
+ raise ValueError(
+ f'Expected {value!r} to be no more than {self.maxvalue!r}'
+ )
+
+ class String(Validator):
+
+ def __init__(self, minsize=None, maxsize=None, predicate=None):
+ self.minsize = minsize
+ self.maxsize = maxsize
+ self.predicate = predicate
+
+ def validate(self, value):
+ if not isinstance(value, str):
+ raise TypeError(f'Expected {value!r} to be an str')
+ if self.minsize is not None and len(value) < self.minsize:
+ raise ValueError(
+ f'Expected {value!r} to be no smaller than {self.minsize!r}'
+ )
+ if self.maxsize is not None and len(value) > self.maxsize:
+ raise ValueError(
+ f'Expected {value!r} to be no bigger than {self.maxsize!r}'
+ )
+ if self.predicate is not None and not self.predicate(value):
+ raise ValueError(
+ f'Expected {self.predicate} to be true for {value!r}'
+ )
+
+
+Practical application
+---------------------
+
+Here's how the data validators can be used in a real class:
+
+.. testcode::
+
+ class Component:
+
+ name = String(minsize=3, maxsize=10, predicate=str.isupper)
+ kind = OneOf('wood', 'metal', 'plastic')
+ quantity = Number(minvalue=0)
+
+ def __init__(self, name, kind, quantity):
+ self.name = name
+ self.kind = kind
+ self.quantity = quantity
+
+The descriptors prevent invalid instances from being created:
+
+.. doctest::
+
+ >>> Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase
+ Traceback (most recent call last):
+ ...
+ ValueError: Expected to be true for 'Widget'
+
+ >>> Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled
+ Traceback (most recent call last):
+ ...
+ ValueError: Expected 'metle' to be one of {'metal', 'plastic', 'wood'}
+
+ >>> Component('WIDGET', 'metal', -5) # Blocked: -5 is negative
+ Traceback (most recent call last):
+ ...
+ ValueError: Expected -5 to be at least 0
+ >>> Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number
+ Traceback (most recent call last):
+ ...
+ TypeError: Expected 'V' to be an int or float
+
+ >>> c = Component('WIDGET', 'metal', 5) # Allowed: The inputs are valid
+
+
+Technical Tutorial
+^^^^^^^^^^^^^^^^^^
+
+What follows is a more technical tutorial for the mechanics and details of how
+descriptors work.
+
+
Abstract
--------
Defines descriptors, summarizes the protocol, and shows how descriptors are
-called. Examines a custom descriptor and several built-in Python descriptors
-including functions, properties, static methods, and class methods. Shows how
-each works by giving a pure Python equivalent and a sample application.
+called. Provides an example showing how object relational mappings work.
Learning about descriptors not only provides access to a larger toolset, it
-creates a deeper understanding of how Python works and an appreciation for the
-elegance of its design.
+creates a deeper understanding of how Python works.
-Definition and Introduction
+Definition and introduction
---------------------------
-In general, a descriptor is an object attribute with "binding behavior", one
-whose attribute access has been overridden by methods in the descriptor
-protocol. Those methods are :meth:`__get__`, :meth:`__set__`, and
-:meth:`__delete__`. If any of those methods are defined for an object, it is
-said to be a descriptor.
+In general, a descriptor is an attribute value that has one of the methods in
+the descriptor protocol. Those methods are :meth:`__get__`, :meth:`__set__`,
+and :meth:`__delete__`. If any of those methods are defined for an the
+attribute, it is said to be a :term:`descriptor`.
The default behavior for attribute access is to get, set, or delete the
attribute from an object's dictionary. For instance, ``a.x`` has a lookup chain
starting with ``a.__dict__['x']``, then ``type(a).__dict__['x']``, and
-continuing through the base classes of ``type(a)`` excluding metaclasses. If the
+continuing through the method resolution order of ``type(a)``. If the
looked-up value is an object defining one of the descriptor methods, then Python
may override the default behavior and invoke the descriptor method instead.
Where this occurs in the precedence chain depends on which descriptor methods
were defined.
Descriptors are a powerful, general purpose protocol. They are the mechanism
-behind properties, methods, static methods, class methods, and :func:`super()`.
-They are used throughout Python itself to implement the new style classes
-introduced in version 2.2. Descriptors simplify the underlying C-code and offer
-a flexible set of new tools for everyday Python programs.
+behind properties, methods, static methods, class methods, and
+:func:`super()`. They are used throughout Python itself. Descriptors
+simplify the underlying C code and offer a flexible set of new tools for
+everyday Python programs.
-Descriptor Protocol
+Descriptor protocol
-------------------
``descr.__get__(self, obj, type=None) -> value``
@@ -60,7 +531,7 @@ as an attribute.
If an object defines :meth:`__set__` or :meth:`__delete__`, it is considered
a data descriptor. Descriptors that only define :meth:`__get__` are called
-non-data descriptors (they are typically used for methods but other uses are
+non-data descriptors (they are often used for methods but other uses are
possible).
Data and non-data descriptors differ in how overrides are calculated with
@@ -75,124 +546,372 @@ called. Defining the :meth:`__set__` method with an exception raising
placeholder is enough to make it a data descriptor.
-Invoking Descriptors
---------------------
+Overview of descriptor invocation
+---------------------------------
+
+A descriptor can be called directly with ``desc.__get__(obj)`` or
+``desc.__get__(None, cls)``.
+
+But it is more common for a descriptor to be invoked automatically from
+attribute access.
+
+The expression ``obj.x`` looks up the attribute ``x`` in the chain of
+namespaces for ``obj``. If the search finds a descriptor outside of the
+instance ``__dict__``, its :meth:`__get__` method is invoked according to the
+precedence rules listed below.
+
+The details of invocation depend on whether ``obj`` is an object, class, or
+instance of super.
+
+
+Invocation from an instance
+---------------------------
+
+Instance lookup scans through a chain of namespaces giving data descriptors
+the highest priority, followed by instance variables, then non-data
+descriptors, then class variables, and lastly :meth:`__getattr__` if it is
+provided.
-A descriptor can be called directly by its method name. For example,
-``d.__get__(obj)``.
+If a descriptor is found for ``a.x``, then it is invoked with:
+``desc.__get__(a, type(a))``.
-Alternatively, it is more common for a descriptor to be invoked automatically
-upon attribute access. For example, ``obj.d`` looks up ``d`` in the dictionary
-of ``obj``. If ``d`` defines the method :meth:`__get__`, then ``d.__get__(obj)``
-is invoked according to the precedence rules listed below.
+The logic for a dotted lookup is in :meth:`object.__getattribute__`. Here is
+a pure Python equivalent:
-The details of invocation depend on whether ``obj`` is an object or a class.
+.. testcode::
+
+ def object_getattribute(obj, name):
+ "Emulate PyObject_GenericGetAttr() in Objects/object.c"
+ null = object()
+ objtype = type(obj)
+ cls_var = getattr(objtype, name, null)
+ descr_get = getattr(type(cls_var), '__get__', null)
+ if descr_get is not null:
+ if (hasattr(type(cls_var), '__set__')
+ or hasattr(type(cls_var), '__delete__')):
+ return descr_get(cls_var, obj, objtype) # data descriptor
+ if hasattr(obj, '__dict__') and name in vars(obj):
+ return vars(obj)[name] # instance variable
+ if descr_get is not null:
+ return descr_get(cls_var, obj, objtype) # non-data descriptor
+ if cls_var is not null:
+ return cls_var # class variable
+ raise AttributeError(name)
-For objects, the machinery is in :meth:`object.__getattribute__` which
-transforms ``b.x`` into ``type(b).__dict__['x'].__get__(b, type(b))``. The
-implementation works through a precedence chain that gives data descriptors
-priority over instance variables, instance variables priority over non-data
-descriptors, and assigns lowest priority to :meth:`__getattr__` if provided.
-The full C implementation can be found in :c:func:`PyObject_GenericGetAttr()` in
-:source:`Objects/object.c`.
-For classes, the machinery is in :meth:`type.__getattribute__` which transforms
-``B.x`` into ``B.__dict__['x'].__get__(None, B)``. In pure Python, it looks
-like::
+.. testcode::
+ :hide:
- def __getattribute__(self, key):
- "Emulate type_getattro() in Objects/typeobject.c"
- v = object.__getattribute__(self, key)
- if hasattr(v, '__get__'):
- return v.__get__(None, self)
- return v
+ # Test the fidelity of object_getattribute() by comparing it with the
+ # normal object.__getattribute__(). The former will be accessed by
+ # square brackets and the latter by the dot operator.
+
+ class Object:
+
+ def __getitem__(obj, name):
+ try:
+ return object_getattribute(obj, name)
+ except AttributeError:
+ if not hasattr(type(obj), '__getattr__'):
+ raise
+ return type(obj).__getattr__(obj, name) # __getattr__
+
+ class DualOperator(Object):
+
+ x = 10
+
+ def __init__(self, z):
+ self.z = z
+
+ @property
+ def p2(self):
+ return 2 * self.x
+
+ @property
+ def p3(self):
+ return 3 * self.x
+
+ def m5(self, y):
+ return 5 * y
+
+ def m7(self, y):
+ return 7 * y
+
+ def __getattr__(self, name):
+ return ('getattr_hook', self, name)
+
+ class DualOperatorWithSlots:
+
+ __getitem__ = Object.__getitem__
+
+ __slots__ = ['z']
+
+ x = 15
+
+ def __init__(self, z):
+ self.z = z
+
+ @property
+ def p2(self):
+ return 2 * self.x
+
+ def m5(self, y):
+ return 5 * y
+
+ def __getattr__(self, name):
+ return ('getattr_hook', self, name)
+
+
+.. doctest::
+ :hide:
+
+ >>> a = DualOperator(11)
+ >>> vars(a).update(p3 = '_p3', m7 = '_m7')
+ >>> a.x == a['x'] == 10
+ True
+ >>> a.z == a['z'] == 11
+ True
+ >>> a.p2 == a['p2'] == 20
+ True
+ >>> a.p3 == a['p3'] == 30
+ True
+ >>> a.m5(100) == a.m5(100) == 500
+ True
+ >>> a.m7 == a['m7'] == '_m7'
+ True
+ >>> a.g == a['g'] == ('getattr_hook', a, 'g')
+ True
+
+ >>> b = DualOperatorWithSlots(22)
+ >>> b.x == b['x'] == 15
+ True
+ >>> b.z == b['z'] == 22
+ True
+ >>> b.p2 == b['p2'] == 30
+ True
+ >>> b.m5(200) == b['m5'](200) == 1000
+ True
+ >>> b.g == b['g'] == ('getattr_hook', b, 'g')
+ True
+
+
+Interestingly, attribute lookup doesn't call :meth:`object.__getattribute__`
+directly. Instead, both the dot operator and the :func:`getattr` function
+perform attribute lookup by way of a helper function:
+
+.. testcode::
+
+ def getattr_hook(obj, name):
+ "Emulate slot_tp_getattr_hook() in Objects/typeobject.c"
+ try:
+ return obj.__getattribute__(name)
+ except AttributeError:
+ if not hasattr(type(obj), '__getattr__'):
+ raise
+ return type(obj).__getattr__(obj, name) # __getattr__
+
+So if :meth:`__getattr__` exists, it is called whenever :meth:`__getattribute__`
+raises :exc:`AttributeError` (either directly or in one of the descriptor calls).
+
+Also, if a user calls :meth:`object.__getattribute__` directly, the
+:meth:`__getattr__` hook is bypassed entirely.
+
+
+Invocation from a class
+-----------------------
+
+The logic for a dotted lookup such as ``A.x`` is in
+:meth:`type.__getattribute__`. The steps are similar to those for
+:meth:`object.__getattribute__` but the instance dictionary lookup is replaced
+by a search through the class's :term:`method resolution order`.
+
+If a descriptor is found, it is invoked with ``desc.__get__(None, A)``.
+
+The full C implementation can be found in :c:func:`type_getattro()` and
+:c:func:`_PyType_Lookup()` in :source:`Objects/typeobject.c`.
+
+
+Invocation from super
+---------------------
+
+The logic for super's dotted lookup is in the :meth:`__getattribute__` method for
+object returned by :class:`super()`.
+
+A dotted lookup such as ``super(A, obj).m`` searches ``obj.__class__.__mro__``
+for the base class ``B`` immediately following ``A`` and then returns
+``B.__dict__['m'].__get__(obj, A)``. If not a descriptor, ``m`` is returned
+unchanged.
+
+The full C implementation can be found in :c:func:`super_getattro()` in
+:source:`Objects/typeobject.c`. A pure Python equivalent can be found in
+`Guido's Tutorial
+`_.
+
+
+Summary of invocation logic
+---------------------------
+
+The mechanism for descriptors is embedded in the :meth:`__getattribute__()`
+methods for :class:`object`, :class:`type`, and :func:`super`.
The important points to remember are:
-* descriptors are invoked by the :meth:`__getattribute__` method
-* overriding :meth:`__getattribute__` prevents automatic descriptor calls
+* Descriptors are invoked by the :meth:`__getattribute__` method.
+
+* Classes inherit this machinery from :class:`object`, :class:`type`, or
+ :func:`super`.
+
+* Overriding :meth:`__getattribute__` prevents automatic descriptor calls
+ because all the descriptor logic is in that method.
+
* :meth:`object.__getattribute__` and :meth:`type.__getattribute__` make
- different calls to :meth:`__get__`.
-* data descriptors always override instance dictionaries.
-* non-data descriptors may be overridden by instance dictionaries.
+ different calls to :meth:`__get__`. The first includes the instance and may
+ include the class. The second puts in ``None`` for the instance and always
+ includes the class.
-The object returned by ``super()`` also has a custom :meth:`__getattribute__`
-method for invoking descriptors. The attribute lookup ``super(B, obj).m`` searches
-``obj.__class__.__mro__`` for the base class ``A`` immediately following ``B``
-and then returns ``A.__dict__['m'].__get__(obj, B)``. If not a descriptor,
-``m`` is returned unchanged. If not in the dictionary, ``m`` reverts to a
-search using :meth:`object.__getattribute__`.
+* Data descriptors always override instance dictionaries.
-The implementation details are in :c:func:`super_getattro()` in
-:source:`Objects/typeobject.c`. and a pure Python equivalent can be found in
-`Guido's Tutorial`_.
+* Non-data descriptors may be overridden by instance dictionaries.
-.. _`Guido's Tutorial`: https://www.python.org/download/releases/2.2.3/descrintro/#cooperation
-The details above show that the mechanism for descriptors is embedded in the
-:meth:`__getattribute__()` methods for :class:`object`, :class:`type`, and
-:func:`super`. Classes inherit this machinery when they derive from
-:class:`object` or if they have a meta-class providing similar functionality.
-Likewise, classes can turn-off descriptor invocation by overriding
-:meth:`__getattribute__()`.
+Automatic name notification
+---------------------------
+Sometimes it is desirable for a descriptor to know what class variable name it
+was assigned to. When a new class is created, the :class:`type` metaclass
+scans the dictionary of the new class. If any of the entries are descriptors
+and if they define :meth:`__set_name__`, that method is called with two
+arguments. The *owner* is the class where the descriptor is used, and the
+*name* is the class variable the descriptor was assigned to.
-Descriptor Example
-------------------
+The implementation details are in :c:func:`type_new()` and
+:c:func:`set_names()` in :source:`Objects/typeobject.c`.
-The following code creates a class whose objects are data descriptors which
-print a message for each get or set. Overriding :meth:`__getattribute__` is
-alternate approach that could do this for every attribute. However, this
-descriptor is useful for monitoring just a few chosen attributes::
+Since the update logic is in :meth:`type.__new__`, notifications only take
+place at the time of class creation. If descriptors are added to the class
+afterwards, :meth:`__set_name__` will need to be called manually.
- class RevealAccess:
- """A data descriptor that sets and returns values
- normally and prints a message logging their access.
- """
- def __init__(self, initval=None, name='var'):
- self.val = initval
- self.name = name
+ORM example
+-----------
- def __get__(self, obj, objtype):
- print('Retrieving', self.name)
- return self.val
+The following code is simplified skeleton showing how data descriptors could
+be used to implement an `object relational mapping
+`_.
- def __set__(self, obj, val):
- print('Updating', self.name)
- self.val = val
+The essential idea is that the data is stored in an external database. The
+Python instances only hold keys to the database's tables. Descriptors take
+care of lookups or updates:
- >>> class MyClass:
- ... x = RevealAccess(10, 'var "x"')
- ... y = 5
- ...
- >>> m = MyClass()
- >>> m.x
- Retrieving var "x"
- 10
- >>> m.x = 20
- Updating var "x"
- >>> m.x
- Retrieving var "x"
- 20
- >>> m.y
- 5
+.. testcode::
+
+ class Field:
+
+ def __set_name__(self, owner, name):
+ self.fetch = f'SELECT {name} FROM {owner.table} WHERE {owner.key}=?;'
+ self.store = f'UPDATE {owner.table} SET {name}=? WHERE {owner.key}=?;'
+
+ def __get__(self, obj, objtype=None):
+ return conn.execute(self.fetch, [obj.key]).fetchone()[0]
+
+ def __set__(self, obj, value):
+ conn.execute(self.store, [value, obj.key])
+ conn.commit()
+
+We can use the :class:`Field` class to define `models
+`_ that describe the schema for
+each table in a database:
+
+.. testcode::
+
+ class Movie:
+ table = 'Movies' # Table name
+ key = 'title' # Primary key
+ director = Field()
+ year = Field()
+
+ def __init__(self, key):
+ self.key = key
+
+ class Song:
+ table = 'Music'
+ key = 'title'
+ artist = Field()
+ year = Field()
+ genre = Field()
+
+ def __init__(self, key):
+ self.key = key
-The protocol is simple and offers exciting possibilities. Several use cases are
-so common that they have been packaged into individual function calls.
-Properties, bound methods, static methods, and class methods are all
-based on the descriptor protocol.
+To use the models, first connect to the database::
+
+ >>> import sqlite3
+ >>> conn = sqlite3.connect('entertainment.db')
+
+An interactive session shows how data is retrieved from the database and how
+it can be updated:
+
+.. testsetup::
+
+ song_data = [
+ ('Country Roads', 'John Denver', 1972),
+ ('Me and Bobby McGee', 'Janice Joplin', 1971),
+ ('Coal Miners Daughter', 'Loretta Lynn', 1970),
+ ]
+
+ movie_data = [
+ ('Star Wars', 'George Lucas', 1977),
+ ('Jaws', 'Steven Spielberg', 1975),
+ ('Aliens', 'James Cameron', 1986),
+ ]
+
+ import sqlite3
+
+ conn = sqlite3.connect(':memory:')
+ conn.execute('CREATE TABLE Music (title text, artist text, year integer);')
+ conn.execute('CREATE INDEX MusicNdx ON Music (title);')
+ conn.executemany('INSERT INTO Music VALUES (?, ?, ?);', song_data)
+ conn.execute('CREATE TABLE Movies (title text, director text, year integer);')
+ conn.execute('CREATE INDEX MovieNdx ON Music (title);')
+ conn.executemany('INSERT INTO Movies VALUES (?, ?, ?);', movie_data)
+ conn.commit()
+
+.. doctest::
+
+ >>> Movie('Star Wars').director
+ 'George Lucas'
+ >>> jaws = Movie('Jaws')
+ >>> f'Released in {jaws.year} by {jaws.director}'
+ 'Released in 1975 by Steven Spielberg'
+
+ >>> Song('Country Roads').artist
+ 'John Denver'
+
+ >>> Movie('Star Wars').director = 'J.J. Abrams'
+ >>> Movie('Star Wars').director
+ 'J.J. Abrams'
+
+
+Pure Python Equivalents
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The descriptor protocol is simple and offers exciting possibilities. Several
+use cases are so common that they have been prepackaged into built-in tools.
+Properties, bound methods, static methods, class methods, and \_\_slots\_\_ are
+all based on the descriptor protocol.
Properties
----------
Calling :func:`property` is a succinct way of building a data descriptor that
-triggers function calls upon access to an attribute. Its signature is::
+triggers a function call upon access to an attribute. Its signature is::
- property(fget=None, fset=None, fdel=None, doc=None) -> property attribute
+ property(fget=None, fset=None, fdel=None, doc=None) -> property
-The documentation shows a typical use to define a managed attribute ``x``::
+The documentation shows a typical use to define a managed attribute ``x``:
+
+.. testcode::
class C:
def getx(self): return self.__x
@@ -201,7 +920,9 @@ The documentation shows a typical use to define a managed attribute ``x``::
x = property(getx, setx, delx, "I'm the 'x' property.")
To see how :func:`property` is implemented in terms of the descriptor protocol,
-here is a pure Python equivalent::
+here is a pure Python equivalent:
+
+.. testcode::
class Property:
"Emulate PyProperty_Type() in Objects/descrobject.c"
@@ -240,6 +961,57 @@ here is a pure Python equivalent::
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)
+.. testcode::
+ :hide:
+
+ # Verify the Property() emulation
+
+ class CC:
+ def getx(self):
+ return self.__x
+ def setx(self, value):
+ self.__x = value
+ def delx(self):
+ del self.__x
+ x = Property(getx, setx, delx, "I'm the 'x' property.")
+
+ # Now do it again but use the decorator style
+
+ class CCC:
+ @Property
+ def x(self):
+ return self.__x
+ @x.setter
+ def x(self, value):
+ self.__x = value
+ @x.deleter
+ def x(self):
+ del self.__x
+
+
+.. doctest::
+ :hide:
+
+ >>> cc = CC()
+ >>> hasattr(cc, 'x')
+ False
+ >>> cc.x = 33
+ >>> cc.x
+ 33
+ >>> del cc.x
+ >>> hasattr(cc, 'x')
+ False
+
+ >>> ccc = CCC()
+ >>> hasattr(ccc, 'x')
+ False
+ >>> ccc.x = 333
+ >>> ccc.x == 333
+ True
+ >>> del ccc.x
+ >>> hasattr(ccc, 'x')
+ False
+
The :func:`property` builtin helps whenever a user interface has granted
attribute access and then subsequent changes require the intervention of a
method.
@@ -248,100 +1020,137 @@ For instance, a spreadsheet class may grant access to a cell value through
``Cell('b10').value``. Subsequent improvements to the program require the cell
to be recalculated on every access; however, the programmer does not want to
affect existing client code accessing the attribute directly. The solution is
-to wrap access to the value attribute in a property data descriptor::
+to wrap access to the value attribute in a property data descriptor:
+
+.. testcode::
class Cell:
- . . .
- def getvalue(self):
+ ...
+
+ @property
+ def value(self):
"Recalculate the cell before returning value"
self.recalc()
return self._value
- value = property(getvalue)
+Either the built-in :func:`property` or our :func:`Property` equivalent would
+work in this example.
-Functions and Methods
+
+Functions and methods
---------------------
Python's object oriented features are built upon a function based environment.
Using non-data descriptors, the two are merged seamlessly.
-Class dictionaries store methods as functions. In a class definition, methods
-are written using :keyword:`def` or :keyword:`lambda`, the usual tools for
-creating functions. Methods only differ from regular functions in that the
-first argument is reserved for the object instance. By Python convention, the
-instance reference is called *self* but may be called *this* or any other
-variable name.
+Functions stored in class dictionaries get turned into methods when invoked.
+Methods only differ from regular functions in that the object instance is
+prepended to the other arguments. By convention, the instance is called
+*self* but could be called *this* or any other variable name.
+
+Methods can be created manually with :class:`types.MethodType` which is
+roughly equivalent to:
-To support method calls, functions include the :meth:`__get__` method for
-binding methods during attribute access. This means that all functions are
-non-data descriptors which return bound methods when they are invoked from an
-object. In pure Python, it works like this::
+.. testcode::
+
+ class MethodType:
+ "Emulate Py_MethodType in Objects/classobject.c"
+
+ def __init__(self, func, obj):
+ self.__func__ = func
+ self.__self__ = obj
+
+ def __call__(self, *args, **kwargs):
+ func = self.__func__
+ obj = self.__self__
+ return func(obj, *args, **kwargs)
+
+To support automatic creation of methods, functions include the
+:meth:`__get__` method for binding methods during attribute access. This
+means that functions are non-data descriptors that return bound methods
+during dotted lookup from an instance. Here's how it works:
+
+.. testcode::
class Function:
- . . .
+ ...
+
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None:
return self
- return types.MethodType(self, obj)
+ return MethodType(self, obj)
-Running the interpreter shows how the function descriptor works in practice::
+Running the following class in the interpreter shows how the function
+descriptor works in practice:
- >>> class D:
- ... def f(self, x):
- ... return x
- ...
- >>> d = D()
+.. testcode::
+
+ class D:
+ def f(self, x):
+ return x
+
+The function has a :term:`qualified name` attribute to support introspection:
+
+.. doctest::
+
+ >>> D.f.__qualname__
+ 'D.f'
+
+Accessing the function through the class dictionary does not invoke
+:meth:`__get__`. Instead, it just returns the underlying function object::
- # Access through the class dictionary does not invoke __get__.
- # It just returns the underlying function object.
>>> D.__dict__['f']
- # Dotted access from a class calls __get__() which just returns
- # the underlying function unchanged.
+Dotted access from a class calls :meth:`__get__` which just returns the
+underlying function unchanged::
+
>>> D.f
- # The function has a __qualname__ attribute to support introspection
- >>> D.f.__qualname__
- 'D.f'
+The interesting behavior occurs during dotted access from an instance. The
+dotted lookup calls :meth:`__get__` which returns a bound method object::
- # Dotted access from an instance calls __get__() which returns the
- # function wrapped in a bound method object
+ >>> d = D()
>>> d.f
>
- # Internally, the bound method stores the underlying function and
- # the bound instance.
+Internally, the bound method stores the underlying function and the bound
+instance::
+
>>> d.f.__func__
-
+
+
>>> d.f.__self__
<__main__.D object at 0x1012e1f98>
+If you have ever wondered where *self* comes from in regular methods or where
+*cls* comes from in class methods, this is it!
+
-Static Methods and Class Methods
---------------------------------
+Static methods
+--------------
Non-data descriptors provide a simple mechanism for variations on the usual
patterns of binding functions into methods.
To recap, functions have a :meth:`__get__` method so that they can be converted
to a method when accessed as attributes. The non-data descriptor transforms an
-``obj.f(*args)`` call into ``f(obj, *args)``. Calling ``klass.f(*args)``
+``obj.f(*args)`` call into ``f(obj, *args)``. Calling ``cls.f(*args)``
becomes ``f(*args)``.
This chart summarizes the binding and its two most useful variants:
+-----------------+----------------------+------------------+
| Transformation | Called from an | Called from a |
- | | Object | Class |
+ | | object | class |
+=================+======================+==================+
| function | f(obj, \*args) | f(\*args) |
+-----------------+----------------------+------------------+
| staticmethod | f(\*args) | f(\*args) |
+-----------------+----------------------+------------------+
- | classmethod | f(type(obj), \*args) | f(klass, \*args) |
+ | classmethod | f(type(obj), \*args) | f(cls, \*args) |
+-----------------+----------------------+------------------+
Static methods return the underlying function without changes. Calling either
@@ -362,21 +1171,27 @@ in statistical work but does not directly depend on a particular dataset.
It can be called either from an object or the class: ``s.erf(1.5) --> .9332`` or
``Sample.erf(1.5) --> .9332``.
-Since staticmethods return the underlying function with no changes, the example
-calls are unexciting::
+Since static methods return the underlying function with no changes, the
+example calls are unexciting:
+
+.. testcode::
+
+ class E:
+ @staticmethod
+ def f(x):
+ print(x)
+
+.. doctest::
- >>> class E:
- ... def f(x):
- ... print(x)
- ... f = staticmethod(f)
- ...
>>> E.f(3)
3
>>> E().f(3)
3
Using the non-data descriptor protocol, a pure Python version of
-:func:`staticmethod` would look like this::
+:func:`staticmethod` would look like this:
+
+.. doctest::
class StaticMethod:
"Emulate PyStaticMethod_Type() in Objects/funcobject.c"
@@ -387,44 +1202,59 @@ Using the non-data descriptor protocol, a pure Python version of
def __get__(self, obj, objtype=None):
return self.f
+
+Class methods
+-------------
+
Unlike static methods, class methods prepend the class reference to the
argument list before calling the function. This format is the same
-for whether the caller is an object or a class::
+for whether the caller is an object or a class:
- >>> class E:
- ... def f(klass, x):
- ... return klass.__name__, x
- ... f = classmethod(f)
- ...
- >>> print(E.f(3))
- ('E', 3)
- >>> print(E().f(3))
- ('E', 3)
+.. testcode::
+
+ class F:
+ @classmethod
+ def f(cls, x):
+ return cls.__name__, x
+.. doctest::
-This behavior is useful whenever the function only needs to have a class
-reference and does not care about any underlying data. One use for classmethods
-is to create alternate class constructors. In Python 2.3, the classmethod
-:func:`dict.fromkeys` creates a new dictionary from a list of keys. The pure
-Python equivalent is::
+ >>> F.f(3)
+ ('F', 3)
+ >>> F().f(3)
+ ('F', 3)
- class Dict:
- . . .
- def fromkeys(klass, iterable, value=None):
+This behavior is useful whenever the method only needs to have a class
+reference and does not rely on data stored in a specific instance. One use for
+class methods is to create alternate class constructors. For example, the
+classmethod :func:`dict.fromkeys` creates a new dictionary from a list of
+keys. The pure Python equivalent is:
+
+.. testcode::
+
+ class Dict(dict):
+ @classmethod
+ def fromkeys(cls, iterable, value=None):
"Emulate dict_fromkeys() in Objects/dictobject.c"
- d = klass()
+ d = cls()
for key in iterable:
d[key] = value
return d
- fromkeys = classmethod(fromkeys)
-Now a new dictionary of unique keys can be constructed like this::
+Now a new dictionary of unique keys can be constructed like this:
+
+.. doctest::
- >>> Dict.fromkeys('abracadabra')
- {'a': None, 'r': None, 'b': None, 'c': None, 'd': None}
+ >>> d = Dict.fromkeys('abracadabra')
+ >>> type(d) is Dict
+ True
+ >>> d
+ {'a': None, 'b': None, 'r': None, 'c': None, 'd': None}
Using the non-data descriptor protocol, a pure Python version of
-:func:`classmethod` would look like this::
+:func:`classmethod` would look like this:
+
+.. testcode::
class ClassMethod:
"Emulate PyClassMethod_Type() in Objects/funcobject.c"
@@ -432,10 +1262,301 @@ Using the non-data descriptor protocol, a pure Python version of
def __init__(self, f):
self.f = f
- def __get__(self, obj, klass=None):
- if klass is None:
- klass = type(obj)
- def newfunc(*args):
- return self.f(klass, *args)
- return newfunc
+ def __get__(self, obj, cls=None):
+ if cls is None:
+ cls = type(obj)
+ if hasattr(obj, '__get__'):
+ return self.f.__get__(cls)
+ return MethodType(self.f, cls)
+
+.. testcode::
+ :hide:
+
+ # Verify the emulation works
+ class T:
+ @ClassMethod
+ def cm(cls, x, y):
+ return (cls, x, y)
+
+.. doctest::
+ :hide:
+
+ >>> T.cm(11, 22)
+ (, 11, 22)
+
+ # Also call it from an instance
+ >>> t = T()
+ >>> t.cm(11, 22)
+ (, 11, 22)
+
+The code path for ``hasattr(obj, '__get__')`` was added in Python 3.9 and
+makes it possible for :func:`classmethod` to support chained decorators.
+For example, a classmethod and property could be chained together:
+.. testcode::
+
+ class G:
+ @classmethod
+ @property
+ def __doc__(cls):
+ return f'A doc for {cls.__name__!r}'
+
+.. doctest::
+
+ >>> G.__doc__
+ "A doc for 'G'"
+
+
+Member objects and __slots__
+----------------------------
+
+When a class defines ``__slots__``, it replaces instance dictionaries with a
+fixed-length array of slot values. From a user point of view that has
+several effects:
+
+1. Provides immediate detection of bugs due to misspelled attribute
+assignments. Only attribute names specified in ``__slots__`` are allowed:
+
+.. testcode::
+
+ class Vehicle:
+ __slots__ = ('id_number', 'make', 'model')
+
+.. doctest::
+
+ >>> auto = Vehicle()
+ >>> auto.id_nubmer = 'VYE483814LQEX'
+ Traceback (most recent call last):
+ ...
+ AttributeError: 'Vehicle' object has no attribute 'id_nubmer'
+
+2. Helps create immutable objects where descriptors manage access to private
+attributes stored in ``__slots__``:
+
+.. testcode::
+
+ class Immutable:
+
+ __slots__ = ('_dept', '_name') # Replace the instance dictionary
+
+ def __init__(self, dept, name):
+ self._dept = dept # Store to private attribute
+ self._name = name # Store to private attribute
+
+ @property # Read-only descriptor
+ def dept(self):
+ return self._dept
+
+ @property
+ def name(self): # Read-only descriptor
+ return self._name
+
+.. doctest::
+
+ >>> mark = Immutable('Botany', 'Mark Watney')
+ >>> mark.dept
+ 'Botany'
+ >>> mark.dept = 'Space Pirate'
+ Traceback (most recent call last):
+ ...
+ AttributeError: can't set attribute
+ >>> mark.location = 'Mars'
+ Traceback (most recent call last):
+ ...
+ AttributeError: 'Immutable' object has no attribute 'location'
+
+3. Saves memory. On a 64-bit Linux build, an instance with two attributes
+takes 48 bytes with ``__slots__`` and 152 bytes without. This `flyweight
+design pattern `_ likely only
+matters when a large number of instances are going to be created.
+
+4. Blocks tools like :func:`functools.cached_property` which require an
+instance dictionary to function correctly:
+
+.. testcode::
+
+ from functools import cached_property
+
+ class CP:
+ __slots__ = () # Eliminates the instance dict
+
+ @cached_property # Requires an instance dict
+ def pi(self):
+ return 4 * sum((-1.0)**n / (2.0*n + 1.0)
+ for n in reversed(range(100_000)))
+
+.. doctest::
+
+ >>> CP().pi
+ Traceback (most recent call last):
+ ...
+ TypeError: No '__dict__' attribute on 'CP' instance to cache 'pi' property.
+
+It is not possible to create an exact drop-in pure Python version of
+``__slots__`` because it requires direct access to C structures and control
+over object memory allocation. However, we can build a mostly faithful
+simulation where the actual C structure for slots is emulated by a private
+``_slotvalues`` list. Reads and writes to that private structure are managed
+by member descriptors:
+
+.. testcode::
+
+ null = object()
+
+ class Member:
+
+ def __init__(self, name, clsname, offset):
+ 'Emulate PyMemberDef in Include/structmember.h'
+ # Also see descr_new() in Objects/descrobject.c
+ self.name = name
+ self.clsname = clsname
+ self.offset = offset
+
+ def __get__(self, obj, objtype=None):
+ 'Emulate member_get() in Objects/descrobject.c'
+ # Also see PyMember_GetOne() in Python/structmember.c
+ value = obj._slotvalues[self.offset]
+ if value is null:
+ raise AttributeError(self.name)
+ return value
+
+ def __set__(self, obj, value):
+ 'Emulate member_set() in Objects/descrobject.c'
+ obj._slotvalues[self.offset] = value
+
+ def __delete__(self, obj):
+ 'Emulate member_delete() in Objects/descrobject.c'
+ value = obj._slotvalues[self.offset]
+ if value is null:
+ raise AttributeError(self.name)
+ obj._slotvalues[self.offset] = null
+
+ def __repr__(self):
+ 'Emulate member_repr() in Objects/descrobject.c'
+ return f''
+
+The :meth:`type.__new__` method takes care of adding member objects to class
+variables:
+
+.. testcode::
+
+ class Type(type):
+ 'Simulate how the type metaclass adds member objects for slots'
+
+ def __new__(mcls, clsname, bases, mapping):
+ 'Emuluate type_new() in Objects/typeobject.c'
+ # type_new() calls PyTypeReady() which calls add_methods()
+ slot_names = mapping.get('slot_names', [])
+ for offset, name in enumerate(slot_names):
+ mapping[name] = Member(name, clsname, offset)
+ return type.__new__(mcls, clsname, bases, mapping)
+
+The :meth:`object.__new__` method takes care of creating instances that have
+slots instead of an instance dictionary. Here is a rough simulation in pure
+Python:
+
+.. testcode::
+
+ class Object:
+ 'Simulate how object.__new__() allocates memory for __slots__'
+
+ def __new__(cls, *args):
+ 'Emulate object_new() in Objects/typeobject.c'
+ inst = super().__new__(cls)
+ if hasattr(cls, 'slot_names'):
+ empty_slots = [null] * len(cls.slot_names)
+ object.__setattr__(inst, '_slotvalues', empty_slots)
+ return inst
+
+ def __setattr__(self, name, value):
+ 'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
+ cls = type(self)
+ if hasattr(cls, 'slot_names') and name not in cls.slot_names:
+ raise AttributeError(
+ f'{type(self).__name__!r} object has no attribute {name!r}'
+ )
+ super().__setattr__(name, value)
+
+ def __delattr__(self, name):
+ 'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
+ cls = type(self)
+ if hasattr(cls, 'slot_names') and name not in cls.slot_names:
+ raise AttributeError(
+ f'{type(self).__name__!r} object has no attribute {name!r}'
+ )
+ super().__delattr__(name)
+
+To use the simulation in a real class, just inherit from :class:`Object` and
+set the :term:`metaclass` to :class:`Type`:
+
+.. testcode::
+
+ class H(Object, metaclass=Type):
+ 'Instance variables stored in slots'
+
+ slot_names = ['x', 'y']
+
+ def __init__(self, x, y):
+ self.x = x
+ self.y = y
+
+At this point, the metaclass has loaded member objects for *x* and *y*::
+
+ >>> from pprint import pp
+ >>> pp(dict(vars(H)))
+ {'__module__': '__main__',
+ '__doc__': 'Instance variables stored in slots',
+ 'slot_names': ['x', 'y'],
+ '__init__': ,
+ 'x': ,
+ 'y': }
+
+.. doctest::
+ :hide:
+
+ # We test this separately because the preceding section is not
+ # doctestable due to the hex memory address for the __init__ function
+ >>> isinstance(vars(H)['x'], Member)
+ True
+ >>> isinstance(vars(H)['y'], Member)
+ True
+
+When instances are created, they have a ``slot_values`` list where the
+attributes are stored:
+
+.. doctest::
+
+ >>> h = H(10, 20)
+ >>> vars(h)
+ {'_slotvalues': [10, 20]}
+ >>> h.x = 55
+ >>> vars(h)
+ {'_slotvalues': [55, 20]}
+
+Misspelled or unassigned attributes will raise an exception:
+
+.. doctest::
+
+ >>> h.xz
+ Traceback (most recent call last):
+ ...
+ AttributeError: 'H' object has no attribute 'xz'
+
+.. doctest::
+ :hide:
+
+ # Examples for deleted attributes are not shown because this section
+ # is already a bit lengthy. We still test that code here.
+ >>> del h.x
+ >>> hasattr(h, 'x')
+ False
+
+ # Also test the code for uninitialized slots
+ >>> class HU(Object, metaclass=Type):
+ ... slot_names = ['x', 'y']
+ ...
+ >>> hu = HU()
+ >>> hasattr(hu, 'x')
+ False
+ >>> hasattr(hu, 'y')
+ False
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
index 7a7a4cf..02cd70f 100644
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -696,7 +696,7 @@ The add_argument() method
* const_ - A constant value required by some action_ and nargs_ selections.
* default_ - The value produced if the argument is absent from the
- command line.
+ command line and if it is absent from the namespace object.
* type_ - The type to which the command-line argument should be converted.
@@ -1006,6 +1006,14 @@ was not present at the command line::
>>> parser.parse_args([])
Namespace(foo=42)
+If the target namespace already has an attribute set, the action *default*
+will not over write it::
+
+ >>> parser = argparse.ArgumentParser()
+ >>> parser.add_argument('--foo', default=42)
+ >>> parser.parse_args([], namespace=argparse.Namespace(foo=101))
+ Namespace(foo=101)
+
If the ``default`` value is a string, the parser parses the value as if it
were a command-line argument. In particular, the parser applies any type_
conversion argument, if provided, before setting the attribute on the
@@ -1133,20 +1141,9 @@ container should match the type_ specified::
Any container can be passed as the *choices* value, so :class:`list` objects,
:class:`set` objects, and custom containers are all supported.
-This includes :class:`enum.Enum`, which could be used to restrain
-argument's choices; if we reuse previous rock/paper/scissors game example,
-this could be as follows::
-
- >>> from enum import Enum
- >>> class GameMove(Enum):
- ... ROCK = 'rock'
- ... PAPER = 'paper'
- ... SCISSORS = 'scissors'
- ...
- >>> parser = argparse.ArgumentParser(prog='game.py')
- >>> parser.add_argument('move', type=GameMove, choices=GameMove)
- >>> parser.parse_args(['rock'])
- Namespace(move=)
+
+Use of :class:`enum.Enum` is not recommended because it is difficult to
+control its appearance in usage, help, and error messages.
required
diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst
index 755c60f..95cb017 100644
--- a/Doc/library/ast.rst
+++ b/Doc/library/ast.rst
@@ -1503,6 +1503,13 @@ Async and await
fields as :class:`For` and :class:`With`, respectively. Only valid in the
body of an :class:`AsyncFunctionDef`.
+.. note::
+ When a string is parsed by :func:`ast.parse`, operator nodes (subclasses
+ of :class:`ast.operator`, :class:`ast.unaryop`, :class:`ast.cmpop`,
+ :class:`ast.boolop` and :class:`ast.expr_context`) on the returned tree
+ will be singletons. Changes to one will be reflected in all other
+ occurrences of the same value (e.g. :class:`ast.Add`).
+
:mod:`ast` Helpers
------------------
@@ -1753,6 +1760,34 @@ and classes for traversing abstract syntax trees:
Added the *indent* option.
+.. _ast-compiler-flags:
+
+Compiler Flags
+--------------
+
+The following flags may be passed to :func:`compile` in order to change
+effects on the compilation of a program:
+
+.. data:: PyCF_ALLOW_TOP_LEVEL_AWAIT
+
+ Enables support for top-level ``await``, ``async for``, ``async with``
+ and async comprehensions.
+
+ .. versionadded:: 3.8
+
+.. data:: PyCF_ONLY_AST
+
+ Generates and returns an abstract syntax tree instead of returning a
+ compiled code object.
+
+.. data:: PyCF_TYPE_COMMENTS
+
+ Enables support for :pep:`484` and :pep:`526` style type comments
+ (``# type: ``, ``# type: ignore ``).
+
+ .. versionadded:: 3.8
+
+
.. _ast-cli:
Command-Line Usage
diff --git a/Doc/library/asyncio-policy.rst b/Doc/library/asyncio-policy.rst
index d9d3232..5e69525 100644
--- a/Doc/library/asyncio-policy.rst
+++ b/Doc/library/asyncio-policy.rst
@@ -209,7 +209,7 @@ implementation used by the asyncio event loop:
It works reliably even when the asyncio event loop is run in a non-main OS thread.
There is no noticeable overhead when handling a big number of children (*O(1)* each
- time a child terminates), but stating a thread per process requires extra memory.
+ time a child terminates), but starting a thread per process requires extra memory.
This watcher is used by default.
@@ -219,7 +219,7 @@ implementation used by the asyncio event loop:
This implementation registers a :py:data:`SIGCHLD` signal handler on
instantiation. That can break third-party code that installs a custom handler for
- `SIGCHLD`. signal).
+ :py:data:`SIGCHLD` signal.
The watcher avoids disrupting other code spawning processes
by polling every process explicitly on a :py:data:`SIGCHLD` signal.
diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
index eb1312a..b033034 100644
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -110,10 +110,8 @@ Creating Subprocesses
.. note::
- The default asyncio event loop implementation on **Windows** does not
- support subprocesses. Subprocesses are available for Windows if a
- :class:`ProactorEventLoop` is used.
- See :ref:`Subprocess Support on Windows `
+ Subprocesses are available for Windows if a :class:`ProactorEventLoop` is
+ used. See :ref:`Subprocess Support on Windows `
for details.
.. seealso::
diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst
index 99f0125..c638f12 100644
--- a/Doc/library/asyncio-task.rst
+++ b/Doc/library/asyncio-task.rst
@@ -504,10 +504,10 @@ Waiting Primitives
return_when=ALL_COMPLETED)
Run :ref:`awaitable objects ` in the *aws*
- set concurrently and block until the condition specified
+ iterable concurrently and block until the condition specified
by *return_when*.
- The *aws* set must not be empty.
+ The *aws* iterable must not be empty.
Returns two sets of Tasks/Futures: ``(done, pending)``.
@@ -593,9 +593,9 @@ Waiting Primitives
.. function:: as_completed(aws, \*, loop=None, timeout=None)
Run :ref:`awaitable objects ` in the *aws*
- set concurrently. Return an iterator of coroutines.
+ iterable concurrently. Return an iterator of coroutines.
Each coroutine returned can be awaited to get the earliest next
- result from the set of the remaining awaitables.
+ result from the iterable of the remaining awaitables.
Raises :exc:`asyncio.TimeoutError` if the timeout occurs before
all Futures are done.
@@ -618,7 +618,7 @@ Running in Threads
Asynchronously run function *func* in a separate thread.
Any \*args and \*\*kwargs supplied for this function are directly passed
- to *func*. Also, the current :class:`contextvars.Context` is propogated,
+ to *func*. Also, the current :class:`contextvars.Context` is propagated,
allowing context variables from the event loop thread to be accessed in the
separate thread.
diff --git a/Doc/library/audit_events.rst b/Doc/library/audit_events.rst
index 3c68a15..367d56e 100644
--- a/Doc/library/audit_events.rst
+++ b/Doc/library/audit_events.rst
@@ -19,3 +19,29 @@ information on handling these events.
specific documentation for actual events raised.
.. audit-event-table::
+
+The following events are raised internally and do not correspond to any
+public API of CPython:
+
++--------------------------+-------------------------------------------+
+| Audit event | Arguments |
++==========================+===========================================+
+| _winapi.CreateFile | ``file_name``, ``desired_access``, |
+| | ``share_mode``, ``creation_disposition``, |
+| | ``flags_and_attributes`` |
++--------------------------+-------------------------------------------+
+| _winapi.CreateJunction | ``src_path``, ``dst_path`` |
++--------------------------+-------------------------------------------+
+| _winapi.CreateNamedPipe | ``name``, ``open_mode``, ``pipe_mode`` |
++--------------------------+-------------------------------------------+
+| _winapi.CreatePipe | |
++--------------------------+-------------------------------------------+
+| _winapi.CreateProcess | ``application_name``, ``command_line``, |
+| | ``current_directory`` |
++--------------------------+-------------------------------------------+
+| _winapi.OpenProcess | ``process_id``, ``desired_access`` |
++--------------------------+-------------------------------------------+
+| _winapi.TerminateProcess | ``handle``, ``exit_code`` |
++--------------------------+-------------------------------------------+
+| ctypes.PyObj_FromPtr | ``obj`` |
++--------------------------+-------------------------------------------+
diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst
index 85cdc16..637baf4 100644
--- a/Doc/library/bz2.rst
+++ b/Doc/library/bz2.rst
@@ -266,7 +266,6 @@ Below are some examples of typical usage of the :mod:`bz2` module.
Using :func:`compress` and :func:`decompress` to demonstrate round-trip compression:
>>> import bz2
-
>>> data = b"""\
... Donec rhoncus quis sapien sit amet molestie. Fusce scelerisque vel augue
... nec ullamcorper. Nam rutrum pretium placerat. Aliquam vel tristique lorem,
@@ -275,11 +274,9 @@ Using :func:`compress` and :func:`decompress` to demonstrate round-trip compress
... Aliquam pharetra lacus non risus vehicula rutrum. Maecenas aliquam leo
... felis. Pellentesque semper nunc sit amet nibh ullamcorper, ac elementum
... dolor luctus. Curabitur lacinia mi ornare consectetur vestibulum."""
-
>>> c = bz2.compress(data)
>>> len(data) / len(c) # Data compression ratio
1.513595166163142
-
>>> d = bz2.decompress(c)
>>> data == d # Check equality to original object after round-trip
True
@@ -287,7 +284,6 @@ Using :func:`compress` and :func:`decompress` to demonstrate round-trip compress
Using :class:`BZ2Compressor` for incremental compression:
>>> import bz2
-
>>> def gen_data(chunks=10, chunksize=1000):
... """Yield incremental blocks of chunksize bytes."""
... for _ in range(chunks):
@@ -310,7 +306,6 @@ while ordered, repetitive data usually yields a high compression ratio.
Writing and reading a bzip2-compressed file in binary mode:
>>> import bz2
-
>>> data = b"""\
... Donec rhoncus quis sapien sit amet molestie. Fusce scelerisque vel augue
... nec ullamcorper. Nam rutrum pretium placerat. Aliquam vel tristique lorem,
@@ -319,14 +314,11 @@ Writing and reading a bzip2-compressed file in binary mode:
... Aliquam pharetra lacus non risus vehicula rutrum. Maecenas aliquam leo
... felis. Pellentesque semper nunc sit amet nibh ullamcorper, ac elementum
... dolor luctus. Curabitur lacinia mi ornare consectetur vestibulum."""
-
>>> with bz2.open("myfile.bz2", "wb") as f:
... # Write compressed data to file
... unused = f.write(data)
-
>>> with bz2.open("myfile.bz2", "rb") as f:
... # Decompress data from file
... content = f.read()
-
>>> content == data # Check equality to original object after round-trip
True
diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst
index db0e25b..2345e78 100644
--- a/Doc/library/collections.abc.rst
+++ b/Doc/library/collections.abc.rst
@@ -291,7 +291,7 @@ Notes on using :class:`Set` and :class:`MutableSet` as a mixin:
:meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set.
If the :class:`Set` mixin is being used in a class with a different
constructor signature, you will need to override :meth:`_from_iterable`
- with a classmethod that can construct new instances from
+ with a classmethod or regular method that can construct new instances from
an iterable argument.
(2)
diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst
index 549ac1b..2ffdb49 100644
--- a/Doc/library/collections.rst
+++ b/Doc/library/collections.rst
@@ -849,6 +849,9 @@ they add the ability to access fields by name instead of position index.
Named tuple instances do not have per-instance dictionaries, so they are
lightweight and require no more memory than regular tuples.
+ To support pickling, the named tuple class should be assigned to a variable
+ that matches *typename*.
+
.. versionchanged:: 3.1
Added support for *rename*.
diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst
index 675a9ff..61d6c11 100644
--- a/Doc/library/concurrent.futures.rst
+++ b/Doc/library/concurrent.futures.rst
@@ -236,9 +236,9 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
An :class:`Executor` subclass that executes calls asynchronously using a pool
of at most *max_workers* processes. If *max_workers* is ``None`` or not
given, it will default to the number of processors on the machine.
- If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError`
+ If *max_workers* is less than or equal to ``0``, then a :exc:`ValueError`
will be raised.
- On Windows, *max_workers* must be equal or lower than ``61``. If it is not
+ On Windows, *max_workers* must be less than or equal to ``61``. If it is not
then :exc:`ValueError` will be raised. If *max_workers* is ``None``, then
the default chosen will be at most ``61``, even if more processors are
available.
@@ -250,7 +250,7 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
each worker process; *initargs* is a tuple of arguments passed to the
initializer. Should *initializer* raise an exception, all currently
pending jobs will raise a :exc:`~concurrent.futures.process.BrokenProcessPool`,
- as well any attempt to submit more jobs to the pool.
+ as well as any attempt to submit more jobs to the pool.
.. versionchanged:: 3.3
When one of the worker processes terminates abruptly, a
diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst
index 2d6c6d0..bf32d3e 100644
--- a/Doc/library/ctypes.rst
+++ b/Doc/library/ctypes.rst
@@ -1326,6 +1326,21 @@ way is to instantiate one of the following classes:
libraries use the standard C calling convention, and are assumed to return
:c:type:`int`.
+ On Windows creating a :class:`CDLL` instance may fail even if the DLL name
+ exists. When a dependent DLL of the loaded DLL is not found, a
+ :exc:`OSError` error is raised with the message *"[WinError 126] The
+ specified module could not be found".* This error message does not contain
+ the name of the missing DLL because the Windows API does not return this
+ information making this error hard to diagnose. To resolve this error and
+ determine which DLL is not found, you need to find the list of dependent
+ DLLs and determine which one is not found using Windows debugging and
+ tracing tools.
+
+.. seealso::
+
+ `Microsoft DUMPBIN tool `_
+ -- A tool to find DLL dependents.
+
.. class:: OleDLL(name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False, winmode=0)
@@ -1618,7 +1633,7 @@ They are instances of a private class:
``ctypes.seh_exception`` with argument ``code`` will be raised, allowing an
audit hook to replace the exception with its own.
-.. audit-event:: ctypes.call_function func_pointer,arguments ctype-foreign-functions
+.. audit-event:: ctypes.call_function func_pointer,arguments foreign-functions
Some ways to invoke foreign function calls may raise an auditing event
``ctypes.call_function`` with arguments ``function pointer`` and ``arguments``.
@@ -2545,4 +2560,3 @@ Arrays and pointers
Returns the object to which to pointer points. Assigning to this
attribute changes the pointer to point to the assigned object.
-
diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst
index d0307bd..c1e72d1 100644
--- a/Doc/library/dis.rst
+++ b/Doc/library/dis.rst
@@ -861,7 +861,7 @@ All of the following opcodes use their arguments.
.. opcode:: LIST_TO_TUPLE
- Pops a list from the stack and pushes a tuple containing the same values.
+ Pops a list from the stack and pushes a tuple containing the same values.
.. versionadded:: 3.9
@@ -889,7 +889,7 @@ All of the following opcodes use their arguments.
.. opcode:: DICT_MERGE
- Like :opcode:`DICT_UPDATE` but raises an exception for duplicate keys.
+ Like :opcode:`DICT_UPDATE` but raises an exception for duplicate keys.
.. versionadded:: 3.9
@@ -907,14 +907,14 @@ All of the following opcodes use their arguments.
.. opcode:: IS_OP (invert)
- Performs ``is`` comparison, or ``is not`` if ``invert`` is 1.
+ Performs ``is`` comparison, or ``is not`` if ``invert`` is 1.
.. versionadded:: 3.9
.. opcode:: CONTAINS_OP (invert)
- Performs ``in`` comparison, or ``not in`` if ``invert`` is 1.
+ Performs ``in`` comparison, or ``not in`` if ``invert`` is 1.
.. versionadded:: 3.9
@@ -955,8 +955,8 @@ All of the following opcodes use their arguments.
.. opcode:: JUMP_IF_NOT_EXC_MATCH (target)
- Tests whether the second value on the stack is an exception matching TOS,
- and jumps if it is not. Pops two values from the stack.
+ Tests whether the second value on the stack is an exception matching TOS,
+ and jumps if it is not. Pops two values from the stack.
.. versionadded:: 3.9
diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index 124085e..9c12b6c 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -150,8 +150,8 @@ are always available. They are listed here in alphabetical order.
* If it is an *integer*, the array will have that size and will be
initialized with null bytes.
- * If it is an object conforming to the *buffer* interface, a read-only buffer
- of the object will be used to initialize the bytes array.
+ * If it is an object conforming to the :ref:`buffer interface `,
+ a read-only buffer of the object will be used to initialize the bytes array.
* If it is an *iterable*, it must be an iterable of integers in the range
``0 <= x < 256``, which are used as the initial contents of the array.
@@ -245,26 +245,24 @@ are always available. They are listed here in alphabetical order.
interactive statement (in the latter case, expression statements that
evaluate to something other than ``None`` will be printed).
- The optional arguments *flags* and *dont_inherit* control which :ref:`future
- statements ` affect the compilation of *source*. If neither
- is present (or both are zero) the code is compiled with those future
- statements that are in effect in the code that is calling :func:`compile`. If the
- *flags* argument is given and *dont_inherit* is not (or is zero) then the
- future statements specified by the *flags* argument are used in addition to
- those that would be used anyway. If *dont_inherit* is a non-zero integer then
- the *flags* argument is it -- the future statements in effect around the call
- to compile are ignored.
-
- Future statements are specified by bits which can be bitwise ORed together to
- specify multiple statements. The bitfield required to specify a given feature
- can be found as the :attr:`~__future__._Feature.compiler_flag` attribute on
- the :class:`~__future__._Feature` instance in the :mod:`__future__` module.
-
- The optional argument *flags* also controls whether the compiled source is
- allowed to contain top-level ``await``, ``async for`` and ``async with``.
- When the bit ``ast.PyCF_ALLOW_TOP_LEVEL_AWAIT`` is set, the return code
- object has ``CO_COROUTINE`` set in ``co_code``, and can be interactively
- executed via ``await eval(code_object)``.
+ The optional arguments *flags* and *dont_inherit* control which
+ :ref:`compiler options ` should be activated
+ and which :ref:`future features ` should be allowed. If neither
+ is present (or both are zero) the code is compiled with the same flags that
+ affect the code that is calling :func:`compile`. If the *flags*
+ argument is given and *dont_inherit* is not (or is zero) then the compiler
+ options and the future statements specified by the *flags* argument are used
+ in addition to those that would be used anyway. If *dont_inherit* is a
+ non-zero integer then the *flags* argument is it -- the flags (future
+ features and compiler options) in the surrounding code are ignored.
+
+ Compiler options and future statements are specified by bits which can be
+ bitwise ORed together to specify multiple options. The bitfield required to
+ specify a given future feature can be found as the
+ :attr:`~__future__._Feature.compiler_flag` attribute on the
+ :class:`~__future__._Feature` instance in the :mod:`__future__` module.
+ :ref:`Compiler flags ` can be found in :mod:`ast`
+ module, with ``PyCF_`` prefix.
The argument *optimize* specifies the optimization level of the compiler; the
default value of ``-1`` selects the optimization level of the interpreter as
@@ -768,6 +766,8 @@ are always available. They are listed here in alphabetical order.
.. impl-detail:: This is the address of the object in memory.
+ .. audit-event:: builtins.id id id
+
.. function:: input([prompt])
diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst
index 14aa184..4869b67 100644
--- a/Doc/library/functools.rst
+++ b/Doc/library/functools.rst
@@ -73,16 +73,31 @@ The :mod:`functools` module defines the following functions:
def variance(self):
return statistics.variance(self._data)
- .. versionadded:: 3.8
+ Note, this decorator interferes with the operation of :pep:`412`
+ key-sharing dictionaries. This means that instance dictionaries
+ can take more space than usual.
- .. note::
+ Also, this decorator requires that the ``__dict__`` attribute on each instance
+ be a mutable mapping. This means it will not work with some types, such as
+ metaclasses (since the ``__dict__`` attributes on type instances are
+ read-only proxies for the class namespace), and those that specify
+ ``__slots__`` without including ``__dict__`` as one of the defined slots
+ (as such classes don't provide a ``__dict__`` attribute at all).
+
+ If a mutable mapping is not available or if space-efficient key sharing
+ is desired, an effect similar to :func:`cached_property` can be achieved
+ by a stacking :func:`property` on top of :func:`cache`::
- This decorator requires that the ``__dict__`` attribute on each instance
- be a mutable mapping. This means it will not work with some types, such as
- metaclasses (since the ``__dict__`` attributes on type instances are
- read-only proxies for the class namespace), and those that specify
- ``__slots__`` without including ``__dict__`` as one of the defined slots
- (as such classes don't provide a ``__dict__`` attribute at all).
+ class DataSet:
+ def __init__(self, sequence_of_numbers):
+ self._data = sequence_of_numbers
+
+ @property
+ @cache
+ def stdev(self):
+ return statistics.stdev(self._data)
+
+ .. versionadded:: 3.8
.. function:: cmp_to_key(func)
@@ -651,4 +666,4 @@ callable, weak referencable, and can have attributes. There are some important
differences. For instance, the :attr:`~definition.__name__` and :attr:`__doc__` attributes
are not created automatically. Also, :class:`partial` objects defined in
classes behave like static methods and do not transform into bound methods
-during instance attribute look-up.
\ No newline at end of file
+during instance attribute look-up.
diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst
index 43096b0..a59a5d3 100644
--- a/Doc/library/idle.rst
+++ b/Doc/library/idle.rst
@@ -527,30 +527,33 @@ by typing '_' after '.', either before or after the box is opened.
Calltips
^^^^^^^^
-A calltip is shown when one types :kbd:`(` after the name of an *accessible*
-function. A name expression may include dots and subscripts. A calltip
-remains until it is clicked, the cursor is moved out of the argument area,
-or :kbd:`)` is typed. When the cursor is in the argument part of a definition,
-the menu or shortcut display a calltip.
-
-A calltip consists of the function signature and the first line of the
-docstring. For builtins without an accessible signature, the calltip
-consists of all lines up the fifth line or the first blank line. These
-details may change.
-
-The set of *accessible* functions depends on what modules have been imported
-into the user process, including those imported by Idle itself,
-and what definitions have been run, all since the last restart.
+A calltip is shown automatically when one types :kbd:`(` after the name
+of an *accessible* function. A function name expression may include
+dots and subscripts. A calltip remains until it is clicked, the cursor
+is moved out of the argument area, or :kbd:`)` is typed. Whenever the
+cursor is in the argument part of a definition, select Edit and "Show
+Call Tip" on the menu or enter its shortcut to display a calltip.
+
+The calltip consists of the function's signature and docstring up to
+the latter's first blank line or the fifth non-blank line. (Some builtin
+functions lack an accessible signature.) A '/' or '*' in the signature
+indicates that the preceding or following arguments are passed by
+position or name (keyword) only. Details are subject to change.
+
+In Shell, the accessible functions depends on what modules have been
+imported into the user process, including those imported by Idle itself,
+and which definitions have been run, all since the last restart.
For example, restart the Shell and enter ``itertools.count(``. A calltip
-appears because Idle imports itertools into the user process for its own use.
-(This could change.) Enter ``turtle.write(`` and nothing appears. Idle does
-not import turtle. The menu or shortcut do nothing either. Enter
-``import turtle`` and then ``turtle.write(`` will work.
-
-In an editor, import statements have no effect until one runs the file. One
-might want to run a file after writing the import statements at the top,
-or immediately run an existing file before editing.
+appears because Idle imports itertools into the user process for its own
+use. (This could change.) Enter ``turtle.write(`` and nothing appears.
+Idle does not itself import turtle. The menu entry and shortcut also do
+nothing. Enter ``import turtle``. Thereafter, ``turtle.write(``
+will display a calltip.
+
+In an editor, import statements have no effect until one runs the file.
+One might want to run a file after writing import statements, after
+adding function definitions, or after opening an existing file.
.. _code-context:
diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst
index 02ecfd9..65681ec 100644
--- a/Doc/library/imaplib.rst
+++ b/Doc/library/imaplib.rst
@@ -174,9 +174,9 @@ example of usage.
.. seealso::
- Documents describing the protocol, and sources and binaries for servers
- implementing it, can all be found at the University of Washington's *IMAP
- Information Center* (https://www.washington.edu/imap/).
+ Documents describing the protocol, sources for servers
+ implementing it, by the University of Washington's IMAP Information Center
+ can all be found at (**Source Code**) https://github.com/uw-imap/imap (**Not Maintained**).
.. _imap4-objects:
diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
index b51db9e..9027ba5 100644
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -438,8 +438,9 @@ ABC hierarchy::
package. This attribute is not set on modules.
- :attr:`__package__`
- The parent package for the module/package. If the module is
- top-level then it has a value of the empty string. The
+ The fully-qualified name of the package under which the module was
+ loaded as a submodule (or the empty string for top-level modules).
+ For packages, it is the same as :attr:`__name__`. The
:func:`importlib.util.module_for_loader` decorator can handle the
details for :attr:`__package__`.
@@ -1344,8 +1345,8 @@ find and load modules.
(``__loader__``)
- The loader to use for loading. For namespace packages this should be
- set to ``None``.
+ The :term:`Loader ` that should be used when loading
+ the module. :term:`Finders ` should always set this.
.. attribute:: origin
@@ -1378,8 +1379,9 @@ find and load modules.
(``__package__``)
- (Read-only) Fully-qualified name of the package to which the module
- belongs as a submodule (or ``None``).
+ (Read-only) The fully-qualified name of the package under which the module
+ should be loaded as a submodule (or the empty string for top-level modules).
+ For packages, it is the same as :attr:`__name__`.
.. attribute:: has_location
diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst
index 5f5e664..d6d1f1e 100644
--- a/Doc/library/ipaddress.rst
+++ b/Doc/library/ipaddress.rst
@@ -202,6 +202,32 @@ write code that handles both IP versions correctly. Address objects are
.. _iana-ipv4-special-registry: https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
.. _iana-ipv6-special-registry: https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml
+.. method:: IPv4Address.__format__(fmt)
+
+ Returns a string representation of the IP address, controlled by
+ an explicit format string.
+ *fmt* can be one of the following: ``'s'``, the default option,
+ equivalent to :func:`str`, ``'b'`` for a zero-padded binary string,
+ ``'X'`` or ``'x'`` for an uppercase or lowercase hexadecimal
+ representation, or ``'n'``, which is equivalent to ``'b'`` for IPv4
+ addresses and ``'x'`` for IPv6. For binary and hexadecimal
+ representations, the form specifier ``'#'`` and the grouping option
+ ``'_'`` are available. ``__format__`` is used by ``format``, ``str.format``
+ and f-strings.
+
+ >>> format(ipaddress.IPv4Address('192.168.0.1'))
+ '192.168.0.1'
+ >>> '{:#b}'.format(ipaddress.IPv4Address('192.168.0.1'))
+ '0b11000000101010000000000000000001'
+ >>> f'{ipaddress.IPv6Address("2001:db8::1000"):s}'
+ '2001:db8::1000'
+ >>> format(ipaddress.IPv6Address('2001:db8::1000'), '_X')
+ '2001_0DB8_0000_0000_0000_0000_0000_1000'
+ >>> '{:#_n}'.format(ipaddress.IPv6Address('2001:db8::1000'))
+ '0x2001_0db8_0000_0000_0000_0000_0000_1000'
+
+ .. versionadded:: 3.9
+
.. class:: IPv6Address(address)
@@ -246,8 +272,8 @@ write code that handles both IP versions correctly. Address objects are
groups consisting entirely of zeroes included.
- For the following attributes, see the corresponding documentation of the
- :class:`IPv4Address` class:
+ For the following attributes and methods, see the corresponding
+ documentation of the :class:`IPv4Address` class:
.. attribute:: packed
.. attribute:: reverse_pointer
@@ -297,6 +323,12 @@ write code that handles both IP versions correctly. Address objects are
the embedded ``(server, client)`` IP address pair. For any other
address, this property will be ``None``.
+.. method:: IPv6Address.__format__(fmt)
+
+ Refer to the corresponding method documentation in
+ :class:`IPv4Address`.
+
+ .. versionadded:: 3.9
Conversion to Strings and Integers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst
index 107bc51..3de66c9 100644
--- a/Doc/library/itertools.rst
+++ b/Doc/library/itertools.rst
@@ -755,7 +755,7 @@ which incur interpreter overhead.
"Count how many times the predicate is true"
return sum(map(pred, iterable))
- def padnone(iterable):
+ def pad_none(iterable):
"""Returns the sequence elements and then returns None indefinitely.
Useful for emulating the behavior of the built-in map() function.
@@ -809,7 +809,7 @@ which incur interpreter overhead.
nexts = cycle(islice(nexts, num_active))
def partition(pred, iterable):
- 'Use a predicate to partition entries into false entries and true entries'
+ "Use a predicate to partition entries into false entries and true entries"
# partition(is_odd, range(10)) --> 0 2 4 6 8 and 1 3 5 7 9
t1, t2 = tee(iterable)
return filterfalse(pred, t1), filter(pred, t2)
@@ -881,7 +881,7 @@ which incur interpreter overhead.
def random_product(*args, repeat=1):
"Random selection from itertools.product(*args, **kwds)"
pools = [tuple(pool) for pool in args] * repeat
- return tuple(random.choice(pool) for pool in pools)
+ return tuple(map(random.choice, pools))
def random_permutation(iterable, r=None):
"Random selection from itertools.permutations(iterable, r)"
@@ -900,11 +900,11 @@ which incur interpreter overhead.
"Random selection from itertools.combinations_with_replacement(iterable, r)"
pool = tuple(iterable)
n = len(pool)
- indices = sorted(random.randrange(n) for i in range(r))
+ indices = sorted(random.choices(range(n), k=r))
return tuple(pool[i] for i in indices)
def nth_combination(iterable, r, index):
- 'Equivalent to list(combinations(iterable, r))[index]'
+ "Equivalent to list(combinations(iterable, r))[index]"
pool = tuple(iterable)
n = len(pool)
if r < 0 or r > n:
diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst
index f82a3b2..94d95d1 100644
--- a/Doc/library/mailbox.rst
+++ b/Doc/library/mailbox.rst
@@ -426,17 +426,14 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF.
.. seealso::
- `maildir man page from qmail `_
- The original specification of the format.
+ `maildir man page from Courier `_
+ A specification of the format. Describes a common extension for
+ supporting folders.
`Using maildir format `_
Notes on Maildir by its inventor. Includes an updated name-creation scheme and
details on "info" semantics.
- `maildir man page from Courier `_
- Another specification of the format. Describes a common extension for supporting
- folders.
-
.. _mailbox-mbox:
@@ -485,11 +482,8 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF.
.. seealso::
- `mbox man page from qmail `_
- A specification of the format and its variations.
-
`mbox man page from tin `_
- Another specification of the format, with details on locking.
+ A specification of the format, with details on locking.
`Configuring Netscape Mail on Unix: Why The Content-Length Format is Bad `_
An argument for using the original mbox format rather than a variation.
diff --git a/Doc/library/math.rst b/Doc/library/math.rst
index 6ec1fee..b20e557 100644
--- a/Doc/library/math.rst
+++ b/Doc/library/math.rst
@@ -130,7 +130,7 @@ Number-theoretic and representation functions
Return the greatest common divisor of the specified integer arguments.
If any of the arguments is nonzero, then the returned value is the largest
- positive integer that is a divisor af all arguments. If all arguments
+ positive integer that is a divisor of all arguments. If all arguments
are zero, then the returned value is ``0``. ``gcd()`` without arguments
returns ``0``.
diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst
index c9992ee..ab84d39 100644
--- a/Doc/library/multiprocessing.rst
+++ b/Doc/library/multiprocessing.rst
@@ -98,7 +98,7 @@ to start a process. These *start methods* are
*spawn*
The parent process starts a fresh python interpreter process. The
child process will only inherit those resources necessary to run
- the process objects :meth:`~Process.run` method. In particular,
+ the process object's :meth:`~Process.run` method. In particular,
unnecessary file descriptors and handles from the parent process
will not be inherited. Starting a process using this method is
rather slow compared to using *fork* or *forkserver*.
@@ -2569,9 +2569,9 @@ Address Formats
filesystem.
* An ``'AF_PIPE'`` address is a string of the form
- :samp:`r'\\\\.\\pipe\\{PipeName}'`. To use :func:`Client` to connect to a named
- pipe on a remote computer called *ServerName* one should use an address of the
- form :samp:`r'\\\\{ServerName}\\pipe\\{PipeName}'` instead.
+ :samp:`r'\\\\.\\pipe\\{PipeName}'`. To use :func:`Client` to connect to a named
+ pipe on a remote computer called *ServerName* one should use an address of the
+ form :samp:`r'\\\\{ServerName}\\pipe\\{PipeName}'` instead.
Note that any string beginning with two backslashes is assumed by default to be
an ``'AF_PIPE'`` address rather than an ``'AF_UNIX'`` address.
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
index 64bc735..6e287ab 100644
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -3695,8 +3695,8 @@ written in Python, such as a mail server's external command delivery program.
The positional-only arguments *path*, *args*, and *env* are similar to
:func:`execve`.
- The *path* parameter is the path to the executable file.The *path* should
- contain a directory.Use :func:`posix_spawnp` to pass an executable file
+ The *path* parameter is the path to the executable file. The *path* should
+ contain a directory. Use :func:`posix_spawnp` to pass an executable file
without directory.
The *file_actions* argument may be a sequence of tuples describing actions
diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst
index 8e8e377..b293adf 100644
--- a/Doc/library/platform.rst
+++ b/Doc/library/platform.rst
@@ -209,13 +209,6 @@ Windows Platform
which means the OS version uses debugging code, i.e. code that checks arguments,
ranges, etc.
- .. note::
-
- This function works best with Mark Hammond's
- :mod:`win32all` package installed, but also on Python 2.3 and
- later (support for this was added in Python 2.6). It obviously
- only runs on Win32 compatible platforms.
-
.. function:: win32_edition()
Returns a string representing the current Windows edition. Possible
diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst
index 2f349b3..9bf9212 100644
--- a/Doc/library/poplib.rst
+++ b/Doc/library/poplib.rst
@@ -67,7 +67,7 @@ The :mod:`poplib` module provides two classes:
.. audit-event:: poplib.connect self,host,port poplib.POP3_SSL
- .. audit-event:: poplib.putline self,line popplib.POP3_SSL
+ .. audit-event:: poplib.putline self,line poplib.POP3_SSL
All commands will raise an :ref:`auditing event `
``poplib.putline`` with arguments ``self`` and ``line``,
diff --git a/Doc/library/random.rst b/Doc/library/random.rst
index 90366f4..8154dfc 100644
--- a/Doc/library/random.rst
+++ b/Doc/library/random.rst
@@ -253,6 +253,8 @@ Functions for sequences
order so that the sample is reproducible.
+.. _real-valued-distributions:
+
Real-valued distributions
-------------------------
@@ -317,6 +319,13 @@ be found in any statistics text.
deviation. This is slightly faster than the :func:`normalvariate` function
defined below.
+ Multithreading note: When two threads call this function
+ simultaneously, it is possible that they will receive the
+ same return value. This can be avoided in three ways.
+ 1) Have each thread use a different instance of the random
+ number generator. 2) Put locks around all calls. 3) Use the
+ slower, but thread-safe :func:`normalvariate` function instead.
+
.. function:: lognormvariate(mu, sigma)
@@ -391,8 +400,8 @@ change across Python versions, but two aspects are guaranteed not to change:
.. _random-examples:
-Examples and Recipes
---------------------
+Examples
+--------
Basic examples::
@@ -536,3 +545,58 @@ Simulation of arrival times and service deliveries for a multiserver queue::
a tutorial by `Peter Norvig `_ covering
the basics of probability theory, how to write simulations, and
how to perform data analysis using Python.
+
+
+Recipes
+-------
+
+The default :func:`.random` returns multiples of 2â»âµÂ³ in the range
+*0.0 ⤠x < 1.0*. All such numbers are evenly spaced and are exactly
+representable as Python floats. However, many other representable
+floats in that interval are not possible selections. For example,
+``0.05954861408025609`` isn't an integer multiple of 2â»âµÂ³.
+
+The following recipe takes a different approach. All floats in the
+interval are possible selections. The mantissa comes from a uniform
+distribution of integers in the range *2âµÂ² ⤠mantissa < 2âµÂ³*. The
+exponent comes from a geometric distribution where exponents smaller
+than *-53* occur half as often as the next larger exponent.
+
+::
+
+ from random import Random
+ from math import ldexp
+
+ class FullRandom(Random):
+
+ def random(self):
+ mantissa = 0x10_0000_0000_0000 | self.getrandbits(52)
+ exponent = -53
+ x = 0
+ while not x:
+ x = self.getrandbits(32)
+ exponent += x.bit_length() - 32
+ return ldexp(mantissa, exponent)
+
+All :ref:`real valued distributions `
+in the class will use the new method::
+
+ >>> fr = FullRandom()
+ >>> fr.random()
+ 0.05954861408025609
+ >>> fr.expovariate(0.25)
+ 8.87925541791544
+
+The recipe is conceptually equivalent to an algorithm that chooses from
+all the multiples of 2â»Â¹â°â·â´ in the range *0.0 ⤠x < 1.0*. All such
+numbers are evenly spaced, but most have to be rounded down to the
+nearest representable Python float. (The value 2â»Â¹â°â·â´ is the smallest
+positive unnormalized float and is equal to ``math.ulp(0.0)``.)
+
+
+.. seealso::
+
+ `Generating Pseudo-random Floating-Point Values
+ `_ a
+ paper by Allen B. Downey describing ways to generate more
+ fine-grained floats than normally generated by :func:`.random`.
diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
index 1b094ae..3f51227 100644
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -158,9 +158,9 @@ Directory and files operations
.. function:: copy(src, dst, *, follow_symlinks=True)
Copies the file *src* to the file or directory *dst*. *src* and *dst*
- should be strings. If *dst* specifies a directory, the file will be
- copied into *dst* using the base filename from *src*. Returns the
- path to the newly created file.
+ should be :term:`path-like objects ` or strings. If
+ *dst* specifies a directory, the file will be copied into *dst* using the
+ base filename from *src*. Returns the path to the newly created file.
If *follow_symlinks* is false, and *src* is a symbolic link,
*dst* will be created as a symbolic link. If *follow_symlinks*
@@ -349,7 +349,7 @@ Directory and files operations
will be created in or as *dst* and *src* will be removed.
If *copy_function* is given, it must be a callable that takes two arguments
- *src* and *dst*, and will be used to copy *src* to *dest* if
+ *src* and *dst*, and will be used to copy *src* to *dst* if
:func:`os.rename` cannot be used. If the source is a directory,
:func:`copytree` is called, passing it the :func:`copy_function`. The
default *copy_function* is :func:`copy2`. Using :func:`~shutil.copy` as the
diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst
index 05b285e..e1daeff 100644
--- a/Doc/library/signal.rst
+++ b/Doc/library/signal.rst
@@ -117,7 +117,7 @@ The variables defined in the :mod:`signal` module are:
Child process stopped or terminated.
- .. availability:: Windows.
+ .. availability:: Unix.
.. data:: SIGCLD
diff --git a/Doc/library/site.rst b/Doc/library/site.rst
index b424e1b..2e3646f 100644
--- a/Doc/library/site.rst
+++ b/Doc/library/site.rst
@@ -231,7 +231,9 @@ Module contents
Return the path of the user-specific site-packages directory,
:data:`USER_SITE`. If it is not initialized yet, this function will also set
- it, respecting :envvar:`PYTHONNOUSERSITE` and :data:`USER_BASE`.
+ it, respecting :data:`USER_BASE`. To determine if the user-specific
+ site-packages was added to ``sys.path`` :data:`ENABLE_USER_SITE` should be
+ used.
.. versionadded:: 3.2
diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst
index a88e358..c1a20fe 100644
--- a/Doc/library/smtplib.rst
+++ b/Doc/library/smtplib.rst
@@ -115,7 +115,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions).
If the *timeout* parameter is set to be zero, it will raise a
:class:`ValueError` to prevent the creation of a non-blocking socket
-.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None,
+.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, \
source_address=None[, timeout])
The LMTP protocol, which is very similar to ESMTP, is heavily based on the
diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index 5bcac20..faf8a76 100755
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -1091,6 +1091,19 @@ The :mod:`socket` module also offers various network-related services:
.. versionchanged:: 3.8
Windows support was added.
+ .. note::
+
+ On Windows network interfaces have different names in different contexts
+ (all names are examples):
+
+ * UUID: ``{FB605B73-AAC2-49A6-9A2F-25416AEA0573}``
+ * name: ``ethernet_32770``
+ * friendly name: ``vEthernet (nat)``
+ * description: ``Hyper-V Virtual Ethernet Adapter``
+
+ This function returns names of the second form from the list, ``ethernet_32770``
+ in this example case.
+
.. function:: if_nametoindex(if_name)
@@ -1105,6 +1118,9 @@ The :mod:`socket` module also offers various network-related services:
.. versionchanged:: 3.8
Windows support was added.
+ .. seealso::
+ "Interface name" is a name as documented in :func:`if_nameindex`.
+
.. function:: if_indextoname(if_index)
@@ -1119,6 +1135,9 @@ The :mod:`socket` module also offers various network-related services:
.. versionchanged:: 3.8
Windows support was added.
+ .. seealso::
+ "Interface name" is a name as documented in :func:`if_nameindex`.
+
.. _socket-objects:
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
index c3f5c04..a48cfa1 100644
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -4119,6 +4119,12 @@ The constructors for both classes work the same:
objects. If *iterable* is not specified, a new empty set is
returned.
+ Sets can be created by several means:
+
+ * Use a comma-separated list of elements within braces: ``{'jack', 'sjoerd'}``
+ * Use a set comprehension: ``{c for c in 'abracadabra' if c not in 'abc'}``
+ * Use the type constructor: ``set()``, ``set('foobar')``, ``set(['a', 'b', 'foo'])``
+
Instances of :class:`set` and :class:`frozenset` provide the following
operations:
@@ -4311,6 +4317,14 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098:
Return a new dictionary initialized from an optional positional argument
and a possibly empty set of keyword arguments.
+ Dictionaries can be created by several means:
+
+ * Use a comma-separated list of ``key: value`` pairs within braces:
+ ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: 'jack', 4127: 'sjoerd'}``
+ * Use a dict comprehension: ``{}``, ``{x: x ** 2 for x in range(10)}``
+ * Use the type constructor: ``dict()``,
+ ``dict([('foo', 100), ('bar', 200)])``, ``dict(foo=100, bar=200)``
+
If no positional argument is given, an empty dictionary is created.
If a positional argument is given and it is a mapping object, a dictionary
is created with the same key-value pairs as the mapping object. Otherwise,
@@ -4717,6 +4731,200 @@ Compared to the overhead of setting up the runtime context, the overhead of a
single class dictionary lookup is negligible.
+.. _types-genericalias:
+
+Generic Alias Type
+==================
+
+.. index::
+ object: GenericAlias
+ pair: Generic; Alias
+
+``GenericAlias`` objects are created by subscripting a class (usually a
+container), such as ``list[int]``. They are intended primarily for
+:term:`type annotations `.
+
+Usually, the :ref:`subscription ` of container objects calls the
+method :meth:`__getitem__` of the object. However, the subscription of some
+containers' classes may call the classmethod :meth:`__class_getitem__` of the
+class instead. The classmethod :meth:`__class_getitem__` should return a
+``GenericAlias`` object.
+
+.. note::
+ If the :meth:`__getitem__` of the class' metaclass is present, it will take
+ precedence over the :meth:`__class_getitem__` defined in the class (see
+ :pep:`560` for more details).
+
+The ``GenericAlias`` object acts as a proxy for :term:`generic types
+`, implementing *parameterized generics* - a specific instance
+of a generic which provides the types for container elements.
+
+The user-exposed type for the ``GenericAlias`` object can be accessed from
+:class:`types.GenericAlias` and used for :func:`isinstance` checks. It can
+also be used to create ``GenericAlias`` objects directly.
+
+.. describe:: T[X, Y, ...]
+
+ Creates a ``GenericAlias`` representing a type ``T`` containing elements
+ of types *X*, *Y*, and more depending on the ``T`` used.
+ For example, a function expecting a :class:`list` containing
+ :class:`float` elements::
+
+ def average(values: list[float]) -> float:
+ return sum(values) / len(values)
+
+ Another example for :term:`mapping` objects, using a :class:`dict`, which
+ is a generic type expecting two type parameters representing the key type
+ and the value type. In this example, the function expects a ``dict`` with
+ keys of type :class:`str` and values of type :class:`int`::
+
+ def send_post_request(url: str, body: dict[str, int]) -> None:
+ ...
+
+The builtin functions :func:`isinstance` and :func:`issubclass` do not accept
+``GenericAlias`` types for their second argument::
+
+ >>> isinstance([1, 2], list[str])
+ Traceback (most recent call last):
+ File "", line 1, in
+ TypeError: isinstance() argument 2 cannot be a parameterized generic
+
+The Python runtime does not enforce :term:`type annotations `.
+This extends to generic types and their type parameters. When creating
+an object from a ``GenericAlias``, container elements are not checked
+against their type. For example, the following code is discouraged, but will
+run without errors::
+
+ >>> t = list[str]
+ >>> t([1, 2, 3])
+ [1, 2, 3]
+
+Furthermore, parameterized generics erase type parameters during object
+creation::
+
+ >>> t = list[str]
+ >>> type(t)
+
+
+ >>> l = t()
+ >>> type(l)
+
+
+Calling :func:`repr` or :func:`str` on a generic shows the parameterized type::
+
+ >>> repr(list[int])
+ 'list[int]'
+
+ >>> str(list[int])
+ 'list[int]'
+
+The :meth:`__getitem__` method of generics will raise an exception to disallow
+mistakes like ``dict[str][str]``::
+
+ >>> dict[str][str]
+ Traceback (most recent call last):
+ File "", line 1, in
+ TypeError: There are no type variables left in dict[str]
+
+However, such expressions are valid when :ref:`type variables ` are
+used. The index must have as many elements as there are type variable items
+in the ``GenericAlias`` object's :attr:`__args__ `. ::
+
+ >>> from typing import TypeVar
+ >>> Y = TypeVar('Y')
+ >>> dict[str, Y][int]
+ dict[str, int]
+
+
+Standard Generic Collections
+----------------------------
+
+These standard library collections support parameterized generics.
+
+* :class:`tuple`
+* :class:`list`
+* :class:`dict`
+* :class:`set`
+* :class:`frozenset`
+* :class:`type`
+* :class:`collections.deque`
+* :class:`collections.defaultdict`
+* :class:`collections.OrderedDict`
+* :class:`collections.Counter`
+* :class:`collections.ChainMap`
+* :class:`collections.abc.Awaitable`
+* :class:`collections.abc.Coroutine`
+* :class:`collections.abc.AsyncIterable`
+* :class:`collections.abc.AsyncIterator`
+* :class:`collections.abc.AsyncGenerator`
+* :class:`collections.abc.Iterable`
+* :class:`collections.abc.Iterator`
+* :class:`collections.abc.Generator`
+* :class:`collections.abc.Reversible`
+* :class:`collections.abc.Container`
+* :class:`collections.abc.Collection`
+* :class:`collections.abc.Callable`
+* :class:`collections.abc.Set`
+* :class:`collections.abc.MutableSet`
+* :class:`collections.abc.Mapping`
+* :class:`collections.abc.MutableMapping`
+* :class:`collections.abc.Sequence`
+* :class:`collections.abc.MutableSequence`
+* :class:`collections.abc.ByteString`
+* :class:`collections.abc.MappingView`
+* :class:`collections.abc.KeysView`
+* :class:`collections.abc.ItemsView`
+* :class:`collections.abc.ValuesView`
+* :class:`contextlib.AbstractContextManager`
+* :class:`contextlib.AbstractAsyncContextManager`
+* :ref:`re.Pattern `
+* :ref:`re.Match `
+
+
+Special Attributes of Generic Alias
+-----------------------------------
+
+All parameterized generics implement special read-only attributes.
+
+.. attribute:: genericalias.__origin__
+
+ This attribute points at the non-parameterized generic class::
+
+ >>> list[int].__origin__
+
+
+
+.. attribute:: genericalias.__args__
+
+ This attribute is a :class:`tuple` (possibly of length 1) of generic
+ types passed to the original :meth:`__class_getitem__`
+ of the generic container::
+
+ >>> dict[str, list[int]].__args__
+ (, list[int])
+
+
+.. attribute:: genericalias.__parameters__
+
+ This attribute is a lazily computed tuple (possibly empty) of unique type
+ variables found in ``__args__``::
+
+ >>> from typing import TypeVar
+
+ >>> T = TypeVar('T')
+ >>> list[T].__parameters__
+ (~T,)
+
+
+.. seealso::
+
+ * :pep:`585` -- "Type Hinting Generics In Standard Collections"
+ * :meth:`__class_getitem__` -- Used to implement parameterized generics.
+ * :ref:`generics` -- Generics in the :mod:`typing` module.
+
+.. versionadded:: 3.9
+
+
.. _typesother:
Other Built-in Types
diff --git a/Doc/library/string.rst b/Doc/library/string.rst
index 91f43e9..5542e9b 100644
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -384,10 +384,10 @@ following:
The ``'#'`` option causes the "alternate form" to be used for the
conversion. The alternate form is defined differently for different
-types. This option is only valid for integer, float, complex and
-Decimal types. For integers, when binary, octal, or hexadecimal output
+types. This option is only valid for integer, float and complex
+types. For integers, when binary, octal, or hexadecimal output
is used, this option adds the prefix respective ``'0b'``, ``'0o'``, or
-``'0x'`` to the output value. For floats, complex and Decimal the
+``'0x'`` to the output value. For float and complex the
alternate form causes the result of the conversion to always contain a
decimal-point character, even if no digits follow it. Normally, a
decimal-point character appears in the result of these conversions
@@ -476,20 +476,36 @@ with the floating point presentation types listed below (except
``'n'`` and ``None``). When doing so, :func:`float` is used to convert the
integer to a floating point number before formatting.
-The available presentation types for floating point and decimal values are:
+The available presentation types for :class:`float` and
+:class:`~decimal.Decimal` values are:
+---------+----------------------------------------------------------+
| Type | Meaning |
+=========+==========================================================+
- | ``'e'`` | Exponent notation. Prints the number in scientific |
- | | notation using the letter 'e' to indicate the exponent. |
- | | The default precision is ``6``. |
+ | ``'e'`` | Scientific notation. For a given precision ``p``, |
+ | | formats the number in scientific notation with the |
+ | | letter 'e' separating the coefficient from the exponent. |
+ | | The coefficient has one digit before and ``p`` digits |
+ | | after the decimal point, for a total of ``p + 1`` |
+ | | significant digits. With no precision given, uses a |
+ | | precision of ``6`` digits after the decimal point for |
+ | | :class:`float`, and shows all coefficient digits |
+ | | for :class:`~decimal.Decimal`. If no digits follow the |
+ | | decimal point, the decimal point is also removed unless |
+ | | the ``#`` option is used. |
+---------+----------------------------------------------------------+
- | ``'E'`` | Exponent notation. Same as ``'e'`` except it uses an |
- | | upper case 'E' as the separator character. |
+ | ``'E'`` | Scientific notation. Same as ``'e'`` except it uses |
+ | | an upper case 'E' as the separator character. |
+---------+----------------------------------------------------------+
- | ``'f'`` | Fixed-point notation. Displays the number as a |
- | | fixed-point number. The default precision is ``6``. |
+ | ``'f'`` | Fixed-point notation. For a given precision ``p``, |
+ | | formats the number as a decimal number with exactly |
+ | | ``p`` digits following the decimal point. With no |
+ | | precision given, uses a precision of ``6`` digits after |
+ | | the decimal point for :class:`float`, and uses a |
+ | | precision large enough to show all coefficient digits |
+ | | for :class:`~decimal.Decimal`. If no digits follow the |
+ | | decimal point, the decimal point is also removed unless |
+ | | the ``#`` option is used. |
+---------+----------------------------------------------------------+
| ``'F'`` | Fixed-point notation. Same as ``'f'``, but converts |
| | ``nan`` to ``NAN`` and ``inf`` to ``INF``. |
@@ -518,7 +534,10 @@ The available presentation types for floating point and decimal values are:
| | the precision. |
| | |
| | A precision of ``0`` is treated as equivalent to a |
- | | precision of ``1``. The default precision is ``6``. |
+ | | precision of ``1``. With no precision given, uses a |
+ | | precision of ``6`` significant digits for |
+ | | :class:`float`, and shows all coefficient digits |
+ | | for :class:`~decimal.Decimal`. |
+---------+----------------------------------------------------------+
| ``'G'`` | General format. Same as ``'g'`` except switches to |
| | ``'E'`` if the number gets too large. The |
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index 880f252..3df529f 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -31,16 +31,22 @@ always available.
When an auditing event is raised through the :func:`sys.audit` function, each
hook will be called in the order it was added with the event name and the
tuple of arguments. Native hooks added by :c:func:`PySys_AddAuditHook` are
- called first, followed by hooks added in the current interpreter.
+ called first, followed by hooks added in the current interpreter. Hooks
+ can then log the event, raise an exception to abort the operation,
+ or terminate the process entirely.
.. audit-event:: sys.addaudithook "" sys.addaudithook
- Raise an auditing event ``sys.addaudithook`` with no arguments. If any
+ Calling :func:`sys.addaudithook` will itself raise an auditing event
+ named ``sys.addaudithook`` with no arguments. If any
existing hooks raise an exception derived from :class:`RuntimeError`, the
new hook will not be added and the exception suppressed. As a result,
callers cannot assume that their hook has been added unless they control
all existing hooks.
+ See the :ref:`audit events table ` for all events raised by
+ CPython, and :pep:`578` for the original design discussion.
+
.. versionadded:: 3.8
.. versionchanged:: 3.8.1
@@ -79,14 +85,23 @@ always available.
.. index:: single: auditing
- Raise an auditing event with any active hooks. The event name is a string
- identifying the event and its associated schema, which is the number and
- types of arguments. The schema for a given event is considered public and
- stable API and should not be modified between releases.
-
- This function will raise the first exception raised by any hook. In general,
- these errors should not be handled and should terminate the process as
- quickly as possible.
+ Raise an auditing event and trigger any active auditing hooks.
+ *event* is a string identifying the event, and *args* may contain
+ optional arguments with more information about the event. The
+ number and types of arguments for a given event are considered a
+ public and stable API and should not be modified between releases.
+
+ For example, one auditing event is named ``os.chdir``. This event has
+ one argument called *path* that will contain the requested new
+ working directory.
+
+ :func:`sys.audit` will call the existing auditing hooks, passing
+ the event name and arguments, and will re-raise the first exception
+ from any hook. In general, if an exception is raised, it should not
+ be handled and the process should be terminated as quickly as
+ possible. This allows hook implementations to decide how to respond
+ to particular events: they can merely log the event or abort the
+ operation by raising an exception.
Hooks are added using the :func:`sys.addaudithook` or
:c:func:`PySys_AddAuditHook` functions.
diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst
index cca466b..7a114fd 100644
--- a/Doc/library/tarfile.rst
+++ b/Doc/library/tarfile.rst
@@ -445,10 +445,11 @@ be finalized; only the internally used file object will be closed. See the
.. method:: TarFile.extractfile(member)
- Extract a member from the archive as a file object. *member* may be a filename
- or a :class:`TarInfo` object. If *member* is a regular file or a link, an
- :class:`io.BufferedReader` object is returned. Otherwise, :const:`None` is
- returned.
+ Extract a member from the archive as a file object. *member* may be
+ a filename or a :class:`TarInfo` object. If *member* is a regular file or
+ a link, an :class:`io.BufferedReader` object is returned. For all other
+ existing members, :const:`None` is returned. If *member* does not appear
+ in the archive, :exc:`KeyError` is raised.
.. versionchanged:: 3.3
Return an :class:`io.BufferedReader` object.
diff --git a/Doc/library/test.rst b/Doc/library/test.rst
index f7e6eba..e24f69c 100644
--- a/Doc/library/test.rst
+++ b/Doc/library/test.rst
@@ -1609,6 +1609,8 @@ script execution tests.
The :mod:`test.support.bytecode_helper` module provides support for testing
and inspecting bytecode generation.
+.. versionadded:: 3.9
+
The module defines the following class:
.. class:: BytecodeTestCase(unittest.TestCase)
diff --git a/Doc/library/types.rst b/Doc/library/types.rst
index 79acdf4..0fe3822 100644
--- a/Doc/library/types.rst
+++ b/Doc/library/types.rst
@@ -109,6 +109,11 @@ Standard names are defined for the following types:
The type of user-defined functions and functions created by
:keyword:`lambda` expressions.
+ .. audit-event:: function.__new__ code types.FunctionType
+
+ The audit event only occurs for direct instantiation of function objects,
+ and is not raised for normal compilation.
+
.. data:: GeneratorType
@@ -138,10 +143,11 @@ Standard names are defined for the following types:
The type for code objects such as returned by :func:`compile`.
- .. audit-event:: code.__new__ code,filename,name,argcount,posonlyargcount,kwonlyargcount,nlocals,stacksize,flags CodeType
+ .. audit-event:: code.__new__ code,filename,name,argcount,posonlyargcount,kwonlyargcount,nlocals,stacksize,flags types.CodeType
Note that the audited arguments may not match the names or positions
- required by the initializer.
+ required by the initializer. The audit event only occurs for direct
+ instantiation of code objects, and is not raised for normal compilation.
.. method:: CodeType.replace(**kwargs)
@@ -236,6 +242,25 @@ Standard names are defined for the following types:
Defaults to ``None``. Previously the attribute was optional.
+.. class:: GenericAlias(t_origin, t_args)
+
+ The type of :ref:`parameterized generics ` such as
+ ``list[int]``.
+
+ ``t_origin`` should be a non-parameterized generic class, such as ``list``,
+ ``tuple`` or ``dict``. ``t_args`` should be a :class:`tuple` (possibly of
+ length 1) of types which parameterize ``t_origin``::
+
+ >>> from types import GenericAlias
+
+ >>> list[int] == GenericAlias(list, (int,))
+ True
+ >>> dict[str, int] == GenericAlias(dict, (str, int))
+ True
+
+ .. versionadded:: 3.9
+
+
.. class:: TracebackType(tb_next, tb_frame, tb_lasti, tb_lineno)
The type of traceback objects such as found in ``sys.exc_info()[2]``.
@@ -359,7 +384,9 @@ Additional Utility Classes and Functions
return "{}({})".format(type(self).__name__, ", ".join(items))
def __eq__(self, other):
- return self.__dict__ == other.__dict__
+ if isinstance(self, SimpleNamespace) and isinstance(other, SimpleNamespace):
+ return self.__dict__ == other.__dict__
+ return NotImplemented
``SimpleNamespace`` may be useful as a replacement for ``class NS: pass``.
However, for a structured record type use :func:`~collections.namedtuple`
diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index 3900e49..af2cafb 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -509,7 +509,8 @@ These can be used as types in annotations using ``[]``, each having a unique syn
is equivalent to ``Tuple[Any, ...]``, and in turn to :class:`tuple`.
.. deprecated:: 3.9
- :class:`builtins.tuple ` now supports ``[]``. See :pep:`585`.
+ :class:`builtins.tuple ` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. data:: Union
@@ -583,7 +584,8 @@ These can be used as types in annotations using ``[]``, each having a unique syn
:class:`collections.abc.Callable`.
.. deprecated:: 3.9
- :class:`collections.abc.Callable` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Callable` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: Type(Generic[CT_co])
@@ -620,7 +622,7 @@ These can be used as types in annotations using ``[]``, each having a unique syn
:ref:`type variables `, and unions of any of these types.
For example::
- def new_non_team_user(user_class: Type[Union[BaseUser, ProUser]]): ...
+ def new_non_team_user(user_class: Type[Union[BasicUser, ProUser]]): ...
``Type[Any]`` is equivalent to ``Type`` which in turn is equivalent
to ``type``, which is the root of Python's metaclass hierarchy.
@@ -628,7 +630,8 @@ These can be used as types in annotations using ``[]``, each having a unique syn
.. versionadded:: 3.5.2
.. deprecated:: 3.9
- :class:`builtins.type ` now supports ``[]``. See :pep:`585`.
+ :class:`builtins.type ` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. data:: Literal
@@ -652,6 +655,12 @@ These can be used as types in annotations using ``[]``, each having a unique syn
.. versionadded:: 3.8
+ .. versionchanged:: 3.9.1
+ ``Literal`` now de-duplicates parameters. Equality comparison of
+ ``Literal`` objects are no longer order dependent. ``Literal`` objects
+ will now raise a :exc:`TypeError` exception during equality comparisons
+ if one of their parameters are not :term:`immutable`.
+
.. data:: ClassVar
Special type construct to mark class variables.
@@ -821,7 +830,7 @@ These are not used in annotations. They are building blocks for creating generic
Type variables exist primarily for the benefit of static type
checkers. They serve as the parameters for generic types as well
- as for generic function definitions. See class Generic for more
+ as for generic function definitions. See :class:`Generic` for more
information on generic types. Generic functions work as follows::
def repeat(x: T, n: int) -> Sequence[T]:
@@ -1060,7 +1069,8 @@ Corresponding to built-in types
...
.. deprecated:: 3.9
- :class:`builtins.dict ` now supports ``[]``. See :pep:`585`.
+ :class:`builtins.dict ` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: List(list, MutableSequence[T])
@@ -1080,7 +1090,8 @@ Corresponding to built-in types
return [item for item in vector if item > 0]
.. deprecated:: 3.9
- :class:`builtins.list ` now supports ``[]``. See :pep:`585`.
+ :class:`builtins.list ` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: Set(set, MutableSet[T])
@@ -1089,14 +1100,16 @@ Corresponding to built-in types
to use an abstract collection type such as :class:`AbstractSet`.
.. deprecated:: 3.9
- :class:`builtins.set ` now supports ``[]``. See :pep:`585`.
+ :class:`builtins.set ` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: FrozenSet(frozenset, AbstractSet[T_co])
A generic version of :class:`builtins.frozenset `.
.. deprecated:: 3.9
- :class:`builtins.frozenset ` now supports ``[]``. See :pep:`585`.
+ :class:`builtins.frozenset ` now supports ``[]``. See
+ :pep:`585` and :ref:`types-genericalias`.
.. note:: :data:`Tuple` is a special form.
@@ -1110,7 +1123,8 @@ Corresponding to types in :mod:`collections`
.. versionadded:: 3.5.2
.. deprecated:: 3.9
- :class:`collections.defaultdict` now supports ``[]``. See :pep:`585`.
+ :class:`collections.defaultdict` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: OrderedDict(collections.OrderedDict, MutableMapping[KT, VT])
@@ -1119,7 +1133,8 @@ Corresponding to types in :mod:`collections`
.. versionadded:: 3.7.2
.. deprecated:: 3.9
- :class:`collections.OrderedDict` now supports ``[]``. See :pep:`585`.
+ :class:`collections.OrderedDict` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: ChainMap(collections.ChainMap, MutableMapping[KT, VT])
@@ -1129,7 +1144,8 @@ Corresponding to types in :mod:`collections`
.. versionadded:: 3.6.1
.. deprecated:: 3.9
- :class:`collections.ChainMap` now supports ``[]``. See :pep:`585`.
+ :class:`collections.ChainMap` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: Counter(collections.Counter, Dict[T, int])
@@ -1139,7 +1155,8 @@ Corresponding to types in :mod:`collections`
.. versionadded:: 3.6.1
.. deprecated:: 3.9
- :class:`collections.Counter` now supports ``[]``. See :pep:`585`.
+ :class:`collections.Counter` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: Deque(deque, MutableSequence[T])
@@ -1149,7 +1166,8 @@ Corresponding to types in :mod:`collections`
.. versionadded:: 3.6.1
.. deprecated:: 3.9
- :class:`collections.deque` now supports ``[]``. See :pep:`585`.
+ :class:`collections.deque` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
Other concrete types
""""""""""""""""""""
@@ -1174,7 +1192,8 @@ Other concrete types
``Match[bytes]``. These types are also in the ``typing.re`` namespace.
.. deprecated:: 3.9
- Classes ``Pattern`` and ``Match`` from :mod:`re` now support ``[]``. See :pep:`585`.
+ Classes ``Pattern`` and ``Match`` from :mod:`re` now support ``[]``.
+ See :pep:`585` and :ref:`types-genericalias`.
.. class:: Text
@@ -1201,7 +1220,8 @@ Corresponding to collections in :mod:`collections.abc`
A generic version of :class:`collections.abc.Set`.
.. deprecated:: 3.9
- :class:`collections.abc.Set` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Set` now supports ``[]``. See :pep:`585` and
+ :ref:`types-genericalias`.
.. class:: ByteString(Sequence[int])
@@ -1214,7 +1234,8 @@ Corresponding to collections in :mod:`collections.abc`
annotate arguments of any of the types mentioned above.
.. deprecated:: 3.9
- :class:`collections.abc.ByteString` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.ByteString` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Collection(Sized, Iterable[T_co], Container[T_co])
@@ -1223,28 +1244,32 @@ Corresponding to collections in :mod:`collections.abc`
.. versionadded:: 3.6.0
.. deprecated:: 3.9
- :class:`collections.abc.Collection` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Collection` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Container(Generic[T_co])
A generic version of :class:`collections.abc.Container`.
.. deprecated:: 3.9
- :class:`collections.abc.Container` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Container` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: ItemsView(MappingView, Generic[KT_co, VT_co])
A generic version of :class:`collections.abc.ItemsView`.
.. deprecated:: 3.9
- :class:`collections.abc.ItemsView` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.ItemsView` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: KeysView(MappingView[KT_co], AbstractSet[KT_co])
A generic version of :class:`collections.abc.KeysView`.
.. deprecated:: 3.9
- :class:`collections.abc.KeysView` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.KeysView` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Mapping(Sized, Collection[KT], Generic[VT_co])
@@ -1255,49 +1280,56 @@ Corresponding to collections in :mod:`collections.abc`
return word_list[word]
.. deprecated:: 3.9
- :class:`collections.abc.Mapping` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Mapping` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: MappingView(Sized, Iterable[T_co])
A generic version of :class:`collections.abc.MappingView`.
.. deprecated:: 3.9
- :class:`collections.abc.MappingView` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.MappingView` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: MutableMapping(Mapping[KT, VT])
A generic version of :class:`collections.abc.MutableMapping`.
.. deprecated:: 3.9
- :class:`collections.abc.MutableMapping` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.MutableMapping` now supports ``[]``. See
+ :pep:`585` and :ref:`types-genericalias`.
.. class:: MutableSequence(Sequence[T])
A generic version of :class:`collections.abc.MutableSequence`.
.. deprecated:: 3.9
- :class:`collections.abc.MutableSequence` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.MutableSequence` now supports ``[]``. See
+ :pep:`585` and :ref:`types-genericalias`.
.. class:: MutableSet(AbstractSet[T])
A generic version of :class:`collections.abc.MutableSet`.
.. deprecated:: 3.9
- :class:`collections.abc.MutableSet` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.MutableSet` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Sequence(Reversible[T_co], Collection[T_co])
A generic version of :class:`collections.abc.Sequence`.
.. deprecated:: 3.9
- :class:`collections.abc.Sequence` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Sequence` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: ValuesView(MappingView[VT_co])
A generic version of :class:`collections.abc.ValuesView`.
.. deprecated:: 3.9
- :class:`collections.abc.ValuesView` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.ValuesView` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
Corresponding to other types in :mod:`collections.abc`
""""""""""""""""""""""""""""""""""""""""""""""""""""""
@@ -1307,14 +1339,16 @@ Corresponding to other types in :mod:`collections.abc`
A generic version of :class:`collections.abc.Iterable`.
.. deprecated:: 3.9
- :class:`collections.abc.Iterable` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Iterable` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Iterator(Iterable[T_co])
A generic version of :class:`collections.abc.Iterator`.
.. deprecated:: 3.9
- :class:`collections.abc.Iterator` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Iterator` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Generator(Iterator[T_co], Generic[T_co, T_contra, V_co])
@@ -1348,7 +1382,8 @@ Corresponding to other types in :mod:`collections.abc`
start += 1
.. deprecated:: 3.9
- :class:`collections.abc.Generator` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Generator` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Hashable
@@ -1359,7 +1394,8 @@ Corresponding to other types in :mod:`collections.abc`
A generic version of :class:`collections.abc.Reversible`.
.. deprecated:: 3.9
- :class:`collections.abc.Reversible` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Reversible` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Sized
@@ -1384,7 +1420,8 @@ Asynchronous programming
.. versionadded:: 3.5.3
.. deprecated:: 3.9
- :class:`collections.abc.Coroutine` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Coroutine` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: AsyncGenerator(AsyncIterator[T_co], Generic[T_co, T_contra])
@@ -1420,7 +1457,8 @@ Asynchronous programming
.. versionadded:: 3.6.1
.. deprecated:: 3.9
- :class:`collections.abc.AsyncGenerator` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.AsyncGenerator` now supports ``[]``. See
+ :pep:`585` and :ref:`types-genericalias`.
.. class:: AsyncIterable(Generic[T_co])
@@ -1429,7 +1467,8 @@ Asynchronous programming
.. versionadded:: 3.5.2
.. deprecated:: 3.9
- :class:`collections.abc.AsyncIterable` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.AsyncIterable` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: AsyncIterator(AsyncIterable[T_co])
@@ -1438,7 +1477,8 @@ Asynchronous programming
.. versionadded:: 3.5.2
.. deprecated:: 3.9
- :class:`collections.abc.AsyncIterator` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.AsyncIterator` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
.. class:: Awaitable(Generic[T_co])
@@ -1447,7 +1487,8 @@ Asynchronous programming
.. versionadded:: 3.5.2
.. deprecated:: 3.9
- :class:`collections.abc.Awaitable` now supports ``[]``. See :pep:`585`.
+ :class:`collections.abc.Awaitable` now supports ``[]``. See :pep:`585`
+ and :ref:`types-genericalias`.
Context manager types
@@ -1461,7 +1502,8 @@ Context manager types
.. versionadded:: 3.6.0
.. deprecated:: 3.9
- :class:`collections.contextlib.AbstractContextManager` now supports ``[]``. See :pep:`585`.
+ :class:`contextlib.AbstractContextManager` now supports ``[]``. See
+ :pep:`585` and :ref:`types-genericalias`.
.. class:: AsyncContextManager(Generic[T_co])
@@ -1471,7 +1513,8 @@ Context manager types
.. versionadded:: 3.6.2
.. deprecated:: 3.9
- :class:`collections.contextlib.AbstractAsyncContextManager` now supports ``[]``. See :pep:`585`.
+ :class:`contextlib.AbstractAsyncContextManager` now supports ``[]``. See
+ :pep:`585` and :ref:`types-genericalias`.
Protocols
---------
@@ -1650,6 +1693,9 @@ Introspection helpers
For a typing object of the form ``X[Y, Z, ...]`` these functions return
``X`` and ``(Y, Z, ...)``. If ``X`` is a generic alias for a builtin or
:mod:`collections` class, it gets normalized to the original class.
+ If ``X`` is a :class:`Union` or :class:`Literal` contained in another
+ generic type, the order of ``(Y, Z, ...)`` may be different from the order
+ of the original arguments ``[Y, Z, ...]`` due to type caching.
For unsupported objects return ``None`` and ``()`` correspondingly.
Examples::
@@ -1691,7 +1737,7 @@ Constant
If ``from __future__ import annotations`` is used in Python 3.7 or later,
annotations are not evaluated at function definition time.
- Instead, the are stored as strings in ``__annotations__``,
+ Instead, they are stored as strings in ``__annotations__``,
This makes it unnecessary to use quotes around the annotation.
(see :pep:`563`).
diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
index 285bb9e..9fc6620 100644
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -593,8 +593,9 @@ The following decorators and exception implement test skipping and expected fail
.. decorator:: expectedFailure
- Mark the test as an expected failure. If the test fails it will be
- considered a success. If the test passes, it will be considered a failure.
+ Mark the test as an expected failure or error. If the test fails or errors
+ it will be considered a success. If the test passes, it will be considered
+ a failure.
.. exception:: SkipTest(reason)
@@ -896,8 +897,7 @@ Test cases
.. method:: assertIs(first, second, msg=None)
assertIsNot(first, second, msg=None)
- Test that *first* and *second* evaluate (or don't evaluate) to the
- same object.
+ Test that *first* and *second* are (or are not) the same object.
.. versionadded:: 3.1
@@ -1088,7 +1088,8 @@ Test cases
If given, *logger* should be a :class:`logging.Logger` object or a
:class:`str` giving the name of a logger. The default is the root
- logger, which will catch all messages.
+ logger, which will catch all messages that were not blocked by a
+ non-propagating descendent logger.
If given, *level* should be either a numeric logging level or
its string equivalent (for example either ``"ERROR"`` or
@@ -1945,7 +1946,7 @@ Loading and running tests
A list containing 2-tuples of :class:`TestCase` instances and strings
holding formatted tracebacks. Each tuple represents an expected failure
- of the test case.
+ or error of the test case.
.. attribute:: unexpectedSuccesses
@@ -2071,8 +2072,8 @@ Loading and running tests
.. method:: addExpectedFailure(test, err)
- Called when the test case *test* fails, but was marked with the
- :func:`expectedFailure` decorator.
+ Called when the test case *test* fails or errors, but was marked with
+ the :func:`expectedFailure` decorator.
The default implementation appends a tuple ``(test, formatted_err)`` to
the instance's :attr:`expectedFailures` attribute, where *formatted_err*
diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst
index 2c78cd9..bf72c46 100644
--- a/Doc/library/xml.dom.minidom.rst
+++ b/Doc/library/xml.dom.minidom.rst
@@ -132,7 +132,7 @@ module documentation. This section lists the differences between the API and
... # Work with dom.
-.. method:: Node.writexml(writer, indent="", addindent="", newl="",
+.. method:: Node.writexml(writer, indent="", addindent="", newl="", \
encoding=None, standalone=None)
Write XML to the writer object. The writer receives texts but not bytes as input,
@@ -174,7 +174,7 @@ module documentation. This section lists the differences between the API and
The :meth:`toxml` method now preserves the attribute order specified
by the user.
-.. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None,
+.. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None, \
standalone=None)
Return a pretty-printed version of the document. *indent* specifies the
diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst
index 7725e4d..f4bccf6 100644
--- a/Doc/library/xml.etree.elementtree.rst
+++ b/Doc/library/xml.etree.elementtree.rst
@@ -251,12 +251,18 @@ We can remove elements using :meth:`Element.remove`. Let's say we want to
remove all countries with a rank higher than 50::
>>> for country in root.findall('country'):
+ ... # using root.findall() to avoid removal during traversal
... rank = int(country.find('rank').text)
... if rank > 50:
... root.remove(country)
...
>>> tree.write('output.xml')
+Note that concurrent modification while iterating can lead to problems,
+just like when iterating and modifying Python lists or dicts.
+Therefore, the example first collects all matching elements with
+``root.findall()``, and only then iterates over the list of matches.
+
Our XML now looks like this:
.. code-block:: xml
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index fc304a1..8906387 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -187,6 +187,24 @@ Ellipsis
related to mathematical numbers, but subject to the limitations of numerical
representation in computers.
+ The string representations of the numeric classes, computed by
+ :meth:`__repr__` and :meth:`__str__`, have the following
+ properties:
+
+ * They are valid numeric literals which, when passed to their
+ class constructor, produce an object having the value of the
+ original numeric.
+
+ * The representation is in base 10, when possible.
+
+ * Leading zeros, possibly excepting a single zero before a
+ decimal point, are not shown.
+
+ * Trailing zeros, possibly excepting a single zero after a
+ decimal point, are not shown.
+
+ * A sign is shown only when the number is negative.
+
Python distinguishes between integers, floating point numbers, and complex
numbers:
@@ -1377,12 +1395,14 @@ Basic customization
context (e.g., in the condition of an ``if`` statement), Python will call
:func:`bool` on the value to determine if the result is true or false.
- By default, :meth:`__ne__` delegates to :meth:`__eq__` and
- inverts the result unless it is ``NotImplemented``. There are no other
- implied relationships among the comparison operators, for example,
- the truth of ``(x` ``object.__getattr__`` with arguments
+ ``obj`` and ``name``.
+
.. method:: object.__setattr__(self, name, value)
@@ -1551,12 +1577,24 @@ access (use of, assignment to, or deletion of ``x.name``) for class instances.
call the base class method with the same name, for example,
``object.__setattr__(self, name, value)``.
+ .. audit-event:: object.__setattr__ obj,name,value object.__setattr__
+
+ For certain sensitive attribute assignments, raises an
+ :ref:`auditing event ` ``object.__setattr__`` with arguments
+ ``obj``, ``name``, ``value``.
+
.. method:: object.__delattr__(self, name)
Like :meth:`__setattr__` but for attribute deletion instead of assignment. This
should only be implemented if ``del obj.name`` is meaningful for the object.
+ .. audit-event:: object.__delattr__ obj,name object.__delattr__
+
+ For certain sensitive attribute deletions, raises an
+ :ref:`auditing event ` ``object.__delattr__`` with arguments
+ ``obj`` and ``name``.
+
.. method:: object.__dir__(self)
@@ -2130,7 +2168,7 @@ Emulating callable objects
.. index:: pair: call; instance
Called when the instance is "called" as a function; if this method is defined,
- ``x(arg1, arg2, ...)`` is a shorthand for ``x.__call__(arg1, arg2, ...)``.
+ ``x(arg1, arg2, ...)`` roughly translates to ``type(x).__call__(x, arg1, ...)``.
.. _sequence-types:
@@ -2376,10 +2414,11 @@ left undefined.
.. note::
- If the right operand's type is a subclass of the left operand's type and that
- subclass provides the reflected method for the operation, this method will be
- called before the left operand's non-reflected method. This behavior allows
- subclasses to override their ancestors' operations.
+ If the right operand's type is a subclass of the left operand's type and
+ that subclass provides a different implementation of the reflected method
+ for the operation, this method will be called before the left operand's
+ non-reflected method. This behavior allows subclasses to override their
+ ancestors' operations.
.. method:: object.__iadd__(self, other)
@@ -2778,6 +2817,6 @@ An example of an asynchronous context manager class::
methodâthat will instead have the opposite effect of explicitly
*blocking* such fallback.
-.. [#] For operands of the same type, it is assumed that if the non-reflected method
- (such as :meth:`__add__`) fails the operation is not supported, which is why the
- reflected method is not called.
+.. [#] For operands of the same type, it is assumed that if the non-reflected
+ method -- such as :meth:`__add__` -- fails then the overall operation is not
+ supported, which is why the reflected method is not called.
diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst
index b68c298..512aa5a 100644
--- a/Doc/reference/expressions.rst
+++ b/Doc/reference/expressions.rst
@@ -162,6 +162,8 @@ ambiguities and allow common typos to pass uncaught.
Displays for lists, sets and dictionaries
-----------------------------------------
+.. index:: single: comprehensions
+
For constructing a list, a set or a dictionary Python provides special syntax
called "displays", each of them in two flavors:
@@ -260,6 +262,7 @@ Set displays
.. index::
pair: set; display
+ pair: set; comprehensions
object: set
single: {} (curly brackets); set expression
single: , (comma); expression list
@@ -287,6 +290,7 @@ Dictionary displays
.. index::
pair: dictionary; display
+ pair: dictionary; comprehensions
key, datum, key/datum pair
object: dictionary
single: {} (curly brackets); dictionary expression
@@ -796,8 +800,8 @@ Subscriptions
object: dictionary
pair: sequence; item
-A subscription selects an item of a sequence (string, tuple or list) or mapping
-(dictionary) object:
+Subscription of a sequence (string, tuple or list) or mapping (dictionary)
+object usually selects an item from the collection:
.. productionlist:: python-grammar
subscription: `primary` "[" `expression_list` "]"
@@ -833,6 +837,11 @@ this method will need to explicitly add that support.
A string's items are characters. A character is not a separate data type but a
string of exactly one character.
+Subscription of certain :term:`classes ` or :term:`types `
+creates a :ref:`generic alias `.
+In this case, user-defined classes can support subscription by providing a
+:meth:`__class_getitem__` classmethod.
+
.. _slicings:
diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst
index 4c36e15..c595242 100644
--- a/Doc/reference/import.rst
+++ b/Doc/reference/import.rst
@@ -680,7 +680,7 @@ Here are the exact rules used:
Cached bytecode invalidation
----------------------------
-Before Python loads cached bytecode from ``.pyc`` file, it checks whether the
+Before Python loads cached bytecode from a ``.pyc`` file, it checks whether the
cache is up-to-date with the source ``.py`` file. By default, Python does this
by storing the source's last-modified timestamp and size in the cache file when
writing it. At runtime, the import system then validates the cache file by
@@ -857,9 +857,8 @@ module. ``find_spec()`` returns a fully populated spec for the module.
This spec will always have "loader" set (with one exception).
To indicate to the import machinery that the spec represents a namespace
-:term:`portion`, the path entry finder sets "loader" on the spec to
-``None`` and "submodule_search_locations" to a list containing the
-portion.
+:term:`portion`, the path entry finder sets "submodule_search_locations" to
+a list containing the portion.
.. versionchanged:: 3.4
:meth:`~importlib.abc.PathEntryFinder.find_spec` replaced
@@ -875,18 +874,7 @@ portion.
:meth:`~importlib.abc.PathEntryFinder.find_loader` takes one argument, the
fully qualified name of the module being imported. ``find_loader()``
returns a 2-tuple where the first item is the loader and the second item
- is a namespace :term:`portion`. When the first item (i.e. the loader) is
- ``None``, this means that while the path entry finder does not have a
- loader for the named module, it knows that the path entry contributes to
- a namespace portion for the named module. This will almost always be the
- case where Python is asked to import a namespace package that has no
- physical presence on the file system. When a path entry finder returns
- ``None`` for the loader, the second item of the 2-tuple return value must
- be a sequence, although it can be empty.
-
- If ``find_loader()`` returns a non-``None`` loader value, the portion is
- ignored and the loader is returned from the path based finder, terminating
- the search through the path entries.
+ is a namespace :term:`portion`.
For backwards compatibility with other implementations of the import
protocol, many path entry finders also support the same,
diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst
index 93be327..f8ab2e9 100644
--- a/Doc/reference/simple_stmts.rst
+++ b/Doc/reference/simple_stmts.rst
@@ -874,8 +874,8 @@ can appear before a future statement are:
* blank lines, and
* other future statements.
-The only feature in Python 3.7 that requires using the future statement is
-``annotations``.
+The only feature that requires using the future statement is
+``annotations`` (see :pep:`563`).
All historical features enabled by the future statement are still recognized
by Python 3. The list includes ``absolute_import``, ``division``,
diff --git a/Doc/tools/extensions/c_annotations.py b/Doc/tools/extensions/c_annotations.py
index fa8244a..76c9d92 100644
--- a/Doc/tools/extensions/c_annotations.py
+++ b/Doc/tools/extensions/c_annotations.py
@@ -79,9 +79,9 @@ class Annotations(dict):
classes=['stableabi']))
if par['objtype'] != 'function':
continue
- if not par[0].has_key('names') or not par[0]['names']:
+ if not par[0].has_key('ids') or not par[0]['ids']:
continue
- name = par[0]['names'][0]
+ name = par[0]['ids'][0]
if name.startswith("c."):
name = name[2:]
entry = self.get(name)
diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py
index f08f4ab..2fad9ec 100644
--- a/Doc/tools/extensions/pyspecific.py
+++ b/Doc/tools/extensions/pyspecific.py
@@ -31,7 +31,12 @@ from sphinx.util import status_iterator, logging
from sphinx.util.nodes import split_explicit_title
from sphinx.writers.text import TextWriter, TextTranslator
from sphinx.writers.latex import LaTeXTranslator
-from sphinx.domains.python import PyModulelevel, PyClassmember
+
+try:
+ from sphinx.domains.python import PyFunction, PyMethod
+except ImportError:
+ from sphinx.domains.python import PyClassmember as PyMethod
+ from sphinx.domains.python import PyModulelevel as PyFunction
# Support for checking for suspicious markup
@@ -238,17 +243,18 @@ class PyDecoratorMixin(object):
return False
-class PyDecoratorFunction(PyDecoratorMixin, PyModulelevel):
+class PyDecoratorFunction(PyDecoratorMixin, PyFunction):
def run(self):
# a decorator function is a function after all
self.name = 'py:function'
- return PyModulelevel.run(self)
+ return PyFunction.run(self)
-class PyDecoratorMethod(PyDecoratorMixin, PyClassmember):
+# TODO: Use sphinx.domains.python.PyDecoratorMethod when possible
+class PyDecoratorMethod(PyDecoratorMixin, PyMethod):
def run(self):
self.name = 'py:method'
- return PyClassmember.run(self)
+ return PyMethod.run(self)
class PyCoroutineMixin(object):
@@ -265,31 +271,31 @@ class PyAwaitableMixin(object):
return ret
-class PyCoroutineFunction(PyCoroutineMixin, PyModulelevel):
+class PyCoroutineFunction(PyCoroutineMixin, PyFunction):
def run(self):
self.name = 'py:function'
- return PyModulelevel.run(self)
+ return PyFunction.run(self)
-class PyCoroutineMethod(PyCoroutineMixin, PyClassmember):
+class PyCoroutineMethod(PyCoroutineMixin, PyMethod):
def run(self):
self.name = 'py:method'
- return PyClassmember.run(self)
+ return PyMethod.run(self)
-class PyAwaitableFunction(PyAwaitableMixin, PyClassmember):
+class PyAwaitableFunction(PyAwaitableMixin, PyFunction):
def run(self):
self.name = 'py:function'
- return PyClassmember.run(self)
+ return PyFunction.run(self)
-class PyAwaitableMethod(PyAwaitableMixin, PyClassmember):
+class PyAwaitableMethod(PyAwaitableMixin, PyMethod):
def run(self):
self.name = 'py:method'
- return PyClassmember.run(self)
+ return PyMethod.run(self)
-class PyAbstractMethod(PyClassmember):
+class PyAbstractMethod(PyMethod):
def handle_signature(self, sig, signode):
ret = super(PyAbstractMethod, self).handle_signature(sig, signode)
@@ -299,7 +305,7 @@ class PyAbstractMethod(PyClassmember):
def run(self):
self.name = 'py:method'
- return PyClassmember.run(self)
+ return PyMethod.run(self)
# Support for documenting version of removal in deprecations
diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index c51f178..1a1c7d0 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -15,7 +15,6 @@
'3.8': '3.8',
'3.7': '3.7',
'3.6': '3.6',
- '3.5': '3.5',
'2.7': '2.7',
};
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index 7be8d0a..c9777c6 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -5,14 +5,13 @@ c-api/sequence,,:i2,o[i1:i2]
c-api/tuple,,:high,p[low:high]
c-api/unicode,,:end,str[start:end]
c-api/unicode,,:start,unicode[start:start+length]
-distutils/examples,267,`,This is the description of the ``foobar`` package.
+distutils/examples,,`,This is the description of the ``foobar`` package.
distutils/setupscript,,::,
extending/embedding,,:numargs,"if(!PyArg_ParseTuple(args, "":numargs""))"
extending/extending,,:myfunction,"PyArg_ParseTuple(args, ""D:myfunction"", &c);"
extending/extending,,:set,"if (PyArg_ParseTuple(args, ""O:set_callback"", &temp)) {"
extending/newtypes,,:call,"if (!PyArg_ParseTuple(args, ""sss:call"", &arg1, &arg2, &arg3)) {"
faq/programming,,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr("
-faq/programming,,::,for x in sequence[::-1]:
faq/programming,,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,"
faq/programming,,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,"
faq/windows,,:d48eceb,"Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32"
@@ -24,6 +23,9 @@ howto/curses,,:blue,"2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white.
howto/curses,,:magenta,"2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white. The"
howto/curses,,:cyan,"2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white. The"
howto/curses,,:white,"2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white. The"
+howto/descriptor,,:root,"INFO:root"
+howto/descriptor,,:Updating,"root:Updating"
+howto/descriptor,,:Accessing,"root:Accessing"
howto/instrumentation,,::,python$target:::function-entry
howto/instrumentation,,:function,python$target:::function-entry
howto/instrumentation,,::,python$target:::function-return
@@ -147,8 +149,10 @@ library/ipaddress,,:db8,IPv6Address('2001:db8::')
library/ipaddress,,::,IPv6Address('2001:db8::')
library/ipaddress,,:db8,>>> ipaddress.IPv6Address('2001:db8::1000')
library/ipaddress,,::,>>> ipaddress.IPv6Address('2001:db8::1000')
-library/ipaddress,,:db8,IPv6Address('2001:db8::1000')
-library/ipaddress,,::,IPv6Address('2001:db8::1000')
+library/ipaddress,,:db8,'2001:db8::1000'
+library/ipaddress,,::,'2001:db8::1000'
+library/ipaddress,231,:db8,">>> f'{ipaddress.IPv6Address(""2001:db8::1000""):s}'"
+library/ipaddress,231,::,">>> f'{ipaddress.IPv6Address(""2001:db8::1000""):s}'"
library/ipaddress,,::,IPv6Address('ff02::5678%1')
library/ipaddress,,::,fe80::1234
library/ipaddress,,:db8,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer"
diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html
index 7a40be7..1c1cb54 100644
--- a/Doc/tools/templates/indexsidebar.html
+++ b/Doc/tools/templates/indexsidebar.html
@@ -7,7 +7,6 @@
diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst
index ff4c797..5c6b65f 100644
--- a/Doc/tutorial/datastructures.rst
+++ b/Doc/tutorial/datastructures.rst
@@ -78,7 +78,7 @@ objects:
Return the number of times *x* appears in the list.
-.. method:: list.sort(key=None, reverse=False)
+.. method:: list.sort(*, key=None, reverse=False)
:noindex:
Sort the items of the list in place (the arguments can be used for sort
diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst
index 0ce9646..efe44da 100644
--- a/Doc/tutorial/errors.rst
+++ b/Doc/tutorial/errors.rst
@@ -273,15 +273,15 @@ Exception Chaining
==================
The :keyword:`raise` statement allows an optional :keyword:`from` which enables
-chaining exceptions by setting the ``__cause__`` attribute of the raised
-exception. For example::
+chaining exceptions. For example::
- raise RuntimeError from OSError
+ # exc must be exception instance or None.
+ raise RuntimeError from exc
This can be useful when you are transforming exceptions. For example::
>>> def func():
- ... raise IOError
+ ... raise IOError
...
>>> try:
... func()
@@ -297,12 +297,11 @@ This can be useful when you are transforming exceptions. For example::
Traceback (most recent call last):
File "", line 4, in
- RuntimeError
+ RuntimeError: Failed to open database
-The expression following the :keyword:`from` must be either an exception or
-``None``. Exception chaining happens automatically when an exception is raised
-inside an exception handler or :keyword:`finally` section. Exception chaining
-can be disabled by using ``from None`` idiom:
+Exception chaining happens automatically when an exception is raised inside an
+:keyword:`except` or :keyword:`finally` section. Exception chaining can be
+disabled by using ``from None`` idiom:
>>> try:
... open('database.sqlite')
@@ -313,6 +312,8 @@ can be disabled by using ``from None`` idiom:
File "", line 4, in
RuntimeError
+For more information about chaining mechanics, see :ref:`bltin-exceptions`.
+
.. _tut-userexceptions:
diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst
index 366a532..4e27cff 100644
--- a/Doc/tutorial/inputoutput.rst
+++ b/Doc/tutorial/inputoutput.rst
@@ -329,11 +329,16 @@ equivalent :keyword:`try`\ -\ :keyword:`finally` blocks::
If you're not using the :keyword:`with` keyword, then you should call
``f.close()`` to close the file and immediately free up any system
-resources used by it. If you don't explicitly close a file, Python's
-garbage collector will eventually destroy the object and close the
-open file for you, but the file may stay open for a while. Another
-risk is that different Python implementations will do this clean-up at
-different times.
+resources used by it.
+
+.. warning::
+ Calling ``f.write()`` without using the :keyword:`!with` keyword or calling
+ ``f.close()`` **might** result in the arguments
+ of ``f.write()`` not being completely written to the disk, even if the
+ program exits successfully.
+
+..
+ See also https://bugs.python.org/issue17852
After a file object is closed, either by a :keyword:`with` statement
or by calling ``f.close()``, attempts to use the file object will
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
index f91ab02..95342f3 100644
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -555,7 +555,7 @@ conflict.
the interactive session. You can also change the prompts :data:`sys.ps1` and
:data:`sys.ps2` and the hook :data:`sys.__interactivehook__` in this file.
- .. audit-event:: cpython.run_startup filename PYTHONSTARTUP
+ .. audit-event:: cpython.run_startup filename envvar-PYTHONSTARTUP
Raises an :ref:`auditing event ` ``cpython.run_startup`` with
the filename as the argument when called on startup.
diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc
index c8f6e8f..5e724cd 100644
--- a/Doc/using/venv-create.inc
+++ b/Doc/using/venv-create.inc
@@ -67,7 +67,7 @@ The command, if run with ``-h``, will show the available options::
Once an environment has been created, you may wish to activate it, e.g. by
sourcing an activate script in its bin directory.
-.. versionchanged:: 3.8
+.. versionchanged:: 3.9
Add ``--upgrade-deps`` option to upgrade pip + setuptools to the latest on PyPI
.. versionchanged:: 3.4
diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
index b95a43c..d0c342e 100644
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -23,8 +23,8 @@ available for application-local distributions.
As specified in :pep:`11`, a Python release only supports a Windows platform
while Microsoft considers the platform under extended support. This means that
-Python |version| supports Windows Vista and newer. If you require Windows XP
-support then please install Python 3.4.
+Python |version| supports Windows 8.1 and newer. If you require Windows 7
+support, please install Python 3.8.
There are a number of different installers available for Windows, each with
certain benefits and downsides.
@@ -103,9 +103,9 @@ paths longer than this would not resolve and errors would result.
In the latest versions of Windows, this limitation can be expanded to
approximately 32,000 characters. Your administrator will need to activate the
-"Enable Win32 long paths" group policy, or set the registry value
-``HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem@LongPathsEnabled``
-to ``1``.
+"Enable Win32 long paths" group policy, or set ``LongPathsEnabled`` to ``1``
+in the registry key
+``HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem``.
This allows the :func:`open` function, the :mod:`os` module and most other
path functionality to accept and return paths longer than 260 characters.
diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index a2fa178..0b4820f 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -870,8 +870,18 @@ clipboard. Converting strings from Tcl to Python and back now never fails.
(Many people worked on this for eight years but the problem was finally
solved by Serhiy Storchaka in :issue:`13153`.)
+New in 3.8.1:
+
+Add option to toggle cursor blink off. (Contributed by Zackery Spytz
+in :issue:`4603`.)
+
+Escape key now closes IDLE completion windows. (Contributed by Johnny
+Najera in :issue:`38944`.)
+
The changes above have been backported to 3.7 maintenance releases.
+Add keywords to module name completion list. (Contributed by Terry J.
+Reedy in :issue:`37765`.)
inspect
-------
@@ -2103,9 +2113,6 @@ Changes in the C API
(Contributed by Antoine Pitrou in :issue:`32388`.)
-* The :c:func:`PyCode_New` has a new parameter in the second position (*posonlyargcount*)
- to support :pep:`570`, indicating the number of positional-only arguments.
-
* The functions :c:func:`PyNode_AddChild` and :c:func:`PyParser_AddToken` now accept
two additional ``int`` arguments *end_lineno* and *end_col_offset*.
diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index 9a09e71..f8f421b 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -135,8 +135,8 @@ More generally, try to run your tests in the :ref:`Python Development Mode
` which helps to prepare your code to make it compatible with the
next Python version.
-Note: a number of pre-existing deprecatations were removed in this version
-of Python as well. Consult the :ref:`removed-in-python-39` section.
+Note: a number of pre-existing deprecations were removed in this version of
+Python as well. Consult the :ref:`removed-in-python-39` section.
New Features
@@ -781,41 +781,41 @@ Here's a summary of performance improvements from Python 3.4 through Python 3.9:
-------------- --- --- --- --- --- ---
Variable and attribute read access:
- read_local 7.1 7.1 5.4 5.1 3.9 4.0
- read_nonlocal 7.1 8.1 5.8 5.4 4.4 4.8
- read_global 15.5 19.0 14.3 13.6 7.6 7.7
- read_builtin 21.1 21.6 18.5 19.0 7.5 7.7
- read_classvar_from_class 25.6 26.5 20.7 19.5 18.4 18.6
- read_classvar_from_instance 22.8 23.5 18.8 17.1 16.4 20.1
- read_instancevar 32.4 33.1 28.0 26.3 25.4 27.7
- read_instancevar_slots 27.8 31.3 20.8 20.8 20.2 24.5
- read_namedtuple 73.8 57.5 45.0 46.8 18.4 23.2
- read_boundmethod 37.6 37.9 29.6 26.9 27.7 45.9
+ read_local 7.1 7.1 5.4 5.1 3.9 3.9
+ read_nonlocal 7.1 8.1 5.8 5.4 4.4 4.5
+ read_global 15.5 19.0 14.3 13.6 7.6 7.8
+ read_builtin 21.1 21.6 18.5 19.0 7.5 7.8
+ read_classvar_from_class 25.6 26.5 20.7 19.5 18.4 17.9
+ read_classvar_from_instance 22.8 23.5 18.8 17.1 16.4 16.9
+ read_instancevar 32.4 33.1 28.0 26.3 25.4 25.3
+ read_instancevar_slots 27.8 31.3 20.8 20.8 20.2 20.5
+ read_namedtuple 73.8 57.5 45.0 46.8 18.4 18.7
+ read_boundmethod 37.6 37.9 29.6 26.9 27.7 41.1
Variable and attribute write access:
- write_local 8.7 9.3 5.5 5.3 4.3 4.2
- write_nonlocal 10.5 11.1 5.6 5.5 4.7 4.9
- write_global 19.7 21.2 18.0 18.0 15.8 17.2
- write_classvar 92.9 96.0 104.6 102.1 39.2 43.2
- write_instancevar 44.6 45.8 40.0 38.9 35.5 40.7
- write_instancevar_slots 35.6 36.1 27.3 26.6 25.7 27.7
+ write_local 8.7 9.3 5.5 5.3 4.3 4.3
+ write_nonlocal 10.5 11.1 5.6 5.5 4.7 4.8
+ write_global 19.7 21.2 18.0 18.0 15.8 16.7
+ write_classvar 92.9 96.0 104.6 102.1 39.2 39.8
+ write_instancevar 44.6 45.8 40.0 38.9 35.5 37.4
+ write_instancevar_slots 35.6 36.1 27.3 26.6 25.7 25.8
Data structure read access:
- read_list 24.2 24.5 20.8 20.8 19.0 21.1
- read_deque 24.7 25.5 20.2 20.6 19.8 21.6
- read_dict 24.3 25.7 22.3 23.0 21.0 22.5
- read_strdict 22.6 24.3 19.5 21.2 18.9 21.6
+ read_list 24.2 24.5 20.8 20.8 19.0 19.5
+ read_deque 24.7 25.5 20.2 20.6 19.8 20.2
+ read_dict 24.3 25.7 22.3 23.0 21.0 22.4
+ read_strdict 22.6 24.3 19.5 21.2 18.9 21.5
Data structure write access:
- write_list 27.1 28.5 22.5 21.6 20.0 21.6
- write_deque 28.7 30.1 22.7 21.8 23.5 23.2
- write_dict 31.4 33.3 29.3 29.2 24.7 27.8
- write_strdict 28.4 29.9 27.5 25.2 23.1 29.8
+ write_list 27.1 28.5 22.5 21.6 20.0 20.0
+ write_deque 28.7 30.1 22.7 21.8 23.5 21.7
+ write_dict 31.4 33.3 29.3 29.2 24.7 25.4
+ write_strdict 28.4 29.9 27.5 25.2 23.1 24.5
Stack (or queue) operations:
- list_append_pop 93.4 112.7 75.4 74.2 50.8 53.9
- deque_append_pop 43.5 57.0 49.4 49.2 42.5 45.5
- deque_append_popleft 43.7 57.3 49.7 49.7 42.8 45.5
+ list_append_pop 93.4 112.7 75.4 74.2 50.8 50.6
+ deque_append_pop 43.5 57.0 49.4 49.2 42.5 44.2
+ deque_append_popleft 43.7 57.3 49.7 49.7 42.8 46.4
Timing loop:
loop_overhead 0.5 0.6 0.4 0.3 0.3 0.3
@@ -891,7 +891,7 @@ Deprecated
and will be removed in future Python versions. ``value`` itself should be
used instead of ``Index(value)``. ``Tuple(slices, Load())`` should be
used instead of ``ExtSlice(slices)``.
- (Contributed by Serhiy Storchaka in :issue:`32892`.)
+ (Contributed by Serhiy Storchaka in :issue:`34822`.)
* :mod:`ast` classes ``Suite``, ``Param``, ``AugLoad`` and ``AugStore``
are considered deprecated and will be removed in future Python versions.
@@ -1177,7 +1177,7 @@ CPython bytecode changes
* ``COMPARE_OP`` for rich comparisons
* ``IS_OP`` for 'is' and 'is not' tests
- * ``CONTAINS_OP`` for 'in' and 'is not' tests
+ * ``CONTAINS_OP`` for 'in' and 'not in' tests
* ``JUMP_IF_NOT_EXC_MATCH`` for checking exceptions in 'try-except'
statements.
@@ -1289,6 +1289,10 @@ New Features
representation of a function-like object.
(Patch by Jeroen Demeyer in :issue:`37645`.)
+* Added :c:func:`PyObject_CallOneArg` for calling an object with one
+ positional argument
+ (Patch by Jeroen Demeyer in :issue:`37483`.)
+
Porting to Python 3.9
---------------------
@@ -1450,3 +1454,46 @@ Removed
``PyNullImporter_Type``, ``PyCmpWrapper_Type``, ``PySortWrapper_Type``,
``PyNoArgsFunction``.
(Contributed by Pablo Galindo Salgado in :issue:`39372`.)
+
+Notable changes in Python 3.9.1
+===============================
+
+typing
+------
+
+The behavior of :class:`typing.Literal` was changed to conform with :pep:`586`
+and to match the behavior of static type checkers specified in the PEP.
+
+1. ``Literal`` now de-duplicates parameters.
+2. Equality comparisons between ``Literal`` objects are now order independent.
+3. ``Literal`` comparisons now respect types. For example,
+ ``Literal[0] == Literal[False]`` previously evaluated to ``True``. It is
+ now ``False``. To support this change, the internally used type cache now
+ supports differentiating types.
+4. ``Literal`` objects will now raise a :exc:`TypeError` exception during
+ equality comparisons if one of their parameters are not :term:`immutable`.
+ Note that declaring ``Literal`` with mutable parameters will not throw
+ an error::
+
+ >>> from typing import Literal
+ >>> Literal[{0}]
+ >>> Literal[{0}] == Literal[{False}]
+ Traceback (most recent call last):
+ File "", line 1, in
+ TypeError: unhashable type: 'set'
+
+(Contributed by Yurii Karabas in :issue:`42345`.)
+
+macOS 11.0 (Big Sur) and Apple Silicon Mac support
+--------------------------------------------------
+
+As of 3.9.1, Python now fully supports building and running on macOS 11.0
+(Big Sur) and on Apple Silicon Macs (based on the ``ARM64`` architecture).
+A new universal build variant, ``universal2``, is now available to natively
+support both ``ARM64`` and ``Intel 64`` in one set of executables. Binaries
+can also now be built on current versions of macOS to be deployed on a range
+of older macOS versions (tested to 10.9) while making some newer OS
+functions and options conditionally available based on the operating system
+version in use at runtime ("weaklinking").
+
+(Contributed by Ronald Oussoren and Lawrence D'Anna in :issue:`41100`.)
diff --git a/Grammar/python.gram b/Grammar/python.gram
index 334a7f5..ce78397 100644
--- a/Grammar/python.gram
+++ b/Grammar/python.gram
@@ -182,7 +182,7 @@ with_stmt[stmt_ty]:
| ASYNC 'with' a=','.with_item+ ':' tc=[TYPE_COMMENT] b=block {
CHECK_VERSION(5, "Async with statements are", _Py_AsyncWith(a, b, NEW_TYPE_COMMENT(p, tc), EXTRA)) }
with_item[withitem_ty]:
- | e=expression 'as' t=target &(',' | ')' | ':') { _Py_withitem(e, t, p->arena) }
+ | e=expression 'as' t=star_target &(',' | ')' | ':') { _Py_withitem(e, t, p->arena) }
| invalid_with_item
| e=expression { _Py_withitem(e, NULL, p->arena) }
@@ -301,7 +301,6 @@ block[asdl_seq*] (memo):
| simple_stmt
| invalid_block
-expressions_list[asdl_seq*]: a=','.star_expression+ [','] { a }
star_expressions[expr_ty]:
| a=star_expression b=(',' c=star_expression { c })+ [','] {
_Py_Tuple(CHECK(_PyPegen_seq_insert_in_front(p, a, b)), Load, EXTRA) }
@@ -413,7 +412,7 @@ compare_op_bitwise_or_pair[CmpopExprPair*]:
| is_bitwise_or
eq_bitwise_or[CmpopExprPair*]: '==' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Eq, a) }
noteq_bitwise_or[CmpopExprPair*]:
- | (tok='!=' {_PyPegen_check_barry_as_flufl(p) ? NULL : tok}) a=bitwise_or {_PyPegen_cmpop_expr_pair(p, NotEq, a) }
+ | (tok='!=' { _PyPegen_check_barry_as_flufl(p, tok) ? NULL : tok}) a=bitwise_or {_PyPegen_cmpop_expr_pair(p, NotEq, a) }
lte_bitwise_or[CmpopExprPair*]: '<=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, LtE, a) }
lt_bitwise_or[CmpopExprPair*]: '<' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Lt, a) }
gte_bitwise_or[CmpopExprPair*]: '>=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, GtE, a) }
@@ -460,6 +459,7 @@ await_primary[expr_ty] (memo):
| AWAIT a=primary { CHECK_VERSION(5, "Await expressions are", _Py_Await(a, EXTRA)) }
| primary
primary[expr_ty]:
+ | invalid_primary # must be before 'primay genexp' because of invalid_genexp
| a=primary '.' b=NAME { _Py_Attribute(a, b->v.Name.id, Load, EXTRA) }
| a=primary b=genexp { _Py_Call(a, CHECK(_PyPegen_singleton_seq(p, b)), NULL, EXTRA) }
| a=primary '(' b=[arguments] ')' {
@@ -502,11 +502,11 @@ group[expr_ty]:
| '(' a=(yield_expr | named_expression) ')' { a }
| invalid_group
genexp[expr_ty]:
- | '(' a=expression ~ b=for_if_clauses ')' { _Py_GeneratorExp(a, b, EXTRA) }
+ | '(' a=named_expression ~ b=for_if_clauses ')' { _Py_GeneratorExp(a, b, EXTRA) }
| invalid_comprehension
-set[expr_ty]: '{' a=expressions_list '}' { _Py_Set(a, EXTRA) }
+set[expr_ty]: '{' a=star_named_expressions '}' { _Py_Set(a, EXTRA) }
setcomp[expr_ty]:
- | '{' a=expression ~ b=for_if_clauses '}' { _Py_SetComp(a, b, EXTRA) }
+ | '{' a=named_expression ~ b=for_if_clauses '}' { _Py_SetComp(a, b, EXTRA) }
| invalid_comprehension
dict[expr_ty]:
| '{' a=[double_starred_kvpairs] '}' {
@@ -534,7 +534,7 @@ yield_expr[expr_ty]:
arguments[expr_ty] (memo):
| a=args [','] &')' { a }
- | incorrect_arguments
+ | invalid_arguments
args[expr_ty]:
| a=','.(starred_expression | named_expression !'=')+ b=[',' k=kwargs {k}] { _PyPegen_collect_call_seqs(p, a, b, EXTRA) }
| a=kwargs { _Py_Call(_PyPegen_dummy_name(p),
@@ -619,7 +619,7 @@ t_atom[expr_ty]:
# From here on, there are rules for invalid syntax with specialised error messages
-incorrect_arguments:
+invalid_arguments:
| args ',' '*' { RAISE_SYNTAX_ERROR("iterable argument unpacking follows keyword argument unpacking") }
| a=expression for_if_clauses ',' [args | expression for_if_clauses] {
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "Generator expression must be parenthesized") }
@@ -664,6 +664,8 @@ invalid_del_stmt:
RAISE_SYNTAX_ERROR_INVALID_TARGET(DEL_TARGETS, a) }
invalid_block:
| NEWLINE !INDENT { RAISE_INDENTATION_ERROR("expected an indented block") }
+invalid_primary:
+ | primary a='{' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "invalid syntax") }
invalid_comprehension:
| ('[' | '(' | '{') a=starred_expression for_if_clauses {
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "iterable unpacking cannot be used in comprehension") }
diff --git a/Include/cpython/fileobject.h b/Include/cpython/fileobject.h
index 57eac13..3005ce1 100644
--- a/Include/cpython/fileobject.h
+++ b/Include/cpython/fileobject.h
@@ -8,14 +8,6 @@ extern "C" {
PyAPI_FUNC(char *) Py_UniversalNewlineFgets(char *, int, FILE*, PyObject *);
-#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03060000
-PyAPI_DATA(const char *) Py_FileSystemDefaultEncodeErrors;
-#endif
-
-#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
-PyAPI_DATA(int) Py_UTF8Mode;
-#endif
-
/* The std printer acts as a preliminary sys.stderr until the new io
infrastructure is in place. */
PyAPI_FUNC(PyObject *) PyFile_NewStdPrinter(int);
diff --git a/Include/fileobject.h b/Include/fileobject.h
index 456887e..6ec2994 100644
--- a/Include/fileobject.h
+++ b/Include/fileobject.h
@@ -20,8 +20,15 @@ PyAPI_FUNC(int) PyObject_AsFileDescriptor(PyObject *);
If non-NULL, this is different than the default encoding for strings
*/
PyAPI_DATA(const char *) Py_FileSystemDefaultEncoding;
+#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03060000
+PyAPI_DATA(const char *) Py_FileSystemDefaultEncodeErrors;
+#endif
PyAPI_DATA(int) Py_HasFileSystemDefaultEncoding;
+#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
+PyAPI_DATA(int) Py_UTF8Mode;
+#endif
+
/* A routine to check if a file descriptor can be select()-ed. */
#ifdef _MSC_VER
/* On Windows, any socket fd can be select()-ed, no matter how high */
diff --git a/Include/internal/pycore_byteswap.h b/Include/internal/pycore_byteswap.h
index 975e150..2b20fc6 100644
--- a/Include/internal/pycore_byteswap.h
+++ b/Include/internal/pycore_byteswap.h
@@ -15,12 +15,9 @@ extern "C" {
# error "this header requires Py_BUILD_CORE define"
#endif
-#if ((defined(__GNUC__) \
- && ((__GNUC__ >= 5) || (__GNUC__ == 4) && (__GNUC_MINOR__ >= 8))) \
- || (defined(__clang__) \
- && (__clang_major__ >= 4 \
- || (__clang_major__ == 3 && __clang_minor__ >= 2))))
- /* __builtin_bswap16() is available since GCC 4.8 and clang 3.2,
+#if defined(__GNUC__) \
+ && ((__GNUC__ >= 5) || (__GNUC__ == 4) && (__GNUC_MINOR__ >= 8))
+ /* __builtin_bswap16() is available since GCC 4.8,
__builtin_bswap32() is available since GCC 4.3,
__builtin_bswap64() is available since GCC 4.3. */
# define _PY_HAVE_BUILTIN_BSWAP
@@ -34,7 +31,7 @@ extern "C" {
static inline uint16_t
_Py_bswap16(uint16_t word)
{
-#ifdef _PY_HAVE_BUILTIN_BSWAP
+#if defined(_PY_HAVE_BUILTIN_BSWAP) || _Py__has_builtin(__builtin_bswap16)
return __builtin_bswap16(word);
#elif defined(_MSC_VER)
Py_BUILD_ASSERT(sizeof(word) == sizeof(unsigned short));
@@ -49,7 +46,7 @@ _Py_bswap16(uint16_t word)
static inline uint32_t
_Py_bswap32(uint32_t word)
{
-#ifdef _PY_HAVE_BUILTIN_BSWAP
+#if defined(_PY_HAVE_BUILTIN_BSWAP) || _Py__has_builtin(__builtin_bswap32)
return __builtin_bswap32(word);
#elif defined(_MSC_VER)
Py_BUILD_ASSERT(sizeof(word) == sizeof(unsigned long));
@@ -66,7 +63,7 @@ _Py_bswap32(uint32_t word)
static inline uint64_t
_Py_bswap64(uint64_t word)
{
-#ifdef _PY_HAVE_BUILTIN_BSWAP
+#if defined(_PY_HAVE_BUILTIN_BSWAP) || _Py__has_builtin(__builtin_bswap64)
return __builtin_bswap64(word);
#elif defined(_MSC_VER)
return _byteswap_uint64(word);
diff --git a/Include/internal/pycore_pylifecycle.h b/Include/internal/pycore_pylifecycle.h
index b76bb2c..50ab645 100644
--- a/Include/internal/pycore_pylifecycle.h
+++ b/Include/internal/pycore_pylifecycle.h
@@ -68,6 +68,7 @@ extern void _PyFloat_Fini(void);
extern void _PySlice_Fini(void);
extern void _PyAsyncGen_Fini(void);
+extern int _PySignal_Init(int install_signal_handlers);
extern void PyOS_FiniInterrupts(void);
extern void _PyExc_Fini(void);
diff --git a/Include/patchlevel.h b/Include/patchlevel.h
index a9e8ef1..0b5d280 100644
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -18,12 +18,12 @@
/*--start constants--*/
#define PY_MAJOR_VERSION 3
#define PY_MINOR_VERSION 9
-#define PY_MICRO_VERSION 0
+#define PY_MICRO_VERSION 1
#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL
#define PY_RELEASE_SERIAL 0
/* Version as a string */
-#define PY_VERSION "3.9.0"
+#define PY_VERSION "3.9.1"
/*--end constants--*/
/* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2.
diff --git a/Include/pyport.h b/Include/pyport.h
index 6505af4..4bd4eb4 100644
--- a/Include/pyport.h
+++ b/Include/pyport.h
@@ -863,4 +863,16 @@ extern _invalid_parameter_handler _Py_silent_invalid_parameter_handler;
# define _Py_NO_RETURN
#endif
+
+// Preprocessor check for a builtin preprocessor function. Always return 0
+// if __has_builtin() macro is not defined.
+//
+// __has_builtin() is available on clang and GCC 10.
+#ifdef __has_builtin
+# define _Py__has_builtin(x) __has_builtin(x)
+#else
+# define _Py__has_builtin(x) 0
+#endif
+
+
#endif /* Py_PYPORT_H */
diff --git a/Lib/_osx_support.py b/Lib/_osx_support.py
index e9efce7..37975fe 100644
--- a/Lib/_osx_support.py
+++ b/Lib/_osx_support.py
@@ -52,7 +52,7 @@ def _find_executable(executable, path=None):
return executable
-def _read_output(commandstring):
+def _read_output(commandstring, capture_stderr=False):
"""Output from successful command execution or None"""
# Similar to os.popen(commandstring, "r").read(),
# but without actually using os.popen because that
@@ -67,7 +67,10 @@ def _read_output(commandstring):
os.getpid(),), "w+b")
with contextlib.closing(fp) as fp:
- cmd = "%s 2>/dev/null >'%s'" % (commandstring, fp.name)
+ if capture_stderr:
+ cmd = "%s >'%s' 2>&1" % (commandstring, fp.name)
+ else:
+ cmd = "%s 2>/dev/null >'%s'" % (commandstring, fp.name)
return fp.read().decode('utf-8').strip() if not os.system(cmd) else None
@@ -110,6 +113,26 @@ def _get_system_version():
return _SYSTEM_VERSION
+_SYSTEM_VERSION_TUPLE = None
+def _get_system_version_tuple():
+ """
+ Return the macOS system version as a tuple
+
+ The return value is safe to use to compare
+ two version numbers.
+ """
+ global _SYSTEM_VERSION_TUPLE
+ if _SYSTEM_VERSION_TUPLE is None:
+ osx_version = _get_system_version()
+ if osx_version:
+ try:
+ _SYSTEM_VERSION_TUPLE = tuple(int(i) for i in osx_version.split('.'))
+ except ValueError:
+ _SYSTEM_VERSION_TUPLE = ()
+
+ return _SYSTEM_VERSION_TUPLE
+
+
def _remove_original_values(_config_vars):
"""Remove original unmodified values for testing"""
# This is needed for higher-level cross-platform tests of get_platform.
@@ -125,6 +148,33 @@ def _save_modified_value(_config_vars, cv, newvalue):
_config_vars[_INITPRE + cv] = oldvalue
_config_vars[cv] = newvalue
+
+_cache_default_sysroot = None
+def _default_sysroot(cc):
+ """ Returns the root of the default SDK for this system, or '/' """
+ global _cache_default_sysroot
+
+ if _cache_default_sysroot is not None:
+ return _cache_default_sysroot
+
+ contents = _read_output('%s -c -E -v - "):
+ in_incdirs = True
+ elif line.startswith("End of search list"):
+ in_incdirs = False
+ elif in_incdirs:
+ line = line.strip()
+ if line == '/usr/include':
+ _cache_default_sysroot = '/'
+ elif line.endswith(".sdk/usr/include"):
+ _cache_default_sysroot = line[:-12]
+ if _cache_default_sysroot is None:
+ _cache_default_sysroot = '/'
+
+ return _cache_default_sysroot
+
def _supports_universal_builds():
"""Returns True if universal builds are supported on this system"""
# As an approximation, we assume that if we are running on 10.4 or above,
@@ -132,14 +182,18 @@ def _supports_universal_builds():
# builds, in particular -isysroot and -arch arguments to the compiler. This
# is in support of allowing 10.4 universal builds to run on 10.3.x systems.
- osx_version = _get_system_version()
- if osx_version:
- try:
- osx_version = tuple(int(i) for i in osx_version.split('.'))
- except ValueError:
- osx_version = ''
+ osx_version = _get_system_version_tuple()
return bool(osx_version >= (10, 4)) if osx_version else False
+def _supports_arm64_builds():
+ """Returns True if arm64 builds are supported on this system"""
+ # There are two sets of systems supporting macOS/arm64 builds:
+ # 1. macOS 11 and later, unconditionally
+ # 2. macOS 10.15 with Xcode 12.2 or later
+ # For now the second category is ignored.
+ osx_version = _get_system_version_tuple()
+ return osx_version >= (11, 0) if osx_version else False
+
def _find_appropriate_compiler(_config_vars):
"""Find appropriate C compiler for extension module builds"""
@@ -331,6 +385,12 @@ def compiler_fixup(compiler_so, cc_args):
except ValueError:
break
+ elif not _supports_arm64_builds():
+ # Look for "-arch arm64" and drop that
+ for idx in reversed(range(len(compiler_so))):
+ if compiler_so[idx] == '-arch' and compiler_so[idx+1] == "arm64":
+ del compiler_so[idx:idx+2]
+
if 'ARCHFLAGS' in os.environ and not stripArch:
# User specified different -arch flags in the environ,
# see also distutils.sysconfig
@@ -481,6 +541,8 @@ def get_platform_osx(_config_vars, osname, release, machine):
if len(archs) == 1:
machine = archs[0]
+ elif archs == ('arm64', 'x86_64'):
+ machine = 'universal2'
elif archs == ('i386', 'ppc'):
machine = 'fat'
elif archs == ('i386', 'x86_64'):
diff --git a/Lib/ast.py b/Lib/ast.py
index d860917..ecd4895 100644
--- a/Lib/ast.py
+++ b/Lib/ast.py
@@ -662,17 +662,23 @@ class _Precedence(IntEnum):
except ValueError:
return self
+
+_SINGLE_QUOTES = ("'", '"')
+_MULTI_QUOTES = ('"""', "'''")
+_ALL_QUOTES = (*_SINGLE_QUOTES, *_MULTI_QUOTES)
+
class _Unparser(NodeVisitor):
"""Methods in this class recursively traverse an AST and
output source code for the abstract syntax; original formatting
is disregarded."""
- def __init__(self):
+ def __init__(self, *, _avoid_backslashes=False):
self._source = []
self._buffer = []
self._precedences = {}
self._type_ignores = {}
self._indent = 0
+ self._avoid_backslashes = _avoid_backslashes
def interleave(self, inter, f, seq):
"""Call f on each item in seq, calling inter() in between."""
@@ -1067,15 +1073,85 @@ class _Unparser(NodeVisitor):
with self.block(extra=self.get_type_comment(node)):
self.traverse(node.body)
+ def _str_literal_helper(
+ self, string, *, quote_types=_ALL_QUOTES, escape_special_whitespace=False
+ ):
+ """Helper for writing string literals, minimizing escapes.
+ Returns the tuple (string literal to write, possible quote types).
+ """
+ def escape_char(c):
+ # \n and \t are non-printable, but we only escape them if
+ # escape_special_whitespace is True
+ if not escape_special_whitespace and c in "\n\t":
+ return c
+ # Always escape backslashes and other non-printable characters
+ if c == "\\" or not c.isprintable():
+ return c.encode("unicode_escape").decode("ascii")
+ return c
+
+ escaped_string = "".join(map(escape_char, string))
+ possible_quotes = quote_types
+ if "\n" in escaped_string:
+ possible_quotes = [q for q in possible_quotes if q in _MULTI_QUOTES]
+ possible_quotes = [q for q in possible_quotes if q not in escaped_string]
+ if not possible_quotes:
+ # If there aren't any possible_quotes, fallback to using repr
+ # on the original string. Try to use a quote from quote_types,
+ # e.g., so that we use triple quotes for docstrings.
+ string = repr(string)
+ quote = next((q for q in quote_types if string[0] in q), string[0])
+ return string[1:-1], [quote]
+ if escaped_string:
+ # Sort so that we prefer '''"''' over """\""""
+ possible_quotes.sort(key=lambda q: q[0] == escaped_string[-1])
+ # If we're using triple quotes and we'd need to escape a final
+ # quote, escape it
+ if possible_quotes[0][0] == escaped_string[-1]:
+ assert len(possible_quotes[0]) == 3
+ escaped_string = escaped_string[:-1] + "\\" + escaped_string[-1]
+ return escaped_string, possible_quotes
+
+ def _write_str_avoiding_backslashes(self, string, *, quote_types=_ALL_QUOTES):
+ """Write string literal value with a best effort attempt to avoid backslashes."""
+ string, quote_types = self._str_literal_helper(string, quote_types=quote_types)
+ quote_type = quote_types[0]
+ self.write(f"{quote_type}{string}{quote_type}")
+
def visit_JoinedStr(self, node):
self.write("f")
- self._fstring_JoinedStr(node, self.buffer_writer)
- self.write(repr(self.buffer))
+ if self._avoid_backslashes:
+ self._fstring_JoinedStr(node, self.buffer_writer)
+ self._write_str_avoiding_backslashes(self.buffer)
+ return
+
+ # If we don't need to avoid backslashes globally (i.e., we only need
+ # to avoid them inside FormattedValues), it's cosmetically preferred
+ # to use escaped whitespace. That is, it's preferred to use backslashes
+ # for cases like: f"{x}\n". To accomplish this, we keep track of what
+ # in our buffer corresponds to FormattedValues and what corresponds to
+ # Constant parts of the f-string, and allow escapes accordingly.
+ buffer = []
+ for value in node.values:
+ meth = getattr(self, "_fstring_" + type(value).__name__)
+ meth(value, self.buffer_writer)
+ buffer.append((self.buffer, isinstance(value, Constant)))
+ new_buffer = []
+ quote_types = _ALL_QUOTES
+ for value, is_constant in buffer:
+ # Repeatedly narrow down the list of possible quote_types
+ value, quote_types = self._str_literal_helper(
+ value, quote_types=quote_types,
+ escape_special_whitespace=is_constant
+ )
+ new_buffer.append(value)
+ value = "".join(new_buffer)
+ quote_type = quote_types[0]
+ self.write(f"{quote_type}{value}{quote_type}")
def visit_FormattedValue(self, node):
self.write("f")
self._fstring_FormattedValue(node, self.buffer_writer)
- self.write(repr(self.buffer))
+ self._write_str_avoiding_backslashes(self.buffer)
def _fstring_JoinedStr(self, node, write):
for value in node.values:
@@ -1090,11 +1166,13 @@ class _Unparser(NodeVisitor):
def _fstring_FormattedValue(self, node, write):
write("{")
- unparser = type(self)()
+ unparser = type(self)(_avoid_backslashes=True)
unparser.set_precedence(_Precedence.TEST.next(), node.value)
expr = unparser.visit(node.value)
if expr.startswith("{"):
write(" ") # Separate pair of opening brackets as "{ {"
+ if "\\" in expr:
+ raise ValueError("Unable to avoid backslash in f-string expression part")
write(expr)
if node.conversion != -1:
conversion = chr(node.conversion)
@@ -1111,33 +1189,17 @@ class _Unparser(NodeVisitor):
self.write(node.id)
def _write_docstring(self, node):
- def esc_char(c):
- if c in ("\n", "\t"):
- # In the AST form, we don't know the author's intentation
- # about how this should be displayed. We'll only escape
- # \n and \t, because they are more likely to be unescaped
- # in the source
- return c
- return c.encode('unicode_escape').decode('ascii')
-
self.fill()
if node.kind == "u":
self.write("u")
-
- value = node.value
- if value:
- # Preserve quotes in the docstring by escaping them
- value = "".join(map(esc_char, value))
- if value[-1] == '"':
- value = value.replace('"', '\\"', -1)
- value = value.replace('"""', '""\\"')
-
- self.write(f'"""{value}"""')
+ self._write_str_avoiding_backslashes(node.value, quote_types=_MULTI_QUOTES)
def _write_constant(self, value):
if isinstance(value, (float, complex)):
# Substitute overflowing decimal literal for AST infinities.
self.write(repr(value).replace("inf", _INFSTR))
+ elif self._avoid_backslashes and isinstance(value, str):
+ self._write_str_avoiding_backslashes(value)
else:
self.write(repr(value))
diff --git a/Lib/asyncio/base_futures.py b/Lib/asyncio/base_futures.py
index 22f2980..2c01ac9 100644
--- a/Lib/asyncio/base_futures.py
+++ b/Lib/asyncio/base_futures.py
@@ -1,6 +1,7 @@
__all__ = ()
import reprlib
+from _thread import get_ident
from . import format_helpers
@@ -41,6 +42,16 @@ def _format_callbacks(cb):
return f'cb=[{cb}]'
+# bpo-42183: _repr_running is needed for repr protection
+# when a Future or Task result contains itself directly or indirectly.
+# The logic is borrowed from @reprlib.recursive_repr decorator.
+# Unfortunately, the direct decorator usage is impossible because of
+# AttributeError: '_asyncio.Task' object has no attribute '__module__' error.
+#
+# After fixing this thing we can return to the decorator based approach.
+_repr_running = set()
+
+
def _future_repr_info(future):
# (Future) -> str
"""helper function for Future.__repr__"""
@@ -49,9 +60,17 @@ def _future_repr_info(future):
if future._exception is not None:
info.append(f'exception={future._exception!r}')
else:
- # use reprlib to limit the length of the output, especially
- # for very long strings
- result = reprlib.repr(future._result)
+ key = id(future), get_ident()
+ if key in _repr_running:
+ result = '...'
+ else:
+ _repr_running.add(key)
+ try:
+ # use reprlib to limit the length of the output, especially
+ # for very long strings
+ result = reprlib.repr(future._result)
+ finally:
+ _repr_running.discard(key)
info.append(f'result={result}')
if future._callbacks:
info.append(_format_callbacks(future._callbacks))
diff --git a/Lib/asyncio/exceptions.py b/Lib/asyncio/exceptions.py
index e03602e..f07e448 100644
--- a/Lib/asyncio/exceptions.py
+++ b/Lib/asyncio/exceptions.py
@@ -34,8 +34,9 @@ class IncompleteReadError(EOFError):
- expected: total number of expected bytes (or None if unknown)
"""
def __init__(self, partial, expected):
+ r_expected = 'undefined' if expected is None else repr(expected)
super().__init__(f'{len(partial)} bytes read on a total of '
- f'{expected!r} expected bytes')
+ f'{r_expected} expected bytes')
self.partial = partial
self.expected = expected
diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py
index 8b05434..f486b67 100644
--- a/Lib/asyncio/tasks.py
+++ b/Lib/asyncio/tasks.py
@@ -373,7 +373,7 @@ ALL_COMPLETED = concurrent.futures.ALL_COMPLETED
async def wait(fs, *, loop=None, timeout=None, return_when=ALL_COMPLETED):
"""Wait for the Futures and coroutines given by fs to complete.
- The sequence futures must not be empty.
+ The fs iterable must not be empty.
Coroutines will be wrapped in Tasks.
@@ -400,13 +400,15 @@ async def wait(fs, *, loop=None, timeout=None, return_when=ALL_COMPLETED):
"and scheduled for removal in Python 3.10.",
DeprecationWarning, stacklevel=2)
- if any(coroutines.iscoroutine(f) for f in set(fs)):
+ fs = set(fs)
+
+ if any(coroutines.iscoroutine(f) for f in fs):
warnings.warn("The explicit passing of coroutine objects to "
"asyncio.wait() is deprecated since Python 3.8, and "
"scheduled for removal in Python 3.11.",
DeprecationWarning, stacklevel=2)
- fs = {ensure_future(f, loop=loop) for f in set(fs)}
+ fs = {ensure_future(f, loop=loop) for f in fs}
return await _wait(fs, timeout, return_when, loop)
@@ -573,7 +575,7 @@ def as_completed(fs, *, loop=None, timeout=None):
Note: The futures 'f' are not necessarily members of fs.
"""
if futures.isfuture(fs) or coroutines.iscoroutine(fs):
- raise TypeError(f"expect a list of futures, not {type(fs).__name__}")
+ raise TypeError(f"expect an iterable of futures, not {type(fs).__name__}")
from .queues import Queue # Import here to avoid circular import problem.
done = Queue(loop=loop)
diff --git a/Lib/binhex.py b/Lib/binhex.py
index 9559f46..ace5217 100644
--- a/Lib/binhex.py
+++ b/Lib/binhex.py
@@ -117,12 +117,12 @@ class _Hqxcoderengine:
first = 0
while first <= len(self.hqxdata) - self.linelen:
last = first + self.linelen
- self.ofp.write(self.hqxdata[first:last] + b'\n')
+ self.ofp.write(self.hqxdata[first:last] + b'\r')
self.linelen = LINELEN
first = last
self.hqxdata = self.hqxdata[first:]
if force:
- self.ofp.write(self.hqxdata + b':\n')
+ self.ofp.write(self.hqxdata + b':\r')
def close(self):
if self.data:
diff --git a/Lib/cProfile.py b/Lib/cProfile.py
index 4f20203..59b4699 100755
--- a/Lib/cProfile.py
+++ b/Lib/cProfile.py
@@ -152,6 +152,11 @@ def main():
(options, args) = parser.parse_args()
sys.argv[:] = args
+ # The script that we're profiling may chdir, so capture the absolute path
+ # to the output file at startup.
+ if options.outfile is not None:
+ options.outfile = os.path.abspath(options.outfile)
+
if len(args) > 0:
if options.module:
code = "run_module(modname, run_name='__main__')"
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index efd654e..bc69a67 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -949,7 +949,7 @@ class ChainMap(_collections_abc.MutableMapping):
def __iter__(self):
d = {}
for mapping in reversed(self.maps):
- d.update(mapping) # reuses stored hash values if possible
+ d.update(dict.fromkeys(mapping)) # reuses stored hash values if possible
return iter(d)
def __contains__(self, key):
diff --git a/Lib/copyreg.py b/Lib/copyreg.py
index dfc463c..7ab8c12 100644
--- a/Lib/copyreg.py
+++ b/Lib/copyreg.py
@@ -48,6 +48,7 @@ def _reconstructor(cls, base, state):
return obj
_HEAPTYPE = 1<<9
+_new_type = type(int.__new__)
# Python code for object.__reduce_ex__ for protocols 0 and 1
@@ -57,6 +58,9 @@ def _reduce_ex(self, proto):
for base in cls.__mro__:
if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE:
break
+ new = base.__new__
+ if isinstance(new, _new_type) and new.__self__ is base:
+ break
else:
base = object # not really reachable
if base is object:
diff --git a/Lib/ctypes/macholib/dyld.py b/Lib/ctypes/macholib/dyld.py
index 9d86b05..1c3f8fd 100644
--- a/Lib/ctypes/macholib/dyld.py
+++ b/Lib/ctypes/macholib/dyld.py
@@ -6,6 +6,11 @@ import os
from ctypes.macholib.framework import framework_info
from ctypes.macholib.dylib import dylib_info
from itertools import *
+try:
+ from _ctypes import _dyld_shared_cache_contains_path
+except ImportError:
+ def _dyld_shared_cache_contains_path(*args):
+ raise NotImplementedError
__all__ = [
'dyld_find', 'framework_find',
@@ -122,8 +127,15 @@ def dyld_find(name, executable_path=None, env=None):
dyld_executable_path_search(name, executable_path),
dyld_default_search(name, env),
), env):
+
if os.path.isfile(path):
return path
+ try:
+ if _dyld_shared_cache_contains_path(path):
+ return path
+ except NotImplementedError:
+ pass
+
raise ValueError("dylib %s could not be found" % (name,))
def framework_find(fn, executable_path=None, env=None):
diff --git a/Lib/ctypes/test/test_find.py b/Lib/ctypes/test/test_find.py
index b99fdcb..92ac184 100644
--- a/Lib/ctypes/test/test_find.py
+++ b/Lib/ctypes/test/test_find.py
@@ -1,4 +1,5 @@
import unittest
+import unittest.mock
import os.path
import sys
import test.support
@@ -72,7 +73,7 @@ class Test_OpenGL_libs(unittest.TestCase):
@unittest.skipUnless(sys.platform.startswith('linux'),
'Test only valid for Linux')
-class LibPathFindTest(unittest.TestCase):
+class FindLibraryLinux(unittest.TestCase):
def test_find_on_libpath(self):
import subprocess
import tempfile
@@ -111,6 +112,15 @@ class LibPathFindTest(unittest.TestCase):
# LD_LIBRARY_PATH)
self.assertEqual(find_library(libname), 'lib%s.so' % libname)
+ def test_find_library_with_gcc(self):
+ with unittest.mock.patch("ctypes.util._findSoname_ldconfig", lambda *args: None):
+ self.assertNotEqual(find_library('c'), None)
+
+ def test_find_library_with_ld(self):
+ with unittest.mock.patch("ctypes.util._findSoname_ldconfig", lambda *args: None), \
+ unittest.mock.patch("ctypes.util._findLib_gcc", lambda *args: None):
+ self.assertNotEqual(find_library('c'), None)
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/ctypes/test/test_macholib.py b/Lib/ctypes/test/test_macholib.py
index 6b35269..a1bac26 100644
--- a/Lib/ctypes/test/test_macholib.py
+++ b/Lib/ctypes/test/test_macholib.py
@@ -45,19 +45,22 @@ def find_lib(name):
class MachOTest(unittest.TestCase):
@unittest.skipUnless(sys.platform == "darwin", 'OSX-specific test')
def test_find(self):
-
- self.assertEqual(find_lib('pthread'),
- '/usr/lib/libSystem.B.dylib')
+ # On Mac OS 11, system dylibs are only present in the shared cache,
+ # so symlinks like libpthread.dylib -> libSystem.B.dylib will not
+ # be resolved by dyld_find
+ self.assertIn(find_lib('pthread'),
+ ('/usr/lib/libSystem.B.dylib', '/usr/lib/libpthread.dylib'))
result = find_lib('z')
# Issue #21093: dyld default search path includes $HOME/lib and
# /usr/local/lib before /usr/lib, which caused test failures if
# a local copy of libz exists in one of them. Now ignore the head
# of the path.
- self.assertRegex(result, r".*/lib/libz\..*.*\.dylib")
+ self.assertRegex(result, r".*/lib/libz.*\.dylib")
- self.assertEqual(find_lib('IOKit'),
- '/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit')
+ self.assertIn(find_lib('IOKit'),
+ ('/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit',
+ '/System/Library/Frameworks/IOKit.framework/IOKit'))
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/ctypes/test/test_wintypes.py b/Lib/ctypes/test/test_wintypes.py
index 71442df..243d596 100644
--- a/Lib/ctypes/test/test_wintypes.py
+++ b/Lib/ctypes/test/test_wintypes.py
@@ -1,12 +1,13 @@
-import sys
import unittest
+# also work on POSIX
+
from ctypes import *
+from ctypes import wintypes
+
-@unittest.skipUnless(sys.platform.startswith('win'), 'Windows-only test')
class WinTypesTest(unittest.TestCase):
def test_variant_bool(self):
- from ctypes import wintypes
# reads 16-bits from memory, anything non-zero is True
for true_value in (1, 32767, 32768, 65535, 65537):
true = POINTER(c_int16)(c_int16(true_value))
@@ -37,5 +38,6 @@ class WinTypesTest(unittest.TestCase):
vb.value = []
self.assertIs(vb.value, False)
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/ctypes/util.py b/Lib/ctypes/util.py
index 01176bf..0c2510e 100644
--- a/Lib/ctypes/util.py
+++ b/Lib/ctypes/util.py
@@ -93,6 +93,12 @@ elif os.name == "posix":
# Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
import re, tempfile
+ def _is_elf(filename):
+ "Return True if the given file is an ELF file"
+ elf_header = b'\x7fELF'
+ with open(filename, 'br') as thefile:
+ return thefile.read(4) == elf_header
+
def _findLib_gcc(name):
# Run GCC's linker with the -t (aka --trace) option and examine the
# library name it prints out. The GCC command will fail because we
@@ -130,10 +136,17 @@ elif os.name == "posix":
# Raised if the file was already removed, which is the normal
# behaviour of GCC if linking fails
pass
- res = re.search(expr, trace)
+ res = re.findall(expr, trace)
if not res:
return None
- return os.fsdecode(res.group(0))
+
+ for file in res:
+ # Check if the given file is an elf file: gcc can report
+ # some files that are linker scripts and not actual
+ # shared objects. See bpo-41976 for more details
+ if not _is_elf(file):
+ continue
+ return os.fsdecode(file)
if sys.platform == "sunos5":
@@ -299,9 +312,14 @@ elif os.name == "posix":
stderr=subprocess.PIPE,
universal_newlines=True)
out, _ = p.communicate()
- res = re.search(expr, os.fsdecode(out))
- if res:
- result = res.group(0)
+ res = re.findall(expr, os.fsdecode(out))
+ for file in res:
+ # Check if the given file is an elf file: gcc can report
+ # some files that are linker scripts and not actual
+ # shared objects. See bpo-41976 for more details
+ if not _is_elf(file):
+ continue
+ return os.fsdecode(file)
except Exception:
pass # result will be None
return result
@@ -309,7 +327,7 @@ elif os.name == "posix":
def find_library(name):
# See issue #9998
return _findSoname_ldconfig(name) or \
- _get_soname(_findLib_gcc(name) or _findLib_ld(name))
+ _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
################################################################
# test code
diff --git a/Lib/datetime.py b/Lib/datetime.py
index 2294ac2..e508d99 100644
--- a/Lib/datetime.py
+++ b/Lib/datetime.py
@@ -1579,7 +1579,7 @@ class time:
self._tzinfo = tzinfo
def __reduce_ex__(self, protocol):
- return (time, self._getstate(protocol))
+ return (self.__class__, self._getstate(protocol))
def __reduce__(self):
return self.__reduce_ex__(2)
diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py
index 0d1bd03..f50edd2 100644
--- a/Lib/distutils/spawn.py
+++ b/Lib/distutils/spawn.py
@@ -54,8 +54,8 @@ def spawn(cmd, search_path=1, verbose=0, dry_run=0):
global _cfg_target, _cfg_target_split
if _cfg_target is None:
from distutils import sysconfig
- _cfg_target = sysconfig.get_config_var(
- 'MACOSX_DEPLOYMENT_TARGET') or ''
+ _cfg_target = str(sysconfig.get_config_var(
+ 'MACOSX_DEPLOYMENT_TARGET') or '')
if _cfg_target:
_cfg_target_split = [int(x) for x in _cfg_target.split('.')]
if _cfg_target:
diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py
index 5e47e07..1b034c9 100644
--- a/Lib/distutils/tests/test_build_ext.py
+++ b/Lib/distutils/tests/test_build_ext.py
@@ -455,7 +455,7 @@ class BuildExtTestCase(TempdirManager,
deptarget = sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET')
if deptarget:
# increment the minor version number (i.e. 10.6 -> 10.7)
- deptarget = [int(x) for x in deptarget.split('.')]
+ deptarget = [int(x) for x in str(deptarget).split('.')]
deptarget[-1] += 1
deptarget = '.'.join(str(i) for i in deptarget)
self._try_compile_deployment_target('<', deptarget)
@@ -488,16 +488,20 @@ class BuildExtTestCase(TempdirManager,
# get the deployment target that the interpreter was built with
target = sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET')
- target = tuple(map(int, target.split('.')[0:2]))
+ target = tuple(map(int, str(target).split('.')[0:2]))
# format the target value as defined in the Apple
# Availability Macros. We can't use the macro names since
# at least one value we test with will not exist yet.
- if target[1] < 10:
+ if target[:2] < (10, 10):
# for 10.1 through 10.9.x -> "10n0"
target = '%02d%01d0' % target
else:
# for 10.10 and beyond -> "10nn00"
- target = '%02d%02d00' % target
+ if len(target) >= 2:
+ target = '%02d%02d00' % target
+ else:
+ # 11 and later can have no minor version (11 instead of 11.0)
+ target = '%02d0000' % target
deptarget_ext = Extension(
'deptarget',
[deptarget_c],
diff --git a/Lib/distutils/unixccompiler.py b/Lib/distutils/unixccompiler.py
index 4d7a6de..f0792de 100644
--- a/Lib/distutils/unixccompiler.py
+++ b/Lib/distutils/unixccompiler.py
@@ -290,7 +290,7 @@ class UnixCCompiler(CCompiler):
cflags = sysconfig.get_config_var('CFLAGS')
m = re.search(r'-isysroot\s*(\S+)', cflags)
if m is None:
- sysroot = '/'
+ sysroot = _osx_support._default_sysroot(sysconfig.get_config_var('CC'))
else:
sysroot = m.group(1)
diff --git a/Lib/email/generator.py b/Lib/email/generator.py
index ae670c2..c9b1216 100644
--- a/Lib/email/generator.py
+++ b/Lib/email/generator.py
@@ -186,7 +186,11 @@ class Generator:
# If we munged the cte, copy the message again and re-fix the CTE.
if munge_cte:
msg = deepcopy(msg)
- msg.replace_header('content-transfer-encoding', munge_cte[0])
+ # Preserve the header order if the CTE header already exists.
+ if msg.get('content-transfer-encoding') is None:
+ msg['Content-Transfer-Encoding'] = munge_cte[0]
+ else:
+ msg.replace_header('content-transfer-encoding', munge_cte[0])
msg.replace_header('content-type', munge_cte[1])
# Write the headers. First we see if the message object wants to
# handle that itself. If not, we'll do it generically.
diff --git a/Lib/ensurepip/_bundled/pip-20.2.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-20.2.1-py2.py3-none-any.whl
deleted file mode 100644
index 3d0d3f8..0000000
Binary files a/Lib/ensurepip/_bundled/pip-20.2.1-py2.py3-none-any.whl and /dev/null differ
diff --git a/Lib/graphlib.py b/Lib/graphlib.py
index 948f62f..d0e7a48 100644
--- a/Lib/graphlib.py
+++ b/Lib/graphlib.py
@@ -22,7 +22,8 @@ class _NodeInfo:
class CycleError(ValueError):
- """Subclass of ValueError raised by TopologicalSorterif cycles exist in the graph
+ """Subclass of ValueError raised by TopologicalSorter.prepare if cycles
+ exist in the working graph.
If multiple cycles exist, only one undefined choice among them will be reported
and included in the exception. The detected cycle can be accessed via the second
@@ -129,7 +130,7 @@ class TopologicalSorter:
return result
def is_active(self):
- """Return True if more progress can be made and ``False`` otherwise.
+ """Return ``True`` if more progress can be made and ``False`` otherwise.
Progress can be made if cycles do not block the resolution and either there
are still nodes ready that haven't yet been returned by "get_ready" or the
@@ -149,7 +150,7 @@ class TopologicalSorter:
"""Marks a set of nodes returned by "get_ready" as processed.
This method unblocks any successor of each node in *nodes* for being returned
- in the future by a a call to "get_ready"
+ in the future by a call to "get_ready".
Raises :exec:`ValueError` if any node in *nodes* has already been marked as
processed by a previous call to this method, if a node was not added to the
diff --git a/Lib/http/server.py b/Lib/http/server.py
index fa204fb..def05f4 100644
--- a/Lib/http/server.py
+++ b/Lib/http/server.py
@@ -1123,12 +1123,7 @@ class CGIHTTPRequestHandler(SimpleHTTPRequestHandler):
referer = self.headers.get('referer')
if referer:
env['HTTP_REFERER'] = referer
- accept = []
- for line in self.headers.getallmatchingheaders('accept'):
- if line[:1] in "\t\n\r ":
- accept.append(line.strip())
- else:
- accept = accept + line[7:].split(',')
+ accept = self.headers.get_all('accept', ())
env['HTTP_ACCEPT'] = ','.join(accept)
ua = self.headers.get('user-agent')
if ua:
diff --git a/Lib/idlelib/Icons/README.txt b/Lib/idlelib/Icons/README.txt
index 8b47162..d91c4d5 100644
--- a/Lib/idlelib/Icons/README.txt
+++ b/Lib/idlelib/Icons/README.txt
@@ -7,3 +7,7 @@ https://www.doxdesk.com/software/py/pyicons.html
Various different formats and sizes are available at this GitHub Pull Request:
https://github.com/python/cpython/pull/17473
+
+The idle.ico file was created with ImageMagick:
+
+ $ convert idle_16.png idle_32.png idle_48.png idle_256.png idle.ico
diff --git a/Lib/idlelib/Icons/idle.ico b/Lib/idlelib/Icons/idle.ico
index 3357aef..2aa9a83 100644
Binary files a/Lib/idlelib/Icons/idle.ico and b/Lib/idlelib/Icons/idle.ico differ
diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index fd76207..869be0a 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -1,8 +1,32 @@
+What's New in IDLE 3.9.1
+Released on 2020-12-07?
+======================================
+
+
+bpo-42508: Keep IDLE running on macOS. Remove obsolete workaround
+that prevented running files with shortcuts when using new universal2
+installers built on macOS 11.
+
+bpo-42426: Fix reporting offset of the RE error in searchengine.
+
+bpo-42416: Get docstrings for IDLE calltips more often
+by using inspect.getdoc.
+
+bpo-33987: Mostly finish using ttk widgets, mainly for editor,
+settings, and searches. Some patches by Mark Roseman.
+
+bpo-41775: Make 'IDLE Shell' the shell title.
+
+bpo-35764: Rewrite the Calltips doc section.
+
+bpo-40181: In calltips, stop reminding that '/' marks the end of
+positional-only arguments.
+
+
What's New in IDLE 3.9.0 (since 3.8.0)
Released on 2020-10-05?
======================================
-
bpo-41468: Improve IDLE run crash error message (which users should
never see).
diff --git a/Lib/idlelib/calltip.py b/Lib/idlelib/calltip.py
index d4092c7..40bc5a0 100644
--- a/Lib/idlelib/calltip.py
+++ b/Lib/idlelib/calltip.py
@@ -55,18 +55,50 @@ class Calltip:
self.open_calltip(False)
def open_calltip(self, evalfuncs):
- self.remove_calltip_window()
+ """Maybe close an existing calltip and maybe open a new calltip.
+ Called from (force_open|try_open|refresh)_calltip_event functions.
+ """
hp = HyperParser(self.editwin, "insert")
sur_paren = hp.get_surrounding_brackets('(')
+
+ # If not inside parentheses, no calltip.
if not sur_paren:
+ self.remove_calltip_window()
return
+
+ # If a calltip is shown for the current parentheses, do
+ # nothing.
+ if self.active_calltip:
+ opener_line, opener_col = map(int, sur_paren[0].split('.'))
+ if (
+ (opener_line, opener_col) ==
+ (self.active_calltip.parenline, self.active_calltip.parencol)
+ ):
+ return
+
hp.set_index(sur_paren[0])
- expression = hp.get_expression()
+ try:
+ expression = hp.get_expression()
+ except ValueError:
+ expression = None
if not expression:
+ # No expression before the opening parenthesis, e.g.
+ # because it's in a string or the opener for a tuple:
+ # Do nothing.
return
+
+ # At this point, the current index is after an opening
+ # parenthesis, in a section of code, preceded by a valid
+ # expression. If there is a calltip shown, it's not for the
+ # same index and should be closed.
+ self.remove_calltip_window()
+
+ # Simple, fast heuristic: If the preceding expression includes
+ # an opening parenthesis, it likely includes a function call.
if not evalfuncs and (expression.find('(') != -1):
return
+
argspec = self.fetch_tip(expression)
if not argspec:
return
@@ -118,7 +150,6 @@ _INDENT = ' '*4 # for wrapped signatures
_first_param = re.compile(r'(?<=\()\w*\,?\s*')
_default_callable_argspec = "See source or doc"
_invalid_method = "invalid method signature"
-_argument_positional = " # '/' marks preceding args as positional-only."
def get_argspec(ob):
'''Return a string describing the signature of a callable object, or ''.
@@ -134,6 +165,7 @@ def get_argspec(ob):
ob_call = ob.__call__
except BaseException: # Buggy user object could raise anything.
return '' # No popup for non-callables.
+ # For Get_argspecTest.test_buggy_getattr_class, CallA() & CallB().
fob = ob_call if isinstance(ob_call, types.MethodType) else ob
# Initialize argspec and wrap it to get lines.
@@ -146,9 +178,6 @@ def get_argspec(ob):
else:
argspec = ''
- if '/' in argspec and len(argspec) < _MAX_COLS - len(_argument_positional):
- # Add explanation TODO remove after 3.7, before 3.9.
- argspec += _argument_positional
if isinstance(fob, type) and argspec == '()':
# If fob has no argument, use default callable argspec.
argspec = _default_callable_argspec
@@ -157,10 +186,7 @@ def get_argspec(ob):
if len(argspec) > _MAX_COLS else [argspec] if argspec else [])
# Augment lines from docstring, if any, and join to get argspec.
- if isinstance(ob_call, types.MethodType):
- doc = ob_call.__doc__
- else:
- doc = getattr(ob, "__doc__", "")
+ doc = inspect.getdoc(ob)
if doc:
for line in doc.split('\n', _MAX_LINES)[:_MAX_LINES]:
line = line.strip()
diff --git a/Lib/idlelib/configdialog.py b/Lib/idlelib/configdialog.py
index 8259649..a84e1c5 100644
--- a/Lib/idlelib/configdialog.py
+++ b/Lib/idlelib/configdialog.py
@@ -67,7 +67,6 @@ class ConfigDialog(Toplevel):
if not _utest:
self.withdraw()
- self.configure(borderwidth=5)
self.title(title or 'IDLE Preferences')
x = parent.winfo_rootx() + 20
y = parent.winfo_rooty() + (30 if not _htest else 150)
@@ -97,6 +96,7 @@ class ConfigDialog(Toplevel):
"""Create and place widgets for tabbed dialog.
Widgets Bound to self:
+ frame: encloses all other widgets
note: Notebook
highpage: HighPage
fontpage: FontPage
@@ -109,7 +109,9 @@ class ConfigDialog(Toplevel):
load_configs: Load pages except for extensions.
activate_config_changes: Tell editors to reload.
"""
- self.note = note = Notebook(self)
+ self.frame = frame = Frame(self, padding="5px")
+ self.frame.grid(sticky="nwes")
+ self.note = note = Notebook(frame)
self.highpage = HighPage(note)
self.fontpage = FontPage(note, self.highpage)
self.keyspage = KeysPage(note)
@@ -148,7 +150,7 @@ class ConfigDialog(Toplevel):
padding_args = {}
else:
padding_args = {'padding': (6, 3)}
- outer = Frame(self, padding=2)
+ outer = Frame(self.frame, padding=2)
buttons_frame = Frame(outer, padding=2)
self.buttons = {}
for txt, cmd in (
@@ -687,7 +689,7 @@ class HighPage(Frame):
def __init__(self, master):
super().__init__(master)
- self.cd = master.master
+ self.cd = master.winfo_toplevel()
self.style = Style(master)
self.create_page_highlight()
self.load_theme_cfg()
@@ -1346,7 +1348,7 @@ class KeysPage(Frame):
def __init__(self, master):
super().__init__(master)
- self.cd = master.master
+ self.cd = master.winfo_toplevel()
self.create_page_keys()
self.load_key_cfg()
diff --git a/Lib/idlelib/help.html b/Lib/idlelib/help.html
index b2853cf..170999e 100644
--- a/Lib/idlelib/help.html
+++ b/Lib/idlelib/help.html
@@ -1,23 +1,24 @@
-
+
- IDLE — Python 3.10.0a0 documentation
+
+ IDLE — Python 3.10.0a1 documentation
-
-
-
-
-
+
+
+
+
+
-
+
@@ -71,11 +72,12 @@
@@ -426,30 +428,30 @@ and that other files do not. Run Python code with the Run menu.
the Command key on macOS.
Backspace deletes to the left; Del deletes to the right
-
C-Backspace delete word left; C-Del delete word to the right
-
Arrow keys and Page Up/Page Down to move around
-
C-LeftArrow and C-RightArrow moves by words
+
C-Backspace delete word left; C-Del delete word to the right
+
Arrow keys and PageUp/PageDown to move around
+
C-LeftArrow and C-RightArrow moves by words
Home/End go to begin/end of line
-
C-Home/C-End go to begin/end of file
+
C-Home/C-End go to begin/end of file
Some useful Emacs bindings are inherited from Tcl/Tk:
-
C-a beginning of line
-
C-e end of line
-
C-k kill line (but doesnât put it in clipboard)
-
C-l center window around the insertion point
-
C-b go backward one character without deleting (usually you can
+
C-a beginning of line
+
C-e end of line
+
C-k kill line (but doesnât put it in clipboard)
+
C-l center window around the insertion point
+
C-b go backward one character without deleting (usually you can
also use the cursor key for this)
-
C-f go forward one character without deleting (usually you can
+
C-f go forward one character without deleting (usually you can
also use the cursor key for this)
-
C-p go up one line (usually you can also use the cursor key for
+
C-p go up one line (usually you can also use the cursor key for
this)
-
C-d delete next character
+
C-d delete next character
-
Standard keybindings (like C-c to copy and C-v to paste)
+
Standard keybindings (like C-c to copy and C-v to paste)
may work. Keybindings are selected in the Configure IDLE dialog.
@@ -486,7 +488,7 @@ one can specify a drive first.) Move into subdirectories by typing a
directory name and a separator.
Instead of waiting, or after a box is closed, open a completion box
immediately with Show Completions on the Edit menu. The default hot
-key is C-space. If one types a prefix for the desired name
+key is C-space. If one types a prefix for the desired name
before opening the box, the first match or near miss is made visible.
The result is the same as if one enters a prefix
after the box is displayed. Show Completions after a quote completes
@@ -509,26 +511,29 @@ by typing â_â after â.â, either before or after the box is opened.
A calltip is shown when one types ( after the name of an accessible
-function. A name expression may include dots and subscripts. A calltip
-remains until it is clicked, the cursor is moved out of the argument area,
-or ) is typed. When the cursor is in the argument part of a definition,
-the menu or shortcut display a calltip.
-
A calltip consists of the function signature and the first line of the
-docstring. For builtins without an accessible signature, the calltip
-consists of all lines up the fifth line or the first blank line. These
-details may change.
-
The set of accessible functions depends on what modules have been imported
-into the user process, including those imported by Idle itself,
-and what definitions have been run, all since the last restart.
+
A calltip is shown automatically when one types ( after the name
+of an accessible function. A function name expression may include
+dots and subscripts. A calltip remains until it is clicked, the cursor
+is moved out of the argument area, or ) is typed. Whenever the
+cursor is in the argument part of a definition, select Edit and âShow
+Call Tipâ on the menu or enter its shortcut to display a calltip.
+
The calltip consists of the functionâs signature and docstring up to
+the latterâs first blank line or the fifth non-blank line. (Some builtin
+functions lack an accessible signature.) A â/â or â*â in the signature
+indicates that the preceding or following arguments are passed by
+position or name (keyword) only. Details are subject to change.
+
In Shell, the accessible functions depends on what modules have been
+imported into the user process, including those imported by Idle itself,
+and which definitions have been run, all since the last restart.
For example, restart the Shell and enter itertools.count(. A calltip
-appears because Idle imports itertools into the user process for its own use.
-(This could change.) Enter turtle.write( and nothing appears. Idle does
-not import turtle. The menu or shortcut do nothing either. Enter
-importturtle and then turtle.write( will work.
-
In an editor, import statements have no effect until one runs the file. One
-might want to run a file after writing the import statements at the top,
-or immediately run an existing file before editing.
+appears because Idle imports itertools into the user process for its own
+use. (This could change.) Enter turtle.write( and nothing appears.
+Idle does not itself import turtle. The menu entry and shortcut also do
+nothing. Enter importturtle. Thereafter, turtle.write(
+will display a calltip.
+
In an editor, import statements have no effect until one runs the file.
+One might want to run a file after writing import statements, after
+adding function definitions, or after opening an existing file.
@@ -556,14 +561,14 @@ If one pastes more that one statement into Shell, the result will be a
The editing features described in previous subsections work when entering
code interactively. IDLEâs Shell window also responds to the following keys.
-
C-c interrupts executing command
-
C-d sends end-of-file; closes window if typed at a >>> prompt
-
Alt-/ (Expand word) is also useful to reduce typing
+
C-c interrupts executing command
+
C-d sends end-of-file; closes window if typed at a >>> prompt
+
Alt-/ (Expand word) is also useful to reduce typing
Command history
-
Alt-p retrieves previous command matching what you have typed. On
-macOS use C-p.
-
Alt-n retrieves next. On macOS use C-n.
+
Alt-p retrieves previous command matching what you have typed. On
+macOS use C-p.
+
Alt-n retrieves next. On macOS use C-n.
Return while on any previous command retrieves that command