-

25.5. IDLE¶

+

26.5. IDLE¶

Source code: Lib/idlelib/

-

IDLE is Python’s Integrated Development and Learning Environment.

+

IDLE is Python’s Integrated Development and Learning Environment.

IDLE has the following features:

    -
  • coded in 100% pure Python, using the tkinter GUI toolkit
    • +
  • coded in 100% pure Python, using the tkinter GUI toolkit
  • cross-platform: works mostly the same on Windows, Unix, and Mac OS X
  • Python shell window (interactive interpreter) with colorizing of code input, output, and error messages
    • @@ -111,24 +107,24 @@ of global and local namespaces
  • configuration, browsers, and other dialogs
-

25.5.2. Editing and navigation¶

-

In this section, ‘C’ refers to the Control key on Windows and Unix and -the Command key on Mac OSX.

+

26.5.2. Editing and navigation¶

+

In this section, ‘C’ refers to the Control key on Windows and Unix and +the Command key on Mac OSX.

    -

  • Backspace deletes to the left; Del deletes to the right

    +

  • Backspace deletes to the left; Del deletes to the right

    • -

  • C-Backspace delete word left; C-Del delete word to the right

    +

  • C-Backspace delete word left; C-Del delete word to the right

    • -

  • Arrow keys and Page Up/Page Down to move around

    +

  • Arrow keys and Page Up/Page Down to move around

    • -

  • C-LeftArrow and C-RightArrow moves by words

    +

  • C-LeftArrow and C-RightArrow moves by words

    • -

  • Home/End go to begin/end of line

    +

  • Home/End go to begin/end of line

    • -

  • C-Home/C-End go to begin/end of file

    +

  • C-Home/C-End go to begin/end of file

  • Some useful Emacs bindings are inherited from Tcl/Tk:

      -
    • C-a beginning of line
      • -
    • C-e end of line
      • -
    • C-k kill line (but doesn’t put it in clipboard)
      • -
    • C-l center window around the insertion point
      • -
    • C-b go backward one character without deleting (usually you can +
    • C-a beginning of line
      • +
    • C-e end of line
      • +
    • C-k kill line (but doesn’t put it in clipboard)
      • +
    • C-l center window around the insertion point
      • +
    • C-b go backward one character without deleting (usually you can also use the cursor key for this)
      • -
    • C-f go forward one character without deleting (usually you can +
    • C-f go forward one character without deleting (usually you can also use the cursor key for this)
      • -
    • C-p go up one line (usually you can also use the cursor key for +
    • C-p go up one line (usually you can also use the cursor key for this)
      • -
    • C-d delete next character
      • +
    • C-d delete next character
-

Standard keybindings (like C-c to copy and C-v to paste) +

Standard keybindings (like C-c to copy and C-v to paste) may work. Keybindings are selected in the Configure IDLE dialog.

-

25.5.2.1. Automatic indentation¶

+

26.5.2.1. Automatic indentation¶

After a block-opening statement, the next line is indented by 4 spaces (in the Python Shell window by one tab). After certain keywords (break, return etc.) -the next line is dedented. In leading indentation, Backspace deletes up -to 4 spaces if they are there. Tab inserts spaces (in the Python +the next line is dedented. In leading indentation, Backspace deletes up +to 4 spaces if they are there. Tab inserts spaces (in the Python Shell window one tab), number depends on Indent width. Currently, tabs are restricted to four spaces due to Tcl/Tk limitations.

See also the indent/dedent region commands in the edit menu.

-

25.5.2.2. Completions¶

+

26.5.2.2. Completions¶

Completions are supplied for functions, classes, and attributes of classes, both built-in and user-defined. Completions are also provided for filenames.

The AutoCompleteWindow (ACW) will open after a predefined delay (default is -two seconds) after a ‘.’ or (in a string) an os.sep is typed. If after one +two seconds) after a ‘.’ or (in a string) an os.sep is typed. If after one of those characters (plus zero or more other characters) a tab is typed the ACW will open immediately if a possible continuation is found.

If there is only one possible completion for the characters entered, a -Tab will supply that completion without opening the ACW.

-

‘Show Completions’ will force open a completions window, by default the -C-space will open a completions window. In an empty +Tab will supply that completion without opening the ACW.

+

‘Show Completions’ will force open a completions window, by default the +C-space will open a completions window. In an empty string, this will contain the files in the current directory. On a blank line, it will contain the built-in and user-defined functions and classes in the current namespaces, plus any modules imported. If some characters have been entered, the ACW will attempt to be more specific.

If a string of characters is typed, the ACW selection will jump to the -entry most closely matching those characters. Entering a tab will +entry most closely matching those characters. Entering a tab will cause the longest non-ambiguous match to be entered in the Editor window or -Shell. Two tab in a row will supply the current ACW selection, as +Shell. Two tab in a row will supply the current ACW selection, as will return or a double click. Cursor keys, Page Up/Down, mouse selection, and the scroll wheel all operate on the ACW.

-

“Hidden” attributes can be accessed by typing the beginning of hidden -name after a ‘.’, e.g. ‘_’. This allows access to modules with -__all__ set, or to class-private attributes.

-

Completions and the ‘Expand Word’ facility can save a lot of typing!

+

“Hidden” attributes can be accessed by typing the beginning of hidden +name after a ‘.’, e.g. ‘_’. This allows access to modules with +__all__ set, or to class-private attributes.

+

Completions and the ‘Expand Word’ facility can save a lot of typing!

Completions are currently limited to those in the namespaces. Names in -an Editor window which are not via __main__ and sys.modules will +an Editor window which are not via __main__ and sys.modules will not be found. Run the module once with your imports to correct this situation. Note that IDLE itself places quite a few modules in sys.modules, so much can be found by default, e.g. the re module.

-

If you don’t like the ACW popping up unbidden, simply make the delay +

If you don’t like the ACW popping up unbidden, simply make the delay longer or disable the extension.

-

25.5.2.3. Calltips¶

-

A calltip is shown when one types ( after the name of an accessible +

26.5.2.3. Calltips¶

+

A calltip is shown when one types ( after the name of an accessible function. A name expression may include dots and subscripts. A calltip remains until it is clicked, the cursor is moved out of the argument area, -or ) is typed. When the cursor is in the argument part of a definition, +or ) is typed. When the cursor is in the argument part of a definition, the menu or shortcut display a calltip.

A calltip consists of the function signature and the first line of the docstring. For builtins without an accessible signature, the calltip @@ -457,40 +454,40 @@ details may change.

The set of accessible functions depends on what modules have been imported into the user process, including those imported by Idle itself, and what definitions have been run, all since the last restart.

-

For example, restart the Shell and enter itertools.count(. A calltip +

For example, restart the Shell and enter itertools.count(. A calltip appears because Idle imports itertools into the user process for its own use. -(This could change.) Enter turtle.write( and nothing appears. Idle does +(This could change.) Enter turtle.write( and nothing appears. Idle does not import turtle. The menu or shortcut do nothing either. Enter -import turtle and then turtle.write( will work.

+import turtle and then turtle.write( will work.

In an editor, import statements have no effect until one runs the file. One might want to run a file after writing the import statements at the top, or immediately run an existing file before editing.

-

25.5.2.4. Python Shell window¶

+

26.5.2.4. Python Shell window¶

    -

  • C-c interrupts executing command

    +

  • C-c interrupts executing command

    • -

  • C-d sends end-of-file; closes window if typed at a >>> prompt

    +

  • C-d sends end-of-file; closes window if typed at a >>> prompt

    • -

  • Alt-/ (Expand word) is also useful to reduce typing

    +

  • Alt-/ (Expand word) is also useful to reduce typing

    Command history

      -
    • Alt-p retrieves previous command matching what you have typed. On -OS X use C-p.
      • -
    • Alt-n retrieves next. On OS X use C-n.
      • -
    • Return while on any previous command retrieves that command
      • +
    • Alt-p retrieves previous command matching what you have typed. On +OS X use C-p.
      • +
    • Alt-n retrieves next. On OS X use C-n.
      • +
    • Return while on any previous command retrieves that command
-

25.5.2.5. Text colors¶

+

26.5.2.5. Text colors¶

Idle defaults to black on white text, but colors text with special meanings. For the shell, these are shell output, shell error, user output, and user error. For Python code, at the shell prompt or in an editor, these are -keywords, builtin class and function names, names following class and -def, strings, and comments. For any text window, these are the cursor (when +keywords, builtin class and function names, names following class and +def, strings, and comments. For any text window, these are the cursor (when present), found text (when possible), and selected text.

Text coloring is done in the background, so uncolorized text is occasionally visible. To change the color scheme, use the Configure IDLE dialog @@ -499,22 +496,22 @@ text in popups and dialogs is not user-configurable.

-

25.5.3. Startup and code execution¶

-

Upon startup with the -s option, IDLE will execute the file referenced by -the environment variables IDLESTARTUP or PYTHONSTARTUP. -IDLE first checks for IDLESTARTUP; if IDLESTARTUP is present the file -referenced is run. If IDLESTARTUP is not present, IDLE checks for -PYTHONSTARTUP. Files referenced by these environment variables are +

26.5.3. Startup and code execution¶

+

Upon startup with the -s option, IDLE will execute the file referenced by +the environment variables IDLESTARTUP or PYTHONSTARTUP. +IDLE first checks for IDLESTARTUP; if IDLESTARTUP is present the file +referenced is run. If IDLESTARTUP is not present, IDLE checks for +PYTHONSTARTUP. Files referenced by these environment variables are convenient places to store functions that are used frequently from the IDLE shell, or for executing import statements to import common modules.

-

In addition, Tk also loads a startup file if it is present. Note that the -Tk file is loaded unconditionally. This additional file is .Idle.py and is -looked for in the user’s home directory. Statements in this file will be +

In addition, Tk also loads a startup file if it is present. Note that the +Tk file is loaded unconditionally. This additional file is .Idle.py and is +looked for in the user’s home directory. Statements in this file will be executed in the Tk namespace, so this file is not useful for importing -functions to be used from IDLE’s Python shell.

+functions to be used from IDLE’s Python shell.

-

25.5.3.1. Command line usage¶

-
idle.py [-c command] [-d] [-e] [-h] [-i] [-r file] [-s] [-t title] [-] [arg] ...
+
26.5.3.1. Command line usage¶

+
idle.py [-c command] [-d] [-e] [-h] [-i] [-r file] [-s] [-t title] [-] [arg] ...
 
 -c command  run command in the shell window
 -d          enable debugger and open shell window
@@ -529,21 +526,21 @@ functions to be used from IDLE’s Python shell.

 

 
If there are arguments:

 
    
-
  • If -, -c, or r is used, all arguments are placed in
-sys.argv[1:...] and sys.argv[0] is set to '', '-c',
-or '-r'.  No editor window is opened, even if that is the default
+
  • If -, -c, or r is used, all arguments are placed in
+sys.argv[1:...] and sys.argv[0] is set to '', '-c',
+or '-r'.  No editor window is opened, even if that is the default
 set in the Options dialog.
    • 
 
  • Otherwise, arguments are files opened for editing and
-sys.argv reflects the arguments passed to IDLE itself.
    • 
+sys.argv reflects the arguments passed to IDLE itself.
 

 

 

-
25.5.3.2. Startup failure¶

+
26.5.3.2. Startup failure¶

 
IDLE uses a socket to communicate between the IDLE GUI process and the user
 code execution process.  A connection must be established whenever the Shell
 starts or restarts.  (The latter is indicated by a divider line that says
-‘RESTART’). If the user process fails to connect to the GUI process, it
-displays a Tk error box with a ‘cannot connect’ message that directs the
+‘RESTART’). If the user process fails to connect to the GUI process, it
+displays a Tk error box with a ‘cannot connect’ message that directs the
 user here.  It then exits.

 
A common cause of failure is a user-written file with the same name as a
 standard library module, such as random.py and tkinter.py. When such a
@@ -565,46 +562,46 @@ Manager to detect and stop one.  Sometimes a restart initiated by a program
 crash or Keyboard Interrupt (control-C) may fail to connect.  Dismissing
 the error box or Restart Shell on the Shell menu may fix a temporary problem.

 
When IDLE first starts, it attempts to read user configuration files in
-~/.idlerc/ (~ is one’s home directory).  If there is a problem, an error
+~/.idlerc/ (~ is one’s home directory).  If there is a problem, an error
 message should be displayed.  Leaving aside random disk glitches, this can
 be prevented by never editing the files by hand, using the configuration
 dialog, under Options, instead Options.  Once it happens, the solution may
 be to delete one or more of the configuration files.

 
If IDLE quits with no message, and it was not started from a console, try
-starting from a console (python -m idlelib) and see if a message appears.

+starting from a console (python -m idlelib) and see if a message appears.

 

 

-
25.5.3.3. IDLE-console differences¶

+
26.5.3.3. IDLE-console differences¶

 
With rare exceptions, the result of executing Python code with IDLE is
 intended to be the same as executing the same code in a console window.
 However, the different interface and operation occasionally affect
-visible results.  For instance, sys.modules starts with more entries.

-
IDLE also replaces sys.stdin, sys.stdout, and sys.stderr with
+visible results.  For instance, sys.modules starts with more entries.

+
IDLE also replaces sys.stdin, sys.stdout, and sys.stderr with
 objects that get input from and send output to the Shell window.
 When Shell has the focus, it controls the keyboard and screen.  This is
 normally transparent, but functions that directly access the keyboard
-and screen will not work.  If sys is reset with importlib.reload(sys),
-IDLE’s changes are lost and things like input, raw_input, and
-print will not work correctly.

-
With IDLE’s Shell, one enters, edits, and recalls complete statements.
+and screen will not work.  If sys is reset with importlib.reload(sys),
+IDLE’s changes are lost and things like input, raw_input, and
+print will not work correctly.

+
With IDLE’s Shell, one enters, edits, and recalls complete statements.
 Some consoles only work with a single physical line at a time.  IDLE uses
-exec to run each statement.  As a result, '__builtins__' is always
+exec to run each statement.  As a result, '__builtins__' is always
 defined for each statement.

 

 

-
25.5.3.4. Developing tkinter applications¶

+
26.5.3.4. Developing tkinter applications¶

 
IDLE is intentionally different from standard Python in order to
-facilitate development of tkinter programs.  Enter import tkinter as tk;
+facilitate development of tkinter programs.  Enter import tkinter as tk;
 root = tk.Tk() in standard Python and nothing appears.  Enter the same
 in IDLE and a tk window appears.  In standard Python, one must also enter
-root.update() to see the window.  IDLE does the equivalent in the
+root.update() to see the window.  IDLE does the equivalent in the
 background, about 20 times a second, which is about every 50 milleseconds.
-Next enter b = tk.Button(root, text='button'); b.pack().  Again,
-nothing visibly changes in standard Python until one enters root.update().

-
Most tkinter programs run root.mainloop(), which usually does not
+Next enter b = tk.Button(root, text='button'); b.pack().  Again,
+nothing visibly changes in standard Python until one enters root.update().

+
Most tkinter programs run root.mainloop(), which usually does not
 return until the tk app is destroyed.  If the program is run with
-python -i or from an IDLE editor, a >>> shell prompt does not
-appear until mainloop() returns, at which time there is nothing left
+python -i or from an IDLE editor, a >>> shell prompt does not
+appear until mainloop() returns, at which time there is nothing left
 to interact with.

 
When running a tkinter program from an IDLE editor, one can comment out
 the mainloop call.  One then gets a shell prompt immediately and can
@@ -612,7 +609,7 @@ interact with the live application.  One just has to remember to
 re-enable the mainloop call when running in standard Python.

 

 

-
25.5.3.5. Running without a subprocess¶

+
26.5.3.5. Running without a subprocess¶

 
By default, IDLE executes user code in a separate subprocess via a socket,
 which uses the internal loopback interface.  This connection is not
 externally visible and no data is sent to or received from the Internet.
@@ -638,24 +635,24 @@ with the default subprocess if at all possible.
-

25.5.4. Help and preferences¶

+

26.5.4. Help and preferences¶

-

25.5.4.1. Additional help sources¶

-

IDLE includes a help menu entry called “Python Docs” that will open the +

26.5.4.1. Additional help sources¶

+

IDLE includes a help menu entry called “Python Docs” that will open the extensive sources of help, including tutorials, available at docs.python.org. Selected URLs can be added or removed from the help menu at any time using the Configure IDLE dialog. See the IDLE help option in the help menu of IDLE for more information.

-

25.5.4.2. Setting preferences¶

+

26.5.4.2. Setting preferences¶

The font preferences, highlighting, keys, and general preferences can be changed via Configure IDLE on the Option menu. Keys can be user defined; IDLE ships with four built-in key sets. In addition, a user can create a custom key set in the Configure IDLE dialog under the keys tab.

-

25.5.4.3. Extensions¶

+

26.5.4.3. Extensions¶

IDLE contains an extension facility. Preferences for extensions can be changed with the Extensions tab of the preferences dialog. See the beginning of config-extensions.def in the idlelib directory for further @@ -673,40 +670,40 @@ also used for testing.

Table Of Contents

    -
  • 25.5. IDLE
      -
    • 25.5.1. Menus
        -
      • 25.5.1.1. File menu (Shell and Editor)
        • -
      • 25.5.1.2. Edit menu (Shell and Editor)
        • -
      • 25.5.1.3. Format menu (Editor window only)
        • -
      • 25.5.1.4. Run menu (Editor window only)
        • -
      • 25.5.1.5. Shell menu (Shell window only)
        • -
      • 25.5.1.6. Debug menu (Shell window only)
        • -
      • 25.5.1.7. Options menu (Shell and Editor)
        • -
      • 25.5.1.8. Window menu (Shell and Editor)
        • -
      • 25.5.1.9. Help menu (Shell and Editor)
        • -
      • 25.5.1.10. Context Menus
        • +
      • 26.5. IDLE
          +
        • 26.5.1. Menus
          • -
        • 25.5.2. Editing and navigation
            -
          • 25.5.2.1. Automatic indentation
            • -
          • 25.5.2.2. Completions
            • -
          • 25.5.2.3. Calltips
            • -
          • 25.5.2.4. Python Shell window
            • -
          • 25.5.2.5. Text colors
            • +
          • 26.5.2. Editing and navigation
            • -
          • 25.5.3. Startup and code execution
              -
            • 25.5.3.1. Command line usage
              • -
            • 25.5.3.2. Startup failure
              • -
            • 25.5.3.3. IDLE-console differences
              • -
            • 25.5.3.4. Developing tkinter applications
              • -
            • 25.5.3.5. Running without a subprocess
              • +
            • 26.5.3. Startup and code execution
              • -
            • 25.5.4. Help and preferences @@ -715,16 +712,16 @@ also used for testing.

              Previous topic

              25.4. tkinter.scrolledtext — Scrolled Text Widget

              + title="previous chapter">26.4. tkinter.scrolledtext — Scrolled Text Widget

              Next topic

              25.6. Other Graphical User Interface Packages

              + title="next chapter">26.6. Other Graphical User Interface Packages

              This Page

              - © Copyright 2001-2017, Python Software Foundation. + © Copyright 2001-2018, Python Software Foundation.
              + The Python Software Foundation is a non-profit corporation. - Please donate. +Please donate. +

              - Last updated on Sep 15, 2017. - Found a bug? + + Last updated on Jun 10, 2018. + Found a bug?
              - Created using Sphinx 1.3.6. + + Created using Sphinx 1.7.4.
              diff --git a/Lib/idlelib/idle_test/test_codecontext.py b/Lib/idlelib/idle_test/test_codecontext.py new file mode 100644 index 0000000..2e59b85 --- /dev/null +++ b/Lib/idlelib/idle_test/test_codecontext.py @@ -0,0 +1,407 @@ +"""Test idlelib.codecontext. + +Coverage: 100% +""" + +import re + +import unittest +from unittest import mock +from test.support import requires +from tkinter import Tk, Frame, Text, TclError + +import idlelib.codecontext as codecontext +from idlelib import config + + +usercfg = codecontext.idleConf.userCfg +testcfg = { + 'main': config.IdleUserConfParser(''), + 'highlight': config.IdleUserConfParser(''), + 'keys': config.IdleUserConfParser(''), + 'extensions': config.IdleUserConfParser(''), +} +code_sample = """\ + +class C1(): + # Class comment. + def __init__(self, a, b): + self.a = a + self.b = b + def compare(self): + if a > b: + return a + elif a < b: + return b + else: + return None +""" + + +class DummyEditwin: + def __init__(self, root, frame, text): + self.root = root + self.top = root + self.text_frame = frame + self.text = text + + +class CodeContextTest(unittest.TestCase): + + @classmethod + def setUpClass(cls): + requires('gui') + root = cls.root = Tk() + root.withdraw() + frame = cls.frame = Frame(root) + text = cls.text = Text(frame) + text.insert('1.0', code_sample) + # Need to pack for creation of code context text widget. + frame.pack(side='left', fill='both', expand=1) + text.pack(side='top', fill='both', expand=1) + cls.editor = DummyEditwin(root, frame, text) + codecontext.idleConf.userCfg = testcfg + + @classmethod + def tearDownClass(cls): + codecontext.idleConf.userCfg = usercfg + cls.editor.text.delete('1.0', 'end') + del cls.editor, cls.frame, cls.text + cls.root.update_idletasks() + cls.root.destroy() + del cls.root + + def setUp(self): + self.text.yview(0) + self.cc = codecontext.CodeContext(self.editor) + + def tearDown(self): + if self.cc.context: + self.cc.context.destroy() + # Explicitly call __del__ to remove scheduled scripts. + self.cc.__del__() + del self.cc.context, self.cc + + def test_init(self): + eq = self.assertEqual + ed = self.editor + cc = self.cc + + eq(cc.editwin, ed) + eq(cc.text, ed.text) + eq(cc.textfont, ed.text['font']) + self.assertIsNone(cc.context) + eq(cc.info, [(0, -1, '', False)]) + eq(cc.topvisible, 1) + eq(self.root.tk.call('after', 'info', self.cc.t1)[1], 'timer') + eq(self.root.tk.call('after', 'info', self.cc.t2)[1], 'timer') + + def test_del(self): + self.cc.__del__() + with self.assertRaises(TclError) as msg: + self.root.tk.call('after', 'info', self.cc.t1) + self.assertIn("doesn't exist", msg) + with self.assertRaises(TclError) as msg: + self.root.tk.call('after', 'info', self.cc.t2) + self.assertIn("doesn't exist", msg) + # For coverage on the except. Have to delete because the + # above Tcl error is caught by after_cancel. + del self.cc.t1, self.cc.t2 + self.cc.__del__() + + def test_reload(self): + codecontext.CodeContext.reload() + self.assertEqual(self.cc.colors, {'background': 'lightgray', + 'foreground': '#000000'}) + self.assertEqual(self.cc.context_depth, 15) + + def test_toggle_code_context_event(self): + eq = self.assertEqual + cc = self.cc + toggle = cc.toggle_code_context_event + + # Make sure code context is off. + if cc.context: + toggle() + + # Toggle on. + eq(toggle(), 'break') + self.assertIsNotNone(cc.context) + eq(cc.context['font'], cc.textfont) + eq(cc.context['fg'], cc.colors['foreground']) + eq(cc.context['bg'], cc.colors['background']) + eq(cc.context.get('1.0', 'end-1c'), '') + + # Toggle off. + eq(toggle(), 'break') + self.assertIsNone(cc.context) + + def test_get_context(self): + eq = self.assertEqual + gc = self.cc.get_context + + # stopline must be greater than 0. + with self.assertRaises(AssertionError): + gc(1, stopline=0) + + eq(gc(3), ([(2, 0, 'class C1():', 'class')], 0)) + + # Don't return comment. + eq(gc(4), ([(2, 0, 'class C1():', 'class')], 0)) + + # Two indentation levels and no comment. + eq(gc(5), ([(2, 0, 'class C1():', 'class'), + (4, 4, ' def __init__(self, a, b):', 'def')], 0)) + + # Only one 'def' is returned, not both at the same indent level. + eq(gc(10), ([(2, 0, 'class C1():', 'class'), + (7, 4, ' def compare(self):', 'def'), + (8, 8, ' if a > b:', 'if')], 0)) + + # With 'elif', also show the 'if' even though it's at the same level. + eq(gc(11), ([(2, 0, 'class C1():', 'class'), + (7, 4, ' def compare(self):', 'def'), + (8, 8, ' if a > b:', 'if'), + (10, 8, ' elif a < b:', 'elif')], 0)) + + # Set stop_line to not go back to first line in source code. + # Return includes stop_line. + eq(gc(11, stopline=2), ([(2, 0, 'class C1():', 'class'), + (7, 4, ' def compare(self):', 'def'), + (8, 8, ' if a > b:', 'if'), + (10, 8, ' elif a < b:', 'elif')], 0)) + eq(gc(11, stopline=3), ([(7, 4, ' def compare(self):', 'def'), + (8, 8, ' if a > b:', 'if'), + (10, 8, ' elif a < b:', 'elif')], 4)) + eq(gc(11, stopline=8), ([(8, 8, ' if a > b:', 'if'), + (10, 8, ' elif a < b:', 'elif')], 8)) + + # Set stop_indent to test indent level to stop at. + eq(gc(11, stopindent=4), ([(7, 4, ' def compare(self):', 'def'), + (8, 8, ' if a > b:', 'if'), + (10, 8, ' elif a < b:', 'elif')], 4)) + # Check that the 'if' is included. + eq(gc(11, stopindent=8), ([(8, 8, ' if a > b:', 'if'), + (10, 8, ' elif a < b:', 'elif')], 8)) + + def test_update_code_context(self): + eq = self.assertEqual + cc = self.cc + # Ensure code context is active. + if not cc.context: + cc.toggle_code_context_event() + + # Invoke update_code_context without scrolling - nothing happens. + self.assertIsNone(cc.update_code_context()) + eq(cc.info, [(0, -1, '', False)]) + eq(cc.topvisible, 1) + + # Scroll down to line 1. + cc.text.yview(1) + cc.update_code_context() + eq(cc.info, [(0, -1, '', False)]) + eq(cc.topvisible, 2) + eq(cc.context.get('1.0', 'end-1c'), '') + + # Scroll down to line 2. + cc.text.yview(2) + cc.update_code_context() + eq(cc.info, [(0, -1, '', False), (2, 0, 'class C1():', 'class')]) + eq(cc.topvisible, 3) + eq(cc.context.get('1.0', 'end-1c'), 'class C1():') + + # Scroll down to line 3. Since it's a comment, nothing changes. + cc.text.yview(3) + cc.update_code_context() + eq(cc.info, [(0, -1, '', False), (2, 0, 'class C1():', 'class')]) + eq(cc.topvisible, 4) + eq(cc.context.get('1.0', 'end-1c'), 'class C1():') + + # Scroll down to line 4. + cc.text.yview(4) + cc.update_code_context() + eq(cc.info, [(0, -1, '', False), + (2, 0, 'class C1():', 'class'), + (4, 4, ' def __init__(self, a, b):', 'def')]) + eq(cc.topvisible, 5) + eq(cc.context.get('1.0', 'end-1c'), 'class C1():\n' + ' def __init__(self, a, b):') + + # Scroll down to line 11. Last 'def' is removed. + cc.text.yview(11) + cc.update_code_context() + eq(cc.info, [(0, -1, '', False), + (2, 0, 'class C1():', 'class'), + (7, 4, ' def compare(self):', 'def'), + (8, 8, ' if a > b:', 'if'), + (10, 8, ' elif a < b:', 'elif')]) + eq(cc.topvisible, 12) + eq(cc.context.get('1.0', 'end-1c'), 'class C1():\n' + ' def compare(self):\n' + ' if a > b:\n' + ' elif a < b:') + + # No scroll. No update, even though context_depth changed. + cc.update_code_context() + cc.context_depth = 1 + eq(cc.info, [(0, -1, '', False), + (2, 0, 'class C1():', 'class'), + (7, 4, ' def compare(self):', 'def'), + (8, 8, ' if a > b:', 'if'), + (10, 8, ' elif a < b:', 'elif')]) + eq(cc.topvisible, 12) + eq(cc.context.get('1.0', 'end-1c'), 'class C1():\n' + ' def compare(self):\n' + ' if a > b:\n' + ' elif a < b:') + + # Scroll up. + cc.text.yview(5) + cc.update_code_context() + eq(cc.info, [(0, -1, '', False), + (2, 0, 'class C1():', 'class'), + (4, 4, ' def __init__(self, a, b):', 'def')]) + eq(cc.topvisible, 6) + # context_depth is 1. + eq(cc.context.get('1.0', 'end-1c'), ' def __init__(self, a, b):') + + def test_jumptoline(self): + eq = self.assertEqual + cc = self.cc + jump = cc.jumptoline + + if not cc.context: + cc.toggle_code_context_event() + + # Empty context. + cc.text.yview(f'{2}.0') + cc.update_code_context() + eq(cc.topvisible, 2) + cc.context.mark_set('insert', '1.5') + jump() + eq(cc.topvisible, 1) + + # 4 lines of context showing. + cc.text.yview(f'{12}.0') + cc.update_code_context() + eq(cc.topvisible, 12) + cc.context.mark_set('insert', '3.0') + jump() + eq(cc.topvisible, 8) + + # More context lines than limit. + cc.context_depth = 2 + cc.text.yview(f'{12}.0') + cc.update_code_context() + eq(cc.topvisible, 12) + cc.context.mark_set('insert', '1.0') + jump() + eq(cc.topvisible, 8) + + @mock.patch.object(codecontext.CodeContext, 'update_code_context') + def test_timer_event(self, mock_update): + # Ensure code context is not active. + if self.cc.context: + self.cc.toggle_code_context_event() + self.cc.timer_event() + mock_update.assert_not_called() + + # Activate code context. + self.cc.toggle_code_context_event() + self.cc.timer_event() + mock_update.assert_called() + + def test_config_timer_event(self): + eq = self.assertEqual + cc = self.cc + save_font = cc.text['font'] + save_colors = codecontext.CodeContext.colors + test_font = 'FakeFont' + test_colors = {'background': '#222222', 'foreground': '#ffff00'} + + # Ensure code context is not active. + if cc.context: + cc.toggle_code_context_event() + + # Nothing updates on inactive code context. + cc.text['font'] = test_font + codecontext.CodeContext.colors = test_colors + cc.config_timer_event() + eq(cc.textfont, save_font) + eq(cc.contextcolors, save_colors) + + # Activate code context, but no change to font or color. + cc.toggle_code_context_event() + cc.text['font'] = save_font + codecontext.CodeContext.colors = save_colors + cc.config_timer_event() + eq(cc.textfont, save_font) + eq(cc.contextcolors, save_colors) + eq(cc.context['font'], save_font) + eq(cc.context['background'], save_colors['background']) + eq(cc.context['foreground'], save_colors['foreground']) + + # Active code context, change font. + cc.text['font'] = test_font + cc.config_timer_event() + eq(cc.textfont, test_font) + eq(cc.contextcolors, save_colors) + eq(cc.context['font'], test_font) + eq(cc.context['background'], save_colors['background']) + eq(cc.context['foreground'], save_colors['foreground']) + + # Active code context, change color. + cc.text['font'] = save_font + codecontext.CodeContext.colors = test_colors + cc.config_timer_event() + eq(cc.textfont, save_font) + eq(cc.contextcolors, test_colors) + eq(cc.context['font'], save_font) + eq(cc.context['background'], test_colors['background']) + eq(cc.context['foreground'], test_colors['foreground']) + codecontext.CodeContext.colors = save_colors + cc.config_timer_event() + + +class HelperFunctionText(unittest.TestCase): + + def test_get_spaces_firstword(self): + get = codecontext.get_spaces_firstword + test_lines = ( + (' first word', (' ', 'first')), + ('\tfirst word', ('\t', 'first')), + (' \u19D4\u19D2: ', (' ', '\u19D4\u19D2')), + ('no spaces', ('', 'no')), + ('', ('', '')), + ('# TEST COMMENT', ('', '')), + (' (continuation)', (' ', '')) + ) + for line, expected_output in test_lines: + self.assertEqual(get(line), expected_output) + + # Send the pattern in the call. + self.assertEqual(get(' (continuation)', + c=re.compile(r'^(\s*)([^\s]*)')), + (' ', '(continuation)')) + + def test_get_line_info(self): + eq = self.assertEqual + gli = codecontext.get_line_info + lines = code_sample.splitlines() + + # Line 1 is not a BLOCKOPENER. + eq(gli(lines[0]), (codecontext.INFINITY, '', False)) + # Line 2 is a BLOCKOPENER without an indent. + eq(gli(lines[1]), (0, 'class C1():', 'class')) + # Line 3 is not a BLOCKOPENER and does not return the indent level. + eq(gli(lines[2]), (codecontext.INFINITY, ' # Class comment.', False)) + # Line 4 is a BLOCKOPENER and is indented. + eq(gli(lines[3]), (4, ' def __init__(self, a, b):', 'def')) + # Line 8 is a different BLOCKOPENER and is indented. + eq(gli(lines[7]), (8, ' if a > b:', 'if')) + # Test tab. + eq(gli('\tif a == b:'), (1, '\tif a == b:', 'if')) + + +if __name__ == '__main__': + unittest.main(verbosity=2) diff --git a/Lib/idlelib/idle_test/test_configdialog.py b/Lib/idlelib/idle_test/test_configdialog.py index 982dc0b..26aba32 100644 --- a/Lib/idlelib/idle_test/test_configdialog.py +++ b/Lib/idlelib/idle_test/test_configdialog.py @@ -1170,7 +1170,7 @@ class GenPageTest(unittest.TestCase): def test_context(self): self.page.context_int.delete(0, 'end') self.page.context_int.insert(0, '1') - self.assertEqual(extpage, {'CodeContext': {'numlines': '1'}}) + self.assertEqual(extpage, {'CodeContext': {'maxlines': '1'}}) def test_source_selected(self): d = self.page diff --git a/Lib/idlelib/idle_test/test_editmenu.py b/Lib/idlelib/idle_test/test_editmenu.py index 17eb25c..1747847 100644 --- a/Lib/idlelib/idle_test/test_editmenu.py +++ b/Lib/idlelib/idle_test/test_editmenu.py @@ -1,6 +1,6 @@ '''Test (selected) IDLE Edit menu items. -Edit modules have their own test files files +Edit modules have their own test files ''' from test.support import requires requires('gui') diff --git a/Lib/idlelib/idle_test/test_text.py b/Lib/idlelib/idle_test/test_text.py index a5ba7bb..0f31179 100644 --- a/Lib/idlelib/idle_test/test_text.py +++ b/Lib/idlelib/idle_test/test_text.py @@ -9,7 +9,7 @@ from _tkinter import TclError class TextTest(object): "Define items common to both sets of tests." - hw = 'hello\nworld' # Several tests insert this after after initialization. + hw = 'hello\nworld' # Several tests insert this after initialization. hwn = hw+'\n' # \n present at initialization, before insert # setUpClass defines cls.Text and maybe cls.root. diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py index ee13131..ddfb56a 100755 --- a/Lib/idlelib/pyshell.py +++ b/Lib/idlelib/pyshell.py @@ -8,6 +8,17 @@ except ImportError: print("** IDLE can't import Tkinter.\n" "Your Python may not be configured for Tk. **", file=sys.__stderr__) raise SystemExit(1) + +# Valid arguments for the ...Awareness call below are defined in the following. +# https://msdn.microsoft.com/en-us/library/windows/desktop/dn280512(v=vs.85).aspx +if sys.platform == 'win32': + import ctypes + PROCESS_SYSTEM_DPI_AWARE = 1 + try: + ctypes.OleDLL('shcore').SetProcessDpiAwareness(PROCESS_SYSTEM_DPI_AWARE) + except (AttributeError, OSError): + pass + import tkinter.messagebox as tkMessageBox if TkVersion < 8.5: root = Tk() # otherwise create root in main diff --git a/Lib/inspect.py b/Lib/inspect.py index dc2aab2..362ae9a 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -2399,6 +2399,16 @@ _VAR_POSITIONAL = _ParameterKind.VAR_POSITIONAL _KEYWORD_ONLY = _ParameterKind.KEYWORD_ONLY _VAR_KEYWORD = _ParameterKind.VAR_KEYWORD +_PARAM_NAME_MAPPING = { + _POSITIONAL_ONLY: 'positional-only', + _POSITIONAL_OR_KEYWORD: 'positional or keyword', + _VAR_POSITIONAL: 'variadic positional', + _KEYWORD_ONLY: 'keyword-only', + _VAR_KEYWORD: 'variadic keyword' +} + +_get_paramkind_descr = _PARAM_NAME_MAPPING.__getitem__ + class Parameter: """Represents a parameter in a function signature. @@ -2433,15 +2443,14 @@ class Parameter: empty = _empty def __init__(self, name, kind, *, default=_empty, annotation=_empty): - - if kind not in (_POSITIONAL_ONLY, _POSITIONAL_OR_KEYWORD, - _VAR_POSITIONAL, _KEYWORD_ONLY, _VAR_KEYWORD): - raise ValueError("invalid value for 'Parameter.kind' attribute") - self._kind = kind - + try: + self._kind = _ParameterKind(kind) + except ValueError: + raise ValueError(f'value {kind!r} is not a valid Parameter.kind') if default is not _empty: - if kind in (_VAR_POSITIONAL, _VAR_KEYWORD): - msg = '{} parameters cannot have default values'.format(kind) + if self._kind in (_VAR_POSITIONAL, _VAR_KEYWORD): + msg = '{} parameters cannot have default values' + msg = msg.format(_get_paramkind_descr(self._kind)) raise ValueError(msg) self._default = default self._annotation = annotation @@ -2450,19 +2459,21 @@ class Parameter: raise ValueError('name is a required attribute for Parameter') if not isinstance(name, str): - raise TypeError("name must be a str, not a {!r}".format(name)) + msg = 'name must be a str, not a {}'.format(type(name).__name__) + raise TypeError(msg) if name[0] == '.' and name[1:].isdigit(): # These are implicit arguments generated by comprehensions. In # order to provide a friendlier interface to users, we recast # their name as "implicitN" and treat them as positional-only. # See issue 19611. - if kind != _POSITIONAL_OR_KEYWORD: - raise ValueError( - 'implicit arguments must be passed in as {}'.format( - _POSITIONAL_OR_KEYWORD - ) + if self._kind != _POSITIONAL_OR_KEYWORD: + msg = ( + 'implicit arguments must be passed as ' + 'positional or keyword arguments, not {}' ) + msg = msg.format(_get_paramkind_descr(self._kind)) + raise ValueError(msg) self._kind = _POSITIONAL_ONLY name = 'implicit{}'.format(name[1:]) @@ -2730,8 +2741,12 @@ class Signature: name = param.name if kind < top_kind: - msg = 'wrong parameter order: {!r} before {!r}' - msg = msg.format(top_kind, kind) + msg = ( + 'wrong parameter order: {} parameter before {} ' + 'parameter' + ) + msg = msg.format(_get_paramkind_descr(top_kind), + _get_paramkind_descr(kind)) raise ValueError(msg) elif kind > top_kind: kind_defaults = False diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py index 70746f8..583f02a 100644 --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1498,45 +1498,28 @@ class IPv4Network(_BaseV4, _BaseNetwork): # Constructing from a packed address or integer if isinstance(address, (int, bytes)): - self.network_address = IPv4Address(address) - self.netmask, self._prefixlen = self._make_netmask(self._max_prefixlen) - #fixme: address/network test here. - return - - if isinstance(address, tuple): - if len(address) > 1: - arg = address[1] - else: - # We weren't given an address[1] - arg = self._max_prefixlen - self.network_address = IPv4Address(address[0]) - self.netmask, self._prefixlen = self._make_netmask(arg) - packed = int(self.network_address) - if packed & int(self.netmask) != packed: - if strict: - raise ValueError('%s has host bits set' % self) - else: - self.network_address = IPv4Address(packed & - int(self.netmask)) - return - + addr = address + mask = self._max_prefixlen + # Constructing from a tuple (addr, [mask]) + elif isinstance(address, tuple): + addr = address[0] + mask = address[1] if len(address) > 1 else self._max_prefixlen # Assume input argument to be string or any object representation # which converts into a formatted IP prefix string. - addr = _split_optional_netmask(address) - self.network_address = IPv4Address(self._ip_int_from_string(addr[0])) - - if len(addr) == 2: - arg = addr[1] else: - arg = self._max_prefixlen - self.netmask, self._prefixlen = self._make_netmask(arg) - - if strict: - if (IPv4Address(int(self.network_address) & int(self.netmask)) != - self.network_address): + args = _split_optional_netmask(address) + addr = self._ip_int_from_string(args[0]) + mask = args[1] if len(args) == 2 else self._max_prefixlen + + self.network_address = IPv4Address(addr) + self.netmask, self._prefixlen = self._make_netmask(mask) + packed = int(self.network_address) + if packed & int(self.netmask) != packed: + if strict: raise ValueError('%s has host bits set' % self) - self.network_address = IPv4Address(int(self.network_address) & - int(self.netmask)) + else: + self.network_address = IPv4Address(packed & + int(self.netmask)) if self._prefixlen == (self._max_prefixlen - 1): self.hosts = self.__iter__ @@ -2191,46 +2174,30 @@ class IPv6Network(_BaseV6, _BaseNetwork): """ _BaseNetwork.__init__(self, address) - # Efficient constructor from integer or packed address - if isinstance(address, (bytes, int)): - self.network_address = IPv6Address(address) - self.netmask, self._prefixlen = self._make_netmask(self._max_prefixlen) - return - - if isinstance(address, tuple): - if len(address) > 1: - arg = address[1] - else: - arg = self._max_prefixlen - self.netmask, self._prefixlen = self._make_netmask(arg) - self.network_address = IPv6Address(address[0]) - packed = int(self.network_address) - if packed & int(self.netmask) != packed: - if strict: - raise ValueError('%s has host bits set' % self) - else: - self.network_address = IPv6Address(packed & - int(self.netmask)) - return - + # Constructing from a packed address or integer + if isinstance(address, (int, bytes)): + addr = address + mask = self._max_prefixlen + # Constructing from a tuple (addr, [mask]) + elif isinstance(address, tuple): + addr = address[0] + mask = address[1] if len(address) > 1 else self._max_prefixlen # Assume input argument to be string or any object representation # which converts into a formatted IP prefix string. - addr = _split_optional_netmask(address) - - self.network_address = IPv6Address(self._ip_int_from_string(addr[0])) - - if len(addr) == 2: - arg = addr[1] else: - arg = self._max_prefixlen - self.netmask, self._prefixlen = self._make_netmask(arg) - - if strict: - if (IPv6Address(int(self.network_address) & int(self.netmask)) != - self.network_address): + args = _split_optional_netmask(address) + addr = self._ip_int_from_string(args[0]) + mask = args[1] if len(args) == 2 else self._max_prefixlen + + self.network_address = IPv6Address(addr) + self.netmask, self._prefixlen = self._make_netmask(mask) + packed = int(self.network_address) + if packed & int(self.netmask) != packed: + if strict: raise ValueError('%s has host bits set' % self) - self.network_address = IPv6Address(int(self.network_address) & - int(self.netmask)) + else: + self.network_address = IPv6Address(packed & + int(self.netmask)) if self._prefixlen == (self._max_prefixlen - 1): self.hosts = self.__iter__ diff --git a/Lib/json/scanner.py b/Lib/json/scanner.py index 86426cd..6e19ca6 100644 --- a/Lib/json/scanner.py +++ b/Lib/json/scanner.py @@ -68,6 +68,6 @@ def py_make_scanner(context): finally: memo.clear() - return _scan_once + return scan_once make_scanner = c_make_scanner or py_make_scanner diff --git a/Lib/lib2to3/main.py b/Lib/lib2to3/main.py index 1a1df01..d6b7088 100644 --- a/Lib/lib2to3/main.py +++ b/Lib/lib2to3/main.py @@ -80,7 +80,7 @@ class StdoutRefactoringTool(refactor.MultiprocessRefactoringTool): filename += self._append_suffix if orig_filename != filename: output_dir = os.path.dirname(filename) - if not os.path.isdir(output_dir): + if not os.path.isdir(output_dir) and output_dir: os.makedirs(output_dir) self.log_message('Writing converted %s to %s.', orig_filename, filename) diff --git a/Lib/locale.py b/Lib/locale.py index 4de0090..e85f34e 100644 --- a/Lib/locale.py +++ b/Lib/locale.py @@ -828,6 +828,16 @@ for k, v in sorted(locale_encoding_alias.items()): # # SS 2014-10-01: # Updated alias mapping with glibc 2.19 supported locales. +# +# SS 2018-05-05: +# Updated alias mapping with glibc 2.27 supported locales. +# +# These are the differences compared to the old mapping (Python 3.6.5 +# and older): +# +# updated 'ca_es@valencia' -> 'ca_ES.ISO8859-15@valencia' to 'ca_ES.UTF-8@valencia' +# updated 'kk_kz' -> 'kk_KZ.RK1048' to 'kk_KZ.ptcp154' +# updated 'russian' -> 'ru_RU.ISO8859-5' to 'ru_RU.KOI8-R' locale_alias = { 'a3': 'az_AZ.KOI8-C', @@ -838,10 +848,13 @@ locale_alias = { 'aa_et': 'aa_ET.UTF-8', 'af': 'af_ZA.ISO8859-1', 'af_za': 'af_ZA.ISO8859-1', + 'agr_pe': 'agr_PE.UTF-8', + 'ak_gh': 'ak_GH.UTF-8', 'am': 'am_ET.UTF-8', 'am_et': 'am_ET.UTF-8', 'american': 'en_US.ISO8859-1', 'an_es': 'an_ES.ISO8859-15', + 'anp_in': 'anp_IN.UTF-8', 'ar': 'ar_AA.ISO8859-6', 'ar_aa': 'ar_AA.ISO8859-6', 'ar_ae': 'ar_AE.ISO8859-6', @@ -859,6 +872,7 @@ locale_alias = { 'ar_qa': 'ar_QA.ISO8859-6', 'ar_sa': 'ar_SA.ISO8859-6', 'ar_sd': 'ar_SD.ISO8859-6', + 'ar_ss': 'ar_SS.UTF-8', 'ar_sy': 'ar_SY.ISO8859-6', 'ar_tn': 'ar_TN.ISO8859-6', 'ar_ye': 'ar_YE.ISO8859-6', @@ -870,6 +884,7 @@ locale_alias = { 'az': 'az_AZ.ISO8859-9E', 'az_az': 'az_AZ.ISO8859-9E', 'az_az.iso88599e': 'az_AZ.ISO8859-9E', + 'az_ir': 'az_IR.UTF-8', 'be': 'be_BY.CP1251', 'be@latin': 'be_BY.UTF-8@latin', 'be_bg.utf8': 'bg_BG.UTF-8', @@ -880,7 +895,10 @@ locale_alias = { 'ber_ma': 'ber_MA.UTF-8', 'bg': 'bg_BG.CP1251', 'bg_bg': 'bg_BG.CP1251', + 'bhb_in.utf8': 'bhb_IN.UTF-8', 'bho_in': 'bho_IN.UTF-8', + 'bho_np': 'bho_NP.UTF-8', + 'bi_vu': 'bi_VU.UTF-8', 'bn_bd': 'bn_BD.UTF-8', 'bn_in': 'bn_IN.UTF-8', 'bo_cn': 'bo_CN.UTF-8', @@ -905,13 +923,17 @@ locale_alias = { 'ca': 'ca_ES.ISO8859-1', 'ca_ad': 'ca_AD.ISO8859-1', 'ca_es': 'ca_ES.ISO8859-1', - 'ca_es@valencia': 'ca_ES.ISO8859-15@valencia', + 'ca_es@valencia': 'ca_ES.UTF-8@valencia', 'ca_fr': 'ca_FR.ISO8859-1', 'ca_it': 'ca_IT.ISO8859-1', 'catalan': 'ca_ES.ISO8859-1', + 'ce_ru': 'ce_RU.UTF-8', 'cextend': 'en_US.ISO8859-1', 'chinese-s': 'zh_CN.eucCN', 'chinese-t': 'zh_TW.eucTW', + 'chr_us': 'chr_US.UTF-8', + 'ckb_iq': 'ckb_IQ.UTF-8', + 'cmn_tw': 'cmn_TW.UTF-8', 'crh_ua': 'crh_UA.UTF-8', 'croatian': 'hr_HR.ISO8859-2', 'cs': 'cs_CZ.ISO8859-2', @@ -933,6 +955,7 @@ locale_alias = { 'de_be': 'de_BE.ISO8859-1', 'de_ch': 'de_CH.ISO8859-1', 'de_de': 'de_DE.ISO8859-1', + 'de_it': 'de_IT.ISO8859-1', 'de_li.utf8': 'de_LI.UTF-8', 'de_lu': 'de_LU.ISO8859-1', 'deutsch': 'de_DE.ISO8859-1', @@ -959,10 +982,12 @@ locale_alias = { 'en_gb': 'en_GB.ISO8859-1', 'en_hk': 'en_HK.ISO8859-1', 'en_ie': 'en_IE.ISO8859-1', + 'en_il': 'en_IL.UTF-8', 'en_in': 'en_IN.ISO8859-1', 'en_ng': 'en_NG.UTF-8', 'en_nz': 'en_NZ.ISO8859-1', 'en_ph': 'en_PH.ISO8859-1', + 'en_sc.utf8': 'en_SC.UTF-8', 'en_sg': 'en_SG.ISO8859-1', 'en_uk': 'en_GB.ISO8859-1', 'en_us': 'en_US.ISO8859-1', @@ -973,6 +998,7 @@ locale_alias = { 'en_zw.utf8': 'en_ZS.UTF-8', 'eng_gb': 'en_GB.ISO8859-1', 'english': 'en_EN.ISO8859-1', + 'english.iso88591': 'en_US.ISO8859-1', 'english_uk': 'en_GB.ISO8859-1', 'english_united-states': 'en_US.ISO8859-1', 'english_united-states.437': 'C', @@ -1053,12 +1079,14 @@ locale_alias = { 'gv': 'gv_GB.ISO8859-1', 'gv_gb': 'gv_GB.ISO8859-1', 'ha_ng': 'ha_NG.UTF-8', + 'hak_tw': 'hak_TW.UTF-8', 'he': 'he_IL.ISO8859-8', 'he_il': 'he_IL.ISO8859-8', 'hebrew': 'he_IL.ISO8859-8', 'hi': 'hi_IN.ISCII-DEV', 'hi_in': 'hi_IN.ISCII-DEV', 'hi_in.isciidev': 'hi_IN.ISCII-DEV', + 'hif_fj': 'hif_FJ.UTF-8', 'hne': 'hne_IN.UTF-8', 'hne_in': 'hne_IN.UTF-8', 'hr': 'hr_HR.ISO8859-2', @@ -1113,7 +1141,8 @@ locale_alias = { 'ka_ge.georgianacademy': 'ka_GE.GEORGIAN-ACADEMY', 'ka_ge.georgianps': 'ka_GE.GEORGIAN-PS', 'ka_ge.georgianrs': 'ka_GE.GEORGIAN-ACADEMY', - 'kk_kz': 'kk_KZ.RK1048', + 'kab_dz': 'kab_DZ.UTF-8', + 'kk_kz': 'kk_KZ.ptcp154', 'kl': 'kl_GL.ISO8859-1', 'kl_gl': 'kl_GL.ISO8859-1', 'km_kh': 'km_KH.UTF-8', @@ -1139,6 +1168,7 @@ locale_alias = { 'li_nl': 'li_NL.UTF-8', 'lij_it': 'lij_IT.UTF-8', 'lithuanian': 'lt_LT.ISO8859-13', + 'ln_cd': 'ln_CD.UTF-8', 'lo': 'lo_LA.MULELAO-1', 'lo_la': 'lo_LA.MULELAO-1', 'lo_la.cp1133': 'lo_LA.IBM-CP1133', @@ -1148,13 +1178,18 @@ locale_alias = { 'lt_lt': 'lt_LT.ISO8859-13', 'lv': 'lv_LV.ISO8859-13', 'lv_lv': 'lv_LV.ISO8859-13', + 'lzh_tw': 'lzh_TW.UTF-8', 'mag_in': 'mag_IN.UTF-8', 'mai': 'mai_IN.UTF-8', 'mai_in': 'mai_IN.UTF-8', + 'mai_np': 'mai_NP.UTF-8', + 'mfe_mu': 'mfe_MU.UTF-8', 'mg_mg': 'mg_MG.ISO8859-15', 'mhr_ru': 'mhr_RU.UTF-8', 'mi': 'mi_NZ.ISO8859-1', 'mi_nz': 'mi_NZ.ISO8859-1', + 'miq_ni': 'miq_NI.UTF-8', + 'mjw_in': 'mjw_IN.UTF-8', 'mk': 'mk_MK.ISO8859-5', 'mk_mk': 'mk_MK.ISO8859-5', 'ml': 'ml_IN.UTF-8', @@ -1168,7 +1203,7 @@ locale_alias = { 'mt': 'mt_MT.ISO8859-3', 'mt_mt': 'mt_MT.ISO8859-3', 'my_mm': 'my_MM.UTF-8', - 'nan_tw@latin': 'nan_TW.UTF-8@latin', + 'nan_tw': 'nan_TW.UTF-8', 'nb': 'nb_NO.ISO8859-1', 'nb_no': 'nb_NO.ISO8859-1', 'nds_de': 'nds_DE.UTF-8', @@ -1207,6 +1242,8 @@ locale_alias = { 'pa_in': 'pa_IN.UTF-8', 'pa_pk': 'pa_PK.UTF-8', 'pap_an': 'pap_AN.UTF-8', + 'pap_aw': 'pap_AW.UTF-8', + 'pap_cw': 'pap_CW.UTF-8', 'pd': 'pd_US.ISO8859-1', 'pd_de': 'pd_DE.ISO8859-1', 'pd_us': 'pd_US.ISO8859-1', @@ -1225,6 +1262,8 @@ locale_alias = { 'pt': 'pt_PT.ISO8859-1', 'pt_br': 'pt_BR.ISO8859-1', 'pt_pt': 'pt_PT.ISO8859-1', + 'quz_pe': 'quz_PE.UTF-8', + 'raj_in': 'raj_IN.UTF-8', 'ro': 'ro_RO.ISO8859-2', 'ro_ro': 'ro_RO.ISO8859-2', 'romanian': 'ro_RO.ISO8859-2', @@ -1232,7 +1271,7 @@ locale_alias = { 'ru_ru': 'ru_RU.UTF-8', 'ru_ua': 'ru_UA.KOI8-U', 'rumanian': 'ro_RO.ISO8859-2', - 'russian': 'ru_RU.ISO8859-5', + 'russian': 'ru_RU.KOI8-R', 'rw': 'rw_RW.ISO8859-1', 'rw_rw': 'rw_RW.ISO8859-1', 'sa_in': 'sa_IN.UTF-8', @@ -1244,12 +1283,14 @@ locale_alias = { 'sd_pk': 'sd_PK.UTF-8', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8@latin', + 'sgs_lt': 'sgs_LT.UTF-8', 'sh': 'sr_RS.UTF-8@latin', 'sh_ba.iso88592@bosnia': 'sr_CS.ISO8859-2', 'sh_hr': 'sh_HR.ISO8859-2', 'sh_hr.iso88592': 'hr_HR.ISO8859-2', 'sh_sp': 'sr_CS.ISO8859-2', 'sh_yu': 'sr_RS.UTF-8@latin', + 'shn_mm': 'shn_MM.UTF-8', 'shs_ca': 'shs_CA.UTF-8', 'si': 'si_LK.UTF-8', 'si_lk': 'si_LK.UTF-8', @@ -1263,6 +1304,7 @@ locale_alias = { 'slovak': 'sk_SK.ISO8859-2', 'slovene': 'sl_SI.ISO8859-2', 'slovenian': 'sl_SI.ISO8859-2', + 'sm_ws': 'sm_WS.UTF-8', 'so_dj': 'so_DJ.ISO8859-1', 'so_et': 'so_ET.UTF-8', 'so_ke': 'so_KE.ISO8859-1', @@ -1309,6 +1351,7 @@ locale_alias = { 'ta_in.tscii': 'ta_IN.TSCII-0', 'ta_in.tscii0': 'ta_IN.TSCII-0', 'ta_lk': 'ta_LK.UTF-8', + 'tcy_in.utf8': 'tcy_IN.UTF-8', 'te': 'te_IN.UTF-8', 'te_in': 'te_IN.UTF-8', 'tg': 'tg_TJ.KOI8-C', @@ -1318,6 +1361,7 @@ locale_alias = { 'th_th.tactis': 'th_TH.TIS620', 'th_th.tis620': 'th_TH.TIS620', 'thai': 'th_TH.ISO8859-11', + 'the_np': 'the_NP.UTF-8', 'ti_er': 'ti_ER.UTF-8', 'ti_et': 'ti_ET.UTF-8', 'tig_er': 'tig_ER.UTF-8', @@ -1326,6 +1370,8 @@ locale_alias = { 'tl_ph': 'tl_PH.ISO8859-1', 'tn': 'tn_ZA.ISO8859-15', 'tn_za': 'tn_ZA.ISO8859-15', + 'to_to': 'to_TO.UTF-8', + 'tpi_pg': 'tpi_PG.UTF-8', 'tr': 'tr_TR.ISO8859-9', 'tr_cy': 'tr_CY.ISO8859-9', 'tr_tr': 'tr_TR.ISO8859-9', @@ -1368,6 +1414,7 @@ locale_alias = { 'yi_us': 'yi_US.CP1255', 'yo_ng': 'yo_NG.UTF-8', 'yue_hk': 'yue_HK.UTF-8', + 'yuw_pg': 'yuw_PG.UTF-8', 'zh': 'zh_CN.eucCN', 'zh_cn': 'zh_CN.gb2312', 'zh_cn.big5': 'zh_TW.big5', diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py index 3617dbb..e2084f5 100644 --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -473,7 +473,8 @@ class Formatter(object): Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with - the optional datefmt argument (if omitted, you get the ISO8601 format). + the optional datefmt argument. If datefmt is omitted, you get an + ISO8601-like (or RFC 3339-like) format. Use a style parameter of '%', '{' or '$' to specify that you want to use one of %-formatting, :meth:`str.format` (``{}``) formatting or @@ -501,13 +502,13 @@ class Formatter(object): in formatters to provide for any specific requirement, but the basic behaviour is as follows: if datefmt (a string) is specified, it is used with time.strftime() to format the creation time of the - record. Otherwise, the ISO8601 format is used. The resulting - string is returned. This function uses a user-configurable function - to convert the creation time to a tuple. By default, time.localtime() - is used; to change this for a particular formatter instance, set the - 'converter' attribute to a function with the same signature as - time.localtime() or time.gmtime(). To change it for all formatters, - for example if you want all logging times to be shown in GMT, + record. Otherwise, an ISO8601-like (or RFC 3339-like) format is used. + The resulting string is returned. This function uses a user-configurable + function to convert the creation time to a tuple. By default, + time.localtime() is used; to change this for a particular formatter + instance, set the 'converter' attribute to a function with the same + signature as time.localtime() or time.gmtime(). To change it for all + formatters, for example if you want all logging times to be shown in GMT, set the 'converter' attribute in the Formatter class. """ ct = self.converter(record.created) diff --git a/Lib/pydoc.py b/Lib/pydoc.py index 34e2fab..8c70754 100644 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -1716,8 +1716,9 @@ class Helper: } # Either add symbols to this dictionary or to the symbols dictionary # directly: Whichever is easier. They are merged later. + _strprefixes = [p + q for p in ('b', 'f', 'r', 'u') for q in ("'", '"')] _symbols_inverse = { - 'STRINGS' : ("'", "'''", "r'", "b'", '"""', '"', 'r"', 'b"'), + 'STRINGS' : ("'", "'''", '"', '"""', *_strprefixes), 'OPERATORS' : ('+', '-', '*', '**', '/', '//', '%', '<<', '>>', '&', '|', '^', '~', '<', '>', '<=', '>=', '==', '!=', '<>'), 'COMPARISON' : ('<', '>', '<=', '>=', '==', '!=', '<>'), @@ -1874,7 +1875,13 @@ has the same effect as typing a particular string at the help> prompt. if not request: break except (KeyboardInterrupt, EOFError): break - request = replace(request, '"', '', "'", '').strip() + request = request.strip() + + # Make sure significant trailing quoting marks of literals don't + # get deleted while cleaning input + if (len(request) > 2 and request[0] == request[-1] in ("'", '"') + and request[0] not in request[1:-1]): + request = request[1:-1] if request.lower() in ('q', 'quit'): break if request == 'help': self.intro() @@ -2266,6 +2273,10 @@ def _start_server(urlhandler, port): def stop(self): """Stop the server and this thread nicely""" self.docserver.quit = True + self.join() + # explicitly break a reference cycle: DocServer.callback + # has indirectly a reference to ServerThread. + self.docserver = None self.serving = False self.url = None diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index f37e672..0c307a1 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Tue Mar 13 21:13:16 2018 +# Autogenerated by Sphinx on Tue Jun 12 00:16:23 2018 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' @@ -106,7 +106,7 @@ topics = {'assert': 'The "assert" statement\n' ' corresponding targets.\n' '\n' ' * If the target list contains one target prefixed with an\n' - ' asterisk, called a "starred" target: The object must be ' + ' asterisk, called a “starred” target: The object must be ' 'an\n' ' iterable with at least as many items as there are targets ' 'in the\n' @@ -203,10 +203,10 @@ topics = {'assert': 'The "assert" statement\n' ' If the primary is a mutable sequence object (such as a ' 'list), the\n' ' subscript must yield an integer. If it is negative, the ' - "sequence's\n" + 'sequence’s\n' ' length is added to it. The resulting value must be a ' 'nonnegative\n' - " integer less than the sequence's length, and the sequence is " + ' integer less than the sequence’s length, and the sequence is ' 'asked\n' ' to assign the assigned object to its item with that index. ' 'If the\n' @@ -216,7 +216,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' If the primary is a mapping object (such as a dictionary), ' 'the\n' - " subscript must have a type compatible with the mapping's key " + ' subscript must have a type compatible with the mapping’s key ' 'type,\n' ' and the mapping is then asked to create a key/datum pair ' 'which maps\n' @@ -239,12 +239,12 @@ topics = {'assert': 'The "assert" statement\n' 'expressions are\n' ' evaluated, insofar they are present; defaults are zero and ' 'the\n' - " sequence's length. The bounds should evaluate to integers. " + ' sequence’s length. The bounds should evaluate to integers. ' 'If\n' - " either bound is negative, the sequence's length is added to " + ' either bound is negative, the sequence’s length is added to ' 'it. The\n' ' resulting bounds are clipped to lie between zero and the ' - "sequence's\n" + 'sequence’s\n' ' length, inclusive. Finally, the sequence object is asked to ' 'replace\n' ' the slice with the items of the assigned sequence. The ' @@ -265,7 +265,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'Although the definition of assignment implies that overlaps ' 'between\n' - "the left-hand side and the right-hand side are 'simultaneous' " + 'the left-hand side and the right-hand side are ‘simultaneous’ ' '(for\n' 'example "a, b = b, a" swaps two variables), overlaps *within* ' 'the\n' @@ -464,7 +464,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'All literals correspond to immutable data types, and hence ' 'the\n' - "object's identity is less important than its value. " + 'object’s identity is less important than its value. ' 'Multiple\n' 'evaluations of literals with the same value (either the ' 'same\n' @@ -600,11 +600,11 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'Note: Setting module "__class__" only affects lookups ' 'made using the\n' - ' attribute access syntax -- directly accessing the ' + ' attribute access syntax – directly accessing the ' 'module globals\n' ' (whether by code within the module, or via a reference ' 'to the\n' - " module's globals dictionary) is unaffected.\n" + ' module’s globals dictionary) is unaffected.\n' '\n' 'Changed in version 3.5: "__class__" module attribute is ' 'now writable.\n' @@ -618,13 +618,12 @@ topics = {'assert': 'The "assert" statement\n' 'containing the method (a so-called *descriptor* class) ' 'appears in an\n' '*owner* class (the descriptor must be in either the ' - "owner's class\n" + 'owner’s class\n' 'dictionary or in the class dictionary for one of its ' 'parents). In the\n' - 'examples below, "the attribute" refers to the attribute ' + 'examples below, “the attribute” refers to the attribute ' 'whose name is\n' - "the key of the property in the owner class' " - '"__dict__".\n' + 'the key of the property in the owner class’ "__dict__".\n' '\n' 'object.__get__(self, instance, owner)\n' '\n' @@ -681,8 +680,8 @@ topics = {'assert': 'The "assert" statement\n' '====================\n' '\n' 'In general, a descriptor is an object attribute with ' - '"binding\n' - 'behavior", one whose attribute access has been ' + '“binding\n' + 'behavior”, one whose attribute access has been ' 'overridden by methods\n' 'in the descriptor protocol: "__get__()", "__set__()", ' 'and\n' @@ -692,7 +691,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'The default behavior for attribute access is to get, ' 'set, or delete\n' - "the attribute from an object's dictionary. For instance, " + 'the attribute from an object’s dictionary. For instance, ' '"a.x" has a\n' 'lookup chain starting with "a.__dict__[\'x\']", then\n' '"type(a).__dict__[\'x\']", and continuing through the ' @@ -747,7 +746,7 @@ topics = {'assert': 'The "assert" statement\n' 'does not define "__get__()", then accessing the ' 'attribute will return\n' 'the descriptor object itself unless there is a value in ' - "the object's\n" + 'the object’s\n' 'instance dictionary. If the descriptor defines ' '"__set__()" and/or\n' '"__delete__()", it is a data descriptor; if it defines ' @@ -868,7 +867,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' '* Nonempty *__slots__* does not work for classes derived ' 'from\n' - ' "variable-length" built-in types such as "int", ' + ' “variable-length” built-in types such as "int", ' '"bytes" and "tuple".\n' '\n' '* Any non-string iterable may be assigned to ' @@ -1011,7 +1010,7 @@ topics = {'assert': 'The "assert" statement\n' 'while\n' 'floor division of integers results in an integer; the result is ' 'that\n' - "of mathematical division with the 'floor' function applied to the\n" + 'of mathematical division with the ‘floor’ function applied to the\n' 'result. Division by zero raises the "ZeroDivisionError" ' 'exception.\n' '\n' @@ -1087,10 +1086,10 @@ topics = {'assert': 'The "assert" statement\n' '************\n' '\n' 'Code objects are used by the implementation to ' - 'represent "pseudo-\n' - 'compiled" executable Python code such as a function ' + 'represent “pseudo-\n' + 'compiled” executable Python code such as a function ' 'body. They differ\n' - "from function objects because they don't contain a " + 'from function objects because they don’t contain a ' 'reference to their\n' 'global execution environment. Code objects are ' 'returned by the built-\n' @@ -1121,7 +1120,7 @@ topics = {'assert': 'The "assert" statement\n' 'bltin-null-object': 'The Null Object\n' '***************\n' '\n' - "This object is returned by functions that don't " + 'This object is returned by functions that don’t ' 'explicitly return a\n' 'value. It supports no special operations. There is ' 'exactly one null\n' @@ -1134,7 +1133,7 @@ topics = {'assert': 'The "assert" statement\n' '************\n' '\n' 'Type objects represent the various object types. An ' - "object's type is\n" + 'object’s type is\n' 'accessed by the built-in function "type()". There are ' 'no special\n' 'operations on types. The standard module "types" ' @@ -1217,7 +1216,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'object.__call__(self[, args...])\n' '\n' - ' Called when the instance is "called" as a function; if ' + ' Called when the instance is “called” as a function; if ' 'this method\n' ' is defined, "x(arg1, arg2, ...)" is a shorthand for\n' ' "x.__call__(arg1, arg2, ...)".\n', @@ -1277,7 +1276,7 @@ topics = {'assert': 'The "assert" statement\n' 'values are calculated, once, when the function is defined; thus, a\n' 'mutable object such as a list or dictionary used as default value ' 'will\n' - "be shared by all calls that don't specify an argument value for " + 'be shared by all calls that don’t specify an argument value for ' 'the\n' 'corresponding slot; this should usually be avoided.) If there are ' 'any\n' @@ -1290,7 +1289,7 @@ topics = {'assert': 'The "assert" statement\n' '**CPython implementation detail:** An implementation may provide\n' 'built-in functions whose positional parameters do not have names, ' 'even\n' - "if they are 'named' for the purpose of documentation, and which\n" + 'if they are ‘named’ for the purpose of documentation, and which\n' 'therefore cannot be supplied by keyword. In CPython, this is the ' 'case\n' 'for functions implemented in C that use "PyArg_ParseTuple()" to ' @@ -1322,16 +1321,17 @@ topics = {'assert': 'The "assert" statement\n' 'must evaluate to an *iterable*. Elements from these iterables are\n' 'treated as if they were additional positional arguments. For the ' 'call\n' - '"f(x1, x2, *y, x3, x4)", if *y* evaluates to a sequence *y1*, ...,\n' - '*yM*, this is equivalent to a call with M+4 positional arguments ' - '*x1*,\n' - '*x2*, *y1*, ..., *yM*, *x3*, *x4*.\n' + '"f(x1, x2, *y, x3, x4)", if *y* evaluates to a sequence *y1*, …, ' + '*yM*,\n' + 'this is equivalent to a call with M+4 positional arguments *x1*, ' + '*x2*,\n' + '*y1*, …, *yM*, *x3*, *x4*.\n' '\n' 'A consequence of this is that although the "*expression" syntax ' 'may\n' 'appear *after* explicit keyword arguments, it is processed ' '*before*\n' - 'the keyword arguments (and any "**expression" arguments -- see ' + 'the keyword arguments (and any "**expression" arguments – see ' 'below).\n' 'So:\n' '\n' @@ -1378,7 +1378,7 @@ topics = {'assert': 'The "assert" statement\n' 'exception. How this value is computed depends on the type of the\n' 'callable object.\n' '\n' - 'If it is---\n' + 'If it is—\n' '\n' 'a user-defined function:\n' ' The code block for the function is executed, passing it the\n' @@ -1437,10 +1437,10 @@ topics = {'assert': 'The "assert" statement\n' ' class Foo(object):\n' ' pass\n' '\n' - "The class's suite is then executed in a new execution frame (see\n" + 'The class’s suite is then executed in a new execution frame (see\n' 'Naming and binding), using a newly created local namespace and the\n' 'original global namespace. (Usually, the suite contains mostly\n' - "function definitions.) When the class's suite finishes execution, " + 'function definitions.) When the class’s suite finishes execution, ' 'its\n' 'execution frame is discarded but its local namespace is saved. [4] ' 'A\n' @@ -1452,7 +1452,7 @@ topics = {'assert': 'The "assert" statement\n' 'namespace.\n' '\n' 'The order in which attributes are defined in the class body is\n' - 'preserved in the new class\'s "__dict__". Note that this is ' + 'preserved in the new class’s "__dict__". Note that this is ' 'reliable\n' 'only right after the class is created and only for classes that ' 'were\n' @@ -1476,13 +1476,13 @@ topics = {'assert': 'The "assert" statement\n' 'for\n' 'function decorators. The result is then bound to the class name.\n' '\n' - "**Programmer's note:** Variables defined in the class definition " + '**Programmer’s note:** Variables defined in the class definition ' 'are\n' 'class attributes; they are shared by instances. Instance ' 'attributes\n' 'can be set in a method with "self.name = value". Both class and\n' 'instance attributes are accessible through the notation ' - '""self.name"",\n' + '“"self.name"”,\n' 'and an instance attribute hides a class attribute with the same ' 'name\n' 'when accessed in this way. Class attributes can be used as ' @@ -1519,15 +1519,15 @@ topics = {'assert': 'The "assert" statement\n' 'y" is\n' 'found to be false).\n' '\n' - 'Formally, if *a*, *b*, *c*, ..., *y*, *z* are expressions and ' + 'Formally, if *a*, *b*, *c*, …, *y*, *z* are expressions and ' '*op1*,\n' - '*op2*, ..., *opN* are comparison operators, then "a op1 b op2 ' - 'c ... y\n' + '*op2*, …, *opN* are comparison operators, then "a op1 b op2 c ' + '... y\n' 'opN z" is equivalent to "a op1 b and b op2 c and ... y opN ' 'z", except\n' 'that each expression is evaluated at most once.\n' '\n' - 'Note that "a op1 b op2 c" doesn\'t imply any kind of ' + 'Note that "a op1 b op2 c" doesn’t imply any kind of ' 'comparison between\n' '*a* and *c*, so that, e.g., "x < y > z" is perfectly legal ' '(though\n' @@ -1548,7 +1548,7 @@ topics = {'assert': 'The "assert" statement\n' 'rather\n' 'abstract notion in Python: For example, there is no canonical ' 'access\n' - "method for an object's value. Also, there is no requirement " + 'method for an object’s value. Also, there is no requirement ' 'that the\n' 'value of an object should be constructed in a particular way, ' 'e.g.\n' @@ -1602,7 +1602,7 @@ topics = {'assert': 'The "assert" statement\n' 'most\n' 'important built-in types.\n' '\n' - '* Numbers of built-in numeric types (Numeric Types --- int, ' + '* Numbers of built-in numeric types (Numeric Types — int, ' 'float,\n' ' complex) and of the standard library types ' '"fractions.Fraction" and\n' @@ -1889,9 +1889,9 @@ topics = {'assert': 'The "assert" statement\n' 'compound\n' 'statements.\n' '\n' - "A compound statement consists of one or more 'clauses.' A " + 'A compound statement consists of one or more ‘clauses.’ A ' 'clause\n' - "consists of a header and a 'suite.' The clause headers of a\n" + 'consists of a header and a ‘suite.’ The clause headers of a\n' 'particular compound statement are all at the same indentation ' 'level.\n' 'Each clause header begins with a uniquely identifying keyword ' @@ -1899,12 +1899,12 @@ topics = {'assert': 'The "assert" statement\n' 'with a colon. A suite is a group of statements controlled by a\n' 'clause. A suite can be one or more semicolon-separated simple\n' 'statements on the same line as the header, following the ' - "header's\n" + 'header’s\n' 'colon, or it can be one or more indented statements on ' 'subsequent\n' 'lines. Only the latter form of a suite can contain nested ' 'compound\n' - "statements; the following is illegal, mostly because it wouldn't " + 'statements; the following is illegal, mostly because it wouldn’t ' 'be\n' 'clear to which "if" clause a following "else" clause would ' 'belong:\n' @@ -1941,7 +1941,7 @@ topics = {'assert': 'The "assert" statement\n' '"DEDENT". Also note that optional continuation clauses always ' 'begin\n' 'with a keyword that cannot start a statement, thus there are no\n' - 'ambiguities (the \'dangling "else"\' problem is solved in Python ' + 'ambiguities (the ‘dangling "else"’ problem is solved in Python ' 'by\n' 'requiring nested "if" statements to be indented).\n' '\n' @@ -1992,7 +1992,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'A "break" statement executed in the first suite terminates the ' 'loop\n' - 'without executing the "else" clause\'s suite. A "continue" ' + 'without executing the "else" clause’s suite. A "continue" ' 'statement\n' 'executed in the first suite skips the rest of the suite and goes ' 'back\n' @@ -2031,7 +2031,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'A "break" statement executed in the first suite terminates the ' 'loop\n' - 'without executing the "else" clause\'s suite. A "continue" ' + 'without executing the "else" clause’s suite. A "continue" ' 'statement\n' 'executed in the first suite skips the rest of the suite and ' 'continues\n' @@ -2058,7 +2058,7 @@ topics = {'assert': 'The "assert" statement\n' 'to at\n' 'all by the loop. Hint: the built-in function "range()" returns ' 'an\n' - "iterator of integers suitable to emulate the effect of Pascal's " + 'iterator of integers suitable to emulate the effect of Pascal’s ' '"for i\n' ':= a to b do"; e.g., "list(range(3))" returns the list "[0, 1, ' '2]".\n' @@ -2120,7 +2120,7 @@ topics = {'assert': 'The "assert" statement\n' 'expression\n' 'is evaluated, and the clause matches the exception if the ' 'resulting\n' - 'object is "compatible" with the exception. An object is ' + 'object is “compatible” with the exception. An object is ' 'compatible\n' 'with an exception if it is the class or a base class of the ' 'exception\n' @@ -2147,7 +2147,7 @@ topics = {'assert': 'The "assert" statement\n' 'assigned to\n' 'the target specified after the "as" keyword in that except ' 'clause, if\n' - "present, and the except clause's suite is executed. All except\n" + 'present, and the except clause’s suite is executed. All except\n' 'clauses must have an executable block. When the end of this ' 'block is\n' 'reached, execution continues normally after the entire try ' @@ -2183,7 +2183,7 @@ topics = {'assert': 'The "assert" statement\n' 'alive\n' 'until the next garbage collection occurs.\n' '\n' - "Before an except clause's suite is executed, details about the\n" + 'Before an except clause’s suite is executed, details about the\n' 'exception are stored in the "sys" module and can be accessed ' 'via\n' '"sys.exc_info()". "sys.exc_info()" returns a 3-tuple consisting ' @@ -2204,8 +2204,8 @@ topics = {'assert': 'The "assert" statement\n' 'are\n' 'not handled by the preceding "except" clauses.\n' '\n' - 'If "finally" is present, it specifies a \'cleanup\' handler. ' - 'The "try"\n' + 'If "finally" is present, it specifies a ‘cleanup’ handler. The ' + '"try"\n' 'clause is executed, including any "except" and "else" clauses. ' 'If an\n' 'exception occurs in any of the clauses and is not handled, the\n' @@ -2236,13 +2236,12 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'When a "return", "break" or "continue" statement is executed in ' 'the\n' - '"try" suite of a "try"..."finally" statement, the "finally" ' - 'clause is\n' - 'also executed \'on the way out.\' A "continue" statement is ' + '"try" suite of a "try"…"finally" statement, the "finally" clause ' + 'is\n' + 'also executed ‘on the way out.’ A "continue" statement is ' 'illegal in\n' 'the "finally" clause. (The reason is a problem with the current\n' - 'implementation --- this restriction may be lifted in the ' - 'future).\n' + 'implementation — this restriction may be lifted in the future).\n' '\n' 'The return value of a function is determined by the last ' '"return"\n' @@ -2274,14 +2273,14 @@ topics = {'assert': 'The "assert" statement\n' 'with\n' 'methods defined by a context manager (see section With ' 'Statement\n' - 'Context Managers). This allows common ' - '"try"..."except"..."finally"\n' - 'usage patterns to be encapsulated for convenient reuse.\n' + 'Context Managers). This allows common "try"…"except"…"finally" ' + 'usage\n' + 'patterns to be encapsulated for convenient reuse.\n' '\n' ' with_stmt ::= "with" with_item ("," with_item)* ":" suite\n' ' with_item ::= expression ["as" target]\n' '\n' - 'The execution of the "with" statement with one "item" proceeds ' + 'The execution of the "with" statement with one “item” proceeds ' 'as\n' 'follows:\n' '\n' @@ -2289,9 +2288,9 @@ topics = {'assert': 'The "assert" statement\n' '"with_item")\n' ' is evaluated to obtain a context manager.\n' '\n' - '2. The context manager\'s "__exit__()" is loaded for later use.\n' + '2. The context manager’s "__exit__()" is loaded for later use.\n' '\n' - '3. The context manager\'s "__enter__()" method is invoked.\n' + '3. The context manager’s "__enter__()" method is invoked.\n' '\n' '4. If a target was included in the "with" statement, the return\n' ' value from "__enter__()" is assigned to it.\n' @@ -2308,8 +2307,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' '5. The suite is executed.\n' '\n' - '6. The context manager\'s "__exit__()" method is invoked. If ' - 'an\n' + '6. The context manager’s "__exit__()" method is invoked. If an\n' ' exception caused the suite to be exited, its type, value, ' 'and\n' ' traceback are passed as arguments to "__exit__()". Otherwise, ' @@ -2349,7 +2347,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'See also:\n' '\n' - ' **PEP 343** - The "with" statement\n' + ' **PEP 343** - The “with” statement\n' ' The specification, background, and examples for the Python ' '"with"\n' ' statement.\n' @@ -2418,25 +2416,24 @@ topics = {'assert': 'The "assert" statement\n' '"func".\n' '\n' 'When one or more *parameters* have the form *parameter* "="\n' - '*expression*, the function is said to have "default parameter ' - 'values."\n' + '*expression*, the function is said to have “default parameter ' + 'values.”\n' 'For a parameter with a default value, the corresponding ' '*argument* may\n' - "be omitted from a call, in which case the parameter's default " + 'be omitted from a call, in which case the parameter’s default ' 'value is\n' 'substituted. If a parameter has a default value, all following\n' - 'parameters up until the ""*"" must also have a default value --- ' - 'this\n' - 'is a syntactic restriction that is not expressed by the ' - 'grammar.\n' + 'parameters up until the “"*"” must also have a default value — ' + 'this is\n' + 'a syntactic restriction that is not expressed by the grammar.\n' '\n' '**Default parameter values are evaluated from left to right when ' 'the\n' 'function definition is executed.** This means that the ' 'expression is\n' 'evaluated once, when the function is defined, and that the same ' - '"pre-\n' - 'computed" value is used for each call. This is especially ' + '“pre-\n' + 'computed” value is used for each call. This is especially ' 'important\n' 'to understand when a default parameter is a mutable object, such ' 'as a\n' @@ -2462,26 +2459,26 @@ topics = {'assert': 'The "assert" statement\n' 'mentioned in\n' 'the parameter list, either from position arguments, from ' 'keyword\n' - 'arguments, or from default values. If the form ""*identifier"" ' + 'arguments, or from default values. If the form “"*identifier"” ' 'is\n' 'present, it is initialized to a tuple receiving any excess ' 'positional\n' 'parameters, defaulting to the empty tuple. If the form\n' - '""**identifier"" is present, it is initialized to a new ordered\n' + '“"**identifier"” is present, it is initialized to a new ordered\n' 'mapping receiving any excess keyword arguments, defaulting to a ' 'new\n' - 'empty mapping of the same type. Parameters after ""*"" or\n' - '""*identifier"" are keyword-only parameters and may only be ' + 'empty mapping of the same type. Parameters after “"*"” or\n' + '“"*identifier"” are keyword-only parameters and may only be ' 'passed\n' 'used keyword arguments.\n' '\n' - 'Parameters may have annotations of the form "": expression"" ' + 'Parameters may have annotations of the form “": expression"” ' 'following\n' 'the parameter name. Any parameter may have an annotation even ' 'those\n' 'of the form "*identifier" or "**identifier". Functions may ' 'have\n' - '"return" annotation of the form ""-> expression"" after the ' + '“return” annotation of the form “"-> expression"” after the ' 'parameter\n' 'list. These annotations can be any valid Python expression and ' 'are\n' @@ -2491,7 +2488,7 @@ topics = {'assert': 'The "assert" statement\n' 'code.\n' 'The presence of annotations does not change the semantics of a\n' 'function. The annotation values are available as values of a\n' - "dictionary keyed by the parameters' names in the " + 'dictionary keyed by the parameters’ names in the ' '"__annotations__"\n' 'attribute of the function object.\n' '\n' @@ -2502,16 +2499,16 @@ topics = {'assert': 'The "assert" statement\n' 'lambda\n' 'expression is merely a shorthand for a simplified function ' 'definition;\n' - 'a function defined in a ""def"" statement can be passed around ' + 'a function defined in a “"def"” statement can be passed around ' 'or\n' 'assigned to another name just like a function defined by a ' 'lambda\n' - 'expression. The ""def"" form is actually more powerful since ' + 'expression. The “"def"” form is actually more powerful since ' 'it\n' 'allows the execution of multiple statements and annotations.\n' '\n' - "**Programmer's note:** Functions are first-class objects. A " - '""def""\n' + '**Programmer’s note:** Functions are first-class objects. A ' + '“"def"”\n' 'statement executed inside a function definition defines a local\n' 'function that can be returned or passed around. Free variables ' 'used\n' @@ -2555,12 +2552,12 @@ topics = {'assert': 'The "assert" statement\n' ' class Foo(object):\n' ' pass\n' '\n' - "The class's suite is then executed in a new execution frame " + 'The class’s suite is then executed in a new execution frame ' '(see\n' 'Naming and binding), using a newly created local namespace and ' 'the\n' 'original global namespace. (Usually, the suite contains mostly\n' - "function definitions.) When the class's suite finishes " + 'function definitions.) When the class’s suite finishes ' 'execution, its\n' 'execution frame is discarded but its local namespace is saved. ' '[4] A\n' @@ -2573,7 +2570,7 @@ topics = {'assert': 'The "assert" statement\n' 'namespace.\n' '\n' 'The order in which attributes are defined in the class body is\n' - 'preserved in the new class\'s "__dict__". Note that this is ' + 'preserved in the new class’s "__dict__". Note that this is ' 'reliable\n' 'only right after the class is created and only for classes that ' 'were\n' @@ -2598,14 +2595,14 @@ topics = {'assert': 'The "assert" statement\n' 'function decorators. The result is then bound to the class ' 'name.\n' '\n' - "**Programmer's note:** Variables defined in the class definition " + '**Programmer’s note:** Variables defined in the class definition ' 'are\n' 'class attributes; they are shared by instances. Instance ' 'attributes\n' 'can be set in a method with "self.name = value". Both class ' 'and\n' 'instance attributes are accessible through the notation ' - '""self.name"",\n' + '“"self.name"”,\n' 'and an instance attribute hides a class attribute with the same ' 'name\n' 'when accessed in this way. Class attributes can be used as ' @@ -2741,20 +2738,19 @@ topics = {'assert': 'The "assert" statement\n' ' exception. That new exception causes the old one to be ' 'lost.\n' '\n' - '[2] Currently, control "flows off the end" except in the case ' + '[2] Currently, control “flows off the end” except in the case ' 'of\n' ' an exception or the execution of a "return", "continue", or\n' ' "break" statement.\n' '\n' '[3] A string literal appearing as the first statement in the\n' - ' function body is transformed into the function\'s "__doc__"\n' - " attribute and therefore the function's *docstring*.\n" + ' function body is transformed into the function’s "__doc__"\n' + ' attribute and therefore the function’s *docstring*.\n' '\n' '[4] A string literal appearing as the first statement in the ' 'class\n' - ' body is transformed into the namespace\'s "__doc__" item ' - 'and\n' - " therefore the class's *docstring*.\n", + ' body is transformed into the namespace’s "__doc__" item and\n' + ' therefore the class’s *docstring*.\n', 'context-managers': 'With Statement Context Managers\n' '*******************************\n' '\n' @@ -2784,7 +2780,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' Enter the runtime context related to this object. The ' '"with"\n' - " statement will bind this method's return value to the " + ' statement will bind this method’s return value to the ' 'target(s)\n' ' specified in the "as" clause of the statement, if ' 'any.\n' @@ -2809,11 +2805,11 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' Note that "__exit__()" methods should not reraise the ' 'passed-in\n' - " exception; this is the caller's responsibility.\n" + ' exception; this is the caller’s responsibility.\n' '\n' 'See also:\n' '\n' - ' **PEP 343** - The "with" statement\n' + ' **PEP 343** - The “with” statement\n' ' The specification, background, and examples for the ' 'Python "with"\n' ' statement.\n', @@ -2839,7 +2835,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'When a description of an arithmetic operator below uses the ' 'phrase\n' - '"the numeric arguments are converted to a common type," this ' + '“the numeric arguments are converted to a common type,” this ' 'means\n' 'that the operator implementation for built-in types works as ' 'follows:\n' @@ -2857,7 +2853,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'Some additional rules apply for certain operators (e.g., a ' 'string as a\n' - "left argument to the '%' operator). Extensions must define " + 'left argument to the ‘%’ operator). Extensions must define ' 'their own\n' 'conversion behavior.\n', 'customization': 'Basic customization\n' @@ -2881,7 +2877,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' Typical implementations create a new instance of the ' 'class by\n' - ' invoking the superclass\'s "__new__()" method using\n' + ' invoking the superclass’s "__new__()" method using\n' ' "super().__new__(cls[, ...])" with appropriate arguments ' 'and then\n' ' modifying the newly-created instance as necessary before ' @@ -2890,7 +2886,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' If "__new__()" returns an instance of *cls*, then the ' 'new\n' - ' instance\'s "__init__()" method will be invoked like\n' + ' instance’s "__init__()" method will be invoked like\n' ' "__init__(self[, ...])", where *self* is the new ' 'instance and the\n' ' remaining arguments are the same as were passed to ' @@ -2898,7 +2894,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' If "__new__()" does not return an instance of *cls*, ' 'then the new\n' - ' instance\'s "__init__()" method will not be invoked.\n' + ' instance’s "__init__()" method will not be invoked.\n' '\n' ' "__new__()" is intended mainly to allow subclasses of ' 'immutable\n' @@ -2916,7 +2912,7 @@ topics = {'assert': 'The "assert" statement\n' 'those\n' ' passed to the class constructor expression. If a base ' 'class has an\n' - ' "__init__()" method, the derived class\'s "__init__()" ' + ' "__init__()" method, the derived class’s "__init__()" ' 'method, if\n' ' any, must explicitly call it to ensure proper ' 'initialization of the\n' @@ -2937,7 +2933,7 @@ topics = {'assert': 'The "assert" statement\n' 'is also\n' ' called a finalizer or (improperly) a destructor. If a ' 'base class\n' - ' has a "__del__()" method, the derived class\'s ' + ' has a "__del__()" method, the derived class’s ' '"__del__()" method,\n' ' if any, must explicitly call it to ensure proper ' 'deletion of the\n' @@ -2958,11 +2954,11 @@ topics = {'assert': 'The "assert" statement\n' 'for\n' ' objects that still exist when the interpreter exits.\n' '\n' - ' Note: "del x" doesn\'t directly call "x.__del__()" --- ' - 'the former\n' + ' Note: "del x" doesn’t directly call "x.__del__()" — the ' + 'former\n' ' decrements the reference count for "x" by one, and the ' 'latter is\n' - ' only called when "x"\'s reference count reaches zero.\n' + ' only called when "x"’s reference count reaches zero.\n' '\n' ' **CPython implementation detail:** It is possible for a ' 'reference\n' @@ -2974,7 +2970,7 @@ topics = {'assert': 'The "assert" statement\n' 'reference\n' ' cycles is when an exception has been caught in a local ' 'variable.\n' - " The frame's locals then reference the exception, which " + ' The frame’s locals then reference the exception, which ' 'references\n' ' its own traceback, which references the locals of all ' 'frames caught\n' @@ -3021,7 +3017,7 @@ topics = {'assert': 'The "assert" statement\n' 'object.__repr__(self)\n' '\n' ' Called by the "repr()" built-in function to compute the ' - '"official"\n' + '“official”\n' ' string representation of an object. If at all possible, ' 'this\n' ' should look like a valid Python expression that could be ' @@ -3035,7 +3031,7 @@ topics = {'assert': 'The "assert" statement\n' ' value must be a string object. If a class defines ' '"__repr__()" but\n' ' not "__str__()", then "__repr__()" is also used when an ' - '"informal"\n' + '“informal”\n' ' string representation of instances of that class is ' 'required.\n' '\n' @@ -3047,7 +3043,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' Called by "str(object)" and the built-in functions ' '"format()" and\n' - ' "print()" to compute the "informal" or nicely printable ' + ' "print()" to compute the “informal” or nicely printable ' 'string\n' ' representation of an object. The return value must be a ' 'string\n' @@ -3075,7 +3071,7 @@ topics = {'assert': 'The "assert" statement\n' 'extension,\n' ' evaluation of formatted string literals and the ' '"str.format()"\n' - ' method, to produce a "formatted" string representation ' + ' method, to produce a “formatted” string representation ' 'of an\n' ' object. The "format_spec" argument is a string that ' 'contains a\n' @@ -3106,7 +3102,7 @@ topics = {'assert': 'The "assert" statement\n' 'object.__gt__(self, other)\n' 'object.__ge__(self, other)\n' '\n' - ' These are the so-called "rich comparison" methods. The\n' + ' These are the so-called “rich comparison” methods. The\n' ' correspondence between operator symbols and method names ' 'is as\n' ' follows: "x>> import pdb\n' @@ -3356,11 +3352,11 @@ topics = {'assert': 'The "assert" statement\n' 'post-\n' 'mortem debugging (or after normal exit of the program), pdb ' 'will\n' - "restart the program. Automatic restarting preserves pdb's state " + 'restart the program. Automatic restarting preserves pdb’s state ' '(such\n' 'as breakpoints) and in most cases is more useful than quitting ' 'the\n' - "debugger upon program's exit.\n" + 'debugger upon program’s exit.\n' '\n' 'New in version 3.2: "pdb.py" now accepts a "-c" option that ' 'executes\n' @@ -3539,7 +3535,7 @@ topics = {'assert': 'The "assert" statement\n' 'the last command was a "list" command, the next 11 lines are ' 'listed.\n' '\n' - "Commands that the debugger doesn't recognize are assumed to be " + 'Commands that the debugger doesn’t recognize are assumed to be ' 'Python\n' 'statements and are executed in the context of the program being\n' 'debugged. Python statements can also be prefixed with an ' @@ -3550,7 +3546,7 @@ topics = {'assert': 'The "assert" statement\n' 'function.\n' 'When an exception occurs in such a statement, the exception name ' 'is\n' - "printed but the debugger's state is not changed.\n" + 'printed but the debugger’s state is not changed.\n' '\n' 'The debugger supports aliases. Aliases can have parameters ' 'which\n' @@ -3567,7 +3563,7 @@ topics = {'assert': 'The "assert" statement\n' 'first\n' '";;" pair, even if it is in the middle of a quoted string.\n' '\n' - 'If a file ".pdbrc" exists in the user\'s home directory or in ' + 'If a file ".pdbrc" exists in the user’s home directory or in ' 'the\n' 'current directory, it is read in and executed as if it had been ' 'typed\n' @@ -3623,7 +3619,7 @@ topics = {'assert': 'The "assert" statement\n' 'prefixed\n' ' with a filename and a colon, to specify a breakpoint in ' 'another\n' - " file (probably one that hasn't been loaded yet). The file " + ' file (probably one that hasn’t been loaded yet). The file ' 'is\n' ' searched on "sys.path". Note that each breakpoint is ' 'assigned a\n' @@ -3731,7 +3727,7 @@ topics = {'assert': 'The "assert" statement\n' 'which\n' ' list to execute.\n' '\n' - " If you use the 'silent' command in the command list, the " + ' If you use the ‘silent’ command in the command list, the ' 'usual\n' ' message about stopping at a breakpoint is not printed. This ' 'may be\n' @@ -3790,13 +3786,13 @@ topics = {'assert': 'The "assert" statement\n' 'the\n' ' bottom-most frame. This lets you jump back and execute code ' 'again,\n' - " or jump forward to skip code that you don't want to run.\n" + ' or jump forward to skip code that you don’t want to run.\n' '\n' - ' It should be noted that not all jumps are allowed -- for ' - 'instance\n' - ' it is not possible to jump into the middle of a "for" loop or ' - 'out\n' - ' of a "finally" clause.\n' + ' It should be noted that not all jumps are allowed – for ' + 'instance it\n' + ' is not possible to jump into the middle of a "for" loop or ' + 'out of a\n' + ' "finally" clause.\n' '\n' 'l(ist) [first[, last]]\n' '\n' @@ -3839,8 +3835,8 @@ topics = {'assert': 'The "assert" statement\n' ' value.\n' '\n' ' Note: "print()" can also be used, but is not a debugger ' - 'command\n' - ' --- this executes the Python "print()" function.\n' + 'command —\n' + ' this executes the Python "print()" function.\n' '\n' 'pp expression\n' '\n' @@ -4010,7 +4006,7 @@ topics = {'assert': 'The "assert" statement\n' 'dictionary:\n' 'each key object is used as a key into the dictionary to store the\n' 'corresponding datum. This means that you can specify the same key\n' - "multiple times in the key/datum list, and the final dictionary's " + 'multiple times in the key/datum list, and the final dictionary’s ' 'value\n' 'for that key will be the last one given.\n' '\n' @@ -4024,7 +4020,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'A dict comprehension, in contrast to list and set comprehensions,\n' 'needs two expressions separated with a colon followed by the usual\n' - '"for" and "if" clauses. When the comprehension is run, the ' + '“for” and “if” clauses. When the comprehension is run, the ' 'resulting\n' 'key and value elements are inserted in the new dictionary in the ' 'order\n' @@ -4098,7 +4094,7 @@ topics = {'assert': 'The "assert" statement\n' 'error (such as division by zero). A Python program can also\n' 'explicitly raise an exception with the "raise" statement. ' 'Exception\n' - 'handlers are specified with the "try" ... "except" statement. ' + 'handlers are specified with the "try" … "except" statement. ' 'The\n' '"finally" clause of such a statement can be used to specify ' 'cleanup\n' @@ -4106,7 +4102,7 @@ topics = {'assert': 'The "assert" statement\n' 'whether an\n' 'exception occurred or not in the preceding code.\n' '\n' - 'Python uses the "termination" model of error handling: an ' + 'Python uses the “termination” model of error handling: an ' 'exception\n' 'handler can find out what happened and continue execution at ' 'an outer\n' @@ -4174,7 +4170,7 @@ topics = {'assert': 'The "assert" statement\n' 'argument to the interpreter) is a code block. A script command ' '(a\n' 'command specified on the interpreter command line with the ' - "'**-c**'\n" + '‘**-c**’\n' 'option) is a code block. The string argument passed to the ' 'built-in\n' 'functions "eval()" and "exec()" is a code block.\n' @@ -4183,7 +4179,7 @@ topics = {'assert': 'The "assert" statement\n' 'contains\n' 'some administrative information (used for debugging) and ' 'determines\n' - "where and how execution continues after the code block's " + 'where and how execution continues after the code block’s ' 'execution has\n' 'completed.\n' '\n' @@ -4258,7 +4254,7 @@ topics = {'assert': 'The "assert" statement\n' 'nearest\n' 'enclosing scope. The set of all such scopes visible to a code ' 'block\n' - "is called the block's *environment*.\n" + 'is called the block’s *environment*.\n' '\n' 'When a name is not found at all, a "NameError" exception is ' 'raised. If\n' @@ -4336,7 +4332,7 @@ topics = {'assert': 'The "assert" statement\n' 'the class. The scope of names defined in a class block is ' 'limited to\n' 'the class block; it does not extend to the code blocks of ' - 'methods --\n' + 'methods –\n' 'this includes comprehensions and generator expressions since ' 'they are\n' 'implemented using a function scope. This means that the ' @@ -4364,7 +4360,7 @@ topics = {'assert': 'The "assert" statement\n' 'global\n' 'namespace; this should be a dictionary or a module (in the ' 'latter case\n' - "the module's dictionary is used). By default, when in the " + 'the module’s dictionary is used). By default, when in the ' '"__main__"\n' 'module, "__builtins__" is the built-in module "builtins"; when ' 'in any\n' @@ -4419,7 +4415,7 @@ topics = {'assert': 'The "assert" statement\n' 'error (such as division by zero). A Python program can also\n' 'explicitly raise an exception with the "raise" statement. ' 'Exception\n' - 'handlers are specified with the "try" ... "except" statement. ' + 'handlers are specified with the "try" … "except" statement. ' 'The\n' '"finally" clause of such a statement can be used to specify ' 'cleanup\n' @@ -4427,7 +4423,7 @@ topics = {'assert': 'The "assert" statement\n' 'whether an\n' 'exception occurred or not in the preceding code.\n' '\n' - 'Python uses the "termination" model of error handling: an ' + 'Python uses the “termination” model of error handling: an ' 'exception\n' 'handler can find out what happened and continue execution at an ' 'outer\n' @@ -4507,7 +4503,7 @@ topics = {'assert': 'The "assert" statement\n' '(a.k.a. a\n' '*singleton*); it is optional in all other cases. A single ' 'expression\n' - "without a trailing comma doesn't create a tuple, but rather " + 'without a trailing comma doesn’t create a tuple, but rather ' 'yields the\n' 'value of that expression. (To create an empty tuple, use an ' 'empty pair\n' @@ -4568,8 +4564,7 @@ topics = {'assert': 'The "assert" statement\n' 'terminates.\n' '\n' 'A "break" statement executed in the first suite terminates the loop\n' - 'without executing the "else" clause\'s suite. A "continue" ' - 'statement\n' + 'without executing the "else" clause’s suite. A "continue" statement\n' 'executed in the first suite skips the rest of the suite and ' 'continues\n' 'with the next item, or with the "else" clause if there is no next\n' @@ -4591,7 +4586,7 @@ topics = {'assert': 'The "assert" statement\n' 'Names in the target list are not deleted when the loop is finished,\n' 'but if the sequence is empty, they will not have been assigned to at\n' 'all by the loop. Hint: the built-in function "range()" returns an\n' - 'iterator of integers suitable to emulate the effect of Pascal\'s "for ' + 'iterator of integers suitable to emulate the effect of Pascal’s "for ' 'i\n' ':= a to b do"; e.g., "list(range(3))" returns the list "[0, 1, 2]".\n' '\n' @@ -4627,7 +4622,7 @@ topics = {'assert': 'The "assert" statement\n' 'are\n' 'differences.\n' '\n' - 'Format strings contain "replacement fields" surrounded by ' + 'Format strings contain “replacement fields” surrounded by ' 'curly braces\n' '"{}". Anything that is not contained in braces is ' 'considered literal\n' @@ -4670,29 +4665,27 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'The *field_name* itself begins with an *arg_name* that is ' 'either a\n' - "number or a keyword. If it's a number, it refers to a " + 'number or a keyword. If it’s a number, it refers to a ' 'positional\n' - "argument, and if it's a keyword, it refers to a named " + 'argument, and if it’s a keyword, it refers to a named ' 'keyword\n' 'argument. If the numerical arg_names in a format string ' 'are 0, 1, 2,\n' - '... in sequence, they can all be omitted (not just some) ' - 'and the\n' - 'numbers 0, 1, 2, ... will be automatically inserted in that ' - 'order.\n' - 'Because *arg_name* is not quote-delimited, it is not ' - 'possible to\n' - 'specify arbitrary dictionary keys (e.g., the strings ' - '"\'10\'" or\n' - '"\':-]\'") within a format string. The *arg_name* can be ' - 'followed by any\n' - 'number of index or attribute expressions. An expression of ' - 'the form\n' - '"\'.name\'" selects the named attribute using "getattr()", ' - 'while an\n' - 'expression of the form "\'[index]\'" does an index lookup ' - 'using\n' - '"__getitem__()".\n' + '… in sequence, they can all be omitted (not just some) and ' + 'the numbers\n' + '0, 1, 2, … will be automatically inserted in that order. ' + 'Because\n' + '*arg_name* is not quote-delimited, it is not possible to ' + 'specify\n' + 'arbitrary dictionary keys (e.g., the strings "\'10\'" or ' + '"\':-]\'") within\n' + 'a format string. The *arg_name* can be followed by any ' + 'number of index\n' + 'or attribute expressions. An expression of the form ' + '"\'.name\'" selects\n' + 'the named attribute using "getattr()", while an expression ' + 'of the form\n' + '"\'[index]\'" does an index lookup using "__getitem__()".\n' '\n' 'Changed in version 3.1: The positional argument specifiers ' 'can be\n' @@ -4747,7 +4740,7 @@ topics = {'assert': 'The "assert" statement\n' 'alignment,\n' 'padding, decimal precision and so on. Each value type can ' 'define its\n' - 'own "formatting mini-language" or interpretation of the ' + 'own “formatting mini-language” or interpretation of the ' '*format_spec*.\n' '\n' 'Most built-in types support a common formatting ' @@ -4773,7 +4766,7 @@ topics = {'assert': 'The "assert" statement\n' 'Format Specification Mini-Language\n' '==================================\n' '\n' - '"Format specifications" are used within replacement fields ' + '“Format specifications” are used within replacement fields ' 'contained\n' 'within a format string to define how individual values are ' 'presented\n' @@ -4815,13 +4808,13 @@ topics = {'assert': 'The "assert" statement\n' 'character that can be any character and defaults to a space ' 'if\n' 'omitted. It is not possible to use a literal curly brace ' - '(""{"" or\n' - '""}"") as the *fill* character in a formatted string ' + '(“"{"” or\n' + '“"}"”) as the *fill* character in a formatted string ' 'literal or when\n' 'using the "str.format()" method. However, it is possible ' 'to insert a\n' 'curly brace with a nested replacement field. This ' - "limitation doesn't\n" + 'limitation doesn’t\n' 'affect the "format()" function.\n' '\n' 'The meaning of the various alignment options is as ' @@ -4850,10 +4843,10 @@ topics = {'assert': 'The "assert" statement\n' 'the sign (if any) |\n' ' | | but before the digits. This is used for ' 'printing fields |\n' - " | | in the form '+000000120'. This alignment " + ' | | in the form ‘+000000120’. This alignment ' 'option is only |\n' ' | | valid for numeric types. It becomes the ' - "default when '0' |\n" + 'default when ‘0’ |\n' ' | | immediately precedes the field ' 'width. |\n' ' ' @@ -4902,7 +4895,7 @@ topics = {'assert': 'The "assert" statement\n' ' ' '+-----------+------------------------------------------------------------+\n' '\n' - 'The "\'#\'" option causes the "alternate form" to be used ' + 'The "\'#\'" option causes the “alternate form” to be used ' 'for the\n' 'conversion. The alternate form is defined differently for ' 'different\n' @@ -5071,7 +5064,7 @@ topics = {'assert': 'The "assert" statement\n' '+===========+============================================================+\n' ' | "\'e\'" | Exponent notation. Prints the number in ' 'scientific |\n' - " | | notation using the letter 'e' to indicate " + ' | | notation using the letter ‘e’ to indicate ' 'the exponent. |\n' ' | | The default precision is ' '"6". |\n' @@ -5079,7 +5072,7 @@ topics = {'assert': 'The "assert" statement\n' '+-----------+------------------------------------------------------------+\n' ' | "\'E\'" | Exponent notation. Same as "\'e\'" ' 'except it uses an upper |\n' - " | | case 'E' as the separator " + ' | | case ‘E’ as the separator ' 'character. |\n' ' ' '+-----------+------------------------------------------------------------+\n' @@ -5216,7 +5209,7 @@ topics = {'assert': 'The "assert" statement\n' "{longitude}'.format(**coord)\n" " 'Coordinates: 37.24N, -115.81W'\n" '\n' - "Accessing arguments' attributes:\n" + 'Accessing arguments’ attributes:\n' '\n' ' >>> c = 3-5j\n' " >>> ('The complex number {0} is formed from the real " @@ -5234,7 +5227,7 @@ topics = {'assert': 'The "assert" statement\n' ' >>> str(Point(4, 2))\n' " 'Point(4, 2)'\n" '\n' - "Accessing arguments' items:\n" + 'Accessing arguments’ items:\n' '\n' ' >>> coord = (3, 5)\n' " >>> 'X: {0[0]}; Y: {0[1]}'.format(coord)\n" @@ -5396,25 +5389,24 @@ topics = {'assert': 'The "assert" statement\n' '"func".\n' '\n' 'When one or more *parameters* have the form *parameter* "="\n' - '*expression*, the function is said to have "default parameter ' - 'values."\n' + '*expression*, the function is said to have “default parameter ' + 'values.”\n' 'For a parameter with a default value, the corresponding ' '*argument* may\n' - "be omitted from a call, in which case the parameter's default " + 'be omitted from a call, in which case the parameter’s default ' 'value is\n' 'substituted. If a parameter has a default value, all following\n' - 'parameters up until the ""*"" must also have a default value --- ' - 'this\n' - 'is a syntactic restriction that is not expressed by the ' - 'grammar.\n' + 'parameters up until the “"*"” must also have a default value — ' + 'this is\n' + 'a syntactic restriction that is not expressed by the grammar.\n' '\n' '**Default parameter values are evaluated from left to right when ' 'the\n' 'function definition is executed.** This means that the ' 'expression is\n' 'evaluated once, when the function is defined, and that the same ' - '"pre-\n' - 'computed" value is used for each call. This is especially ' + '“pre-\n' + 'computed” value is used for each call. This is especially ' 'important\n' 'to understand when a default parameter is a mutable object, such ' 'as a\n' @@ -5440,26 +5432,26 @@ topics = {'assert': 'The "assert" statement\n' 'mentioned in\n' 'the parameter list, either from position arguments, from ' 'keyword\n' - 'arguments, or from default values. If the form ""*identifier"" ' + 'arguments, or from default values. If the form “"*identifier"” ' 'is\n' 'present, it is initialized to a tuple receiving any excess ' 'positional\n' 'parameters, defaulting to the empty tuple. If the form\n' - '""**identifier"" is present, it is initialized to a new ordered\n' + '“"**identifier"” is present, it is initialized to a new ordered\n' 'mapping receiving any excess keyword arguments, defaulting to a ' 'new\n' - 'empty mapping of the same type. Parameters after ""*"" or\n' - '""*identifier"" are keyword-only parameters and may only be ' + 'empty mapping of the same type. Parameters after “"*"” or\n' + '“"*identifier"” are keyword-only parameters and may only be ' 'passed\n' 'used keyword arguments.\n' '\n' - 'Parameters may have annotations of the form "": expression"" ' + 'Parameters may have annotations of the form “": expression"” ' 'following\n' 'the parameter name. Any parameter may have an annotation even ' 'those\n' 'of the form "*identifier" or "**identifier". Functions may ' 'have\n' - '"return" annotation of the form ""-> expression"" after the ' + '“return” annotation of the form “"-> expression"” after the ' 'parameter\n' 'list. These annotations can be any valid Python expression and ' 'are\n' @@ -5469,7 +5461,7 @@ topics = {'assert': 'The "assert" statement\n' 'code.\n' 'The presence of annotations does not change the semantics of a\n' 'function. The annotation values are available as values of a\n' - "dictionary keyed by the parameters' names in the " + 'dictionary keyed by the parameters’ names in the ' '"__annotations__"\n' 'attribute of the function object.\n' '\n' @@ -5480,16 +5472,16 @@ topics = {'assert': 'The "assert" statement\n' 'lambda\n' 'expression is merely a shorthand for a simplified function ' 'definition;\n' - 'a function defined in a ""def"" statement can be passed around ' + 'a function defined in a “"def"” statement can be passed around ' 'or\n' 'assigned to another name just like a function defined by a ' 'lambda\n' - 'expression. The ""def"" form is actually more powerful since ' + 'expression. The “"def"” form is actually more powerful since ' 'it\n' 'allows the execution of multiple statements and annotations.\n' '\n' - "**Programmer's note:** Functions are first-class objects. A " - '""def""\n' + '**Programmer’s note:** Functions are first-class objects. A ' + '“"def"”\n' 'statement executed inside a function definition defines a local\n' 'function that can be returned or passed around. Free variables ' 'used\n' @@ -5533,8 +5525,7 @@ topics = {'assert': 'The "assert" statement\n' 'change\n' 'the meaning of the program.\n' '\n' - '**Programmer\'s note:** "global" is a directive to the parser. ' - 'It\n' + '**Programmer’s note:** "global" is a directive to the parser. It\n' 'applies only to code parsed at the same time as the "global"\n' 'statement. In particular, a "global" statement contained in a ' 'string\n' @@ -5591,7 +5582,7 @@ topics = {'assert': 'The "assert" statement\n' 'within the\n' ' context of a class definition, are re-written to use a ' 'mangled form\n' - ' to help avoid name clashes between "private" attributes of ' + ' to help avoid name clashes between “private” attributes of ' 'base and\n' ' derived classes. See section Identifiers (Names).\n', 'identifiers': 'Identifiers and keywords\n' @@ -5739,7 +5730,7 @@ topics = {'assert': 'The "assert" statement\n' 'within the\n' ' context of a class definition, are re-written to use a ' 'mangled form\n' - ' to help avoid name clashes between "private" attributes of ' + ' to help avoid name clashes between “private” attributes of ' 'base and\n' ' derived classes. See section Identifiers (Names).\n', 'if': 'The "if" statement\n' @@ -5820,7 +5811,7 @@ topics = {'assert': 'The "assert" statement\n' 'either\n' 'that the module could not be located, *or* that an error occurred\n' 'while initializing the module, which includes execution of the\n' - "module's code.\n" + 'module’s code.\n' '\n' 'If the requested module is retrieved successfully, it will be ' 'made\n' @@ -5831,7 +5822,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' '* If no other name is specified, and the module being imported is ' 'a\n' - " top level module, the module's name is bound in the local " + ' top level module, the module’s name is bound in the local ' 'namespace\n' ' as a reference to the imported module\n' '\n' @@ -5884,7 +5875,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'The *public names* defined by a module are determined by checking ' 'the\n' - 'module\'s namespace for a variable named "__all__"; if defined, it ' + 'module’s namespace for a variable named "__all__"; if defined, it ' 'must\n' 'be a sequence of strings which are names defined or imported by ' 'that\n' @@ -5892,7 +5883,7 @@ topics = {'assert': 'The "assert" statement\n' 'and\n' 'are required to exist. If "__all__" is not defined, the set of ' 'public\n' - "names includes all names found in the module's namespace which do " + 'names includes all names found in the module’s namespace which do ' 'not\n' 'begin with an underscore character ("\'_\'"). "__all__" should ' 'contain\n' @@ -5902,8 +5893,7 @@ topics = {'assert': 'The "assert" statement\n' 'were\n' 'imported and used within the module).\n' '\n' - 'The wild card form of import --- "from module import *" --- is ' - 'only\n' + 'The wild card form of import — "from module import *" — is only\n' 'allowed at the module level. Attempting to use it in class or\n' 'function definitions will raise a "SyntaxError".\n' '\n' @@ -6011,7 +6001,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' import __future__ [as name]\n' '\n' - "That is not a future statement; it's an ordinary import statement " + 'That is not a future statement; it’s an ordinary import statement ' 'with\n' 'no special semantics or syntax restrictions.\n' '\n' @@ -6022,8 +6012,7 @@ topics = {'assert': 'The "assert" statement\n' 'the\n' 'future statement. This can be controlled by optional arguments ' 'to\n' - '"compile()" --- see the documentation of that function for ' - 'details.\n' + '"compile()" — see the documentation of that function for details.\n' '\n' 'A future statement typed at an interactive interpreter prompt ' 'will\n' @@ -6130,13 +6119,13 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'Lambda expressions (sometimes called lambda forms) are used to ' 'create\n' - 'anonymous functions. The expression "lambda arguments: ' + 'anonymous functions. The expression "lambda parameters: ' 'expression"\n' 'yields a function object. The unnamed object behaves like a ' 'function\n' 'object defined with:\n' '\n' - ' def (arguments):\n' + ' def (parameters):\n' ' return expression\n' '\n' 'See section Function definitions for the syntax of parameter ' @@ -6229,7 +6218,7 @@ topics = {'assert': 'The "assert" statement\n' 'nearest\n' 'enclosing scope. The set of all such scopes visible to a code ' 'block\n' - "is called the block's *environment*.\n" + 'is called the block’s *environment*.\n' '\n' 'When a name is not found at all, a "NameError" exception is ' 'raised. If\n' @@ -6303,7 +6292,7 @@ topics = {'assert': 'The "assert" statement\n' 'the class. The scope of names defined in a class block is limited ' 'to\n' 'the class block; it does not extend to the code blocks of methods ' - '--\n' + '–\n' 'this includes comprehensions and generator expressions since they ' 'are\n' 'implemented using a function scope. This means that the ' @@ -6330,7 +6319,7 @@ topics = {'assert': 'The "assert" statement\n' 'global\n' 'namespace; this should be a dictionary or a module (in the latter ' 'case\n' - "the module's dictionary is used). By default, when in the " + 'the module’s dictionary is used). By default, when in the ' '"__main__"\n' 'module, "__builtins__" is the built-in module "builtins"; when in ' 'any\n' @@ -6406,7 +6395,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'Note that numeric literals do not include a sign; a phrase like ' '"-1"\n' - 'is actually an expression composed of the unary operator \'"-"\' ' + 'is actually an expression composed of the unary operator ‘"-"‘ ' 'and the\n' 'literal "1".\n', 'numeric-types': 'Emulating numeric types\n' @@ -6496,15 +6485,15 @@ topics = {'assert': 'The "assert" statement\n' '"__rpow__()" (the\n' ' coercion rules would become too complicated).\n' '\n' - " Note: If the right operand's type is a subclass of the " + ' Note: If the right operand’s type is a subclass of the ' 'left\n' - " operand's type and that subclass provides the " + ' operand’s type and that subclass provides the ' 'reflected method\n' ' for the operation, this method will be called before ' 'the left\n' - " operand's non-reflected method. This behavior allows " + ' operand’s non-reflected method. This behavior allows ' 'subclasses\n' - " to override their ancestors' operations.\n" + ' to override their ancestors’ operations.\n' '\n' 'object.__iadd__(self, other)\n' 'object.__isub__(self, other)\n' @@ -6542,7 +6531,7 @@ topics = {'assert': 'The "assert" statement\n' 'certain\n' ' situations, augmented assignment can result in ' 'unexpected errors\n' - " (see Why does a_tuple[i] += ['item'] raise an exception " + ' (see Why does a_tuple[i] += [‘item’] raise an exception ' 'when the\n' ' addition works?), but this behavior is in fact part of ' 'the data\n' @@ -6604,18 +6593,18 @@ topics = {'assert': 'The "assert" statement\n' 'objects': 'Objects, values and types\n' '*************************\n' '\n' - "*Objects* are Python's abstraction for data. All data in a " + '*Objects* are Python’s abstraction for data. All data in a ' 'Python\n' 'program is represented by objects or by relations between ' 'objects. (In\n' - 'a sense, and in conformance to Von Neumann\'s model of a "stored\n' - 'program computer," code is also represented by objects.)\n' + 'a sense, and in conformance to Von Neumann’s model of a “stored\n' + 'program computer,” code is also represented by objects.)\n' '\n' - "Every object has an identity, a type and a value. An object's\n" + 'Every object has an identity, a type and a value. An object’s\n' '*identity* never changes once it has been created; you may think ' 'of it\n' - 'as the object\'s address in memory. The \'"is"\' operator ' - 'compares the\n' + 'as the object’s address in memory. The ‘"is"’ operator compares ' + 'the\n' 'identity of two objects; the "id()" function returns an integer\n' 'representing its identity.\n' '\n' @@ -6623,14 +6612,14 @@ topics = {'assert': 'The "assert" statement\n' 'memory\n' 'address where "x" is stored.\n' '\n' - "An object's type determines the operations that the object " + 'An object’s type determines the operations that the object ' 'supports\n' - '(e.g., "does it have a length?") and also defines the possible ' + '(e.g., “does it have a length?”) and also defines the possible ' 'values\n' 'for objects of that type. The "type()" function returns an ' - "object's\n" + 'object’s\n' 'type (which is an object itself). Like its identity, an ' - "object's\n" + 'object’s\n' '*type* is also unchangeable. [1]\n' '\n' 'The *value* of some objects can change. Objects whose value can\n' @@ -6639,14 +6628,14 @@ topics = {'assert': 'The "assert" statement\n' 'once they are created are called *immutable*. (The value of an\n' 'immutable container object that contains a reference to a ' 'mutable\n' - "object can change when the latter's value is changed; however " + 'object can change when the latter’s value is changed; however ' 'the\n' 'container is still considered immutable, because the collection ' 'of\n' 'objects it contains cannot be changed. So, immutability is not\n' 'strictly the same as having an unchangeable value, it is more ' 'subtle.)\n' - "An object's mutability is determined by its type; for instance,\n" + 'An object’s mutability is determined by its type; for instance,\n' 'numbers, strings and tuples are immutable, while dictionaries ' 'and\n' 'lists are mutable.\n' @@ -6654,9 +6643,9 @@ topics = {'assert': 'The "assert" statement\n' 'Objects are never explicitly destroyed; however, when they ' 'become\n' 'unreachable they may be garbage-collected. An implementation is\n' - 'allowed to postpone garbage collection or omit it altogether --- ' - 'it is\n' - 'a matter of implementation quality how garbage collection is\n' + 'allowed to postpone garbage collection or omit it altogether — it ' + 'is a\n' + 'matter of implementation quality how garbage collection is\n' 'implemented, as long as no objects are collected that are still\n' 'reachable.\n' '\n' @@ -6676,13 +6665,14 @@ topics = {'assert': 'The "assert" statement\n' '(so\n' 'you should always close files explicitly).\n' '\n' - "Note that the use of the implementation's tracing or debugging\n" + 'Note that the use of the implementation’s tracing or debugging\n' 'facilities may keep objects alive that would normally be ' 'collectable.\n' - 'Also note that catching an exception with a \'"try"..."except"\'\n' - 'statement may keep objects alive.\n' + 'Also note that catching an exception with a ‘"try"…"except"’ ' + 'statement\n' + 'may keep objects alive.\n' '\n' - 'Some objects contain references to "external" resources such as ' + 'Some objects contain references to “external” resources such as ' 'open\n' 'files or windows. It is understood that these resources are ' 'freed\n' @@ -6693,14 +6683,13 @@ topics = {'assert': 'The "assert" statement\n' 'release the external resource, usually a "close()" method. ' 'Programs\n' 'are strongly recommended to explicitly close such objects. The\n' - '\'"try"..."finally"\' statement and the \'"with"\' statement ' - 'provide\n' + '‘"try"…"finally"’ statement and the ‘"with"’ statement provide\n' 'convenient ways to do this.\n' '\n' 'Some objects contain references to other objects; these are ' 'called\n' '*containers*. Examples of containers are tuples, lists and\n' - "dictionaries. The references are part of a container's value. " + 'dictionaries. The references are part of a container’s value. ' 'In\n' 'most cases, when we talk about the value of a container, we imply ' 'the\n' @@ -6757,7 +6746,7 @@ topics = {'assert': 'The "assert" statement\n' '| "lambda" | ' 'Lambda expression |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "if" -- "else" | ' + '| "if" – "else" | ' 'Conditional expression |\n' '+-------------------------------------------------+---------------------------------------+\n' '| "or" | ' @@ -6840,7 +6829,7 @@ topics = {'assert': 'The "assert" statement\n' 'application.\n' '\n' '[2] If x is very close to an exact integer multiple of ' - "y, it's\n" + 'y, it’s\n' ' possible for "x//y" to be one larger than ' '"(x-x%y)//y" due to\n' ' rounding. In such cases, Python returns the latter ' @@ -6851,8 +6840,8 @@ topics = {'assert': 'The "assert" statement\n' '\n' '[3] The Unicode standard distinguishes between *code ' 'points* (e.g.\n' - ' U+0041) and *abstract characters* (e.g. "LATIN ' - 'CAPITAL LETTER A").\n' + ' U+0041) and *abstract characters* (e.g. “LATIN ' + 'CAPITAL LETTER A”).\n' ' While most abstract characters in Unicode are only ' 'represented\n' ' using one code point, there is a number of abstract ' @@ -6860,8 +6849,8 @@ topics = {'assert': 'The "assert" statement\n' ' that can in addition be represented using a sequence ' 'of more than\n' ' one code point. For example, the abstract character ' - '"LATIN\n' - ' CAPITAL LETTER C WITH CEDILLA" can be represented as ' + '“LATIN\n' + ' CAPITAL LETTER C WITH CEDILLA” can be represented as ' 'a single\n' ' *precomposed character* at code position U+00C7, or ' 'as a sequence\n' @@ -6877,9 +6866,9 @@ topics = {'assert': 'The "assert" statement\n' 'to humans. For\n' ' example, ""\\u00C7" == "\\u0043\\u0327"" is "False", ' 'even though both\n' - ' strings represent the same abstract character "LATIN ' + ' strings represent the same abstract character “LATIN ' 'CAPITAL\n' - ' LETTER C WITH CEDILLA".\n' + ' LETTER C WITH CEDILLA”.\n' '\n' ' To compare strings at the level of abstract ' 'characters (that is,\n' @@ -6909,10 +6898,11 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' pass_stmt ::= "pass"\n' '\n' - '"pass" is a null operation --- when it is executed, nothing ' - 'happens.\n' - 'It is useful as a placeholder when a statement is required\n' - 'syntactically, but no code needs to be executed, for example:\n' + '"pass" is a null operation — when it is executed, nothing happens. ' + 'It\n' + 'is useful as a placeholder when a statement is required ' + 'syntactically,\n' + 'but no code needs to be executed, for example:\n' '\n' ' def f(arg): pass # a function that does nothing (yet)\n' '\n' @@ -6968,7 +6958,7 @@ topics = {'assert': 'The "assert" statement\n' '"BaseException". If it is a class, the exception instance will be\n' 'obtained when needed by instantiating the class with no arguments.\n' '\n' - "The *type* of the exception is the exception instance's class, the\n" + 'The *type* of the exception is the exception instance’s class, the\n' '*value* is the instance itself.\n' '\n' 'A traceback object is normally created automatically when an ' @@ -7014,7 +7004,7 @@ topics = {'assert': 'The "assert" statement\n' 'inside\n' 'an exception handler or a "finally" clause: the previous exception ' 'is\n' - 'then attached as the new exception\'s "__context__" attribute:\n' + 'then attached as the new exception’s "__context__" attribute:\n' '\n' ' >>> try:\n' ' ... print(1 / 0)\n' @@ -7114,7 +7104,7 @@ topics = {'assert': 'The "assert" statement\n' '"clear()",\n' '"setdefault()", "pop()", "popitem()", "copy()", and ' '"update()"\n' - "behaving similar to those for Python's standard dictionary " + 'behaving similar to those for Python’s standard dictionary ' 'objects.\n' 'The "collections" module provides a "MutableMapping" ' 'abstract base\n' @@ -7140,7 +7130,7 @@ topics = {'assert': 'The "assert" statement\n' 'the\n' '"__contains__()" method to allow efficient use of the "in" ' 'operator;\n' - 'for mappings, "in" should search the mapping\'s keys; for ' + 'for mappings, "in" should search the mapping’s keys; for ' 'sequences, it\n' 'should search through the values. It is further ' 'recommended that both\n' @@ -7158,7 +7148,7 @@ topics = {'assert': 'The "assert" statement\n' 'Should return\n' ' the length of the object, an integer ">=" 0. Also, an ' 'object that\n' - ' doesn\'t define a "__bool__()" method and whose ' + ' doesn’t define a "__bool__()" method and whose ' '"__len__()" method\n' ' returns zero is considered to be false in a Boolean ' 'context.\n' @@ -7315,7 +7305,7 @@ topics = {'assert': 'The "assert" statement\n' 'values or\n' ' the key-item pairs.\n' '\n' - ' For objects that don\'t define "__contains__()", the ' + ' For objects that don’t define "__contains__()", the ' 'membership test\n' ' first tries iteration via "__iter__()", then the old ' 'sequence\n' @@ -7409,7 +7399,7 @@ topics = {'assert': 'The "assert" statement\n' 'object.__dict__\n' '\n' ' A dictionary or other mapping object used to store an ' - "object's\n" + 'object’s\n' ' (writable) attributes.\n' '\n' 'instance.__class__\n' @@ -7469,15 +7459,15 @@ topics = {'assert': 'The "assert" statement\n' 'to\n' ' "[1.0, 2.0]", and similarly for tuples.\n' '\n' - "[3] They must have since the parser can't tell the type of " + '[3] They must have since the parser can’t tell the type of ' 'the\n' ' operands.\n' '\n' '[4] Cased characters are those with general category ' 'property\n' - ' being one of "Lu" (Letter, uppercase), "Ll" (Letter, ' + ' being one of “Lu” (Letter, uppercase), “Ll” (Letter, ' 'lowercase),\n' - ' or "Lt" (Letter, titlecase).\n' + ' or “Lt” (Letter, titlecase).\n' '\n' '[5] To format only a tuple you should therefore provide a\n' ' singleton tuple whose only element is the tuple to be ' @@ -7489,7 +7479,7 @@ topics = {'assert': 'The "assert" statement\n' 'special\n' 'syntax (such as arithmetic operations or subscripting and ' 'slicing) by\n' - "defining methods with special names. This is Python's " + 'defining methods with special names. This is Python’s ' 'approach to\n' '*operator overloading*, allowing classes to define their own ' 'behavior\n' @@ -7523,7 +7513,7 @@ topics = {'assert': 'The "assert" statement\n' 'elements, but\n' 'extracting a slice may not make sense. (One example of this ' 'is the\n' - '"NodeList" interface in the W3C\'s Document Object Model.)\n' + '"NodeList" interface in the W3C’s Document Object Model.)\n' '\n' '\n' 'Basic customization\n' @@ -7547,7 +7537,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' Typical implementations create a new instance of the ' 'class by\n' - ' invoking the superclass\'s "__new__()" method using\n' + ' invoking the superclass’s "__new__()" method using\n' ' "super().__new__(cls[, ...])" with appropriate arguments ' 'and then\n' ' modifying the newly-created instance as necessary before ' @@ -7556,7 +7546,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' If "__new__()" returns an instance of *cls*, then the ' 'new\n' - ' instance\'s "__init__()" method will be invoked like\n' + ' instance’s "__init__()" method will be invoked like\n' ' "__init__(self[, ...])", where *self* is the new instance ' 'and the\n' ' remaining arguments are the same as were passed to ' @@ -7564,7 +7554,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' If "__new__()" does not return an instance of *cls*, then ' 'the new\n' - ' instance\'s "__init__()" method will not be invoked.\n' + ' instance’s "__init__()" method will not be invoked.\n' '\n' ' "__new__()" is intended mainly to allow subclasses of ' 'immutable\n' @@ -7582,7 +7572,7 @@ topics = {'assert': 'The "assert" statement\n' 'those\n' ' passed to the class constructor expression. If a base ' 'class has an\n' - ' "__init__()" method, the derived class\'s "__init__()" ' + ' "__init__()" method, the derived class’s "__init__()" ' 'method, if\n' ' any, must explicitly call it to ensure proper ' 'initialization of the\n' @@ -7603,8 +7593,8 @@ topics = {'assert': 'The "assert" statement\n' 'is also\n' ' called a finalizer or (improperly) a destructor. If a ' 'base class\n' - ' has a "__del__()" method, the derived class\'s ' - '"__del__()" method,\n' + ' has a "__del__()" method, the derived class’s "__del__()" ' + 'method,\n' ' if any, must explicitly call it to ensure proper deletion ' 'of the\n' ' base class part of the instance.\n' @@ -7624,11 +7614,11 @@ topics = {'assert': 'The "assert" statement\n' 'for\n' ' objects that still exist when the interpreter exits.\n' '\n' - ' Note: "del x" doesn\'t directly call "x.__del__()" --- ' - 'the former\n' + ' Note: "del x" doesn’t directly call "x.__del__()" — the ' + 'former\n' ' decrements the reference count for "x" by one, and the ' 'latter is\n' - ' only called when "x"\'s reference count reaches zero.\n' + ' only called when "x"’s reference count reaches zero.\n' '\n' ' **CPython implementation detail:** It is possible for a ' 'reference\n' @@ -7640,7 +7630,7 @@ topics = {'assert': 'The "assert" statement\n' 'reference\n' ' cycles is when an exception has been caught in a local ' 'variable.\n' - " The frame's locals then reference the exception, which " + ' The frame’s locals then reference the exception, which ' 'references\n' ' its own traceback, which references the locals of all ' 'frames caught\n' @@ -7686,7 +7676,7 @@ topics = {'assert': 'The "assert" statement\n' 'object.__repr__(self)\n' '\n' ' Called by the "repr()" built-in function to compute the ' - '"official"\n' + '“official”\n' ' string representation of an object. If at all possible, ' 'this\n' ' should look like a valid Python expression that could be ' @@ -7700,7 +7690,7 @@ topics = {'assert': 'The "assert" statement\n' ' value must be a string object. If a class defines ' '"__repr__()" but\n' ' not "__str__()", then "__repr__()" is also used when an ' - '"informal"\n' + '“informal”\n' ' string representation of instances of that class is ' 'required.\n' '\n' @@ -7712,7 +7702,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' Called by "str(object)" and the built-in functions ' '"format()" and\n' - ' "print()" to compute the "informal" or nicely printable ' + ' "print()" to compute the “informal” or nicely printable ' 'string\n' ' representation of an object. The return value must be a ' 'string\n' @@ -7740,7 +7730,7 @@ topics = {'assert': 'The "assert" statement\n' 'extension,\n' ' evaluation of formatted string literals and the ' '"str.format()"\n' - ' method, to produce a "formatted" string representation of ' + ' method, to produce a “formatted” string representation of ' 'an\n' ' object. The "format_spec" argument is a string that ' 'contains a\n' @@ -7771,7 +7761,7 @@ topics = {'assert': 'The "assert" statement\n' 'object.__gt__(self, other)\n' 'object.__ge__(self, other)\n' '\n' - ' These are the so-called "rich comparison" methods. The\n' + ' These are the so-called “rich comparison” methods. The\n' ' correspondence between operator symbols and method names ' 'is as\n' ' follows: "x=" 0. Also, an ' 'object that\n' - ' doesn\'t define a "__bool__()" method and whose ' + ' doesn’t define a "__bool__()" method and whose ' '"__len__()" method\n' ' returns zero is considered to be false in a Boolean ' 'context.\n' @@ -8977,7 +8967,7 @@ topics = {'assert': 'The "assert" statement\n' 'values or\n' ' the key-item pairs.\n' '\n' - ' For objects that don\'t define "__contains__()", the ' + ' For objects that don’t define "__contains__()", the ' 'membership test\n' ' first tries iteration via "__iter__()", then the old ' 'sequence\n' @@ -9073,15 +9063,15 @@ topics = {'assert': 'The "assert" statement\n' '"__rpow__()" (the\n' ' coercion rules would become too complicated).\n' '\n' - " Note: If the right operand's type is a subclass of the " + ' Note: If the right operand’s type is a subclass of the ' 'left\n' - " operand's type and that subclass provides the reflected " + ' operand’s type and that subclass provides the reflected ' 'method\n' ' for the operation, this method will be called before ' 'the left\n' - " operand's non-reflected method. This behavior allows " + ' operand’s non-reflected method. This behavior allows ' 'subclasses\n' - " to override their ancestors' operations.\n" + ' to override their ancestors’ operations.\n' '\n' 'object.__iadd__(self, other)\n' 'object.__isub__(self, other)\n' @@ -9119,7 +9109,7 @@ topics = {'assert': 'The "assert" statement\n' 'certain\n' ' situations, augmented assignment can result in unexpected ' 'errors\n' - " (see Why does a_tuple[i] += ['item'] raise an exception " + ' (see Why does a_tuple[i] += [‘item’] raise an exception ' 'when the\n' ' addition works?), but this behavior is in fact part of ' 'the data\n' @@ -9209,7 +9199,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' Enter the runtime context related to this object. The ' '"with"\n' - " statement will bind this method's return value to the " + ' statement will bind this method’s return value to the ' 'target(s)\n' ' specified in the "as" clause of the statement, if any.\n' '\n' @@ -9233,11 +9223,11 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' Note that "__exit__()" methods should not reraise the ' 'passed-in\n' - " exception; this is the caller's responsibility.\n" + ' exception; this is the caller’s responsibility.\n' '\n' 'See also:\n' '\n' - ' **PEP 343** - The "with" statement\n' + ' **PEP 343** - The “with” statement\n' ' The specification, background, and examples for the ' 'Python "with"\n' ' statement.\n' @@ -9248,9 +9238,9 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'For custom classes, implicit invocations of special methods ' 'are only\n' - "guaranteed to work correctly if defined on an object's type, " + 'guaranteed to work correctly if defined on an object’s type, ' 'not in\n' - "the object's instance dictionary. That behaviour is the " + 'the object’s instance dictionary. That behaviour is the ' 'reason why\n' 'the following code raises an exception:\n' '\n' @@ -9284,7 +9274,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'Incorrectly attempting to invoke an unbound method of a ' 'class in this\n' - "way is sometimes referred to as 'metaclass confusion', and " + 'way is sometimes referred to as ‘metaclass confusion’, and ' 'is avoided\n' 'by bypassing the instance when looking up special methods:\n' '\n' @@ -9297,7 +9287,7 @@ topics = {'assert': 'The "assert" statement\n' 'interest of\n' 'correctness, implicit special method lookup generally also ' 'bypasses\n' - 'the "__getattribute__()" method even of the object\'s ' + 'the "__getattribute__()" method even of the object’s ' 'metaclass:\n' '\n' ' >>> class Meta(type):\n' @@ -9577,11 +9567,11 @@ topics = {'assert': 'The "assert" statement\n' 'Alphabetic\n' ' characters are those characters defined in the Unicode ' 'character\n' - ' database as "Letter", i.e., those with general category ' + ' database as “Letter”, i.e., those with general category ' 'property\n' - ' being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note ' + ' being one of “Lm”, “Lt”, “Lu”, “Ll”, or “Lo”. Note ' 'that this is\n' - ' different from the "Alphabetic" property defined in the ' + ' different from the “Alphabetic” property defined in the ' 'Unicode\n' ' Standard.\n' '\n' @@ -9595,7 +9585,7 @@ topics = {'assert': 'The "assert" statement\n' 'in base 10,\n' ' e.g. U+0660, ARABIC-INDIC DIGIT ZERO. Formally a ' 'decimal character\n' - ' is a character in the Unicode General Category "Nd".\n' + ' is a character in the Unicode General Category “Nd”.\n' '\n' 'str.isdigit()\n' '\n' @@ -9654,7 +9644,7 @@ topics = {'assert': 'The "assert" statement\n' 'characters are\n' ' those characters defined in the Unicode character ' 'database as\n' - ' "Other" or "Separator", excepting the ASCII space ' + ' “Other” or “Separator”, excepting the ASCII space ' '(0x20) which is\n' ' considered printable. (Note that printable characters ' 'in this\n' @@ -9672,9 +9662,9 @@ topics = {'assert': 'The "assert" statement\n' 'Whitespace\n' ' characters are those characters defined in the Unicode ' 'character\n' - ' database as "Other" or "Separator" and those with ' + ' database as “Other” or “Separator” and those with ' 'bidirectional\n' - ' property being one of "WS", "B", or "S".\n' + ' property being one of “WS”, “B”, or “S”.\n' '\n' 'str.istitle()\n' '\n' @@ -10132,9 +10122,9 @@ topics = {'assert': 'The "assert" statement\n' '"str.upper().isupper()" might be\n' ' "False" if "s" contains uncased characters or if the ' 'Unicode\n' - ' category of the resulting character(s) is not "Lu" ' + ' category of the resulting character(s) is not “Lu” ' '(Letter,\n' - ' uppercase), but e.g. "Lt" (Letter, titlecase).\n' + ' uppercase), but e.g. “Lt” (Letter, titlecase).\n' '\n' ' The uppercasing algorithm used is described in section ' '3.13 of the\n' @@ -10229,9 +10219,9 @@ topics = {'assert': 'The "assert" statement\n' 'literals,\n' '"\'\\U\'" and "\'\\u\'" escapes in raw strings are not treated ' 'specially.\n' - "Given that Python 2.x's raw unicode literals behave differently " + 'Given that Python 2.x’s raw unicode literals behave differently ' 'than\n' - 'Python 3.x\'s the "\'ur\'" syntax is not supported.\n' + 'Python 3.x’s the "\'ur\'" syntax is not supported.\n' '\n' 'New in version 3.3: The "\'rb\'" prefix of raw bytes literals has ' 'been\n' @@ -10255,7 +10245,7 @@ topics = {'assert': 'The "assert" statement\n' 'In triple-quoted literals, unescaped newlines and quotes are ' 'allowed\n' '(and are retained), except that three unescaped quotes in a row\n' - 'terminate the literal. (A "quote" is the character used to open ' + 'terminate the literal. (A “quote” is the character used to open ' 'the\n' 'literal, i.e. either "\'" or """.)\n' '\n' @@ -10434,13 +10424,13 @@ topics = {'assert': 'The "assert" statement\n' 'item whose\n' 'index is that value (counting from zero). Since the support ' 'for\n' - "negative indices and slicing occurs in the object's " + 'negative indices and slicing occurs in the object’s ' '"__getitem__()"\n' 'method, subclasses overriding this method will need to ' 'explicitly add\n' 'that support.\n' '\n' - "A string's items are characters. A character is not a " + 'A string’s items are characters. A character is not a ' 'separate data\n' 'type but a string of exactly one character.\n', 'truth': 'Truth Value Testing\n' @@ -10497,7 +10487,7 @@ topics = {'assert': 'The "assert" statement\n' 'less except clause, if present, must be last; it matches any\n' 'exception. For an except clause with an expression, that expression\n' 'is evaluated, and the clause matches the exception if the resulting\n' - 'object is "compatible" with the exception. An object is compatible\n' + 'object is “compatible” with the exception. An object is compatible\n' 'with an exception if it is the class or a base class of the ' 'exception\n' 'object or a tuple containing an item compatible with the exception.\n' @@ -10519,7 +10509,7 @@ topics = {'assert': 'The "assert" statement\n' 'When a matching except clause is found, the exception is assigned to\n' 'the target specified after the "as" keyword in that except clause, ' 'if\n' - "present, and the except clause's suite is executed. All except\n" + 'present, and the except clause’s suite is executed. All except\n' 'clauses must have an executable block. When the end of this block ' 'is\n' 'reached, execution continues normally after the entire try ' @@ -10549,7 +10539,7 @@ topics = {'assert': 'The "assert" statement\n' 'cycle with the stack frame, keeping all locals in that frame alive\n' 'until the next garbage collection occurs.\n' '\n' - "Before an except clause's suite is executed, details about the\n" + 'Before an except clause’s suite is executed, details about the\n' 'exception are stored in the "sys" module and can be accessed via\n' '"sys.exc_info()". "sys.exc_info()" returns a 3-tuple consisting of ' 'the\n' @@ -10563,7 +10553,7 @@ topics = {'assert': 'The "assert" statement\n' 'the end of the "try" clause. [2] Exceptions in the "else" clause are\n' 'not handled by the preceding "except" clauses.\n' '\n' - 'If "finally" is present, it specifies a \'cleanup\' handler. The ' + 'If "finally" is present, it specifies a ‘cleanup’ handler. The ' '"try"\n' 'clause is executed, including any "except" and "else" clauses. If ' 'an\n' @@ -10591,12 +10581,10 @@ topics = {'assert': 'The "assert" statement\n' 'execution of the "finally" clause.\n' '\n' 'When a "return", "break" or "continue" statement is executed in the\n' - '"try" suite of a "try"..."finally" statement, the "finally" clause ' - 'is\n' - 'also executed \'on the way out.\' A "continue" statement is illegal ' - 'in\n' + '"try" suite of a "try"…"finally" statement, the "finally" clause is\n' + 'also executed ‘on the way out.’ A "continue" statement is illegal in\n' 'the "finally" clause. (The reason is a problem with the current\n' - 'implementation --- this restriction may be lifted in the future).\n' + 'implementation — this restriction may be lifted in the future).\n' '\n' 'The return value of a function is determined by the last "return"\n' 'statement executed. Since the "finally" clause always executes, a\n' @@ -10631,7 +10619,7 @@ topics = {'assert': 'The "assert" statement\n' 'will often be provided via the standard library instead.\n' '\n' 'Some of the type descriptions below contain a paragraph listing\n' - "'special attributes.' These are attributes that provide access to " + '‘special attributes.’ These are attributes that provide access to ' 'the\n' 'implementation and are not intended for general use. Their ' 'definition\n' @@ -10644,7 +10632,7 @@ topics = {'assert': 'The "assert" statement\n' 'It\n' ' is used to signify the absence of a value in many situations, ' 'e.g.,\n' - " it is returned from functions that don't explicitly return\n" + ' it is returned from functions that don’t explicitly return\n' ' anything. Its truth value is false.\n' '\n' 'NotImplemented\n' @@ -10695,7 +10683,7 @@ topics = {'assert': 'The "assert" statement\n' 'shift\n' ' and mask operations, a binary representation is assumed, ' 'and\n' - " negative numbers are represented in a variant of 2's\n" + ' negative numbers are represented in a variant of 2’s\n' ' complement which gives the illusion of an infinite string ' 'of\n' ' sign bits extending to the left.\n' @@ -10749,7 +10737,7 @@ topics = {'assert': 'The "assert" statement\n' 'items\n' ' of a sequence. When the length of a sequence is *n*, the index ' 'set\n' - ' contains the numbers 0, 1, ..., *n*-1. Item *i* of sequence *a* ' + ' contains the numbers 0, 1, …, *n*-1. Item *i* of sequence *a* ' 'is\n' ' selected by "a[i]".\n' '\n' @@ -10759,8 +10747,8 @@ topics = {'assert': 'The "assert" statement\n' 'implies\n' ' that the index set is renumbered so that it starts at 0.\n' '\n' - ' Some sequences also support "extended slicing" with a third ' - '"step"\n' + ' Some sequences also support “extended slicing” with a third ' + '“step”\n' ' parameter: "a[i:j:k]" selects all items of *a* with index *x* ' 'where\n' ' "x = i + n*k", *n* ">=" "0" and *i* "<=" *x* "<" *j*.\n' @@ -10785,7 +10773,7 @@ topics = {'assert': 'The "assert" statement\n' 'code\n' ' points. All the code points in the range "U+0000 - ' 'U+10FFFF"\n' - " can be represented in a string. Python doesn't have a " + ' can be represented in a string. Python doesn’t have a ' '"char"\n' ' type; instead, every code point in the string is ' 'represented\n' @@ -10804,7 +10792,7 @@ topics = {'assert': 'The "assert" statement\n' ' The items of a tuple are arbitrary Python objects. Tuples ' 'of\n' ' two or more items are formed by comma-separated lists of\n' - " expressions. A tuple of one item (a 'singleton') can be\n" + ' expressions. A tuple of one item (a ‘singleton’) can be\n' ' formed by affixing a comma to an expression (an expression ' 'by\n' ' itself does not create a tuple, since parentheses must be\n' @@ -10910,7 +10898,7 @@ topics = {'assert': 'The "assert" statement\n' 'object\n' ' identity, the reason being that the efficient implementation ' 'of\n' - " dictionaries requires a key's hash value to remain constant.\n" + ' dictionaries requires a key’s hash value to remain constant.\n' ' Numeric types used for keys obey the normal rules for ' 'numeric\n' ' comparison: if two numbers compare equal (e.g., "1" and ' @@ -10935,7 +10923,7 @@ topics = {'assert': 'The "assert" statement\n' ' definition (see section Function definitions). It should be\n' ' called with an argument list containing the same number of ' 'items\n' - " as the function's formal parameter list.\n" + ' as the function’s formal parameter list.\n' '\n' ' Special attributes:\n' '\n' @@ -10945,8 +10933,8 @@ topics = {'assert': 'The "assert" statement\n' '| |\n' ' ' '+===========================+=================================+=============+\n' - ' | "__doc__" | The function\'s ' - 'documentation | Writable |\n' + ' | "__doc__" | The function’s documentation ' + '| Writable |\n' ' | | string, or "None" if ' '| |\n' ' | | unavailable; not inherited by ' @@ -10955,12 +10943,12 @@ topics = {'assert': 'The "assert" statement\n' '| |\n' ' ' '+---------------------------+---------------------------------+-------------+\n' - ' | "__name__" | The function\'s ' - 'name | Writable |\n' + ' | "__name__" | The function’s name ' + '| Writable |\n' ' ' '+---------------------------+---------------------------------+-------------+\n' - ' | "__qualname__" | The function\'s *qualified ' - 'name* | Writable |\n' + ' | "__qualname__" | The function’s *qualified name* ' + '| Writable |\n' ' | | New in version 3.3. ' '| |\n' ' ' @@ -10993,9 +10981,9 @@ topics = {'assert': 'The "assert" statement\n' '+---------------------------+---------------------------------+-------------+\n' ' | "__globals__" | A reference to the dictionary ' '| Read-only |\n' - " | | that holds the function's " + ' | | that holds the function’s ' '| |\n' - ' | | global variables --- the global ' + ' | | global variables — the global ' '| |\n' ' | | namespace of the module in ' '| |\n' @@ -11013,7 +11001,7 @@ topics = {'assert': 'The "assert" statement\n' '| Read-only |\n' ' | | contain bindings for the ' '| |\n' - " | | function's free variables. " + ' | | function’s free variables. ' '| |\n' ' ' '+---------------------------+---------------------------------+-------------+\n' @@ -11036,7 +11024,7 @@ topics = {'assert': 'The "assert" statement\n' ' ' '+---------------------------+---------------------------------+-------------+\n' '\n' - ' Most of the attributes labelled "Writable" check the type of ' + ' Most of the attributes labelled “Writable” check the type of ' 'the\n' ' assigned value.\n' '\n' @@ -11052,7 +11040,7 @@ topics = {'assert': 'The "assert" statement\n' ' attributes on built-in functions may be supported in the\n' ' future.*\n' '\n' - " Additional information about a function's definition can be\n" + ' Additional information about a function’s definition can be\n' ' retrieved from its code object; see the description of ' 'internal\n' ' types below.\n' @@ -11065,7 +11053,7 @@ topics = {'assert': 'The "assert" statement\n' ' Special read-only attributes: "__self__" is the class ' 'instance\n' ' object, "__func__" is the function object; "__doc__" is the\n' - ' method\'s documentation (same as "__func__.__doc__"); ' + ' method’s documentation (same as "__func__.__doc__"); ' '"__name__"\n' ' is the method name (same as "__func__.__name__"); ' '"__module__"\n' @@ -11089,7 +11077,7 @@ topics = {'assert': 'The "assert" statement\n' 'instances,\n' ' its "__self__" attribute is the instance, and the method ' 'object\n' - ' is said to be bound. The new method\'s "__func__" attribute ' + ' is said to be bound. The new method’s "__func__" attribute ' 'is\n' ' the original function object.\n' '\n' @@ -11122,7 +11110,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' When an instance method object is derived from a class ' 'method\n' - ' object, the "class instance" stored in "__self__" will ' + ' object, the “class instance” stored in "__self__" will ' 'actually\n' ' be the class itself, so that calling either "x.f(1)" or ' '"C.f(1)"\n' @@ -11155,7 +11143,7 @@ topics = {'assert': 'The "assert" statement\n' 'object\n' ' which can be used to execute the body of the function: ' 'calling\n' - ' the iterator\'s "iterator.__next__()" method will cause the\n' + ' the iterator’s "iterator.__next__()" method will cause the\n' ' function to execute until it provides a value using the ' '"yield"\n' ' statement. When the function executes a "return" statement ' @@ -11184,7 +11172,7 @@ topics = {'assert': 'The "assert" statement\n' 'for"\n' ' statement to execute the body of the function.\n' '\n' - ' Calling the asynchronous iterator\'s "aiterator.__anext__()"\n' + ' Calling the asynchronous iterator’s "aiterator.__anext__()"\n' ' method will return an *awaitable* which when awaited will\n' ' execute until it provides a value using the "yield" ' 'expression.\n' @@ -11203,9 +11191,9 @@ topics = {'assert': 'The "assert" statement\n' 'of\n' ' the arguments are determined by the C function. Special ' 'read-\n' - ' only attributes: "__doc__" is the function\'s documentation\n' + ' only attributes: "__doc__" is the function’s documentation\n' ' string, or "None" if unavailable; "__name__" is the ' - "function's\n" + 'function’s\n' ' name; "__self__" is set to "None" (but see the next item);\n' ' "__module__" is the name of the module the function was ' 'defined\n' @@ -11248,16 +11236,16 @@ topics = {'assert': 'The "assert" statement\n' ' translated to lookups in this dictionary, e.g., "m.x" is ' 'equivalent\n' ' to "m.__dict__["x"]". A module object does not contain the code\n' - " object used to initialize the module (since it isn't needed " + ' object used to initialize the module (since it isn’t needed ' 'once\n' ' the initialization is done).\n' '\n' - " Attribute assignment updates the module's namespace dictionary,\n" + ' Attribute assignment updates the module’s namespace dictionary,\n' ' e.g., "m.x = 1" is equivalent to "m.__dict__["x"] = 1".\n' '\n' - ' Predefined (writable) attributes: "__name__" is the module\'s ' + ' Predefined (writable) attributes: "__name__" is the module’s ' 'name;\n' - ' "__doc__" is the module\'s documentation string, or "None" if\n' + ' "__doc__" is the module’s documentation string, or "None" if\n' ' unavailable; "__annotations__" (optional) is a dictionary\n' ' containing *variable annotations* collected during module body\n' ' execution; "__file__" is the pathname of the file from which ' @@ -11270,7 +11258,7 @@ topics = {'assert': 'The "assert" statement\n' 'is\n' ' the pathname of the shared library file.\n' '\n' - ' Special read-only attribute: "__dict__" is the module\'s ' + ' Special read-only attribute: "__dict__" is the module’s ' 'namespace\n' ' as a dictionary object.\n' '\n' @@ -11299,7 +11287,7 @@ topics = {'assert': 'The "assert" statement\n' ' classes. This search of the base classes uses the C3 method\n' ' resolution order which behaves correctly even in the presence ' 'of\n' - " 'diamond' inheritance structures where there are multiple\n" + ' ‘diamond’ inheritance structures where there are multiple\n' ' inheritance paths leading back to a common ancestor. Additional\n' ' details on the C3 MRO used by Python can be found in the\n' ' documentation accompanying the 2.3 release at\n' @@ -11308,7 +11296,7 @@ topics = {'assert': 'The "assert" statement\n' ' When a class attribute reference (for class "C", say) would ' 'yield a\n' ' class method object, it is transformed into an instance method\n' - ' object whose "__self__" attributes is "C". When it would yield ' + ' object whose "__self__" attribute is "C". When it would yield ' 'a\n' ' static method object, it is transformed into the object wrapped ' 'by\n' @@ -11318,7 +11306,7 @@ topics = {'assert': 'The "assert" statement\n' 'differ\n' ' from those actually contained in its "__dict__".\n' '\n' - " Class attribute assignments update the class's dictionary, " + ' Class attribute assignments update the class’s dictionary, ' 'never\n' ' the dictionary of a base class.\n' '\n' @@ -11330,11 +11318,11 @@ topics = {'assert': 'The "assert" statement\n' 'is\n' ' the module name in which the class was defined; "__dict__" is ' 'the\n' - ' dictionary containing the class\'s namespace; "__bases__" is a ' + ' dictionary containing the class’s namespace; "__bases__" is a ' 'tuple\n' ' containing the base classes, in the order of their occurrence ' 'in\n' - ' the base class list; "__doc__" is the class\'s documentation ' + ' the base class list; "__doc__" is the class’s documentation ' 'string,\n' ' or "None" if undefined; "__annotations__" (optional) is a\n' ' dictionary containing *variable annotations* collected during ' @@ -11347,7 +11335,7 @@ topics = {'assert': 'The "assert" statement\n' ' A class instance has a namespace implemented as a dictionary ' 'which\n' ' is the first place in which attribute references are searched.\n' - " When an attribute is not found there, and the instance's class " + ' When an attribute is not found there, and the instance’s class ' 'has\n' ' an attribute by that name, the search continues with the class\n' ' attributes. If a class attribute is found that is a ' @@ -11356,17 +11344,17 @@ topics = {'assert': 'The "assert" statement\n' 'object\n' ' whose "__self__" attribute is the instance. Static method and\n' ' class method objects are also transformed; see above under\n' - ' "Classes". See section Implementing Descriptors for another way ' + ' “Classes”. See section Implementing Descriptors for another way ' 'in\n' ' which attributes of a class retrieved via its instances may ' 'differ\n' - ' from the objects actually stored in the class\'s "__dict__". If ' + ' from the objects actually stored in the class’s "__dict__". If ' 'no\n' - " class attribute is found, and the object's class has a\n" + ' class attribute is found, and the object’s class has a\n' ' "__getattr__()" method, that is called to satisfy the lookup.\n' '\n' - " Attribute assignments and deletions update the instance's\n" - " dictionary, never a class's dictionary. If the class has a\n" + ' Attribute assignments and deletions update the instance’s\n' + ' dictionary, never a class’s dictionary. If the class has a\n' ' "__setattr__()" or "__delattr__()" method, this is called ' 'instead\n' ' of updating the instance dictionary directly.\n' @@ -11377,7 +11365,7 @@ topics = {'assert': 'The "assert" statement\n' ' Special method names.\n' '\n' ' Special attributes: "__dict__" is the attribute dictionary;\n' - ' "__class__" is the instance\'s class.\n' + ' "__class__" is the instance’s class.\n' '\n' 'I/O objects (also known as file objects)\n' ' A *file object* represents an open file. Various shortcuts are\n' @@ -11389,7 +11377,7 @@ topics = {'assert': 'The "assert" statement\n' ' provided by extension modules).\n' '\n' ' The objects "sys.stdin", "sys.stdout" and "sys.stderr" are\n' - " initialized to file objects corresponding to the interpreter's\n" + ' initialized to file objects corresponding to the interpreter’s\n' ' standard input, output and error streams; they are all open in ' 'text\n' ' mode and therefore follow the interface defined by the\n' @@ -11407,7 +11395,7 @@ topics = {'assert': 'The "assert" statement\n' ' or *bytecode*. The difference between a code object and a\n' ' function object is that the function object contains an ' 'explicit\n' - " reference to the function's globals (the module in which it " + ' reference to the function’s globals (the module in which it ' 'was\n' ' defined), while a code object contains no context; also the\n' ' default argument values are stored in the function object, ' @@ -11499,7 +11487,7 @@ topics = {'assert': 'The "assert" statement\n' 'is\n' ' used by the debugger); "f_lineno" is the current line number ' 'of\n' - ' the frame --- writing to this from within a trace function ' + ' the frame — writing to this from within a trace function ' 'jumps\n' ' to the given line (only for the bottom-most frame). A ' 'debugger\n' @@ -11616,7 +11604,7 @@ topics = {'assert': 'The "assert" statement\n' ' is retrieved from classes and class instances. The behaviour ' 'of\n' ' class method objects upon such retrieval is described above,\n' - ' under "User-defined methods". Class method objects are ' + ' under “User-defined methods”. Class method objects are ' 'created\n' ' by the built-in "classmethod()" constructor.\n', 'typesfunctions': 'Functions\n' @@ -11636,8 +11624,8 @@ topics = {'assert': 'The "assert" statement\n' 'different object types.\n' '\n' 'See Function definitions for more information.\n', - 'typesmapping': 'Mapping Types --- "dict"\n' - '************************\n' + 'typesmapping': 'Mapping Types — "dict"\n' + '**********************\n' '\n' 'A *mapping* object maps *hashable* values to arbitrary ' 'objects.\n' @@ -11648,7 +11636,7 @@ topics = {'assert': 'The "assert" statement\n' 'in "list", "set", and "tuple" classes, and the "collections" ' 'module.)\n' '\n' - "A dictionary's keys are *almost* arbitrary values. Values " + 'A dictionary’s keys are *almost* arbitrary values. Values ' 'that are\n' 'not *hashable*, that is, values containing lists, ' 'dictionaries or\n' @@ -11824,13 +11812,13 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' items()\n' '\n' - ' Return a new view of the dictionary\'s items ("(key, ' + ' Return a new view of the dictionary’s items ("(key, ' 'value)"\n' ' pairs). See the documentation of view objects.\n' '\n' ' keys()\n' '\n' - " Return a new view of the dictionary's keys. See the\n" + ' Return a new view of the dictionary’s keys. See the\n' ' documentation of view objects.\n' '\n' ' pop(key[, default])\n' @@ -11878,14 +11866,14 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' values()\n' '\n' - " Return a new view of the dictionary's values. See " + ' Return a new view of the dictionary’s values. See ' 'the\n' ' documentation of view objects.\n' '\n' ' Dictionaries compare equal if and only if they have the ' 'same "(key,\n' - ' value)" pairs. Order comparisons (\'<\', \'<=\', \'>=\', ' - "'>') raise\n" + ' value)" pairs. Order comparisons (‘<’, ‘<=’, ‘>=’, ‘>’) ' + 'raise\n' ' "TypeError".\n' '\n' 'See also: "types.MappingProxyType" can be used to create a ' @@ -11899,7 +11887,7 @@ topics = {'assert': 'The "assert" statement\n' 'The objects returned by "dict.keys()", "dict.values()" and\n' '"dict.items()" are *view objects*. They provide a dynamic ' 'view on the\n' - "dictionary's entries, which means that when the dictionary " + 'dictionary’s entries, which means that when the dictionary ' 'changes,\n' 'the view reflects these changes.\n' '\n' @@ -11921,7 +11909,7 @@ topics = {'assert': 'The "assert" statement\n' 'which is\n' ' non-random, varies across Python implementations, and ' 'depends on\n' - " the dictionary's history of insertions and deletions. If " + ' the dictionary’s history of insertions and deletions. If ' 'keys,\n' ' values and items views are iterated over with no ' 'intervening\n' @@ -11941,7 +11929,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'x in dictview\n' '\n' - ' Return "True" if *x* is in the underlying dictionary\'s ' + ' Return "True" if *x* is in the underlying dictionary’s ' 'keys, values\n' ' or items (in the latter case, *x* should be a "(key, ' 'value)"\n' @@ -12056,7 +12044,7 @@ topics = {'assert': 'The "assert" statement\n' 'The only special operation on a module is attribute access: ' '"m.name",\n' 'where *m* is a module and *name* accesses a name defined in ' - "*m*'s\n" + '*m*’s\n' 'symbol table. Module attributes can be assigned to. (Note ' 'that the\n' '"import" statement is not, strictly speaking, an operation ' @@ -12069,14 +12057,14 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'A special attribute of every module is "__dict__". This is ' 'the\n' - "dictionary containing the module's symbol table. Modifying " + 'dictionary containing the module’s symbol table. Modifying ' 'this\n' - "dictionary will actually change the module's symbol table, " + 'dictionary will actually change the module’s symbol table, ' 'but direct\n' 'assignment to the "__dict__" attribute is not possible (you ' 'can write\n' '"m.__dict__[\'a\'] = 1", which defines "m.a" to be "1", but ' - "you can't\n" + 'you can’t\n' 'write "m.__dict__ = {}"). Modifying "__dict__" directly is ' 'not\n' 'recommended.\n' @@ -12087,8 +12075,8 @@ topics = {'assert': 'The "assert" statement\n' 'written as\n' '"".\n', - 'typesseq': 'Sequence Types --- "list", "tuple", "range"\n' - '*******************************************\n' + 'typesseq': 'Sequence Types — "list", "tuple", "range"\n' + '*****************************************\n' '\n' 'There are three basic sequence types: lists, tuples, and range\n' 'objects. Additional sequence types tailored for processing of ' @@ -12262,7 +12250,7 @@ topics = {'assert': 'The "assert" statement\n' '*j* are\n' ' reduced to "len(s) - 1" if they are greater. If *i* or *j* ' 'are\n' - ' omitted or "None", they become "end" values (which end ' + ' omitted or "None", they become “end” values (which end ' 'depends on\n' ' the sign of *k*). Note, *k* cannot be zero. If *k* is ' '"None", it\n' @@ -12293,7 +12281,7 @@ topics = {'assert': 'The "assert" statement\n' 'documentation\n' '\n' '7. Some sequence types (such as "range") only support item\n' - " sequences that follow specific patterns, and hence don't " + ' sequences that follow specific patterns, and hence don’t ' 'support\n' ' sequence concatenation or repetition.\n' '\n' @@ -12450,7 +12438,7 @@ topics = {'assert': 'The "assert" statement\n' ' sequence.\n' '\n' '5. "clear()" and "copy()" are included for consistency with the\n' - " interfaces of mutable containers that don't support slicing\n" + ' interfaces of mutable containers that don’t support slicing\n' ' operations (such as "dict" and "set")\n' '\n' ' New in version 3.3: "clear()" and "copy()" methods.\n' @@ -12490,7 +12478,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' The constructor builds a list whose items are the same and in ' 'the\n' - " same order as *iterable*'s items. *iterable* may be either " + ' same order as *iterable*’s items. *iterable* may be either ' 'a\n' ' sequence, a container that supports iteration, or an ' 'iterator\n' @@ -12557,8 +12545,8 @@ topics = {'assert': 'The "assert" statement\n' 'is\n' ' stable if it guarantees not to change the relative order ' 'of\n' - ' elements that compare equal --- this is helpful for ' - 'sorting in\n' + ' elements that compare equal — this is helpful for sorting ' + 'in\n' ' multiple passes (for example, sort by department, then by ' 'salary\n' ' grade).\n' @@ -12604,7 +12592,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' ' The constructor builds a tuple whose items are the same and ' 'in the\n' - " same order as *iterable*'s items. *iterable* may be either " + ' same order as *iterable*’s items. *iterable* may be either ' 'a\n' ' sequence, a container that supports iteration, or an ' 'iterator\n' @@ -12733,7 +12721,7 @@ topics = {'assert': 'The "assert" statement\n' 'Range objects implement the "collections.abc.Sequence" ABC, and\n' 'provide features such as containment tests, element index ' 'lookup,\n' - 'slicing and support for negative indices (see Sequence Types --- ' + 'slicing and support for negative indices (see Sequence Types — ' 'list,\n' 'tuple, range):\n' '\n' @@ -12771,7 +12759,7 @@ topics = {'assert': 'The "assert" statement\n' 'constant\n' 'time instead of iterating through all items.\n' '\n' - "Changed in version 3.3: Define '==' and '!=' to compare range " + 'Changed in version 3.3: Define ‘==’ and ‘!=’ to compare range ' 'objects\n' 'based on the sequence of values they define (instead of ' 'comparing\n' @@ -12910,7 +12898,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' '5. "clear()" and "copy()" are included for consistency ' 'with the\n' - " interfaces of mutable containers that don't support " + ' interfaces of mutable containers that don’t support ' 'slicing\n' ' operations (such as "dict" and "set")\n' '\n' @@ -12965,7 +12953,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'A "break" statement executed in the first suite terminates the ' 'loop\n' - 'without executing the "else" clause\'s suite. A "continue" ' + 'without executing the "else" clause’s suite. A "continue" ' 'statement\n' 'executed in the first suite skips the rest of the suite and goes ' 'back\n' @@ -12975,21 +12963,22 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'The "with" statement is used to wrap the execution of a block with\n' 'methods defined by a context manager (see section With Statement\n' - 'Context Managers). This allows common "try"..."except"..."finally"\n' - 'usage patterns to be encapsulated for convenient reuse.\n' + 'Context Managers). This allows common "try"…"except"…"finally" ' + 'usage\n' + 'patterns to be encapsulated for convenient reuse.\n' '\n' ' with_stmt ::= "with" with_item ("," with_item)* ":" suite\n' ' with_item ::= expression ["as" target]\n' '\n' - 'The execution of the "with" statement with one "item" proceeds as\n' + 'The execution of the "with" statement with one “item” proceeds as\n' 'follows:\n' '\n' '1. The context expression (the expression given in the "with_item")\n' ' is evaluated to obtain a context manager.\n' '\n' - '2. The context manager\'s "__exit__()" is loaded for later use.\n' + '2. The context manager’s "__exit__()" is loaded for later use.\n' '\n' - '3. The context manager\'s "__enter__()" method is invoked.\n' + '3. The context manager’s "__enter__()" method is invoked.\n' '\n' '4. If a target was included in the "with" statement, the return\n' ' value from "__enter__()" is assigned to it.\n' @@ -13003,7 +12992,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' '5. The suite is executed.\n' '\n' - '6. The context manager\'s "__exit__()" method is invoked. If an\n' + '6. The context manager’s "__exit__()" method is invoked. If an\n' ' exception caused the suite to be exited, its type, value, and\n' ' traceback are passed as arguments to "__exit__()". Otherwise, ' 'three\n' @@ -13039,7 +13028,7 @@ topics = {'assert': 'The "assert" statement\n' '\n' 'See also:\n' '\n' - ' **PEP 343** - The "with" statement\n' + ' **PEP 343** - The “with” statement\n' ' The specification, background, and examples for the Python ' '"with"\n' ' statement.\n', diff --git a/Lib/random.py b/Lib/random.py index 0152e5e..7a2585e 100644 --- a/Lib/random.py +++ b/Lib/random.py @@ -241,6 +241,8 @@ class Random(_random.Random): "enough bits to choose from a population range this large.\n" "To remove the range limitation, add a getrandbits() method.") return int(random() * n) + if n == 0: + raise ValueError("Boundary cannot be zero") rem = maxsize % n limit = (maxsize - rem) / maxsize # int(limit * maxsize) % n == 0 r = random() diff --git a/Lib/shelve.py b/Lib/shelve.py index 581baf1..dfd7316 100644 --- a/Lib/shelve.py +++ b/Lib/shelve.py @@ -235,7 +235,7 @@ def open(filename, flag='c', protocol=None, writeback=False): filename and more than one file may be created. The optional flag parameter has the same interpretation as the flag parameter of dbm.open(). The optional protocol parameter specifies the - version of the pickle protocol (0, 1, or 2). + version of the pickle protocol. See the module's __doc__ string for an overview of the interface. """ diff --git a/Lib/site.py b/Lib/site.py index 0fc9200..1bd63cc 100644 --- a/Lib/site.py +++ b/Lib/site.py @@ -104,11 +104,11 @@ def abs_paths(): continue # don't mess with a PEP 302-supplied __file__ try: m.__file__ = os.path.abspath(m.__file__) - except (AttributeError, OSError): + except (AttributeError, OSError, TypeError): pass try: m.__cached__ = os.path.abspath(m.__cached__) - except (AttributeError, OSError): + except (AttributeError, OSError, TypeError): pass diff --git a/Lib/socketserver.py b/Lib/socketserver.py index 41a3766..c4d544b 100644 --- a/Lib/socketserver.py +++ b/Lib/socketserver.py @@ -547,8 +547,10 @@ if hasattr(os, "fork"): timeout = 300 active_children = None max_children = 40 + # If true, server_close() waits until all child processes complete. + _block_on_close = False - def collect_children(self): + def collect_children(self, *, blocking=False): """Internal routine to wait for children that have exited.""" if self.active_children is None: return @@ -572,7 +574,8 @@ if hasattr(os, "fork"): # Now reap all defunct children. for pid in self.active_children.copy(): try: - pid, _ = os.waitpid(pid, os.WNOHANG) + flags = 0 if blocking else os.WNOHANG + pid, _ = os.waitpid(pid, flags) # if the child hasn't exited yet, pid will be 0 and ignored by # discard() below self.active_children.discard(pid) @@ -621,6 +624,10 @@ if hasattr(os, "fork"): finally: os._exit(status) + def server_close(self): + super().server_close() + self.collect_children(blocking=self._block_on_close) + class ThreadingMixIn: """Mix-in class to handle each request in a new thread.""" @@ -628,6 +635,11 @@ class ThreadingMixIn: # Decides how threads will act upon termination of the # main process daemon_threads = False + # If true, server_close() waits until all non-daemonic threads terminate. + _block_on_close = False + # For non-daemonic threads, list of threading.Threading objects + # used by server_close() to wait for all threads completion. + _threads = None def process_request_thread(self, request, client_address): """Same as in BaseServer but as a thread. @@ -647,8 +659,21 @@ class ThreadingMixIn: t = threading.Thread(target = self.process_request_thread, args = (request, client_address)) t.daemon = self.daemon_threads + if not t.daemon and self._block_on_close: + if self._threads is None: + self._threads = [] + self._threads.append(t) t.start() + def server_close(self): + super().server_close() + if self._block_on_close: + threads = self._threads + self._threads = None + if threads: + for thread in threads: + thread.join() + if hasattr(os, "fork"): class ForkingUDPServer(ForkingMixIn, UDPServer): pass diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 6146235..2cb5434 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -136,7 +136,7 @@ def _sanitize_params(prefix, suffix, dir): class _RandomNameSequence: """An instance of _RandomNameSequence generates an endless sequence of unpredictable strings which can safely be incorporated - into file names. Each string is six characters long. Multiple + into file names. Each string is eight characters long. Multiple threads can safely use the same instance at the same time. _RandomNameSequence is an iterator.""" @@ -173,7 +173,9 @@ def _candidate_tempdir_list(): # Failing that, try OS-specific locations. if _os.name == 'nt': - dirlist.extend([ r'c:\temp', r'c:\tmp', r'\temp', r'\tmp' ]) + dirlist.extend([ _os.path.expanduser(r'~\AppData\Local\Temp'), + _os.path.expandvars(r'%SYSTEMROOT%\Temp'), + r'c:\temp', r'c:\tmp', r'\temp', r'\tmp' ]) else: dirlist.extend([ '/tmp', '/var/tmp', '/usr/tmp' ]) diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index dd0a9d7..0a82026 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -20,14 +20,14 @@ import logging import struct import operator import weakref -import test.support +from test import support import test.support.script_helper # Skip tests if _multiprocessing wasn't built. -_multiprocessing = test.support.import_module('_multiprocessing') +_multiprocessing = support.import_module('_multiprocessing') # Skip tests if sem_open implementation is broken. -test.support.import_module('multiprocessing.synchronize') +support.import_module('multiprocessing.synchronize') # import threading after _multiprocessing to raise a more relevant error # message: "No module named _multiprocessing". _multiprocessing is not compiled # without thread support. @@ -567,8 +567,8 @@ class _TestSubclassingProcess(BaseTestCase): if self.TYPE == "threads": self.skipTest('test not appropriate for {}'.format(self.TYPE)) - testfn = test.support.TESTFN - self.addCleanup(test.support.unlink, testfn) + testfn = support.TESTFN + self.addCleanup(support.unlink, testfn) proc = self.Process(target=self._test_stderr_flush, args=(testfn,)) proc.start() proc.join() @@ -597,8 +597,8 @@ class _TestSubclassingProcess(BaseTestCase): if self.TYPE == 'threads': self.skipTest('test not appropriate for {}'.format(self.TYPE)) - testfn = test.support.TESTFN - self.addCleanup(test.support.unlink, testfn) + testfn = support.TESTFN + self.addCleanup(support.unlink, testfn) for reason in ( [1, 2, 3], @@ -853,7 +853,7 @@ class _TestQueue(BaseTestCase): close_queue(queue) def test_no_import_lock_contention(self): - with test.support.temp_cwd(): + with support.temp_cwd(): module_name = 'imported_by_an_imported_module' with open(module_name + '.py', 'w') as f: f.write("""if 1: @@ -866,7 +866,7 @@ class _TestQueue(BaseTestCase): del q """) - with test.support.DirsOnSysPath(os.getcwd()): + with support.DirsOnSysPath(os.getcwd()): try: __import__(module_name) except pyqueue.Empty: @@ -891,7 +891,7 @@ class _TestQueue(BaseTestCase): class NotSerializable(object): def __reduce__(self): raise AttributeError - with test.support.captured_stderr(): + with support.captured_stderr(): q = self.Queue() q.put(NotSerializable()) q.put(True) @@ -2194,7 +2194,7 @@ class _TestPool(BaseTestCase): self.assertIs(type(cause), multiprocessing.pool.RemoteTraceback) self.assertIn('raise RuntimeError(123) # some comment', cause.tb) - with test.support.captured_stderr() as f1: + with support.captured_stderr() as f1: try: raise exc except RuntimeError: @@ -2476,7 +2476,7 @@ class _TestRemoteManager(BaseTestCase): authkey = os.urandom(32) manager = QueueManager( - address=(test.support.HOST, 0), authkey=authkey, serializer=SERIALIZER + address=(support.HOST, 0), authkey=authkey, serializer=SERIALIZER ) manager.start() @@ -2513,7 +2513,7 @@ class _TestManagerRestart(BaseTestCase): def test_rapid_restart(self): authkey = os.urandom(32) manager = QueueManager( - address=(test.support.HOST, 0), authkey=authkey, serializer=SERIALIZER) + address=(support.HOST, 0), authkey=authkey, serializer=SERIALIZER) srvr = manager.get_server() addr = srvr.address # Close the connection.Listener socket which gets opened as a part @@ -2736,14 +2736,14 @@ class _TestConnection(BaseTestCase): p = self.Process(target=self._writefd, args=(child_conn, b"foo")) p.daemon = True p.start() - self.addCleanup(test.support.unlink, test.support.TESTFN) - with open(test.support.TESTFN, "wb") as f: + self.addCleanup(support.unlink, support.TESTFN) + with open(support.TESTFN, "wb") as f: fd = f.fileno() if msvcrt: fd = msvcrt.get_osfhandle(fd) reduction.send_handle(conn, fd, p.pid) p.join() - with open(test.support.TESTFN, "rb") as f: + with open(support.TESTFN, "rb") as f: self.assertEqual(f.read(), b"foo") @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction") @@ -2762,8 +2762,8 @@ class _TestConnection(BaseTestCase): p = self.Process(target=self._writefd, args=(child_conn, b"bar", True)) p.daemon = True p.start() - self.addCleanup(test.support.unlink, test.support.TESTFN) - with open(test.support.TESTFN, "wb") as f: + self.addCleanup(support.unlink, support.TESTFN) + with open(support.TESTFN, "wb") as f: fd = f.fileno() for newfd in range(256, MAXFD): if not self._is_fd_assigned(newfd): @@ -2776,7 +2776,7 @@ class _TestConnection(BaseTestCase): finally: os.close(newfd) p.join() - with open(test.support.TESTFN, "rb") as f: + with open(support.TESTFN, "rb") as f: self.assertEqual(f.read(), b"bar") @classmethod @@ -2986,7 +2986,7 @@ class _TestPicklingConnections(BaseTestCase): l.close() l = socket.socket() - l.bind((test.support.HOST, 0)) + l.bind((support.HOST, 0)) l.listen() conn.send(l.getsockname()) new_conn, addr = l.accept() @@ -3336,7 +3336,7 @@ class _TestFinalize(BaseTestCase): gc.set_threshold(5, 5, 5) threads = [threading.Thread(target=run_finalizers), threading.Thread(target=make_finalizers)] - with test.support.start_threads(threads): + with support.start_threads(threads): time.sleep(4.0) # Wait a bit to trigger race condition finish = True if exc is not None: @@ -3697,7 +3697,7 @@ class TestWait(unittest.TestCase): def test_wait_socket(self, slow=False): from multiprocessing.connection import wait l = socket.socket() - l.bind((test.support.HOST, 0)) + l.bind((support.HOST, 0)) l.listen() addr = l.getsockname() readers = [] @@ -3910,11 +3910,11 @@ class TestNoForkBomb(unittest.TestCase): sm = multiprocessing.get_start_method() name = os.path.join(os.path.dirname(__file__), 'mp_fork_bomb.py') if sm != 'fork': - rc, out, err = test.support.script_helper.assert_python_failure(name, sm) + rc, out, err = support.script_helper.assert_python_failure(name, sm) self.assertEqual(out, b'') self.assertIn(b'RuntimeError', err) else: - rc, out, err = test.support.script_helper.assert_python_ok(name, sm) + rc, out, err = support.script_helper.assert_python_ok(name, sm) self.assertEqual(out.rstrip(), b'123') self.assertEqual(err, b'') @@ -4021,6 +4021,9 @@ class TestCloseFds(unittest.TestCase): class TestIgnoreEINTR(unittest.TestCase): + # Sending CONN_MAX_SIZE bytes into a multiprocessing pipe must block + CONN_MAX_SIZE = max(support.PIPE_MAX_SIZE, support.SOCK_MAX_SIZE) + @classmethod def _test_ignore(cls, conn): def handler(signum, frame): @@ -4029,7 +4032,7 @@ class TestIgnoreEINTR(unittest.TestCase): conn.send('ready') x = conn.recv() conn.send(x) - conn.send_bytes(b'x'*(1024*1024)) # sending 1 MB should block + conn.send_bytes(b'x' * cls.CONN_MAX_SIZE) @unittest.skipUnless(hasattr(signal, 'SIGUSR1'), 'requires SIGUSR1') def test_ignore(self): @@ -4048,7 +4051,7 @@ class TestIgnoreEINTR(unittest.TestCase): self.assertEqual(conn.recv(), 1234) time.sleep(0.1) os.kill(p.pid, signal.SIGUSR1) - self.assertEqual(conn.recv_bytes(), b'x'*(1024*1024)) + self.assertEqual(conn.recv_bytes(), b'x' * self.CONN_MAX_SIZE) time.sleep(0.1) p.join() finally: @@ -4144,7 +4147,7 @@ class TestStartMethod(unittest.TestCase): if multiprocessing.get_start_method() != 'forkserver': self.skipTest("test only relevant for 'forkserver' method") name = os.path.join(os.path.dirname(__file__), 'mp_preload.py') - rc, out, err = test.support.script_helper.assert_python_ok(name) + rc, out, err = support.script_helper.assert_python_ok(name) out = out.decode() err = err.decode() if out.rstrip() != 'ok' or err != '': @@ -4278,7 +4281,7 @@ class BaseMixin(object): def tearDownClass(cls): # bpo-26762: Some multiprocessing objects like Pool create reference # cycles. Trigger a garbage collection to break these cycles. - test.support.gc_collect() + support.gc_collect() processes = set(multiprocessing.process._dangling) - set(cls.dangling[0]) if processes: @@ -4457,7 +4460,7 @@ def install_tests_in_module_dict(remote_globs, start_method): # bpo-26762: Some multiprocessing objects like Pool create reference # cycles. Trigger a garbage collection to break these cycles. - test.support.gc_collect() + support.gc_collect() multiprocessing.set_start_method(old_start_method[0], force=True) # pause a bit so we don't get warning about dangling threads/processes @@ -4479,7 +4482,7 @@ def install_tests_in_module_dict(remote_globs, start_method): if need_sleep: time.sleep(0.5) multiprocessing.process._cleanup() - test.support.gc_collect() + support.gc_collect() remote_globs['setUpModule'] = setUpModule remote_globs['tearDownModule'] = tearDownModule diff --git a/Lib/test/bisect.py b/Lib/test/bisect.py index 0fd577d..968537e 100755 --- a/Lib/test/bisect.py +++ b/Lib/test/bisect.py @@ -38,7 +38,7 @@ def write_tests(filename, tests): def write_output(filename, tests): if not filename: return - print("Write %s tests into %s" % (len(tests), filename)) + print("Writing %s tests into %s" % (len(tests), filename)) write_tests(filename, tests) return filename @@ -133,11 +133,11 @@ def main(): print("ran %s tests/%s" % (ntest, len(tests))) print("exit", exitcode) if exitcode: - print("Tests failed: use this new subtest") + print("Tests failed: continuing with this subtest") tests = subtests output = write_output(args.output, tests) else: - print("Tests succeeded: skip this subtest, try a new subbset") + print("Tests succeeded: skipping this subtest, trying a new subset") print() iteration += 1 except KeyboardInterrupt: diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py index 96120c3..d3fa753 100644 --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -2254,25 +2254,24 @@ class TestDateTime(TestDate): base = cls(2000, 2, 29) self.assertRaises(ValueError, base.replace, year=2001) + @support.run_with_tz('EDT4') def test_astimezone(self): - return # The rest is no longer applicable - # Pretty boring! The TZ test is more interesting here. astimezone() - # simply can't be applied to a naive object. dt = self.theclass.now() - f = FixedOffset(44, "") - self.assertRaises(ValueError, dt.astimezone) # naive + f = FixedOffset(44, "0044") + dt_utc = dt.replace(tzinfo=timezone(timedelta(hours=-4), 'EDT')) + self.assertEqual(dt.astimezone(), dt_utc) # naive self.assertRaises(TypeError, dt.astimezone, f, f) # too many args self.assertRaises(TypeError, dt.astimezone, dt) # arg wrong type - self.assertRaises(ValueError, dt.astimezone, f) # naive - self.assertRaises(ValueError, dt.astimezone, tz=f) # naive + dt_f = dt.replace(tzinfo=f) + timedelta(hours=4, minutes=44) + self.assertEqual(dt.astimezone(f), dt_f) # naive + self.assertEqual(dt.astimezone(tz=f), dt_f) # naive class Bogus(tzinfo): def utcoffset(self, dt): return None def dst(self, dt): return timedelta(0) bog = Bogus() self.assertRaises(ValueError, dt.astimezone, bog) # naive - self.assertRaises(ValueError, - dt.replace(tzinfo=bog).astimezone, f) + self.assertEqual(dt.replace(tzinfo=bog).astimezone(f), dt_f) class AlsoBogus(tzinfo): def utcoffset(self, dt): return timedelta(0) @@ -2280,6 +2279,14 @@ class TestDateTime(TestDate): alsobog = AlsoBogus() self.assertRaises(ValueError, dt.astimezone, alsobog) # also naive + class Broken(tzinfo): + def utcoffset(self, dt): return 1 + def dst(self, dt): return 1 + broken = Broken() + dt_broken = dt.replace(tzinfo=broken) + with self.assertRaises(TypeError): + dt_broken.astimezone() + def test_subclass_datetime(self): class C(self.theclass): diff --git a/Lib/test/libregrtest/cmdline.py b/Lib/test/libregrtest/cmdline.py index 4999fa7..dab17c3 100644 --- a/Lib/test/libregrtest/cmdline.py +++ b/Lib/test/libregrtest/cmdline.py @@ -96,7 +96,7 @@ resources to test. Currently only the following are defined: largefile - It is okay to run some test that may create huge files. These tests can take a long time and may - consume >2GB of disk space temporarily. + consume >2 GiB of disk space temporarily. network - It is okay to run tests that use external network resource, e.g. testing SSL support for sockets. diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index 15e5152..3429b37 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -8,7 +8,6 @@ import re import sys import sysconfig import tempfile -import textwrap import time import unittest from test.libregrtest.cmdline import _parse_args @@ -18,6 +17,7 @@ from test.libregrtest.runtest import ( INTERRUPTED, CHILD_ERROR, PROGRESS_MIN_TIME, format_test_result) from test.libregrtest.setup import setup_tests +from test.libregrtest.utils import removepy, count, format_duration, printlist from test import support try: import gc @@ -41,16 +41,6 @@ else: TEMPDIR = os.path.abspath(TEMPDIR) -def format_duration(seconds): - if seconds < 1.0: - return '%.0f ms' % (seconds * 1e3) - if seconds < 60.0: - return '%.0f sec' % seconds - - minutes, seconds = divmod(seconds, 60.0) - return '%.0f min %.0f sec' % (minutes, seconds) - - class Regrtest: """Execute a test suite. @@ -88,6 +78,8 @@ class Regrtest: self.skipped = [] self.resource_denieds = [] self.environment_changed = [] + self.rerun = [] + self.first_result = None self.interrupted = False # used by --slow @@ -132,8 +124,9 @@ class Regrtest: # "[ 51/405/1] test_tcl passed" line = f"{test_index:{self.test_count_width}}{self.test_count}" - if self.bad and not self.ns.pgo: - line = f"{line}/{len(self.bad)}" + fails = len(self.bad) + len(self.environment_changed) + if fails and not self.ns.pgo: + line = f"{line}/{fails}" line = f"[{line}] {test}" # add the system load prefix: "load avg: 1.80 " @@ -280,10 +273,13 @@ class Regrtest: self.ns.verbose = True self.ns.failfast = False self.ns.verbose3 = False - self.ns.match_tests = None + self.first_result = self.get_tests_result() + + print() print("Re-running failed tests in verbose mode") - for test in self.bad[:]: + self.rerun = self.bad[:] + for test in self.rerun: print("Re-running test %r in verbose mode" % test, flush=True) try: self.ns.verbose = True @@ -301,22 +297,27 @@ class Regrtest: print(count(len(self.bad), 'test'), "failed again:") printlist(self.bad) + self.display_result() + def display_result(self): + # If running the test suite for PGO then no one cares about results. + if self.ns.pgo: + return + + print() + print("== Tests result: %s ==" % self.get_tests_result()) + if self.interrupted: - # print a newline after ^C print() + # print a newline after ^C print("Test suite interrupted by signal SIGINT.") executed = set(self.good) | set(self.bad) | set(self.skipped) omitted = set(self.selected) - executed print(count(len(omitted), "test"), "omitted:") printlist(omitted) - # If running the test suite for PGO then no one cares about - # results. - if self.ns.pgo: - return - if self.good and not self.ns.quiet: + print() if (not self.bad and not self.skipped and not self.interrupted @@ -347,6 +348,11 @@ class Regrtest: print(count(len(self.skipped), "test"), "skipped:") printlist(self.skipped) + if self.rerun: + print() + print("%s:" % count(len(self.rerun), "re-run test")) + printlist(self.rerun) + def run_tests_sequential(self): if self.ns.trace: import trace @@ -431,6 +437,24 @@ class Regrtest: % (locale.getpreferredencoding(False), sys.getfilesystemencoding())) + def get_tests_result(self): + result = [] + if self.bad: + result.append("FAILURE") + elif self.ns.fail_env_changed and self.environment_changed: + result.append("ENV CHANGED") + + if self.interrupted: + result.append("INTERRUPTED") + + if not result: + result.append("SUCCESS") + + result = ', '.join(result) + if self.first_result: + result = '%s then %s' % (self.first_result, result) + return result + def run_tests(self): # For a partial run, we do not need to clutter the output. if (self.ns.header @@ -472,16 +496,7 @@ class Regrtest: print() duration = time.monotonic() - self.start_time print("Total duration: %s" % format_duration(duration)) - - if self.bad: - result = "FAILURE" - elif self.interrupted: - result = "INTERRUPTED" - elif self.ns.fail_env_changed and self.environment_changed: - result = "ENV CHANGED" - else: - result = "SUCCESS" - print("Tests result: %s" % result) + print("Tests result: %s" % self.get_tests_result()) if self.ns.runleaks: os.system("leaks %d" % os.getpid()) @@ -548,37 +563,6 @@ class Regrtest: sys.exit(0) -def removepy(names): - if not names: - return - for idx, name in enumerate(names): - basename, ext = os.path.splitext(name) - if ext == '.py': - names[idx] = basename - - -def count(n, word): - if n == 1: - return "%d %s" % (n, word) - else: - return "%d %ss" % (n, word) - - -def printlist(x, width=70, indent=4, file=None): - """Print the elements of iterable x to stdout. - - Optional arg width (default 70) is the maximum line length. - Optional arg indent (default 4) is the number of blanks with which to - begin each line. - """ - - blanks = ' ' * indent - # Print the sorted list: 'x' may be a '--random' list or a set() - print(textwrap.fill(' '.join(str(elt) for elt in sorted(x)), width, - initial_indent=blanks, subsequent_indent=blanks), - file=file) - - def main(tests=None, **kwargs): """Run the Python suite.""" Regrtest().main(tests=tests, **kwargs) diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py index 0bd8288..18d5bd0 100644 --- a/Lib/test/libregrtest/refleak.py +++ b/Lib/test/libregrtest/refleak.py @@ -7,36 +7,6 @@ from inspect import isabstract from test import support -try: - MAXFD = os.sysconf("SC_OPEN_MAX") -except Exception: - MAXFD = 256 - - -def fd_count(): - """Count the number of open file descriptors""" - if sys.platform.startswith(('linux', 'freebsd')): - try: - names = os.listdir("/proc/self/fd") - return len(names) - except FileNotFoundError: - pass - - count = 0 - for fd in range(MAXFD): - try: - # Prefer dup() over fstat(). fstat() can require input/output - # whereas dup() doesn't. - fd2 = os.dup(fd) - except OSError as e: - if e.errno != errno.EBADF: - raise - else: - os.close(fd2) - count += 1 - return count - - def dash_R(the_module, test, indirect_test, huntrleaks): """Run a test multiple times, looking for reference leaks. @@ -182,7 +152,7 @@ def dash_R_cleanup(fs, ps, pic, zdc, abcs): func1 = sys.getallocatedblocks func2 = sys.gettotalrefcount gc.collect() - return func1(), func2(), fd_count() + return func1(), func2(), support.fd_count() def clear_caches(): diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py index aefec12..12bf422 100644 --- a/Lib/test/libregrtest/runtest.py +++ b/Lib/test/libregrtest/runtest.py @@ -103,6 +103,9 @@ def runtest(ns, test): faulthandler.dump_traceback_later(ns.timeout, exit=True) try: support.set_match_tests(ns.match_tests) + # reset the environment_altered flag to detect if a test altered + # the environment + support.environment_altered = False if ns.failfast: support.failfast = True if output_on_failure: diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py index 779ff01..108b56d 100644 --- a/Lib/test/libregrtest/runtest_mp.py +++ b/Lib/test/libregrtest/runtest_mp.py @@ -17,6 +17,7 @@ from test.libregrtest.runtest import ( runtest, INTERRUPTED, CHILD_ERROR, PROGRESS_MIN_TIME, format_test_result) from test.libregrtest.setup import setup_tests +from test.libregrtest.utils import format_duration # Display the running tests if nothing happened last N seconds @@ -171,7 +172,8 @@ def run_tests_multiprocess(regrtest): continue dt = time.monotonic() - worker.start_time if dt >= PROGRESS_MIN_TIME: - running.append('%s (%.0f sec)' % (current_test, dt)) + text = '%s (%s)' % (current_test, format_duration(dt)) + running.append(text) return running finished = 0 @@ -187,7 +189,7 @@ def run_tests_multiprocess(regrtest): except queue.Empty: running = get_running(workers) if running and not regrtest.ns.pgo: - print('running: %s' % ', '.join(running)) + print('running: %s' % ', '.join(running), flush=True) continue test, stdout, stderr, result = item @@ -239,6 +241,6 @@ def run_tests_multiprocess(regrtest): line = "Waiting for %s (%s tests)" % (', '.join(running), len(running)) if dt >= WAIT_PROGRESS: line = "%s since %.0f sec" % (line, dt) - print(line) + print(line, flush=True) for worker in workers: worker.join(WAIT_PROGRESS) diff --git a/Lib/test/libregrtest/save_env.py b/Lib/test/libregrtest/save_env.py index 8309f26..3c45621 100644 --- a/Lib/test/libregrtest/save_env.py +++ b/Lib/test/libregrtest/save_env.py @@ -268,7 +268,13 @@ class saved_test_environment: def __exit__(self, exc_type, exc_val, exc_tb): saved_values = self.saved_values del self.saved_values - support.gc_collect() # Some resources use weak references + + # Some resources use weak references + support.gc_collect() + + # Read support.environment_altered, set by support helper functions + self.changed |= support.environment_altered + for name, get, restore in self.resource_info(): current = get() original = saved_values.pop(name) diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py new file mode 100644 index 0000000..85049cb --- /dev/null +++ b/Lib/test/libregrtest/utils.py @@ -0,0 +1,47 @@ +import os.path +import textwrap + + +def format_duration(seconds): + if seconds < 1.0: + return '%.0f ms' % (seconds * 1e3) + if seconds < 60.0: + return '%.0f sec' % seconds + + minutes, seconds = divmod(seconds, 60.0) + hours, minutes = divmod(minutes, 60.0) + if hours: + return '%.0f hour %.0f min' % (hours, minutes) + else: + return '%.0f min %.0f sec' % (minutes, seconds) + + +def removepy(names): + if not names: + return + for idx, name in enumerate(names): + basename, ext = os.path.splitext(name) + if ext == '.py': + names[idx] = basename + + +def count(n, word): + if n == 1: + return "%d %s" % (n, word) + else: + return "%d %ss" % (n, word) + + +def printlist(x, width=70, indent=4, file=None): + """Print the elements of iterable x to stdout. + + Optional arg width (default 70) is the maximum line length. + Optional arg indent (default 4) is the number of blanks with which to + begin each line. + """ + + blanks = ' ' * indent + # Print the sorted list: 'x' may be a '--random' list or a set() + print(textwrap.fill(' '.join(str(elt) for elt in sorted(x)), width, + initial_indent=blanks, subsequent_indent=blanks), + file=file) diff --git a/Lib/test/lock_tests.py b/Lib/test/lock_tests.py index c177570..5b1f033 100644 --- a/Lib/test/lock_tests.py +++ b/Lib/test/lock_tests.py @@ -31,6 +31,9 @@ class Bunch(object): self.started = [] self.finished = [] self._can_exit = not wait_before_exit + self.wait_thread = support.wait_threads_exit() + self.wait_thread.__enter__() + def task(): tid = threading.get_ident() self.started.append(tid) @@ -40,6 +43,7 @@ class Bunch(object): self.finished.append(tid) while not self._can_exit: _wait() + try: for i in range(n): start_new_thread(task, ()) @@ -54,6 +58,8 @@ class Bunch(object): def wait_for_finished(self): while len(self.finished) < self.n: _wait() + # Wait for threads exit + self.wait_thread.__exit__(None, None, None) def do_finish(self): self._can_exit = True @@ -220,20 +226,23 @@ class LockTests(BaseLockTests): # Lock needs to be released before re-acquiring. lock = self.locktype() phase = [] + def f(): lock.acquire() phase.append(None) lock.acquire() phase.append(None) - start_new_thread(f, ()) - while len(phase) == 0: - _wait() - _wait() - self.assertEqual(len(phase), 1) - lock.release() - while len(phase) == 1: + + with support.wait_threads_exit(): + start_new_thread(f, ()) + while len(phase) == 0: + _wait() _wait() - self.assertEqual(len(phase), 2) + self.assertEqual(len(phase), 1) + lock.release() + while len(phase) == 1: + _wait() + self.assertEqual(len(phase), 2) def test_different_thread(self): # Lock can be released from a different thread. @@ -304,6 +313,7 @@ class RLockTests(BaseLockTests): self.assertRaises(RuntimeError, lock.release) finally: b.do_finish() + b.wait_for_finished() def test__is_owned(self): lock = self.locktype() diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index a1c0bd7..764057a 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -2667,29 +2667,30 @@ class AbstractPicklerUnpicklerObjectTests(unittest.TestCase): # object again, the third serialized form should be identical to the # first one we obtained. data = ["abcdefg", "abcdefg", 44] - f = io.BytesIO() - pickler = self.pickler_class(f) + for proto in protocols: + f = io.BytesIO() + pickler = self.pickler_class(f, proto) - pickler.dump(data) - first_pickled = f.getvalue() + pickler.dump(data) + first_pickled = f.getvalue() - # Reset BytesIO object. - f.seek(0) - f.truncate() + # Reset BytesIO object. + f.seek(0) + f.truncate() - pickler.dump(data) - second_pickled = f.getvalue() + pickler.dump(data) + second_pickled = f.getvalue() - # Reset the Pickler and BytesIO objects. - pickler.clear_memo() - f.seek(0) - f.truncate() + # Reset the Pickler and BytesIO objects. + pickler.clear_memo() + f.seek(0) + f.truncate() - pickler.dump(data) - third_pickled = f.getvalue() + pickler.dump(data) + third_pickled = f.getvalue() - self.assertNotEqual(first_pickled, second_pickled) - self.assertEqual(first_pickled, third_pickled) + self.assertNotEqual(first_pickled, second_pickled) + self.assertEqual(first_pickled, third_pickled) def test_priming_pickler_memo(self): # Verify that we can set the Pickler's memo attribute. diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index f1b0233..9242a36 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -275,6 +275,14 @@ def collect_readline(info_add): copy_attributes(info_add, readline, 'readline.%s', attributes, formatter=format_attr) + if not hasattr(readline, "_READLINE_LIBRARY_VERSION"): + # _READLINE_LIBRARY_VERSION has been added to CPython 3.7 + doc = getattr(readline, '__doc__', '') + if 'libedit readline' in doc: + info_add('readline.library', 'libedit readline') + elif 'GNU readline' in doc: + info_add('readline.library', 'GNU readline') + def collect_gdb(info_add): import subprocess @@ -489,6 +497,34 @@ def collect_test_support(info_add): call_func(info_add, 'test_support.python_is_optimized', support, 'python_is_optimized') +def collect_cc(info_add): + import subprocess + import sysconfig + + CC = sysconfig.get_config_var('CC') + if not CC: + return + + try: + import shlex + args = shlex.split(CC) + except ImportError: + args = CC.split() + args.append('--version') + proc = subprocess.Popen(args, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + universal_newlines=True) + stdout = proc.communicate()[0] + if proc.returncode: + # CC --version failed: ignore error + return + + text = stdout.splitlines()[0] + text = normalize_text(text) + info_add('CC.version', text) + + def collect_info(info): error = False info_add = info.add @@ -515,6 +551,7 @@ def collect_info(info): collect_decimal, collect_testcapi, collect_resource, + collect_cc, # Collecting from tests should be last as they have side effects. collect_test_socket, diff --git a/Lib/test/signalinterproctester.py b/Lib/test/signalinterproctester.py index d3ae170..877be51 100644 --- a/Lib/test/signalinterproctester.py +++ b/Lib/test/signalinterproctester.py @@ -74,10 +74,13 @@ class InterProcessSignalTests(unittest.TestCase): # Nothing should happen: SIGUSR2 is ignored child.wait() - signal.alarm(1) - self.wait_signal(None, 'SIGALRM', KeyboardInterrupt) - self.assertEqual(self.got_signals, {'SIGHUP': 1, 'SIGUSR1': 1, - 'SIGALRM': 0}) + try: + signal.alarm(1) + self.wait_signal(None, 'SIGALRM', KeyboardInterrupt) + self.assertEqual(self.got_signals, {'SIGHUP': 1, 'SIGUSR1': 1, + 'SIGALRM': 0}) + finally: + signal.alarm(0) if __name__ == "__main__": diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 74d4355..20c1edc 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -107,7 +107,7 @@ __all__ = [ "check_warnings", "check_no_resource_warning", "EnvironmentVarGuard", "run_with_locale", "swap_item", "swap_attr", "Matcher", "set_memlimit", "SuppressCrashReport", "sortdict", - "run_with_tz", "PGO", "missing_compiler_executable", + "run_with_tz", "PGO", "missing_compiler_executable", "fd_count", ] class Error(Exception): @@ -366,6 +366,20 @@ if sys.platform.startswith("win"): _force_run(fullname, os.unlink, fullname) _waitfor(_rmtree_inner, path, waitall=True) _waitfor(lambda p: _force_run(p, os.rmdir, p), path) + + def _longpath(path): + try: + import ctypes + except ImportError: + # No ctypes means we can't expands paths. + pass + else: + buffer = ctypes.create_unicode_buffer(len(path) * 2) + length = ctypes.windll.kernel32.GetLongPathNameW(path, buffer, + len(buffer)) + if length: + return buffer[:length] + return path else: _unlink = os.unlink _rmdir = os.rmdir @@ -392,6 +406,9 @@ else: _rmtree_inner(path) os.rmdir(path) + def _longpath(path): + return path + def unlink(filename): try: _unlink(filename) @@ -2052,6 +2069,14 @@ def modules_cleanup(oldmodules): #======================================================================= # Threading support to prevent reporting refleaks when running regrtest.py -R +# Flag used by saved_test_environment of test.libregrtest.save_env, +# to check if a test modified the environment. The flag should be set to False +# before running a new test. +# +# For example, threading_cleanup() sets the flag is the function fails +# to cleanup threads. +environment_altered = False + # NOTE: we use thread._count() rather than threading.enumerate() (or the # moral equivalent thereof) because a threading.Thread object is still alive # until its __bootstrap() method has returned, even after it has been @@ -2095,6 +2120,42 @@ def reap_threads(func): threading_cleanup(*key) return decorator + +@contextlib.contextmanager +def wait_threads_exit(timeout=60.0): + """ + bpo-31234: Context manager to wait until all threads created in the with + statement exit. + + Use _thread.count() to check if threads exited. Indirectly, wait until + threads exit the internal t_bootstrap() C function of the _thread module. + + threading_setup() and threading_cleanup() are designed to emit a warning + if a test leaves running threads in the background. This context manager + is designed to cleanup threads started by the _thread.start_new_thread() + which doesn't allow to wait for thread exit, whereas thread.Thread has a + join() method. + """ + old_count = _thread._count() + try: + yield + finally: + start_time = time.monotonic() + deadline = start_time + timeout + while True: + count = _thread._count() + if count <= old_count: + break + if time.monotonic() > deadline: + dt = time.monotonic() - start_time + msg = (f"wait_threads() failed to cleanup {count - old_count} " + f"threads after {dt:.1f} seconds " + f"(count: {count}, old count: {old_count})") + raise AssertionError(msg) + time.sleep(0.010) + gc_collect() + + def reap_children(): """Use this function at the end of test_main() whenever sub-processes are started. This will help ensure that no extra children (zombies) @@ -2333,13 +2394,15 @@ def can_xattr(): if not hasattr(os, "setxattr"): can = False else: - tmp_fp, tmp_name = tempfile.mkstemp() + tmp_dir = tempfile.mkdtemp() + tmp_fp, tmp_name = tempfile.mkstemp(dir=tmp_dir) try: with open(TESTFN, "wb") as fp: try: # TESTFN & tempfile may use different file systems with # different capabilities os.setxattr(tmp_fp, b"user.test", b"") + os.setxattr(tmp_name, b"trusted.foo", b"42") os.setxattr(fp.fileno(), b"user.test", b"") # Kernels < 2.6.39 don't respect setxattr flags. kernel_version = platform.release() @@ -2350,6 +2413,7 @@ def can_xattr(): finally: unlink(TESTFN) unlink(tmp_name) + rmdir(tmp_dir) _can_xattr = can return can @@ -2403,7 +2467,7 @@ def check__all__(test_case, module, name_of_module=None, extra=(), The 'extra' argument can be a set of names that wouldn't otherwise be automatically detected as "public", like objects without a proper - '__module__' attriubute. If provided, it will be added to the + '__module__' attribute. If provided, it will be added to the automatically detected ones. The 'blacklist' argument can be a set of names that must not be treated @@ -2671,6 +2735,65 @@ def disable_faulthandler(): faulthandler.enable(file=fd, all_threads=True) +def fd_count(): + """Count the number of open file descriptors. + """ + if sys.platform.startswith(('linux', 'freebsd')): + try: + names = os.listdir("/proc/self/fd") + # Substract one because listdir() opens internally a file + # descriptor to list the content of the /proc/self/fd/ directory. + return len(names) - 1 + except FileNotFoundError: + pass + + MAXFD = 256 + if hasattr(os, 'sysconf'): + try: + MAXFD = os.sysconf("SC_OPEN_MAX") + except OSError: + pass + + old_modes = None + if sys.platform == 'win32': + # bpo-25306, bpo-31009: Call CrtSetReportMode() to not kill the process + # on invalid file descriptor if Python is compiled in debug mode + try: + import msvcrt + msvcrt.CrtSetReportMode + except (AttributeError, ImportError): + # no msvcrt or a release build + pass + else: + old_modes = {} + for report_type in (msvcrt.CRT_WARN, + msvcrt.CRT_ERROR, + msvcrt.CRT_ASSERT): + old_modes[report_type] = msvcrt.CrtSetReportMode(report_type, 0) + + try: + count = 0 + for fd in range(MAXFD): + try: + # Prefer dup() over fstat(). fstat() can require input/output + # whereas dup() doesn't. + fd2 = os.dup(fd) + except OSError as e: + if e.errno != errno.EBADF: + raise + else: + os.close(fd2) + count += 1 + finally: + if old_modes is not None: + for report_type in (msvcrt.CRT_WARN, + msvcrt.CRT_ERROR, + msvcrt.CRT_ASSERT): + msvcrt.CrtSetReportMode(report_type, old_modes[report_type]) + + return count + + class SaveSignals: """ Save an restore signal handlers. diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index a5c4a8e..41a7f41 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -4831,7 +4831,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs=None, metavar=tuple()) def test_nargs_None_metavar_length1(self): - self.do_test_no_exception(nargs=None, metavar=("1")) + self.do_test_no_exception(nargs=None, metavar=("1",)) def test_nargs_None_metavar_length2(self): self.do_test_exception(nargs=None, metavar=("1", "2")) @@ -4848,7 +4848,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs="?", metavar=tuple()) def test_nargs_optional_metavar_length1(self): - self.do_test_no_exception(nargs="?", metavar=("1")) + self.do_test_no_exception(nargs="?", metavar=("1",)) def test_nargs_optional_metavar_length2(self): self.do_test_exception(nargs="?", metavar=("1", "2")) @@ -4865,7 +4865,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs="*", metavar=tuple()) def test_nargs_zeroormore_metavar_length1(self): - self.do_test_no_exception(nargs="*", metavar=("1")) + self.do_test_exception(nargs="*", metavar=("1",)) def test_nargs_zeroormore_metavar_length2(self): self.do_test_no_exception(nargs="*", metavar=("1", "2")) @@ -4882,7 +4882,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs="+", metavar=tuple()) def test_nargs_oneormore_metavar_length1(self): - self.do_test_no_exception(nargs="+", metavar=("1")) + self.do_test_exception(nargs="+", metavar=("1",)) def test_nargs_oneormore_metavar_length2(self): self.do_test_no_exception(nargs="+", metavar=("1", "2")) @@ -4899,7 +4899,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_no_exception(nargs="...", metavar=tuple()) def test_nargs_remainder_metavar_length1(self): - self.do_test_no_exception(nargs="...", metavar=("1")) + self.do_test_no_exception(nargs="...", metavar=("1",)) def test_nargs_remainder_metavar_length2(self): self.do_test_no_exception(nargs="...", metavar=("1", "2")) @@ -4916,7 +4916,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs="A...", metavar=tuple()) def test_nargs_parser_metavar_length1(self): - self.do_test_no_exception(nargs="A...", metavar=("1")) + self.do_test_no_exception(nargs="A...", metavar=("1",)) def test_nargs_parser_metavar_length2(self): self.do_test_exception(nargs="A...", metavar=("1", "2")) @@ -4933,7 +4933,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs=1, metavar=tuple()) def test_nargs_1_metavar_length1(self): - self.do_test_no_exception(nargs=1, metavar=("1")) + self.do_test_no_exception(nargs=1, metavar=("1",)) def test_nargs_1_metavar_length2(self): self.do_test_exception(nargs=1, metavar=("1", "2")) @@ -4950,7 +4950,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs=2, metavar=tuple()) def test_nargs_2_metavar_length1(self): - self.do_test_no_exception(nargs=2, metavar=("1")) + self.do_test_exception(nargs=2, metavar=("1",)) def test_nargs_2_metavar_length2(self): self.do_test_no_exception(nargs=2, metavar=("1", "2")) @@ -4967,7 +4967,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs=3, metavar=tuple()) def test_nargs_3_metavar_length1(self): - self.do_test_no_exception(nargs=3, metavar=("1")) + self.do_test_exception(nargs=3, metavar=("1",)) def test_nargs_3_metavar_length2(self): self.do_test_exception(nargs=3, metavar=("1", "2")) @@ -4994,6 +4994,30 @@ class TestImportStar(TestCase): ] self.assertEqual(sorted(items), sorted(argparse.__all__)) + +class TestWrappingMetavar(TestCase): + + def setUp(self): + self.parser = ErrorRaisingArgumentParser( + 'this_is_spammy_prog_with_a_long_name_sorry_about_the_name' + ) + # this metavar was triggering library assertion errors due to usage + # message formatting incorrectly splitting on the ] chars within + metavar = '' + self.parser.add_argument('--proxy', metavar=metavar) + + def test_help_with_metavar(self): + help_text = self.parser.format_help() + self.assertEqual(help_text, textwrap.dedent('''\ + usage: this_is_spammy_prog_with_a_long_name_sorry_about_the_name + [-h] [--proxy ] + + optional arguments: + -h, --help show this help message and exit + --proxy + ''')) + + def test_main(): support.run_unittest(__name__) # Remove global references to avoid looking like we have refleaks. diff --git a/Lib/test/test_asyncgen.py b/Lib/test/test_asyncgen.py index 34ab8a0..45cc275 100644 --- a/Lib/test/test_asyncgen.py +++ b/Lib/test/test_asyncgen.py @@ -114,6 +114,31 @@ class AsyncGenTest(unittest.TestCase): def async_iterate(g): res = [] while True: + an = g.__anext__() + try: + while True: + try: + an.__next__() + except StopIteration as ex: + if ex.args: + res.append(ex.args[0]) + break + else: + res.append('EMPTY StopIteration') + break + except StopAsyncIteration: + raise + except Exception as ex: + res.append(str(type(ex))) + break + except StopAsyncIteration: + res.append('STOP') + break + return res + + def async_iterate(g): + res = [] + while True: try: g.__anext__().__next__() except StopAsyncIteration: @@ -300,6 +325,37 @@ class AsyncGenTest(unittest.TestCase): "non-None value .* async generator"): gen().__anext__().send(100) + def test_async_gen_exception_11(self): + def sync_gen(): + yield 10 + yield 20 + + def sync_gen_wrapper(): + yield 1 + sg = sync_gen() + sg.send(None) + try: + sg.throw(GeneratorExit()) + except GeneratorExit: + yield 2 + yield 3 + + async def async_gen(): + yield 10 + yield 20 + + async def async_gen_wrapper(): + yield 1 + asg = async_gen() + await asg.asend(None) + try: + await asg.athrow(GeneratorExit()) + except GeneratorExit: + yield 2 + yield 3 + + self.compare_generators(sync_gen_wrapper(), async_gen_wrapper()) + def test_async_gen_api_01(self): async def gen(): yield 123 diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py index 830f0d8..42c0707 100644 --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -1750,5 +1750,6 @@ class RunningLoopTests(unittest.TestCase): outer_loop.close() + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py index 2c4629a..548a461 100644 --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -362,6 +362,24 @@ class EventLoopTestsMixin: self.assertEqual(res, 'yo') self.assertNotEqual(thread_id, threading.get_ident()) + def test_run_in_executor_cancel(self): + called = False + + def patched_call_soon(*args): + nonlocal called + called = True + + def run(): + time.sleep(0.05) + + f2 = self.loop.run_in_executor(None, run) + f2.cancel() + self.loop.close() + self.loop.call_soon = patched_call_soon + self.loop.call_soon_threadsafe = patched_call_soon + time.sleep(0.4) + self.assertFalse(called) + def test_reader_callback(self): r, w = test_utils.socketpair() r.setblocking(False) @@ -2449,6 +2467,28 @@ class HandleTests(test_utils.TestCase): # built-in async_gen.asend(). self.assertEqual(coroutines._format_coroutine(coro), 'Coro()') + coro = Coro() + coro.__qualname__ = 'AAA' + coro.cr_code = None + self.assertEqual(coroutines._format_coroutine(coro), 'AAA()') + + coro = Coro() + coro.__qualname__ = 'AAA' + coro.cr_code = None + coro.cr_frame = None + self.assertEqual(coroutines._format_coroutine(coro), 'AAA()') + + coro = Coro() + coro.__qualname__ = None + coro.cr_code = None + coro.cr_frame = None + self.assertEqual(coroutines._format_coroutine(coro), f'{repr(coro)}()') + + coro = Coro() + coro.cr_code = None + coro.cr_frame = None + self.assertEqual(coroutines._format_coroutine(coro), f'{repr(coro)}()') + class TimerTests(unittest.TestCase): diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py index d76da66..edf0461 100644 --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -330,7 +330,7 @@ class ProactorSocketTransportTests(test_utils.TestCase): def test_pause_resume_reading(self): tr = self.socket_transport() futures = [] - for msg in [b'data1', b'data2', b'data3', b'data4', b'']: + for msg in [b'data1', b'data2', b'data3', b'data4', b'data5', b'']: f = asyncio.Future(loop=self.loop) f.set_result(msg) futures.append(f) @@ -352,6 +352,13 @@ class ProactorSocketTransportTests(test_utils.TestCase): self.protocol.data_received.assert_called_with(b'data3') self.loop._run_once() self.protocol.data_received.assert_called_with(b'data4') + + tr.pause_reading() + tr.resume_reading() + self.loop.call_exception_handler = mock.Mock() + self.loop._run_once() + self.loop.call_exception_handler.assert_not_called() + self.protocol.data_received.assert_called_with(b'data5') tr.close() diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py index 830b15c..c359b45 100644 --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -832,6 +832,21 @@ class SelectorTransportTests(test_utils.TestCase): self.assertIsNone(tr._protocol) self.assertIsNone(tr._loop) + def test__add_reader(self): + tr = self.create_transport() + tr._buffer.extend(b'1') + tr._add_reader(7, mock.sentinel) + self.assertTrue(self.loop.readers) + + tr._force_close(None) + + self.assertTrue(tr.is_closing()) + self.assertFalse(self.loop.readers) + + # can not add readers after closing + tr._add_reader(7, mock.sentinel) + self.assertFalse(self.loop.readers) + class SelectorSocketTransportTests(test_utils.TestCase): @@ -1172,6 +1187,12 @@ class SelectorSocketTransportTests(test_utils.TestCase): self.sock.shutdown.assert_called_with(socket.SHUT_WR) tr.close() + def test_write_eof_after_close(self): + tr = self.socket_transport() + tr.close() + self.loop.run_until_complete(asyncio.sleep(0)) + tr.write_eof() + @mock.patch('asyncio.base_events.logger') def test_transport_close_remove_writer(self, m_log): remove_writer = self.loop._remove_writer = mock.Mock() diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py index 9b198bf..4ffbc0f 100644 --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -59,9 +59,9 @@ class SslProtoHandshakeTests(test_utils.TestCase): return [] waiter.cancel() - self.connection_made(ssl_proto, do_handshake=do_handshake) with test_utils.disable_logger(): + self.connection_made(ssl_proto, do_handshake=do_handshake) self.loop.run_until_complete(handshake_fut) def test_eof_received_waiter(self): diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py index e8822c3..0b2b17a 100644 --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -210,6 +210,7 @@ class SubprocessMixin: @asyncio.coroutine def write_stdin(proc, data): + yield from asyncio.sleep(0.5, loop=self.loop) proc.stdin.write(data) yield from proc.stdin.drain() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index f41160b..e8ec09e 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1991,7 +1991,7 @@ class BaseTaskTests: def test_cancel_wait_for(self): self._test_cancel_wait_for(60.0) - def test_cancel_gather(self): + def test_cancel_gather_1(self): """Ensure that a gathering future refuses to be cancelled once all children are done""" loop = asyncio.new_event_loop() @@ -2021,6 +2021,33 @@ class BaseTaskTests: self.assertFalse(gather_task.cancelled()) self.assertEqual(gather_task.result(), [42]) + def test_cancel_gather_2(self): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + async def test(): + time = 0 + while True: + time += 0.05 + await asyncio.gather(asyncio.sleep(0.05, loop=loop), + return_exceptions=True, + loop=loop) + if time > 1: + return + + async def main(): + qwe = self.new_task(loop, test()) + await asyncio.sleep(0.2, loop=loop) + qwe.cancel() + try: + await qwe + except asyncio.CancelledError: + pass + else: + self.fail('gather did not propagate the cancellation request') + + loop.run_until_complete(main()) + def test_exception_traceback(self): # See http://bugs.python.org/issue28843 @@ -2049,7 +2076,11 @@ class BaseTaskTests: self.assertFalse(m_log.error.called) with self.assertRaises(ValueError): - self.new_task(self.loop, coro()) + gen = coro() + try: + self.new_task(self.loop, gen) + finally: + gen.close() self.assertTrue(m_log.error.called) message = m_log.error.call_args[0][0] diff --git a/Lib/test/test_bdb.py b/Lib/test/test_bdb.py new file mode 100644 index 0000000..a366678 --- /dev/null +++ b/Lib/test/test_bdb.py @@ -0,0 +1,1153 @@ +""" Test the bdb module. + + A test defines a list of tuples that may be seen as paired tuples, each + pair being defined by 'expect_tuple, set_tuple' as follows: + + ([event, [lineno[, co_name[, eargs]]]]), (set_type, [sargs]) + + * 'expect_tuple' describes the expected current state of the Bdb instance. + It may be the empty tuple and no check is done in that case. + * 'set_tuple' defines the set_*() method to be invoked when the Bdb + instance reaches this state. + + Example of an 'expect_tuple, set_tuple' pair: + + ('line', 2, 'tfunc_main'), ('step', ) + + Definitions of the members of the 'expect_tuple': + event: + Name of the trace event. The set methods that do not give back + control to the tracer [1] do not trigger a tracer event and in + that case the next 'event' may be 'None' by convention, its value + is not checked. + [1] Methods that trigger a trace event are set_step(), set_next(), + set_return(), set_until() and set_continue(). + lineno: + Line number. Line numbers are relative to the start of the + function when tracing a function in the test_bdb module (i.e. this + module). + co_name: + Name of the function being currently traced. + eargs: + A tuple: + * On an 'exception' event the tuple holds a class object, the + current exception must be an instance of this class. + * On a 'line' event, the tuple holds a dictionary and a list. The + dictionary maps each breakpoint number that has been hit on this + line to its hits count. The list holds the list of breakpoint + number temporaries that are being deleted. + + Definitions of the members of the 'set_tuple': + set_type: + The type of the set method to be invoked. This may + be the type of one of the Bdb set methods: 'step', 'next', + 'until', 'return', 'continue', 'break', 'quit', or the type of one + of the set methods added by test_bdb.Bdb: 'ignore', 'enable', + 'disable', 'clear', 'up', 'down'. + sargs: + The arguments of the set method if any, packed in a tuple. +""" + +import bdb as _bdb +import sys +import os +import unittest +import textwrap +import importlib +import linecache +from contextlib import contextmanager +from itertools import islice, repeat +import test.support + +class BdbException(Exception): pass +class BdbError(BdbException): """Error raised by the Bdb instance.""" +class BdbSyntaxError(BdbException): """Syntax error in the test case.""" +class BdbNotExpectedError(BdbException): """Unexpected result.""" + +# When 'dry_run' is set to true, expect tuples are ignored and the actual +# state of the tracer is printed after running each set_*() method of the test +# case. The full list of breakpoints and their attributes is also printed +# after each 'line' event where a breakpoint has been hit. +dry_run = 0 + +def reset_Breakpoint(): + _bdb.Breakpoint.next = 1 + _bdb.Breakpoint.bplist = {} + _bdb.Breakpoint.bpbynumber = [None] + +def info_breakpoints(): + bp_list = [bp for bp in _bdb.Breakpoint.bpbynumber if bp] + if not bp_list: + return '' + + header_added = False + for bp in bp_list: + if not header_added: + info = 'BpNum Temp Enb Hits Ignore Where\n' + header_added = True + + disp = 'yes ' if bp.temporary else 'no ' + enab = 'yes' if bp.enabled else 'no ' + info += ('%-5d %s %s %-4d %-6d at %s:%d' % + (bp.number, disp, enab, bp.hits, bp.ignore, + os.path.basename(bp.file), bp.line)) + if bp.cond: + info += '\n\tstop only if %s' % (bp.cond,) + info += '\n' + return info + +class Bdb(_bdb.Bdb): + """Extend Bdb to enhance test coverage.""" + + def trace_dispatch(self, frame, event, arg): + self.currentbp = None + return super().trace_dispatch(frame, event, arg) + + def set_break(self, filename, lineno, temporary=False, cond=None, + funcname=None): + if isinstance(funcname, str): + if filename == __file__: + globals_ = globals() + else: + module = importlib.import_module(filename[:-3]) + globals_ = module.__dict__ + func = eval(funcname, globals_) + code = func.__code__ + filename = code.co_filename + lineno = code.co_firstlineno + funcname = code.co_name + + res = super().set_break(filename, lineno, temporary=temporary, + cond=cond, funcname=funcname) + if isinstance(res, str): + raise BdbError(res) + return res + + def get_stack(self, f, t): + self.stack, self.index = super().get_stack(f, t) + self.frame = self.stack[self.index][0] + return self.stack, self.index + + def set_ignore(self, bpnum): + """Increment the ignore count of Breakpoint number 'bpnum'.""" + bp = self.get_bpbynumber(bpnum) + bp.ignore += 1 + + def set_enable(self, bpnum): + bp = self.get_bpbynumber(bpnum) + bp.enabled = True + + def set_disable(self, bpnum): + bp = self.get_bpbynumber(bpnum) + bp.enabled = False + + def set_clear(self, fname, lineno): + err = self.clear_break(fname, lineno) + if err: + raise BdbError(err) + + def set_up(self): + """Move up in the frame stack.""" + if not self.index: + raise BdbError('Oldest frame') + self.index -= 1 + self.frame = self.stack[self.index][0] + + def set_down(self): + """Move down in the frame stack.""" + if self.index + 1 == len(self.stack): + raise BdbError('Newest frame') + self.index += 1 + self.frame = self.stack[self.index][0] + +class Tracer(Bdb): + """A tracer for testing the bdb module.""" + + def __init__(self, expect_set, skip=None, dry_run=False, test_case=None): + super().__init__(skip=skip) + self.expect_set = expect_set + self.dry_run = dry_run + self.header = ('Dry-run results for %s:' % test_case if + test_case is not None else None) + self.init_test() + + def init_test(self): + self.cur_except = None + self.expect_set_no = 0 + self.breakpoint_hits = None + self.expected_list = list(islice(self.expect_set, 0, None, 2)) + self.set_list = list(islice(self.expect_set, 1, None, 2)) + + def trace_dispatch(self, frame, event, arg): + # On an 'exception' event, call_exc_trace() in Python/ceval.c discards + # a BdbException raised by the Tracer instance, so we raise it on the + # next trace_dispatch() call that occurs unless the set_quit() or + # set_continue() method has been invoked on the 'exception' event. + if self.cur_except is not None: + raise self.cur_except + + if event == 'exception': + try: + res = super().trace_dispatch(frame, event, arg) + return res + except BdbException as e: + self.cur_except = e + return self.trace_dispatch + else: + return super().trace_dispatch(frame, event, arg) + + def user_call(self, frame, argument_list): + # Adopt the same behavior as pdb and, as a side effect, skip also the + # first 'call' event when the Tracer is started with Tracer.runcall() + # which may be possibly considered as a bug. + if not self.stop_here(frame): + return + self.process_event('call', frame, argument_list) + self.next_set_method() + + def user_line(self, frame): + self.process_event('line', frame) + + if self.dry_run and self.breakpoint_hits: + info = info_breakpoints().strip('\n') + # Indent each line. + for line in info.split('\n'): + print(' ' + line) + self.delete_temporaries() + self.breakpoint_hits = None + + self.next_set_method() + + def user_return(self, frame, return_value): + self.process_event('return', frame, return_value) + self.next_set_method() + + def user_exception(self, frame, exc_info): + self.exc_info = exc_info + self.process_event('exception', frame) + self.next_set_method() + + def do_clear(self, arg): + # The temporary breakpoints are deleted in user_line(). + bp_list = [self.currentbp] + self.breakpoint_hits = (bp_list, bp_list) + + def delete_temporaries(self): + if self.breakpoint_hits: + for n in self.breakpoint_hits[1]: + self.clear_bpbynumber(n) + + def pop_next(self): + self.expect_set_no += 1 + try: + self.expect = self.expected_list.pop(0) + except IndexError: + raise BdbNotExpectedError( + 'expect_set list exhausted, cannot pop item %d' % + self.expect_set_no) + self.set_tuple = self.set_list.pop(0) + + def process_event(self, event, frame, *args): + # Call get_stack() to enable walking the stack with set_up() and + # set_down(). + tb = None + if event == 'exception': + tb = self.exc_info[2] + self.get_stack(frame, tb) + + # A breakpoint has been hit and it is not a temporary. + if self.currentbp is not None and not self.breakpoint_hits: + bp_list = [self.currentbp] + self.breakpoint_hits = (bp_list, []) + + # Pop next event. + self.event= event + self.pop_next() + if self.dry_run: + self.print_state(self.header) + return + + # Validate the expected results. + if self.expect: + self.check_equal(self.expect[0], event, 'Wrong event type') + self.check_lno_name() + + if event in ('call', 'return'): + self.check_expect_max_size(3) + elif len(self.expect) > 3: + if event == 'line': + bps, temporaries = self.expect[3] + bpnums = sorted(bps.keys()) + if not self.breakpoint_hits: + self.raise_not_expected( + 'No breakpoints hit at expect_set item %d' % + self.expect_set_no) + self.check_equal(bpnums, self.breakpoint_hits[0], + 'Breakpoint numbers do not match') + self.check_equal([bps[n] for n in bpnums], + [self.get_bpbynumber(n).hits for + n in self.breakpoint_hits[0]], + 'Wrong breakpoint hit count') + self.check_equal(sorted(temporaries), self.breakpoint_hits[1], + 'Wrong temporary breakpoints') + + elif event == 'exception': + if not isinstance(self.exc_info[1], self.expect[3]): + self.raise_not_expected( + "Wrong exception at expect_set item %d, got '%s'" % + (self.expect_set_no, self.exc_info)) + + def check_equal(self, expected, result, msg): + if expected == result: + return + self.raise_not_expected("%s at expect_set item %d, got '%s'" % + (msg, self.expect_set_no, result)) + + def check_lno_name(self): + """Check the line number and function co_name.""" + s = len(self.expect) + if s > 1: + lineno = self.lno_abs2rel() + self.check_equal(self.expect[1], lineno, 'Wrong line number') + if s > 2: + self.check_equal(self.expect[2], self.frame.f_code.co_name, + 'Wrong function name') + + def check_expect_max_size(self, size): + if len(self.expect) > size: + raise BdbSyntaxError('Invalid size of the %s expect tuple: %s' % + (self.event, self.expect)) + + def lno_abs2rel(self): + fname = self.canonic(self.frame.f_code.co_filename) + lineno = self.frame.f_lineno + return ((lineno - self.frame.f_code.co_firstlineno + 1) + if fname == self.canonic(__file__) else lineno) + + def lno_rel2abs(self, fname, lineno): + return (self.frame.f_code.co_firstlineno + lineno - 1 + if (lineno and self.canonic(fname) == self.canonic(__file__)) + else lineno) + + def get_state(self): + lineno = self.lno_abs2rel() + co_name = self.frame.f_code.co_name + state = "('%s', %d, '%s'" % (self.event, lineno, co_name) + if self.breakpoint_hits: + bps = '{' + for n in self.breakpoint_hits[0]: + if bps != '{': + bps += ', ' + bps += '%s: %s' % (n, self.get_bpbynumber(n).hits) + bps += '}' + bps = '(' + bps + ', ' + str(self.breakpoint_hits[1]) + ')' + state += ', ' + bps + elif self.event == 'exception': + state += ', ' + self.exc_info[0].__name__ + state += '), ' + return state.ljust(32) + str(self.set_tuple) + ',' + + def print_state(self, header=None): + if header is not None and self.expect_set_no == 1: + print() + print(header) + print('%d: %s' % (self.expect_set_no, self.get_state())) + + def raise_not_expected(self, msg): + msg += '\n' + msg += ' Expected: %s\n' % str(self.expect) + msg += ' Got: ' + self.get_state() + raise BdbNotExpectedError(msg) + + def next_set_method(self): + set_type = self.set_tuple[0] + args = self.set_tuple[1] if len(self.set_tuple) == 2 else None + set_method = getattr(self, 'set_' + set_type) + + # The following set methods give back control to the tracer. + if set_type in ('step', 'continue', 'quit'): + set_method() + return + elif set_type in ('next', 'return'): + set_method(self.frame) + return + elif set_type == 'until': + lineno = None + if args: + lineno = self.lno_rel2abs(self.frame.f_code.co_filename, + args[0]) + set_method(self.frame, lineno) + return + + # The following set methods do not give back control to the tracer and + # next_set_method() is called recursively. + if (args and set_type in ('break', 'clear', 'ignore', 'enable', + 'disable')) or set_type in ('up', 'down'): + if set_type in ('break', 'clear'): + fname, lineno, *remain = args + lineno = self.lno_rel2abs(fname, lineno) + args = [fname, lineno] + args.extend(remain) + set_method(*args) + elif set_type in ('ignore', 'enable', 'disable'): + set_method(*args) + elif set_type in ('up', 'down'): + set_method() + + # Process the next expect_set item. + # It is not expected that a test may reach the recursion limit. + self.event= None + self.pop_next() + if self.dry_run: + self.print_state() + else: + if self.expect: + self.check_lno_name() + self.check_expect_max_size(3) + self.next_set_method() + else: + raise BdbSyntaxError('"%s" is an invalid set_tuple' % + self.set_tuple) + +class TracerRun(): + """Provide a context for running a Tracer instance with a test case.""" + + def __init__(self, test_case, skip=None): + self.test_case = test_case + self.dry_run = test_case.dry_run + self.tracer = Tracer(test_case.expect_set, skip=skip, + dry_run=self.dry_run, test_case=test_case.id()) + self._original_tracer = None + + def __enter__(self): + # test_pdb does not reset Breakpoint class attributes on exit :-( + reset_Breakpoint() + self._original_tracer = sys.gettrace() + return self.tracer + + def __exit__(self, type_=None, value=None, traceback=None): + reset_Breakpoint() + sys.settrace(self._original_tracer) + + not_empty = '' + if self.tracer.set_list: + not_empty += 'All paired tuples have not been processed, ' + not_empty += ('the last one was number %d' % + self.tracer.expect_set_no) + + # Make a BdbNotExpectedError a unittest failure. + if type_ is not None and issubclass(BdbNotExpectedError, type_): + if isinstance(value, BaseException) and value.args: + err_msg = value.args[0] + if not_empty: + err_msg += '\n' + not_empty + if self.dry_run: + print(err_msg) + return True + else: + self.test_case.fail(err_msg) + else: + assert False, 'BdbNotExpectedError with empty args' + + if not_empty: + if self.dry_run: + print(not_empty) + else: + self.test_case.fail(not_empty) + +def run_test(modules, set_list, skip=None): + """Run a test and print the dry-run results. + + 'modules': A dictionary mapping module names to their source code as a + string. The dictionary MUST include one module named + 'test_module' with a main() function. + 'set_list': A list of set_type tuples to be run on the module. + + For example, running the following script outputs the following results: + + ***************************** SCRIPT ******************************** + + from test.test_bdb import run_test, break_in_func + + code = ''' + def func(): + lno = 3 + + def main(): + func() + lno = 7 + ''' + + set_list = [ + break_in_func('func', 'test_module.py'), + ('continue', ), + ('step', ), + ('step', ), + ('step', ), + ('quit', ), + ] + + modules = { 'test_module': code } + run_test(modules, set_list) + + **************************** results ******************************** + + 1: ('line', 2, 'tfunc_import'), ('next',), + 2: ('line', 3, 'tfunc_import'), ('step',), + 3: ('call', 5, 'main'), ('break', ('test_module.py', None, False, None, 'func')), + 4: ('None', 5, 'main'), ('continue',), + 5: ('line', 3, 'func', ({1: 1}, [])), ('step',), + BpNum Temp Enb Hits Ignore Where + 1 no yes 1 0 at test_module.py:2 + 6: ('return', 3, 'func'), ('step',), + 7: ('line', 7, 'main'), ('step',), + 8: ('return', 7, 'main'), ('quit',), + + ************************************************************************* + + """ + def gen(a, b): + try: + while 1: + x = next(a) + y = next(b) + yield x + yield y + except StopIteration: + return + + # Step over the import statement in tfunc_import using 'next' and step + # into main() in test_module. + sl = [('next', ), ('step', )] + sl.extend(set_list) + + test = BaseTestCase() + test.dry_run = True + test.id = lambda : None + test.expect_set = list(gen(repeat(()), iter(sl))) + with create_modules(modules): + sys.path.append(os.getcwd()) + with TracerRun(test, skip=skip) as tracer: + tracer.runcall(tfunc_import) + +@contextmanager +def create_modules(modules): + with test.support.temp_cwd(): + try: + for m in modules: + fname = m + '.py' + with open(fname, 'w') as f: + f.write(textwrap.dedent(modules[m])) + linecache.checkcache(fname) + importlib.invalidate_caches() + yield + finally: + for m in modules: + test.support.forget(m) + +def break_in_func(funcname, fname=__file__, temporary=False, cond=None): + return 'break', (fname, None, temporary, cond, funcname) + +TEST_MODULE = 'test_module' +TEST_MODULE_FNAME = TEST_MODULE + '.py' +def tfunc_import(): + import test_module + test_module.main() + +def tfunc_main(): + lno = 2 + tfunc_first() + tfunc_second() + lno = 5 + lno = 6 + lno = 7 + +def tfunc_first(): + lno = 2 + lno = 3 + lno = 4 + +def tfunc_second(): + lno = 2 + +class BaseTestCase(unittest.TestCase): + """Base class for all tests.""" + + dry_run = dry_run + + def fail(self, msg=None): + # Override fail() to use 'raise from None' to avoid repetition of the + # error message and traceback. + raise self.failureException(msg) from None + +class StateTestCase(BaseTestCase): + """Test the step, next, return, until and quit 'set_' methods.""" + + def test_step(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('step', ), + ('line', 2, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_step_next_on_last_statement(self): + for set_type in ('step', 'next'): + with self.subTest(set_type=set_type): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('break', (__file__, 3)), + ('None', 1, 'tfunc_first'), ('continue', ), + ('line', 3, 'tfunc_first', ({1:1}, [])), (set_type, ), + ('line', 4, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_next(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('next', ), + ('line', 4, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_second'), ('step', ), + ('line', 2, 'tfunc_second'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_next_over_import(self): + code = """ + def main(): + lno = 3 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('next', ), + ('line', 3, 'tfunc_import'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_next_on_plain_statement(self): + # Check that set_next() is equivalent to set_step() on a plain + # statement. + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('next', ), + ('line', 2, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_next_in_caller_frame(self): + # Check that set_next() in the caller frame causes the tracer + # to stop next in the caller frame. + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('up', ), + ('None', 3, 'tfunc_main'), ('next', ), + ('line', 4, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_return(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('step', ), + ('line', 2, 'tfunc_first'), ('return', ), + ('return', 4, 'tfunc_first'), ('step', ), + ('line', 4, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_return_in_caller_frame(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('up', ), + ('None', 3, 'tfunc_main'), ('return', ), + ('return', 7, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_until(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('step', ), + ('line', 2, 'tfunc_first'), ('until', (4, )), + ('line', 4, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_until_with_too_large_count(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), break_in_func('tfunc_first'), + ('None', 2, 'tfunc_main'), ('continue', ), + ('line', 2, 'tfunc_first', ({1:1}, [])), ('until', (9999, )), + ('return', 4, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_until_in_caller_frame(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('up', ), + ('None', 3, 'tfunc_main'), ('until', (6, )), + ('line', 6, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_skip(self): + # Check that tracing is skipped over the import statement in + # 'tfunc_import()'. + code = """ + def main(): + lno = 3 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('step', ), + ('line', 3, 'tfunc_import'), ('quit', ), + ] + skip = ('importlib*', TEST_MODULE) + with TracerRun(self, skip=skip) as tracer: + tracer.runcall(tfunc_import) + + def test_down(self): + # Check that set_down() raises BdbError at the newest frame. + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('down', ), + ] + with TracerRun(self) as tracer: + self.assertRaises(BdbError, tracer.runcall, tfunc_main) + + def test_up(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('up', ), + ('None', 3, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + +class BreakpointTestCase(BaseTestCase): + """Test the breakpoint set method.""" + + def test_bp_on_non_existent_module(self): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('break', ('/non/existent/module.py', 1)) + ] + with TracerRun(self) as tracer: + self.assertRaises(BdbError, tracer.runcall, tfunc_import) + + def test_bp_after_last_statement(self): + code = """ + def main(): + lno = 3 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('break', (TEST_MODULE_FNAME, 4)) + ] + with TracerRun(self) as tracer: + self.assertRaises(BdbError, tracer.runcall, tfunc_import) + + def test_temporary_bp(self): + code = """ + def func(): + lno = 3 + + def main(): + for i in range(2): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, True), + ('None', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, True), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [1])), ('continue', ), + ('line', 3, 'func', ({2:1}, [2])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_disabled_temporary_bp(self): + code = """ + def func(): + lno = 3 + + def main(): + for i in range(3): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, True), + ('None', 2, 'tfunc_import'), ('disable', (2, )), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [])), ('enable', (2, )), + ('None', 3, 'func'), ('disable', (1, )), + ('None', 3, 'func'), ('continue', ), + ('line', 3, 'func', ({2:1}, [2])), ('enable', (1, )), + ('None', 3, 'func'), ('continue', ), + ('line', 3, 'func', ({1:2}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_bp_condition(self): + code = """ + def func(a): + lno = 3 + + def main(): + for i in range(3): + func(i) + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, False, 'a == 2'), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:3}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_bp_exception_on_condition_evaluation(self): + code = """ + def func(a): + lno = 3 + + def main(): + func(0) + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, False, '1 / 0'), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_bp_ignore_count(self): + code = """ + def func(): + lno = 3 + + def main(): + for i in range(2): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), ('ignore', (1, )), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:2}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_ignore_count_on_disabled_bp(self): + code = """ + def func(): + lno = 3 + + def main(): + for i in range(3): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), ('ignore', (1, )), + ('None', 2, 'tfunc_import'), ('disable', (1, )), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({2:1}, [])), ('enable', (1, )), + ('None', 3, 'func'), ('continue', ), + ('line', 3, 'func', ({2:2}, [])), ('continue', ), + ('line', 3, 'func', ({1:2}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_clear_two_bp_on_same_line(self): + code = """ + def func(): + lno = 3 + lno = 4 + + def main(): + for i in range(3): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('break', (TEST_MODULE_FNAME, 3)), + ('None', 2, 'tfunc_import'), ('break', (TEST_MODULE_FNAME, 3)), + ('None', 2, 'tfunc_import'), ('break', (TEST_MODULE_FNAME, 4)), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [])), ('continue', ), + ('line', 4, 'func', ({3:1}, [])), ('clear', (TEST_MODULE_FNAME, 3)), + ('None', 4, 'func'), ('continue', ), + ('line', 4, 'func', ({3:2}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_clear_at_no_bp(self): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('clear', (__file__, 1)) + ] + with TracerRun(self) as tracer: + self.assertRaises(BdbError, tracer.runcall, tfunc_import) + +class RunTestCase(BaseTestCase): + """Test run, runeval and set_trace.""" + + def test_run_step(self): + # Check that the bdb 'run' method stops at the first line event. + code = """ + lno = 2 + """ + self.expect_set = [ + ('line', 2, ''), ('step', ), + ('return', 2, ''), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.run(compile(textwrap.dedent(code), '', 'exec')) + + def test_runeval_step(self): + # Test bdb 'runeval'. + code = """ + def main(): + lno = 3 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 1, ''), ('step', ), + ('call', 2, 'main'), ('step', ), + ('line', 3, 'main'), ('step', ), + ('return', 3, 'main'), ('step', ), + ('return', 1, ''), ('quit', ), + ] + import test_module + with TracerRun(self) as tracer: + tracer.runeval('test_module.main()', globals(), locals()) + +class IssuesTestCase(BaseTestCase): + """Test fixed bdb issues.""" + + def test_step_at_return_with_no_trace_in_caller(self): + # Issue #13183. + # Check that the tracer does step into the caller frame when the + # trace function is not set in that frame. + code_1 = """ + from test_module_2 import func + def main(): + func() + lno = 5 + """ + code_2 = """ + def func(): + lno = 3 + """ + modules = { + TEST_MODULE: code_1, + 'test_module_2': code_2, + } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', 'test_module_2.py'), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [])), ('step', ), + ('return', 3, 'func'), ('step', ), + ('line', 5, 'main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_next_until_return_in_generator(self): + # Issue #16596. + # Check that set_next(), set_until() and set_return() do not treat the + # `yield` and `yield from` statements as if they were returns and stop + # instead in the current frame. + code = """ + def test_gen(): + yield 0 + lno = 4 + return 123 + + def main(): + it = test_gen() + next(it) + next(it) + lno = 11 + """ + modules = { TEST_MODULE: code } + for set_type in ('next', 'until', 'return'): + with self.subTest(set_type=set_type): + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('test_gen', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'test_gen', ({1:1}, [])), (set_type, ), + ] + + if set_type == 'return': + self.expect_set.extend( + [('exception', 10, 'main', StopIteration), ('step',), + ('return', 10, 'main'), ('quit', ), + ] + ) + else: + self.expect_set.extend( + [('line', 4, 'test_gen'), ('quit', ),] + ) + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_next_command_in_generator_for_loop(self): + # Issue #16596. + code = """ + def test_gen(): + yield 0 + lno = 4 + yield 1 + return 123 + + def main(): + for i in test_gen(): + lno = 10 + lno = 11 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('test_gen', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'test_gen', ({1:1}, [])), ('next', ), + ('line', 4, 'test_gen'), ('next', ), + ('line', 5, 'test_gen'), ('next', ), + ('line', 6, 'test_gen'), ('next', ), + ('exception', 9, 'main', StopIteration), ('step', ), + ('line', 11, 'main'), ('quit', ), + + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_next_command_in_generator_with_subiterator(self): + # Issue #16596. + code = """ + def test_subgen(): + yield 0 + return 123 + + def test_gen(): + x = yield from test_subgen() + return 456 + + def main(): + for i in test_gen(): + lno = 12 + lno = 13 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('test_gen', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 7, 'test_gen', ({1:1}, [])), ('next', ), + ('line', 8, 'test_gen'), ('next', ), + ('exception', 11, 'main', StopIteration), ('step', ), + ('line', 13, 'main'), ('quit', ), + + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_return_command_in_generator_with_subiterator(self): + # Issue #16596. + code = """ + def test_subgen(): + yield 0 + return 123 + + def test_gen(): + x = yield from test_subgen() + return 456 + + def main(): + for i in test_gen(): + lno = 12 + lno = 13 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('test_subgen', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'test_subgen', ({1:1}, [])), ('return', ), + ('exception', 7, 'test_gen', StopIteration), ('return', ), + ('exception', 11, 'main', StopIteration), ('step', ), + ('line', 13, 'main'), ('quit', ), + + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + +def test_main(): + test.support.run_unittest( + StateTestCase, + RunTestCase, + BreakpointTestCase, + IssuesTestCase, + ) + +if __name__ == "__main__": + test_main() diff --git a/Lib/test/test_cgitb.py b/Lib/test/test_cgitb.py index a87a422..e299ec3 100644 --- a/Lib/test/test_cgitb.py +++ b/Lib/test/test_cgitb.py @@ -45,6 +45,7 @@ class TestCgitb(unittest.TestCase): out = out.decode(sys.getfilesystemencoding()) self.assertIn("ValueError", out) self.assertIn("Hello World", out) + self.assertIn("<module>", out) # By default we emit HTML markup. self.assertIn('

              ', out) self.assertIn('

              ', out) diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py index 2b79a17..d0b44c4 100644 --- a/Lib/test/test_coroutines.py +++ b/Lib/test/test_coroutines.py @@ -1229,7 +1229,9 @@ class CoroutineTest(unittest.TestCase): pass with self.assertRaisesRegex( - TypeError, "object int can't be used in 'await' expression"): + TypeError, + "'async with' received an object from __aenter__ " + "that does not implement __await__: int"): # it's important that __aexit__ wasn't called run_async(foo()) @@ -1241,6 +1243,7 @@ class CoroutineTest(unittest.TestCase): def __aexit__(self, *e): return 444 + # Exit with exception async def foo(): async with CM(): 1/0 @@ -1249,7 +1252,9 @@ class CoroutineTest(unittest.TestCase): run_async(foo()) except TypeError as exc: self.assertRegex( - exc.args[0], "object int can't be used in 'await' expression") + exc.args[0], + "'async with' received an object from __aexit__ " + "that does not implement __await__: int") self.assertTrue(exc.__context__ is not None) self.assertTrue(isinstance(exc.__context__, ZeroDivisionError)) else: @@ -1266,18 +1271,58 @@ class CoroutineTest(unittest.TestCase): def __aexit__(self, *e): return 456 + # Normal exit async def foo(): nonlocal CNT async with CM(): CNT += 1 + with self.assertRaisesRegex( + TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: int"): + run_async(foo()) + self.assertEqual(CNT, 1) - + # Exit with 'break' + async def foo(): + nonlocal CNT + for i in range(2): + async with CM(): + CNT += 1 + break with self.assertRaisesRegex( - TypeError, "object int can't be used in 'await' expression"): + TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: int"): + run_async(foo()) + self.assertEqual(CNT, 2) + # Exit with 'continue' + async def foo(): + nonlocal CNT + for i in range(2): + async with CM(): + CNT += 1 + continue + with self.assertRaisesRegex( + TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: int"): run_async(foo()) + self.assertEqual(CNT, 3) - self.assertEqual(CNT, 1) + # Exit with 'return' + async def foo(): + nonlocal CNT + async with CM(): + CNT += 1 + return + with self.assertRaisesRegex( + TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: int"): + run_async(foo()) + self.assertEqual(CNT, 4) def test_with_9(self): @@ -1892,6 +1937,36 @@ class CoroutineTest(unittest.TestCase): run_async(run_gen()), ([], [121])) + def test_comp_4_2(self): + async def f(it): + for i in it: + yield i + + async def run_list(): + return [i + 10 async for i in f(range(5)) if 0 < i < 4] + self.assertEqual( + run_async(run_list()), + ([], [11, 12, 13])) + + async def run_set(): + return {i + 10 async for i in f(range(5)) if 0 < i < 4} + self.assertEqual( + run_async(run_set()), + ([], {11, 12, 13})) + + async def run_dict(): + return {i + 10: i + 100 async for i in f(range(5)) if 0 < i < 4} + self.assertEqual( + run_async(run_dict()), + ([], {11: 101, 12: 102, 13: 103})) + + async def run_gen(): + gen = (i + 10 async for i in f(range(5)) if 0 < i < 4) + return [g + 100 async for g in gen] + self.assertEqual( + run_async(run_gen()), + ([], [111, 112, 113])) + def test_comp_5(self): async def f(it): for i in it: diff --git a/Lib/test/test_dbm.py b/Lib/test/test_dbm.py index f0a428d..1884b5c 100644 --- a/Lib/test/test_dbm.py +++ b/Lib/test/test_dbm.py @@ -38,8 +38,7 @@ def delete_files(): class AnyDBMTestCase: - _dict = {'0': b'', - 'a': b'Python:', + _dict = {'a': b'Python:', 'b': b'Programming', 'c': b'the', 'd': b'way', @@ -76,22 +75,29 @@ class AnyDBMTestCase: def test_anydbm_creation_n_file_exists_with_invalid_contents(self): # create an empty file test.support.create_empty_file(_fname) - - f = dbm.open(_fname, 'n') - self.addCleanup(f.close) - self.assertEqual(len(f), 0) + with dbm.open(_fname, 'n') as f: + self.assertEqual(len(f), 0) def test_anydbm_modification(self): self.init_db() f = dbm.open(_fname, 'c') self._dict['g'] = f[b'g'] = b"indented" self.read_helper(f) + # setdefault() works as in the dict interface + self.assertEqual(f.setdefault(b'xxx', b'foo'), b'foo') + self.assertEqual(f[b'xxx'], b'foo') f.close() def test_anydbm_read(self): self.init_db() f = dbm.open(_fname, 'r') self.read_helper(f) + # get() works as in the dict interface + self.assertEqual(f.get(b'a'), self._dict['a']) + self.assertEqual(f.get(b'xxx', b'foo'), b'foo') + self.assertIsNone(f.get(b'xxx')) + with self.assertRaises(KeyError): + f[b'xxx'] f.close() def test_anydbm_keys(self): @@ -100,6 +106,20 @@ class AnyDBMTestCase: keys = self.keys_helper(f) f.close() + def test_empty_value(self): + if getattr(dbm._defaultmod, 'library', None) == 'Berkeley DB': + self.skipTest("Berkeley DB doesn't distinguish the empty value " + "from the absent one") + f = dbm.open(_fname, 'c') + self.assertEqual(f.keys(), []) + f[b'empty'] = b'' + self.assertEqual(f.keys(), [b'empty']) + self.assertIn(b'empty', f) + self.assertEqual(f[b'empty'], b'') + self.assertEqual(f.get(b'empty'), b'') + self.assertEqual(f.setdefault(b'empty'), b'') + f.close() + def test_anydbm_access(self): self.init_db() f = dbm.open(_fname, 'r') diff --git a/Lib/test/test_dbm_dumb.py b/Lib/test/test_dbm_dumb.py index 2606ba7..58173a2 100644 --- a/Lib/test/test_dbm_dumb.py +++ b/Lib/test/test_dbm_dumb.py @@ -74,6 +74,9 @@ class DumbDBMTestCase(unittest.TestCase): f = dumbdbm.open(_fname, 'w') self._dict[b'g'] = f[b'g'] = b"indented" self.read_helper(f) + # setdefault() works as in the dict interface + self.assertEqual(f.setdefault(b'xxx', b'foo'), b'foo') + self.assertEqual(f[b'xxx'], b'foo') f.close() def test_dumbdbm_read(self): @@ -86,6 +89,12 @@ class DumbDBMTestCase(unittest.TestCase): with self.assertWarnsRegex(DeprecationWarning, 'The database is opened for reading only'): del f[b'a'] + # get() works as in the dict interface + self.assertEqual(f.get(b'b'), self._dict[b'b']) + self.assertEqual(f.get(b'xxx', b'foo'), b'foo') + self.assertIsNone(f.get(b'xxx')) + with self.assertRaises(KeyError): + f[b'xxx'] f.close() def test_dumbdbm_keys(self): diff --git a/Lib/test/test_dbm_gnu.py b/Lib/test/test_dbm_gnu.py index d96df92..463d343 100644 --- a/Lib/test/test_dbm_gnu.py +++ b/Lib/test/test_dbm_gnu.py @@ -32,9 +32,12 @@ class TestGdbm(unittest.TestCase): self.assertIn(key, key_set) key_set.remove(key) key = self.g.nextkey(key) - self.assertRaises(KeyError, lambda: self.g['xxx']) # get() and setdefault() work as in the dict interface + self.assertEqual(self.g.get(b'a'), b'b') + self.assertIsNone(self.g.get(b'xxx')) self.assertEqual(self.g.get(b'xxx', b'foo'), b'foo') + with self.assertRaises(KeyError): + self.g['xxx'] self.assertEqual(self.g.setdefault(b'xxx', b'foo'), b'foo') self.assertEqual(self.g[b'xxx'], b'foo') diff --git a/Lib/test/test_dbm_ndbm.py b/Lib/test/test_dbm_ndbm.py index fb7d0e8..49b88f5 100644 --- a/Lib/test/test_dbm_ndbm.py +++ b/Lib/test/test_dbm_ndbm.py @@ -18,7 +18,7 @@ class DbmTestCase(unittest.TestCase): def test_keys(self): self.d = dbm.ndbm.open(self.filename, 'c') - self.assertTrue(self.d.keys() == []) + self.assertEqual(self.d.keys(), []) self.d['a'] = 'b' self.d[b'bytes'] = b'data' self.d['12345678910'] = '019237410982340912840198242' @@ -26,6 +26,28 @@ class DbmTestCase(unittest.TestCase): self.assertIn('a', self.d) self.assertIn(b'a', self.d) self.assertEqual(self.d[b'bytes'], b'data') + # get() and setdefault() work as in the dict interface + self.assertEqual(self.d.get(b'a'), b'b') + self.assertIsNone(self.d.get(b'xxx')) + self.assertEqual(self.d.get(b'xxx', b'foo'), b'foo') + with self.assertRaises(KeyError): + self.d['xxx'] + self.assertEqual(self.d.setdefault(b'xxx', b'foo'), b'foo') + self.assertEqual(self.d[b'xxx'], b'foo') + self.d.close() + + def test_empty_value(self): + if dbm.ndbm.library == 'Berkeley DB': + self.skipTest("Berkeley DB doesn't distinguish the empty value " + "from the absent one") + self.d = dbm.ndbm.open(self.filename, 'c') + self.assertEqual(self.d.keys(), []) + self.d['empty'] = '' + self.assertEqual(self.d.keys(), [b'empty']) + self.assertIn(b'empty', self.d) + self.assertEqual(self.d[b'empty'], b'') + self.assertEqual(self.d.get(b'empty'), b'') + self.assertEqual(self.d.setdefault(b'empty'), b'') self.d.close() def test_modes(self): diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 57094ad..45d80ce 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -4450,19 +4450,19 @@ class Coverage(unittest.TestCase): def test_round(self): # Python3 behavior: round() returns Decimal Decimal = self.decimal.Decimal - getcontext = self.decimal.getcontext + localcontext = self.decimal.localcontext - c = getcontext() - c.prec = 28 + with localcontext() as c: + c.prec = 28 - self.assertEqual(str(Decimal("9.99").__round__()), "10") - self.assertEqual(str(Decimal("9.99e-5").__round__()), "0") - self.assertEqual(str(Decimal("1.23456789").__round__(5)), "1.23457") - self.assertEqual(str(Decimal("1.2345").__round__(10)), "1.2345000000") - self.assertEqual(str(Decimal("1.2345").__round__(-10)), "0E+10") + self.assertEqual(str(Decimal("9.99").__round__()), "10") + self.assertEqual(str(Decimal("9.99e-5").__round__()), "0") + self.assertEqual(str(Decimal("1.23456789").__round__(5)), "1.23457") + self.assertEqual(str(Decimal("1.2345").__round__(10)), "1.2345000000") + self.assertEqual(str(Decimal("1.2345").__round__(-10)), "0E+10") - self.assertRaises(TypeError, Decimal("1.23").__round__, "5") - self.assertRaises(TypeError, Decimal("1.23").__round__, 5, 8) + self.assertRaises(TypeError, Decimal("1.23").__round__, "5") + self.assertRaises(TypeError, Decimal("1.23").__round__, 5, 8) def test_create_decimal(self): c = self.decimal.Context() @@ -5406,7 +5406,7 @@ class CWhitebox(unittest.TestCase): # SSIZE_MIN x = (1, (), -sys.maxsize-1) - self.assertEqual(str(c.create_decimal(x)), '-0E-1000026') + self.assertEqual(str(c.create_decimal(x)), '-0E-1000007') self.assertRaises(InvalidOperation, Decimal, x) x = (1, (0, 1, 2), -sys.maxsize-1) diff --git a/Lib/test/test_difflib.py b/Lib/test/test_difflib.py index aaefe6d..745ccbd 100644 --- a/Lib/test/test_difflib.py +++ b/Lib/test/test_difflib.py @@ -93,6 +93,14 @@ class TestSFbugs(unittest.TestCase): self.assertEqual("+ \t\tI am a bug", diff[2]) self.assertEqual("? +\n", diff[3]) + def test_mdiff_catch_stop_iteration(self): + # Issue #33224 + self.assertEqual( + list(difflib._mdiff(["2"], ["3"], 1)), + [((1, '\x00-2\x01'), (1, '\x00+3\x01'), True)], + ) + + patch914575_from1 = """ 1. Beautiful is beTTer than ugly. 2. Explicit is better than implicit. diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index b9b5ac2..1cee368 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -597,7 +597,7 @@ Filename: (.*) Argument count: 0 Kw-only arguments: 0 Number of locals: 2 -Stack size: 17 +Stack size: 16 Flags: OPTIMIZED, NEWLOCALS, NOFREE, COROUTINE Constants: 0: None diff --git a/Lib/test/test_file.py b/Lib/test/test_file.py index 65be30f..9890b8c 100644 --- a/Lib/test/test_file.py +++ b/Lib/test/test_file.py @@ -8,6 +8,7 @@ import io import _pyio as pyio from test.support import TESTFN +from test import support from collections import UserList class AutoFileTests: @@ -19,7 +20,7 @@ class AutoFileTests: def tearDown(self): if self.f: self.f.close() - os.remove(TESTFN) + support.unlink(TESTFN) def testWeakRefs(self): # verify weak references @@ -137,8 +138,12 @@ class PyAutoFileTests(AutoFileTests, unittest.TestCase): class OtherFileTests: + def tearDown(self): + support.unlink(TESTFN) + def testModeStrings(self): # check invalid mode strings + self.open(TESTFN, 'wb').close() for mode in ("", "aU", "wU+", "U+", "+U", "rU+"): try: f = self.open(TESTFN, mode) @@ -185,7 +190,6 @@ class OtherFileTests: # SF bug # "file.truncate fault on windows" - os.unlink(TESTFN) f = self.open(TESTFN, 'wb') try: @@ -209,7 +213,6 @@ class OtherFileTests: self.fail("File size after ftruncate wrong %d" % size) finally: f.close() - os.unlink(TESTFN) def testIteration(self): # Test the complex interaction when mixing file-iteration and the @@ -230,87 +233,84 @@ class OtherFileTests: methods = [("readline", ()), ("read", ()), ("readlines", ()), ("readinto", (array("b", b" "*100),))] + # Prepare the testfile + bag = self.open(TESTFN, "wb") + bag.write(filler * nchunks) + bag.writelines(testlines) + bag.close() + # Test for appropriate errors mixing read* and iteration + for methodname, args in methods: + f = self.open(TESTFN, 'rb') + if next(f) != filler: + self.fail, "Broken testfile" + meth = getattr(f, methodname) + meth(*args) # This simply shouldn't fail + f.close() + + # Test to see if harmless (by accident) mixing of read* and + # iteration still works. This depends on the size of the internal + # iteration buffer (currently 8192,) but we can test it in a + # flexible manner. Each line in the bag o' ham is 4 bytes + # ("h", "a", "m", "\n"), so 4096 lines of that should get us + # exactly on the buffer boundary for any power-of-2 buffersize + # between 4 and 16384 (inclusive). + f = self.open(TESTFN, 'rb') + for i in range(nchunks): + next(f) + testline = testlines.pop(0) try: - # Prepare the testfile - bag = self.open(TESTFN, "wb") - bag.write(filler * nchunks) - bag.writelines(testlines) - bag.close() - # Test for appropriate errors mixing read* and iteration - for methodname, args in methods: - f = self.open(TESTFN, 'rb') - if next(f) != filler: - self.fail, "Broken testfile" - meth = getattr(f, methodname) - meth(*args) # This simply shouldn't fail - f.close() + line = f.readline() + except ValueError: + self.fail("readline() after next() with supposedly empty " + "iteration-buffer failed anyway") + if line != testline: + self.fail("readline() after next() with empty buffer " + "failed. Got %r, expected %r" % (line, testline)) + testline = testlines.pop(0) + buf = array("b", b"\x00" * len(testline)) + try: + f.readinto(buf) + except ValueError: + self.fail("readinto() after next() with supposedly empty " + "iteration-buffer failed anyway") + line = buf.tobytes() + if line != testline: + self.fail("readinto() after next() with empty buffer " + "failed. Got %r, expected %r" % (line, testline)) + + testline = testlines.pop(0) + try: + line = f.read(len(testline)) + except ValueError: + self.fail("read() after next() with supposedly empty " + "iteration-buffer failed anyway") + if line != testline: + self.fail("read() after next() with empty buffer " + "failed. Got %r, expected %r" % (line, testline)) + try: + lines = f.readlines() + except ValueError: + self.fail("readlines() after next() with supposedly empty " + "iteration-buffer failed anyway") + if lines != testlines: + self.fail("readlines() after next() with empty buffer " + "failed. Got %r, expected %r" % (line, testline)) + f.close() - # Test to see if harmless (by accident) mixing of read* and - # iteration still works. This depends on the size of the internal - # iteration buffer (currently 8192,) but we can test it in a - # flexible manner. Each line in the bag o' ham is 4 bytes - # ("h", "a", "m", "\n"), so 4096 lines of that should get us - # exactly on the buffer boundary for any power-of-2 buffersize - # between 4 and 16384 (inclusive). - f = self.open(TESTFN, 'rb') - for i in range(nchunks): - next(f) - testline = testlines.pop(0) - try: - line = f.readline() - except ValueError: - self.fail("readline() after next() with supposedly empty " - "iteration-buffer failed anyway") - if line != testline: - self.fail("readline() after next() with empty buffer " - "failed. Got %r, expected %r" % (line, testline)) - testline = testlines.pop(0) - buf = array("b", b"\x00" * len(testline)) + # Reading after iteration hit EOF shouldn't hurt either + f = self.open(TESTFN, 'rb') + try: + for line in f: + pass try: + f.readline() f.readinto(buf) + f.read() + f.readlines() except ValueError: - self.fail("readinto() after next() with supposedly empty " - "iteration-buffer failed anyway") - line = buf.tobytes() - if line != testline: - self.fail("readinto() after next() with empty buffer " - "failed. Got %r, expected %r" % (line, testline)) - - testline = testlines.pop(0) - try: - line = f.read(len(testline)) - except ValueError: - self.fail("read() after next() with supposedly empty " - "iteration-buffer failed anyway") - if line != testline: - self.fail("read() after next() with empty buffer " - "failed. Got %r, expected %r" % (line, testline)) - try: - lines = f.readlines() - except ValueError: - self.fail("readlines() after next() with supposedly empty " - "iteration-buffer failed anyway") - if lines != testlines: - self.fail("readlines() after next() with empty buffer " - "failed. Got %r, expected %r" % (line, testline)) - f.close() - - # Reading after iteration hit EOF shouldn't hurt either - f = self.open(TESTFN, 'rb') - try: - for line in f: - pass - try: - f.readline() - f.readinto(buf) - f.read() - f.readlines() - except ValueError: - self.fail("read* failed after next() consumed file") - finally: - f.close() + self.fail("read* failed after next() consumed file") finally: - os.unlink(TESTFN) + f.close() class COtherFileTests(OtherFileTests, unittest.TestCase): open = io.open @@ -319,11 +319,5 @@ class PyOtherFileTests(OtherFileTests, unittest.TestCase): open = staticmethod(pyio.open) -def tearDownModule(): - # Historically, these tests have been sloppy about removing TESTFN. - # So get rid of it no matter what. - if os.path.exists(TESTFN): - os.unlink(TESTFN) - if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py index 565633f..bb74982 100644 --- a/Lib/test/test_fileinput.py +++ b/Lib/test/test_fileinput.py @@ -8,6 +8,7 @@ import re import fileinput import collections import builtins +import tempfile import unittest try: @@ -32,20 +33,15 @@ from unittest import mock # all the work, and a few functions (input, etc.) that use a global _state # variable. -# Write lines (a list of lines) to temp file number i, and return the -# temp file's name. -def writeTmp(i, lines, mode='w'): # opening in text mode is the default - name = TESTFN + str(i) - f = open(name, mode) - for line in lines: - f.write(line) - f.close() - return name - -def remove_tempfiles(*names): - for name in names: - if name: - safe_unlink(name) +class BaseTests: + # Write a content (str or bytes) to temp file, and return the + # temp file's name. + def writeTmp(self, content, *, mode='w'): # opening in text mode is the default + fd, name = tempfile.mkstemp() + self.addCleanup(support.unlink, name) + with open(fd, mode) as f: + f.write(content) + return name class LineReader: @@ -83,23 +79,19 @@ class LineReader: def close(self): pass -class BufferSizesTests(unittest.TestCase): +class BufferSizesTests(BaseTests, unittest.TestCase): def test_buffer_sizes(self): # First, run the tests with default and teeny buffer size. for round, bs in (0, 0), (1, 30): - t1 = t2 = t3 = t4 = None - try: - t1 = writeTmp(1, ["Line %s of file 1\n" % (i+1) for i in range(15)]) - t2 = writeTmp(2, ["Line %s of file 2\n" % (i+1) for i in range(10)]) - t3 = writeTmp(3, ["Line %s of file 3\n" % (i+1) for i in range(5)]) - t4 = writeTmp(4, ["Line %s of file 4\n" % (i+1) for i in range(1)]) - if bs: - with self.assertWarns(DeprecationWarning): - self.buffer_size_test(t1, t2, t3, t4, bs, round) - else: + t1 = self.writeTmp(''.join("Line %s of file 1\n" % (i+1) for i in range(15))) + t2 = self.writeTmp(''.join("Line %s of file 2\n" % (i+1) for i in range(10))) + t3 = self.writeTmp(''.join("Line %s of file 3\n" % (i+1) for i in range(5))) + t4 = self.writeTmp(''.join("Line %s of file 4\n" % (i+1) for i in range(1))) + if bs: + with self.assertWarns(DeprecationWarning): self.buffer_size_test(t1, t2, t3, t4, bs, round) - finally: - remove_tempfiles(t1, t2, t3, t4) + else: + self.buffer_size_test(t1, t2, t3, t4, bs, round) def buffer_size_test(self, t1, t2, t3, t4, bs=0, round=0): pat = re.compile(r'LINE (\d+) OF FILE (\d+)') @@ -186,74 +178,59 @@ class UnconditionallyRaise: self.invoked = True raise self.exception_type() -class FileInputTests(unittest.TestCase): +class FileInputTests(BaseTests, unittest.TestCase): def test_zero_byte_files(self): - t1 = t2 = t3 = t4 = None - try: - t1 = writeTmp(1, [""]) - t2 = writeTmp(2, [""]) - t3 = writeTmp(3, ["The only line there is.\n"]) - t4 = writeTmp(4, [""]) - fi = FileInput(files=(t1, t2, t3, t4)) - - line = fi.readline() - self.assertEqual(line, 'The only line there is.\n') - self.assertEqual(fi.lineno(), 1) - self.assertEqual(fi.filelineno(), 1) - self.assertEqual(fi.filename(), t3) - - line = fi.readline() - self.assertFalse(line) - self.assertEqual(fi.lineno(), 1) - self.assertEqual(fi.filelineno(), 0) - self.assertEqual(fi.filename(), t4) - fi.close() - finally: - remove_tempfiles(t1, t2, t3, t4) + t1 = self.writeTmp("") + t2 = self.writeTmp("") + t3 = self.writeTmp("The only line there is.\n") + t4 = self.writeTmp("") + fi = FileInput(files=(t1, t2, t3, t4)) + + line = fi.readline() + self.assertEqual(line, 'The only line there is.\n') + self.assertEqual(fi.lineno(), 1) + self.assertEqual(fi.filelineno(), 1) + self.assertEqual(fi.filename(), t3) + + line = fi.readline() + self.assertFalse(line) + self.assertEqual(fi.lineno(), 1) + self.assertEqual(fi.filelineno(), 0) + self.assertEqual(fi.filename(), t4) + fi.close() def test_files_that_dont_end_with_newline(self): - t1 = t2 = None - try: - t1 = writeTmp(1, ["A\nB\nC"]) - t2 = writeTmp(2, ["D\nE\nF"]) - fi = FileInput(files=(t1, t2)) - lines = list(fi) - self.assertEqual(lines, ["A\n", "B\n", "C", "D\n", "E\n", "F"]) - self.assertEqual(fi.filelineno(), 3) - self.assertEqual(fi.lineno(), 6) - finally: - remove_tempfiles(t1, t2) + t1 = self.writeTmp("A\nB\nC") + t2 = self.writeTmp("D\nE\nF") + fi = FileInput(files=(t1, t2)) + lines = list(fi) + self.assertEqual(lines, ["A\n", "B\n", "C", "D\n", "E\n", "F"]) + self.assertEqual(fi.filelineno(), 3) + self.assertEqual(fi.lineno(), 6) ## def test_unicode_filenames(self): ## # XXX A unicode string is always returned by writeTmp. ## # So is this needed? -## try: -## t1 = writeTmp(1, ["A\nB"]) -## encoding = sys.getfilesystemencoding() -## if encoding is None: -## encoding = 'ascii' -## fi = FileInput(files=str(t1, encoding)) -## lines = list(fi) -## self.assertEqual(lines, ["A\n", "B"]) -## finally: -## remove_tempfiles(t1) +## t1 = self.writeTmp("A\nB") +## encoding = sys.getfilesystemencoding() +## if encoding is None: +## encoding = 'ascii' +## fi = FileInput(files=str(t1, encoding)) +## lines = list(fi) +## self.assertEqual(lines, ["A\n", "B"]) def test_fileno(self): - t1 = t2 = None - try: - t1 = writeTmp(1, ["A\nB"]) - t2 = writeTmp(2, ["C\nD"]) - fi = FileInput(files=(t1, t2)) - self.assertEqual(fi.fileno(), -1) - line =next( fi) - self.assertNotEqual(fi.fileno(), -1) - fi.nextfile() - self.assertEqual(fi.fileno(), -1) - line = list(fi) - self.assertEqual(fi.fileno(), -1) - finally: - remove_tempfiles(t1, t2) + t1 = self.writeTmp("A\nB") + t2 = self.writeTmp("C\nD") + fi = FileInput(files=(t1, t2)) + self.assertEqual(fi.fileno(), -1) + line = next(fi) + self.assertNotEqual(fi.fileno(), -1) + fi.nextfile() + self.assertEqual(fi.fileno(), -1) + line = list(fi) + self.assertEqual(fi.fileno(), -1) def test_opening_mode(self): try: @@ -262,17 +239,13 @@ class FileInputTests(unittest.TestCase): self.fail("FileInput should reject invalid mode argument") except ValueError: pass - t1 = None - try: - # try opening in universal newline mode - t1 = writeTmp(1, [b"A\nB\r\nC\rD"], mode="wb") - with check_warnings(('', DeprecationWarning)): - fi = FileInput(files=t1, mode="U") - with check_warnings(('', DeprecationWarning)): - lines = list(fi) - self.assertEqual(lines, ["A\n", "B\n", "C\n", "D"]) - finally: - remove_tempfiles(t1) + # try opening in universal newline mode + t1 = self.writeTmp(b"A\nB\r\nC\rD", mode="wb") + with check_warnings(('', DeprecationWarning)): + fi = FileInput(files=t1, mode="U") + with check_warnings(('', DeprecationWarning)): + lines = list(fi) + self.assertEqual(lines, ["A\n", "B\n", "C\n", "D"]) def test_stdin_binary_mode(self): with mock.patch('sys.stdin') as m_stdin: @@ -313,8 +286,7 @@ class FileInputTests(unittest.TestCase): self.invoked = True return open(*args) - t = writeTmp(1, ["\n"]) - self.addCleanup(remove_tempfiles, t) + t = self.writeTmp("\n") custom_open_hook = CustomOpenHook() with FileInput([t], openhook=custom_open_hook) as fi: fi.readline() @@ -357,27 +329,22 @@ class FileInputTests(unittest.TestCase): self.assertEqual(fi.readline(), b'') def test_context_manager(self): - try: - t1 = writeTmp(1, ["A\nB\nC"]) - t2 = writeTmp(2, ["D\nE\nF"]) - with FileInput(files=(t1, t2)) as fi: - lines = list(fi) - self.assertEqual(lines, ["A\n", "B\n", "C", "D\n", "E\n", "F"]) - self.assertEqual(fi.filelineno(), 3) - self.assertEqual(fi.lineno(), 6) - self.assertEqual(fi._files, ()) - finally: - remove_tempfiles(t1, t2) + t1 = self.writeTmp("A\nB\nC") + t2 = self.writeTmp("D\nE\nF") + with FileInput(files=(t1, t2)) as fi: + lines = list(fi) + self.assertEqual(lines, ["A\n", "B\n", "C", "D\n", "E\n", "F"]) + self.assertEqual(fi.filelineno(), 3) + self.assertEqual(fi.lineno(), 6) + self.assertEqual(fi._files, ()) def test_close_on_exception(self): + t1 = self.writeTmp("") try: - t1 = writeTmp(1, [""]) with FileInput(files=t1) as fi: raise OSError except OSError: self.assertEqual(fi._files, ()) - finally: - remove_tempfiles(t1) def test_empty_files_list_specified_to_constructor(self): with FileInput(files=[]) as fi: @@ -386,8 +353,7 @@ class FileInputTests(unittest.TestCase): def test__getitem__(self): """Tests invoking FileInput.__getitem__() with the current line number""" - t = writeTmp(1, ["line1\n", "line2\n"]) - self.addCleanup(remove_tempfiles, t) + t = self.writeTmp("line1\nline2\n") with FileInput(files=[t]) as fi: retval1 = fi[0] self.assertEqual(retval1, "line1\n") @@ -397,8 +363,7 @@ class FileInputTests(unittest.TestCase): def test__getitem__invalid_key(self): """Tests invoking FileInput.__getitem__() with an index unequal to the line number""" - t = writeTmp(1, ["line1\n", "line2\n"]) - self.addCleanup(remove_tempfiles, t) + t = self.writeTmp("line1\nline2\n") with FileInput(files=[t]) as fi: with self.assertRaises(RuntimeError) as cm: fi[1] @@ -407,8 +372,7 @@ class FileInputTests(unittest.TestCase): def test__getitem__eof(self): """Tests invoking FileInput.__getitem__() with the line number but at end-of-input""" - t = writeTmp(1, []) - self.addCleanup(remove_tempfiles, t) + t = self.writeTmp('') with FileInput(files=[t]) as fi: with self.assertRaises(IndexError) as cm: fi[0] @@ -422,8 +386,8 @@ class FileInputTests(unittest.TestCase): os_unlink_orig = os.unlink os_unlink_replacement = UnconditionallyRaise(OSError) try: - t = writeTmp(1, ["\n"]) - self.addCleanup(remove_tempfiles, t) + t = self.writeTmp("\n") + self.addCleanup(support.unlink, t + '.bak') with FileInput(files=[t], inplace=True) as fi: next(fi) # make sure the file is opened os.unlink = os_unlink_replacement @@ -442,8 +406,7 @@ class FileInputTests(unittest.TestCase): os_fstat_orig = os.fstat os_fstat_replacement = UnconditionallyRaise(OSError) try: - t = writeTmp(1, ["\n"]) - self.addCleanup(remove_tempfiles, t) + t = self.writeTmp("\n") with FileInput(files=[t], inplace=True) as fi: os.fstat = os_fstat_replacement fi.readline() @@ -462,8 +425,7 @@ class FileInputTests(unittest.TestCase): os_chmod_orig = os.chmod os_chmod_replacement = UnconditionallyRaise(OSError) try: - t = writeTmp(1, ["\n"]) - self.addCleanup(remove_tempfiles, t) + t = self.writeTmp("\n") with FileInput(files=[t], inplace=True) as fi: os.chmod = os_chmod_replacement fi.readline() @@ -482,8 +444,7 @@ class FileInputTests(unittest.TestCase): self.__call__() unconditionally_raise_ValueError = FilenoRaisesValueError() - t = writeTmp(1, ["\n"]) - self.addCleanup(remove_tempfiles, t) + t = self.writeTmp("\n") with FileInput(files=[t]) as fi: file_backup = fi._file try: @@ -530,6 +491,7 @@ class FileInputTests(unittest.TestCase): self.assertRaises(StopIteration, next, fi) self.assertEqual(src.linesread, []) + class MockFileInput: """A class that mocks out fileinput.FileInput for use during unit tests""" diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py index d9ceeb5..67c2a6d 100644 --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -10,12 +10,18 @@ import types from test import support -_testcapi = support.import_module('_testcapi') +try: + import _testcapi +except ImportError: + _testcapi = None # This tests to make sure that if a SIGINT arrives just before we send into a # yield from chain, the KeyboardInterrupt is raised in the innermost # generator (see bpo-30039). +@unittest.skipUnless(_testcapi is not None and + hasattr(_testcapi, "raise_SIGINT_then_send_None"), + "needs _testcapi.raise_SIGINT_then_send_None") class SignalAndYieldFromTest(unittest.TestCase): def generator1(self): diff --git a/Lib/test/test_http_cookies.py b/Lib/test/test_http_cookies.py index 2432e0b..234b9ee 100644 --- a/Lib/test/test_http_cookies.py +++ b/Lib/test/test_http_cookies.py @@ -216,6 +216,16 @@ class CookieTests(unittest.TestCase): with self.assertRaises(cookies.CookieError): C.load(rawdata) + def test_comment_quoting(self): + c = cookies.SimpleCookie() + c['foo'] = '\N{COPYRIGHT SIGN}' + self.assertEqual(str(c['foo']), 'Set-Cookie: foo="\\251"') + c['foo']['comment'] = 'comment \N{COPYRIGHT SIGN}' + self.assertEqual( + str(c['foo']), + 'Set-Cookie: foo="\\251"; Comment="comment \\251"' + ) + class MorselTests(unittest.TestCase): """Tests for the Morsel object.""" diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py index da05da5..6db1f11 100644 --- a/Lib/test/test_idle.py +++ b/Lib/test/test_idle.py @@ -1,19 +1,23 @@ import unittest from test.support import import_module -# Skip test if _thread or _tkinter wasn't built, if idlelib is missing, -# or if tcl/tk is not the 8.5+ needed for ttk widgets. -import_module('threading') # imported by PyShell, imports _thread +# For 3.6, skip test_idle if threads are not supported. +import_module('threading') # Imported by PyShell, imports _thread. + +# Skip test_idle if _tkinter wasn't built, if tkinter is missing, +# if tcl/tk is not the 8.5+ needed for ttk widgets, +# or if idlelib is missing (not installed). tk = import_module('tkinter') # imports _tkinter if tk.TkVersion < 8.5: raise unittest.SkipTest("IDLE requires tk 8.5 or later.") idlelib = import_module('idlelib') -# Before test imports, tell IDLE to avoid changing the environment. +# Before importing and executing more of idlelib, +# tell IDLE to avoid changing the environment. idlelib.testing = True -# unittest.main and test.libregrtest.runtest.runtest_inner -# call load_tests, when present, to discover tests to run. +# Unittest.main and test.libregrtest.runtest.runtest_inner +# call load_tests, when present here, to discover tests to run. from idlelib.idle_test import load_tests if __name__ == '__main__': diff --git a/Lib/test/test_importlib/extension/test_loader.py b/Lib/test/test_importlib/extension/test_loader.py index 8d20040..9ad05fa 100644 --- a/Lib/test/test_importlib/extension/test_loader.py +++ b/Lib/test/test_importlib/extension/test_loader.py @@ -9,7 +9,7 @@ import types import unittest import importlib.util import importlib - +from test.support.script_helper import assert_python_failure class LoaderTests(abc.LoaderTests): @@ -267,6 +267,29 @@ class MultiPhaseExtensionModuleTests(abc.LoaderTests): self.assertEqual(module.__name__, name) self.assertEqual(module.__doc__, "Module named in %s" % lang) + @unittest.skipIf(not hasattr(sys, 'gettotalrefcount'), + '--with-pydebug has to be enabled for this test') + def test_bad_traverse(self): + ''' Issue #32374: Test that traverse fails when accessing per-module + state before Py_mod_exec was executed. + (Multiphase initialization modules only) + ''' + script = """if True: + try: + from test import support + import importlib.util as util + spec = util.find_spec('_testmultiphase') + spec.name = '_testmultiphase_with_bad_traverse' + + with support.SuppressCrashReport(): + m = spec.loader.create_module(spec) + except: + # Prevent Python-level exceptions from + # ending the process with non-zero status + # (We are testing for a crash in C-code) + pass""" + assert_python_failure("-c", script) + (Frozen_MultiPhaseExtensionModuleTests, Source_MultiPhaseExtensionModuleTests diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index 126b26e..d64bd45 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1390,6 +1390,20 @@ class TestGetcallargsFunctions(unittest.TestCase): with self.assertRaisesRegex(TypeError, "'a', 'b' and 'c'"): inspect.getcallargs(f6) + # bpo-33197 + with self.assertRaisesRegex(ValueError, + 'variadic keyword parameters cannot' + ' have default values'): + inspect.Parameter("foo", kind=inspect.Parameter.VAR_KEYWORD, + default=42) + with self.assertRaisesRegex(ValueError, + "value 5 is not a valid Parameter.kind"): + inspect.Parameter("bar", kind=5, default=42) + + with self.assertRaisesRegex(TypeError, + 'name must be a str, not a int'): + inspect.Parameter(123, kind=4) + class TestGetcallargsMethods(TestGetcallargsFunctions): def setUp(self): @@ -2979,7 +2993,8 @@ class TestParameterObject(unittest.TestCase): self.assertIs(p.annotation, p.empty) self.assertEqual(p.kind, inspect.Parameter.POSITIONAL_ONLY) - with self.assertRaisesRegex(ValueError, 'invalid value'): + with self.assertRaisesRegex(ValueError, "value '123' is " + "not a valid Parameter.kind"): inspect.Parameter('foo', default=10, kind='123') with self.assertRaisesRegex(ValueError, 'not a valid parameter name'): @@ -3069,7 +3084,9 @@ class TestParameterObject(unittest.TestCase): self.assertEqual(p2.kind, p2.POSITIONAL_OR_KEYWORD) self.assertNotEqual(p2, p) - with self.assertRaisesRegex(ValueError, 'invalid value for'): + with self.assertRaisesRegex(ValueError, + "value " + "is not a valid Parameter.kind"): p2 = p2.replace(kind=p2.empty) p2 = p2.replace(kind=p2.KEYWORD_ONLY) @@ -3082,7 +3099,9 @@ class TestParameterObject(unittest.TestCase): @cpython_only def test_signature_parameter_implicit(self): with self.assertRaisesRegex(ValueError, - 'implicit arguments must be passed in as'): + 'implicit arguments must be passed as ' + 'positional or keyword arguments, ' + 'not positional-only'): inspect.Parameter('.0', kind=inspect.Parameter.POSITIONAL_ONLY) param = inspect.Parameter( diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 3f3b390..0b5b033 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -1496,6 +1496,7 @@ class CBufferedReaderTest(BufferedReaderTest, SizeofTest): def test_garbage_collection(self): # C BufferedReader objects are collected. # The Python version has __del__, so it ends into gc.garbage instead + self.addCleanup(support.unlink, support.TESTFN) with support.check_warnings(('', ResourceWarning)): rawio = self.FileIO(support.TESTFN, "w+b") f = self.tp(rawio) @@ -1698,6 +1699,7 @@ class BufferedWriterTest(unittest.TestCase, CommonBufferedTests): def test_truncate(self): # Truncate implicitly flushes the buffer. + self.addCleanup(support.unlink, support.TESTFN) with self.open(support.TESTFN, self.write_mode, buffering=0) as raw: bufio = self.tp(raw, 8) bufio.write(b"abcdef") @@ -1710,6 +1712,7 @@ class BufferedWriterTest(unittest.TestCase, CommonBufferedTests): # Ensure that truncate preserves the file position after # writes longer than the buffer size. # Issue: https://bugs.python.org/issue32228 + self.addCleanup(support.unlink, support.TESTFN) with self.open(support.TESTFN, "wb") as f: # Fill with some buffer f.write(b'\x00' * 10000) @@ -1820,6 +1823,7 @@ class CBufferedWriterTest(BufferedWriterTest, SizeofTest): # C BufferedWriter objects are collected, and collecting them flushes # all data to disk. # The Python version has __del__, so it ends into gc.garbage instead + self.addCleanup(support.unlink, support.TESTFN) with support.check_warnings(('', ResourceWarning)): rawio = self.FileIO(support.TESTFN, "w+b") f = self.tp(rawio) @@ -3956,6 +3960,7 @@ class SignalsTest(unittest.TestCase): if isinstance(exc, RuntimeError): self.assertTrue(str(exc).startswith("reentrant call"), str(exc)) finally: + signal.alarm(0) wio.close() os.close(r) @@ -3984,6 +3989,7 @@ class SignalsTest(unittest.TestCase): # - third raw read() returns b"bar" self.assertEqual(decode(rio.read(6)), "foobar") finally: + signal.alarm(0) rio.close() os.close(w) os.close(r) @@ -4052,6 +4058,7 @@ class SignalsTest(unittest.TestCase): self.assertIsNone(error) self.assertEqual(N, sum(len(x) for x in read_results)) finally: + signal.alarm(0) write_finished = True os.close(w) os.close(r) diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py index 5d96330..1cef421 100644 --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -405,6 +405,9 @@ class AddressTestCase_v6(BaseTestCase, CommonTestMixin_v6): class NetmaskTestMixin_v4(CommonTestMixin_v4): """Input validation on interfaces and networks is very similar""" + def test_no_mask(self): + self.assertEqual(str(self.factory('1.2.3.4')), '1.2.3.4/32') + def test_split_netmask(self): addr = "1.2.3.4/32/24" with self.assertAddressError("Only one '/' permitted in %r" % addr): @@ -1039,10 +1042,30 @@ class IpaddrUnitTest(unittest.TestCase): self.assertEqual(ipaddress.IPv4Address('1.2.3.1'), hosts[0]) self.assertEqual(ipaddress.IPv4Address('1.2.3.254'), hosts[-1]) + ipv6_network = ipaddress.IPv6Network('2001:658:22a:cafe::/120') + hosts = list(ipv6_network.hosts()) + self.assertEqual(255, len(hosts)) + self.assertEqual(ipaddress.IPv6Address('2001:658:22a:cafe::1'), hosts[0]) + self.assertEqual(ipaddress.IPv6Address('2001:658:22a:cafe::ff'), hosts[-1]) + # special case where only 1 bit is left for address - self.assertEqual([ipaddress.IPv4Address('2.0.0.0'), - ipaddress.IPv4Address('2.0.0.1')], - list(ipaddress.ip_network('2.0.0.0/31').hosts())) + addrs = [ipaddress.IPv4Address('2.0.0.0'), + ipaddress.IPv4Address('2.0.0.1')] + str_args = '2.0.0.0/31' + tpl_args = ('2.0.0.0', 31) + self.assertEqual(addrs, list(ipaddress.ip_network(str_args).hosts())) + self.assertEqual(addrs, list(ipaddress.ip_network(tpl_args).hosts())) + self.assertEqual(list(ipaddress.ip_network(str_args).hosts()), + list(ipaddress.ip_network(tpl_args).hosts())) + + addrs = [ipaddress.IPv6Address('2001:658:22a:cafe::'), + ipaddress.IPv6Address('2001:658:22a:cafe::1')] + str_args = '2001:658:22a:cafe::/127' + tpl_args = ('2001:658:22a:cafe::', 127) + self.assertEqual(addrs, list(ipaddress.ip_network(str_args).hosts())) + self.assertEqual(addrs, list(ipaddress.ip_network(tpl_args).hosts())) + self.assertEqual(list(ipaddress.ip_network(str_args).hosts()), + list(ipaddress.ip_network(tpl_args).hosts())) def testFancySubnetting(self): self.assertEqual(sorted(self.ipv4_network.subnets(prefixlen_diff=3)), diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py index 1ad37ae..9317951 100644 --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -1172,6 +1172,7 @@ class TestBasicOps(unittest.TestCase): (10, 20, 3), (10, 3, 20), (10, 20), + (10, 10), (10, 3), (20,) ]: @@ -1198,6 +1199,10 @@ class TestBasicOps(unittest.TestCase): self.assertEqual(list(islice(it, 3)), list(range(3))) self.assertEqual(list(it), list(range(3, 10))) + it = iter(range(10)) + self.assertEqual(list(islice(it, 3, 3)), []) + self.assertEqual(list(it), list(range(3, 10))) + # Test invalid arguments ra = range(10) self.assertRaises(TypeError, islice, ra) @@ -1571,6 +1576,48 @@ class TestExamples(unittest.TestCase): self.assertEqual(list(takewhile(lambda x: x<5, [1,4,6,4,1])), [1,4]) +class TestPurePythonRoughEquivalents(unittest.TestCase): + + @staticmethod + def islice(iterable, *args): + s = slice(*args) + start, stop, step = s.start or 0, s.stop or sys.maxsize, s.step or 1 + it = iter(range(start, stop, step)) + try: + nexti = next(it) + except StopIteration: + # Consume *iterable* up to the *start* position. + for i, element in zip(range(start), iterable): + pass + return + try: + for i, element in enumerate(iterable): + if i == nexti: + yield element + nexti = next(it) + except StopIteration: + # Consume to *stop*. + for i, element in zip(range(i + 1, stop), iterable): + pass + + def test_islice_recipe(self): + self.assertEqual(list(self.islice('ABCDEFG', 2)), list('AB')) + self.assertEqual(list(self.islice('ABCDEFG', 2, 4)), list('CD')) + self.assertEqual(list(self.islice('ABCDEFG', 2, None)), list('CDEFG')) + self.assertEqual(list(self.islice('ABCDEFG', 0, None, 2)), list('ACEG')) + # Test items consumed. + it = iter(range(10)) + self.assertEqual(list(self.islice(it, 3)), list(range(3))) + self.assertEqual(list(it), list(range(3, 10))) + it = iter(range(10)) + self.assertEqual(list(self.islice(it, 3, 3)), []) + self.assertEqual(list(it), list(range(3, 10))) + # Test that slice finishes in predictable state. + c = count() + self.assertEqual(list(self.islice(c, 1, 3, 50)), [1]) + self.assertEqual(next(c), 3) + + class TestGC(unittest.TestCase): def makecycle(self, iterator, container): @@ -2118,6 +2165,11 @@ Samuele ... "Return first n items of the iterable as a list" ... return list(islice(iterable, n)) +>>> def prepend(value, iterator): +... "Prepend a single value in front of an iterator" +... # prepend(1, [2, 3, 4]) -> 1 2 3 4 +... return chain([value], iterator) + >>> def enumerate(iterable, start=0): ... return zip(count(start), iterable) @@ -2125,6 +2177,17 @@ Samuele ... "Return function(0), function(1), ..." ... return map(function, count(start)) +>>> import collections +>>> def consume(iterator, n=None): +... "Advance the iterator n-steps ahead. If n is None, consume entirely." +... # Use functions that consume iterators at C speed. +... if n is None: +... # feed the entire iterator into a zero-length deque +... collections.deque(iterator, maxlen=0) +... else: +... # advance to the empty slice starting at position n +... next(islice(iterator, n, n), None) + >>> def nth(iterable, n, default=None): ... "Returns the nth item or a default value" ... return next(islice(iterable, n, None), default) @@ -2259,12 +2322,23 @@ perform as purported. >>> take(10, count()) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] +>>> list(prepend(1, [2, 3, 4])) +[1, 2, 3, 4] + >>> list(enumerate('abc')) [(0, 'a'), (1, 'b'), (2, 'c')] >>> list(islice(tabulate(lambda x: 2*x), 4)) [0, 2, 4, 6] +>>> it = iter(range(10)) +>>> consume(it, 3) +>>> next(it) +3 +>>> consume(it) +>>> next(it, 'Done') +'Done' + >>> nth('abcde', 3) 'd' @@ -2353,6 +2427,7 @@ def test_main(verbose=None): test_classes = (TestBasicOps, TestVariousIteratorArgs, TestGC, RegressionTests, LengthTransparency, SubclassWithKwargsTest, TestExamples, + TestPurePythonRoughEquivalents, SizeofTest) support.run_unittest(*test_classes) diff --git a/Lib/test/test_json/test_decode.py b/Lib/test/test_json/test_decode.py index 7e568be..738f109 100644 --- a/Lib/test/test_json/test_decode.py +++ b/Lib/test/test_json/test_decode.py @@ -58,7 +58,9 @@ class TestDecode: def test_keys_reuse(self): s = '[{"a_key": 1, "b_\xe9": 2}, {"a_key": 3, "b_\xe9": 4}]' self.check_keys_reuse(s, self.loads) - self.check_keys_reuse(s, self.json.decoder.JSONDecoder().decode) + decoder = self.json.decoder.JSONDecoder() + self.check_keys_reuse(s, decoder.decode) + self.assertFalse(decoder.memo) def test_extra_data(self): s = '[1, 2, 3]5' diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py index 650d737..a5ec2c5 100644 --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -430,7 +430,7 @@ class NormalizeTest(unittest.TestCase): def test_valencia_modifier(self): self.check('ca_ES.UTF-8@valencia', 'ca_ES.UTF-8@valencia') - self.check('ca_ES@valencia', 'ca_ES.ISO8859-15@valencia') + self.check('ca_ES@valencia', 'ca_ES.UTF-8@valencia') self.check('ca@valencia', 'ca_ES.ISO8859-1@valencia') def test_devanagari_modifier(self): diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index fc06713..d341ef8 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -883,6 +883,7 @@ if threading: """ allow_reuse_address = True + _block_on_close = True def __init__(self, addr, handler, poll_interval=0.5, bind_and_activate=True): @@ -915,6 +916,8 @@ if threading: before calling :meth:`start`, so that the server will set up the socket and listen on it. """ + _block_on_close = True + def __init__(self, addr, handler, poll_interval=0.5, bind_and_activate=True): class DelegatingUDPRequestHandler(DatagramRequestHandler): @@ -1474,11 +1477,11 @@ class SocketHandlerTest(BaseTest): def tearDown(self): """Shutdown the TCP server.""" try: - if self.server: - self.server.stop(2.0) if self.sock_hdlr: self.root_logger.removeHandler(self.sock_hdlr) self.sock_hdlr.close() + if self.server: + self.server.stop(2.0) finally: BaseTest.tearDown(self) diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py index 56d85e7..c188a9d 100644 --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -734,6 +734,13 @@ class MmapTests(unittest.TestCase): self.assertRaises(ValueError, m.write_byte, 42) self.assertRaises(ValueError, m.write, b'abc') + def test_concat_repeat_exception(self): + m = mmap.mmap(-1, 16) + with self.assertRaises(TypeError): + m + m + with self.assertRaises(TypeError): + m * 2 + class LargeMmapTests(unittest.TestCase): diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py index 4a3571d..03a8a63 100644 --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -322,21 +322,19 @@ class RoundtripLegalSyntaxTestCase(unittest.TestCase): # An absolutely minimal test of position information. Better # tests would be a big project. code = "def f(x):\n return x + 1" - st1 = parser.suite(code) - st2 = st1.totuple(line_info=1, col_info=1) + st = parser.suite(code) def walk(tree): node_type = tree[0] next = tree[1] - if isinstance(next, tuple): + if isinstance(next, (tuple, list)): for elt in tree[1:]: for x in walk(elt): yield x else: yield tree - terminals = list(walk(st2)) - self.assertEqual([ + expected = [ (1, 'def', 1, 0), (1, 'f', 1, 4), (7, '(', 1, 5), @@ -352,8 +350,25 @@ class RoundtripLegalSyntaxTestCase(unittest.TestCase): (4, '', 2, 16), (6, '', 2, -1), (4, '', 2, -1), - (0, '', 2, -1)], - terminals) + (0, '', 2, -1), + ] + + self.assertEqual(list(walk(st.totuple(line_info=True, col_info=True))), + expected) + self.assertEqual(list(walk(st.totuple())), + [(t, n) for t, n, l, c in expected]) + self.assertEqual(list(walk(st.totuple(line_info=True))), + [(t, n, l) for t, n, l, c in expected]) + self.assertEqual(list(walk(st.totuple(col_info=True))), + [(t, n, c) for t, n, l, c in expected]) + self.assertEqual(list(walk(st.tolist(line_info=True, col_info=True))), + [list(x) for x in expected]) + self.assertEqual(list(walk(parser.st2tuple(st, line_info=True, + col_info=True))), + expected) + self.assertEqual(list(walk(parser.st2list(st, line_info=True, + col_info=True))), + [list(x) for x in expected]) def test_extended_unpacking(self): self.check_suite("*a = y") diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index a4d2f8d..bf9467e 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1531,7 +1531,7 @@ class _BasePathTest(object): # resolves to 'dirB/..' first before resolving to parent of dirB. self._check_resolve_relative(p, P(BASE, 'foo', 'in', 'spam'), False) # Now create absolute symlinks - d = tempfile.mkdtemp(suffix='-dirD') + d = support._longpath(tempfile.mkdtemp(suffix='-dirD')) self.addCleanup(support.rmtree, d) os.symlink(os.path.join(d), join('dirA', 'linkX')) os.symlink(join('dirB'), os.path.join(d, 'linkY')) @@ -2145,6 +2145,9 @@ class PosixPathTest(_BasePathTest, unittest.TestCase): otherhome = pwdent.pw_dir.rstrip('/') if othername != username and otherhome: break + else: + othername = username + otherhome = userhome p1 = P('~/Documents') p2 = P('~' + username + '/Documents') diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py index ca9bc62..234c855 100644 --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -217,11 +217,12 @@ class DummyPOP3Server(asyncore.dispatcher, threading.Thread): def run(self): self.active = True self.__flag.set() - while self.active and asyncore.socket_map: - self.active_lock.acquire() - asyncore.loop(timeout=0.1, count=1) - self.active_lock.release() - asyncore.close_all(ignore_all=True) + try: + while self.active and asyncore.socket_map: + with self.active_lock: + asyncore.loop(timeout=0.1, count=1) + finally: + asyncore.close_all(ignore_all=True) def stop(self): assert self.active diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index 296e328..eb3ef84 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -230,7 +230,12 @@ class PosixTester(unittest.TestCase): except OSError as inst: # issue10812, ZFS doesn't appear to support posix_fallocate, # so skip Solaris-based since they are likely to have ZFS. - if inst.errno != errno.EINVAL or not sys.platform.startswith("sunos"): + # issue33655: Also ignore EINVAL on *BSD since ZFS is also + # often used there. + if inst.errno == errno.EINVAL and sys.platform.startswith( + ('sunos', 'freebsd', 'netbsd', 'openbsd', 'gnukfreebsd')): + raise unittest.SkipTest("test may fail on ZFS filesystems") + else: raise finally: os.close(fd) diff --git a/Lib/test/test_profile.py b/Lib/test/test_profile.py index 1fc3c42..a998266 100644 --- a/Lib/test/test_profile.py +++ b/Lib/test/test_profile.py @@ -51,13 +51,18 @@ class ProfileTest(unittest.TestCase): results = self.do_profiling() expected = self.get_expected_output() self.assertEqual(results[0], 1000) + fail = [] for i, method in enumerate(self.methodnames): - if results[i+1] != expected[method]: - print("Stats.%s output for %s doesn't fit expectation!" % - (method, self.profilerclass.__name__)) - print('\n'.join(unified_diff( - results[i+1].split('\n'), - expected[method].split('\n')))) + a = expected[method] + b = results[i+1] + if a != b: + fail.append(f"\nStats.{method} output for " + f"{self.profilerclass.__name__} " + "does not fit expectation:") + fail.extend(unified_diff(a.split('\n'), b.split('\n'), + lineterm="")) + if fail: + self.fail("\n".join(fail)) def test_calling_conventions(self): # Issue #5330: profile and cProfile wouldn't report C functions called diff --git a/Lib/test/test_pty.py b/Lib/test/test_pty.py index 45ebea5..cd934bf 100644 --- a/Lib/test/test_pty.py +++ b/Lib/test/test_pty.py @@ -69,14 +69,11 @@ class PtyTest(unittest.TestCase): def setUp(self): # isatty() and close() can hang on some platforms. Set an alarm # before running the test to make sure we don't hang forever. - self.old_alarm = signal.signal(signal.SIGALRM, self.handle_sig) + old_alarm = signal.signal(signal.SIGALRM, self.handle_sig) + self.addCleanup(signal.signal, signal.SIGALRM, old_alarm) + self.addCleanup(signal.alarm, 0) signal.alarm(10) - def tearDown(self): - # remove alarm, restore old alarm handler - signal.alarm(0) - signal.signal(signal.SIGALRM, self.old_alarm) - def handle_sig(self, sig, frame): self.fail("isatty hung") diff --git a/Lib/test/test_random.py b/Lib/test/test_random.py index 3e57a82..10f431a 100644 --- a/Lib/test/test_random.py +++ b/Lib/test/test_random.py @@ -644,7 +644,10 @@ class MersenneTwister_TestBasicOps(TestBasicOps, unittest.TestCase): # Population range too large (n >= maxsize) self.gen._randbelow(maxsize+1, maxsize = maxsize) self.gen._randbelow(5640, maxsize = maxsize) - + # issue 33203: test that _randbelow raises ValueError on + # n == 0 also in its getrandbits-independent branch. + with self.assertRaises(ValueError): + self.gen._randbelow(0, maxsize=maxsize) # This might be going too far to test a single line, but because of our # noble aim of achieving 100% test coverage we need to write a case in # which the following line in Random._randbelow() gets executed: diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index f63ed64..e2b5228 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -389,6 +389,7 @@ class BaseTestCase(unittest.TestCase): def check_executed_tests(self, output, tests, skipped=(), failed=(), env_changed=(), omitted=(), + rerun=(), randomize=False, interrupted=False, fail_env_changed=False): if isinstance(tests, str): @@ -401,6 +402,8 @@ class BaseTestCase(unittest.TestCase): env_changed = [env_changed] if isinstance(omitted, str): omitted = [omitted] + if isinstance(rerun, str): + rerun = [rerun] executed = self.parse_executed_tests(output) if randomize: @@ -435,6 +438,14 @@ class BaseTestCase(unittest.TestCase): regex = list_regex('%s test%s omitted', omitted) self.check_line(output, regex) + if rerun: + regex = list_regex('%s re-run test%s', rerun) + self.check_line(output, regex) + self.check_line(output, "Re-running failed tests in verbose mode") + for name in rerun: + regex = "Re-running test %r in verbose mode" % name + self.check_line(output, regex) + good = (len(tests) - len(skipped) - len(failed) - len(omitted) - len(env_changed)) if good: @@ -446,14 +457,19 @@ class BaseTestCase(unittest.TestCase): if interrupted: self.check_line(output, 'Test suite interrupted by signal SIGINT.') + result = [] if failed: - result = 'FAILURE' - elif interrupted: - result = 'INTERRUPTED' + result.append('FAILURE') elif fail_env_changed and env_changed: - result = 'ENV CHANGED' - else: - result = 'SUCCESS' + result.append('ENV CHANGED') + if interrupted: + result.append('INTERRUPTED') + if not result: + result.append('SUCCESS') + result = ', '.join(result) + if rerun: + self.check_line(output, 'Tests result: %s' % result) + result = 'FAILURE then %s' % result self.check_line(output, 'Tests result: %s' % result) def parse_random_seed(self, output): @@ -593,6 +609,8 @@ class ProgramsTestCase(BaseTestCase): def test_pcbuild_rt(self): # PCbuild\rt.bat script = os.path.join(ROOT_DIR, r'PCbuild\rt.bat') + if not os.path.isfile(script): + self.skipTest(f'File "{script}" does not exist') rt_args = ["-q"] # Quick, don't run tests twice if platform.architecture()[0] == '64bit': rt_args.append('-x64') # 64-bit build @@ -835,22 +853,10 @@ class ArgsTestCase(BaseTestCase): import os import unittest - # Issue #25306: Disable popups and logs to stderr on assertion - # failures in MSCRT - try: - import msvcrt - msvcrt.CrtSetReportMode - except (ImportError, AttributeError): - # no Windows, o release build - pass - else: - for m in [msvcrt.CRT_WARN, msvcrt.CRT_ERROR, msvcrt.CRT_ASSERT]: - msvcrt.CrtSetReportMode(m, 0) - class FDLeakTest(unittest.TestCase): def test_leak(self): fd = os.open(__file__, os.O_RDONLY) - # bug: never cloes the file descriptor + # bug: never close the file descriptor """) self.check_leak(code, 'file descriptors') @@ -962,6 +968,21 @@ class ArgsTestCase(BaseTestCase): self.check_executed_tests(output, [testname], env_changed=testname, fail_env_changed=True) + def test_rerun_fail(self): + code = textwrap.dedent(""" + import unittest + + class Tests(unittest.TestCase): + def test_bug(self): + # test always fail + self.fail("bug") + """) + testname = self.create_test(code=code) + + output = self.run_tests("-w", testname, exitcode=2) + self.check_executed_tests(output, [testname], + failed=testname, rerun=testname) + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py index e47344c..3f30185 100644 --- a/Lib/test/test_robotparser.py +++ b/Lib/test/test_robotparser.py @@ -249,6 +249,33 @@ Disallow: /cyberworld/map/ bad = ['/cyberworld/map/index.html'] +class StringFormattingTest(BaseRobotTest, unittest.TestCase): + robots_txt = """\ +User-agent: * +Crawl-delay: 1 +Request-rate: 3/15 +Disallow: /cyberworld/map/ # This is an infinite virtual URL space + +# Cybermapper knows where to go. +User-agent: cybermapper +Disallow: /some/path + """ + + expected_output = """\ +User-agent: cybermapper +Disallow: /some/path + +User-agent: * +Crawl-delay: 1 +Request-rate: 3/15 +Disallow: /cyberworld/map/ + +""" + + def test_string_formatting(self): + self.assertEqual(str(self.parser), self.expected_output) + + class RobotHandler(BaseHTTPRequestHandler): def do_GET(self): diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py index 852b2fe..23dfd06 100644 --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -372,17 +372,19 @@ class BaseSelectorTestCase(unittest.TestCase): orig_alrm_handler = signal.signal(signal.SIGALRM, handler) self.addCleanup(signal.signal, signal.SIGALRM, orig_alrm_handler) - self.addCleanup(signal.alarm, 0) - signal.alarm(1) + try: + signal.alarm(1) - s.register(rd, selectors.EVENT_READ) - t = time() - # select() is interrupted by a signal which raises an exception - with self.assertRaises(InterruptSelect): - s.select(30) - # select() was interrupted before the timeout of 30 seconds - self.assertLess(time() - t, 5.0) + s.register(rd, selectors.EVENT_READ) + t = time() + # select() is interrupted by a signal which raises an exception + with self.assertRaises(InterruptSelect): + s.select(30) + # select() was interrupted before the timeout of 30 seconds + self.assertLess(time() - t, 5.0) + finally: + signal.alarm(0) @unittest.skipUnless(hasattr(signal, "alarm"), "signal.alarm() required for this test") @@ -394,17 +396,19 @@ class BaseSelectorTestCase(unittest.TestCase): orig_alrm_handler = signal.signal(signal.SIGALRM, lambda *args: None) self.addCleanup(signal.signal, signal.SIGALRM, orig_alrm_handler) - self.addCleanup(signal.alarm, 0) - signal.alarm(1) + try: + signal.alarm(1) - s.register(rd, selectors.EVENT_READ) - t = time() - # select() is interrupted by a signal, but the signal handler doesn't - # raise an exception, so select() should by retries with a recomputed - # timeout - self.assertFalse(s.select(1.5)) - self.assertGreaterEqual(time() - t, 1.0) + s.register(rd, selectors.EVENT_READ) + t = time() + # select() is interrupted by a signal, but the signal handler doesn't + # raise an exception, so select() should by retries with a recomputed + # timeout + self.assertFalse(s.select(1.5)) + self.assertGreaterEqual(time() - t, 1.0) + finally: + signal.alarm(0) class ScalableSelectorMixIn: @@ -450,7 +454,14 @@ class ScalableSelectorMixIn: self.skipTest("FD limit reached") raise - self.assertEqual(NUM_FDS // 2, len(s.select())) + try: + fds = s.select() + except OSError as e: + if e.errno == errno.EINVAL and sys.platform == 'darwin': + # unexplainable errors on macOS don't need to fail the test + self.skipTest("Invalid argument error calling poll()") + raise + self.assertEqual(NUM_FDS // 2, len(fds)) class DefaultSelectorTestCase(BaseSelectorTestCase): diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py index faa4868..e41ddfd 100644 --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -245,6 +245,9 @@ class ThreadableTest: self.server_ready.set() def _setUp(self): + self.wait_threads = support.wait_threads_exit() + self.wait_threads.__enter__() + self.server_ready = threading.Event() self.client_ready = threading.Event() self.done = threading.Event() @@ -271,6 +274,7 @@ class ThreadableTest: def _tearDown(self): self.__tearDown() self.done.wait() + self.wait_threads.__exit__(None, None, None) if self.queue.qsize(): exc = self.queue.get() @@ -3697,7 +3701,6 @@ class InterruptedTimeoutBase(unittest.TestCase): orig_alrm_handler = signal.signal(signal.SIGALRM, lambda signum, frame: 1 / 0) self.addCleanup(signal.signal, signal.SIGALRM, orig_alrm_handler) - self.addCleanup(self.setAlarm, 0) # Timeout for socket operations timeout = 4.0 @@ -3734,9 +3737,12 @@ class InterruptedRecvTimeoutTest(InterruptedTimeoutBase, UDPTestBase): def checkInterruptedRecv(self, func, *args, **kwargs): # Check that func(*args, **kwargs) raises # errno of EINTR when interrupted by a signal. - self.setAlarm(self.alarm_time) - with self.assertRaises(ZeroDivisionError) as cm: - func(*args, **kwargs) + try: + self.setAlarm(self.alarm_time) + with self.assertRaises(ZeroDivisionError) as cm: + func(*args, **kwargs) + finally: + self.setAlarm(0) def testInterruptedRecvTimeout(self): self.checkInterruptedRecv(self.serv.recv, 1024) @@ -3792,10 +3798,13 @@ class InterruptedSendTimeoutTest(InterruptedTimeoutBase, # Check that func(*args, **kwargs), run in a loop, raises # OSError with an errno of EINTR when interrupted by a # signal. - with self.assertRaises(ZeroDivisionError) as cm: - while True: - self.setAlarm(self.alarm_time) - func(*args, **kwargs) + try: + with self.assertRaises(ZeroDivisionError) as cm: + while True: + self.setAlarm(self.alarm_time) + func(*args, **kwargs) + finally: + self.setAlarm(0) # Issue #12958: The following tests have problems on OS X prior to 10.7 @support.requires_mac_ver(10, 7) @@ -4525,8 +4534,8 @@ class TCPTimeoutTest(SocketTCPTest): raise Alarm old_alarm = signal.signal(signal.SIGALRM, alarm_handler) try: - signal.alarm(2) # POSIX allows alarm to be up to 1 second early try: + signal.alarm(2) # POSIX allows alarm to be up to 1 second early foo = self.serv.accept() except socket.timeout: self.fail("caught timeout instead of Alarm") @@ -5586,7 +5595,7 @@ class LinuxKernelCryptoAPI(unittest.TestCase): @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") class TestMSWindowsTCPFlags(unittest.TestCase): knownTCPFlags = { - # avaliable since long time ago + # available since long time ago 'TCP_MAXSEG', 'TCP_NODELAY', # available starting with Windows 10 1607 diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py index 4362133..8177c41 100644 --- a/Lib/test/test_socketserver.py +++ b/Lib/test/test_socketserver.py @@ -48,11 +48,11 @@ def receive(sock, n, timeout=20): if HAVE_UNIX_SOCKETS and HAVE_FORKING: class ForkingUnixStreamServer(socketserver.ForkingMixIn, socketserver.UnixStreamServer): - pass + _block_on_close = True class ForkingUnixDatagramServer(socketserver.ForkingMixIn, socketserver.UnixDatagramServer): - pass + _block_on_close = True @contextlib.contextmanager @@ -62,24 +62,14 @@ def simple_subprocess(testcase): if pid == 0: # Don't raise an exception; it would be caught by the test harness. os._exit(72) - yield None - pid2, status = os.waitpid(pid, 0) - testcase.assertEqual(pid2, pid) - testcase.assertEqual(72 << 8, status) - - -def close_server(server): - server.server_close() - - if hasattr(server, 'active_children'): - # ForkingMixIn: Manually reap all child processes, since server_close() - # calls waitpid() in non-blocking mode using the WNOHANG flag. - for pid in server.active_children.copy(): - try: - os.waitpid(pid, 0) - except ChildProcessError: - pass - server.active_children.clear() + try: + yield None + except: + raise + finally: + pid2, status = os.waitpid(pid, 0) + testcase.assertEqual(pid2, pid) + testcase.assertEqual(72 << 8, status) @unittest.skipUnless(threading, 'Threading required for this test.') @@ -115,6 +105,8 @@ class SocketServerTest(unittest.TestCase): def make_server(self, addr, svrcls, hdlrbase): class MyServer(svrcls): + _block_on_close = True + def handle_error(self, request, client_address): self.close_request(request) raise @@ -156,8 +148,12 @@ class SocketServerTest(unittest.TestCase): if verbose: print("waiting for server") server.shutdown() t.join() - close_server(server) + server.server_close() self.assertEqual(-1, server.socket.fileno()) + if HAVE_FORKING and isinstance(server, socketserver.ForkingMixIn): + # bpo-31151: Check that ForkingMixIn.server_close() waits until + # all children completed + self.assertFalse(server.active_children) if verbose: print("done") def stream_examine(self, proto, addr): @@ -280,7 +276,7 @@ class SocketServerTest(unittest.TestCase): s.shutdown() for t, s in threads: t.join() - close_server(s) + s.server_close() def test_tcpserver_bind_leak(self): # Issue #22435: the server socket wouldn't be closed if bind()/listen() @@ -344,6 +340,8 @@ class ErrorHandlerTest(unittest.TestCase): class BaseErrorTestServer(socketserver.TCPServer): + _block_on_close = True + def __init__(self, exception): self.exception = exception super().__init__((HOST, 0), BadHandler) @@ -352,7 +350,7 @@ class BaseErrorTestServer(socketserver.TCPServer): try: self.handle_request() finally: - close_server(self) + self.server_close() self.wait_done() def handle_error(self, request, client_address): @@ -386,7 +384,7 @@ class ThreadingErrorTestServer(socketserver.ThreadingMixIn, if HAVE_FORKING: class ForkingErrorTestServer(socketserver.ForkingMixIn, BaseErrorTestServer): - pass + _block_on_close = True class SocketWriterTest(unittest.TestCase): @@ -398,7 +396,7 @@ class SocketWriterTest(unittest.TestCase): self.server.request_fileno = self.request.fileno() server = socketserver.TCPServer((HOST, 0), Handler) - self.addCleanup(close_server, server) + self.addCleanup(server.server_close) s = socket.socket( server.address_family, socket.SOCK_STREAM, socket.IPPROTO_TCP) with s: @@ -422,7 +420,7 @@ class SocketWriterTest(unittest.TestCase): self.server.sent2 = self.wfile.write(big_chunk) server = socketserver.TCPServer((HOST, 0), Handler) - self.addCleanup(close_server, server) + self.addCleanup(server.server_close) interrupted = threading.Event() def signal_handler(signum, frame): @@ -498,7 +496,7 @@ class MiscTestCase(unittest.TestCase): s.close() server.handle_request() self.assertEqual(server.shutdown_called, 1) - close_server(server) + server.server_close() if __name__ == "__main__": diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 8dd3b41..9785a59 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1687,6 +1687,7 @@ class SimpleBackgroundTests(unittest.TestCase): self.assertEqual(len(ctx.get_ca_certs()), 1) @needs_sni + @unittest.skipUnless(hasattr(ssl, "PROTOCOL_TLSv1_2"), "needs TLS 1.2") def test_context_setget(self): # Check that the context of a connected socket can be replaced. ctx1 = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index aca1411..9ebe322 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -6,6 +6,7 @@ import sys import platform import signal import io +import itertools import os import errno import tempfile @@ -2108,6 +2109,55 @@ class POSIXProcessTestCase(BaseTestCase): self.check_swap_fds(2, 0, 1) self.check_swap_fds(2, 1, 0) + def _check_swap_std_fds_with_one_closed(self, from_fds, to_fds): + saved_fds = self._save_fds(range(3)) + try: + for from_fd in from_fds: + with tempfile.TemporaryFile() as f: + os.dup2(f.fileno(), from_fd) + + fd_to_close = (set(range(3)) - set(from_fds)).pop() + os.close(fd_to_close) + + arg_names = ['stdin', 'stdout', 'stderr'] + kwargs = {} + for from_fd, to_fd in zip(from_fds, to_fds): + kwargs[arg_names[to_fd]] = from_fd + + code = textwrap.dedent(r''' + import os, sys + skipped_fd = int(sys.argv[1]) + for fd in range(3): + if fd != skipped_fd: + os.write(fd, str(fd).encode('ascii')) + ''') + + skipped_fd = (set(range(3)) - set(to_fds)).pop() + + rc = subprocess.call([sys.executable, '-c', code, str(skipped_fd)], + **kwargs) + self.assertEqual(rc, 0) + + for from_fd, to_fd in zip(from_fds, to_fds): + os.lseek(from_fd, 0, os.SEEK_SET) + read_bytes = os.read(from_fd, 1024) + read_fds = list(map(int, read_bytes.decode('ascii'))) + msg = textwrap.dedent(f""" + When testing {from_fds} to {to_fds} redirection, + parent descriptor {from_fd} got redirected + to descriptor(s) {read_fds} instead of descriptor {to_fd}. + """) + self.assertEqual([to_fd], read_fds, msg) + finally: + self._restore_fds(saved_fds) + + # Check that subprocess can remap std fds correctly even + # if one of them is closed (#32844). + def test_swap_std_fds_with_one_closed(self): + for from_fds in itertools.combinations(range(3), 2): + for to_fds in itertools.permutations(range(3), 2): + self._check_swap_std_fds_with_one_closed(from_fds, to_fds) + def test_surrogates_error_message(self): def prepare(): raise ValueError("surrogate:\uDCff") diff --git a/Lib/test/test_sundry.py b/Lib/test/test_sundry.py index 1fb9964..fc5ac0a 100644 --- a/Lib/test/test_sundry.py +++ b/Lib/test/test_sundry.py @@ -6,8 +6,7 @@ import unittest class TestUntestedModules(unittest.TestCase): def test_untested_modules_can_be_imported(self): - untested = ('bdb', 'encodings', 'formatter', - 'nturl2path', 'tabnanny') + untested = ('encodings', 'formatter', 'nturl2path', 'tabnanny') with support.check_warnings(quiet=True): for name in untested: try: diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py index fff0fc7..6e0c122 100644 --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -451,6 +451,17 @@ class TestSupport(unittest.TestCase): self.assertTrue(support.match_test(test_access)) self.assertFalse(support.match_test(test_chdir)) + def test_fd_count(self): + # We cannot test the absolute value of fd_count(): on old Linux + # kernel or glibc versions, os.urandom() keeps a FD open on + # /dev/urandom device and Python has 4 FD opens instead of 3. + start = support.fd_count() + fd = os.open(__file__, os.O_RDONLY) + try: + more = support.fd_count() + finally: + os.close(fd) + self.assertEqual(more - start, 1) # XXX -follows a list of untested API # make_legacy_pyc diff --git a/Lib/test/test_sys_settrace.py b/Lib/test/test_sys_settrace.py index 46593cf..efee5d6 100644 --- a/Lib/test/test_sys_settrace.py +++ b/Lib/test/test_sys_settrace.py @@ -7,8 +7,9 @@ import difflib import gc from functools import wraps + class tracecontext: - """Contex manager that traces its enter and exit.""" + """Context manager that traces its enter and exit.""" def __init__(self, output, value): self.output = output self.value = value @@ -19,6 +20,41 @@ class tracecontext: def __exit__(self, *exc_info): self.output.append(-self.value) +class asynctracecontext: + """Asynchronous context manager that traces its aenter and aexit.""" + def __init__(self, output, value): + self.output = output + self.value = value + + async def __aenter__(self): + self.output.append(self.value) + + async def __aexit__(self, *exc_info): + self.output.append(-self.value) + +async def asynciter(iterable): + """Convert an iterable to an asynchronous iterator.""" + for x in iterable: + yield x + +def asyncio_run(main): + import asyncio + import asyncio.events + import asyncio.coroutines + assert asyncio.events._get_running_loop() is None + assert asyncio.coroutines.iscoroutine(main) + loop = asyncio.events.new_event_loop() + try: + asyncio.events.set_event_loop(loop) + return loop.run_until_complete(main) + finally: + try: + loop.run_until_complete(loop.shutdown_asyncgens()) + finally: + asyncio.events.set_event_loop(None) + loop.close() + + # A very basic example. If this fails, we're in deep trouble. def basic(): return 1 @@ -591,6 +627,19 @@ class JumpTestCase(unittest.TestCase): sys.settrace(None) self.compare_jump_output(expected, output) + def run_async_test(self, func, jumpFrom, jumpTo, expected, error=None, + event='line', decorated=False): + tracer = JumpTracer(func, jumpFrom, jumpTo, event, decorated) + sys.settrace(tracer.trace) + output = [] + if error is None: + asyncio_run(func(output)) + else: + with self.assertRaisesRegex(*error): + asyncio_run(func(output)) + sys.settrace(None) + self.compare_jump_output(expected, output) + def jump_test(jumpFrom, jumpTo, expected, error=None, event='line'): """Decorator that creates a test that makes a jump from one place to another in the following code. @@ -603,6 +652,18 @@ class JumpTestCase(unittest.TestCase): return test return decorator + def async_jump_test(jumpFrom, jumpTo, expected, error=None, event='line'): + """Decorator that creates a test that makes a jump + from one place to another in the following asynchronous code. + """ + def decorator(func): + @wraps(func) + def test(self): + self.run_async_test(func, jumpFrom, jumpTo, expected, + error=error, event=event, decorated=True) + return test + return decorator + ## The first set of 'jump' tests are for things that are allowed: @jump_test(1, 3, [3]) @@ -634,6 +695,23 @@ class JumpTestCase(unittest.TestCase): output.append(6) output.append(7) + @async_jump_test(4, 5, [3, 5]) + async def test_jump_out_of_async_for_block_forwards(output): + for i in [1]: + async for i in asynciter([1, 2]): + output.append(3) + output.append(4) + output.append(5) + + @async_jump_test(5, 2, [2, 4, 2, 4, 5, 6]) + async def test_jump_out_of_async_for_block_backwards(output): + for i in [1]: + output.append(2) + async for i in asynciter([1]): + output.append(4) + output.append(5) + output.append(6) + @jump_test(1, 2, [3]) def test_jump_to_codeless_line(output): output.append(1) @@ -698,12 +776,24 @@ class JumpTestCase(unittest.TestCase): output.append(2) output.append(3) + @async_jump_test(2, 3, [1, 3]) + async def test_jump_forwards_out_of_async_with_block(output): + async with asynctracecontext(output, 1): + output.append(2) + output.append(3) + @jump_test(3, 1, [1, 2, 1, 2, 3, -2]) def test_jump_backwards_out_of_with_block(output): output.append(1) with tracecontext(output, 2): output.append(3) + @async_jump_test(3, 1, [1, 2, 1, 2, 3, -2]) + async def test_jump_backwards_out_of_async_with_block(output): + output.append(1) + async with asynctracecontext(output, 2): + output.append(3) + @jump_test(2, 5, [5]) def test_jump_forwards_out_of_try_finally_block(output): try: @@ -767,6 +857,14 @@ class JumpTestCase(unittest.TestCase): with tracecontext(output, 4): output.append(5) + @async_jump_test(2, 4, [1, 4, 5, -4]) + async def test_jump_across_async_with(output): + output.append(1) + async with asynctracecontext(output, 2): + output.append(3) + async with asynctracecontext(output, 4): + output.append(5) + @jump_test(4, 5, [1, 3, 5, 6]) def test_jump_out_of_with_block_within_for_block(output): output.append(1) @@ -776,6 +874,15 @@ class JumpTestCase(unittest.TestCase): output.append(5) output.append(6) + @async_jump_test(4, 5, [1, 3, 5, 6]) + async def test_jump_out_of_async_with_block_within_for_block(output): + output.append(1) + for i in [1]: + async with asynctracecontext(output, 3): + output.append(4) + output.append(5) + output.append(6) + @jump_test(4, 5, [1, 2, 3, 5, -2, 6]) def test_jump_out_of_with_block_within_with_block(output): output.append(1) @@ -785,6 +892,15 @@ class JumpTestCase(unittest.TestCase): output.append(5) output.append(6) + @async_jump_test(4, 5, [1, 2, 3, 5, -2, 6]) + async def test_jump_out_of_async_with_block_within_with_block(output): + output.append(1) + with tracecontext(output, 2): + async with asynctracecontext(output, 3): + output.append(4) + output.append(5) + output.append(6) + @jump_test(5, 6, [2, 4, 6, 7]) def test_jump_out_of_with_block_within_finally_block(output): try: @@ -795,6 +911,16 @@ class JumpTestCase(unittest.TestCase): output.append(6) output.append(7) + @async_jump_test(5, 6, [2, 4, 6, 7]) + async def test_jump_out_of_async_with_block_within_finally_block(output): + try: + output.append(2) + finally: + async with asynctracecontext(output, 4): + output.append(5) + output.append(6) + output.append(7) + @jump_test(8, 11, [1, 3, 5, 11, 12]) def test_jump_out_of_complex_nested_blocks(output): output.append(1) @@ -818,6 +944,14 @@ class JumpTestCase(unittest.TestCase): output.append(4) output.append(5) + @async_jump_test(3, 5, [1, 2, 5]) + async def test_jump_out_of_async_with_assignment(output): + output.append(1) + async with asynctracecontext(output, 2) \ + as x: + output.append(4) + output.append(5) + @jump_test(3, 6, [1, 6, 8, 9]) def test_jump_over_return_in_try_finally_block(output): output.append(1) @@ -857,6 +991,17 @@ class JumpTestCase(unittest.TestCase): output.append(7) output.append(8) + @async_jump_test(1, 7, [7, 8]) + async def test_jump_over_async_for_block_before_else(output): + output.append(1) + if not output: # always false + async for i in asynciter([3]): + output.append(4) + else: + output.append(6) + output.append(7) + output.append(8) + # The second set of 'jump' tests are for things that are not allowed: @jump_test(2, 3, [1], (ValueError, 'after')) @@ -908,12 +1053,24 @@ class JumpTestCase(unittest.TestCase): for i in 1, 2: output.append(3) + @async_jump_test(1, 3, [], (ValueError, 'into')) + async def test_no_jump_forwards_into_async_for_block(output): + output.append(1) + async for i in asynciter([1, 2]): + output.append(3) + @jump_test(3, 2, [2, 2], (ValueError, 'into')) def test_no_jump_backwards_into_for_block(output): for i in 1, 2: output.append(2) output.append(3) + @async_jump_test(3, 2, [2, 2], (ValueError, 'into')) + async def test_no_jump_backwards_into_async_for_block(output): + async for i in asynciter([1, 2]): + output.append(2) + output.append(3) + @jump_test(2, 4, [], (ValueError, 'into')) def test_no_jump_forwards_into_while_block(output): i = 1 @@ -936,12 +1093,24 @@ class JumpTestCase(unittest.TestCase): with tracecontext(output, 2): output.append(3) + @async_jump_test(1, 3, [], (ValueError, 'into')) + async def test_no_jump_forwards_into_async_with_block(output): + output.append(1) + async with asynctracecontext(output, 2): + output.append(3) + @jump_test(3, 2, [1, 2, -1], (ValueError, 'into')) def test_no_jump_backwards_into_with_block(output): with tracecontext(output, 1): output.append(2) output.append(3) + @async_jump_test(3, 2, [1, 2, -1], (ValueError, 'into')) + async def test_no_jump_backwards_into_async_with_block(output): + async with asynctracecontext(output, 1): + output.append(2) + output.append(3) + @jump_test(1, 3, [], (ValueError, 'into')) def test_no_jump_forwards_into_try_finally_block(output): output.append(1) @@ -1022,6 +1191,14 @@ class JumpTestCase(unittest.TestCase): with tracecontext(output, 4): output.append(5) + @async_jump_test(3, 5, [1, 2, -2], (ValueError, 'into')) + async def test_no_jump_between_async_with_blocks(output): + output.append(1) + async with asynctracecontext(output, 2): + output.append(3) + async with asynctracecontext(output, 4): + output.append(5) + @jump_test(7, 4, [1, 6], (ValueError, 'into')) def test_no_jump_into_for_block_before_else(output): output.append(1) @@ -1033,6 +1210,17 @@ class JumpTestCase(unittest.TestCase): output.append(7) output.append(8) + @async_jump_test(7, 4, [1, 6], (ValueError, 'into')) + async def test_no_jump_into_async_for_block_before_else(output): + output.append(1) + if not output: # always false + async for i in asynciter([3]): + output.append(4) + else: + output.append(6) + output.append(7) + output.append(8) + def test_no_jump_to_non_integers(self): self.run_test(no_jump_to_non_integers, 2, "Spam", [True]) diff --git a/Lib/test/test_thread.py b/Lib/test/test_thread.py index 2dd1593..52f6c79 100644 --- a/Lib/test/test_thread.py +++ b/Lib/test/test_thread.py @@ -59,12 +59,13 @@ class ThreadRunningTests(BasicThreadTest): self.done_mutex.release() def test_starting_threads(self): - # Basic test for thread creation. - for i in range(NUMTASKS): - self.newtask() - verbose_print("waiting for tasks to complete...") - self.done_mutex.acquire() - verbose_print("all tasks done") + with support.wait_threads_exit(): + # Basic test for thread creation. + for i in range(NUMTASKS): + self.newtask() + verbose_print("waiting for tasks to complete...") + self.done_mutex.acquire() + verbose_print("all tasks done") def test_stack_size(self): # Various stack size tests. @@ -94,12 +95,13 @@ class ThreadRunningTests(BasicThreadTest): verbose_print("trying stack_size = (%d)" % tss) self.next_ident = 0 self.created = 0 - for i in range(NUMTASKS): - self.newtask() + with support.wait_threads_exit(): + for i in range(NUMTASKS): + self.newtask() - verbose_print("waiting for all tasks to complete") - self.done_mutex.acquire() - verbose_print("all tasks done") + verbose_print("waiting for all tasks to complete") + self.done_mutex.acquire() + verbose_print("all tasks done") thread.stack_size(0) @@ -109,25 +111,28 @@ class ThreadRunningTests(BasicThreadTest): mut = thread.allocate_lock() mut.acquire() started = [] + def task(): started.append(None) mut.acquire() mut.release() - thread.start_new_thread(task, ()) - while not started: - time.sleep(POLL_SLEEP) - self.assertEqual(thread._count(), orig + 1) - # Allow the task to finish. - mut.release() - # The only reliable way to be sure that the thread ended from the - # interpreter's point of view is to wait for the function object to be - # destroyed. - done = [] - wr = weakref.ref(task, lambda _: done.append(None)) - del task - while not done: - time.sleep(POLL_SLEEP) - self.assertEqual(thread._count(), orig) + + with support.wait_threads_exit(): + thread.start_new_thread(task, ()) + while not started: + time.sleep(POLL_SLEEP) + self.assertEqual(thread._count(), orig + 1) + # Allow the task to finish. + mut.release() + # The only reliable way to be sure that the thread ended from the + # interpreter's point of view is to wait for the function object to be + # destroyed. + done = [] + wr = weakref.ref(task, lambda _: done.append(None)) + del task + while not done: + time.sleep(POLL_SLEEP) + self.assertEqual(thread._count(), orig) def test_save_exception_state_on_error(self): # See issue #14474 @@ -140,16 +145,14 @@ class ThreadRunningTests(BasicThreadTest): except ValueError: pass real_write(self, *args) - c = thread._count() started = thread.allocate_lock() with support.captured_output("stderr") as stderr: real_write = stderr.write stderr.write = mywrite started.acquire() - thread.start_new_thread(task, ()) - started.acquire() - while thread._count() > c: - time.sleep(POLL_SLEEP) + with support.wait_threads_exit(): + thread.start_new_thread(task, ()) + started.acquire() self.assertIn("Traceback", stderr.getvalue()) @@ -181,13 +184,14 @@ class Barrier: class BarrierTest(BasicThreadTest): def test_barrier(self): - self.bar = Barrier(NUMTASKS) - self.running = NUMTASKS - for i in range(NUMTASKS): - thread.start_new_thread(self.task2, (i,)) - verbose_print("waiting for tasks to end") - self.done_mutex.acquire() - verbose_print("tasks done") + with support.wait_threads_exit(): + self.bar = Barrier(NUMTASKS) + self.running = NUMTASKS + for i in range(NUMTASKS): + thread.start_new_thread(self.task2, (i,)) + verbose_print("waiting for tasks to end") + self.done_mutex.acquire() + verbose_print("tasks done") def task2(self, ident): for i in range(NUMTRIPS): @@ -225,11 +229,10 @@ class TestForkInThread(unittest.TestCase): @unittest.skipUnless(hasattr(os, 'fork'), 'need os.fork') @support.reap_threads def test_forkinthread(self): - running = True status = "not set" def thread1(): - nonlocal running, status + nonlocal status # fork in a thread pid = os.fork() @@ -244,13 +247,11 @@ class TestForkInThread(unittest.TestCase): # parent os.close(self.write_fd) pid, status = os.waitpid(pid, 0) - running = False - thread.start_new_thread(thread1, ()) - self.assertEqual(os.read(self.read_fd, 2), b"OK", - "Unable to fork() in thread") - while running: - time.sleep(POLL_SLEEP) + with support.wait_threads_exit(): + thread.start_new_thread(thread1, ()) + self.assertEqual(os.read(self.read_fd, 2), b"OK", + "Unable to fork() in thread") self.assertEqual(status, 0) def tearDown(self): diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py index b42314f..43ef22b 100644 --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -125,9 +125,10 @@ class ThreadTests(BaseTestCase): done.set() done = threading.Event() ident = [] - _thread.start_new_thread(f, ()) - done.wait() - self.assertIsNotNone(ident[0]) + with support.wait_threads_exit(): + tid = _thread.start_new_thread(f, ()) + done.wait() + self.assertEqual(ident[0], tid) # Kill the "immortal" _DummyThread del threading._active[ident[0]] @@ -165,9 +166,10 @@ class ThreadTests(BaseTestCase): mutex = threading.Lock() mutex.acquire() - tid = _thread.start_new_thread(f, (mutex,)) - # Wait for the thread to finish. - mutex.acquire() + with support.wait_threads_exit(): + tid = _thread.start_new_thread(f, (mutex,)) + # Wait for the thread to finish. + mutex.acquire() self.assertIn(tid, threading._active) self.assertIsInstance(threading._active[tid], threading._DummyThread) #Issue 29376 diff --git a/Lib/test/test_threadsignals.py b/Lib/test/test_threadsignals.py index 9d92742..99b60cd 100644 --- a/Lib/test/test_threadsignals.py +++ b/Lib/test/test_threadsignals.py @@ -4,8 +4,8 @@ import unittest import signal import os import sys -from test.support import run_unittest, import_module -thread = import_module('_thread') +from test import support +thread = support.import_module('_thread') import time if (sys.platform[:3] == 'win'): @@ -39,13 +39,15 @@ def send_signals(): class ThreadSignals(unittest.TestCase): def test_signals(self): - # Test signal handling semantics of threads. - # We spawn a thread, have the thread send two signals, and - # wait for it to finish. Check that we got both signals - # and that they were run by the main thread. - signalled_all.acquire() - self.spawnSignallingThread() - signalled_all.acquire() + with support.wait_threads_exit(): + # Test signal handling semantics of threads. + # We spawn a thread, have the thread send two signals, and + # wait for it to finish. Check that we got both signals + # and that they were run by the main thread. + signalled_all.acquire() + self.spawnSignallingThread() + signalled_all.acquire() + # the signals that we asked the kernel to send # will come back, but we don't know when. # (it might even be after the thread exits @@ -54,9 +56,11 @@ class ThreadSignals(unittest.TestCase): # wait for it return. if signal_blackboard[signal.SIGUSR1]['tripped'] == 0 \ or signal_blackboard[signal.SIGUSR2]['tripped'] == 0: - signal.alarm(1) - signal.pause() - signal.alarm(0) + try: + signal.alarm(1) + signal.pause() + finally: + signal.alarm(0) self.assertEqual( signal_blackboard[signal.SIGUSR1]['tripped'], 1) self.assertEqual( signal_blackboard[signal.SIGUSR1]['tripped_by'], @@ -96,6 +100,7 @@ class ThreadSignals(unittest.TestCase): # after timeout return of lock.acquire() (which can fool assertRaises). self.assertLess(dt, 3.0) finally: + signal.alarm(0) signal.signal(signal.SIGALRM, oldalrm) @unittest.skipIf(USING_PTHREAD_COND, @@ -115,24 +120,28 @@ class ThreadSignals(unittest.TestCase): # thread. def other_thread(): rlock.acquire() - thread.start_new_thread(other_thread, ()) - # Wait until we can't acquire it without blocking... - while rlock.acquire(blocking=False): - rlock.release() - time.sleep(0.01) - signal.alarm(1) - t1 = time.time() - self.assertRaises(KeyboardInterrupt, rlock.acquire, timeout=5) - dt = time.time() - t1 - # See rationale above in test_lock_acquire_interruption - self.assertLess(dt, 3.0) + + with support.wait_threads_exit(): + thread.start_new_thread(other_thread, ()) + # Wait until we can't acquire it without blocking... + while rlock.acquire(blocking=False): + rlock.release() + time.sleep(0.01) + signal.alarm(1) + t1 = time.time() + self.assertRaises(KeyboardInterrupt, rlock.acquire, timeout=5) + dt = time.time() - t1 + # See rationale above in test_lock_acquire_interruption + self.assertLess(dt, 3.0) finally: + signal.alarm(0) signal.signal(signal.SIGALRM, oldalrm) def acquire_retries_on_intr(self, lock): self.sig_recvd = False def my_handler(signal, frame): self.sig_recvd = True + old_handler = signal.signal(signal.SIGUSR1, my_handler) try: def other_thread(): @@ -147,14 +156,16 @@ class ThreadSignals(unittest.TestCase): # the lock acquisition. Then we'll let it run. time.sleep(0.5) lock.release() - thread.start_new_thread(other_thread, ()) - # Wait until we can't acquire it without blocking... - while lock.acquire(blocking=False): - lock.release() - time.sleep(0.01) - result = lock.acquire() # Block while we receive a signal. - self.assertTrue(self.sig_recvd) - self.assertTrue(result) + + with support.wait_threads_exit(): + thread.start_new_thread(other_thread, ()) + # Wait until we can't acquire it without blocking... + while lock.acquire(blocking=False): + lock.release() + time.sleep(0.01) + result = lock.acquire() # Block while we receive a signal. + self.assertTrue(self.sig_recvd) + self.assertTrue(result) finally: signal.signal(signal.SIGUSR1, old_handler) @@ -193,19 +204,20 @@ class ThreadSignals(unittest.TestCase): os.kill(process_pid, signal.SIGUSR1) done.release() - # Send the signals from the non-main thread, since the main thread - # is the only one that can process signals. - thread.start_new_thread(send_signals, ()) - timed_acquire() - # Wait for thread to finish - done.acquire() - # This allows for some timing and scheduling imprecision - self.assertLess(self.end - self.start, 2.0) - self.assertGreater(self.end - self.start, 0.3) - # If the signal is received several times before PyErr_CheckSignals() - # is called, the handler will get called less than 40 times. Just - # check it's been called at least once. - self.assertGreater(self.sigs_recvd, 0) + with support.wait_threads_exit(): + # Send the signals from the non-main thread, since the main thread + # is the only one that can process signals. + thread.start_new_thread(send_signals, ()) + timed_acquire() + # Wait for thread to finish + done.acquire() + # This allows for some timing and scheduling imprecision + self.assertLess(self.end - self.start, 2.0) + self.assertGreater(self.end - self.start, 0.3) + # If the signal is received several times before PyErr_CheckSignals() + # is called, the handler will get called less than 40 times. Just + # check it's been called at least once. + self.assertGreater(self.sigs_recvd, 0) finally: signal.signal(signal.SIGUSR1, old_handler) @@ -219,7 +231,7 @@ def test_main(): oldsigs = registerSignals(handle_signals, handle_signals, handle_signals) try: - run_unittest(ThreadSignals) + support.run_unittest(ThreadSignals) finally: registerSignals(*oldsigs) diff --git a/Lib/test/test_tools/test_i18n.py b/Lib/test/test_tools/test_i18n.py index 5e7c4bf..64b5d09 100644 --- a/Lib/test/test_tools/test_i18n.py +++ b/Lib/test/test_tools/test_i18n.py @@ -2,11 +2,11 @@ import os import unittest -import textwrap +from textwrap import dedent from test.support.script_helper import assert_python_ok from test.test_tools import skip_if_missing, toolsdir -from test.support import temp_cwd +from test.support import temp_cwd, temp_dir skip_if_missing() @@ -107,9 +107,68 @@ class Test_pygettext(unittest.TestCase): # This will raise if the date format does not exactly match. datetime.strptime(creationDate, '%Y-%m-%d %H:%M%z') + def test_funcdocstring(self): + for doc in ('"""doc"""', "r'''doc'''", "R'doc'", 'u"doc"'): + with self.subTest(doc): + msgids = self.extract_docstrings_from_str(dedent('''\ + def foo(bar): + %s + ''' % doc)) + self.assertIn('doc', msgids) + + def test_funcdocstring_bytes(self): + msgids = self.extract_docstrings_from_str(dedent('''\ + def foo(bar): + b"""doc""" + ''')) + self.assertFalse([msgid for msgid in msgids if 'doc' in msgid]) + + def test_funcdocstring_fstring(self): + msgids = self.extract_docstrings_from_str(dedent('''\ + def foo(bar): + f"""doc""" + ''')) + self.assertFalse([msgid for msgid in msgids if 'doc' in msgid]) + + def test_classdocstring(self): + for doc in ('"""doc"""', "r'''doc'''", "R'doc'", 'u"doc"'): + with self.subTest(doc): + msgids = self.extract_docstrings_from_str(dedent('''\ + class C: + %s + ''' % doc)) + self.assertIn('doc', msgids) + + def test_classdocstring_bytes(self): + msgids = self.extract_docstrings_from_str(dedent('''\ + class C: + b"""doc""" + ''')) + self.assertFalse([msgid for msgid in msgids if 'doc' in msgid]) + + def test_classdocstring_fstring(self): + msgids = self.extract_docstrings_from_str(dedent('''\ + class C: + f"""doc""" + ''')) + self.assertFalse([msgid for msgid in msgids if 'doc' in msgid]) + + def test_msgid(self): + msgids = self.extract_docstrings_from_str( + '''_("""doc""" r'str' u"ing")''') + self.assertIn('docstring', msgids) + + def test_msgid_bytes(self): + msgids = self.extract_docstrings_from_str('_(b"""doc""")') + self.assertFalse([msgid for msgid in msgids if 'doc' in msgid]) + + def test_msgid_fstring(self): + msgids = self.extract_docstrings_from_str('_(f"""doc""")') + self.assertFalse([msgid for msgid in msgids if 'doc' in msgid]) + def test_funcdocstring_annotated_args(self): """ Test docstrings for functions with annotated args """ - msgids = self.extract_docstrings_from_str(textwrap.dedent('''\ + msgids = self.extract_docstrings_from_str(dedent('''\ def foo(bar: str): """doc""" ''')) @@ -117,7 +176,7 @@ class Test_pygettext(unittest.TestCase): def test_funcdocstring_annotated_return(self): """ Test docstrings for functions with annotated return type """ - msgids = self.extract_docstrings_from_str(textwrap.dedent('''\ + msgids = self.extract_docstrings_from_str(dedent('''\ def foo(bar) -> str: """doc""" ''')) @@ -125,7 +184,7 @@ class Test_pygettext(unittest.TestCase): def test_funcdocstring_defvalue_args(self): """ Test docstring for functions with default arg values """ - msgids = self.extract_docstrings_from_str(textwrap.dedent('''\ + msgids = self.extract_docstrings_from_str(dedent('''\ def foo(bar=()): """doc""" ''')) @@ -135,7 +194,7 @@ class Test_pygettext(unittest.TestCase): """ Test docstring extraction for multiple functions combining annotated args, annotated return types and default arg values """ - msgids = self.extract_docstrings_from_str(textwrap.dedent('''\ + msgids = self.extract_docstrings_from_str(dedent('''\ def foo1(bar: tuple=()) -> str: """doc1""" @@ -153,8 +212,32 @@ class Test_pygettext(unittest.TestCase): """ Test docstring extraction for a class with colons occuring within the parentheses. """ - msgids = self.extract_docstrings_from_str(textwrap.dedent('''\ + msgids = self.extract_docstrings_from_str(dedent('''\ class D(L[1:2], F({1: 2}), metaclass=M(lambda x: x)): """doc""" ''')) self.assertIn('doc', msgids) + + def test_files_list(self): + """Make sure the directories are inspected for source files + bpo-31920 + """ + text1 = 'Text to translate1' + text2 = 'Text to translate2' + text3 = 'Text to ignore' + with temp_cwd(None), temp_dir(None) as sdir: + os.mkdir(os.path.join(sdir, 'pypkg')) + with open(os.path.join(sdir, 'pypkg', 'pymod.py'), 'w') as sfile: + sfile.write(f'_({text1!r})') + os.mkdir(os.path.join(sdir, 'pkg.py')) + with open(os.path.join(sdir, 'pkg.py', 'pymod2.py'), 'w') as sfile: + sfile.write(f'_({text2!r})') + os.mkdir(os.path.join(sdir, 'CVS')) + with open(os.path.join(sdir, 'CVS', 'pymod3.py'), 'w') as sfile: + sfile.write(f'_({text3!r})') + assert_python_ok(self.script, sdir) + with open('messages.pot') as fp: + data = fp.read() + self.assertIn(f'msgid "{text1}"', data) + self.assertIn(f'msgid "{text2}"', data) + self.assertNotIn(text3, data) diff --git a/Lib/test/test_trace.py b/Lib/test/test_trace.py index e04ca01..55a8bce 100644 --- a/Lib/test/test_trace.py +++ b/Lib/test/test_trace.py @@ -2,6 +2,7 @@ import os import sys from test.support import TESTFN, rmtree, unlink, captured_stdout from test.support.script_helper import assert_python_ok, assert_python_failure +import textwrap import unittest import trace @@ -365,6 +366,46 @@ class Test_Ignore(unittest.TestCase): # Matched before. self.assertTrue(ignore.names(jn('bar', 'baz.py'), 'baz')) +# Created for Issue 31908 -- CLI utility not writing cover files +class TestCoverageCommandLineOutput(unittest.TestCase): + + codefile = 'tmp.py' + coverfile = 'tmp.cover' + + def setUp(self): + with open(self.codefile, 'w') as f: + f.write(textwrap.dedent('''\ + x = 42 + if []: + print('unreachable') + ''')) + + def tearDown(self): + unlink(self.codefile) + unlink(self.coverfile) + + def test_cover_files_written_no_highlight(self): + argv = '-m trace --count'.split() + [self.codefile] + status, stdout, stderr = assert_python_ok(*argv) + self.assertTrue(os.path.exists(self.coverfile)) + with open(self.coverfile) as f: + self.assertEqual(f.read(), + " 1: x = 42\n" + " 1: if []:\n" + " print('unreachable')\n" + ) + + def test_cover_files_written_with_highlight(self): + argv = '-m trace --count --missing'.split() + [self.codefile] + status, stdout, stderr = assert_python_ok(*argv) + self.assertTrue(os.path.exists(self.coverfile)) + with open(self.coverfile) as f: + self.assertEqual(f.read(), textwrap.dedent('''\ + 1: x = 42 + 1: if []: + >>>>>> print('unreachable') + ''')) + class TestCommandLine(unittest.TestCase): def test_failures(self): diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index fe5247c..4843d6f 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -1310,6 +1310,74 @@ class GenericTests(BaseTestCase): with self.assertRaises(Exception): D[T] + def test_new_with_args(self): + + class A(Generic[T]): + pass + + class B: + def __new__(cls, arg): + # call object + obj = super().__new__(cls) + obj.arg = arg + return obj + + # mro: C, A, Generic, B, object + class C(A, B): + pass + + c = C('foo') + self.assertEqual(c.arg, 'foo') + + def test_new_with_args2(self): + + class A: + def __init__(self, arg): + self.from_a = arg + # call object + super().__init__() + + # mro: C, Generic, A, object + class C(Generic[T], A): + def __init__(self, arg): + self.from_c = arg + # call Generic + super().__init__(arg) + + c = C('foo') + self.assertEqual(c.from_a, 'foo') + self.assertEqual(c.from_c, 'foo') + + def test_new_no_args(self): + + class A(Generic[T]): + pass + + with self.assertRaises(TypeError): + A('foo') + + class B: + def __new__(cls): + # call object + obj = super().__new__(cls) + obj.from_b = 'b' + return obj + + # mro: C, A, Generic, B, object + class C(A, B): + def __init__(self, arg): + self.arg = arg + + def __new__(cls, arg): + # call A + obj = super().__new__(cls) + obj.from_c = 'c' + return obj + + c = C('foo') + self.assertEqual(c.arg, 'foo') + self.assertEqual(c.from_b, 'b') + self.assertEqual(c.from_c, 'c') class ClassVarTests(BaseTestCase): @@ -1739,6 +1807,8 @@ class GetTypeHintTests(BaseTestCase): self.assertEqual(gth(HasForeignBaseClass), {'some_xrepr': XRepr, 'other_a': mod_generics_cache.A, 'some_b': mod_generics_cache.B}) + self.assertEqual(gth(XRepr.__new__), + {'x': int, 'y': int}) self.assertEqual(gth(mod_generics_cache.B), {'my_inner_a1': mod_generics_cache.B.A, 'my_inner_a2': mod_generics_cache.B.A, diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py index 13e2dd5..15f73de 100644 --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -177,6 +177,7 @@ class OtherNetworkTests(unittest.TestCase): opener.open(request) self.assertEqual(request.get_header('User-agent'),'Test-Agent') + @unittest.skip('XXX: http://www.imdb.com is gone') def test_sites_no_connection_close(self): # Some sites do not send Connection: close header. # Verify that those work properly. (#issue12576) diff --git a/Lib/test/test_uu.py b/Lib/test/test_uu.py index ad2f2c5..7356478 100644 --- a/Lib/test/test_uu.py +++ b/Lib/test/test_uu.py @@ -6,7 +6,7 @@ Nick Mathewson import unittest from test import support -import sys, os +import sys import uu import io @@ -142,113 +142,61 @@ class UUStdIOTest(unittest.TestCase): class UUFileTest(unittest.TestCase): - def _kill(self, f): - # close and remove file - if f is None: - return - try: - f.close() - except (SystemExit, KeyboardInterrupt): - raise - except: - pass - try: - os.unlink(f.name) - except (SystemExit, KeyboardInterrupt): - raise - except: - pass - def setUp(self): self.tmpin = support.TESTFN + "i" self.tmpout = support.TESTFN + "o" - - def tearDown(self): - del self.tmpin - del self.tmpout + self.addCleanup(support.unlink, self.tmpin) + self.addCleanup(support.unlink, self.tmpout) def test_encode(self): - fin = fout = None - try: - support.unlink(self.tmpin) - fin = open(self.tmpin, 'wb') + with open(self.tmpin, 'wb') as fin: fin.write(plaintext) - fin.close() - fin = open(self.tmpin, 'rb') - fout = open(self.tmpout, 'wb') - uu.encode(fin, fout, self.tmpin, mode=0o644) - fin.close() - fout.close() + with open(self.tmpin, 'rb') as fin: + with open(self.tmpout, 'wb') as fout: + uu.encode(fin, fout, self.tmpin, mode=0o644) - fout = open(self.tmpout, 'rb') + with open(self.tmpout, 'rb') as fout: s = fout.read() - fout.close() - self.assertEqual(s, encodedtextwrapped(0o644, self.tmpin)) + self.assertEqual(s, encodedtextwrapped(0o644, self.tmpin)) - # in_file and out_file as filenames - uu.encode(self.tmpin, self.tmpout, self.tmpin, mode=0o644) - fout = open(self.tmpout, 'rb') + # in_file and out_file as filenames + uu.encode(self.tmpin, self.tmpout, self.tmpin, mode=0o644) + with open(self.tmpout, 'rb') as fout: s = fout.read() - fout.close() - self.assertEqual(s, encodedtextwrapped(0o644, self.tmpin)) - - finally: - self._kill(fin) - self._kill(fout) + self.assertEqual(s, encodedtextwrapped(0o644, self.tmpin)) def test_decode(self): - f = None - try: - support.unlink(self.tmpin) - f = open(self.tmpin, 'wb') + with open(self.tmpin, 'wb') as f: f.write(encodedtextwrapped(0o644, self.tmpout)) - f.close() - f = open(self.tmpin, 'rb') + with open(self.tmpin, 'rb') as f: uu.decode(f) - f.close() - f = open(self.tmpout, 'rb') + with open(self.tmpout, 'rb') as f: s = f.read() - f.close() - self.assertEqual(s, plaintext) - # XXX is there an xp way to verify the mode? - finally: - self._kill(f) + self.assertEqual(s, plaintext) + # XXX is there an xp way to verify the mode? def test_decode_filename(self): - f = None - try: - support.unlink(self.tmpin) - f = open(self.tmpin, 'wb') + with open(self.tmpin, 'wb') as f: f.write(encodedtextwrapped(0o644, self.tmpout)) - f.close() - uu.decode(self.tmpin) + uu.decode(self.tmpin) - f = open(self.tmpout, 'rb') + with open(self.tmpout, 'rb') as f: s = f.read() - f.close() - self.assertEqual(s, plaintext) - finally: - self._kill(f) + self.assertEqual(s, plaintext) def test_decodetwice(self): # Verify that decode() will refuse to overwrite an existing file - f = None - try: - f = io.BytesIO(encodedtextwrapped(0o644, self.tmpout)) - - f = open(self.tmpin, 'rb') + with open(self.tmpin, 'wb') as f: + f.write(encodedtextwrapped(0o644, self.tmpout)) + with open(self.tmpin, 'rb') as f: uu.decode(f) - f.close() - f = open(self.tmpin, 'rb') + with open(self.tmpin, 'rb') as f: self.assertRaises(uu.Error, uu.decode, f) - f.close() - finally: - self._kill(f) def test_main(): support.run_unittest(UUTest, diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index f8af69f..c14f839 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -606,6 +606,7 @@ class ElementTreeTest(unittest.TestCase): self.assertEqual(str(cm.exception), 'junk after document element: line 1, column 12') + self.addCleanup(support.unlink, TESTFN) with open(TESTFN, "wb") as f: f.write(b"junk") it = iterparse(TESTFN) @@ -2794,9 +2795,6 @@ class ElementSlicingTest(unittest.TestCase): class IOTest(unittest.TestCase): - def tearDown(self): - support.unlink(TESTFN) - def test_encoding(self): # Test encoding issues. elem = ET.Element("tag") @@ -2867,12 +2865,14 @@ class IOTest(unittest.TestCase): "" % enc).encode(enc)) def test_write_to_filename(self): + self.addCleanup(support.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''''')) tree.write(TESTFN) with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), b'''''') def test_write_to_text_file(self): + self.addCleanup(support.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''''')) with open(TESTFN, 'w', encoding='utf-8') as f: tree.write(f, encoding='unicode') @@ -2881,6 +2881,7 @@ class IOTest(unittest.TestCase): self.assertEqual(f.read(), b'''''') def test_write_to_binary_file(self): + self.addCleanup(support.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''''')) with open(TESTFN, 'wb') as f: tree.write(f) @@ -2889,6 +2890,7 @@ class IOTest(unittest.TestCase): self.assertEqual(f.read(), b'''''') def test_write_to_binary_file_with_bom(self): + self.addCleanup(support.unlink, TESTFN) tree = ET.ElementTree(ET.XML('''''')) # test BOM writing to buffered file with open(TESTFN, 'wb') as f: diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py index f9bf304..07f7ba0 100644 --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -1179,13 +1179,9 @@ class GzipUtilTestCase(unittest.TestCase): class ServerProxyTestCase(unittest.TestCase): def setUp(self): unittest.TestCase.setUp(self) - if threading: - self.url = URL - else: - # Without threading, http_server() and http_multi_server() will not - # be executed and URL is still equal to None. 'http://' is a just - # enough to choose the scheme (HTTP) - self.url = 'http://' + # Actual value of the URL doesn't matter if it is a string in + # the correct format. + self.url = 'http://fake.localhost' def test_close(self): p = xmlrpclib.ServerProxy(self.url) diff --git a/Lib/threading.py b/Lib/threading.py index bb25565..0ab1e46 100644 --- a/Lib/threading.py +++ b/Lib/threading.py @@ -1093,7 +1093,7 @@ class Thread: def ident(self): """Thread identifier of this thread or None if it has not been started. - This is a nonzero integer. See the thread.get_ident() function. Thread + This is a nonzero integer. See the get_ident() function. Thread identifiers may be recycled when a thread exits and another thread is created. The identifier is available even after the thread has exited. diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index 53bad3f..b78191e 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -3533,7 +3533,7 @@ class Image: self.tk.call('image', 'width', self.name)) class PhotoImage(Image): - """Widget which can display colored images in GIF, PPM/PGM format.""" + """Widget which can display images in PGM, PPM, GIF, PNG format.""" def __init__(self, name=None, cnf={}, master=None, **kw): """Create an image with NAME. @@ -3597,7 +3597,7 @@ class PhotoImage(Image): self.tk.call(args) class BitmapImage(Image): - """Widget which can display a bitmap.""" + """Widget which can display images in XBM format.""" def __init__(self, name=None, cnf={}, master=None, **kw): """Create a bitmap with NAME. diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index ab0db28..cb99193 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -1485,6 +1485,15 @@ class TreeviewTest(AbstractWidgetTest, unittest.TestCase): self.tv.insert('', 'end', text=value), text=None), value) + # test for values which are not None + itemid = self.tv.insert('', 'end', 0) + self.assertEqual(itemid, '0') + itemid = self.tv.insert('', 'end', 0.0) + self.assertEqual(itemid, '0.0') + # this is because False resolves to 0 and element with 0 iid is already present + self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', False) + self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', '') + def test_selection(self): self.assertRaises(TypeError, self.tv.selection, 'spam') diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py index 42e29bd..3cf5faf 100644 --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -1336,7 +1336,7 @@ class Treeview(Widget, tkinter.XView, tkinter.YView): already exist in the tree. Otherwise, a new unique identifier is generated.""" opts = _format_optdict(kw) - if iid: + if iid is not None: res = self.tk.call(self._w, "insert", parent, index, "-id", iid, *opts) else: diff --git a/Lib/trace.py b/Lib/trace.py index 9ba9486..d171577 100755 --- a/Lib/trace.py +++ b/Lib/trace.py @@ -79,9 +79,6 @@ else: PRAGMA_NOCOVER = "#pragma NO COVER" -# Simple rx to find lines with no code. -rx_blank = re.compile(r'^\s*(#.*)?$') - class _Ignore: def __init__(self, modules=None, dirs=None): self._mods = set() if not modules else set(modules) @@ -284,16 +281,15 @@ class CoverageResults: lnotab = _find_executable_linenos(filename) else: lnotab = {} - if lnotab: - source = linecache.getlines(filename) - coverpath = os.path.join(dir, modulename + ".cover") - with open(filename, 'rb') as fp: - encoding, _ = tokenize.detect_encoding(fp.readline) - n_hits, n_lines = self.write_results_file(coverpath, source, - lnotab, count, encoding) - if summary and n_lines: - percent = int(100 * n_hits / n_lines) - sums[modulename] = n_lines, percent, modulename, filename + source = linecache.getlines(filename) + coverpath = os.path.join(dir, modulename + ".cover") + with open(filename, 'rb') as fp: + encoding, _ = tokenize.detect_encoding(fp.readline) + n_hits, n_lines = self.write_results_file(coverpath, source, + lnotab, count, encoding) + if summary and n_lines: + percent = int(100 * n_hits / n_lines) + sums[modulename] = n_lines, percent, modulename, filename if summary and sums: @@ -312,6 +308,7 @@ class CoverageResults: def write_results_file(self, path, lines, lnotab, lines_hit, encoding=None): """Return a coverage results file in path.""" + # ``lnotab`` is a dict of executable lines, or a line number "table" try: outfile = open(path, "w", encoding=encoding) @@ -330,17 +327,13 @@ class CoverageResults: outfile.write("%5d: " % lines_hit[lineno]) n_hits += 1 n_lines += 1 - elif rx_blank.match(line): - outfile.write(" ") - else: - # lines preceded by no marks weren't hit - # Highlight them if so indicated, unless the line contains + elif lineno in lnotab and not PRAGMA_NOCOVER in line: + # Highlight never-executed lines, unless the line contains # #pragma: NO COVER - if lineno in lnotab and not PRAGMA_NOCOVER in line: - outfile.write(">>>>>> ") - n_lines += 1 - else: - outfile.write(" ") + outfile.write(">>>>>> ") + n_lines += 1 + else: + outfile.write(" ") outfile.write(line.expandtabs(8)) return n_hits, n_lines diff --git a/Lib/typing.py b/Lib/typing.py index b5564cc..f2b6aaf 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -1181,10 +1181,18 @@ def _generic_new(base_cls, cls, *args, **kwds): # Assure type is erased on instantiation, # but attempt to store it in __orig_class__ if cls.__origin__ is None: - return base_cls.__new__(cls) + if (base_cls.__new__ is object.__new__ and + cls.__init__ is not object.__init__): + return base_cls.__new__(cls) + else: + return base_cls.__new__(cls, *args, **kwds) else: origin = cls._gorg - obj = base_cls.__new__(origin) + if (base_cls.__new__ is object.__new__ and + cls.__init__ is not object.__init__): + obj = base_cls.__new__(origin) + else: + obj = base_cls.__new__(origin, *args, **kwds) try: obj.__orig_class__ = cls except AttributeError: @@ -2146,6 +2154,7 @@ class NamedTupleMeta(type): "follow default field(s) {default_names}" .format(field_name=field_name, default_names=', '.join(defaults_dict.keys()))) + nm_tpl.__new__.__annotations__ = collections.OrderedDict(types) nm_tpl.__new__.__defaults__ = tuple(defaults) nm_tpl._field_defaults = defaults_dict # update from user namespace without overriding special namedtuple attributes diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index 6be7bb4..86183ad 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -2,7 +2,7 @@ # Test tools for mocking and patching. # Maintained by Michael Foord # Backport for other versions of Python available from -# http://pypi.python.org/pypi/mock +# https://pypi.org/project/mock __all__ = ( 'Mock', diff --git a/Lib/unittest/test/test_discovery.py b/Lib/unittest/test/test_discovery.py index 227b44a..204043b 100644 --- a/Lib/unittest/test/test_discovery.py +++ b/Lib/unittest/test/test_discovery.py @@ -4,10 +4,11 @@ import re import sys import types import pickle -import builtins from test import support +import test.test_importlib.util import unittest +import unittest.mock import unittest.test @@ -820,7 +821,6 @@ class TestDiscovery(unittest.TestCase): def test_discovery_from_dotted_namespace_packages(self): loader = unittest.TestLoader() - orig_import = __import__ package = types.ModuleType('package') package.__path__ = ['/a', '/b'] package.__spec__ = types.SimpleNamespace( @@ -832,11 +832,6 @@ class TestDiscovery(unittest.TestCase): sys.modules[packagename] = package return package - def cleanup(): - builtins.__import__ = orig_import - self.addCleanup(cleanup) - builtins.__import__ = _import - _find_tests_args = [] def _find_tests(start_dir, pattern, namespace=None): _find_tests_args.append((start_dir, pattern)) @@ -844,28 +839,34 @@ class TestDiscovery(unittest.TestCase): loader._find_tests = _find_tests loader.suiteClass = list - suite = loader.discover('package') + + with unittest.mock.patch('builtins.__import__', _import): + # Since loader.discover() can modify sys.path, restore it when done. + with support.DirsOnSysPath(): + # Make sure to remove 'package' from sys.modules when done. + with test.test_importlib.util.uncache('package'): + suite = loader.discover('package') + self.assertEqual(suite, ['/a/tests', '/b/tests']) def test_discovery_failed_discovery(self): loader = unittest.TestLoader() package = types.ModuleType('package') - orig_import = __import__ def _import(packagename, *args, **kwargs): sys.modules[packagename] = package return package - def cleanup(): - builtins.__import__ = orig_import - self.addCleanup(cleanup) - builtins.__import__ = _import - - with self.assertRaises(TypeError) as cm: - loader.discover('package') - self.assertEqual(str(cm.exception), - 'don\'t know how to discover from {!r}' - .format(package)) + with unittest.mock.patch('builtins.__import__', _import): + # Since loader.discover() can modify sys.path, restore it when done. + with support.DirsOnSysPath(): + # Make sure to remove 'package' from sys.modules when done. + with test.test_importlib.util.uncache('package'): + with self.assertRaises(TypeError) as cm: + loader.discover('package') + self.assertEqual(str(cm.exception), + 'don\'t know how to discover from {!r}' + .format(package)) if __name__ == '__main__': diff --git a/Lib/urllib/robotparser.py b/Lib/urllib/robotparser.py index daac29c..883ef24 100644 --- a/Lib/urllib/robotparser.py +++ b/Lib/urllib/robotparser.py @@ -190,7 +190,10 @@ class RobotFileParser: return self.default_entry.req_rate def __str__(self): - return ''.join([str(entry) + "\n" for entry in self.entries]) + entries = self.entries + if self.default_entry is not None: + entries = entries + [self.default_entry] + return '\n'.join(map(str, entries)) + '\n' class RuleLine: @@ -222,10 +225,15 @@ class Entry: def __str__(self): ret = [] for agent in self.useragents: - ret.extend(["User-agent: ", agent, "\n"]) - for line in self.rulelines: - ret.extend([str(line), "\n"]) - return ''.join(ret) + ret.append(f"User-agent: {agent}") + if self.delay is not None: + ret.append(f"Crawl-delay: {self.delay}") + if self.req_rate is not None: + rate = self.req_rate + ret.append(f"Request-rate: {rate.requests}/{rate.seconds}") + ret.extend(map(str, self.rulelines)) + ret.append('') # for compatibility + return '\n'.join(ret) def applies_to(self, useragent): """check if this entry applies to the specified agent""" diff --git a/Lib/uuid.py b/Lib/uuid.py index 32a48ea..db8b2ef 100644 --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -438,7 +438,7 @@ def _ipconfig_getnode(): with proc: for line in proc.stdout: value = line.split(':')[-1].strip().lower() - if re.match('([0-9a-f][0-9a-f]-){5}[0-9a-f][0-9a-f]', value): + if re.fullmatch('(?:[0-9a-f][0-9a-f]-){5}[0-9a-f][0-9a-f]', value): return int(value.replace('-', ''), 16) def _netbios_getnode(): diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index c33737e..d76b2c0 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -210,9 +210,9 @@ def library_recipes(): result.extend([ dict( - name="OpenSSL 1.0.2n", - url="https://www.openssl.org/source/openssl-1.0.2n.tar.gz", - checksum='13bdc1b1d1ff39b6fd42a255e74676a4', + name="OpenSSL 1.0.2o", + url="https://www.openssl.org/source/openssl-1.0.2o.tar.gz", + checksum='44279b8557c3247cbe324e2322ecd114', buildrecipe=build_universal_openssl, configure=None, install=None, @@ -1523,6 +1523,10 @@ def buildDMG(): shellQuote(os.path.join(WORKDIR, 'installer')), shellQuote(imagepath + ".tmp.dmg" ))) + # Try to mitigate race condition in certain versions of macOS, e.g. 10.9, + # when hdiutil fails with "Resource busy" + + time.sleep(10) if not os.path.exists(os.path.join(WORKDIR, "mnt")): os.mkdir(os.path.join(WORKDIR, "mnt")) diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf index e814659..e8deea1 100644 --- a/Mac/BuildScript/resources/ReadMe.rtf +++ b/Mac/BuildScript/resources/ReadMe.rtf @@ -1,4 +1,4 @@ -{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf200 +{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf400 {\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fmodern\fcharset0 CourierNewPSMT;} {\colortbl;\red255\green255\blue255;} {\*\expandedcolortbl;;} @@ -9,13 +9,10 @@ \ \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 -\b \cf0 \ul \ulc0 Which installer variant should I use? [CHANGED in 3.6.5] +\b \cf0 \ul \ulc0 Which installer variant should I use? [CHANGED in 3.6.6] \b0 \ulnone \ \ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 - -\b \cf0 **NEW** -\b0 With Python 3.6.5, the python.org website now provides two installer variants for download: one that installs a +With Python 3.6.5, the python.org website now provides two installer variants for download: one that installs a \i 64-bit-only \i0 Python capable of running on \i macOS 10.9 (Mavericks) @@ -26,9 +23,14 @@ \i0 or later. (This ReadMe was installed with the \i $MACOSX_DEPLOYMENT_TARGET \i0 variant.) Previous Python 3.6.x releases only provided the 10.6 or later installer. If you are running on macOS 10.9 or later and if you have no need for compatibility with older systems, use the 10.9 variant. Use the 10.6 variant if you are running on macOS 10.6 through 10.8, if you need to maintain compatibility with previous 3.6.x releases, or if you want to produce standalone applications that can run on systems from 10.6. The Pythons installed by these installers are built with private copies of some third-party libraries not included with or newer than those in macOS itself. The list of these libraries varies by installer variant and is included at the end of the License.rtf file.\ +\ +\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0 + +\b \cf0 CHANGED in 3.6.6: +\b0 the 10.9+ 64-bit-only installer variant is now the default download. The 10.6+ variant is available from the $FULL_VERSION release page.\ \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 -\b \cf0 \ul \ulc0 \ +\b \cf0 \ul \ Certificate verification and OpenSSL\ \b0 \ulnone \ @@ -42,7 +44,7 @@ This variant of Python 3.6 now includes its own private copy of OpenSSL 1.0.2. \f1 /Applications/Python 3.6 \f0 to install a curated bundle of default root certificates from the third-party \f1 certifi -\f0 package ({\field{\*\fldinst{HYPERLINK "https://pypi.python.org/pypi/certifi"}}{\fldrslt https://pypi.python.org/pypi/certifi}}). If you choose to use +\f0 package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}). If you choose to use \f1 certifi \f0 , you should consider subscribing to the{\field{\*\fldinst{HYPERLINK "https://certifi.io/en/latest/"}}{\fldrslt project's email update service}} to be notified when the certificate bundle is updated.\ \ @@ -56,13 +58,19 @@ The bundled \ The 10.9+ installer variant comes with its own private version of Tcl/Tk 8.6. It does not use system-supplied or third-party supplied versions of Tcl/Tk.\ \ -For the 10.6+ variant, you continue to need to install a newer third-party version of the +For the 10.6+ variant in 3.6.6, you continue to need to install a newer third-party version of the \i Tcl/Tk \i0 8.5 (not 8.6) frameworks to use IDLE or other programs that use the Tkinter graphical user interface toolkit. Visit {\field{\*\fldinst{HYPERLINK "https://www.python.org/download/mac/tcltk/"}}{\fldrslt https://www.python.org/download/mac/tcltk/}} for current information about supported and recommended versions of \i Tcl/Tk \i0 for this version of Python and of macOS.\ +\ +\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 -\b \ul \ +\b \cf0 NOTE: +\b0 As of the next 3.6.x release, 3.6.7, the 10.6+ variant will also include Tcl/Tk 8.6.\ +\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 + +\b \cf0 \ul \ Other changes\ \b0 \ulnone \ diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf index cac9626..22a794a 100644 --- a/Mac/BuildScript/resources/Welcome.rtf +++ b/Mac/BuildScript/resources/Welcome.rtf @@ -1,4 +1,4 @@ -{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf200 +{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf400 \cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;} {\colortbl;\red255\green255\blue255;} {\*\expandedcolortbl;;} @@ -21,4 +21,7 @@ \b NEW in 3.6.5: \b0 two installer variants (10.9+ 64-bit-only, 10.6+ 64-/32-bit), built-in Tcl/Tk 8.6 support in the 10.9+ variant (no additional third-party downloads!)\ -} \ No newline at end of file +\ + +\b CHANGED in 3.6.6: +\b0 the 10.9+ 64-bit-only installer variant is now the default download} \ No newline at end of file diff --git a/Mac/BuildScript/resources/install_certificates.command b/Mac/BuildScript/resources/install_certificates.command index 1d2e2d8..19b4ada 100755 --- a/Mac/BuildScript/resources/install_certificates.command +++ b/Mac/BuildScript/resources/install_certificates.command @@ -6,7 +6,7 @@ # # sample script to install or update a set of default Root Certificates # for the ssl module. Uses the certificates provided by the certifi package: -# https://pypi.python.org/pypi/certifi +# https://pypi.org/project/certifi/ import os import os.path diff --git a/Mac/BuildScript/scripts/postflight.ensurepip b/Mac/BuildScript/scripts/postflight.ensurepip index 3074fa3..36d0594 100755 --- a/Mac/BuildScript/scripts/postflight.ensurepip +++ b/Mac/BuildScript/scripts/postflight.ensurepip @@ -12,6 +12,11 @@ umask 022 "${FWK}/bin/python${PYVER}" -E -s -m ensurepip --upgrade +# bpo-33290: An earlier "pip3 install --upgrade pip" may have installed +# a "pip" in the fw bin directory. For a py3 install, remove it. + +rm -f "${FWK}/bin/pip" + "${FWK}/bin/python${PYVER}" -E -s -Wi \ "${FWK}/lib/python${PYVER}/compileall.py" -q -j0 \ -f -x badsyntax \ diff --git a/Mac/Makefile.in b/Mac/Makefile.in index 1255b13..95fd4a2 100644 --- a/Mac/Makefile.in +++ b/Mac/Makefile.in @@ -221,10 +221,6 @@ install_IDLE: -test -d "$(DESTDIR)$(PYTHONAPPSDIR)/IDLE.app" && rm -rf "$(DESTDIR)$(PYTHONAPPSDIR)/IDLE.app" /bin/cp -PR "$(srcdir)/IDLE/IDLE.app" "$(DESTDIR)$(PYTHONAPPSDIR)" ln -sf "$(INSTALLED_PYTHONAPP)" "$(DESTDIR)$(PYTHONAPPSDIR)/IDLE.app/Contents/MacOS/Python" -ifneq ($(LIPO_32BIT_FLAGS),) - rm "$(DESTDIR)$(PYTHONAPPSDIR)/IDLE.app/Contents/MacOS/Python" - lipo $(LIPO_32BIT_FLAGS) -output "$(DESTDIR)$(PYTHONAPPSDIR)/IDLE.app/Contents/MacOS/Python" "$(BUILDPYTHON)" -endif sed -e "s!%prefix%!$(prefix)!g" -e 's!%exe%!$(PYTHONFRAMEWORK)!g' < "$(srcdir)/IDLE/IDLE.app/Contents/MacOS/IDLE" > "$(DESTDIR)$(PYTHONAPPSDIR)/IDLE.app/Contents/MacOS/IDLE" sed "s!%version%!`$(RUNSHARED) $(BUILDPYTHON) -c 'import platform; print(platform.python_version())'`!g" < "$(srcdir)/IDLE/IDLE.app/Contents/Info.plist" > "$(DESTDIR)$(PYTHONAPPSDIR)/IDLE.app/Contents/Info.plist" if [ -f "$(DESTDIR)$(LIBDEST)/idlelib/config-main.def" ]; then \ diff --git a/Makefile.pre.in b/Makefile.pre.in index 1c9ffea..d912a19 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -578,11 +578,15 @@ $(srcdir)/Modules/_blake2/blake2s_impl.c: $(srcdir)/Modules/_blake2/blake2b_impl # Under GNU make, MAKEFLAGS are sorted and normalized; the 's' for # -s, --silent or --quiet is always the first char. # Under BSD make, MAKEFLAGS might be " -s -v x=y". +# Ignore macros passed by GNU make, passed after -- sharedmods: $(BUILDPYTHON) pybuilddir.txt Modules/_math.o - @case "$$MAKEFLAGS" in \ + @case "`echo X $$MAKEFLAGS | sed 's/^X //;s/ -- .*//'`" in \ *\ -s*|s*) quiet="-q";; \ *) quiet="";; \ esac; \ + echo "$(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' \ + _TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \ + $(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build"; \ $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' \ _TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \ $(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build diff --git a/Misc/ACKS b/Misc/ACKS index b2033ee..93e65a4 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -154,6 +154,7 @@ Mike Bland Martin Bless Pablo Bleyer Erik van Blokland +Stéphane Blondon Eric Blossom Sergey Bobrov Finn Bock @@ -414,6 +415,7 @@ Ulrich Eckhardt David Edelsohn John Edmonds Grant Edwards +Zvi Effron John Ehresman Tal Einat Eric Eisner @@ -672,6 +674,7 @@ Ken Howard Brad Howes Mike Hoy Ben Hoyt +Miro Hrončok Chiu-Hsiang Hsu Chih-Hao Huang Christian Hudon @@ -834,6 +837,7 @@ Maksim Kozyarchuk Stefan Krah Rolf Krahl Bob Kras +Oleg Krasnikov Sebastian Kreft Holger Krekel Michael Kremer @@ -1220,6 +1224,7 @@ Zero Piraeus Antoine Pitrou Jean-François Piéronne Oleg Plakhotnyuk +Marcel Plch Remi Pointel Jon Poler Ariel Poliak @@ -1661,6 +1666,7 @@ David Watson Aaron Watters Henrik Weber Leon Weber +Steve Weber Corran Webster Glyn Webster Phil Webster diff --git a/Misc/NEWS b/Misc/NEWS index 051d78b..c23bcfd 100644 --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,326 @@ Python News +++++++++++ +What's New in Python 3.6.6 final? +================================= + +*Release date: 2018-06-27* + +There were no new changes in version 3.6.6. + + + +What's New in Python 3.6.6 release candidate 1? +=============================================== + +*Release date: 2018-06-11* + +Core and Builtins +----------------- + +- bpo-33786: Fix asynchronous generators to handle GeneratorExit in athrow() + correctly + +- bpo-30654: Fixed reset of the SIGINT handler to SIG_DFL on interpreter + shutdown even when there was a custom handler set previously. Patch by + Philipp Kerling. + +- bpo-33622: Fixed a leak when the garbage collector fails to add an object + with the ``__del__`` method or referenced by it into the + :data:`gc.garbage` list. :c:func:`PyGC_Collect` can now be called when an + exception is set and preserves it. + +- bpo-31849: Fix signed/unsigned comparison warning in pyhash.c. + +- bpo-33391: Fix a leak in set_symmetric_difference(). + +- bpo-28055: Fix unaligned accesses in siphash24(). Patch by Rolf Eike Beer. + +- bpo-33231: Fix potential memory leak in ``normalizestring()``. + +- bpo-29922: Improved error messages in 'async with' when ``__aenter__()`` + or ``__aexit__()`` return non-awaitable object. + +- bpo-33199: Fix ``ma_version_tag`` in dict implementation is uninitialized + when copying from key-sharing dict. + +- bpo-33041: Fixed jumping when the function contains an ``async for`` loop. + +- bpo-32282: Fix an unnecessary ifdef in the include of VersionHelpers.h in + socketmodule on Windows. + +- bpo-21983: Fix a crash in `ctypes.cast()` in case the type argument is a + ctypes structured data type. Patch by Eryk Sun and Oren Milman. + +Library +------- + +- bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. Patch by + Steve Weber. + +- bpo-33812: Datetime instance d with non-None tzinfo, but with + d.tzinfo.utcoffset(d) returning None is now treated as naive by the + astimezone() method. + +- bpo-30805: Avoid race condition with debug logging + +- bpo-33767: The concatenation (``+``) and repetition (``*``) sequence + operations now raise :exc:`TypeError` instead of :exc:`SystemError` when + performed on :class:`mmap.mmap` objects. Patch by Zackery Spytz. + +- bpo-32684: Fix gather to propagate cancellation of itself even with + return_exceptions. + +- bpo-33674: Fix a race condition in SSLProtocol.connection_made() of + asyncio.sslproto: start immediately the handshake instead of using + call_soon(). Previously, data_received() could be called before the + handshake started, causing the handshake to hang or fail. + +- bpo-31467: Fixed bug where calling write_eof() on a + _SelectorSocketTransport after it's already closed raises AttributeError. + +- bpo-33672: Fix Task.__repr__ crash with Cython's bogus coroutines + +- bpo-33469: Fix RuntimeError after closing loop that used run_in_executor + +- bpo-11874: Use a better regex when breaking usage into wrappable parts. + Avoids bogus assertion errors from custom metavar strings. + +- bpo-30877: Fixed a bug in the Python implementation of the JSON decoder + that prevented the cache of parsed strings from clearing after finishing + the decoding. Based on patch by c-fos. + +- bpo-33548: tempfile._candidate_tempdir_list should consider common TEMP + locations + +- bpo-33542: Prevent ``uuid.get_node`` from using a DUID instead of a MAC on + Windows. Patch by Zvi Effron + +- bpo-26819: Fix race condition with `ReadTransport.resume_reading` in + Windows proactor event loop. + +- bpo-28556: Minor fixes in typing module: add annotations to + ``NamedTuple.__new__``, pass ``*args`` and ``**kwds`` in + ``Generic.__new__``. Original PRs by Paulius Šarka and Chad Dombrova. + +- bpo-20087: Updated alias mapping with glibc 2.27 supported locales. + +- bpo-33422: Fix trailing quotation marks getting deleted when looking up + byte/string literals on pydoc. Patch by Andrés Delfino. + +- bpo-33197: Update error message when constructing invalid + inspect.Parameters Patch by Dong-hee Na. + +- bpo-33383: Fixed crash in the get() method of the :mod:`dbm.ndbm` database + object when it is called with a single argument. + +- bpo-33329: Fix multiprocessing regression on newer glibcs + +- bpo-991266: Fix quoting of the ``Comment`` attribute of + :class:`http.cookies.SimpleCookie`. + +- bpo-33131: Upgrade bundled version of pip to 10.0.1. + +- bpo-33308: Fixed a crash in the :mod:`parser` module when converting an ST + object to a tree of tuples or lists with ``line_info=False`` and + ``col_info=True``. + +- bpo-33263: Fix FD leak in `_SelectorSocketTransport` Patch by Vlad + Starostin. + +- bpo-33256: Fix display of ```` call in the html produced by + ``cgitb.html()``. Patch by Stéphane Blondon. + +- bpo-33203: ``random.Random.choice()`` now raises ``IndexError`` for empty + sequences consistently even when called from subclasses without a + ``getrandbits()`` implementation. + +- bpo-33224: Update difflib.mdiff() for PEP 479. Convert an uncaught + StopIteration in a generator into a return-statement. + +- bpo-33209: End framing at the end of C implementation of + :func:`pickle.Pickler.dump`. + +- bpo-32861: The urllib.robotparser's ``__str__`` representation now + includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. + Patch by Michael Lazar. + +- bpo-33096: Allow ttk.Treeview.insert to insert iid that has a false + boolean value. Note iid=0 and iid=False would be same. Patch by Garvit + Khatri. + +- bpo-33127: The ssl module now compiles with LibreSSL 2.7.1. + +- bpo-33021: Release the GIL during fstat() calls, avoiding hang of all + threads when calling mmap.mmap(), os.urandom(), and random.seed(). Patch + by Nir Soffer. + +- bpo-27683: Fix a regression in :mod:`ipaddress` that result of + :meth:`hosts` is empty when the network is constructed by a tuple + containing an integer mask and only 1 bit left for addresses. + +- bpo-32844: Fix wrong redirection of a low descriptor (0 or 1) to stderr in + subprocess if another low descriptor is closed. + +- bpo-31908: Fix output of cover files for ``trace`` module command-line + tool. Previously emitted cover files only when ``--missing`` option was + used. Patch by Michael Selik. + +- bpo-31457: If nested log adapters are used, the inner ``process()`` + methods are no longer omitted. + +- bpo-16865: Support arrays >=2GiB in :mod:`ctypes`. Patch by Segev Finer. + +- bpo-31238: pydoc: the stop() method of the private ServerThread class now + waits until DocServer.serve_until_quit() completes and then explicitly + sets its docserver attribute to None to break a reference cycle. + +Documentation +------------- + +- bpo-33503: Fix broken pypi link + +- bpo-33421: Add missing documentation for ``typing.AsyncContextManager``. + +- bpo-33378: Add Korean language switcher for https://docs.python.org/3/ + +- bpo-33276: Clarify that the ``__path__`` attribute on modules cannot be + just any value. + +- bpo-33201: Modernize documentation for writing C extension types. + +- bpo-33195: Deprecate ``Py_UNICODE`` usage in ``c-api/arg`` document. + ``Py_UNICODE`` related APIs are deprecated since Python 3.3, but it is + missed in the document. + +- bpo-33126: Document PyBuffer_ToContiguous(). + +- bpo-27212: Modify documentation for the :func:`islice` recipe to consume + initial values up to the start index. + +- bpo-28247: Update :mod:`zipapp` documentation to describe how to make + standalone applications. + +- bpo-18802: Documentation changes for ipaddress. Patch by Jon Foster and + Berker Peksag. + +- bpo-27428: Update documentation to clarify that ``WindowsRegistryFinder`` + implements ``MetaPathFinder``. (Patch by Himanshu Lakhara) + +- bpo-8243: Add a note about curses.addch and curses.addstr exception + behavior when writing outside a window, or pad. + +- bpo-31432: Clarify meaning of CERT_NONE, CERT_OPTIONAL, and CERT_REQUIRED + flags for ssl.SSLContext.verify_mode. + +Tests +----- + +- bpo-33655: Ignore test_posix_fallocate failures on BSD platforms that + might be due to running on ZFS. + +- bpo-19417: Add test_bdb.py. + +Build +----- + +- bpo-5755: Move ``-Wstrict-prototypes`` option to ``CFLAGS_NODIST`` from + ``OPT``. This option emitted annoying warnings when building extension + modules written in C++. + +- bpo-33614: Ensures module definition files for the stable ABI on Windows + are correctly regenerated. + +- bpo-33522: Enable CI builds on Visual Studio Team Services at + https://python.visualstudio.com/cpython + +- bpo-33012: Add ``-Wno-cast-function-type`` for gcc 8 for silencing + warnings about function casts like casting to PyCFunction in method + definition lists. + +- bpo-33394: Enable the verbose build for extension modules, when GNU make + is passed macros on the command line. + +Windows +------- + +- bpo-33184: Update Windows installer to OpenSSL 1.0.2o. + +macOS +----- + +- bpo-33184: Update macOS installer build to use OpenSSL 1.0.2o. + +IDLE +---- + +- bpo-33656: On Windows, add API call saying that tk scales for DPI. On + Windows 8.1+ or 10, with DPI compatibility properties of the Python binary + unchanged, and a monitor resolution greater than 96 DPI, this should make + text and lines sharper. It should otherwise have no effect. + +- bpo-33768: Clicking on a context line moves that line to the top of the + editor window. + +- bpo-33763: IDLE: Use read-only text widget for code context instead of + label widget. + +- bpo-33664: Scroll IDLE editor text by lines. Previously, the mouse wheel + and scrollbar slider moved text by a fixed number of pixels, resulting in + partial lines at the top of the editor box. The change also applies to + the shell and grep output windows, but not to read-only text views. + +- bpo-33679: Enable theme-specific color configuration for Code Context. Use + the Highlights tab to see the setting for built-in themes or add settings + to custom themes. + +- bpo-33642: Display up to maxlines non-blank lines for Code Context. If + there is no current context, show a single blank line. + +- bpo-33628: IDLE: Cleanup codecontext.py and its test. + +- bpo-33564: IDLE's code context now recognizes async as a block opener. + +- bpo-29706: IDLE now colors async and await as keywords in 3.6. They become + full keywords in 3.7. + +- bpo-21474: Update word/identifier definition from ascii to unicode. In + text and entry boxes, this affects selection by double-click, movement + left/right by control-left/right, and deletion left/right by + control-BACKSPACE/DEL. + +- bpo-33204: IDLE: consistently color invalid string prefixes. A 'u' string + prefix cannot be paired with either 'r' or 'f'. Consistently color as much + of the prefix, starting at the right, as is valid. Revise and extend + colorizer test. + +- bpo-32831: Add docstrings and tests for codecontext. + +Tools/Demos +----------- + +- bpo-33189: :program:`pygettext.py` now recognizes only literal strings as + docstrings and translatable strings, and rejects bytes literals and + f-string expressions. + +- bpo-31920: Fixed handling directories as arguments in the ``pygettext`` + script. Based on patch by Oleg Krasnikov. + +- bpo-29673: Fix pystackv and pystack gdbinit macros. + +- bpo-32885: Add an ``-n`` flag for ``Tools/scripts/pathfix.py`` to disbale + automatic backup creation (files with ``~`` suffix). + +- bpo-31583: Fix 2to3 for using with --add-suffix option but without + --output-dir option for relative path to files in current directory. + +C API +----- + +- bpo-32374: Document that m_traverse for multi-phase initialized modules + can be called with m_state=NULL, and add a sanity check + + What's New in Python 3.6.5 final? ================================= @@ -52,7 +372,7 @@ Core and Builtins Chandra. - bpo-32583: Fix possible crashing in builtin Unicode decoders caused by - write out-of- bound errors when using customized decode error handlers. + write out-of-bound errors when using customized decode error handlers. - bpo-26163: Improved frozenset() hash to create more distinct hash values when faced with datasets containing many similar values. @@ -561,8 +881,8 @@ Library ``sqlite3.Cursor`` object more than once. Patch by Oren Milman. - bpo-31672: ``idpattern`` in ``string.Template`` matched some non-ASCII - characters. Now it uses ``-i`` regular expression local flag to avoid non- - ASCII characters. + characters. Now it uses ``-i`` regular expression local flag to avoid + non-ASCII characters. - bpo-31764: Prevent a crash in ``sqlite3.Cursor.close()`` in case the ``Cursor`` object is uninitialized. Patch by Oren Milman. @@ -611,8 +931,8 @@ Library - bpo-31482: ``random.seed()`` now works with bytes in version=1 - bpo-31334: Fix ``poll.poll([timeout])`` in the ``select`` module for - arbitrary negative timeouts on all OSes where it can only be a non- - negative integer or -1. Patch by Riccardo Coccioli. + arbitrary negative timeouts on all OSes where it can only be a + non-negative integer or -1. Patch by Riccardo Coccioli. - bpo-31310: multiprocessing's semaphore tracker should be launched again if crashed. @@ -630,8 +950,8 @@ Documentation documentation example code. Patch by Brad Smith. - bpo-30085: The operator functions without double underscores are preferred - for clarity. The one with underscores are only kept for back- - compatibility. + for clarity. The one with underscores are only kept for + back-compatibility. Tests ----- @@ -745,10 +1065,10 @@ IDLE - bpo-31459: Rename IDLE's module browser from Class Browser to Module Browser. The original module-level class and method browser became a module browser, with the addition of module-level functions, years ago. - Nested classes and functions were added yesterday. For back- - compatibility, the virtual event <>, which appears on - the Keys tab of the Settings dialog, is not changed. Patch by Cheryl - Sabella. + Nested classes and functions were added yesterday. For + back-compatibility, the virtual event <>, which + appears on the Keys tab of the Settings dialog, is not changed. Patch by + Cheryl Sabella. - bpo-31500: Default fonts now are scaled on HiDPI displays. @@ -879,7 +1199,7 @@ Core and Builtins - bpo-30703: Improve signal delivery. Avoid using Py_AddPendingCall from signal handler, to avoid calling - signal- unsafe functions. The tests I'm adding here fail without the rest + signal-unsafe functions. The tests I'm adding here fail without the rest of the patch, on Linux and OS X. This means our signal delivery logic had defects (some signals could be lost). @@ -937,8 +1257,8 @@ Library - bpo-29212: Fix concurrent.futures.thread.ThreadPoolExecutor threads to have a non repr() based thread name by default when no thread_name_prefix - is supplied. They will now identify themselves as "ThreadPoolExecutor- - y_n". + is supplied. They will now identify themselves as + "ThreadPoolExecutor-y_n". - bpo-9146: Fix a segmentation fault in _hashopenssl when standard hash functions such as md5 are not available in the linked OpenSSL library. As @@ -986,7 +1306,7 @@ Library contains CR or LF. Patch by Dong-hee Na. - bpo-30595: multiprocessing.Queue.get() with a timeout now polls its reader - in non- blocking mode if it succeeded to aquire the lock but the acquire + in non-blocking mode if it succeeded to aquire the lock but the acquire took longer than the timeout. - bpo-29403: Fix ``unittest.mock``'s autospec to not fail on method-bound @@ -1023,7 +1343,7 @@ Library certain cases. - bpo-30879: os.listdir() and os.scandir() now emit bytes names when called - with bytes- like argument. + with bytes-like argument. - bpo-30746: Prohibited the '=' character in environment variable names in ``os.putenv()`` and ``os.spawn*()``. @@ -1125,15 +1445,15 @@ IDLE The main difference for users is that user configurable key bindings for builtin features are now handled uniformly. Now, editing a binding in a keyset only affects its value in the keyset. All bindings are defined - together in the system-specific default keysets in config- extensions.def. - All custom keysets are saved as a whole in config- extension.cfg. All - take effect as soon as one clicks Apply or Ok. + together in the system-specific default keysets in config-extensions.def. + All custom keysets are saved as a whole in config-extension.cfg. All take + effect as soon as one clicks Apply or Ok. The affected events are '<>', '<>', '<>', '<>', '<>', - '<>', '<>', and '<>'. Any - (global) customizations made before 3.6.3 will not affect their keyset- - specific customization after 3.6.3. and vice versa. + '<>', '<>', and '<>'. Any (global) + customizations made before 3.6.3 will not affect their keyset-specific + customization after 3.6.3. and vice versa. Inital patch by Charles Wohlganger. @@ -1305,7 +1625,7 @@ Security infinite loop DoS), CVE-2016-9063 (Integer overflow, re-fix), CVE-2016-0718 (Fix regression bugs from 2.2.0's fix to CVE-2016-0718) and CVE-2012-0876 (Counter hash flooding with SipHash). Note: the - CVE-2016-5300 (Use os- specific entropy sources like getrandom) doesn't + CVE-2016-5300 (Use os-specific entropy sources like getrandom) doesn't impact Python, since Python already gets entropy from the OS to set the expat secret using ``XML_SetHashSalt()``. @@ -1334,8 +1654,8 @@ Core and Builtins mutated during searching, inserting or comparing. Based on patches by Duane Griffin and Tim Mitchell. -- bpo-25794: Fixed type.__setattr__() and type.__delattr__() for non- - interned attribute names. Based on patch by Eryk Sun. +- bpo-25794: Fixed type.__setattr__() and type.__delattr__() for + non-interned attribute names. Based on patch by Eryk Sun. - bpo-30039: If a KeyboardInterrupt happens when the interpreter is in the middle of resuming a chain of nested 'yield from' or 'await' calls, it's @@ -1381,8 +1701,9 @@ Library - bpo-23890: unittest.TestCase.assertRaises() now manually breaks a reference cycle to not keep objects alive longer than expected. -- bpo-30149: inspect.signature() now supports callables with variable- - argument parameters wrapped with partialmethod. Patch by Dong-hee Na. +- bpo-30149: inspect.signature() now supports callables with + variable-argument parameters wrapped with partialmethod. Patch by Dong-hee + Na. - bpo-30645: Fix path calculation in imp.load_package(), fixing it for cases when a package is only shipped with bytecodes. Patch by Alexandru @@ -1465,9 +1786,9 @@ Library - bpo-30048: Fixed ``Task.cancel()`` can be ignored when the task is running coroutine and the coroutine returned without any more ``await``. -- bpo-30266: contextlib.AbstractContextManager now supports anti- - registration by setting __enter__ = None or __exit__ = None, following the - pattern introduced in bpo-25958. Patch by Jelle Zijlstra. +- bpo-30266: contextlib.AbstractContextManager now supports + anti-registration by setting __enter__ = None or __exit__ = None, + following the pattern introduced in bpo-25958. Patch by Jelle Zijlstra. - bpo-30298: Weaken the condition of deprecation warnings for inline modifiers. Now allowed several subsequential inline modifiers at the start @@ -1489,8 +1810,8 @@ Library when Ctrl-C is received. - bpo-28556: Various updates to typing module: add typing.NoReturn type, use - WrapperDescriptorType, minor bug-fixes. Original PRs by Jim Fasarakis- - Hilliard and Ivan Levkivskyi. + WrapperDescriptorType, minor bug-fixes. Original PRs by Jim + Fasarakis-Hilliard and Ivan Levkivskyi. - bpo-30205: Fix getsockname() for unbound AF_UNIX sockets on Linux. @@ -1577,8 +1898,8 @@ IDLE scrollbar; selecting an item by hitting Return. Hangs on MacOSX should no longer happen. Patch by Louie Lu. -- bpo-25514: Add doc subsubsection about IDLE failure to start. Popup no- - connection message directs users to this section. +- bpo-25514: Add doc subsubsection about IDLE failure to start. Popup + no-connection message directs users to this section. - bpo-30642: Fix reference leaks in IDLE tests. Patches by Louie Lu and Terry Jan Reedy. @@ -1602,15 +1923,15 @@ Build ----- - bpo-29941: Add ``--with-assertions`` configure flag to explicitly enable C - ``assert()`` checks. Defaults to off. ``--with-pydebug`` implies ``--with- - assertions``. + ``assert()`` checks. Defaults to off. ``--with-pydebug`` implies + ``--with-assertions``. -- bpo-28787: Fix out-of-tree builds of Python when configured with ``--with - --dtrace``. +- bpo-28787: Fix out-of-tree builds of Python when configured with + ``--with--dtrace``. - bpo-29243: Prevent unnecessary rebuilding of Python during ``make test``, ``make install`` and some other make targets when configured with - ``--enable- optimizations``. + ``--enable-optimizations``. - bpo-23404: Don't regenerate generated files based on file modification time anymore: the action is now explicit. Replace ``make touch`` with @@ -2132,7 +2453,7 @@ Core and Builtins when decode astral characters. Patch by Xiang Zhang. - bpo-19398: Extra slash no longer added to sys.path components in case of - empty compile- time PYTHONPATH components. + empty compile-time PYTHONPATH components. - bpo-28665: Improve speed of the STORE_DEREF opcode by 40%. @@ -2451,8 +2772,8 @@ Library - bpo-27358: Optimized merging var-keyword arguments and improved error message when passing a non-mapping as a var-keyword argument. -- bpo-28257: Improved error message when passing a non-iterable as a var- - positional argument. Added opcode BUILD_TUPLE_UNPACK_WITH_CALL. +- bpo-28257: Improved error message when passing a non-iterable as a + var-positional argument. Added opcode BUILD_TUPLE_UNPACK_WITH_CALL. - bpo-28322: Fixed possible crashes when unpickle itertools objects from incorrect pickle data. Based on patch by John Leitch. @@ -2787,7 +3108,7 @@ Library middle of the regular expression. - bpo-26885: xmlrpc now supports unmarshalling additional data types used by - Apache XML- RPC implementation for numerics and None. + Apache XML-RPC implementation for numerics and None. - bpo-28070: Fixed parsing inline verbose flag in regular expressions. @@ -2972,8 +3293,8 @@ Library http.client.HTTPConnection requests. The urllib.request.AbstractHTTPHandler class does not enforce a Content-Length header any more. If a HTTP request has a file or iterable body, but no - Content-Length header, the library now falls back to use chunked transfer- - encoding. + Content-Length header, the library now falls back to use chunked + transfer-encoding. - A new version of typing.py from https://github.com/python/typing: - Collection (only for 3.6) (Issue #27598) - Add FrozenSet to __all__ @@ -3067,8 +3388,8 @@ Build - bpo-27983: Cause lack of llvm-profdata tool when using clang as required for PGO linking to be a configure time error rather than make time when - --with- optimizations is enabled. Also improve our ability to find the - llvm- profdata tool on MacOS and some Linuxes. + ``--with-optimizations`` is enabled. Also improve our ability to find the + llvm-profdata tool on MacOS and some Linuxes. - bpo-21590: Support for DTrace and SystemTap probes. @@ -3353,8 +3674,8 @@ Build Hsuan Yen. - bpo-27641: The configure script now inserts comments into the makefile to - prevent the pgen and _freeze_importlib executables from being cross- - compiled. + prevent the pgen and _freeze_importlib executables from being + cross-compiled. - bpo-26662: Set PYTHON_FOR_GEN in configure as the Python program to be used for file generation during the build. @@ -4136,7 +4457,7 @@ Core and Builtins - bpo-25267: The UTF-8 encoder is now up to 75 times as fast for error handlers: ``ignore``, ``replace``, ``surrogateescape``, ``surrogatepass``. - Patch co- written with Serhiy Storchaka. + Patch co-written with Serhiy Storchaka. - bpo-25280: Import trace messages emitted in verbose (-v) mode are no longer formatted twice. @@ -4148,7 +4469,7 @@ Core and Builtins - bpo-25003: On Solaris 11.3 or newer, os.urandom() now uses the getrandom() function instead of the getentropy() function. The getentropy() function is blocking to generate very good quality entropy, os.urandom() doesn't - need such high- quality entropy. + need such high-quality entropy. - bpo-9232: Modify Python's grammar to allow trailing commas in the argument list of a function declaration. For example, "def f(\*, a = 3,): pass" is @@ -4193,9 +4514,9 @@ Library - bpo-17214: The "urllib.request" module now percent-encodes non-ASCII bytes found in redirect target URLs. Some servers send Location header fields - with non- ASCII bytes, but "http.client" requires the request target to be - ASCII- encodable, otherwise a UnicodeEncodeError is raised. Based on - patch by Christian Heimes. + with non-ASCII bytes, but "http.client" requires the request target to be + ASCII-encodable, otherwise a UnicodeEncodeError is raised. Based on patch + by Christian Heimes. - bpo-27033: The default value of the decode_data parameter for smtpd.SMTPChannel and smtpd.SMTPServer constructors is changed to False. @@ -4269,8 +4590,8 @@ Library Xiang Zhang. - bpo-26804: urllib.request will prefer lower_case proxy environment - variables over UPPER_CASE or Mixed_Case ones. Patch contributed by Hans- - Peter Jansen. + variables over UPPER_CASE or Mixed_Case ones. Patch contributed by + Hans-Peter Jansen. - bpo-26837: assertSequenceEqual() now correctly outputs non-stringified differing items (like bytes in the -b mode). This affects @@ -4521,8 +4842,8 @@ Library - bpo-25911: Restored support of bytes paths in os.walk() on Windows. -- bpo-26045: Add UTF-8 suggestion to error message when posting a non- - Latin-1 string with http.client. +- bpo-26045: Add UTF-8 suggestion to error message when posting a + non-Latin-1 string with http.client. - bpo-26039: Added zipfile.ZipInfo.from_file() and zipinfo.ZipInfo.is_dir(). Patch by Thomas Kluyver. @@ -4535,8 +4856,8 @@ Library and "keywords" attributes of functools.partial have now always types tuple and dict correspondingly. -- bpo-26202: copy.deepcopy() now correctly copies range() objects with non- - atomic attributes. +- bpo-26202: copy.deepcopy() now correctly copies range() objects with + non-atomic attributes. - bpo-23076: Path.glob() now raises a ValueError if it's called with an invalid pattern. Patch by Thomas Nyberg. @@ -4979,12 +5300,12 @@ IDLE - bpo-25173: Associate tkinter messageboxes with a specific widget. For Mac OSX, make them a 'sheet'. Patch by Mark Roseman. -- bpo-25198: Enhance the initial html viewer now used for Idle Help. * - Properly indent fixed-pitch text (patch by Mark Roseman). * Give code - snippet a very Sphinx- like light blueish-gray background. * Re-use - initial width and height set by users for shell and editor. * When the - Table of Contents (TOC) menu is used, put the section header at the top of - the screen. +- bpo-25198: Enhance the initial html viewer now used for Idle Help. + Properly indent fixed-pitch text (patch by Mark Roseman). Give code + snippet a very Sphinx-like light blueish-gray background. Re-use initial + width and height set by users for shell and editor. When the Table of + Contents (TOC) menu is used, put the section header at the top of the + screen. - bpo-25225: Condense and rewrite Idle doc section on text colors. @@ -5086,7 +5407,7 @@ Build via macros (in particular on Android). Patch by Chi Hsuan Yen. - bpo-22359: Disable the rules for running _freeze_importlib and pgen when - cross- compiling. The output of these programs is normally saved with the + cross-compiling. The output of these programs is normally saved with the source code anyway, and is still regenerated when doing a native build. Patch by Xavier de Gaye. @@ -5159,9 +5480,9 @@ Tools/Demos ----------- - bpo-26799: Fix python-gdb.py: don't get C types once when the Python code - is loaded, but get C types on demand. The C types can change if python- - gdb.py is loaded before the Python executable. Patch written by Thomas - Ilsche. + is loaded, but get C types on demand. The C types can change if + python-gdb.py is loaded before the Python executable. Patch written by + Thomas Ilsche. - bpo-26271: Fix the Freeze tool to properly use flags passed through configure. Patch by Daniel Shaulov. @@ -5229,7 +5550,7 @@ Core and Builtins when decode astral characters. Patch by Xiang Zhang. - bpo-19398: Extra slash no longer added to sys.path components in case of - empty compile- time PYTHONPATH components. + empty compile-time PYTHONPATH components. - bpo-28426: Fixed potential crash in PyUnicode_AsDecodedObject() in debug build. @@ -5313,7 +5634,7 @@ Core and Builtins - bpo-27419: Standard __import__() no longer look up "__import__" in globals or builtins for importing submodules or "from import". Fixed handling an - error of non- string package name. + error of non-string package name. - bpo-27083: Respect the PYTHONCASEOK environment variable under Windows. @@ -5859,8 +6180,8 @@ Build - bpo-27983: Cause lack of llvm-profdata tool when using clang as required for PGO linking to be a configure time error rather than make time when - --with- optimizations is enabled. Also improve our ability to find the - llvm- profdata tool on MacOS and some Linuxes. + ``--with-optimizations`` is enabled. Also improve our ability to find the + llvm-profdata tool on MacOS and some Linuxes. - bpo-26307: The profile-opt build now applies PGO to the built-in modules. @@ -5878,8 +6199,8 @@ Build Hsuan Yen. - bpo-27641: The configure script now inserts comments into the makefile to - prevent the pgen and _freeze_importlib executables from being cross- - compiled. + prevent the pgen and _freeze_importlib executables from being + cross-compiled. - bpo-26662: Set PYTHON_FOR_GEN in configure as the Python program to be used for file generation during the build. @@ -6123,9 +6444,9 @@ Library - bpo-17214: The "urllib.request" module now percent-encodes non-ASCII bytes found in redirect target URLs. Some servers send Location header fields - with non- ASCII bytes, but "http.client" requires the request target to be - ASCII- encodable, otherwise a UnicodeEncodeError is raised. Based on - patch by Christian Heimes. + with non-ASCII bytes, but "http.client" requires the request target to be + ASCII-encodable, otherwise a UnicodeEncodeError is raised. Based on patch + by Christian Heimes. - bpo-26892: Honor debuglevel flag in urllib.request.HTTPHandler. Patch contributed by Chi Hsuan Yen. @@ -6168,8 +6489,8 @@ Library Xiang Zhang. - bpo-26804: urllib.request will prefer lower_case proxy environment - variables over UPPER_CASE or Mixed_Case ones. Patch contributed by Hans- - Peter Jansen. + variables over UPPER_CASE or Mixed_Case ones. Patch contributed by + Hans-Peter Jansen. - bpo-26837: assertSequenceEqual() now correctly outputs non-stringified differing items (like bytes in the -b mode). This affects @@ -6332,8 +6653,8 @@ Library - bpo-25911: Restored support of bytes paths in os.walk() on Windows. -- bpo-26045: Add UTF-8 suggestion to error message when posting a non- - Latin-1 string with http.client. +- bpo-26045: Add UTF-8 suggestion to error message when posting a + non-Latin-1 string with http.client. - bpo-12923: Reset FancyURLopener's redirect counter even if there is an exception. Based on patches by Brian Brazil and Daniel Rocco. @@ -6343,8 +6664,8 @@ Library and "keywords" attributes of functools.partial have now always types tuple and dict correspondingly. -- bpo-26202: copy.deepcopy() now correctly copies range() objects with non- - atomic attributes. +- bpo-26202: copy.deepcopy() now correctly copies range() objects with + non-atomic attributes. - bpo-23076: Path.glob() now raises a ValueError if it's called with an invalid pattern. Patch by Thomas Nyberg. @@ -6578,7 +6899,7 @@ Build de Gaye. - bpo-22359: Disable the rules for running _freeze_importlib and pgen when - cross- compiling. The output of these programs is normally saved with the + cross-compiling. The output of these programs is normally saved with the source code anyway, and is still regenerated when doing a native build. Patch by Xavier de Gaye. @@ -6650,9 +6971,9 @@ Tools/Demos ----------- - bpo-26799: Fix python-gdb.py: don't get C types once when the Python code - is loaded, but get C types on demand. The C types can change if python- - gdb.py is loaded before the Python executable. Patch written by Thomas - Ilsche. + is loaded, but get C types on demand. The C types can change if + python-gdb.py is loaded before the Python executable. Patch written by + Thomas Ilsche. - bpo-26271: Fix the Freeze tool to properly use flags passed through configure. Patch by Daniel Shaulov. @@ -6747,7 +7068,7 @@ Core and Builtins - bpo-25003: On Solaris 11.3 or newer, os.urandom() now uses the getrandom() function instead of the getentropy() function. The getentropy() function is blocking to generate very good quality entropy, os.urandom() doesn't - need such high- quality entropy. + need such high-quality entropy. - bpo-25182: The stdprinter (used as sys.stderr before the io module is imported at startup) now uses the backslashreplace error handler. @@ -7055,12 +7376,12 @@ IDLE - bpo-25173: Associate tkinter messageboxes with a specific widget. For Mac OSX, make them a 'sheet'. Patch by Mark Roseman. -- bpo-25198: Enhance the initial html viewer now used for Idle Help. * - Properly indent fixed-pitch text (patch by Mark Roseman). * Give code - snippet a very Sphinx- like light blueish-gray background. * Re-use - initial width and height set by users for shell and editor. * When the - Table of Contents (TOC) menu is used, put the section header at the top of - the screen. +- bpo-25198: Enhance the initial html viewer now used for Idle Help. + Properly indent fixed-pitch text (patch by Mark Roseman). Give code + snippet a very Sphinx-like light blueish-gray background. Re-use initial + width and height set by users for shell and editor. When the Table of + Contents (TOC) menu is used, put the section header at the top of the + screen. - bpo-25225: Condense and rewrite Idle doc section on text colors. @@ -7331,10 +7652,11 @@ IDLE - bpo-23672: Allow Idle to edit and run files with astral chars in name. Patch by Mohd Sanad Zaki Rizvi. -- bpo-24745: Idle editor default font. Switch from Courier to platform- - sensitive TkFixedFont. This should not affect current customized font - selections. If there is a problem, edit $HOME/.idlerc/config-main.cfg and - remove 'fontxxx' entries from [Editor Window]. Patch by Mark Roseman. +- bpo-24745: Idle editor default font. Switch from Courier to + platform-sensitive TkFixedFont. This should not affect current customized + font selections. If there is a problem, edit + $HOME/.idlerc/config-main.cfg and remove 'fontxxx' entries from [Editor + Window]. Patch by Mark Roseman. - bpo-21192: Idle editor. When a file is run, put its name in the restart bar. Do not print false prompts. Original patch by Adnan Umer. @@ -7360,8 +7682,8 @@ Tests ----- - bpo-24751: When running regrtest with the ``-w`` command line option, a - test run is no longer marked as a failure if all tests succeed when re- - run. + test run is no longer marked as a failure if all tests succeed when + re-run. What's New in Python 3.5.0 beta 4? @@ -7437,7 +7759,7 @@ Library inside a data segment. - bpo-15014: SMTP.auth() and SMTP.login() now support RFC 4954's optional - initial- response argument to the SMTP AUTH command. + initial-response argument to the SMTP AUTH command. - bpo-24669: Fix inspect.getsource() for 'async def' functions. Patch by Kai Groner. @@ -7501,7 +7823,7 @@ Library called. Based on patch by Martin Panter. - bpo-20387: Restore semantic round-trip correctness in tokenize/untokenize - for tab- indented blocks. + for tab-indented blocks. - bpo-24456: Fixed possible buffer over-read in adpcm2lin() and lin2adpcm() functions of the audioop module. @@ -8083,8 +8405,8 @@ Tests Tools/Demos ----------- -- bpo-18128: pygettext now uses standard +NNNN format in the POT-Creation- - Date header. +- bpo-18128: pygettext now uses standard +NNNN format in the + POT-Creation-Date header. - bpo-23935: Argument Clinic's understanding of format units accepting bytes, bytearrays, and buffers is now consistent with both the @@ -8127,8 +8449,8 @@ Core and Builtins - bpo-23681: The -b option now affects comparisons of bytes with int. -- bpo-23632: Memoryviews now allow tuple indexing (including for multi- - dimensional memoryviews). +- bpo-23632: Memoryviews now allow tuple indexing (including for + multi-dimensional memoryviews). - bpo-23192: Fixed generator lambdas. Patch by Bruno Cauet. @@ -8139,7 +8461,7 @@ Library ------- - bpo-14260: The groupindex attribute of regular expression pattern object - now is non- modifiable mapping. + now is non-modifiable mapping. - bpo-23792: Ignore KeyboardInterrupt when the pydoc pager is active. This mimics the behavior of the standard unix pagers, and prevents pipepager @@ -8487,8 +8809,8 @@ Core and Builtins - bpo-20335: bytes constructor now raises TypeError when encoding or errors is specified with non-string argument. Based on patch by Renaud Blanch. -- bpo-22834: If the current working directory ends up being set to a non- - existent directory then import will no longer raise FileNotFoundError. +- bpo-22834: If the current working directory ends up being set to a + non-existent directory then import will no longer raise FileNotFoundError. - bpo-22869: Move the interpreter startup & shutdown code to a new dedicated pylifecycle.c module @@ -8705,8 +9027,8 @@ Library state is now always restored or swapped, not only if why is WHY_YIELD or WHY_RETURN. Patch co-written with Antoine Pitrou. -- bpo-14099: Restored support of writing ZIP files to tellable but non- - seekable streams. +- bpo-14099: Restored support of writing ZIP files to tellable but + non-seekable streams. - bpo-14099: Writing to ZipFile and reading multiple ZipExtFiles is threadsafe now. @@ -8958,9 +9280,9 @@ Library - bpo-18216: gettext now raises an error when a .mo file has an unsupported major version number. Patch by Aaron Hill. -- bpo-13918: Provide a locale.delocalize() function which can remove locale- - specific number formatting from a string representing a number, without - then converting it to a specific type. Patch by Cédric Krier. +- bpo-13918: Provide a locale.delocalize() function which can remove + locale-specific number formatting from a string representing a number, + without then converting it to a specific type. Patch by Cédric Krier. - bpo-22676: Make the pickling of global objects which don't have a __module__ attribute less slow. @@ -8978,9 +9300,9 @@ Library without executing the test suite. The new `errors` attribute on TestLoader exposes these non-fatal errors encountered during discovery. -- bpo-21991: Make email.headerregistry's header 'params' attributes be read- - only (MappingProxyType). Previously the dictionary was modifiable but a - new one was created on each access of the attribute. +- bpo-21991: Make email.headerregistry's header 'params' attributes be + read-only (MappingProxyType). Previously the dictionary was modifiable + but a new one was created on each access of the attribute. - bpo-22638: SSLv3 is now disabled throughout the standard library. It can still be enabled by instantiating a SSLContext manually. @@ -9274,7 +9596,7 @@ Library - bpo-5411: Added support for the "xztar" format in the shutil module. - bpo-21121: Don't force 3rd party C extensions to be built with - -Werror=declaration- after-statement. + ``-Werror=declaration-after-statement``. - bpo-21975: Fixed crash when using uninitialized sqlite3.Row (in particular when unpickling pickled sqlite3.Row). sqlite3.Row is now initialized in @@ -9810,7 +10132,7 @@ Library - bpo-20378: Improve repr of inspect.Signature and inspect.Parameter. - bpo-20816: Fix inspect.getcallargs() to raise correct TypeError for - missing keyword- only arguments. Patch by Jeremiah Lowin. + missing keyword-only arguments. Patch by Jeremiah Lowin. - bpo-20817: Fix inspect.getcallargs() to fail correctly if more than 3 arguments are missing. Patch by Jeremiah Lowin. @@ -9821,8 +10143,9 @@ Library wording by David Gutteridge) - bpo-21117: Fix inspect.signature to better support functools.partial. Due - to the specifics of functools.partial implementation, positional-or- - keyword arguments passed as keyword arguments become keyword-only. + to the specifics of functools.partial implementation, + positional-or-keyword arguments passed as keyword arguments become + keyword-only. - bpo-20334: inspect.Signature and inspect.Parameter are now hashable. Thanks to Antony Lee for bug reports and suggestions. @@ -9935,8 +10258,8 @@ IDLE - bpo-21284: Paragraph reformat test passes after user changes reformat width. -- bpo-17654: Ensure IDLE menus are customized properly on OS X for non- - framework builds and for all variants of Tk. +- bpo-17654: Ensure IDLE menus are customized properly on OS X for + non-framework builds and for all variants of Tk. - bpo-23180: Rename IDLE "Windows" menu item to "Window". Patch by Al Sweigart. @@ -9953,8 +10276,8 @@ Build - bpo-22592: Drop support of the Borland C compiler to build Python. The distutils module still supports it to build extensions. -- bpo-22591: Drop support of MS-DOS, especially of the DJGPP compiler (MS- - DOS port of GCC). +- bpo-22591: Drop support of MS-DOS, especially of the DJGPP compiler + (MS-DOS port of GCC). - bpo-16537: Check whether self.extensions is empty in setup.py. Patch by Jonathan Hosmer. @@ -10008,7 +10331,7 @@ Build - bpo-21811: Anticipated fixes to support OS X versions > 10.9. - bpo-21166: Prevent possible segfaults and other random failures of python - --generate- posix-vars in pybuilddir.txt build target. + ``--generate-posix-vars`` in pybuilddir.txt build target. - bpo-18096: Fix library order returned by python-config. diff --git a/Misc/gdbinit b/Misc/gdbinit index 3b6fe50..bb10bdb 100644 --- a/Misc/gdbinit +++ b/Misc/gdbinit @@ -36,8 +36,8 @@ define pylocals set $_i = 0 while $_i < f->f_code->co_nlocals if f->f_localsplus + $_i != 0 - set $_names = co->co_varnames - set $_name = _PyUnicode_AsString(PyTuple_GetItem($_names, $_i)) + set $_names = f->f_code->co_varnames + set $_name = PyUnicode_AsUTF8(PyTuple_GetItem($_names, $_i)) printf "%s:\n", $_name pyo f->f_localsplus[$_i] end @@ -76,8 +76,8 @@ define pyframev end define pyframe - set $__fn = _PyUnicode_AsString(co->co_filename) - set $__n = _PyUnicode_AsString(co->co_name) + set $__fn = PyUnicode_AsUTF8(f->f_code->co_filename) + set $__n = PyUnicode_AsUTF8(f->f_code->co_name) printf "%s (", $__fn lineno printf "): %s\n", $__n @@ -102,7 +102,7 @@ end #end define printframe - if $pc > PyEval_EvalFrameEx && $pc < PyEval_EvalCodeEx + if $pc > PyEval_EvalFrameEx && $pc < _PyEval_EvalFrameDefault pyframe else frame @@ -129,7 +129,7 @@ end # print the entire Python call stack define pystack while $pc < Py_Main || $pc > Py_GetArgcArgv - if $pc > PyEval_EvalFrameEx && $pc < PyEval_EvalCodeEx + if $pc > PyEval_EvalFrameEx && $pc < _PyEval_EvalFrameDefault pyframe end up-silently 1 @@ -140,7 +140,7 @@ end # print the entire Python call stack - verbose mode define pystackv while $pc < Py_Main || $pc > Py_GetArgcArgv - if $pc > PyEval_EvalFrameEx && $pc < PyEval_EvalCodeEx + if $pc > PyEval_EvalFrameEx && $pc < _PyEval_EvalFrameDefault pyframev end up-silently 1 diff --git a/Misc/python.man b/Misc/python.man index ad3ea6a..b2e9437 100644 --- a/Misc/python.man +++ b/Misc/python.man @@ -445,7 +445,7 @@ Developer resources: https://devguide.python.org/ .br Downloads: https://www.python.org/downloads/ .br -Module repository: https://pypi.python.org/ +Module repository: https://pypi.org/ .br Newsgroups: comp.lang.python, comp.lang.python.announce .SH LICENSING diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index f533871..1430097 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -2248,19 +2248,19 @@ set_exception: } if (res == 1) { /* `result` is a generator */ - PyObject *ret; - ret = task_set_error_soon( + o = task_set_error_soon( task, PyExc_RuntimeError, "yield was used instead of yield from for " - "generator in task %R with %S", task, result); + "generator in task %R with %R", task, result); Py_DECREF(result); - return ret; + return o; } /* The `result` is none of the above */ - Py_DECREF(result); - return task_set_error_soon( + o = task_set_error_soon( task, PyExc_RuntimeError, "Task got bad yield: %R", result); + Py_DECREF(result); + return o; self_await: o = task_set_error_soon( diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index af20d6e..d7b344b 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -574,7 +574,7 @@ deque_concat(dequeobject *deque, PyObject *other) return new_deque; } -static void +static int deque_clear(dequeobject *deque) { block *b; @@ -586,7 +586,7 @@ deque_clear(dequeobject *deque) PyObject **itemptr, **limit; if (Py_SIZE(deque) == 0) - return; + return 0; /* During the process of clearing a deque, decrefs can cause the deque to mutate. To avoid fatal confusion, we have to make the @@ -647,7 +647,7 @@ deque_clear(dequeobject *deque) } CHECK_END(leftblock->rightlink); freeblock(leftblock); - return; + return 0; alternate_method: while (Py_SIZE(deque)) { @@ -655,6 +655,7 @@ deque_clear(dequeobject *deque) assert (item != NULL); Py_DECREF(item); } + return 0; } static PyObject * diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 258e805..69d73f5 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -1392,8 +1392,7 @@ PyCArrayType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) StgDictObject *stgdict; StgDictObject *itemdict; PyObject *length_attr, *type_attr; - long length; - int overflow; + Py_ssize_t length; Py_ssize_t itemsize, itemalign; /* create the new instance (which is a class, @@ -1415,14 +1414,15 @@ PyCArrayType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) Py_XDECREF(length_attr); goto error; } - length = PyLong_AsLongAndOverflow(length_attr, &overflow); - if (overflow) { - PyErr_SetString(PyExc_OverflowError, - "The '_length_' attribute is too large"); - Py_DECREF(length_attr); + length = PyLong_AsSsize_t(length_attr); + Py_DECREF(length_attr); + if (length == -1 && PyErr_Occurred()) { + if (PyErr_ExceptionMatches(PyExc_OverflowError)) { + PyErr_SetString(PyExc_OverflowError, + "The '_length_' attribute is too large"); + } goto error; } - Py_DECREF(length_attr); type_attr = PyObject_GetAttrString((PyObject *)result, "_type_"); if (!type_attr) { @@ -5327,7 +5327,7 @@ cast_check_pointertype(PyObject *arg) if (PyCFuncPtrTypeObject_Check(arg)) return 1; dict = PyType_stgdict(arg); - if (dict) { + if (dict != NULL && dict->proto != NULL) { if (PyUnicode_Check(dict->proto) && (strchr("sPzUZXO", PyUnicode_AsUTF8(dict->proto)[0]))) { /* simple pointer types, c_void_p, c_wchar_p, BSTR, ... */ diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c index 7403e95..148bd80 100644 --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -5201,6 +5201,7 @@ datetime_astimezone(PyDateTime_DateTime *self, PyObject *args, PyObject *kw) return NULL; if (!HASTZINFO(self) || self->tzinfo == Py_None) { + naive: self_tzinfo = local_timezone_from_local(self); if (self_tzinfo == NULL) return NULL; @@ -5221,6 +5222,16 @@ datetime_astimezone(PyDateTime_DateTime *self, PyObject *args, PyObject *kw) Py_DECREF(self_tzinfo); if (offset == NULL) return NULL; + else if(offset == Py_None) { + Py_DECREF(offset); + goto naive; + } + else if (!PyDelta_Check(offset)) { + Py_DECREF(offset); + PyErr_Format(PyExc_TypeError, "utcoffset() returned %.200s," + " expected timedelta or None", Py_TYPE(offset)->tp_name); + return NULL; + } /* result = self - offset */ result = (PyDateTime_DateTime *)add_datetime_timedelta(self, (PyDateTime_Delta *)offset, -1); diff --git a/Modules/_dbmmodule.c b/Modules/_dbmmodule.c index 8429536..2fe1de7 100644 --- a/Modules/_dbmmodule.c +++ b/Modules/_dbmmodule.c @@ -275,7 +275,7 @@ static PySequenceMethods dbm_as_sequence = { _dbm.dbm.get key: str(accept={str, robuffer}, zeroes=True) - default: object(c_default="NULL") = b'' + default: object = None / Return the value for key if present, otherwise default. @@ -284,7 +284,7 @@ Return the value for key if present, otherwise default. static PyObject * _dbm_dbm_get_impl(dbmobject *self, const char *key, Py_ssize_clean_t key_length, PyObject *default_value) -/*[clinic end generated code: output=b44f95eba8203d93 input=a3a279957f85eb6d]*/ +/*[clinic end generated code: output=b44f95eba8203d93 input=b788eba0ffad2e91]*/ /*[clinic end generated code: output=4f5c0e523eaf1251 input=9402c0af8582dc69]*/ { datum dbm_key, val; diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 7e9bb98..792c760 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4061,9 +4061,10 @@ dump(PicklerObject *self, PyObject *obj) } if (save(self, obj, 0) < 0 || - _Pickler_Write(self, &stop_op, 1) < 0) + _Pickler_Write(self, &stop_op, 1) < 0 || + _Pickler_CommitFrame(self) < 0) return -1; - + self->framing = 0; return 0; } diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index 05a08eb..ad934df 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -424,7 +424,7 @@ child_exec(char *const exec_array[], either 0, 1 or 2, it is possible that it is overwritten (#12607). */ if (c2pwrite == 0) POSIX_CALL(c2pwrite = dup(c2pwrite)); - if (errwrite == 0 || errwrite == 1) + while (errwrite == 0 || errwrite == 1) POSIX_CALL(errwrite = dup(errwrite)); /* Dup fds for child. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 1c6aa54..32cd306 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -1125,8 +1125,8 @@ int pysqlite_check_thread(pysqlite_Connection* self) if (self->check_same_thread) { if (PyThread_get_thread_ident() != self->thread_ident) { PyErr_Format(pysqlite_ProgrammingError, - "SQLite objects created in a thread can only be used in that same thread." - "The object was created in thread id %ld and this is thread id %ld", + "SQLite objects created in a thread can only be used in that same thread. " + "The object was created in thread id %ld and this is thread id %ld.", self->thread_ident, PyThread_get_thread_ident()); return 0; } diff --git a/Modules/_ssl.c b/Modules/_ssl.c index c54e43c..5e007da 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -106,6 +106,12 @@ struct py_ssl_library_code { #if (OPENSSL_VERSION_NUMBER >= 0x10100000L) && !defined(LIBRESSL_VERSION_NUMBER) # define OPENSSL_VERSION_1_1 1 +# define PY_OPENSSL_1_1_API 1 +#endif + +/* LibreSSL 2.7.0 provides necessary OpenSSL 1.1.0 APIs */ +#if defined(LIBRESSL_VERSION_NUMBER) && LIBRESSL_VERSION_NUMBER >= 0x2070000fL +# define PY_OPENSSL_1_1_API 1 #endif /* Openssl comes with TLSv1.1 and TLSv1.2 between 1.0.0h and 1.0.1 @@ -152,16 +158,18 @@ struct py_ssl_library_code { #define INVALID_SOCKET (-1) #endif -#ifdef OPENSSL_VERSION_1_1 -/* OpenSSL 1.1.0+ */ -#ifndef OPENSSL_NO_SSL2 -#define OPENSSL_NO_SSL2 -#endif -#else /* OpenSSL < 1.1.0 */ -#if defined(WITH_THREAD) +/* OpenSSL 1.0.2 and LibreSSL needs extra code for locking */ +#if !defined(OPENSSL_VERSION_1_1) && defined(WITH_THREAD) #define HAVE_OPENSSL_CRYPTO_LOCK #endif +#if defined(OPENSSL_VERSION_1_1) && !defined(OPENSSL_NO_SSL2) +#define OPENSSL_NO_SSL2 +#endif + +#ifndef PY_OPENSSL_1_1_API +/* OpenSSL 1.1 API shims for OpenSSL < 1.1.0 and LibreSSL < 2.7.0 */ + #define TLS_method SSLv23_method #define TLS_client_method SSLv23_client_method #define TLS_server_method SSLv23_server_method @@ -227,7 +235,7 @@ SSL_SESSION_get_ticket_lifetime_hint(const SSL_SESSION *s) return s->tlsext_tick_lifetime_hint; } -#endif /* OpenSSL < 1.1.0 or LibreSSL */ +#endif /* OpenSSL < 1.1.0 or LibreSSL < 2.7.0 */ enum py_ssl_error { diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index 4daa34e..43f8034 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -10,6 +10,10 @@ typedef struct { PyObject *x_attr; /* Attributes dictionary */ } ExampleObject; +typedef struct { + PyObject *integer; +} testmultiphase_state; + /* Example methods */ static int @@ -19,11 +23,10 @@ Example_traverse(ExampleObject *self, visitproc visit, void *arg) return 0; } -static int +static void Example_finalize(ExampleObject *self) { Py_CLEAR(self->x_attr); - return 0; } static PyObject * @@ -219,18 +222,21 @@ static int execfunc(PyObject *m) } /* Helper for module definitions; there'll be a lot of them */ -#define TEST_MODULE_DEF(name, slots, methods) { \ + +#define TEST_MODULE_DEF_EX(name, slots, methods, statesize, traversefunc) { \ PyModuleDef_HEAD_INIT, /* m_base */ \ name, /* m_name */ \ PyDoc_STR("Test module " name), /* m_doc */ \ - 0, /* m_size */ \ + statesize, /* m_size */ \ methods, /* m_methods */ \ slots, /* m_slots */ \ - NULL, /* m_traverse */ \ + traversefunc, /* m_traverse */ \ NULL, /* m_clear */ \ NULL, /* m_free */ \ } +#define TEST_MODULE_DEF(name, slots, methods) TEST_MODULE_DEF_EX(name, slots, methods, 0, NULL) + PyModuleDef_Slot main_slots[] = { {Py_mod_exec, execfunc}, {0, NULL}, @@ -614,6 +620,44 @@ PyInit__testmultiphase_exec_unreported_exception(PyObject *spec) return PyModuleDef_Init(&def_exec_unreported_exception); } +static int +bad_traverse(PyObject *self, visitproc visit, void *arg) { + testmultiphase_state *m_state; + + m_state = PyModule_GetState(self); + Py_VISIT(m_state->integer); + return 0; +} + +static int +execfunc_with_bad_traverse(PyObject *mod) { + testmultiphase_state *m_state; + + m_state = PyModule_GetState(mod); + if (m_state == NULL) { + return -1; + } + + m_state->integer = PyLong_FromLong(0x7fffffff); + Py_INCREF(m_state->integer); + + return 0; +} + +static PyModuleDef_Slot slots_with_bad_traverse[] = { + {Py_mod_exec, execfunc_with_bad_traverse}, + {0, NULL} +}; + +static PyModuleDef def_with_bad_traverse = TEST_MODULE_DEF_EX( + "_testmultiphase_with_bad_traverse", slots_with_bad_traverse, NULL, + sizeof(testmultiphase_state), bad_traverse); + +PyMODINIT_FUNC +PyInit__testmultiphase_with_bad_traverse(PyObject *spec) { + return PyModuleDef_Init(&def_with_bad_traverse); +} + /*** Helper for imp test ***/ static PyModuleDef imp_dummy_def = TEST_MODULE_DEF("imp_dummy", main_slots, testexport_methods); @@ -623,3 +667,4 @@ PyInit_imp_dummy(PyObject *spec) { return PyModuleDef_Init(&imp_dummy_def); } + diff --git a/Modules/audioop.c b/Modules/audioop.c index d582042..4aad2e5 100644 --- a/Modules/audioop.c +++ b/Modules/audioop.c @@ -20,10 +20,17 @@ static const unsigned int masks[] = {0, 0xFF, 0xFFFF, 0xFFFFFF, 0xFFFFFFFF}; static int fbound(double val, double minval, double maxval) { - if (val > maxval) + if (val > maxval) { val = maxval; - else if (val < minval + 1) + } + else if (val < minval + 1.0) { val = minval; + } + + /* Round towards minus infinity (-inf) */ + val = floor(val); + + /* Cast double to integer: round towards zero */ return (int)val; } @@ -924,9 +931,8 @@ audioop_mul_impl(PyObject *module, Py_buffer *fragment, int width, for (i = 0; i < fragment->len; i += width) { double val = GETRAWSAMPLE(width, fragment->buf, i); - val *= factor; - val = floor(fbound(val, minval, maxval)); - SETRAWSAMPLE(width, ncp, i, (int)val); + int ival = fbound(val * factor, minval, maxval); + SETRAWSAMPLE(width, ncp, i, ival); } return rv; } @@ -973,9 +979,9 @@ audioop_tomono_impl(PyObject *module, Py_buffer *fragment, int width, for (i = 0; i < len; i += width*2) { double val1 = GETRAWSAMPLE(width, cp, i); double val2 = GETRAWSAMPLE(width, cp, i + width); - double val = val1*lfactor + val2*rfactor; - val = floor(fbound(val, minval, maxval)); - SETRAWSAMPLE(width, ncp, i/2, val); + double val = val1 * lfactor + val2 * rfactor; + int ival = fbound(val, minval, maxval); + SETRAWSAMPLE(width, ncp, i/2, ival); } return rv; } @@ -1021,8 +1027,8 @@ audioop_tostereo_impl(PyObject *module, Py_buffer *fragment, int width, for (i = 0; i < fragment->len; i += width) { double val = GETRAWSAMPLE(width, fragment->buf, i); - int val1 = (int)floor(fbound(val*lfactor, minval, maxval)); - int val2 = (int)floor(fbound(val*rfactor, minval, maxval)); + int val1 = fbound(val * lfactor, minval, maxval); + int val2 = fbound(val * rfactor, minval, maxval); SETRAWSAMPLE(width, ncp, i*2, val1); SETRAWSAMPLE(width, ncp, i*2 + width, val2); } @@ -1080,7 +1086,7 @@ audioop_add_impl(PyObject *module, Py_buffer *fragment1, else { double fval = (double)val1 + (double)val2; /* truncate in case of overflow */ - newval = (int)floor(fbound(fval, minval, maxval)); + newval = fbound(fval, minval, maxval); } SETRAWSAMPLE(width, ncp, i, newval); diff --git a/Modules/clinic/_dbmmodule.c.h b/Modules/clinic/_dbmmodule.c.h index 7dd529a..9a35311 100644 --- a/Modules/clinic/_dbmmodule.c.h +++ b/Modules/clinic/_dbmmodule.c.h @@ -39,7 +39,7 @@ _dbm_dbm_keys(dbmobject *self, PyObject *Py_UNUSED(ignored)) } PyDoc_STRVAR(_dbm_dbm_get__doc__, -"get($self, key, default=b\'\', /)\n" +"get($self, key, default=None, /)\n" "--\n" "\n" "Return the value for key if present, otherwise default."); @@ -57,7 +57,7 @@ _dbm_dbm_get(dbmobject *self, PyObject *args) PyObject *return_value = NULL; const char *key; Py_ssize_clean_t key_length; - PyObject *default_value = NULL; + PyObject *default_value = Py_None; if (!PyArg_ParseTuple(args, "s#|O:get", &key, &key_length, &default_value)) { @@ -141,4 +141,4 @@ dbmopen(PyObject *module, PyObject *args) exit: return return_value; } -/*[clinic end generated code: output=001fabffcecb99f1 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=919cc4337be4a5d3 input=a9049054013a1b77]*/ diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c index 754348e..3bddc40 100644 --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -751,10 +751,8 @@ debug_cycle(const char *msg, PyObject *op) * garbage list (a Python list), else only the objects in finalizers with * __del__ methods are appended to garbage. All objects in finalizers are * merged into the old list regardless. - * Returns 0 if all OK, <0 on error (out of memory to grow the garbage list). - * The finalizers list is made empty on a successful return. */ -static int +static void handle_legacy_finalizers(PyGC_Head *finalizers, PyGC_Head *old) { PyGC_Head *gc = finalizers->gc.gc_next; @@ -769,12 +767,11 @@ handle_legacy_finalizers(PyGC_Head *finalizers, PyGC_Head *old) if ((debug & DEBUG_SAVEALL) || has_legacy_finalizer(op)) { if (PyList_Append(garbage, op) < 0) - return -1; + break; } } gc_list_merge(finalizers, old); - return 0; } /* Run first-time finalizers (if any) on all the objects in collectable. @@ -1045,7 +1042,7 @@ collect(int generation, Py_ssize_t *n_collected, Py_ssize_t *n_uncollectable, * reachable list of garbage. The programmer has to deal with * this if they insist on creating this type of structure. */ - (void)handle_legacy_finalizers(&finalizers, old); + handle_legacy_finalizers(&finalizers, old); /* Clear free list only during the collection of the highest * generation */ @@ -1109,9 +1106,12 @@ invoke_gc_callback(const char *phase, int generation, PyObject *r, *cb = PyList_GET_ITEM(callbacks, i); Py_INCREF(cb); /* make sure cb doesn't go away */ r = PyObject_CallFunction(cb, "sO", phase, info); - Py_XDECREF(r); - if (r == NULL) + if (r == NULL) { PyErr_WriteUnraisable(cb); + } + else { + Py_DECREF(r); + } Py_DECREF(cb); } Py_XDECREF(info); @@ -1588,8 +1588,11 @@ PyGC_Collect(void) if (collecting) n = 0; /* already collecting, don't do anything */ else { + PyObject *exc, *value, *tb; collecting = 1; + PyErr_Fetch(&exc, &value, &tb); n = collect_with_callback(NUM_GENERATIONS - 1); + PyErr_Restore(exc, value, tb); collecting = 0; } @@ -1773,6 +1776,7 @@ _PyObject_GC_Resize(PyVarObject *op, Py_ssize_t nitems) { const size_t basicsize = _PyObject_VAR_SIZE(Py_TYPE(op), nitems); PyGC_Head *g = AS_GC(op); + assert(!IS_TRACKED(op)); if (basicsize > PY_SSIZE_T_MAX - sizeof(PyGC_Head)) return (PyVarObject *)PyErr_NoMemory(); g = (PyGC_Head *)PyObject_REALLOC(g, sizeof(PyGC_Head) + basicsize); diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index 426b7ca..ebef501 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -840,24 +840,6 @@ mmap_subscript(mmap_object *self, PyObject *item) } } -static PyObject * -mmap_concat(mmap_object *self, PyObject *bb) -{ - CHECK_VALID(NULL); - PyErr_SetString(PyExc_SystemError, - "mmaps don't support concatenation"); - return NULL; -} - -static PyObject * -mmap_repeat(mmap_object *self, Py_ssize_t n) -{ - CHECK_VALID(NULL); - PyErr_SetString(PyExc_SystemError, - "mmaps don't support repeat operation"); - return NULL; -} - static int mmap_ass_item(mmap_object *self, Py_ssize_t i, PyObject *v) { @@ -977,8 +959,8 @@ mmap_ass_subscript(mmap_object *self, PyObject *item, PyObject *value) static PySequenceMethods mmap_as_sequence = { (lenfunc)mmap_length, /*sq_length*/ - (binaryfunc)mmap_concat, /*sq_concat*/ - (ssizeargfunc)mmap_repeat, /*sq_repeat*/ + 0, /*sq_concat*/ + 0, /*sq_repeat*/ (ssizeargfunc)mmap_item, /*sq_item*/ 0, /*sq_slice*/ (ssizeobjargproc)mmap_ass_item, /*sq_ass_item*/ @@ -1078,6 +1060,7 @@ static PyObject * new_mmap_object(PyTypeObject *type, PyObject *args, PyObject *kwdict) { struct _Py_stat_struct status; + int fstat_result = -1; mmap_object *m_obj; Py_ssize_t map_size; off_t offset = 0; @@ -1094,7 +1077,7 @@ new_mmap_object(PyTypeObject *type, PyObject *args, PyObject *kwdict) return NULL; if (map_size < 0) { PyErr_SetString(PyExc_OverflowError, - "memory mapped length must be postiive"); + "memory mapped length must be positive"); return NULL; } if (offset < 0) { @@ -1143,8 +1126,14 @@ new_mmap_object(PyTypeObject *type, PyObject *args, PyObject *kwdict) if (fd != -1) (void)fcntl(fd, F_FULLFSYNC); #endif - if (fd != -1 && _Py_fstat_noraise(fd, &status) == 0 - && S_ISREG(status.st_mode)) { + + if (fd != -1) { + Py_BEGIN_ALLOW_THREADS + fstat_result = _Py_fstat_noraise(fd, &status); + Py_END_ALLOW_THREADS + } + + if (fd != -1 && fstat_result == 0 && S_ISREG(status.st_mode)) { if (map_size == 0) { if (status.st_size == 0) { PyErr_SetString(PyExc_ValueError, @@ -1274,7 +1263,7 @@ new_mmap_object(PyTypeObject *type, PyObject *args, PyObject *kwdict) if (map_size < 0) { PyErr_SetString(PyExc_OverflowError, - "memory mapped length must be postiive"); + "memory mapped length must be positive"); return NULL; } if (offset < 0) { diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c index a444335..710efc2 100644 --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -136,18 +136,18 @@ node2tuple(node *n, /* node to convert */ goto error; (void) addelem(result, 1, w); - if (lineno == 1) { + if (lineno) { w = PyLong_FromLong(n->n_lineno); if (w == NULL) goto error; (void) addelem(result, 2, w); } - if (col_offset == 1) { + if (col_offset) { w = PyLong_FromLong(n->n_col_offset); if (w == NULL) goto error; - (void) addelem(result, 3, w); + (void) addelem(result, 2 + lineno, w); } } else { diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index 0b9e8f1..a93dc64 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -119,13 +119,6 @@ static PyObject *DefaultHandler; static PyObject *IgnoreHandler; static PyObject *IntHandler; -/* On Solaris 8, gcc will produce a warning that the function - declaration is not a prototype. This is caused by the definition of - SIG_DFL as (void (*)())0; the correct declaration would have been - (void (*)(int))0. */ - -static PyOS_sighandler_t old_siginthandler = SIG_DFL; - #ifdef MS_WINDOWS static HANDLE sigint_event = NULL; #endif @@ -759,7 +752,6 @@ iterable_to_sigset(PyObject *iterable, sigset_t *mask) int result = -1; PyObject *iterator, *item; long signum; - int err; sigemptyset(mask); @@ -781,11 +773,14 @@ iterable_to_sigset(PyObject *iterable, sigset_t *mask) Py_DECREF(item); if (signum == -1 && PyErr_Occurred()) goto error; - if (0 < signum && signum < NSIG) - err = sigaddset(mask, (int)signum); - else - err = 1; - if (err) { + if (0 < signum && signum < NSIG) { + /* bpo-33329: ignore sigaddset() return value as it can fail + * for some reserved signals, but we want the `range(1, NSIG)` + * idiom to allow selecting all valid signals. + */ + (void) sigaddset(mask, (int)signum); + } + else { PyErr_Format(PyExc_ValueError, "signal number %ld out of range", signum); goto error; @@ -1286,7 +1281,7 @@ PyInit__signal(void) /* Install default int handler */ Py_INCREF(IntHandler); Py_SETREF(Handlers[SIGINT].func, IntHandler); - old_siginthandler = PyOS_setsig(SIGINT, signal_handler); + PyOS_setsig(SIGINT, signal_handler); } #ifdef SIGHUP @@ -1492,14 +1487,11 @@ finisignal(void) int i; PyObject *func; - PyOS_setsig(SIGINT, old_siginthandler); - old_siginthandler = SIG_DFL; - for (i = 1; i < NSIG; i++) { func = Handlers[i].func; _Py_atomic_store_relaxed(&Handlers[i].tripped, 0); Handlers[i].func = NULL; - if (i != SIGINT && func != NULL && func != Py_None && + if (func != NULL && func != Py_None && func != DefaultHandler && func != IgnoreHandler) PyOS_setsig(i, SIG_DFL); Py_XDECREF(func); diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 36920f7..ff73a3f 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -312,10 +312,8 @@ http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/net/getaddrinfo.c.diff?r1=1.82& # include # endif -#if defined(_MSC_VER) && _MSC_VER >= 1800 /* Provides the IsWindows7SP1OrGreater() function */ #include -#endif /* remove some flags on older version Windows during run-time. https://msdn.microsoft.com/en-us/library/windows/desktop/ms738596.aspx */ @@ -6571,15 +6569,7 @@ PyInit__socket(void) #ifdef MS_WINDOWS if (support_wsa_no_inherit == -1) { -#if defined(_MSC_VER) && _MSC_VER >= 1800 support_wsa_no_inherit = IsWindows7SP1OrGreater(); -#else - DWORD version = GetVersion(); - DWORD major = (DWORD)LOBYTE(LOWORD(version)); - DWORD minor = (DWORD)HIBYTE(LOWORD(version)); - /* need Windows 7 SP1, 2008 R2 SP1 or later */ - support_wsa_no_inherit = major > 6 || (major == 6 && minor >= 1); -#endif } #endif diff --git a/Modules/xxlimited.c b/Modules/xxlimited.c index 5041ac8..5586989 100644 --- a/Modules/xxlimited.c +++ b/Modules/xxlimited.c @@ -47,11 +47,10 @@ Xxo_traverse(XxoObject *self, visitproc visit, void *arg) return 0; } -static int +static void Xxo_finalize(XxoObject *self) { Py_CLEAR(self->x_attr); - return 0; } static PyObject * diff --git a/Modules/xxsubtype.c b/Modules/xxsubtype.c index 8d0d6ae..11242d7 100644 --- a/Modules/xxsubtype.c +++ b/Modules/xxsubtype.c @@ -239,7 +239,7 @@ spam_bench(PyObject *self, PyObject *args) int n = 1000; time_t t0, t1; - if (!PyArg_ParseTuple(args, "OS|i", &obj, &name, &n)) + if (!PyArg_ParseTuple(args, "OU|i", &obj, &name, &n)) return NULL; t0 = clock(); while (--n >= 0) { diff --git a/Objects/dictobject.c b/Objects/dictobject.c index 8862be8..ddd0582 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -2654,6 +2654,7 @@ PyDict_Copy(PyObject *o) split_copy->ma_values = newvalues; split_copy->ma_keys = mp->ma_keys; split_copy->ma_used = mp->ma_used; + split_copy->ma_version_tag = DICT_NEXT_VERSION(); DK_INCREF(mp->ma_keys); for (i = 0, n = size; i < n; i++) { PyObject *value = mp->ma_values[i]; diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 1f134aa..5540679 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -1,4 +1,3 @@ - /* Float object implementation */ /* XXX There should be overflow checks here, but it's hard to check @@ -48,7 +47,7 @@ static PyStructSequence_Field floatinfo_fields[] = { "is representable"}, {"max_10_exp", "DBL_MAX_10_EXP -- maximum int e such that 10**e " "is representable"}, - {"min", "DBL_MIN -- Minimum positive normalizer float"}, + {"min", "DBL_MIN -- Minimum positive normalized float"}, {"min_exp", "DBL_MIN_EXP -- minimum int e such that radix**(e-1) " "is a normalized float"}, {"min_10_exp", "DBL_MIN_10_EXP -- minimum int e such that 10**e is " @@ -58,7 +57,7 @@ static PyStructSequence_Field floatinfo_fields[] = { {"epsilon", "DBL_EPSILON -- Difference between 1 and the next " "representable float"}, {"radix", "FLT_RADIX -- radix of exponent"}, - {"rounds", "FLT_ROUNDS -- addition rounds"}, + {"rounds", "FLT_ROUNDS -- rounding mode"}, {0} }; diff --git a/Objects/genobject.c b/Objects/genobject.c index 1c29e29..f226dbe 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -1939,21 +1939,20 @@ yield_close: return NULL; check_error: - if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration)) { + if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration) || + PyErr_ExceptionMatches(PyExc_GeneratorExit)) + { o->agt_state = AWAITABLE_STATE_CLOSED; if (o->agt_args == NULL) { /* when aclose() is called we don't want to propagate - StopAsyncIteration; just raise StopIteration, signalling - that 'aclose()' is done. */ + StopAsyncIteration or GeneratorExit; just raise + StopIteration, signalling that this 'aclose()' await + is done. + */ PyErr_Clear(); PyErr_SetNone(PyExc_StopIteration); } } - else if (PyErr_ExceptionMatches(PyExc_GeneratorExit)) { - o->agt_state = AWAITABLE_STATE_CLOSED; - PyErr_Clear(); /* ignore these errors */ - PyErr_SetNone(PyExc_StopIteration); - } return NULL; } diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index f357af2..c597932 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -20,6 +20,17 @@ static PyMemberDef module_members[] = { {0} }; + +/* Helper for sanity check for traverse not handling m_state == NULL + * Issue #32374 */ +#ifdef Py_DEBUG +static int +bad_traverse_test(PyObject *self, void *arg) { + assert(self != NULL); + return 0; +} +#endif + PyTypeObject PyModuleDef_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "moduledef", /* tp_name */ @@ -338,6 +349,16 @@ PyModule_FromDefAndSpec2(struct PyModuleDef* def, PyObject *spec, int module_api } } + /* Sanity check for traverse not handling m_state == NULL + * This doesn't catch all possible cases, but in many cases it should + * make many cases of invalid code crash or raise Valgrind issues + * sooner than they would otherwise. + * Issue #32374 */ +#ifdef Py_DEBUG + if (def->m_traverse != NULL) { + def->m_traverse(m, bad_traverse_test, NULL); + } +#endif Py_DECREF(nameobj); return m; diff --git a/Objects/setobject.c b/Objects/setobject.c index c742041..96485f8 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1740,8 +1740,10 @@ set_symmetric_difference(PySetObject *so, PyObject *other) if (otherset == NULL) return NULL; rv = set_symmetric_difference_update(otherset, (PyObject *)so); - if (rv == NULL) + if (rv == NULL) { + Py_DECREF(otherset); return NULL; + } Py_DECREF(rv); return (PyObject *)otherset; } diff --git a/Objects/typeobject.c b/Objects/typeobject.c index e843204..69b1878 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -6740,9 +6740,9 @@ static slotdef slotdefs[] = { SQSLOT("__add__", sq_concat, NULL, wrap_binaryfunc, "__add__($self, value, /)\n--\n\nReturn self+value."), SQSLOT("__mul__", sq_repeat, NULL, wrap_indexargfunc, - "__mul__($self, value, /)\n--\n\nReturn self*value.n"), + "__mul__($self, value, /)\n--\n\nReturn self*value."), SQSLOT("__rmul__", sq_repeat, NULL, wrap_indexargfunc, - "__rmul__($self, value, /)\n--\n\nReturn self*value."), + "__rmul__($self, value, /)\n--\n\nReturn value*self."), SQSLOT("__getitem__", sq_item, slot_sq_item, wrap_sq_item, "__getitem__($self, key, /)\n--\n\nReturn self[key]."), SQSLOT("__setitem__", sq_ass_item, slot_sq_ass_item, wrap_sq_setitem, diff --git a/PCbuild/find_msbuild.bat b/PCbuild/find_msbuild.bat index 4676395..8abad58 100644 --- a/PCbuild/find_msbuild.bat +++ b/PCbuild/find_msbuild.bat @@ -48,5 +48,14 @@ @exit /b 1 :found -@echo Using %MSBUILD% (found in the %_Py_MSBuild_Source%) +@pushd %MSBUILD% >nul 2>nul +@if not ERRORLEVEL 1 @( + @if exist msbuild.exe @(set MSBUILD="%CD%\msbuild.exe") else @(set MSBUILD=) + @popd +) + +@if defined MSBUILD @echo Using %MSBUILD% (found in the %_Py_MSBuild_Source%) +@if not defined MSBUILD @echo Failed to find MSBuild @set _Py_MSBuild_Source= +@if not defined MSBUILD @exit /b 1 +@exit /b 0 diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index dab9464..4e5cbc6 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -41,7 +41,7 @@ echo.Fetching external libraries... set libraries= set libraries=%libraries% bzip2-1.0.6 -if NOT "%IncludeSSL%"=="false" set libraries=%libraries% openssl-1.0.2k +if NOT "%IncludeSSL%"=="false" set libraries=%libraries% openssl-1.0.2o set libraries=%libraries% sqlite-3.21.0.0 if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tcl-core-8.6.6.0 if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tk-8.6.6.0 diff --git a/PCbuild/python.props b/PCbuild/python.props index cc6f6a4..a146e65 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -49,7 +49,7 @@ $(ExternalsDir)sqlite-3.21.0.0\ $(ExternalsDir)bzip2-1.0.6\ $(ExternalsDir)xz-5.2.2\ - $(ExternalsDir)openssl-1.0.2k\ + $(ExternalsDir)openssl-1.0.2o\ $(opensslDir)include32 $(opensslDir)include64 $(ExternalsDir)\nasm-2.11.06\ diff --git a/PCbuild/python3dll.vcxproj b/PCbuild/python3dll.vcxproj index 125e071..a6b9f38 100644 --- a/PCbuild/python3dll.vcxproj +++ b/PCbuild/python3dll.vcxproj @@ -89,14 +89,18 @@ - + <_DefLines Remove="@(_DefLines)" /> <_Lines Remove="@(_Lines)" /> + <_OriginalLines Remove="@(_OriginalLines)" /> + + + <_Pattern1>(=python$(MajorVersionNumber)$(MinorVersionNumber))\. <_Sub1>$1_d. @@ -109,17 +113,23 @@ - + + - + <_DefLines Remove="@(_DefLines)" /> <_Lines Remove="@(_Lines)" /> + <_OriginalLines Remove="@(_OriginalLines)" /> + + + <_Pattern>^[\w.]+=.+?\.([^ ]+).*$ <_Sub>$1 @@ -132,6 +142,8 @@ <_Lines Include="@(_Symbols->'%(Symbol)')" /> - + + \ No newline at end of file diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt index 5fed7aa..fa53e81 100644 --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -177,7 +177,7 @@ _lzma Homepage: http://tukaani.org/xz/ _ssl - Python wrapper for version 1.0.2k of the OpenSSL secure sockets + Python wrapper for version 1.0.2o of the OpenSSL secure sockets library, which is built by ssl.vcxproj Homepage: http://www.openssl.org/ diff --git a/PCbuild/rt.bat b/PCbuild/rt.bat index e73ac04..be3513f 100644 --- a/PCbuild/rt.bat +++ b/PCbuild/rt.bat @@ -7,7 +7,7 @@ rem -q "quick" -- normally the tests are run twice, the first time rem after deleting all the .pyc files reachable from Lib/. rem -q runs the tests just once, and without deleting .pyc files. rem -x64 Run the 64-bit build of python (or python_d if -d was specified) -rem from the 'amd64' dir instead of the 32-bit build in this dir. +rem When omitted, uses %PREFIX% if set or the 32-bit build rem All leading instances of these switches are shifted off, and rem whatever remains (up to 9 arguments) is passed to regrtest.py. rem For example, @@ -28,28 +28,29 @@ rem rt -u "network,largefile" setlocal set pcbuild=%~dp0 -set prefix=%pcbuild%win32\ set suffix= set qmode= set dashO= set regrtestargs= +set exe= :CheckOpts if "%1"=="-O" (set dashO=-O) & shift & goto CheckOpts if "%1"=="-q" (set qmode=yes) & shift & goto CheckOpts if "%1"=="-d" (set suffix=_d) & shift & goto CheckOpts -if "%1"=="-x64" (set prefix=%pcbuild%amd64\) & shift & goto CheckOpts +if "%1"=="-x64" (set prefix=%pcbuild%amd64) & shift & goto CheckOpts if NOT "%1"=="" (set regrtestargs=%regrtestargs% %1) & shift & goto CheckOpts -set exe=%prefix%python%suffix%.exe -set cmd="%exe%" %dashO% -Wd -E -bb -m test %regrtestargs% +if not defined prefix set prefix=%pcbuild%win32 +set exe=%prefix%\python%suffix%.exe +set cmd="%exe%" %dashO% -u -Wd -E -bb -m test %regrtestargs% if defined qmode goto Qmode echo Deleting .pyc files ... "%exe%" "%pcbuild%rmpyc.py" echo Cleaning _pth files ... -if exist %prefix%*._pth del %prefix%*._pth +if exist %prefix%\*._pth del %prefix%\*._pth echo on %cmd% diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py index 6302af6..90c5fea 100644 --- a/Parser/asdl_c.py +++ b/Parser/asdl_c.py @@ -643,10 +643,11 @@ ast_traverse(AST_object *self, visitproc visit, void *arg) return 0; } -static void +static int ast_clear(AST_object *self) { Py_CLEAR(self->dict); + return 0; } static int diff --git a/Python/Python-ast.c b/Python/Python-ast.c index 212211c..8e383ad 100644 --- a/Python/Python-ast.c +++ b/Python/Python-ast.c @@ -525,10 +525,11 @@ ast_traverse(AST_object *self, visitproc visit, void *arg) return 0; } -static void +static int ast_clear(AST_object *self) { Py_CLEAR(self->dict); + return 0; } static int diff --git a/Python/_warnings.c b/Python/_warnings.c index 0077b9b..e96e154 100644 --- a/Python/_warnings.c +++ b/Python/_warnings.c @@ -63,6 +63,13 @@ get_warnings_attr(const char *attr, int try_import) } } else { + /* if we're so late into Python finalization that the module dict is + gone, then we can't even use PyImport_GetModule without triggering + an interpreter abort. + */ + if (!PyThreadState_GET()->interp->modules) { + return NULL; + } all_modules = PyImport_GetModuleDict(); warnings_module = PyDict_GetItem(all_modules, warnings_str); diff --git a/Python/ceval.c b/Python/ceval.c index 9ad582b..646f6a5 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -66,6 +66,7 @@ static PyObject * unicode_concatenate(PyObject *, PyObject *, static PyObject * special_lookup(PyObject *, _Py_Identifier *); static int check_args_iterable(PyObject *func, PyObject *vararg); static void format_kwargs_mapping_error(PyObject *func, PyObject *kwargs); +static void format_awaitable_error(PyTypeObject *, int); #define NAME_ERROR_MSG \ "name '%.200s' is not defined" @@ -2040,6 +2041,11 @@ _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag) PyObject *iterable = TOP(); PyObject *iter = _PyCoro_GetAwaitableIter(iterable); + if (iter == NULL) { + format_awaitable_error(Py_TYPE(iterable), + _Py_OPCODE(next_instr[-2])); + } + Py_DECREF(iterable); if (iter != NULL && PyCoro_CheckExact(iter)) { @@ -3882,7 +3888,7 @@ too_many_positional(PyCodeObject *co, Py_ssize_t given, Py_ssize_t defcount, } /* This is gonna seem *real weird*, but if you put some other code between - PyEval_EvalFrame() and PyEval_EvalCodeEx() you will need to adjust + PyEval_EvalFrame() and _PyEval_EvalFrameDefault() you will need to adjust the test in the if statements in Misc/gdbinit (pystack and pystackv). */ static PyObject * @@ -5403,6 +5409,25 @@ format_exc_unbound(PyCodeObject *co, int oparg) } } +static void +format_awaitable_error(PyTypeObject *type, int prevopcode) +{ + if (type->tp_as_async == NULL || type->tp_as_async->am_await == NULL) { + if (prevopcode == BEFORE_ASYNC_WITH) { + PyErr_Format(PyExc_TypeError, + "'async with' received an object from __aenter__ " + "that does not implement __await__: %.100s", + type->tp_name); + } + else if (prevopcode == WITH_CLEANUP_START) { + PyErr_Format(PyExc_TypeError, + "'async with' received an object from __aexit__ " + "that does not implement __await__: %.100s", + type->tp_name); + } + } +} + static PyObject * unicode_concatenate(PyObject *v, PyObject *w, PyFrameObject *f, const _Py_CODEUNIT *next_instr) diff --git a/Python/codecs.c b/Python/codecs.c index fe57d0d..4ff8301 100644 --- a/Python/codecs.c +++ b/Python/codecs.c @@ -77,8 +77,6 @@ PyObject *normalizestring(const char *string) } p[i] = '\0'; v = PyUnicode_FromString(p); - if (v == NULL) - return NULL; PyMem_Free(p); return v; } diff --git a/Python/compile.c b/Python/compile.c index aae3300..c1df7e8 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -2237,23 +2237,19 @@ compiler_async_for(struct compiler *c, stmt_ty s) ADDOP(c, DUP_TOP); ADDOP_O(c, LOAD_GLOBAL, stop_aiter_error, names); ADDOP_I(c, COMPARE_OP, PyCmp_EXC_MATCH); - ADDOP_JABS(c, POP_JUMP_IF_FALSE, try_cleanup); - - ADDOP(c, POP_TOP); - ADDOP(c, POP_TOP); - ADDOP(c, POP_TOP); - ADDOP(c, POP_EXCEPT); /* for SETUP_EXCEPT */ - ADDOP(c, POP_BLOCK); /* for SETUP_LOOP */ - ADDOP_JABS(c, JUMP_ABSOLUTE, after_loop_else); - - - compiler_use_next_block(c, try_cleanup); + ADDOP_JABS(c, POP_JUMP_IF_TRUE, try_cleanup); ADDOP(c, END_FINALLY); compiler_use_next_block(c, after_try); VISIT_SEQ(c, stmt, s->v.AsyncFor.body); ADDOP_JABS(c, JUMP_ABSOLUTE, try); + compiler_use_next_block(c, try_cleanup); + ADDOP(c, POP_TOP); + ADDOP(c, POP_TOP); + ADDOP(c, POP_TOP); + ADDOP(c, POP_EXCEPT); /* for SETUP_EXCEPT */ + ADDOP(c, POP_TOP); /* for correct calculation of stack effect */ ADDOP(c, POP_BLOCK); /* for SETUP_LOOP */ compiler_pop_fblock(c, LOOP, try); @@ -2603,8 +2599,7 @@ compiler_import_as(struct compiler *c, identifier name, identifier asname) PyUnicode_GET_LENGTH(name)); if (!attr) return 0; - ADDOP_O(c, LOAD_ATTR, attr, names); - Py_DECREF(attr); + ADDOP_N(c, LOAD_ATTR, attr, names); pos = dot + 1; } } @@ -2632,8 +2627,7 @@ compiler_import(struct compiler *c, stmt_ty s) if (level == NULL) return 0; - ADDOP_O(c, LOAD_CONST, level, consts); - Py_DECREF(level); + ADDOP_N(c, LOAD_CONST, level, consts); ADDOP_O(c, LOAD_CONST, Py_None, consts); ADDOP_NAME(c, IMPORT_NAME, alias->name, names); @@ -2666,9 +2660,7 @@ static int compiler_from_import(struct compiler *c, stmt_ty s) { Py_ssize_t i, n = asdl_seq_LEN(s->v.ImportFrom.names); - - PyObject *names = PyTuple_New(n); - PyObject *level; + PyObject *level, *names; static PyObject *empty_string; if (!empty_string) { @@ -2677,14 +2669,15 @@ compiler_from_import(struct compiler *c, stmt_ty s) return 0; } - if (!names) - return 0; - level = PyLong_FromLong(s->v.ImportFrom.level); if (!level) { - Py_DECREF(names); return 0; } + ADDOP_N(c, LOAD_CONST, level, consts); + + names = PyTuple_New(n); + if (!names) + return 0; /* build up the names */ for (i = 0; i < n; i++) { @@ -2695,16 +2688,12 @@ compiler_from_import(struct compiler *c, stmt_ty s) if (s->lineno > c->c_future->ff_lineno && s->v.ImportFrom.module && _PyUnicode_EqualToASCIIString(s->v.ImportFrom.module, "__future__")) { - Py_DECREF(level); Py_DECREF(names); return compiler_error(c, "from __future__ imports must occur " "at the beginning of the file"); } + ADDOP_N(c, LOAD_CONST, names, consts); - ADDOP_O(c, LOAD_CONST, level, consts); - Py_DECREF(level); - ADDOP_O(c, LOAD_CONST, names, consts); - Py_DECREF(names); if (s->v.ImportFrom.module) { ADDOP_NAME(c, IMPORT_NAME, s->v.ImportFrom.module, names); } @@ -2727,7 +2716,6 @@ compiler_from_import(struct compiler *c, stmt_ty s) store_name = alias->asname; if (!compiler_nameop(c, store_name, Store)) { - Py_DECREF(names); return 0; } } @@ -3105,8 +3093,7 @@ compiler_nameop(struct compiler *c, identifier name, expr_context_ty ctx) "param invalid for local variable"); return 0; } - ADDOP_O(c, op, mangled, varnames); - Py_DECREF(mangled); + ADDOP_N(c, op, mangled, varnames); return 1; case OP_GLOBAL: switch (ctx) { @@ -3750,7 +3737,7 @@ compiler_async_comprehension_generator(struct compiler *c, _Py_IDENTIFIER(StopAsyncIteration); comprehension_ty gen; - basicblock *anchor, *if_cleanup, *try, + basicblock *if_cleanup, *try, *after_try, *except, *try_cleanup; Py_ssize_t i, n; @@ -3761,12 +3748,11 @@ compiler_async_comprehension_generator(struct compiler *c, try = compiler_new_block(c); after_try = compiler_new_block(c); - try_cleanup = compiler_new_block(c); except = compiler_new_block(c); if_cleanup = compiler_new_block(c); - anchor = compiler_new_block(c); + try_cleanup = compiler_new_block(c); - if (if_cleanup == NULL || anchor == NULL || + if (if_cleanup == NULL || try == NULL || after_try == NULL || except == NULL || try_cleanup == NULL) { return 0; @@ -3807,16 +3793,7 @@ compiler_async_comprehension_generator(struct compiler *c, ADDOP(c, DUP_TOP); ADDOP_O(c, LOAD_GLOBAL, stop_aiter_error, names); ADDOP_I(c, COMPARE_OP, PyCmp_EXC_MATCH); - ADDOP_JABS(c, POP_JUMP_IF_FALSE, try_cleanup); - - ADDOP(c, POP_TOP); - ADDOP(c, POP_TOP); - ADDOP(c, POP_TOP); - ADDOP(c, POP_EXCEPT); /* for SETUP_EXCEPT */ - ADDOP_JABS(c, JUMP_ABSOLUTE, anchor); - - - compiler_use_next_block(c, try_cleanup); + ADDOP_JABS(c, POP_JUMP_IF_TRUE, try_cleanup); ADDOP(c, END_FINALLY); compiler_use_next_block(c, after_try); @@ -3865,7 +3842,12 @@ compiler_async_comprehension_generator(struct compiler *c, } compiler_use_next_block(c, if_cleanup); ADDOP_JABS(c, JUMP_ABSOLUTE, try); - compiler_use_next_block(c, anchor); + + compiler_use_next_block(c, try_cleanup); + ADDOP(c, POP_TOP); + ADDOP(c, POP_TOP); + ADDOP(c, POP_TOP); + ADDOP(c, POP_EXCEPT); /* for SETUP_EXCEPT */ ADDOP(c, POP_TOP); return 1; @@ -4561,11 +4543,11 @@ compiler_annassign(struct compiler *c, stmt_ty s) if (s->v.AnnAssign.simple && (c->u->u_scope_type == COMPILER_SCOPE_MODULE || c->u->u_scope_type == COMPILER_SCOPE_CLASS)) { + VISIT(c, expr, s->v.AnnAssign.annotation); mangled = _Py_Mangle(c->u->u_private, targ->v.Name.id); if (!mangled) { return 0; } - VISIT(c, expr, s->v.AnnAssign.annotation); /* ADDOP_N decrefs its argument */ ADDOP_N(c, STORE_ANNOTATION, mangled, names); } diff --git a/Python/import.c b/Python/import.c index 4b3bedf..fa0a9e1 100644 --- a/Python/import.c +++ b/Python/import.c @@ -354,12 +354,16 @@ PyImport_Cleanup(void) if (Py_VerboseFlag) PySys_WriteStderr("# clear builtins._\n"); - PyDict_SetItemString(interp->builtins, "_", Py_None); + if (PyDict_SetItemString(interp->builtins, "_", Py_None) < 0) { + PyErr_Clear(); + } for (p = sys_deletes; *p != NULL; p++) { if (Py_VerboseFlag) PySys_WriteStderr("# clear sys.%s\n", *p); - PyDict_SetItemString(interp->sysdict, *p, Py_None); + if (PyDict_SetItemString(interp->sysdict, *p, Py_None) < 0) { + PyErr_Clear(); + } } for (p = sys_files; *p != NULL; p+=2) { if (Py_VerboseFlag) @@ -367,7 +371,9 @@ PyImport_Cleanup(void) value = PyDict_GetItemString(interp->sysdict, *(p+1)); if (value == NULL) value = Py_None; - PyDict_SetItemString(interp->sysdict, *p, value); + if (PyDict_SetItemString(interp->sysdict, *p, value) < 0) { + PyErr_Clear(); + } } /* We prepare a list which will receive (name, weakref) tuples of @@ -381,14 +387,17 @@ PyImport_Cleanup(void) #define STORE_MODULE_WEAKREF(name, mod) \ if (weaklist != NULL) { \ PyObject *wr = PyWeakref_NewRef(mod, NULL); \ - if (name && wr) { \ + if (wr) { \ PyObject *tup = PyTuple_Pack(2, name, wr); \ - PyList_Append(weaklist, tup); \ + if (!tup || PyList_Append(weaklist, tup) < 0) { \ + PyErr_Clear(); \ + } \ Py_XDECREF(tup); \ + Py_DECREF(wr); \ } \ - Py_XDECREF(wr); \ - if (PyErr_Occurred()) \ + else { \ PyErr_Clear(); \ + } \ } /* Remove all modules from sys.modules, hoping that garbage collection @@ -399,7 +408,9 @@ PyImport_Cleanup(void) if (Py_VerboseFlag && PyUnicode_Check(key)) PySys_FormatStderr("# cleanup[2] removing %U\n", key); STORE_MODULE_WEAKREF(key, value); - PyDict_SetItem(modules, key, Py_None); + if (PyDict_SetItem(modules, key, Py_None) < 0) { + PyErr_Clear(); + } } } @@ -472,6 +483,7 @@ PyImport_Cleanup(void) /* Once more */ _PyGC_CollectNoFail(); +#undef CLEAR_MODULE #undef STORE_MODULE_WEAKREF } diff --git a/Python/pyhash.c b/Python/pyhash.c index a2ec230..13f6da1 100644 --- a/Python/pyhash.c +++ b/Python/pyhash.c @@ -272,8 +272,8 @@ fnv(const void *src, Py_ssize_t len) x = (_PyHASH_MULTIPLIER * x) ^ (Py_uhash_t) *p++; x ^= (Py_uhash_t) len; x ^= (Py_uhash_t) _Py_HashSecret.fnv.suffix; - if (x == -1) { - x = -2; + if (x == (Py_uhash_t) -1) { + x = (Py_uhash_t) -2; } return x; } @@ -369,7 +369,7 @@ siphash24(const void *src, Py_ssize_t src_sz) { uint64_t k0 = _le64toh(_Py_HashSecret.siphash.k0); uint64_t k1 = _le64toh(_Py_HashSecret.siphash.k1); uint64_t b = (uint64_t)src_sz << 56; - const uint64_t *in = (uint64_t*)src; + const uint8_t *in = (uint8_t*)src; uint64_t v0 = k0 ^ 0x736f6d6570736575ULL; uint64_t v1 = k1 ^ 0x646f72616e646f6dULL; @@ -378,12 +378,13 @@ siphash24(const void *src, Py_ssize_t src_sz) { uint64_t t; uint8_t *pt; - uint8_t *m; while (src_sz >= 8) { - uint64_t mi = _le64toh(*in); - in += 1; - src_sz -= 8; + uint64_t mi; + memcpy(&mi, in, sizeof(mi)); + mi = _le64toh(mi); + in += sizeof(mi); + src_sz -= sizeof(mi); v3 ^= mi; DOUBLE_ROUND(v0,v1,v2,v3); v0 ^= mi; @@ -391,15 +392,14 @@ siphash24(const void *src, Py_ssize_t src_sz) { t = 0; pt = (uint8_t *)&t; - m = (uint8_t *)in; switch (src_sz) { - case 7: pt[6] = m[6]; /* fall through */ - case 6: pt[5] = m[5]; /* fall through */ - case 5: pt[4] = m[4]; /* fall through */ - case 4: memcpy(pt, m, sizeof(uint32_t)); break; - case 3: pt[2] = m[2]; /* fall through */ - case 2: pt[1] = m[1]; /* fall through */ - case 1: pt[0] = m[0]; /* fall through */ + case 7: pt[6] = in[6]; /* fall through */ + case 6: pt[5] = in[5]; /* fall through */ + case 5: pt[4] = in[4]; /* fall through */ + case 4: memcpy(pt, in, sizeof(uint32_t)); break; + case 3: pt[2] = in[2]; /* fall through */ + case 2: pt[1] = in[1]; /* fall through */ + case 1: pt[0] = in[0]; /* fall through */ } b |= _le64toh(t); diff --git a/Python/random.c b/Python/random.c index e0ee153..c62cbeb 100644 --- a/Python/random.c +++ b/Python/random.c @@ -301,10 +301,15 @@ dev_urandom(char *buffer, Py_ssize_t size, int raise) if (raise) { struct _Py_stat_struct st; + int fstat_result; if (urandom_cache.fd >= 0) { + Py_BEGIN_ALLOW_THREADS + fstat_result = _Py_fstat_noraise(urandom_cache.fd, &st); + Py_END_ALLOW_THREADS + /* Does the fd point to the same thing as before? (issue #21207) */ - if (_Py_fstat_noraise(urandom_cache.fd, &st) + if (fstat_result || st.st_dev != urandom_cache.st_dev || st.st_ino != urandom_cache.st_ino) { /* Something changed: forget the cached fd (but don't close it, diff --git a/README.rst b/README.rst index 21b6084..1dab9c8 100644 --- a/README.rst +++ b/README.rst @@ -1,4 +1,4 @@ -This is Python version 3.6.5 +This is Python version 3.6.6 ============================ .. image:: https://travis-ci.org/python/cpython.svg?branch=3.6 @@ -61,6 +61,11 @@ You can pass many options to the configure script; run ``./configure --help`` to find out more. On macOS and Cygwin, the executable is called ``python.exe``; elsewhere it's just ``python``. +If you are running on macOS with the latest updates installed, make sure to install +openSSL or some other SSL software along with Homebrew or another package manager. +If issues persist, see https://devguide.python.org/setup/#macos-and-os-x for more +information. + On macOS, if you have configured Python with ``--enable-framework``, you should use ``make frameworkinstall`` to do the installation. Note that this installs the Python executable in a place that is not normally on your PATH, @@ -91,15 +96,16 @@ below. Profile Guided Optimization --------------------------- -PGO takes advantage of recent versions of the GCC or Clang compilers. If ran, -``make profile-opt`` will do several steps. +PGO takes advantage of recent versions of the GCC or Clang compilers. If used, +either via ``configure --enable-optimizations`` above or by manually running +``make profile-opt`` regardless of configure flags it will do several steps. First, the entire Python directory is cleaned of temporary files that may have resulted in a previous compilation. Then, an instrumented version of the interpreter is built, using suitable -compiler flags for each flavour. Note that this is just an intermediary step -and the binary resulted after this step is not good for real life workloads, as +compiler flags for each flavour. Note that this is just an intermediary step. +The binary resulting from this step is not good for real life workloads as it has profiling instructions embedded inside. After this instrumented version of the interpreter is built, the Makefile will @@ -171,9 +177,10 @@ something is wrong. By default, tests are prevented from overusing resources like disk space and memory. To enable these tests, run ``make testall``. -If any tests fail, you can re-run the failing test(s) in verbose mode:: +If any tests fail, you can re-run the failing test(s) in verbose mode. For +example, if ``test_os`` and ``test_gdb`` failed, you can run:: - make test TESTOPTS="-v test_that_failed" + make test TESTOPTS="-v test_os test_gdb" If the failure persists and appears to be a problem with Python rather than your environment, you can `file a bug report `_ and diff --git a/Tools/i18n/pygettext.py b/Tools/i18n/pygettext.py index 0f0395a..b46dd33 100755 --- a/Tools/i18n/pygettext.py +++ b/Tools/i18n/pygettext.py @@ -232,6 +232,10 @@ def escape_nonascii(s, encoding): return ''.join(escapes[b] for b in s.encode(encoding)) +def is_literal_string(s): + return s[0] in '\'"' or (s[0] in 'rRuU' and s[1] in '\'"') + + def safe_eval(s): # unwrap quotes, safely return eval(s, {'__builtins__':{}}, {}) @@ -259,24 +263,6 @@ def containsAny(str, set): return 1 in [c in str for c in set] -def _visit_pyfiles(list, dirname, names): - """Helper for getFilesForName().""" - # get extension for python source files - if '_py_ext' not in globals(): - global _py_ext - _py_ext = importlib.machinery.SOURCE_SUFFIXES[0] - - # don't recurse into CVS directories - if 'CVS' in names: - names.remove('CVS') - - # add all *.py files to list - list.extend( - [os.path.join(dirname, file) for file in names - if os.path.splitext(file)[1] == _py_ext] - ) - - def getFilesForName(name): """Get a list of module files for a filename, a module or package name, or a directory. @@ -302,7 +288,17 @@ def getFilesForName(name): if os.path.isdir(name): # find all python files in directory list = [] - os.walk(name, _visit_pyfiles, list) + # get extension for python source files + _py_ext = importlib.machinery.SOURCE_SUFFIXES[0] + for root, dirs, files in os.walk(name): + # don't recurse into CVS directories + if 'CVS' in dirs: + dirs.remove('CVS') + # add all *.py files to list + list.extend( + [os.path.join(root, file) for file in files + if os.path.splitext(file)[1] == _py_ext] + ) return list elif os.path.exists(name): # a single file @@ -325,8 +321,8 @@ class TokenEater: def __call__(self, ttype, tstring, stup, etup, line): # dispatch ## import token -## print >> sys.stderr, 'ttype:', token.tok_name[ttype], \ -## 'tstring:', tstring +## print('ttype:', token.tok_name[ttype], 'tstring:', tstring, +## file=sys.stderr) self.__state(ttype, tstring, stup[0]) def __waiting(self, ttype, tstring, lineno): @@ -335,7 +331,7 @@ class TokenEater: if opts.docstrings and not opts.nodocstrings.get(self.__curfile): # module docstring? if self.__freshmodule: - if ttype == tokenize.STRING: + if ttype == tokenize.STRING and is_literal_string(tstring): self.__addentry(safe_eval(tstring), lineno, isdocstring=1) self.__freshmodule = 0 elif ttype not in (tokenize.COMMENT, tokenize.NL): @@ -361,7 +357,7 @@ class TokenEater: def __suitedocstring(self, ttype, tstring, lineno): # ignore any intervening noise - if ttype == tokenize.STRING: + if ttype == tokenize.STRING and is_literal_string(tstring): self.__addentry(safe_eval(tstring), lineno, isdocstring=1) self.__state = self.__waiting elif ttype not in (tokenize.NEWLINE, tokenize.INDENT, @@ -386,7 +382,7 @@ class TokenEater: if self.__data: self.__addentry(EMPTYSTRING.join(self.__data)) self.__state = self.__waiting - elif ttype == tokenize.STRING: + elif ttype == tokenize.STRING and is_literal_string(tstring): self.__data.append(safe_eval(tstring)) elif ttype not in [tokenize.COMMENT, token.INDENT, token.DEDENT, token.NEWLINE, tokenize.NL]: diff --git a/Tools/scripts/checkpip.py b/Tools/scripts/checkpip.py index 8a64eda..a4a9ddf 100755 --- a/Tools/scripts/checkpip.py +++ b/Tools/scripts/checkpip.py @@ -14,7 +14,7 @@ def main(): for project, version in ensurepip._PROJECTS: data = json.loads(urllib.request.urlopen( - "https://pypi.python.org/pypi/{}/json".format(project), + "https://pypi.org/pypi/{}/json".format(project), cadefault=True, ).read().decode("utf8")) upstream_version = data["info"]["version"] diff --git a/Tools/scripts/patchcheck.py b/Tools/scripts/patchcheck.py index 1154b81..4ab6b77 100755 --- a/Tools/scripts/patchcheck.py +++ b/Tools/scripts/patchcheck.py @@ -45,16 +45,6 @@ def status(message, modal=False, info=None): return decorated_fxn -def mq_patches_applied(): - """Check if there are any applied MQ patches.""" - cmd = 'hg qapplied' - with subprocess.Popen(cmd.split(), - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) as st: - bstdout, _ = st.communicate() - return st.returncode == 0 and bstdout - - def get_git_branch(): """Get the symbolic name for the current git branch""" cmd = "git rev-parse --abbrev-ref HEAD".split() @@ -99,16 +89,8 @@ def get_base_branch(): @status("Getting the list of files that have been added/changed", info=lambda x: n_files_str(len(x))) def changed_files(base_branch=None): - """Get the list of changed or added files from Mercurial or git.""" - if os.path.isdir(os.path.join(SRCDIR, '.hg')): - if base_branch is not None: - sys.exit('need a git checkout to check PR status') - cmd = 'hg status --added --modified --no-status' - if mq_patches_applied(): - cmd += ' --rev qparent' - with subprocess.Popen(cmd.split(), stdout=subprocess.PIPE) as st: - filenames = [x.decode().rstrip() for x in st.stdout] - elif os.path.exists(os.path.join(SRCDIR, '.git')): + """Get the list of changed or added files from git.""" + if os.path.exists(os.path.join(SRCDIR, '.git')): # We just use an existence check here as: # directory = normal git checkout/clone # file = git worktree directory @@ -130,7 +112,7 @@ def changed_files(base_branch=None): filename = filename.split(' -> ', 2)[1].strip() filenames.append(filename) else: - sys.exit('need a Mercurial or git checkout to get modified files') + sys.exit('need a git checkout to get modified files') filenames2 = [] for filename in filenames: diff --git a/Tools/scripts/pathfix.py b/Tools/scripts/pathfix.py index 562bbc7..c5bf984 100755 --- a/Tools/scripts/pathfix.py +++ b/Tools/scripts/pathfix.py @@ -7,8 +7,9 @@ # Directories are searched recursively for files whose name looks # like a python module. # Symbolic links are always ignored (except as explicit directory -# arguments). Of course, the original file is kept as a back-up -# (with a "~" attached to its name). +# arguments). +# The original file is kept as a back-up (with a "~" attached to its name), +# -n flag can be used to disable this. # # Undoubtedly you can do this using find and sed or perl, but this is # a nice example of Python code that recurses down a directory tree @@ -31,14 +32,17 @@ rep = sys.stdout.write new_interpreter = None preserve_timestamps = False +create_backup = True + def main(): global new_interpreter global preserve_timestamps - usage = ('usage: %s -i /interpreter -p file-or-directory ...\n' % + global create_backup + usage = ('usage: %s -i /interpreter -p -n file-or-directory ...\n' % sys.argv[0]) try: - opts, args = getopt.getopt(sys.argv[1:], 'i:p') + opts, args = getopt.getopt(sys.argv[1:], 'i:pn') except getopt.error as msg: err(str(msg) + '\n') err(usage) @@ -48,6 +52,8 @@ def main(): new_interpreter = a.encode() if o == '-p': preserve_timestamps = True + if o == '-n': + create_backup = False if not new_interpreter or not new_interpreter.startswith(b'/') or \ not args: err('-i option or file-or-directory missing\n') @@ -134,10 +140,16 @@ def fix(filename): except OSError as msg: err('%s: warning: chmod failed (%r)\n' % (tempname, msg)) # Then make a backup of the original file as filename~ - try: - os.rename(filename, filename + '~') - except OSError as msg: - err('%s: warning: backup failed (%r)\n' % (filename, msg)) + if create_backup: + try: + os.rename(filename, filename + '~') + except OSError as msg: + err('%s: warning: backup failed (%r)\n' % (filename, msg)) + else: + try: + os.remove(filename) + except OSError as msg: + err('%s: warning: removing failed (%r)\n' % (filename, msg)) # Now move the temp file to the original file try: os.rename(tempname, filename) diff --git a/Tools/ssl/multissltests.py b/Tools/ssl/multissltests.py index ce5bbd8..f3241cd 100755 --- a/Tools/ssl/multissltests.py +++ b/Tools/ssl/multissltests.py @@ -57,8 +57,9 @@ LIBRESSL_OLD_VERSIONS = [ ] LIBRESSL_RECENT_VERSIONS = [ - "2.5.3", "2.5.5", + "2.6.4", + "2.7.1", ] # store files in ../multissl @@ -122,6 +123,11 @@ parser.add_argument( action='store_true', help="Don't run tests, only compile _ssl.c and _hashopenssl.c." ) +parser.add_argument( + '--system', + default='', + help="Override the automatic system type detection." +) class AbstractBuilder(object): @@ -149,6 +155,7 @@ class AbstractBuilder(object): # build directory (removed after install) self.build_dir = os.path.join( self.src_dir, self.build_template.format(version)) + self.system = args.system def __str__(self): return "<{0.__class__.__name__} for {0.version}>".format(self) @@ -253,9 +260,13 @@ class AbstractBuilder(object): cwd = self.build_dir cmd = ["./config", "shared", "--prefix={}".format(self.install_dir)] cmd.extend(self.compile_args) - self._subprocess_call(cmd, cwd=cwd) + env = None + if self.system: + env = os.environ.copy() + env['SYSTEM'] = self.system + self._subprocess_call(cmd, cwd=cwd, env=env) # Old OpenSSL versions do not support parallel builds. - self._subprocess_call(["make", "-j1"], cwd=cwd) + self._subprocess_call(["make", "-j1"], cwd=cwd, env=env) def _make_install(self, remove=True): self._subprocess_call(["make", "-j1", "install"], cwd=self.build_dir) diff --git a/configure b/configure index a707743..986ab87 100755 --- a/configure +++ b/configure @@ -3003,7 +3003,7 @@ if test "${enable_universalsdk+set}" = set; then : enableval=$enable_universalsdk; case $enableval in yes) - # Locate the best usable SDK, see Mac/README.txt for more + # Locate the best usable SDK, see Mac/README for more # information enableval="`/usr/bin/xcodebuild -version -sdk macosx Path 2>/dev/null`" if ! ( echo $enableval | grep -E '\.sdk' 1>/dev/null ) @@ -3527,7 +3527,7 @@ fi if test "$ac_sys_system" = "Darwin" then # Compiler selection on MacOSX is more complicated than - # AC_PROG_CC can handle, see Mac/README.txt for more + # AC_PROG_CC can handle, see Mac/README for more # information if test -z "${CC}" then @@ -6865,9 +6865,6 @@ if test "${OPT-unset}" = "unset" then case $GCC in yes) - if test "$CC" != 'g++' ; then - STRICT_PROTO="-Wstrict-prototypes" - fi # For gcc 4.x we need to use -fwrapv so lets check if its supported if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then WRAP="-fwrapv" @@ -6914,8 +6911,6 @@ then ;; esac - OPT="$OPT $STRICT_PROTO" - case $ac_sys_system in SCO_SV*) OPT="$OPT -m486 -DSCO5" ;; @@ -7186,6 +7181,47 @@ $as_echo "$ac_cv_disable_missing_field_initializers" >&6; } CFLAGS_NODIST="$CFLAGS_NODIST -Wno-missing-field-initializers" fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking if we can turn off $CC invalid function cast warning" >&5 +$as_echo_n "checking if we can turn off $CC invalid function cast warning... " >&6; } + ac_save_cc="$CC" + CC="$CC -Wcast-function-type -Werror" + if ${ac_cv_disable_cast_function_type+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + + +int +main () +{ + + ; + return 0; +} + +_ACEOF +if ac_fn_c_try_compile "$LINENO"; then : + + ac_cv_disable_cast_function_type=yes + +else + + ac_cv_disable_cast_function_type=no + +fi +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +fi + + CC="$ac_save_cc" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_disable_cast_function_type" >&5 +$as_echo "$ac_cv_disable_cast_function_type" >&6; } + + if test $ac_cv_disable_cast_function_type = yes + then + CFLAGS_NODIST="$CFLAGS_NODIST -Wno-cast-function-type" + fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking if we can turn on $CC mixed sign comparison warning" >&5 $as_echo_n "checking if we can turn on $CC mixed sign comparison warning... " >&6; } ac_save_cc="$CC" @@ -7283,6 +7319,47 @@ fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_enable_unreachable_code_warning" >&5 $as_echo "$ac_cv_enable_unreachable_code_warning" >&6; } + { $as_echo "$as_me:${as_lineno-$LINENO}: checking if we can turn on $CC strict-prototypes warning" >&5 +$as_echo_n "checking if we can turn on $CC strict-prototypes warning... " >&6; } + ac_save_cc="$CC" + CC="$CC -Werror -Wstrict-prototypes" + if ${ac_cv_enable_enable_strict_prototypes_warning+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + + +int +main () +{ + + ; + return 0; +} + +_ACEOF +if ac_fn_c_try_compile "$LINENO"; then : + + ac_cv_enable_strict_prototypes_warning=yes + +else + + ac_cv_enable_strict_prototypes_warning=no + +fi +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +fi + + CC="$ac_save_cc" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_enable_strict_prototypes_warning" >&5 +$as_echo "$ac_cv_enable_strict_prototypes_warning" >&6; } + + if test $ac_cv_enable_strict_prototypes_warning = yes + then + CFLAGS_NODIST="$CFLAGS_NODIST -Wstrict-prototypes" + fi + # if using gcc on alpha, use -mieee to get (near) full IEEE 754 # support. Without this, treatment of subnormals doesn't follow # the standard. diff --git a/configure.ac b/configure.ac index 0ec0936..338be3b 100644 --- a/configure.ac +++ b/configure.ac @@ -155,7 +155,7 @@ AC_ARG_ENABLE(universalsdk, [ case $enableval in yes) - # Locate the best usable SDK, see Mac/README.txt for more + # Locate the best usable SDK, see Mac/README for more # information enableval="`/usr/bin/xcodebuild -version -sdk macosx Path 2>/dev/null`" if ! ( echo $enableval | grep -E '\.sdk' 1>/dev/null ) @@ -632,7 +632,7 @@ fi if test "$ac_sys_system" = "Darwin" then # Compiler selection on MacOSX is more complicated than - # AC_PROG_CC can handle, see Mac/README.txt for more + # AC_PROG_CC can handle, see Mac/README for more # information if test -z "${CC}" then @@ -1464,9 +1464,6 @@ if test "${OPT-unset}" = "unset" then case $GCC in yes) - if test "$CC" != 'g++' ; then - STRICT_PROTO="-Wstrict-prototypes" - fi # For gcc 4.x we need to use -fwrapv so lets check if its supported if "$CC" -v --help 2>/dev/null |grep -- -fwrapv > /dev/null; then WRAP="-fwrapv" @@ -1513,8 +1510,6 @@ then ;; esac - OPT="$OPT $STRICT_PROTO" - case $ac_sys_system in SCO_SV*) OPT="$OPT -m486 -DSCO5" ;; @@ -1666,6 +1661,26 @@ yes) CFLAGS_NODIST="$CFLAGS_NODIST -Wno-missing-field-initializers" fi + AC_MSG_CHECKING(if we can turn off $CC invalid function cast warning) + ac_save_cc="$CC" + CC="$CC -Wcast-function-type -Werror" + AC_CACHE_VAL(ac_cv_disable_cast_function_type, + AC_COMPILE_IFELSE( + [ + AC_LANG_PROGRAM([[]], [[]]) + ],[ + ac_cv_disable_cast_function_type=yes + ],[ + ac_cv_disable_cast_function_type=no + ])) + CC="$ac_save_cc" + AC_MSG_RESULT($ac_cv_disable_cast_function_type) + + if test $ac_cv_disable_cast_function_type = yes + then + CFLAGS_NODIST="$CFLAGS_NODIST -Wno-cast-function-type" + fi + AC_MSG_CHECKING(if we can turn on $CC mixed sign comparison warning) ac_save_cc="$CC" CC="$CC -Wsign-compare" @@ -1721,6 +1736,26 @@ yes) fi AC_MSG_RESULT($ac_cv_enable_unreachable_code_warning) + AC_MSG_CHECKING(if we can turn on $CC strict-prototypes warning) + ac_save_cc="$CC" + CC="$CC -Werror -Wstrict-prototypes" + AC_CACHE_VAL(ac_cv_enable_enable_strict_prototypes_warning, + AC_COMPILE_IFELSE( + [ + AC_LANG_PROGRAM([[]], [[]]) + ],[ + ac_cv_enable_strict_prototypes_warning=yes + ],[ + ac_cv_enable_strict_prototypes_warning=no + ])) + CC="$ac_save_cc" + AC_MSG_RESULT($ac_cv_enable_strict_prototypes_warning) + + if test $ac_cv_enable_strict_prototypes_warning = yes + then + CFLAGS_NODIST="$CFLAGS_NODIST -Wstrict-prototypes" + fi + # if using gcc on alpha, use -mieee to get (near) full IEEE 754 # support. Without this, treatment of subnormals doesn't follow # the standard.