Merge branch 'main' into add_return_none

Doc/bugs.rst

@@ -19,6 +19,9 @@ If you find a bug in this documentation or would like to propose an improvement,
 please submit a bug report on the :ref:`tracker <using-the-tracker>`.  If you
 have a suggestion on how to fix it, include that as well.
 
+You can also open a discussion item on our
+`Documentation Discourse forum <https://discuss.python.org/c/documentation/26>`_.
+
 If you're short on time, you can also email documentation bug reports to
 docs@python.org (behavioral bugs can be sent to python-list@python.org).
 'docs@' is a mailing list run by volunteers; your request will be noticed,

Doc/library/asyncio-eventloop.rst

@@ -186,19 +186,24 @@ Running and stopping the loop
 .. coroutinemethod:: loop.shutdown_default_executor(timeout=None)
 
    Schedule the closure of the default executor and wait for it to join all of
-   the threads in the :class:`ThreadPoolExecutor`. After calling this method, a
-   :exc:`RuntimeError` will be raised if :meth:`loop.run_in_executor` is called
-   while using the default executor.
+   the threads in the :class:`~concurrent.futures.ThreadPoolExecutor`.
+   Once this method has been called,
+   using the default executor with :meth:`loop.run_in_executor`
+   will raise a :exc:`RuntimeError`.
 
-   The *timeout* parameter specifies the amount of time the executor will
-   be given to finish joining. The default value is ``None``, which means the
-   executor will be given an unlimited amount of time.
+   The *timeout* parameter specifies the amount of time
+   (in :class:`float` seconds) the executor will be given to finish joining.
+   With the default, ``None``,
+   the executor is allowed an unlimited amount of time.
 
-   If the timeout duration is reached, a warning is emitted and executor is
-   terminated without waiting for its threads to finish joining.
+   If the *timeout* is reached, a :exc:`RuntimeWarning` is emitted
+   and the default executor is terminated
+   without waiting for its threads to finish joining.
 
-   Note that there is no need to call this function when
-   :func:`asyncio.run` is used.
+   .. note::
+
+      Do not call this method when using :func:`asyncio.run`,
+      as the latter handles default executor shutdown automatically.
 
    .. versionadded:: 3.9
 
@@ -213,22 +218,23 @@ Scheduling callbacks
    Schedule the *callback* :term:`callback` to be called with
    *args* arguments at the next iteration of the event loop.
 
+   Return an instance of :class:`asyncio.Handle`,
+   which can be used later to cancel the callback.
+
    Callbacks are called in the order in which they are registered.
    Each callback will be called exactly once.
 
-   An optional keyword-only *context* argument allows specifying a
+   The optional keyword-only *context* argument specifies a
    custom :class:`contextvars.Context` for the *callback* to run in.
-   The current context is used when no *context* is provided.
-
-   An instance of :class:`asyncio.Handle` is returned, which can be
-   used later to cancel the callback.
+   Callbacks use the current context when no *context* is provided.
 
-   This method is not thread-safe.
+   Unlike :meth:`call_soon_threadsafe`, this method is not thread-safe.
 
 .. method:: loop.call_soon_threadsafe(callback, *args, context=None)
 
-   A thread-safe variant of :meth:`call_soon`.  Must be used to
-   schedule callbacks *from another thread*.
+   A thread-safe variant of :meth:`call_soon`. When scheduling callbacks from
+   another thread, this function *must* be used, since :meth:`call_soon` is not
+   thread-safe.
 
    Raises :exc:`RuntimeError` if called on a loop that's been closed.
    This can happen on a secondary thread when the main application is

Doc/requirements.txt

@@ -8,7 +8,7 @@ sphinx==4.5.0
 blurb
 
 sphinx-lint==0.6.7
-sphinxext-opengraph>=0.7.1
+sphinxext-opengraph==0.7.5
 
 # The theme used by the documentation is stored separately, so we need
 # to install that as well.

Doc/whatsnew/3.12.rst

@@ -479,7 +479,7 @@ APIs:
 * :class:`webbrowser.MacOSX` (:gh:`86421`)
 
 Pending Removal in Python 3.14
-==============================
+------------------------------
 
 * Deprecated the following :mod:`importlib.abc` classes, scheduled for removal in
   Python 3.14:

Lib/enum.py

@@ -1429,12 +1429,11 @@ def _missing_(cls, value):
                     % (cls.__name__, value, unknown, bin(unknown))
                     )
         # normal Flag?
-        __new__ = getattr(cls, '__new_member__', None)
-        if cls._member_type_ is object and not __new__:
+        if cls._member_type_ is object:
             # construct a singleton enum pseudo-member
             pseudo_member = object.__new__(cls)
         else:
-            pseudo_member = (__new__ or cls._member_type_.__new__)(cls, value)
+            pseudo_member = cls._member_type_.__new__(cls, value)
         if not hasattr(pseudo_member, '_value_'):
             pseudo_member._value_ = value
         if member_value:

Lib/test/test_capi/test_eval_code_ex.py

@@ -0,0 +1,56 @@
+import unittest
+
+from test.support import import_helper
+
+
+# Skip this test if the _testcapi module isn't available.
+_testcapi = import_helper.import_module('_testcapi')
+
+
+class PyEval_EvalCodeExTests(unittest.TestCase):
+
+    def test_simple(self):
+        def f():
+            return a
+
+        self.assertEqual(_testcapi.eval_code_ex(f.__code__, dict(a=1)), 1)
+
+    # Need to force the compiler to use LOAD_NAME
+    # def test_custom_locals(self):
+    #     def f():
+    #         return
+
+    def test_with_args(self):
+        def f(a, b, c):
+            return a
+
+        self.assertEqual(_testcapi.eval_code_ex(f.__code__, {}, {}, (1, 2, 3)), 1)
+
+    def test_with_kwargs(self):
+        def f(a, b, c):
+            return a
+
+        self.assertEqual(_testcapi.eval_code_ex(f.__code__, {}, {}, (), dict(a=1, b=2, c=3)), 1)
+
+    def test_with_default(self):
+        def f(a):
+            return a
+
+        self.assertEqual(_testcapi.eval_code_ex(f.__code__, {}, {}, (), {}, (1,)), 1)
+
+    def test_with_kwarg_default(self):
+        def f(*, a):
+            return a
+
+        self.assertEqual(_testcapi.eval_code_ex(f.__code__, {}, {}, (), {}, (), dict(a=1)), 1)
+
+    def test_with_closure(self):
+        a = 1
+        def f():
+            return a
+
+        self.assertEqual(_testcapi.eval_code_ex(f.__code__, {}, {}, (), {}, (), {}, f.__closure__), 1)
+
+
+if __name__ == "__main__":
+    unittest.main()

Lib/test/test_capi/test_misc.py

@@ -1413,6 +1413,9 @@ def callback():
         ret = assert_python_ok('-X', 'tracemalloc', '-c', code)
         self.assertIn(b'callback called', ret.out)
 
+    def test_gilstate_matches_current(self):
+        _testcapi.test_current_tstate_matches()
+
 
 class Test_testcapi(unittest.TestCase):
     locals().update((name, getattr(_testcapi, name))

Lib/test/test_ctypes/test_pep3118.py

@@ -87,14 +87,14 @@ class PackedPoint(Structure):
     _fields_ = [("x", c_long), ("y", c_long)]
 
 class PointMidPad(Structure):
-    _fields_ = [("x", c_byte), ("y", c_uint64)]
+    _fields_ = [("x", c_byte), ("y", c_uint)]
 
 class PackedPointMidPad(Structure):
     _pack_ = 2
     _fields_ = [("x", c_byte), ("y", c_uint64)]
 
 class PointEndPad(Structure):
-    _fields_ = [("x", c_uint64), ("y", c_byte)]
+    _fields_ = [("x", c_uint), ("y", c_byte)]
 
 class PackedPointEndPad(Structure):
     _pack_ = 2
@@ -199,14 +199,14 @@ class Complete(Structure):
 
     ## structures and unions
 
-    (Point2,                    "T{<l:x:<l:y:}".replace('l', s_long),  (),  Point2),
-    (Point,                     "T{<l:x:<l:y:}".replace('l', s_long),  (),  Point),
-    (PackedPoint,               "T{<l:x:<l:y:}".replace('l', s_long),  (),  PackedPoint),
-    (PointMidPad,               "T{<b:x:7x<Q:y:}",                     (),  PointMidPad),
-    (PackedPointMidPad,         "T{<b:x:x<Q:y:}",                      (),  PackedPointMidPad),
-    (PointEndPad,               "T{<Q:x:<b:y:7x}",                     (),  PointEndPad),
-    (PackedPointEndPad,         "T{<Q:x:<b:y:x}",                      (),  PackedPointEndPad),
-    (EmptyStruct,               "T{}",                                 (),  EmptyStruct),
+    (Point2,                    "T{<l:x:<l:y:}".replace('l', s_long),   (),  Point2),
+    (Point,                     "T{<l:x:<l:y:}".replace('l', s_long),   (),  Point),
+    (PackedPoint,               "T{<l:x:<l:y:}".replace('l', s_long),   (),  PackedPoint),
+    (PointMidPad,               "T{<b:x:3x<I:y:}".replace('I', s_uint), (),  PointMidPad),
+    (PackedPointMidPad,         "T{<b:x:x<Q:y:}",                       (),  PackedPointMidPad),
+    (PointEndPad,               "T{<I:x:<b:y:3x}".replace('I', s_uint), (),  PointEndPad),
+    (PackedPointEndPad,         "T{<Q:x:<b:y:x}",                       (),  PackedPointEndPad),
+    (EmptyStruct,               "T{}",                                  (),  EmptyStruct),
     # the pep doesn't support unions
     (aUnion,                    "B",                                   (),  aUnion),
     # structure with sub-arrays

Lib/test/test_enum.py

@@ -2855,6 +2855,46 @@ class NTEnum(Enum):
                 [TTuple(id=0, a=0, blist=[]), TTuple(id=1, a=2, blist=[4]), TTuple(id=2, a=4, blist=[0, 1, 2])],
                 )
 
+    def test_flag_with_custom_new(self):
+        class FlagFromChar(IntFlag):
+            def __new__(cls, c):
+                value = 1 << c
+                self = int.__new__(cls, value)
+                self._value_ = value
+                return self
+            #
+            a = ord('a')
+        #
+        self.assertEqual(FlagFromChar.a, 158456325028528675187087900672)
+        self.assertEqual(FlagFromChar.a|1, 158456325028528675187087900673)
+        #
+        #
+        class FlagFromChar(Flag):
+            def __new__(cls, c):
+                value = 1 << c
+                self = object.__new__(cls)
+                self._value_ = value
+                return self
+            #
+            a = ord('a')
+            z = 1
+        #
+        self.assertEqual(FlagFromChar.a.value, 158456325028528675187087900672)
+        self.assertEqual((FlagFromChar.a|FlagFromChar.z).value, 158456325028528675187087900674)
+        #
+        #
+        class FlagFromChar(int, Flag, boundary=KEEP):
+            def __new__(cls, c):
+                value = 1 << c
+                self = int.__new__(cls, value)
+                self._value_ = value
+                return self
+            #
+            a = ord('a')
+        #
+        self.assertEqual(FlagFromChar.a, 158456325028528675187087900672)
+        self.assertEqual(FlagFromChar.a|1, 158456325028528675187087900673)
+
 class TestOrder(unittest.TestCase):
     "test usage of the `_order_` attribute"
 

Lib/test/test_hashlib.py

@@ -22,6 +22,7 @@
 from test.support import _4G, bigmemtest
 from test.support.import_helper import import_fresh_module
 from test.support import os_helper
+from test.support import requires_resource
 from test.support import threading_helper
 from test.support import warnings_helper
 from http.client import HTTPException
@@ -354,6 +355,15 @@ def test_large_update(self):
             self.assertEqual(m1.digest(*args), m4_copy.digest(*args))
             self.assertEqual(m4.digest(*args), m4_digest)
 
+    @requires_resource('cpu')
+    def test_sha256_update_over_4gb(self):
+        zero_1mb = b"\0" * 1024 * 1024
+        h = hashlib.sha256()
+        for i in range(0, 4096):
+            h.update(zero_1mb)
+        h.update(b"hello world")
+        self.assertEqual(h.hexdigest(), "a5364f7a52ebe2e25f1838a4ca715a893b6fd7a23f2a0d9e9762120da8b1bf53")
+
     def check(self, name, data, hexdigest, shake=False, **kwargs):
         length = len(hexdigest)//2
         hexdigest = hexdigest.lower()

Lib/test/test_typing.py

@@ -4892,6 +4892,18 @@ class NontotalMovie(TypedDict, total=False):
     title: Required[str]
     year: int
 
+class ParentNontotalMovie(TypedDict, total=False):
+    title: Required[str]
+
+class ChildTotalMovie(ParentNontotalMovie):
+    year: NotRequired[int]
+
+class ParentDeeplyAnnotatedMovie(TypedDict):
+    title: Annotated[Annotated[Required[str], "foobar"], "another level"]
+
+class ChildDeeplyAnnotatedMovie(ParentDeeplyAnnotatedMovie):
+    year: NotRequired[Annotated[int, 2000]]
+
 class AnnotatedMovie(TypedDict):
     title: Annotated[Required[str], "foobar"]
     year: NotRequired[Annotated[int, 2000]]
@@ -5221,6 +5233,17 @@ def test_get_type_hints_typeddict(self):
             'a': Annotated[Required[int], "a", "b", "c"]
         })
 
+        self.assertEqual(get_type_hints(ChildTotalMovie), {"title": str, "year": int})
+        self.assertEqual(get_type_hints(ChildTotalMovie, include_extras=True), {
+            "title": Required[str], "year": NotRequired[int]
+        })
+
+        self.assertEqual(get_type_hints(ChildDeeplyAnnotatedMovie), {"title": str, "year": int})
+        self.assertEqual(get_type_hints(ChildDeeplyAnnotatedMovie, include_extras=True), {
+            "title": Annotated[Required[str], "foobar", "another level"],
+            "year": NotRequired[Annotated[int, 2000]]
+        })
+
     def test_get_type_hints_collections_abc_callable(self):
         # https://github.com/python/cpython/issues/91621
         P = ParamSpec('P')
@@ -6381,6 +6404,16 @@ def test_required_notrequired_keys(self):
         self.assertEqual(WeirdlyQuotedMovie.__optional_keys__,
                          frozenset({"year"}))
 
+        self.assertEqual(ChildTotalMovie.__required_keys__,
+                         frozenset({"title"}))
+        self.assertEqual(ChildTotalMovie.__optional_keys__,
+                         frozenset({"year"}))
+
+        self.assertEqual(ChildDeeplyAnnotatedMovie.__required_keys__,
+                         frozenset({"title"}))
+        self.assertEqual(ChildDeeplyAnnotatedMovie.__optional_keys__,
+                         frozenset({"year"}))
+
     def test_multiple_inheritance(self):
         class One(TypedDict):
             one: int

Makefile.pre.in

@@ -2612,7 +2612,7 @@ MODULE__HASHLIB_DEPS=$(srcdir)/Modules/hashlib.h
 MODULE__IO_DEPS=$(srcdir)/Modules/_io/_iomodule.h
 MODULE__MD5_DEPS=$(srcdir)/Modules/hashlib.h
 MODULE__SHA1_DEPS=$(srcdir)/Modules/hashlib.h
-MODULE__SHA256_DEPS=$(srcdir)/Modules/hashlib.h
+MODULE__SHA256_DEPS=$(srcdir)/Modules/hashlib.h $(srcdir)/Modules/_hacl/include/krml/FStar_UInt_8_16_32_64.h $(srcdir)/Modules/_hacl/include/krml/lowstar_endianness.h $(srcdir)/Modules/_hacl/include/krml/internal/target.h $(srcdir)/Modules/_hacl/Hacl_Streaming_SHA2.h
 MODULE__SHA3_DEPS=$(srcdir)/Modules/_sha3/sha3.c $(srcdir)/Modules/_sha3/sha3.h $(srcdir)/Modules/hashlib.h
 MODULE__SHA512_DEPS=$(srcdir)/Modules/hashlib.h
 MODULE__SOCKET_DEPS=$(srcdir)/Modules/socketmodule.h $(srcdir)/Modules/addrinfo.h $(srcdir)/Modules/getaddrinfo.c $(srcdir)/Modules/getnameinfo.c

Misc/ACKS

@@ -1414,6 +1414,7 @@ Jean-François Piéronne
 Oleg Plakhotnyuk
 Anatoliy Platonov
 Marcel Plch
+Kirill Podoprigora
 Remi Pointel
 Jon Poler
 Ariel Poliak

Misc/NEWS.d/next/Core and Builtins/2023-01-28-20-31-42.gh-issue-101372.8BcpCC.rst

@@ -0,0 +1,2 @@
+Fix :func:`~unicodedata.is_normalized` to properly handle the UCD 3.2.0
+cases. Patch by Dong-hee Na.

Misc/NEWS.d/next/Core and Builtins/2023-01-30-11-56-09.gh-issue-59956.7xqnC_.rst

@@ -0,0 +1,3 @@
+The GILState API is now partially compatible with subinterpreters.
+Previously, ``PyThreadState_GET()`` and ``PyGILState_GetThisThreadState()``
+would get out of sync, causing inconsistent behavior and crashes.

Misc/NEWS.d/next/Core and Builtins/2023-02-06-20-13-36.gh-issue-92173.RQE0mk.rst

@@ -0,0 +1,2 @@
+Fix the ``defs`` and ``kwdefs`` arguments to :c:func:`PyEval_EvalCodeEx`
+and a reference leak in that function.

Misc/NEWS.d/next/Library/2023-02-05-14-39-49.gh-issue-101541.Mo3ppp.rst

@@ -0,0 +1 @@
+[Enum] - fix psuedo-flag creation

Misc/NEWS.d/next/Security/2022-11-08-12-06-52.gh-issue-99108.4Wrsuh.rst

@@ -0,0 +1,4 @@
+Replace the builtin :mod:`hashlib` implementations of SHA2-224 and SHA2-256
+originally from LibTomCrypt with formally verified, side-channel resistant
+code from the `HACL* <https://github.com/hacl-star/hacl-star/>`_ project.  The
+builtins remain a fallback only used when OpenSSL does not provide them.

Misc/NEWS.d/next/Windows/2023-02-03-17-53-06.gh-issue-101543.cORAT4.rst

@@ -0,0 +1,2 @@
+Ensure the install path in the registry is only used when the standard
+library hasn't been located in any other way.

Modules/Setup.stdlib.in

@@ -79,7 +79,7 @@
 # hashing builtins, can be disabled with --without-builtin-hashlib-hashes
 @MODULE__MD5_TRUE@_md5 md5module.c
 @MODULE__SHA1_TRUE@_sha1 sha1module.c
-@MODULE__SHA256_TRUE@_sha256 sha256module.c
+@MODULE__SHA256_TRUE@_sha256 sha256module.c _hacl/Hacl_Streaming_SHA2.c
 @MODULE__SHA512_TRUE@_sha512 sha512module.c
 @MODULE__SHA3_TRUE@_sha3 _sha3/sha3module.c
 @MODULE__BLAKE2_TRUE@_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c