Merge branch 'main' into issue-99631-2

Author: furkanonder

Doc/c-api/module.rst

@@ -288,22 +288,40 @@ An alternate way to specify extensions is to request "multi-phase initialization
 Extension modules created this way behave more like Python modules: the
 initialization is split between the *creation phase*, when the module object
 is created, and the *execution phase*, when it is populated.
-The distinction is similar to the :py:meth:`!__new__` and :py:meth:`!__init__` methods
-of classes.
+The distinction is similar to the :py:meth:`~object.__new__` and
+:py:meth:`~object.__init__` methods of classes.
 
 Unlike modules created using single-phase initialization, these modules are not
-singletons: if the *sys.modules* entry is removed and the module is re-imported,
-a new module object is created, and the old module is subject to normal garbage
-collection -- as with Python modules.
-By default, multiple modules created from the same definition should be
-independent: changes to one should not affect the others.
-This means that all state should be specific to the module object (using e.g.
-using :c:func:`PyModule_GetState`), or its contents (such as the module's
-:attr:`~object.__dict__` or individual classes created with :c:func:`PyType_FromSpec`).
+singletons.
+For example, if the :py:attr:`sys.modules` entry is removed and the module
+is re-imported, a new module object is created, and typically populated with
+fresh method and type objects.
+The old module is subject to normal garbage collection.
+This mirrors the behavior of pure-Python modules.
+
+Additional module instances may be created in
+:ref:`sub-interpreters <sub-interpreter-support>`
+or after after Python runtime reinitialization
+(:c:func:`Py_Finalize` and :c:func:`Py_Initialize`).
+In these cases, sharing Python objects between module instances would likely
+cause crashes or undefined behavior.
+
+To avoid such issues, each instance of an extension module should
+be *isolated*: changes to one instance should not implicitly affect the others,
+and all state, including references to Python objects, should be specific to
+a particular module instance.
+See :ref:`isolating-extensions-howto` for more details and a practical guide.
+
+A simpler way to avoid these issues is
+:ref:`raising an error on repeated initialization <isolating-extensions-optout>`.
 
 All modules created using multi-phase initialization are expected to support
-:ref:`sub-interpreters <sub-interpreter-support>`. Making sure multiple modules
-are independent is typically enough to achieve this.
+:ref:`sub-interpreters <sub-interpreter-support>`, or otherwise explicitly
+signal a lack of support.
+This is usually achieved by isolation or blocking repeated initialization,
+as above.
+A module may also be limited to the main interpreter using
+the :c:data:`Py_mod_multiple_interpreters` slot.
 
 To request multi-phase initialization, the initialization function
 (PyInit_modulename) returns a :c:type:`PyModuleDef` instance with non-empty

Doc/howto/isolating-extensions.rst

@@ -168,7 +168,7 @@ possible, consider explicit locking.
 If it is necessary to use process-global state, the simplest way to
 avoid issues with multiple interpreters is to explicitly prevent a
 module from being loaded more than once per process—see
-`Opt-Out: Limiting to One Module Object per Process`_.
+:ref:`isolating-extensions-optout`.
 
 
 Managing Per-Module State
@@ -207,6 +207,8 @@ An example of a module with per-module state is currently available as
 example module initialization shown at the bottom of the file.
 
 
+.. _isolating-extensions-optout:
+
 Opt-Out: Limiting to One Module Object per Process
 --------------------------------------------------
 

Doc/library/ast.rst

@@ -2445,8 +2445,9 @@ and classes for traversing abstract syntax trees:
    indents that many spaces per level.  If *indent* is a string (such as ``"\t"``),
    that string is used to indent each level.
 
-   If *show_empty* is ``False`` (the default), empty lists and fields that are ``None``
-   will be omitted from the output.
+   If *show_empty* is false (the default), optional empty lists will be
+   omitted from the output.
+   Optional ``None`` values are always omitted.
 
    .. versionchanged:: 3.9
       Added the *indent* option.

Include/internal/mimalloc/mimalloc/types.h

@@ -50,6 +50,32 @@ terms of the MIT license. A copy of the license can be found in the file
 #define mi_decl_cache_align
 #endif
 
+#if (MI_DEBUG)
+#if defined(_MSC_VER)
+#define mi_decl_noreturn        __declspec(noreturn)
+#elif (defined(__GNUC__) && (__GNUC__ >= 3)) || defined(__clang__)
+#define mi_decl_noreturn        __attribute__((__noreturn__))
+#else
+#define mi_decl_noreturn
+#endif
+
+/*
+ * 'cold' attribute seems to have been fully supported since GCC 4.x.
+ * See https://github.com/gcc-mirror/gcc/commit/52bf96d2f299e9e6.
+ */
+#if (defined(__GNUC__) && (__GNUC__ >= 4)) || defined(__clang__)
+#define mi_decl_cold            __attribute__((cold))
+#else
+#define mi_decl_cold
+#endif
+
+#if (defined(__GNUC__) && defined(__THROW))
+#define mi_decl_throw           __THROW
+#else
+#define mi_decl_throw
+#endif
+#endif
+
 // ------------------------------------------------------
 // Variants
 // ------------------------------------------------------
@@ -582,7 +608,8 @@ struct mi_heap_s {
 
 #if (MI_DEBUG)
 // use our own assertion to print without memory allocation
-void _mi_assert_fail(const char* assertion, const char* fname, unsigned int line, const char* func );
+mi_decl_noreturn mi_decl_cold mi_decl_throw
+void _mi_assert_fail(const char* assertion, const char* fname, unsigned int line, const char* func);
 #define mi_assert(expr)     ((expr) ? (void)0 : _mi_assert_fail(#expr,__FILE__,__LINE__,__func__))
 #else
 #define mi_assert(x)

Include/internal/pycore_crossinterp.h

@@ -317,7 +317,9 @@ typedef enum error_code {
     _PyXI_ERR_ALREADY_RUNNING = -4,
     _PyXI_ERR_MAIN_NS_FAILURE = -5,
     _PyXI_ERR_APPLY_NS_FAILURE = -6,
-    _PyXI_ERR_NOT_SHAREABLE = -7,
+    _PyXI_ERR_PRESERVE_FAILURE = -7,
+    _PyXI_ERR_EXC_PROPAGATION_FAILURE = -8,
+    _PyXI_ERR_NOT_SHAREABLE = -9,
 } _PyXI_errcode;
 
 
@@ -350,16 +352,33 @@ typedef struct xi_session _PyXI_session;
 PyAPI_FUNC(_PyXI_session *) _PyXI_NewSession(void);
 PyAPI_FUNC(void) _PyXI_FreeSession(_PyXI_session *);
 
+typedef struct {
+    PyObject *preserved;
+    PyObject *excinfo;
+    _PyXI_errcode errcode;
+} _PyXI_session_result;
+PyAPI_FUNC(void) _PyXI_ClearResult(_PyXI_session_result *);
+
 PyAPI_FUNC(int) _PyXI_Enter(
     _PyXI_session *session,
     PyInterpreterState *interp,
-    PyObject *nsupdates);
-PyAPI_FUNC(void) _PyXI_Exit(_PyXI_session *session);
-
-PyAPI_FUNC(PyObject *) _PyXI_GetMainNamespace(_PyXI_session *);
-
-PyAPI_FUNC(PyObject *) _PyXI_ApplyCapturedException(_PyXI_session *session);
-PyAPI_FUNC(int) _PyXI_HasCapturedException(_PyXI_session *session);
+    PyObject *nsupdates,
+    _PyXI_session_result *);
+PyAPI_FUNC(int) _PyXI_Exit(
+    _PyXI_session *,
+    _PyXI_errcode,
+    _PyXI_session_result *);
+
+PyAPI_FUNC(PyObject *) _PyXI_GetMainNamespace(
+    _PyXI_session *,
+    _PyXI_errcode *);
+
+PyAPI_FUNC(int) _PyXI_Preserve(
+    _PyXI_session *,
+    const char *,
+    PyObject *,
+    _PyXI_errcode *);
+PyAPI_FUNC(PyObject *) _PyXI_GetPreserved(_PyXI_session_result *, const char *);
 
 
 /*************/

Lib/ast.py

@@ -147,18 +147,16 @@ def _format(node, level=0):
                 if value is None and getattr(cls, name, ...) is None:
                     keywords = True
                     continue
-                if (
-                    not show_empty
-                    and (value is None or value == [])
-                    # Special cases:
-                    # `Constant(value=None)` and `MatchSingleton(value=None)`
-                    and not isinstance(node, (Constant, MatchSingleton))
-                ):
-                    args_buffer.append(repr(value))
-                    continue
-                elif not keywords:
-                    args.extend(args_buffer)
-                    args_buffer = []
+                if not show_empty:
+                    if value == []:
+                        field_type = cls._field_types.get(name, object)
+                        if getattr(field_type, '__origin__', ...) is list:
+                            if not keywords:
+                                args_buffer.append(repr(value))
+                            continue
+                    if not keywords:
+                        args.extend(args_buffer)
+                        args_buffer = []
                 value, simple = _format(value, level)
                 allsimple = allsimple and simple
                 if keywords:

Lib/test/_code_definitions.py

@@ -57,6 +57,15 @@ def spam_with_globals_and_builtins():
     print(res)
 
 
+def spam_full_args(a, b, /, c, d, *args, e, f, **kwargs):
+    return (a, b, c, d, e, f, args, kwargs)
+
+
+def spam_full_args_with_defaults(a=-1, b=-2, /, c=-3, d=-4, *args,
+                                 e=-5, f=-6, **kwargs):
+    return (a, b, c, d, e, f, args, kwargs)
+
+
 def spam_args_attrs_and_builtins(a, b, /, c, d, *args, e, f, **kwargs):
     if args.__len__() > 2:
         return None
@@ -67,6 +76,10 @@ def spam_returns_arg(x):
     return x
 
 
+def spam_raises():
+    raise Exception('spam!')
+
+
 def spam_with_inner_not_closure():
     def eggs():
         pass
@@ -177,8 +190,11 @@ def ham_C_closure(z):
     spam_minimal,
     spam_with_builtins,
     spam_with_globals_and_builtins,
+    spam_full_args,
+    spam_full_args_with_defaults,
     spam_args_attrs_and_builtins,
     spam_returns_arg,
+    spam_raises,
     spam_with_inner_not_closure,
     spam_with_inner_closure,
     spam_annotated,
@@ -219,8 +235,10 @@ def ham_C_closure(z):
     spam,
     spam_minimal,
     spam_with_builtins,
+    spam_full_args,
     spam_args_attrs_and_builtins,
     spam_returns_arg,
+    spam_raises,
     spam_annotated,
     spam_with_inner_not_closure,
     spam_with_inner_closure,
@@ -238,6 +256,7 @@ def ham_C_closure(z):
 STATELESS_CODE = [
     *STATELESS_FUNCTIONS,
     script_with_globals,
+    spam_full_args_with_defaults,
     spam_with_globals_and_builtins,
     spam_full,
 ]
@@ -248,6 +267,7 @@ def ham_C_closure(z):
     script_with_explicit_empty_return,
     spam_minimal,
     spam_with_builtins,
+    spam_raises,
     spam_with_inner_not_closure,
     spam_with_inner_closure,
 ]

Lib/test/support/interpreters/__init__.py

@@ -226,33 +226,32 @@ def exec(self, code, /):
         if excinfo is not None:
             raise ExecutionFailed(excinfo)
 
-    def call(self, callable, /):
-        """Call the object in the interpreter with given args/kwargs.
+    def _call(self, callable, args, kwargs):
+        res, excinfo = _interpreters.call(self._id, callable, args, kwargs, restrict=True)
+        if excinfo is not None:
+            raise ExecutionFailed(excinfo)
+        return res
 
-        Only functions that take no arguments and have no closure
-        are supported.
+    def call(self, callable, /, *args, **kwargs):
+        """Call the object in the interpreter with given args/kwargs.
 
-        The return value is discarded.
+        Nearly all callables, args, kwargs, and return values are
+        supported.  All "shareable" objects are supported, as are
+        "stateless" functions (meaning non-closures that do not use
+        any globals).  This method will fall back to pickle.
 
         If the callable raises an exception then the error display
-        (including full traceback) is send back between the interpreters
+        (including full traceback) is sent back between the interpreters
         and an ExecutionFailed exception is raised, much like what
         happens with Interpreter.exec().
         """
-        # XXX Support args and kwargs.
-        # XXX Support arbitrary callables.
-        # XXX Support returning the return value (e.g. via pickle).
-        excinfo = _interpreters.call(self._id, callable, restrict=True)
-        if excinfo is not None:
-            raise ExecutionFailed(excinfo)
+        return self._call(callable, args, kwargs)
 
-    def call_in_thread(self, callable, /):
+    def call_in_thread(self, callable, /, *args, **kwargs):
         """Return a new thread that calls the object in the interpreter.
 
         The return value and any raised exception are discarded.
         """
-        def task():
-            self.call(callable)
-        t = threading.Thread(target=task)
+        t = threading.Thread(target=self._call, args=(callable, args, kwargs))
         t.start()
         return t

Lib/test/test_ast/test_ast.py

@@ -1543,18 +1543,42 @@ def check_text(code, empty, full, **kwargs):
             full="MatchSingleton(value=None)",
         )
 
+        check_node(
+            ast.MatchSingleton(value=[]),
+            empty="MatchSingleton(value=[])",
+            full="MatchSingleton(value=[])",
+        )
+
         check_node(
             ast.Constant(value=None),
             empty="Constant(value=None)",
             full="Constant(value=None)",
         )
 
+        check_node(
+            ast.Constant(value=[]),
+            empty="Constant(value=[])",
+            full="Constant(value=[])",
+        )
+
         check_node(
             ast.Constant(value=''),
             empty="Constant(value='')",
             full="Constant(value='')",
         )
 
+        check_node(
+            ast.Interpolation(value=ast.Constant(42), str=None, conversion=-1),
+            empty="Interpolation(value=Constant(value=42), str=None, conversion=-1)",
+            full="Interpolation(value=Constant(value=42), str=None, conversion=-1)",
+        )
+
+        check_node(
+            ast.Interpolation(value=ast.Constant(42), str=[], conversion=-1),
+            empty="Interpolation(value=Constant(value=42), str=[], conversion=-1)",
+            full="Interpolation(value=Constant(value=42), str=[], conversion=-1)",
+        )
+
         check_text(
             "def a(b: int = 0, *, c): ...",
             empty="Module(body=[FunctionDef(name='a', args=arguments(args=[arg(arg='b', annotation=Name(id='int', ctx=Load()))], kwonlyargs=[arg(arg='c')], kw_defaults=[None], defaults=[Constant(value=0)]), body=[Expr(value=Constant(value=Ellipsis))])])",