pythongh-128384: Use a context variable for warnings.catch_warnings (pythongh-130010)

Make `warnings.catch_warnings()` use a context variable for holding
the warning filtering state if the `sys.flags.context_aware_warnings`
flag is set to true.  This makes using the context manager thread-safe in
multi-threaded programs.

Add the `sys.flags.thread_inherit_context` flag.  If true, starting a new
thread with `threading.Thread` will use a copy of the context
from the caller of `Thread.start()`.

Both these flags are set to true by default for the free-threaded build
and false for the default build.

Move the Python implementation of warnings.py into _py_warnings.py.

Make _contextvars a builtin module.

Co-authored-by: Kumar Aditya <[email protected]>

Author: 2 people

Doc/howto/free-threading-python.rst

@@ -152,3 +152,33 @@ to re-enable it in a thread-safe way in the 3.14 release.  This overhead is
 expected to be reduced in upcoming Python release.   We are aiming for an
 overhead of 10% or less on the pyperformance suite compared to the default
 GIL-enabled build.
+
+
+Behavioral changes
+==================
+
+This section describes CPython behavioural changes with the free-threaded
+build.
+
+
+Context variables
+-----------------
+
+In the free-threaded build, the flag :data:`~sys.flags.thread_inherit_context`
+is set to true by default which causes threads created with
+:class:`threading.Thread` to start with a copy of the
+:class:`~contextvars.Context()` of the caller of
+:meth:`~threading.Thread.start`.  In the default GIL-enabled build, the flag
+defaults to false so threads start with an
+empty :class:`~contextvars.Context()`.
+
+
+Warning filters
+---------------
+
+In the free-threaded build, the flag :data:`~sys.flags.context_aware_warnings`
+is set to true by default.  In the default GIL-enabled build, the flag defaults
+to false.  If the flag is true then the :class:`warnings.catch_warnings`
+context manager uses a context variable for warning filters.  If the flag is
+false then :class:`~warnings.catch_warnings` modifies the global filters list,
+which is not thread-safe.  See the :mod:`warnings` module for more details.

Doc/library/decimal.rst

@@ -1884,13 +1884,20 @@ the current thread.
 
 If :func:`setcontext` has not been called before :func:`getcontext`, then
 :func:`getcontext` will automatically create a new context for use in the
-current thread.
-
-The new context is copied from a prototype context called *DefaultContext*. To
-control the defaults so that each thread will use the same values throughout the
-application, directly modify the *DefaultContext* object. This should be done
-*before* any threads are started so that there won't be a race condition between
-threads calling :func:`getcontext`. For example::
+current thread.  New context objects have default values set from the
+:data:`decimal.DefaultContext` object.
+
+The :data:`sys.flags.thread_inherit_context` flag affects the context for
+new threads.  If the flag is false, new threads will start with an empty
+context.  In this case, :func:`getcontext` will create a new context object
+when called and use the default values from *DefaultContext*.  If the flag
+is true, new threads will start with a copy of context from the caller of
+:meth:`threading.Thread.start`.
+
+To control the defaults so that each thread will use the same values throughout
+the application, directly modify the *DefaultContext* object. This should be
+done *before* any threads are started so that there won't be a race condition
+between threads calling :func:`getcontext`. For example::
 
    # Set applicationwide defaults for all threads about to be launched
    DefaultContext.prec = 12

Doc/library/sys.rst

@@ -535,7 +535,8 @@ always available. Unless explicitly noted otherwise, all variables are read-only
 .. data:: flags
 
    The :term:`named tuple` *flags* exposes the status of command line
-   flags. The attributes are read only.
+   flags.  Flags should only be accessed only by name and not by index.  The
+   attributes are read only.
 
    .. list-table::
 
@@ -594,6 +595,18 @@ always available. Unless explicitly noted otherwise, all variables are read-only
       * - .. attribute:: flags.warn_default_encoding
         - :option:`-X warn_default_encoding <-X>`
 
+      * - .. attribute:: flags.gil
+        - :option:`-X gil <-X>` and :envvar:`PYTHON_GIL`
+
+      * - .. attribute:: flags.thread_inherit_context
+        - :option:`-X thread_inherit_context <-X>` and
+          :envvar:`PYTHON_THREAD_INHERIT_CONTEXT`
+
+      * - .. attribute:: flags.context_aware_warnings
+        - :option:`-X thread_inherit_context <-X>` and
+          :envvar:`PYTHON_CONTEXT_AWARE_WARNINGS`
+
+
    .. versionchanged:: 3.2
       Added ``quiet`` attribute for the new :option:`-q` flag.
 
@@ -620,6 +633,15 @@ always available. Unless explicitly noted otherwise, all variables are read-only
    .. versionchanged:: 3.11
       Added the ``int_max_str_digits`` attribute.
 
+   .. versionchanged:: 3.13
+      Added the ``gil`` attribute.
+
+   .. versionchanged:: 3.14
+      Added the ``thread_inherit_context`` attribute.
+
+   .. versionchanged:: 3.14
+      Added the ``context_aware_warnings`` attribute.
+
 
 .. data:: float_info
 

Doc/library/threading.rst

@@ -334,7 +334,7 @@ since it is impossible to detect the termination of alien threads.
 
 
 .. class:: Thread(group=None, target=None, name=None, args=(), kwargs={}, *, \
-                  daemon=None)
+                  daemon=None, context=None)
 
    This constructor should always be called with keyword arguments.  Arguments
    are:
@@ -359,6 +359,16 @@ since it is impossible to detect the termination of alien threads.
    If ``None`` (the default), the daemonic property is inherited from the
    current thread.
 
+   *context* is the :class:`~contextvars.Context` value to use when starting
+   the thread.  The default value is ``None`` which indicates that the
+   :data:`sys.flags.thread_inherit_context` flag controls the behaviour.  If
+   the flag is true, threads will start with a copy of the context of the
+   caller of :meth:`~Thread.start`.  If false, they will start with an empty
+   context.  To explicitly start with an empty context, pass a new instance of
+   :class:`~contextvars.Context()`.  To explicitly start with a copy of the
+   current context, pass the value from :func:`~contextvars.copy_context`. The
+   flag defaults true on free-threaded builds and false otherwise.
+
    If the subclass overrides the constructor, it must make sure to invoke the
    base class constructor (``Thread.__init__()``) before doing anything else to
    the thread.
@@ -369,6 +379,9 @@ since it is impossible to detect the termination of alien threads.
    .. versionchanged:: 3.10
       Use the *target* name if *name* argument is omitted.
 
+   .. versionchanged:: 3.14
+      Added the *context* parameter.
+
    .. method:: start()
 
       Start the thread's activity.

Doc/library/warnings.rst

@@ -324,11 +324,13 @@ the warning using the :class:`catch_warnings` context manager::
 While within the context manager all warnings will simply be ignored. This
 allows you to use known-deprecated code without having to see the warning while
 not suppressing the warning for other code that might not be aware of its use
-of deprecated code.  Note: this can only be guaranteed in a single-threaded
-application. If two or more threads use the :class:`catch_warnings` context
-manager at the same time, the behavior is undefined.
+of deprecated code.
 
+    .. note::
 
+        See :ref:`warning-concurrent-safe` for details on the
+        concurrency-safety of the :class:`catch_warnings` context manager when
+        used in programs using multiple threads or async functions.
 
 .. _warning-testing:
 
@@ -364,10 +366,13 @@ the warning has been cleared.
 Once the context manager exits, the warnings filter is restored to its state
 when the context was entered. This prevents tests from changing the warnings
 filter in unexpected ways between tests and leading to indeterminate test
-results. The :func:`showwarning` function in the module is also restored to
-its original value.  Note: this can only be guaranteed in a single-threaded
-application. If two or more threads use the :class:`catch_warnings` context
-manager at the same time, the behavior is undefined.
+results.
+
+    .. note::
+
+        See :ref:`warning-concurrent-safe` for details on the
+        concurrency-safety of the :class:`catch_warnings` context manager when
+        used in programs using multiple threads or async functions.
 
 When testing multiple operations that raise the same kind of warning, it
 is important to test them in a manner that confirms each operation is raising
@@ -615,12 +620,71 @@ Available Context Managers
 
     .. note::
 
-        The :class:`catch_warnings` manager works by replacing and
-        then later restoring the module's
-        :func:`showwarning` function and internal list of filter
-        specifications.  This means the context manager is modifying
-        global state and therefore is not thread-safe.
+        See :ref:`warning-concurrent-safe` for details on the
+        concurrency-safety of the :class:`catch_warnings` context manager when
+        used in programs using multiple threads or async functions.
+
 
     .. versionchanged:: 3.11
 
         Added the *action*, *category*, *lineno*, and *append* parameters.
+
+
+.. _warning-concurrent-safe:
+
+Concurrent safety of Context Managers
+-------------------------------------
+
+The behavior of :class:`catch_warnings` context manager depends on the
+:data:`sys.flags.context_aware_warnings` flag.  If the flag is true, the
+context manager behaves in a concurrent-safe fashion and otherwise not.
+Concurrent-safe means that it is both thread-safe and safe to use within
+:ref:`asyncio coroutines <coroutine>` and tasks.  Being thread-safe means
+that behavior is predictable in a multi-threaded program.  The flag defaults
+to true for free-threaded builds and false otherwise.
+
+If the :data:`~sys.flags.context_aware_warnings` flag is false, then
+:class:`catch_warnings` will modify the global attributes of the
+:mod:`warnings` module.  This is not safe if used within a concurrent program
+(using multiple threads or using asyncio coroutines).  For example, if two
+or more threads use the :class:`catch_warnings` class at the same time, the
+behavior is undefined.
+
+If the flag is true, :class:`catch_warnings` will not modify global
+attributes and will instead use a :class:`~contextvars.ContextVar` to
+store the newly established warning filtering state.  A context variable
+provides thread-local storage and it makes the use of :class:`catch_warnings`
+thread-safe.
+
+The *record* parameter of the context handler also behaves differently
+depending on the value of the flag.  When *record* is true and the flag is
+false, the context manager works by replacing and then later restoring the
+module's :func:`showwarning` function.  That is not concurrent-safe.
+
+When *record* is true and the flag is true, the :func:`showwarning` function
+is not replaced.  Instead, the recording status is indicated by an internal
+property in the context variable.  In this case, the :func:`showwarning`
+function will not be restored when exiting the context handler.
+
+The :data:`~sys.flags.context_aware_warnings` flag can be set the :option:`-X
+context_aware_warnings<-X>` command-line option or by the
+:envvar:`PYTHON_CONTEXT_AWARE_WARNINGS` environment variable.
+
+    .. note::
+
+        It is likely that most programs that desire thread-safe
+        behaviour of the warnings module will also want to set the
+        :data:`~sys.flags.thread_inherit_context` flag to true.  That flag
+        causes threads created by :class:`threading.Thread` to start
+        with a copy of the context variables from the thread starting
+        it.  When true, the context established by :class:`catch_warnings`
+        in one thread will also apply to new threads started by it.  If false,
+        new threads will start with an empty warnings context variable,
+        meaning that any filtering that was established by a
+        :class:`catch_warnings` context manager will no longer be active.
+
+.. versionchanged:: 3.14
+
+   Added the :data:`sys.flags.context_aware_warnings` flag and the use of a
+   context variable for :class:`catch_warnings` if the flag is true.  Previous
+   versions of Python acted as if the flag was always set to false.

Doc/using/cmdline.rst

@@ -639,6 +639,23 @@ Miscellaneous options
 
      .. versionadded:: 3.13
 
+   * :samp:`-X thread_inherit_context={0,1}` causes :class:`~threading.Thread`
+     to, by default, use a copy of context of of the caller of
+     ``Thread.start()`` when starting.  Otherwise, threads will start
+     with an empty context.  If unset, the value of this option defaults
+     to ``1`` on free-threaded builds and to ``0`` otherwise.  See also
+     :envvar:`PYTHON_THREAD_INHERIT_CONTEXT`.
+
+     .. versionadded:: 3.14
+
+   * :samp:`-X context_aware_warnings={0,1}` causes the
+     :class:`warnings.catch_warnings` context manager to use a
+     :class:`~contextvars.ContextVar` to store warnings filter state.  If
+     unset, the value of this option defaults to ``1`` on free-threaded builds
+     and to ``0`` otherwise.  See also :envvar:`PYTHON_CONTEXT_AWARE_WARNINGS`.
+
+     .. versionadded:: 3.14
+
    It also allows passing arbitrary values and retrieving them through the
    :data:`sys._xoptions` dictionary.
 
@@ -1241,6 +1258,26 @@ conflict.
 
    .. versionadded:: 3.13
 
+.. envvar:: PYTHON_THREAD_INHERIT_CONTEXT
+
+   If this variable is set to ``1`` then :class:`~threading.Thread` will,
+   by default, use a copy of context of of the caller of ``Thread.start()``
+   when starting.  Otherwise, new threads will start with an empty context.
+   If unset, this variable defaults to ``1`` on free-threaded builds and to
+   ``0`` otherwise.  See also :option:`-X thread_inherit_context<-X>`.
+
+   .. versionadded:: 3.14
+
+.. envvar:: PYTHON_CONTEXT_AWARE_WARNINGS
+
+   If set to ``1`` then the :class:`warnings.catch_warnings` context
+   manager will use a :class:`~contextvars.ContextVar` to store warnings
+   filter state.  If unset, this variable defaults to ``1`` on
+   free-threaded builds and to ``0`` otherwise.  See :option:`-X
+   context_aware_warnings<-X>`.
+
+   .. versionadded:: 3.14
+
 Debug-mode variables
 ~~~~~~~~~~~~~~~~~~~~
 

Include/cpython/initconfig.h

@@ -180,6 +180,8 @@ typedef struct PyConfig {
     int use_frozen_modules;
     int safe_path;
     int int_max_str_digits;
+    int thread_inherit_context;
+    int context_aware_warnings;
 #ifdef __APPLE__
     int use_system_logger;
 #endif

Include/internal/pycore_global_objects_fini_generated.h

@@ -244,6 +244,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(_feature_version)
         STRUCT_FOR_ID(_field_types)
         STRUCT_FOR_ID(_fields_)
+        STRUCT_FOR_ID(_filters)
         STRUCT_FOR_ID(_finalizing)
         STRUCT_FOR_ID(_find_and_load)
         STRUCT_FOR_ID(_fix_up_module)

Include/internal/pycore_global_strings.h

@@ -582,6 +582,7 @@ struct _warnings_runtime_state {
     PyObject *default_action; /* String */
     _PyRecursiveMutex lock;
     long filters_version;
+    PyObject *context;
 };
 
 struct _Py_mem_interp_free_queue {