ENH: Add ITK_PYTHON_RELEASE_GIL option and SWIG -threads flag

[thewtex]

• Added cmake option ITK_PYTHON_RELEASE_GIL (default ON) in WrappingOptions.cmake
• Modified Python CMakeLists.txt to add -threads flag to SWIG when option is enabled
• Created test_gil_release.py to verify GIL is released during ITK operations
• Added test to CMakeLists.txt

Backport from main for ITK 5.4.

[greptile-apps]

Greptile Summary

This PR backports the ITK_PYTHON_RELEASE_GIL CMake option (default ON) from main to release-5.4, adding the SWIG -threads flag so the Python GIL is released during ITK C++ operations, and includes a threading-concurrency test to verify the behavior.

• PythonGILReleaseTest is registered unconditionally in Tests/CMakeLists.txt but will always fail when ITK_PYTHON_RELEASE_GIL=OFF since the GIL won't be released; the test registration should be guarded with if(ITK_PYTHON_RELEASE_GIL).

Confidence Score: 4/5

Safe to merge after adding the ITK_PYTHON_RELEASE_GIL guard around the test registration.

One P1 issue: the GIL release test is unconditionally registered and will fail for any build with ITK_PYTHON_RELEASE_GIL=OFF; the fix is a one-line CMake guard.

Wrapping/Generators/Python/Tests/CMakeLists.txt — needs ITK_PYTHON_RELEASE_GIL guard around PythonGILReleaseTest.

Important Files Changed

Filename	Overview
Wrapping/WrappingOptions.cmake	Adds cmake_dependent_option ITK_PYTHON_RELEASE_GIL (default ON when ITK_WRAP_PYTHON) — correct usage and placement.
Wrapping/Generators/Python/CMakeLists.txt	Conditionally appends -threads to SWIG command based on ITK_PYTHON_RELEASE_GIL; correct scoping with unset cleanup.
Wrapping/Generators/Python/Tests/CMakeLists.txt	PythonGILReleaseTest registered unconditionally — will fail when ITK_PYTHON_RELEASE_GIL=OFF.
Wrapping/Generators/Python/Tests/test_gil_release.py	GIL release test using timing heuristics; logic is reasonable but timing-based pass/fail can be flaky on constrained machines.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[CMake Configure] --> B{ITK_WRAP_PYTHON?}
    B -- No --> C[ITK_PYTHON_RELEASE_GIL forced OFF]
    B -- Yes --> D{ITK_PYTHON_RELEASE_GIL
default ON}
    D -- ON --> E[SWIG -threads flag added]
    D -- OFF --> F[No -threads flag]
    E --> G[Bindings release GIL
on each C++ call]
    F --> H[Bindings hold GIL
during C++ calls]
    G --> I{BUILD_TESTING?}
    H --> I
    I -- Yes --> J[PythonGILReleaseTest
registered unconditionally]
    J --> K[Test asserts 2x speedup
across 4 threads]
    K --> L{GIL released?}
    L -- Yes --> M[PASS]
    L -- No --> N[FAIL - even for valid OFF config]

Loading

_{Reviews (1): Last reviewed commit: "ENH: Add ITK_PYTHON_RELEASE_GIL option a..." | Re-trigger Greptile}

[blowekamp]

The Python GIL may need to be required on call back such as in ITK commands, and I think ITK python has a filter callback to implement a filter in python too?

Example: https://github.com/SimpleITK/SimpleITK/blob/40e9920342a2587a60490f93ec894341660cdcf3/Wrapping/Python/sitkPyCommand.cxx#L33

[thewtex]

@blowekamp yes, that's here: https://github.com/InsightSoftwareConsortium/ITK/blob/release-5.4/Wrapping/Generators/Python/PyUtils/itkPyCommand.cxx

[hjmjohnson]

Detailed Review: GIL Release via SWIG -threads

Summary

Tested across Python 3.10-3.14 with comprehensive gotcha tests. The PR's approach is fundamentally sound — itkPyCommand.cxx already correctly re-acquires the GIL for Python callbacks. However, there are several concerns that should be addressed before merging.

Test Results (Python 3.10-3.14, all versions identical)

Test	Result	Notes
Callback during filter execution	PASS	PyGILState_Ensure in itkPyCommand works correctly
Concurrent callbacks from 4 threads	PASS	No crashes, no data corruption
Object destruction under concurrent access	PASS	SmartPointer prevents premature free
Concurrent image read/write (4 threads)	PASS	Separate images, no races
Signal handling while GIL released	PASS	SIGUSR1 delivered correctly
GIL reacquisition deadlock detection	PASS	No deadlock in 10s timeout
Stress test (200 create/destroy, 4 threads)	PASS	No crashes

Known Gotchas Found in Research

1. SWIG director methods don't auto-acquire GIL (CRITICAL)

When C++ calls back into Python via a SWIG director (virtual method override in Python), the generated code does NOT acquire the GIL before calling the Python method. This causes segfaults. (swig/swig#3091)

ITK mitigates this via manual PyGILState_Ensure in itkPyCommand.cxx, but any OTHER Python callback path (custom directors, Python-implemented filters) is at risk.

Recommendation: Add a comment/warning in the PR description that any new Python callback code MUST use PyGILState_Ensure/PyGILState_Release.

2. C++ exceptions + thread state = potential crash

When -threads is enabled, SWIG calls PyEval_SaveThread() before C++ and PyEval_RestoreThread() after. If C++ throws an exception, the thread state restoration may not happen correctly in all SWIG versions, leading to crashes.

Tested: Exception propagation works correctly in current ITK builds (RuntimeError from file-not-found propagates cleanly). This appears fixed in modern SWIG versions.

3. Process exit race condition

If the main thread calls sys.exit() while a C++ filter is still running in another thread (GIL released), the interpreter cleanup can trigger std::terminate.

Recommendation: Document that users should ensure all ITK operations complete before process exit. This is inherent to any GIL-releasing code, not specific to this PR.

4. SwigPyIterator::value() called without GIL

The generated wrapper calls SwigPyIterator::value() (which returns PyObject*) inside a SWIG_PYTHON_THREAD_BEGIN_ALLOW / END_ALLOW block. This means Python object creation happens without the GIL — undefined behavior.

In practice this hasn't caused issues because ITK Python rarely uses SWIG iterator objects directly, but it's a latent bug.

5. @blowekamp's concern: Python filter callbacks

As noted, itkPyCommand.cxx correctly handles this with PyGILState_Ensure. The callback re-acquires the GIL, does Python work, releases it, and returns to C++. Tested and verified — no deadlocks, no crashes.

What the PR Does Right

• Uses CMake option (ITK_PYTHON_RELEASE_GIL, default ON) for easy opt-out
• Is a non-invasive change (just adds -threads flag to SWIG command)
• itkPyCommand.cxx already has correct GIL handling for callbacks
• Test verifies concurrent execution actually occurs

Suggestions

1. The test is fragile — timing-based parallel speedup detection can fail on slow/loaded CI runners. Consider a more deterministic test (e.g., verify that a callback can run while another thread's filter is executing).

2. Add %nothread for known-problematic wrappers — consider using %nothread on any methods that are known to throw exceptions frequently, to avoid the exception+thread-state edge case.

3. Document the callback requirement — add a note in the migration guide or docs that any Python code that receives C++ callbacks must ensure GIL handling.

4. Consider defaulting to OFF for 5.4 backport — this is a behavioral change. For a patch release, OFF by default (opt-in) would be safer.

[hjmjohnson]

Comprehensive GIL release safety test results

Created test_gil_release_safety.py (commit 55eeaa9ab0 on branch gil-release-tests) covering 8 known SWIG `-threads` gotchas. Each test references the SWIG issue or Python doc that motivated it.

Results: ITK_PYTHON_RELEASE_GIL=ON

Python	Tests	Passed	Failed	Skipped
3.10.20	15	15	0	0
3.11.15	15	15	0	0
3.12.13	15	15	0	0
3.13.12	15	15	0	0
3.14.3	15	15	0	0

Results: ITK_PYTHON_RELEASE_GIL=OFF

Python	Tests	Passed	Failed	Skipped
3.13.12	15	15	0	0

Tests pass regardless of the setting because they test safety invariants (no crashes, no deadlocks, no data corruption) rather than concurrency speedup.

Tests

#	What	Gotcha tested	Reference
1	Callback during filter execution	Director GIL re-acquisition	swig/swig#3091, #2670
2	Concurrent callbacks (4 threads)	PyGILState re-entrancy	Python C API docs
3	Exception propagation (single + 4 threads)	SaveThread/RestoreThread + throw	SWIG-devel thread
4	Object destruction under concurrency	Refcount under GIL release	Python refcounting docs
5	Concurrent image processing (4 threads)	Data races in ITK internals	ITK MultiThreaderBase
6	Signal handling while GIL released	SIGINT/SIGUSR1 delivery	Python signal docs
7	GIL reacquisition deadlock	Callback → PyGILState_Ensure	Python GIL state docs
8	Stress test (200 cycles, 4 threads)	Intermittent crashes	swig/swig#2396, #3121

The test file is ready to be included in this PR or submitted as a separate PR.

[hjmjohnson]

This is great!