gh-153568: Pause the GC while converting the AST to Python objects

[pablogsal]

Building the Python object tree out of the C AST allocates thousands of container objects in a burst, which repeatedly trips the garbage collector into traversing the half-built tree. Those nodes cannot be part of a reference cycle until the conversion exposes them, so a collection during the conversion can never free anything of this tree. Pausing the collector around the conversion (restoring the previous state, so a user-disabled GC stays disabled) removes those wasted traversals.

Benchmark (running ast.parse over 8 of the largest stdlib files, 1.3 MB, 20 times per run; pinned cores):

build	time per run	speedup
main	133.6 ms
this PR	120.5 ms	1.11x faster

Executed instructions (stable under machine load, perf stat) drop by 5.6%.

• Issue: Speed up the parser #153568

[maurycy]

My, probably naive, question here is what about free-threading?

There seems to be a single global (per interpret) state:

cpython/Python/gc_free_threading.c

Lines 2527 to 2539 in 9d231cb

	int
	PyGC_Enable(void)
	{
	GCState *gcstate = get_gc_state();
	return _Py_atomic_exchange_int(&gcstate->enabled, 1);
	}
	int
	PyGC_Disable(void)
	{
	GCState *gcstate = get_gc_state();
	return _Py_atomic_exchange_int(&gcstate->enabled, 0);
	}

cpython/Python/gc_free_threading.c

Lines 1656 to 1661 in 9d231cb

	static GCState *
	get_gc_state(void)
	{
	PyInterpreterState *interp = _PyInterpreterState_GET();
	return &interp->gc;
	}

For example: a thread disabled GC (in the window between here) but the parser would enable it again?

[pablogsal]

Yes this will always be racy because is currently not possible to know if this was us or someone else doing it. This is something we need to consider when reviewing this optimization

[maurycy]

Maybe this is a more general problem?

Quick ripgrep reveals:

cpython/Modules/_posixsubprocess.c

Lines 1063 to 1067 in 1b9fe5c

	/* We need to call gc.disable() when we'll be calling preexec_fn */
	if (preexec_fn != Py_None) {
	need_to_reenable_gc = PyGC_Disable();
	}

cpython/Modules/_posixsubprocess.c

Lines 1326 to 1329 in 1b9fe5c

	if (need_to_reenable_gc) {
	PyGC_Enable();
	}

In reverse, maybe even more places would gain from disabling changing the default GC behaviour (disabling it?).