fix(scheduler): defer loading dag_run.conf to avoid memory spike with large run config (#71267) [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT]

[waterWang]

Problem

When a DAG with dynamic task mapping is triggered with a large run config (512KB–2MB JSON), the scheduler memory spikes by 5–6× during scheduling of the dynamic tasks. Once the dagrun completes, the memory returns to normal.

Reported in #71267.

Root Cause

The scheduler's critical section query at _executable_task_instances_to_queued uses joinedload(TI.dag_run) (line 808), which eagerly loads the full dag_run row — including the conf column — for every task instance in the SQL result set.

When 500+ mapped task instances all share the same dag_run, the joined SQL result set carries the 2MiB conf column 500+ times. This results in approximately 1 GiB of redundant conf data being transferred from the database and materialized in Python memory, causing the memory spike.

Fix

Add .defer(DagRun.conf) to the eager-load chain. The scheduler never reads dag_run.conf in the critical section, so deferring it eliminates the bloat while keeping all other columns and the secondary selectinload of created_dag_version available.

Verification

• SQLAlchemy standalone test confirms defer correctly excludes the conf column from the SELECT statement
• The existing test test_executable_task_instances_no_per_ti_queries still passes (no N+1 regression)
• No query count changes — the defer only affects which columns are loaded, not how many queries are issued

Closes #71267

[Prab-27]

Thanks @waterWang !
would you please fix the title of this PR - you don't need to add any issue number or hash here
and you could remove fix ():

[kaxil]

Closing this as part of a cleanup of a large batch of PRs opened in quick succession from this account.

18 PRs have been opened here in the past two weeks and none have merged. Several show signs of being generated and submitted without review: #71432 and #71433 are the same change across the same five files, opened two minutes apart, and several titles carry a leaked agent identifier that other contributors already flagged as garbled text on #70629 and #71322.

Airflow is maintained by volunteers. Every PR costs reviewer time and CI capacity, so a high volume of unvetted submissions has a real cost to the project.

You are welcome to keep contributing. Please open one change at a time, run it locally against the tests, and read the contributors' guide before submitting. If you think a specific change here is correct, comment with the reasoning and a maintainer can reopen it.