Core: Fix background thread leak in ScanTaskIterable

[sejal-gupta-ksolves]

Closes: #16758

Problem

When downstream query engines (such as StarRocks, Trino, or Spark) cancel or abort a REST table scan early due to client disconnects, timeouts, or query limits, they trigger the cleanup sequence on the outer execution container.

In Apache Iceberg, ScanTaskIterable.close() was implemented as an empty no-op method. Because this outer close() call failed to cascade the shutdown signal to the underlying data structures:

• The internal shutdown state atomic flag remained false.
• Background PlanTaskWorker threads continued running indefinitely.
• Once the internal taskQueue reached its 1000 item capacity limit, all active worker threads became permanently deadlocked inside offerWithTimeout(), leading to thread pool exhaustion on the engine coordinator side.

Solution

1. Implemented State Tracking and Cleanup: Added thread-safe execution barriers inside ScanTaskIterable.close() utilizing shutdown.compareAndSet(false, true).
2. Queue Eviction Matrix: Updated the close block to explicitly flush taskQueue, planTasks, and initialFileScanTasks lists upon termination. This allows background threads stuck in an offer wait cycle to instantly unblock, evaluate the flipped shutdown state, and exit gracefully.
3. Decoupled Iterator Lifecycle: Refactored the internal ScanTasksIterator.close() block to eliminate redundant code duplication, rewriting it to delegate its cleanup tasks straight up to ScanTaskIterable.this.close(). This ensures unified thread termination safety across all potential entry points.
4. Regression Test Addition: Designed and integrated TestScanTaskIterableLeak under the org.apache.iceberg.rest test package, proving that active planning thread allocations successfully scale back down to 0 upon premature termination.

Verification Testing

# 1. Clean format code using Spotless rules
./gradlew spotlessApply

# 2. Run static quality analysis lint checks on modified packages
./gradlew :iceberg-core:compileJava :iceberg-core:compileTestJava

# 3. Verify the core build pass and execute the regression test case
./gradlew :iceberg-core:test --tests "org.apache.iceberg.rest.TestScanTaskIterableLeak" --info

[chenwyi2]

At present, In StarRocks FE deployment, when a query is cancelled or times out during the DeployScanRanges / scan-planning phase, close() can be invoked?

[sejal-gupta-ksolves]

@chenwyi2 Yes, close() will be invoked automatically by StarRocks FE without requiring any downstream code changes.

During query planning and range deployment, StarRocks processes the file splits via Iceberg's CloseableIterable scan stream. When a query is cancelled or times out, StarRocks' coordinator explicitly cascades a .close() signal down to this execution stream wrapper to free resources.

Because this patch unifies the cleanup paths by having the iterator directly delegate to ScanTaskIterable.this.close(), any termination signal triggered by StarRocks will instantly flip the shutdown barrier, empty the planning queues, and gracefully exit the blocked background worker threads.

[chenwyi2]

but when i test the case, kill the query and use jstack, see the thread, ScanTaskIterable is still not closed.

2026-06-16 16:53:22.901+08:00 INFO (starrocks-mysql-nio-pool-0|166) [StmtExecutor] execute Exception, sql: SELECT * FROM xxx, error: killed manually: KILL QUERY 16777572

"iceberg_catalog-sr-iceberg-worker-pool-0" #775 daemon prio=5 os_prio=0 cpu=1518.93ms elapsed=33.72s tid=0x00007f47883b35e0 nid=0x85ccb runnable [0x00007f47709fe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.16/Native Method)
- parking to wait for <0x000010001f3635e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.16/LockSupport.java:252)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@17.0.16/AbstractQueuedSynchronizer.java:1679)
at java.util.concurrent.LinkedBlockingQueue.offer(java.base@17.0.16/LinkedBlockingQueue.java:378)
at org.apache.iceberg.rest.ScanTaskIterable$PlanTaskWorker.offerWithTimeout(ScanTaskIterable.java:161)
at org.apache.iceberg.rest.ScanTaskIterable$PlanTaskWorker.processPlanTask(ScanTaskIterable.java:224)
at org.apache.iceberg.rest.ScanTaskIterable$PlanTaskWorker.run(ScanTaskIterable.java:139)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.16/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.16/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.16/Thread.java:840)

it seems that when killing the query, close() is not invoked automatically by StarRocks FE ?

maybe i should cherry-pick StarRocks/starrocks#68678 this pr

[sejal-gupta-ksolves]

@chenwyi2 Thank you for the jstack trace—you are completely right.

The thread dump confirms that StarRocks FE is not calling close() during a manual KILL QUERY, leaving the worker threads permanently blocked on taskQueue.offer(...).

This Iceberg PR provides the framework to clean up those background pools, but it depends on the engine to trigger it. Cherry-picking that StarRocks PR to ensure .close() is explicitly invoked on query cancellation is exactly the right downstream fix. Once both sides are patched, the thread leak will be fully resolved!

[singhpk234]

@chenwyi2 seems like its explictly calls iterator close
https://github.com/StarRocks/starrocks/pull/68678/changes#diff-6c25423f964e0e7910311e2cb6fba148f1def8754ce9e8a292e1c906ec3f14c2R1129

seems like starrocks is already calling the iterator close... i believe closing iterator is correct thing to do .