[Performance] Mitigate redundant Reconciles

[vicentefb]

This issue/investigation is derived from having opened these two PRs #508 and #509 where we reduce the number of conflicts by using .Patch() instead of .Update() to update the status of the sandbox and sandboxclaim resources.

During high-concurrency scale testing (kube-api-qps=1000, sandbox-concurrent-workers=400), we observed a high volume of 409 Conflict: the object has been modified errors across both the Sandbox and SandboxClaim controllers.

The SandboxReconciler uses `Owns(&corev1.Pod{}) to watch for changes on its child Pods. When a new sandbox pod is provisioned, the Kubelet rapidly transitions the pod through multiple setup phases (Scheduled -> Initialized -> PodIP assigned -> ContainersReady -> Running) in a fraction of a second.

The following test run shows what's happening in baseline main (without using .Patch() in sandbox or sandboxclaim controller).

Test Conditions: BURST=1, WARMPOOL=2 (Sandbox Status Unpatched, SandboxClaim Status Unpatched)

Agent Sandbox Claim Startup latency (ms):

• "StartupLatency50": 653.8145060214703,
• "StartupLatency90": 1150.0000000000002,
• "StartupLatency99": 2364.999999999999

API Server View (HTTP Audit Logs): This trace isolates the exact moment the Kubelet spam causes the controllers to repeatedly crash into each other over a single second.

Step	Delta (ms)	Method	Target Resource	Result / Notes
1	0	PATCH	sandboxclaims/agent-claim-4	🟢 Start: Test Runner injects Claim
2	17	UPDATE	sandboxes/warmpool-0-6qkhx	🟢 Claim Controller adopts Sandbox
3	28	DELETE	networkpolicies/...	🟡 404 Not Found
4	32	CREATE	events/agent-claim-4...	🟢 K8s records adoption event
5	42	CREATE	sandboxes	🟢 WarmPool orders replacement
6	45	UPDATE	sandboxclaims/agent-claim-4/status	🟢 Claim Controller updates status
7	56	DELETE	networkpolicies/...	🟡 404 Not Found (Retry)
8	58	PATCH	sandboxwarmpools/warmpool-0/status	🟢 WarmPool updates pool status
9	67	UPDATE	sandboxclaims/agent-claim-4/status	🔴 409 Conflict (Claim collides w/ itself)
10	68	UPDATE	pods/warmpool-0-6qkhx	🟢 Sandbox Controller updates pod
11	71	PATCH	sandboxes/warmpool-0-qcblt	🟢 WarmPool configures replacement
12	77	DELETE	networkpolicies/...	🟡 404 Not Found (Retry)
13	85	PATCH	sandboxwarmpools/warmpool-0/status	🟢 WarmPool updates pool status
14	90	PATCH	sandboxes/warmpool-0-6qkhx/status	🟢 SUCCESS: Sandbox Status patched
15	97	PATCH	sandboxwarmpools/warmpool-0/status	🟢 WarmPool updates pool status
16	111	DELETE	networkpolicies/...	🟡 404 Not Found (Retry)
17	127	CREATE	pods/warmpool-0-qcblt	🟢 Sandbox Controller provisions pod
18	129	UPDATE	sandboxclaims/agent-claim-4/status	🟢 Retry Succeeds: Claim status resolves
19	132	UPDATE	pods/warmpool-0-6qkhx	🟢 Sandbox Controller updates pod
20	147	PATCH	sandboxes/warmpool-0-6qkhx/status	🟢 SUCCESS: Sandbox Status patched again
21	150	DELETE	networkpolicies/...	🟡 404 Not Found (Retry)
22	152	PATCH	sandboxes/warmpool-0-qcblt	🟢 WarmPool configures replacement
23	166	CREATE	services/warmpool-0-qcblt	🟢 Service provisioned
24	191	UPDATE	pods/warmpool-0-6qkhx	🟢 Sandbox Controller updates pod
25	193	PATCH	sandboxes/warmpool-0-qcblt/status	🟢 SUCCESS: New Sandbox Status patched
26	244	UPDATE	pods/warmpool-0-qcblt	🔴 409 Conflict (Sandbox collides w/ Kubelet)
27	264	PATCH	sandboxes/warmpool-0-qcblt/status	🟢 SUCCESS: New Sandbox Status patched again
28	312	UPDATE	pods/warmpool-0-qcblt	🟢 Sandbox Controller updates pod
29	361	UPDATE	pods/warmpool-0-qcblt	🟢 Sandbox Controller updates pod
30	380	PATCH	sandboxes/warmpool-0-qcblt/status	🟢 SUCCESS: New Sandbox Status patched again
31	715	UPDATE	pods/warmpool-0-qcblt	🟢 Sandbox Controller updates pod
32	734	PATCH	sandboxes/warmpool-0-qcblt/status	🟢 SUCCESS: New Sandbox Status patched again
33	758	PATCH	sandboxwarmpools/warmpool-0/status	🟢 WarmPool updates pool status
34	796	UPDATE	pods/warmpool-0-qcblt	🟢 Sandbox Controller updates pod

This trace tracks the reconcileID generated by controller-runtime for a single replacement pod (warmpool-0-qcblt). It proves the Kubelet's rapid status updates are generating excessive Informer notifications, causing the worker thread to execute 7 full loops in under 1 second.

Step	Delta (ms)	Reconcile ID	Controller Action / Event
1	0	9a3bf8f1...	🟢 Loop 1: "Creating a new Pod" & "Creating a new Headless Service"
2	124	ce0ba836...	🟡 Loop 2: "Found Pod" (Woken up by Kubelet Scheduled event)
3	196	ce0ba836...	🔴 CRASH: "failed to update pod... the object has been modified"
4	196	ce0ba836...	🔴 CRASH: "Failed to update sandbox status" (409 Conflict)
5	196	9c2064ce...	🟡 Loop 3: "Found Pod" (Woken up by Kubelet Initialized event)
6	1232	8f2137f8...	🟡 Loop 4: "Found Pod" (Woken up by Kubelet network/IP assignment)
7	1308	258ff9f0...	🟡 Loop 5: "Found Pod" (Woken up by Kubelet Ready event)
-	-	-	(Note: Controller continues to loop for subsequent Kubelet phase shifts)

[vicentefb]

/assign

[vicentefb]

Code in #532

With Predicate only: commit

• "StartupLatency50": 687.5,
• "StartupLatency90": 1825.0000000000005,
• "StartupLatency99": 2432.5

Step	Delta (ms)	Reconcile ID	Event
1	0	f9f84ff3...	🟢 Loop 1: "Creating a new Pod"
2	87	c89334dc...	🟡 Loop 2: "Found Pod" (Triggered by Kubelet Scheduled)
3	138	c89334dc...	🔴 CRASH: "failed to update pod... the object has been modified"
4	583	34278a4c...	🟢 Loop 3: "Found Pod" (Triggered by Kubelet Ready / Running)

Added controller-runtime Predicates to drop the Kubelet noise, successfully reducing wakeups from 7 down to 3. Latency increased becaues during the first valid wakeup (Pod Scheduled), the controller attempted to Update both the Pod labels and the Sandbox Status. Because we were still using Update, it collided with the Kubelet, resulting in a Double Crash (both updates failed). The controller sat idle for ~400ms before retrying, affecting our p90 latency.

With Predicate + Sandbox Patch Status:

• "StartupLatency50": 570.2904641157868,
• "StartupLatency90": 1149.9999999999995,
• "StartupLatency99": 2365

Step	Timestamp (s)	Delta (ms)	Reconcile ID	Controller Action / Event
1	03.218	0	6d784320...	🟢 Loop 1: "Creating a new Pod" & "Headless Service"
2	03.311	93	8714b30a...	🟡 Loop 2: "Found Pod" (Predicate allows 'Scheduled' phase change)
3	03.371	153	8714b30a...	🔴 CRASH: "failed to update pod... the object has been modified"
4	03.377	159	040ce5c7...	🟢 Loop 3: "Found Pod" (Instant Workqueue retry after the crash)
5	03.855	637	e69276e9...	🟢 Loop 4: "Found Pod" (Predicate allows final 'Ready' phase change)

The p90 latency dropped by 700 milliseconds (from 1.8 seconds down to 1.1 seconds).

With Predicate + Sandbox Patch Status + SandboxClaim Patch Status: commit

• "StartupLatency50": 658.4820200024881,
• "StartupLatency90": 1324.0436462788507,
• "StartupLatency99": 2382.4043646278847

This trace proves that all 409 Conflicts have been eradicated from the CRs. The only remaining collision is the collision on the physical Pod.

Step	Delta (ms)	Method	Target Resource	Result / Notes
1	0	CREATE	sandboxes	🟢 WarmPool orders replacement warmpool-0-9h5lr
2	27	PATCH	sandboxes/warmpool-0-9h5lr	🟢 WarmPool configures sandbox
3	74	CREATE	pods/warmpool-0-9h5lr	🟢 Sandbox Controller provisions pod
4	93	PATCH	sandboxes/warmpool-0-9h5lr	🟢 Sandbox Controller updates labels
5	125	PATCH	sandboxes/warmpool-0-9h5lr/status	🟢 SUCCESS: Sandbox Status Patched
6	132	CREATE	services/warmpool-0-9h5lr	🟢 Service provisioned
7	170	UPDATE	pods/warmpool-0-9h5lr	🔴 409 Conflict (Sandbox Controller collides w/ Kubelet)
8	174	PATCH	sandboxes/warmpool-0-9h5lr/status	🟢 SUCCESS: Sandbox Status Patched
9	191	PATCH	sandboxes/warmpool-0-9h5lr/status	🟢 SUCCESS: Sandbox Status Patched
10	239	UPDATE	pods/warmpool-0-9h5lr	🟢 Retry Succeeds: Pod labels applied
11	1300	PATCH	sandboxes/warmpool-0-9h5lr/status	🟢 SUCCESS: Sandbox Status Patched (Ready)

Step	Delta (ms)	Reconcile ID	Controller Action / Event
1	0	b13cfe94...	🟢 Loop 1: "Creating a new Pod" & "Headless Service"
2	98	743b7733...	🟡 Loop 2: "Found Pod" (Predicate allows 'Scheduled' event)
3	165	743b7733...	🔴 CRASH: "failed to update pod... the object has been modified"
4	170	7de34a7b...	🟢 Loop 3: "Found Pod" (Instant Workqueue retry, just 5ms later)
5	1218	c65a9880...	🟢 Loop 4: "Found Pod" (Predicate allows final 'Ready' event)