OCPBUGS-77056: Make external cert validation asynchronous (v3 — fix re-admission after secret deletion) by bentito · Pull Request #828 · openshift/router

bentito · 2026-07-29T13:26:53Z

> Draft — waiting on revert of #825 (v2) to land before rebase.

Background

This is the fourth attempt at landing asynchronous external certificate validation. PR #825 (v2) fixed the x509: ECDSA verification failure race conditions from #822 but introduced a new regression: 5 out of 10 payload-aggregate runs failed on RouteExternalCertificate tests.

v2 payload results (5/10 — not 10/10)

The /payload-aggregate run for #825 (results) showed 5 successes, 5 failures. The failing runs had two blocking RouteExternalCertificate test failures:

"the secret is deleted then routes are not reachable" — timed out at 900s
"the secret is updated but RBAC permissions are dropped then routes are not reachable" — timed out at 900s

Notably, zero x509 or ECDSA errors appeared anywhere in the logs — the v2 race condition fixes (DeepCopy, atomic PEM write) eliminated those. The new failure mode is a status transition timeout: the route never reaches Admitted=False and the test polls until the 15-minute timeout.

Root cause: SARCompleted re-admits a deleted-secret route

After DeleteFunc rejects a route (Admitted=False, ValidationFailed), the status update triggers a re-enqueue. The subsequent HandleRoute(Modified) call:

DeepCopies the route (capturing Admitted=False in the local copy)
Calls validate() — SAR check passes (RBAC is still valid, only the secret was deleted)
Calls populateRouteTLSFromSecret() — GetSecret succeeds because the informer cache hasn't propagated the deletion yet
Plugin chain accepts the route
The SARCompleted guard checks the DeepCopied route, sees Admitted=False (no ext-cert reason), and writes SARCompleted — flipping the route back to Admitted=True

The route bounces between Admitted=False (from DeleteFunc) and Admitted=True (from SARCompleted), and the E2E test polls for Admitted=False until timeout.

What's new in this PR (net new vs #825)

One commit on top of the full v2 (#825) changeset:

Fix: check `deletedSecrets` before writing SARCompleted

Before emitting RecordRouteUpdate(SARCompleted), check p.deletedSecrets.Load(key). If the secret has been marked as deleted by the DeleteFunc, skip the write. This prevents re-admission during the informer cache propagation window.

Test: `TestDeletedSecretDoesNotGetReadmitted`

Reproduces the exact failure:

Admits a route with an external cert (SARCompleted written)
Fires DeleteFunc (route rejected with ValidationFailed)
Calls HandleRoute(Modified) while GetSecret still succeeds (cache race)
Asserts the route is NOT re-admitted

Fails on v2 code (2 SARCompleted writes = re-admission), passes after the fix.

Update: root cause found and fixed — synchronous certificate refresh

Post-review, payload-aggregate kept showing the RouteExternalCertificate "secret is updated but RBAC permissions are dropped" test failing on roughly half of the runs, despite passing PR-time tests every time. We iterated through several mitigations narrowing when SARCompleted fires and adding a delayed RBAC re-check after a secret update — these reduced but didn't eliminate the flake.

Root cause: UpdateFunc only wrote a status condition on secret change. The actual certificate refresh (populateRouteTLSFromSecret) only ran when the router's own route-watch happened to redeliver the route as Modified — an indirect round trip through its own status write. Found by locally injecting API-server latency and packet loss into the router's connection (emulating HyperShift's separated control plane), since this never reproduced under CI's PR-time tests or low-latency local runs. Under injected latency/loss, that round trip could be delayed or dropped, leaving a route serving a stale certificate indefinitely until an unrelated later event (e.g. the next secret rotation) happened to reprocess it. This is also the direct answer to @jcmoraisjr's review question about whether the 3s delayed re-check was "time enough" — that goroutine only ever re-ran the SAR check, never the cert refresh, so it provided no protection against this specific failure mode.

Fix: UpdateFunc now calls validate() and populateRouteTLSFromSecret() synchronously and pushes the result through the plugin chain directly, instead of depending on the router observing its own write. The delayed re-check goroutine is unchanged and now purely serves its original purpose: catching RBAC revocations that haven't propagated yet.

Simplification: also investigated increasing writerlease's worker count (1→N), reasoning a single worker could head-of-line-block status writes under high latency. Built a local repro harness (6 routes sharing one secret, matching CI's topology, with tc netem injecting 1s±300ms latency, 10% loss, 2% duplication into the router's API-server connection) and empirically disproved this — increasing workers made no measurable difference, and the failure signature (all 6 routes in a namespace failing or succeeding together) ruled out a per-worker queuing bottleneck. Kept writerlease at its original single worker rather than adding concurrency that wasn't fixing anything.

Validation — same local harness, before vs. after the synchronous-refresh fix, under identical injected latency/loss:

Before: 137/360 checks timed out (38%)
After: 3/360 timed out (0.8%) — residual failures show a distinct, much rarer race between this fix's synchronous path and the router's independent periodic re-validation running concurrently on a different goroutine; tracked separately as a follow-up, not blocking.

CI: /payload-aggregate periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance 10 — 10/10 clean, zero RouteExternalCertificate failures (only pre-existing, unrelated non-blocking flakes elsewhere in the suite).

Test plan

go test -race ./pkg/router/controller/ — all pass, including TestDeletedSecretDoesNotGetReadmitted
make verify and make check — pass
/payload-aggregate periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance 10 — 10/10, zero RouteExternalCertificate failures
Scale test: 2000 ext-cert routes admitted in ~120s (verified: 2000/2000 in 138s, 0 rejections, on a fresh 4.21 cluster with the synchronous-refresh fix)

🤖 Generated with Claude Code

openshift-ci-robot · 2026-07-29T13:27:00Z

@bentito: This pull request references Jira Issue OCPBUGS-77056, which is invalid:

expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Draft — waiting on revert of #825 (v2) to land before rebase.

Background

This is the fourth attempt at landing asynchronous external certificate validation. PR #825 (v2) fixed the x509: ECDSA verification failure race conditions from #822 but introduced a new regression: 5 out of 10 payload-aggregate runs failed on RouteExternalCertificate tests.

v2 payload results (5/10 — not 10/10)

The /payload-aggregate run for #825 (results) showed 5 successes, 5 failures. The failing runs had two blocking RouteExternalCertificate test failures:

"the secret is deleted then routes are not reachable" — timed out at 900s

"the secret is updated but RBAC permissions are dropped then routes are not reachable" — timed out at 900s

Notably, zero x509 or ECDSA errors appeared anywhere in the logs — the v2 race condition fixes (DeepCopy, atomic PEM write) eliminated those. The new failure mode is a status transition timeout: the route never reaches Admitted=False and the test polls until the 15-minute timeout.

Root cause: SARCompleted re-admits a deleted-secret route

After DeleteFunc rejects a route (Admitted=False, ValidationFailed), the status update triggers a re-enqueue. The subsequent HandleRoute(Modified) call:

DeepCopies the route (capturing Admitted=False in the local copy)

Calls validate() — SAR check passes (RBAC is still valid, only the secret was deleted)

Calls populateRouteTLSFromSecret() — GetSecret succeeds because the informer cache hasn't propagated the deletion yet

Plugin chain accepts the route

The SARCompleted guard checks the DeepCopied route, sees Admitted=False (no ext-cert reason), and writes SARCompleted — flipping the route back to Admitted=True

The route bounces between Admitted=False (from DeleteFunc) and Admitted=True (from SARCompleted), and the E2E test polls for Admitted=False until timeout.

What's new in this PR (net new vs #825)

One commit on top of the full v2 (#825) changeset:

Fix: check deletedSecrets before writing SARCompleted

Before emitting RecordRouteUpdate(SARCompleted), check p.deletedSecrets.Load(key). If the secret has been marked as deleted by the DeleteFunc, skip the write. This prevents re-admission during the informer cache propagation window.

Test: TestDeletedSecretDoesNotGetReadmitted

Reproduces the exact failure:

Admits a route with an external cert (SARCompleted written)

Fires DeleteFunc (route rejected with ValidationFailed)

Calls HandleRoute(Modified) while GetSecret still succeeds (cache race)

Asserts the route is NOT re-admitted

Fails on v2 code (2 SARCompleted writes = re-admission), passes after the fix.

Test plan

go test -race ./pkg/router/controller/ — all pass, including TestDeletedSecretDoesNotGetReadmitted

make verify and make check — pass

/payload-aggregate-with-prs periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance 20 — target 20/20

Scale test: 2000 ext-cert routes admitted in ~120s

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-07-29T13:27:02Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

The pull request adds shared secret informer management and cache-aware external-certificate SAR validation. It updates route status handling, secret event processing, ingress condition comparison, and related tests. WriterLease gains configurable workers and non-blocking follower scheduling. Debug image builds accept GOARCH, Kubernetes client rate limits increase, certificate writes become atomic, and test setup is adjusted.

Possibly related PRs

openshift/router#822: Contains matching changes across external-certificate validation, secret management, status handling, writer leases, and build configuration.
openshift/router#824: Directly overlaps with changes in router validation, secret management, status handling, writer leases, and debug builds.
openshift/router#829: Directly overlaps with changes across external-certificate validation, secret management, status handling, writer leases, certificate writes, and debug image builds.

Suggested reviewers: melvinjoseph86, davidesalerno, thealisyed, redhat-chai-bot

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

Ignore

❌ Failed checks (1 error, 1 warning)

Check name	Status	Explanation	Resolution
Ote Binary Stdout Contract	❌ Error	TestMain still writes to stdout at router_test.go:107, :110, and :183 via fmt.Println/Printf, so the test binary can emit non-JSON process-level output.	Redirect all TestMain diagnostics to os.Stderr, and verify process-level setup has no stdout writes.
Docstring Coverage	⚠️ Warning	Docstring coverage is 52.63% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (13 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The patch adds standard Go tests and static subtest names only; the repository has no Ginkgo dependency or It/Describe/Context/When titles, and no title contains run-dependent data.
Test Structure And Quality	✅ Passed	Not applicable: all eight changed test files use standard testing.T/Test/t.Run; no Ginkgo or Gomega imports or spec declarations were found.
Microshift Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added. The changed tests are Go unit tests using testing.T, so the MicroShift API compatibility check is not applicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The PR adds only standard Go unit tests in pkg/router/controller; no Ginkgo e2e tests or multi-node/SNO assumptions were added.
Topology-Aware Scheduling Compatibility	✅ Passed	The PR changes controller, validation, client, and test code but adds no manifests or scheduling constructs: no affinity, topology spread, node selectors, tolerations, replicas, or PDB changes were...
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; changed tests use standard Go Test functions and add no IPv4 literals or external connectivity.
No-Weak-Crypto	✅ Passed	The complete PR diff adds no MD5, SHA1, DES/3DES, RC4, Blowfish, or ECB usage; crypto changes use standard TLS/X.509 APIs and do not compare secret values.
Container-Privileges	✅ Passed	The PR changes no Kubernetes/container manifests and adds no privilege declarations; the only hostNetwork: true is pre-existing in deploy/router.yaml and unchanged.
No-Sensitive-Data-In-Logs	✅ Passed	Added logs contain resource names, namespaces, file paths, statuses, and errors only; inspection found no secret bytes, passwords, tokens, API keys, PII, or session IDs logged.
Title check	✅ Passed	The title clearly describes the main change: asynchronous external certificate validation and the secret-deletion re-admission fix.
Description check	✅ Passed	The description directly explains the external certificate validation changes, regression fix, tests, and validation results.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci · 2026-07-29T13:27:06Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

…d Informer secret monitoring

This prevents the router from dropping its leader lease under high concurrency when updating route statuses, which was causing a 60-second stall in scale tests.

This commit modifies DeleteFunc inside route_secret_manager.go to transition the route's ingress condition to ExternalCertificateValidationFailed on secret deletion, rather than the transient ExternalCertificateSecretDeleted status. Reasoning and Analysis: - The RouteExternalCertificate e2e conformance tests in hypershift expect the route to transition to 'ExternalCertificateValidationFailed' (with ConditionFalse) when a referenced TLS secret is deleted. - Previously, when DeleteFunc fired, it recorded the intermediate reason 'ExternalCertificateSecretDeleted'. It was expected that the subsequent standard route controller Modified reconciliation would trigger validate(), which would then transition to the final 'ExternalCertificateValidationFailed' reason. - However, relying on this multi-step watch-triggered transition is race-prone and does not guarantee completion before the test polls. - By immediately and directly recording 'ExternalCertificateValidationFailed' on secret deletion, we satisfy the E2E test assertion requirements instantly, unblocking hypershift conformance payload nightlies. Related Changes: - Updated pkg/router/controller/route_secret_manager_test.go unit tests (TestSecretDelete and TestSecretRecreation) to expect the updated rejection reason.

This commit adds 'system:serviceaccounts', 'system:serviceaccounts:openshift-ingress', and 'system:authenticated' standard service account groups to all SubjectAccessReviewSpecs. Previously, only the routerServiceAccount name was passed, leading to false-negative denials if permissions are granted via standard service account group roles on the target cluster. Specifying the standard groups ensures complete and correct RBAC evaluation.

…ests This commit improves testing hygiene in factory_endpointslices_test.go by replacing the global os.Setenv call with t.Setenv. Using t.Setenv ensures that the KUBE_FEATURE_WatchListClient environment override is automatically scoped and cleanly torn down after each test execution, avoiding potential side effects or pollution on other test suites.

This commit refines the deletion message inside DeleteFunc to say 'external certificate validation failed: secret ... deleted for route ...' This ensures complete semantic consistency with the ExternalCertificateValidationFailed rejection reason, providing clear and non-contradictory diagnostic information for operators inspecting the route condition.

…ence This commit updates vendor/modules.txt to remove the reference to github.com/openshift/library-go/pkg/authorization/authorizationutil, which is no longer used by the router following our asynchronous external certificate validation refactoring.

This commit modifies StartFakeServerForTest in pkg/router/template/configmanager/haproxy/testing/haproxy.go to use the pattern 'fake-haproxy-*' instead of combining the long test name in the prefix. This avoids reaching the hard 104-character limit on Darwin (macOS) Unix socket paths when running local tests, ensuring all tests compile and execute successfully on both macOS and Linux environments.

…ert validation Three tests that prove the bugs causing the x509: ECDSA verification failure after PR openshift#822 merged: 1. TestPopulateRouteTLSRace: fails with -race, proving populateRouteTLSFromSecret mutates the shared informer cache object while the informer goroutine concurrently reads it via DeepCopy. 2. TestWriteCertificateAtomicity: fails consistently, proving os.WriteFile truncates the PEM file before writing — concurrent readers (HAProxy during reload) observe empty files 3% of the time. 3. TestSARCompletedFeedbackLoop: passes, documenting that every HandleRoute unconditionally emits RecordRouteUpdate(SARCompleted), creating a re-enqueue loop that doubles cert writes and reloads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ilure Three fixes for the bugs that caused the second revert (PR openshift#824): 1. DeepCopy route before mutating TLS fields: HandleRoute now DeepCopies the route before populateRouteTLSFromSecret writes Certificate/Key in-place. Secret handlers (Add/Update/DeleteFunc) also DeepCopy after fetching from the lister. This eliminates the data race between the main controller goroutine and informer goroutines that share the same route pointer from the informer cache. 2. Atomic PEM file write: WriteCertificate now writes to a temp file and renames into place via os.Rename, which is atomic on Linux. Previously os.WriteFile truncated the file before writing, creating a window where HAProxy could read an empty PEM during reload. 3. Guard SARCompleted feedback loop: Only emit RecordRouteUpdate with SARCompleted when the route doesn't already have an ext-cert admitted reason. Previously every HandleRoute unconditionally wrote SARCompleted, causing a re-enqueue loop that doubled cert writes and HAProxy reloads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Decouple writerlease worker count from the SAR semaphore. Status writes to the API server don't need 50 concurrent workers — that level of parallelism causes a storm of concurrent status writes, leading to write conflicts and rapid-fire route re-enqueues that widen race windows. Use a single worker, matching the pre-async behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Test doc comments were written for the "reproduce the bug" commit and still described pre-fix behavior. Update them to describe the invariants the tests now protect rather than the bugs they originally exposed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Address review feedback from jcmoraisjr and coderabbitai: check the error return from HandleRoute and WriteCertificate in the concurrent test goroutines instead of silently discarding them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…t route After the DeleteFunc handler rejects a route (Admitted=False), the status update triggers a re-enqueue. If the subsequent HandleRoute succeeds (because GetSecret still returns the secret from the informer cache during the deletion propagation window), the SARCompleted guard would see no ext-cert admitted reason on the DeepCopied route and write SARCompleted — flipping the route back to Admitted=True. This caused the E2E test "the secret is deleted then routes are not reachable" to poll for Admitted=False until the 15-minute timeout. Fix: check the deletedSecrets map before writing SARCompleted. If the secret has been marked as deleted by the DeleteFunc, skip the write. Adds TestDeletedSecretDoesNotGetReadmitted which fails on unfixed code (2 SARCompleted writes = re-admission) and passes after the fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

bentito · 2026-08-07T15:02:38Z

/payload-aggregate periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade 10

neisw · 2026-08-07T17:28:20Z

/payload-aggregate periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance 10

openshift-ci · 2026-08-07T17:28:24Z

@neisw: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/62c688c0-9285-11f1-9d0b-16796c10e774-0

bentito · 2026-08-07T19:12:14Z

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv4 periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview

bentito · 2026-08-07T19:12:15Z

/payload-aggregate periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade 10

openshift-ci · 2026-08-07T19:12:17Z

@bentito: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv4
periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/e67f5b70-9293-11f1-89d0-b10883231955-0

openshift-ci · 2026-08-07T19:12:19Z

@bentito: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/e72bef20-9293-11f1-9946-51693c1235dd-0

Temporary diagnostic commit — to be reverted before merge. The secret informer handlers (Add/Update/Delete) log at V(4), which is invisible in CI at default verbosity. This makes it impossible to determine whether informer events are being delivered when investigating failures. Raise the log level to V(2) and add a "Secret refresh completed" log after the cert is pushed through the plugin chain and Commit() is called. This gives us definitive evidence of whether the informer fires during the "secret is updated then routes are reachable" E2E test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Two fixes for HyperShift CI failures where the "secret is updated then routes are reachable" E2E test times out: 1. Register route with SharedSecretManager BEFORE SAR validation. Previously, validateAndRegister() called validate() first and RegisterRoute() second. If the initial SAR check failed (common in HyperShift due to RBAC propagation delays across API server replicas), the route was never registered with the secret informer. It would never receive UpdateFunc events and could never pick up secret changes — permanently orphaned from the informer. Now RegisterRoute() runs first. If SAR fails, the route is still rejected (not admitted), but it stays registered so future informer events can trigger re-evaluation when RBAC propagates. 2. Set per-secret informer resyncPeriod from 0 to 30 seconds. With resyncPeriod=0, the informer relies entirely on the watch connection for event delivery. If a watch event is lost (silent watch stall, proxy connection issue, API server load), there is no recovery mechanism. CI logs show the router reloading HAProxy 36+ times over 3 minutes without ever picking up the new cert, strongly suggesting the watch event was never delivered. With resyncPeriod=30s, the informer periodically re-dispatches cached state to handlers, providing a recovery path after watch reconnection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

bentito · 2026-08-08T00:18:21Z

/payload-aggregate periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance 10

openshift-ci · 2026-08-08T00:18:54Z

@bentito: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/aa8d8940-92be-11f1-9541-e50d2b6a527d-0

neisw · 2026-08-08T15:53:36Z

/payload-aggregate periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade 10

openshift-ci · 2026-08-08T15:53:38Z

@neisw: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5102c790-9341-11f1-9ea5-0dc88e6158bd-0

neisw · 2026-08-08T15:54:02Z

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv4 periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview

openshift-ci · 2026-08-08T15:54:04Z

@neisw: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv4
periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/60a9bcd0-9341-11f1-867b-bff27a687075-0

neisw · 2026-08-09T12:43:30Z

/payload-aggregate periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade 10

openshift-ci · 2026-08-09T12:43:35Z

@neisw: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ed8d8e20-93ef-11f1-9eeb-37b4e969cade-0

…diagnostics" This reverts commit 25df784.

bentito · 2026-08-09T13:24:33Z

/payload-aggregate periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance 10

bentito · 2026-08-09T13:24:37Z

/payload-aggregate periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade 10

openshift-ci · 2026-08-09T13:24:38Z

@bentito: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a975db60-93f5-11f1-905d-2f552bde3df6-0

openshift-ci · 2026-08-09T13:24:40Z

@bentito: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/abcb3d10-93f5-11f1-8f73-71fb6dacab1c-0

bentito · 2026-08-09T13:24:41Z

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv4 periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview

openshift-ci · 2026-08-09T13:24:44Z

@bentito: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv4
periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ae2cd3c0-93f5-11f1-8198-bba396a059ec-0

bentito · 2026-08-09T20:29:44Z

/test perfscale-aws-ingress-perf

bentito · 2026-08-09T20:31:31Z

@neisw this seems to be doing quite well on the aggregate payload tests now. going to ask for human review and verify again

neisw · 2026-08-09T20:35:30Z

/unhold
I agree @bentito - thanks for the extra effort on this!

openshift-ci · 2026-08-09T23:18:44Z

@bentito: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

jcmoraisjr

Changes look good. I focused on the latest ones after the last review, maybe I'm repeating myself or other previous comments, so sorry if this is happening.

jcmoraisjr · 2026-08-10T13:58:58Z

 // isRouteAdmittedTrue returns true if the given route has been admitted
 // by the current router, otherwise false.
 func isRouteAdmittedTrue(route *routev1.Route, routerName string) bool {


nit: this func is not being used anymore.

jcmoraisjr · 2026-08-10T14:01:28Z

+// lockRoute acquires the per-route lock for key, creating it on first use,
+// and returns a function to release it.
+func (p *RouteSecretManager) lockRoute(key string) func() {
+	value, _ := p.routeLocks.LoadOrStore(key, &sync.Mutex{})
+	mu := value.(*sync.Mutex)
+	mu.Lock()
+	return mu.Unlock
 }


nit: a key as a types.NamespacedName{} avoids a string concat in the hotpath

jcmoraisjr · 2026-08-10T14:06:09Z

+	for _, h := range handlers {
+		switch eventType {
+		case "Add":
+			if h.AddFunc != nil {
+				h.AddFunc(obj)
+			}
+		case "Update":
+			if h.UpdateFunc != nil {
+				h.UpdateFunc(oldObj, obj)
+			}


nit: is handling hundreds of routes pointing to the same secret a valid scenario? If so maybe parallelizing the UpdateFunc() is a valid followup update?

jcmoraisjr · 2026-08-10T14:08:45Z

+			Annotations: map[string]string{
+				"router.openshift.io/cert-resource-version": "100",
+			},


nit: maybe use the certResourceVersionAnnotation const?

jcmoraisjr · 2026-08-10T14:26:45Z

 		ActiveServiceUnits:    activeServiceUnits,
 		HTTPResponseHeaders:   httpResponseHeadersList,
 		HTTPRequestHeaders:    httpRequestHeadersList,
+		CertResourceVersion:   route.Annotations["router.openshift.io/cert-resource-version"],


nit: same here, what about reusing certResourceVersionAnnotation const.

jcmoraisjr · 2026-08-10T14:28:30Z

+		if isStaleExtCert(newConfig, &existingConfig) {
+			log.V(4).Info("dropping stale external certificate update",
+				"namespace", route.Namespace, "name", route.Name,
+				"existingVersion", existingConfig.CertResourceVersion,
+				"incomingVersion", newConfig.CertResourceVersion)
+			return
+		}


What is the chance for this guard being dropping legit changes to routes?

jcmoraisjr · 2026-08-10T14:32:21Z

+func isStaleExtCert(incoming, existing *ServiceAliasConfig) bool {
+	if incoming.CertResourceVersion == "" || existing.CertResourceVersion == "" {
+		return false
+	}
+	inV, errIn := strconv.ParseInt(incoming.CertResourceVersion, 10, 64)
+	exV, errEx := strconv.ParseInt(existing.CertResourceVersion, 10, 64)
+	if errIn != nil || errEx != nil {
+		return false
+	}
+	return inV <= exV
+}


resourceVersion is considered an opaque attribute. TIL it can be safely compared when using the same api. Wondering however if using resourceversion.CompareResourceVersion() from "k8s.io/apimachinery/pkg/util/resourceversion" would be a better approach.

jcmoraisjr · 2026-08-10T14:38:39Z

+			unlock := p.lockRoute(key)
+			if err := p.populateRouteTLSFromSecret(route); err != nil {
+				unlock()
+				return
 			}
+			if err := p.plugin.HandleRoute(watch.Modified, route); err != nil {
+				log.Error(err, "failed to propagate route after secret update", "namespace", namespace, "route", routeName)
+			}
+			unlock()


If HandleRoute() panics and the caller recovers and don't let the process to crash, the route should continue locked indefinitely, potentially deadlocking the next goroutine trying to acquire a lock from that route. Wondering if it'd be safer to defer unlock inside an inline func().

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2026

openshift-ci-robot mentioned this pull request Jul 29, 2026

OCPBUGS-77056: Revert #825 "Make external cert validation asynchronous (v2 — race condition fixes)" #829

Merged

openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 29, 2026

bentito and others added 19 commits July 30, 2026 08:53

OCPBUGS-77056: Asynchronous external certificate validation and Hybri…

d8e8d3e

…d Informer secret monitoring

OCPBUGS-77056: Retry on write conflicts without dropping writerlease

ed0e5f6

This prevents the router from dropping its leader lease under high concurrency when updating route statuses, which was causing a 60-second stall in scale tests.

Remove library-go replace directive and update vendor

73ef8d9

address review comment: use types.NamespacedName for informer key

203d09a

address review comment: remove unnecessary lock from StatusAdmitter

4abff8f

Addressed several refactor needs from PR comments

11bc0d0

Addressing coderabbit PR comments

7702535

bentito force-pushed the OCPBUGS-77056-async-sar-resurrect-v3 branch from 7092d0f to 8bb1e1e Compare July 30, 2026 12:53

bentito marked this pull request as ready for review July 30, 2026 12:53

bentito and others added 2 commits August 7, 2026 20:17

Revert "OCPBUGS-77056: Raise secret handler log level to V(2) for CI …

91025fd

…diagnostics" This reverts commit 25df784.

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 9, 2026

jcmoraisjr reviewed Aug 10, 2026

View reviewed changes

Uh oh!

Conversation

bentito commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

v2 payload results (5/10 — not 10/10)

Root cause: SARCompleted re-admits a deleted-secret route

What's new in this PR (net new vs #825)

Fix: check deletedSecrets before writing SARCompleted

Test: TestDeletedSecretDoesNotGetReadmitted

Update: root cause found and fixed — synchronous certificate refresh

Test plan

Uh oh!

openshift-ci-robot commented Jul 29, 2026

Background

v2 payload results (5/10 — not 10/10)

Root cause: SARCompleted re-admits a deleted-secret route

What's new in this PR (net new vs #825)

Fix: check deletedSecrets before writing SARCompleted

Test: TestDeletedSecretDoesNotGetReadmitted

Test plan

Uh oh!

coderabbitai Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Pre-merge checks failed

❌ Failed checks (1 error, 1 warning)

Uh oh!

openshift-ci Bot commented Jul 29, 2026

Uh oh!

bentito commented Aug 7, 2026

Uh oh!

neisw commented Aug 7, 2026

Uh oh!

openshift-ci Bot commented Aug 7, 2026

Uh oh!

bentito commented Aug 7, 2026

Uh oh!

bentito commented Aug 7, 2026

Uh oh!

openshift-ci Bot commented Aug 7, 2026

Uh oh!

openshift-ci Bot commented Aug 7, 2026

Uh oh!

bentito commented Aug 8, 2026

Uh oh!

openshift-ci Bot commented Aug 8, 2026

Uh oh!

neisw commented Aug 8, 2026

Uh oh!

openshift-ci Bot commented Aug 8, 2026

Uh oh!

neisw commented Aug 8, 2026

Uh oh!

openshift-ci Bot commented Aug 8, 2026

Uh oh!

neisw commented Aug 9, 2026

Uh oh!

openshift-ci Bot commented Aug 9, 2026

Uh oh!

bentito commented Aug 9, 2026

Uh oh!

bentito commented Aug 9, 2026

Uh oh!

openshift-ci Bot commented Aug 9, 2026

Uh oh!

openshift-ci Bot commented Aug 9, 2026

Uh oh!

bentito commented Aug 9, 2026

Uh oh!

openshift-ci Bot commented Aug 9, 2026

Uh oh!

bentito commented Aug 9, 2026

Uh oh!

bentito commented Aug 9, 2026

Uh oh!

neisw commented Aug 9, 2026

Uh oh!

openshift-ci Bot commented Aug 9, 2026

bentito commented Jul 29, 2026 •

edited

Loading

Fix: check `deletedSecrets` before writing SARCompleted

Test: `TestDeletedSecretDoesNotGetReadmitted`

Fix: check `deletedSecrets` before writing SARCompleted

Test: `TestDeletedSecretDoesNotGetReadmitted`

coderabbitai Bot commented Jul 29, 2026 •

edited

Loading