Reconcile orphaned output secrets after agent update failures

[ycombinator]

What is the problem this PR solves?

#7533 prevents an ambiguous .fleet-agents update failure from deleting a secret that Elasticsearch may already have referenced. The safe failure behavior can leave an orphaned .fleet-secrets document when the update truly did not commit.

This failure mode was observed under load during a 100k-agent scale test.

How does this PR solve the problem?

Add an out-of-band, candidate-based reconciler:

• The failed update path only enqueues the newly created secret and returns; it performs no cleanup I/O in the check-in request.
• Candidates remain in memory for a 10-minute grace period.
• The reconciler rereads the agent with a fresh background context and preserves candidates referenced by either the current output or output API-key retirement history.
• A secret must be observed unreferenced twice, at least 5 minutes apart, before deletion.
• Read and delete failures retain the candidate for retry.
• Work is capped at 100 candidate checks per minute per Fleet Server to avoid creating a cleanup load spike.
• A full queue fails safe by leaking a secret rather than blocking a check-in or risking deletion.

Candidates intentionally do not survive a Fleet Server restart. A restart during the reconciliation window can therefore leave an orphaned secret, but cannot damage an agent by deleting a referenced secret.

This PR is stacked on and depends on #7533. Its implementation is the second commit (b3f11ce9). Once #7533 merges, GitHub will remove the shared first commit from this PR's diff.

How to test

• go test ./internal/pkg/policy ./internal/pkg/gc ./internal/pkg/api ./internal/pkg/server
• mage test:unit

mage check:all currently reports the same 58 pre-existing linter findings as #7533 in unrelated files. None are in files changed by this PR.

Design Checklist

• The reconciler is local to each horizontally scaled Fleet Server and requires no cross-instance coordination because each secret candidate is created by one request on one instance.
• The queue, grace period, confirmation reads, and per-run cap are designed for 100k-agent deployments.
• Every uncertain state fails safe by retaining the secret.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.md

Configuration changes are not applicable. Package documentation was updated and the changelog entry is supplied as a fragment.

[blakerouse]

This looks good.

[github-actions]

TL;DR

The integration-test Buildkite step failed (go test ... -tags=integration) but the provided log excerpt does not include the actual failing package/test, so no code-level root cause can be confirmed from current evidence. Immediate next action is to rerun the same step with full test output (or attach complete logs) so the first failing test can be identified.

Remediation

• Re-run the :smartbear-testexecute: Run integration tests step and capture full go test output (including the first --- FAIL: / failing package line), e.g. preserve complete stdout or use go test -json artifact output.
• Once the first failing test is visible, map it to the changed PR files (notably internal/pkg/api/handleCheckin.go, internal/pkg/policy/policy_output.go, and internal/pkg/gc/orphaned_output_secrets.go) and apply a targeted fix.

Investigation details

Root Cause

From the available artifact, the failure is inconclusive: all visible tests are passing, then the log ends with a bare FAIL and command exit code 1, without the failing test/package context. That means a specific code bug cannot be reliably attributed yet.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16025
• Job/step: :smartbear-testexecute: Run integration tests
• Key log excerpt (/tmp/gh-aw/buildkite-logs/fleet-server-smartbear-testexecute-run-integration-tests.txt):

ok  github.com/elastic/fleet-server/v7/internal/pkg/ver 1.039s
?   github.com/elastic/fleet-server/v7/version [no test files]
FAIL
Error: running "go test -v -tags=integration,grpcnotrace -count=1 -race -p 1 ./..." failed with exit code 1

• The excerpt starts mid-stream (=== RUN TestSet/a.b.2.c) and contains no --- FAIL: section identifying the first failing test.

Verification

• Not run in this workflow: integration reproduction requires full Buildkite job context/services and the provided artifact is truncated.

Follow-up

• I checked open flaky-test issues; none clearly match this specific signature from the provided excerpt.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[github-actions]

@Mergifyio backport 9.5 9.4 9.3 8.19

[mergify]

backport 9.5 9.4 9.3 8.19

✅ Backports have been created

Details

• #7549 [9.5](backport #7534) Reconcile orphaned output secrets after agent update failures has been created for branch 9.5 but encountered conflicts

Cherry-pick of 229f161 has failed:

On branch mergify/bp/9.5/pr-7534
Your branch is up to date with 'origin/9.5'.

You are currently cherry-picking commit 229f161.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539201-reconcile-orphaned-output-secrets.yaml
	modified:   internal/pkg/api/handleCheckin.go
	modified:   internal/pkg/gc/doc.go
	new file:   internal/pkg/gc/orphaned_output_secrets.go
	new file:   internal/pkg/gc/orphaned_output_secrets_test.go
	modified:   internal/pkg/policy/policy_output_integration_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go
	both modified:   internal/pkg/server/fleet.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7550 [9.4](backport #7534) Reconcile orphaned output secrets after agent update failures has been created for branch 9.4 but encountered conflicts

Cherry-pick of 229f161 has failed:

On branch mergify/bp/9.4/pr-7534
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 229f161.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539201-reconcile-orphaned-output-secrets.yaml
	modified:   internal/pkg/api/handleCheckin.go
	modified:   internal/pkg/gc/doc.go
	new file:   internal/pkg/gc/orphaned_output_secrets.go
	new file:   internal/pkg/gc/orphaned_output_secrets_test.go
	modified:   internal/pkg/policy/policy_output_integration_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go
	both modified:   internal/pkg/server/fleet.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7551 [9.3](backport #7534) Reconcile orphaned output secrets after agent update failures has been created for branch 9.3 but encountered conflicts

Cherry-pick of 229f161 has failed:

On branch mergify/bp/9.3/pr-7534
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 229f161.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539201-reconcile-orphaned-output-secrets.yaml
	modified:   internal/pkg/api/handleCheckin.go
	modified:   internal/pkg/gc/doc.go
	new file:   internal/pkg/gc/orphaned_output_secrets.go
	new file:   internal/pkg/gc/orphaned_output_secrets_test.go
	modified:   internal/pkg/policy/policy_output_integration_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go
	both modified:   internal/pkg/server/fleet.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7552 [8.19](backport #7534) Reconcile orphaned output secrets after agent update failures has been created for branch 8.19 but encountered conflicts

Cherry-pick of 229f161 has failed:

On branch mergify/bp/8.19/pr-7534
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 229f161.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539201-reconcile-orphaned-output-secrets.yaml
	modified:   internal/pkg/api/handleCheckin.go
	modified:   internal/pkg/gc/doc.go
	new file:   internal/pkg/gc/orphaned_output_secrets.go
	new file:   internal/pkg/gc/orphaned_output_secrets_test.go
	modified:   internal/pkg/policy/policy_output_integration_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go
	both modified:   internal/pkg/server/fleet.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally