[9.3](backport #7534) Reconcile orphaned output secrets after agent update failures

[mergify]

What is the problem this PR solves?

#7533 prevents an ambiguous .fleet-agents update failure from deleting a secret that Elasticsearch may already have referenced. The safe failure behavior can leave an orphaned .fleet-secrets document when the update truly did not commit.

This failure mode was observed under load during a 100k-agent scale test.

How does this PR solve the problem?

Add an out-of-band, candidate-based reconciler:

• The failed update path only enqueues the newly created secret and returns; it performs no cleanup I/O in the check-in request.
• Candidates remain in memory for a 10-minute grace period.
• The reconciler rereads the agent with a fresh background context and preserves candidates referenced by either the current output or output API-key retirement history.
• A secret must be observed unreferenced twice, at least 5 minutes apart, before deletion.
• Read and delete failures retain the candidate for retry.
• Work is capped at 100 candidate checks per minute per Fleet Server to avoid creating a cleanup load spike.
• A full queue fails safe by leaking a secret rather than blocking a check-in or risking deletion.

Candidates intentionally do not survive a Fleet Server restart. A restart during the reconciliation window can therefore leave an orphaned secret, but cannot damage an agent by deleting a referenced secret.

This PR is stacked on and depends on #7533. Its implementation is the second commit (b3f11ce9). Once #7533 merges, GitHub will remove the shared first commit from this PR's diff.

How to test

• go test ./internal/pkg/policy ./internal/pkg/gc ./internal/pkg/api ./internal/pkg/server
• mage test:unit

mage check:all currently reports the same 58 pre-existing linter findings as #7533 in unrelated files. None are in files changed by this PR.

Design Checklist

• The reconciler is local to each horizontally scaled Fleet Server and requires no cross-instance coordination because each secret candidate is created by one request on one instance.
• The queue, grace period, confirmation reads, and per-run cap are designed for 100k-agent deployments.
• Every uncertain state fails safe by retaining the secret.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.md

Configuration changes are not applicable. Package documentation was updated and the changelog entry is supplied as a fragment.

---

This is an automatic backport of pull request #7534 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 229f161 has failed:

On branch mergify/bp/9.3/pr-7534
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 229f161.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539201-reconcile-orphaned-output-secrets.yaml
	modified:   internal/pkg/api/handleCheckin.go
	modified:   internal/pkg/gc/doc.go
	new file:   internal/pkg/gc/orphaned_output_secrets.go
	new file:   internal/pkg/gc/orphaned_output_secrets_test.go
	modified:   internal/pkg/policy/policy_output_integration_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go
	both modified:   internal/pkg/server/fleet.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

This pull request is now in conflicts. Could you fix it @mergify[bot]? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/9.3/pr-7534 upstream/mergify/bp/9.3/pr-7534
git merge upstream/9.3
git push upstream mergify/bp/9.3/pr-7534

[github-actions]

TL;DR

Buildkite build 16062 failed in both unit-test jobs because commit 70ce6e7 removed OutputSecretCandidate from production code but left a test helper in internal/pkg/policy/policy_output_test.go that still references that type. Remove that stale helper (or restore the type), then rerun CI.

Remediation

• Keep the follow-up fix from commit 4a24c67 (it removes the stale recordingOutputSecretCandidateCollector helper referencing OutputSecretCandidate) or apply the equivalent change.
• Re-run the two failed steps: Run unit tests (Linux) and Run unit tests: MacOS 13.

Investigation details

Root Cause

This is a code bug introduced in backport commit 70ce6e7:

• internal/pkg/policy/policy_output.go removes OutputSecretCandidate and related collector types.
• internal/pkg/policy/policy_output_test.go still had a helper using OutputSecretCandidate at lines 73 and 76 in that commit, so package test compilation fails.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16062
• Failed jobs/steps:
  • Run unit tests
  • Run unit tests: MacOS 13
• Buildkite log excerpt (both jobs):

Error: running "go test -tags=grpcnotrace -v -race -coverprofile=build/coverage-<os>.out ./..." failed with exit code 1

• Reproduced on the exact failing commit (70ce6e7):

# github.com/elastic/fleet-server/v7/internal/pkg/policy [github.com/elastic/fleet-server/v7/internal/pkg/policy.test]
internal/pkg/policy/policy_output_test.go:73:15: undefined: OutputSecretCandidate
internal/pkg/policy/policy_output_test.go:76:65: undefined: OutputSecretCandidate
FAIL	github.com/elastic/fleet-server/v7/internal/pkg/policy [build failed]

• No matching open flaky-test issue was found for this failure signature.

Verification

• Correlated Buildkite failures with the commit sequence on PR [9.3](backport #7534) Reconcile orphaned output secrets after agent update failures #7551 (70ce6e7 failed; 4a24c67 immediately removes the stale helper).

Follow-up

• After rerun, if anything else fails, share the new failing step/log and I can triage the next blocker quickly.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[ycombinator]

Closing: the reconciler in #7534 depends on WriteSecret/DeleteSecret from #7416, which has not been backported to this branch. Since output secrets are never written in this branch, orphaned secrets cannot occur and this backport has no meaningful effect.