[9.5](backport #7533) Retain output secret when agent update fails

[mergify]

What is the problem this PR solves?

When Fleet Server creates an output API key, it stores the encoded key in .fleet-secrets and then updates the agent document in .fleet-agents with the secret reference. Elasticsearch can commit that update while the client still receives an error, for example when the request context expires while waiting for the response. This can happen under scale load and was observed during a 100k-agent scale test.

The existing error path immediately deleted the secret. In the ambiguous-commit case, this left the agent document pointing at a missing secret and subsequent check-ins failed while resolving the output API key.

How does this PR solve the problem?

Retain the newly created secret whenever the agent update returns an error. This chooses a possible orphaned secret over damaging an agent document with a dangling reference.

A follow-up PR adds conservative, out-of-band reconciliation for these retained candidates.

How to test

• go test ./internal/pkg/policy -run TestPolicyOutputESPrepare -count=1
• mage test:unit

mage check:all currently reports 58 pre-existing linter findings in unrelated files under the repository's pinned Go/toolchain configuration. The files changed by this PR are clean.

Design Checklist

• The solution is stateless and assumes a horizontally scaled Fleet Server deployment.
• The changed path is intended for 100k-agent deployments.
• The error behavior is fail-safe: an uncertain write retains the secret so a committed agent reference remains usable.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.md

Documentation and configuration changes are not applicable. The changelog entry is supplied as a fragment.

---

This is an automatic backport of pull request #7533 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 7fb25fd has failed:

On branch mergify/bp/9.5/pr-7533
Your branch is up to date with 'origin/9.5'.

You are currently cherry-picking commit 7fb25fd.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539200-retain-output-secret-on-agent-update-failure.yaml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

TL;DR

Buildkite build 16051 is failing in unit-test steps, but the provided job logs only contain tail output and do not include the first failing package/test line, so the underlying code-level root cause is not observable from this run’s available logs. Immediate action: rerun with full go test output (or attach full JUnit details) and then fix the first failing package.

Remediation

• Re-run the failing Buildkite steps with untruncated test output (capture and expose the first --- FAIL: / FAIL\t<package> block from mage test:unit).
• Publish build/*.xml test report contents (or failing test names) in the step logs/artifacts so the exact package/test can be traced deterministically.
• After the failing package is visible, run targeted repro locally (same tags: grpcnotrace and requirefips where applicable) and patch that package.

Investigation details

Root Cause

Current evidence is consistent with an infrastructure/observability failure in CI diagnostics (insufficient log detail), not a confirmed code defect from this dataset.

The four failed jobs all end with only a generic wrapper error:

• Error: running "go test ... ./..." failed with exit code 1

but none of the available log files include the failing package/test stanza needed to attribute a code bug:

• /tmp/gh-aw/buildkite-logs/fleet-server-smartbear-testexecute-run-unit-tests.txt
• /tmp/gh-aw/buildkite-logs/fleet-server-smartbear-testexecute-run-unit-tests-with-fips.txt
• /tmp/gh-aw/buildkite-logs/fleet-server-smartbear-testexecute-run-fips140only-unit-tests.txt
• /tmp/gh-aw/buildkite-logs/fleet-server-smartbear-testexecute-run-unit-tests-macos-13.txt

Each file ends with successful output from later packages (for example internal/pkg/ver) and a final FAIL summary, which indicates the actionable failure happened earlier in output not present in the provided excerpts.

Also verified: this is not the prior merge-marker failure mode from earlier comments. The current PR diff for internal/pkg/policy/policy_output.go and internal/pkg/policy/policy_output_test.go no longer contains conflict markers; latest changes are clean backport edits.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16051
• Failed jobs:
  • Run unit tests
  • Run unit tests with FIPS
  • Run fips140=only unit tests
  • Run unit tests: MacOS 13
• Key log excerpt (all four jobs):

FAIL
Error: running "go test ... ./..." failed with exit code 1

Verification

Could not perform root-cause attribution from the available Buildkite log excerpts because they do not contain the failing test/package section.

Follow-up

If the rerun still fails and surfaces a specific test/package, I can provide a concrete code-level diagnosis and patch guidance for that exact failure.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.