[9.4](backport #7533) Retain output secret when agent update fails

[mergify]

What is the problem this PR solves?

When Fleet Server creates an output API key, it stores the encoded key in .fleet-secrets and then updates the agent document in .fleet-agents with the secret reference. Elasticsearch can commit that update while the client still receives an error, for example when the request context expires while waiting for the response. This can happen under scale load and was observed during a 100k-agent scale test.

The existing error path immediately deleted the secret. In the ambiguous-commit case, this left the agent document pointing at a missing secret and subsequent check-ins failed while resolving the output API key.

How does this PR solve the problem?

Retain the newly created secret whenever the agent update returns an error. This chooses a possible orphaned secret over damaging an agent document with a dangling reference.

A follow-up PR adds conservative, out-of-band reconciliation for these retained candidates.

How to test

• go test ./internal/pkg/policy -run TestPolicyOutputESPrepare -count=1
• mage test:unit

mage check:all currently reports 58 pre-existing linter findings in unrelated files under the repository's pinned Go/toolchain configuration. The files changed by this PR are clean.

Design Checklist

• The solution is stateless and assumes a horizontally scaled Fleet Server deployment.
• The changed path is intended for 100k-agent deployments.
• The error behavior is fail-safe: an uncertain write retains the secret so a committed agent reference remains usable.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.md

Documentation and configuration changes are not applicable. The changelog entry is supplied as a fragment.

---

This is an automatic backport of pull request #7533 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 7fb25fd has failed:

On branch mergify/bp/9.4/pr-7533
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 7fb25fd.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539200-retain-output-secret-on-agent-update-failure.yaml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

TL;DR

The current Buildkite unit-test failures are caused by a deterministic test expectation mismatch in internal/pkg/policy/policy_output_test.go, not by merge markers. The backported test expects WriteSecret(...) to be called, but the 9.4 prepareElasticsearch path never calls WriteSecret, so the mock fails every run.

Remediation

• Update TestPolicyOutputESPrepare/Secret is retained when agent document update fails to match 9.4 behavior (remove the WriteSecret expectation and validate only the intended failure-path contract for this branch).
• Re-run unit tests (.buildkite/scripts/unit_test.sh / mage test:unit) after adjusting that test.

Investigation details

Root Cause

This is a test failure introduced in the backport test content:

• The failing subtest sets bulker.On("WriteSecret", ...) in internal/pkg/policy/policy_output_test.go:310 and then asserts expectations at internal/pkg/policy/policy_output_test.go:326.
• In this branch’s implementation, prepareElasticsearch generates an API key and immediately builds agent-update fields (internal/pkg/policy/policy_output.go:256-333), but there is no WriteSecret call in that path (or file).

That mismatch makes the mock expectation impossible to satisfy.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16050
• Jobs/steps: Run unit tests (Linux), Run unit tests: MacOS 13
• Buildkite log excerpts (both jobs):
  • Error: running "go test -tags=grpcnotrace -v -race -coverprofile=build/coverage-*.out ./..." failed with exit code 1
  • FAIL

Reproduction on the Buildkite commit 95cfef4939cee444e962f996220b7c877a940f26 shows the exact failing expectation:

--- FAIL: TestPolicyOutputESPrepare/Secret_is_retained_when_agent_document_update_fails
policy_output_test.go:326: FAIL: WriteSecret(string,string)
policy_output_test.go:326: FAIL: 2 out of 3 expectation(s) were met.

Verification

• Reproduced with:
  • go test -race ./internal/pkg/policy -count=1
  • at commit 95cfef4939cee444e962f996220b7c877a940f26

Follow-up

No flaky-test signature was identified here; this appears deterministic from a test/branch-behavior mismatch.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.