[8.19](backport #7533) Retain output secret when agent update fails

[mergify]

What is the problem this PR solves?

When Fleet Server creates an output API key, it stores the encoded key in .fleet-secrets and then updates the agent document in .fleet-agents with the secret reference. Elasticsearch can commit that update while the client still receives an error, for example when the request context expires while waiting for the response. This can happen under scale load and was observed during a 100k-agent scale test.

The existing error path immediately deleted the secret. In the ambiguous-commit case, this left the agent document pointing at a missing secret and subsequent check-ins failed while resolving the output API key.

How does this PR solve the problem?

Retain the newly created secret whenever the agent update returns an error. This chooses a possible orphaned secret over damaging an agent document with a dangling reference.

A follow-up PR adds conservative, out-of-band reconciliation for these retained candidates.

How to test

• go test ./internal/pkg/policy -run TestPolicyOutputESPrepare -count=1
• mage test:unit

mage check:all currently reports 58 pre-existing linter findings in unrelated files under the repository's pinned Go/toolchain configuration. The files changed by this PR are clean.

Design Checklist

• The solution is stateless and assumes a horizontally scaled Fleet Server deployment.
• The changed path is intended for 100k-agent deployments.
• The error behavior is fail-safe: an uncertain write retains the secret so a committed agent reference remains usable.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.md

Documentation and configuration changes are not applicable. The changelog entry is supplied as a fragment.

---

This is an automatic backport of pull request #7533 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 7fb25fd has failed:

On branch mergify/bp/8.19/pr-7533
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 7fb25fd.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539200-retain-output-secret-on-agent-update-failure.yaml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

TL;DR

The current failures are no longer consistent with the prior policy_output backport-mismatch diagnosis; the policy_output path now passes locally. A reproducible failure is in internal/pkg/es/client_test.go (TestConnectionTLS) under FIPS-enabled runs, where the test’s expectation logic does not match observed crypto behavior.

Remediation

• Update TestConnectionTLS in internal/pkg/es/client_test.go to stop keying expected outcome solely on fips140.Enabled() and instead align with the actual FIPS mode semantics used in CI (GOFIPS140/runtime mode), or split assertions by explicit mode.
• Re-run the unit matrix after adjusting the test expectation; if non-FIPS jobs still fail, capture/upload full failing package output (not tail-only) to identify any second root cause.

Investigation details

Root Cause

I could not extract the exact failing test name from the provided Buildkite logs because each pre-fetched log file only contains the tail and ends at:

• Error: running "go test ... ./..." failed with exit code 1

However, on PR head ca17b48c3a59e61840797da883be733112e0b1d6, I reproduced a concrete FIPS-mode unit failure:

• internal/pkg/es/client_test.go:210 in TestConnectionTLS
• error: tls: certificate uses RSA-1024 public key which is not allowed by FIPS 140-3 (minimum 2048 bits)

This points to a test expectation mismatch around FIPS-mode detection/behavior in:

• internal/pkg/es/client_test.go:207-210 (branching only on fips140.Enabled())
• embedded test fixture key: internal/pkg/es/client_test.go:217-221 (fips_invalid.key, 1024-bit RSA)

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16056
• Jobs/steps (all four failed):
  • :smartbear-testexecute: Run unit tests
  • :smartbear-testexecute: Run fips140=on unit tests with FIPS provider and microsoft/go
  • :smartbear-testexecute: Run fips140=only unit tests with FIPS provider and upstream go
  • :smartbear-testexecute: Run unit tests: MacOS 13
• Key Buildkite excerpt (from all provided tails):
  • Error: running "go test ... ./..." failed with exit code 1
• Local reproduction (same PR head):
  • go test -tags=grpcnotrace,requirefips -v -race -coverprofile=/tmp/gh-aw/agent/coverage-fips.out ./...
  • Failing excerpt:
    • --- FAIL: TestConnectionTLS
    • client_test.go:210: Received unexpected error: tls: certificate uses RSA-1024 public key which is not allowed by FIPS 140-3 (minimum 2048 bits)

Verification

• go test -tags=grpcnotrace ./internal/pkg/policy -run TestPolicyOutputESPrepare -count=1 -v (pass)
• go test -tags=grpcnotrace -v -race -coverprofile=/tmp/gh-aw/agent/coverage-linux.out ./... (pass)
• go test -tags=grpcnotrace,requirefips -v -race -coverprofile=/tmp/gh-aw/agent/coverage-fips.out ./... (fails in TestConnectionTLS)

Follow-up

No open flaky-test issue matched TestConnectionTLS in this repo. If this is expected instability across FIPS modes, consider opening/labeling one so future detective passes can auto-reference it.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.