Retain output secret when agent update fails

[ycombinator]

What is the problem this PR solves?

When Fleet Server creates an output API key, it stores the encoded key in .fleet-secrets and then updates the agent document in .fleet-agents with the secret reference. Elasticsearch can commit that update while the client still receives an error, for example when the request context expires while waiting for the response. This can happen under scale load and was observed during a 100k-agent scale test.

The existing error path immediately deleted the secret. In the ambiguous-commit case, this left the agent document pointing at a missing secret and subsequent check-ins failed while resolving the output API key.

How does this PR solve the problem?

Retain the newly created secret whenever the agent update returns an error. This chooses a possible orphaned secret over damaging an agent document with a dangling reference.

A follow-up PR adds conservative, out-of-band reconciliation for these retained candidates.

How to test

• go test ./internal/pkg/policy -run TestPolicyOutputESPrepare -count=1
• mage test:unit

mage check:all currently reports 58 pre-existing linter findings in unrelated files under the repository's pinned Go/toolchain configuration. The files changed by this PR are clean.

Design Checklist

• The solution is stateless and assumes a horizontally scaled Fleet Server deployment.
• The changed path is intended for 100k-agent deployments.
• The error behavior is fail-safe: an uncertain write retains the secret so a committed agent reference remains usable.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.md

Documentation and configuration changes are not applicable. The changelog entry is supplied as a fragment.

[blakerouse]

Actually after reviewing #7534 I am okay with this actually.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[github-actions]

@Mergifyio backport 9.5 9.4 9.3 8.19

[mergify]

backport 9.5 9.4 9.3 8.19

✅ Backports have been created

Details

• #7544 [9.5](backport #7533) Retain output secret when agent update fails has been created for branch 9.5 but encountered conflicts

Cherry-pick of 7fb25fd has failed:

On branch mergify/bp/9.5/pr-7533
Your branch is up to date with 'origin/9.5'.

You are currently cherry-picking commit 7fb25fd.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539200-retain-output-secret-on-agent-update-failure.yaml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7545 [9.4](backport #7533) Retain output secret when agent update fails has been created for branch 9.4 but encountered conflicts

Cherry-pick of 7fb25fd has failed:

On branch mergify/bp/9.4/pr-7533
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 7fb25fd.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539200-retain-output-secret-on-agent-update-failure.yaml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7546 [9.3](backport #7533) Retain output secret when agent update fails has been created for branch 9.3 but encountered conflicts

Cherry-pick of 7fb25fd has failed:

On branch mergify/bp/9.3/pr-7533
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 7fb25fd.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539200-retain-output-secret-on-agent-update-failure.yaml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7547 [8.19](backport #7533) Retain output secret when agent update fails has been created for branch 8.19 but encountered conflicts

Cherry-pick of 7fb25fd has failed:

On branch mergify/bp/8.19/pr-7533
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 7fb25fd.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1785539200-retain-output-secret-on-agent-update-failure.yaml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/policy/policy_output.go
	both modified:   internal/pkg/policy/policy_output_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally