Store output API key secrets in .fleet-secrets instead of .fleet-agents

[ycombinator]

What is the problem this PR solves?

The kibana_system built-in role has broad read (actually all) access to .fleet-agents*. Fleet Server currently writes output API key secrets as plaintext id:secret strings into outputs.{name}.api_key on agent documents in .fleet-agents-7. Any principal with kibana_system credentials can therefore extract these secrets and use them to write to data streams that kibana_system itself cannot access — a privilege escalation.

How does this PR solve the problem?

Instead of storing the raw id:secret string in .fleet-agents-7, Fleet Server now:

1. On new key generation (policy_output.go): writes the key material to .fleet-secrets via POST /_fleet/secret and stores only a $co.elastic.secret{<secretId>} reference in outputs.{name}.api_key.
2. At agent checkin (policy_output.go): detects the reference format and resolves it via GET /_fleet/secret/{id} before injecting api_key into the policy payload sent to the agent.
3. On ACK / key rotation (handleAck.go): deletes the retired secret from .fleet-secrets via DELETE /_fleet/secret/{id}. The secret ID is carried in a new secret_id field on ToRetireAPIKeyIdsItems (schema change in model/schema.json).

Backward compatibility: existing plaintext values already stored in .fleet-agents-7 continue to work unchanged. The check secret.ParseSecretReference(output.APIKey) returns false for plain id:secret strings, so they are used directly. Agents migrate to the new format naturally on the next key rotation.

Dependency: requires elastic/elasticsearch#154498 (granting write_fleet_secrets cluster privilege to the elastic/fleet-server service account) to be merged and deployed first.

How to test this PR locally

Run the integration test suite against a stack with elastic/elasticsearch#154498 applied:

mage integration:test

Verify that:

• Newly enrolled agents receive a valid api_key in checkin responses
• .fleet-agents-7 documents show $co.elastic.secret{...} in outputs.*.api_key rather than a plaintext key
• .fleet-secrets contains the corresponding entries
• After a policy change that rotates the key, the old secret is deleted from .fleet-secrets

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer. (Secrets are resolved per-request from ES; no local state.)
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected. (Each checkin that needs to rotate a key adds one POST /_fleet/secret write; reads are already part of the checkin path via ReadSecrets.)
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc. (Secret writes/deletes are only on key rotation, not every checkin.)

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Requires Grant write_fleet_secrets privilege to elastic/fleet-server service account elasticsearch#154498

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

TL;DR

Both cloud E2E jobs fail before the E2E suite starts because the provisioned deployment has no Fleet endpoint: Terraform outputs fleet_url = "", and .buildkite/scripts/cloud_e2e_test.sh exits with status 1 when FLEET_SERVER_URL is empty. This is a shared deployment-startup failure, not a FIPS-specific failure.

Remediation

• Inspect the failed deployment’s Integrations Server / Elastic Agent startup logs to determine why the Fleet endpoint was not published, then fix that startup error and rerun the cloud E2E jobs.
• Verify the 9.6.0-SNAPSHOT stack used by the deployment includes the Elasticsearch write_fleet_secrets change required by this PR; the PR explicitly depends on that change and the new Fleet Server path calls POST /_fleet/secret.
• Confirm Terraform outputs a non-empty fleet_url before mage test:cloudE2ERun is invoked.

Investigation details

Root Cause

The confirmed failure is the cloud E2E precondition. In .buildkite/scripts/cloud_e2e_test.sh, FLEET_SERVER_URL is read from the Terraform state and the script exits 1 when it is empty. Both failed artifacts show fleet_url = "" in the Terraform output immediately before the deployment is destroyed. dev-tools/cloud/terraform/outputs.tf derives this value from ec_deployment.deployment.integrations_server.endpoints.fleet, so the endpoint was unavailable from the provisioned deployment.

The captured artifacts do not include the earlier Elastic Agent/Integrations Server startup error, so they cannot establish whether the endpoint was missing because the image failed to start, the snapshot lacked the required Elasticsearch privilege, or another deployment issue occurred. The PR changes no cloud Terraform or E2E orchestration code; both failures use the same shared script.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/15692
• Jobs: Cloud e2e Test and Cloud e2e FIPS Test
• Key log excerpt from both artifacts: fleet_url = "" -> null (Terraform output), followed by Error: The command exited with status 1 after cleanup.
• .buildkite/scripts/cloud_e2e_test.sh: explicitly reports FLEET_SERVER_URL is empty, cloud e2e tests cannot be executed and exits 1.
• No matching flaky-test issue was found.

Verification

Not run locally: the failure depends on the external Elastic Cloud deployment, and the provided logs omit the deployment startup log that caused the Fleet endpoint to be unavailable.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[github-actions]

TL;DR

Both Cloud e2E jobs fail before the Go tests start because Terraform returns an empty fleet_url; .buildkite/scripts/cloud_e2e_test.sh then exits with status 1. This is a cloud deployment/output issue, not evidence of a failure in the PR’s secret-handling code.

Remediation

• Inspect the complete test:cloudE2EUp output and the resulting ECH deployment to determine why integrations_server.endpoints.fleet is absent; rerun after a Fleet endpoint is provisioned and terraform output fleet_url is non-empty.
• Preserve the full pre-cleanup log on the next run; the supplied artifacts contain only the teardown and the generic exit-status line.
• Once the endpoint is available, rerun both standard and FIPS cloud e2E jobs. If the tests then fail on POST /_fleet/secret, verify the required Elasticsearch change elastic/elasticsearch#154498 is in the selected snapshot (dev-tools/integration/.env still points to 9.6.0-b07e0453-SNAPSHOT).

Investigation details

Root Cause

The captured teardown plan for both jobs reports fleet_url = "" while the Elasticsearch, Kibana, and integrations-server HTTPS endpoints are populated. In .buildkite/scripts/cloud_e2e_test.sh:31-35, that output is assigned to FLEET_SERVER_URL and an empty value causes the script to exit at line 44, before mage test:cloudE2ERun at line 48. The Terraform output is explicitly empty when integrations_server.endpoints is null (dev-tools/cloud/terraform/outputs.tf:28-30).

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/15692
• Jobs: Cloud e2e Test and Cloud e2e FIPS Test
• Key log evidence in both artifacts: fleet_url = "" followed by Error: The command exited with status 1; both logs then show only deployment destruction.
• The existing flaky-test issue [Flaky Test] TestStandAloneRunningSuite/TestOpAMP signal: killed #6590 concerns TestStandAloneRunningSuite/TestOpAMP, which is unrelated to these cloud jobs.

Verification

• Not run locally: the cloud E2E path requires the provisioned ECH environment, and the provided logs do not contain the deployment command’s failure details or the FLEET_SERVER_URL is empty line.

Follow-up

If a full rerun reaches the Go tests, separately validate the PR’s documented Elasticsearch dependency before attributing any later / _fleet/secret error to this build.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[blakerouse]

Overall the code looks good, but I do have a worry. Correct me if I am off base here.

Being that the API key is now in secrets and this is not running like a transactional database across two tables there is a case that might not be handled correctly.

Write the secret to the secret store, but then writing the reference into .fleet-agents fails. I don't see where if that fails the secret is then deleted. I feel like it would need to be deleted then, because it is just left in there with no reference.

Also what happens if writing into the secret store fails? Does that just error back to the caller and they make the API call again? Seems like it, I believe that is correct.

[ycombinator]

Write the secret to the secret store, but then writing the reference into .fleet-agents fails. I don't see where if that fails the secret is then deleted. I feel like it would need to be deleted then, because it is just left in there with no reference.

Yep, good catch. This is a gap. Fixed now in 0818c0e.

Also what happens if writing into the secret store fails? Does that just error back to the caller and they make the API call again? Seems like it, I believe that is correct.

Yes, that's what happens. This early return when bulker.WriteSecret fails means no secret is written and the Agent doc is not updated either. When Agent makes the API call again, the whole flow will be retried.

[blakerouse]

Nice! Thanks for the fixes.