Add checkin capacity-rejection rate metric to /stats

[ycombinator]

What is the problem this PR solves?

The EPA autoscaler for Fleet Server (configured in fleet-controller) currently scales on http_server.tcp_active (active TCP connections). This has repeatedly failed to scale up pods fast enough under load in Serverless scale tests — including the 20k → 30k drone check-in test tracked in elastic/ingest-dev#7719, and other runs such as a recent 5k test where the autoscaler did not add pods when it should have.

The root cause: when a pod is saturated, agents being turned away don't count as "active" connections, so tcp_active understates real demand and the autoscaler under-reacts.

In this comment, @cmacknz suggested scaling on the rate of check-ins rejected because Fleet Server was at capacity instead — a server-side saturation signal that (unlike checkin volume) can't be inflated by a valid-but-misbehaving or malicious client.

Fleet Server already counts these rejections (http_server.routes.checkin.limit_max), but it's a monotonically increasing counter, not a rate — not directly usable as an autoscaler trigger (the EPA reads metric values as levels, not deltas).

How does this PR solve the problem?

Adds a new gauge, http_server.routes.checkin.limit_max_rate, to the internal /stats endpoint. A new background subsystem (api.RunCheckinRejectionRateSampler, wired into Fleet.runSubsystems alongside the other subsystems) samples the existing limit_max counter every 10s and publishes the delta as a rejections/sec rate.

This is intentionally computed inside Fleet Server, not left to the consumer (e.g. an autoscaler) to derive from the raw counter, so the rate is available directly via /stats in a form ready to use as a metric target.

This PR only adds the metric; it does not change any existing behavior or defaults. A follow-up draft PR to fleet-controller (elastic/fleet-controller#623) wires the EPA to this new metric as an additional (not replacing) autoscaling trigger.

⚠️ Deployment sequencing note: the companion fleet-controller change must not be enabled until this change has been rolled out to all Fleet Server pods. The EPA has no fallback for a metric that's missing from a pod's /stats response — a missing metric aborts that reconcile cycle entirely (including evaluation of the existing tcp_active trigger), rather than degrading gracefully. See the fleet-controller PR for how this is gated (metric is opt-in via config, defaulted off).

How to test this PR locally

1. Run fleet-server with the internal HTTP server enabled (http.enabled: true, default port 5066).
2. curl localhost:5066/stats | jq .http_server.routes.checkin.limit_max_rate — should read 0 at idle.
3. Drive check-ins past checkin_limit.max (e.g. via a load test) and confirm the value rises, then falls back to 0 once rejections stop.
4. Or run go test ./internal/pkg/api/... -run TestComputeRate and -run TestRunCheckinRejectionRateSampler for the unit-level behavior, and go test -tags integration ./internal/pkg/api/... -run TestMetricsEndpoints for the /stats JSON shape.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer. (Per-pod gauge, sampled from per-pod in-memory counters only — no shared state.)
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected. (Intended as part of the ingest-dev#7719 scale test follow-up; not yet run at 100K+.)
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc. (Purely observational — a read-only sampler goroutine on a 10s ticker; adds no load to the request path.)

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (none needed — no new config)
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Relates elastic/ingest-dev#7719

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[macdewee]

LGTM

[ycombinator]

@cavokz I am unable to add you as a reviewer on this PR but would like your input as well. If not specifically on the implementation then at least on the idea behind this enhancement. Thank you!

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[cmacknz]

Nice, will be very curious to see how well this works once wired up to the autoscaler.