Export metrics from the standalone symphony-server

[kriszyp]

Why

symphony's counters are only reachable through the napi metrics() call, so a deployment running symphony-server as its own OS process — which is how host-manager supervises it — has no way to read them at all. The per-listener counters were worse off: they were incremented on every hot path and then never read by anything.

Companion PR: HarperFast/host-manager#166 turns the endpoint on.

What

Export path — ts/admin.ts

An optional admin block in the server config exposes GET /metrics (Prometheus text), /metrics.json, and /health over a Unix socket, a loopback TCP port, or both. Omitted by default; nothing is exposed unless configured.

{ "admin": { "socketPath": "/run/symphony/admin.sock", "port": 9095, "host": "127.0.0.1" } }

Blocked and error counts are emitted only under their reason label — the labelled series sum to the total by construction, so a separate unlabelled metric would be a second representation of the same number (sum without(reason)).

Counters — src/metrics.rs

• Per-listener counters are surfaced through metrics(), one entry per listener with its address and mode.
• total_errors was a single bucket for TLS handshake failure, upstream connect failure, idle timeout and copy error — four different alerts collapsed into one number. Errors and blocks are now broken out by reason, using a fixed-size array indexed by enum discriminant so the hot path stays allocation-free.
• Byte counters, via a CountingStream wrapper on the client side. Reading copy_bidirectional's return value instead would lose every byte of any session that ends by idle timeout or reset — most long-lived ones.
• Suspended connections record how they ended; the route table exposes its live and cert-failing route counts.

What gets exported

12 metric names. Every reason is emitted even at zero, so a dashboard series exists before the first incident rather than appearing mid-outage.

Metric	Type	Labels	Meaning
symphony_build_info	gauge	version	Always 1; the version rides the label
symphony_start_time_seconds	gauge	—	Process start, unix seconds
symphony_config_reload_time_seconds	gauge	—	Last successful config reconcile
symphony_routes	gauge	proxy	Routes in the live table, including the default route
symphony_routes_failing	gauge	proxy	Routes whose cert failed to build — dropped, or serving a carried-forward last-good cert
symphony_suspended_pending	gauge	proxy	Connections held awaiting resolveConnection()
symphony_suspended_total	counter	proxy, outcome	outcome is resolved or unresolved
symphony_listener_active_connections	gauge	proxy, listener, mode	Currently being proxied
symphony_listener_accepted_total	counter	proxy, listener, mode	Connections accepted for proxying
symphony_listener_bytes_received_total	counter	proxy, listener, mode	client → upstream
symphony_listener_bytes_sent_total	counter	proxy, listener, mode	upstream → client
symphony_listener_blocked_total	counter	+ reason	Rejected before proxying
symphony_listener_errors_total	counter	+ reason	Failed after admission

Block reasons — max_connections, cidr_blocked, ja3_blocked, ja4_blocked, incomplete_handshake, no_sni, rate_limited, too_many_connections, penalty_boxed

Error reasons — no_route, route_rate_limited, suspend_unresolved, tls_handshake, tls_missing_cert, upstream_connect, idle_timeout, stream, http_header

idle_timeout does not currently mean what it says. forward() wraps the copy in a tokio::time::timeout, which is a hard deadline that never resets on I/O — so idleTimeoutMs cuts busy long-lived connections at their total duration, not quiet ones. That is pre-existing on main (this branch only changed the error mapping around it) and is filed as #34. Until it is fixed, this bucket counts duration-cap terminations.

Sample scrape

Trimmed from real renderer output — a :443 TLS proxy and a :80 HTTP one:

# HELP symphony_build_info Always 1; the version is carried in the label.
# TYPE symphony_build_info gauge
symphony_build_info{version="1.1.0"} 1
# HELP symphony_routes Routes in the live table, including the default route.
# TYPE symphony_routes gauge
symphony_routes{proxy="443"} 138
symphony_routes{proxy="80"} 138
# HELP symphony_routes_failing Routes whose cert failed to build — dropped, or serving a carried-forward last-good cert.
# TYPE symphony_routes_failing gauge
symphony_routes_failing{proxy="443"} 1
# HELP symphony_listener_active_connections Connections currently being proxied.
# TYPE symphony_listener_active_connections gauge
symphony_listener_active_connections{proxy="443",listener="0.0.0.0:443",mode="tls"} 412
symphony_listener_active_connections{proxy="80",listener="0.0.0.0:80",mode="http"} 3
# HELP symphony_listener_accepted_total Connections accepted for proxying.
# TYPE symphony_listener_accepted_total counter
symphony_listener_accepted_total{proxy="443",listener="0.0.0.0:443",mode="tls"} 1482137
# HELP symphony_listener_bytes_sent_total Bytes written to clients (upstream → client).
# TYPE symphony_listener_bytes_sent_total counter
symphony_listener_bytes_sent_total{proxy="443",listener="0.0.0.0:443",mode="tls"} 812384123991
# HELP symphony_listener_blocked_total Connections rejected before proxying, by reason.
# TYPE symphony_listener_blocked_total counter
symphony_listener_blocked_total{proxy="443",listener="0.0.0.0:443",mode="tls",reason="cidr_blocked"} 3
symphony_listener_blocked_total{proxy="443",listener="0.0.0.0:443",mode="tls",reason="rate_limited"} 24
symphony_listener_blocked_total{proxy="443",listener="0.0.0.0:443",mode="tls",reason="no_sni"} 0
# HELP symphony_listener_errors_total Connections that failed, by reason.
# TYPE symphony_listener_errors_total counter
symphony_listener_errors_total{proxy="443",listener="0.0.0.0:443",mode="tls",reason="no_route"} 12
symphony_listener_errors_total{proxy="443",listener="0.0.0.0:443",mode="tls",reason="tls_handshake"} 5
symphony_listener_errors_total{proxy="443",listener="0.0.0.0:443",mode="tls",reason="upstream_connect"} 4
symphony_listener_errors_total{proxy="80",listener="0.0.0.0:80",mode="http",reason="http_header"} 7

/metrics.json

The same snapshot, unrendered — for a collector that would rather not parse text:

{
  "pid": 1234,
  "version": "1.1.0",
  "startedAt": "2026-07-27T09:00:00.000Z",
  "reloadedAt": "2026-07-27T14:12:03.000Z",
  "proxies": [
    {
      "ports": "443",
      "metrics": {
        "activeConnections": 412,
        "blockedConnections": 27,
        "pendingSuspended": 0,
        "suspendedResolved": 0,
        "suspendedUnresolved": 0,
        "routes": 138,
        "failingRoutes": 1,
        "listeners": [
          {
            "address": "0.0.0.0:443",
            "mode": "tls",
            "activeConnections": 412,
            "accepted": 1482137,
            "blocked": 27,
            "errors": 348,
            "bytesReceived": 91238412734,
            "bytesSent": 812384123991,
            "blockedByReason": [{ "reason": "rate_limited", "count": 24 }],
            "errorsByReason": [{ "reason": "upstream_connect", "count": 4 }]
          }
        ]
      }
    }
  ]
}

Cardinality

The proxy label is the port-set of the config entry a listener belongs to; each entry keeps its own route table. On a Fabric host that is 5 port-sets × 1 listener × (4 + 9 + 9 series) ≈ 110 series — flat, regardless of how many tenants the host fronts, because no label carries a route or SNI.

Invariants worth reviewing

Three places where the obvious implementation is subtly wrong, and the fix is structural rather than a guard:

Totals are derived, never maintained alongside their parts. A total_blocked incremented next to its reason counter is two non-atomic writes, so a scrape landing between them sees a total that disagrees with its own breakdown — the invariant would hold only while the proxy is idle, precisely when nobody is reading it. Totals are summed from the same values reported alongside them.

The idle timeout is its own error kind, not an inferred one. Classifying on io::ErrorKind::TimedOut would misfile a peer's kernel-level ETIMEDOUT as an idle timeout.

A live Unix socket is never deleted. Reclaiming a stale socket requires ECONNREFUSED specifically (an EACCES from a restrictively-permissioned live socket is not evidence nobody is listening) and an inode that is actually a socket (a socketPath misconfigured onto a regular file would otherwise be deleted). The bind happens on a pid-unique temp path that is renamed into place, so there is no probe→unlink→bind window for a second process to delete a socket the first has already bound. On shutdown the published path is unlinked only while its inode is still ours. Same family as the status.json ownership guard.

Behaviour changes worth flagging

• A bind failure never affects proxying — logged and retried on a timer rather than thrown out of a reconcile. This is load-bearing during a version upgrade, where the incumbent still holds the socket while the successor is already serving traffic through SO_REUSEPORT.
• blockedConnections now includes maxConnections rejections. It didn't before, which meant the proxy-wide total wasn't the sum of its listeners. Small semantic change to a shipped field.

Security posture

Metric labels are proxy, listener, mode, reason, outcome, version — no SNI, hostname, or other tenant identifier. That is what makes a loopback binding acceptable for consumers; a per-route label would turn aggregate proxy health into a list of which customers a host fronts. There is a test pinning the allowed label set so adding one is a deliberate, visible decision.

Testing

101 JS tests + 101 Rust tests, clippy clean. New coverage: per-listener breakdown and exact byte counts, each error classification (no_route, upstream_connect, idle_timeout vs stream), Prometheus output shape and sample grouping, the endpoint over both transports, stale-socket reclaim after a real SIGKILL, and — the inverse, which is the one that costs something to get wrong — a live socket surviving a would-be reclaimer whose probe fails with EACCES.

Generated with Claude Opus 5.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive metrics and monitoring system for the symphony proxy, including detailed in-process metrics, a Prometheus text renderer, and an optional out-of-process admin HTTP endpoint over Unix sockets or TCP. While the implementation is thorough and well-tested, several critical issues were identified: the idle timeout implementation in src/proxy_conn.rs acts as a maximum connection duration limit that will prematurely terminate active long-lived connections; a subtle race condition in ts/admin.ts could lead to the incorrect deletion of a successor's live socket on shutdown; the socket probe lacks a timeout and could hang indefinitely; and test teardown blocks in __test__/metrics.spec.ts cause unnecessary 3-second delays by failing to check for process signal termination.

[sleekmountaincat]

.____     ________________________   
|    |   /  _____/\__    ___/     \  
|    |  /   \  ___  |    | /  \ /  \ 
|    |__\    \_\  \ |    |/    Y    \
|_______ \______  / |____|\____|__  /
        \/      \/                \/