Export metrics from the standalone symphony-server

CLAUDE.md

@@ -12,6 +12,19 @@ symphony is a **napi-rs cdylib** loaded by Node.js. The tokio multi-thread runti
 
 For consumers that want symphony as its own OS process rather than embedded in their Node app, the package ships a `symphony-server` bin. It reads a JSON config file (`{ version, proxies: [{ listeners, routes }] }` — one entry per port-set, since the route table is per-proxy), constructs a `SymphonyProxy` per entry, and **watches the config file** to hot-reload (route change → `updateConfig`; listener change → recreate that proxy). Cert material may be given inline (`certChain`/`privateKey`) or by path (`certChainFile`/`privateKeyFile`) — the path form is resolved in `server.ts` only, so the napi `CertConfig` stays inline-only. It writes a `status.json` (`{ pid, version, ports, ... }`) for supervisors, and handles `SIGHUP` (reload) / `SIGTERM`/`SIGINT` (graceful stop). host-manager uses this to supervise symphony out-of-process.
 
+### Admin/metrics endpoint (`ts/admin.ts`)
+
+An optional `admin` block in the config file (`{ socketPath?, socketMode?, port?, host? }`) makes `symphony-server` expose `GET /metrics` (Prometheus text), `/metrics.json`, and `/health` over a Unix socket, a loopback TCP port, or both. It exists because an out-of-process symphony has no reachable napi `metrics()` — the endpoint is the only export path for that deployment.
+
+Three properties are load-bearing and easy to regress:
+
+- **It must never affect proxying.** A bind failure is logged and retried on a 5s timer instead of throwing out of `doReconcile()`. This is not defensiveness for its own sake: during a version upgrade host-manager runs both processes concurrently (the Rust listeners overlap via `SO_REUSEPORT`, which a Node HTTP server has no equivalent for), so the successor *will* lose the admin bind for a few seconds and must keep serving traffic anyway.
+- **A stale Unix socket is reclaimed, a live one is not.** Three things protect this, and each was a real hole once: the probe counts a path reclaimable only on `ECONNREFUSED` (an `EACCES` from a restrictively-permissioned live socket is not evidence nobody is listening); the inode must actually be a socket (a `socketPath` misconfigured onto a regular file would otherwise be deleted); and the bind happens on a pid-unique temp path that is `rename`d into place, so there is no probe→unlink→bind window for a second process to delete a socket the first has already bound. On shutdown the published path is unlinked only while its inode is still the one we put there. Same family as the `status.json` ownership guard.
+- **Counters are read per request**, not cached at reconcile, so a scrape never serves numbers frozen at the last config reload.
+- **Totals are derived, never maintained alongside their parts.** `blocked`/`errors` are summed from the per-reason values in the same snapshot, and the proxy-wide blocked total from the listener values. A separate `total_blocked` incremented next to its reason counter is two non-atomic writes: a scrape landing between them sees a total that disagrees with its own breakdown, so the invariant would hold only while the proxy is idle — precisely when nobody is reading it.
+
+Prometheus shape: blocked/error counts are emitted **only** under their `reason` label (they sum to the unlabeled total, so a separate total would be a second representation of the same number), and the proxy-wide active gauge is `sum without(listener)`. `renderPrometheus` is exported from the package for embedded consumers.
+
 The server also **watches the cert/key files referenced by the config** (grouped by parent dir, deduped, re-derived on every reconcile so watchers don't leak) → a debounced `reconcile()` on change, so an on-disk cert renewal is picked up live without a `config.json` write or restart. Two details make a listener-level cert rotation actually apply: the per-proxy `listenerSig` is computed over the *resolved* listeners (cert contents included), so a rotated `defaultCert`/mTLS file changes the signature and forces a recreate rather than a route-only hot-swap against the frozen `default_listener_tls`. Basename-filtered dir watching handles in-place / rename rotation (what host-manager does); k8s projected-volume `..data` symlink swaps are not yet covered. Cert-failure resilience lives in `router.rs::build_route_table`: a route whose cert can't be built (e.g. rustls `KeyMismatch` from a rotated key vs a stale inlined chain) is isolated — one bad tenant cert never aborts the whole table; on a hot-swap the last-good route is carried forward for that SNI (mid-rotation the old cert is still valid), and on initial build the SNI is simply dropped.
 
 ### Data flow
@@ -46,7 +59,7 @@ TCP accept (SO_REUSEPORT per worker thread)
 | `src/proxy_conn.rs` | Per-connection handler: the full 7-step flow |
 | `src/protection.rs` | IP rate limiting, concurrency, CIDR lists, JA3 blocking |
 | `src/suspended.rs` | Pending-connection registry (DashMap + oneshot channels) |
-| `src/metrics.rs` | AtomicU64 counters: active, accepted, errors, blocked |
+| `src/metrics.rs` | AtomicU64 counters (active, accepted, bytes, per-reason blocks/errors) + `CountingStream` |
 | `src/error.rs` | SymphonyError enum → napi::Error conversion |
 
 ---
@@ -151,6 +164,18 @@ napi `Buffer` contains raw pointers (`*mut napi_env__`, `*mut napi_ref__`) that
 4. Add a new `kind` case in `parse_upstream_spec()` in `proxy.rs`
 5. Add a test
 
+### Adding a new metric
+
+1. Add the counter to `ListenerMetrics`/`GlobalMetrics` in `metrics.rs`, or a variant to the
+   `labeled_enum!` block for `BlockKind`/`ErrorKind` — the variant list drives the counter array,
+   the label, and the export, so there is no second list to update.
+2. Increment it at the call site. A new `protection::BlockReason` variant will fail to compile
+   until `From<&BlockReason> for BlockKind` maps it — that is deliberate, so a new protection
+   check can't land in an unlabeled bucket.
+3. Surface it in `JsProxyMetrics`/`JsListenerMetrics` (`proxy.rs`), `ts/types.ts`, and the
+   mapping in `ts/proxy.ts`.
+4. Add the sample to `renderPrometheus` in `ts/admin.ts`, and a case in `__test__/metrics.spec.ts`.
+
 ### Adding a new napi method
 
 1. Implement in `proxy.rs` with `#[napi]`
@@ -169,6 +194,7 @@ Tests live in `__test__/` and use Node's built-in `node:test` runner.
 - **`protection.spec.ts`** — rate limit token bucket exhaustion, CIDR blocklist in `blockedIps()`
 - **`suspended.spec.ts`** — hold → resolve → proxy, hold → null → close, hold → timeout → drop
 - **`mtls.spec.ts`** — mTLS termination + PROXY v2 TLV forwarding of the client cert chain (0xE2, SSL TLV 0x20); skips without openssl
+- **`metrics.spec.ts`** — per-listener breakdown and byte counting, `renderPrometheus` output shape, the admin endpoint over UDS + TCP, and stale-socket reclaim after a `SIGKILL`
 
 Build and run:
 ```bash

README.md

@@ -599,23 +599,115 @@ attestation roadmap, and the shared-responsibility split for DDoS — see
 
 ## Metrics & monitoring
 
+### In-process (`proxy.metrics()`)
+
 ```typescript
 const m = proxy.metrics();
-// m.activeConnections — connections being proxied right now
-// m.blockedConnections — total blocked since start
-// m.pendingSuspended — connections currently held waiting for resolveConnection()
+// Proxy-wide
+// m.activeConnections     — connections being proxied right now
+// m.blockedConnections    — total rejected since start (protection + maxConnections)
+// m.pendingSuspended      — connections held waiting for resolveConnection()
+// m.suspendedResolved     — suspended connections that were resolved with a route
+// m.suspendedUnresolved   — suspended connections that timed out or were rejected
+// m.routes                — routes in the live table, including the default route
+// m.failingRoutes         — routes whose cert failed to build (see "Per-route certificates")
+
+// Per listener, in configuration order
+for (const l of m.listeners) {
+  // l.address, l.mode ('tls' | 'http')
+  // l.activeConnections, l.accepted
+  // l.bytesReceived — bytes read from clients   (client → upstream)
+  // l.bytesSent     — bytes written to clients  (upstream → client)
+  //   Counted where the proxy sees the bytes. On a terminated-TLS route that is the plaintext
+  //   stream, and the handshake — which precedes the counter — is excluded. On a passthrough
+  //   route the proxy has no plaintext view and forwards wire bytes, so the handshake records
+  //   are part of the stream and are counted.
+  // l.blockedByReason — [{ reason: 'rate_limited', count: 12 }, ...]
+  // l.errorsByReason  — [{ reason: 'upstream_connect', count: 3 }, ...]
+}
 
 const blocked = proxy.blockedIps();
 // blocked.rateLimited — IPs with a depleted per-second or sustained token bucket
 // blocked.concurrencyLimited — IPs at their maxConcurrentPerIp limit
 // blocked.cidrBlocklist — the configured static CIDR blocklist
 // blocked.penaltyBoxed — IPs currently in the penalty box
+```
+
+Every reason is reported on every call, including reasons still at zero, so a dashboard series
+exists before the first incident rather than appearing mid-outage. `l.blocked` / `l.errors` are
+summed from the very reason values reported alongside them, and `m.blockedConnections` from the
+listener values in the same snapshot — so a reading taken mid-traffic is internally consistent
+rather than only adding up while the proxy is idle.
+
+**Block reasons:** `max_connections`, `cidr_blocked`, `ja3_blocked`, `ja4_blocked`,
+`incomplete_handshake`, `no_sni`, `rate_limited`, `too_many_connections`, `penalty_boxed`.
+
+**Error reasons:** `no_route`, `route_rate_limited`, `suspend_unresolved`, `tls_handshake`,
+`tls_missing_cert`, `upstream_connect`, `idle_timeout`, `stream`, `http_header`.
+
+> `idle_timeout` counts terminations by `idleTimeoutMs`, which today is a *total duration* cap
+> rather than an idleness one — see [#34](https://github.com/HarperFast/symphony/issues/34). Busy
+> long-lived connections land in this bucket, not just quiet ones.
+
+### Out-of-process (`symphony-server` admin endpoint)
 
-setInterval(() => {
-  console.log('active:', proxy.metrics().activeConnections);
-}, 10_000);
+When symphony runs as its own process there is no JS API to call, so the server bin can expose
+the same numbers over HTTP. Add an `admin` block to the config file:
+
+```json
+{
+  "version": 1,
+  "admin": {
+    "socketPath": "/run/symphony/admin.sock",
+    "socketMode": 432,
+    "port": 9095,
+    "host": "127.0.0.1"
+  },
+  "proxies": [ ... ]
+}
 ```
 
+Both bindings are optional; give either or both. Omit the `admin` block entirely and nothing is
+exposed. `socketPath` may be relative to the config file's directory, and is chmodded to
+`socketMode` (default `0o660`) after bind. `host` defaults to `127.0.0.1` — metrics carry no
+tenant identifiers, but there is still no reason to publish them off-box.
+
+| Route | Response |
+|---|---|
+| `GET /metrics` | Prometheus text exposition (v0.0.4) |
+| `GET /metrics.json` | the same snapshot as JSON |
+| `GET /health` | `{ ok, pid, version, ports }` |
+
+```
+$ curl --unix-socket /run/symphony/admin.sock http://localhost/metrics
+# HELP symphony_build_info Always 1; the version is carried in the label.
+# TYPE symphony_build_info gauge
+symphony_build_info{version="0.5.0"} 1
+...
+symphony_listener_accepted_total{proxy="80,443",listener="0.0.0.0:443",mode="tls"} 148213
+symphony_listener_blocked_total{proxy="80,443",listener="0.0.0.0:443",mode="tls",reason="rate_limited"} 27
+symphony_listener_errors_total{proxy="80,443",listener="0.0.0.0:443",mode="tls",reason="upstream_connect"} 4
+```
+
+The `proxy` label is the port-set of the proxy entry the listener belongs to (each config entry
+gets its own route table). Blocked and error counts are only ever emitted with their `reason`
+label — the labelled series sum to the total, so use `sum without(reason)` rather than looking
+for a separate unlabelled metric. Likewise the proxy-wide active-connection gauge is
+`sum without(listener) (symphony_listener_active_connections)`.
+
+The endpoint is strictly read-only and best-effort: it never blocks proxying, and a bind failure
+is logged and retried every 5s rather than aborting startup. That matters during a version
+upgrade, where the incumbent still holds the socket while the replacement is already serving
+traffic through `SO_REUSEPORT` — the successor picks up the admin endpoint once the old process
+exits. A socket file left behind by a `SIGKILL`ed process is reclaimed automatically, but only
+after a connect probe proves nobody is listening on it.
+
+The Prometheus renderer is internal to the standalone server and is not exported from the
+package root: a snapshot carries that process's pid, timestamps, and port-set grouping, which an
+embedded consumer would have to synthesise. An embedded proxy has `proxy.metrics()` directly. If
+a caller genuinely needs Prometheus text from an embedded proxy, open an issue — the right shape
+is a `ProxyMetrics`-based renderer, not this one.
+
 ---
 
 ## Hot config updates