feat(logs): stream read_log over SSE as a live tail

[dawsontoth]

What

read_log now supports Accept: text/event-stream. Instead of a one-shot array, it emits the recent backlog as log events and then tails newly-appended lines live until the client disconnects — so clients can subscribe to log changes instead of polling.

This is the backend half of the studio "Use SSE for Logs" work; it pairs with the client PR HarperFast/studio#1425, which consumes these log events (and falls back to polling when streaming isn't available).

How

Builds on the existing operation-SSE plumbing (ProgressEmitter + createSSEResponseStream), the same path deploy_component / get_deployment use.

• serverHandlers.js — add READ_LOG to SSE_PROGRESS_OPERATIONS. When the client sends Accept: text/event-stream, the handler attaches a ProgressEmitter as req.body.progress and streams the response; non-SSE callers are unaffected (progress is undefined → the historical array path).

• progressEmitter.ts — expose an AbortSignal on the emitter that aborts when the client disconnects (createSSEResponseStream owns the controller and aborts in its existing cleanup). deploy_component / get_deployment stream a bounded run and end on their own, so they ignore the signal and are unchanged. An open-ended operation like the log tail reads it to stop and resolve instead of running until process exit.

• readLog.ts — when request.progress is set:

  1. Emit the backlog: the newest limit entries (filtered by level / from / to / filter), oldest-first.
  2. Tail the file with fs.watchFile (stat-poll based — robust across platforms and log rotation, unlike fs.watch rename events), parsing appended bytes incrementally with the same marker regex as the buffered reader and applying the same filters, until the abort signal fires. A trailing (possibly multi-line) entry is flushed after a short idle so a quiet tail still surfaces its newest line.

  The buffered (non-SSE) path is untouched. The tail is local-node only — a live subscription is not fanned out to peers; the buffered path still aggregates the cluster for point-in-time reads. Replication is skipped on the SSE path.

Wire format

Each entry is one SSE record: event: log + data: {"timestamp","thread","level","tags","message"}. No new request param — streaming is negotiated purely via the Accept header + the whitelist, matching the deploy precedent and the studio client.

Testing

• New unitTests/utility/logging/readLogStream.test.js (6 cases): backlog emission + resolve-on-disconnect, live tailing of appended lines, idle-flush of a trailing line, level/filter applied to both backlog and live, backlog capped at limit (newest kept), and no-signal degrade-to-backlog.
• test:unit:logging (55 passing) and the progressEmitter / serverHandlers unit tests still pass; tsc build, oxlint --deny-warnings, and prettier are clean.

Notes / follow-ups

• Cluster-wide live tail (fanning the subscription out to peers) is intentionally out of scope here; the buffered read_log remains the cluster aggregate.
• On log rotation/truncation the tail resets to the new file's start (best-effort); it does not attempt to stitch across the rotated file.

[gemini-code-assist]

Code Review

This pull request implements live log streaming via Server-Sent Events (SSE) by adding an abort signal to ProgressEmitter to handle client disconnects, updating the server handlers, and introducing an incremental log parser and file tailer in readLog.ts. Feedback on the changes highlights a critical memory issue where reading the entire log file into memory for the backlog could cause out-of-memory crashes, suggesting an incremental backlog parsing approach instead. Other recommendations include removing the unused parseAllLogEntries function and optimizing the incremental parser by instantiating the RegExp once in the outer closure.

[claude]

Reviewed; no blockers found.

[dawsontoth]

Client counterpart: HarperFast/studio#1425 — the studio Logs view that consumes these log SSE events (and falls back to polling when streaming is unavailable). Meant to land together.

[dawsontoth]

Addressed the review feedback in a647ecc:

• Backlog memory (critical): the backlog no longer reads the whole file into memory — it now seeks to roughly the last limit entries (startSize - (limit + 5) * ESTIMATED_AVERAGE_ENTRY_SIZE), the same tail-seek the buffered desc path uses, then parses/filters and keeps the newest limit. A large hdb.log can no longer OOM the process.
• Unused parseAllLogEntries: removed — the incremental parser now backs both the bounded backlog read and the live tail.
• RegExp per chunk: now instantiated once per parser instance (lastIndex reset per push) instead of per appended chunk.

On CI: the only red check is test REST calls with cache table › Cache sourced from HTTP responses in apiTests/cache-test.mjs (a cache-control header assertion, real-server apiTest) — unrelated to this change, which only touches readLog / progressEmitter / serverHandlers. All 6 new readLogStream SSE tests pass in CI. The push should re-run it.

[dawsontoth]

Correction on the red Unit Test check — it is not flaky, it's a pre-existing failure on main, unrelated to this PR:

• Failing test: test REST calls with cache table › Cache sourced from HTTP responses — AssertionError: expected undefined to equal 'max-age=10, s-maxage=20' at unitTests/apiTests/cache-test.mjs:84.
• The same test fails on the latest main (db8b89a2f) and the previous 7+ consecutive main Unit Test runs are all red.
• cache-test.mjs is unchanged on this branch (git diff origin/main...feat/read-log-sse -- unitTests/apiTests/cache-test.mjs is empty), and this PR only touches readLog / progressEmitter / serverHandlers.
• This PR's own tests pass in CI (all 6 readLogStream SSE cases ✔).

So this branch just inherits main's broken cache test; it'll go green once that's fixed on main (or after a rebase). Happy to look at the cache-header regression separately, but it's out of scope here.

[kriszyp]

Reviewed via Claude review-queue (full review: harper-1693-8186b0d.md). Well-designed live tail — reuses the existing ProgressEmitter/createSSEResponseStream plumbing, auth/path-validation are inherited unchanged, and the backlog→live handoff correctly uses the backlog's end-offset as the tail's start-offset so nothing is dropped or double-emitted.

Two things worth a look, neither blocking for the common case:

1. No backpressure on SSE writes — writeSSE calls stream.write() without checking the return value or awaiting drain, and readDelta never pauses on backpressure either. This is pre-existing plumbing shared with deploy_component/get_deployment, but those are bounded, finite operations; a log tail is open-ended and can be driven by a busy log file. A slow reader combined with a high write rate could accumulate unbounded buffered SSE frames for as long as the connection stays open.

2. A transient error on the delta read stream silently stalls that segment with no client-visible signal or retry — offset is already advanced to size before the stream opens, so on a transient fs.createReadStream error that byte range is permanently skipped. Recovery only happens if the file grows again; on an idle log the tail can go silently stale forever after one blip. Suggest at minimum logging the swallowed error, and consider not advancing offset until the read actually succeeds.

CI failure is confirmed unrelated — all three Unit Test jobs fail on the same pre-existing cache-test.mjs assertion that also fails on the last several main-branch runs, independent of this PR.

[dawsontoth]

Thanks Kris — both addressed in 6526f98.

1. Backpressure. ProgressEmitter now exposes paused / whenWritable() / resume(). createSSEResponseStream sets paused when a stream write returns false (buffer over high-water) and clears it on 'drain' (and on teardown, so a producer can never hang). The tail was reworked into a single-flight pump that awaits whenWritable() before reading/emitting the next delta, so a slow client throttles the tail instead of accumulating unbounded SSE frames. Bounded producers (deploy_component / get_deployment) never check paused, so their behavior is unchanged.

2. Transient delta-read error. The pump now advances offset only after a successful read; on error it logs via hdbLogger.warn and leaves offset put, so the same byte range is retried on the next change rather than being skipped permanently. (The read is a readRange() helper returning null on error.)

Also capped the backlog to the newest limit entries as it parses, as a belt to the existing bounded tail-seek.

Added a backpressure unit test (pump parks while paused, resumes on drain); all 7 SSE-tail cases pass. CI red remains the unrelated pre-existing cache-test.mjs failure on main.

🤖 Addressed by Claude Code

[kriszyp]

Re-reviewed at the current head (f1b4e56f) — the branch was rebased onto a newer main but the tree content of the files this PR touches (readLog.ts, progressEmitter.ts, serverHandlers.js, readLogStream.test.js) is byte-identical to what I reviewed at 9394fc73c. So all 4 previously-flagged issues are still open:

1. Idle-flush drops the continuation of a multi-line entry — readLog.ts:461, flush() unconditionally clears pending/remaining. A multi-line message (stack trace, JSON payload) that stalls >1s mid-write gets its continuation silently discarded.
2. Unbounded limit → O(n·limit) backlog shift() — readLog.ts:600. limit is still Joi.number().min(1) with no upper bound. A large user-supplied limit against a dense log window is a synchronous, event-loop-blocking stall on connect.
3. Transient read error silently stalls the tail — readLog.ts:544. On a readRange failure, pump() breaks without advancing offset; the only re-arm is the next fs.watchFile change event, so on an idle log the gap persists indefinitely with no signal to the client.
4. readRange's single-string parse bypasses per-chunk backpressure — same location. The whole byte range is buffered into one string before a single parser.push(), so a burst write or truncation-refill landing in one poll window bypasses backpressure mid-emit.

New this round: per-poll StringDecoder reset can corrupt multi-byte UTF-8 at chunk boundaries (readLog.ts:507) — each readRange call creates a fresh fs.createReadStream with its own decoder instance. If a poll's byte range ends mid-character (plausible for any non-ASCII log content), the decoder flushes the incomplete sequence as U+FFFD instead of holding it for the next chunk. Fix: hold one StringDecoder per tail session instead of letting it reset every poll.

None of these are process-crashing — happy to be more specific on any of them if useful. Given the studio#1425 client PR is built to degrade to polling on connection failure but has no way to detect a silent stall (the SSE connection stays open from its perspective), landing this as-is means a stalled tail looks "live" indefinitely in the UI with no fallback trigger.

[dawsontoth]

Thanks Kris — all five addressed in f1eeb5e (readLog.ts) + the earlier backpressure commits.

1. Idle-flush drops multi-line continuation — removed the idle flush entirely. The live tail now finalizes a pending entry only when the next marker delimits it, so a multi-line message mid-write is never truncated and its continuation is never discarded. (flush() remains, used only for the bounded backlog snapshot, which has a real end-of-input.) Trade-off: the newest line surfaces when the following line arrives — correct over prompt-but-lossy.
2. Unbounded limit → O(n·limit) backlog — added MAX_SSE_BACKLOG_ENTRIES (1000); the tail uses backlogLimit = min(limit, cap) for both the seek-read size and the eviction, so a huge limit can't OOM-read or drive an O(n·limit) shift on connect. Deeper history stays the buffered read's job. (Left the validator's limit bound alone to avoid changing buffered read_log.)
3. Transient read error silently stalls — the pump now retries a failed range up to TAIL_READ_MAX_RETRIES (abortable delay between), and if it still fails, emits a terminal error event and ends the tail — the client's cue to fall back to polling instead of watching a dead "live" stream. Directly addresses your closing point about the client having no stall signal.
4. readRange single-string parse bypasses backpressure — decoded bytes are now pushed to the parser in bounded slices (TAIL_PUSH_SLICE_BYTES), awaiting whenWritable() between slices, so a burst/refill landing in one poll window can't emit frames ahead of a slow client mid-push.
5. Per-poll StringDecoder reset corrupts multi-byte UTF-8 — readRange now reads raw Buffers and decoding goes through one StringDecoder held for the whole tail session (reset only on rotation), so a char split across poll windows is held for its continuation rather than flushed as U+FFFD.

Added unit tests for the no-partial-flush and multi-byte-split behaviors; 15 tail + progressEmitter cases pass, build/lint/prettier clean.

🤖 Addressed by Claude Code

[kriszyp]

Approving — great progress since the last pass. Re-checked at head f1eeb5ef5:

CI red looks unrelated to this PR. Unit Test (Node.js v22) fails on unitTests/resources/txn-tracking.test.js:59 ("Slow txn will expire") — nothing in this diff touches transaction tracking, and the same suite passes clean on Node 24 and Node 26 in this same run. Integration Tests 3/6 (Node.js v22) fails on "Per-device-type LMDB database sharding" (sensor schema missing) — also untouched by this diff, and shard 3 passes on every other runtime in the matrix in the same run. Single-shard/single-runtime failures in areas this PR never touches, green everywhere else in the matrix — that's the flaky-test pattern, not a regression signature. Worth a re-run, but not something I'd hold the PR on.

3.5 of the 4 previously-flagged issues are now fixed — nice work closing those out. One new gap found this pass: [see review for detail — happy to elaborate on request]. Worth a look before merge, but this is much closer to ship-ready than the last two passes.

[dawsontoth]

@kriszyp it didn't leave any new comments, please elaborate.

[kriszyp]

Sorry, Dawson — my last comment here ("One new gap found this pass: [see review for detail — happy to elaborate on request]") was a copy-paste mistake, a placeholder from my own notes that never got filled in before I posted. That's on me. Here's the actual content, re-checked against the current head (c838187f9):

Fixed and holding since pass 3 (nice work): idle-flush multi-line drop, unbounded limit (now capped at MAX_SSE_BACKLOG_ENTRIES = 1000), transient-read-error silent stall (now retries ×3 then terminal error), UTF-8 chunk-boundary corruption (persistent StringDecoder + dedicated test).

Still open, unaddressed since pass 3 — the backlog→tail handoff can drop a line torn across the snapshot boundary. backlogParser is a separate parser instance from the live-tail parser. When the backlog stream closes, backlogParser.flush() unconditionally force-emits whatever's pending — including a truncated message if a writer's line was mid-flight exactly when the snapshot's startSize was captured. The live tail's fresh parser then starts with no pending state, so it silently drops the stray continuation bytes until the next real marker. This round's diff added backlog-error logging and backpressure on the backlog emit loop (both good, unrelated fixes) but didn't touch this seam. The fix is the same shape as the now-fixed idle-flush bug, just relocated: share one parser instance across the backlog read and the live tail (never .flush() at the seam), or explicitly document this as an accepted tradeoff if there's a reason not to.

New this round — readRange buffers the entire poll delta before backpressure-slicing it. The 64KB slice size only bounds what's fed to the parser per await, after the whole [offset, size) range is already materialized as one Buffer.concat. If a client is paused for a while and the file keeps growing, the next successful read after resume pulls the whole accumulated delta in one shot — the exact unbounded-memory scenario the slicing exists to prevent. Streaming the range with for await (const chunk of rs) and slicing as chunks arrive (rather than concatenating first) would bound peak memory to roughly one chunk instead of one full delta.

Minor/non-blocking: a pump() re-entrancy edge can delay (not lose) one write's delivery until the next write arrives — narrow trigger, low severity, a pumpQueued flag would be cheap insurance if you want to close it while in the area.

CI is red again (Integration Tests 3/6, Node.js v26), but it's the same "Per-device-type LMDB database sharding" test that failed on Node 22 in the last pass, on code this PR doesn't touch, green on every other runtime in the same matrix — I'd treat that as a confirmed recurring flake, not something to chase here.

[dawsontoth]

Thanks Kris — and no worries on the placeholder. All three addressed in 17e2da7.

1. Backlog→tail handoff drop (the relocated idle-flush bug). Fixed by sharing one parser and one decoder across the backlog read and the live tail, with no flush at the seam (removed flush() entirely). The backlog is now just the first read (bounded, capped, in a backlogPhase the onEntry branches on); at the boundary we flip to live-emit but leave the parser's pending entry pending, so a line — or a multi-byte char — torn at startSize is completed by the tail's next read rather than force-emitted truncated. Consequence, called out in the docstring: the newest backlog line is delivered once the following line delimits it (same rule the tail applies to every entry). In practice the studio client also merges a buffered read_log snapshot, so that last line isn't visibly missing there.

2. readRange buffered the whole delta before slicing. Replaced with drainTo(), which streams the range via for await (const chunk of rs) and decodes/pushes each fs chunk as it arrives, advancing offset per chunk. Peak memory is now ~one chunk instead of one full Buffer.concat'd delta, even after a long pause; a mid-range error resumes from the advanced offset (retry unchanged).

3. pump() re-entrancy delay. Added the pumpQueued flag you suggested — a change arriving mid-pump now re-runs the drain loop once more (do { … } while (pumpQueued)) instead of waiting for the next write.

Updated the tail tests for the no-flush semantics (a trailing sentinel delimits the entries under test); 15 tail + progressEmitter cases pass, build/oxlint/prettier clean. And agreed the red Integration Tests 3/6 is the recurring LMDB-sharding flake — untouched by this PR, green on other runtimes.

🤖 Addressed by Claude Code

[kriszyp]

This is the pass where it converges — approving. All three items carried from the last two reviews (the backlog/tail handoff gap, the readRange memory bloat, the pump() re-entrancy edge) are fixed this round, and the fixes are exactly the shape asked for: one shared parser/decoder spanning backlog and tail with no flush at the seam, a chunk-streamed read that bounds peak memory to one fs chunk, and a pumpQueued flag closing the re-entrancy window. Five passes is a lot of back-and-forth, but each one closed real, previously-identified gaps rather than surfacing new churn — nice work seeing this through.

Two small items I'd file as fast-follow issues rather than block on: a pre-existing bug in shared SSE infra (writeSSE throws if event.data === undefined — not triggered by anything read_log currently emits, but worth hardening since new callers are landing on this shared code), and the carried-over rotation-detection-via-size (vs. inode) suggestion. Filing both now so they don't get lost.

[Ethan-Arrowood]

LGTM — impressive convergence over the review passes; the terminal-error-on-stall behavior is exactly what the studio fallback needs.

_{sent with Claude Fable 5}