Reader infrastructure: incremental cursors and git-canonical project keys

[willwashburn]

Context

Two related reader upgrades, bundled because they both land across packages/reader + packages/ledger and share test infrastructure.

Part A: Incremental file cursor

Today parseClaudeSession in packages/reader/src/claude.ts re-parses every session JSONL file in full on every run. On a user with months of sessions this is the hot path for burn summary, and it creates unnecessary duplicate-append pressure on the ledger.

Plan (pattern cribbed from TokenTracker rollout.js:74-183):

• Persist cursor state in \$RELAYBURN_HOME/cursors.json:

  { \"files\": { \"/abs/path/session.jsonl\": { \"inode\": 12345, \"offsetBytes\": 98765, \"mtimeMs\": 1700000000000 } } }

• On next run: fstat each session file. If inode unchanged and mtime ≥ stored mtime, seek to offsetBytes and parse only the tail. Otherwise treat as a new file and parse from zero (log rotation / file replacement).
• Tail-safety: only advance offsetBytes past the last complete newline. Never record a position mid-line.
• Concurrency: file-lock cursors.json during write (session read is append-only and safe without one).

Part B: Git-canonical project key

Today TurnRecord.project is set to cwd at claude.ts:141. That means /Users/will/Projects/burn and /Users/will/burn-worktree-2 — the same repo — roll up separately, and nothing rolls up across machines.

Plan (pattern from TokenTracker rollout.js:1608-1630):

• Add a small helper: resolveProject(cwd): { project: string, projectKey?: string }.
• Walk up from cwd looking for .git/config. Parse [remote \"origin\"] url. Canonicalize to host/owner/repo (strip .git, strip git@host:, normalize https://host/ to host/).
• Keep project: cwd for backward compatibility; add projectKey as the rollup key.
• Queries group by projectKey when present, fall back to project.

Part C: Ledger idempotency

While we're in the ledger: dedup by (source, sessionId, messageId) hash on append. Prevents double-counting when a session is re-parsed (which will still happen anytime a file is rewritten — e.g. Claude Code's session save cadence).

• Maintain a secondary sidecar index at \$RELAYBURN_HOME/ledger.idx — a simple newline-delimited list of hashes, or a Bloom filter if memory becomes a concern.
• On appendTurns, skip any turn whose hash is already indexed.
• Expose a ledger.rebuildIndex() for recovery.

Acceptance

• Second burn summary run over the same data is ≥ 10× faster than the first (measured on a fixture with ≥ 100k turns).
• A session from /Users/will/Projects/burn and a copy at /tmp/burn-worktree-2 roll up under the same projectKey in burn summary.
• Repeated parse of the same session file produces zero duplicate ledger entries. Verified by a test that parses the same fixture twice and asserts ledger byte-length is unchanged on the second pass.
• Log rotation (inode change) correctly triggers full re-parse of the new file.

Depends on

Nothing. Can land in parallel with #1.

[willwashburn]

Dedup pattern refinement from tokscale (crates/tokscale-core/src/sessions/opencode.rs:185-197, 217-223).

Problem with ID-only dedup

The spec in this issue uses (source, sessionId, messageId) hash as the dedup key. That's correct for the common case but misses two scenarios:

1. Path migrations with ID reassignment. Same logical session arriving via two readers (SQLite + legacy JSON during an OpenCode upgrade) may have subtly different messageId generation. Tokscale runs into this for OpenCode and handles it explicitly.
2. Replayed / rewritten session files where content is identical but IDs regenerate.

Tokscale's approach: content fingerprint as secondary check

They hash (timestamp, model, tokens, cost) — a content fingerprint independent of IDs. On ingest, a turn is considered duplicate if either the ID hash matches or the content fingerprint matches a recently-ingested turn (within some window).

They also maintain a migration cache at ~/.cache/tokscale/opencode-migration.json that records when a full migration has completed, letting them skip redundant scans of the now-deprecated path.

Proposed refinement to #4's spec

Keep the primary (source, sessionId, messageId) hash as the fast path — cheap, covers 99% of cases, no false positives.

Add a secondary check: when a new turn's ID-hash doesn't match anything in the index, also check a content-fingerprint hash (ts, model, input+output tokens, cacheRead, cacheCreate) against a smaller rolling window (e.g. last 10,000 turns). Two-tier dedup.

• Primary: ID-hash lookup, O(1), catches exact re-parses.
• Secondary: content-fingerprint lookup on unseen IDs, O(1) in a rolling window, catches migration cases.

False-positive risk on the secondary: two legitimately distinct turns with identical timestamp-to-the-second, identical model, identical token counts. Possible but rare, and the rolling window bounds it. For extra safety, include a short prefix of the first tool call's argsHash in the fingerprint.

Migration-cache note

If burn grows a migration-aware path (e.g. OpenCode legacy → SQLite), consider tokscale's migration-cache pattern so we don't re-hash the full historical dataset on every ingest after migration completes.

Priority

Low-medium. The primary ID-hash dedup is sufficient for current scope (Claude Code JSONL + OpenCode). Add the secondary check when a second path for the same source lands — realistically, that's the OpenCode SQLite+JSON dual-path, which PR #9 handles via the SQLite path only (legacy JSON support deferred).

[willwashburn]

All three parts plus the follow-up landed:

• Part A (incremental cursor): packages/ledger/src/cursors.ts persists {inode, offsetBytes, mtimeMs} in cursors.json; rotation detection + tail-seek in packages/cli/src/ingest.ts:100-132.
• Part B (git-canonical project key): resolveProject() in packages/reader/src/git.ts:11 — walks up for .git/config, canonicalizes remote "origin" to host/owner/repo. Wired through claude/codex/opencode readers.
• Part C (ledger idempotency): packages/ledger/src/index-sidecar.ts with rebuildIndex() at :111. Two-tier dedup from the follow-up comment is in too: both turnIdHash and turnContentFingerprint exported; rebuildIndex() returns { ids, content }. CLI: burn rebuild and burn rebuild-index.