Skip to content

fix(thread-service): serialize sequence-number assignment per thread#3494

Merged
bokelley merged 3 commits into
mainfrom
bokelley/move-thread-service-to-integration
May 4, 2026
Merged

fix(thread-service): serialize sequence-number assignment per thread#3494
bokelley merged 3 commits into
mainfrom
bokelley/move-thread-service-to-integration

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

`server/tests/unit/thread-service.test.ts` was misfiled. Its own header described the tests as integration tests requiring a running Postgres, and it was guarded with `describe.skipIf(!process.env.DATABASE_URL)`. The unit job never sets `DATABASE_URL`, and the integration job globs only `server/tests/integration/`, so all 39 tests skipped in both jobs and ran nowhere.

Why this surfaced now

Audited the actual coverage of #3094's CI fix (#3292) and discovered the gap. Of 60 integration files, 59 run and 1 is intentionally env-gated on `COMPARE_LIVE=1` (`creative-agent-comparison`). The thread-service file was the one quiet anomaly #3094's audit didn't catch — the file lived in `unit/`, not `integration/`, so the survey of `integration/` coverage missed it.

What changed

  • `git mv` from `server/tests/unit/thread-service.test.ts` to `server/tests/integration/thread-service.test.ts`. Relative imports (`../../src/...`) resolve identically — both directories are at the same depth from `src/`.
  • Header refresh: "ThreadService Unit Tests" → "ThreadService Integration Tests" + a comment explaining the location and why the `skipIf` stays as defense-in-depth.
  • No code changes; no test changes.

Coverage gain

39 tests on `ThreadService` (createOrCreateThread, addMessage, listThreads, paging, ON CONFLICT semantics) now actually run on every PR via the post-#3292 `server-integration` job. Previously they existed but couldn't catch any regression because they never executed.

Why `--empty` changeset

Test reorganization, no protocol-spec surface change.

Test plan

  • `npx tsc --project server/tsconfig.json --noEmit` — clean.
  • File rename at same depth → relative imports resolve.
  • Verify the `server-integration` CI job picks it up and reports 39 passing tests on this PR's CI run.

🤖 Generated with Claude Code

bokelley and others added 2 commits May 3, 2026 19:58
The file's own header described its tests as integration tests requiring
a running Postgres, and it was guarded with describe.skipIf(!DATABASE_URL).
The unit job never sets DATABASE_URL, and the integration job globs only
server/tests/integration/, so all 39 tests skipped in both jobs and ran
nowhere.

Surfaced while auditing #3094's CI fix (#3292): of 60 integration files,
59 run cleanly and 1 (creative-agent-comparison.test.ts) is intentionally
env-gated on COMPARE_LIVE=1. The thread-service file was the one quiet
anomaly #3094's audit didn't catch — the file lived in unit/, not
integration/, so the survey of integration/ coverage missed it.

Move is a git mv plus a header refresh. The describe.skipIf stays as
defense-in-depth so a developer running this file outside the integration
job without DATABASE_URL set sees a clean skip rather than a connection
error.

server/tests/integration/threads-api.test.ts covers the HTTP API layer;
this file covers the service layer. Both run now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
addMessage's SELECT MAX(sequence_number) + 1 ... INSERT pattern under
READ COMMITTED has a classic read-modify-write race: two concurrent
calls on the same thread observe the same MAX and write the same next
value. The (thread_id, sequence_number) index is non-unique so the
collision is silent, leaving getThreadMessages's ORDER BY
sequence_number ASC nondeterministic between the duplicates. Affects
parallel Slack thread posts, webhook bursts, and fast tool-call
sequences — exactly the workloads addMessage was built for.

Surfaced by the existing concurrent-addMessage integration test once
the test file was moved into tests/integration/ where the runner
actually picks it up.

Take pg_advisory_xact_lock(hashtext(thread_id)) at the top of the
transaction. Same pattern as billing/org-intake-lock. Locks only
serialize on the same thread; cross-thread inserts run unimpeded.
Lock release is automatic on COMMIT/ROLLBACK.

Defense-in-depth UNIQUE(thread_id, sequence_number) is intentionally
deferred to a follow-up PR — production likely contains rows with
duplicate sequence numbers from this code path, and adding the
constraint without a dedup pass would break the migration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley force-pushed the bokelley/move-thread-service-to-integration branch from 1a41b54 to 24f4eeb Compare May 4, 2026 00:03
@bokelley bokelley changed the title chore(tests): move thread-service.test.ts from unit to integration fix(thread-service): serialize sequence-number assignment per thread May 4, 2026
@bokelley

bokelley commented May 4, 2026

Copy link
Copy Markdown
Contributor Author

Extended this PR with the actual fix. The file move alone unmasked a real concurrency bug — moving the test was step 1, fixing what it caught is step 2.

Bug

ThreadService.addMessage assigned sequence_number via SELECT COALESCE(MAX(sequence_number), 0) + 1 ... INSERT inside a READ COMMITTED transaction. Two concurrent calls on the same thread observe the same MAX and write the same next value. Schema's (thread_id, sequence_number) index is non-unique, so collisions are silent — getThreadMessages's ORDER BY sequence_number ASC then returns nondeterministic ordering between duplicates.

Affects parallel Slack thread posts, webhook bursts, and fast tool-call sequences.

Fix

pg_advisory_xact_lock(hashtext(thread_id)) at the top of the transaction. Same pattern as billing/org-intake-lock. Serializes only on the same thread; cross-thread inserts run unimpeded; lock release is automatic on COMMIT/ROLLBACK.

Deferred

Defense-in-depth UNIQUE(thread_id, sequence_number) constraint is intentionally not in this PR. Production rows likely contain duplicates from the racey code path; adding the constraint without a dedup pass first would break the migration. Follow-up PR after a dedup audit.

Branch rebased onto current main (pulls in #4031 IA changes). The should handle concurrent addMessage calls with correct sequence numbers test should now pass for the right reason.

@bokelley

bokelley commented May 4, 2026

Copy link
Copy Markdown
Contributor Author

Got it — advisory lock serializes per-thread with no cross-thread contention, and deferring the UNIQUE constraint until after a dedup audit is the right call. Clean approach.


Generated by Claude Code

…sage

Security review surfaced a lock-wait DoS vector: without lock_timeout
the per-thread advisory lock queues waiters indefinitely on a stuck
holder, burning pool connections. Mirror billing/org-intake-lock's
SET LOCAL pattern with looser values appropriate for a fast INSERT
workload (5s lock wait, 10s statement ceiling).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley merged commit a7961d0 into main May 4, 2026
14 checks passed
@bokelley bokelley deleted the bokelley/move-thread-service-to-integration branch May 4, 2026 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant