Skip to content

feat: NIP-29 native compatibility — standard nostr clients can chat on Sprout - #63

Merged
tlongwell-block merged 27 commits into
mainfrom
nip-native-migration
Mar 14, 2026
Merged

feat: NIP-29 native compatibility — standard nostr clients can chat on Sprout#63
tlongwell-block merged 27 commits into
mainfrom
nip-native-migration

Conversation

@tlongwell-block

@tlongwell-block tlongwell-block commented Mar 14, 2026

Copy link
Copy Markdown
Collaborator

What

Makes Sprout compatible with standard NIP-29 nostr clients (Chachi, 0xchat, any NDK-based app) for basic group chat. The Sprout desktop client and MCP/ACP agent harness retain all current functionality — nothing removed or degraded.

The change in one sentence

KIND_STREAM_MESSAGE moves from 40001 to 9 (NIP-29 group chat), three small features are added, and ~120 test literals are updated.

Why

Internal users want to use open-source nostr clients for basic chat on Sprout relays. Until now, Sprout used a custom kind (40001) that no standard client understands. NIP-29 (relay-based groups) defines kind:9 as the standard group chat message — widely adopted by NDK, Chachi, and 0xchat.

Changes

WI-1: Kind constant change

KIND_STREAM_MESSAGE: 40001 → 9 in kind.rs. The ~62 Rust references auto-propagate at compile time. Hardcoded literals fixed manually across desktop (relayClient.ts, kinds.ts, hooks.ts) and MCP (server.rs). Note: hooks.ts used JavaScript's numeric separator (4_0001 = 40001), which was invisible to rg '40001' — caught via desktop e2e test failure.

WI-2: NIP-29 group discovery emission

New emit_group_discovery_events() in side_effects.rs. Emits relay-signed kind:39000 (group metadata), kind:39001 (group admins), and kind:39002 (group members) events whenever channels are created, updated, or membership changes. Called from both WebSocket NIP-29 handlers and the REST API channel creation endpoint.

  • Events stored channel-scoped (channel_id = Some(...)) so existing access control applies — private channel member lists are only visible to members
  • Addressable event replacement is transactionalreplace_addressable_event() wraps insert + soft-delete in a single DB transaction with SELECT ... FOR UPDATE serialization
  • Composite index idx_events_addressable on (kind, pubkey, channel_id, deleted_at) enables InnoDB next-key gap locking even on cold addresses (no existing rows)
  • Insert-first ordering guarantees at least one active row at all times
  • Discovery events cleaned up on group deletion via soft_delete_discovery_events()
  • Common sign/store/fanout logic extracted into emit_addressable_discovery_event() helper

WI-3: Pubkey allowlist auth

New pubkey_allowlist table (BINARY(32) for fixed-size cryptographic keys) + SPROUT_PUBKEY_ALLOWLIST env var. When enabled, NIP-42 pubkey-only connections (no JWT/API token) are restricted to allowlisted pubkeys. Users with valid API tokens or Okta JWTs bypass the allowlist entirely. Fail-closed on DB errors with explicit warning logs for observability.

WI-4: Desktop transition

Legacy kind:40001 added to CHANNEL_EVENT_KINDS so the desktop sees pre-migration messages during any transition period.

WI-5: Literal updates + data migration

~120 hardcoded 40001 references updated to 9 across test files, comments, TOML config strings, and documentation. SQL migration (20260318000002) converts existing kind:40001 rows to kind:9.

WI-6: Integration test

New test_nip29_standard_client_flow — end-to-end test exercising: group discovery query (39000 + 39001 + 39002), kind:9 send/receive, kind:7 reaction, kind:5 deletion, and h-tag enforcement.

What standard clients get

Feature Works? Notes
Read/send channel messages kind:9 + h tag
Reactions kind:7 (already standard)
Deletions kind:5 (already standard)
Group discovery kind:39000/39001/39002
NIP-42 auth Pubkey allowlist when enabled
Message edits kind:40003 (Sprout-only)
Rich content kind:40002 (Sprout-only)
Threads ⚠️ q-tag replies work; Sprout thread view deferred

What doesn't change

  • Edit/diff/pin/rich kinds (40002–40008) — unchanged
  • Agent jobs, workflows, huddles, presence — unchanged
  • Proxy (auto-updates via constant propagation: kind:9 → kind:42)
  • MCP/ACP tools (use constant, auto-update)
  • Desktop rendering logic — unchanged

Testing

Suite Result
Unit tests 140/140 ✅
REST API e2e 48/48 ✅
WebSocket relay e2e 19/19 ✅
MCP e2e 14/14 ✅
Multi-agent ACP e2e All exercises ✅
Third-party nostr client (nak 0.18.7) All NIP-29 features ✅
Desktop smoke e2e 57/57 ✅

Crossfire Review

Reviewer Rounds Final Score Status
Opus R1→R2 9/10 APPROVE All 8 R1 issues fixed
Codex CLI R1→R2→R3→R4 9/10 APPROVE All issues fixed incl. gap lock index

Issues fixed across iterations

  1. Race condition → Transactional replace_addressable_event() with insert-first ordering
  2. Concurrent writer serializationSELECT ... FOR UPDATE with backing composite index for gap locking
  3. Discovery cleanup on deletesoft_delete_discovery_events() in handle_delete_group
  4. Silent error discarding → Explicit match + warn! on DB errors (auth, membership, system messages)
  5. No data migration → SQL migration for kind:40001→9
  6. Dead code → Removed add_to_allowlist()
  7. Code duplication → Extracted emit_addressable_discovery_event() helper
  8. VARBINARY(32)BINARY(32) for fixed-size keys
  9. Missing config test → Assertion for default
  10. E2E coverage → Test verifies all three discovery kinds (39000/39001/39002)
  11. Hidden numeric separator4_0001 in hooks.ts (JS numeric separator = 40001) broke optimistic reply rendering

Deferred to Phase 2

  • Live global discovery fan-out for open channels (historical queries work today)
  • Forum/canvas kind migration
  • NIP-50 search for standard clients
  • Admin REST endpoint for allowlist management
  • q-tag threading acceptance in desktop
  • E2E test for allowlist auth rejection (requires test infrastructure for per-test env vars)
  • Concurrent regression test for replace_addressable_event

Stats

35 files changed, +683 insertions, -159 deletions

- Collapse nested if in auth.rs allowlist check (clippy::collapsible_if)
- Fix biome import formatting in relayClient.ts
- Fix biome array formatting in e2eBridge.ts
- Replace separate soft_delete + insert with atomic replace_addressable_event()
  that wraps both in a single transaction, preventing duplicate addressable events
  under concurrent membership changes
- Extract emit_addressable_discovery_event() helper to eliminate 3x code duplication
  in emit_group_discovery_events()
- Add soft_delete_discovery_events() for bulk cleanup of 39000/39001/39002
- Clean up discovery events in handle_delete_group so deleted groups stop appearing
  in group discovery queries
- Remove dead add_to_allowlist() (zero callers) and soft_delete_previous_addressable()
  (replaced by transactional method)
…gration

BINARY(32) enforces exact 32-byte length at the DB level, preventing
malformed rows. VARBINARY(32) only sets a max, not a minimum.
One-time migration ensures historical messages stored as kind:40001
appear in kind:9 subscriptions. The desktop client's legacy 40001
entry in CHANNEL_EVENT_KINDS becomes unnecessary once this runs.
…stem message errors

- Remove redundant kind_i32 parameter from replace_addressable_event() —
  derive kind solely from the event to eliminate dual-derivation fragility
- Replace silent let _ = on system message insert with logged warning,
  consistent with error handling in the rest of the file
…ro-row window

Reverse the operation order: insert the new event first, then soft-delete
previous active events (excluding the just-inserted row by ID). This
guarantees at least one active addressable event at all times — the
previous delete-then-insert ordering could leave zero active rows if
INSERT IGNORE no-oped on a duplicate event ID.
Replace unwrap_or(false) with explicit match that logs the DB error
with connection and pubkey context. Fail-closed behavior preserved,
but infra failures are now distinguishable from 'pubkey not in allowlist'
in relay logs.
Expand test_nip29_standard_client_flow to query and assert that group
admins (39001) and group members (39002) discovery events are emitted
alongside the existing 39000 metadata check.
…FOR UPDATE

Add an exclusive row lock (SELECT ... FOR UPDATE) at the start of the
replace_addressable_event transaction. This serializes concurrent writers
for the same logical address (kind, pubkey, channel_id) — the second
transaction blocks until the first commits, preventing duplicate active
rows. Combined with insert-first ordering, this guarantees exactly one
active addressable event per group at all times.
…nt denial

Replace unwrap_or(false) on get_channel() with explicit match that logs
the DB error with connection and channel context. Fail-closed behavior
preserved, but infra failures are now distinguishable from 'channel is
not open' in relay logs.
Add idx_events_addressable on (kind, pubkey, channel_id, deleted_at) so
that SELECT ... FOR UPDATE in replace_addressable_event() can take a
next-key lock even on a cold address (no existing rows). Without this
index, InnoDB has no key range to lock, allowing two concurrent
first-time emissions to both insert — violating the single-active-row
invariant. Update the code comment to document the index dependency.
JavaScript's numeric separator (4_0001 = 40001) was invisible to our
rg '40001' search. Optimistic reply events had kind:40001 instead of
kind:9, causing isTimelineContentEvent to filter them out — replies
never appeared in the timeline. Use KIND_STREAM_MESSAGE constant.
@tlongwell-block
tlongwell-block merged commit 124047e into main Mar 14, 2026
8 checks passed
@tlongwell-block
tlongwell-block deleted the nip-native-migration branch March 14, 2026 14:51
tlongwell-block added a commit that referenced this pull request Mar 14, 2026
…-heartbeat

* origin/main:
  feat: NIP-29 native compatibility — standard nostr clients can chat on Sprout (#63)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant