P0: Fix homelab message sending and prove deployed E2E with Tawny and Kelpie

[jusso-dev]

Parent: #21

Goal

Fix the current authenticated homelab message-send failure, then prove Muster's critical product and real Tawny/Kelpie flows against the deployed Intel Ubuntu stack. Mock or local-only success does not satisfy this issue.

Current blocker (reported 2026-07-26)

Sending a message from the homelab Muster UI currently throws an error. First action: reproduce with the deployed image, capture the sanitised browser console/network response, server trace ID, and service logs, then add a failing regression test before fixing it.

Scope

Deployed product E2E

• Run Playwright against the actual homelab URL and exact deployed image digest, not a local dev server.
• Authenticate using private test credentials supplied only through the test environment.
• Cover login/logout, room navigation, send by button and Enter, Shift+Enter newline, optimistic state, reload persistence, thread reply, reaction, task creation, delegation, search, and mobile viewport.
• Use two authenticated browser contexts for real-time delivery, unread state, thread updates, and organisation isolation.
• Exercise duplicate submits, retry, refresh during send, Redis interruption, worker/web restart, and recovery without duplicate durable records.
• Capture sanitised trace, screenshots, console, request/response metadata, server trace IDs, and container logs on failure.

Real Tawny and Kelpie

• Discover deployed/upstream HTTP contracts.
• Generate least-privilege API credentials using supported mechanisms.
• Store credentials only in private homelab secrets; never commit, print, screenshot, or log them.
• Configure connector health, webhooks/polling, retry, and delivery logs.
• Exercise Tawny inventory, hunt, and approval-gated safe response.
• Exercise Kelpie search, create/update, timeline, observable, and evidence references.
• Keep Muster on 3004, Kelpie on 3000, and docs on 3010.

Guardrails

• Use synthetic organisation, users, rooms, messages, indicators, cases, and response targets.
• Scope every assertion by organisation and verify cross-tenant denial.
• Dangerous actions require server-side capability checks and an approval record; default E2E actions remain non-destructive.
• Never expose real credentials or sensitive homelab data in GitHub artifacts.
• PostgreSQL is authoritative; Redis/BullMQ outages may degrade execution but must not invent durable state.

Acceptance criteria

• The reported message-send error is reproduced, diagnosed, fixed, and protected by a deployed regression test.
• All Muster services and real connectors report truthful health.
• Critical chat/task flows pass against the deployed homelab in desktop and mobile viewports.
• Two-browser real-time tests pass; reload and restart preserve durable state.
• Failure/retry tests produce no duplicate message, task, action, or case.
• Real Tawny hunt result reaches a Muster room/task history.
• Approved safe response reaches Tawny once; result records once.
• Workflow creates/updates a real synthetic Kelpie case and timeline.
• Cross-tenant, unauthorised, expired-session, and dangerous-action paths fail closed.
• Secrets are absent from Git, logs, screenshots, traces, and test artifacts.
• Evidence records image digest, request IDs, timestamps, trace IDs, delivery records, and exact test command.

Verification

• Extensive Playwright suite pointed at deployed homelab.
• Connector integration tests against deployed services.
• Restart/recovery, idempotency, and Redis degradation tests.
• API key rotation/revocation test.
• Homelab runbook execution from a clean synthetic state.

[jusso-dev]

Stop-ship deployed slice completed and pushed to main in a55f7d39714e0e11a4d633e833b4dd784bc36928.

Evidence (synthetic/private credentials only):

• Reproduced against prior deployed image sha256:192a28ca9c5ddf3731253504d58a5e1d2ad21cb3ad8ae674813c65462a957e3d: authenticated button send entered the error state, preserved the draft, emitted no POST, no console error, and therefore no server trace ID.
• Root cause: crypto.randomUUID() is unavailable on the insecure LAN HTTP origin and threw before fetch.
• Regression: tests/homelab/message-send.spec.ts uses the deployed URL only, private env credentials, retained failure trace/screenshots, and sanitised diagnostics. A unit regression covers the getRandomValues UUID-v4 fallback.
• Full pnpm run check passed before push.
• Deployed exact revision as image sha256:62ad9b06418a34e49cd8d85912af05e21d24779217a014f893ed8fa2d9dc5908; web, worker, and agent gateway are healthy.
• Exact deployed command passed: MUSTER_BASE_URL=http://192.168.1.19:3004 pnpm exec playwright test --config=playwright.homelab.config.ts --project=chromium tests/homelab/message-send.spec.ts (credentials supplied privately via environment).
• Successful durable write audit trace: 3a0c66ae-e1b6-45f5-8ccd-e11926f50380 at 2026-07-26 14:41:20 UTC.
• PostgreSQL proof: messages 5 -> 6, audit events 0 -> 1, outbox events 0 -> 1; the original five-message digest remained c851fb25fd806a8f346c37518213bed0, proving preservation, with exactly one new durable record.
• Private rollback: PostgreSQL dump SHA-256 58f2a6280497f99e0a796b39a2333d20953c8331945689206d6fbd2cfc4a4478; prior app image retained as rollback-issue10-192a28c.
• Port-3010 handbook updated with the verified revision/image, persistence, recovery, and truthful note that external connectors remain mocked.

Per parent #21, this issue stays open: broader chat/realtime/resilience acceptance follows #22/#23, and real Tawny/Kelpie acceptance returns after #13.

[jusso-dev]

Real Tawny/Kelpie slice completed and deployed.

Commits:

• Muster runtime: f480e3a39ecacf63636a78788ccb177006921c43
• Anonymous-auth test isolation: eb3100eec6d50f223f5b58cd4c97dbb2719ac181
• Clean-install CI selector repair: ea71c6c
• Tawny supported API-token response access: 85d0709 on Tawny main

Evidence (all targets/data synthetic; credentials remained private):

• pnpm check passed all 22 packages. Integration tests passed 9/9; DB integration tests passed 6/6.
• Deployed real-system Playwright passed 1/1 with trace/screenshots disabled: Tawny inventory and hunt, room/task delivery, approval-gated isolation of an inert synthetic endpoint, Kelpie approved case creation, comment, observable, contained-status update, case readback, room evidence, internal duplicate replay, and unauthenticated denial.
• Deployed image: sha256:045f4d8a2d6ff7646ca32339e66d403cc430830d57310bafc42b4b163db6783c; web, worker, and agent gateway are healthy with zero restarts.
• Tawny proof: 3 test executions produced 3 unique idempotency markers, maximum 1 response action per marker.
• Kelpie proof: 3 test executions produced 3 unique idempotency markers, maximum 1 case per marker; each verified timeline and observable persistence.
• PostgreSQL after proof: 25 rooms, 67 memberships, 65 messages, 9 evidence records, 187 audit events, 161 outbox events, 20 connectors, 17 credentials, 26 query templates, 17 query runs, 15 deliveries, and 7 approvals. Audit chain: 0 link mismatches, 0 sequence gaps.
• A pre-update database backup and rollback image were retained privately. Port-3010 handbook was updated. No connector secret appeared in Git, logs, screenshots, traces, API projections, or this comment.

Issue remains open for the remaining chat/two-context/restart/Redis/API-key-rotation acceptance.