Hosted relay: write path 500 / DB pool timeout; accepted DM events not fanned out; WS subscription silently stops delivering

[gundamkid]

Describe the bug
Messages sent through the hosted relay (wss://gundamkid.communities.buzz.xyz) intermittently fail with relay error 500: internal server error / relay error 400: database error: pool timed out while waiting for an open connection. Separately, DM-channel events that are accepted (accepted:true) and queryable via messages get are NOT fanned out to the other member's clients when they carry only an h tag (no p tag) — stored but never pushed. Additionally, the WebSocket subscription silently stops delivering inbound events after ~2h of uptime while the connection still logs as established (no disconnect/reconnect is emitted), and even p-tagged DMs stop arriving; switching the client to polling (messages get) receives them fine.

Steps to reproduce

1. Connect a Nostr client (Hermes Agent gateway "buzz" platform, Linux buzz CLI from block/buzz main) to wss://gundamkid.communities.buzz.xyz and subscribe to a DM channel.
2. From another client (e.g. buzz messages send --channel <dm-channel-id> --content ...), send a message into that DM channel.
3. Observe: the event is returned as accepted:true and is readable via buzz messages get, but the subscriber's WebSocket never delivers it (no inbound event received; no error logged).
4. During the fault window, attempt a write from the subscribed client: the relay returns relay error 500: internal server error (then, on retry) relay error 400: database error: pool timed out while waiting for an open connection, and at times the write times out after 30s (exit 124).
5. (WebSocket case) Leave the client connected ~2h — inbound events stop arriving entirely, including p-tagged DMs, with no disconnect/reconnect logged. NIP-11 read path still responds.

Expected behavior

• Accepted events are fanned out to all matching subscriptions in real time, regardless of whether they carry a p tag.
• The write path does not return 500 / DB pool timeouts under normal community load.
• A WebSocket subscription that stops receiving events is either kept alive by the relay or closed cleanly so the client reconnects — not silently dead.

Version and platform
Find your version at the bottom of the Settings sidebar. Write "unknown" if you can't determine it.

• Buzz version: hosted platform (relay crate version reported by NIP-11: 0.2.0); client = Hermes Agent gateway buzz adapter using Linux buzz CLI built from block/buzz main
• OS: WSL2 (Linux) client; relay is Block-hosted

Logs / additional context
All timestamps UTC, 2026-08-04. Full gateway log excerpt:

16:14:21,009 ERROR hermes_plugins.buzz_platform.adapter: Buzz: failed to list channels
16:27:47,214 WARNING gateway.platforms.base: [Buzz] Send failed (attempt 1/2, retrying in 2.9s): relay_error: relay error 500: internal server error (exit 2)
16:27:59,403 WARNING gateway.platforms.base: [Buzz] Send failed (attempt 2/2, retrying in 4.1s): relay_error: relay error 400: error: database error: database error: pool timed out while waiting for an open connection (exit 2)
16:28:32,656 WARNING gateway.platforms.base: [Buzz] Send failed: timeout: buzz messages timed out after 30.0s (exit 124)
15:46:57,451 INFO gateway.run: inbound message: platform=buzz user=<pubkey> chat=<dm-channel> msg='[content omitted]'
15:48:32,983 INFO gateway.run: response ready: platform=buzz chat=<dm-channel> time=95.5s api_calls=7 response=1290 chars
15:48:35,184 INFO gateway.platforms.base: [Buzz] Sending response (1290 chars) to <dm-channel>
-- no inbound events received after this point until the gateway was restarted at 16:22 --
16:22:30,454 INFO hermes_plugins.buzz_platform.adapter: Buzz: connected to wss://gundamkid.communities.buzz.xyz as Thrall_Hermes, watching 4 channel(s) via poll, poll interval 4.0s

Observation notes:

• The 15:48 inbound arrived over the WebSocket transport; nothing after 15:48 was received by the WS transport even though p-tagged DMs were sent. Switching the adapter to transport: poll (CLI messages get every 4s) restored inbound delivery immediately.
• NIP-11 probe during the fault window returned HTTP 200 with full relay metadata while write requests were failing — consistent with DB pool exhaustion on the write path.

[ianzencom]

Additional observation – agent initialization timeouts across multiple runtimes

I’m seeing related symptoms on hosted relays that may be connected to the DB pool / fan-out issues described here.

What happens:

• Managed agents (Cursor, Claude, and Grok-based) start briefly, then die.
• They never stay responsive. Messages can still be sent from the desktop, but agents do not reply.
• This occurs on multiple machines and across different communities (not limited to one community).

From the agent harness logs (~/Library/Application Support/xyz.block.buzz.app/agents/logs/):

agent initialize failed: Request timeout - agent did not respond within 60s
...
Error: all 10 agents failed to start - cannot continue

The harness connects to the relay (wss://kikeriki.communities.buzz.xyz in my case), but the agent process never completes initialization within the 60s window. After repeated failures it gives up.

Pattern:

• Agents flap online → timeout → offline.
• Happens with completely different agent backends, so it does not appear to be a single runtime bug.
• Timing and behaviour are consistent with the relay being under pressure (pool timeouts, slow responses, or partial fan-out failures) rather than a pure client-side problem.

Happy to provide more log excerpts or test anything if useful.

[geniusblack9999]

Confirming this on a second hosted community (<community>.communities.buzz.xyz), with a controlled test that narrows the WS symptom: subscriptions can be born dead, not only die after ~2h of uptime, and the failure is per-subscription on one relay at one moment rather than per-connection or per-client.

Environment

• Client: Buzz Desktop v0.5.3 (macOS 26.5.2, Apple Silicon), 6 managed agents via buzz-acp
• Runtimes: claude-agent-acp v0.64.0, codex-acp v1.1.7, goose
• Relay: hosted, different community from the original report; NIP-11 version: 0.2.0

Test: 6 agents, same relay, same channel, same instant — 2 receive, 4 get nothing

All 6 agents are members of one channel and each logged an explicit subscribed to channel <id> for it. I then sent 6 owner-signed mentions (one per agent) into that channel within ~5 seconds.

Every mention was accepted:true and every one is readable via messages get. Only 2 of the 6 agents' WebSockets ever delivered them:

subscribe times (all to the same channel id)
  16:44:18.925  Opus     -> deaf
  16:44:20.418  Gemini   -> deaf
  16:44:24.607  Luna     -> deaf
  16:47:06.634  Terra    -> deaf
  16:47:xx       Sol     -> received
  16:48:xx       Sonnet  -> received

16:50:01  6 mentions sent, all accepted:true, all present in `messages get`
16:50:21  Sonnet replied
16:50:50  Sol replied
          Opus / Gemini / Luna / Terra: no inbound event, no error, no reconnect logged

Terra subscribed 3 minutes before the mention and received nothing, so this is not a slow leak — the subscription never worked from the start. Note also that Opus/Gemini/Luna subscribed during a window when the relay was returning error: database error on a subscription close (see below), while Sol and Sonnet subscribed a few minutes later and work.

Local causes ruled out

For all 6 processes simultaneously:

• process alive; lsof shows exactly one ESTABLISHED socket to the same Cloudflare edge (104.18.17.37:443)
• each logged connected to relay, subscribed to membership notifications, owner resolved from BUZZ_AUTH_TAG, discovered N channel(s), and subscribed to channel <test-channel>
• no disconnect, no reconnect, no error, nothing at all logged after presence set to online

The 4 deaf agents are indistinguishable from the 2 working ones on the client side. Confirming your polling observation: messages get returns the events fine while the WS delivers nothing.

Corroborating the other two failure modes

Same relay, same day, from agent logs:

15:13:42  WARN relay NOTICE: rate-limited: too many concurrent requests
15:13:42  WARN rate-limit gate armed via NOTICE until ~4.4s from now
16:00:26  WARN autonomous reconnect attempt 1 failed: WebSocket error: HTTP 503 Service Unavailable
16:00:32  WARN subscription ch-<id> closed by relay: error: database error
16:44:33  WARN initial relay connect attempt 1 failed: Auth failed: error: internal error checking restriction state
16:44:53  WARN initial relay connect attempt 3 failed: Auth failed: error: internal error checking restriction state
16:46:17  WARN initial relay connect failed with terminal error: WebSocket error: HTTP 404 Not Found

So alongside the write-path 500 / DB pool timeout, the same relay also intermittently fails the WS handshake (404, 503) and the auth restriction check (internal error checking restriction state) — the latter looks like the same DB exhaustion surfacing on the auth path. database error on a subscription close is presumably the same root cause hitting the fanout path, which would explain born-dead subscriptions directly.

Yesterday the same relay was intermittently 500ing POST /query channel discovery, which kills buzz-acp outright since 500 is classified non-retryable while 503 is retried — filed as #4561. Possibly also related: #4667 (agent acks a mention but never responds after reconnect + rate-limit).

One observation that may hinder triage

This relay's NIP-11 version stayed 0.2.0 across a build change:

• 2026-08-03: "supported_extensions":["nip-er","nip-pl"]
• 2026-08-04: "supported_extensions":["nip-er"]

Same version string, different advertised extensions. If hosted relays ship without a version bump, reports can't be correlated to builds — worth a distinct build/commit identifier in NIP-11.

Impact

For an agent fleet this is silent and total: buzz-acp has no way to detect a subscription that was accepted but never delivers, so agents sit indefinitely showing presence: online while ignoring every mention. Restarting is a coin flip on whether the new subscription happens to work. Agreeing strongly with the expected behavior as written — a subscription that stops delivering needs to be closed cleanly so the client reconnects, since clients cannot currently distinguish "no traffic" from "dead subscription".

[jack-owen]

Thanks for writing this up — happy to add a third data point, since I saw the same thing today on a different community.

Different hosted community, same day (2026-08-04 UTC), from my agent harness log:

16:44:22  WARN relay connection lost — reconnecting…
16:44:29  WARN autonomous reconnect attempt 1 failed: Connection closed
16:44:39  WARN autonomous reconnect attempt 2 failed: Connection closed
16:44:51  WARN autonomous reconnect attempt 3 failed: Connection closed
16:45:01  WARN autonomous reconnect attempt 4 failed: Auth failed: error: internal error checking restriction state
16:45:15  WARN autonomous reconnect attempt 5 failed: Connection closed
16:45:24  WARN relay reconnect failed: Connection closed
16:45:32  WARN relay reconnect failed: Connection closed
16:45:50  WARN relay reconnect failed: WebSocket error: HTTP error: 404 Not Found
16:46:00  WARN relay reconnect failed: WebSocket error: HTTP error: 404 Not Found
16:46:23  INFO relay reconnected
16:46:45  WARN thread context fetch timed out — will retry
17:30:36  WARN relay NOTICE: rate-limited: too many concurrent requests

The internal error checking restriction state auth failure and the HTTP 404 on reconnect land in the same ~16:44–16:46 window as the other reports above, on yet another community/relay host — seems more like a shared infra incident than something scoped to one community.

[vangsteven2]

🚀✨ 性能瓶颈一般在 IO 还是 CPU

[mckenziecraig554]

📌🧠🛠️ 模块边界划分得很克制，喜欢

[hollanddonald2]

🛠️🔥📦 想问下生产环境怎么做降级兜底

[geniusblack9999]

Follow-up to my earlier comment: I A/B'd the same 6-agent fleet across two completely different networks to rule out client-side and network causes. The result argues the fault is per-subscription, not per-client, per-connection, or per-network.

Method

Same machine, same 6 buzz-acp agents, same relay, same channel, same test (one owner-signed mention per agent, then wait). Run A on a corporate wifi, Run B tethered to a phone hotspot (cellular, no corporate middleboxes at all). Agents were restarted between runs, so each run established fresh subscriptions.

Result — the affected set reshuffles randomly

Agent	Run A: corporate wifi	Run B: cellular hotspot
Sonnet	replied	silent
Sol	replied	replied
Gemini	silent	replied
Opus	silent	replied
Luna	silent	silent
Terra	silent	silent
	2 / 6	3 / 6

Gemini and Opus were deaf on wifi and work on cellular. Sonnet worked on wifi and went deaf on cellular. If this were a client bug or network interference, the same agents would fail in both runs, and cellular would not break an agent that had been working. A different random subset each time is what per-subscription server-side failure looks like.

Both runs are broken; the one-agent difference is noise at n=6, not a network gradient.

Write path also 500s on cellular

From the hotspot, sending a single message required retries:

Opus   try1  relay error 500: internal server error
Terra  try1  relay error 500: internal server error
Terra  try2  relay error 500: internal server error
Terra  try3  sent

So the write-path 500 in the original report reproduces on a network with no corporate security in the path.

Client-side and network causes ruled out, in both runs

• Genuine leaf certificate (CN=communities.buzz.xyz, issuer Google Trust Services) on both networks — no TLS interception. No proxy configured (scutil --proxy empty), no inspection CA (Zscaler/Netskope/Palo/Fortinet) in the system keychain.
• All 6 processes alive, each with exactly one ESTABLISHED socket to the Cloudflare edge, in both runs.
• All 6 logged subscribed to channel <test-channel> in both runs, seconds before the mentions were sent.
• Deaf agents logged nothing after presence set to online — no disconnect, no reconnect, no error.
• All mentions returned accepted:true and are readable via messages get in both runs, including the ones no agent ever received.

Worth noting the deaf agents sit on healthy sockets with no reconnect churn at all during the fault window — so this is not the client failing to notice a dropped TCP connection. The subscription is registered and simply never receives fanout.

Suggestion

Since a subscription can be born dead and clients cannot distinguish "quiet channel" from "dead subscription", some server-side liveness signal would make this detectable — either a periodic ping on subscriptions that have delivered nothing, or closing the subscription cleanly on the fanout error (the subscription ch-<id> closed by relay: error: database error path already does this, so the fanout failure seems to sometimes close cleanly and sometimes leave the subscription silently registered).

[gibsoncaroline2876]

希望能出一版架构简图 🔥📦