Skip to content

fix(acp): re-subscribe channels the relay layer has dropped - #5014

Open
joe-rodgers wants to merge 1 commit into
block:mainfrom
joe-rodgers:fix/acp-subscribe-confirmation
Open

fix(acp): re-subscribe channels the relay layer has dropped#5014
joe-rodgers wants to merge 1 commit into
block:mainfrom
joe-rodgers:fix/acp-subscribe-confirmation

Conversation

@joe-rodgers

Copy link
Copy Markdown

Problem

subscribed_channel_ids in the harness main loop records the intent to enqueue a subscribe command, not a confirmed subscription: subscribe_channel / subscribe_channel_from only queue a REQ on the background task's command channel and return Ok.

The relay layer can then permanently drop a channel behind the harness's back — drop_channel_on_access_denied (member evicted / open→private flip) and the two "missing filter — dropping" paths in drain_rate_limited_pending / drain_resubscribe_retry. All three clear active_subscriptions / active_filters, so reconnect never restores the channel either.

Nothing told lib.rs. The if subscribed_channel_ids.contains(&ch) guard on the member-added path then made every subsequent membership notification for that channel a silent no-op, and the channel stayed dead until the process restarted — with no log line saying so.

Fix

The background task records each permanent drop into a DroppedChannels set shared with HarnessRelay. The main loop drains it via take_dropped_channels() before the "already subscribed" guard runs and clears those ids, so the next member-added notification re-subscribes instead of short-circuiting. A fresh Subscribe command clears the drop mark so a rebuilt subscription is not torn down again. Every drop now logs at warn.

Testing

Relay-side: drop is reported, connection-level denials are not, re-subscribe clears the mark, missing-filter paths report. Harness-side: a dropped channel takes the subscribe path on the next membership notification, unrelated channels untouched, repeatable.

cargo test -p buzz-acp --release: 675 passed, 0 failed. cargo fmt --check clean.

Context

Found on a long-running self-hosted fleet where individual agents would go quiet in one channel while still answering in others, with nothing in the logs to distinguish it from an idle agent.

`subscribed_channel_ids` in the harness main loop records the intent to
enqueue a subscribe command, not a confirmed subscription:
`subscribe_channel` / `subscribe_channel_from` only queue a REQ on the
background task's command channel and return Ok.

The relay layer can then permanently drop a channel behind the harness's
back — `drop_channel_on_access_denied` (member evicted / open→private
flip) and the two "missing filter — dropping" paths in
`drain_rate_limited_pending` / `drain_resubscribe_retry`. All three clear
`active_subscriptions` / `active_filters`, so reconnect never restores
the channel either.

Nothing told lib.rs. The `if subscribed_channel_ids.contains(&ch)` guard
on the member-added path then made every subsequent membership
notification for that channel a silent no-op, and the channel stayed
dead until the process restarted, with no log line saying so.

Fix: the background task records each permanent drop into a
`DroppedChannels` set shared with `HarnessRelay`. The main loop drains it
via `take_dropped_channels()` before the "already subscribed" guard runs
and clears those ids, so the next member-added notification re-subscribes
instead of short-circuiting. A fresh Subscribe command clears the drop
mark so a rebuilt subscription is not torn down again. Every drop now
logs at warn.

Tests: relay-side (drop is reported, connection-level denials are not,
re-subscribe clears the mark, missing-filter paths report) and
harness-side (a dropped channel takes the subscribe path on the next
membership notification; unrelated channels untouched; repeatable).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015KzSArAnn5xntFSNurJDH3
@joe-rodgers
joe-rodgers requested a review from a team as a code owner August 6, 2026 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant