Summary
A devsy SSH session to a dev container drops completely during short network
stalls that a plain ssh session over the same path rides through without
noticing. In a measured episode the network pause lasted 39 s, but devsy's
internal SSH keep-alive closed the transport after only ~8 s (2 missed
intervals). Once the transport is closed, the next periodic workspace refresh
fails to open a channel and the whole tunnel (container + host) is torn down,
killing the user's SSH session.
The trigger is the network; the bug is that devsy's keep-alive tolerance is an
order of magnitude lower than stock OpenSSH, so transient stalls that normal ssh
absorbs become hard disconnects.
There is also a secondary bug: on reconnect the forwarded port is not released
(bind: address already in use).
Environment
|
|
| devsy |
v1.5.2 (devpod base v0.26.1) |
| Client |
Ubuntu 24.04.4, OpenSSH_9.6p1 |
| Host (ssh provider) |
Ubuntu 24.04.3, Docker 29.6.2 |
| Provider |
ssh, USE_BUILTIN_SSH=false, INACTIVITY_TIMEOUT=2h |
| devcontainer |
mcr.microsoft.com/devcontainers/base:ubuntu-24.04 |
| Connect |
ssh <ws>.devsy → ProxyCommand devsy workspace ssh --stdio … |
Symptom
While working, the SSH session to the container closes unexpectedly
(client_loop: send disconnect: Broken pipe). It happens irregularly. Tunnel
lifetime before a teardown varies wildly — from 1m39s to 1h9m33s — i.e. it
is event-driven, not a fixed timer. The flakier the network, the more often
it happens.
Root cause
The decisive line comes from devsy's internal SSH client
(golang.org/x/crypto/ssh), and it appears on every drop (5 of 5 drops in a
~14 h run):
2026/07/26 11:48:13 ssh: connection keep-alive timeout after 2 intervals; closing transport
Full causal chain of one drop (devsy zap lines are local +0200; the Go
keep-alive line is UTC — same episode):
# devsy runs a periodic "refresh" every ~30s, opening a channel into the container:
13:48:06.302 DEBUG Start refresh
13:48:06.306 DEBUG Run command in container: ".../agent internal agent workspace update-config …"
13:48:06.896 DEBUG Out:
# a network stall begins (~39s). The internal keep-alive gives up after 2 intervals:
11:48:13 ssh: connection keep-alive timeout after 2 intervals; closing transport # UTC = 13:48:13 local
# the next refresh tries to open a channel on the now-dead transport:
13:48:36.899 DEBUG Start refresh
13:48:36.904 DEBUG Run command in container: ".../agent internal agent workspace update-config …"
# channel open fails → the whole tunnel is torn down:
13:48:43.838 DEBUG connection to container closed
13:48:43.839 DEBUG Container tunnel exited
13:48:43.839 ERROR Error updating remote workspace: failed to create SSH session: ssh: unexpected packet in response to channel open: <nil>
13:48:43.845 DEBUG Tunnel to host closed
# the outer OpenSSH client sees the ProxyCommand stdout close:
client_loop: send disconnect: Broken pipe
Why this is a devsy issue, not just a bad network
A control session to the same container was kept open in parallel over a
different path (ssh <host> → docker exec, plain OpenSSH, separate TCP
connection) with a 5 s heartbeat. During the same stall the control session
also paused — for exactly 39 s — but it did not disconnect: TCP
retransmitted and the buffered heartbeats arrived in one burst when the network
recovered.
11:48:00.037Z BEAT
11:48:05.082Z BEAT # last beat before the stall
… 39 s of silence …
11:48:44.149Z BEAT ┐
11:48:44.196Z BEAT │ 7 heartbeats flushed at once —
11:48:44.199Z BEAT │ the connection was paused but survived
11:48:44.202Z BEAT │
11:48:44.205Z BEAT │
11:48:44.208Z BEAT │
11:48:44.210Z BEAT ┘
11:48:45.050Z BEAT # back to normal, no reconnect
The difference is keep-alive tolerance:
- OpenSSH (control):
ServerAliveInterval 30, ServerAliveCountMax 4 → tolerates ~120 s.
- devsy (internal Go SSH): closes the transport after 2 intervals (~8 s here).
Same stall, same path: control survives, devsy tears down.
Note: this is not the INACTIVITY_TIMEOUT (2h) — the drops happen under
constant traffic, and an idle parallel session dropped on the same stalls too.
Secondary bug: forwarded port leaks on reconnect
After a drop devsy reconnects, but the previous forward is never released:
13:48:51.170 ERROR error port forwarding localhost:7777 to 7777: listen tcp 127.0.0.1:7777: bind: address already in use
This occurred 11 times in the run (once per reconnect) — forwarded ports stay
broken after a drop until the whole session is restarted.
Expected behavior
- A short network pause (seconds to a few tens of seconds) should not kill the
user's SSH session — as it doesn't with plain ssh.
- Losing the internal transport should trigger a transparent tunnel reconnect,
not a teardown of the whole session.
- On reconnect, previous port-forward listeners should be released before
re-binding.
Suggested fixes
- Make the internal keep-alive tolerance configurable and/or more lenient.
The current "2 intervals" is much stricter than OpenSSH defaults and turns
brief stalls (that plain ssh absorbs) into teardowns. Grep target:
keep-alive timeout after … intervals; closing transport.
- Survive transport loss by reconnecting the tunnel instead of tearing down
the user-facing session.
- Release the port-forward listener before reconnecting (fixes the
bind: address already in use on 7777).
Reproduction
Occurs on its own with a flaky network. It should reproduce deterministically by
stalling the transport for longer than 2 keep-alive intervals (e.g. tc netem
delay/loss of ~40 s, or pausing the ProxyCommand process), while a parallel plain
ssh session over the same path survives.
Full logs
Sanitized logs (host/user/IP redacted, gzip+base64 workspace payloads omitted)
attached as a gist: https://gist.github.com/tihonove/3377865bc8fde6bcff2e30e5bfb6149e
01-drop-sequence.log — full devsy debug around one drop (keep-alive timeout → teardown → reconnect)02-control-survived.log — control heartbeat showing the 39 s stall it survived03-summary.txt — all keep-alive timeouts, tunnel lifetimes, port-forward leak count04-environment.txt — versions and provider config
Happy to capture more episodes or run a deterministic repro on request.
Summary
A devsy SSH session to a dev container drops completely during short network
stalls that a plain
sshsession over the same path rides through withoutnoticing. In a measured episode the network pause lasted 39 s, but devsy's
internal SSH keep-alive closed the transport after only ~8 s (2 missed
intervals). Once the transport is closed, the next periodic workspace refresh
fails to open a channel and the whole tunnel (container + host) is torn down,
killing the user's SSH session.
The trigger is the network; the bug is that devsy's keep-alive tolerance is an
order of magnitude lower than stock OpenSSH, so transient stalls that normal ssh
absorbs become hard disconnects.
There is also a secondary bug: on reconnect the forwarded port is not released
(
bind: address already in use).Environment
ssh,USE_BUILTIN_SSH=false,INACTIVITY_TIMEOUT=2hmcr.microsoft.com/devcontainers/base:ubuntu-24.04ssh <ws>.devsy→ ProxyCommanddevsy workspace ssh --stdio …Symptom
While working, the SSH session to the container closes unexpectedly
(
client_loop: send disconnect: Broken pipe). It happens irregularly. Tunnellifetime before a teardown varies wildly — from
1m39sto1h9m33s— i.e. itis event-driven, not a fixed timer. The flakier the network, the more often
it happens.
Root cause
The decisive line comes from devsy's internal SSH client
(
golang.org/x/crypto/ssh), and it appears on every drop (5 of 5 drops in a~14 h run):
Full causal chain of one drop (devsy zap lines are local
+0200; the Gokeep-alive line is UTC — same episode):
Why this is a devsy issue, not just a bad network
A control session to the same container was kept open in parallel over a
different path (
ssh <host>→docker exec, plain OpenSSH, separate TCPconnection) with a 5 s heartbeat. During the same stall the control session
also paused — for exactly 39 s — but it did not disconnect: TCP
retransmitted and the buffered heartbeats arrived in one burst when the network
recovered.
The difference is keep-alive tolerance:
ServerAliveInterval 30,ServerAliveCountMax 4→ tolerates ~120 s.Same stall, same path: control survives, devsy tears down.
Note: this is not the
INACTIVITY_TIMEOUT(2h) — the drops happen underconstant traffic, and an idle parallel session dropped on the same stalls too.
Secondary bug: forwarded port leaks on reconnect
After a drop devsy reconnects, but the previous forward is never released:
This occurred 11 times in the run (once per reconnect) — forwarded ports stay
broken after a drop until the whole session is restarted.
Expected behavior
user's SSH session — as it doesn't with plain ssh.
not a teardown of the whole session.
re-binding.
Suggested fixes
The current "2 intervals" is much stricter than OpenSSH defaults and turns
brief stalls (that plain ssh absorbs) into teardowns. Grep target:
keep-alive timeout after … intervals; closing transport.the user-facing session.
bind: address already in useon 7777).Reproduction
Occurs on its own with a flaky network. It should reproduce deterministically by
stalling the transport for longer than 2 keep-alive intervals (e.g.
tc netemdelay/loss of ~40 s, or pausing the ProxyCommand process), while a parallel plain
ssh session over the same path survives.
Full logs
Sanitized logs (host/user/IP redacted, gzip+base64 workspace payloads omitted)
attached as a gist: https://gist.github.com/tihonove/3377865bc8fde6bcff2e30e5bfb6149e
01-drop-sequence.log— full devsy debug around one drop (keep-alive timeout → teardown → reconnect)02-control-survived.log— control heartbeat showing the 39 s stall it survived03-summary.txt— all keep-alive timeouts, tunnel lifetimes, port-forward leak count04-environment.txt— versions and provider configHappy to capture more episodes or run a deterministic repro on request.