Mailroom EMFILE resilience

[DanGould]

Symptoms

mailroom has been intermittently stopped serving requests and logging errno 24 (EMFILE, "Too many open files"). on one host the accept loop hung accept4() failed (24: Too many open files). on payjo.in the directory's mailbox reads started 500ing because the process has been hitting the file descriptor limit. This PR is a necessary but insufficient change raising that limit, so when there's a blowup it doesn't crash so quickly.

Bug

The actual bug seems to be that payjoin-mailroom accepts connections through tokio_listener behind axum::serve

1. tokio_listener's accept loop sorts errors into two buckets. Per-connection errors (a client reset mid-handshake — ECONNRESET/ECONNABORTED/ECONNREFUSED) → retry immediately, the next connection is probably fine. Everything else, including EMFILE/ENFILE, is returned as fatal — unless its sleep_on_errors option is on, in which case it sleeps 1s and retries (the pause gives fds time to free). The default is sleep_on_errors = false, so EMFILE escaped as fatal.
2. axum 0.8's listener adapter handles a fatal accept error by logging it and then calling std::future::pending() — a future that never resolves. That parks the accept loop forever. The process stays alive but won't serve any new requests because they can't open fds.. And because it's hung, not crashed, systemd Restart=on-failure never fires (there's no exit to detect). Only a human restart brings it back.

The one-line fix: set sleep_on_errors = true Now EMFILE → sleep 1s → retry accept(). Once connections drain and fds free, accept() succeeds and the server self-heals — no restart. It briefly degrades (new connections wait/refuse during the saturated second) instead
of dying permanently.

This does two things: raise the fd limit and survive hitting it. This does NOT yet address the underlying problem. But this is the fire extinguisher we use first so that we have some time to address the core issues.

Underlying Issue

axum::serve enforces no keep-alive/idle timeout and no connection cap, so idle inbound connections pile up until they exhaust the limit ~1k idle ESTABLISHED inbound sockets, accept queue overflowing. We ship this PR first but the real fix is long-poll-safe timeout + connection caps in a follow-up PR. Multiplexing with h2/h3 between relay<->director hops is how this scales beyond that.

Disclosure: co-authored by Claude; reviewed by Codex

Pull Request Checklist

Please confirm the following before requesting review:

• I have disclosed my use of
  AI
  in the body of this PR.
• I have read CONTRIBUTING.md and rebased my branch to produce hygienic commits.

[coveralls]

Coverage Report for CI Build 27063085831

Coverage decreased (-0.006%) to 85.37%

Details

• Coverage decreased (-0.006%) from the base build.
• Patch coverage: 3 uncovered changes across 1 file (0 of 3 lines covered, 0.0%).
• No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
payjoin-mailroom/src/lib.rs	3	0	0.0%

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	14689
Covered Lines:	12540
Line Coverage:	85.37%
Coverage Strength:	371.48 hits per line

---

💛 - Coveralls

[benalleng]

CACK 609c5b2

Is this really where some hardware limitations come into play setting this? How were these 16bit:19bit limits chosen?

[DanGould]

I asked the same questions drafting this

Is this really where some hardware limitations come into play setting this?

How were these 16bit:19bit limits chosen?

65536 is like 2G which seems reasonable given our actual hardware in the wild, and it's just the length of the leash not somewhere we want to operate at, which is why hardware limits aren't really the issue here. The issue here is a leak where fds accumulate and are never realased.
524288 is the default limit that I've chosen to explicitly keep, if you don't specify then the one number becomes both the hard and soft limit. We could still move the hard limit up if the hard limit is higher in case we come against this wall in the future, giving enough operating margin to fix a problem in prod if it were to come up again afaiu.

[benalleng]

We could still move the hard limit up if the hard limit is higher in case we come against this wall in the future, giving enough operating margin to fix a problem in prod if it were to come up again afaiu.

In my opinion I think the next lever we should pull is on the integration side with a bucket of directories but we can look into that on the payjoin-cli side.

ACK 609c5b2

[DanGould]

Bucket of directories doesn't really solve this if directories never give up idle connections. We gotta give up idle connections. more directories is basically just still more buckets till failure rather than emptying buckets when we can. Let's do both.