fix: peers struggle to grow past ~10 connections during bootstrap

[iduartgomez]

Problem

Peers have difficulty climbing past ~10 connections during the bootstrap phase. The compounded failure probability across multiple independent steps (routing, acceptance, NAT traversal) makes each connection attempt unlikely to succeed, and the current parameters don't compensate for this.

Root Cause Analysis

Connection growth requires bilateral cooperation — both the initiator and the terminus peer must succeed through several independent steps:

1. Initiator picks a target location and sends a CONNECT
2. CONNECT must route successfully (max 10 hops, greedy forwarding)
3. Terminus peer must accept (probabilistic Kleinberg filter)
4. NAT traversal must succeed

Each step has independent failure probability. The compounded success rate is quite low.

Key Throttling Mechanisms

Mechanism	Current Value	Effect
KLEINBERG_FILTER_MIN_CONNECTIONS	3	Above 3 connections, acceptance becomes probabilistic (50-100%) — premature for bootstrap
MAX_CONCURRENT_CONNECTIONS	3	Only 3 outbound attempts at a time, regardless of bootstrap phase
Accept-at-terminus rule	Always on	Each CONNECT gets exactly one acceptance chance at the terminus — if rejected, the entire attempt fails
Location-based backoff	On failure	Failed CONNECTs trigger exponential backoff per Location, blocking retries to that ring region

The Kleinberg gap score is not meaningful with <12 connections (too few data points for a useful distribution), yet the probabilistic filter kicks in at just 3 connections.

Proposed Solutions

1. Raise KLEINBERG_FILTER_MIN_CONNECTIONS to min_connections / 2 (~12)

Below ~12 connections, accept everything — gap-based filtering during early bootstrap is premature optimization. The gap score isn't meaningful with so few connections anyway.

Impact: Simple parameter change. Peers below 12 connections would accept all inbound CONNECTs, dramatically increasing bilateral success rate during bootstrap.

2. Increase MAX_CONCURRENT_CONNECTIONS during bootstrap

Currently hardcoded to 3 regardless of phase. Below min_connections, allowing 5–8 concurrent outbound attempts would increase the rate of successful connections since each attempt is independent.

Impact: Simple parameter change. More attempts per tick = faster growth through the early phase.

3. (More invasive) Allow non-terminus acceptance during bootstrap

If a relay peer is below min_connections / 2, it could accept the joiner even if it can forward closer. This breaks the strict accept-at-terminus rule but only during early growth, meaning each CONNECT has multiple acceptance chances rather than a single shot at the terminus.

Impact: Protocol-level change, most impactful but needs careful evaluation of topology quality tradeoffs.

Relevant Code

• crates/core/src/ring/connection_manager.rs:469-648 — should_accept() with Kleinberg filter
• crates/core/src/ring/connection_manager.rs:580 — KLEINBERG_FILTER_MIN_CONNECTIONS = 3
• crates/core/src/ring.rs:1790 — MAX_CONCURRENT_CONNECTIONS = 3
• crates/core/src/ring.rs:1766 — connection_maintenance() loop
• crates/core/src/topology.rs:339 — adjust_topology()
• crates/core/src/operations/connect.rs:452 — accept-at-terminus logic

[github-actions]

🤖 Auto-labeled

Applied labels:

• T-bug (90% confidence)
• P-high (85% confidence)
• E-medium (80% confidence)
• A-networking (95% confidence)
• S-needs-design (80% confidence)

Reasoning: The report describes a functional problem where peers cannot reliably grow past ~10 connections during bootstrap due to overly aggressive throttling and probabilistic filters; this is symptomatic of a bug in bootstrap behavior (T-bug). It impacts peer connectivity and likely affects many nodes, so P-high is appropriate. The proposed fixes range from simple parameter tweaks to protocol changes; evaluating and implementing safely requires moderate effort and design work (E-medium) and falls squarely within networking/topology code (A-networking). Because several proposed mitigations (including protocol-level changes) require architectural discussion and trade-off analysis, marking S-needs-design is warranted.

Previous labels: none

If these labels are incorrect, please update them. This helps improve the auto-labeling system.

[sanity]

@claude Can you confirm this:

Accept-at-terminus rule Always on Each CONNECT gets exactly one acceptance chance at the terminus — if rejected, the entire attempt fails

Any peer that forwards a connect request should be able to accept it - potentially multiple acceptances per connect request, is this not the case?

[github-actions]

Claude finished @sanity's task in 1m 6s —— View job

---

Analysis of Accept-at-Terminus Rule

• Read relevant connect.rs and connection_manager.rs code
• Trace the CONNECT forwarding and acceptance logic
• Answer the question about accept-at-terminus behavior

---

The claim in the issue is accurate. The current code explicitly implements an "accept only at terminus" rule. Here's the relevant logic from crates/core/src/operations/connect.rs:507-531:

// ACCEPT ONLY AT TERMINUS: Relays only accept when they can't forward to a closer peer.
// ...
// Algorithm:
// 1. First, check if we can forward to a closer peer
// 2. If we can forward: forward only (don't accept)
// 3. If we can't forward (terminus): accept if should_accept() allows

let can_forward = self.forwarded_to.is_none() && self.request.ttl > 0;
let next_hop = if can_forward {
    ctx.select_next_hop(...)
} else {
    None
};
let is_terminus = next_hop.is_none();

Forwarding and accepting are mutually exclusive — the acceptance check at line 569 requires is_terminus && self.forwarded_to.is_none(). Any peer that can route closer simply forwards and never accepts.

There is one partial exception: if the terminus peer rejects (e.g., should_accept() returns false due to capacity), the code at lines 595–618 routes the request uphill (away from the target) to give other peers a chance. This means a rejected terminus can result in a secondary acceptance opportunity — but still only at that uphill peer's terminus, not at the original forwarding hops.

So to directly answer your question: No, forwarding peers cannot accept under the current implementation. Each CONNECT has exactly one acceptance opportunity at the closest reachable peer, with a fallback uphill route if that peer rejects. This is intentional — the comment explains it prevents early relays (gateway, first hops) from forming non-local ring connections.

Whether this should be changed (to allow any forwarding peer below min_connections / 2 to accept, as proposed in Solution 3 of the issue) is the design question. The current code enforces strict terminus-only acceptance.

[iduartgomez]

Deep Dive: Why Growth Stalls in Practice

Traced through the actual code paths and a live peer's connection table (9 peers after 1.5 hours) to understand the mechanics.

Two Acquisition Loops, Both Under-Delivering

1. initial_join_procedure (gateway-directed) — goes dormant too early

Once both gateways show as "connected/pending", this loop enters a no-op branch (unconnected_count == 0, line 2424 of connect.rs) and stops initiating any new connections. It only sends CONNECTs to gateways, not through them to new targets. So for a peer with 2 gateways connected but only 9 ring peers, this entire loop is idle.

2. connection_maintenance (ring-directed) — low concurrency

• Ticks every 5s during bootstrap (good)
• But caps at MAX_CONCURRENT_CONNECTIONS = 3 in-flight CONNECTs at a time
• Each CONNECT must: route successfully (≤10 hops) → terminus accepts → NAT traversal succeeds
• Observed NAT success rate: 1/20 recent (5%)
• At 3 concurrent attempts with 5% NAT success: ~0.15 successful connections per 5-second tick

Compounding Failure Probability

Each connection attempt has independent failure at multiple steps:

Step	Estimated Success Rate
Routing finds a candidate	~90% (can fail with few connections)
CONNECT reaches terminus (≤10 hops)	~80%
Terminus accepts (Kleinberg filter, 50%+ floor)	~65%
NAT traversal succeeds	~5-12% (observed)
Compounded	~2-4%

With 3 concurrent attempts every 5s, that's ~36 attempts/minute × 3% success ≈ ~1 new connection per minute optimistically. In practice, backoff on failed locations further reduces this.

Key Findings

1. initial_join_procedure is the biggest gap — it goes completely idle once gateways are connected, leaving only connection_maintenance as the sole driver. It should be sending CONNECTs through gateways to new targets, not just to gateways.

2. MAX_CONCURRENT_CONNECTIONS = 3 is too low for bootstrap — with a 95%+ per-attempt failure rate (dominated by NAT), you need many more in-flight attempts to achieve reasonable growth. Consider 8-10 during bootstrap.

3. NAT failure rate may be the dominant bottleneck — even with perfect acceptance logic, 5% NAT success means ~20 attempts per successful connection. This is potentially orthogonal to the topology pickiness question but compounds with it.

4. PR fix: use Kleinberg-optimal gap_target for bootstrap connections #3503's gap_target fix helps distribution but not growth rate — routing 93% of requests through gap_target improves where connections form, but doesn't increase the rate at which they form.

Recommended Priorities

1. Keep initial_join_procedure active below min_connections even when gateways are connected — route through them
2. Raise MAX_CONCURRENT_CONNECTIONS during bootstrap (e.g., 8)
3. Investigate NAT failure rate separately — if hole punching is 5% success, that's the ceiling on everything else
4. Raise KLEINBERG_FILTER_MIN_CONNECTIONS from 3 to ~12 (per original issue description)

[iduartgomez]

Telemetry Analysis: Peer 37TR5xUwAbEKxKHrb (location 0.247, IP 79.247.216.34)

Correlated a live peer's connection table with the central telemetry server to trace exactly what's happening during the bootstrap stall.

The Smoking Gun: 7 Outbound vs 715 Inbound

Metric	Count
connect_request_sent (outbound)	7
connect_request_received (inbound relay)	715
connect_connected (unique)	782
Unique peer addresses connected	169

The peer sent only 7 outbound CONNECT requests in 2 hours, but relayed 715 inbound requests for other peers. It's doing tons of work for the network but barely acquiring connections for itself.

All 7 Outbound Requests Are Gateway-Directed (is_initial: True)

7.5m ago: target=xaS6mFwJ8DzE4sGH@82.11.195.238 (peer, not gateway!)  desired_loc=0.253
6.5m ago: target=4626yNNSwrp2p6ZiW@81.108.124.131 (peer)               desired_loc=0.250
1.8m ago: target=4ftT57SaDhXxmPsCU@100.27.151.80:31337 (gateway)        desired_loc=0.484
1.8m ago: target=i4QWRJcwgtBxtQf6@5.9.111.215:31337 (gateway)           desired_loc=0.315
0.8m ago: target=4ftT57SaDhXxmPsCU@100.27.151.80:31337 (gateway)        desired_loc=0.474
0.7m ago: target=i4QWRJcwgtBxtQf6@5.9.111.215:31337 (gateway)           desired_loc=0.058
0.5m ago: target=i4QWRJcwgtBxtQf6@5.9.111.215:31337 (gateway)           desired_loc=0.322

Zero requests came from connection_maintenance/acquire_new. The ring-directed acquisition path is completely silent. All outbound activity comes from initial_join_procedure which only fires when gateways are unconnected.

Connection Count: Plateaued at 9-10 with Constant Churn

Connection count histogram (782 unique connect events):
  count=0:  12 times  (bootstrap restarts)
  count=1:   6 times
  count=2:   4 times
  count=9:  46 times
  count=10: 699 times  ← stuck here
  count=11:  1 times   ← briefly exceeded

The peer hits 10 connections, oscillates between 9-10 with constant churn (169 unique peer addresses over 2h = connections forming and dropping), but never climbs past 10-11.

Timeline

• 47m ago: First connection (count=0 → 1)
• 47m → 17m: 30-minute gap with no telemetry — struggling through initial bootstrap
• 17m → now: Burst of activity, stabilized at count=10 with constant churn

Root Cause: connection_maintenance Is Not Initiating

The connection_maintenance loop should be calling acquire_new() every 5 seconds below min_connections=25, generating connect_request_sent events via ring-directed targeting. It generated zero. Possible explanations:

1. acquire_new returns None every time — with ~9-10 connections, the routing table may not have candidates for the gap-target locations. acquire_new selects the k-best peer to route through, and if none are close enough to the target, it returns None.

2. adjust_topology returns NoChange — if resource usage is in the 50-90% "happy zone", it won't request new connections even though we're well below min_connections=25. But this shouldn't apply below min_connections (the code checks current_connections < min_connections first).

3. pending_conn_adds drains but all attempts fail before telemetry emission — the connect_request_sent event is emitted in join_ring_request, which acquire_new calls. If acquire_new can't find a routing candidate, it returns before calling join_ring_request.

Most likely: explanation #1 — acquire_new can't find routing candidates for gap-targeted locations because the peer's 9-10 connections are clustered in the 0.3-0.7 range and there are no close peers to route through for targets near the ring boundaries (0.0-0.1, 0.9-1.0).

Implications

This confirms the issue is NOT about acceptance pickiness — it's about outbound initiation being dead. The peer has plenty of inbound connection offers (715 relayed requests) but its own outbound machinery (connection_maintenance) is completely non-functional.

The fix priority should be:

1. Diagnose why acquire_new returns None for peers with ~10 connections — likely needs to fall back to routing through ANY connected peer, not just the "best" one for the target location
2. Keep initial_join_procedure active even when gateways are connected — it should send CONNECTs through gateways to reach new targets
3. Increase MAX_CONCURRENT_CONNECTIONS during bootstrap as a multiplier on whatever attempts do get through

[sanity]

Issue Priority Assessment

Generated: 2026-03-12T22:24:10.070147923+00:00
Issue hash: 066fc710

Value Assessment

Initial join now keeps contributing even when gateways show as “connected”: the loop in initial_join_procedure continues routing CONNECTs through already attached gateways whenever we remain below min_connections and the deficit exceeds the gateway count, so the gateway path is no longer idle once ring-directed acquisition is awake (crates/core/src/operations/connect.rs:2350, crates/core/src/operations/connect.rs:2431). Likewise, connection_maintenance calls calculate_max_concurrent_connections every tick to scale the concurrency cap up to roughly half of min_connections when well below the target, so the low “3-in-flight” ceiling described in the issue is no longer hardcoded and now rises toward 11 slots at zero connections (crates/core/src/ring.rs:2022, crates/core/src/ring.rs:2483). Those two changes directly attack the “no outbound CONNECTs after the gateways are marked pending” problem that telemetry honed in on, so a recent checkout already addresses that portion of the report.

The probabilistic accept filter, however, still flips on when a node has only three active peer links. KLEINBERG_FILTER_MIN_CONNECTIONS remains at 3, and should_accept returns true only when the open count is below that constant before it even consults the gap score floor (crates/core/src/ring/connection_manager.rs:55, crates/core/src/ring/connection_manager.rs:569). Raising that threshold to something closer to min_connections / 2 (∼12 when min_connections is 25) would eliminate rejections during most of bootstrap and meaningfully raise the bilateral success probability because a typical node still sees the filter drop from 0.9 to 0.3 while it sits below 12 peers, including the NAT-failure-heavy tail the issue describes.

Users who start with few peers still get stuck in the 9–10 range because the Kleinberg filter is already enforcing selectivity long before the topology has enough neighbors to make clustering meaningful, and the bootstrap logic keeps waiting on min_connections before it considers itself healthy (crates/core/src/operations/connect.rs:2379). That stagnant state is a practical blocker for any feature that requires a ready ring (e.g., transaction acceptance or participation in routing quotas) even though it doesn’t block other work in the codebase. Because increasing this acceptance threshold is a trivial parameter tweak with no urgent deadline, the issue is moderate priority: it aligns with the project’s reliability goal and would raise the baseline experience for new peers, but it can wait until an upcoming sprint rather than needing immediate attention.

Cost Assessment

Raising the cost floor: the relevant entry points the issue calls out are still essentially the same in the current tree, so the action plan is still applicable. The Kleinberg filter still forces a probabilistic gate with only three connections (crates/core/src/ring/connection_manager.rs:55), the bootstrap acquisition loop still caps at three concurrent candidates and only scales via calculate_max_concurrent_connections (crates/core/src/ring.rs:1768 and crates/core/src/ring.rs:2483), and the initial/gateway join logic plus terminus-only acceptance still look as described (initial_join_procedure at crates/core/src/operations/connect.rs:2347, acquire_new at crates/core/src/ring.rs:2313, and the terminus acceptance enforcement at crates/core/src/operations/connect.rs:507). There have been no significant rewrites that would force a different approach, so the adjustments proposed in the issue remain actionable today.

Scope-wise the work touches multiple layers. At minimum KLEINBERG_FILTER_MIN_CONNECTIONS in connection_manager.rs needs to be raised (and the sliding-floor math may need to readjust), the connection maintenance machinery in ring.rs needs new heuristics for higher max concurrency and perhaps smarter acquisition that keeps poking the gateway path (especially where acquire_new currently returns None because no routing candidate exists for a target), and the initial_join_procedure in connect.rs must be extended to keep submitting CONNECTs “through” gateways even after the gateways themselves are connected. The accept-at-terminus rule likewise resides in operations/connect.rs:507–644, so relaxing it during bootstrap would require threading state through the relay decisions without breaking the uphill/TTL handling around lines 595–636. Each of those snippets is fairly intertwined with backoff, admission reservations, and routing metadata, so the implementation is non-trivial: you’ll need to ensure pending reservations, topology adjustments, and live transaction tracking stay consistent while the concurrency limits, forward routing, and acceptance conditions differ between bootstrap and steady state.

Non-code dependencies are minimal—no external approvals or services are needed—but the risk floor is high. Touching connection-growth parameters affects NAT traversal behavior, topology quality, and the heuristics enforced by should_accept() and the ConnectionEvaluator, so the change could easily harm steady-state topology or flood gateways if the gating/tracking logic isn’t carefully sandboxed to the <min_connections regime. Testing effort isn’t trivial either: existing unit tests around calculate_max_concurrent_connections (crates/core/src/ring.rs:2533) would need extension, and reliable verification likely requires simulated or instrumented bootstrap runs to confirm the new limits actually increase outbound attempts without breaking relay routing. For those reasons this is a medium-to-high cost change despite being parameter-tweaky—careful refactors and regression tests are required to keep the rest of the bootstrap and relay logic stable.

---

Generated by issue-prioritizer