feat: automatic node eviction on low disk

[jacderida]

Summary

Automatic node eviction when a node's disk runs low, driven by the shared daemon so it applies to
both the ant CLI and the ant-gui app.

• When free space at a node's data directory falls to 500 MB (the reserve at which ant-node
  itself refuses to store chunks), the daemon stops the smallest co-located node and deletes its
  data directory to reclaim space for the others. Smallest-first minimises the re-replication
  cascade on the network.
• Evicted nodes get a persisted evicted status with an explanatory reason (survives daemon
  restarts), are never restarted, and can be dismissed with ant node dismiss <id>.
• Adds an extensible fleet-health layer (green/warning/critical) that warns before an
  eviction and names the next candidate — shown in ant node status and served at a new
  GET /api/v1/health.
• A node that is the only one on its partition is never evicted; that surfaces as critical
  health so the operator can act manually.
• start-all/stop-all skip evicted nodes; single start/stop reject them with a clear message.
• Windows: data-directory deletion retries with backoff so space is reclaimed once the OS
  releases the just-killed process's file handles (LMDB map / the node's own copied binary).
• Thresholds (500 MB evict / 1024 MB warn) are fixed internal constants — not user-configurable.
• Backward compatible: the new eviction field is #[serde(default)], so pre-upgrade
  node_registry.json files load unchanged.

Test plan

• cargo test -p ant-core green — unit + daemon integration, including new tests for candidate
  selection, health levels, Evicted status derivation, pre-upgrade registry load, and the
  delete-retry helper.
• Live testnets: filled a capped 2 GB loopback partition on Linux and macOS (two testnets each)
  — node evicted at 500 MB free, data directory removed, status shows evicted, dismiss works, and
  the fleet-health indicator escalated green → warning → critical.
• Windows: verified the data directory is now removed on eviction (the retry fix).

🤖 Generated with Claude Code

[dirvine]

Hermes review-team summary

Verdict: hold for two eviction-safety fixes. I reviewed this together with a 6-agent review team and then independently verified the concrete findings locally.

Blockers

1. Eviction candidate sizing should not follow symlinks or use logical file length

ant-core/src/node/daemon/disk.rs says symlinks are not followed, but DirEntry::metadata() follows symlinks. That means a symlinked directory/file inside a node data dir can make the size walk escape the node tree and count unrelated data.

The same function uses meta.len(), which is logical file length rather than allocated/reclaimable disk usage. For sparse/LMDB-style files this can mis-rank nodes for the “smallest data dir” eviction policy and misreport reclaimed bytes. Since eviction is automatic and destructive, the candidate should be based on actual reclaimable usage inside the node tree.

Suggested direction:

• use non-following metadata/traversal (symlink_metadata() or equivalent);
• on Unix, prefer allocated blocks (MetadataExt::blocks() * 512) for regular files;
• add regression tests for symlink escape and sparse-file sizing.

2. Eviction/start race between stop/delete and persisted marker

evict_node stops the node, then deletes the data dir, then persists config.eviction / marks runtime Evicted. In the window after stop_node() returns and before the eviction marker is visible, a concurrent POST /nodes/{id}/start can pass the current guards and spawn the node while its data dir is being deleted.

Suggested direction:

• make the node terminal/unstartable before the stop/delete window, e.g. persist an eviction/evicting marker first, or add a supervisor Evicting state that start rejects;
• make rollback semantics explicit if stop/delete fails.

Related existing issue, not introduced here

UpgradeScheduled is a live process state, but Supervisor::is_running() only treats exact Running as running. That can allow unsafe remove/refuse stop/double-start behaviour around upgrade-scheduled nodes. I verified this appears to exist on main already, so I’m not treating it as introduced by this PR, but it sits on the same safety boundary.

CI / non-blocking notes

• cargo fmt --all -- --check and rustdoc currently fail for PR-caused formatting/doc-link issues; these are mechanical.
• The security audit failure is already present on main (quinn-proto / RUSTSEC-2026-0185 via saorsa-transport), so I am not treating it as PR-caused.
• Focused local checks passed for the disk/health/eviction tests, but the blockers above are not currently covered by tests.

[jacderida]

Thanks for the thorough review @dirvine — both blockers were real. Addressed in 7bbb49e.

1. Eviction candidate sizing (symlinks + logical length)

• dir_size now examines entries with symlink_metadata and skips symlinks entirely, so the walk can't follow a link out of the node's data directory and count unrelated data.
• It sums allocated disk usage (blocks() * 512 on Unix) instead of len(), so LMDB's sparse data.mdb no longer mis-ranks the "smallest" node or inflates the reported reclaimed bytes. Non-Unix falls back to logical length (std exposes no allocated size there).
• Added regression tests for both the symlink-escape and sparse-file cases.

2. Eviction/start race

• evict_node now makes the node unstartable before the stop/delete window. It sets an in-memory evicting flag under the supervisor write lock — which start_node checks under that same lock, so a concurrent POST /nodes/{id}/start is atomically excluded — and persists the eviction marker first (so the terminal state also survives a daemon crash mid-delete).
• Rollback semantics are now explicit: once the process is stopped the node stays Evicted whether or not the delete succeeds; on delete failure the reason text and reclaimed_bytes reflect that no space was reclaimed, and it logs at error level.
• Added a regression test that start_node is refused while a node is evicting.

CI

• cargo fmt --all applied and the two rustdoc links fixed (the private-item link and the unresolved EvictionRecord link). Verified locally: fmt clean, cargo doc -D warnings clean, clippy -D warnings clean, unit tests green.

Out of scope (agree with your assessment)

• The Security Audit failure (quinn-proto RUSTSEC-2026-0185 via saorsa-transport) is pre-existing on main.
• The is_running() / UpgradeScheduled gap is pre-existing and adjacent — happy to open a follow-up issue to track it if that's useful.

[dirvine]

Re-reviewed current head 7bbb49eaf6c702550a7d73b9d4f0876a1fab11b6 after the eviction-safety fixes.

The two issues I previously flagged are addressed:

• dir_size now uses non-following symlink_metadata, skips symlinks, uses allocated blocks on Unix, and has regression coverage for symlink escape and sparse-file sizing.
• eviction now marks a node as evicting and persists an eviction marker before the stop/delete window, so start_node rejects it while deletion is in progress.

Focused local checks passed:

• cargo test -p ant-core --lib disk -- --test-threads=1 — 12 passed
• cargo test -p ant-core --lib eviction -- --test-threads=1 — 5 passed
• cargo test -p ant-core --lib start_node_rejects_a_node_being_evicted -- --test-threads=1 — 1 passed
• cargo fmt --all -- --check — passed
• RUSTDOCFLAGS='-D warnings' cargo doc --all-features --no-deps — passed

CI note: the remaining Security Audit failure is the pre-existing quinn-proto advisory already present on main, not introduced by this PR. E2E jobs were still pending when checked, so this approval is for the code review; branch protection should still enforce final CI.