feat(vardiff): replace threshold-ladder with adaptive EWMA algorithm

[gimballock]

Summary

Replace the threshold-ladder vardiff algorithm with the decline-safety-selected
champion: a three-stage adaptive composition (EWMA estimator / sign-persistence
adaptive boundary / accelerating partial retarget) that is selected to never enter
the over-difficulty death-spiral on a sustained decline.

This is the clean, production-ready extraction of the shipping algorithm from the
simulation framework in #2154 — a single flat-struct replacement of VardiffState,
no scaffolding. (Note: an earlier revision of this PR shipped the mid-arc contender
selected by a scalar fitness sweep; this revision ships the champion selected by the
decline-safety constraint that superseded it. See "Selection" below.)

Motivation

The existing vardiff was observed producing excessive jitter at moderate share rates
and sluggish reaction to genuine hashrate drops at high share rates. Investigation
with the simulation framework (#2154) showed these are design limitations, not
tuning problems:

1. Reactivity degrades with share rate. After convergence the old algorithm
   averages over an ever-growing window; old stable data dilutes a new signal. At
   60+ shares/min, only 9–16% of genuine −50% drops are detected within 5 minutes.
2. Jitter and reactivity are coupled. The fixed 15% threshold is simultaneously
   too loose at high share rates (noise floor 3–9%) and too tight at low rates.
3. Full retarget overshoots on noisy estimates. Jumping the full distance to a
   noisy estimate overshoots during cold-start (~30%).

(REMOVED the old Motivation #4 — "tightening rejects in-flight shares." That framing
is incorrect: the pool validates each share against the per-job target snapshotted at
job creation (job_id_to_target), which the vardiff path never mutates, so a
difficulty change does not reject in-flight shares. Thanks to @vnprc for the catch.
The asymmetry in the new algorithm is justified by decline-safety, not lost work —
see Stage 2.)

The algorithm

Three sequential stages per evaluation tick (behaviorally identical to the
champion_composed reference validated in #2154):

Stage 1: EWMA Estimator (tau = 360s)

Exponentially-weighted moving average, 360-second time constant. The 360s window is
the floor of the τ-safety-valley: shorter windows fail the slow-decline safety gate
at sparse share rates (they lurch on noise and can talk themselves into
over-difficulty); longer ones only add lag. On fire the EWMA rescales by the
difficulty ratio rather than resetting, preserving history.

Stage 2: Adaptive Boundary (seam at 6 shares/min)

• Below 6 shares/min: PoissonCI — wide confidence interval, prevents premature
  fires on sparse data (small miners / bitaxe).
• At or above 6 shares/min: sign-persistence CUSUM — a sequential-testing
  boundary that protects the dangerous (tightening) direction two ways: an
  asymmetric multiplier (tightening requires 8× the evidence of loosening) and a
  sign-persistence discount (the threshold relaxes only after consecutive
  same-direction ticks accumulate, so a single lucky streak can't trip a tightening).
  This is decline-safety / dangerous-direction protection: tightening into a
  falling miner is the move that spirals, so the boundary resists it as a constraint
  — not a "rejecting in-flight shares" cost (it doesn't; see Motivation).

Stage 3: Accelerating Partial Retarget

Each fire moves 20% toward the estimate; consecutive same-direction fires ramp the
step by 0.05 per fire, capping at 60% — converging faster when the signal is clearly
real, without overshooting on a single noisy sample.

Selection: decline-safety, not fitness

The champion is the gentlest composition that clears the decline-safety gate —
selected by a minimax over a sustained-decline grid (the death-spiral test), as a
hard constraint, not by a weighted fitness score. An earlier revision of this PR
used the fitness-sweep winner (EWMA 120 / CUSUM-at-10 / tighten 3×); that contender
fails the slow-decline gate at sparse rates, which is why the selection criterion
moved to decline-safety and the parameters changed (notably tighten-multiplier 3 → 8:
a hard safety constraint demands more tightening-reluctance than a stability-weighted
fitness score did). Full derivation: #2154 / information-floor.md.

Results (champion, regenerated; 1000 trials/cell, deterministic seeding)

Metric	Old (SPM 6)	Champion (SPM 6)	Champion (SPM 30)
Cold-start overshoot (p99)	28%	5.7%	0%
Jitter (mean fires/min)	0.033	0.038	0.003
Convergence time (p90)	9m	12m	9m
Detect −50% drop (within 5m)	87%	92.8%	100%
Settled accuracy (p50)	—	6.6%	0.0%
Detect +50% increase (within 5m)	10%	39%	2.2%

Read the +50%-increase row honestly: the champion is deliberately slow to react
to a rising miner (2.2% at SPM 30), far slower than the contender was. That is the
decline-safety tradeoff working as designed — tightening (the response to a faster
miner) is the dangerous direction, so the champion requires sustained evidence before
raising difficulty. A miner that genuinely speeds up is still caught, just over
minutes of consistent signal rather than on one lucky tick. Jitter at SPM 6 is a hair
higher than the contender (0.038 vs 0.033) for the same reason — the slow window
trades a little steady-state churn for decline-safety. These are the right trades for
a controller whose job is to never strand a declining miner.

Breaking changes

• Adds private fields to VardiffState. Requires channels_sv2 major version bump.
• shares_since_last_update semantics: "shares since last evaluation tick" (the EWMA
  consumes and zeroes the counter each tick).
  Public constructor API (new, new_with_min) and Vardiff trait unchanged.

Test plan

• cargo test -p channels_sv2 — 14 vardiff property tests (champion contract)
• cargo test --verbose — full workspace passes
• cargo fmt --check — clean
• Re-deploy to testnet4 with live miners (the prior testnet run was the contender)

[vnprc]

We discussed this PR at @TriangleBitDevs yesterday and I think we came away with some valuable feedback. Shoutout @gimballock for presenting your work!

Both boundaries apply asymmetric cost: 3x more evidence required to tighten difficulty than to loosen, reflecting that tightening rejects in-flight shares while loosening is free.

I don't believe the asymmetric boundary cost is a useful optimization. This seems to stem from a misunderstanding of how difficulty adjustments interact with existing jobs.

Difficulty updates do not invalidate existing jobs. Those jobs remain valid until an on_set_new_prev_hash message is received, as indicated by this code comment. In other words, the old difficulty target remains valid until a new block is found.

The lines of code that prove it:

1. Target snapshotted at job creation. self.target is captured into the map per job ID: standard extended
2. Share validated against the per-job snapshot, not current channel target. job_target from the map is what the share hash is checked against: standard extended
3. update_channel (vardiff path) only mutates self.target, never touches job_id_to_target, so existing jobs retain their original target: standard extended
4. The map is wiped on new block: standard extended

I think we also exposed a similar misunderstanding of how shares rejected due to low difficulty impact profitability of a miner. They don't.

Any shares rejected due to low difficulty do not represent lost value in aggregate because shares found at or above the new difficulty threshold are worth proportionally more. In other words, the shares that pass the new difficulty threshold make up for the lost value of any shares that do not pass the new threshold. Rejected shares are a usability problem because they seem to indicate an error to the human monitoring these metrics. Assumptions that these shares lead to lost value for the miner arise from misunderstanding the nuances of share value calculation.

[gimballock]

reflecting that tightening rejects in-flight shares while loosening is free.

You're right that the pool's job_id_to_target snapshot design prevents difficulty adjustments from rejecting in-flight shares — the "rejecting in-flight shares" framing in the docs is incorrect and we'll fix it. Good catch, and thanks for inviting me to speak at your BitDevs.

The asymmetric boundaries aren't justified by that rationale though — they're justified by our simulation framework results. We swept tighten_multiplier over [1.0, 1.25, 1.5, 2.0, 2.5, 3.0] across 1000 Monte Carlo trials per cell and found the multiplier 3.0 produced the highest fitness score.

What the asymmetry actually does: it suppresses false tightenings from lucky streaks, reducing steady-state jitter and preventing large upward difficulty jumps. The cost is slower convergence when tightening is genuinely needed (e.g. hardware upgrade).

One caveat we're transparent about: our fitness metric weights stability heavily (jitter + step_magnitude_safety = 50%) over reactivity (25%) and convergence (15%). Under more balanced weights (30/30/30/10), the convergence penalty gets amplified 2x while the stability gains get discounted — a lower multiplier like 1.5–2.0 would likely rank higher, producing faster adaptation to genuine hashrate increases at the cost of more false fires and higher jitter on transient spikes.

So the 3.0 constant is optimal given our prioritization, not in any absolute sense - it optimizes for "never surprise the miner" over "track hashrate changes quickly." We're open to discussing whether the weighting reflects the right tradeoffs for the upstream project, and whether a lower multiplier (with its faster convergence but higher churn) better serves the broader community of pool operators.

[gimballock]

Following up on this — your correction is now fully incorporated, and it ended up
reshaping more than the wording.

1. The in-flight-shares framing is retracted, not just patched. You were right on the
   mechanism (job_id_to_target snapshots the target at job creation; the vardiff path
   never mutates it; the map only clears on a new block) and right on the aggregate-value
   point (shares rejected for low difficulty aren't lost value — the shares that clear
   the new target are worth proportionally more). The PR no longer claims a lost-work
   cost; the motivation section says so explicitly and credits the catch.

2. The asymmetry's justification itself changed since my last reply — and your
   objection is part of why. I defended it then on the fitness sweep (tighten_multiplier
   3.0 won), and flagged that the metric weighted stability heavily. Since then the
   selection criterion changed: the shipped controller isn't the fitness-pick anymore —
   it's the gentlest composition that clears a decline-safety gate (it must not enter the
   over-difficulty death-spiral on a sustained decline; the earlier EWMA(120)/CUSUM
   contender actually fails that gate). So the asymmetry isn't there because stability
   was over-weighted in a fitness function — it's there because tightening into a falling
   miner is the dangerous direction, and the controller has to resist it as a hard
   constraint. That directly addresses your weighting concern: it's no longer a weighting
   tradeoff, it's a safety floor. (It also made the asymmetry stronger — tighten
   multiplier went 3 → 8 — which is what you'd expect when you move from a fitness score
   to an unforgiving constraint.)

3. Consequently the algorithm in this PR changed. It's now the decline-safety champion
   (EWMA 360 / sign-persistence adaptive boundary, seam at 6 / accelerating partial
   retarget), the same one [Draft] feat(vardiff): decline-safe vardiff champion + simulation framework + proof #2154 derives — the previous revision here was the
   pre-decline-safety contender. Thanks again for the catch; it's a better PR for it.