LDI3-01: structured local-preference objectives (SLM-128)#359
Conversation
Add a shared, architecture-neutral objective library consumed by both the causal and TwoTower local trainers (one implementation, no duplicated math): - Legal-Set FTPO with `pairwise` (weighted G x B set margin, per-state mean so large sets don't dominate) and `mass` (legal probability-mass margin) variants. - tab_barrier: TAB-PO-inspired additive, separately-metered SFT anchor that lifts verified-critical, under-confident good actions only; reports barrier-active fraction and likelihood-erosion rate. - tbpo_inspired: TokenRatio/TBPO-inspired state-normalized action-ratio control vs a reference at the same state, with optional advantage baseline; disabled when legal-state support is inadequate. Typed, versioned, fail-closed config with a fingerprint that is part of run and adapter identity. Objectives are adapted (not reproduced); names carry `_inspired` accordingly. Non-trainable (constraint-shadow) views and out-of-legal action ids are refused (the E284 lesson). Verified: 14 tests including torch.autograd.gradcheck for all four objective/variant forms, hand-computed toy values, numeric stability at tiny masses, barrier gating + erosion, and config round-trip; existing unlikelihood/ftpo behavior untouched (152 preference tests pass). Adds a no-model-update objective report + committed fixture JSON. No quality claim. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VHgqD6cnywdnaXP9gRFCwt
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 50 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…iteria (LDI3-01) Reconcile the LDI3-01 objective library on main (from #359) with the acceptance criteria it did not yet cover, taking the remaining features from the parallel SLM-128 implementation. All additions are backward compatible (new config fields default off, new params default None), so #359's existing behavior and its tests are unchanged. - Ambiguous/unobserved actions: `_prepare` now extracts them from the ObjectiveView, validates them inside the legal set, enforces disjoint good/bad/ambiguous/unobserved partitions, and reports their legal mass separately (the issue requires good/bad/ambiguous/unobserved mass be reported separately and never mislabeled). - Reference/locality tethers: add `non_target_tether` / `target_tether` / `target_grace` config plus `_locality_tether`, composed onto every objective in `structured_decision_loss` and metered independently of the barrier so the target anchor never double-counts. - Semantic role weighting: wire the previously-dead `default_role_weight` and add an optional `good_role_weights` so structural tokens can be down-weighted in the barrier anchor; report `mean_role_weight`. Adds 3 tests (partition-mass reporting + legal validation, tether composition + separate metering + reference requirement, role-weight down-weighting + honored default). Full tests/test_harnesses/preference green (186), ruff + repo_policy clean. No model update, no quality claim. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U5Fy8PnMTV54p675y9T3B8
…ved mass, locality tethers, role weighting (#366) Complete the LDI3-01 structured-objective library on main (from #359) with the acceptance criteria it did not yet cover, reconciled from the parallel SLM-128 work. All additive and backward compatible. - Ambiguous/unobserved actions: extracted from the ObjectiveView, validated inside the legal set, disjoint-partition enforced, and their legal mass reported separately (never mislabeled as good/bad targets). - Reference/locality tethers: non_target_tether / target_tether / target_grace config + _locality_tether, composed onto every objective and metered independently of the barrier so the target anchor never double-counts. - Semantic role weighting: wires the previously-dead default_role_weight and adds optional good_role_weights so structural tokens can be down-weighted. 3 new tests; existing #359 tests unchanged; full preference suite green (186). No model update, no quality claim.
1 participant
Summary
Implements SLM-128 (LDI3-01): three OpenUI-native structured preference objectives in a shared, architecture-neutral library consumed by both the causal and TwoTower local trainers — one implementation, no per-architecture duplicated math. Objectives are adapted, not reproduced (names carry
_inspired). No model update / no quality claim.Unlike the LDI campaign harnesses, this issue is fully CPU-completable (out of scope: "no quality-bearing matrix run"), and it's gradcheck-verified.
What's here
src/slm_training/harnesses/preference/structured_objectives.py— objectives + config.scripts/report_structured_objectives.py+docs/design/ldi3-01-structured-objective-report-20260718.json— no-model-update evidence report.tests/test_harnesses/preference/test_structured_objectives.py— 14 tests.docs/design/ldi3-01-structured-objectives-20260718.md— design note.Objectives
legal_set_ftpo):pairwise(weighted G×B set margin, per-state mean so large sets don't dominate) andmass(softplus(margin + log P_B − log P_G)over legal-softmax masses; sums over the legal set only).tab_barrier(TAB-PO-inspired): pairwise preference + an additive, separately-metered SFT anchor−log p(g)applied only to verified-critical, under-confident good actions (zero for confident); reports barrier-active fraction + likelihood-erosion rate.tbpo_inspired(TokenRatio/TBPO-inspired): state-normalized good/bad log-ratio control vs a reference at the same state; optional advantage baseline; disabled when support is inadequate.Typed, versioned, fail-closed config (unknown fields rejected) with a fingerprint that is part of run/adapter identity.
Verification
pytest tests/test_harnesses/preference→ 152 passed (14 new + 138 existing, no breakage)torch.autograd.gradcheckpasses for all four objective/variant formsruffclean ·repo_policyok ·git diff --checkcleanScope
Objectives + config + gradient correctness + fixture report. Wiring the new names into the trainer dispatch for a quality-bearing run is deferred to the LDI3 matrix issues (explicitly out of scope here). Existing
unlikelihood/ftpo_*behavior is untouched.🤖 Generated with Claude Code
Generated by Claude Code