Skip to content

test(retrieval): gate graph boost end to end [release:v0.28.0] - #29

Draft
tcballard wants to merge 2 commits into
mainfrom
agent/graph-boost-eval
Draft

test(retrieval): gate graph boost end to end [release:v0.28.0]#29
tcballard wants to merge 2 commits into
mainfrom
agent/graph-boost-eval

Conversation

@tcballard

@tcballard tcballard commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds deterministic release coverage for the v0.28.0 graph-boost lexical floor.

Adds:

  • four adversarial search-artifacts component cases over a 39-artifact corpus
  • a black-box explain/counterfactual test and a non-gating ratio sweep
  • two DecisionGrounding scenarios that exercise the same failure through an agent's four-artifact grounding budget

Companion implementation: asdecided/core#449.

Scope

Included

  • a focused lexical policy, graph-popular hub, and seven connected catalogue distractors
  • category floors requiring P@1, R@5, and MRR of 1.0 for all graph-gate cases
  • proof that the pre-floor RRF formula would rank a weaker candidate first
  • proof that explain mode is deterministic and does not change result membership
  • a 0.75–0.95 diagnostic sweep that identifies 0.85 as the first passing grid point
  • two downstream prohibition scenarios requiring the governing decision to be first inside the four-item RAC budget
  • explicit execution of that slice in the DecisionGrounding Core-integration job

Excluded

  • tuning Core's ranking constant from DecisionGrounding outcomes
  • embeddings, an LLM judge, or probabilistic scoring
  • a new standalone benchmark family
  • changing the pinned release installer before a v0.28.0 release and checksum exist

Product / Architecture Decisions

  • search-artifacts owns the component regression because it observes production rank order directly.
  • DecisionGrounding remains a small downstream acceptance slice: it verifies that corrected order reaches the agent and changes adherence.
  • The scorecard consumes production order verbatim. The ratio sweep is separate, counterfactual, and non-gating.
  • The fixtures are intentionally relationship-heavy so a future formula change cannot turn this into a no-op test.

Verification

Ran

  • parsed every repository JSON file
  • compiled the three added Python test/diagnostic modules
  • parsed the updated DecisionGrounding workflow YAML
  • decided validate search-artifacts/corpus — 39 valid artifacts
  • decided relationships search-artifacts/corpus --validate — 33 relationships, 0 issues
  • decided validate decisiongrounding/scenarios_retrieval_gate/corpus — 9 valid artifacts
  • decided relationships decisiongrounding/scenarios_retrieval_gate/corpus --validate — 18 relationships, 0 issues
  • retrieval-gate tests without Core — 3 passed, 2 skipped
  • repository invariant tests — 4 passed
  • git diff --check

Regression Evidence Against v0.26.0

  • graph-gate P@1: 0.0; MRR: 0.245833
  • overall P@1: 0.857143; MRR: 0.892262
  • both downstream RAC cases omit the governing decision from the four-artifact budget
  • ratio sweep: 0.85 is the first tested ratio with P@1 of 1.0 across all four cases

These old-release failures are intentional evidence that the fixtures detect the defect.

Release Dependency

This draft must remain blocked until:

  • Core PR #449 passes and merges.
  • Release-prep PR asdecided/core#450 merges.
  • v0.28.0 is published with native artifacts.
  • .github/scripts/install-asdecided.sh records the v0.28.0 checksums.
  • both benchmark workflows install v0.28.0.
  • the full component and DecisionGrounding batteries pass against that released binary.

Current CI installs v0.26.0 for the component suite and v0.23.1 for the DecisionGrounding Core job, so release-dependent checks are expected to remain red until that follow-up commit.

Review Path

Suggested order: graph fixture corpus and queries, black-box test and ratio sweep, DecisionGrounding scenarios and assertions, then baseline and documentation.

Notes For Reviewer

The benchmark does not import Core code. All scored behavior crosses the released decided CLI boundary, preserving the external-system-under-test contract.

Adds deterministic component and downstream adherence coverage for the v0.28.0 retrieval gate.

Signed-off-by: Tom Ballard <tom.ballard08@googlemail.com>
Signed-off-by: Tom Ballard <tom.ballard08@googlemail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant