7/7 Fix the retrieval path, and enforce the claim ladder - #308
Closed
Alek99 wants to merge 3 commits into
Conversation
Everything in the previous six commits is worthless if the next agent still opens `styles.py`, counts ten properties, and concludes the library is barely customizable. That is a retrieval failure, not a capability one, and it needs its own fix. `CLAUDE.md` — which is `AGENTS.md`, they are the same file — now names both registries in its opening paragraph, before the layout section anyone skims past, and says explicitly that `styles.py` looks like the whole answer and is not. `limitations-and-alpha-status.md` and the capability matrix cross-link in both directions. The README carries one pointer. An `xy-evaluate` skill carries the same map for agents that never see the checkout, and `tests/test_evaluation_skill.py` fails when a path it names moves — a skill full of dead pointers reproduces the original failure one level up. Two stale pointers fixed while in there: `CLAUDE.md` sent readers to `scripts/bench.py` and `scripts/bench_vs.py`, which have lived in `benchmarks/` for some time. The claim ladder is committed in the comparison document, and its last rung is now mechanically enforced rather than trusted. `check_claim_guardrails.py` covered performance superlatives and nothing else — "XY is the most customizable charting library" passed it clean, which I verified before writing this. It now rejects that shape along with "fully customizable", "style anything", and "more extensible than any …", and requires a comparative claim against a named library to carry its dimension and its evidence. It also scans `README.md`, which it previously did not, and which is where a slogan is most likely to be written and least likely to be reviewed. The new rule caught two live overclaims in `docs/index.md` on its first run. Both are rewritten to say what is actually true and to link the matrix.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
…schema-coverage' into claude/customization-capability-matrix-vckjxg-7-retrieval-path
Merging this PR will not alter performance
Comparing 
Footnotes
|
The two rendering fixes in this branch are the kind that a reader either takes on faith or reproduces by hand. Three figures in spec/assets remove that choice, rendered by checking out the pre-change tree into a worktree, building its Rust core and its client, and running the same figure through both — so the only difference between the columns is the change. Two facts fall out of the frames and are worth stating because they are the ones a reviewer would otherwise have to verify: The native raster output is byte-identical before and after. `stroke_shaped` delegates to the original `stroke` for round/round, and the pixels agree with `np.array_equal`, so nothing that already renders moves. The browser output changed in exactly 1024 pixels, in two clusters at the polyline's two ends. That is the whole diff: the caps, and nothing else. The area-outline figure is the same comparison for the quieter half of the bug, where the SVG writer named its join and let the cap default to the format's flat one — which `_pdf` then read back flat while the rasterizer drew round.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #307. Last of the stack.
Everything in the previous six PRs is worthless if the next agent still opens
styles.py, counts ten properties, and concludes the library is barely customizable. That is a retrieval failure, not a capability one, and it needs its own fix.Where the pointers now are
CLAUDE.md— which isAGENTS.md, they are the same file via symlink — names both registries in its opening paragraph, before the layout section anyone skims past, and says explicitly thatstyles.pylooks like the whole answer and is not.limitations-and-alpha-status.md↔ the capability matrix, both directions.xy-evaluateskill under.agents/skills/, carrying the same map for agents that never see the checkout.tests/test_evaluation_skill.pyfails when a path it names moves — a skill full of dead pointers reproduces the original failure one level up.Two stale pointers fixed in passing:
CLAUDE.mdsent readers toscripts/bench.pyandscripts/bench_vs.py, which have lived inbenchmarks/for some time.The claim ladder, enforced rather than trusted
The ladder is committed in the comparison document. Its last rung now has teeth.
check_claim_guardrails.pycovered performance superlatives and nothing else. I verified before writing this that "XY is the most customizable charting library" passed it clean, exit 0 — and thatREADME.mdwas not in its scanned set at all, which is where a slogan is most likely to be written and least likely to be reviewed.It now rejects that shape along with "fully customizable", "style anything", and "more extensible than any …", and requires a comparative claim against a named library to carry its dimension and its evidence. A properly qualified claim still passes — the comparison document's own rows are the shapes worth copying.
It caught two live overclaims on its first run
docs/index.mdsaid the library "is completely customizable" and led a bullet with "Completely customizable." Both rewritten to say what is true and link the matrix.The eval
Hand a fresh agent the clone and ask "is xy more customizable than Plotly?". If it answers from
styles.pyalone and says no, the retrieval path is still broken. Worth running before each release rather than after.uv run pytest2365 passed · pre-commit clean · codespell clean on all changed docs.Generated by Claude Code