test: lint comfy hint-strings + snapshot the command inventory#516
Open
mattmillerai wants to merge 1 commit into
Open
test: lint comfy hint-strings + snapshot the command inventory#516mattmillerai wants to merge 1 commit into
mattmillerai wants to merge 1 commit into
Conversation
Add two anti-rot CLI-surface guards, mirroring the MCP server's EXPECTED_TOOLS + description-budget snapshot pattern: - Hint-string lint: extract every backtick/shell-hint `comfy …` invocation from help text, docstrings, hint/error strings, examples, and bundled skill docs, then validate each against the registered command set by walking the Typer app tree. Fails naming the offending string + file. A regression fixture pins that it flags the historical `comfy auth login` rot (and the sibling `auth whoami`). - Command-inventory snapshot: snapshot the full command tree so surface additions/removals/hides are explicit diffs in review. Regenerate with UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py. The lint immediately surfaced live rot: five hint strings pointed at the nonexistent `comfy auth whoami` (deliberately removed — see test_legacy_auth_whoami_is_gone). Route them to the real `comfy cloud whoami`. The remaining `comfy query` rot (CQL surface documented but no command registered) is baselined for a separate fix.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Judge call failed (status=parse_error): Could not parse JSON findings from output. First 500 chars:
I have enough to adjudicate. The shell/web tools are blocked in this environment, so I verified everything by reading the actual file and reasoning from Python internals.
Key adjudication decisions:
- **Dropped the gemini "high" AttributeError claim (line 172).** The claim that `ast.Constant` nodes inside f-strings lack `.lineno` in Python 3.10/3.11 is false — parsed f-string literal parts carry valid (if historically imprecise) position info; `.lineno` never raises `AttributeError` on parsed
Re-trigger by removing and re-adding the cursor-review label.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI-5
Some of the CLI's help messages and error hints told you to run commands that
don't exist — like
comfy auth login(the real one iscomfy cloud login).Nobody noticed for a long time. This adds two tests that act like a spell-checker
for command names:
comfy …command mentioned in help text, hints,and error messages and checks each one is a real registered command. If a
string points at a command that doesn't exist, the test fails and names the
exact string + file.
up as an obvious diff in review (mirrors the MCP server's
EXPECTED_TOOLSpattern).
Turning the lint on immediately caught live rot — five hints pointed at the
removed
comfy auth whoami— so those are corrected tocomfy cloud whoamitoo.What & why
The MCP server prevents help/hint rot with
EXPECTED_TOOLS+ description-budgetsnapshot tests; the CLI had no equivalent, which is how seven strings referenced
the nonexistent
comfy auth loginfor who knows how long.1. Hint-string lint (
tests/comfy_cli/test_cli_surface.py)comfy …invocation — backtick-quoted,run:/$-prefixed shell hints, and whole-string command lines — from allPython string literals (docstrings,
help=/hint=/message=kwargs,examples) and bundled skill docs, scanned via AST so comments and code
identifiers are never mistaken for references.
tree (
build_help_json). Correctly accepts groups, leaves-plus-args(
comfy generate list), and hidden commands (comfy agent-review); flags the"valid group + nonexistent subcommand" rot class (
comfy auth login/whoami).like "the comfy skills into Claude Code" is not mistaken for a command
(verified: zero false positives across the whole package).
2. Command-inventory snapshot — snapshots every command path (groups +
leaves, with a
[hidden]marker) totests/comfy_cli/fixtures/cli_command_inventory.txt.Regenerate after an intentional surface change with one command:
Both run under the normal
pytestinvocation, so they gate every PR in CI.Rot the lint surfaced (and what this PR does about it)
comfy auth whoami(5 hints) →comfy cloud whoami.auth whoamiwasdeliberately removed (there is an existing
test_legacy_auth_whoami_is_gone);the hints were stale. All five are cloud-auth contexts, so they route to the
real
comfy cloud whoami. Fixed here.comfy query(2 refs — an example + acomfy run-clidemo step). The CQLquery surface is documented but no
querycommand is registered on the app.Whether to wire one up or rewrite the examples is a product call, so it is
baselined (
BASELINE_INVALID_REFS) for a separate fix rather than bundledinto this guard.
test_baseline_is_not_stalefails the moment it's fixed,forcing the baseline entry to be removed so the allowlist can't outlive the bug.
comfy versionis allowlisted — it's the--versionflag's output-envelopecommand name (in
discovery.COMMAND_SCHEMAS), not an invokable subcommand.Acceptance
auth loginrefs — pinned by the regressionfixture in
test_lint_catches_auth_login_and_friends(backtick,run:, andwhole-string forms), per the ticket's "add one as a regression fixture" note.
pytest, picked up by the existing pytestworkflow.
Notes for the reviewer
auth loginhints tocomfy cloud login; the live lint depends on those being fixed to stay green.GitHub will retarget this to
mainwhen that PR merges.auth whoami → cloud whoamicorrections the lint immediately surfaced.tests/comfy_clisuite green (2475 passed, 13 skipped);ruff check+ruff formatclean.Judgment calls
auth whoamirot inline (clean, unambiguous, test-backed) butbaselined
comfy query(needs a product decision) — kept this PR focused onthe guard plus the rot that's mechanically safe to fix.
*.mdskill docs too (agent-facing help); all currentrefs there are valid.