Skip to content

test: lint comfy hint-strings + snapshot the command inventory#516

Open
mattmillerai wants to merge 1 commit into
matt/be-2975-fix-auth-login-refsfrom
matt/be-2979-hint-lint-discover-snapshot
Open

test: lint comfy hint-strings + snapshot the command inventory#516
mattmillerai wants to merge 1 commit into
matt/be-2975-fix-auth-login-refsfrom
matt/be-2979-hint-lint-discover-snapshot

Conversation

@mattmillerai

Copy link
Copy Markdown
Collaborator

ELI-5

Some of the CLI's help messages and error hints told you to run commands that
don't exist — like comfy auth login (the real one is comfy cloud login).
Nobody noticed for a long time. This adds two tests that act like a spell-checker
for command names:

  1. A lint that reads every comfy … command mentioned in help text, hints,
    and error messages and checks each one is a real registered command. If a
    string points at a command that doesn't exist, the test fails and names the
    exact string + file.
  2. A snapshot of the full command list, so adding or removing a command shows
    up as an obvious diff in review (mirrors the MCP server's EXPECTED_TOOLS
    pattern).

Turning the lint on immediately caught live rot — five hints pointed at the
removed comfy auth whoami — so those are corrected to comfy cloud whoami too.

What & why

The MCP server prevents help/hint rot with EXPECTED_TOOLS + description-budget
snapshot tests; the CLI had no equivalent, which is how seven strings referenced
the nonexistent comfy auth login for who knows how long.

1. Hint-string lint (tests/comfy_cli/test_cli_surface.py)

  • Extracts every recognizable comfy … invocation — backtick-quoted,
    run:/$-prefixed shell hints, and whole-string command lines — from all
    Python string literals (docstrings, help=/hint=/message= kwargs,
    examples) and bundled skill docs, scanned via AST so comments and code
    identifiers are never mistaken for references.
  • Validates each against the registered command set by walking the Typer app
    tree (build_help_json). Correctly accepts groups, leaves-plus-args
    (comfy generate list), and hidden commands (comfy agent-review); flags the
    "valid group + nonexistent subcommand" rot class (comfy auth login/whoami).
  • Fails naming the offending string + file. Extraction is marker-gated, so prose
    like "the comfy skills into Claude Code" is not mistaken for a command
    (verified: zero false positives across the whole package).

2. Command-inventory snapshot — snapshots every command path (groups +
leaves, with a [hidden] marker) to tests/comfy_cli/fixtures/cli_command_inventory.txt.
Regenerate after an intentional surface change with one command:

UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py

Both run under the normal pytest invocation, so they gate every PR in CI.

Rot the lint surfaced (and what this PR does about it)

  • comfy auth whoami (5 hints) → comfy cloud whoami. auth whoami was
    deliberately removed (there is an existing test_legacy_auth_whoami_is_gone);
    the hints were stale. All five are cloud-auth contexts, so they route to the
    real comfy cloud whoami. Fixed here.
  • comfy query (2 refs — an example + a comfy run-cli demo step). The CQL
    query surface is documented but no query command is registered on the app.
    Whether to wire one up or rewrite the examples is a product call, so it is
    baselined (BASELINE_INVALID_REFS) for a separate fix rather than bundled
    into this guard. test_baseline_is_not_stale fails the moment it's fixed,
    forcing the baseline entry to be removed so the allowlist can't outlive the bug.
  • comfy version is allowlisted — it's the --version flag's output-envelope
    command name (in discovery.COMMAND_SCHEMAS), not an invokable subcommand.

Acceptance

  1. Would have caught all 7 auth login refs — pinned by the regression
    fixture in test_lint_catches_auth_login_and_friends (backtick, run:, and
    whole-string forms), per the ticket's "add one as a regression fixture" note.
  2. Snapshot update path documented — one command, above and in the test.
  3. Runs in CI on every PR — plain pytest, picked up by the existing pytest
    workflow.

Notes for the reviewer

  • Stacked PR. Based on the branch that routes the seven auth login hints to
    comfy cloud login; the live lint depends on those being fixed to stay green.
    GitHub will retarget this to main when that PR merges.
  • The two source-file kinds of change are: the new tests/snapshot, and the
    auth whoami → cloud whoami corrections the lint immediately surfaced.
  • Full tests/comfy_cli suite green (2475 passed, 13 skipped); ruff check +
    ruff format clean.

Judgment calls

  • Fixed the auth whoami rot inline (clean, unambiguous, test-backed) but
    baselined comfy query (needs a product decision) — kept this PR focused on
    the guard plus the rot that's mechanically safe to fix.
  • The lint scans bundled *.md skill docs too (agent-facing help); all current
    refs there are valid.

Add two anti-rot CLI-surface guards, mirroring the MCP server's
EXPECTED_TOOLS + description-budget snapshot pattern:

- Hint-string lint: extract every backtick/shell-hint `comfy …`
  invocation from help text, docstrings, hint/error strings, examples,
  and bundled skill docs, then validate each against the registered
  command set by walking the Typer app tree. Fails naming the offending
  string + file. A regression fixture pins that it flags the historical
  `comfy auth login` rot (and the sibling `auth whoami`).
- Command-inventory snapshot: snapshot the full command tree so surface
  additions/removals/hides are explicit diffs in review. Regenerate with
  UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py.

The lint immediately surfaced live rot: five hint strings pointed at the
nonexistent `comfy auth whoami` (deliberately removed — see
test_legacy_auth_whoami_is_gone). Route them to the real `comfy cloud
whoami`. The remaining `comfy query` rot (CQL surface documented but no
command registered) is baselined for a separate fix.
@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 39ac0922-7ac7-497c-afa8-55f5beee6b93

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch matt/be-2979-hint-lint-discover-snapshot
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch matt/be-2979-hint-lint-discover-snapshot

Comment @coderabbitai help to get the list of available commands.

@mattmillerai mattmillerai marked this pull request as ready for review July 14, 2026 20:07
@mattmillerai mattmillerai added cursor-review Request Cursor bot review agent-coded PR authored by the agent-work loop labels Jul 14, 2026
@dosubot dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Jul 14, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

⚠️ Review failed

Judge call failed (status=parse_error): Could not parse JSON findings from output. First 500 chars:
I have enough to adjudicate. The shell/web tools are blocked in this environment, so I verified everything by reading the actual file and reasoning from Python internals.

Key adjudication decisions:

- **Dropped the gemini "high" AttributeError claim (line 172).** The claim that `ast.Constant` nodes inside f-strings lack `.lineno` in Python 3.10/3.11 is false — parsed f-string literal parts carry valid (if historically imprecise) position info; `.lineno` never raises `AttributeError` on parsed 

Re-trigger by removing and re-adding the cursor-review label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded PR authored by the agent-work loop cursor-review Request Cursor bot review size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant