Evaluate AGENTS.md and skills with Codex#70120
Conversation
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-3-test
jason810496
left a comment
jason810496
left a comment
There was a problem hiding this comment.
Thanks for adding the Codex support!
| installed_version = json.loads(package_json.read_text())["version"] | ||
| except (OSError, KeyError, ValueError): | ||
| continue | ||
| if installed_version == expected_version: |
There was a problem hiding this comment.
Does this means that we force the contributor to use the exact same version we pin?
| additional_dependencies: ['promptfoo@0.121.17', '@anthropic-ai/claude-agent-sdk@0.3.185'] | ||
| additional_dependencies: | ||
| - 'promptfoo@0.121.17' | ||
| - '@anthropic-ai/claude-agent-sdk@0.3.185' | ||
| - '@openai/codex-sdk@0.144.6' |
There was a problem hiding this comment.
Or does it means we will use the Claude / Codex install by prek? Would it be possible to relex as to use system wise CLI first then fallback to prek installed?
| entry: ./dev/skill-evals/eval.py | ||
| language: node | ||
| language_version: '22.22.0' | ||
| additional_dependencies: ['promptfoo@0.121.17', '@anthropic-ai/claude-agent-sdk@0.3.185'] |
There was a problem hiding this comment.
I think it's fine for promptfoo as most of the user won't have it but I think we can assume most of the contributor have either Claude or Codex. Since we will pull ~310–350 MB Codex CLI binary, including Claude-only users now.
| return {"id": "anthropic:claude-agent-sdk", "label": label, "config": config} | ||
|
|
||
|
|
||
| def build_codex_provider( |
There was a problem hiding this comment.
Not a real comment but a interesting finding by claude.
Fortunately, our next steps is shirking the AGENTS.md haha.
Codex silently truncates AGENTS.md at 32 KiB, and Airflow's AGENTS.md is already 35,417 bytes, so a Codex-runtime eval can measure nothing while recording proof it ran (CONFIRMED)
Codex's DEFAULT_PROJECT_DOC_MAX_BYTES is 32,768 and codex-rs prefix-truncates project docs silently; build_codex_provider sets cli_config only for history persistence and never raises project_doc_max_bytes, even though promptfoo spreads cli_config straight into Codex config overrides. Since arms differ only in AGENTS.md, any guidance edit landing past the 32,768-byte boundary (the file's last ~2.6 KB, where new guidance is typically appended) produces byte-identical visible guidance in both arms — the exact "eval silently measures nothing" failure mode check_claude_md_symlink guards against on the Claude side, and the hash is still recorded as proof. Fix is one line: add project_doc_max_bytes to cli_config.
| SDK_PACKAGES = { | ||
| "claude": ("@anthropic-ai/claude-agent-sdk", "0.3.185"), | ||
| "codex": ("@openai/codex-sdk", "0.144.6"), | ||
| } |
There was a problem hiding this comment.
Or another direction, we could define the lower bound for the CLI version. (Thought I have no idea how should we pin the version at this moment).
2 participants
cc: @jason810496
Follow up on #69308 by adding Codex as an alternative runtime for evaluating AGENTS.md and skills through promptfoo's built-in Codex SDK provider.
Run the evaluation with Codex:
The evaluation completed all six cases with no provider errors. Four behavioral assertions passed and two existing ambiguous newsfragment cases failed.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.