Add Trustabl Agent Scanner to CI

[Joshua - Trustabl (joshua-trustabl)]

We came across your repo and we like that you've created a framework for evaluating AI agents and LLMs in a traceable and adaptable way. We scanned the repo, and noticed agent runtime reliability findings that might be worth reviewing.

1. [MEDIUM] Project uses default OpenAI tracing
   What it means: The project uses the OpenAI Agents SDK with default tracing enabled.

2. [MEDIUM] LlmAgent has no description
   File: examples/phoenix_auto_trace/travel_google_adk.py, The llm_agent instance lacks a description. It matters because Google ADK routes delegation between agents using the description= field on each LlmAgent.

3. [MEDIUM] Agent has no safety_settings
   File: examples/phoenix_auto_trace/travel_google_adk.py, The llm_agent instance lacks safety_settings. It matters because Google ADK agents use Gemini models whose configurable content filters default to OFF when safety_settings is not provided.

Recommendations are based on our understanding of agent runtime reliability, some findings may be intentional. Please let us know if this was intentional or if our findings are helpful so we can improve the accuracy of the scanner.

Best,
Trustabl.ai
Open-source AI agent reliability scanner (runs locally, GitHub Action)

[Chang Liu (changliu2)]

Thanks for the report — the individual findings are worth a look and we'll triage them separately. However, I can't merge this workflow as written; there are two blocking problems and a few hardening gaps.

Blocking:

1. Unpinned third-party action + unpinned scanner binary. trustabl/trustabl-action@v0 is a mutable lightweight tag (currently 57d4363, i.e. v0.3.1, while v0.4.1 exists) that the maintainer can repoint at any commit. Worse, pinning the action alone is insufficient: action.yml defaults version: latest, so at runtime the action resolves trustabl/trustabl's latest GitHub release and downloads and executes a prebuilt binary in our runner. The SHA256 check in install.ts verifies against checksums.txt published in the same release, so it guards against transport tampering but gives us no protection if the release itself is malicious or the account is compromised. This is arbitrary code execution in our CI on a repo that is ~2.5 months old with a single maintaining org. Our own scorecard.yml pins third-party actions by full commit SHA — this workflow regresses that posture and our OpenSSF Scorecard Pinned-Dependencies score.

2. This will fail CI red on every PR and every push to main, starting immediately. The PR description presents this as advisory, and the defaults (severity-threshold: none, risk-score-threshold: 0) look non-gating — but src/gate.ts fails the job unconditionally when the scanner's native exit code is 1, which the action's own comment documents as "medium+ or --strict". Your PR body lists three MEDIUM findings in this repo today, two of them in examples/. So this job goes red on merge and stays red, blocking unrelated work. If the intent is advisory, it needs continue-on-error: true and must not be added to required checks.

Also needed before merge: job-scoped rather than workflow-scoped write permissions, persist-credentials: false on checkout (matching scorecard.yml), and a timeout-minutes. Nice-to-have: exclude examples/ — that directory is 20+ intentionally minimal, single-SDK demo scripts, and "no description=" / "no safety_settings" on a 15-line demo is exactly the false-positive class that trains reviewers to ignore the check.

Happy to reconsider with the action pinned to a full SHA, version: pinned to an exact engine release, and the job explicitly advisory.

Inline notes

.github/workflows/trustabl.yml:17 — Critical — unpinned third-party action and unpinned runtime binary.

      - uses: trustabl/trustabl-action@v0

@v0 is a mutable lightweight tag. Additionally action.yml defaults version: latest, so the action downloads and executes trustabl/trustabl's newest release binary at runtime — pinning the action SHA alone does not close this.
Suggested fix:

      - uses: trustabl/trustabl-action@973f666d20b5fbb2e6a4511bd3846e965a08c28b # v0.4.1
        with:
          version: v0.1.6   # pin the engine; do not use 'latest'

.github/workflows/trustabl.yml:13-17 — High — job fails on every run despite non-gating defaults. src/gate.ts fails the job when the engine's native exit is 1 (medium+ findings), regardless of severity-threshold/risk-score-threshold. Three MEDIUM findings already exist per the PR description.
Suggested fix: add continue-on-error: true to the scan job (or set an explicit gating policy and accept red CI), and do not add this to branch-protection required checks.

.github/workflows/trustabl.yml:7-10 — Medium — write permissions granted workflow-wide instead of job-scoped. security-events: write and pull-requests: write at the top level are inherited by any job added to this file later.
Suggested fix: set permissions: contents: read at workflow level and move security-events: write / pull-requests: write under jobs.scan.permissions.

.github/workflows/trustabl.yml:16 — Medium — checkout leaves the GITHUB_TOKEN in .git/config. A third-party action runs in the same job and same workspace. scorecard.yml already sets persist-credentials: false for exactly this reason.
Suggested fix:

      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
        with:
          persist-credentials: false

.github/workflows/trustabl.yml:12-14 — Low — no job timeout / no concurrency cancellation. Suggested fix: add timeout-minutes: 15 to the job and a concurrency: group keyed on github.ref with cancel-in-progress: true.

.github/workflows/trustabl.yml:2-5 — Low — scans examples/, guaranteed false-positive noise. Suggested fix: add paths-ignore: ['examples/**', 'docs/**', '**/*.md'], or restrict the scan target: to the package source.

Must fix before merge

1. Pin trustabl/trustabl-action to a full commit SHA and pin the version: input to an exact engine release — unpinned action + version: latest = arbitrary code execution in CI from a 2.5-month-old single-maintainer project.
2. Make the job explicitly advisory (continue-on-error: true) — it fails on medium+ findings today regardless of the non-gating-looking defaults, and this repo already has three.
3. Move security-events: write / pull-requests: write to the job level; keep top-level contents: read.
4. Add persist-credentials: false to actions/checkout (matches scorecard.yml).

Nice to have

• timeout-minutes on the job; concurrency group with cancel-in-progress.
• paths-ignore for examples/, docs/, **/*.md to suppress demo-code false positives.
• Pin actions/checkout@v4 to a SHA for consistency with scorecard.yml.

Not an issue (checked, clean)

• Trigger is pull_request, not pull_request_target — no fork-PR secret exposure.
• No secrets/API keys referenced; enrich/llm-key are opt-in and unused here.
• The download path does verify SHA256 — the gap is release-level trust, not transport.