Skip to content

fix(claude-sdk): workspace skills not discovered by Claude Agent SDK provider #876

@christso

Description

@christso

Problem

The claude-sdk provider does not discover workspace-level skills from .claude/skills/. When running skill-trigger evals with provider: claude-sdk, positive test cases always fail because the agent doesn't know the skill exists.

Evidence

Running multi-provider-skill-trigger.EVAL.yaml (from examples/features/agent-skills-evals/) with --target claude-sdk:

Test claude-sdk result
should-trigger-direct-request (should_trigger: true) FAIL — skill not found
should-trigger-casual-phrasing (should_trigger: true) FAIL — skill not found
should-not-trigger-unrelated (should_trigger: false) PASS
should-not-trigger-near-miss (should_trigger: false) PASS

The same eval passes all positive cases with claude-cli (claude -p subprocess).

claude-sdk session init — NO workspace skills or plugins:

"skills": ["keybindings-help", "debug", "claude-developer-platform"],
"plugins": []

claude-cli session init — workspace skills discovered automatically:

"skills": ["keybindings-help", "debug", "claude-developer-platform", "csv-analyzer", ...],
"plugins": [{"name": "frontend-design", ...}, ...]

Root Cause

claude -p CLI auto-discovers .claude/skills/ from the workspace cwd. The Claude Agent SDK's query() does not — it needs explicit configuration.

The AgentV ClaudeSdkProvider (in packages/core/src/evaluation/providers/claude-sdk.ts) passes only: model, cwd, systemPrompt, maxTurns, maxBudgetUsd, env. It does NOT pass skills or plugin config.

SDK Capabilities (already available)

The SDK (@anthropic-ai/claude-agent-sdk v0.2.49, types in sdk.d.ts) supports:

  • Options.plugins?: SdkPluginConfig[] — load local plugins: [{ type: 'local', path: './my-plugin' }]
    • BUT .claude/skills/ are bare skill files (SKILL.md), not plugins with plugin.json manifests
  • AgentDefinition.skills?: string[] — preload skill names into agent context
  • Options.allowedTools?: string[] / Options.disallowedTools?: string[] — tool control

Files to Modify

File Change
packages/core/src/evaluation/providers/claude-sdk.ts Pass skills/plugins to SDK query() options
packages/core/src/evaluation/providers/targets.ts Extend ClaudeResolvedConfig if new target fields needed
examples/features/.agentv/targets.yaml Add claude-sdk target definition

Design Question

.claude/skills/ directories are bare SKILL.md files, not structured plugins. The SDK's plugins option expects { type: 'local', path } pointing to a plugin with plugin.json. Options to bridge this:

  1. Auto-detect and inject via system prompt — read SKILL.md files from workspace .claude/skills/ and append to systemPrompt. Pragmatic, works today, no SDK changes needed.
  2. Wrap as temporary plugin — create an ephemeral plugin manifest pointing to the workspace skills. More complex but uses the SDK's native plugin system.
  3. Check if SDK auto-discovers from cwd — verify whether setting cwd to the workspace root triggers built-in skill discovery in newer SDK versions. If so, this might be a version issue.

Acceptance Criteria

  • multi-provider-skill-trigger.EVAL.yaml passes all 4 tests with --target claude-sdk
  • Workspace .claude/skills/ are available to the SDK session
  • No manual skill configuration required in targets.yaml for workspace skills

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions