-
Notifications
You must be signed in to change notification settings - Fork 0
fix(claude-sdk): workspace skills not discovered by Claude Agent SDK provider #876
Description
Problem
The claude-sdk provider does not discover workspace-level skills from .claude/skills/. When running skill-trigger evals with provider: claude-sdk, positive test cases always fail because the agent doesn't know the skill exists.
Evidence
Running multi-provider-skill-trigger.EVAL.yaml (from examples/features/agent-skills-evals/) with --target claude-sdk:
| Test | claude-sdk result |
|---|---|
| should-trigger-direct-request (should_trigger: true) | FAIL — skill not found |
| should-trigger-casual-phrasing (should_trigger: true) | FAIL — skill not found |
| should-not-trigger-unrelated (should_trigger: false) | PASS |
| should-not-trigger-near-miss (should_trigger: false) | PASS |
The same eval passes all positive cases with claude-cli (claude -p subprocess).
claude-sdk session init — NO workspace skills or plugins:
"skills": ["keybindings-help", "debug", "claude-developer-platform"],
"plugins": []claude-cli session init — workspace skills discovered automatically:
"skills": ["keybindings-help", "debug", "claude-developer-platform", "csv-analyzer", ...],
"plugins": [{"name": "frontend-design", ...}, ...]Root Cause
claude -p CLI auto-discovers .claude/skills/ from the workspace cwd. The Claude Agent SDK's query() does not — it needs explicit configuration.
The AgentV ClaudeSdkProvider (in packages/core/src/evaluation/providers/claude-sdk.ts) passes only: model, cwd, systemPrompt, maxTurns, maxBudgetUsd, env. It does NOT pass skills or plugin config.
SDK Capabilities (already available)
The SDK (@anthropic-ai/claude-agent-sdk v0.2.49, types in sdk.d.ts) supports:
Options.plugins?: SdkPluginConfig[]— load local plugins:[{ type: 'local', path: './my-plugin' }]- BUT
.claude/skills/are bare skill files (SKILL.md), not plugins withplugin.jsonmanifests
- BUT
AgentDefinition.skills?: string[]— preload skill names into agent contextOptions.allowedTools?: string[]/Options.disallowedTools?: string[]— tool control
Files to Modify
| File | Change |
|---|---|
packages/core/src/evaluation/providers/claude-sdk.ts |
Pass skills/plugins to SDK query() options |
packages/core/src/evaluation/providers/targets.ts |
Extend ClaudeResolvedConfig if new target fields needed |
examples/features/.agentv/targets.yaml |
Add claude-sdk target definition |
Design Question
.claude/skills/ directories are bare SKILL.md files, not structured plugins. The SDK's plugins option expects { type: 'local', path } pointing to a plugin with plugin.json. Options to bridge this:
- Auto-detect and inject via system prompt — read
SKILL.mdfiles from workspace.claude/skills/and append tosystemPrompt. Pragmatic, works today, no SDK changes needed. - Wrap as temporary plugin — create an ephemeral plugin manifest pointing to the workspace skills. More complex but uses the SDK's native plugin system.
- Check if SDK auto-discovers from cwd — verify whether setting
cwdto the workspace root triggers built-in skill discovery in newer SDK versions. If so, this might be a version issue.
Acceptance Criteria
-
multi-provider-skill-trigger.EVAL.yamlpasses all 4 tests with--target claude-sdk - Workspace
.claude/skills/are available to the SDK session - No manual skill configuration required in targets.yaml for workspace skills
Metadata
Metadata
Assignees
Labels
Type
Projects
Status