From b7523d407e80991d94439353af70ecfe2d0c13c9 Mon Sep 17 00:00:00 2001 From: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com> Date: Mon, 18 May 2026 18:27:54 +1000 Subject: [PATCH 1/3] chore: add gap-analysis maintainer skill Adds a Claude Code skill at .claude/skills/gap-analysis/ that audits each provider adapter against upstream docs and produces a dated markdown report under .agent/gap-analysis/. Triggered via /gap-analysis with scopes , feature , models, or --all. The skill is read-only: it never edits feature-support.ts or any model-meta.ts directly. Also gitignores .agent/gap-analysis/ and .agent/triage/ since those are local agent scratch output. Co-Authored-By: Claude Opus 4.7 (1M context) --- .claude/skills/gap-analysis/SKILL.md | 93 ++++++++++ .../references/audit-checklist.md | 171 ++++++++++++++++++ .../references/provider-doc-urls.md | 92 ++++++++++ .../references/report-template.md | 123 +++++++++++++ .gitignore | 4 + 5 files changed, 483 insertions(+) create mode 100644 .claude/skills/gap-analysis/SKILL.md create mode 100644 .claude/skills/gap-analysis/references/audit-checklist.md create mode 100644 .claude/skills/gap-analysis/references/provider-doc-urls.md create mode 100644 .claude/skills/gap-analysis/references/report-template.md diff --git a/.claude/skills/gap-analysis/SKILL.md b/.claude/skills/gap-analysis/SKILL.md new file mode 100644 index 000000000..4a1a6e8d5 --- /dev/null +++ b/.claude/skills/gap-analysis/SKILL.md @@ -0,0 +1,93 @@ +--- +name: gap-analysis +description: > + Audit TanStack AI provider adapters for feature parity gaps and outdated + model lists. Triggered as /gap-analysis |models|--all>. + Produces a dated markdown report under .agent/gap-analysis/. Maintainer + tool — does not edit feature-support.ts or model-meta.ts directly. +--- + +# Gap Analysis — TanStack AI adapter audit + +You are auditing TanStack AI's provider adapters against each provider's +upstream documentation. This is a **maintainer** tool. Your only output is a +markdown report under `.agent/gap-analysis/`. **Do not edit source files.** + +## Invocation + +| Args | Scope | +| --------------------------------- | ------------------------------------------------------ | +| `` (e.g. `openai`) | One provider — all four audit dimensions. | +| `feature ` (e.g. `tts`) | One feature row of the matrix across all providers. | +| `models` | New-model diff for every provider. | +| `--all` | Full sweep (fan out subagents, one per provider). | +| _(none)_ | Ask the user which scope via AskUserQuestion. | + +## Workflow + +1. **Parse scope.** If missing, AskUserQuestion with the four options above. +2. **Load the truth files**, then read the per-scope inputs you need: + - Matrix: `testing/e2e/src/lib/feature-support.ts` + - Types: `testing/e2e/src/lib/types.ts` (Provider + Feature unions, ALL_PROVIDERS, ALL_FEATURES) + - Adapter index: `packages/typescript/ai-/src/index.ts` + - Model meta: `packages/typescript/ai-/src/model-meta.ts` + - Core types: `packages/typescript/ai/src/types.ts` (Modality, ContentPart, ToolCall) +3. **Research upstream.** Use WebFetch against the curated URLs in + [references/provider-doc-urls.md](references/provider-doc-urls.md). When a + doc page has moved, fall back to WebSearch. For SDK API surface details + use the `context7` MCP server (`mcp__plugin_context7_context7__resolve-library-id` + then `mcp__plugin_context7_context7__query-docs`). +4. **Walk the audit dimensions** in [references/audit-checklist.md](references/audit-checklist.md): + 1. New models + 2. Cross-adapter feature parity + 3. Untracked features + 4. Capability-flag drift +5. **Fan out** for `--all`: launch one `Explore` subagent per provider, max 3 + in parallel. Each subagent returns the four-dimension findings for its + provider; you synthesise into the combined report. +6. **Write the report** to `.agent/gap-analysis/YYYY-MM-DD-.md` using + [references/report-template.md](references/report-template.md). Date is + today's ISO date. `` is `openai` / `feature-tts` / `models` / `all`. +7. **Print the report path and a 5-line summary** to the user. + +## Critical rules + +1. **Never edit `feature-support.ts` or any `model-meta.ts`.** The report is + read-only — the maintainer applies changes. +2. **Always reference line numbers** when citing exclusions (e.g., + `feature-support.ts:57`) so the maintainer can jump to them. +3. **Distinguish three gap classes** in the report: + - **Real gap** — upstream supports it, TanStack AI doesn't, no exclusion comment. + - **Tested gap** — TanStack AI doesn't list it but there's an exclusion + comment in `feature-support.ts` (e.g., aimock format limitation). Not + actionable code-wise; surface in "Out-of-scope" section. + - **Stale capability flag** — `model-meta.ts` declares a capability the + model no longer has, or omits one it now has. +4. **Cite sources.** Every claim "upstream supports X" must link the upstream + doc page you read. No claims from training data. +5. **Use today's date** from the system context (currentDate). Never invent. +6. **Quote the relevant snippet** from `feature-support.ts` when flagging a + parity gap, so the report is self-contained. + +## Known providers + +`openai`, `anthropic`, `gemini`, `ollama`, `grok`, `groq`, `openrouter`, `fal` +(media-only), `elevenlabs` (TTS-only). The feature matrix tracks the first +seven; `fal` and `elevenlabs` only appear in model/media audits. + +## Known features (22) + +`chat`, `one-shot-text`, `reasoning`, `multi-turn`, `tool-calling`, +`parallel-tool-calls`, `tool-approval`, `text-tool-text`, `structured-output`, +`structured-output-stream`, `agentic-structured`, `multimodal-image`, +`multimodal-structured`, `summarize`, `summarize-stream`, `image-gen`, `tts`, +`transcription`, `video-gen`. + +## Verification before finishing + +Before printing the summary: +- Report file exists and is non-empty. +- `git status` shows only new files under `.agent/gap-analysis/` — nothing + under `packages/` or `testing/` should have been modified. Run `git status` + and confirm. +- Every "real gap" entry has an upstream doc URL. diff --git a/.claude/skills/gap-analysis/references/audit-checklist.md b/.claude/skills/gap-analysis/references/audit-checklist.md new file mode 100644 index 000000000..c1d79d935 --- /dev/null +++ b/.claude/skills/gap-analysis/references/audit-checklist.md @@ -0,0 +1,171 @@ +# Audit checklist + +Walk this checklist for each scope. Each dimension produces a section of the +final report. Skip dimensions that don't apply to the scope (e.g., +`feature ` only needs dimension 2). + +--- + +## 1. New models + +**Input:** `packages/typescript/ai-/src/model-meta.ts` +**Upstream:** the provider's models page (see provider-doc-urls.md). + +Steps: +1. Read the local `model-meta.ts` and extract the model-id constants + (e.g. `OPENAI_CHAT_MODELS`, `ANTHROPIC_MODELS`, `GEMINI_MODELS`, etc.). + The constants are usually exported `as const` records keyed by model id. +2. WebFetch the upstream models page. Extract the canonical model ids. +3. **Diff**: produce two lists — + - Missing locally (upstream lists, we don't) + - Stale locally (we list, upstream has deprecated or removed) +4. For each missing model, capture from upstream: + - Context window + - Max output tokens + - Knowledge cutoff date (if listed) + - Pricing (input / cached input / output per 1M tokens) + - Capabilities array (text / image / audio / video in/out, tools, reasoning) +5. Cross-reference with the existing `ModelMeta` shape in that provider's + `model-meta.ts` so the report's suggested-addition block matches the + shape the maintainer can paste in. + +**Priority rubric:** +- New flagship / GA model → **high** +- New preview / experimental model → **medium** +- Deprecated upstream but still in local → **medium** (action: deprecate) +- Minor variant (e.g., size suffix) → **low** + +--- + +## 2. Cross-adapter feature parity + +**Input:** `testing/e2e/src/lib/feature-support.ts` (the matrix). +**Upstream:** each provider's API reference / capabilities page. + +Steps: +1. Read the matrix. For each `feature ∈ ALL_FEATURES`: + - List `providers_with = matrix[feature]`. + - Compute `providers_without = ALL_PROVIDERS - providers_with`. +2. For each `(feature, provider)` in `providers_without`: + - Check if `feature-support.ts` has an inline comment immediately above + the feature's `Set` that excludes this provider (e.g., `// Gemini + excluded: …`). If yes → **tested gap**, surface in "Out-of-scope" with + line-number citation. + - Otherwise, WebFetch the provider's API reference and search for the + capability. Map TanStack feature → upstream capability using the table + below. If upstream supports it → **real gap**, high or medium priority. + If upstream does not support it → not a gap, omit. +3. Also flag any provider in `providers_with` that the upstream docs say + has **removed** support (rare, but possible after deprecation). + +### Feature → upstream capability map + +| TanStack feature | Upstream capability to look for | +| -------------------------- | -------------------------------------------------------- | +| `chat` | Any chat/messages/completions endpoint | +| `one-shot-text` | Non-streaming completion | +| `reasoning` | Reasoning / thinking / chain-of-thought tokens | +| `multi-turn` | Conversation / message history support | +| `tool-calling` | Function calling / tool use | +| `parallel-tool-calls` | Multiple tool calls in one turn (parallel) | +| `tool-approval` | Pause-before-execute / user-confirm tool hooks | +| `text-tool-text` | Tool call interleaved with text in the same turn | +| `structured-output` | JSON schema / response format / structured outputs | +| `structured-output-stream` | Streaming JSON schema responses | +| `agentic-structured` | Structured output + tool calling combined | +| `multimodal-image` | Image input (vision) | +| `multimodal-structured` | Image input + structured output | +| `summarize` | Any non-chat completion useful for summarization | +| `summarize-stream` | Streaming summarize | +| `image-gen` | Image generation endpoint | +| `tts` | Text-to-speech endpoint | +| `transcription` | Speech-to-text endpoint | +| `video-gen` | Video generation endpoint | + +**Priority rubric:** +- Real gap on a flagship feature (`tool-calling`, `structured-output`, + `multimodal-image`) → **high** +- Real gap on a media feature (`image-gen`, `tts`, `transcription`, + `video-gen`) → **medium** +- Tested gap (documented exclusion) → **out-of-scope** + +--- + +## 3. Untracked features + +**Input:** the union of `ALL_FEATURES` plus the four media adapters +(`image`, `tts`, `transcription`, `video`). +**Upstream:** provider's API reference top-level navigation. + +Steps: +1. Read the upstream API reference's top-level sections. +2. For each section, ask: does TanStack AI have a concept for this? Examples + of "novel" upstream capabilities to look out for: + - Prompt caching (Anthropic, OpenAI) + - Batch API + - Files API / assistants API + - Fine-tuning + - Moderation + - Real-time / live API (already partially supported on openai, grok) + - Embeddings (already supported on openai; gap if elsewhere) + - Computer use / browser tools + - Memory / sessions +3. Cross-check by grep'ing the adapter source: + - `Grep "" packages/typescript/ai-/src/` + - If zero matches → likely untracked. +4. For each untracked feature, note: + - Upstream URL + - One-line summary of what it does + - Whether other TanStack AI adapters expose anything analogous + +**Priority rubric:** +- Capability that meaningfully changes app architecture (caching, batch, + realtime) → **high** +- Capability that's a sibling of existing features → **medium** +- Provider-specific niche → **low** + +--- + +## 4. Capability flag drift + +**Input:** every provider's `model-meta.ts` `supports.{input,output,endpoints,features,tools}` arrays. +**Upstream:** model-level capability tables on the provider's models page. + +Steps: +1. Read every provider's `model-meta.ts`. For each model entry, list its + `supports` arrays. +2. Build a cross-provider table: for the **chat** model class, which keys + are commonly declared (e.g., `endpoints: ['chat']`)? Flag outliers. +3. For each provider's flagship model, WebFetch the upstream model card and + check whether the local `supports` arrays match. Common drifts: + - `supports.input` missing `image` after vision was added + - `supports.features` missing `structured_outputs` after GA + - `supports.tools` missing a newly launched provider tool + - `context_window` stale (upstream raised the cap) + - `knowledge_cutoff` stale +4. Pricing drift is **out of scope** for an audit (changes too often) — + note in the report only if upstream announcement is < 30 days old. + +**Priority rubric:** +- Missing capability flag for an in-use feature → **high** (causes type + errors or feature unavailable) +- Stale context window / output tokens → **medium** +- Stale knowledge cutoff label → **low** + +--- + +## Subagent dispatch (for `--all` scope) + +When fan-out is needed, launch one `Explore` subagent per provider with a +prompt of this shape: + +> Audit the `` adapter at `packages/typescript/ai-/` +> against upstream docs at the URLs in +> `.claude/skills/gap-analysis/references/provider-doc-urls.md`. Walk +> dimensions 1, 3, and 4 from `audit-checklist.md`. Skip dimension 2 (the +> orchestrator handles cross-provider parity centrally). Return findings as +> markdown sections matching the report template — High / Medium / Low / +> Out-of-scope — with upstream URLs cited for every claim. + +Run at most 3 in parallel. Aggregate their returned markdown into the +combined report. diff --git a/.claude/skills/gap-analysis/references/provider-doc-urls.md b/.claude/skills/gap-analysis/references/provider-doc-urls.md new file mode 100644 index 000000000..2b67eb647 --- /dev/null +++ b/.claude/skills/gap-analysis/references/provider-doc-urls.md @@ -0,0 +1,92 @@ +# Provider documentation URLs + +Curated entry points for each provider's docs. Use these as WebFetch targets. +If a page 404s or has moved, fall back to WebSearch with the provider name +plus the section keyword (e.g., "openai models page 2026"). + +When a section needs SDK-level API detail (e.g., "what parameters does the +`messages.create` call accept?"), prefer the `context7` MCP server over +WebFetch — call `resolve-library-id` with the SDK npm name, then `query-docs`. + +--- + +## openai + +- Models: https://platform.openai.com/docs/models +- API reference: https://platform.openai.com/docs/api-reference +- Changelog: https://platform.openai.com/docs/changelog +- Cookbook (capability examples): https://cookbook.openai.com/ +- npm SDK: https://www.npmjs.com/package/openai +- context7 lib id hint: `openai/openai-node` + +## anthropic + +- Models: https://docs.anthropic.com/en/docs/about-claude/models +- API reference: https://docs.anthropic.com/en/api/getting-started +- Release notes: https://docs.anthropic.com/en/release-notes/api +- Tool-use docs: https://docs.anthropic.com/en/docs/build-with-claude/tool-use +- Prompt caching: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching +- npm SDK: https://www.npmjs.com/package/@anthropic-ai/sdk +- context7 lib id hint: `anthropics/anthropic-sdk-typescript` + +## gemini (Google) + +- Models: https://ai.google.dev/gemini-api/docs/models +- API reference: https://ai.google.dev/api +- What's new: https://ai.google.dev/gemini-api/docs/changelog +- Imagen (image-gen): https://ai.google.dev/gemini-api/docs/imagen +- Lyria (audio-gen): https://ai.google.dev/gemini-api/docs/music-generation +- npm SDK: https://www.npmjs.com/package/@google/genai +- context7 lib id hint: `googleapis/js-genai` + +## ollama + +- Models library: https://ollama.com/library +- API docs: https://github.com/ollama/ollama/blob/main/docs/api.md +- Tool calling: https://ollama.com/blog/tool-support +- npm SDK: https://www.npmjs.com/package/ollama +- context7 lib id hint: `ollama/ollama-js` + +## grok (xAI) + +- Models: https://docs.x.ai/docs/models +- API reference: https://docs.x.ai/docs/api-reference +- Capabilities: https://docs.x.ai/docs/guides +- Changelog: https://docs.x.ai/docs/release-notes +- (Uses OpenAI-compatible HTTP API; SDK is the openai package.) + +## groq + +- Models: https://console.groq.com/docs/models +- API reference: https://console.groq.com/docs/api-reference +- Tool use: https://console.groq.com/docs/tool-use +- npm SDK: https://www.npmjs.com/package/groq-sdk +- context7 lib id hint: `groq/groq-typescript` + +## openrouter + +- Models: https://openrouter.ai/models +- API docs: https://openrouter.ai/docs/quickstart +- Provider routing: https://openrouter.ai/docs/features/provider-routing +- (Proxies many providers; uses OpenAI-compatible API.) + +## fal (media-only) + +- Models catalog: https://fal.ai/models +- API docs: https://docs.fal.ai/ +- npm SDK: https://www.npmjs.com/package/@fal-ai/client +- context7 lib id hint: `fal-ai/fal-js` + +## elevenlabs (TTS-only) + +- Voices / models: https://elevenlabs.io/docs/api-reference/voices +- TTS API: https://elevenlabs.io/docs/api-reference/text-to-speech +- npm SDK: https://www.npmjs.com/package/@elevenlabs/elevenlabs-js + +--- + +## Maintenance + +If you find a URL has permanently moved while running an audit, update this +file in the same report turn — the maintainer will see the change in +`git status` alongside the new report. diff --git a/.claude/skills/gap-analysis/references/report-template.md b/.claude/skills/gap-analysis/references/report-template.md new file mode 100644 index 000000000..788653c61 --- /dev/null +++ b/.claude/skills/gap-analysis/references/report-template.md @@ -0,0 +1,123 @@ +# Report template + +Fill this template and write the result to +`.agent/gap-analysis/-.md`. + +`` is one of: +- `` — e.g. `openai` +- `feature-` — e.g. `feature-tts` +- `models` +- `all` + +Replace every `{{placeholder}}`. Drop sections that have zero entries +(except **Summary**, which is always present). + +--- + +```markdown +# Gap Analysis — {{scope-human}} — {{YYYY-MM-DD}} + +> Source-of-truth matrix: `testing/e2e/src/lib/feature-support.ts` +> Local model lists: `packages/typescript/ai-*/src/model-meta.ts` +> Generated by `.claude/skills/gap-analysis` — review before applying. + +## Summary + +- **{{N_high}}** high-priority gaps +- **{{N_medium}}** medium-priority gaps +- **{{N_low}}** low-priority items +- **{{N_outofscope}}** documented exclusions (no action) + +{{one-paragraph headline: the single most important finding}} + +--- + +## High priority — real gaps + +> Upstream supports it; TanStack AI does not; no exclusion comment. + +- **[{{provider}}] {{feature-or-capability}}** + - Upstream: [{{doc-title}}]({{doc-url}}) + - Current state: {{quote relevant matrix line or model-meta excerpt with file:line}} + - Suggested change: {{file path and one-line patch description}} + - Effort: {{S / M / L}} + +{{repeat block per gap}} + +--- + +## Medium priority — new / stale models + +- **[{{provider}}]** missing models: `{{model-id}}` ({{context-window}}, {{kn-cutoff}}), `{{model-id}}` ({{…}}) + - Upstream: [{{provider models page}}]({{url}}) + - Suggested addition to `packages/typescript/ai-{{provider}}/src/model-meta.ts`: + ```ts + {{paste-ready ModelMeta entry skeleton}} + ``` + +- **[{{provider}}]** deprecated upstream, still listed locally: `{{model-id}}` — remove or mark deprecated. + +{{repeat per provider that has model drift}} + +--- + +## Medium priority — untracked features + +- **[{{provider}}] {{capability}}** + - Upstream: [{{doc-title}}]({{doc-url}}) + - What it does: {{one sentence}} + - Analogous TanStack feature: {{name or "none"}} + - Suggested home: {{adapter sub-package / new sub-adapter / new feature in matrix}} + +--- + +## Low priority — capability flag drift + +- **[{{provider}} / {{model-id}}]** `supports.{{key}}` is `{{current}}`, upstream says `{{should-be}}` — see {{model-meta path:line}}. + +--- + +## Out-of-scope — documented exclusions + +> Listed for completeness; no action required. Each links to the comment +> in `feature-support.ts` that explains why. + +- **[{{provider}} × {{feature}}]** — {{reason from comment}} (`testing/e2e/src/lib/feature-support.ts:{{line}}`) + +--- + +## Suggested follow-ups + +Ordered list of concrete files to touch, one per follow-up: + +1. **`{{path}}`** — {{one-line rationale}} +2. **`{{path}}`** — {{one-line rationale}} + +--- + +## Audit metadata + +- Scope: `{{scope-raw}}` +- Date: {{YYYY-MM-DD}} +- Providers audited: {{comma-separated}} +- Features audited: {{comma-separated or "all 22"}} +- Upstream pages fetched: {{count}} +- Subagents launched: {{count or "none"}} +- Time: {{wall-clock}} (optional) +``` + +--- + +## Conventions inside the report + +- **Always cite line numbers** when quoting from `feature-support.ts` or + any `model-meta.ts` — the maintainer should be able to ⌘-click straight + to them. +- **Quote the upstream source URL** for every "real gap" claim. No claim + from training data. +- **Paste-ready code blocks** for suggested `ModelMeta` additions — match + the existing shape in that provider's `model-meta.ts` exactly. +- **Drop empty sections.** If there are no high-priority gaps, omit that + header entirely (except Summary, which always renders). +- **Date format:** ISO `YYYY-MM-DD` taken from the system `currentDate`. No + invented dates. diff --git a/.gitignore b/.gitignore index 6c359c884..c05210e1b 100644 --- a/.gitignore +++ b/.gitignore @@ -73,3 +73,7 @@ docs/superpowers .claude/settings.local.json .claude/worktrees/* solo.yml + +# Agent scratch output (gap-analysis reports, triage notes — generated locally) +.agent/gap-analysis/ +.agent/triage/ From 679e86c8122e30f4ac0f9c613072e86070707114 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Mon, 18 May 2026 08:29:30 +0000 Subject: [PATCH 2/3] ci: apply automated fixes --- .claude/skills/gap-analysis/SKILL.md | 15 +++--- .../references/audit-checklist.md | 52 +++++++++++-------- .../references/report-template.md | 5 +- 3 files changed, 41 insertions(+), 31 deletions(-) diff --git a/.claude/skills/gap-analysis/SKILL.md b/.claude/skills/gap-analysis/SKILL.md index 4a1a6e8d5..0e0242009 100644 --- a/.claude/skills/gap-analysis/SKILL.md +++ b/.claude/skills/gap-analysis/SKILL.md @@ -15,13 +15,13 @@ markdown report under `.agent/gap-analysis/`. **Do not edit source files.** ## Invocation -| Args | Scope | -| --------------------------------- | ------------------------------------------------------ | -| `` (e.g. `openai`) | One provider — all four audit dimensions. | -| `feature ` (e.g. `tts`) | One feature row of the matrix across all providers. | -| `models` | New-model diff for every provider. | -| `--all` | Full sweep (fan out subagents, one per provider). | -| _(none)_ | Ask the user which scope via AskUserQuestion. | +| Args | Scope | +| -------------------------------- | --------------------------------------------------- | +| `` (e.g. `openai`) | One provider — all four audit dimensions. | +| `feature ` (e.g. `tts`) | One feature row of the matrix across all providers. | +| `models` | New-model diff for every provider. | +| `--all` | Full sweep (fan out subagents, one per provider). | +| _(none)_ | Ask the user which scope via AskUserQuestion. | ## Workflow @@ -86,6 +86,7 @@ seven; `fal` and `elevenlabs` only appear in model/media audits. ## Verification before finishing Before printing the summary: + - Report file exists and is non-empty. - `git status` shows only new files under `.agent/gap-analysis/` — nothing under `packages/` or `testing/` should have been modified. Run `git status` diff --git a/.claude/skills/gap-analysis/references/audit-checklist.md b/.claude/skills/gap-analysis/references/audit-checklist.md index c1d79d935..c9c6f02c7 100644 --- a/.claude/skills/gap-analysis/references/audit-checklist.md +++ b/.claude/skills/gap-analysis/references/audit-checklist.md @@ -12,6 +12,7 @@ final report. Skip dimensions that don't apply to the scope (e.g., **Upstream:** the provider's models page (see provider-doc-urls.md). Steps: + 1. Read the local `model-meta.ts` and extract the model-id constants (e.g. `OPENAI_CHAT_MODELS`, `ANTHROPIC_MODELS`, `GEMINI_MODELS`, etc.). The constants are usually exported `as const` records keyed by model id. @@ -30,6 +31,7 @@ Steps: shape the maintainer can paste in. **Priority rubric:** + - New flagship / GA model → **high** - New preview / experimental model → **medium** - Deprecated upstream but still in local → **medium** (action: deprecate) @@ -43,13 +45,14 @@ Steps: **Upstream:** each provider's API reference / capabilities page. Steps: + 1. Read the matrix. For each `feature ∈ ALL_FEATURES`: - List `providers_with = matrix[feature]`. - Compute `providers_without = ALL_PROVIDERS - providers_with`. 2. For each `(feature, provider)` in `providers_without`: - Check if `feature-support.ts` has an inline comment immediately above the feature's `Set` that excludes this provider (e.g., `// Gemini - excluded: …`). If yes → **tested gap**, surface in "Out-of-scope" with +excluded: …`). If yes → **tested gap**, surface in "Out-of-scope" with line-number citation. - Otherwise, WebFetch the provider's API reference and search for the capability. Map TanStack feature → upstream capability using the table @@ -60,29 +63,30 @@ Steps: ### Feature → upstream capability map -| TanStack feature | Upstream capability to look for | -| -------------------------- | -------------------------------------------------------- | -| `chat` | Any chat/messages/completions endpoint | -| `one-shot-text` | Non-streaming completion | -| `reasoning` | Reasoning / thinking / chain-of-thought tokens | -| `multi-turn` | Conversation / message history support | -| `tool-calling` | Function calling / tool use | -| `parallel-tool-calls` | Multiple tool calls in one turn (parallel) | -| `tool-approval` | Pause-before-execute / user-confirm tool hooks | -| `text-tool-text` | Tool call interleaved with text in the same turn | -| `structured-output` | JSON schema / response format / structured outputs | -| `structured-output-stream` | Streaming JSON schema responses | -| `agentic-structured` | Structured output + tool calling combined | -| `multimodal-image` | Image input (vision) | -| `multimodal-structured` | Image input + structured output | -| `summarize` | Any non-chat completion useful for summarization | -| `summarize-stream` | Streaming summarize | -| `image-gen` | Image generation endpoint | -| `tts` | Text-to-speech endpoint | -| `transcription` | Speech-to-text endpoint | -| `video-gen` | Video generation endpoint | +| TanStack feature | Upstream capability to look for | +| -------------------------- | -------------------------------------------------- | +| `chat` | Any chat/messages/completions endpoint | +| `one-shot-text` | Non-streaming completion | +| `reasoning` | Reasoning / thinking / chain-of-thought tokens | +| `multi-turn` | Conversation / message history support | +| `tool-calling` | Function calling / tool use | +| `parallel-tool-calls` | Multiple tool calls in one turn (parallel) | +| `tool-approval` | Pause-before-execute / user-confirm tool hooks | +| `text-tool-text` | Tool call interleaved with text in the same turn | +| `structured-output` | JSON schema / response format / structured outputs | +| `structured-output-stream` | Streaming JSON schema responses | +| `agentic-structured` | Structured output + tool calling combined | +| `multimodal-image` | Image input (vision) | +| `multimodal-structured` | Image input + structured output | +| `summarize` | Any non-chat completion useful for summarization | +| `summarize-stream` | Streaming summarize | +| `image-gen` | Image generation endpoint | +| `tts` | Text-to-speech endpoint | +| `transcription` | Speech-to-text endpoint | +| `video-gen` | Video generation endpoint | **Priority rubric:** + - Real gap on a flagship feature (`tool-calling`, `structured-output`, `multimodal-image`) → **high** - Real gap on a media feature (`image-gen`, `tts`, `transcription`, @@ -98,6 +102,7 @@ Steps: **Upstream:** provider's API reference top-level navigation. Steps: + 1. Read the upstream API reference's top-level sections. 2. For each section, ask: does TanStack AI have a concept for this? Examples of "novel" upstream capabilities to look out for: @@ -119,6 +124,7 @@ Steps: - Whether other TanStack AI adapters expose anything analogous **Priority rubric:** + - Capability that meaningfully changes app architecture (caching, batch, realtime) → **high** - Capability that's a sibling of existing features → **medium** @@ -132,6 +138,7 @@ Steps: **Upstream:** model-level capability tables on the provider's models page. Steps: + 1. Read every provider's `model-meta.ts`. For each model entry, list its `supports` arrays. 2. Build a cross-provider table: for the **chat** model class, which keys @@ -147,6 +154,7 @@ Steps: note in the report only if upstream announcement is < 30 days old. **Priority rubric:** + - Missing capability flag for an in-use feature → **high** (causes type errors or feature unavailable) - Stale context window / output tokens → **medium** diff --git a/.claude/skills/gap-analysis/references/report-template.md b/.claude/skills/gap-analysis/references/report-template.md index 788653c61..0b63b4682 100644 --- a/.claude/skills/gap-analysis/references/report-template.md +++ b/.claude/skills/gap-analysis/references/report-template.md @@ -4,6 +4,7 @@ Fill this template and write the result to `.agent/gap-analysis/-.md`. `` is one of: + - `` — e.g. `openai` - `feature-` — e.g. `feature-tts` - `models` @@ -14,7 +15,7 @@ Replace every `{{placeholder}}`. Drop sections that have zero entries --- -```markdown +````markdown # Gap Analysis — {{scope-human}} — {{YYYY-MM-DD}} > Source-of-truth matrix: `testing/e2e/src/lib/feature-support.ts` @@ -104,7 +105,7 @@ Ordered list of concrete files to touch, one per follow-up: - Upstream pages fetched: {{count}} - Subagents launched: {{count or "none"}} - Time: {{wall-clock}} (optional) -``` +```` --- From 3fb5adfc9ad7f82da9e7423c6418fa7864b1590f Mon Sep 17 00:00:00 2001 From: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com> Date: Mon, 18 May 2026 19:28:04 +1000 Subject: [PATCH 3/3] chore(gap-analysis): address coderabbit + add telemetry-parity dimension - Fix feature count drift (22 -> 19) and point at ALL_FEATURES as canonical. - Align media adapter names with the feature table (image-gen, video-gen). - Rewrite pricing-drift rule as an unambiguous conditional. - Defer URL-file updates to a follow-up PR to preserve the clean-tree rule. - Add audit dimension 5: telemetry / observability parity (usage tokens, cache/reasoning counts, request ids, logging asymmetry across adapters). Co-Authored-By: Claude Opus 4.7 (1M context) --- .claude/skills/gap-analysis/SKILL.md | 9 ++- .../references/audit-checklist.md | 71 +++++++++++++++++-- .../references/provider-doc-urls.md | 9 ++- .../references/report-template.md | 19 ++++- 4 files changed, 95 insertions(+), 13 deletions(-) diff --git a/.claude/skills/gap-analysis/SKILL.md b/.claude/skills/gap-analysis/SKILL.md index 0e0242009..bc5842693 100644 --- a/.claude/skills/gap-analysis/SKILL.md +++ b/.claude/skills/gap-analysis/SKILL.md @@ -42,8 +42,10 @@ markdown report under `.agent/gap-analysis/`. **Do not edit source files.** 2. Cross-adapter feature parity 3. Untracked features 4. Capability-flag drift + 5. Telemetry / observability parity (usage tokens, cache/reasoning + counts, request ids, logging asymmetry) 5. **Fan out** for `--all`: launch one `Explore` subagent per provider, max 3 - in parallel. Each subagent returns the four-dimension findings for its + in parallel. Each subagent returns the five-dimension findings for its provider; you synthesise into the combined report. 6. **Write the report** to `.agent/gap-analysis/YYYY-MM-DD-.md` using [references/report-template.md](references/report-template.md). Date is @@ -75,7 +77,10 @@ markdown report under `.agent/gap-analysis/`. **Do not edit source files.** (media-only), `elevenlabs` (TTS-only). The feature matrix tracks the first seven; `fal` and `elevenlabs` only appear in model/media audits. -## Known features (22) +## Known features (19) + +Canonical list is `ALL_FEATURES` in `testing/e2e/src/lib/types.ts` — always +re-read it; this list is a snapshot: `chat`, `one-shot-text`, `reasoning`, `multi-turn`, `tool-calling`, `parallel-tool-calls`, `tool-approval`, `text-tool-text`, `structured-output`, diff --git a/.claude/skills/gap-analysis/references/audit-checklist.md b/.claude/skills/gap-analysis/references/audit-checklist.md index c9c6f02c7..de6220771 100644 --- a/.claude/skills/gap-analysis/references/audit-checklist.md +++ b/.claude/skills/gap-analysis/references/audit-checklist.md @@ -98,7 +98,7 @@ excluded: …`). If yes → **tested gap**, surface in "Out-of-scope" with ## 3. Untracked features **Input:** the union of `ALL_FEATURES` plus the four media adapters -(`image`, `tts`, `transcription`, `video`). +(`image-gen`, `tts`, `transcription`, `video-gen`). **Upstream:** provider's API reference top-level navigation. Steps: @@ -150,8 +150,9 @@ Steps: - `supports.tools` missing a newly launched provider tool - `context_window` stale (upstream raised the cap) - `knowledge_cutoff` stale -4. Pricing drift is **out of scope** for an audit (changes too often) — - note in the report only if upstream announcement is < 30 days old. +4. Pricing drift is generally **out of scope** for an audit (changes too + often), but must be included in the report when an upstream pricing + announcement is less than 30 days old. **Priority rubric:** @@ -162,6 +163,60 @@ Steps: --- +## 5. Telemetry / observability parity + +**Input:** every adapter's stream-emit and final-response paths in +`packages/typescript/ai-/src/adapters/*.ts`. +**Upstream:** the provider's API reference for the response envelope (usage, +cost, cache, reasoning, request-id, safety/moderation fields). + +The goal is to flag adapters that drop telemetry their upstream returns, or +that surface it less completely than a sibling adapter does. Pricing is +not computed locally — but if upstream returns billable counts (cached +tokens, reasoning tokens, image-token splits, etc.) the adapter should pass +them through so callers can price downstream. + +Steps: + +1. For each adapter, grep the streaming and non-streaming paths for the + fields it forwards into `StreamChunk` / final response usage: + - `Grep "usage|promptTokens|completionTokens|cached|reasoning|cache_creation|cache_read" packages/typescript/ai-/src/adapters/` +2. Build a cross-adapter table of which fields each adapter emits. Rows + below are examples — extend per provider: + + | Telemetry field | openai | anthropic | gemini | ollama | grok | groq | + | --------------------------- | ------ | --------- | ------ | ------ | ---- | ---- | + | prompt / input tokens | | | | | | | + | completion / output tokens | | | | | | | + | total tokens | | | | | | | + | cached input tokens | | | | | | | + | cache-creation tokens | | | | | | | + | reasoning tokens | | | | | | | + | image / audio token splits | | | | | | | + | request-id / response-id | | | | | | | + | upstream cost (if returned) | | | | | | | + | safety / moderation flags | | | | | | | + | finish-reason / stop-reason | | | | | | | + +3. WebFetch the upstream API reference to confirm which fields are actually + returned. A blank cell where upstream returns the field → telemetry gap. + A blank cell where upstream doesn't return the field → not a gap. +4. Also check logging parity: does one adapter `console.warn` / + `console.debug` on retry/rate-limit/parse-failure while another swallows + it silently? Grep `console\.(warn|error|debug|info)` and `logger` per + adapter and note asymmetries. + +**Priority rubric:** + +- Adapter drops a field upstream returns that another adapter forwards + (e.g., cached tokens) → **high** (breaks downstream cost accounting). +- Adapter drops a field upstream returns but no sibling forwards it + either → **medium** (gap, but not a regression). +- Logging asymmetry (one adapter warns, another swallows) → **low** + unless it hides a class of errors callers need to handle. + +--- + ## Subagent dispatch (for `--all` scope) When fan-out is needed, launch one `Explore` subagent per provider with a @@ -170,10 +225,12 @@ prompt of this shape: > Audit the `` adapter at `packages/typescript/ai-/` > against upstream docs at the URLs in > `.claude/skills/gap-analysis/references/provider-doc-urls.md`. Walk -> dimensions 1, 3, and 4 from `audit-checklist.md`. Skip dimension 2 (the -> orchestrator handles cross-provider parity centrally). Return findings as -> markdown sections matching the report template — High / Medium / Low / -> Out-of-scope — with upstream URLs cited for every claim. +> dimensions 1, 3, 4, and 5 from `audit-checklist.md`. Skip dimension 2 +> (the orchestrator handles cross-provider parity centrally) — but do +> emit dimension-5 telemetry rows in the per-provider format; the +> orchestrator stitches them into the cross-adapter table. Return +> findings as markdown sections matching the report template — High / +> Medium / Low / Out-of-scope — with upstream URLs cited for every claim. Run at most 3 in parallel. Aggregate their returned markdown into the combined report. diff --git a/.claude/skills/gap-analysis/references/provider-doc-urls.md b/.claude/skills/gap-analysis/references/provider-doc-urls.md index 2b67eb647..2dc1818d4 100644 --- a/.claude/skills/gap-analysis/references/provider-doc-urls.md +++ b/.claude/skills/gap-analysis/references/provider-doc-urls.md @@ -87,6 +87,9 @@ WebFetch — call `resolve-library-id` with the SDK npm name, then `query-docs`. ## Maintenance -If you find a URL has permanently moved while running an audit, update this -file in the same report turn — the maintainer will see the change in -`git status` alongside the new report. +Audit runs are read-only outside `.agent/gap-analysis/`. If you find a URL +has permanently moved while running an audit, **do not edit this file** in +the same turn (that would violate the clean-working-tree rule in +`SKILL.md`). Instead, note the broken/moved URL and its replacement in the +report's _Suggested follow-ups_ section so the maintainer can apply the +update in a separate PR. diff --git a/.claude/skills/gap-analysis/references/report-template.md b/.claude/skills/gap-analysis/references/report-template.md index 0b63b4682..468b04234 100644 --- a/.claude/skills/gap-analysis/references/report-template.md +++ b/.claude/skills/gap-analysis/references/report-template.md @@ -78,6 +78,23 @@ Replace every `{{placeholder}}`. Drop sections that have zero entries --- +## Telemetry / observability parity + +> Per dimension 5 in audit-checklist.md. Compares what each adapter +> forwards into `StreamChunk` / final response vs. what upstream returns. + +- **[{{provider}}] drops `{{field}}`** that upstream returns (e.g., + `cache_read_input_tokens`, `reasoning_tokens`, `cached_tokens`). + - Upstream evidence: [{{doc-title}}]({{doc-url}}) + - Sibling adapters that already forward it: {{list}} + - Adapter path: `packages/typescript/ai-{{provider}}/src/adapters/{{file}}:{{line}}` + - Effort: {{S / M / L}} + +- **[{{provider}}] logging asymmetry** — {{e.g., adapter swallows parse + failures while `` warns}}; consider matching. + +--- + ## Out-of-scope — documented exclusions > Listed for completeness; no action required. Each links to the comment @@ -101,7 +118,7 @@ Ordered list of concrete files to touch, one per follow-up: - Scope: `{{scope-raw}}` - Date: {{YYYY-MM-DD}} - Providers audited: {{comma-separated}} -- Features audited: {{comma-separated or "all 22"}} +- Features audited: {{comma-separated or "all (per ALL_FEATURES)"}} - Upstream pages fetched: {{count}} - Subagents launched: {{count or "none"}} - Time: {{wall-clock}} (optional)