fix: pivot installer from Catwalk to Models.dev by iceglober · Pull Request #124 · iceglober/harness-opencode

iceglober · 2026-04-26T17:30:49Z

Pivot installer from Catwalk to Models.dev (fix amazon-bedrock prefix)

Goal

OpenCode's runtime resolves <provider_id>/<model_id> against Models.dev's registry (verified from docs at https://opencode.ai/docs/models/: "OpenCode uses the AI SDK and Models.dev to support 75+ LLM providers"). The plugin's installer fetches provider/model data from a different registry — Catwalk at https://catwalk.charm.land/v2/providers — to offer provider selection and suggest deep/mid/fast tiers. The two registries disagree on provider IDs for at least Bedrock (Catwalk: "bedrock", Models.dev: "amazon-bedrock") and Vertex (Catwalk: no entry; Models.dev: "google-vertex-anthropic"). The installer's hardcoded MODEL_PRESETS, its dynamic Catwalk-driven path, the validator's suggestion map, and multiple test fixtures all emit Catwalk-style IDs. OpenCode then rejects those IDs at agent invocation with Agent <name>'s configured model <id> is not valid.

Fix by pivoting the installer to use Models.dev as its data source — the same registry the runtime consults. This collapses the dual-source bug class entirely: what the installer writes into opencode.json is what the runtime validates against.

Out-of-the-box-broken providers before this fix: Bedrock, Vertex. Anthropic (direct) has worked because its provider ID happens to be identical in both registries.

Constraints

Zero user-filesystem-writes invariant at plugin startup. No auto-rewrite of existing opencode.json configs — only the install flow (user-invoked) writes files.
Must not break the installer on Catwalk outage — Models.dev availability is now the critical dependency, but both sources can fail, so retain a hardcoded fallback preset list covering Anthropic, Bedrock, Vertex.
suggestTiers(provider) produces ${provider.id}/${model.id} refs — when we switch the input shape from Catwalk's schema to Models.dev's, the ref format must match what the runtime expects.
The Bedrock CRIS decision: installer default is amazon-bedrock/global.anthropic.* (decided above). Models without a global. variant fall back to non-prefixed anthropic.* (example: Opus 4.1 has no global. variant on Models.dev).
The legacy @glrs-dev/harness-opencode@<pre-fix> tuple-form harness.models block may still exist in user configs with Catwalk-style IDs (bedrock/anthropic.*). The validator must flag these with an amazon-bedrock/global.anthropic.* suggestion.
Existing passing tests that lock Catwalk-style IDs as correct (test/catwalk.test.ts:105-121 "produces correct refs for real Bedrock provider shape") must be rewritten or deleted, not preserved.
bun run build && bun run typecheck && bun test must pass.
Changeset: minor bump (user-visible fix, changes the installer's output and flags previously-silently-accepted IDs — this is not a pure patch).

Acceptance criteria

File-level changes

src/cli/models-dev.ts (NEW)

Change: Implement Models.dev client. Fetch https://models.dev/api.json, parse top-level object keyed by provider ID (not an array — verify during implementation; the explore-agent returned a map shape when finding "amazon-bedrock" as top-level key). Export the typed provider/model shapes needed by the installer. Implement suggestTiersFromModelsDev mirroring the cost-sorted algorithm from src/cli/catwalk.ts:59-95. Implement pickBedrockTierIds with the global. preference.
Why: Put the new source of truth in its own module. Keep the Catwalk module alone (deletion is a separate decision).
Risk: medium. New HTTP client. Models.dev's response is large (~1.8 MB per the earlier fetch); need to handle parse/network failures. Fallback to MODEL_PRESETS on any failure.

src/cli/install.ts

Change:
- Update MODEL_PRESETS entries for Bedrock and Vertex (line 66-88) per acceptance criteria. Leave Anthropic entry unchanged.
- Replace import from ./catwalk.js with import from ./models-dev.js (line 33). Update fetchProviders → fetchModelsDevProviders, suggestTiers → suggestTiersFromModelsDev.
- Update the provider-picker UX (lines 379-418) to iterate Models.dev's provider list. Provider IDs from Models.dev are directly usable as runtime IDs, so no translation layer needed.
- Update line 457 example text.
Why: Main path correction.
Risk: medium. Installer is user-facing; if the fetch fails, user sees a broken flow. Mitigation: retain hardcoded MODEL_PRESETS fallback; always be ready to offer presets if fetch fails.

src/cli/catwalk.ts (DELETE)

Change: Delete the file. Only two consumers: src/cli/install.ts:33 (changing) and test/catwalk.test.ts:2 (also deleting). Confirmed via grep -rn "catwalk" src/ test/ — no other importers.
Why: Catwalk is the wrong source of truth; keeping a deprecated module adds maintenance burden for zero benefit.
Risk: low. Audit done.

src/model-validator.ts

Change: Extend LEGACY_PRE_100_PATTERN to also flag ^bedrock\/ and ^vertex(ai)?\/ catch-alls (these provider prefixes are never valid at runtime; any ID starting with them is broken). Rename LEGACY_TO_CATWALK → LEGACY_TO_MODELS_DEV. Rewrite all suggestion values to Models.dev IDs. Add entries for the off-by-one forms (bedrock/anthropic.*). Update module-level JSDoc explaining the dual-registry problem and why this validator exists.
Why: Runtime-warning symmetry with the install-time fix. Users whose configs were generated by an old installer get the same Models.dev-valid suggestion the new installer produces.
Risk: low. Pure function. Regex extension is additive-union.

test/model-validator.test.ts

Change: See acceptance criteria. Fundamentally: strings I previously asserted were "valid" are now genuinely invalid (Models.dev confirms). Move them to the invalid bucket with correct suggestions. Add new positive cases with Models.dev-format IDs.
Why: Test the correct behavior. Without this, the suite actively locks in the bug.
Risk: low. Test-only.

test/catwalk.test.ts (DELETE)

Change: Delete the file. src/cli/catwalk.ts is being deleted too; no point keeping the test for a deleted module.
Why: No orphaned tests.
Risk: low.

test/models-dev.test.ts (NEW)

Change: Unit tests for the new module. Three test buckets: fetch (mocked), suggestTiers (fixture data, cost sorting), pickBedrockTierIds (CRIS preference + fallback).
Why: Lock in the new source-of-truth module's contract.
Risk: none.

test/harness-models.test.ts

Change: Update string literals. Split the existing "does not warn when overrides are valid" test into two cases: one for Anthropic (anthropic/claude-opus-4-7 — still valid), one for Models.dev Bedrock (amazon-bedrock/global.anthropic.* — now valid). Move any bedrock/anthropic.* strings from "valid" assertions to "flags bad ID" assertions.
Why: Align with validator's new behavior.
Risk: low. Test-only.

test/doctor.test.ts

Change: Update assertion strings at lines 117-118 to match new Models.dev-style suggestions.
Why: Align with validator.
Risk: low.

.changeset/pivot-installer-to-models-dev.md (NEW)

Change: Create via bunx changeset, pick minor. Body:

Pivot the installer's provider/model data source from Catwalk (catwalk.charm.land) to Models.dev (models.dev/api.json), matching what OpenCode's runtime uses to validate model IDs.

Previously, the installer emitted provider IDs from Catwalk's registry (bedrock/anthropic.claude-opus-4-6, vertexai/claude-opus-4-6@20250610) that OpenCode's runtime rejects at agent invocation with Agent <name>'s configured model <id> is not valid. Models.dev uses different provider IDs (amazon-bedrock, google-vertex-anthropic) for the same providers. Bedrock and Vertex presets have been broken out of the box since this ID schism was introduced upstream; only the Anthropic preset happened to work because its provider ID is identical in both registries.

The Bedrock preset now emits amazon-bedrock/global.anthropic.claude-* IDs (using AWS CRIS global cross-region inference for the highest availability). The Vertex preset now emits google-vertex-anthropic/claude-*@default IDs. The Anthropic preset is unchanged.

The plugin's runtime validator (src/model-validator.ts) now flags any model override starting with bedrock/ or vertex(ai)/ as invalid and suggests the Models.dev-valid replacement. If you hit ProviderModelNotFoundError or configured model ... is not valid after a recent OpenCode upgrade, run bunx @glrs-dev/harness-opencode doctor — it will enumerate the bad overrides and the correct Models.dev IDs.

Note for existing installations: your opencode.json is never auto-rewritten. The doctor tells you the exact line to change. If you had a working anthropic/* or amazon-bedrock/anthropic.*-v1 config, nothing changes. If you had a Catwalk-style bedrock/anthropic.* config, you will now see warnings until you update it.
Why: Release contract; changelog entry describes breaking scope to users.
Risk: none.

Test plan

Unit (new module): bun test test/models-dev.test.ts.
Unit (validator): bun test test/model-validator.test.ts.
Integration (resolver warn-dedup): bun test test/harness-models.test.ts.
Integration (doctor): bun test test/doctor.test.ts.
Legacy / deletion: bun test test/catwalk.test.ts if retained; otherwise confirm file is gone and no other test imports from src/cli/catwalk.ts.
Full suite: bun test. Expect 0 failures.
Typecheck: bun run typecheck — no errors. Models.dev response types must flow through.
Build: bun run build — dist output includes models-dev.js/.d.ts.
Dry-run publish: npm publish --dry-run — verify tarball content.
Smoke:
- node -e "const v = require('./dist/model-validator.js'); console.log(v.validateModelOverride('bedrock/anthropic.claude-opus-4-6'))" → { valid: false, suggestion: 'amazon-bedrock/global.anthropic.claude-opus-4-7', ... }.
- Sandbox install: XDG_CONFIG_HOME=/tmp/smoke node dist/cli.js install, pick AWS Bedrock, verify amazon-bedrock/global.anthropic. 3× in /tmp/smoke/opencode/opencode.json.

Out of scope

Replacing the validator entirely with a live Models.dev manifest lookup at plugin startup. Higher blast radius, needs offline cache, needs graceful degradation on network failure. Possible follow-up; not this PR.
Fixing other providers beyond Bedrock/Vertex/Anthropic (Gemini has "gemini" in Catwalk, likely "google" or "google-generative-ai" in Models.dev — unverified). MODEL_PRESETS currently only ships those three; if a user hits a Gemini bug they'll file a separate issue.
Auto-rewriting user configs. Explicit non-goal per the plugin's zero-user-filesystem-writes invariant.
Reworking src/config-hook.ts's resolveHarnessModels(). It already calls the validator; the validator is the only thing that changes.

Open questions

All resolved during grounding pass.

catwalk.ts consumers: Only src/cli/install.ts:33 and test/catwalk.test.ts:2 import it. Decision: delete both files and remove the dependency entirely. Cleaner than deprecating in-place.
Models.dev response shape: Top-level is Record<string, Provider> (object map keyed by provider ID). Provider.models is also an object map keyed by model ID. Iterate via Object.entries(...) both levels.
Error handling: fetchModelsDevProviders() wraps fetch + JSON.parse in try/catch, returns null on any failure (network, HTTP non-2xx, parse error, unexpected shape). Installer falls through to hardcoded MODEL_PRESETS.
Cost fields: Models.dev uses model.cost.input and model.cost.output (numbers, USD per 1M tokens). cost.cache_read and cost.cache_write are optional. Catwalk's flat cost_per_1m_in / cost_per_1m_out do not exist on Models.dev — the new tier suggester reads model.cost.input + model.cost.output for cost-sort.

Concrete type definitions (resolved)

// src/cli/models-dev.ts

export interface ModelsDevModel {
  id: string;
  name: string;
  family?: string;
  attachment?: boolean;
  reasoning?: boolean;
  tool_call?: boolean;
  structured_output?: boolean;
  temperature?: boolean;
  knowledge?: string;
  release_date?: string;
  last_updated?: string;
  modalities?: { input: string[]; output: string[] };
  open_weights?: boolean;
  cost?: {
    input?: number;
    output?: number;
    cache_read?: number;
    cache_write?: number;
  };
  limit?: { context?: number; output?: number };
}

export interface ModelsDevProvider {
  id: string;
  name: string;
  env?: string[];
  npm?: string;
  doc?: string;
  models: Record<string, ModelsDevModel>;
}

export type ModelsDevRegistry = Record<string, ModelsDevProvider>;

// Fetch URL: https://models.dev/api.json
// Returns null on any failure — caller falls through to MODEL_PRESETS.
export async function fetchModelsDevProviders(): Promise<ModelsDevProvider[] | null>;

// Cost-sort: deep = highest input+output cost, fast = lowest, mid = middle.
// Matches Catwalk's suggestTiers algorithm.
export function suggestTiersFromModelsDev(
  provider: ModelsDevProvider,
): { deep: string; mid: string; fast: string };

// Bedrock-specialized: prefer `global.anthropic.*` variants; fall back to
// non-prefixed `anthropic.*` where `global.` doesn't exist for a model family.
// Input: the amazon-bedrock provider entry. Output: deep/mid/fast refs.
export function pickBedrockTierIds(
  provider: ModelsDevProvider,
): { deep: string; mid: string; fast: string };

Progress log

OpenCode validates model IDs against the Models.dev registry at runtime (per https://opencode.ai/docs/models/), but the installer was using Catwalk — which uses different provider IDs for AWS Bedrock (`bedrock` vs `amazon-bedrock`) and Google Vertex (`vertexai` vs `google-vertex-anthropic`). Every Bedrock and Vertex preset emitted by the installer crashed at agent invocation with `Agent … configured model … is not valid`. Only Anthropic direct worked because its provider ID is identical in both registries. Pivot the installer to Models.dev — the same source of truth the runtime consults — eliminating the ID schism at the root. New module `src/cli/models-dev.ts` fetches, types, and tier-suggests from `models.dev/api.json`. Bedrock preset now uses `amazon-bedrock/global.anthropic.*` (CRIS global route for highest availability); Vertex preset uses `google-vertex-anthropic/*@default`. Anthropic preset unchanged. Validator (`src/model-validator.ts`) now flags any ID with a `bedrock/` or `vertex(ai)/` prefix as invalid and suggests the correct Models.dev replacement. Runtime warns users on next startup; `doctor` enumerates offenders. No auto-rewrite of user config. Delete `src/cli/catwalk.ts` and `test/catwalk.test.ts` — no other consumers. Add `test/models-dev.test.ts` (16 new tests covering fetch, suggestTiers, pickBedrockTierIds). Update all existing tests to assert the new behavior; the "valid IDs" buckets previously locked Catwalk-style IDs as correct (they weren't). Full suite: 878 pass, 0 fail, 47 files. Plan: ~/.glorious/opencode/glorious-opencode/plans/fix-amazon-bedrock-provider-prefix.md

iceglober force-pushed the fix/pilot-agents-provider-model-not-found branch from 308ffc3 to a6b7910 Compare April 26, 2026 17:49

iceglober merged commit fafe250 into main Apr 26, 2026
2 checks passed

harness-opencode-ci Bot mentioned this pull request Apr 26, 2026

chore: version packages #123

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: pivot installer from Catwalk to Models.dev#124

fix: pivot installer from Catwalk to Models.dev#124
iceglober merged 1 commit into
mainfrom
fix/pilot-agents-provider-model-not-found

iceglober commented Apr 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

iceglober commented Apr 26, 2026

Pivot installer from Catwalk to Models.dev (fix amazon-bedrock prefix)

Goal

Constraints

Acceptance criteria

File-level changes

src/cli/models-dev.ts (NEW)

src/cli/install.ts

src/cli/catwalk.ts (DELETE)

src/model-validator.ts

test/model-validator.test.ts

test/catwalk.test.ts (DELETE)

test/models-dev.test.ts (NEW)

test/harness-models.test.ts

test/doctor.test.ts

.changeset/pivot-installer-to-models-dev.md (NEW)

Test plan

Out of scope

Open questions

Concrete type definitions (resolved)

Progress log

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant