Extract the model catalog, add discovery probe automation, and onboard 26 models - #127
Merged
alexanderguy merged 27 commits intoAug 10, 2026
Conversation
The model and provider facts were hand-authored inline in dev tooling and reachable only by standing up the seed, its database, and the hub. Making them a published, dependency-free package gives internal trees and external consumers a typed source of what models and providers exist without a running system. Capabilities are baked to literals so importing pulls no discovery rig, while each offering keeps its discovery source and curated tags as provenance. A guard re-bakes from that provenance against the discovery matrix and pins the capability vocabulary and provider base URLs, so the duplication the zero-dependency contract requires cannot silently drift.
The seed and the live-session recorder now consume the published @intx/inference-catalog instead of the inline dev-tooling modules, which are removed. Dev-only credentials and pricing, deliberately excluded from the published package, stay as seed-local fixtures joined to the catalog by provider and model, with a coverage check that fails loudly on a gap rather than seeding a partial catalog. Offering capabilities now come from the package's baked literal, which the package guard pins equal to what the discovery matrix resolves, so the seeded rows are unchanged.
Dev pricing is a fake seed fixture, so a catalog offering can be seeded without one. Model that as an explicit choice rather than an inference: an offering must be either priced or listed in the unpriced set, never neither and never both. A dropped or mistyped price on a priced model still aborts the seed, so the fail-loud typo-catch is preserved while price-free offerings become possible.
The discovery support matrix has probed several models the catalog did not yet carry. Add them as offerings on the existing providers, with capabilities baked from the matrix and provenance preserved. Shared models fan out across the relays that serve them; reasoning-bearing opencode-zen offerings carry the reasoning quirks, extended from the Kimi backends where that requirement is verified to the others on the same relay wire. The new offerings ship without dev pricing, which stays a seed-only fixture, so they are listed in the seed's unpriced set rather than given fake prices.
Probe claude-opus-4-8 across the Anthropic capability set and commit the captured session fixtures. The model rejects the classic thinking.type.enabled wire with a 400 and requires the adaptive wire, so it joins the adaptive-thinking set the request builder keys on; without that it never reaches the thinking-with-tools captures.
Add claude-opus-4-8 as an Anthropic Direct offering now that the discovery matrix carries its captured capabilities. The capabilities are baked from the matrix and the offering ships unpriced, like the other newly added models.
Probe the net-new opencode-zen relay models live and commit the captured fixtures. Each model's capabilities reflect what it demonstrated on the wire, not merely an HTTP 200: vision and reasoning_content vary widely per model. grok-4.5 (the relay reports its upstream endpoint unavailable) and the deprecated mimo-v2-omni and mimo-v2-pro are probed but excluded.
Add the twelve viable opencode-zen relay models probed live as OpenCode Zen v1 offerings. Each offering advertises only the capabilities the model demonstrated on the wire, so vision and reasoning vary across them; reasoning-bearing offerings carry the reasoning quirks and the rest ship an empty bag. All ship unpriced.
Probe claude-opus-4-5/4-6/4-7 and claude-sonnet-4-5/4-6 across the Anthropic capability set and commit the captured fixtures. Adaptive extended thinking arrived at 4.6, so opus-4-6, opus-4-7 and sonnet-4-6 join the adaptive-thinking set while the 4.5 snapshots keep the classic wire; the set is mirrored in the discovery and runtime providers.
Add claude-opus-4-5/4-6/4-7 and claude-sonnet-4-5/4-6 as Anthropic Direct offerings now that the discovery matrix carries their captured capabilities. Capabilities are baked from the matrix and all ship unpriced.
gemini-3.1-pro-preview rejects a zero thinking budget with a 400, so it joins the thinking-mandatory set that requests the dynamic budget instead. gemini-3-flash-preview accepts a zero budget and stays out of that set.
A provider's request builder throws when it cannot construct a request for a (model, capability) pair. Until now that was a bare Error, indistinguishable from a malformed intent or a genuine builder bug. The discovery probe needs to treat the former as an unsupported outcome while letting the latter propagate, so the builders raise a typed error the probe can catch by identity.
google-genai used TEXT_MODELS/IMAGE_MODELS membership both to pick a model's request shape and to reject any model absent from both sets. The rejection blocked discovery of new models, though the sets are a curated record rather than a wall the probe should hit. Membership now only classifies: a known model keeps its set-derived class; an unknown model takes the caller's override, defaulting to text, so a new chat model probes without being pre-registered. An unknown image model must declare its class, since the image request shape cannot be inferred from identity.
The capture bundle records response headers and body but not the HTTP status line, so nothing offline can tell a genuine 2xx capture from a 4xx or 5xx error body. runCapture now returns the final status and exchange count. A non-2xx response also ends the capture at once: it persists the error bytes for inspection but does not parse them (a 4xx/5xx body need not be JSON) and does not write the success-path manifest, so a failed capture is never recorded as a session.
A capture that returned HTTP 200 is not evidence the capability works: the body can be an empty shell, a soft decline, or an error payload the wire delivered with a 200. classifyReplay drives the captured response through the production runInference decoder and reads the resulting event stream, so a cell counts as captured only when the runtime produces real content. A decode error, an invariant violation, or a replay that throws all resolve to misled rather than crashing a batch.
bin/discover replays the curated support matrix, so it can only touch models already listed. Bringing a new model in needs the opposite: a sweep of a model the matrix does not know across every capability its provider can build. probe makes one live call per capability and records the outcome only the live call can see: unsupported when the request cannot be built, http-error on a non-2xx, or a 2xx capture. Captures land in a scratch tree, never the committed corpus, because baking an accepted model is a separate, deliberate step.
probe records the HTTP outcome of a capture but not whether the 200 it got is genuine. classify-sessions is the offline read side: it replays each capability session under a model directory through the production decoder and reports what the decode implies: captured, refused, or misled. It makes no network call, so it reruns freely against a scratch probe tree or the committed corpus without paying.
Most carry the full Chat Completions captured surface. gpt-4-turbo takes image and document input but rejects json_schema structured output; the original text-only gpt-4 rejects both structured output and image or document parts. None surface a reasoning field on this wire, the o-series reasoning models included, so reasoning stays unsupported here and is reachable only through the Responses API.
Batch classification must never crash on one undecodable session; that contract had no regression guard until now.
The probe's argument parsing ran only inside the CLI's top-level entry point, so its index-mutating value reader and validation had no coverage. Splitting it into a shared module makes it testable and adds two behaviors the tests pin: --model-class is rejected for a non-google provider instead of silently ignored, and a call-failed cell (an unexpected error, not a normal discovery outcome) makes the command exit non-zero.
The response-header redaction list dropped openai-organization but not openai-project, so the project identifier that pairs with it rode into every committed OpenAI capture. It is an identifier, not a credential, but it belongs redacted alongside the organization header. Add it to the list so future captures redact it, and scrub the value from the existing fixtures.
alexanderguy
changed the title
The catalog splits its flat model list from its offerings across two files, so an offering can name a model the list never carries. The seed resolved offerings by model name and silently skipped a miss, shipping a dev catalog that lacked an intended model. It now fails on the miss, and a catalog guard test rejects the drift at build time.
No caller reads exchangeCount; the probe consumes only finalStatus, so the field was speculative surface on a returned type.
The discover command writes into the committed sessions tree, so a non-2xx response now stops it loudly and removes the partial bundle instead of exiting 0 with an unreplayable session. The probe prints a distinct ASSUMED marker when it defaults an unknown google-genai model to the text class, rather than a label that read identically for a known model. A shared formatter now backs both discovery CLIs' provider help, and guard tests pin the gemini and openai advertised rosters to their classifier and matrix sources.
The runtime adapter and the discovery probe each own a copy of which anthropic models use the adaptive thinking wire, kept in sync only by a comment. If they drift, a captured fixture builds a different request shape than production sends and stops proving the real wire. Both sets are now exported and a guard test asserts they are equal, so adding a model to one but not the other fails the build.
Member
Author
Self-reviewWhole-branch review ( One known exception, tracked separately as INTR-449: the anthropic adaptive-thinking |
alexanderguy
deleted the
intr-445-extract-the-model-catalog-into-intxinference-catalog-a
branch
August 10, 2026 03:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@intx/inference-catalog, a standalone, zero-runtime-dependency, publicly-published package the dev seed consumes (layering dev-only pricing on top). Capabilities are baked as resolved literals with preserved provenance (discoverySource+curatedCapabilities); guard tests keep them derived from the discovery matrix and keep every offering's model present in the catalog.bin/probesweeps a model that need not be in the support matrix across every capability its provider can build, recording the HTTP outcome only the live call can see;bin/classify-sessionsclassifies the captures offline by replaying them through the production decoder. Supporting changes: a typedCapabilityNotBuildableError; google's model sets become a request-shape classifier rather than a discovery gate (an unknown chat model probes without pre-registration);runCapturereturns the HTTP outcome and stops on a non-2xx; a replay-based semantic classifier maps a capture to the support-matrix outcome vocabulary.openai-projectresponse header in captured fixtures, alongside the already-redactedopenai-organization.Verification
make allpasses at HEAD: lint,tsctypecheck, admin-ui build, and the full test suite (0 failures).runInferencedecoder, so a baked capability means the runtime actually decodes real content — not merely that a request returned 200.Closes INTR-445
Closes INTR-232