Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Changelog

All notable changes to the ContextualWisdomLab organization workflow policy are
documented in this file.

The format follows Keep a Changelog, and releaseable changes use Semantic
Versioning where the central workflow repository publishes a versioned release.

## [Unreleased]

### Added

- Import the exact, receipt-verified `contextual-orchestrator` fallback-policy
module for Noema, OpenCode Agent, and Strix.
- Add a strict shared model manifest with explicit cost tier, repository
visibility, required credential name, capability, and deterministic priority.
- Add fail-closed supply-chain verification for the vendored source commit and
Git blob identities.
- Add 74 integration, vendored-policy, and adapter regression tests plus
operator and doctoring documentation.

### Changed

- Noema now exhausts eligible public NVIDIA NIM free candidates before an
explicitly configured custom fallback.
- OpenCode Agent now places every eligible NVIDIA NIM, OpenCode free, and
included-quota GitHub Models candidate before paid provider candidates.
- Strix now uses the same policy order while preserving its existing provider
transports, report parsing, severity threshold, and reviewer credentials.

### Security

- Private and internal repositories are excluded from public-only hosted trial
candidates.
- Model pool, vendor receipt, import path, file type, JSON size, duplicate key,
and source-identity drift fail closed without exposing secret values.
1 change: 1 addition & 0 deletions config/llm-fallback-policy.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"agents":{"noema":{"candidates":[{"candidate_id":"noema_nim_ultra","capabilities":["text","structured_output"],"cost_tier":"free","model":"nvidia/nemotron-3-ultra-550b-a55b","priority":10,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_NIM_API_KEY"]},{"candidate_id":"noema_nim_super_49b","capabilities":["text","structured_output"],"cost_tier":"free","model":"nvidia/llama-3.3-nemotron-super-49b-v1.5","priority":20,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_NIM_API_KEY"]},{"candidate_id":"noema_nim_super_120b","capabilities":["text","structured_output"],"cost_tier":"free","model":"nvidia/nemotron-3-super-120b-a12b","priority":30,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_NIM_API_KEY"]},{"candidate_id":"noema_custom","capabilities":["text","structured_output"],"cost_tier":"paid","model":"configured/noema-custom","priority":1000,"provider":"noema-custom","repository_visibilities":["public","private","internal"],"required_credentials":["NOEMA_CUSTOM_LLM_CONFIGURED"]}]},"opencode-review":{"candidates":[{"candidate_id":"oc_nim_super_49b","capabilities":["text","code_review"],"cost_tier":"free","model":"nvidia-nim/nvidia/llama-3.3-nemotron-super-49b-v1.5","priority":10,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_API_KEY"]},{"candidate_id":"oc_nim_ultra_253b","capabilities":["text","code_review"],"cost_tier":"free","model":"nvidia-nim/nvidia/llama-3.1-nemotron-ultra-253b-v1","priority":20,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_API_KEY"]},{"candidate_id":"oc_nim_super_120b","capabilities":["text","code_review"],"cost_tier":"free","model":"nvidia-nim/nvidia/nemotron-3-super-120b-a12b","priority":30,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_API_KEY"]},{"candidate_id":"oc_nim_ultra_550b","capabilities":["text","code_review"],"cost_tier":"free","model":"nvidia-nim/nvidia/nemotron-3-ultra-550b-a55b","priority":40,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_API_KEY"]},{"candidate_id":"oc_nim_llama_70b","capabilities":["text","code_review"],"cost_tier":"free","model":"nvidia-nim/meta/llama-3.3-70b-instruct","priority":50,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_API_KEY"]},{"candidate_id":"oc_nim_deepseek_v4","capabilities":["text","code_review"],"cost_tier":"free","model":"nvidia-nim/deepseek-ai/deepseek-v4-pro","priority":60,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_API_KEY"]},{"candidate_id":"oc_nim_codestral","capabilities":["text","code_review"],"cost_tier":"free","model":"nvidia-nim/mistralai/codestral-22b-instruct-v0.1","priority":70,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["NVIDIA_API_KEY"]},{"candidate_id":"oc_free_nemotron","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/nemotron-3-ultra-free","priority":100,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_deepseek","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/deepseek-v4-flash-free","priority":110,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_north","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/north-mini-code-free","priority":120,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_laguna","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/laguna-s-2.1-free","priority":130,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_ling","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/ling-3.0-flash-free","priority":140,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_pickle","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/big-pickle","priority":150,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_mimo","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/mimo-v2.5-free","priority":160,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_hy3","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/hy3-free","priority":170,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_minimax","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/minimax-m3-free","priority":180,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_glm","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/glm-5-free","priority":190,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_kimi","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/kimi-k2.5-free","priority":200,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_free_qwen","capabilities":["text","code_review"],"cost_tier":"free","model":"opencode-free/qwen3.6-plus-free","priority":210,"provider":"opencode-free","repository_visibilities":["public"]},{"candidate_id":"oc_github_deepseek_v3","capabilities":["text","code_review"],"cost_tier":"free","model":"github-models/deepseek/deepseek-v3-0324","priority":300,"provider":"github-models","repository_visibilities":["public","private","internal"]},{"candidate_id":"oc_github_gpt41","capabilities":["text","code_review"],"cost_tier":"free","model":"github-models/openai/gpt-4.1","priority":310,"provider":"github-models","repository_visibilities":["public","private","internal"]},{"candidate_id":"oc_github_gpt5","capabilities":["text","code_review"],"cost_tier":"free","model":"github-models/openai/gpt-5","priority":320,"provider":"github-models","repository_visibilities":["public","private","internal"]},{"candidate_id":"oc_github_gpt5_chat","capabilities":["text","code_review"],"cost_tier":"free","model":"github-models/openai/gpt-5-chat","priority":330,"provider":"github-models","repository_visibilities":["public","private","internal"]},{"candidate_id":"oc_github_o3","capabilities":["text","code_review"],"cost_tier":"free","model":"github-models/openai/o3","priority":340,"provider":"github-models","repository_visibilities":["public","private","internal"]},{"candidate_id":"oc_github_dsr1_0528","capabilities":["text","code_review"],"cost_tier":"free","model":"github-models/deepseek/deepseek-r1-0528","priority":350,"provider":"github-models","repository_visibilities":["public","private","internal"]},{"candidate_id":"oc_github_dsr1","capabilities":["text","code_review"],"cost_tier":"free","model":"github-models/deepseek/deepseek-r1","priority":360,"provider":"github-models","repository_visibilities":["public","private","internal"]},{"candidate_id":"oc_terra","capabilities":["text","code_review"],"cost_tier":"paid","model":"opencode/gpt-5.6-terra","priority":1000,"provider":"opencode","repository_visibilities":["public","private","internal"],"required_credentials":["OPENCODE_API_KEY"]},{"candidate_id":"oc_luna","capabilities":["text","code_review"],"cost_tier":"paid","model":"openai/gpt-5.6-luna","priority":1010,"provider":"openai","repository_visibilities":["public","private","internal"],"required_credentials":["OPENAI_API_KEY"]},{"candidate_id":"oc_openrouter_deepseek","capabilities":["text","code_review"],"cost_tier":"paid","model":"openrouter/deepseek/deepseek-v3.2","priority":1020,"provider":"openrouter","repository_visibilities":["public","private","internal"],"required_credentials":["OPENROUTER_API_KEY"]},{"candidate_id":"oc_openrouter_qwen","capabilities":["text","code_review"],"cost_tier":"paid","model":"openrouter/qwen/qwen3-coder","priority":1030,"provider":"openrouter","repository_visibilities":["public","private","internal"],"required_credentials":["OPENROUTER_API_KEY"]}]},"strix":{"candidates":[{"candidate_id":"strix_nim_ultra","capabilities":["text","security_review"],"cost_tier":"free","model":"nvidia_nim/nvidia/nemotron-3-ultra-550b-a55b","priority":10,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["STRIX_PRIMARY_KEY_CONFIGURED"]},{"candidate_id":"strix_nim_super_49b","capabilities":["text","security_review"],"cost_tier":"free","model":"nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5","priority":20,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["STRIX_PRIMARY_KEY_CONFIGURED"]},{"candidate_id":"strix_nim_super_120b","capabilities":["text","security_review"],"cost_tier":"free","model":"nvidia_nim/nvidia/nemotron-3-super-120b-a12b","priority":30,"provider":"nvidia-nim","repository_visibilities":["public"],"required_credentials":["STRIX_PRIMARY_KEY_CONFIGURED"]},{"candidate_id":"strix_openrouter_free","capabilities":["text","security_review"],"cost_tier":"free","model":"openrouter/free","priority":100,"provider":"openrouter","repository_visibilities":["public","private","internal"],"required_credentials":["STRIX_PRIMARY_KEY_CONFIGURED"]},{"candidate_id":"strix_configured_github_primary","capabilities":["text","security_review"],"cost_tier":"free","model":"configured/strix-github-primary","priority":190,"provider":"github-models","repository_visibilities":["public","private","internal"],"required_credentials":["STRIX_PRIMARY_KEY_CONFIGURED"]},{"candidate_id":"strix_github_o3","capabilities":["text","security_review"],"cost_tier":"free","model":"github_models/openai/o3","priority":200,"provider":"github-models","repository_visibilities":["public","private","internal"],"required_credentials":["STRIX_GITHUB_MODELS_CONFIGURED"]},{"candidate_id":"strix_github_gpt5_chat","capabilities":["text","security_review"],"cost_tier":"free","model":"github_models/openai/gpt-5-chat","priority":210,"provider":"github-models","repository_visibilities":["public","private","internal"],"required_credentials":["STRIX_GITHUB_MODELS_CONFIGURED"]},{"candidate_id":"strix_configured_paid_primary","capabilities":["text","security_review"],"cost_tier":"paid","model":"configured/strix-paid-primary","priority":900,"provider":"configured","repository_visibilities":["public","private","internal"],"required_credentials":["STRIX_PRIMARY_KEY_CONFIGURED"]},{"candidate_id":"strix_openai_luna","capabilities":["text","security_review"],"cost_tier":"paid","model":"openai_direct/gpt-5.6-luna","priority":1000,"provider":"openai","repository_visibilities":["public","private","internal"],"required_credentials":["STRIX_PRIMARY_KEY_CONFIGURED"]},{"candidate_id":"strix_vertex_pro","capabilities":["text","security_review"],"cost_tier":"paid","model":"vertex_ai/gemini-3.1-pro-preview-customtools","priority":1010,"provider":"vertex-ai","repository_visibilities":["public","private","internal"]},{"candidate_id":"strix_vertex_flash","capabilities":["text","security_review"],"cost_tier":"paid","model":"vertex_ai/gemini-2.5-flash","priority":1020,"provider":"vertex-ai","repository_visibilities":["public","private","internal"]}]}},"schema_version":1}
135 changes: 135 additions & 0 deletions docs/doctoring/shared-llm-fallback-policy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Doctoring record: shared free-first LLM fallback policy

## Clinical finding

The central review workflows had three different model-selection contracts.
OpenCode Agent already had a broad pool and retries, Strix had provider-specific
fallbacks, and Noema made one model call. Their credentials, result schemas,
review identities, and security gates were intentionally different, but model
cost ordering was not governed by one auditable policy. This created four
risks: paid inference could run before an available free candidate, free-to-free
fallback was inconsistent, repository-visibility constraints could drift, and
provider pricing changes had no single review surface.

## Intervention

A pure policy module was added to `contextual-orchestrator` and imported into
the central `.github` repository through an exact-commit vendoring receipt. It
performs no network I/O. It validates trusted candidate metadata and returns a
deterministic eligible sequence in which all free candidates precede all paid
candidates. Thin adapters hand that sequence to the existing Noema, OpenCode,
and Strix execution engines.

The transport boundary is deliberate. Combining the agents into one HTTP
client would also combine privileges and could weaken current-head validation,
reviewer authentication, report parsing, or provider-specific credential
handling. The shared module therefore owns only candidate validation and
ordering; each agent retains its existing acceptance and security contract.

## Evidence-based rationale

LLM cascade research demonstrates that lower-cost models can be attempted
before escalation, but also shows that useful routing depends on task-specific
quality estimation. FrugalGPT reports large cost reductions from cascades;
RouteLLM learns cost-quality routing from preference data; and cascade-routing
research formalizes when routing and cascading can be combined. The present
implementation is intentionally the deterministic baseline: it enforces an
operator-selected budget boundary but does not claim to predict review quality.
A learned router may be added only after it is calibrated on the exact code
review and security tasks and preserves the selected free-before-paid policy.

Current provider documentation also shows that “free” is contractual and
mutable. GitHub Models includes rate-limited free usage, but an organization can
opt into paid usage. OpenRouter free variants and the `openrouter/free` router
have changing availability and lower rate limits. NVIDIA describes hosted API
access as a free development/prototyping endpoint that may be throttled. The
manifest therefore requires explicit `cost_tier` metadata and never infers cost
from a model name.

## Safety and privacy controls

- Public hosted candidates are ineligible for private and internal repositories.
- Noema's existing reviewer token hierarchy is unchanged.
- OpenCode's provider keys remain scoped to the privileged review job and its
unchanged core continues to reject synthetic approval after exhaustion.
- Strix's existing per-model key/API-base selection and severity gate remain
authoritative.
- Secret values are not persisted in the manifest, plan, receipt, diagnostics,
or test evidence.
- Vendor files are verified as regular non-symlink files against exact Git blob
identities before import.
- The manifest and receipt reject duplicate JSON keys, unknown fields, unsafe
identifiers, duplicate logical targets, unsupported schema versions, and
empty eligible pools.
- Provider exceptions are summarized by type/status rather than response body,
reducing accidental prompt or credential disclosure.

## Verification record

The implementation-specific test suite contains 74 regression tests covering:

- committed manifest plus exact vendored module integration for all three agents;
- free-before-paid and free-to-free ordering;
- stable priority and declaration-order ties;
- repository visibility, capability, and credential-name filtering;
- configured-pool drift and duplicate rejection;
- vendor receipt, source commit, file map, symlink, and Git blob verification;
- bounded UTF-8 JSON, duplicate-key, and import-path hardening;
- Noema fallback, environment restoration, secret-free failure diagnostics,
and preservation of the original single-model core;
- OpenCode adapter delegation and no-model behavior;
- Strix public NIM, GitHub Models, configured-primary, and fail-closed adapter
behavior.

Local exact-slice results before PR creation:

- 74 tests passed;
- `contextual_fallback_policy.py`: 174 statements, 56 branches, 100%;
- central Python policy surface: 487 statements, 180 branches, 100%;
- Noema wrapper: 93 statements, 36 branches, 100%;
- contextual-orchestrator policy source: 270 statements, 94 branches, 100%;
- all newly public Python symbols have docstrings;
- Bash syntax checks passed for the OpenCode adapter and Strix model utility.

Repository-wide GitHub checks on the exact PR head remain the authoritative
merge gate because they also execute the pre-existing Noema, OpenCode, Strix,
SAST, supply-chain, and required-workflow contracts.

## APA 7 references

Chen, L., Zaharia, M., & Zou, J. (2023). *FrugalGPT: How to use large language
models while reducing cost and improving performance* [Preprint]. arXiv.
https://doi.org/10.48550/arXiv.2305.05176

Dekoninck, J., Baader, M., & Vechev, M. (2024). *A unified approach to routing
and cascading for LLMs* [Preprint]. arXiv.
https://doi.org/10.48550/arXiv.2410.10347

GitHub. (n.d.). *GitHub Models billing*. Retrieved August 5, 2026, from
https://docs.github.com/en/billing/concepts/product-billing/github-models

NVIDIA. (n.d.). *Get started with NVIDIA NIM for LLMs*. Retrieved August 5,
2026, from
https://docs.nvidia.com/nim/large-language-models/1.10.0/getting-started.html

NVIDIA. (n.d.). *NVIDIA NIM model API: Free endpoint and API trial terms*.
Retrieved August 5, 2026, from https://build.nvidia.com/

Nottingham, M., & Fielding, R. (2012). *Additional HTTP status codes*
(RFC 6585). RFC Editor. https://doi.org/10.17487/RFC6585

Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E.,
Kadous, M. W., & Stoica, I. (2024). *RouteLLM: Learning to route LLMs with
preference data* [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2406.18665

OpenRouter. (n.d.-a). *Free models router*. Retrieved August 5, 2026, from
https://openrouter.ai/docs/guides/routing/routers/free-router

OpenRouter. (n.d.-b). *Free variant*. Retrieved August 5, 2026, from
https://openrouter.ai/docs/guides/routing/model-variants/free

OpenRouter. (n.d.-c). *Model fallbacks*. Retrieved August 5, 2026, from
https://openrouter.ai/docs/guides/routing/model-fallbacks

Rescorla, E., Nottingham, M., & Bishop, M. (2022). *HTTP semantics* (RFC 9110).
RFC Editor. https://doi.org/10.17487/RFC9110
Loading
Loading