feat: opt-in exact-match LLM response cache (stage 1/5)#404
feat: opt-in exact-match LLM response cache (stage 1/5)#404zhongxuanwang-nv wants to merge 3 commits into
Conversation
WalkthroughAdds an opt-in exact-match LLM response cache with canonical keying, TTL-backed in-memory or Redis storage, buffered and streaming interception, provider-native replay, telemetry, configuration validation, gateway handling, and Rust, Go, Python, and Node APIs. ChangesResponse cache feature
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant Client
participant Gateway
participant ResponseCacheIntercept
participant CacheStore
participant Provider
Client->>Gateway: submit LLM request
Gateway->>ResponseCacheIntercept: execute buffered or streaming request
ResponseCacheIntercept->>CacheStore: lookup canonical key
alt cache hit
CacheStore-->>ResponseCacheIntercept: cached response aggregate
ResponseCacheIntercept-->>Gateway: return response or replay chunks
else cache miss
ResponseCacheIntercept->>Provider: execute upstream request
Provider-->>ResponseCacheIntercept: response or stream
ResponseCacheIntercept->>CacheStore: store eligible result
ResponseCacheIntercept-->>Gateway: return live response or stream
end
Gateway-->>Client: response with appropriate content type
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
github-actions
Bot
added
lang:go
zhongxuanwang-nv
changed the title
License DiffCompared against Lockfile license changesLockfile License ChangesRustAdded
Removed
Updated/Changed
NodeAdded
Removed
Updated/Changed
|
4b575ad to
22b5fdb
Compare
22b5fdb to
17520f9
Compare
17520f9 to
043afc6
Compare
043afc6 to
b149846
Compare
b149846 to
7253a68
Compare
7253a68 to
4363be4
Compare
4363be4 to
83db5c1
Compare
An opt-in feature of the adaptive plugin (a response_cache config section, not a new plugin kind): managed LLM calls are keyed by a SHA-256 fingerprint of the normalized request and repeats are served from the store — instant, free, reproducible. Buffered and streaming calls share one keyspace; a streamed miss is teed and stored as its aggregated response, a hit replays provider-native chunks. Keying auto-detects the provider surface from the request shape and trusts the decode only where it is faithful: known-lossy shapes and decodes that fail to round-trip fall back to raw-body fingerprinting, which can only cost a miss. Stateful calls (Responses persistence, conversations, containers) and nondeterministic calls under the safety toggle bypass entirely. Stored answers must be complete: non-null errors, non-final statuses, truncated or lossily-collected streams, and upstream failures are never cached — the CLI gateway relays failed upstream replies to the client verbatim while keeping them invisible to the execution chain. The in-memory store is bounded by an honest resident-size budget with oldest-first eviction; the Redis backend runs under hard deadlines and re-checks entry expiry. Everything fails open: any cache error falls through to a live call. Same config surface in Rust, Python, Node, and Go; doctor reports configuration and backend reachability; hit/miss/ bypass marks carry fingerprints and savings, never bodies. Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
83db5c1 to
96df1d9
Compare
zhongxuanwang-nv
added
the
DO NOT MERGE
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/adaptive/src/plugin_component.rs`:
- Around line 337-355: Update validate_response_cache_backend_config_fields (or
its validate_response_cache caller) to validate Redis backend values for type,
requiring both url and key_prefix to be strings when provided. Report invalid
types through the existing ConfigDiagnostic validation path, preventing
build_redis_store from falling back to the shared prefix for mistyped key_prefix
values while preserving the canonical adaptive schema behavior.
In `@crates/adaptive/src/response_cache/intercept.rs`:
- Around line 269-291: Update the response-cache completion logic around the
collector loop and the aggregate cache condition to track each observed chat
choice index and whether that choice has reached a finish reason, requiring
every seen choice to terminate before caching. Replace the existing any-choice
completion behavior with an all-seen-choices check, while preserving
single-choice and non-chat stream behavior. Add a test covering a multi-choice
stream where one choice is truncated and verifying the aggregate is not cached.
In `@crates/adaptive/src/response_cache/key.rs`:
- Around line 424-989: Move the response-cache test modules out of source files
into the crate integration-test tree: relocate the tests under
crates/adaptive/src/response_cache/key.rs lines 424-989, replay.rs lines
264-368, and mark.rs lines 213-293 into crates/adaptive/tests/, preserving their
existing coverage and adapting imports or visibility as needed; remove the
in-source test modules after relocation.
In `@crates/adaptive/src/response_cache/replay.rs`:
- Around line 28-50: Update replay_is_lossy so an unrecognized response surface
is treated as lossy rather than returning false. Change the
detect_response_surface miss path to return true, forcing streaming replay to
use the provider instead of emitting a single aggregate frame; preserve the
existing codec comparison logic for recognized surfaces.
In `@crates/adaptive/src/runtime/validation.rs`:
- Around line 194-203: Update runtime validation and shared Redis cache setup
around config.namespace so an empty namespace cannot enable cross-tenant
response reuse by default. Report it as an error or require an explicit
unsafe-sharing opt-in, and ensure cache scope includes tenant isolation and root
UUID isolation rather than relying only on authentication-excluded request keys.
Preserve namespaced cache behavior for valid configurations.
- Around line 181-193: Update the Redis URL validation condition in the backend
configuration check to treat URLs containing only whitespace as missing:
retrieve the string from backend.config.url, trim it, and reject it when the
trimmed value is empty. Preserve acceptance of non-empty URLs and the existing
response_cache_error diagnostic.
In `@crates/adaptive/tests/integration/response_cache_benchmark_tests.rs`:
- Around line 201-205: Update the repeat and bypass benchmark assertions to
enforce a complete cache-status partition: assert hits + misses + bypasses
equals total_requests, and assert the status excluded by each scenario is zero.
Apply this in both assertion blocks so duplicate or missing lifecycle statuses
cannot pass.
In `@crates/adaptive/tests/integration/response_cache_tests.rs`:
- Around line 212-238: The integration tests need coverage for configured
nondeterministic request bypasses. Add a test alongside
stateful_responses_calls_bypass_the_cache that activates ResponseCacheConfig
with cache_nondeterministic disabled, repeats a request using a nonzero
temperature or top_p, and verifies the provider call counter is 2 and both cache
lifecycle results are marked bypass.
- Around line 1032-1074: Strengthen cache_coexists_with_acg_execution_intercept
by observing an ACG lifecycle event or request transformation during the first
cache miss, then assert the second request is served by the outer cache without
invoking the inner ACG intercept again. Keep the existing provider-count
assertion, and verify middleware ordering through the captured ACG behavior
rather than relying only on response caching.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: cfe4d74d-7700-4f8e-a185-1442a614e9d1
📒 Files selected for processing (31)
crates/adaptive/Cargo.tomlcrates/adaptive/src/config.rscrates/adaptive/src/lib.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/redis.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/runtime/features.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/src/doctor.rscrates/cli/src/gateway.rscrates/cli/tests/coverage/doctor_tests.rscrates/cli/tests/coverage/gateway_tests.rscrates/cli/tests/coverage/server_tests.rscrates/node/adaptive.d.tscrates/node/adaptive.jsgo/nemo_relay/adaptive.gogo/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive_runtime_test.gopython/nemo_relay/adaptive.pypython/nemo_relay/adaptive.pyipython/tests/test_adaptive_config.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (32)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/gateway_tests.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rscrates/cli/tests/coverage/doctor_tests.rscrates/adaptive/src/lib.rscrates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/replay.rscrates/cli/tests/coverage/server_tests.rscrates/adaptive/src/response_cache/intercept.rscrates/cli/src/doctor.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/cli/src/gateway.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/src/lib.rscrates/adaptive/Cargo.tomlcrates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/gateway_tests.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rscrates/cli/tests/coverage/doctor_tests.rscrates/adaptive/src/lib.rscrates/adaptive/tests/integration/response_cache_common.rspython/nemo_relay/adaptive.pycrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rspython/tests/test_adaptive_config.pycrates/adaptive/src/response_cache/replay.rscrates/cli/tests/coverage/server_tests.rscrates/adaptive/src/response_cache/intercept.rscrates/cli/src/doctor.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/cli/src/gateway.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/gateway_tests.rscrates/node/adaptive.jscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rscrates/cli/tests/coverage/doctor_tests.rscrates/adaptive/src/lib.rscrates/adaptive/tests/integration/response_cache_common.rscrates/node/adaptive.d.tspython/nemo_relay/adaptive.pycrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rspython/tests/test_adaptive_config.pycrates/adaptive/src/response_cache/replay.rscrates/cli/tests/coverage/server_tests.rscrates/adaptive/src/response_cache/intercept.rscrates/cli/src/doctor.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/cli/src/gateway.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/gateway_tests.rscrates/node/adaptive.jscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rsgo/nemo_relay/adaptive/adaptive.gocrates/cli/tests/coverage/doctor_tests.rscrates/adaptive/src/lib.rsgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_common.rscrates/node/adaptive.d.tspython/nemo_relay/adaptive.pycrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rsgo/nemo_relay/adaptive_runtime_test.gopython/tests/test_adaptive_config.pycrates/adaptive/src/response_cache/replay.rscrates/cli/tests/coverage/server_tests.rscrates/adaptive/src/response_cache/intercept.rscrates/cli/src/doctor.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/cli/src/gateway.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/gateway_tests.rscrates/node/adaptive.jscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rsgo/nemo_relay/adaptive/adaptive.gocrates/cli/tests/coverage/doctor_tests.rscrates/adaptive/src/lib.rsgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_common.rscrates/node/adaptive.d.tscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rsgo/nemo_relay/adaptive_runtime_test.gocrates/adaptive/src/response_cache/replay.rscrates/cli/tests/coverage/server_tests.rscrates/adaptive/src/response_cache/intercept.rscrates/cli/src/doctor.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/cli/src/gateway.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/adaptive/src/runtime/features.rscrates/adaptive/src/lib.rspython/nemo_relay/adaptive.pycrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rspython/tests/test_adaptive_config.pycrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/response_cache/intercept.rscrates/cli/src/doctor.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/cli/src/gateway.rscrates/adaptive/src/response_cache/key.rs
crates/adaptive/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep
crates/adaptivealigned with the canonical adaptive config schema, built-in section helpers, plugin lifecycle, and validation/report behavior.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/src/lib.rscrates/adaptive/Cargo.tomlcrates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/gateway_tests.rscrates/node/adaptive.jscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rsgo/nemo_relay/adaptive/adaptive.gopython/nemo_relay/adaptive.pyicrates/cli/tests/coverage/doctor_tests.rscrates/adaptive/src/lib.rscrates/adaptive/Cargo.tomlgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_common.rscrates/node/adaptive.d.tspython/nemo_relay/adaptive.pycrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rsgo/nemo_relay/adaptive_runtime_test.gopython/tests/test_adaptive_config.pycrates/adaptive/src/response_cache/replay.rscrates/cli/tests/coverage/server_tests.rscrates/adaptive/src/response_cache/intercept.rscrates/cli/src/doctor.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/cli/src/gateway.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/src/lib.rscrates/adaptive/Cargo.tomlcrates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/gateway_tests.rscrates/node/adaptive.jscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rsgo/nemo_relay/adaptive/adaptive.gocrates/cli/tests/coverage/doctor_tests.rscrates/adaptive/src/lib.rsgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_common.rscrates/node/adaptive.d.tspython/nemo_relay/adaptive.pycrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rsgo/nemo_relay/adaptive_runtime_test.gopython/tests/test_adaptive_config.pycrates/adaptive/src/response_cache/replay.rscrates/cli/tests/coverage/server_tests.rscrates/adaptive/src/response_cache/intercept.rscrates/cli/src/doctor.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/cli/src/gateway.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/redis.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/src/lib.rscrates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/tests/integration/response_cache_tests.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/cli/tests/coverage/gateway_tests.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/doctor_tests.rscrates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rsgo/nemo_relay/adaptive_runtime_test.gopython/tests/test_adaptive_config.pycrates/cli/tests/coverage/server_tests.rscrates/adaptive/tests/integration/response_cache_tests.rs
crates/node/**/*.{js,ts,jsx,tsx,json}
📄 CodeRabbit inference engine (.agents/skills/test-node-binding/SKILL.md)
Format changed Node files with
npm run format --workspace=nemo-relay-node
Files:
crates/node/adaptive.jscrates/node/adaptive.d.ts
crates/node/**/*.{js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
camelCasefor Node.js public APIs.
Files:
crates/node/adaptive.jscrates/node/adaptive.d.ts
**/*.{py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Keep Python, Go, and Node.js config objects and subscriber/exporter methods aligned so all bindings expose the same logical knobs and semantics.
Files:
crates/node/adaptive.jsgo/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive.gocrates/node/adaptive.d.tspython/nemo_relay/adaptive.pygo/nemo_relay/adaptive_runtime_test.gopython/tests/test_adaptive_config.py
crates/{python,ffi,node}/**/*
⚙️ CodeRabbit configuration file
crates/{python,ffi,node}/**/*: Treat binding changes as public API changes. Check for parity with the other language bindings, FFI ownership/lifetime safety,
callback error propagation, stable type conversion, and consistent async/stream semantics.
Flag changes that update one binding without corresponding tests or documentation for the same surface elsewhere.
Files:
crates/node/adaptive.jscrates/node/adaptive.d.ts
go/nemo_relay/**/*.go
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
go/nemo_relay/**/*.go: Format changed Go packages withcd go/nemo_relay && go fmt ./...
Run Go tests withjust test-goto build and test the NeMo Relay Go binding
Usejust build-gowhen you want an explicit build-only pass or need the artifact for other work
Usejust ci=true test-gowhen you need the CI-style coverage and JUnit path
On macOS, setDYLD_LIBRARY_PATHto the../../target/releasedirectory before running the rawgo testcommand directlyUse
PascalCasefor public Go APIs.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive.gogo/nemo_relay/adaptive_runtime_test.go
**/*.go
📄 CodeRabbit inference engine (CONTRIBUTING.md)
When changing the experimental Go binding, format Go code with
gofmtand keepgo vet ./...passing.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive.gogo/nemo_relay/adaptive_runtime_test.go
{crates/python/src/py_api/mod.rs,python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go,crates/node/src/api/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update the language-native bindings for every exposed surface in Python, Go, and Node.js.
Files:
go/nemo_relay/adaptive/adaptive.gopython/nemo_relay/adaptive.pyigo/nemo_relay/adaptive.gopython/nemo_relay/adaptive.pygo/nemo_relay/adaptive_runtime_test.go
{python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update language wrapper helpers such as Python wrapper modules, Python type stubs, and Go shorthand packages when the new behavior belongs in those helper layers.
Files:
go/nemo_relay/adaptive/adaptive.gopython/nemo_relay/adaptive.pyigo/nemo_relay/adaptive.gopython/nemo_relay/adaptive.pygo/nemo_relay/adaptive_runtime_test.go
go/nemo_relay/adaptive/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep Go adaptive helpers under
go/nemo_relay/adaptivesynchronized with the shared adaptive config model and lifecycle.
Files:
go/nemo_relay/adaptive/adaptive.go
go/nemo_relay/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep shared plugin helpers in
go/nemo_relayaligned with plugin registration, composition, and lifecycle behavior.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive.gogo/nemo_relay/adaptive_runtime_test.go
go/nemo_relay/**/*
⚙️ CodeRabbit configuration file
go/nemo_relay/**/*: Review Go binding changes for cgo memory ownership, race safety, callback cleanup, idiomatic exported APIs, and parity with Rust/FFI behavior.
Any API change should include focused Go tests and consider race-test behavior.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive.gogo/nemo_relay/adaptive_runtime_test.go
python/nemo_relay/**/*
⚙️ CodeRabbit configuration file
python/nemo_relay/**/*: Review Python wrapper changes for typed API consistency, contextvars-based scope isolation, async behavior, and parity with the native extension.
Stubs and runtime implementations should stay aligned.
Files:
python/nemo_relay/adaptive.pyipython/nemo_relay/adaptive.py
**/Cargo.toml
📄 CodeRabbit inference engine (.agents/skills/prepare-code-freeze/SKILL.md)
Confirm or infer the target release version from
upstream/main:Cargo.toml. Derive the release branch asrelease/<major>.<minor>.Keep Rust package names and workspace metadata in
Cargo.tomlinternally consistent across the project.
Files:
crates/adaptive/Cargo.toml
**/*.toml
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all TOML files using the
#comment form.
Files:
crates/adaptive/Cargo.toml
crates/node/**/*.{ts,tsx,d.ts}
📄 CodeRabbit inference engine (.agents/skills/test-node-binding/SKILL.md)
Use
npm run check:docstrings --workspace=nemo-relay-nodeto validate public API docstring checks when surface docs changed
Files:
crates/node/adaptive.d.ts
python/nemo_relay/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Python wrapper modules live under
python/nemo_relay/, and the native extension is built fromcrates/pythonwithmaturin.
Files:
python/nemo_relay/adaptive.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: When changing the Python wrapper package, tests, or docs tooling, lint with Ruff (E,F,W,I), format with Ruff formatter (120-character lines, double quotes), and passtytype checking.
Add the SPDX license header to all Python source files using the#comment form.
Files:
python/nemo_relay/adaptive.pypython/tests/test_adaptive_config.py
python/nemo_relay/{adaptive.py,plugin.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep Python adaptive/plugin wrappers in
python/nemo_relay/adaptive.pyandpython/nemo_relay/plugin.pysynchronized with the shared adaptive/plugin boundary and lifecycle.
Files:
python/nemo_relay/adaptive.py
python/tests/**/*.py
📄 CodeRabbit inference engine (.agents/skills/test-python-binding/SKILL.md)
python/tests/**/*.py: Pytest is used to run tests.
Do not add@pytest.mark.asyncioto any test; async tests are automatically detected and run by the async runner.
Do not add a-> Nonereturn type annotation to test functions.
When mocking a class, do not define a new class; useunittest.mock.MagicMockorunittest.mock.AsyncMock, with thespecconstructor argument when necessary.
Name mocked classes with themockprefix, notfake.
Prefer pytest fixtures over helper methods.
Do not repeat fixtures; if a fixture is needed in multiple test files, place it in aconftest.pyfile.
When creating a fixture, use@pytest.fixture(name="<fixture_name>"[, scope="<scope>"])and define the fixture function asdef <fixture_name>_fixture() -> <return_type>:; only specifyscopewhen it is notfunction.
Preferpytest.mark.parametrizeover creating individual tests for different input types.
Files:
python/tests/test_adaptive_config.py
🧠 Learnings (1)
📚 Learning: 2026-05-07T18:04:44.387Z
Learnt from: mnajafian-nv
Repo: NVIDIA/NeMo-Flow PR: 67
File: integrations/openclaw/src/modules.ts:1-2
Timestamp: 2026-05-07T18:04:44.387Z
Learning: In NVIDIA/NeMo-Flow, TypeScript source files should use `//` line comments for SPDX headers (e.g., `// SPDX-FileCopyrightText: ...` and `// SPDX-License-Identifier: ...`) rather than C-style block comments (`/* ... */`). The repo’s copyright checker enforces this mapping, so `//` SPDX headers in `.ts` files should not be flagged as a style violation.
Applied to files:
crates/node/adaptive.d.ts
🔇 Additional comments (34)
go/nemo_relay/adaptive_runtime_test.go (1)
172-237: LGTM!python/tests/test_adaptive_config.py (1)
18-18: LGTM!Also applies to: 160-199
crates/adaptive/src/response_cache/key.rs (1)
1-423: LGTM!crates/adaptive/src/response_cache/intercept.rs (1)
1-268: LGTM!Also applies to: 292-338, 353-456
crates/adaptive/src/response_cache/replay.rs (1)
1-27: LGTM!Also applies to: 51-263
crates/adaptive/src/response_cache/mark.rs (1)
1-212: LGTM!crates/cli/src/gateway.rs (1)
283-292: LGTM!Also applies to: 322-374, 388-388, 405-413, 455-478, 563-575
crates/cli/tests/coverage/gateway_tests.rs (1)
455-464: LGTM!crates/cli/tests/coverage/server_tests.rs (1)
18-19: LGTM!Also applies to: 100-106, 2692-3008
crates/adaptive/Cargo.toml (1)
25-25: LGTM!Also applies to: 37-37, 50-57
crates/node/adaptive.d.ts (1)
55-67: LGTM!Also applies to: 77-77, 261-274
crates/node/adaptive.js (1)
150-178: LGTM!Also applies to: 237-237
go/nemo_relay/adaptive.go (1)
20-20: LGTM!Also applies to: 70-112, 185-216
go/nemo_relay/adaptive/adaptive.go (1)
55-60: LGTM!Also applies to: 122-136
crates/adaptive/src/runtime/validation.rs (1)
7-10: LGTM!Also applies to: 86-180, 204-236
crates/adaptive/src/redis.rs (1)
34-50: LGTM!Also applies to: 75-75
crates/cli/src/doctor.rs (1)
21-22: LGTM!Also applies to: 705-711, 742-821
crates/cli/tests/coverage/doctor_tests.rs (1)
1005-1040: LGTM!Also applies to: 1042-1067, 1069-1103
crates/adaptive/src/config.rs (1)
10-11: LGTM!Also applies to: 37-40, 56-56, 194-242, 313-319, 392-410
crates/adaptive/src/lib.rs (1)
33-34: LGTM!Also applies to: 46-47, 56-56
crates/adaptive/src/response_cache/config.rs (1)
1-56: LGTM!crates/adaptive/src/response_cache/mod.rs (1)
1-27: LGTM!python/nemo_relay/adaptive.py (1)
254-301: LGTM!Also applies to: 314-314, 329-342, 439-439
python/nemo_relay/adaptive.pyi (1)
184-216: LGTM!Also applies to: 229-240
crates/adaptive/src/plugin_component.rs (1)
185-185: LGTM!Also applies to: 288-332
crates/adaptive/src/response_cache/store.rs (1)
1-683: LGTM!crates/adaptive/src/runtime/features.rs (1)
37-45: LGTM!Also applies to: 477-482, 777-819
crates/adaptive/tests/integration/response_cache_benchmark_tests.rs (2)
1-200: LGTM!
206-268: LGTM!Also applies to: 273-367
crates/adaptive/tests/integration/response_cache_common.rs (1)
1-58: LGTM!crates/adaptive/tests/integration/response_cache_tests.rs (3)
1-211: LGTM!
239-1031: LGTM!
1075-1134: LGTM!crates/adaptive/tests/unit/config_tests.rs (1)
93-93: LGTM!Also applies to: 122-136
…-stage1 Resolve conflicts with the upstream module splits and config changes: - crates/adaptive/src/redis.rs: keep the shared connect() helper and move the upstream plugin-resource logging into it so both RedisBackend and the response-cache store emit the operational logging events. - crates/cli/src/diagnostics/mod.rs (was doctor.rs): keep only the imports the merged module still uses. - crates/cli/src/gateway/mod.rs (was gateway.rs): adopt the upstream session_finish flag while preserving the intercept short-circuit body handling and upstream-failure relay. - Switch response-cache integration tests to initialize_plugins_exact so ambient plugins.toml discovery cannot leak into the asserted config (observability config v2 now rejects legacy ATOF fields as errors). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
crates/adaptive/src/config.rs (1)
221-232: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winDisable nondeterministic caching by default.
cache_nondeterministic: truemakes an otherwise default response-cache configuration replay stochastic provider output. This conflicts with the stated bypass rule for nondeterministic calls and can return stale responses for later equivalent requests. Default this tofalse; callers can explicitly opt in.Proposed fix
- cache_nondeterministic: true, + cache_nondeterministic: false,🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/adaptive/src/config.rs` around lines 221 - 232, Update the Default implementation for the configuration so cache_nondeterministic is false instead of true. Keep the existing explicit opt-in behavior available through the configuration field and leave all other default values unchanged.crates/adaptive/src/runtime/features.rs (1)
488-490: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winFail open when optional cache-store initialization fails.
ResponseCacheFeature::registerpropagatesbuild_store(...).await?, andregister_featurethen rolls back the runtime. An unavailable Redis cache therefore prevents live provider calls entirely, contradicting the fail-open cache contract. DowngradeAdaptiveError::Storagefrom cache-store construction to a warning and skip cache interceptor registration; continue propagating invalid configuration errors.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/adaptive/src/runtime/features.rs` around lines 488 - 490, The response-cache setup around ResponseCacheFeature::register and register_feature must fail open when cache-store construction returns AdaptiveError::Storage: log a warning, skip registering the cache interceptor, and allow live provider calls to continue. Preserve propagation of invalid configuration errors and do not roll back the runtime for storage initialization failures.crates/adaptive/tests/integration/response_cache_benchmark_tests.rs (1)
300-317: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAssert that each run produces the expected five hits.
Equality alone passes when both runs return zero because caching never activated.
Proposed fix
let first_run = run_once("bench_determinism_a").await; let second_run = run_once("bench_determinism_b").await; + assert_eq!(first_run, 5, "the first workload must cache all five repeats"); + assert_eq!(second_run, 5, "the fresh cache must cache all five repeats"); assert_eq!( first_run, second_run,As per path instructions, “Tests should cover the behavior promised by the changed API surface.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/adaptive/tests/integration/response_cache_benchmark_tests.rs` around lines 300 - 317, Update the determinism test around run_once so each invocation asserts that its hits count equals the expected five hits, in addition to retaining the comparison between first_run and second_run. This must fail when caching is inactive and still verify that results do not leak across plugin re-initialization.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@crates/adaptive/src/config.rs`:
- Around line 221-232: Update the Default implementation for the configuration
so cache_nondeterministic is false instead of true. Keep the existing explicit
opt-in behavior available through the configuration field and leave all other
default values unchanged.
In `@crates/adaptive/src/runtime/features.rs`:
- Around line 488-490: The response-cache setup around
ResponseCacheFeature::register and register_feature must fail open when
cache-store construction returns AdaptiveError::Storage: log a warning, skip
registering the cache interceptor, and allow live provider calls to continue.
Preserve propagation of invalid configuration errors and do not roll back the
runtime for storage initialization failures.
In `@crates/adaptive/tests/integration/response_cache_benchmark_tests.rs`:
- Around line 300-317: Update the determinism test around run_once so each
invocation asserts that its hits count equals the expected five hits, in
addition to retaining the comparison between first_run and second_run. This must
fail when caching is inactive and still verify that results do not leak across
plugin re-initialization.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 98963021-fc91-4a67-99e0-d4ee10c28aaa
📒 Files selected for processing (13)
crates/adaptive/Cargo.tomlcrates/adaptive/src/config.rscrates/adaptive/src/redis.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/src/diagnostics/mod.rscrates/cli/src/gateway/mod.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/tests/coverage/shared/server_tests.rs
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (16)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/cli/src/gateway/mod.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/config.rscrates/adaptive/Cargo.tomlcrates/adaptive/src/redis.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/cli/src/gateway/mod.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/cli/src/gateway/mod.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/cli/src/gateway/mod.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/cli/src/gateway/mod.rs
crates/adaptive/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep
crates/adaptivealigned with the canonical adaptive config schema, built-in section helpers, plugin lifecycle, and validation/report behavior.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/config.rscrates/adaptive/Cargo.tomlcrates/adaptive/src/redis.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/Cargo.tomlcrates/adaptive/src/redis.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/cli/src/gateway/mod.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/config.rscrates/adaptive/Cargo.tomlcrates/adaptive/src/redis.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/cli/src/gateway/mod.rs
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}: Expose the new middleware surface with parity in every affected language binding.
Add focused tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback error propagation, scope-local inheritance and cleanup, event input/output semantics, and binding parity.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/redis.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/cli/src/gateway/mod.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/config.rscrates/adaptive/src/redis.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/adaptive/tests/integration/response_cache_common.rscrates/adaptive/tests/unit/config_tests.rscrates/cli/tests/coverage/shared/gateway_tests.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/cli/tests/coverage/shared/server_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/cli/src/diagnostics/mod.rscrates/adaptive/src/config.rscrates/adaptive/src/redis.rscrates/adaptive/src/runtime/features.rscrates/cli/src/gateway/mod.rs
**/Cargo.toml
📄 CodeRabbit inference engine (.agents/skills/prepare-code-freeze/SKILL.md)
Confirm or infer the target release version from
upstream/main:Cargo.toml. Derive the release branch asrelease/<major>.<minor>.Keep Rust package names and workspace metadata in
Cargo.tomlinternally consistent across the project.
Files:
crates/adaptive/Cargo.toml
**/*.toml
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all TOML files using the
#comment form.
Files:
crates/adaptive/Cargo.toml
🔇 Additional comments (11)
crates/adaptive/src/config.rs (1)
40-40: 📐 Maintainability & Code QualityVerify the required validation matrix.
No validation output is provided for these Rust and
crates/adaptivechanges. Please runcargo fmt --all,cargo clippy --workspace --all-targets -- -D warnings,just test-rust, and the repository’s Rust/Python/Go/Node matrix before handoff. As per coding guidelines, “Ifcrates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.”Source: Coding guidelines
crates/adaptive/tests/integration/response_cache_tests.rs (3)
212-238: Configured nondeterministic bypass coverage is still missing.This was already flagged in the previous review.
1033-1074: The test still does not prove that the ACG intercept executes or that middleware ordering is correct.This was already flagged in the previous review.
123-210: LGTM!Also applies to: 240-273
crates/cli/src/diagnostics/mod.rs (1)
30-31: LGTM!Also applies to: 532-538, 569-570
crates/cli/src/gateway/mod.rs (1)
180-187: LGTM!Also applies to: 219-242, 251-277, 291-291, 310-318, 360-383, 472-484
crates/cli/tests/coverage/shared/doctor_tests.rs (1)
1050-1084: LGTM!Also applies to: 1086-1111, 1113-1147
crates/cli/tests/coverage/shared/gateway_tests.rs (1)
15-15: LGTM!Also applies to: 911-920
crates/cli/tests/coverage/shared/server_tests.rs (1)
18-19: LGTM!Also applies to: 101-107, 3168-3223, 3225-3289, 3291-3350, 3352-3416, 3418-3483
crates/adaptive/tests/integration/response_cache_common.rs (2)
34-44: LGTM!
34-39: 📐 Maintainability & Code QualityConfirm the mandatory Rust and cross-language validation completed.
Please verify
cargo fmt --all, strict workspace Clippy,just test-rust, the full Rust/Python/Go/Node matrix, and the final all-files pre-commit pass.As per coding guidelines, “Changes to
crates/coreorcrates/adaptivemust run the full language matrix.”Source: Coding guidelines
- Track per-choice completion in the streaming tee: a chat stream is terminal only when every choice that appeared carries a finish_reason, so a clean close after only some choices finished is a truncation and is never cached. - Treat an unrecognized aggregate shape as lossy for the streaming hit tier: with no native chunk synthesis, the tier runs live instead of serving one aggregate-shaped frame; buffered hits are unaffected. - Reject a wrong-typed redis key_prefix (it silently fell back to the shared default prefix) and treat a blank redis url as missing. - Assert the hit/miss/bypass partition exhaustively in the benchmark accounting tests. - Move the response-cache unit tests out of the src tree into tests/unit/response_cache/, matching the crate's #[path]-included test file convention. Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/adaptive/src/response_cache/store.rs`:
- Around line 500-501: Before handoff, validate the response_cache store test
integration around the tests module by running cargo fmt --all, cargo clippy
--workspace --all-targets -- -D warnings, just test-rust, the complete
Rust/Python/Go/Node test matrix, and uv run pre-commit run --all-files; resolve
any failures before completing the change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 57ce3305-e947-4d3d-9462-b12f292c63c9
📒 Files selected for processing (12)
crates/adaptive/src/response_cache/intercept.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/tests/unit/response_cache/store_tests.rs
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (14)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
crates/adaptive/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep
crates/adaptivealigned with the canonical adaptive config schema, built-in section helpers, plugin lifecycle, and validation/report behavior.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}: Expose the new middleware surface with parity in every affected language binding.
Add focused tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback error propagation, scope-local inheritance and cleanup, event input/output semantics, and binding parity.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/response_cache/store.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/adaptive/tests/unit/response_cache/mark_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/replay.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/intercept.rscrates/adaptive/src/response_cache/store.rs
🔇 Additional comments (16)
crates/adaptive/src/runtime/validation.rs (2)
7-10: LGTM!Also applies to: 86-91, 99-159, 160-163, 167-193, 205-247
164-166: 🎯 Functional CorrectnessRust 1.86+ already supports
Option::is_none_or, so this is fine.> Likely an incorrect or invalid review comment.crates/adaptive/src/response_cache/store.rs (1)
478-483: 🗄️ Data Integrity & IntegrationCheck
key_prefixhandling inbuild_store. A present non-stringbackend.config.key_prefixshould not fall back tonemo-relay:llm-cache:; ifcheck_backend_healthreaches this path directly, the default can hide malformed config and merge Redis keyspaces.crates/adaptive/src/response_cache/mark.rs (1)
214-215: LGTM!crates/adaptive/tests/integration/response_cache_benchmark_tests.rs (1)
206-209: LGTM!Also applies to: 277-278
crates/adaptive/tests/integration/response_cache_tests.rs (1)
921-935: LGTM!Also applies to: 965-994
crates/adaptive/tests/unit/response_cache/key_tests.rs (1)
1-567: LGTM!crates/adaptive/tests/unit/response_cache/mark_tests.rs (2)
1-34: LGTM!Also applies to: 53-83
35-52: 🩺 Stability & AvailabilityScope the pricing-resolver override to this test. If the resolver is shared process-wide, wrap this setup/reset in the repo’s pricing test lock and an RAII guard so a panic can’t leak the override into later tests.
crates/adaptive/tests/unit/response_cache/replay_tests.rs (2)
1-8: LGTM!Also applies to: 24-110
9-23: 🎯 Functional CorrectnessNo change needed.
system_fingerprint,service_tier, andlogprobsare buffered-response metadata, not streamed chunk content, so leaving them out here does not make the replay lossy.> Likely an incorrect or invalid review comment.crates/adaptive/tests/unit/response_cache/store_tests.rs (1)
1-187: LGTM!crates/adaptive/src/response_cache/intercept.rs (2)
19-19: LGTM!Also applies to: 269-277, 290-299, 338-373
255-302: 📐 Maintainability & Code QualityRun the required validation matrix before handoff.
Please run
just test-rust,cargo fmt --all,cargo clippy --workspace --all-targets -- -D warnings, the full Rust/Python/Go/Node matrix, anduv run pre-commit run --all-files; no results are included in the review context.As per coding guidelines, Rust changes in
crates/adaptiverequire the full language matrix and pre-commit validation before review or handoff.Source: Coding guidelines
crates/adaptive/src/response_cache/key.rs (1)
425-426: LGTM!crates/adaptive/src/response_cache/replay.rs (1)
19-21: LGTM!Also applies to: 34-39, 267-268
1 participant
Overview
First stage of a five-stage series (exact match → logical keys → tool-result cache → semantic → streaming semantic), each stage a single reviewable commit with a matching docs PR. This stage is fully standalone.
Adds an opt-in, exact-match response cache for managed LLM calls as a
response_cachesection of the adaptive plugin config (no new plugin kind, no call-site changes). Repeated identical calls are served from an in-memory or Redis store instead of re-running the provider — instant repeats in dev loops and evals, and reproducible runs via thebypass_ratetrust dial. Off by default; every cache error fails open to a live call.Details
crates/adaptive/src/response_cache/key.rs): requests are keyed by a SHA-256 fingerprint of the RFC 8785-canonicalized, normalized request. The provider surface is auto-detected from the request shape (nothing to configure); the decode is trusted only where faithful — known-lossy shapes and decodes that fail a structural round-trip (tools, conversation, tool_choice, generation params) fall back to raw-body fingerprinting, which can only cost a miss, never a wrong reuse. Tool-call IDs are normalized; only allowlisted headers are keyed; namespace/provider/schema-version are folded in. Requests carrying integers beyond ±2^53 bypass (RFC 8785 serializes numbers as f64, so distinct IDs could share one key), as do Responses prompt-template references (server-side state); wrong-typedparallel_tool_calls/top_logprobsfall back to raw keying like the other guarded scalars.response_cache/intercept.rs): only complete, replayable answers are stored — non-nullerrorbodies, non-finalstatus, truncated streams (no protocol-terminal event), and lossily-collected aggregates (thinking blocks, refusal-only choices) are never cached. Stateful calls (Responses persist-by-default withoutstore: false,conversation,container) and nondeterministic calls undercache_nondeterministic: falsebypass entirely.response_cache/store.rs): in-memory backend bounded by an honest resident-size budget with oldest-first eviction; Redis backend (feature-gated) with hard per-op deadlines, native expiry plus an embedded-expiry re-check.crates/cli/src/gateway.rs): non-2xx / non-JSON upstream replies flow back asErrso nothing downstream can cache them, while the client still receives the upstream response verbatim. The verbatim relay is gated on the upstream exchange itself having failed, so a runtime rejection after a successful call (e.g. an execution intercept blocking a response) surfaces as the translated error rather than a passthrough of the blocked body. Short-circuited hits carry correct content types.skip_keys, and auth headers inheader_allowlist. Same typed config in Rust, Python, Node, and Go;nemo-relay doctorreports configuration and backend reachability.response_cachehit/miss/bypass marks carry key fingerprints and savings, never bodies. Entries are stored unredacted (PII sanitize guardrails rewrite telemetry only), so shared Redis deployments must be trusted and namespaced — documented.Validation run on this branch:
cargo test -p nemo-relay-adaptive --lib --tests --features nemo-relay/openinference(all targets green, ~550 tests incl. integration + benchmark suites), gateway HTTP cache tests (server::tests::gateway_*, real mock upstream),cargo fmt --all --check,cargo clippy -- -D warningson both crates. Python/Go surfaces are config mirrors validated by the shared Rust validator; their suites cover round-trip + validation parity. Known pre-existing sandbox failures (plugin-shim sidecar tests) are unrelated.Where should the reviewer start?
crates/adaptive/src/response_cache/key.rs— the key-derivation contract and the lossy-decode guards (the core correctness decisions: when to trust the normalized decode vs fall back to raw keying).crates/adaptive/tests/integration/response_cache_tests.rs— the behavioral contract as tests: exact repeats hit, guards force misses instead of wrong reuses, streams store only on terminal events.crates/cli/src/gateway.rs— the upstream-failure boundary (design decision: upstream failures relayed verbatim but invisible to the execution chain; post-success runtime rejections surface as errors).Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
codec::resolve:request_codec,streaming_codec,response_codec,detect_response_surface,ProviderSurface::codec_name); no local surface→codec dispatch remains in the feature.Summary by CodeRabbit
Content-Typehandling for cached responses.