docs: document the opt-in response cache (stage 1/5)#405
docs: document the opt-in response cache (stage 1/5)#405zhongxuanwang-nv wants to merge 1 commit into
Conversation
WalkthroughThe Adaptive plugin documentation now covers opt-in response caching, including configuration, runtime setup, cache eligibility and replay, key construction, observability, configurable fields, storage warnings, and validation failures. ChangesAdaptive response cache
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
github-actions
Bot
added
size:XXL
License DiffCompared against Lockfile license changesLockfile License ChangesRustAdded
Removed
Updated/Changed
NodeAdded
Removed
Updated/Changed
PythonAdded
Removed
Updated/Changed
Status output |
be2d512 to
fa77ecb
Compare
fa77ecb to
1e7cafa
Compare
1e7cafa to
c381f1f
Compare
c381f1f to
d1089ab
Compare
d1089ab to
f369cd2
Compare
f369cd2 to
bb99464
Compare
bb99464 to
2b4a5e8
Compare
github-actions
Bot
added
size:M
Adds a Response Cache page to the adaptive plugin docs: when to use it, a plugins.toml quickstart, per-language plugin configuration, the complete-answers-only storage rules with streaming aggregation and native replay, exact-request key normalization, the response_cache mark and doctor surfaces, a field reference, and the unredacted-at-rest caveat for shared Redis deployments. Links the page from the adaptive About and Configuration pages. Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
2b4a5e8 to
47df001
Compare
zhongxuanwang-nv
added
the
DO NOT MERGE
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/configure-plugins/adaptive/about.mdx`:
- Around line 55-56: The adaptive documentation link changes require link
validation. Run just docs-linkcheck and fix any broken links across
docs/configure-plugins/adaptive/about.mdx lines 55-56,
docs/configure-plugins/adaptive/configuration.mdx lines 38-40, and
docs/configure-plugins/adaptive/response-cache.mdx lines 16-20; make no direct
changes unless the check identifies invalid links.
In `@docs/configure-plugins/adaptive/response-cache.mdx`:
- Around line 10-14: Update the response-cache description to avoid
unconditional guarantees that repeats are instant, free, or always served from
storage. State that cache reuse can avoid the provider call when a valid entry
is available, while preserving that expiry, bypass settings, cache errors, and
store latency may result in a live request or additional cost; retain the claim
that cached responses preserve usage fields and response shape.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 5a728a28-1e7e-4002-9e0c-683197ab1efb
📒 Files selected for processing (3)
docs/configure-plugins/adaptive/about.mdxdocs/configure-plugins/adaptive/configuration.mdxdocs/configure-plugins/adaptive/response-cache.mdx
📜 Review details
🧰 Additional context used
📓 Path-based instructions (7)
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
In MDX files, top-of-file comments must use JSX comment delimiters (
{/*to open and*/}to close); do not use HTML comments for MDX SPDX headers
Files:
docs/configure-plugins/adaptive/about.mdxdocs/configure-plugins/adaptive/configuration.mdxdocs/configure-plugins/adaptive/response-cache.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Update
README.md,fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.
**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such asRELEASING.md, not as user-facing docs pages orCHANGELOG.md
Keep stable user-facing wrappers atscripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pagesIf links in documentation change, run
just docs-linkcheck.
Files:
docs/configure-plugins/adaptive/about.mdxdocs/configure-plugins/adaptive/configuration.mdxdocs/configure-plugins/adaptive/response-cache.mdx
**/*.{md,markdown,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.
Files:
docs/configure-plugins/adaptive/about.mdxdocs/configure-plugins/adaptive/configuration.mdxdocs/configure-plugins/adaptive/response-cache.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/configure-plugins/adaptive/about.mdxdocs/configure-plugins/adaptive/configuration.mdxdocs/configure-plugins/adaptive/response-cache.mdx
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
docs/configure-plugins/adaptive/about.mdxdocs/configure-plugins/adaptive/configuration.mdxdocs/configure-plugins/adaptive/response-cache.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/configure-plugins/adaptive/about.mdxdocs/configure-plugins/adaptive/configuration.mdxdocs/configure-plugins/adaptive/response-cache.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/configure-plugins/adaptive/about.mdxdocs/configure-plugins/adaptive/configuration.mdxdocs/configure-plugins/adaptive/response-cache.mdx
🔇 Additional comments (3)
docs/configure-plugins/adaptive/about.mdx (1)
25-25: LGTM!docs/configure-plugins/adaptive/response-cache.mdx (2)
1-7: LGTM!
177-183: 🎯 Functional CorrectnessKeep
wait_for_idle()as a direct call
wait_for_idle()is synchronous in the Python API, so this example should notawaitit.> Likely an incorrect or invalid review comment.
| - [Response Cache](/configure-plugins/adaptive/response-cache) explains the opt-in LLM response cache: | ||
| turning it on, what gets cached, and how savings are reported. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial
🧩 Analysis chain
🏁 Script executed:
just docs-linkcheckRepository: NVIDIA/NeMo-Relay
Length of output: 197
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n --hidden --glob '!/.git' 'docs-linkcheck|linkcheck|justfile|Justfile' .Repository: NVIDIA/NeMo-Relay
Length of output: 2517
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n --hidden --glob '!/.git' 'docs-linkcheck|linkcheck' . || trueRepository: NVIDIA/NeMo-Relay
Length of output: 1034
Run just docs-linkcheck for the adaptive docs link changes.
📍 Affects 3 files
docs/configure-plugins/adaptive/about.mdx#L55-L56(this comment)docs/configure-plugins/adaptive/configuration.mdx#L38-L40docs/configure-plugins/adaptive/response-cache.mdx#L16-L20
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/configure-plugins/adaptive/about.mdx` around lines 55 - 56, The adaptive
documentation link changes require link validation. Run just docs-linkcheck and
fix any broken links across docs/configure-plugins/adaptive/about.mdx lines
55-56, docs/configure-plugins/adaptive/configuration.mdx lines 38-40, and
docs/configure-plugins/adaptive/response-cache.mdx lines 16-20; make no direct
changes unless the check identifies invalid links.
Source: Coding guidelines
| Use the response cache when the same LLM request is made more than once and the | ||
| repeat should be served from a store instead of paying for the call again. A | ||
| repeat of an identical request returns the earlier answer instantly, at no | ||
| cost, and unchanged — usage fields intact, so a cached reuse is | ||
| shape-identical to a live call. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Avoid unconditional “instant” and “no cost” guarantees.
A matching request can still run live due to expiry, bypass_rate, or fail-open cache errors; Redis hits also have store latency/cost. Describe avoidance of the provider call conditionally.
Proposed wording
- repeat of an identical request returns the earlier answer instantly, at no
- cost, and unchanged — usage fields intact, so a cached reuse is
- shape-identical to a live call.
+ an eligible request matching an unexpired cache entry is served from the
+ store without calling the provider. Cached responses preserve the stored
+ response shape and usage fields.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Use the response cache when the same LLM request is made more than once and the | |
| repeat should be served from a store instead of paying for the call again. A | |
| repeat of an identical request returns the earlier answer instantly, at no | |
| cost, and unchanged — usage fields intact, so a cached reuse is | |
| shape-identical to a live call. | |
| Use the response cache when the same LLM request is made more than once and the | |
| repeat should be served from a store instead of paying for the call again. An | |
| eligible request matching an unexpired cache entry is served from the store | |
| without calling the provider. Cached responses preserve the stored response | |
| shape and usage fields. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/configure-plugins/adaptive/response-cache.mdx` around lines 10 - 14,
Update the response-cache description to avoid unconditional guarantees that
repeats are instant, free, or always served from storage. State that cache reuse
can avoid the provider call when a valid entry is available, while preserving
that expiry, bypass settings, cache errors, and store latency may result in a
live request or additional cost; retain the claim that cached responses preserve
usage fields and response shape.
1 participant
Overview
Documents the opt-in response cache on the documentation site: a new Response Cache page in the Adaptive plugin section, written in plain language with runnable examples — when to use it, a
plugins.tomlquickstart, per-language plugin configuration (Python/Node.js/Rust), what gets cached (complete answers only, streaming aggregation and native replay), how cache keys are built, observability (nemo-relay doctor,response_cachemarks), a full field reference, and the unredacted-at-rest caveat for shared Redis deployments. Docs PR 1/5 of the response-cache series.Details
docs/configure-plugins/adaptive/response-cache.mdx(position 5 in the Adaptive plugin section), following the ACG page skeleton: intro → When To Use It →plugins.tomlexample → Plugin Configuration tabs → Manual API tabs → What Gets Cached → Cache Keys → Observability → Fields → Common Validation Failures.docs/configure-plugins/adaptive/about.mdx(coordination list and Pages list) anddocs/configure-plugins/adaptive/configuration.mdx(Component Shape table and area-pages paragraph).fern checkandfern docs broken-links --strict(the only reported broken links are the pre-existing generated API-reference pages, regenerated byjust docs).Where should the reviewer start?
docs/configure-plugins/adaptive/response-cache.mdx— the page is the whole change (one commit, docs files only).Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit