feat(observability): OpenAI answer spend snapshot on the deep health probe#585
Conversation
…probe
Wave 2a of the repo-productivity program. Turns the already-persisted per-answer
token counts (rag_retrieval_logs.metadata.answer.tokens) into a cost signal so a
budget regression — like the reasoning-token starvation incident, or a pricing/
traffic change — is visible before the bill arrives.
- src/lib/observability/spend-metrics.ts: spendSnapshot() aggregates answer-path
token usage over a trailing window, derives USD from a configurable per-token
price, and splits by route (fast/strong) and model, with a 24h projection and
an alert flag. Zero-migration (reads existing JSONB); reasoning tokens surfaced
but billed within output (never double-counted); cached input billed at the
cheaper rate. Throws on query error so the caller nulls the block.
- health-response.ts: optional `spend` block gated exactly like `slo`/`cache`
(token-authorized deep probe + healthy Supabase, errors → null, never flips
liveness). Suppressible via includeSpend: false.
- env.ts: OPENAI_PRICE_{INPUT,CACHED_INPUT,OUTPUT}_PER_MTOK (operator-tunable
placeholders) + SPEND_ALERT_DAILY_USD.
- tests/spend-metrics.test.ts: 9 offline tests (cost math, route/model split,
daily projection + alert, empty window, truncated sample, error path).
Verified: 20/20 vitest (spend + health-route + answer-slo), tsc --noEmit, eslint,
prettier — all green. No live provider calls.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds OpenAI pricing and spend-alert environment settings, implements token-based spend aggregation, and exposes spend snapshots through authorized deep health probes with test coverage. ChangesAnswer Spend Telemetry
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant HealthProbe
participant SpendSnapshot
participant Supabase
HealthProbe->>Supabase: validate deep health state
HealthProbe->>SpendSnapshot: request configured spend snapshot
SpendSnapshot->>Supabase: query retrieval log token metadata
Supabase-->>SpendSnapshot: return rows
SpendSnapshot-->>HealthProbe: return aggregated spend
HealthProbe-->>HealthProbe: serialize spend in response
🚥 Pre-merge checks | ✅ 5 | ❌ 6❌ Failed checks (6 inconclusive)
✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 323d9cb5d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex resolve actionable Codex review findings for this pull request and current head using the repository instructions. This is the pull request's single automatic repair pass: do not perform a fresh review, create new standalone findings, or request another review. Work only the existing unresolved Codex threads on the current head. Always fix P0 and P1 findings. For P2 and lower findings, fix only clear, scoped, low-risk issues; otherwise disposition them with a concise reason. After fixing or dispositioning a thread, reply in that thread with as the first line, followed by a concise summary; that marker authorizes the workflow to close that exact thread. If human input or new authorization is required, do not use the marker and leave the thread open with the blocker. Finish only after every actionable thread is fixed or dispositioned and closed, or explicitly left open for a human decision. Do not update the branch from main, address unrelated reviews, broaden scope, or create more than one scoped fix commit. Do not use external APIs, paid services, credentials, dependency changes, or broad refactors unless explicitly authorized. Add targeted tests where behavior changes and run the narrowest relevant validation. |
|
To use Codex here, create a Codex account and connect to github. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/lib/observability/spend-metrics.ts (1)
142-147: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd an explicit order before
.limit(rowCap)for deterministic truncation.Without
.order(), PostgREST/Postgres row order under the cap is unspecified, so whensampleTruncatedis true the specific 5000 rows retained (and thus the totals/projection) can vary between otherwise-identical calls. Ordering bycreated_at(e.g. descending, to keep the most recent activity when truncated) would make the lower-bound behavior documented in thesampleTruncatedcomment (Line 55-56) actually deterministic.♻️ Suggested fix
const result = await client .from("rag_retrieval_logs") .select("query_class, metadata") .gt("created_at", sinceIso) .eq("metadata->answer->>log_source", "answer") + .order("created_at", { ascending: false }) .limit(rowCap);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/observability/spend-metrics.ts` around lines 142 - 147, Update the query chain in the spend-metrics retrieval flow to add an explicit created_at ordering before limit(rowCap), using descending order to retain the most recent rows when truncated. Keep the existing filters and cap unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/lib/observability/spend-metrics.ts`:
- Around line 142-147: Update the query chain in the spend-metrics retrieval
flow to add an explicit created_at ordering before limit(rowCap), using
descending order to retain the most recent rows when truncated. Keep the
existing filters and cap unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 86a19291-5edc-4ca3-8634-5de8a47d9988
📒 Files selected for processing (4)
src/lib/env.tssrc/lib/health-response.tssrc/lib/observability/spend-metrics.tstests/spend-metrics.test.ts
1 participant
What
Wave 2a of the repo-productivity program. Turns the already-persisted per-answer token counts (
rag_retrieval_logs.metadata.answer.tokens) into a cost signal on the deep/api/healthprobe, so a budget regression — like the reasoning-token starvation incident, or a pricing/traffic change — is visible before the bill arrives.src/lib/observability/spend-metrics.ts—spendSnapshot()aggregates answer-path token usage over a trailing window, derives USD from a configurable per-token price, splits by route (fast/strong) and model, and adds a 24h projection + alert flag. Zero-migration (reads existing JSONB). Reasoning tokens are surfaced but billed within output (never double-counted); cached input billed at the cheaper rate. Throws on query error so the caller nulls the block.src/lib/health-response.ts— optionalspendblock gated exactly likeslo/cache(token-authorized deep probe + healthy Supabase, errors →null, never flips liveness). Suppressible viaincludeSpend: false.src/lib/env.ts—OPENAI_PRICE_{INPUT,CACHED_INPUT,OUTPUT}_PER_MTOK(operator-tunable placeholders — set to live OpenAI price) +SPEND_ALERT_DAILY_USD.tests/spend-metrics.test.ts— 9 offline tests (cost math, route/model split, daily projection + alert, empty window, truncated sample, error path).Verification
20/20 vitest (spend + health-route + answer-slo),
tsc --noEmit, eslint, prettier — all green. No live provider calls; the block only activates behind the existing secret-gated deep probe.Operator note
The three
OPENAI_PRICE_*env vars ship as placeholders — set them to the live OpenAI price for the answer model to make the USD figures accurate.SPEND_ALERT_DAILY_USD=0(default) leaves the alert flag off.🤖 Generated with Claude Code