fix(addie): surface compliance diagnostics and bump SDK#5763
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
There was a problem hiding this comment.
Code's clean and well-fenced. It can't merge in this state, and one hunk is a stale rewrite of a fix that already landed. Rebase and drop that hunk, and this is an approve.
Why this isn't an approve yet
The PR is CONFLICTING / DIRTY. The conflict is server/src/training-agent/brand-handlers.ts:940. This hunk rewrites the old const endMs = new Date(campaign.end_date).getTime() block to append T23:59:59Z, but main already shipped #5760 (16e3d58f) which rewrote the same block to compare against startOfTodayMs. The PR branched before #5760 merged.
The two fixes are not equivalent. #5760 keeps a campaign ending today licensable by comparing endMs < startOfTodayMs (00:00:00Z today). This PR's version compares end_date+23:59:59Z against Date.now(). Resolving the conflict by taking this side silently reverts #5760's deliberate calendar-day semantics. Drop the brand-handlers change entirely, keep #5760, rebase. That clears the conflict and the regression in one move.
Things I checked
- Prompt-injection surface holds.
declaredProtocolandexpectedProtocolcome from the remote agent'ssupported_protocols. Both are sanitized at the source viasanitizeClassifiedValue(raw, 80)(compliance-testing.ts:296), then double-fenced throughfenceAgentValuebefore reaching Addie's markdown (member-tools.ts:4789,5040). Headline asserts single-line.security-reviewer: no High/Medium. - The re-probe is not a new SSRF or auth-leak target.
classifyCapabilityResolutionErrorWithDeclaredProtocols(member-tools.ts:580) re-hits the sameresolved.resolvedUrlwith the sameauthOptionthe maincomply()path already used this invocation. Single-shot, try/caught, logsprobeErrornot the body. Gated tokind === 'specialism_parent_protocol_missing'. - Diagnostics stay owner-gated.
first_failure_validationsrides the sameincludeDiagnosticsflag asfirst_failure_message; unauthorized callers get[]. The two omitting callers are anis_ownerpath and the caller's own just-run serialization, not cross-tenant reads.security-reviewerconfirmed tenant isolation. - LATERAL join keys correctly.
agent_url AND run_id AND storyboard_id,failed_validations_jsonb IS NOT NULL, deterministicORDER BY d.captured_at DESC, d.id DESC LIMIT 1. NULLrun_idcomparison yields[]safely. No app-level N+1 — it's one statement on the bulk path. - No changeset required. No
static/schemas/source/**, nodocs/reference/**, no wire surface. Internal server diagnostics only. Scope check claim is sound.
Follow-ups (non-blocking — file as issues)
first_failed_step_id IS NULLattaches an arbitrary step's validations (compliance-db.ts:1077,1199). When the first-failed step id is null, the LATERAL returns the most-recent failing step's validations, which the UI then labels as the "first failure." Deterministic viad.id DESC, best-effort owner-debug surface — but the label implies a precision it doesn't have.- Raw jsonb array is uncapped in the REST response (
registry-api.ts:394).normalizeValidationListpasses the agent-influenced payload straight throughz.array(z.any()); the UI slices to 3 client-side but the API returns the full array. Owner-only and escaped on render, so not a leak — consider a server-side length cap for symmetry. - Confirm
typecheckis green on a clean SDK 9.3.0 install.knownProtocolsFromIndexnow readsindex.protocols?.map(p => p.id)(compliance-testing.ts:275). Runtime-safe via?./?? [], but it depends onComplianceIndex.protocolsexisting on the 9.3.0 type. PR claims typecheck passed; let CI on a clean install confirm.
Minor nits (non-blocking)
- Script path form is inconsistent.
test:server-unitnow passestests/unit/(config-root-relative) while siblingtest:server-integrationpasses theserver/-prefixedserver/tests/integration(package.json:94). Both resolve viaroot: __dirname, but the differing conventions read as a mistake.
Rebase onto main, drop the brand-handlers hunk in favor of #5760, and I'll approve.
44a73e4 to
53ea4ae
Compare
There was a problem hiding this comment.
Clean, well-scoped diagnostics surfacing plus a near-miss protocol classifier. The right shape: agent-declared strings stay sanitized at extraction and re-fenced at the markdown sink, and the new diagnostics stay gated behind includeDiagnostics.
Things I checked
- Classifier precedence (
server/src/addie/services/compliance-testing.ts:301-345):nearMissProtocolDeclarationfires on hyphen→underscore confusion (media-buyvsmedia_buy) before theknownProtocolsFromIndex()smuggling defense, and only whendeclaredProtocol !== parentProtocol.index.protocols?.map(...)is optional-chained, so a missing SDK field won't throw. Presenter parity at:432-447is complete across all four sinks; headline asserts no CR/LF. - Protocol-id semantics (
ad-tech-protocol-expert: sound): canonicalsupported_protocolsenum is snake_case perstatic/schemas/source/protocol/get-adcp-capabilities-response.json— kebab-case is reserved for compliance URL paths and the task-categorization axis. A hyphen→underscore collapse can only map onto the same logical protocol in a closed 7-value enum, so the heuristic cannot conflate two distinct ids. Guidance is correct in context. - Fencing on the LLM path (
security-reviewer: no High/Medium): both newunrecognized_supported_protocolbranches inmember-tools.ts(:4789-4798,:5040-5049) wrapdeclaredProtocol/expectedProtocol/specialisminfenceAgentValue(..., 80)— double sanitization. The reprobe atmember-tools.ts:580-595re-hits the samevalidateAgentUrl-checked endpoint (no new SSRF surface) and logs failures vialogger.warnrather than leaking to the model. - SQL LATERAL joins (
compliance-db.ts:1073-1089,:1196-1212): predicate scopes byagent_url/run_id/storyboard_id/step_id;ORDER BY captured_at DESC, d.id DESC LIMIT 1is fully deterministic (idBIGSERIAL);LEFT JOIN LATERAL ... LIMIT 1cannot fan out rows. NULLrun_idyields NULL validations, which is safe. Onlyfailed_validations_jsonbis pulled — wire-payload columns left out. - Test-script fix (
package.json:94):vitest run --config server/vitest.config.ts tests/unit/runs the sameserver/tests/unit/*.test.tsfiles but under the server config'sfileParallelism: false, which guards the sharedpoolsingleton the old root-config invocation raced. Same tests, run safely. - No changeset is correct: changed files are all under
server/plus the SDK bump — none matchPROTOCOL_SCOPED_PATHSinscripts/check-changeset-protocol-scope.cjs. No wire-shape change.
Follow-ups (non-blocking — file as issues)
- No test coverage for the reprobe path.
classifyCapabilityResolutionErrorWithDeclaredProtocolsand the twounrecognized_supported_protocoltool-handler branches are untested — only the pure classifier/presenter got unit tests. A test exercisingspecialism_parent_protocol_missing→ reprobe → reclassify would lock in the kind-guard, thecaps.profile?.supported_protocols ?? []fallback, and the catch-and-fall-back-to-initialbehavior. index.protocolsmay be a no-op.knownProtocolsFromIndexnow readsindex.protocols?.map(p => p.id). Worth confirming SDK 9.3.0's index actually shipsprotocols[], otherwise that half of the cross-check is dead code (thespecialisms[].protocolset still gates — not a correctness or security issue).
Minor nits (non-blocking)
- Stale PR body. The Summary claims a "date-only brand rights end-date validation" fix, but no rights file is in the diff — that landed as
16e3d58already onmain. Trim the body to the eight files actually changed. - Title isn't conventional-commits (
[codex] Surface compliance diagnostics and bump SDK). The underlying commit conforms; if the squash uses the PR title as subject, it won't.
LGTM. Follow-ups noted below.
53ea4ae to
77c6b2a
Compare
There was a problem hiding this comment.
Clean, contained diagnostic surface plus a first-party SDK bump. Right shape: sanitize-at-classify, fence-at-LLM-sink, gate-diagnostics-behind-ownership — the PR follows the existing pattern at every new site.
Things I checked
- Tenant isolation on the new SQL. Both
LEFT JOIN LATERALblocks incompliance-db.ts(getStoryboardStatuses + bulk variant) scope diagnostics byagent_url+run_id+storyboard_id, and the outer query is already filtered to the requestedagent_url. No cross-owner join path.security-reviewer: PASS, no leak. - Prompt-injection fencing. Every agent-controlled value in the two new
unrecognized_supported_protocolbranches (member-tools.ts:4789-4798,5040-5048) goes throughfenceAgentValue; the classifier inputs go throughsanitizeClassifiedValuefirst. Double-protected. The unit test asserts the headline carries no\r\n. - Diagnostics gating.
first_failure_validationsis gated behindincludeDiagnosticsinserializeStoryboardRunStatus(registry-api.ts:410-412), which resolves to owner/static-admin only. Non-owners get[]. test:server-unitpath.vitest run --config server/vitest.config.ts tests/unit/resolves the positional against the configroot: __dirname(=server/), landing on the sameserver/tests/unit/set. No repo-roottests/unit/exists, confirming config-relative resolution. The new form additionally inheritsfileParallelism: false— which is the stated point (shared server singletons).- Protocol-id claim.
media_buy(snake_case) is the canonicalsupported_protocolsvalue perstatic/schemas/source/protocol/get-adcp-capabilities-response.json;media-buyis not in that enum. Hyphen→underscore is the right normalization direction for this field.ad-tech-protocol-expert: sound. - No changeset is correct. No file under
static/schemas/**ordist/schemas/**changed; scope check passes. Diagnostic text only, no wire surface touched.
Follow-ups (non-blocking — file as issues)
- NULL-step fallback can mismatch the message.
compliance-db.ts:1075-1084(and the batch at:1199-1208): whenfirst_failed_step_id IS NULLbut failures exist, the lateral returns the most-recent diagnostic for any step in the run, which may not be the step whosefirst_failure_messagethe UI renders beside it. Either surface nothing on the NULL branch or note the fallback is intentional. - Over-broad message string. "Protocol ids use underscores, not hyphens" is true for
supported_protocolsbut false as a universal AdCP statement —adcp-protocol.json(tasks-list, webhook payloads, compliance storyboard paths) is kebab-case by design. Scope the sentence to the field, e.g. "In `supported_protocols`, protocol ids use underscores." Correct where it fires today; just fragile if reused. - Redundant re-probe.
classifyCapabilityResolutionErrorWithDeclaredProtocols(member-tools.ts:580-595) fires a secondget_adcp_capabilitiesround-trip in the catch path rather than threading thesupported_protocolsalready fetched in the try block. Only on the rarespecialism_parent_protocol_missingkind, wrapped in try/catch with warn-and-fallback — fine to ship, cheap to tighten later. - Index coverage. Neither existing diagnostics index fully covers the lateral's
(agent_url, run_id, storyboard_id)+ORDER BY captured_at DESC.LIMIT 1per row keeps it acceptable now; a composite index would make it index-only if the batch endpoint gets hot.
Minor nits (non-blocking)
- Ordering comment.
nearMissProtocolDeclarationruns before the defense-in-depthknown.has(parentProtocol)check (compliance-testing.ts:333-344). Safe because the regex is start-anchored and the string is resolver-produced, but a one-line comment noting the ordering is deliberate would save the next reader the trace.
One observation: the PR body lists a "Fix date-only brand rights end-date validation" bullet and a brand-sandbox-tools.test.ts validation run, but no such change is in this diff — that work landed in 16e3d58 on main. Stale description, not a code problem.
LGTM. Follow-ups noted below.
77c6b2a to
88379ce
Compare
There was a problem hiding this comment.
Clean, contained diagnostics fix. The classifier near-miss path and the LATERAL-join surfacing are the right shape: a hyphen/underscore protocol-id mismatch (media-buy vs media_buy) gets a specific, actionable diagnosis instead of falling through to unknown_specialism.
Things I checked
- Classifier ordering (
compliance-testing.ts:301-318): near-miss is evaluated before the known-protocol defense-in-depth check. Correct —unrecognized_supported_protocolis the more specific diagnosis, andnearMissProtocolDeclarationonly fires when a sanitized declared value normalizes (-→_) to the already-regex-validatedparentProtocol. An attacker can't smuggle an arbitrary string intodeclaredProtocol. - LLM-context fencing (
member-tools.ts:4789-4798,:5040-5048): all four agent-declared strings (declaredProtocol,expectedProtocol,specialism,parentProtocol) are double-sanitized —sanitizeClassifiedValue(strips\r\n, backticks, control chars, 80-char cap) at classify time, thenfenceAgentValueat render time.security-reviewer: PASS, no bypass. - Data exposure (
compliance-db.tsLATERAL JOIN →registry-api.tsfirst_failure_validations): gated behindcanViewAgentDebugData(static-admin or verified owner); non-owners get[]. Payloads are redacted at write time viastripValidationRequestResponse+redactForDiagnosticsbefore storage, so the join surfaces an already-redacted column.security-reviewer: PASS. test:server-unitswitch (package.json):vitest run --config server/vitest.config.ts tests/unit/resolves toserver/tests/unit/because the config setsroot: __dirname(=server/), and now picks upfileParallelism: false. That's the actual fix — stops the shareddb/index.tspool singleton from being nulled mid-query by a sibling file'safterAll(closeDatabase).index.protocolsnull-safety (compliance-testing.ts:266-275):index.protocols?.map(p => p.id).filter(Boolean) ?? []is safe for missing field / missing id. Typecheck passes against SDK 9.3.0.- Dashboard render (
dashboard-agents.html:1726-1738): agent-controlled values flow throughescapeHtmlinto text/<code>contexts. No XSS. - SDK bump 9.2.0→9.3.0 pinned exact with updated integrity hash; no new deps, no install scripts. Changeset omission is defensible — no protocol release-surface change, and the PR body confirms
check-changeset-protocol-scope.cjspasses. Thestatic/openapi/registry.yamldelta is generated, not hand-edited.
Follow-ups (non-blocking — file as issues)
- Nullable
run_id→ silent empty validations (compliance-db.ts:1078-1086and the batch query ~:1196-1204):agent_storyboard_status.run_idis nullable (migration 390), so the LATERAL predicated.run_id = s.run_idnever matches whens.run_id IS NULLandfirst_failure_validationscomes back empty even when diagnostics exist. Best-effort is fine, but a one-line comment would save a future "why are validations missing" investigation. first_failed_step_id IS NULLfallback (compliance-db.ts:1080-1086): with no first-failed step, the join attaches the most-recent (captured_at DESC, id DESC) failing diagnostic for any step of the run. The surfaced validations then don't correspond to a named step. Acceptable as best-effort.- Redaction by key-name/format only (
compliance-testing.ts:1086-1103, pre-existing — not regressed here): a non-standard plaintext secret echoed by a hostile agent into a validationexpected/actualunder a benign key name survives to storage. Exposure is owner/admin-only andrequest/responseblobs are already stripped, so impact is bounded. Worth a belt-and-suspenders pass if this view widens.
The body still lists a "date-only brand rights end-date validation" fix that isn't in this diff — landed separately in 16e3d58 on main. Stale bullet, not a blocker.
LGTM. Follow-ups noted below.
Summary
@adcp/sdkto9.3.0soseed_accountroutes through the SDK fix forseed.account.supported_protocolsnear-misses distinctly in Addie compliance tools, e.g.media-buyvsmedia_buy.rights_constraint.valid_until.Validation
npx vitest run server/tests/unit/capability-resolution-error-classifier.test.ts tests/addie/member-tools.test.ts server/tests/unit/compliance-db-last-write-wins.test.tsnpx vitest run tests/addie/brand-sandbox-tools.test.ts -t "rights_constraint uses date-time format" --pool=threadsnpm run test:server-unitnpm run typechecknode scripts/check-changeset-protocol-scope.cjs origin/maingit diff --checkNo changeset: no protocol release-surface changes detected; scope check passes.