feat: buyer-artifact-grounded agent testing tools#1564
Merged
Conversation
Add evaluate_agent_quality and compare_media_kit tools to Addie's member toolset for agent quality coaching. evaluate_agent_quality runs SDK comply() and returns structured results for conversational coaching. compare_media_kit lets publishers compare their stated inventory against what their agent actually returns via get_products. Refactor training agent to use SDK request types throughout instead of Record<string, unknown> with manual casts. Handlers now cast args to typed SDK requests at entry, eliminating ~50 individual field casts. Derive extension types structurally (PackageUpdate, Destination, SignalFilters) for SDK types not re-exported from the main entry point. Type shared/formats.ts with TrainingFormat interface. Fix compare_media_kit gap analysis reading non-existent p.channel/p.format fields (now correctly iterates p.channels array and p.format_ids array). Add input validation and data delimiters for prompt injection defense on user-provided content. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
buildAuthOption now returns proper { type: 'basic', username, password }
for basic auth agents instead of incorrectly wrapping credentials as
bearer tokens. The SDK's comply() and AdCPClient both support basic auth
natively.
compare_media_kit now runs get_products briefs concurrently via
Promise.all instead of sequentially, reducing latency proportionally
to the number of verticals tested.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NovaMind AI publisher used event_type 'agent_session' which is not a valid EventType in the AdCP schema. The SDK's comply() correctly rejected this during schema validation, cascading failures across media buy lifecycle tests. Fix: change to 'custom' (valid EventType), tighten PricingTemplate.eventType from string to EventType, and remove the unsafe cast in buildPricingOption. Also fix smoke test health check to accept standalone agent responses. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge main into agent-test-framework branch, resolving conflicts in training-agent files. Fix all pre-existing typecheck errors across the codebase: - testing/index.ts: add `type` keyword to type re-exports (isolatedModules) - tsconfig.json: add paths for @adcp/client subpath exports - billing.ts: narrow Stripe Customer|DeletedCustomer union after .deleted guard - http.ts: same Stripe union narrowing - stripe-client.ts: fix void truthiness check on Product.deleted - bolt-app.ts: cast Slack streaming chunks (undocumented API feature) - Test files: fix mock typing (jest.fn<any>), add missing required fields, remove unused @ts-expect-error directives Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cross-session media buy lookup: get_media_buys and get_media_buy_delivery now search all sessions when explicit IDs aren't found locally, matching real seller behavior where media_buy_ids is a global lookup - SDK field aliases: brief→signal_spec, signal_id→signal_agent_segment_id, singular destination, plural media_buy_ids on delivery endpoint - include_snapshot support on get_media_buys response - Bid price validation: auction pricing now requires bid_price - activate_signal relaxed required fields for SDK-style calls - get_adcp_capabilities now reports signals in supported_protocols - creative_id validation: reject creatives without IDs per spec (buyer assigns creative_id, not seller) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add cross-session fallback to handleUpdateMediaBuy, matching the pattern already applied to get_media_buys and get_media_buy_delivery. The SDK's compliance test creates and updates media buys in separate MCP requests, which land in different sessions with the stateless transport. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Mar 21, 2026
Merge latest main into agent-test-framework. Resolve 9 conflicts: - training-agent: keep SDK typed imports/handlers, merge new episode duration and special episode handling from main - bolt-app: keep streaming chunks cast (undocumented Slack API) - admin HTML/routes: merge new favicon and middleware changes - types: keep SDK-derived types over local alternatives
Resolve 8 conflicts from main merge (bolt-app, testing/index, stripe-client, http, billing, product-factory, test utilities). Install @types/multer for new portraits route from main. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Includes fixes for comply() signal field names (#359), inline creative creative_id (#360), and getPlatformTypesWithLabels (#361). Adapt to stricter types: derive SpecialCategory from Episode, cast params to Record<string, unknown>, narrow property description. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use adcpError() from SDK for all error responses, providing L3 transport (structuredContent.adcp_error + JSON text fallback + isError) - Replace custom validation_error codes with standard AdCP error codes: PRODUCT_NOT_FOUND, BUDGET_TOO_LOW, INVALID_REQUEST, SERVICE_UNAVAILABLE - Move budget validation before product lookup so negative budgets return BUDGET_TOO_LOW instead of PRODUCT_NOT_FOUND - Add root POST route to standalone server for SDK error transport tests - Remove tsconfig paths workaround (SDK 4.13.0 typesVersions handles it) Error Compliance track: 0/3 → 3/3 passing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace compare_media_kit with two tools grounded in real buyer artifacts: - test_rfp_response: Takes an RFP brief, calls get_products with brief mode, runs deterministic gap analysis (channels, formats, budget, KPIs). Supports publisher_response for comparison. - test_io_execution: Takes IO line items, maps each to agent products via normalized channel/format/pricing scoring, constructs the exact create_media_buy JSON a buyer agent would send. Optional execute mode. Training agent improvements: - Channel-aware brief matching (+10 per channel match vs keyword) - Proposal completion in brief mode (pulls missing allocated products) - Fix viewpoint_multi_screen proposal suffix (premium → video_premium) - Startup warning when proposals reference missing products - invalidateCache now clears cachedProposals Security hardening: - SSRF protection: validateAgentUrl blocks metadata endpoints, private IPs, and requires HTTPS in production - Boundary tags around external agent response data - Error message truncation (500 char limit) - Use original agentUrl in error messages (not resolved URL) Also: upgrade @adcp/client to 4.14.0 (state machine compliance) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bokelley
changed the title
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- scripts/test-buyer-artifacts.ts: e2e tests for test_rfp_response and test_io_execution against the training agent - scripts/test-comply.ts: comply() test runner - specs/buyer-artifact-testing.md: design spec for buyer artifact tools Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add 30s timeout on all outbound executeTask calls (SSRF mitigation) - Replace console.warn with structured logger in product-factory - Fix stale dry_run reference in prompts.ts (renamed to execute) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add 'audio' → 'streaming_audio' alias to prevent false mismatches - Remove unused quantity field from test_io_execution schema - Update spec to use execute instead of dry_run Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_rfp_responseandtest_io_executionreplace the circularcompare_media_kitapproach with buyer-artifact-grounded testingtest_rfp_response: Takes a real RFP, callsget_productswith brief mode, runs deterministic gap analysis (channels, formats, budget, KPIs), and compares against the publisher's actual sales response when providedtest_io_execution: Takes real IO line items, maps each to agent products using normalized channel/format/pricing scoring, constructs the exactcreate_media_buyJSON a buyer agent would send, with optional execute modepremium→video_premium), added startup warnings for broken proposals, brief-mode proposal completion so keyword capping doesn't break proposalsTest plan
npm test— 499 tests passnpx tsc --noEmit— clean type check🤖 Generated with Claude Code