Skip to content

docs(learning): add A2B hands-on lab module — Testing Your First Agent#3398

Merged
bokelley merged 5 commits into
mainfrom
claude/issue-2571-a2b-testing-lab
May 14, 2026
Merged

docs(learning): add A2B hands-on lab module — Testing Your First Agent#3398
bokelley merged 5 commits into
mainfrom
claude/issue-2571-a2b-testing-lab

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Closes #2571

Adds a free hands-on lab module (A2B: Testing Your First Agent) as a supplement to A2. Addresses the gap Addie's weekly insights surfaced: ~75 learners/week complete A2 conceptually but lack "operational muscle memory" for actual agent calls. The module provides copy-paste curl examples for every step of the buy flow, explicit handling of all three create_media_buy response shapes, dry-run creative validation, tasks/get async polling, and a five-entry common-errors table.

Non-breaking justification: Additive only — new MDX doc page, new DB migration row (with ON CONFLICT DO UPDATE for idempotency), two new entries in TypeScript object literals, one nav item, one changeset file. No schema changes, no protocol changes, no existing content modified except the sort_order bump for A3 (3→4) and the free-modules string in certification-tools.ts.

Files changed:

  • docs/learning/foundations/a2b-testing-your-first-agent.mdx — new module page with curl reference, error table, assessment rubric
  • server/src/db/migrations/448_a2b_testing_lab.sql — inserts A2B (sort_order=3, is_free=true, prerequisites={A2}); bumps A3 sort_order 3→4
  • server/src/addie/mcp/certification-tools.ts — adds A2B to MODULE_ILLUSTRATION_TOPICS and MODULE_RESOURCES; updates free-modules list string
  • docs.json — adds A2B to the "Basics (free)" nav group between A2 and A3
  • .changeset/a2b-testing-lab.md--empty changeset (non-protocol change)

Companion issue: #2570 (Media Buy Lifecycle doc) is still open. The reading-list card "Media buy lifecycle" currently links to /docs/media-buy/media-buys (the existing page covering all status states). Update the link once #2570 ships.

Nit (noted for reviewer): The curl example uses idempotency_key: "mb-lab-20260428-001" with a hardcoded date as a concrete placeholder. A reader following the guide should use a fresh value — this is clear from the surrounding prose but could be made more explicit with a <date> placeholder form if desired.

Pre-PR review:

  • education-expert: approved — two blockers found and fixed (tasks/get method form; Media Buy Lifecycle link label); assessment rubric dimensions verified orthogonal; content completeness confirmed against all acceptance criteria
  • code-reviewer: approved — no blockers after fixes; nits noted above

Triage-managed PR. This bot does not currently iterate on
review comments or PR conversation threads (only on the source
issue). To unblock:

  • Push fixup commits directly: gh pr checkout <num>
    fix → push.
  • Or re-trigger: comment /triage execute on the source
    issue.

See #3121
for context.

Session: https://claude.ai/code/session_01PFHFqGU1pQdxswb7xswSwY


Generated by Claude Code

@bokelley bokelley added the claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage. label Apr 28, 2026
@bokelley

Copy link
Copy Markdown
Contributor Author

Server integration tests failing. Likely: test setup needs DB init or mocks. Check failure detail and either mock the missing dep or update test setup.

@bokelley

Copy link
Copy Markdown
Contributor Author

Fixed. Root cause: digest-email-recipients.test.ts had expect(r.cert_total_modules).toBe(3) hardcoded for track A — written when track A had three modules (A1, A2, A3). Migration 448 adds A2B, so track A now has four, and the assertion failed.

Fix in 51ad809: replaced the hardcoded 3 with a live SELECT COUNT(*) FROM certification_modules WHERE track_id = 'A' query, consistent with the pattern already used for the track-B and tie-break tests in the same file. No logic changes — test-only.


Generated by Claude Code

@bokelley

Copy link
Copy Markdown
Contributor Author

Noted — 51ad809 looks good. Replacing the hardcoded 3 with a live COUNT(*) against certification_modules is the right approach and consistent with the existing track-B and tie-break patterns. No further action needed here.


Generated by Claude Code

@bokelley

Copy link
Copy Markdown
Contributor Author

Acknowledged — no further action from triage on this thread.


Generated by Claude Code

@bokelley

Copy link
Copy Markdown
Contributor Author

Routine: this PR has 4 CI failures driven by two root causes — fix both in one rebase pass:

1. Migration renumber (448 → 451)

Main now has:

448_backfill_primary_organization_id.sql
449_auto_provision_brand_hierarchy_children.sql
450_auto_provision_hierarchy_enabled_at.sql

Rename 448_a2b_testing_lab.sql451_a2b_testing_lab.sql to clear the dup-prefix check.

2. Rebase + TypeScript / integration test conflicts

server/src/addie/mcp/certification-tools.ts and server/tests/integration/digest-email-recipients.test.ts likely conflict with recent cert and digest PRs that landed (#3405-related work, #3395 cert rubric, etc.). Rebase against origin/main and resolve — keep this PR's A2B-specific additions, take main's other changes.

Once both are resolved:

  • No duplicate migration numbers should pass (no more 448 dup)
  • Built migrations against Postgres should pass (renamed migration applies cleanly)
  • TypeScript Build should pass (post-rebase imports/types align)
  • Server integration tests should pass (test conflicts resolved)

Push the rebased branch and CI should clear all four.

@bokelley

bokelley commented May 3, 2026

Copy link
Copy Markdown
Contributor Author

Held back from docs automerge bundle. CI failures: TypeScript Build, integration tests, migration number conflicts (branch is stale vs main migrations).

Failure logs are 6+ days old; a rebase onto current main (which absorbed the IA Phase 2/3 doc moves in #4022 and #4031) would either fix link/build drift or surface real issues. Once CI is green, this is automerge-eligible.

@bokelley bokelley force-pushed the claude/issue-2571-a2b-testing-lab branch from 51ad809 to 0daf5d7 Compare May 3, 2026 23:54
@bokelley

bokelley commented May 3, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current main and pushed.

What changed:

Build + test result:

  • npm run build && npm run typecheck — clean.
  • npm test — 1 pre-existing failure in lint-storyboard-context-output-paths (sales-guaranteed/create_media_buy — task_completion.media_buy_id (path_not_in_schema)). That storyboard is identical to main; the failure exists on main independently of this PR.

Should be CI-green (or fail only on the pre-existing lint) after this push.


Generated by Claude Code

@bokelley

bokelley commented May 3, 2026

Copy link
Copy Markdown
Contributor Author

Got it — rebase clean, migration renumber to 466 noted. The lint-storyboard-context-output-paths failure on sales-guaranteed/create_media_buy is pre-existing on main and unrelated to this PR's diff.

PR looks ready for CI green + human review.


Generated by Claude Code


Generated by Claude Code

@bokelley

bokelley commented May 3, 2026

Copy link
Copy Markdown
Contributor Author

Confirmed — lint-storyboard-context-output-paths failure on sales-guaranteed/create_media_buy is pre-existing on main and outside this PR's diff. No action needed here. PR is ready for CI green + human review to proceed.


Generated by Claude Code

claude and others added 5 commits May 14, 2026 06:59
New free Academy module A2B between A2 and A3 (sort_order 3; A3 bumped to 4).
Adds step-by-step curl examples for MCP session init, get_products, create_media_buy
(all three response shapes), sync_creatives with dry_run, and tasks/get async
polling. Four-exercise assessment rubric: practical_knowledge 40%, problem_solving
30%, error_recovery 20%, conceptual_understanding 10%. Closes #2571.

https://claude.ai/code/session_01PFHFqGU1pQdxswb7xswSwY
…examples

- account.brand: bare string → brand-ref object {domain: "nova-motors.com"}
- account.operator: bare slug → domain string "pinnacle-media.com"
- tasks/get params: "id" → "task_id" + add "include_result: true"
- tasks/get result field: "artifact" → "result" (per tasks-get-response.json)
- error field: "task.message" → "task.error.message" (MDX + SQL lesson_plan)
- docs.json: add A2B to latest version's "Basics (free)" nav group

https://claude.ai/code/session_01PFHFqGU1pQdxswb7xswSwY
Adding A2B to certification_modules (migration 448) bumps track A from 3
to 4 modules, breaking the 'counts tested_out the same as completed' test
which hardcoded cert_total_modules = 3.  Replace with a live COUNT query
matching the pattern already used in the track-B and tie-break cases in
the same file.

https://claude.ai/code/session_01UBnL5mMQJJ9ui2D7ndyj3u
…main

Migration 448 was claimed by `448_backfill_primary_organization_id.sql`
merged to main while this branch was open. Renumber to 466 (next after 465).

https://claude.ai/code/session_0186LE5uSHeWNCjmHWrSaxUp
Slot 466 has 466_adcp_state.sql on main; A2B's earlier 448→466 renumber
collided with that landing. Next free slot at rebase time is 478.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley force-pushed the claude/issue-2571-a2b-testing-lab branch from 0daf5d7 to e60a4a9 Compare May 14, 2026 11:01
@bokelley bokelley marked this pull request as ready for review May 14, 2026 11:01
@bokelley bokelley merged commit 34a8b38 into main May 14, 2026
16 of 17 checks passed
@bokelley bokelley deleted the claude/issue-2571-a2b-testing-lab branch May 14, 2026 11:01
bokelley added a commit that referenced this pull request May 14, 2026
Migration 478 shipped success_criteria as plain strings; SuccessCriterion is
{ id, text } and demonstrations-fairness asserts every criterion has an id, so
main has been red on Server integration tests since PR #3398. Migration 479
converts each string criterion to { id: <ex>_c<n>, text: <string> }. Idempotent
— pre-existing object criteria pass through unchanged.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

academy: A2.5 Testing Your First Agent — hands-on lab module

2 participants