test(junior): Rework testing architecture

package.json

@@ -27,7 +27,7 @@
     "test:watch": "pnpm --filter @sentry/junior test:watch",
     "evals": "pnpm --filter @sentry/junior-evals evals",
     "evals:record": "pnpm --filter @sentry/junior-evals evals:record",
-    "typecheck": "pnpm --filter @sentry/junior-plugin-api typecheck && pnpm --filter @sentry/junior-scheduler typecheck && pnpm --filter @sentry/junior typecheck && pnpm --filter @sentry/junior-dashboard typecheck && pnpm --filter @sentry/junior-testing typecheck && pnpm --filter @sentry/junior-example typecheck",
+    "typecheck": "pnpm --filter @sentry/junior-plugin-api typecheck && pnpm --filter @sentry/junior-scheduler typecheck && pnpm --filter @sentry/junior typecheck && pnpm --filter @sentry/junior-evals typecheck && pnpm --filter @sentry/junior-dashboard typecheck && pnpm --filter @sentry/junior-testing typecheck && pnpm --filter @sentry/junior-example typecheck",
     "skills:check": "pnpm --filter @sentry/junior skills:check",
     "test:ci": "pnpm --filter @sentry/junior build && pnpm --filter @sentry/junior-dashboard build && pnpm --filter @sentry/junior test:coverage && pnpm --filter @sentry/junior-dashboard test:coverage"
   },

packages/junior-evals/README.md

@@ -23,7 +23,7 @@ Quick mapping:
 - `evals/*`: Integration-style coverage for conversation-level agent behavior and quality scoring through the runtime harness.
 - `tests/unit/*` (or non-integration tests): isolated logic/invariant tests.
 
-This separation is enforced by `pnpm --filter @sentry/junior run test:slack-boundary`.
+This separation is enforced by `pnpm --filter @sentry/junior run test:boundaries`.
 
 ## What Is In Scope
 
@@ -59,22 +59,28 @@ For each `it()` case inside a `describeEval()` suite:
 2. Create a fresh runtime instance for the case via the chat composition root; do not mutate the production singleton runtime.
 3. Route message events through real ingress + queue-worker behavior, with only the external queue transport replaced by an in-memory harness shim.
 4. Return observed artifacts as JSON for LLM judgment, including structured `assistant_posts` with text plus actual attached-file metadata, and Slack-visible metadata.
+   The output also includes compact `turn_diagnostics` so evals can assert user-facing runtime metadata such as selected thinking level without scraping logs.
    The helper pretty-prints this JSON so failure output stays readable in local runs and CI.
 5. `vitest-evals` scores the output against `criteria` (A–E → 1.0–0.0).
 
 Harness override knobs (in `EvalOverrides`):
 
-- `auto_complete_mcp_oauth`: after our app genuinely starts an MCP OAuth flow for the listed providers, the harness immediately completes the fake provider callback.
-- `auto_complete_oauth`: after our app genuinely starts a generic OAuth flow for the listed providers, the harness immediately completes the fake provider callback.
-- `credential_providers`: seed normal provider credentials for the listed providers. GitHub uses dummy GitHub App env vars plus an intercepted installation-token exchange; Sentry uses the normal OAuth token store.
-- `fail_reply_call`: force a non-retryable reply failure on a specific call.
-- `mock_image_generation`: stub the image-generation HTTP response with a valid image payload while still exercising the real attachment path.
-- `plugin_dirs`: load plugin fixtures from eval-local directories without adding workspace packages.
-- `reply_texts`: override returned reply text per call.
-- `reply_timeout_ms`: lower or set the per-reply harness timeout for a specific scenario. It cannot exceed 30 seconds.
-- `subscribed_decisions`: controls the subscribed-message reply gate in the harness. If you use it, do not claim that reply-selection behavior is being validated by the eval itself.
-
-These knobs work by overriding services on the eval-local runtime instance. They must not reintroduce mutable global runtime behavior seams.
+- `auth.autoCompleteMcpOAuth`: after our app genuinely starts an MCP OAuth flow for the listed providers, the harness immediately completes the fake provider callback.
+- `auth.autoCompleteOAuth`: after our app genuinely starts a generic OAuth flow for the listed providers, the harness immediately completes the fake provider callback.
+- `auth.credentialProviders`: seed normal provider credentials for the listed providers. GitHub uses dummy GitHub App env vars plus an intercepted installation-token exchange; Sentry uses the normal OAuth token store.
+- `plugins.pluginDirs`: load plugin fixtures from eval-local directories without adding workspace packages.
+- `plugins.pluginPackages`: load named workspace plugin packages for plugin-specific behavior evals.
+- `plugins.skillDirs`: load skill fixture directories into the real reply-generation path.
+- `replyGeneration.cannedResults`: return structured reply results for downstream delivery or resilience scenarios.
+- `replyGeneration.cannedTexts`: return reply text per successful call for downstream delivery scenarios.
+- `replyGeneration.failCall`: force a non-retryable reply failure on a specific call.
+- `replyGeneration.mockImageGeneration`: stub the image-generation HTTP response with a valid image payload while still exercising the real attachment path.
+- `replyGeneration.timeoutMs`: lower or set the per-reply harness timeout for a specific scenario. It cannot exceed 30 seconds.
+- `replyGeneration.unsetGatewayCredentials`: remove gateway credentials for the duration of real reply generation when the scenario explicitly covers missing credential behavior.
+- `subscribedReplyDecisions`: controls the subscribed-message reply gate in the harness. If you use it, do not claim that reply-selection behavior is being validated by the eval itself.
+
+These knobs configure role-named scenario adapters on the eval-local runtime instance. They must not reintroduce mutable global runtime behavior seams or nested production service override bags.
+`replyGeneration.cannedTexts` and `replyGeneration.cannedResults` bypass real reply generation, so use them only for downstream delivery behavior, not prompt, model-routing, or thinking-level coverage.
 
 Tool replay:
 
@@ -106,7 +112,7 @@ Evals require real Vercel Sandbox access. If sandbox bootstrap fails, the eval f
 
 - Add core cases under `evals/core/*.eval.ts` and plugin-specific cases under `evals/<plugin>/` using `describeEval()` with `slackEvals`.
 - Use event builders (`mention`, `threadMessage`, `threadStart`) from `evals/helpers.ts`.
-- Use `auto_complete_mcp_oauth` or `auto_complete_oauth` when the harness should instantly complete the fake provider callback after our app has genuinely initiated auth.
+- Use `auth.autoCompleteMcpOAuth` or `auth.autoCompleteOAuth` when the harness should instantly complete the fake provider callback after our app has genuinely initiated auth.
 - For multi-turn, pass the same `thread` override so events land in one thread.
 - Keep each case focused on one primary behavior.
 - Encode all expectations in `criteria`; do not add deterministic inline assertions.
@@ -127,6 +133,7 @@ Do not do these in eval files:
 
 - Do not import `@/chat/slack/*` directly.
 - Do not use MSW Slack helpers (`queueSlackApiResponse`, `getCapturedSlackApiCalls`, `queueSlackApiError`, `queueSlackRateLimit`).
+- Do not import raw Slack capture wrappers. Use eval artifact helpers that expose Slack-visible posts, reactions, canvases, or files instead.
 - Do not validate raw Slack Web API request payload shapes from evals.
 - Do not validate implementation internals (exact tool names, sandbox IDs, or other non-user-visible details) unless the scenario explicitly evaluates those surfaces.