From 31e99e7b9bea3fd352c816fdb1582b8707c027ab Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 07:13:17 +0000 Subject: [PATCH] fix(ci): eval canary must run as public-owner sentinel, not E2E user The nightly eval canary pinned the eval owner to the E2E user via RAG_EVAL_OWNER_EMAIL. Since the corpus went all-public (owner_id = NULL), that user owns 0 retrievable documents, so the golden retrieval eval returns 0 candidates and fails --fail-on-threshold every run. resolveEvalOwnerId (scripts/eval-utils.ts) already defaults to the public-owner sentinel 00000000-0000-0000-0000-000000000000 when no owner is set (the fix established in the eval-owner-default change). Drop the owner pin so the canary inherits that sentinel and scopes to the public corpus, and remove the now-unused E2E_USER_EMAIL from the required-secrets preflight. Note: this removes the *owner* failure only. The canary's current red is a stale/unregistered OPENAI_API_KEY repo secret ("Unregistered API key" at the embedding step); rotating that secret is an operator action. With both addressed the canary should pass in one run. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01R5KT4sYGB8TGQ4jV888EPH --- .github/workflows/eval-canary.yml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/eval-canary.yml b/.github/workflows/eval-canary.yml index f756c3c17..0697554b3 100644 --- a/.github/workflows/eval-canary.yml +++ b/.github/workflows/eval-canary.yml @@ -38,7 +38,13 @@ env: NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY: placeholder-ci-anon-key SUPABASE_SERVICE_ROLE_KEY: ${{ secrets.SUPABASE_SERVICE_ROLE_KEY }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} - RAG_EVAL_OWNER_EMAIL: ${{ secrets.E2E_USER_EMAIL }} + # The live corpus is all-public (owner_id = NULL). Owner-scoped retrieval + # fail-closes on a null owner, so the eval must run as the public-owner + # sentinel. resolveEvalOwnerId (scripts/eval-utils.ts) defaults to that + # sentinel when no owner is set, so deliberately do NOT pin an eval owner + # here — pinning the E2E user (who owns 0 public docs) returns 0 candidates + # and fails the golden threshold. Set RAG_EVAL_OWNER_ID only if the corpus + # is ever re-scoped to a real owner. jobs: eval-canary: @@ -56,7 +62,6 @@ jobs: missing="" [ -z "$SUPABASE_SERVICE_ROLE_KEY" ] && missing="$missing SUPABASE_SERVICE_ROLE_KEY" [ -z "$OPENAI_API_KEY" ] && missing="$missing OPENAI_API_KEY" - [ -z "$RAG_EVAL_OWNER_EMAIL" ] && missing="$missing E2E_USER_EMAIL" if [ -n "$missing" ]; then echo "::error::Eval canary cannot run — missing repo secrets:$missing" exit 1