Skip to content

fix(ci): eval canary must run as public-owner sentinel, not E2E user #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 1 commit into from
Jul 8, 2026
+7 −2
Merged

fix(ci): eval canary must run as public-owner sentinel, not E2E user #383

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/eval-canary.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,13 @@ env:
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY: placeholder-ci-anon-key
SUPABASE_SERVICE_ROLE_KEY: ${{ secrets.SUPABASE_SERVICE_ROLE_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
RAG_EVAL_OWNER_EMAIL: ${{ secrets.E2E_USER_EMAIL }}
# The live corpus is all-public (owner_id = NULL). Owner-scoped retrieval
# fail-closes on a null owner, so the eval must run as the public-owner
# sentinel. resolveEvalOwnerId (scripts/eval-utils.ts) defaults to that
# sentinel when no owner is set, so deliberately do NOT pin an eval owner
# here — pinning the E2E user (who owns 0 public docs) returns 0 candidates
# and fails the golden threshold. Set RAG_EVAL_OWNER_ID only if the corpus
# is ever re-scoped to a real owner.

jobs:
eval-canary:
Expand All @@ -56,7 +62,6 @@ jobs:
missing=""
[ -z "$SUPABASE_SERVICE_ROLE_KEY" ] && missing="$missing SUPABASE_SERVICE_ROLE_KEY"
[ -z "$OPENAI_API_KEY" ] && missing="$missing OPENAI_API_KEY"
[ -z "$RAG_EVAL_OWNER_EMAIL" ] && missing="$missing E2E_USER_EMAIL"
if [ -n "$missing" ]; then
echo "::error::Eval canary cannot run — missing repo secrets:$missing"
exit 1
Expand Down
Loading