Skip to content

ci: add action-level PR harness for cache and environment actions#114

Open
mmcky wants to merge 6 commits into
mainfrom
ci/pr-action-harness
Open

ci: add action-level PR harness for cache and environment actions#114
mmcky wants to merge 6 commits into
mainfrom
ci/pr-action-harness

Conversation

@mmcky

@mmcky mmcky commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

First stage of #100 (the PR-time harness — harness A in the design proposal there). Stages 2–4 (the test-lectures canary repo, preview/publish coverage, tag-push trigger) need a separate repo and secrets and stay with the issue, which should remain open after this merges.

What this adds

A new workflow, .github/workflows/test-actions.yml, that tests the composite actions themselves — the first automated tests this layer has ever had. It invokes the actions via ./ local paths, so it exercises the code on the PR, not a released ref, and it runs on every PR that touches a covered action (plus pushes to main and manual dispatch). It is the permanent form of the throwaway harness that verified the #104 fix, which shipped dead on arrival in v0.8.0 and survived eight months because nothing in CI exercised it.

Job group What it asserts
restore-jupyter-cache unit jobs (7, seconds each, no conda) Both cache types × both modes: genuine miss reports empty outputs; read-only and save-mode restores report the hit with the exact expected key (the save-mode job is the #104 regression scenario); fail-on-miss fires on a genuine miss and stays quiet on a hit; payloads round-trip byte-for-byte via a STAMP sentinel
setup-environment chain (2 jobs) Standard-mode conda cache: cold run misses, second run gets an exact cache hit — the #33/#78 path that CI had never confirmed (PLAN item 9, now closed)
build-lectures Real HTML build with the default -W --keep-going, run on the cache-restored conda env (proving the cached env is functional, not just present), with the executed cell's output asserted in the HTML
build-jupyter-cache (3 jobs) Smoke build of the fixture with all outputs asserted, then a full round-trip: restore-jupyter-cache restores both the build cache and the execution cache it saved — the real cache.yml → ci.yml consumer flow
harness-summary Single needs:-fan-in job as a branch-protection anchor

What the harness caught on its first run

Both are real consumer-facing defects in the standard-runner (non-container) path, invisible in production because the lecture repos build in containers. Both fixes are included here, and the harness run on this PR is the regression test for both.

1. setup-environment: the conda env cache could be saved but never restored. The cache was rooted at ${CONDA}/envs, whose parent is root-owned on hosted runners — every restore died in tar (Cannot utime/Cannot change mode: Operation not permitted) and actions/cache reported a miss, so every standard-mode run paid the full environment solve and a partial extraction could leave a mangled env behind. The cache is now rooted at the runner-owned ${CONDA}/envs/<environment-name>. This is precisely the #33/#78 validation gap PLAN item 9 flagged ("if ${CONDA}/envs is wrong on a hosted runner it's a one-line fix" — it was).

2. build-lectures: a successful build failed the step anyway on hosted runners. The build step runs in a login shell (bash -l, required for conda activation) and ended with exit $BUILD_EXIT_CODE. The exit builtin triggers ~/.bash_logout processing, where Ubuntu's clear_console fails on a headless runner at SHLVL=1 and its status overrides the one passed to exit — instrumentation showed jb exit code: 0, reached exit, code: 0, then Process completed with exit code 1. The step now returns its status by ending the script instead of calling exit. Containers are unaffected (root has no ~/.bash_logout).

One consequence of fix 2: build-jupyter-cache's internal build-lectures@v0 still carries the old exit pattern until the v0 tag moves, so the bjc-smoke job removes ~/.bash_logout as a marked temporary workaround — itself a live demonstration of the sibling-pin constraint the #100 proposal describes. Drop that step after the next release.

Determinism

Every job salts its fixture with run_id-run_attempt, so cache keys are unique per run and a miss assertion can never be polluted by caches from earlier runs, re-runs, or other branches. Miss assertions use the build cache type only: the execution type's bare jupyter-cache- restore-key fallback (deliberate — .jupyter_cache is content-addressed and self-revalidating) makes a genuine execution miss unconstructible once any execution cache exists in scope, so those jobs assert on exact key match and sentinel content instead, which a wrong restore cannot fake.

The fixture

.github/fixtures/mini-lectures/ is a two-page jupyter-book with execute_notebooks: cache and one real executed code cell, so builds populate a genuine _build/.jupyter_cache — the artifact these actions exist to move around. The existing container fixture builds with execution off and structurally cannot exercise that (noted in #108). The environment is intentionally minimal (python + jupyter-book + ipykernel): this harness tests action logic, not the science stack, which belongs to the stage-2 canary on the real containers.

What this deliberately does not cover

build-jupyter-cache's internal setup-environment/build-lectures calls execute @v0, not the PR — GitHub forbids expressions in uses:, so that chain is untestable pre-release by construction (the smoke job documents this in place). The publish and preview actions need real deploy targets and secrets. Both are stage 2+ of #100.

The full harness is green on this PR: all 13 jobs, including both round-trips, with the two fixes above in place.

🤖 Generated with Claude Code

First stage of #100. The action logic shipped with no automated tests
— restore-jupyter-cache's save mode was dead on arrival in v0.8.0
(#104) and survived eight months because nothing in CI exercised it.

Adds .github/workflows/test-actions.yml, which runs the actions via ./
local paths (the PR's code, not a released ref) against per-run salted
fixtures, asserting on outputs and the filesystem:

- restore-jupyter-cache: both cache types x both modes, fail-on-miss
  firing and staying quiet, payload round-trips
- setup-environment: standard-mode conda cache two-run miss->hit chain
  (PLAN item 9 — the #33/#78 path CI had never confirmed)
- build-lectures: real executed HTML build on the cache-restored env
- build-jupyter-cache: smoke build, then a full round-trip into
  restore-jupyter-cache for both cache types

The committed .github/fixtures/mini-lectures fixture executes a real
code cell so _build/.jupyter_cache is genuinely populated — the
container test book builds with execution off and cannot.

Not covered, by construction: build-jupyter-cache's internal @v0
sibling calls (GitHub forbids expressions in uses:) and the
publish/preview actions (need real deploy targets). Both belong to the
post-release canary proposed in #100.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 23, 2026 03:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the first permanent, action-level CI harness for this repository’s composite actions (stage 1 of #100), ensuring PRs exercise the PR’s action code via local uses: ./... paths and validating cache/env behavior that previously went untested in CI.

Changes:

  • Introduces .github/workflows/test-actions.yml, a PR/push/dispatch workflow that asserts composite-action outputs and restored filesystem state across restore-jupyter-cache, setup-environment, build-lectures, and build-jupyter-cache.
  • Adds a committed minimal Jupyter Book fixture (.github/fixtures/mini-lectures/) that executes notebooks with caching enabled to produce a real _build/.jupyter_cache.
  • Documents the harness in TESTING.md, and records it in CHANGELOG.md and PLAN.md.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.

Show a summary per file
File Description
TESTING.md Documents the new action-level PR harness and what it covers.
PLAN.md Marks the standard-mode conda-cache CI coverage item as completed via test-actions.yml.
CHANGELOG.md Adds an Unreleased entry describing the new harness and fixture.
.github/workflows/test-actions.yml New CI workflow that runs action-level assertions and provides a single fan-in job for branch protection.
.github/fixtures/mini-lectures/README.md Explains the purpose and design of the new minimal, executed fixture.
.github/fixtures/mini-lectures/lectures/lecture1.md Adds an executed code cell whose output is asserted in the harness.
.github/fixtures/mini-lectures/lectures/intro.md Minimal landing page for the fixture book.
.github/fixtures/mini-lectures/lectures/_toc.yml Defines a tiny two-page Jupyter Book structure.
.github/fixtures/mini-lectures/lectures/_config.yml Enables execute_notebooks: cache and avoids warnings under -W by building only TOC files.
.github/fixtures/mini-lectures/environment.yml Minimal environment for harness builds (python + jupyter-book + ipykernel).

mmcky and others added 5 commits July 23, 2026 13:52
…rent

Caught by the harness on its first run. The standard-mode conda cache
was rooted at ${CONDA}/envs, whose parent is root-owned on hosted
runners: the save works, but every restore dies in tar (Cannot
utime/chmod on envs) and actions/cache reports a miss — so the cache
has never restored, every run pays the full solve, and a partial
extraction can leave a mangled env behind for conda env update to
patch. Root the cache at the runner-owned envs/<environment-name>
instead, which round-trips cleanly.

Old-path caches become unreachable (path is part of the cache
version); the first run after this re-saves under the new path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second defect caught by the harness. The build step runs in a login
shell (bash -l, needed for conda activation) and ended with
exit $BUILD_EXIT_CODE. On hosted runners the exit builtin triggers
~/.bash_logout processing at SHLVL=1, where Ubuntu's clear_console
fails on a headless runner and its status overrides the one passed to
exit — a successful jb build printed its banner and the step still
returned 1. Verified by instrumentation: "jb exit code: 0",
"reached exit, code: 0", then "Process completed with exit code 1".

Return the status by ending the script instead of calling exit.
Containers are unaffected (root has no ~/.bash_logout), which is why
production lecture builds never surfaced this.

The harness's bjc-smoke job removes ~/.bash_logout as a temporary
workaround because build-jupyter-cache's internal build-lectures@v0
still carries the old exit pattern until the v0 tag moves.

Also removes the temporary diagnostic job and instrumentation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants