ci: add action-level PR harness for cache and environment actions#114
Open
mmcky wants to merge 6 commits into
Open
ci: add action-level PR harness for cache and environment actions#114mmcky wants to merge 6 commits into
mmcky wants to merge 6 commits into
Conversation
First stage of #100. The action logic shipped with no automated tests — restore-jupyter-cache's save mode was dead on arrival in v0.8.0 (#104) and survived eight months because nothing in CI exercised it. Adds .github/workflows/test-actions.yml, which runs the actions via ./ local paths (the PR's code, not a released ref) against per-run salted fixtures, asserting on outputs and the filesystem: - restore-jupyter-cache: both cache types x both modes, fail-on-miss firing and staying quiet, payload round-trips - setup-environment: standard-mode conda cache two-run miss->hit chain (PLAN item 9 — the #33/#78 path CI had never confirmed) - build-lectures: real executed HTML build on the cache-restored env - build-jupyter-cache: smoke build, then a full round-trip into restore-jupyter-cache for both cache types The committed .github/fixtures/mini-lectures fixture executes a real code cell so _build/.jupyter_cache is genuinely populated — the container test book builds with execution off and cannot. Not covered, by construction: build-jupyter-cache's internal @v0 sibling calls (GitHub forbids expressions in uses:) and the publish/preview actions (need real deploy targets). Both belong to the post-release canary proposed in #100. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds the first permanent, action-level CI harness for this repository’s composite actions (stage 1 of #100), ensuring PRs exercise the PR’s action code via local uses: ./... paths and validating cache/env behavior that previously went untested in CI.
Changes:
- Introduces
.github/workflows/test-actions.yml, a PR/push/dispatch workflow that asserts composite-action outputs and restored filesystem state acrossrestore-jupyter-cache,setup-environment,build-lectures, andbuild-jupyter-cache. - Adds a committed minimal Jupyter Book fixture (
.github/fixtures/mini-lectures/) that executes notebooks with caching enabled to produce a real_build/.jupyter_cache. - Documents the harness in
TESTING.md, and records it inCHANGELOG.mdandPLAN.md.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
TESTING.md |
Documents the new action-level PR harness and what it covers. |
PLAN.md |
Marks the standard-mode conda-cache CI coverage item as completed via test-actions.yml. |
CHANGELOG.md |
Adds an Unreleased entry describing the new harness and fixture. |
.github/workflows/test-actions.yml |
New CI workflow that runs action-level assertions and provides a single fan-in job for branch protection. |
.github/fixtures/mini-lectures/README.md |
Explains the purpose and design of the new minimal, executed fixture. |
.github/fixtures/mini-lectures/lectures/lecture1.md |
Adds an executed code cell whose output is asserted in the harness. |
.github/fixtures/mini-lectures/lectures/intro.md |
Minimal landing page for the fixture book. |
.github/fixtures/mini-lectures/lectures/_toc.yml |
Defines a tiny two-page Jupyter Book structure. |
.github/fixtures/mini-lectures/lectures/_config.yml |
Enables execute_notebooks: cache and avoids warnings under -W by building only TOC files. |
.github/fixtures/mini-lectures/environment.yml |
Minimal environment for harness builds (python + jupyter-book + ipykernel). |
…rent
Caught by the harness on its first run. The standard-mode conda cache
was rooted at ${CONDA}/envs, whose parent is root-owned on hosted
runners: the save works, but every restore dies in tar (Cannot
utime/chmod on envs) and actions/cache reports a miss — so the cache
has never restored, every run pays the full solve, and a partial
extraction can leave a mangled env behind for conda env update to
patch. Root the cache at the runner-owned envs/<environment-name>
instead, which round-trips cleanly.
Old-path caches become unreachable (path is part of the cache
version); the first run after this re-saves under the new path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second defect caught by the harness. The build step runs in a login shell (bash -l, needed for conda activation) and ended with exit $BUILD_EXIT_CODE. On hosted runners the exit builtin triggers ~/.bash_logout processing at SHLVL=1, where Ubuntu's clear_console fails on a headless runner and its status overrides the one passed to exit — a successful jb build printed its banner and the step still returned 1. Verified by instrumentation: "jb exit code: 0", "reached exit, code: 0", then "Process completed with exit code 1". Return the status by ending the script instead of calling exit. Containers are unaffected (root has no ~/.bash_logout), which is why production lecture builds never surfaced this. The harness's bjc-smoke job removes ~/.bash_logout as a temporary workaround because build-jupyter-cache's internal build-lectures@v0 still carries the old exit pattern until the v0 tag moves. Also removes the temporary diagnostic job and instrumentation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First stage of #100 (the PR-time harness — harness A in the design proposal there). Stages 2–4 (the
test-lecturescanary repo, preview/publish coverage, tag-push trigger) need a separate repo and secrets and stay with the issue, which should remain open after this merges.What this adds
A new workflow,
.github/workflows/test-actions.yml, that tests the composite actions themselves — the first automated tests this layer has ever had. It invokes the actions via./local paths, so it exercises the code on the PR, not a released ref, and it runs on every PR that touches a covered action (plus pushes tomainand manual dispatch). It is the permanent form of the throwaway harness that verified the #104 fix, which shipped dead on arrival in v0.8.0 and survived eight months because nothing in CI exercised it.restore-jupyter-cacheunit jobs (7, seconds each, no conda)fail-on-missfires on a genuine miss and stays quiet on a hit; payloads round-trip byte-for-byte via a STAMP sentinelsetup-environmentchain (2 jobs)build-lectures-W --keep-going, run on the cache-restored conda env (proving the cached env is functional, not just present), with the executed cell's output asserted in the HTMLbuild-jupyter-cache(3 jobs)restore-jupyter-cacherestores both the build cache and the execution cache it saved — the real cache.yml → ci.yml consumer flowharness-summaryneeds:-fan-in job as a branch-protection anchorWhat the harness caught on its first run
Both are real consumer-facing defects in the standard-runner (non-container) path, invisible in production because the lecture repos build in containers. Both fixes are included here, and the harness run on this PR is the regression test for both.
1.
setup-environment: the conda env cache could be saved but never restored. The cache was rooted at${CONDA}/envs, whose parent is root-owned on hosted runners — every restore died in tar (Cannot utime/Cannot change mode: Operation not permitted) andactions/cachereported a miss, so every standard-mode run paid the full environment solve and a partial extraction could leave a mangled env behind. The cache is now rooted at the runner-owned${CONDA}/envs/<environment-name>. This is precisely the #33/#78 validation gap PLAN item 9 flagged ("if${CONDA}/envsis wrong on a hosted runner it's a one-line fix" — it was).2.
build-lectures: a successful build failed the step anyway on hosted runners. The build step runs in a login shell (bash -l, required for conda activation) and ended withexit $BUILD_EXIT_CODE. The exit builtin triggers~/.bash_logoutprocessing, where Ubuntu'sclear_consolefails on a headless runner atSHLVL=1and its status overrides the one passed toexit— instrumentation showedjb exit code: 0,reached exit, code: 0, thenProcess completed with exit code 1. The step now returns its status by ending the script instead of callingexit. Containers are unaffected (root has no~/.bash_logout).One consequence of fix 2:
build-jupyter-cache's internalbuild-lectures@v0still carries the old exit pattern until the v0 tag moves, so thebjc-smokejob removes~/.bash_logoutas a marked temporary workaround — itself a live demonstration of the sibling-pin constraint the #100 proposal describes. Drop that step after the next release.Determinism
Every job salts its fixture with
run_id-run_attempt, so cache keys are unique per run and a miss assertion can never be polluted by caches from earlier runs, re-runs, or other branches. Miss assertions use the build cache type only: the execution type's barejupyter-cache-restore-key fallback (deliberate —.jupyter_cacheis content-addressed and self-revalidating) makes a genuine execution miss unconstructible once any execution cache exists in scope, so those jobs assert on exact key match and sentinel content instead, which a wrong restore cannot fake.The fixture
.github/fixtures/mini-lectures/is a two-page jupyter-book withexecute_notebooks: cacheand one real executed code cell, so builds populate a genuine_build/.jupyter_cache— the artifact these actions exist to move around. The existing container fixture builds with execution off and structurally cannot exercise that (noted in #108). The environment is intentionally minimal (python + jupyter-book + ipykernel): this harness tests action logic, not the science stack, which belongs to the stage-2 canary on the real containers.What this deliberately does not cover
build-jupyter-cache's internalsetup-environment/build-lecturescalls execute@v0, not the PR — GitHub forbids expressions inuses:, so that chain is untestable pre-release by construction (the smoke job documents this in place). The publish and preview actions need real deploy targets and secrets. Both are stage 2+ of #100.The full harness is green on this PR: all 13 jobs, including both round-trips, with the two fixes above in place.
🤖 Generated with Claude Code