-
Notifications
You must be signed in to change notification settings - Fork 0
fix(coverage): keep native fuzz locks out of generic images #763
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
seonghobae
merged 3 commits into
fix/opencode-coverage-failure-diagnostics
from
fix/coverage-skip-native-fuzz-locks
Aug 5, 2026
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # Changelog | ||
|
|
||
| All notable changes to the ContextualWisdomLab central GitHub control plane are documented in this file. | ||
|
|
||
| The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versioned releases follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). | ||
|
|
||
| ## [Unreleased] | ||
|
|
||
| ### Fixed | ||
|
|
||
| - Keep the native Atheris fuzz-engine lock in dedicated repository fuzz workflows instead of installing it in the generic OpenCode coverage image; immutable hash-pinned property and regression test locks remain eligible for central coverage materialization. | ||
|
|
||
| ### Documentation | ||
|
|
||
| - Add an APA 7 doctoring record for the generic coverage/native fuzz-engine dependency boundary, exact-base trust model, verification fixture, limitations, and rollback requirements. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # Generic coverage versus native fuzz-engine lock boundary | ||
|
|
||
| ## Decision | ||
|
|
||
| The central OpenCode coverage image materializes immutable, hash-pinned dependencies needed to import selected production modules and run their ordinary tests. It does not install native coverage-guided fuzz engines that are executed only by dedicated repository fuzz workflows. | ||
|
|
||
| `requirements-atheris.txt` is therefore classified as a native fuzz-engine lock and excluded from generic coverage materialization. The classification is exact-name based and path-independent. Hash-pinned property and regression locks such as `requirements-property.txt` and `requirements-fuzz-regression.txt` remain eligible. | ||
|
|
||
| ## Technical rationale | ||
|
|
||
| Atheris is a coverage-guided native Python fuzzer built on libFuzzer. Its runtime role is to instrument and repeatedly execute fuzz targets, not to provide application imports required by an ordinary coverage.py test run. Installing an interpreter- and platform-specific native fuzz runtime in every generic coverage image adds an unrelated native artifact compatibility gate before application coverage begins. | ||
|
|
||
| Coverage.py measures execution of Python programs and can report statement and branch coverage for the selected test process without Atheris. The central reviewer therefore preserves two independent verification layers: | ||
|
|
||
| 1. repository Fuzz workflows install and execute the native fuzz engine against real fuzz targets; | ||
| 2. central OpenCode coverage evidence installs ordinary import/test dependencies and measures the selected production surface. | ||
|
|
||
| Separating these layers avoids converting a native fuzz toolchain mismatch into a source-coverage review failure while retaining both gates. | ||
|
|
||
| ## Trust boundary | ||
|
|
||
| The materializer still reads every candidate only from the exact validated pull-request base commit. Pull-request-mutated dependency files never enter the networked image-build stage. Every included lock must remain hash-pinned, and malformed Git metadata, unsafe paths, non-blob entries, unpinned requirements, and unsafe output destinations remain fail-closed. | ||
|
|
||
| The exclusion reduces trusted inputs. It does not introduce an unhashed fallback, download a replacement package, or suppress an application/test import failure. Dedicated Fuzz required workflows continue to install `requirements-atheris.txt` directly. | ||
|
|
||
| ## Verification evidence | ||
|
|
||
| A real temporary Git repository fixture contains: | ||
|
|
||
| - `fuzz/requirements-atheris.txt`; | ||
| - `fuzz/requirements-property.txt`; | ||
| - `services/example_service/requirements-fuzz-regression.txt`. | ||
|
|
||
| The test commits these files as the immutable base, materializes that exact revision, and proves that only the property and regression locks appear in the generated manifest. A second contract proves exact-name classification so a substring or directory name cannot broaden the exclusion. | ||
|
|
||
| The changed helper and integration path are subject to the central 100% statement, branch, and docstring gates. | ||
|
|
||
| ## Operational limits | ||
|
|
||
| The exact-name set initially contains only `requirements-atheris.txt`. Another native engine must not be added through a wildcard or informal comment. It requires separate artifact-role evidence, a regression fixture, review, and changelog entry. | ||
|
|
||
| This boundary does not claim that Atheris is optional for fuzzing. It is optional only for the generic OpenCode import/coverage image. Repositories remain responsible for realistic dedicated fuzz execution and crash-regression evidence. | ||
|
|
||
| ## Rollback | ||
|
|
||
| Rollback removes the exact-name classifier and its fixture. Before rollback, operators must confirm that every supported central coverage interpreter can install every repository's Atheris lock and that doing so provides coverage evidence not already supplied by the dedicated Fuzz workflow. Otherwise rollback recreates the false-negative review condition documented here. | ||
|
|
||
| ## APA 7 references | ||
|
|
||
| Batchelder, N. (2026). *Coverage.py documentation*. https://coverage.readthedocs.io/ | ||
|
|
||
| Google. (2026). *Atheris: A coverage-guided, native Python fuzzer* [Computer software]. GitHub. https://github.com/google/atheris | ||
|
|
||
| Python Packaging Authority. (2026). *Dependency specifiers*. Python Packaging User Guide. https://packaging.python.org/en/latest/specifications/dependency-specifiers/ | ||
|
|
||
| Semgrep, Inc. (2026). *Sample continuous integration configurations*. https://semgrep.dev/docs/semgrep-ci/sample-ci-configs |
129 changes: 129 additions & 0 deletions
129
docs/superpowers/plans/2026-08-05-coverage-native-fuzz-lock-boundary.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| # Coverage Native-Fuzz Lock Boundary Implementation Plan | ||
|
|
||
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. | ||
|
|
||
| **Goal:** Prevent native fuzz-engine toolchain locks from entering generic OpenCode coverage images while retaining hash-pinned property and test dependencies. | ||
|
|
||
| **Architecture:** Add one exact-name lock-role classifier to the trusted-base Python dependency materializer and evaluate it before blob selection. Protect the boundary with real temporary-Git fixtures, 100% production statement/branch coverage, docstrings, and source-backed doctoring. | ||
|
|
||
| **Tech Stack:** Python 3.10+, `pathlib`, Git CLI read-only commands, pytest, pytest-cov. | ||
|
|
||
| ## Global Constraints | ||
|
|
||
| - Continue reading dependency metadata only from the validated base commit. | ||
| - Do not change OpenCode, Noema, Strix, NVIDIA NIM, or reviewer credential names/scopes. | ||
| - Do not weaken `--require-hashes`, output bounds, symlink rejection, or malformed-tree failure. | ||
| - Every changed production helper must have a docstring and 100% statement/branch coverage. | ||
| - Document current authoritative sources in APA 7 format. | ||
| - Update `CHANGELOG.md`. | ||
|
|
||
| --- | ||
|
|
||
| ### Task 1: Add failing real-repository coverage-role evidence | ||
|
|
||
| **Files:** | ||
| - Create: `tests/test_coverage_native_fuzz_lock_boundary.py` | ||
|
|
||
| **Interfaces:** | ||
| - Consumes: `materializer.materialize(repo, base_sha, output)` | ||
| - Produces: a fixture proving `requirements-atheris.txt` is excluded while property/test locks remain | ||
|
|
||
| - [ ] **Step 1: Create a temporary Git base with three hash locks** | ||
|
|
||
| Add: | ||
|
|
||
| - `fuzz/requirements-atheris.txt`; | ||
| - `fuzz/requirements-property.txt`; | ||
| - `services/example/requirements-fuzz-regression.txt`. | ||
|
|
||
| - [ ] **Step 2: Assert only the latter two enter the manifest** | ||
|
|
||
| The exact Atheris name must be absent. The nonexact similarly named lock proves the classifier is not substring-based. | ||
|
|
||
| - [ ] **Step 3: Run the focused test and verify RED** | ||
|
|
||
| Run: `python -m pytest tests/test_coverage_native_fuzz_lock_boundary.py -q` | ||
|
|
||
| Expected: FAIL because all three files are currently selected. | ||
|
|
||
| - [ ] **Step 4: Commit the failing test** | ||
|
|
||
| ```bash | ||
| git add tests/test_coverage_native_fuzz_lock_boundary.py | ||
| git commit -m "test(coverage): exclude native fuzz engine locks" | ||
| ``` | ||
|
|
||
| ### Task 2: Implement the exact-name lock-role boundary | ||
|
|
||
| **Files:** | ||
| - Modify: `scripts/ci/materialize_base_python_requirements.py` | ||
|
|
||
| **Interfaces:** | ||
| - Produces: `_is_native_fuzz_engine_lock_name(name: str) -> bool` | ||
| - Updates: `_is_candidate_lock_name(name: str) -> bool` | ||
|
|
||
| - [ ] **Step 1: Add an immutable exact-name set** | ||
|
|
||
| The initial set contains only `requirements-atheris.txt`. | ||
|
|
||
| - [ ] **Step 2: Add the pure classifier with explanatory docstring** | ||
|
|
||
| Return true only for exact members of the immutable set. | ||
|
|
||
| - [ ] **Step 3: Exclude the native toolchain before ordinary candidate matching** | ||
|
|
||
| Keep every existing candidate and content check unchanged for other files, including the shared bounded and credential-redacting failure-summary publisher. | ||
|
|
||
| - [ ] **Step 4: Run focused tests and verify GREEN** | ||
|
|
||
| Run: `python -m pytest tests/test_coverage_native_fuzz_lock_boundary.py -q` | ||
|
|
||
| Expected: PASS. | ||
|
|
||
| - [ ] **Step 5: Run coverage and docstring gates** | ||
|
|
||
| Run the repository's full Python test, branch-coverage, compile, formatting, static-security, and interrogate commands. | ||
|
|
||
| - [ ] **Step 6: Commit** | ||
|
|
||
| ```bash | ||
| git add scripts/ci/materialize_base_python_requirements.py tests/test_coverage_native_fuzz_lock_boundary.py .github/workflows/opencode-coverage-diagnostics-ci.yml | ||
| git commit -m "fix(coverage): skip native fuzz engine locks" | ||
| ``` | ||
|
|
||
| ### Task 3: Record doctoring and release evidence | ||
|
|
||
| **Files:** | ||
| - Create: `docs/doctoring/coverage-native-fuzz-lock-boundary.md` | ||
| - Create or modify: `CHANGELOG.md` | ||
|
|
||
| **Interfaces:** | ||
| - Produces: operational rationale, standards traceability, and Unreleased evidence | ||
|
|
||
| - [ ] **Step 1: Document the role boundary** | ||
|
|
||
| Record why Atheris belongs to dedicated fuzz execution rather than generic import coverage, the immutable-base trust boundary, limitations, and rollback. | ||
|
|
||
| - [ ] **Step 2: Add APA 7 references** | ||
|
|
||
| Cite official Atheris, Python packaging, coverage.py, and Semgrep/GitHub Actions material relevant to the decision. | ||
|
|
||
| - [ ] **Step 3: Update the changelog** | ||
|
|
||
| Add the generic coverage materializer correction under `Unreleased / Fixed`. | ||
|
|
||
| - [ ] **Step 4: Run full exact-slice verification and commit** | ||
|
|
||
| ```bash | ||
| git add docs/doctoring/coverage-native-fuzz-lock-boundary.md CHANGELOG.md | ||
| git commit -m "docs(coverage): record native fuzz lock boundary" | ||
| ``` | ||
|
|
||
| ### Task 4: Validate, review, and integrate | ||
|
|
||
| - [ ] **Step 1: Open a focused PR closing #762** | ||
| - [ ] **Step 2: Resolve every automated and human review finding** | ||
| - [ ] **Step 3: Re-run all exact-head central checks and independent review** | ||
| - [ ] **Step 4: Merge prerequisite #759 without administrative bypass** | ||
| - [ ] **Step 5: Retarget this PR to integrated `main`, rerun exact-head gates, and merge without bypass** | ||
| - [ ] **Step 6: Re-dispatch coverage review for contextual-orchestrator #96, merge it when green, then continue its dependency-ordered PR queue** |
47 changes: 47 additions & 0 deletions
47
docs/superpowers/specs/2026-08-05-coverage-native-fuzz-lock-boundary-design.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # Coverage Native-Fuzz Lock Boundary Design | ||
|
|
||
| ## Status | ||
|
|
||
| Approved for autonomous implementation under issue #762. | ||
|
|
||
| ## Problem | ||
|
|
||
| The central trusted-base dependency materializer currently treats every hash-pinned `requirements*.txt` file as part of the generic offline coverage image. That is too broad: native coverage-guided fuzz engines are execution toolchains for dedicated fuzz jobs, not import dependencies for ordinary statement/branch coverage. Selecting `requirements-atheris.txt` can make a coverage review fail on an interpreter-specific native wheel before any selected application or test code runs. | ||
|
|
||
| ## Decision | ||
|
|
||
| Classify exact native fuzz-engine lock names as coverage-incompatible. The first supported name is `requirements-atheris.txt`. | ||
|
|
||
| The classification is path-independent so standalone repositories and nested MSA modules receive the same treatment. Hash-pinned property/unit-test locks such as `requirements-property.txt` remain eligible. Exact-base Git reads, immutable-source selection, hash requirements, bounded output, and deterministic ordering remain unchanged. | ||
|
|
||
| ## Components | ||
|
|
||
| ### Lock-role classifier | ||
|
|
||
| A pure helper receives one file name and returns whether it represents a native fuzz runtime. The helper uses an immutable exact-name set rather than substring matching, which avoids excluding unrelated dependencies such as `requirements-fuzz-regression.txt`. | ||
|
|
||
| ### Materializer integration | ||
|
|
||
| `base_hash_locks` checks the role before reading or exporting the blob. Excluded native fuzz locks never enter the Docker build context or trusted coverage image. Dedicated repository fuzz workflows continue installing those locks directly. | ||
|
|
||
| ### Verification | ||
|
|
||
| A real temporary Git repository proves that: | ||
|
|
||
| - `fuzz/requirements-atheris.txt` is excluded; | ||
| - `fuzz/requirements-property.txt` remains materialized; | ||
| - a similarly named non-exact lock remains eligible; | ||
| - existing exact-base, hash, symlink, malformed-tree, uv, and CLI contracts remain green; | ||
| - changed production helpers retain 100% statement/branch coverage and docstrings. | ||
|
|
||
| ## Security boundary | ||
|
|
||
| The change reduces trusted coverage-image inputs; it does not accept PR-controlled dependency metadata or weaken `--require-hashes`. Exclusion is limited to a toolchain that the generic coverage job never executes. Adding another excluded name requires a separate reviewed change, evidence, and test. | ||
|
|
||
| ## Non-goals | ||
|
|
||
| - changing the dedicated Fuzz required workflow; | ||
| - changing OpenCode/Noema/Strix models or credentials; | ||
| - changing NVIDIA NIM keys; | ||
| - suppressing coverage failures in selected application/test dependencies; | ||
| - interpreting arbitrary requirement contents or comments as trusted role metadata. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import subprocess | ||
| from pathlib import Path | ||
|
|
||
| from scripts.ci import materialize_base_python_requirements as materializer | ||
|
|
||
|
|
||
| def _git(repo: Path, *args: str) -> str: | ||
| """Run one deterministic Git command inside a temporary fixture repository.""" | ||
|
|
||
| return subprocess.run( | ||
| ["git", "-C", str(repo), *args], | ||
| check=True, | ||
| capture_output=True, | ||
| text=True, | ||
| ).stdout.strip() | ||
|
|
||
|
|
||
| def _hashed_requirement(package_name: str, digest_character: str) -> str: | ||
| """Create one syntactically hash-pinned requirement fixture line.""" | ||
|
|
||
| return ( | ||
| f"{package_name}==1.0.0 --hash=sha256:" | ||
| f"{digest_character * 64}\n" | ||
| ) | ||
|
|
||
|
|
||
| def test_generic_coverage_excludes_only_the_exact_native_atheris_lock( | ||
| tmp_path: Path, | ||
| ) -> None: | ||
| """Coverage retains test dependencies but never installs the Atheris toolchain.""" | ||
|
|
||
| repo = tmp_path / "repository" | ||
| repo.mkdir() | ||
| _git(repo, "init") | ||
| _git(repo, "config", "user.name", "Coverage Boundary Test") | ||
| _git(repo, "config", "user.email", "coverage-boundary@example.invalid") | ||
|
|
||
| fuzz_directory = repo / "fuzz" | ||
| fuzz_directory.mkdir() | ||
| (fuzz_directory / "requirements-atheris.txt").write_text( | ||
| _hashed_requirement("atheris", "a"), | ||
| encoding="utf-8", | ||
| ) | ||
| (fuzz_directory / "requirements-property.txt").write_text( | ||
| _hashed_requirement("hypothesis", "b"), | ||
| encoding="utf-8", | ||
| ) | ||
|
|
||
| service_directory = repo / "services" / "example_service" | ||
| service_directory.mkdir(parents=True) | ||
| (service_directory / "requirements-fuzz-regression.txt").write_text( | ||
| _hashed_requirement("pytest", "c"), | ||
| encoding="utf-8", | ||
| ) | ||
|
|
||
| _git(repo, "add", ".") | ||
| _git(repo, "commit", "-m", "base dependency roles") | ||
| base_sha = _git(repo, "rev-parse", "HEAD") | ||
|
|
||
| output_directory = tmp_path / "materialized" | ||
| manifest = materializer.materialize(repo, base_sha, output_directory) | ||
|
|
||
| assert [entry["source"] for entry in manifest] == [ | ||
| "fuzz/requirements-property.txt", | ||
| "services/example_service/requirements-fuzz-regression.txt", | ||
| ] | ||
| assert "requirements-atheris.txt" not in ( | ||
| output_directory / "manifest.json" | ||
| ).read_text(encoding="utf-8") | ||
|
|
||
|
|
||
| def test_native_fuzz_engine_classifier_uses_exact_file_names() -> None: | ||
| """Role classification cannot expand through substrings or directory names.""" | ||
|
|
||
| assert materializer._is_native_fuzz_engine_lock_name( | ||
| "requirements-atheris.txt" | ||
| ) | ||
| assert not materializer._is_native_fuzz_engine_lock_name( | ||
| "requirements-atheris-regression.txt" | ||
| ) | ||
| assert not materializer._is_native_fuzz_engine_lock_name( | ||
| "requirements-property.txt" | ||
| ) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
P1 Core changed surface (exact-name native fuzz-engine lock classifier and its fixture) is not inspectable in the trusted evidence - fail-closed until hunks are supplied
scripts/ci/materialize_base_python_requirements.py:24Suggested diff