P2 pilot: add the pandas_panel trio (realwage, countries, employ) with manifests#17
Merged
Conversation
…h manifests Second pilot (PLAN Phase 8, P2) — the cross-series shared-static case: three files consumed by BOTH lecture-python-programming and lecture-python.myst via their own-repo copies today, migrated here to a single canonical copy each. - realwage.csv — OECD real minimum wages (CC BY 4.0); constructed, builder unrecovered (inherited). - employ.csv — Eurostat employment (reuse w/ attribution, Decision 2011/833/EU); constructed, builder unrecovered (inherited). - countries.csv — WorldData.info country table; verbatim. Source is all-rights-reserved (restricted) — cached as an inherited exposure (served by the lectures for years), recorded honestly and logged for licence review in QuantEcon/workspace-lectures#20. AGENTS.md gains a one-line exception for this already-served case; it must be resolved before data.quantecon.org goes public. Integrity: each file is byte-identical across the two consuming repos (compared 2026-07-17) and its manifest sha256 matches the committed bytes, so the repoints cannot change lecture output. Upstream provenance is `unverifiable` for all three (inherited, undated, no builder) — recorded honestly per the P1 pattern. Repoints in the two lecture repos follow as separate PRs. Part of QuantEcon/meta#338; retires the last of the pandas_panel legacy-repo references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 17, 2026
There was a problem hiding this comment.
Pull request overview
Adds the “pandas_panel” P2 pilot datasets to the flat lectures/ namespace with provenance manifests, enabling two lecture series to consume a single canonical copy while recording integrity and licensing constraints.
Changes:
- Added
realwage.csv,employ.csv, andcountries.csvunderlectures/, each with a sidecar manifest capturing provenance, integrity hashes, schema expectations, and consumers. - Recorded inherited/no-builder status for the two constructed datasets via
builder: null+builder_status: unrecovered. - Updated
AGENTS.mdto document an explicit exception for caching already-publicly-served restricted datasets (with required manifest markings and tracking).
Reviewed changes
Copilot reviewed 5 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| lectures/realwage.csv | Adds the OECD real minimum wages dataset snapshot used by the pandas panel lecture. |
| lectures/realwage.csv.yml | Documents provenance/licensing, integrity hash, schema, and dual consumers for realwage.csv. |
| lectures/employ.csv.yml | Documents provenance/licensing, integrity hash, schema, and dual consumers for employ.csv. |
| lectures/countries.csv | Adds the WorldData.info country reference table snapshot (semicolon-delimited) used for continent lookup. |
| lectures/countries.csv.yml | Documents restricted licensing status, integrity hash, schema, and dual consumers for countries.csv. |
| AGENTS.md | Documents the “already served publicly” restricted-file caching exception and required tracking. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Copilot review (data-lectures#17): the column holds non-integer values for territories with hyphenated prefixes (1-684, 599-9, 44-1624, 1-284, 1-670, 1-721, 1-649), so int64 was inaccurate and would break a schema-enforcing validator. pandas reads it as object/string. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mmcky
added a commit
to QuantEcon/lecture-python-programming
that referenced
this pull request
Jul 17, 2026
* P2 pilot: read pandas_panel data from data-lectures Repoint realwage.csv, countries.csv and employ.csv from this repo's own copies under lectures/_static/lecture_specific/pandas_panel/ to the canonical copies in QuantEcon/data-lectures, and delete the local copies. Byte-compare gate (one-time, reproducible from the data-lectures manifests' sha256): the removed files are byte-identical to the migrated copies, so the repoint cannot change lecture output. realwage.csv cd60e36dc4a9e7ac1d7859dc06d5666e32f619dc6272be7c60dd0136c77fd511 countries.csv b334ddac9bf8b1788c6b9abf626959e2ff98a7f40a9dcdaa22e226d84595c3be employ.csv 2ff176e99e4a4669b7b24c603fe1889a2061eff4149d4c356fd84175803149d7 Draft until the data lands: the interim URL 404s until QuantEcon/data-lectures#17 merges. Data PR first, lecture PR second — never a non-default ref. Part of QuantEcon/meta#338. Pairs with QuantEcon/lecture-python.myst (same change). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * P2: repoint pandas_panel URLs to data-lectures (fix missing edit) The prior commit deleted the local copies but did not stage the URL change in pandas_panel.md, so the lecture still pointed at the now-deleted own-repo path. Repoint url1/url2/url3 to the canonical data-lectures copies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mmcky
added a commit
to QuantEcon/lecture-python.myst
that referenced
this pull request
Jul 17, 2026
* P2 pilot: read pandas_panel data from data-lectures Repoint realwage.csv, countries.csv and employ.csv from this repo's own copies under lectures/_static/lecture_specific/pandas_panel/ to the canonical copies in QuantEcon/data-lectures, and delete the local copies. Byte-compare gate (one-time, reproducible from the data-lectures manifests' sha256): the removed files are byte-identical to the migrated copies, so the repoint cannot change lecture output. realwage.csv cd60e36dc4a9e7ac1d7859dc06d5666e32f619dc6272be7c60dd0136c77fd511 countries.csv b334ddac9bf8b1788c6b9abf626959e2ff98a7f40a9dcdaa22e226d84595c3be employ.csv 2ff176e99e4a4669b7b24c603fe1889a2061eff4149d4c356fd84175803149d7 Draft until the data lands: the interim URL 404s until QuantEcon/data-lectures#17 merges. Data PR first, lecture PR second — never a non-default ref. Part of QuantEcon/meta#338. Pairs with QuantEcon/lecture-python-programming (same change). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * P2: repoint pandas_panel URLs to data-lectures (fix missing edit) The prior commit deleted the local copies but did not stage the URL change in pandas_panel.md, so the lecture still pointed at the now-deleted own-repo path. Repoint url1/url2/url3 to the canonical data-lectures copies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Jul 17, 2026
…tracker (#21) * Add the generated data-audit dashboard: scanner, tracker, 3-page site scripts/build_audit.py scans the 8 synced Python-family lecture repos on origin/main (clone + grep — never gh search code, which cannot find URLs), classifies every data reference into the hosting-pattern taxonomy, and renders a 3-page static dashboard (overview / migration tracker / full audit) via scripts/render_audit.py. Three sources of truth, reconciled on every build: - lectures/*.yml manifests — the migrated datasets - migration.yml (new) — the migration lifecycle + PR provenance, kept separate from the manifests so it can be archived when the migration programme completes without touching the permanent dataset records - scripts/audit_annotations.yml (new) — curated judgment (descriptions, provenance classes, live-API pedagogy) for not-yet-migrated references The strict scan fails on an unannotated data reference or a migration.yml status that disagrees with what the lectures actually read, so the dashboard cannot silently rot — the failure mode that killed the hand-built 2026-07-15 audit artifact within two days. Part of #20; taxonomy and design system carried over from that artifact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Deploy the dashboard and the published tree to GitHub Pages .github/workflows/audit-dashboard.yml builds the dashboard from shallow clones of the 8 lecture repos and deploys the default Pages site: dashboard at /, the published lectures/ tree at /lectures/, audit.json alongside for reuse. Triggers: push to main, weekly cron, manual dispatch; on PRs touching the audit inputs it runs the strict build as a guardrail without deploying. Serving lectures/ on Pages pre-stages PLAN Phase 4: when the data.quantecon.org DNS question is resolved, only the custom domain changes. Checkout uses lfs: true so LFS-tracked files publish as bytes, never pointers. Part of #20. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs: record the dashboard, its update rules, and P2 completion README gains the audit-dashboard section; AGENTS.md gains the keep-it-truthful rules (migration.yml updated in the same PR as a landing/repoint; annotations for new references; never commit site/ or audit.json) and the expanded repo map; PLAN.md checks off the Pages deploy (default domain — custom domain stays open), adds the Phase 5 workflow entry, and marks pilot P2 complete (#17, QuantEcon/lecture-python-programming#578, QuantEcon/lecture-python.myst#973). Part of #20. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review: taxonomy gap, status validation, notebook guard - render_audit.py: add the external-web pattern to PATTERN_META so a non-GitHub data URL would surface in the hosting-pattern table and distribution chart instead of being silently omitted (no current ref triggers it; the failure mode was the point) - build_audit.py: validate migration.yml status values in the scan-stage consistency check (an unknown status fails --strict with a clear message); render_audit.py's stepper raises a named error as backstop - build_audit.py: flag any .ipynb under lectures/ outside _static/ as an unscanned_notebooks problem. Every lecture source today is MyST .md and the only notebooks are builder/figure helpers (deliberately not consumers); if a notebook-format lecture ever appears, the build now refuses to miss it silently Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P2 pilot — cross-series shared static (PLAN Phase 8)
The second pilot: three files consumed by two lecture series (
lecture-python-programmingandlecture-python.myst), migrated from each series' own-repo copy to a single canonical copy here. This is the flat-namespace-with-two-consumers case, and it retires the last of thepandas_panellegacy-repo references.What lands here
redistributionrealwage.csvemploy.csvcountries.csvIntegrity (stands in for CI — this repo has none)
Each file is byte-identical across both consuming repos (git blob compare, 2026-07-17), and each manifest's
integrity.sha256matches the committed bytes — so the repoints cannot change lecture output, and the go-live consumed-file invariant holds. Upstream provenance isunverifiablefor all three (inherited, undated, no builder committed) — recorded honestly per the P1 pattern rather than faked.The
countries.csvlicence wrinkleWorldData.info is all-rights-reserved — a no-redistribution term that AGENTS.md would normally block. It is cached here as an inherited exposure: the lectures have served this exact file publicly for years, so caching it here adds no new exposure, while hiding it would obscure a real one. It is recorded honestly (
redistribution: restricted+note) and logged for licence review in QuantEcon/workspace-lectures#20. AGENTS.md gains a one-line exception for the already-served case; the file must be resolved (permission, an open replacement, or removal) beforedata.quantecon.orgis promoted as a public open-data host. The lecture only uses two of its columns (Country (en),Continent) for a continent lookup.Follows
Repoints in the two lecture repos are separate PRs (data PR first, per the ordering rule); each will carry the one-time byte-compare gate in its body. Then P2 is ticked on the cutover tracker #15.
Part of QuantEcon/meta#338 (pilot) and QuantEcon/QuantEcon.manual#108 (convention).