Skip to content

P2 pilot: add the pandas_panel trio (realwage, countries, employ) with manifests#17

Merged
mmcky merged 2 commits into
mainfrom
datasets/p2-pandas-panel
Jul 17, 2026
Merged

P2 pilot: add the pandas_panel trio (realwage, countries, employ) with manifests#17
mmcky merged 2 commits into
mainfrom
datasets/p2-pandas-panel

Conversation

@mmcky

@mmcky mmcky commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

P2 pilot — cross-series shared static (PLAN Phase 8)

The second pilot: three files consumed by two lecture series (lecture-python-programming and lecture-python.myst), migrated from each series' own-repo copy to a single canonical copy here. This is the flat-namespace-with-two-consumers case, and it retires the last of the pandas_panel legacy-repo references.

What lands here

File Source Class Licence redistribution
realwage.csv OECD real minimum wages constructed (builder unrecovered) CC BY 4.0 permitted
employ.csv Eurostat employment constructed (builder unrecovered) Reuse w/ attribution (Decision 2011/833/EU) permitted
countries.csv WorldData.info country table verbatim © WorldData.info, all rights reserved restricted

Integrity (stands in for CI — this repo has none)

Each file is byte-identical across both consuming repos (git blob compare, 2026-07-17), and each manifest's integrity.sha256 matches the committed bytes — so the repoints cannot change lecture output, and the go-live consumed-file invariant holds. Upstream provenance is unverifiable for all three (inherited, undated, no builder committed) — recorded honestly per the P1 pattern rather than faked.

The countries.csv licence wrinkle

WorldData.info is all-rights-reserved — a no-redistribution term that AGENTS.md would normally block. It is cached here as an inherited exposure: the lectures have served this exact file publicly for years, so caching it here adds no new exposure, while hiding it would obscure a real one. It is recorded honestly (redistribution: restricted + note) and logged for licence review in QuantEcon/workspace-lectures#20. AGENTS.md gains a one-line exception for the already-served case; the file must be resolved (permission, an open replacement, or removal) before data.quantecon.org is promoted as a public open-data host. The lecture only uses two of its columns (Country (en), Continent) for a continent lookup.

Follows

Repoints in the two lecture repos are separate PRs (data PR first, per the ordering rule); each will carry the one-time byte-compare gate in its body. Then P2 is ticked on the cutover tracker #15.

Part of QuantEcon/meta#338 (pilot) and QuantEcon/QuantEcon.manual#108 (convention).

…h manifests

Second pilot (PLAN Phase 8, P2) — the cross-series shared-static case: three
files consumed by BOTH lecture-python-programming and lecture-python.myst via
their own-repo copies today, migrated here to a single canonical copy each.

- realwage.csv  — OECD real minimum wages (CC BY 4.0); constructed, builder
  unrecovered (inherited).
- employ.csv    — Eurostat employment (reuse w/ attribution, Decision
  2011/833/EU); constructed, builder unrecovered (inherited).
- countries.csv — WorldData.info country table; verbatim. Source is
  all-rights-reserved (restricted) — cached as an inherited exposure (served by
  the lectures for years), recorded honestly and logged for licence review in
  QuantEcon/workspace-lectures#20. AGENTS.md gains a one-line exception for this
  already-served case; it must be resolved before data.quantecon.org goes
  public.

Integrity: each file is byte-identical across the two consuming repos (compared
2026-07-17) and its manifest sha256 matches the committed bytes, so the repoints
cannot change lecture output. Upstream provenance is `unverifiable` for all
three (inherited, undated, no builder) — recorded honestly per the P1 pattern.

Repoints in the two lecture repos follow as separate PRs. Part of
QuantEcon/meta#338; retires the last of the pandas_panel legacy-repo references.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the “pandas_panel” P2 pilot datasets to the flat lectures/ namespace with provenance manifests, enabling two lecture series to consume a single canonical copy while recording integrity and licensing constraints.

Changes:

  • Added realwage.csv, employ.csv, and countries.csv under lectures/, each with a sidecar manifest capturing provenance, integrity hashes, schema expectations, and consumers.
  • Recorded inherited/no-builder status for the two constructed datasets via builder: null + builder_status: unrecovered.
  • Updated AGENTS.md to document an explicit exception for caching already-publicly-served restricted datasets (with required manifest markings and tracking).

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
lectures/realwage.csv Adds the OECD real minimum wages dataset snapshot used by the pandas panel lecture.
lectures/realwage.csv.yml Documents provenance/licensing, integrity hash, schema, and dual consumers for realwage.csv.
lectures/employ.csv.yml Documents provenance/licensing, integrity hash, schema, and dual consumers for employ.csv.
lectures/countries.csv Adds the WorldData.info country reference table snapshot (semicolon-delimited) used for continent lookup.
lectures/countries.csv.yml Documents restricted licensing status, integrity hash, schema, and dual consumers for countries.csv.
AGENTS.md Documents the “already served publicly” restricted-file caching exception and required tracking.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread lectures/countries.csv.yml Outdated
Copilot review (data-lectures#17): the column holds non-integer values for
territories with hyphenated prefixes (1-684, 599-9, 44-1624, 1-284, 1-670,
1-721, 1-649), so int64 was inaccurate and would break a schema-enforcing
validator. pandas reads it as object/string.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mmcky
mmcky merged commit e73500f into main Jul 17, 2026
1 check passed
@mmcky
mmcky deleted the datasets/p2-pandas-panel branch July 17, 2026 01:23
mmcky added a commit to QuantEcon/lecture-python-programming that referenced this pull request Jul 17, 2026
* P2 pilot: read pandas_panel data from data-lectures

Repoint realwage.csv, countries.csv and employ.csv from this repo's own copies
under lectures/_static/lecture_specific/pandas_panel/ to the canonical copies in
QuantEcon/data-lectures, and delete the local copies.

Byte-compare gate (one-time, reproducible from the data-lectures manifests'
sha256): the removed files are byte-identical to the migrated copies, so the
repoint cannot change lecture output.
  realwage.csv  cd60e36dc4a9e7ac1d7859dc06d5666e32f619dc6272be7c60dd0136c77fd511
  countries.csv b334ddac9bf8b1788c6b9abf626959e2ff98a7f40a9dcdaa22e226d84595c3be
  employ.csv    2ff176e99e4a4669b7b24c603fe1889a2061eff4149d4c356fd84175803149d7

Draft until the data lands: the interim URL 404s until QuantEcon/data-lectures#17
merges. Data PR first, lecture PR second — never a non-default ref.

Part of QuantEcon/meta#338. Pairs with QuantEcon/lecture-python.myst (same change).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* P2: repoint pandas_panel URLs to data-lectures (fix missing edit)

The prior commit deleted the local copies but did not stage the URL change in
pandas_panel.md, so the lecture still pointed at the now-deleted own-repo path.
Repoint url1/url2/url3 to the canonical data-lectures copies.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mmcky added a commit to QuantEcon/lecture-python.myst that referenced this pull request Jul 17, 2026
* P2 pilot: read pandas_panel data from data-lectures

Repoint realwage.csv, countries.csv and employ.csv from this repo's own copies
under lectures/_static/lecture_specific/pandas_panel/ to the canonical copies in
QuantEcon/data-lectures, and delete the local copies.

Byte-compare gate (one-time, reproducible from the data-lectures manifests'
sha256): the removed files are byte-identical to the migrated copies, so the
repoint cannot change lecture output.
  realwage.csv  cd60e36dc4a9e7ac1d7859dc06d5666e32f619dc6272be7c60dd0136c77fd511
  countries.csv b334ddac9bf8b1788c6b9abf626959e2ff98a7f40a9dcdaa22e226d84595c3be
  employ.csv    2ff176e99e4a4669b7b24c603fe1889a2061eff4149d4c356fd84175803149d7

Draft until the data lands: the interim URL 404s until QuantEcon/data-lectures#17
merges. Data PR first, lecture PR second — never a non-default ref.

Part of QuantEcon/meta#338. Pairs with QuantEcon/lecture-python-programming (same change).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* P2: repoint pandas_panel URLs to data-lectures (fix missing edit)

The prior commit deleted the local copies but did not stage the URL change in
pandas_panel.md, so the lecture still pointed at the now-deleted own-repo path.
Repoint url1/url2/url3 to the canonical data-lectures copies.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mmcky added a commit that referenced this pull request Jul 17, 2026
…tracker (#21)

* Add the generated data-audit dashboard: scanner, tracker, 3-page site

scripts/build_audit.py scans the 8 synced Python-family lecture repos on
origin/main (clone + grep — never gh search code, which cannot find URLs),
classifies every data reference into the hosting-pattern taxonomy, and
renders a 3-page static dashboard (overview / migration tracker / full
audit) via scripts/render_audit.py.

Three sources of truth, reconciled on every build:
- lectures/*.yml manifests — the migrated datasets
- migration.yml (new) — the migration lifecycle + PR provenance, kept
  separate from the manifests so it can be archived when the migration
  programme completes without touching the permanent dataset records
- scripts/audit_annotations.yml (new) — curated judgment (descriptions,
  provenance classes, live-API pedagogy) for not-yet-migrated references

The strict scan fails on an unannotated data reference or a migration.yml
status that disagrees with what the lectures actually read, so the
dashboard cannot silently rot — the failure mode that killed the
hand-built 2026-07-15 audit artifact within two days.

Part of #20; taxonomy and design system carried over from that artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Deploy the dashboard and the published tree to GitHub Pages

.github/workflows/audit-dashboard.yml builds the dashboard from shallow
clones of the 8 lecture repos and deploys the default Pages site:
dashboard at /, the published lectures/ tree at /lectures/, audit.json
alongside for reuse. Triggers: push to main, weekly cron, manual
dispatch; on PRs touching the audit inputs it runs the strict build as
a guardrail without deploying.

Serving lectures/ on Pages pre-stages PLAN Phase 4: when the
data.quantecon.org DNS question is resolved, only the custom domain
changes. Checkout uses lfs: true so LFS-tracked files publish as bytes,
never pointers.

Part of #20.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Docs: record the dashboard, its update rules, and P2 completion

README gains the audit-dashboard section; AGENTS.md gains the
keep-it-truthful rules (migration.yml updated in the same PR as a
landing/repoint; annotations for new references; never commit site/ or
audit.json) and the expanded repo map; PLAN.md checks off the Pages
deploy (default domain — custom domain stays open), adds the Phase 5
workflow entry, and marks pilot P2 complete (#17,
QuantEcon/lecture-python-programming#578, QuantEcon/lecture-python.myst#973).

Part of #20.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review: taxonomy gap, status validation, notebook guard

- render_audit.py: add the external-web pattern to PATTERN_META so a
  non-GitHub data URL would surface in the hosting-pattern table and
  distribution chart instead of being silently omitted (no current ref
  triggers it; the failure mode was the point)
- build_audit.py: validate migration.yml status values in the scan-stage
  consistency check (an unknown status fails --strict with a clear
  message); render_audit.py's stepper raises a named error as backstop
- build_audit.py: flag any .ipynb under lectures/ outside _static/ as an
  unscanned_notebooks problem. Every lecture source today is MyST .md and
  the only notebooks are builder/figure helpers (deliberately not
  consumers); if a notebook-format lecture ever appears, the build now
  refuses to miss it silently

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants