feat: Add CMIP7 lazy loading and finalisation support#571
Merged
Conversation
* origin/main: fix: correct the order of the constraints
Contributor
There was a problem hiding this comment.
Pull request overview
Adds CMIP7 two-phase parsing (DRS bootstrap + complete finalisation) and consolidates shared finalisation logic so CMIP6 and CMIP7 follow the same lazy-loading/finalisation pattern via FinaliseableDatasetAdapterMixin.
Changes:
- Introduces CMIP7 DRS and complete parsers and wires CMIP7 adapter to support lazy finalisation.
- Extracts shared finalisation + DB persistence logic into
FinaliseableDatasetAdapterMixin, refactoring CMIP6/CMIP7 adapters to use it. - Expands dataset utilities (version extraction, DRS daterange parsing, time metadata reads) and adds shared/unit + integration tests for both adapters.
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/climate-ref/tests/unit/test_config.py | Adds cmip7_parser to default-config expectations. |
| packages/climate-ref/tests/unit/datasets/test_utils.py | Adds tests for new utils (extract_version_from_path, parse_drs_daterange, parse_datetime). |
| packages/climate-ref/tests/unit/datasets/test_netcdf_utils.py | Adds coverage for new read_time_metadata(). |
| packages/climate-ref/tests/unit/datasets/test_cmip_adapters.py | New shared parameterised unit tests for CMIP6/CMIP7 adapters + mixin finalisation behavior. |
| packages/climate-ref/tests/unit/datasets/test_cmip7.py | Refactors CMIP7 unit tests to use new parser functions and shared coverage. |
| packages/climate-ref/tests/unit/datasets/test_cmip6.py | Removes duplicated CMIP6 tests now covered by shared suites; keeps CMIP6-specific tests. |
| packages/climate-ref/tests/unit/datasets/conftest.py | Removes CMIP6-only cached catalog fixture (now covered elsewhere). |
| packages/climate-ref/tests/integration/datasets/test_adapter_integration.py | New integration tests for discovery, DB round-trips, and finalisation for both adapters. |
| packages/climate-ref/tests/integration/datasets/conftest.py | Integration fixtures to parameterise over adapters and parsers. |
| packages/climate-ref/src/climate_ref/datasets/utils.py | Adds extract_version_from_path() and parse_drs_daterange() utilities. |
| packages/climate-ref/src/climate_ref/datasets/netcdf_utils.py | Adds read_time_metadata() helper. |
| packages/climate-ref/src/climate_ref/datasets/mixins.py | Introduces FinaliseableDatasetAdapterMixin implementation incl. DB persistence. |
| packages/climate-ref/src/climate_ref/datasets/cmip7_parsers.py | New CMIP7 DRS + complete parsing implementations. |
| packages/climate-ref/src/climate_ref/datasets/cmip7.py | Refactors CMIP7 adapter to use mixin and parser dispatch (cmip7_parser). |
| packages/climate-ref/src/climate_ref/datasets/cmip6_parsers.py | Reuses shared version/date-range utilities; removes duplicated daterange parsing. |
| packages/climate-ref/src/climate_ref/datasets/cmip6.py | Refactors CMIP6 adapter to use mixin for finalisation. |
| packages/climate-ref/src/climate_ref/config.py | Adds cmip7_parser configuration option. |
| packages/climate-ref/conftest.py | Adds CMIP7 converted-file fixture and shared adapter test configuration/fixtures. |
| packages/climate-ref-pmp/src/climate_ref_pmp/diagnostics/variability_modes.py | Removes realm facet from requirements. |
| packages/climate-ref-pmp/src/climate_ref_pmp/diagnostics/enso.py | Removes realm facet from requirements. |
| packages/climate-ref-pmp/src/climate_ref_pmp/diagnostics/annual_cycle.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/zec.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/tcre.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/tcr.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/sea_ice_sensitivity.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/sea_ice_area_basic.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/regional_historical_changes.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/enso.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/ecs.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/cloud_scatterplots.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/cloud_radiative_effects.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/climate_drivers_for_fire.py | Removes realm facet from requirements. |
| packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/climate_at_global_warming_levels.py | Removes realm facet from requirements. |
| changelog/571.improvement.md | Notes shared finalisation logic extraction. |
| changelog/571.feature.md | Notes CMIP7 lazy loading + finalisation feature. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lewisjared
added a commit
that referenced
this pull request
Mar 1, 2026
* origin/main: (40 commits) chore: read mandatory variables fix: Better handling of nan values docs: Update documenation about DRS parser docs: changelog update fix: correct the order of the constraints chore: add changelog entries for PR #571 chore: Clean up CMIP6 and CMIP7 test overlap chore: refactor tests chore: update tests chore: remove realm from filtering feat: Add CMIP7 lazy loading chore: Fix regression outputs chore: changelog fix: apply version filtering to the regression analysis chore: Use variable_name chore: fix pre-commit add change log rename double_ITCZ to doubleITCZ for consistency in output file naming chore: add test case for double ITCZ chore(deps-dev): bump jupyterlab in the python-dependencies group ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add lazy loading (DRS + complete parser) and finalisation support for CMIP7 datasets, mirroring the existing CMIP6 pattern via the shared
FinaliseableDatasetAdapterMixin.Key changes
Source:
cmip7_parsers.pywith DRS and complete parsing functions for CMIP7 filesFinaliseableDatasetAdapterMixininmixins.py, used by both CMIP6 and CMIP7 adapterscmip6.pyandcmip6_parsers.pyto use the shared mixin, removing duplicated codecmip7.pyto use the shared mixin for lazy loading and finalisationcmip7_parserconfig option and version filtering supportdatasets/utils.pyanddatasets/netcdf_utils.pyrealmfrom diagnostic data requirement filteringTests:
test_cmip_adapters.py) covering adapter basics, parser dispatch, finalisation edge cases, and persist logic for both CMIP6 and CMIP7test_adapter_integration.py) covering local dataset discovery, round-trip database persistence, and end-to-end finalisation for both adapterstest_cmip6.pyandtest_cmip7.pyto remove duplicated tests now covered by shared suitesOther:
Checklist
Please confirm that this pull request has done the following:
changelog/