Deal with different data_specs_versions in CMIP6

[zklaus]

As discussed in #158, there is a problem that we are slowly approaching: CMIP6 data will not follow a uniform data_specs_version. This matters because it means that for example data from different models will have different standard_names. For example the two already published datasets

{dataset: GISS-E2-1-G, exp: historical, ensemble: r1i1p1f1, mip: Amon, short_name: psl, grid: gn}
{dataset: GFDL-CM4, exp: historical, ensemble: r1i1p1f1, mip: Amon, short_name: psl, grid: gr1}

use standard_name air_pressure_at_sea_level and air_pressure_at_mean_sea_level, respectively.

Both of them do so following the data_specs_version that they give in the file, namely 01.00.23 and 01.00.27. But how do you compare this?

The oldest data_specs_version I have seen in CMIP6 data is 01.00.23, but there is no guarantee that there isn't and older one and I have not checked what has changed since then.

I am not sure what, if anything, to do about it right now, but at some point we will get this kind of error.
Broadly speaking there seem to be three approaches:

• Do nothing
• Support multiple data spec versions
• Choose one data spec version and provide "fixes" to align other data with that.

@jvegasbsc already commented in #158 in support of the third option, saying

Choose one data spec version and provide "fixes" to align other data with that.

I prefer this, with one condition: we choose last version released and keep updating. We should assume that changes are done for a reason. Most variables will be the same in all versions anyway, so I think the number of fixes will be manageable. We can add fixes at PROJECT level that are applied after MODEL fixes to convert data to the latest version of the request.

In your example, we can add a PROJECT fix that changes the standard_name for psl to the new one automatically if it finds the old one.

What do other people think?

[valeriupredoi]

by data specs you mean CMOR tables versions? https://github.com/PCMDI/cmip6-cmor-tables
I don't see 01.00.27 in there...

[jvegreg]

Look into the tables itself. I can not decipher the releases number

https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_3hr.json

[valeriupredoi]

the release number is per branch

[valeriupredoi]

what we can do is we can assemble the peculiarities of each version vs a standard version of our choice in a say, json file, and apply automatic fixes from that json file depending what version the data comes with?

[zklaus]

About the release numbering, this is how I understand it:
There are really two parts to the data in the PCMDI/cmip6-cmor-tables repository. The tables and the controlled vocabulary.

The tables come from the cmip6 data request (the dreq) and the primary source for this is an xml file in the dreq repository. Yes, that is a reinvention of relational databases in xml syntax living in a subversion repository. This is where the 01.00.30 style versions come from and it has the strongest impact on the contents.

The controlled vocabulary (CV) is curated by the CMIP body of the WCRP and lives here. It generally gives additional information about the application of entities in the data request, like the component models used in the different models, and which models and modelling groups have registered to participate in which MIP. It is released in that repository with four part release numbers, currently standing at 6.2.32.0.

The CV is incorporated into the cmip6-cmor-tables repo by aggregating all the individual json files into a big CMIP6_CV.json file.

These two pieces are then released together under yet another release number, currently 6.5.29. When they do this release, they also note which version of CMOR (implying also PrePARE) uses this bundle of information, currently 3.4.0.

And there you have it. I think this is all the version numbers at play here. You will find an overview of the things belonging together under the releases of cmip6-cmor-tables.

Now the trouble is that the only requirement really is the data request and some groups don't use CMOR at all and consequently are not limited to the bundles that have been released as cmip6-cmor-tables, but just use any of the releases of the data request.

[valeriupredoi]

nice overview, cheers K-man @zklaus - I was aware of bits of this process but not the whole big picture 🍺

The ones that don't use CMOR are a total pain in the rear. But even if they use CMOR they use different versions of the tables (which correspond to the names of the branches on the main CMOR gitHub page, Javi!) - what I was proposing would be for us to set one version as reference (the latest one or 01.00.29 since a lot of the people have used that one for CMORizations) and then find out what and how many are the differences vs the previous ones

[zklaus]

Seems this has come around to bite us. How shall we proceed @jvegasbsc ?