TypeXXX missing in a lot of CMIP6 FX files

[valeriupredoi]

An example reported by @bascrezee in #328 and also reported by @ValerioLembo and explained by @schlunma in ESMValGroup/ESMValTool#1272 (comment)

Question to be asked: what is this typeXXX stuff? @zklaus or @jvegasbsc you guys know? If not I'll take it up with CMIP6 data gurus like Ruth Petrie at CEDA or @agstephens

Solution at hand: write fixes seems a bit of overkill, but if this type crap is needed then we should automate it for all models; if it's not, forget about it in the CMOR checks 🍺

[schlunma]

This morning @debe-kevin and me also discovered several datasets where scalar coordinates (e.g. depth, height or the typeXXX ones) are missing. It would be really nice to have automatic fixes for this, because I think that a lot of models are affected.

[valeriupredoi]

one important thing we have to bear in mind is that CMIP6 data is far from final (and the quality of it differs from one ESGF node to another), so before we start writing a high complexity function that addresses automated fixes we have to think twice. But yeah, am in favour of automatic fixes -> @jvegasbsc

[zklaus]

These things (typeXXX) are generally related to vegetation and this is imho the reason why they are a little bit less mature than other parts of the esms that have been around for longer.

One problem that was encountered in the specification of CMIP6 was that there were standard names in CF for some of the needed categories, but not for all. This lead to the slightly inhomogeneous situation were some of that information is purely conveyed by the standard_name attribute and some by additional (scalar) variables.

Because of this, we should tackle this variable by variable. The only variable that I have seen mentioned by name is (fx, sftlf).
Thing is, to my best ability it seems that both in the datarequest and in the table I can not find an indication that a land type should be present. Therefore I think at the moment that models without it are correct and eg IPSL is wrong in including it.

To get a better picture, I'd like to know some of the problematic files. If you know some, please add them into the following list, organized by (table, variable) pairs.

• (fx, sftlf)
• (Ofx, sftof)

- {dataset: CESM2, project: CMIP6, grid: gn, exp: historical, ensemble: r1i1p1f1}
  Variables: treeFrac; grassFrac; shrubFrac; cropFrac, sftlf, sftof

@zklaus the links you refer to are identical, one must be wrong, can you check, correct and remove this comment. Thanks, @schlunma @bascrezee, corrected.

[zklaus]

Please also add example files if you have any! Eg as the {dataset:...} line.

[zklaus]

Digging a bit deeper it seems this is due to a change of the tables: Going from data spec version 01.00.29 to 01.00.30, on 11 March 2019, the typeland dimension vanished from (fx, sftlf) and the typeli dimension from (fx, sftgif). The same commit also removes typesea from (Ofx, sftof).

So these fx/Ofx files go back to #150, #158, and #159. Basically some models have prepared those files with older versions of the tables and in the newer versions these attributes are no longer present.

The Lmon/Eyr files that @bascrezee mentions are different though. But since they are clearly not fx files, I suggest we continue the discussion about those in #328 and focus here on the problems with the fx files and in general on those stemming from the change of tables, ok?

[valeriupredoi]

good digging @zklaus 🔧 @ValerioLembo has mentioned in #1272 that he's started seeing these quite a few - Valerio, care to add more files you've seen and are not mentioned here? 🍺

[ValerioLembo]

Hi there.

I am following the discussion, but I am not replying because I am working in a branch I have opened, testing the fixes I have written for the few models I have spotted (namely, EC-Earth-Veg and GFDL-CM4). As soon as I am done with that, I will commit them and send the PR.

[zklaus]

To maximize the chance of implementing the right fix, I suggest to fully diagnose the problem first. For example I think that the fix suggested in ESMValGroup/ESMValTool#1272 is wrong since it aligns the data with an outdated version of the data spec, whereas in the discussion in #158 (and #159) we agreed to align with the most recent version.

At first glance it seems indeed nonsensical to have, eg, a land type coordinate for sftlf, when there is patently no distinction between land types in that variable. Hence the removal seems to be a bugfix of the outdated version of the specs.

[ValerioLembo]

@zklaus I agree, indeed. The fix suggested by @schlunma is meant to be temporary, and I think the right way to go would be preventing the preprocessor from looking for scalar coordinates that are not needed anywhere else.

I implemented a fix analogously to #230 (as suggested in ESMValGroup/ESMValTool#1272). I am now able to write the "height" scalar coordinate in uas and vas (GFDL-CM4), and the "type" scalar coordinate in sftlf (GFDL-CM4 and EC-Earth-Veg).

I won't PR the changes, but one can see the new files in metadata_CMIP6_fx_files branch.

[zklaus]

The height stuff seems fine to me. I'd like to take a closer look at the type things first, so I suggest you separate the two and submit a pull request for the height stuff. Please add me as a reviewer since I already have some comments 😈.

[ValerioLembo]

@zklaus there you go! #330

[zklaus]

As for the other type things: This seems to be because in ESMValCore we are at the moment also at the (slightly) outdated version of the data spec of 01.00.29 whereas the mismatching data files are by-and-large made with version 01.00.30.

So following the path set out in #158, #159, we should update to the latest version of the tables and then the errors will vanish. Mostly. Then we might need to write some fixes to align older model data like CESM2.