CMIP6 fx files stored in different directories, depending on realm

[DSenftleben]

Not all fx files for CMIP6 models are located in the fx/ directory. For example, 'areacello' is stored under Ofx/, 'sandfrac' under Efx/. See also example below.

Therefore, config-developer.yml needs to be extended to also check the other *fx/ directories when searching for the fx files specified in the recipe.

Example:
/CNRM-CERFACS/CNRM-CM6-1/historical/r1i1p1f2/fx/
/CNRM-CERFACS/CNRM-CM6-1/historical/r1i1p1f2/Efx/
/CNRM-CERFACS/CNRM-CM6-1/historical/r1i1p1f2/Ofx/

[DSenftleben]

Another issue with fx file directory structure:

fx files are stored for each ensemble member and experiment individually, but should be independent of these keys (e.g., areacello is only dependent on grid structure, which is the same for each ensemble member). This causes unnecessary file duplication and the tool to crash for all ensemble members for which the fx files are not placed in their respective directories.

In CMIP5, fx files therefore were stored under r0i0p0. Maybe a similar solution could be implemented for CMIP6?

[valeriupredoi]

cheers for the heads up @DSenftleben - I suggest we wait a bit until things get settled in terms of data storage structure and replication for CMIP6. Eg I am looking on the CEDA archive (BADC) and I am seeing only the amip and the Efx dir is populated like this:
/badc/cmip6/data/CMIP6/CMIP/CNRM-CERFACS/CNRM-CM6-1/amip/r1i1p1f2/Efx/sftgrf/gr/latest/sftgrf_Efx_CNRM-CM6-1_amip_r1i1p1f2_gr.nc

whereas the fx dir is populated as:

[valeriu@jasmin-sci2 ~]$ ls -la /badc/cmip6/data/CMIP6/CMIP/CNRM-CERFACS/CNRM-CM6-1/amip/r1i1p1f2/fx/
total 0
drwxr-x--- 1 badc open 0 Jan 24 17:01 .
drwxr-x--- 1 badc open 0 Jan 16 22:17 ..
drwxr-x--- 1 badc open 0 Jan 16 22:17 areacella
drwxr-x--- 1 badc open 0 Jan 24 17:00 areacellr
drwxr-x--- 1 badc open 0 Jan 16 22:17 mrsofc
drwxr-x--- 1 badc open 0 Jan 16 22:17 orog
drwxr-x--- 1 badc open 0 Jan 16 22:17 rootd
drwxr-x--- 1 badc open 0 Jan 16 22:17 sftgif
drwxr-x--- 1 badc open 0 Jan 16 22:17 sftlf

I'll ask the CEDA guys if this will change in the near future, if not, yes, we'll have to make the changes

[bouweandela]

I noticed that the fx path settings are completely missing from config-developer.yml for CMIP6. Maybe someone can make a pull request to add those and that will also fix this issue?

[zklaus]

Some more context for the practicalities of fx files:

• Some "fx" files might be not so fxed: At least thkcello, masscello, and volcello can live either in Ofx or in (Omon and Odec) depending on the model. My reading of the data request suggests that these are the only ones.
• At least IPSL is embedding at least some fx information (certainly areacello) in their data files. Maybe we can just ignore that, but should inconsistencies surface, we should have a look there.

[valeriupredoi]

Another issue with fx file directory structure:

fx files are stored for each ensemble member and experiment individually, but should be independent of these keys (e.g., areacello is only dependent on grid structure, which is the same for each ensemble member). This causes unnecessary file duplication and the tool to crash for all ensemble members for which the fx files are not placed in their respective directories.

In CMIP5, fx files therefore were stored under r0i0p0. Maybe a similar solution could be implemented for CMIP6?

a heads up that I have also noticed this for IPSL and CNRM-CERFACS: on BADC:

f1 = iris.load_cube('/badc/cmip6/data/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r3i1p1f1/Ofx/areacello/gn/latest/areacello_Ofx_IPSL-CM6A-LR_historical_r3i1p1f1_gn.nc')
f2 = iris.load_cube('/badc/cmip6/data/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r5i1p1f1/Ofx/areacello/gn/latest/areacello_Ofx_IPSL-CM6A-LR_historical_r5i1p1f1_gn.nc')

these files are totally identical (well, data is, almost all metadata is too, the parent ensemble is r1i1p1f1). Does any of you guys know what the strategy for including these files in the DRS is - @tillku is already starting to have headaches for his diagnostic that uses volcello and that one is not yet present at all for any model on ESGF 🍺

[valeriupredoi]

OK guys I got confirmation of the standard CMIP6 fx paths from Ruth Petrie, data guru at STFC UK:

me: So, in a nutshell, should we expect to find fx files in a DRS like

/badc/cmip6/data/CMIP6/CMIP/[institute]/[dataset]/[experiment]/[ensemble]/[fxdir]/

where [fxdir] will be the same for each [ensemble] but have different names (fx, Ofx...) depending on [experiment]?

Ruth: Yes that is correct. It is formally “table_id” instead of “fx_dir” the full list of all the table_ids is at https://github.com/WCRP-CMIP/CMIP6_CVs/blob/master/CMIP6_table_id.json this includes all the tables not just the fx ones but if you scan the list you should find all the different fx tables.

@zklaus : I think from the point of CMIP6 these fields are just normal variables.
Hence, they can be found as any other variable in the respective (Table,
variable) place.
Generally speaking, fixed files
for the ocean are in Ofx, files for atmosphere and land are in fx,
fixed files for the ice sheets over Antarctica and Greenland are in
IfxAnt and IfxGre, respectively, and some extended fixed things are
in Efx.

[zklaus]

Indeed, neither the table nor the variable name depends on the experiment.

Additionally, some of these are actually cell_measures, so the most elegant way, imho, to deal with them would be to add the fx files to the list of input filenames before they are loaded as cubes into iris.
Then, iris will automatically sort them out and make them available via cube.cell_measure().
Grepping the tables, this seems to apply for

• areacella from fx
• areacellg from IfxAnt and IfxGre
• areacello from Ofx
• volcello from either Ofx or (Omon, Oyr, Odec).
• areacellr from fx

[valeriupredoi]

started working on a functional implementation in #1037 - let's take the discussion there, guys 🍺

[schlunma]

Additionally, some of these are actually cell_measures, so the most elegant way, imho, to deal with them would be to add the fx files to the list of input filenames before they are loaded as cubes into iris.
Then, iris will automatically sort them out and make them available via cube.cell_measure().
Grepping the tables, this seems to apply for

Good call, I think this would also remove the UserWarning: Missing CF-netCDF measure variable ... which appears very often (see #853).