Standard variable names in OBS

[mattiarighi]

For cmorizing the observational data we adopt the CMOR standard from the CMIP5 Tables.

The problem is that the metadata defined in the CMIP6 Tables are sometimes different even for the same variable.

For example "Sea Ice Area Fraction" is sic in CMIP5 and siconc in CMIP6, and the mip is also different (OImon vs. SImon). In some cases even the units are different: "Mole Fraction of Ozone" is tro3 with units of 1.e-9 (ppb) in CMIP5 and o3 with units mol/mol in CMIP6.

Since we want our OBS to be usable across the various CMIP phase and we do not want to duplicate the data just changing the name or the units, we would need to find a way to map the CMIP5 metadata
to CMIP6 (or other projects if needed).

The idea would be to have a yml file containing this mapping, using the CMIP5 Tables as reference.

Would that work? @jvegasbsc @bouweandela @valeriupredoi

[DSenftleben]

Adding to this issue, some ocean variables are already regridded to the atmospheric model grid and provided by the model groups. For example, siconc on the atmospheric grid is siconca. Thus, only a very simple 1-to-1 mapping between those variables is required.

[bouweandela]

@jvegasbsc already implemented something related to this, but I'm not sure if it covers this case:https://github.com/ESMValGroup/ESMValTool/blob/91246325f9737d3e0b39eec3515b26374776d986/esmvaltool/cmor/table.py#L66-L70

[mattiarighi]

Good to know, thanks!
But I'm expecting this dictionary to grow, so it would be good to move it to a yml file.

[jvegreg]

Yes, there are a few variables that have been renamed from CMIP5 to CMIP6, as well as some tables. Good news is that the changes are making them more consistent.

[jvegreg]

Since we want our OBS to be usable across the various CMIP phase and we do not want to duplicate the data just changing the name or the units, we would need to find a way to map the CMIP5 metadata
to CMIP6 (or other projects if needed).
The idea would be to have a yml file containing this mapping, using the CMIP5 Tables as reference.

Or we can try to construct this at runtime by comparing the standard_name. It will be a bit more complicated to create but we will not have to worry about maintining in the long term.

[jvegreg]

About the units, we must choose a default project an use it for the default, but I think it will be nice to have them as an option in the preprocessor.

[mattiarighi]

Or we can try to construct this at runtime by comparing the standard_name. It will be a bit more complicated to create but we will not have to worry about maintining in the long term.

If the standard name remains the same across the CMIP projects that would be indeed the best solution.

[DSenftleben]

Hi @jvegasbsc , could this solution also be extended to solve the issue with observations that have old variable names? Like, for example, the sea ice concentration observations still have the variable name sic, which raises the error
esmvaltool._recipe_checks.RecipeError: Unable to load CMOR table 'OBS' for variable 'siconc' with mip 'SImon'.

[npgillett]

Hi all, I'm having problems reading in siconca. I can read in siconc fine, but when I switch to siconca I get:

File "/mnt/lustre01/pf/b/b380746/ESMValTool-AR6/esmvaltool/_recipe.py", line 126, in _add_cmor_info
  format(cmor_table, short_name, mip))
esmvaltool._recipe_checks.RecipeError: Unable to load CMOR table 'CMIP6' for variable 'siconca' with mip 'SImon'

[jvegreg]

Hi all, I'm having problems reading in siconca.

I will take a look

[jvegreg]

Hi @jvegasbsc , could this solution also be extended to solve the issue with observations that have old variable names?

Yes, I will want a solution that works for all conventions we support

[jvegreg]

Ok, the automatic mapping is not possible. siconc and siconca are different variables but have the same standard_name

[DSenftleben]

@npgillett : The way I got siconca to work was adding the line

'sic': 'siconca',

to

_CMIP_5to6_varname = { 'sic': 'siconc', 'sit': 'sithick', 'tro3': 'o3', }

in ESMValTool/esmvaltool/cmor/table.py
However, as Javier just said, there is currently no way to load siconc and siconca simultaneously in the same recipe.

[npgillett]

Thanks very much for the help so far. But unfortunately this doesn't work for me. I changed ESMValTool-AR6/esmvaltool/cmor/table.py to include siconca instead of siconc:

    _CMIP_5to6_varname = {
        'sic': 'siconca',
        'sit': 'sithick',
        'tro3': 'o3',
    }

I still get the error:

  File "/mnt/lustre01/pf/b/b380746/ESMValTool-AR6/esmvaltool/_recipe.py", line 126, in _add_cmor_info
    format(cmor_table, short_name, mip))
esmvaltool._recipe_checks.RecipeError: Unable to load CMOR table 'CMIP6' for variable 'siconca' with mip 'SImon'

You can see my error log here:
/scratch/b/b380746/esmvaltool_output/test_attribute_20190311_172901/run/main_log_debug.txt
Also, even with the above change to table.py the receipe will still read in siconc - it appears not to be using _CMIP_5to6_varname at all.