Generic level checks lead to inconsistent ESMValTool runs

[schlunma]

See #598:

I still have this weird issue that (I think) depends on the order of which the datasets are read:

The following recipe sometimes works, sometimes fails with

esmvalcore.cmor.check.CMORCheckError: There were errors in variable cl:
lev: units should be 1, not m
 lev: has values > valid_max = 1.0

for HadGEM3-GC31-MM and sometimes with the same error for ACCESS-ESM1-5.

Each of these datasets does NOT throw an error when used alone.

What is going on here?

This recipe sometimes works, sometimes not:

# ESMValTool
---
documentation:
  authors:
    - schlund_manuel

  maintainer:
    - schlund_manuel

  references:
    - gregory04grl

  projects:
    - crescendo


diagnostics:

  diag_test:
    variables:
      cl:
        project: CMIP6
        exp: historical
        mip: Amon
        start_year: 2004
        end_year: 2004
    additional_datasets:
      - {dataset: ACCESS-ESM1-5,   ensemble: r1i1p1f1, grid: gn}
      - {dataset: GISS-E2-1-G,     ensemble: r1i1p1f1, grid: gn}
      - {dataset: GISS-E2-1-H,     ensemble: r1i1p1f1, grid: gn}
      - {dataset: HadGEM3-GC31-LL, ensemble: r1i1p1f3, grid: gn}
      - {dataset: HadGEM3-GC31-MM, ensemble: r1i1p1f3, grid: gn}
    scripts:
      null

This recipe always works:

# ESMValTool
---
documentation:
  authors:
    - schlund_manuel

  maintainer:
    - schlund_manuel

  references:
    - gregory04grl

  projects:
    - crescendo


diagnostics:

  diag_test:
    variables:
      cl:
        project: CMIP6
        exp: historical
        mip: Amon
        start_year: 2004
        end_year: 2004
    additional_datasets:
      - {dataset: ACCESS-ESM1-5,   ensemble: r1i1p1f1, grid: gn}
    scripts:
      null

[zklaus]

From the above, it seems that the following datasets exhibit the problem for the cl variable:

- {dataset: ACCESS-ESM1-5, project: CMIP6, exp: historical, mip: Amon, ensemble: r1i1p1f1, grid: gn}
- {dataset: HadGEM3-GC31-MM, project: CMIP6, exp: historical, mip: Amon, ensemble: r1i1p1f3, grid: gn}

I had a look at them and on their own, they seem fine, so I am not yet sure what is going on.

But there seems to be the misconception that the lev variable in the netcdf file is not used for the calculation. This is not correct.

These parametric coordinates work as follows (cf CF-1.7 Sect. 4.3.3 and Appendix D): The standard_name of the coordinate determines which form of parametric coordinate from Appendix D is in effect. Each one of those comes with at least one "Definition", ie a formula that allows the calculation of the coordinate from a number of parameters. Some have two different formulae for slightly different models. This formula can often be found in the formula attribute of the coordinate variable, though that does not seem to be required. In any case, the applicable formula is given in Appendix D of the CF-Conventions and is specified generically using a, ap, b, etc. These parameters are called "formula terms" and are given by variables in the netcdf file. But these variables can have any name; the connection between a formula term and a netcdf variable is made with the formula_terms attribute of the coordinate.

For the above example datasets, we find:

lev:formula_terms = "a: lev b: b orog: orog" ;

meaning that the values of lev are in fact the a in the formula.

[schlunma]

Yes, for these datasets (with hybrid height coordinate) you're absolutely right @zklaus. Sorry, I jumped to conclusions and chose the wrong example. I will give a detailed answer in #882 since this is the wrong place for this discussion.

Unfortunately, that does not explain the problem described in this issue (that sometimes the tool crashes with this recipe, sometimes not).

[sloosvel]

It's just that in some runs the cmor information get's overwritten incorrectly, I'll try to fix it. Since the datasets are using different coordinates for the same variables, in some runs there is this bug happening that coordinates get mismatched.