concatenation of cubes with overlapping data

[mwjury]

Describe the bug
Thanks for all the efforts so far of concatenating overlapping data (as e.g. #280)!
For certain datasets however this does not work satisfactorily. E.g. for tas

 - {project: CMIP5, dataset: GISS-E2-H-CC, ensemble: r1i1p1, exp: [historical, rcp85], start_year: 1950, end_year: 2100}

and uses the files:

historical/mon/atmos/Amon/r1i1p1/v20120723/tas/tas_Amon_GISS-E2-H-CC_historical_r1i1p1_195101-201012.nc
historical/mon/atmos/Amon/r1i1p1/v20120723/tas/tas_Amon_GISS-E2-H-CC_historical_r1i1p1_190101-195012.nc
rcp85/mon/atmos/Amon/r1i1p1/v20160512/tas/tas_Amon_GISS-E2-H-CC_rcp85_r1i1p1_200601-205012.nc
rcp85/mon/atmos/Amon/r1i1p1/v20160512/tas/tas_Amon_GISS-E2-H-CC_rcp85_r1i1p1_205101-210012.nc

The initial concatenation concatenates the first 2 and the last cube to one (from 1950-2010 and 2051-2100) and leaves the 3rd as it is. _concatenate_overlapping_cubes subsequently ignores this 3rd one.

This is caught later on and throws:

esmvalcore.cmor.check.CMORCheckError: There were errors in variable tas:
time: Frequency mon does not match input data

Any idea how to approach this issue?

Please attach

• The recipe that you are trying to run, you can find a copy in the run directory in the output directory concatenate_bug_yml.txt

• The main_log_debug.txt file, this can also be found in the run directory in the output directory
  main_log_debug.txt

[valeriupredoi]

what are time bounds you are asking for in the recipe? Even better, can you pls post the recipe so I can replicate the issue and attempt a fix? Cheers, sorry for a bit of a delay in response 🍺

[mwjury]

Hi @valeriupredoi , thanks! The recipe is in my first message (concatenate_bug_yml.txt) and the error is happening already in the preprocessor after it produces 01_concatenate.nc (this file already has missing time sclices).

[valeriupredoi]

ah cheers, I have a tendency to completely overlook the standard-formatted messages we actually ask the OP to provide 😁 - I'll have a look in a jiffy

[valeriupredoi]

hi @mwjury cheers for pointing out this case - this is an odd one and a very useful one nonetheless! I managed to replicate your error fully and it is indeed coming from the concatenation but it is actually a result of bad iris concatenation:

concatenated = iris.cube.CubeList(cubes).concatenate()

is operating on the four cubes list

tas_Amon_GISS-E2-H-CC_historical_r1i1p1_190101-195012.nc
tas_Amon_GISS-E2-H-CC_historical_r1i1p1_195101-201012.nc
tas_Amon_GISS-E2-H-CC_rcp85_r1i1p1_200601-205012.nc
tas_Amon_GISS-E2-H-CC_rcp85_r1i1p1_205101-210012.nc

and it performs the concatenation of the first two with the fourth with no problem but hits a snag when concatenating that resulting 3-cube-composite with the third (2006-2050) because that's overlapping time-data (well, the third cube has bounds included in the composite cube). BUT the first (iris-standard) concatenation should not happen! I think this should be treated with care by the iris guys, so am gonna ask @bjlittle to have a look and open an iris issue. If they say let it be, then we'll fix this in our code

[mwjury]

Thanks @valeriupredoi, thanks! It would be a pity if we wouldn't be able to include those CMIP5 models in the IPCC figures.
The same issue with very similar files is happening with:

- {project: CMIP5, dataset: GISS-E2-R-CC, ensemble: r1i1p1, exp: [historical, rcp85]}
- {project: CMIP5, dataset: FGOALS-g2, ensemble: r1i1p1, exp: [historical, rcp85]}

Maybe one rather simple solution would be the possibility to provide a start_year / end_year list in the recipe. Thereby one can make sure which years enter subsequently for the single datasets. (If this is not done dataset specific it may lead to other problems as e.g. MOHC/HadGEM2-ES historical runs end in 200511, RCP85 starts in 200512.)

[valeriupredoi]

OK @mwjury - no answer yet from the Iris folk, I will attempt a fix in our code tomorrow 🍺

[valeriupredoi]

@mwjury my apologies it took so long, things are not the same when working from home 😁 I have a nice solution for you in #615 (which I tested with both your use cases from here) 🍺

[mwjury]

@valeriupredoi works like a charm. Thanks!