Dataset problem: Can't concatenate CESM2 historical and ssp245 because of different bounds_dtype in latitude and longitude.

[npgillett]

CESM2 monthly mean tas historical has bounds_dtype float32 and corresponding ssp245 file has bounds_dtype float64. I want to concatenate historical and ssp245 and this gives an error when trying to concatenate cubes. I tried to write a fix for this model, but apparently this attribute is read-only, so it's not possible simply to overwrite one of the attributes. How should I proceed?

• Project (CMIP5/CMIP6/OBS/obs4mips/etc): CMIP6
• Full dataset description (fill out as many as you know, please):
  • experiment: historical, ssp245
  • mip: cmip, scenariomip
  • grid: gn
  • version: See below for filepath.
  • variable used: tas
• Problems encountered (please describe what the actual problems are: e.g. wrong standard name, issue with dimensions):
  Fails with this message:

  File "/home/rng/miniconda3/envs/esmvaltool2/lib/python3.7/site-packages/iris/_concatenate.py", line 480, in match
    raise iris.exceptions.ConcatenateError(msgs)
iris.exceptions.ConcatenateError: failed to concatenate into a single cube.
  Dimension coordinates metadata differ: latitude, longitude != latitude, longitude

The dim_metadata which are compared and which lead to the above failure are the following:

**** 1 [_CoordMetaData(defn=CoordDefn(standard_name='time', long_name=None, var_name='time', units=Unit('days since 1850-1-1 00:00:00', calendar='365_day'), attributes={'title': 'time', 'type': 'double'}, coord_system=None), dims=(0,), points_dtype=dtype('float64'), bounds_dtype=dtype('float64'), kwargs={'scalar': False, 'circular': False, 'order': 1}), _CoordMetaData(defn=CoordDefn(standard_name='latitude', long_name=None, var_name='lat', units=Unit('degrees_north'), attributes={'title': 'Latitude', 'type': 'double', 'valid_max': 90.0, 'valid_min': -90.0}, coord_system=None), dims=(1,), points_dtype=dtype('float64'), bounds_dtype=dtype('float32'), kwargs={'scalar': False, 'circular': False, 'order': 1}), _CoordMetaData(defn=CoordDefn(standard_name='longitude', long_name=None, var_name='lon', units=Unit('degrees_east'), attributes={'title': 'Longitude', 'type': 'double', 'valid_max': 360.0, 'valid_min': 0.0}, coord_system=None), dims=(2,), points_dtype=dtype('float64'), bounds_dtype=dtype('float32'), kwargs={'scalar': False, 'circular': True, 'order': 1})]

**** 2 [_CoordMetaData(defn=CoordDefn(standard_name='time', long_name=None, var_name='time', units=Unit('days since 1850-1-1 00:00:00', calendar='365_day'), attributes={'title': 'time', 'type': 'double'}, coord_system=None), dims=(0,), points_dtype=dtype('float64'), bounds_dtype=dtype('float64'), kwargs={'scalar': False, 'circular': False, 'order': 1}), _CoordMetaData(defn=CoordDefn(standard_name='latitude', long_name=None, var_name='lat', units=Unit('degrees_north'), attributes={'title': 'Latitude', 'type': 'double', 'valid_max': 90.0, 'valid_min': -90.0}, coord_system=None), dims=(1,), points_dtype=dtype('float64'), bounds_dtype=dtype('float64'), kwargs={'scalar': False, 'circular': False, 'order': 1}), _CoordMetaData(defn=CoordDefn(standard_name='longitude', long_name=None, var_name='lon', units=Unit('degrees_east'), attributes={'title': 'Longitude', 'type': 'double', 'valid_max': 360.0, 'valid_min': 0.0}, coord_system=None), dims=(2,), points_dtype=dtype('float64'), bounds_dtype=dtype('float64'), kwargs={'scalar': False, 'circular': True, 'order': 1})]

Note that bounds_type is different for latitude and longitude for the two files.

I tried adding the following to cesm2.py to overwrite bounds_dtype:

            latitude = cube.coord('latitude')
            latitude.bounds_dtype=np.dtype('float64')
            longitude = cube.coord('longitude')
            longitude.bounds_dtype=np.dtype('float64')

But that gave the following error:

  File "/home/rng/miniconda3/envs/esmvaltool2/lib/python3.7/site-packages/esmvalcore/cmor/_fixes/cmip6/cesm2.py", line 30, in fix_metadata
    latitude.bounds_dtype=np.dtype('float64')
AttributeError: can't set attribute

This seems to be because bounds_dtype is read-only (see e.g. here https://scitools.org.uk/iris/docs/latest/whatsnew/2.0.html)

test-cesm.yml.txt

main_log_debug.txt

• Pointer to existing copy of data :

CMIP6/CMIP/NCAR/CESM2/historical/r1i1p1f1/Amon/tas/gn/v20190308/tas_Amon_CESM2_historical_r1i1p1f1_gn_185001-201412.nc
CMIP6/ScenarioMIP/NCAR/CESM2/ssp245/r1i1p1f1/Amon/tas/gn/v20190730/tas_Amon_CESM2_ssp245_r1i1p1f1_gn_201501-206412.nc

[zklaus]

Thanks for this report! The problem with your fix is that bounds_dtype is only a reflection of the actual property of the bounds that you have to change. The following should work instead:

            latitude = cube.coord('latitude')
            latitude.bounds = latitude.bounds.astype(np.float64)
            longitude = cube.coord('longitude')
            longitude.bounds = longitude.bounds.astype(np.float64)

which actually converts the data type. There is a possibility that the converted bounds are identical to the correct double bounds only up to 5 or so decimals, which might prevent concatenation still. Let me know if that happens.

[npgillett]

Thanks very much for the response @zklaus! With your fix above in cesm2.py the fixes now run OK, but the code fails later in concatenate:

  File "/home/rng/miniconda3/envs/esmvaltool2/lib/python3.7/site-packages/esmvalcore/preprocessor/_io.py", line 91, in concatenate
    cube = iris.cube.CubeList(cubes).concatenate_cube()
  File "/home/rng/miniconda3/envs/esmvaltool2/lib/python3.7/site-packages/iris/cube.py", line 511, in concatenate_cube
    raise iris.exceptions.ConcatenateError(msgs)
iris.exceptions.ConcatenateError: failed to concatenate into a single cube.
  An unexpected problem prevented concatenation.
  Expected only a single cube, found 2.

Here is the main_log_debug:

main_log_debug.txt

I couldn't figure out how to check whether it was any difference in bounds that was still causing the problem - I found the logic in the iris code hard to follow!

PS Could you give any advice on how to debug this error message? I'm getting the same error message for a different model/variable as well and it would be good to know how to proceed. Is there a good way to check what is different about the two cubes which is preventing them from being concatenated? Thanks in advance!

[zklaus]

To debug it might be the easiest to in your config-user.yml set

save_intermediary_cubes: true
remove_preproc_dir: false

Then in your output directory in the preproc dir you will find one output cube for every step of the preprocessing. Hopefully, then you can see the data after a number of preprocessing steps and trace back from the last one that worked to the point where the problem occurred.

Having said that, I would expect it to be the problem I mentioned above, ie rounding differences due to data type conversion. I would suggest copying the bounds from the correct ones to the faulty ones.

[npgillett]

Great - thanks for the response. You were correct @zklaus - it was a problem with differences in the latitude bounds at > 5 d.p. There may be a more elegant way to do this, but I was able to fix this just by rounding latitude bounds to 4 d.p. For the record the CESM2 tas files now concatenate successfully with the following fix in cesm2.py (I am not an esmvalcore developer, so will leave this to someone else to fold in).

class Tas(Fix):
    """Fixes for tas."""

    def fix_metadata(self, cubes):
        """Fix latitude_bounds.

        Parameters
        ----------
        cube : iris.cube.CubeList

        Returns
        -------
        iris.cube.Cube

         """
        cube = self.get_cube_from_list(cubes)
        add_scalar_height_coord(cube)
        for cube in cubes:
            latitude = cube.coord('latitude')
            latitude.bounds = latitude.bounds.astype(np.float64)
            latitude.bounds=np.round(latitude.bounds,4)
            longitude = cube.coord('longitude')
            longitude.bounds = longitude.bounds.astype(np.float64)

        return cubes

[npgillett]

@zklaus Sorry for the simple question, but how do I compare the two cubes to see what is different? The other example I mentioned above is NorESM2-LM, which throws the same error when trying to concatenate siconc from historical and ssp245. The error message doesn't tell me what is different about the two cubes, so I don't know what to fix. (Edit: Turned out to be the lat/lon bounds again, but it would be good to know how to check other than guessing).
main_log_debug.txt
How can I compare the two cubes to see what is different? Thanks!

[zklaus]

@npgillett, I wrote down some tips in the form of FAQ here. Let me know if that helps.

[npgillett]

@zklaus Thanks - that did help. For some reason I couldn't fully debug this by looking at the NetCDF file, but that page was useful - based on the advice there I used a merge command inserted into the code to identify overlapping time coordinates in some CESM2 hist-GHG and ssp245-GHG files I was trying to concatenate.

[zklaus]

Interesting! Ok, I suggest that we finish this issue here and then discuss the other problem in a new issue, ok?

Don't worry about being a developer or not, could you do a pull-request to address this issue with the fix you proposed above?

[npgillett]

OK - I will start new threads for the other issues I identified. Currently I have esmvalcore installed from conda i.e. not from github, and I'm not all that proficient with git. I can post my model-specific fixes here, but I'm not sure how to do a pull request.

[npgillett]

Here is a fix file to address the above issue, and also to address a second unrelated issue with siconc in CESM2 files such as this (the typesi fix seems to come from an earlier version of esmvaltool but isn't included in the version of esmvalcore currently on the repo):

DAMIP/NCAR/CESM2/hist-nat/r3i1p1f1/SImon/siconc/gn/v20190730/siconc_SImon_CESM2_hist-nat_r3i1p1f1_gn_185001-189912.nc
Error message:

 File "/home/rng/miniconda3/envs/esmvaltool2/lib/python3.7/site-packages/esmvalcore/cmor/check.py", line 124, in report_errors
    raise CMORCheckError(msg)
esmvalcore.cmor.check.CMORCheckError: There were errors in variable siconc:
typesi: does not exist
in cube:
sea_ice_area_fraction                      (time: 600; cell index along first dimension: 384; cell index along second dimension: 320)

[Edit: Fix below replaced, because original was missing import iris command.]

cesm2.py.txt

[rswamina]

@zklaus and @npgillett - I ran into the same problem when trying to concatenate cubes from the CESM2 model as well. What has been decided on how to fix this?

[valeriupredoi]

if it's an issue related to partial time overlaps and multiple experiments to be concatenated try this #615