CMIP6 IPSL historical and ssp245 concatenate error

[jvegreg]

I am running into a related problem when trying to concatenate data from IPSL historical and ssp245 simulations (tas_Amon_IPSL-CM6A-LR_historical_r2i1p1f1_gr_185001-201412.nc and tas_Amon_IPSL-CM6A-LR_ssp245_r2i1p1f1_gr_201501-210012.nc). The historical data has a time origin of January 1850 and SSP245 data has a time origin of January 2015 . I get this error:

iris.exceptions.ConcatenateError: failed to concatenate into a single cube.
   Cube metadata differs for phenomenon: air_temperature
   Dimension coordinates metadata differ: time != time
   Auxiliary coordinates metadata differ: day_of_month, day_of_year, month_number, year != day_of_month, day_of_year, month_number, year

It appears that _fix_cube_attributes ensures that the global attributes of the two cubes being merged are the same, but not the attributes attached to the coordinates. By inserting print statements, it looks to me like the code correctly accounts for the different time origins of the two input files, and adapts the units of time and the dates accordingly, but it leaves the attribute 'time-origin' attached to the 'time' coordinate of the second file unchanged, which causes check_aux_coords to fail.

I'm not sure if this can be resolved by changing _fix_cube_attributes or just be writing a model-specific fix. Any help would be greatly appreciated!

main_log_debug.txt

Originally posted by @npgillett in ESMValGroup/ESMValTool#538 (comment)

[jvegreg]

From @bouweandela

I'm not sure if this can be resolved by changing _fix_cube_attributes or just be writing a model-specific fix. Any help would be greatly appreciated!

It depends on whether this is a problem with just the IPSL-CM6A-LR model or if the problem is present in many models. If it's just IPSL-CM6A-LR, a model specific fix would make most sense.

@jvegasbsc Can you give some advice on this?

[jvegreg]

From @zklaus

This is a fault in the IPSL data since the reference date in the scenarios is supposed to be the same as in the corresponding historical run.

However, this should not be an issue anymore since we went back to setting the same reference date for all data. This is discussed in a different issue/pr. I'll see if I can find it. The last this was changed was in #108, but we have always changed the reference time. So the problem seems to be that the concatenation happens before the change of reference time, but that is a different problem from this issue. I suggest to close this issue and to follow up in ESMValCore. There is already some activity wrt to combining runs. If that doesn't address the issue, I suggest to open a new issue in ESMValCore.

Relevant issues/prs:

• ESMValGroup/ESMValTool#1329
• Attempt concatenation when cubes have overlapping data #280

[jvegreg]

And now my comment:

. So the problem seems to be that the concatenation happens before the change of reference time, but that is a different problem from this issue.

We are applying that change before concatenation, at the fix metadata step. I will try to debug the issue and let you know what is happening

[jvegreg]

Okay, error located: we are fixing the units but they have a 'time_origin' attribute that corresponds to the reference point that we are not touching

historical:

        double time(time) ;
                time:axis = "T" ;
                time:standard_name = "time" ;
                time:long_name = "Time axis" ;
                time:calendar = "gregorian" ;
                time:units = "days since 1850-01-01 00:00:00" ;
                time:time_origin = "1850-01-01 00:00:00" ;
                time:bounds = "time_bounds" ;

ssp285:

        double time(time) ;
                time:axis = "T" ;
                time:standard_name = "time" ;
                time:long_name = "Time axis" ;
                time:calendar = "gregorian" ;
                time:units = "days since 2015-01-01 00:00:00" ;
                time:time_origin = "2015-01-01 00:00:00" ;
                time:bounds = "time_bounds" ;

[valeriupredoi]

I love the dialogue between @jvegasbsc and friends (mostly @jvegasbsc but also @jvegasbsc ) 😁

good catch! the concatenation happens before any fixes and checks apart from the _fix_cube_attributes(cubes)

🍺

[jvegreg]

I love the dialogue between @jvegasbsc and friends (mostly @jvegasbsc but also @jvegasbsc ) grin

Side effects of being unable to split an issue in two. Even the description that was created automatically when opening the issue from a comment looks like I wrote it

good catch! the concatenation happens before any fixes and checks apart from the _fix_cube_attributes(cubes)

That's incorrect!

The current workflow is:

• load
• fix_metadata (includes check with automatic_fixes=True)
• concatenate
• check_metadata (the final check with automatic_fixes=False)

So we have checked and fixed data arriving to the concatenate. We add an extra check later just in case there is any bug in the automated fixes.

[valeriupredoi]

I stand corrected! I was thinking about extract_time but you are totally right, fixes are applied prior 🍺

[npgillett]

Hi all, Thanks very much for looking into this. I agree that the problem is that the time units are corrected prior to the concatenate, but not the time_origin attribute. This is why my code fails. Will this be addressed by a fix to the code? Or is there something I should do differently to get this to work? Thanks!

[jvegreg]

The 'time_origins' attribute is not part of the standard and is redundant with the units definition.

My approach would be to create a fix for this model. If later on we discover this is quite common, we can add an automatic fix for this

[zklaus]

Agreed. And to be clear: The fix should probably be simply to remove attribute in fix_metadata.

[npgillett]

OK - thanks very much! For the record, I fixed the problem by adding:

        #Remove time_origin attribute, which is redundant.
        for cube in cubes:
            cube.coord('time').attributes={}

to the beginning of fix_metadata in ipsl_cm6a_lr.py.

[zklaus]

@npgillett did you already do this in a pull request here? Otherwise, would you mind opening one? Perhaps it would be slightly less drastic to simply remove the offending attribute, like

cube.coord('time').attributes.pop('time_origin')

[npgillett]

Slight revision required so it doesn't fail on files containing cell_area, which has no time coordinate:

        #Remove time_origin attribute, which is redundant.
        for cube in cubes:
            if cube.name() == 'cell_area':
                continue
            cube.coord('time').attributes={}

[jvegreg]

Don't look for a specific variable, just check if the time coordinate is there:

for cube in cubes
    try:
        cube.coord('time').attributes.pop('time_origin', default=None)
    except iris.exceptions.CoordinateNotFoundError:
        pass