ESMValTool Feasibility to process CORDEX data

[aperezpredictia]

Dear all,

I would like to know if there is any option to use ESMValTool with EURO-CORDEX datasets in order to either process or validate regional climate projections. I have been looking into the config-developer.yml and it seems to not be compatible with CORDEX datasets or at least, this proyect is not defined in the .yml file. Even though there are a lot of examples for CMIP5 format which is similar to CORDEX format, I have not seen any CORDEX dataset in any of the recipes given.

I'm considering ESMValTool as a tool to produce not only global climate proyections, but it should also consider regional models as CORDEX.

Thanks for everything!

[valeriupredoi]

unfortunately I am not very familiar with CORDEX data types (actually I don't know anything in terms of its technical parameters) - @mattiarighi or @alistairsellar may be able to comment. One thing I can ask you would be to please provide us with a practical example of a CORDEX data file (sample data, don't have to be anything large). Cheers 🍺

[aperezpredictia]

They are really similar to CMIP5 with the difference that the file name path has an additional parameter which is regional climate model and the grid is rotated latitude/longitude. See an ncdump of a CORDEX file below:

ncdump -h /predictia/Data/c3s512/data/era5/tas_EUR-44_IPSL-IPSL-CM5A-MR_rcp85_r1i1p1_IPSL-INERIS-WRF331F_v1_day_20810101-20851231.nc 
netcdf tas_EUR-44_IPSL-IPSL-CM5A-MR_rcp85_r1i1p1_IPSL-INERIS-WRF331F_v1_day_20810101-20851231 {
dimensions:
	rlat = 103 ;
	rlon = 106 ;
	vertices = 4 ;
	time = UNLIMITED ; // (1826 currently)
	bnds = 2 ;
variables:
	double lat(rlat, rlon) ;
		lat:bounds = "lat_vertices" ;
		lat:long_name = "latitude coordinate" ;
		lat:standard_name = "latitude" ;
		lat:units = "degrees_north" ;
	double lat_vertices(rlat, rlon, vertices) ;
		lat_vertices:units = "degrees_north" ;
	double lon(rlat, rlon) ;
		lon:bounds = "lon_vertices" ;
		lon:long_name = "longitude coordinate" ;
		lon:standard_name = "longitude" ;
		lon:units = "degrees_east" ;
	double lon_vertices(rlat, rlon, vertices) ;
		lon_vertices:units = "degrees_east" ;
	double time(time) ;
		time:bounds = "time_bnds" ;
		time:units = "days since 1949-12-01 00:00:00" ;
		time:calendar = "gregorian" ;
		time:axis = "T" ;
		time:long_name = "time" ;
		time:standard_name = "time" ;
	double time_bnds(time, bnds) ;
	double rlat(rlat) ;
		rlat:units = "degrees" ;
		rlat:axis = "Y" ;
		rlat:long_name = "latitude in rotated pole grid" ;
		rlat:standard_name = "grid_latitude" ;
	double rlon(rlon) ;
		rlon:units = "degrees" ;
		rlon:axis = "X" ;
		rlon:long_name = "longitude in rotated pole grid" ;
		rlon:standard_name = "grid_longitude" ;
	double height ;
		height:units = "m" ;
		height:axis = "Z" ;
		height:positive = "up" ;
		height:long_name = "height" ;
		height:standard_name = "height" ;
	float tas(time, rlat, rlon) ;
		tas:standard_name = "air_temperature" ;
		tas:long_name = "Near-Surface Air Temperature" ;
		tas:comment = "daily-mean near-surface (usually, 2 meter) air temperature." ;
		tas:units = "K" ;
		tas:original_name = "T2" ;
		tas:cell_methods = "time: mean" ;
		tas:history = "2014-03-11T23:56:36Z altered by CMOR: Treated scalar dimension: \'height\'. 2014-03-11T23:56:49Z altered by CMOR: Converted type from \'d\' to \'f\'." ;
		tas:coordinates = "height lat lon" ;
		tas:missing_value = 1.e+20f ;
		tas:_FillValue = 1.e+20f ;
		tas:associated_files = "gridspecFile: gridspec_atmos_fx_IPSL-INERIS-WRF331F_rcp85_r0i0p0.nc" ;
		tas:grid_mapping = "rotated_latitude_longitude" ;
	int rotated_pole ;
		rotated_pole:grid_mapping_name = "rotated_latitude_longitude" ;
		rotated_pole:grid_north_pole_latitude = 39.25 ;
		rotated_pole:grid_north_pole_longitude = -162. ;

// global attributes:
		:CORDEX_domain = "EUR-44" ;
		:driving_experiment = "IPSL-IPSL-CM5A-MR, rcp85, r1i1p1" ;
		:driving_model_id = "IPSL-IPSL-CM5A-MR" ;
		:driving_model_ensemble_member = "r1i1p1" ;
		:driving_experiment_name = "rcp85" ;
		:rcm_version_id = "v1" ;
		:institution = "IPSL (Institut Pierre Simon Laplace) and INERIS (Institut National de l Environnement industriel et des RISques)" ;
		:institute_id = "IPSL-INERIS" ;
		:experiment_id = "rcp85" ;
		:source = "WRF 3.3.1 F" ;
		:model_id = "IPSL-INERIS-WRF331F" ;
		:forcing = "N/A" ;
		:contact = "robert.vautard@lsce.ipsl.fr Data manager : Robert Vautard - Thomas Noel" ;
		:comment = "be careful : time: mean (interval: 3 hours)" ;
		:references = "Vautard, R. et al., 2012 : The simulation of European heat waves from an ensemble of 1 regional climate models within the EURO-CORDEX project. Climate Dynamics, doi:10.1007/s00382-01-1714-z  Jacob, D. et al., 2013, EURO-CORDEX: New high-resolution climate change projections for European impact research, Regional Environmental change, doi:10.1007/s10113-013-0499-2" ;
		:initialization_method = 1 ;
		:physics_version = 1 ;
		:tracking_id = "e85f4185-c144-4fe2-a29d-27d13e76578a" ;
		:product = "output" ;
		:experiment = "RCP8.5" ;
		:frequency = "day" ;
		:creation_date = "2014-03-11T23:56:49Z" ;
		:history = "2014-03-11T23:56:49Z CMOR rewrote data to comply with CF standards and CORDEX requirements." ;
		:Conventions = "CF-1.4" ;
		:project_id = "CORDEX" ;
		:table_id = "Table day (Sept 2013) 5dadd8e128640b99180329730b4c04fa" ;
		:title = "IPSL-INERIS-WRF331F model output prepared for CORDEX RCP8.5" ;
		:modeling_realm = "atmos" ;
		:realization = 1 ;
		:cmor_version = "2.7.1" ;
}

[zklaus]

This is definitely something that is coming! In the IS-ENES3 project there is some work scheduled to make ESMValTool more applicable to regional data and already now some metrics can be applied and we have some crucial bits in place (for example regridding from projections to global grids for comparison with reanalyses).

You are also completely right that CORDEX data is an excellent candidate since it not only follows very similar standards to CMIP, but also is distributed on the ESGF.

Adding a CORDEX project would be a great step in that direction.

@aperezpredictia, would you be interested in helping with that?

[aperezpredictia]

Hi,
I finally added a CORDEX project to the 'config-developer.yml' and it seems that ESMValTool recognizes it and is able to choose the correct input CORDEX files given in the recipe.

CORDEX:
  input_dir:
    default: '/'
    drs1: '[project]/[product]/[Domain]/[Institution]/[GCMModelName]/[CMIP5ExperimentName]/[CMIP5EnsembleMember]/[RCMModelName]/[RCMVersionID]/[Frequency]/[VariableName]'
  input_file: '[short_name]_[dataset]_[gcminstitute]_[exp]_[ensemble]_[rcmmodelname]_[version]_[mip]_*.nc'
  output_file: '[short_name]_[dataset]_[exp]_[ensemble]_[version]_[mip]_[start_year]-[end_year]'
  cmor_type: 'CMIP5'

Moreover, I have done changes in the regrid part due to several errors that appeared as a consequence of the irregular regrid from the EUROPE region to the GLOBAL region. This errors came especifically as a consequence of the boundaries.... As I have already seen in several issues, ESMValTool is really strict with the boundaries. However, the CORDEX files have not any boundaries for the rlat, rlon nor for the lat, lon coordinates. In order to avoid this problem, I used one of the fixed classes given in ESMValTool which obtained the boundaries for lat, lon through the use of rlat, rlon, which would not be possible with CORDEX (because rlat and rlon have not boundaries) if the _guess_bounds iris function wouldn't exist.

As the last step, I find out a problem with the Iris regridder. If you have one of the variables for the source or the reference grid without a concrete coordinate system, it fails. Therefore, I decided to create a try/except option with which I would be able to define the regridding scheme in case that iris.cube is not able to recognize this characteristic. After several attempts, we have obtained something like this when we regrid a CORDEX EUR-11 file to the observations given by ERA5 in the same time interval with the same time frequency:

It seems that the results are fair with the data provided. However, I would like to know how to take only the European region as a square / rectangle. I have tried to use the projection parameter that you can give to the recipe script, but it does not recognize the EuroPP projection given in Cartopy. Moreover, I have tried to modify the shared/latlon.ncl file in which the function select_region is given. We created the European region giving the max and min latitude and the max and min longitude, but it did not succeed, because for the latlon plot type only the global map can be used.

Here is how I defined it:

begin

  funcname = "select_region"
  scriptname = "diag_scripts/shared/latlon.ncl"
  enter_msg(scriptname, funcname)

  if (region.eq."global") then
    region_array = (/-90., 90., 0., 360./)
    region_array@name = "Global"
    leave_msg(scriptname, funcname)
    return(region_array)
  end if

  if (region.eq."trop") then
    region_array = (/-20., 20., 0., 360./)
    region_array@name = "Tropics"
    leave_msg(scriptname, funcname)
    return(region_array)
  end if
.
.
.

  if (region.eq."eur") then
    region_array = (/27., 72., -22., 45./)
    region_array@name = "Europe CORDEX"
    leave_msg(scriptname, funcname)
    return(region_array)
  end if

  error_msg("f", scriptname, funcname, "unrecognized region " + region)

end

And here it is why the representation is raising an error when the 'eur' one is given:

  if ((ptype.eq."zonal" .or. ptype.eq."latlon") .and. \
      diag_script_info@region.ne."global") then
    error_msg("f", DIAG_SCRIPT, "", "plot_type " + ptype + \
              " implemented only for region 'global'")
  end if

Do you think that there is another option or correction in the .ncl file to obtain the square / rectangular representation that I would like to have? I am new with the manipulation of .ncl files and they are not easy to follow...

Thank you in advance!

[zklaus]

Hi, that sounds like a lot of good work! We would be very interested to get CORDEX and generally regional model support into the tool in general. Do you think you could prepare your work as a number of pull requests? Then I think we can separate all the different issues and address them one by one. What do you think?

[aperezpredictia]

Hi, I prepared it as a pull request: Making ESMValTool available for CORDEX (Regional) Data #182
Let me know if you need any clarification or further implementation!

[bouweandela]

However, I would like to know how to take only the European region as a square / rectangle.

You can use the extract_region preprocessor funtion

I have tried to use the projection parameter that you can give to the recipe script, but it does not recognize the EuroPP projection given in Cartopy.

I'm not sure what you did here, but if this is still relevant after the above comment, can you show the recipe you created?

Moreover, I have tried to modify the shared/latlon.ncl file in which the function select_region is given.

This file is only used by some NCL diagnostic scripts. The preprocessor functions are all in Python and can be found in esmvalcore/preprocessor/*.py

[aperezpredictia]

You can use the extract_region preprocessor funtion

The problem is that the extract_region function extracts a rectangular region. However, as you can see in the CORDEX data representation, this region is not rectangular. Moreover, I have seen the shapefile function, but I do not think it is a good idea to use it. I think that there should exist an option to mask a map with the data of other map.
Explaining it better: if a lat/lon combination has a NA value in CORDEX data, it should have it as well in ERA-5 (these are the two datasets that I'm comparing). When you perform a sum, subtraction, ... between two cubes, iris directly convert it to NA. However, when I want to compare the cycles between ERA5 and CORDEX, I'm not only considering the data points in the CORDEX printed area for ERA5. Do you know what I mean?

[mattiarighi]

The mask_fillvalues preprocessor may be what you need.
See https://esmvaltool.readthedocs.io/projects/ESMValCore/en/latest/esmvalcore/preprocessor.html#masking-of-missing-values.

[aperezpredictia]

The mask_fillvalues preprocessor may be what you need.
See https://esmvaltool.readthedocs.io/projects/ESMValCore/en/latest/esmvalcore/preprocessor.html#masking-of-missing-values.

Thanks! It works now!

[mattiarighi]

Did #183 address this issue?

[zklaus]

I think by and large it does. However, there are a few issues regarding boundaries and regridding open (#184, #185). Perhaps also #186 is related.

Moreover, as we are starting the work on regional models in IS-ENES3, more things are likely to pop up. Would it be ok if I start a project for this work to track the various issues? I think then we can close this one.