Read in CMIP3 datasets

[LisaBock]

I need to read in CMIP3 datasets in the ESMValTool. I started to implement this feature in the code but it was not successful as I did something wrong or missed something for sure.
Is there anyone who could support me?

What I did:
I modified cmor/table.py and config-developer.yml and added the CMOR table file cmor/tables/cmip3/Tables/CMIP3_A1 (see branch version2_development_cmip3).
And I tried it with adding this dataset lines in the recipes/recipe_python.yml:

  - {dataset: ukmo_hadcm3,  project: CMIP3,  mip: A1,  exp: 20c3m,  ensemble: run1,  start_year: 1997,  end_year: 1999}
  - {dataset: ukmo_hadgem1,  project: CMIP3,  mip: A1,  exp: 20c3m,  ensemble: run1,  start_year: 1997,  end_year: 1999}

On DKRZ the data structure is like this:
<CMIP3-directory>/20c3m/atm/mo/pr/ukmo_hadgem1/run1/pr_A1.nc

The error message is:
esmvaltool._recipe_checks.RecipeError: Unable to load CMOR table 'CMIP3' for variable 'tas' with mip 'A1'

[jvegreg]

Hi Lisa

I will take a look. We are talking about pre-CMOR here, so probably it will need quite a lot of manual effort. it looks like there is not even a common file format

[LisaBock]

I just found the definition of output on the PCMDI website. Nothing else.

[jvegreg]

I have updated the branch with a couple of fixes. Now it works, as long as you change the variable in the recipe to tas because it is the one defined in A1

I also added unit testing for the CMIP3 tables

[LisaBock]

Thanks. I will test it on Monday.

[LisaBock]

@jvegasbsc Perfect, now the ESMValTool found my CMIP3 files.

But there are still more problems:

• The ESMValTool wants to find start_year and end_year in the filename which is not always possible in the CMIP3 models.
  File names are e.g. tas_A1_1.nc, tas_A1.20C3M_1.CCSM.atmm.1870-01_cat_1879-12.nc, tas_A1_a42_0007-0147.nc, tas_A1.GISS1.20C3M.run1.nc, tas_A1.nc

• Frequency is named "mo" instead of "mon".

• The next stop was with iris.exceptions.CoordinateNotFoundError: 'Expected to find exactly 1 coordinate, but found none.' when I tried to read the tas variable.

I could not solve all the problems by my own as I am not totally familiar with all the python/iris/cmor coding...

[jvegreg]

Let me take a look...

[jvegreg]

The ESMValTool wants to find start_year and end_year in the filename which is not always possible in the CMIP3 models.

This should be solved now in the CMIP3 branch

[valeriupredoi]

just gotta say - hats off to you guys for doing this, CMIP3 data is probably older than me and I am pretty ancient myself 🤣 One idea - can we use the OBS cmorizers to cmorize CMIP3 data in advance of using it?

[jvegreg]

Now that we are restarting this...

One idea - can we use the OBS cmorizers to cmorize CMIP3 data in advance of using it?

I would like to try avoiding this. It will be a good test to see if we can manage to read loosely structured datasets