Restart a partially completed recipe

[ledm]

Is there any way to Restart a partially completed recipe that fails at the preprocessor stage?

For instance, I'm running a preprocessor over 18 models and it takes several hours but then fails on model 17. This is frustrating, as I have to fix the recipe and re-run the entire process. I want to fix the recipe then re-run only the incomplete datasets. Is that possible at the moment?

We already have the ability to skip completed parts of the diagnostics. This would be very powerful at the preprocessor stage as well.

[bouweandela]

Is there any way to Restart a partially completed recipe that fails at the preprocessor stage?

No

For instance, I'm running a preprocessor over 18 models and it takes several hours but then fails on model 17. This is frustrating, as I have to fix the recipe and re-run the entire process.

I would recommend simply reducing the size of the dataset you're working with for debugging purposes. For example, you could select only a small area using extract_region, select a small number of years by using the command line argument --max-years or a preprocessor function extract_time, reduce the number of vertical levels by using a preprocessor extract_levels, etc.

[ledm]

Thanks for the quick reply, Bouwe.

Unfortunately, reducing the area isn't always an option. For instance, in one case I had run a particularly expensive recipe that took 12 or so hours to complete. Then a week or so later, another CMIP6 model data set appears (or was fixed). In order to add this new dataset, I had to re-run the entire recipe from scratch.

For reasons like this, I feel that this functionality will be essential in the next generation of ESMValTool. Not just for this case, but also for rapid prototyping of recipes. In the meanwhile, can we hack in a solution that will allow recipe preprocessors to be re-run?

[bouweandela]

In the meanwhile, can we hack in a solution that will allow recipe preprocessors to be re-run?

Implementing this in a reliable way is more work than I have time for. I feel that ESMValTool is already suffering enough from user friendliness and reliability problems, so I think that we should first focus on making the current features nice before starting to implement yet more new features and breaking things in the process.

If you want to hack something, you can just run the recipe with only the extra model you're interested in and add the path to the resulting metadata.yml file in the preprocessor output directory to the 'input_files' list in the settings.yml file that is the first command line argument in the diagnostic call. I wouldn't recommend using that for a figure in a paper though, just run the recipe overnight on a cluster.

[bouweandela]

I do agree that this would be a nice feature to have, though I think it would have to be used with care because the resulting plots will probably not be reproducible and doing reproducible science is something we should care about.

For instance, in one case I had run a particularly expensive recipe that took 12 or so hours to complete.

Would it be possible to share this recipe? I don't think it is in ESMValTool, is it? Because as far as I know almost all recipes we currently have in the tool complete within an hour or so. What hardware did you use and which setting did you use for max_pararallel_tasks in config-user.yml? Is it the preprocessing or the diagnostic part that is time consuming?

[valeriupredoi]

@ledm what is the reason why the preprocessor fails? If it's related to dataset's metadata not being compliant with CMOR then you have the --dry-run command line option to go through all the datasets and flag all the metadata-related issues. This runs fast and gets you the errors that you will have to report and then fix. If it's related to other reasons (MemoryError, actual data issue eg division by 0 etc) then your out of luck. But this we can defo add to a future release to do list 🍺

[ledm]

@bouweandela

Would it be possible to share this recipe? I don't think it is in ESMValTool, is it? Because as far as I know almost all recipes we currently have in the tool complete within an hour or so. What hardware did you use and which setting did you use for max_pararallel_tasks in config-user.yml? Is it the preprocessing or the diagnostic part that is time consuming?

This recipe is in the AR6 branch, but if you have access it's:
https://github.com/ESMValGroup/ESMValTool-AR6/blob/ar6_chap_3_ocean_figures/esmvaltool/recipes/recipe_ocean_fig_3_17.yml

The preprocessor is particularly heavy:

  prep_transect: # For extracting a transect
    custom_order: true
    climate_statistics:
      operator: mean
      period: full
    regrid:
      target_grid: 1x1
      scheme: linear
    zonal_means:
      coordinate: longitude
      mean_type: mean
    extract_levels:
      levels: [0.1, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000, 1250, 1500, 1750, 2000,  2400,  2800,  3200,  3600,  4000,  4400,  4800,  5200]
      scheme: linear

The other heavy part is that I'm running over 50 models, and they all need to be on the same veritical and horizontal grids.

@valeriupredoi:

the --dry-run command line option to go through all the datasets and flag all the metadata-related issues.

Doesn't seem to work for me:

esmvaltool -c config-user.yml --dry-run recipe_ocean_fig_3_20_heat_cmip6.yml
usage: esmvaltool [-h] [-v] [-c CONFIG_FILE] [-s]
                  [--max-datasets MAX_DATASETS] [--max-years MAX_YEARS]
                  [--skip-nonexistent]
                  [--diagnostics [DIAGNOSTICS [DIAGNOSTICS ...]]]
                  recipe

esmvaltool: error: unrecognized arguments: --dry-run

[valeriupredoi]

dry run is awaiting merge in #307

[bouweandela]

Thanks @ledm for elaborating. Would it be possible to also share the section with the variables and models? And what setting are you using for max_parallel_tasks?

[ledm]

---
documentation:
  description: |
    Recipe to demonstrate preprocessor slowness.
  authors:
    - demora_lee

  references:
    - demora2018gmd

  projects:
    - ukesm


preprocessors:
  # --------------------------------------------------
  # Transects preprocessors
  # --------------------------------------------------
  prep_transect: # For extracting a transect
    custom_order: true
    climate_statistics:
      operator: mean
      period: full
    regrid:
      target_grid: 1x1
      scheme: linear
    zonal_means:
      coordinate: longitude
      mean_type: mean
    extract_levels:
      levels: [0.1, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000, 1250, 1500, 1750, 2000,  2400,  2800,  3200,  3600,  4000,  4400,  4800,  5200]
      scheme: linear

diagnostics:
  # --------------------------------------------------
  # Transects diagnostics
  # --------------------------------------------------
  diag_transect_thetao:
    description: Extracting along a trajectory
    variables:
      thetao: # Temperature ocean 3D
        preprocessor: prep_transect
        #field: TO3M
        mip: Omon
        exp: historical
        start_year: 1975
        end_year: 2000
    additional_datasets:
        # ALL CMIP5 - 22/8/2019
        # find /badc/cmip5/data/cmip5/output1/*/*/historical/mon/ocean/Omon/r1i1p1/latest/thetao -type d
        - {dataset: BNU-ESM, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CanCM4, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CanESM2, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CMCC-CESM, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CMCC-CM, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CMCC-CMS, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CNRM-CM5-2, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CNRM-CM5, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: ACCESS1-0, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: ACCESS1-3, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CSIRO-Mk3-6-0, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: EC-EARTH, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: IPSL-CM5A-LR, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: IPSL-CM5A-MR, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: IPSL-CM5B-LR, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: FGOALS-g2, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: HadCM3, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: HadGEM2-CC, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: HadGEM2-ES, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: MPI-ESM-LR, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: MPI-ESM-MR, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: MPI-ESM-P, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: MRI-CGCM3, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: MRI-ESM1, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: GISS-E2-H-CC, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: GISS-E2-H, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: GISS-E2-R-CC, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: GISS-E2-R, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CCSM4, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: NorESM1-ME, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: NorESM1-M, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: HadGEM2-AO, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: GFDL-CM2p1, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: GFDL-CM3, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: GFDL-ESM2G, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: GFDL-ESM2M, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CESM1-BGC, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CESM1-CAM5-1-FV2, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CESM1-CAM5, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CESM1-FASTCHEM, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }
        - {dataset: CESM1-WACCM, project: CMIP5, ensemble: r1i1p1, mip: Omon, exp: historical, }

        # All CMIP6 - 22/8/2019
        # find /badc/cmip6/data/CMIP6/CMIP/*/*/historical/r1i1p1f?/Omon/thetao -type d -maxdepth 0
        - {dataset: CAMS-CSM1-0, project: CMIP6, ensemble: r1i1p1f1, grid: gn, mip: Omon, exp: historical, } 
        - {dataset: CanESM5, project: CMIP6, ensemble: r1i1p1f1, grid: gn, mip: Omon, exp: historical, } 
        - {dataset: CNRM-CM6-1, project: CMIP6, ensemble: r1i1p1f2, grid: gn, mip: Omon, exp: historical, }
        - {dataset: CNRM-ESM2-1, project: CMIP6, ensemble: r1i1p1f2, grid: gn, mip: Omon, exp: historical, }
        - {dataset: IPSL-CM6A-LR, project: CMIP6, ensemble: r1i1p1f1, grid: gn, mip: Omon, exp: historical, } 
        - {dataset: HadGEM3-GC31-LL, project: CMIP6, ensemble: r1i1p1f3, grid: gn, mip: Omon, exp: historical, } 
        - {dataset: UKESM1-0-LL, project: CMIP6, ensemble: r1i1p1f2, grid: gn, mip: Omon, exp: historical, } 
        - {dataset: GISS-E2-1-G, project: CMIP6, ensemble: r1i1p1f1, grid: gn, mip: Omon, exp: historical, }
        - {dataset: GISS-E2-1-H, project: CMIP6, ensemble: r1i1p1f1, grid: gr, mip: Omon, exp: historical, }
        - {dataset: CESM2, project: CMIP6, ensemble: r1i1p1f1, grid: gn, mip: Omon, exp: historical, }
        - {dataset: GFDL-CM4, project: CMIP6, ensemble: r1i1p1f1, grid: gn, mip: Omon, exp: historical, }
        - {dataset: SAM0-UNICON, project: CMIP6, ensemble: r1i1p1f1, grid: gn, mip: Omon, exp: historical, } 
        
        - {dataset: WOA,  project: OBS,  type: clim,  version: 2013v2,  start_year: 2000, end_year: 2000,  tier: 2}

I've got max_parallel_tasks: 1.

[bouweandela]

Great, can you try running with max_parallel_tasks: null instead? This should do stuff in parallel and save a lot of time. Or do you run out of memory then?