Add option to expand ensembles

[jvegreg]

In CMIP6, some models are providing 30 ensemble members or more. Processing them all is currently a nightmare of copy pasting and very error prone:

The currently implemented syntax is the following:

  - {<<: *cmip, dataset: IPSL-CM6A-LR, ensemble: 'r[1:30]i1p1f1'}

This will add all datasets from r1i1p1f1 to r30i1p1f1, making dealing with big ensembles easier.

Any suggestions on the syntax are welcome. The current code only supports expanding one of the numbers.

[bouweandela]

Any suggestions on the syntax are welcome. The current code only supports expanding one of the numbers.

Nice addition. I think that normal glob rules provided by fnmatch would be easiest to understand for most users, i.e. '*', '?', '[0-9]', etc. It would be nice to have these available not just for ensemble members, but also for dataset name, start_year, end_year, ...

[jvegreg]

Nice addition. I think that normal glob rules provided by fnmatch would be easiest to understand for most users, i.e. '*', '?', '[0-9]', etc. It would be nice to have these available not just for ensemble members, but also for dataset name, start_year, end_year, ...

This will be nice, but they are two different kind of animals:

• This pull request aims to define multiple datasets with only one definition, but in a strict way. If any of those is missing, recipe will fail

• fn_match will be more like find anything matching this pattern and run it. Will be more difficult, for example, to easily run using inly the first 15 members and ignoring the rest. Also, it is more difficult to implement, as we need to check which ones are available from all the possible combinations

What we can do to avoid confussion is to change the syntax to avoid both paradigms to coexist.

[bouweandela]

find anything matching this pattern and run it.

This is a feature that several users have asked me about. I agree that it is not trivial to implement.

What we can do to avoid confussion is to change the syntax to avoid both paradigms to coexist.

Yes, I think it's important to think about this. Of course we also have the --skip-nonexistent option available..

[bouweandela]

What we can do to avoid confussion is to change the syntax to avoid both paradigms to coexist.

I think the square brackets are not good in that case, because those can also be used in globs. Maybe use (1:30) instead?

[jvegreg]

Good idea. Already implemented

[bouweandela]

Could you also add documentation? This might be a good place.