Use glob in datafinder

[jvegreg]

A small change to allow users to use wildcard characters in path definiton (we already allow it on files).

I added it because of a quirk our observations convention has: variable folder can just be the variable name or it can have appended the original frequency used for the computation:

Example:

• psl monthly means for ERA5 if directly provided by ECMF: /esarchive/recon/ecmwf/era5/daily_mean/psl

• psl monthly means for ERA5 if computed from hourly outputs provided by ECMF: /esarchive/recon/ecmwf/era5/daily_mean/psl_f1h

Both will be found with the new code if folder path is defined whatever\[short_name]* but I will need to add a extra tag for the user to provide with the old version

As a bonus, things like PRIMAVERA workspaces in jasmin (which are primavera1 to primavera5) can be add now as only one 'primavera?'

[valeriupredoi]

not a big fan of glob.glob man, why not use os.walk 🍺

[jvegreg]

Because os.walk does not support wildcards and that is the reason to use glob. Also, I do not want to walk the whole tree

[valeriupredoi]

ok, the "I do not want to walk the whole tree" sold me 😁

[bouweandela]

Nice addition, can you add unit tests?

[mattiarighi]

Can this be extended to also address #250?

[jvegreg]

Can this be extended to also address #250?
Yes, will do it

Another question is how to make both work with #206

[jvegreg]

Done!

[mattiarighi]

We should test this on several machines/drs'.
I added @valeriupredoi and @bascrezee as reviewers.
Also @schlunma could help checking with CMIP6 data at DKRZ (I'm testing CMIP5/OBS/obs4mips).

[valeriupredoi]

Ok Ill test it out myself asap Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Tue, 8 Oct 2019, 09:46 Mattia Righi, ***@***.***> wrote: ***@***.**** approved this pull request. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#210?email_source=notifications&email_token=AG5EFIZR5LNJ3BNLD33UGYTQNRCGNA5CNFSM4IPOCR4KYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCHGIAFQ#pullrequestreview-298614806>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG5EFI5QPFS3JDUADHLTOFLQNRCGNANCNFSM4IPOCR4A> .

[bascrezee]

I am testing using the settings from recipe_check_obs, is that how I am intended to do the testing? @ruthlorenz If you sent me your datasets (e-mail) I can include them as well in this test.

[mattiarighi]

recipe_check_obs is a good test case for this PR.

[schlunma]

Successfully tested for many CMIP5 and CMIP6 models (using this recipe).

[valeriupredoi]

hold merging, this breaks the data finding for fx variables. I am taking a look now

[valeriupredoi]

so why was config-developer.yml not aligned from development?

[mattiarighi]

There were some conflicts which I fixed, maybe I overlooked something.

[mattiarighi]

But now you reverted all the changes of this PR, we use { instead of [...

[valeriupredoi]

yeah I just saw that, need to revert the revert and fix the issues manually

[valeriupredoi]

nobody move an inch, am making the changes now

[valeriupredoi]

ok tested with @ruthlorenz collins recipe, OK to go

[bascrezee]

Testing is ongoing here.

[bascrezee]

Testing is ongoing here.

What is the meaning of this? I do not specify any preprocessor, but it does call a 'default' preprocessor. This seems not to occur for OBS.
INFO Creating preprocessor 'default' task for variable 'tas'

[valeriupredoi]

Testing is ongoing here.

What is the meaning of this? I do not specify any preprocessor, but it does call a 'default' preprocessor. This seems not to occur for OBS.
INFO Creating preprocessor 'default' task for variable 'tas'

any time you don't specify a preprocessor name for a variable, it will create the default one -> this is per variable and not per dataset; the default preprocessor will contain only the basic preproc steps: load, cmor check, fix, extract time etc (note that extract time does not happen for default prprocs for fx varibales that are time-invariant)