Add Listing and finding recipes to the experimental notebook API

doc/api/esmvalcore.api.config.rst

@@ -0,0 +1,141 @@
+.. _api_config:
+
+Configuration
+=============
+
+This section describes the config submodule of the API.
+
+Config
+******
+
+Configuration of ESMValCore/Tool is done via the ``Config`` object.
+The global configuration can be imported from the ``esmvalcore.experimental`` module as ``CFG``:
+
+.. code-block:: python
+
+    >>> from esmvalcore.experimental import CFG
+    >>> CFG
+    Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
+            'compress_netcdf': False,
+            'config_developer_file': None,
+            'config_file': PosixPath('/home/user/.esmvaltool/config-user.yml'),
+            'drs': {'CMIP5': 'default', 'CMIP6': 'default'},
+            'exit_on_warning': False,
+            'log_level': 'info',
+            'max_parallel_tasks': None,
+            'output_dir': PosixPath('/home/user/esmvaltool_output'),
+            'output_file_type': 'png',
+            'profile_diagnostic': False,
+            'remove_preproc_dir': True,
+            'rootpath': {'CMIP5': '~/default_inputpath',
+                         'CMIP6': '~/default_inputpath',
+                         'default': '~/default_inputpath'},
+            'save_intermediary_cubes': False,
+            'write_netcdf': True,
+            'write_plots': True})
+
+The parameters for the user configuration file are listed `here <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file>`__.
+
+``CFG`` is essentially a python dictionary with a few extra functions, similar to ``matplotlib.rcParams``.
+This means that values can be updated like this:
+
+.. code-block:: python
+
+    >>> CFG['output_dir'] = '~/esmvaltool_output'
+    >>> CFG['output_dir']
+    PosixPath('/home/user/esmvaltool_output')
+
+Notice that ``CFG`` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
+All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
+
+.. code-block:: python
+
+    >>> CFG['otoptu_dri'] = '~/esmvaltool_output'
+    InvalidConfigParameter: `otoptu_dri` is not a valid config parameter.
+
+Or, if the value entered cannot be converted to the expected type:
+
+.. code-block:: python
+
+    >>> CFG['max_years'] = '🐜'
+    InvalidConfigParameter: Key `max_years`: Could not convert '🐜' to int
+
+``Config`` is also flexible, so it tries to correct the type of your input if possible:
+
+.. code-block:: python
+
+    >>> CFG['max_years'] = '123'  # str
+    >>> type(CFG['max_years'])
+    int
+
+By default, the config is loaded from the default location (``/home/user/.esmvaltool/config-user.yml``).
+If it does not exist, it falls back to the default values.
+to load a different file:
+
+.. code-block:: python
+
+    >>> CFG.load_from_file('~/my-config.yml')
+
+Or to reload the current config:
+
+.. code-block:: python
+
+    >>> CFG.reload()
+
+
+Session
+*******
+
+Recipes and diagnostics will be run in their own directories.
+This behaviour can be controlled via the ``Session`` object.
+A ``Session`` can be initiated from the global ``Config``.
+
+.. code-block:: python
+
+    >>> session = CFG.start_session(name='my_session')
+
+A ``Session`` is very similar to the config.
+It is also a dictionary, and copies all the keys from the ``Config``.
+At this moment, ``session`` is essentially a copy of ``CFG``:
+
+.. code-block:: python
+
+    >>> print(session == CFG)
+    True
+    >>> session['output_dir'] = '~/my_output_dir'
+    >>> print(session == CFG)  # False
+    False
+
+A ``Session`` also knows about the directories where the data will stored.
+The session name is used to prefix the directories.
+
+.. code-block:: python
+
+    >>> session.session_dir
+    /home/user/my_output_dir/my_session_20201203_155821
+    >>> session.run_dir
+    /home/user/my_output_dir/my_session_20201203_155821/run
+    >>> session.work_dir
+    /home/user/my_output_dir/my_session_20201203_155821/work
+    >>> session.preproc_dir
+    /home/user/my_output_dir/my_session_20201203_155821/preproc
+    >>> session.plot_dir
+    /home/user/my_output_dir/my_session_20201203_155821/plots
+
+Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from the ``Config``.
+
+
+API reference
+*************
+
+.. autoclass:: esmvalcore.experimental.config.CFG
+    :no-inherited-members:
+    :no-show-inheritance:
+
+.. autoclass:: esmvalcore.experimental.config.Config
+    :no-inherited-members:
+    :no-show-inheritance:
+
+.. autoclass:: esmvalcore.experimental.config.Session
+    :no-inherited-members:
+    :no-show-inheritance:

doc/api/esmvalcore.api.recipe_info.rst

@@ -0,0 +1,22 @@
+.. _api_recipe_info:
+
+Recipe Metadata
+===============
+
+This section describes the :py:mod:`esmvalcore.experimental.recipe_info` submodule of the API.
+
+Recipe metadata
+***************
+
+:py:class:`esmvalcore.experimental.recipe_info.RecipeInfo` info is a class that holds metadata from a recipe.
+
+.. code-block:: python
+
+    >>> RecipeInfo('path/to/my_recipe.yml')
+    recipe = RecipeInfo('My recipe')
+
+
+API reference
+*************
+
+.. automodule:: esmvalcore.experimental.recipe_info

doc/api/esmvalcore.api.rst

@@ -7,137 +7,8 @@ This page describes the new ESMValCore API.
 The API module is available in the submodule ``esmvalcore.experimental``.
 The API is under development, so use at your own risk!
 
-Config
-******
+.. toctree::
 
-Configuration of ESMValCore/Tool is done via the ``Config`` object.
-The global configuration can be imported from the ``esmvalcore.experimental`` module as ``CFG``:
-
-.. code-block:: python
-
-    >>> from esmvalcore.experimental import CFG
-    >>> CFG
-    Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
-            'compress_netcdf': False,
-            'config_developer_file': None,
-            'config_file': PosixPath('/home/user/.esmvaltool/config-user.yml'),
-            'drs': {'CMIP5': 'default', 'CMIP6': 'default'},
-            'exit_on_warning': False,
-            'log_level': 'info',
-            'max_parallel_tasks': None,
-            'output_dir': PosixPath('/home/user/esmvaltool_output'),
-            'output_file_type': 'png',
-            'profile_diagnostic': False,
-            'remove_preproc_dir': True,
-            'rootpath': {'CMIP5': '~/default_inputpath',
-                         'CMIP6': '~/default_inputpath',
-                         'default': '~/default_inputpath'},
-            'save_intermediary_cubes': False,
-            'write_netcdf': True,
-            'write_plots': True})
-
-The parameters for the user configuration file are listed `here <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file>`__.
-
-``CFG`` is essentially a python dictionary with a few extra functions, similar to ``matplotlib.rcParams``.
-This means that values can be updated like this:
-
-.. code-block:: python
-
-    >>> CFG['output_dir'] = '~/esmvaltool_output'
-    >>> CFG['output_dir']
-    PosixPath('/home/user/esmvaltool_output')
-
-Notice that ``CFG`` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
-All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
-
-.. code-block:: python
-
-    >>> CFG['otoptu_dri'] = '~/esmvaltool_output'
-    InvalidConfigParameter: `otoptu_dri` is not a valid config parameter.
-
-Or, if the value entered cannot be converted to the expected type:
-
-.. code-block:: python
-
-    >>> CFG['max_years'] = '🐜'
-    InvalidConfigParameter: Key `max_years`: Could not convert '🐜' to int
-
-``Config`` is also flexible, so it tries to correct the type of your input if possible:
-
-.. code-block:: python
-
-    >>> CFG['max_years'] = '123'  # str
-    >>> type(CFG['max_years'])
-    int
-
-By default, the config is loaded from the default location (``/home/user/.esmvaltool/config-user.yml``).
-If it does not exist, it falls back to the default values.
-to load a different file:
-
-.. code-block:: python
-
-    >>> CFG.load_from_file('~/my-config.yml')
-
-Or to reload the current config:
-
-.. code-block:: python
-
-    >>> CFG.reload()
-
-
-Session
-*******
-
-Recipes and diagnostics will be run in their own directories.
-This behaviour can be controlled via the ``Session`` object.
-A ``Session`` can be initiated from the global ``Config``.
-
-.. code-block:: python
-
-    >>> session = CFG.start_session(name='my_session')
-
-A ``Session`` is very similar to the config.
-It is also a dictionary, and copies all the keys from the ``Config``.
-At this moment, ``session`` is essentially a copy of ``CFG``:
-
-.. code-block:: python
-
-    >>> print(session == CFG)
-    True
-    >>> session['output_dir'] = '~/my_output_dir'
-    >>> print(session == CFG)  # False
-    False
-
-A ``Session`` also knows about the directories where the data will stored.
-The session name is used to prefix the directories.
-
-.. code-block:: python
-
-    >>> session.session_dir
-    /home/user/my_output_dir/my_session_20201203_155821
-    >>> session.run_dir
-    /home/user/my_output_dir/my_session_20201203_155821/run
-    >>> session.work_dir
-    /home/user/my_output_dir/my_session_20201203_155821/work
-    >>> session.preproc_dir
-    /home/user/my_output_dir/my_session_20201203_155821/preproc
-    >>> session.plot_dir
-    /home/user/my_output_dir/my_session_20201203_155821/plots
-
-Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from the ``Config``.
-
-
-API reference
-*************
-
-.. autoclass:: esmvalcore.experimental.config.CFG
-    :no-inherited-members:
-    :no-show-inheritance:
-
-.. autoclass:: esmvalcore.experimental.config.Config
-    :no-inherited-members:
-    :no-show-inheritance:
-
-.. autoclass:: esmvalcore.experimental.config.Session
-    :no-inherited-members:
-    :no-show-inheritance:
+   esmvalcore.api.config
+   esmvalcore.api.recipe_info
+   esmvalcore.api.utils

doc/api/esmvalcore.api.utils.rst

@@ -0,0 +1,63 @@
+.. _api_utils:
+
+
+
+Utils
+=====
+
+This section describes the utilities submodule of the API.
+
+Finding recipes
+***************
+
+One of the first thing we may want to do, is to simply get one of the recipes available in ``ESMValTool``
+
+If you already know which recipe you want to load, call :py:func:`esmvalcore.experimental.utils.get_recipe`.
+
+.. code-block:: python
+
+    from esmvalcore.experimental import get_recipe
+    >>> get_recipe('examples/recipe_python')
+    RecipeInfo('Recipe python')
+
+Call the :py:func:`esmvalcore.experimental.utils.get_all_recipes` function to get a list of all available recipes.
+
+.. code-block:: python
+
+    >>> from esmvalcore.experimental import get_all_recipes
+    >>> recipes = get_all_recipes()
+    >>> recipes
+    [RecipeInfo('Recipe perfmetrics cmip5 4cds'),
+     RecipeInfo('Recipe martin18grl'),
+     ...
+     RecipeInfo('Recipe wflow'),
+     RecipeInfo('Recipe pcrglobwb')]
+
+To search for a specific recipe, you can use the :py:meth:`esmvalcore.experimental.utils.RecipeList.find` method. This takes a search query that looks through the recipe metadata and returns any matches. The query can be a regex pattern, so you can make it as complex as you like.
+
+.. code-block:: python
+
+    >>> results = recipes.find('climwip')
+    [RecipeInfo('Recipe climwip')]
+
+The recipes are loaded in a :py:class:`esmvalcore.experimental.recipe_info.RecipeInfo` object, which knows about the documentation, authors, project, and related references of the recipe. It resolves all the tags, so that it knows which institute an author belongs to and which references are associated with the recipe.
+
+This means you can search for something like this:
+
+.. code-block:: python
+
+    >>> recipes.find('Geophysical Research Letters')
+    [RecipeInfo('Recipe martin18grl'),
+     RecipeInfo('Recipe climwip'),
+     RecipeInfo('Recipe ecs constraints'),
+     RecipeInfo('Recipe ecs scatter'),
+     RecipeInfo('Recipe ecs'),
+     RecipeInfo('Recipe seaice')]
+
+
+API reference
+*************
+
+.. automodule:: esmvalcore.experimental.utils
+    :no-inherited-members:
+    :no-show-inheritance:

doc/requirements.txt

@@ -6,6 +6,7 @@ numpy
 pandas
 pillow
 prov[dot]
+pybtex
 pyyaml
 scipy
 shapely[vectorized]

esmvalcore/experimental/__init__.py

@@ -8,7 +8,13 @@
     '\n  More info: https://github.com/ESMValGroup/ESMValCore/issues/498', )
 
 from .config import CFG  # noqa: E402
+from .recipe_info import RecipeInfo  # noqa: E402
+from .utils import RecipeList, get_all_recipes, get_recipe  # noqa: E402
 
 __all__ = [
     'CFG',
+    'get_all_recipes',
+    'get_recipe',
+    'RecipeInfo',
+    'RecipeList',
 ]