cmorize_obs does not work when installing ESMValTool with conda

[YanchunHe]

Hi,

I find the cmorize_obs program simply does not work. It it my first time to use it, so sorry I made sth silly.

$ cmorize_obs
Traceback (most recent call last):
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (matplotlib 3.2.1 (/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages), Requirement.parse('matplotlib<3'), {'ESMValTool'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/projects/NS9252K/conda/esmvaltool/bin/cmorize_obs", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3258, in <module>
    @_call_aside
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside
    f(*args, **kwargs)
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 586, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/projects/NS9252K/conda/esmvaltool/lib/python3.7/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ESMValTool==2.0.0b3' distribution was not found and is required by the application

softeware and system information

$ esmvaltool --version
2.0.0b8

$ python --version
Python 3.7.6

Any suggestions?
Please let me know if you need further information to diagnose this.
Thanks in advance!

Yanchun

[valeriupredoi]

You need to actually install esmvaltool in the ESMValTool source via pip install -e .[develop] - you have installed only esmvalcore in development mode, but the cmorization module is part of esmvaltool

[bouweandela]

Did you follow the installation instructions? As @valeriupredoi suggests, it looks like you may have missed a step somewhere.

[YanchunHe]

Hi @valeriupredoi @bouweandela , thanks a lot for the replies, and sorry for the late response.

Both esmvaltool and ESMValCore are installed by the system admin, so that I don't have permission to install extra package by e.g., pip to the system path.

$ conda list esmval*
# packages in environment at /conda/esmvaltool/2.0.0b3:
#
# Name                    Version                   Build  Channel
esmvalcore                2.0.0b8                    py_0    esmvalgroup
esmvaltool                2.0.0b3                    py_0    esmvalgroup

$ cd /conda/esmvaltool/2.0.0b3/lib/python3.7/site-packages
$ ls esmval*
esmvalcore:
cmor/                 _config.py       __init__.py    _provenance.py     _recipe.py         utils/
config-developer.yml  config-user.yml  _main.py       __pycache__/       recipe_schema.yml  _version.py
config-logging.yml    _data_finder.py  preprocessor/  _recipe_checks.py  _task.py

esmvaltool:
cmorizers/             diag_scripts/  install/            __pycache__/  references/
config-references.yml  __init__.py    interface_scripts/  recipes/      utils/

$ which cmorize_obs
/conda/esmvaltool/2.0.0b3/bin/cmorize_obs

I don't understand what is the development mode, and how to proceed with the pip install for the missing step?

[valeriupredoi]

OK I see your sysadmin has installed esmvaltool from conda via the conda install -c conda-forge -c esmvalgroup esmvaltool which is fine; I confirm that currently the package installs fine and the import works fine too but the command line call raises the same traceback as you are reporting. This is because the package is built with matplotlib<3 (2.2.5) and the environment is created with matplotlib>3 (3.2.1). We have two options:

• @bouweandela creates a new conda package with restrictions to matplotlib relaxed in the package meta
• your sysadmin downgrades matplotlib=3.2.1 to 2.2.5 via conda install -c conda-forge matplotlib=2.2.5

Downgrading matplotlib is minimal effort so I suggest you do that for now, in the meantime we'll fix the package deps too 🍺

[YanchunHe]

Thanks a lot figuring this out!
I will try to ask our sysadmin to downgrade matplotlib to circumvent this.
And please update me here once you release a new package with fix.

[bouweandela]

I'll make a new conda package and also try to run the unit tests as part of the package creation, so this doesn't happen again.

[YanchunHe]

Many thanks! @bouweandela

[bouweandela]

The latest release and conda package (ESMValTool v2.0.0b4) should work, could you ask your system administrator to install it and check if your problem is solved?

[YanchunHe]

@bouweandela Thanks a lot!
Our HPC system is under maintenance this week, so I will ask our sysadmin to upgrade the ESMValTool.
Hopefully it will fix this, and I will update you afterwards.