Use importlib.metadata instead of deprecated pkg_resources

[bouweandela]

Description

Closes #2094

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

Merging #2096 (831f7db) into main (9d6b963) will increase coverage by 0.00%.
The diff coverage is 88.88%.

@@           Coverage Diff           @@
##             main    #2096   +/-   ##
=======================================
  Coverage   93.09%   93.09%           
=======================================
  Files         237      237           
  Lines       12794    12795    +1     
=======================================
+ Hits        11910    11911    +1     
  Misses        884      884           

Impacted Files	Coverage Δ
esmvalcore/_main.py	90.61% <87.50%> (-0.26%)	⬇️
esmvalcore/config/_config.py	100.00% <100.00%> (+1.08%)	⬆️

[bouweandela]

I'm not able to reproduce the issue reported by Codacy locally and usage with a group keyword argument is exactly as in the examples on the docs page, so I'm not sure what is causing it.

[valeriupredoi]

@bouweandela there is an issue with this implementation in Python 3.9, bud. I'll also have a look, possibly Codacy is complaining for the same reason

[valeriupredoi]

first clue as for the fail issue - there are two slightly different packages installed (for all Python versions, not just 3.9):

 + importlib-metadata                    6.7.0  pyha770c72_0             conda-forge/noarch        26kB
 + importlib_metadata                    6.7.0  hd8ed1ab_0               conda-forge/noarch         9kB

this no bueno. Lemme dig deeper

[valeriupredoi]

so importlib_metadata is an old package that was decomissioned by the maintainer in January, see https://anaconda.org/anaconda/importlib_metadata - and they migrated it to importlib-metadata which is at 6.7.0 now (whereas _metadata stopped at 6.0.0). We have two issues here:

• we have a dep in our env that pulls importlib_metadata; we end up with both -metadata and _metadata; the import in module is obv with import importlib_metadata so that ends up importing the old package
• conda forge is silly and reports that _metadata is at the same version 6.7.0 as -metadata (most prob a bug in the ABI) - this needs reported

[valeriupredoi]

@zklaus what's the best way to report a broken package to conda-forge? Via gitter or open an actual issue? 🍺

[valeriupredoi]

OK gotcha - iris needs both importlib_metadata and importlib-metadata (hopefully not via one of their deps)

[valeriupredoi]

OK gotcha - iris needs both importlib_metadata and importlib-metadata (hopefully not via one of their deps)

it's in fact via dask and I stopped pruning environments and solving dep graphs there, I asked on Gitter first https://app.gitter.im/#/room/#conda-forge:matrix.org

[bouweandela]

Note that this module is part of the standard library: https://docs.python.org/3.11/library/importlib.metadata.html. I guess something changed between Python 3.9 and 3.10, though it isn't immediately obvious from the documentation. It looks like API docs are completely missing for the module.

[valeriupredoi]

ah then I chased a red herring then 🐟 - still, for Python 3.9 and only for Python 3.9 there are two importlib-/_resources:

    + importlib-resources                  5.12.0  pyhd8ed1ab_0            conda-forge/noarch         9kB
    + importlib_metadata                    6.7.0  hd8ed1ab_0              conda-forge/noarch         9kB
    + importlib_resources                  5.12.0  pyhd8ed1ab_0            conda-forge/noarch        31kB

Look at the sizes of the pkgs - lemme try try not specifying importlib_resources in our env

[valeriupredoi]

boom! Problem solved - by good old conda-forge packaging 😁 If you want to you can add importlib_metadata as direct requirement for us, but I'd not really bother since that's coming from scipy via iris 👍 Oh and maybe someone should tell Python core devs that importlib.metadata is broke for Python 3.9, but I've had enough conflict resolution done today (elsewhere) 😆

[bouweandela]

OK, but the point of this pull request was to use the Python standard library instead of the package that backports that to older Python versions. I'll have a look if I can get it to work for Python 3.9 when I have some time.

[valeriupredoi]

Why are you keen on using the standard library - as you can see from here, the c-f package is more stable, rather, I thought the idea was to retire pkg_resources 🍺

[zklaus]

I'd like to explain a bit the ecosystem. Pypi decided that - and _ are the same thing, i.e. in URLs, and when installing stuff with Pip you can always use either. Conda-forge does not work like that. If you want to support both formats, you need to provide two packages, which is relatively frequently done with multi-output recipes, i.e. a single recipe producing multiple packages. In that case, it usually creates one of the two packages as a dummy that depends on the other one. Here, the recipe generates the real package as importlib-metadata and a dummy package importlib_metadata that depends on the former.

So when you depend on importlib_metadata you automatically also depend on importlib-metadata, but these are not distinct packages.

ah then I chased a red herring then fish - still, for Python 3.9 and only for Python 3.9 there are two importlib-/_resources:

How did you come to this conclusion? The conda-forge packages are noarch, so there aren't different packages for different Python versions.

[zklaus]

We should probably avoid calling entry_points twice. Do we really need to turn this into a list? Or would if not entry_points(... work?

[zklaus]

so importlib_metadata is an old package that was decomissioned by the maintainer in January, see https://anaconda.org/anaconda/importlib_metadata - [...]

You are looking in the wrong place. We don't care about Anaconda packages; the corresponding Conda-forge package is alive and well at https://anaconda.org/conda-forge/importlib_metadata.

[zklaus]

To record some insight we generated at the workshop yesterday: In Python 3.9, entry_points() does not have a group keyword and instead just returns all entry points as a Python dict keyed by the entry point groups. From Python 3.10 on, entry_points is more powerful and can do some filtering. It also does not return a dict, but a specialized EntryPoints object.

Both dict in Python 3.9 and EntryPoints in 3.10+ support [] access, so we could just use entry_points()["emsvaltool_commands"], which will return all registered esmvaltool commands and raise a KeyError if there none.

The downside is that from 3.10 on, this usage is deprecated and will generate a DeprecationWarning; I could not find out when it will actually be removed.

[zklaus]

Should we really be removing importlib_resources? Note that this is a different thing than importlib_metadata. While importlib_metadata is used to access entry points, distribution package names, etc., importlib_resources is used to access resources, such as binary and text files like recipes in the distribution. Confusingly, both functionalities were provided by pkg_resources.

We should be using importlib_resources (also in the standard lib as importlib.resources) to load packaged recipes and configuration files, though we may not be doing that consequently at the moment. Basically, all uses of __file__ should be replaced with some use of importlib resources. That is a different discussion, but I point it out here to ask whether we really want to remove the dependency on importlib_resources in this PR.

[zklaus]

PS: Seems the only place where this is used is at

ESMValCore/esmvalcore/config/_config.py

Line 24 in 91aa9e4

if sys.version_info[:2] >= (3, 9):

, so if we consider Python 3.8 support to be dropped, we can remove the dependency, but then should also remove the special import code there, probably both together in a separate PR.

[bouweandela]

Yes, I forgot to remove that code when dropping support for Python 3.8 in #2053 and then I spotted the old dependencies, thought I might as well replace them with what we currently need. Did a grep through the code for any uses, but apparently missed the use in esmvalcore/config/_config.py.

[valeriupredoi]

python=3.8 went to the fishes a while back, Klaus 🐟

[zklaus]

Except for the github actions trigger I think this is ok.

As a side note, we might want to change the name for the entry points group to esmvaltool.commands. That would be more in line with the specification and prepare us for possible future entry points esmvaltool.recipes and esmvaltool.diagnostics.

[zklaus]

Needs to be removed prior to merging.

[valeriupredoi]

done! 🍺