Problems importing blosc on HPC cluster

[rabernat]

I am using zarr on an HPC cluster with a rather complex configuration (multiple conda envs, dask distributed, etc.)

Minimal, reproducible code sample, a copy-pastable example if possible

# What should happen (and works on cluster head node)
>>> import zarr
>>> zarr.Blosc
numcodecs.blosc.Blosc

# What happens on a compute node
>>> import zarr
>>> zarr.Blosc
AttributeError: module 'zarr' has no attribute 'Blosc
>>> import numcodecs.blosc
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/numcodecs/blosc.cpython-36m-x86_64-linux-gnu.so)

Problem description

Both examples use the same underlying conda environment and have identical zarr / numcodecs version. However, there is some deep library error which makes Blosc unusable on the compute nodes.

Version and installation information

Please provide the following:

• Value of zarr.__version__: 2.2.0rc4.dev1
• Value of numcodecs.__version__: 0.5.3
• Version of Python interpreter: 3.6.4
• Operating system (Linux/Windows/Mac) Linux
• How Zarr was installed (e.g., "using pip into virtual environment", or "using conda"): pip

[jakirkham]

The error means that the build (i.e. pip install), as we ship sdists on PyPI only, occurred on a machine that used glibc 2.14 or greater, was run on a machine that had a glibc version older than 2.14. IOW it was built on a newer version of Linux than you are running it on. The solution would be to build on the version of Linux you intend to run it on, but for various reasons that may be more complicated. One alternative might be to try the conda-forge binary for numcodecs, which works with glibc 2.12 or newer. If that doesn't work, we can discuss other options.

[jakirkham]

Any luck @rabernat?

[jakirkham]

Going to close as this isn't a problem with numcodecs per se, but an issue with the build (i.e. pip install step) occurring on a newer OS than the target machine's OS. Of course happy to talk with you more about this @rabernat if you would like and discuss other options if the one's suggested above don't work for your use case.

[jakirkham]

FWIW we could probably build binary wheels too as outlined in issue ( #70 ).