Unable to install with pip via main branch

[lsterzinger]

Hello,

I'm attempting to use a software package (https://github.com/intake/fsspec-reference-maker) that depends upon #273. However, the latest release of numcodecs is 0.7.3 which was released a couple months before #273 was merged.

I attempted to install the development version of numcodecs on Ubuntu 20.04.1 LTS (on the Pangeo Cloud) via

pip install git+https://github.com/zarr-developers/numcodecs

but it fails with

  /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/lz4-1.9.2 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.4.4 -Ic-blosc/internal-complibs/zstd-1.4.4/common -Ic-blosc/internal-complibs/zstd-1.4.4/compress -Ic-blosc/internal-complibs/zstd-1.4.4/decompress -Ic-blosc/internal-complibs/zstd-1.4.4/deprecated -Ic-blosc/internal-complibs/zstd-1.4.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.4.4/dll -Ic-blosc/internal-complibs/zstd-1.4.4/legacy -Ic-blosc/internal-complibs/zstd-1.4.4/dll/example -I/srv/conda/envs/notebook/include/python3.8 -c c-blosc/internal-complibs/snappy-1.1.1/snappy-c.cc -o build/temp.linux-x86_64-3.8/c-blosc/internal-complibs/snappy-1.1.1/snappy-c.o -DSHUFFLE_SSE2_ENABLED -DSHUFFLE_AVX2_ENABLED
  x86_64-conda-linux-gnu-cc: fatal error: cannot execute 'cc1plus': execvp: No such file or directory
  compilation terminated.
  [numcodecs] command '/srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc' failed with exit status 1
  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/unixccompiler.py", line 117, in _compile
      self.spawn(compiler_so + cc_args + [src, '-o', obj] +
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/ccompiler.py", line 910, in spawn
      spawn(cmd, dry_run=self.dry_run)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/spawn.py", line 36, in spawn
      _spawn_posix(cmd, search_path, dry_run=dry_run)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/spawn.py", line 157, in _spawn_posix
      raise DistutilsExecError(
  distutils.errors.DistutilsExecError: command '/srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc' failed with exit status 1
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/tmp/pip-req-build-qi5shd1w/setup.py", line 301, in build_extension
      build_ext.build_extension(self, ext)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
      objects = self.compiler.compile(sources,
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/ccompiler.py", line 574, in compile
      self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/unixccompiler.py", line 120, in _compile
      raise CompileError(msg)
  distutils.errors.CompileError: command '/srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc' failed with exit status 1
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-qi5shd1w/setup.py", line 377, in <module>
      run_setup(with_extensions)
    File "/tmp/pip-req-build-qi5shd1w/setup.py", line 326, in run_setup
      setup(
    File "/srv/conda/envs/notebook/lib/python3.8/site-packages/setuptools/__init__.py", line 163, in setup
      return distutils.core.setup(**attrs)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/srv/conda/envs/notebook/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-qi5shd1w/setup.py", line 294, in run
      build_ext.run(self)
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
      self._build_extensions_serial()
    File "/srv/conda/envs/notebook/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-req-build-qi5shd1w/setup.py", line 304, in build_extension
      raise BuildFailed()
  __main__.BuildFailed
  ----------------------------------------
  ERROR: Failed building wheel for numcodecs

and I'm unsure how to proceed. A quick google shows that it may be related to g++ not being installed, but I'm running on the Pangeo Cloud and do not have access to install system compilers/libraries.

I tried installing locally on a macOS device with g++ installed and get a different error

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/lucass/anaconda3/include -arch x86_64 -I/Users/lucass/anaconda3/include -arch x86_64 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zstd-1.4.4 -Ic-blosc/internal-complibs/lz4-1.9.2 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.4.4/compress -Ic-blosc/internal-complibs/zstd-1.4.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.4.4/decompress -Ic-blosc/internal-complibs/zstd-1.4.4/legacy -Ic-blosc/internal-complibs/zstd-1.4.4/common -Ic-blosc/internal-complibs/zstd-1.4.4/dll -Ic-blosc/internal-complibs/zstd-1.4.4/deprecated -Ic-blosc/internal-complibs/zstd-1.4.4/dll/example -I/Users/lucass/anaconda3/include/python3.8 -c c-blosc/internal-complibs/zlib-1.2.8/gzread.c -o build/temp.macosx-10.9-x86_64-3.8/c-blosc/internal-complibs/zlib-1.2.8/gzread.o -msse2 -mavx2 -stdlib=libc++ -DSHUFFLE_SSE2_ENABLED -DSHUFFLE_AVX2_ENABLED
  c-blosc/internal-complibs/zlib-1.2.8/gzread.c:30:15: error: implicit declaration of function 'read' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
          ret = read(state->fd, buf + *have, len - *have);
                ^
  c-blosc/internal-complibs/zlib-1.2.8/gzread.c:30:15: note: did you mean 'fread'?
  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:158:9: note: 'fread' declared here
  size_t   fread(void * __restrict __ptr, size_t __size, size_t __nitems, FILE * __restrict __stream);
           ^
  c-blosc/internal-complibs/zlib-1.2.8/gzread.c:591:11: error: implicit declaration of function 'close' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
      ret = close(state->fd);
            ^
  2 errors generated.
  [numcodecs] command 'gcc' failed with exit status 1

Is there a good way to get around this? Or, if at all possible, would a new release trigger the building of new wheels for pip or a package on conda-forge?

[lsterzinger]

I forgot to mention a couple things:
My python version is 3.8.6 and the macOS machine I attempted to install on locally is an Intel machine, not the newer ARM64.

[joshmoore]

If 0.7.3 works for you, I assume a new release would once deployed to conda-forge would work for you.

I assume there are no objections to a release from anyone. Looking at the merge history:

462278f (origin/master, origin/HEAD) Update cpuinfo to 8.0.0 (#280) (7 days ago) <Florian Jetter>
d16d1ea numcodecs.zfpy is ready (#229) (3 months ago) <haiying xu>
f946d7d Merge pull request #273 from pbranson/shufflecodec-numba (3 months ago) <Martin Durant>
006476a Merge pull request #276 from JacksonMaxfield/master (3 months ago) <Martin Durant>
7ef98f7 Drop out-of-date manual release upload doc bit (#272) (5 months ago) <jakirkham>

I do wonder if this worth a bump to 0.8.0. Anyone?

[jakirkham]

We also publish wheels to PyPI. So one can use that if they are ok with 0.7.3

👍 to bumping. There are build fixes for ARM and a couple new filters that I'm sure people would be excited to have access to 🙂

[jakirkham]

cc @zarr-developers/core-devs (in case others have thoughts on doing a 0.8.0 release)

[martindurant]

Yes please, release early, release often

[joshmoore]

v0.8.0 pushed. Builds should now be running now. Please let us know if you have any issues using them.

[lsterzinger]

This seems to have worked perfectly, thank you all!