Option to install numpy built with Apple's Accelerate BLAS implementation

[ollie-bell]

More of a feature request. Is there a plan to enable installation of numpy built against Apple's Accelerate BLAS implementation when on oxs-arm64? e.g. something similar to conda install -c conda-forge numpy "libblas=*=*accelerate" (based on the instructions here).

This can be achieved by building numpy from source and installing via pip (see these instructions), but it would be great to have a clean conda installation to achieve the same thing.

[h-vetinari]

Conda-forge can only really depend on things it can distribute itself (with some exceedingly rare exceptions that are both rock-stable and also difficult/dangerous to replace; e.g. glibc). Since the accelerate sources are not available, and I don't think the binaries are redistributable (or ABI-stable enough without further metadata), I'm doubtful whether this is feasible.

@isuruf might have a more definite answer.

[isuruf]

That's not the issue. The issue is that Accelerate's lapack version is 3.2.1 which is ancient. Therefore Accelerate can only provide blas which means only numpy can support it. scipy and many other packages cannot.

[ollie-bell]

That's interesting. I guess that reflects Apple's lack of support for Fortran. But yes, now that you mention it, the SciPy docs also say the same.

[h-vetinari]

Isuru found a way around this: conda-forge/blas-feedstock#82

In 2-3 hours, you should be able to try installing conda install numpy libblas=*=*accelerate. :)

I'll start testing this in #252 once it becomes available as well...

[ollie-bell]

Seems to be broken at the moment. I created a fresh environment with Python 3.9 and conda install numpy "libblas=*=*accelerate"

Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:06) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
zsh: killed     python

[jakirkham]

@ollie-bell would suggest sharing the full list of packages from the environment in case there is a clue in there. Also would include conda info. Is this using Intel or ARM?

[ollie-bell]

conda info and conda list are below. I'm on an M1 MacBook, so ARM.

     active environment : test
    active env location : /Users/oliver/miniforge3/envs/test
            shell level : 3
       user config file : /Users/oliver/.condarc
 populated config files : /Users/oliver/miniforge3/.condarc
                          /Users/oliver/.condarc
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.9.9.final.0
       virtual packages : __osx=12.1=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/oliver/miniforge3  (writable)
      conda av data dir : /Users/oliver/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/oliver/miniforge3/pkgs
                          /Users/oliver/.conda/pkgs
       envs directories : /Users/oliver/miniforge3/envs
                          /Users/oliver/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.11.0 requests/2.27.1 CPython/3.9.9 Darwin/21.2.0 OSX/12.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

# Name                    Version                   Build  Channel
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2021.10.8            h4653dfc_0    conda-forge
libblas                   3.9.0           12_osxarm64_accelerate    conda-forge
libcblas                  3.9.0           12_osxarm64_accelerate    conda-forge
libcxx                    12.0.1               h168391b_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0.dev0      11_0_1_hf114ba7_23    conda-forge
libgfortran5              11.0.1.dev0         hf114ba7_23    conda-forge
liblapack                 3.9.0           12_osxarm64_accelerate    conda-forge
libzlib                   1.2.11            hee7b306_1013    conda-forge
llvm-openmp               12.0.1               hf3c4609_1    conda-forge
ncurses                   6.2                  h9aa5885_4    conda-forge
numpy                     1.22.0           py39h61a45d2_0    conda-forge
openssl                   3.0.0                h3422bc3_2    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
python                    3.9.9           h43b31ca_0_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
readline                  8.1                  hedafd6a_0    conda-forge
setuptools                60.5.0           py39h2804cbe_0    conda-forge
sqlite                    3.37.0               h72a2b83_0    conda-forge
tk                        8.6.11               he1e0b03_1    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
zlib                      1.2.11            hee7b306_1013    conda-forge

[h-vetinari]

On osx-x86, the test suite runs in #252 through without failures. However, osx-arm64 (as for @ollie-bell) isn't being tested (only cross-compiled) because we don't have the hardware for it in CI.

[jakirkham]

Does this issue show if you use netlib instead of accelerate?

[ollie-bell]

@jakirkham the issue only shows with accelerate. Note that I can successfully install numpy with accelerate via pip, and it does show ~2x speed-up in the numpy linalg benchmark over the default conda-forge numpy.

[isuruf]

Should be fixed now

[ngam]

@ollie-bell could you report back on the performance please? Do you still see any performance improvement?

[ollie-bell]

@ngam yes indeed. When I run the numpy linalg benchmarks I typically see 2-4x speed up with the new accelerate installation. I assume those benchmarks are robust and representative of real use!