Build failures in Darwin arm64 (Mac M1)

[kueda]

I'm having trouble building from source on an M1 Mac. TL;DR, binaries don't seem to be available, and building from source seems to result to dependencies conflicts between mapnik, node-mapnik, and mapnik-vector-tile, or maybe protozero is an undocumented external requirement. I'm using node 16.15.1, node-mapnik 693e402.

> git clone git@github.com:mapnik/node-mapnik.git
> cd node-mapnik
# Using nvm to use a recent-ish version of node
> nvm use 16.15.1
> npm install
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.

> mapnik@4.5.9 install
> node-pre-gyp install --fallback-to-build

node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@1.0.9
node-pre-gyp info using node@16.15.1 | darwin | arm64
node-pre-gyp info check checked for "/Users/kueda/Downloads/node-mapnik/lib/binding/mapnik.node" (not found)
node-pre-gyp http GET https://mapbox-node-binary.s3.amazonaws.com/mapnik/v4.5.9/Release/darwin-arm64.tar.gz
node-pre-gyp ERR! install response status 403 Forbidden on https://mapbox-node-binary.s3.amazonaws.com/mapnik/v4.5.9/Release/darwin-arm64.tar.gz 
node-pre-gyp WARN Pre-built binaries not installable for mapnik@4.5.9 and node@16.15.1 (node-v93 ABI, unknown) (falling back to source compile with node-gyp) 
node-pre-gyp WARN Hit error response status 403 Forbidden on https://mapbox-node-binary.s3.amazonaws.com/mapnik/v4.5.9/Release/darwin-arm64.tar.gz 
gyp info it worked if it ends with ok
gyp info using node-gyp@9.0.0
gyp info using node@16.15.1 | darwin | arm64
gyp info ok 
gyp info it worked if it ends with ok
gyp info using node-gyp@9.0.0
gyp info using node@16.15.1 | darwin | arm64
gyp info find Python using Python version 3.9.13 found at "/opt/homebrew/opt/python@3.9/bin/python3.9"
gyp info spawn /opt/homebrew/opt/python@3.9/bin/python3.9
gyp info spawn args [
gyp info spawn args   '/Users/kueda/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/kueda/Downloads/node-mapnik/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/kueda/Downloads/node-mapnik/common.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/kueda/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/kueda/Library/Caches/node-gyp/16.15.1/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/Users/kueda/Library/Caches/node-gyp/16.15.1',
gyp info spawn args   '-Dnode_gyp_dir=/Users/kueda/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/Users/kueda/Library/Caches/node-gyp/16.15.1/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/kueda/Downloads/node-mapnik',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
/bin/sh: mapnik-config: command not found
gyp: Call to 'mapnik-config --cflags' returned exit status 127 while in binding.gyp. while trying to load binding.gyp

and then the build just fails, but the root problem there is the binary isn't available, so I should instead be using the more specific source build options at https://github.com/mapnik/node-mapnik/#source-build (right?)

> make release                                                                                                                                                                                                (master) 14:05:39
./scripts/install_deps.sh
Did not detect global clang++ at '/Users/kueda/.mason/mason_packages/osx-arm64/clang++/10.0.0' or /Users/kueda/.mason/mason_packages/osx-arm64/llvm/10.0.0
Downloading https://s3.amazonaws.com/mason-binaries/osx-arm64/clang++/10.0.0.tar.gz
curl: (22) The requested URL returned error: 403
make: *** [mason_packages/.link/bin/mapnik-config] Error 22

Looks like Mason doesn't have a copy of a clang binary for my architecture. Does this have something to do with SSE support?

> SSE_MATH=false make release                                                                                                                                                                                 (master) 14:09:01
./scripts/install_deps.sh
Did not detect global clang++ at '/Users/kueda/.mason/mason_packages/osx-arm64/clang++/10.0.0' or /Users/kueda/.mason/mason_packages/osx-arm64/llvm/10.0.0
Downloading https://s3.amazonaws.com/mason-binaries/osx-arm64/clang++/10.0.0.tar.gz
curl: (22) The requested URL returned error: 403
make: *** [mason_packages/.link/bin/mapnik-config] Error 22

Doesn't look like it. So let's try source build strategy B: install an external copy of mapnik on my system:

> brew install mapnik
==> Downloading https://ghcr.io/v2/homebrew/core/mapnik/manifests/3.1.0_9
Already downloaded: /Users/kueda/Library/Caches/Homebrew/downloads/4028d12c0c89cb35cce279f7489143a8e11fb374595da59b958f45139740deee--mapnik-3.1.0_9.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/mapnik/blobs/sha256:88c2bfae2fc095df973e6c1a24bd6424cdecb15a8cfe9e276cff215740e4150b
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:88c2bfae2fc095df973e6c1a24bd6424cdecb15a8cfe9e276cff215740e4150b?se=2022-06-09T21%3A20%3A00Z&sig=a0B15fG%2FjDAPGmBQKYJergVAI%2BxAntIvkRdQbR%2BPGnA%3D&sp=r&spr=http
######################################################################## 100.0%
==> Pouring mapnik--3.1.0_9.arm64_monterey.bottle.tar.gz
🍺  /opt/homebrew/Cellar/mapnik/3.1.0_9: 507 files, 38.4MB
==> Running `brew cleanup mapnik`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

From the homebrew formula, it seems to be installing the latest available release of mapnik from https://github.com/mapnik/mapnik/releases/download/v3.1.0/mapnik-v3.1.0.tar.bz2. Is mapnik working?

> mapnik-config --prefix                                                                                                                                                                                      (master) 14:14:16
/opt/homebrew

Yes. Let's try npm install again:

> npm install
# ...snip...
gyp info using node-gyp@9.0.0
gyp info using node@16.15.1 | darwin | arm64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  CXX(target) Release/obj.target/mapnik/src/mapnik_logger.o
  CXX(target) Release/obj.target/mapnik/src/node_mapnik.o
In file included from ../src/node_mapnik.cpp:4:
../node_modules/mapnik-vector-tile/src/vector_tile_config.hpp:9:10: fatal error: 'protozero/types.hpp' file not found
#include <protozero/types.hpp>
         ^~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [Release/obj.target/mapnik/src/node_mapnik.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/kueda/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:527:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Darwin 21.3.0
# ...snip...

Nope. Maybe I should try building mapnik from source:

> brew uninstall mapnik
> brew install mapnik -s
# this takes a long time but succeeds without errors
> npm install
# ...snip...
gyp info using node-gyp@9.0.0
gyp info using node@16.15.1 | darwin | arm64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  CXX(target) Release/obj.target/mapnik/src/mapnik_logger.o
  CXX(target) Release/obj.target/mapnik/src/node_mapnik.o
In file included from ../src/node_mapnik.cpp:4:
../node_modules/mapnik-vector-tile/src/vector_tile_config.hpp:9:10: fatal error: 'protozero/types.hpp' file not found
#include <protozero/types.hpp>
         ^~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [Release/obj.target/mapnik/src/node_mapnik.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/kueda/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:527:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
# ...snip...

Maybe I should try building from HEAD:

> brew uninstall mapnik
> brew install mapnik --HEAD # this requires removing the `inreplace` line in the formula
> npm install
# ...snip...
gyp info using node-gyp@9.0.0
gyp info using node@16.15.1 | darwin | arm64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  CXX(target) Release/obj.target/mapnik/src/mapnik_logger.o
  CXX(target) Release/obj.target/mapnik/src/node_mapnik.o
In file included from ../src/node_mapnik.cpp:4:
../node_modules/mapnik-vector-tile/src/vector_tile_config.hpp:9:10: fatal error: 'protozero/types.hpp' file not found
#include <protozero/types.hpp>
         ^~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [Release/obj.target/mapnik/src/node_mapnik.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/kueda/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:527:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
# ...snip...

Anyway, looks like it's trying to find protozero files and they don't exist. I could try installing protozero by hand, but if that's a build requirement, shouldn't that be documented in this repo?

[springmeyer]

@kueda thanks for the detailed report. I don't have time at the moment to dig into the details to comment on each cascade of the failure (but I do appreciate you providing those details to make that possible!). Overall nobody - that I'm aware - on the Mapnik crew has access to an M1 and therefore this problem exists. I'm faintly aware, however that there is a way to force an M1 to use the old architecture. Have you tried that yet (sorry if you did and I missed it). Or perhaps that is not viable for a specific reason?

[kueda]

Thanks for the quick reply, Dane. I hadn't looked into using Rosetta yet, I just assumed someone would have figured out how to do this on an M1 at this point. I'll try Rosetta now. However, I'm not yet convinced the M1 is to blame for the build errors, just the lack of binaries. Has anyone on the dev team tried building everything from source, i.e. not relying on any homebrew or mason binaries for any dependencies?

[springmeyer]

I personally used to build everything from source, but stopped ~2015 once mason got to a point that it was easier (and could also debug via isolated source builds using "mason build"). But mason development is stalled, so it's high time to find an alternative (or like you say, at least double-check source builds to something like /usr/local work). The last person that might be doing source builds is @artemp

[kueda]

Ok, managing separate arm64 and x86_64 dependencies is proving to be a bit beyond me (got as far convincing mason I was on x86 which got me some binary dependencies but not all). I'm hoping you all can figure out an arm64 build solution at some point, but for now I think I just need to work around this, or do work on stuff that depends on mapnik on an x86 machine.

[seeliang]

@kueda , same issue here

[ClayShentrup]

I am also here with same problem.

[hallahan]

Similar problem.

mapbox/mapbox-tile-copy#158

[ClayShentrup]

Overall nobody - that I'm aware - on the Mapnik crew has access to an M1 and therefore this problem exists. I'm faintly aware, however that there is a way to force an M1 to use the old architecture. Have you tried that yet (sorry if you did and I missed it). Or perhaps that is not viable for a specific reason?

Could we send you an M1 laptop? Could you ask if anyone would be open to getting Mapnik working on M1 in exchange for the laptop? It would be worth it for my employer.

[pleary]

I recently switched to an M1 arm64-based Mac and confronted this same issue. The way I've worked past it for now is to essentially install an x86-based version of node.js then install the node-mapnik package and that worked just fine. The more specific details are:

• I use nvm as my node version manager, and iTerm as my terminal
• To install an x86 version of node I made a copy of the iTerm application. I can right-click on the copy, choose Get Info and check the Open with Rosetta option (you may need to have installed Rosetta previously)
• Using the Rosetta-enabled copy of iTerm I installed my preferred version of node with nvm - this installs an x86 version of node
• From there I don't need to use the Rosetta-enabled version of iTerm any more and I can, using the normal arm64 iTerm, cd to my development directory, run nvm use then npm install and all packages including node-mapnik install just fine

I haven't looked into installing two different copies of the same version of node.js, so I'm working with the understanding that if I reference that same nvm-installed version of node in a different development directory, that will also use the same x86 version.

Not everyone will want to use this approach, but I'm happy with it and I prefer it over alternatives like using Docker or VMs or something else that does not feel like my normal workflow. That said I'll happily switch to an arm64 compatible version when that's possible

[zy6p]

Using Rosetta is indeed a solution in previous versions maintained by mason, but it can't help with the latest cmake version of mapnik. All those who want to experience the latest features of the source code compilation and installation still need to solve this problem.

[frewsxcv]

Potentially relevant mapnik issue: mapnik/mapnik#4313

[tpendragon]

Anyone got a nice workaround yet? Running into this issue now.

[tpendragon]

I did a very similar thing to @pleary, but with asdf

1. Find Terminal in Finder, right click, Get Info, check Open with Rosetta
2. Open Terminal, asdf uninstall nodejs
3. asdf install nodejs (my version was defined in a local .tool-versions)
4. yarn install then compiled mapnik.
5. Disable Open with Rosetta for Terminal.