npm_install requires external build tools for installing native modules

[guytt]

🐞 bug report

Affected Rule

The issue is caused by the rule: npm_install

Is this a regression?

Not that I have noticed.

Description

When I use npm_install on a package.json that includes, in its list of dependencies, a native node module, the run of the rule attempts to use build tools external to bazel (e.g. **make**) and fails when such build tools are not present in the linux environment where bazel is being run.

After installing the required tools (i.e. running apt-get install build-essential), running the rule succeeds. This violates bazel's proclaimed hermeticity to some extent.

🔬 Minimal Reproduction

Partial content of WORKSPACE file:

load("@build_bazel_rules_nodejs//:defs.bzl", "node_repositories", "npm_install")

node_repositories(
    package_json = ["//:package.json"],
    node_version = "10.13.0",
)

npm_install(
    name = "node_modules",
    package_json = "//:package.json",
)

Content of pacakge.json:

{
  "name": "mymodule",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "native-module": "^0.11.3"
  }
}

🔥 Exception or Error

gyp ERR! build error
gyp ERR! stack Error: not found: make
gyp ERR! stack     at getNotFoundError (/bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/nodejs/bin/nodejs/lib/node_modules/npm/node_modules/which/which.js:13:12)
gyp ERR! stack     at F (/bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/nodejs/bin/nodejs/lib/node_modules/npm/node_modules/which/which.js:68:19)
gyp ERR! stack     at E (/bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/nodejs/bin/nodejs/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack     at /bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/nodejs/bin/nodejs/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack     at /bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/nodejs/bin/nodejs/lib/node_modules/npm/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/nodejs/bin/nodejs/lib/node_modules/npm/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqWrap.oncomplete (fs.js:154:21)
gyp ERR! System Linux 4.9.125-linuxkit
gyp ERR! command "/bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/nodejs/bin/nodejs/bin/node" "/bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/nodejs/bin/nodejs/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /bazel-smb-share/outputs/9f616bba2eedecf392b1070e3c256f4e/external/cloud-host-worker-deps/node_modules/native-module
gyp ERR! node -v v10.13.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN @wix/wix-express-newrelic-parameters@1.0.481 requires a peer of newrelic@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @wix/cloud-host-worker@1.0.920 No license field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! native-module@0.11.3 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the native-module@0.11.3 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/bazel/.npm/_logs/2019-01-18T19_55_56_115Z-debug.log

🌍 Your Environment

Operating System:

  
4.9.125-linuxkit
  

Output of bazel version:

  
Build label: 0.20.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Nov 30 14:39:01 2018 (1543588741)
Build timestamp: 1543588741
Build timestamp as int: 1543588741
  

Rules version (SHA):

  
0.16.4 (d6fcfaf1380e81ace932b534c4ea99166008f0b6)
  

Encountered as part of using Bazel at Wix.com @ittaiz

[mgechev]

Do you hit this issue by running npm outside of Bazel? I suppose no?

I understand that this violates the hermeticity but it might be a reasonable compromise so that we do not couple rules_nodejs with the external tooling involved in building native node modules.

cc @alexeagle @gregmagolan

[ittaiz]

What do you suggest? That we have every developer install make and various other tools needed (we stumbled on to two more IIRC)? I’m not asking sarcastically (though am a bit worried). Have you encountered this in the past? What did you do? I’d be surprised if @alexeagle never faced this dilemma

…

On Fri, 18 Jan 2019 at 23:42 Minko Gechev ***@***.***> wrote: Do you hit this issue by running npm outside of Bazel? I suppose no? I understand that this violates the hermeticity but it might be a reasonable compromise so that we do not couple rules_nodejs with the external tooling involved in building native node modules. cc @alexeagle <https://github.com/alexeagle> @gregmagolan <https://github.com/gregmagolan> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#506 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF3b45WEA-dundMFR1tY4DQNIbNmNks5vEj-7gaJpZM4aIk_c> .

[guytt]

@mgechev
I hit this issue whenever running npm install in an environment where this external tooling is not installed.
As most installations of node/npm install this additional tooling, I don't hit the issue.

[ittaiz]

@gregmagolan can you update about the toolchain approach we discussed? Thanks!

[gregmagolan]

Yes. @ittaiz @alexeagle and I discussed this at the bazel rules summit last week. The thinking is that we should be able to resolve this with bazel toolchains. We're planning to migrate nodejs to bazel toolchains in the next month so we'll look at this at that point.

[rohansingh]

Doesn't this break determinism as well? The npm process for building native modules seems to break caching for me.

[ittaiz]

Greg, Any chance you have an update?

…

On Sun, 3 Mar 2019 at 5:57 Rohan Singh ***@***.***> wrote: Doesn't this break determinism as well? The npm process for building native modules isn't deterministic or hermetic, and breaks caching. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#506 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFyQj9zcaSOSASoUQG8iYYQGr3vRkks5vS0gNgaJpZM4aIk_c> .

[Toxicable]

Flyby comment: I think the long term solution for this one is getting projects that publish native code to instead publish WASM