3.1.5 breaks global install of dependents on Windows

[cwalther]

On Windows 10 (x86 and x64), installing a package that depends on sqlite3@3.1.5 (or something that resolves to 3.1.5, like ^3.1.4) globally (npm install -g) fails with error EBUSY: resource busy or locked, rmdir from npm install node-pre-gyp. The same thing works as expected with 3.1.4. It also works with 3.1.5 on OS X. Installing locally also works as expected.

I have not understood yet how exactly the error comes about, but it appears to be caused by 9a038b5.

Minimal reproducing example:

C:\Users\Walther\sqtest315>dir
 Volume in drive C has no label.
 Volume Serial Number is 7CE2-F96E

 Directory of C:\Users\Walther\sqtest315

05.10.2016  16:32    <DIR>          .
05.10.2016  16:32    <DIR>          ..
05.10.2016  16:28                85 package.json
               1 File(s)             85 bytes
               2 Dir(s)  26'892'546'048 bytes free

C:\Users\Walther\sqtest315>type package.json
{
"name": "sqtest315",
"version": "1.0.0",
"dependencies": {"sqlite3": "3.1.5"}
}
C:\Users\Walther\sqtest315>npm pack
sqtest315-1.0.0.tgz

C:\Users\Walther\sqtest315>npm install -g sqtest315-1.0.0.tgz

> sqlite3@3.1.5 preinstall C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773
> npm install node-pre-gyp

C:\Users\Walther\node
`-- node-pre-gyp@0.6.30
  +-- mkdirp@0.5.1
  | `-- minimist@0.0.8
  +-- nopt@3.0.6
  | `-- abbrev@1.0.9
  +-- npmlog@4.0.0
  | +-- are-we-there-yet@1.1.2
  | | `-- delegates@1.0.0
  | +-- console-control-strings@1.1.0
  | +-- gauge@2.6.0
  | | +-- aproba@1.0.4
  | | +-- has-color@0.1.7
  | | +-- has-unicode@2.0.1
  | | +-- object-assign@4.1.0
  | | +-- signal-exit@3.0.1
  | | +-- string-width@1.0.2
  | | | +-- code-point-at@1.0.1
  | | | | `-- number-is-nan@1.0.1
  | | | `-- is-fullwidth-code-point@1.0.0
  | | +-- strip-ansi@3.0.1
  | | | `-- ansi-regex@2.0.0
  | | `-- wide-align@1.1.0
  | `-- set-blocking@2.0.0
  +-- rc@1.1.6
  | +-- deep-extend@0.4.1
  | +-- ini@1.3.4
  | +-- minimist@1.2.0
  | `-- strip-json-comments@1.0.4
  +-- request@2.75.0
  | +-- aws-sign2@0.6.0
  | +-- aws4@1.4.1
  | +-- bl@1.1.2
  | | `-- readable-stream@2.0.6
  | +-- caseless@0.11.0
  | +-- combined-stream@1.0.5
  | | `-- delayed-stream@1.0.0
  | +-- extend@3.0.0
  | +-- forever-agent@0.6.1
  | +-- form-data@2.0.0
  | | `-- asynckit@0.4.0
  | +-- har-validator@2.0.6
  | | +-- chalk@1.1.3
  | | | +-- ansi-styles@2.2.1
  | | | +-- escape-string-regexp@1.0.5
  | | | +-- has-ansi@2.0.0
  | | | `-- supports-color@2.0.0
  | | +-- commander@2.9.0
  | | | `-- graceful-readlink@1.0.1
  | | +-- is-my-json-valid@2.15.0
  | | | +-- generate-function@2.0.0
  | | | +-- generate-object-property@1.2.0
  | | | | `-- is-property@1.0.2
  | | | +-- jsonpointer@4.0.0
  | | | `-- xtend@4.0.1
  | | `-- pinkie-promise@2.0.1
  | |   `-- pinkie@2.0.4
  | +-- hawk@3.1.3
  | | +-- boom@2.10.1
  | | +-- cryptiles@2.0.5
  | | +-- hoek@2.16.3
  | | `-- sntp@1.0.9
  | +-- http-signature@1.1.1
  | | +-- assert-plus@0.2.0
  | | +-- jsprim@1.3.1
  | | | +-- extsprintf@1.0.2
  | | | +-- json-schema@0.2.3
  | | | `-- verror@1.3.6
  | | `-- sshpk@1.10.1
  | |   +-- asn1@0.2.3
  | |   +-- assert-plus@1.0.0
  | |   +-- bcrypt-pbkdf@1.0.0
  | |   +-- dashdash@1.14.0
  | |   | `-- assert-plus@1.0.0
  | |   +-- ecc-jsbn@0.1.1
  | |   +-- getpass@0.1.6
  | |   | `-- assert-plus@1.0.0
  | |   +-- jodid25519@1.0.2
  | |   +-- jsbn@0.1.0
  | |   `-- tweetnacl@0.14.3
  | +-- is-typedarray@1.0.0
  | +-- isstream@0.1.2
  | +-- json-stringify-safe@5.0.1
  | +-- mime-types@2.1.12
  | | `-- mime-db@1.24.0
  | +-- node-uuid@1.4.7
  | +-- oauth-sign@0.8.2
  | +-- qs@6.2.1
  | +-- stringstream@0.0.5
  | +-- tough-cookie@2.3.1
  | `-- tunnel-agent@0.4.3
  +-- rimraf@2.5.4
  | `-- glob@7.1.0
  |   +-- fs.realpath@1.0.0
  |   +-- inflight@1.0.5
  |   +-- minimatch@3.0.3
  |   | `-- brace-expansion@1.1.6
  |   |   +-- balanced-match@0.4.2
  |   |   `-- concat-map@0.0.1
  |   +-- once@1.4.0
  |   `-- path-is-absolute@1.0.1
  +-- semver@5.3.0
  +-- tar@2.2.1
  | +-- block-stream@0.0.9
  | +-- fstream@1.0.10
  | | `-- graceful-fs@4.1.9
  | `-- inherits@2.0.3
  `-- tar-pack@3.1.4
    +-- debug@2.2.0
    | `-- ms@0.7.1
    +-- fstream-ignore@1.0.5
    +-- once@1.3.3
    | `-- wrappy@1.0.2
    +-- readable-stream@2.1.5
    | +-- buffer-shims@1.0.0
    | +-- core-util-is@1.0.2
    | +-- isarray@1.0.0
    | +-- process-nextick-args@1.0.7
    | +-- string_decoder@0.10.31
    | `-- util-deprecate@1.0.2
    `-- uid-number@0.0.6

npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Users\\Walther\\node\\node.exe" "C:\\Users\\Walther\\node\\node_modules\\npm\\bin\\npm-cli.js" "install" "node-pre-gyp"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.8
npm ERR! path C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773
npm ERR! code EBUSY
npm ERR! errno -4082
npm ERR! syscall rmdir

npm ERR! EBUSY: resource busy or locked, rmdir 'C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773\npm-debug.log
npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Users\\Walther\\node\\node.exe" "C:\\Users\\Walther\\node\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "sqtest315-1.0.0.tgz"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.8
npm ERR! code ELIFECYCLE

npm ERR! sqlite3@3.1.5 preinstall: `npm install node-pre-gyp`
npm ERR! Exit status 4294963214
npm ERR!
npm ERR! Failed at the sqlite3@3.1.5 preinstall script 'npm install node-pre-gyp'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the sqlite3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm install node-pre-gyp
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs sqlite3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls sqlite3
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Walther\sqtest315\npm-debug.log

C:\Users\Walther\sqtest315>

npm-debug.log.zip

[sam-github]

@springmeyer 9a038b5 is broken, you can't list node-pre-gyp as a dep, and install it in preinstall, I suspect what is happening is that sometimes npm is installing it at the same time as the preinstall script is trying.

The change was committed directly without a PR, so its hard to understand the justificatio.

I'm going to guess its because of the well known problem with npm 2.x that it does not guarantee a package's dependencies are installed before a package's install script runs, so its an attempt to work around this. The only robust fix is to bundle any deps that must be present at the time a package's install script runs. Which is what the code used to do... so why was it changed?

[springmeyer]

you can't list node-pre-gyp as a dep, and install it in preinstall, I suspect what is happening is that sometimes npm is installing it at the same time as the preinstall script is trying.

@sam-github can you point me to npm documentation on this restriction? I've tried and been unable to replicate this problem.

The change was committed directly without a PR, so its hard to understand the justificatio.

Bundling is a headache because security issues in the bundled dependencies cannot be fixed without re-tagging node-pre-gyp, and every single module that depends on it upgrading. Across modules depending on node-pre-gyp this was resulting in a lot of unnecessary time lost and confusion from users that don't know why vulnerable modules are being included (since they are way downstream of the module that includes node-pre-gyp).

Secondarily the size of the package is then large and users were complaining. But this is going to be this way once installed of course, so this is less of a problem.

Some background at mapbox/node-pre-gyp#162 and mapbox/node-pre-gyp#163

[springmeyer]

@sam-github any idea what .sl-blip.js is. I see npm ERR! strong-arc@1.8.8 preinstall: node .sl-blip.js in many of the error messages at strongloop/strong-arc#1733.

In the meantime I'm working on a v3.1.6 release that bundles again to avoid the breakages.

[rmg]

The failing node .sl-blip.js is another example of annoying install script behaviour I think.

One thing you can try w.r.t. install scripts in the future like you did with node-pre-gyp is to do npm install --no-global node-pre-gyp. This will prevent it from being hoisted and installed globally when sqlite3 is being pulled in by a global module, but won't have any effect when not being pulled in via a global install. This is required because all npm CLI flags get turned into env vars, and the propagate to nested npm processes that way.

[springmeyer]

npm install --no-global node-pre-gyp

@rmg ah, that sounds really smart. Do you think that could potentially avoid this Windows-specific EBUSY error as well, or are you recommending as a good idea in general?