Allow users to pass in their own compression-stream function

[nickdesaulniers]

It looks like FF now accepts brotli compression. I was going to take a stab at implementing a native addon that links libbrotli. Does that seem worthwhile?

We could use this: https://www.npmjs.com/package/brotli, which uses an emscriptenized version of brotli.

Does anyone know of any others brotli modules that could be used? Otherwise, I don't mind starting the C++ add on.

I've been taking a look at how the gzip stream is used here, so I think I can make a nice library that implements the same interface.

[dougwilson]

Cool. This library cannot take a dependency on a native addon, due to the situation of native addons and Windows, making it really hard for people to use.

But, I do want to increase the flexibility of this module such that users can define their own encodings, like brotli.

[dougwilson]

Huh, it looks like that module you referenced, brotli, is actually a pure-JS module :)

[nickdesaulniers]

due to the situation of native addons and Windows

can you elaborate on this?

[dougwilson]

When you install Node.js, running npm install native_module will fail because there is no tool chains to compile anything on Windows. As I exclusively develop on Windows for these modules, if I'm not even able to install a module out-of-the-box, then I can't even develop against it, let alone all the other users.

This has been a major issue for years & years and no real progress towards addressing. For example, a standard Node.js install on my machine, when I try to install your nanomsg module, here is the error:

$ npm install nanomsg
|
> nanomsg@2.1.4 install c:\Users\doug.wilson\test2\node_modules\nanomsg
> node-gyp rebuild


c:\Users\doug.wilson\test2\node_modules\nanomsg>node "c:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild
gyp ERR! configure error
gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack     at failNoPython (c:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:103:14)
gyp ERR! stack     at c:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:64:11
gyp ERR! stack     at Object.oncomplete (evalmachine.<anonymous>:107:15)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "c:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd c:\Users\doug.wilson\test2\node_modules\nanomsg
gyp ERR! node -v v0.10.33
gyp ERR! node-gyp -v v1.0.1
gyp ERR! not ok

npm ERR! nanomsg@2.1.4 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the nanomsg@2.1.4 install script.
npm ERR! This is most likely a problem with the nanomsg package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls nanomsg
npm ERR! There is likely additional logging output above.
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "c:\\Program Files\\nodejs\\node.exe" "c:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "nanomsg"
npm ERR! cwd c:\Users\doug.wilson\test2
npm ERR! node -v v0.10.33
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

Just installing Python onto Windows (not a simple task) is only the start. Users then have to figure out how to get the correct version of Visual Studio installed, get the proper environment variables setup so gyp works, etc. It's pretty much an impossible task for all people starting on Node.js on Windows.

[nickdesaulniers]

Isn't this also the case with OSX as command line tools aren't installed by default?

[dougwilson]

I'm not familiar with Mac OS X, but I can say that when I had a module that has a native dependency, I only got endless Windows users making issue reports, but no Mac OS X or Linux users, so there is some large disconnect between those two populations of developers.

You can even find dozens of issues in the Node.js repository (like nodejs/node-v0.x-archive#8005) expressing the issue of how hard it is for typical Windows developers to use native modules. I don't see any of that in the Node.js repo from Mac OS X users.

[dougwilson]

P.S. as far as I can tell, that brotli module doesn't implement a streaming interface, which is kinda a downer :(

[dougwilson]

It looks like there is a native addon https://www.npmjs.com/package/iltorb that provides brotli and even does so as a stream. This means we have here the following wants:

1. Implement a way for users to define their own encodings in the module, which would let a user use a module like https://www.npmjs.com/package/iltorb
2. If someone is willing to either get brotli into Node.js core itself or make a pure-JS brotli implementation that provides a stream interface, we can add it as a dependency for brotli out-of-the-box.