Unicode is big =/ Let's split it up...

[iarna]

This module is necessarily huge (108MB installed, for 6.3.0!), as it encompasses all of unicode. In practice though, I find that I only want a relatively small subset of it at any given time. I would propose making it build split into separate modules for each of the individually loadable pieces, such that a module could depend on unicode-6.3.0-categories-L-regex. From a usage point of view, this seems nearly exclusively a win. The one possible downside would be a huge proliferation of new modules that might, given the current state of npm search, make finding the right module for a new user even more difficult.

Would there be interest in a pull-request to this effect? I'd be happy to put that together if there was interest in using it.

[aredridel]

👍 This would be a huge win I think.

[mathiasbynens]

Out of curiosity, are you using these packages as part of build scripts or as run-time dependencies? I’ve only used them for build scripts (in my dev environment) so far, and in such cases the file size doesn’t really matter.

Making hundreds of packages for each Unicode version sounds like overkill IMHO. Would it help to just make separate packages for unicode-6.3.0-categories, unicode-6.3.0-scripts, unicode-6.3.0-blocks, unicode-6.3.0-bidi, unicode-6.3.0-bidi-mirroring, unicode-6.3.0-bidi-brackets?

(Btw, I would be interested to see how you’re using these packages!)

[mathiasbynens]

FWIW, the download size of the tarball for the unicode-6.3.0 module (http://registry.npmjs.org/unicode-6.3.0/-/unicode-6.3.0-0.1.0.tgz) is ~16.49 MB. Not too bad IMHO.

[iarna]

I'm currently using the module as a runtime dependency for a lexer. Specifically, for the category L regex.

Size-wise, it may compress to 16.49MB compressed, but unextracted it's 108MB– it's a substantial drag on build/install times, and it hugely clutters up end-user apps that have added their node_modules as part of their release process (eg http://www.futurealoof.com/posts/nodemodules-in-git.html et al).

And beyond that, it seems absurd to me to require 108MB of JS just because I need a 5KB regex.

[iarna]

Personally, I'd be ok with exposing only the regex interface this way, as its substantially faster than the other options, and I'd be hard pressed to name a use case where you'd want the others instead of it.

[mathiasbynens]

Interesting. I never intended the data packages to be used for anything other than build scripts.

Have you considered using regenerate in tandem with one of these packages, as devDependencies (rather than dependencies) as part of a build script (rather than at run time)? E.g. you could have a script named build-regex.js that does something like:

var regenerate = require('regenerate');
var L = require('unicode-6.3.0/categories/L/code-points'); // or `…/symbols`, doesn’t matter
console.log(regenerate(L).toString());
// Then, maybe write this to a file

This won’t solve all the problems you outlined but it would solve some. What do you think?

[mathiasbynens]

Personally, I'd be ok with exposing only the regex interface this way, as its substantially faster than the other options […]

Hmm, yeah, maybe we could move/copy the regular expressions to a separate unicode-6.3.0-regex package. The regex data indeed only takes up a small percentage of the package’s total file size.

and I'd be hard pressed to name a use case where you'd want the others instead of it.

FWIW, here’s a real-world example: https://github.com/ariya/esprima/blob/598d21bf26dfa3dd50b00b1a6c975d5612f4c8ab/tools/generate-identifier-regex.js

I could also imagine a situation where you want to construct a regex that matches all Unicode symbols in, say, the Arabic and Greek scripts.

var regenerate = require('regenerate');
var Arabic = require('unicode-6.3.0/scripts/Arabic/code-points'); // or `…/symbols`, doesn’t matter
var Greek = require('unicode-6.3.0/scripts/Greek/code-points'); // or `…/symbols`, doesn’t matter
var set = regenerate()
  .add(Arabic)
  .add(Greek);
console.log(set.toString());

// Then you might want to use a template like this to write the result to a file, along with any regex flags you might need:
// var regex = /<%= set.toString() %>/gim;

[iarna]

Eep, didn't mean to delete that comment. I wish issues were git repositories too. I wonder if I can recover it from email or RSS or suchlike...

[cscott]

@iarna I think i've got your deleted comment here in my email (thanks, github notifications). It was:

I don't think regenerate, as you outline it, would get anything me in particular– at least, not anything I wouldn't get just by having my build copy that one file out of unicode-6.3.0 directly.

(FWIW my actual use case: https://github.com/iarna/sql-lexer/blob/master/sql92/token-matcher-L0.js#L56)

As far as code points go, yes, I see what you mean– combining different classes together.

Out of curiosity, when would you actually use this as a part of a build script? Just generating use-case specific JS?

Given the current state of NPM– the inability to have submodules, I get not wanting to clutter things up. I'll experiment with how a regex only version would feel…

[iarna]

Thanks — wasn't in my email — guess you only get others updates in one's email.

[cscott]

And, responding to your earlier comment -- I would use this as part of a build script if I were implementing a spec which referenced a particular Unicode property. For example, the Java Language Specification says that valid identifier characters are members of a particular set of unicode classes. That set of characters doesn't change often, but it does change from time to time. I would probably use a build script to create the regex for "valid Java identifier character" based on the latest unicode data, and then just use that regex in my lexer.

[iarna]

Ok, so yes, just generating use-case specific javascript files.

So a regex only distribution would only be 1.4MB. And of course one can combine them, eg:

var Arabic = require('unicode-6.3.0/scripts/Arabic/regex');
var Greek = require('unicode-6.3.0/scripts/Greek/regex');
var set = new RegExp('['+Arabic.source+Greek.source+']');
console.log(set.toString());

[mathiasbynens]

There are a few problems with that approach:

• It’s prone to failure. RegExp#toString doesn’t always return a result that is fully equivalent to the source regex. Relevant V8 bug: https://code.google.com/p/v8/issues/detail?id=1982
• The output is not optimal. If in this case the regex for Arabic were to contain [\u1000-\u1FFF] (just an example) and Greek contained [\u2000-\u2FFF], you’d want the combined regex to just combine this into a single range, i.e. [\u1000-\u2FFF]. By using Regenerate instead, you ensure the resulting regular expression is as compact as possible (while remaining ASCII-only).

P.S. Your sample code has some other bugs – the output is wrapped in an extra set of [] and there should probably be a | between Arabic.source and Greek.source. All the more reason to avoid doing this manually, IMHO.

[mathiasbynens]

Perhaps we should remove the regular expressions from the packages altogether. They’re not intended to be toString()ed and concatenated etc. anyway (#14 (comment)). They’re ony useful when you need to match an existing Unicode category/script/block/… as-is at run-time, but in that case installing these big packages might be overkill as @iarna says.

[iarna]

Yeah, I blame early morning muzzyness for the character class mess in the example.

I don't think the overlap from:

var set = new RegExp(Arabic.source+'|'+Greek.source);

Is actually worth worrying about.

If .source is useless =( (although I don't see that– the bug shows that toString() is only useful for debug output, but it doesn't show the same for source) then having character ranges for use in character classes would provide a safer alternative– it might be superior too, as it'd allow concatenation in a single character class, which most regexp engines can optimize pretty well.