src: support UTF-8 in compiled-in JS source files

[bnoordhuis]

Detect it when source files in lib/ are not ASCII. Decode them as UTF-8
and store them as UTF-16 in the binary so they can be used as external
string resources without non-ASCII characters getting mangled.

CI: https://ci.nodejs.org/job/node-test-pull-request/6176/

[bnoordhuis]

I rather feared this approach may be a bit too forward-looking for some compilers... I'll see if I can dumb it down.

[aqrln]

Maybe 'utf-16be' if sys.byteorder == 'little' else 'utf-16le'?

[bnoordhuis]

The big-endian encoding is intentional, the line below stitches the individual octets together again into uint16s. (.encode() returns a byte string in case you're wondering.)

[aqrln]

Ah, I see. Right.

[jasnell]

Just curious, have there been specific issues or is this purely preventative?

[aqrln]

@jasnell yes, reported in #10673

[gibfahn]

What's the reason for using UTF-16 rather than UTF-8 (out of interest) to store the files in the binary?

[jasnell]

I would have to assume it is because the external string API does not directly support UTF8. That is, if you look at ExternalStringResource, it assumes a utf16_t* buffer while ExternalOneByteStringResource assumes const char*. If the strings were stored as UTF8 we would incur an additional cost at startup that is not necessary.

[aqrln]

@gibfahn so that they can be used as external strings. UTF-8 strings must be copied to V8's managed memory. See #5458 by @bnoordhuis.

[addaleax]

Tiny suggestion: Could we be a bit more explicit and rename P to Ptr or Pointer? 😄

[aqrln]

@bnoordhuis ping. Did you have time to dumb it down? 😉 This patch seems to work on GCC 4.8+.

Diff

diff --git a/src/node_javascript.cc b/src/node_javascript.cc
index 8c665381db..07a0daaa83 100644
--- a/src/node_javascript.cc
+++ b/src/node_javascript.cc
@@ -16,16 +16,16 @@ using v8::String;
 template <typename T, size_t N, T P>
 struct ExternalStringResource;
 
-template <size_t N, const char (&P)[N]>
-struct ExternalStringResource<const char[N], N, P>
+template <size_t N, const char* P>
+struct ExternalStringResource<const char*, N, P>
     : public String::ExternalOneByteStringResource {
   const char* data() const override { return P; }
   size_t length() const override { return N; }
   void Dispose() override { /* Default calls `delete this`. */ }
 };
 
-template <size_t N, const uint16_t (&P)[N]>
-struct ExternalStringResource<const uint16_t[N], N, P>
+template <size_t N, const uint16_t* P>
+struct ExternalStringResource<const uint16_t*, N, P>
     : public String::ExternalStringResource {
   const uint16_t* data() const override { return P; }
   size_t length() const override { return N; }
@@ -34,7 +34,7 @@ struct ExternalStringResource<const uint16_t[N], N, P>
 
 // id##_data is defined in node_natives.h.
 #define V(id) \
-  static ExternalStringResource<decltype(id##_data),                          \
+  static ExternalStringResource<decltype(&id##_data[0]),                      \
                                 arraysize(id##_data),                         \
                                 id##_data> id##_external_data;
 NODE_NATIVES_MAP(V)
diff --git a/tools/js2c.py b/tools/js2c.py
index e6c56cf075..8104e50e5a 100755
--- a/tools/js2c.py
+++ b/tools/js2c.py
@@ -189,7 +189,7 @@ NODE_NATIVES_MAP = """\
 SOURCES = """\
 static const uint8_t {escaped_id}_name[] = {{
 {name}}};
-static const {ctype} {escaped_id}_data[] = {{
+static constexpr const {ctype} {escaped_id}_data[] = {{
 {data}}};
 """
 

[bnoordhuis]

@aqrln Thanks. I tried a couple of different approaches but I ended up settling for something that's pretty similar to your patch. :-)

I added a second commit that replaces a few U+2019 quotes with ASCII quotes so that their files can be stored as one-byte strings. Can I get a quick LGTM or two?

EDIT: The CI scoreboard is currently linking to a previous CI build. The correct one is https://ci.nodejs.org/job/node-test-commit/7713/.

[richardlau]

I added a second commit that replaces a few U+2019 quotes with ASCII quotes so that their files can be stored as one-byte strings.

Probably one for another PR, but this sounds like a good candidate for a lint rule.

[bnoordhuis]

Interesting VS build failure...

an expression involving objects with internal linkage cannot be used as a non-type argument

I suppose the compiler is technically correct, and wasn't that the best kind of correct?

EDIT: New attempt: https://ci.nodejs.org/job/node-test-pull-request/6243/

[bnoordhuis]

Different take: moved all the logic to js2c.py.

CI: https://ci.nodejs.org/job/node-test-pull-request/6357/

EDIT: Sigh...

node_javascript.cc(55276): error C2466: cannot allocate an array of constant size 0

[bnoordhuis]

CI is finally green except for that silly issue where the ARM buildbots don't report their status properly.

@addaleax The PR changed quite a bit, perhaps you want to take another look?

[bnoordhuis]

You'd expect INTERMEDIATE_DIR to work here but it doesn't: js2c generates the file but the build doesn't pick it up (and I updated the 'sources' list in case you're wondering.) I decided to leave well enough alone for now.

[jasnell]

Needs a backport PR to land in v4.
Only one of the two commits was cherry-picked into v6 (the second one wasn't needed)