Options for disambiguating the \k backreferences

[littledan]

Backreferences to named capture groups have the surface syntax \k<foo>, but that already has semantics in non-Unicode RegExps. A few options have been discussed for the issue:

1. Named capture groups only usable in Unicode mode
   This was the original proposal, and it's what V8 implements behind a flag. This is one reason why we reserved extra sequences like \k to be a syntax error in Unicode mode--so we could add new features this way. It's the simplest option, and it would give people a carrot to upgrade to using the Unicode flag. Going from 1, we could "always" add 2 or 3 "later". @mathiasbynens and @hashseed have argued for this minimal option.
2. Named capture groups can be used outside of Unicode mode, but named backreferences are only with Unicode mode on
   There seemed to be some concern from the committee that this is something of an unexpected cliff in the middle of the feature. Another argument against it is that we shouldn't add new things to non-Unicode RegExps to encourage people to flip the flag on. This was my funny idea.
3. Disambiguate by making \k it have the new semantics if there are any named capture groups
   This is definitely possible, but more complicated than one might think at first. If there are no named capture groups, then \k can be anywhere, but otherwise, it needs to be followed by < IdentifierName >; this complicates the grammar. Another piece of complexity is that an implementation can't determine whether there are named capture groups on-line, if lookbehind is in play (because lookbehind semantics are executing the RegExp backwards, and this affects captured groups. For example: /(?<=\k<a>(?<a>.))/ matches a zero-length sequence which is preceded by the same character twice. It's definitely unambiguous, just complicated. This was @bakkot's suggestion.

At the September TC39 meeting, we seemed to come to consensus on 3; however, this was without incorporating some feedback from people not present in the room, and without a full understanding of the complexity of 3. With the complexity of 3, and the weird cliff of 2, I'm personally leaning back towards 1. OTOH, 3 feels the most "1JS-y" to me. Any thoughts?

[mathiasbynens]

IMHO the u flag is to regular expressions as strict mode is to all of JavaScript — it should be considered the new default. Any new features should be made available to Unicode mode only. Improving non-Unicode mode is not worth even the smallest amount of complexity.

[hashseed]

I'm a bit torn on this. I also have the sentiment that /u is the strict mode of regexp. But then again, you wouldn't require /u for new features such as lookbehind assertions, right? \k outside of /u is unambiguous, so that argument is void.

[littledan]

@mathiasbynens We made the opposite decision with strict mode. Classes, let, destructuring, etc are all available in sloppy mode, based on @dherman 's 1JS policy, at the cost of a lot of design and implementation complexity, but with the benefit that users don't have to think as much about which mode they're in, and it's easier to incrementally adopt new features in code bases that might be difficult to switch to strict mode.

[littledan]

Waldemar Horwat pointed out to me that we could implement @bakkot's disambiguation strategy (#3 above) in the spec by reparsing from the beginning if a named group is encountered when not in named group mode. In a conversation with @schuay @hashseed and @ErikCorryGoogle, it was noted that V8's RegExp engine does this internally in some cases already, so it seems like we could add another bail-out and reparse-with-a-flag case if a \k preceding a named group.

Although I changed the explainer and spec text to be Unicode-only, I'm now having second thoughts since supporting non-Unicode mode seems pretty doable both on the spec and implement. Thoughts?

[bakkot]

@littledan I think you could even do a bail-out and reparse-with-a-flag if you encountered a \k not immediately followed by <...>, or a \k<...> not preceded or followed by a (?<...>). I expect this almost never happens in practice, so I'd hope it would be a negligible performance hit.

Personally I like my proposal (😄) and I think it's the one most likely to satisfy the committee, if engines feel it's implementable.

[littledan]

Any opinions from other engines? cc @jswalden @syg @bterlson @msaboff @kmiller68

[jswalden]

My knowledge of new (-er than ES5!) regular expression syntax/semantics/flags is minimal for lack of time to follow it (and, somewhat, comparative disinterest). So whether you should listen to this at all, I dunno. And I'm probably unlikely to care too strongly if anyone or the spec thinks differently from me. :-)

If we step past that, I mildly tend toward option 1 for simplicity. Option 2 just seems an awkward splitting of the baby to me: a half-working feature, sometimes. Option 3 is better than 2. But reparsing things is always inelegant, and it's IMO much easier to explain that the functionality just isn't available in non-Unicode regular expressions (versus some more complicated explanation that it's available only sometimes, depending on what other functionality is used -- pattern syntax is hoary enough, and we want to introduce even more complexity?).

The one demerit to option 1 is that you can't just tack a u onto your regular expression to use captures, because u changes other stuff. I'd bet lots of regular expression use parses presumed-ASCII sorts of text -- programmatic grammars, say. If someone tacks a u onto a regex that uses \w to use named captures, (say) an HTTP parser could match more things than intended. This feels like not enough hazard to preclude option 1, but it's not a slam-dunk.

One final overriding consideration: given that we just embed a minimally-modified fork of V8's irregexp, it's doubtful that any option really has different implementation burden for us. Considering only implementation effort, SpiderMonkey probably doesn't care what's decided.

[littledan]

@jswalden, when you say simplicity and minimalism, do you mean simplicity for users, spec authors, or implementers? It's hard for me to see how 3 creates complexity for users, though it definitely creates additional complexity for spec authors and implementers.

[mathiasbynens]

With option 3, what happens if \k is used without any named capture groups in combination with the u flag? /\k/u throws, currently — IMHO it would be confusing if this proposal made it not throw anymore despite that code sample not really using any new functionality.

[littledan]

In the current spec text, it would continue to throw a syntax error, as the \k is expected to have a group named with it for a Unicode RegExp. Also, /\k<foo>/u throws, as foo has no GroupSpecifier for it.

[bterlson]

IMO:

2 is no good.

1 and 3, hard to say without seeing how complex things get, but I did hear some argue in TC39 that there is strong use cases for non-Unicode regexps. I don't necessarily agree, but if it's true, option 3 seems better than option 1. If /u is a regexp strict mode then option 1 is better.