Restrict codepoints of valid identifiers

[stevengj]

As mentioned in #5434, separate from the question of what unicode normalization we should use for identifiers, it would probably be a good idea to restrict the codepoints of valid identifiers. Currently, you can do crazy things like:

julia> ² = 1
1
julia> 2²
2
julia> ２= 1
1
julia> 1 + ２
2
julia> –3 = 3
3
julia> -3 + –3
0

Python 3's valid identifiers provide one possible model.

[stevengj]

Another possible model would be the Fortress language specification, which is fairly detailed (see chapter 5, although it doesn't discuss normalization) and was unburdened by backwards compatibility (unlike Python).

[stevengj]

cc: @malmaud, @jiahao

[JeffBezanson]

It's starting to look like we need more and more of the ICU library. It would be great to rely on libc and call iswalpha, but (1) some libc implementations are quite far behind the unicode standard, and (2) for some reason this function is locale-dependent. I don't really see how whether a character is a letter should depend on locale...

[stevengj]

The utf8proc library will tell us the unicode category of a codepoint, in a locale-independent way.

[JeffBezanson]

Excellent.

[stevengj]

What character categories do we want to allow in identifiers?

Certainly we want Sm (symbol, math) to be allowed, unlike Python.

As another example, Python does not allow Po (punctuation, other) characters in identifiers. Currently, Julia does, so you can have e.g. x′ as an identifier using the prime character. Do we want to allow this?

[StefanKarpinski]

I really like using prime in variable names. However, we probably want to use other mathematical operators as, well, operators. So, I suspect we'll have to go through the math pages and decide on a case-by-case basis whether they should be allowed in identifiers or become operators (and how they should parse).