From 392a2906f6d419de471a25706faa1c6072c3f06f Mon Sep 17 00:00:00 2001 From: Mara Bos Date: Tue, 15 Nov 2022 18:32:53 +0100 Subject: [PATCH 1/8] Add mixed utf8 literals rfc. --- text/0000-mixed-utf8-literals.md | 109 +++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 text/0000-mixed-utf8-literals.md diff --git a/text/0000-mixed-utf8-literals.md b/text/0000-mixed-utf8-literals.md new file mode 100644 index 00000000000..2e565e6b38c --- /dev/null +++ b/text/0000-mixed-utf8-literals.md @@ -0,0 +1,109 @@ +- Feature Name: `mixed_utf8_literals` +- Start Date: 2022-11-15 +- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) + +# Summary +[summary]: #summary + +Allow the exact same characters and escape codes in `"…"` and `b"…"` literals. + +That is: + +- Allow unicode characters, including `\u{…}` escape codes, in byte string literals. E.g. `b"hello\xff我叫\u{1F980}"` +- Allow `\x…` escape codes in regular string literals, as long as they are valid UTF-8. E.g. `"\xf0\x9f\xa6\x80"` + +# Motivation +[motivation]: #motivation + +Byte strings (`[u8]`) are a strict superset of regular (utf-8) strings (`str`), +but Rust's byte string literals are currently not a superset of regular string literals: +they reject non-ascii characters and `\u{…}` escape codes. + +``` +error: non-ASCII character in byte constant + --> src/main.rs:2:16 + | +2 | b"hello\xff你\u{597d}" + | ^^ byte constant must be ASCII + | + +error: unicode escape in byte string + --> src/main.rs:2:17 + | +2 | b"hello\xff你\u{597d}" + | ^^^^^^^^ unicode escape in byte string + | +``` + +This can be annoying when working with "conventionally UTF-8" strings, such as with the popular [`bstr` crate](https://docs.rs/bstr/latest/bstr/). +For example, right now, there is no convenient way to write a literal like `b"hello\xff你好"`. + +Allowing all characters and escape codes in both types of string literals reduces the complexity of the language. +We'd no longer have [different escape codes](https://doc.rust-lang.org/reference/tokens.html#characters-and-strings) +for different literal types. We'd only require regular string literals to be valid UTF-8. + +If we can postpone the UTF-8 validation until the point where tokens are turned into literals, then this not only simplifies the job of the tokenizer, +but allows macros to take string literals with invalid UTF-8 (through `$_:tt` or `TokenTree`). +That can be useful for macros like `cstr!("…")` and `wide!("…")`, etc., which currently unnecessarily result in errors for non-UTF-8 data: + +``` +error: out of range hex escape + --> src/main.rs:3:13 + | +3 | cstr!("¿\xff"); + | ^^^^ must be a character in the range [\x00-\x7f] +``` + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Regular string literals (`""`) must be valid UTF-8. For example, valid strings are `"abc"`, `"🦀"`, `"\u{1F980}"` and `"\xf0\x9f\xa6\x80"`. +`"\x80"` is not valid, however, as that is not valid UTF-8. + +Byte string literals (`b""`) may include non-ascii characters and unicode escape codes (`\u{…}`), which will be encoded as UTF-8. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +The tokenizer should accept all escape codes in both `""` and `b""` literals. +Only a regular string literal is checked for invalid UTF-8, but only at the point where the token is converted to a string literal AST node. + +Just like how `$_:tt` accepts a thousand-digit integer literal but `$_:literal` does not, +a `$_:tt` should accept `"\x80"`, but `$_:literal` should not. +Similar, proc macros should be able to consume invalid UTF-8 string literals as `TokenTree`. + +# Drawbacks +[drawbacks]: #drawbacks + +One might unintentionally write `\xf0` instead of `\u{f0}`. +However, for regular string literals that will result in an error in nearly all cases, since that's not valid UTF-8 by itself. + +# Alternatives +[alternatives]: #alternatives + +- Only extend `b""`, but still don't accept `\x` in regular string literals (`""`). + +- Stabilize `concat_bytes!()` and require writing `"hello\xff你好"` as `concat_bytes!(b"hello\xff", "你好")`. + (Assuming we extend the macro to accept a mix of byte string literals and regular string literals.) + +# Prior art +[prior-art]: #prior-art + +- C and C++ do the same. (Assuming UTF-8 character set.) +- [The `bstr` crate](https://docs.rs/bstr/latest/bstr/) +- Python and Javascript do it differently: `\xff` mean `\u{ff}`, because their strings behave like UTF-32 or UTF-16 rather than UTF-8. + (Also, Python's byte strings "accept" `\u` escape codes as just `'\\', 'u'`, without any warning or error.) + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- Should `concat!("\xf0\x9f", "\xa6\x80")` work? (The string literals are not valid UTF-8 individually, but are valid UTF-8 after being concatenated.) + + (I don't care. I guess we should do whatever is easiest to implement.) + +# Future possibilities +[future-possibilities]: #future-possibilities + +- Update the `concat!()` macro to accept `b""` strings and also not implicitly convert integers to strings, such that `concat!(b"", $x, b"\0")` becomes usable. + (This would need to happen over an edition.) From 4038665a7546b303ca62aa5dfee14e02bbbc2783 Mon Sep 17 00:00:00 2001 From: Mara Bos Date: Tue, 15 Nov 2022 18:48:04 +0100 Subject: [PATCH 2/8] Add RFC number. --- ...{0000-mixed-utf8-literals.md => 3349-mixed-utf8-literals.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename text/{0000-mixed-utf8-literals.md => 3349-mixed-utf8-literals.md} (98%) diff --git a/text/0000-mixed-utf8-literals.md b/text/3349-mixed-utf8-literals.md similarity index 98% rename from text/0000-mixed-utf8-literals.md rename to text/3349-mixed-utf8-literals.md index 2e565e6b38c..b02b1de7ee4 100644 --- a/text/0000-mixed-utf8-literals.md +++ b/text/3349-mixed-utf8-literals.md @@ -1,6 +1,6 @@ - Feature Name: `mixed_utf8_literals` - Start Date: 2022-11-15 -- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- RFC PR: [rust-lang/rfcs#3349](https://github.com/rust-lang/rfcs/pull/3349) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) # Summary From 3409a49b788759c54065d3134af3c7bb2393f02e Mon Sep 17 00:00:00 2001 From: Mara Bos Date: Thu, 17 Nov 2022 19:09:43 +0100 Subject: [PATCH 3/8] Update. --- text/3349-mixed-utf8-literals.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/text/3349-mixed-utf8-literals.md b/text/3349-mixed-utf8-literals.md index b02b1de7ee4..a46177e72d0 100644 --- a/text/3349-mixed-utf8-literals.md +++ b/text/3349-mixed-utf8-literals.md @@ -11,7 +11,7 @@ Allow the exact same characters and escape codes in `"…"` and `b"…"` literal That is: - Allow unicode characters, including `\u{…}` escape codes, in byte string literals. E.g. `b"hello\xff我叫\u{1F980}"` -- Allow `\x…` escape codes in regular string literals, as long as they are valid UTF-8. E.g. `"\xf0\x9f\xa6\x80"` +- Also allow non-ASCII `\x…` escape codes in regular string literals, as long as they are valid UTF-8. E.g. `"\xf0\x9f\xa6\x80"` # Motivation [motivation]: #motivation @@ -102,6 +102,17 @@ However, for regular string literals that will result in an error in nearly all (I don't care. I guess we should do whatever is easiest to implement.) +- How about single byte and character literals? + + - Should `b'\u{30}` work? (It's a unicode escape code, but it's still just one byte in UTF-8.) + + I think yes. I see no reason to disallow it. + + - Should `'\xf0\x9f\xa6\x80'` work? (It's multiple escape codes, but it's still just one character in UTF-8.) + + Probably not, since a `char` is not UTF-8 encoded; it's a single UTF-32 codepoint. + _Decoding_ UTF-8 from `\x` escape codes back into UTF-32 would be a bit surprising. + # Future possibilities [future-possibilities]: #future-possibilities From 679c165cbf65071557e5cef22e94b8eb882b354e Mon Sep 17 00:00:00 2001 From: Mara Bos Date: Thu, 17 Nov 2022 19:12:50 +0100 Subject: [PATCH 4/8] Updateee. --- text/3349-mixed-utf8-literals.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3349-mixed-utf8-literals.md b/text/3349-mixed-utf8-literals.md index a46177e72d0..7cd95235dda 100644 --- a/text/3349-mixed-utf8-literals.md +++ b/text/3349-mixed-utf8-literals.md @@ -82,7 +82,7 @@ However, for regular string literals that will result in an error in nearly all # Alternatives [alternatives]: #alternatives -- Only extend `b""`, but still don't accept `\x` in regular string literals (`""`). +- Only extend `b""` (that is, accept `b"🦀"`), but still do not accept non-ASCII `\x` in regular string literals (that is, keep rejecting `"\xf0\x9f\xa6\x80"`). - Stabilize `concat_bytes!()` and require writing `"hello\xff你好"` as `concat_bytes!(b"hello\xff", "你好")`. (Assuming we extend the macro to accept a mix of byte string literals and regular string literals.) From 3892774269d8dbb5c1e774a3f75333f56df7beeb Mon Sep 17 00:00:00 2001 From: Mara Bos Date: Wed, 30 Nov 2022 13:03:54 +0100 Subject: [PATCH 5/8] Move the "validate later" part to future possibilities. --- text/3349-mixed-utf8-literals.md | 36 +++++++++++++------------------- 1 file changed, 14 insertions(+), 22 deletions(-) diff --git a/text/3349-mixed-utf8-literals.md b/text/3349-mixed-utf8-literals.md index 7cd95235dda..a96e61cec29 100644 --- a/text/3349-mixed-utf8-literals.md +++ b/text/3349-mixed-utf8-literals.md @@ -39,26 +39,15 @@ error: unicode escape in byte string This can be annoying when working with "conventionally UTF-8" strings, such as with the popular [`bstr` crate](https://docs.rs/bstr/latest/bstr/). For example, right now, there is no convenient way to write a literal like `b"hello\xff你好"`. -Allowing all characters and escape codes in both types of string literals reduces the complexity of the language. +Allowing all characters and all known escape codes in both types of string literals reduces the complexity of the language. We'd no longer have [different escape codes](https://doc.rust-lang.org/reference/tokens.html#characters-and-strings) for different literal types. We'd only require regular string literals to be valid UTF-8. -If we can postpone the UTF-8 validation until the point where tokens are turned into literals, then this not only simplifies the job of the tokenizer, -but allows macros to take string literals with invalid UTF-8 (through `$_:tt` or `TokenTree`). -That can be useful for macros like `cstr!("…")` and `wide!("…")`, etc., which currently unnecessarily result in errors for non-UTF-8 data: - -``` -error: out of range hex escape - --> src/main.rs:3:13 - | -3 | cstr!("¿\xff"); - | ^^^^ must be a character in the range [\x00-\x7f] -``` - # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -Regular string literals (`""`) must be valid UTF-8. For example, valid strings are `"abc"`, `"🦀"`, `"\u{1F980}"` and `"\xf0\x9f\xa6\x80"`. +Regular string literals (`""`) must be valid UTF-8. +For example, valid strings are `"abc"`, `"🦀"`, `"\u{1F980}"` and `"\xf0\x9f\xa6\x80"`. `"\x80"` is not valid, however, as that is not valid UTF-8. Byte string literals (`b""`) may include non-ascii characters and unicode escape codes (`\u{…}`), which will be encoded as UTF-8. @@ -66,12 +55,8 @@ Byte string literals (`b""`) may include non-ascii characters and unicode escape # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -The tokenizer should accept all escape codes in both `""` and `b""` literals. -Only a regular string literal is checked for invalid UTF-8, but only at the point where the token is converted to a string literal AST node. - -Just like how `$_:tt` accepts a thousand-digit integer literal but `$_:literal` does not, -a `$_:tt` should accept `"\x80"`, but `$_:literal` should not. -Similar, proc macros should be able to consume invalid UTF-8 string literals as `TokenTree`. +The tokenizer should accept all known escape codes in both `""` and `b""` literals. +Only a regular string literal is checked to be valid UTF-8 afterwards. # Drawbacks [drawbacks]: #drawbacks @@ -92,8 +77,8 @@ However, for regular string literals that will result in an error in nearly all - C and C++ do the same. (Assuming UTF-8 character set.) - [The `bstr` crate](https://docs.rs/bstr/latest/bstr/) -- Python and Javascript do it differently: `\xff` mean `\u{ff}`, because their strings behave like UTF-32 or UTF-16 rather than UTF-8. - (Also, Python's byte strings "accept" `\u` escape codes as just `'\\', 'u'`, without any warning or error.) +- Python and Javascript do it differently: `\xff` means `\u{ff}`, because their strings behave like UTF-32 or UTF-16 rather than UTF-8. + (Also, Python's byte strings "accept" `\u` as just `'\\', 'u'`, without any warning or error.) # Unresolved questions [unresolved-questions]: #unresolved-questions @@ -113,8 +98,15 @@ However, for regular string literals that will result in an error in nearly all Probably not, since a `char` is not UTF-8 encoded; it's a single UTF-32 codepoint. _Decoding_ UTF-8 from `\x` escape codes back into UTF-32 would be a bit surprising. + (But note that `'\x41'` already works, for single byte UTF-8 characters, aka ASCII.) + # Future possibilities [future-possibilities]: #future-possibilities +- Postpone the UTF-8 validation to a later stage, such that macros can accept literals with invalid UTF-8. E.g. `cstr!("\xff")`. + + - If we do that, we could also decide to accept _all_ escape codes, even unknown ones, to allow things like `some_macro!("\a\b\c")`. + (The tokenizer would only need to know about `\"`.) + - Update the `concat!()` macro to accept `b""` strings and also not implicitly convert integers to strings, such that `concat!(b"", $x, b"\0")` becomes usable. (This would need to happen over an edition.) From 32c8d674432a8d3820457a835fbd373f9f1acbfe Mon Sep 17 00:00:00 2001 From: Mara Bos Date: Thu, 10 Aug 2023 11:23:48 +0200 Subject: [PATCH 6/8] Update RFC. --- text/3349-mixed-utf8-literals.md | 56 ++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 20 deletions(-) diff --git a/text/3349-mixed-utf8-literals.md b/text/3349-mixed-utf8-literals.md index a96e61cec29..2e2b121ba4a 100644 --- a/text/3349-mixed-utf8-literals.md +++ b/text/3349-mixed-utf8-literals.md @@ -6,9 +6,9 @@ # Summary [summary]: #summary -Allow the exact same characters and escape codes in `"…"` and `b"…"` literals. +Relax the restrictions on which escape codes are allowed in string, char, and byte literals. -That is: +Most importantly, this means we accept the exact same characters and escape codes in `"…"` and `b"…"` literals. That is: - Allow unicode characters, including `\u{…}` escape codes, in byte string literals. E.g. `b"hello\xff我叫\u{1F980}"` - Also allow non-ASCII `\x…` escape codes in regular string literals, as long as they are valid UTF-8. E.g. `"\xf0\x9f\xa6\x80"` @@ -46,17 +46,46 @@ for different literal types. We'd only require regular string literals to be val # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -Regular string literals (`""`) must be valid UTF-8. +Regular string literals (`""` and `r""`) must be valid UTF-8. For example, valid strings are `"abc"`, `"🦀"`, `"\u{1F980}"` and `"\xf0\x9f\xa6\x80"`. -`"\x80"` is not valid, however, as that is not valid UTF-8. +`"\xff"` is not valid, however, as that is not valid UTF-8. -Byte string literals (`b""`) may include non-ascii characters and unicode escape codes (`\u{…}`), which will be encoded as UTF-8. +Byte string literals (`b""` and `br""`) may include non-ascii characters and unicode escape codes (`\u{…}`), which will be encoded as UTF-8. + +The `char` type does not store UTF-8, so while `'\u{1F980}'` is valid, trying to encode it in UTF-8 as in `'\xf0\x9f\xa6\x80'` is not accepted. +In a char literal (`''`), `\x` may only be used for values 0 through 0x7F. + +Similarly, in a byte literal (`b''`), `\u` may only be used for values 0 through 0x7F, since those are the only code points that are unambiguously represented as a single byte. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -The tokenizer should accept all known escape codes in both `""` and `b""` literals. -Only a regular string literal is checked to be valid UTF-8 afterwards. +The ["characters and strings" section in the Rust Reference](https://doc.rust-lang.org/reference/tokens.html#characters-and-strings) +is updated with the following table: + +  | Example | Characters | Escapes | Validation +-- | -- | -- | -- | +Character | 'H' | All Unicode | ASCII, unicode | Valid unicode code point +String | "hello" | All Unicode | ASCII, high byte, unicode | Valid UTF-8 +Raw string | r#"hello"# | All Unicode | - | Valid UTF-8 +Byte | b'H' | All ASCII | ASCII, high byte | - +Byte string | b"hello" | All Unicode | ASCII, high byte, unicode | - +Raw byte string | br#"hello"# | All Unicode | - | - + +With the following definitions for the escape codes: + +- ASCII: `\'`, `\"`, `\n`, `\r`, `\t`, `\\`, `\0`, `\u{0}` through `\u{7F}`, `\x00` through `\x7F` +- Unicode: `\u{80}` and beyond. +- High byte: `\x80` through `\xFF` + +Compared to before, the tokenizer should start accepting: +- unicode characters in `b""` and `br""` literals (which will be encoded as UTF-8), +- all `\x` escapes in `""` literals, +- all `\u` escapes in `b""` literals (which will be encoded as UTF-8), and +- ASCII `\u` escapes in `b''` literals. + +Regular string literals (`""`) are checked to be valid UTF-8 afterwards. +(Either during tokenization, or at a later point in time. See future possibilities.) # Drawbacks [drawbacks]: #drawbacks @@ -87,19 +116,6 @@ However, for regular string literals that will result in an error in nearly all (I don't care. I guess we should do whatever is easiest to implement.) -- How about single byte and character literals? - - - Should `b'\u{30}` work? (It's a unicode escape code, but it's still just one byte in UTF-8.) - - I think yes. I see no reason to disallow it. - - - Should `'\xf0\x9f\xa6\x80'` work? (It's multiple escape codes, but it's still just one character in UTF-8.) - - Probably not, since a `char` is not UTF-8 encoded; it's a single UTF-32 codepoint. - _Decoding_ UTF-8 from `\x` escape codes back into UTF-32 would be a bit surprising. - - (But note that `'\x41'` already works, for single byte UTF-8 characters, aka ASCII.) - # Future possibilities [future-possibilities]: #future-possibilities From be6cbcbfd691a33861b621a8f499979bdc6b86d9 Mon Sep 17 00:00:00 2001 From: Mara Bos Date: Thu, 10 Aug 2023 11:24:55 +0200 Subject: [PATCH 7/8] Fix table. --- text/3349-mixed-utf8-literals.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/text/3349-mixed-utf8-literals.md b/text/3349-mixed-utf8-literals.md index 2e2b121ba4a..c1911d69d21 100644 --- a/text/3349-mixed-utf8-literals.md +++ b/text/3349-mixed-utf8-literals.md @@ -63,14 +63,14 @@ Similarly, in a byte literal (`b''`), `\u` may only be used for values 0 through The ["characters and strings" section in the Rust Reference](https://doc.rust-lang.org/reference/tokens.html#characters-and-strings) is updated with the following table: -  | Example | Characters | Escapes | Validation --- | -- | -- | -- | -Character | 'H' | All Unicode | ASCII, unicode | Valid unicode code point -String | "hello" | All Unicode | ASCII, high byte, unicode | Valid UTF-8 -Raw string | r#"hello"# | All Unicode | - | Valid UTF-8 -Byte | b'H' | All ASCII | ASCII, high byte | - -Byte string | b"hello" | All Unicode | ASCII, high byte, unicode | - -Raw byte string | br#"hello"# | All Unicode | - | - +|   | Example | Characters | Escapes | Validation | +|-----------------|-------------|-------------|---------------------------|--------------------------| +| Character | 'H' | All Unicode | ASCII, unicode | Valid unicode code point | +| String | "hello" | All Unicode | ASCII, high byte, unicode | Valid UTF-8 | +| Raw string | r#"hello"# | All Unicode | - | Valid UTF-8 | +| Byte | b'H' | All ASCII | ASCII, high byte | - | +| Byte string | b"hello" | All Unicode | ASCII, high byte, unicode | - | +| Raw byte string | br#"hello"# | All Unicode | - | - | With the following definitions for the escape codes: From bf9a91e01a000ca59c58b6e0c9042330a1f7fa56 Mon Sep 17 00:00:00 2001 From: Mara Bos Date: Thu, 10 Aug 2023 11:35:52 +0200 Subject: [PATCH 8/8] Update. --- text/3349-mixed-utf8-literals.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3349-mixed-utf8-literals.md b/text/3349-mixed-utf8-literals.md index c1911d69d21..9741863e970 100644 --- a/text/3349-mixed-utf8-literals.md +++ b/text/3349-mixed-utf8-literals.md @@ -6,7 +6,7 @@ # Summary [summary]: #summary -Relax the restrictions on which escape codes are allowed in string, char, and byte literals. +Relax the restrictions on which characters and escape codes are allowed in string, char, byte string, and byte literals. Most importantly, this means we accept the exact same characters and escape codes in `"…"` and `b"…"` literals. That is: