Give tip for unicode chars which might not be visible when rendered

[chenyukang]

Fixes #100388

[rust-highfive]

r? @petrochenkov

(rust-highfive has picked a reviewer for you, use r? to override)

[compiler-errors]

I think this needs a different heuristic for "is not visible when rendered", since this triggers on every non-ascii char.

Also, why are you checking err.suggestions here?

[chenyukang]

I think this needs a different heuristic for "is not visible when rendered", since this triggers on every non-ascii char.

Also, why are you checking err.suggestions here?

Yes, I'm still thinking any better method for check a unicode...

I'm checking the err.suggestions here, because we already have a strategy to give suggestions for those possible substitution,

rust/compiler/rustc_parse/src/lexer/mod.rs

Line 322 in d76952f

let token = unicode_chars::check_for_substitution(self, start, c, &mut err);

.

My idea is if we already have such kind of suggestion, we won't add more, since it's clear enough:

 --> /home/cat/code/rust/src/test/ui/parser/unicode-quote-chars.rs:2:14
  |
2 |     println!(“hello world”);
  |              ^
  |
= help: Unicode character \u{323} might not be visible when rendered
help: Unicode characters '“' (Left Double Quotation Mark) and '”' (Right Double Quotation Mark) look like '"' (Quotation Mark), but are not

I think = help: Unicode character \u{323} might not be visible when rendered here will be redundant.

[compiler-errors]

I'm checking the err.suggestions here, because we already have a strategy to give suggestions for those possible substitution,

I think if you change your PR to only note characters that have a rendered width of zero (i.e. combining characters, etc) then this check is redundant.

[chenyukang]

I haven't found a better way to figure out "not visible when rendered" chars.
We have a test case in

rust/src/test/ui/issues/issue-29227.rs

Line 125 in b998821

fn is_es_whitespace(self) -> bool {

The unicode_space_separator, unicode_space_separator and combining_spacing_mark categories seem contain the chars which are hard to spot, but those are also not totaly 'not visible`, and I think introduce those big tables into compiler code base maybe too heavy ..

[petrochenkov]

@rustbot author
(to address compiler-errors' comments)

[rustbot]

Error: Parsing shortcut command in comment failed: ...'bot author' | error: expected end of command at >| ' (to addre'...

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

[CAD97]

You'll need to use the unicode table generator to generate a table of the characters with General Category=Nonspacing Mark and/or Bidi_Class=Nonspacing_Mark for which to warn on. Adding the table to rustc is probably fine, adding it to std is nondesirable.

The UI test is unrelated.

[bors]

☔ The latest upstream changes (presumably #102302) made this pull request unmergeable. Please resolve the merge conflicts.

[JohnCSimon]

ping from triage:
@chenyukang

Can you please address the merge conflicts - and post your status on this PR?

FYI: when a PR is ready for review, send a message containing
@rustbot ready to switch to S-waiting-on-review so the PR is in the reviewer's backlog.