Remove token::{Open,Close}Delim.

By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.

PR #137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.

This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
  pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
  `Delimiter` values to `TokenKind`.

Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
-   } else if self.token != token::CloseDelim(Delimiter::Brace) {
+   } else if self.token != token::CloseBrace {
```

Author: nnethercote

src/macros.rs

@@ -722,7 +722,7 @@ fn last_tok(tt: &TokenTree) -> Token {
     match *tt {
         TokenTree::Token(ref t, _) => t.clone(),
         TokenTree::Delimited(delim_span, _, delim, _) => Token {
-            kind: TokenKind::CloseDelim(delim),
+            kind: delim.as_open_token_kind(),
             span: delim_span.close,
         },
     }
@@ -1124,8 +1124,14 @@ fn next_space(tok: &TokenKind) -> SpaceState {
         TokenKind::PathSep
         | TokenKind::Pound
         | TokenKind::Dollar
-        | TokenKind::OpenDelim(_)
-        | TokenKind::CloseDelim(_) => SpaceState::Never,
+        | TokenKind::OpenParen
+        | TokenKind::CloseParen
+        | TokenKind::OpenBrace
+        | TokenKind::CloseBrace
+        | TokenKind::OpenBracket
+        | TokenKind::CloseBracket
+        | TokenKind::OpenInvisible(_)
+        | TokenKind::CloseInvisible(_) => SpaceState::Never,
 
         TokenKind::Literal(..) | TokenKind::Ident(..) | TokenKind::Lifetime(..) => {
             SpaceState::Ident

src/parse/macros/cfg_if.rs

@@ -1,7 +1,7 @@
 use std::panic::{AssertUnwindSafe, catch_unwind};
 
 use rustc_ast::ast;
-use rustc_ast::token::{Delimiter, TokenKind};
+use rustc_ast::token::TokenKind;
 use rustc_parse::exp;
 use rustc_parse::parser::ForceCollect;
 use rustc_span::symbol::kw;
@@ -60,9 +60,7 @@ fn parse_cfg_if_inner<'a>(
             return Err("Expected an opening brace");
         }
 
-        while parser.token != TokenKind::CloseDelim(Delimiter::Brace)
-            && parser.token.kind != TokenKind::Eof
-        {
+        while parser.token != TokenKind::CloseBrace && parser.token.kind != TokenKind::Eof {
             let item = match parser.parse_item(ForceCollect::No) {
                 Ok(Some(item_ptr)) => item_ptr.into_inner(),
                 Ok(None) => continue,