This repository includes:

• a detailed description of the Rust 1.91 lexer (in writeup)
• a Rust reimplementation of the lexer based on that description (in src/reimplementation)
• a manual list of testcases (in src/framework/testcases.rs)
• harnesses for running rustc's lexer in-process via rustc_private (in src/rustc_harness:
  • one comparing the reimplementation to the rustc_parse high-level lexer
  • one comparing the reimplementation to what declarative macros see
• strategies for comparing the reimplementation with rustc_parse's using proptest

See also the rendered description.

Running the tests

See Reimplementation command-line interface for what's available from the CLI:

Note the provided rust-toolchain.toml will cause cargo run to install the required nightly version of rustc.

Building the description

Install mdbook and mdbook-toc:

cargo install mdbook mdbook-toc

Then run

scripts/build_writeup

The output will appear in book/

License

The contents of this repository are distributed under the terms of both the MIT license and the Apache license (Version 2.0).

See LICENSE-APACHE and LICENSE-MIT for details.