rustdoc: Test & document `test_harness` code block attribute by fmease · Pull Request #148183 · rust-lang/rust

fmease · 2025-10-27T17:58:45Z

They were fully untested and undocumented previously despite being stable.
Context: #t-rustdoc > `test_harness` langstr @ 💬.

While at it, I've also improved the documentation for code block attributes in general.
I was only inspired to do so because there was no good place for test_harness and because it got hard to skim due to a lack of subsubsections.

rustbot · 2025-10-27T17:58:49Z

r? @GuillaumeGomez

rustbot has assigned @GuillaumeGomez.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

fmease · 2025-10-27T18:02:07Z

@@ -847,11 +847,12 @@ pub(crate) enum Ignore {
    Some(Vec<String>),
 }



It's not any old ENBF¹ (variant), it's ABNF² specifically (from IETF RFC 5234) which I previously didn't know about (and thus was confused about the syntax).

Footnotes

Extended Backus-Naur form. ↩

Augmented Backus-Naur form. ↩

fmease · 2025-10-27T18:03:07Z

 /// delimited-attribute-list = OPEN-CURLY-BRACKET attribute-list CLOSE-CURLY-BRACKET
 /// token-list = [sep] token *(sep token) [sep]
-/// comment = OPEN_PAREN *(all characters) CLOSE_PAREN
+/// comment = OPEN_PAREN *<all characters except closing parentheses> CLOSE_PAREN


Prose terminals are to be denoted by <…> not by (…) (which is grouping) per IETF RFC 5234 (Ctrl+F for prose-val).

GuillaumeGomez · 2025-10-30T16:24:45Z

-Code blocks can be annotated with attributes that help `rustdoc` do the right
-thing when testing your code:
+Code blocks can be annotated with attributes that tell `rustdoc` how to build and interpret the test.
+They follow the triple backtick in the opening line.


Not necessarily triple, it can be ten if you want. ^^'

So I'd say (or something along the lines):

Suggested change

They follow the triple backtick in the opening line.

They follow the backticks marking the beginning of the code block.

Yea it can also be ~~~ ^^'. I've used triple backtick because that's what's also used at the top of the file. We do link to the CommonMark spec later for the full rules.

But yeah, I should probably rephrase that.

I keep forgetting about ~~~... Markdown is too flexible 😅

Replaced with code fence + a link to the relevant part of the CommonMark spec.

GuillaumeGomez · 2025-10-30T16:26:12Z

+They follow the triple backtick in the opening line.
+As such, they share the place with language strings like `rust` or `text`.
+Multiple attributes can be provided by separating them with commas, spaces or tabs.
+You can also write comments which are enclosed in parentheses `(…)`.


Maybe link to the relevant part of the book which explains the syntax of codeblock attributes instead?

This is the part of the book about codeblock attributes. We didn't have any documentation in the Book about this before.

lolbinarycat · 2025-11-06T16:55:46Z

A quick github code search reveals this is in fact used in the wild.

rustbot · 2026-06-05T13:01:56Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

fmease · 2026-06-05T13:02:18Z

@rustbot review

rust-bors · 2026-06-05T19:18:34Z

This pull request was unapproved.

This PR was contained in a rollup (#157502), which was unapproved.

View changes since this unapproval

fmease · 2026-06-05T21:25:07Z

This is so sad, there's no official (cross-platform) way to pass libtest arguments to the inner test harness, that's pretty sad but kinda obvious in hindsight; --test-threads=1 isn't forwarded which makes sense meaning that one test case has non-deterministic output (duh).

I'm crying, do I need to abuse runtools now?

GuillaumeGomez · 2026-06-05T21:27:17Z

I won't judge if you do. :3

fmease · 2026-06-08T14:11:58Z

@bors try jobs=aarch64-gnu-llvm-21-1

…s, r=<try> rustdoc: Test & document `test_harness` code block attribute try-job: aarch64-gnu-llvm-21-1

rust-bors · 2026-06-08T15:09:54Z

💔 Test for fa2c91f failed: CI. Failed job:

try - aarch64-gnu-llvm-21-1 (web logs, enhanced plaintext logs)

fmease · 2026-06-08T16:32:00Z

@bors try jobs=x86_64-msvc-1,aarch64-gnu-llvm-21-1

…s, r=<try> rustdoc: Test & document `test_harness` code block attribute try-job: x86_64-msvc-1 try-job: aarch64-gnu-llvm-21-1

rust-bors · 2026-06-08T18:58:38Z

💔 Test for 2fe156a failed: CI. Failed job:

try - x86_64-msvc-1 (web logs, enhanced plaintext logs)

Moreover, flesh out the rustdoc book section about code block attributes.

fmease · 2026-06-08T19:47:20Z

@bors try jobs=x86_64-msvc-1

…s, r=<try> rustdoc: Test & document `test_harness` code block attribute try-job: x86_64-msvc-1

fmease · 2026-06-08T20:42:49Z

Final try-builds still needs to come back clean of course but this is now ready for re-review. I've basically split out the problematic part out of the rustdoc-UI test into a new run-make test.

rust-bors · 2026-06-08T22:17:23Z

💔 Test for acf5f03 failed: CI. Failed job:

try - x86_64-msvc-1 (web logs, enhanced plaintext logs)

rust-log-analyzer · 2026-06-08T22:17:26Z

The job x86_64-msvc-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

---- [run-make] tests\run-make\doctests-test_harness stdout ----

error: rmake recipe failed to complete
status: exit code: 101
command: "D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\test\\run-make\\doctests-test_harness\\rmake.exe"
stdout: none
--- stderr -------------------------------
"D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2\\bin\\rustc.exe" "-L" "D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\test\\run-make\\doctests-test_harness\\rmake_out" "doctests.rs" "--crate-type" "lib" "--target=x86_64-pc-windows-msvc"
output status: `exit code: 0`
=== STDOUT ===



=== STDERR ===



"D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2\\bin\\rustc.exe" "-L" "D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\test\\run-make\\doctests-test_harness\\rmake_out" "runtool.rs" "--target=x86_64-pc-windows-msvc"
output status: `exit code: 0`
=== STDOUT ===



=== STDERR ===



"D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2\\bin\\rustdoc.exe" "doctests.rs" "--test" "--test-args=--test-threads=1" "--test-runtool" ".\\runtool..exe" "-L." "--target=x86_64-pc-windows-msvc"
output status: `exit code: 101`
=== STDOUT ===

running 2 tests
test doctests.rs - (line 15) ... FAILED
test doctests.rs - (line 4) ... FAILED

---
@@ -6,55 +6,9 @@
 failures:
 
 ---- doctests.rs - (line 15) stdout ----
-Test executable failed ($STATUS).
-
-stdout:
-
-running 2 tests
-test bad ... FAILED
-test good ... ok
-
-failures:
-
----- bad stdout ----
-
-thread 'bad' ($TID) panicked at doctests.rs:4:5:
-assertion failed: false
-note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
-
-
-failures:
---
-
-
+Couldn't run the test: The system cannot find the file specified. (os error 2)
 ---- doctests.rs - (line 4) stdout ----
-Test executable failed ($STATUS).
-
-stdout:
-
-running 2 tests
-test ill ... FAILED
-test well ... ok
-
-failures:
-
----- ill stdout ----
-
-thread 'ill' ($TID) panicked at doctests.rs:9:6:
-assertion failed: false
-note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
-
-
-failures:
---
   0: std::panicking::panic_handler
             at /rustc/0417c25868d6dfbd1c291dfeae950504faa6f790/library\std\src\panicking.rs:689
   1: core::panicking::panic_fmt
             at /rustc/0417c25868d6dfbd1c291dfeae950504faa6f790/library\core\src\panicking.rs:80
   2: <run_make_support::diff::Diff>::run
   3: rmake::main
   4: rmake::main
   5: std::rt::lang_start::<()>
   6: std::rt::lang_start::<()>
   7: std::rt::lang_start_internal::closure$0
             at /rustc/0417c25868d6dfbd1c291dfeae950504faa6f790/library\std\src\rt.rs:175
   8: std::panicking::catch_unwind::do_call
---

Some tests failed in compiletest suite=run-make mode=run-make host=x86_64-pc-windows-msvc target=x86_64-pc-windows-msvc
Bootstrap failed while executing `test --stage 2 --skip=compiler --skip=src`
Build completed unsuccessfully in 2:22:09
make: *** [Makefile:115: ci-msvc-py] Error 1
  local time: Mon Jun  8 22:17:09 CUT 2026
  network time: Mon, 08 Jun 2026 22:17:09 GMT
##[error]Process completed with exit code 2.
##[group]Run echo "disk usage:"
echo "disk usage:"

rustbot assigned GuillaumeGomez Oct 27, 2025

fmease commented Oct 27, 2025

View reviewed changes

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from eab1d61 to 5e7f083 Compare October 27, 2025 18:05

This comment has been minimized.

Sign in to view

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from 5e7f083 to c687f89 Compare October 27, 2025 18:53

This comment has been minimized.

Sign in to view

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from c687f89 to f71d897 Compare October 27, 2025 19:41

This comment has been minimized.

Sign in to view

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from f71d897 to 9a78729 Compare October 27, 2025 20:24

This comment has been minimized.

Sign in to view

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from 9a78729 to 3a17eba Compare October 27, 2025 23:40

GuillaumeGomez reviewed Oct 30, 2025

View reviewed changes

Comment thread src/doc/rustdoc/src/write-documentation/documentation-tests.md Outdated

fmease mentioned this pull request Nov 21, 2025

cfg(test) is not set within the test code while compiling doctests #148942

Open

epage mentioned this pull request Jan 23, 2026

test_attr_in_doctest emits when the test_harness attribute is present rust-lang/rust-clippy#16447

Closed

fmease added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 1, 2026

fmease mentioned this pull request Jun 5, 2026

rustdoc: Emit a lint warning if doctest contains #[test] functions but isn't marked test_harness #157458

Open

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from 3a17eba to f13978f Compare June 5, 2026 13:01

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 5, 2026

GuillaumeGomez approved these changes Jun 5, 2026

View reviewed changes

rust-bors Bot removed the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jun 5, 2026

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from f13978f to 5706bfe Compare June 8, 2026 14:11

rustbot added the A-run-make Area: port run-make Makefiles to rmake.rs label Jun 8, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 8, 2026

Auto merge of #148183 - fmease:rustdoc-test-n-doc-doctest-test-harnes…

fa2c91f

…s, r=<try> rustdoc: Test & document `test_harness` code block attribute try-job: aarch64-gnu-llvm-21-1

This comment has been minimized.

Sign in to view

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from 5706bfe to 4f88c4d Compare June 8, 2026 16:31

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 8, 2026

Auto merge of #148183 - fmease:rustdoc-test-n-doc-doctest-test-harnes…

2fe156a

…s, r=<try> rustdoc: Test & document `test_harness` code block attribute try-job: x86_64-msvc-1 try-job: aarch64-gnu-llvm-21-1

This comment has been minimized.

Sign in to view

fmease force-pushed the rustdoc-test-n-doc-doctest-test-harness branch from 4f88c4d to a918066 Compare June 8, 2026 19:46

rustdoc: Test & document test_harness code block attribute

a918066

Moreover, flesh out the rustdoc book section about code block attributes.

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 8, 2026

Auto merge of #148183 - fmease:rustdoc-test-n-doc-doctest-test-harnes…

acf5f03

…s, r=<try> rustdoc: Test & document `test_harness` code block attribute try-job: x86_64-msvc-1

fmease added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 8, 2026

rust-bors Bot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 8, 2026

		@@ -847,11 +847,12 @@ pub(crate) enum Ignore {
		Some(Vec<String>),
		}

	They follow the triple backtick in the opening line.
	They follow the backticks marking the beginning of the code block.

Uh oh!

Conversation

fmease commented Oct 27, 2025 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Oct 27, 2025

Uh oh!

fmease Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

fmease Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

GuillaumeGomez Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeGomez Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

fmease Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeGomez Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

fmease Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GuillaumeGomez Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

fmease Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lolbinarycat commented Nov 6, 2025

Uh oh!

rustbot commented Jun 5, 2026

Uh oh!

fmease commented Jun 5, 2026

Uh oh!

rust-bors Bot commented Jun 5, 2026

Uh oh!

fmease commented Jun 5, 2026

Uh oh!

GuillaumeGomez commented Jun 5, 2026

Uh oh!

fmease commented Jun 8, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Jun 8, 2026

Uh oh!

This comment has been minimized.

fmease commented Jun 8, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Jun 8, 2026

Uh oh!

This comment has been minimized.

fmease commented Jun 8, 2026

Uh oh!

This comment has been minimized.

fmease commented Jun 8, 2026

Uh oh!

rust-bors Bot commented Jun 8, 2026

Uh oh!

rust-log-analyzer commented Jun 8, 2026

Uh oh!

Reviewers

Assignees

fmease commented Oct 27, 2025 •

edited by rustbot

Loading

fmease Oct 27, 2025 •

edited

Loading

fmease Oct 27, 2025 •

edited

Loading

fmease Jun 5, 2026 •

edited

Loading

fmease Nov 14, 2025 •

edited

Loading