Add ‘#[serial]’ tests (v2)

[shivjm]

(Fixed version of #42626; sorry for the noise.)

This PR adds a #[serial] attribute that marks tests that should be executed one-by-one rather than concurrently (per #33519).

I’ve confirmed that the libtest and tidy tests pass. I wasn’t able to run the full suite as a lot of the run_make tests seem to fail on my Windows system, but everything passed up to that point.

Rust (and PR!) newbie here, so please do let me know if I’m doing something poorly or incorrectly!

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[leonardo-m]

Perhaps a name like #[serial_test] is better?

[Mark-Simulacrum]

I think #[serial] is fine given that #[test] will also be there, but I could see an argument being made that serial is too generic a name. I do think this needs a feature gate though, which I believe hasn't yet been implemented. I could be wrong though.

[shivjm]

Ah, I didn’t understand how the feature gates work and the example I followed (#42219) is apparently a WIP. Looking at #42088, I need to gate this by:

1. Adding an entry to the declare_features block in feature_gate.rs with a feature name, a version, and an issue number:

(active, serial_tests, "1.19.0", Some(42684))

2. Specifying AttributeGate::Gated for the serial attribute in BUILTIN_ATTRIBUTES (same file), which needs a feature name, an explanation, and a function to check its configuration (auto-generated using cfg_fn):

("serial", Normal, Gated(Stability::Unstable, "serial_tests", "the `#[serial]` attribute is an experimental feature", cfg_fn!(serial_tests))

3. Adding a new document under unstable-book/src/language-features/ that points to this PR.
4. Adding an entry under unstable-book/src/SUMMARY.md that points to that document.

Is this correct? Have I missed anything? I’ll try to get it done pronto, but if not, it will be about a week.

Re: the name of the attribute itself, I can change that if serial is too generic.

[bjorn3]

Few minor things. Overal code looks good to me.

[bjorn3]

Please end the line with a comma, so there is less noise when new fields get added.

[bjorn3]

Same here

[bjorn3]

Same here

[bjorn3]

Same here

[bjorn3]

Please print the result too. Eg

panic!("result not okay: {:?}", result),

[bjorn3]

Same printing thing here.

[bors]

☔ The latest upstream changes (presumably #42664) made this pull request unmergeable. Please resolve the merge conflicts.

[Mark-Simulacrum]

@shivjm I believe this is waiting on some changes from you and a rebase. Let us know if we can help in any way! Feel free to ping me on IRC (simulacrum).

[shivjm]

Sorry, been out of town and then busy catching up! I’ve rebased the PR and (I hope) fixed the issues @bjorn3 pointed out (thanks!). Next up, need to implement the feature gate. I’ll ask for help on IRC.

[shivjm]

I’ve added the serial_tests feature gate. @est31, does this need an entry under unstable-book as well?

[est31]

does this need an entry under unstable-book as well?

The requirement for an entry has been removed. You can add one (with meaningful content) if you want to, if not, a stub will be generated automatically.

[bjorn3]

Lgtm when the trailing newline is added.

[bjorn3]

Please add a trailing newline.

[shivjm]

Done, thanks!

[brson]

Thanks for the patch.

This probably needs more discussion before landing. This is a thing that we've declined to do for many years.

[brson]

cc @nrc

[bors]

☔ The latest upstream changes (presumably #42964) made this pull request unmergeable. Please resolve the merge conflicts.

[shivjm]

Is there anything I can do to help?

[bors]

☔ The latest upstream changes (presumably #42727) made this pull request unmergeable. Please resolve the merge conflicts.

[nrc]

@brson do you have links to previous discussion or could you summarise here please? My opinion (uninformed by previous discussion) is that this is something I've wanted personally and I don't see why we should not support it (as long as it is feature gated etc.) being its own attribute feels kind of wrong (I'd prefer to attach it to #[test], but I suppose that #[should_panic] is a precedent?)

[nrc]

Discussed at the dev-tools meeting today, this problem can be solved today using a mutex (I filed #43155 to document that). Given that this solution feels somewhat ad hoc and there is workaround, we would prefer to punt on adding this in favour of custom test runners or some other long-term solution.

Thanks for the PR @shivjm and sorry to close it out after your work.

[shivjm]

Alright. Thanks for letting me know.

[yshui]

@nrc I don't see why this shouldn't be implemented in the test framework. The mutex work around seems to have some catches, and is non-trivial to get right.

[DevQps]

For everyone getting to this point: This solution is also implemented in a separate crate now: serial_test