Add recycle_on_failure and retries options to runtests

[michel2323]

Two opt-in failure-handling options for runtests, motivated by running oneAPI.jl's test suite on a memory-constrained GPU (8GB Arc A750, --jobs=2), where a baseline run failed 520 tests — 366 of them OutOfGPUMemoryError — cascading across ~20 test files that all pass individually.

recycle_on_failure = true

A test that corrupts process-wide state poisons every later test scheduled onto the same worker. The motivating case is a GPU whose driver ends up in a state where every subsequent allocation in the process fails: in the run above, a single such event turned into ten consecutive failed files on one worker. The Distributed-based harness this package was extracted from recycled a worker after any failed test (# the worker encountered some failure, recycle it so future tests get a fresh environment); this restores that behavior alongside the existing max-rss and crash recycling.

retries = N

Tests that did not pass are re-run up to N times after the main run completes — sequentially, on a single fresh worker, with all other workers stopped. Parallel test runs create resource contention (several workers sharing one GPU, or limited RAM), so a failure can mean "lost the resource race" rather than "broken": re-running on an otherwise-idle system distinguishes the two. Analogous to pytest-rerunfailures / Bazel's flaky_test_attempts, but the retry environment is deliberately quiesced. Only the final attempt of each test enters the results, and retried tests are visibly marked in the output, so flakiness is surfaced rather than hidden.

With both options enabled (plus a per-worker GPU memory budget on the oneAPI.jl side), the run above goes from 10167 pass / 520 errors to 11751 pass / 4 errors with zero OOM errors; one file that failed under concurrency was rescued by its idle retry, and deterministic failures failed again and were reported exactly once.

Both options default to off, preserving current behavior. The package's own test suite passes.

[giordano]

If you have a job using too many resources, maybe you want to mark that one as serial? That's what #125 enables (and I'm not looking forward to rebase it for the umpteenth time)

[michel2323]

"Too many resources" is very system-dependent. I don't want to run large tests sequentially on a system where they can run in parallel. Or am I missing something?

[christiangnrd]

I don't want to run large tests sequentially on a system where they can run in parallel

You could detect vram and use that to set the number of jobs to a lower value than would be otherwise selected.

That said, I do like the idea of getting rid of a worker on failure and I would even go as far as to say that it should become the default in the next breaking version.

I do agree with Mose that #125 should be reviewed and merged first.

[michel2323]

This is based on the experience I have when developing oneAPI.jl locally on my box. Manually setting a '--jobs' value does not guarantee that the tests pass, because the order of the tests is non-deterministic and timing-dependent. Therefore, I have to be overly defensive and set for example --jobs=2.

@christiangnrd I've added a VRAM budget for each worker into the oneAPI.jl tests. But that's not enough for various reasons.

[christiangnrd]

You seem to have a version of the problem that led me to open #77 which is to be implemented by #125. In which case I would designate the bigger tests to be run serially. I think you could also potentially only add them to the serial tests list if you detect lower resources or something. This probably wouldn't completely fix your issue but it could help mitigate it significantly.