CI: Add pytest plugins pytest-xdist and pytest-rerunfailures#3193
Merged
CI: Add pytest plugins pytest-xdist and pytest-rerunfailures#3193
Conversation
seisman
force-pushed
the
pytest/xdist
branch
3 times, most recently
from
d4cd959 to
d451df7
Compare
seisman
changed the title
seisman
changed the title
seisman
added
maintenance
Member
Author
|
Ping @GenericMappingTools/pygmt-maintainers for comments on adding two more pytest plugins. If agreed, then maybe we should also add them to other workflows, like:
|
seisman
added
run/benchmark
Member
Author
|
Edit: Works after the patch PR #3241. |
CodSpeed Performance ReportMerging #3193 will improve performances by ×3.4Comparing Summary
Benchmarks breakdown
|
seisman
commented
May 11, 2024
Member
Author
|
The two pytest plugins are only installed to the "Benchmarks", "Tests" and "Dev Tests" workflows, because we only care about the speed of these three workflows. |
seisman
changed the title
seisman
added
final review call
seisman
removed
final review call
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Our "Tests" workflow sometimes fails due to a few crashed tests, especially on Windows and macOS. The crashes are likely caused by unfixed upstream bugs, and they cause the CI to fail immediately, so other tests are not executed at all, which is very annoying.
Inspired by this post (https://stackoverflow.com/a/32140354), we can use the pytest-xdist to let other tests continue even if one or more tests crash (https://pytest-xdist.readthedocs.io/en/stable/crash.html). The crashed tests are marked as failures.
In addition to
pytest-xdist, we can also use thepytest-rerunfailuresplugin, which can rerun failing tests multiple times. So, if a test crashes but passes when rerun, then the test should be good.Initially, I thought we have to use
-n 1to run the tests using one process because PyGMT uses a single global session and can't be run parallel withoutimport pygmtin each process (xref #217). However, it turns out-n autoworks, which is unexpected. Here,-n automeans using as many processes as the number of cores (2 for Linux/Windows and 3 for macOS with the GitHub-hosted agents), which speed up the CI runs significantly.Here is the comparison of the "Run tests" step of the latest runs of the main branch (https://github.com/GenericMappingTools/pygmt/actions/runs/8825374546) and this PR (https://github.com/GenericMappingTools/pygmt/actions/runs/8825127559):
Looking at the detailed log in https://github.com/GenericMappingTools/pygmt/actions/runs/8825127559/job/24228863150, we can see that the
test_tilemap_no_cliptest crashed but passed in rerun, so the CI run still passes.