fix(ci): run GUI test under pyqt5 instead of skipping - #595
Conversation
test_HighContentScreeningGui.py has been --ignore'd in CI since #391 due to a PySide6 QObject double-init failure. The known-good local recipe (force qtpy/pytest-qt onto PyQt5) works headless, and setup_22.04.sh already installs PyQt5 via apt (python3-pyqt5), so pin QT_API/PYTEST_QT_API=pyqt5 at the job level and drop the --ignore. Verified locally headless under the pin: 5 passed in 17.9s. The PySide6 double-init itself remains unfixed (tracked separately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First CI run showed the pyqt5 pin works (the GUI test passed), but running it in the shared pytest process deterministically segfaults a later Qt test (test_channel_sequence.py:116 qtbot.waitUntil) — reproduced locally on macOS with the same two-file ordering (exit 139). The GUI test leaves Qt/thread state behind (faulthandler dump shows the simulated-camera stream thread and slack-notifier worker still alive). Keep the main run ignoring it and add a second pytest invocation for the GUI test alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
CI finding + fix (second commit): the first run proved the pyqt5 pin works — Fix: keep the main pytest run ignoring the GUI test and run it as its own pytest invocation in a second step — it is now enforced in CI, just process-isolated. This also resolves the original mystery: with no env pin, pytest-qt picked PySide6 for |
With the job-level pin, the main pytest run passed all 1413 tests and then segfaulted at interpreter shutdown — the pin flips pytest-qt/qtpy binding selection for the whole suite, hitting PyQt5's destructor-order-vs-GC conflict at exit (the reason main_hcs.py uses os._exit()). Master runs the main suite unpinned and green, so keep it identical and pin only the isolated GUI-test step, which is the invocation that needs it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Second CI finding: the job-level pin had a wider blast radius than intended — the main run (still Pushed |
Third CI iteration: the isolated GUI-test step passed (5/5) but the process segfaulted during interpreter shutdown — under PyQt5, a process that constructed the full HCS GUI crashes in Qt C++ destructors at exit even when every test passed. This is the same crash main_hcs.py avoids with os._exit(). Add an env-gated (SQUID_PYTEST_HARD_EXIT=1) pytest_unconfigure hook that flushes output and os._exit()s with pytest's real exit status, and set the variable only on the isolated CI step. Verified locally: summary still prints, pass exits 0, failure exits 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Third CI finding + fix ( Fix mirrors the product's own solution: an env-gated ( |
Review finding (Opus): if pytest_sessionfinish never runs (a pytest_sessionstart failure leaves initstate==1, where unconfigure still fires), the previous default of 0 would turn a broken session into a green CI step. Default to 1 so an unrecorded status fails the gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… suite has the same flaky exit segfault The round-4 CI failure on this PR was the main run (byte-identical invocation to master's) segfaulting at interpreter shutdown after 1413 tests passed. Master itself fails the same way — 3 of its last 15 runs: 'N passed' followed by 'Segmentation fault (core dumped)' exit 139 (runs 28883871972, 28762269294, 28559414408). This also retroactively explains this PR's round-2 failure, which was misattributed to the job-level Qt pin. Set SQUID_PYTEST_HARD_EXIT=1 on the main step as well: after the session completes and all reporting is written, the conftest hook os._exit()s with pytest's real status — genuine failures still fail the step, and the ~20%-flaky Qt-teardown crash stops failing green runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Round-4 CI failure root-caused — it's a pre-existing master flake, and this PR now fixes it ( The failing run was the main pytest step — byte-identical invocation to master's ( Fix: set |
…gfaults (#604) ## Problem CI has been failing ~30% of runs since 2026-06-19 with what look like random test failures. They are almost never test failures: in 11 of 12 failed runs examined, **every test passed and the pytest process then segfaulted** (exit 139) — usually during interpreter shutdown (e.g. run 30389568104: `1612 passed` → segfault 26s later), occasionally mid-suite inside the Qt event loop (run 27848898456 has the faulthandler dump). ## Root cause ~40 test call sites build a full simulated `Microscope`/`MultiPointController`; only ~15 ever close one. Each leak leaves behind: - daemon threads: camera `stream_fn` (1ms busy-loop generating frames), slack-notifier workers, laser-engine `_tick_loop` - a prewarmed `JobRunner` **child process** per `MultiPointController` (failed runs show ~17 orphaned `python3` processes at job cleanup; green runs show none) A leftover thread touching a destroyed Qt object mid-suite, or frozen inside Qt/numpy C code during interpreter finalization, segfaults the process. Green vs red is a race on thread timing — hence "random". A visible marker in **every** run (green and red): `Exception in thread Acquisition thread ... TimeoutError: Current mcu operation timed out after 3 [s]` — the watchdog tests abort an acquisition and close the scope without joining the acquisition thread, which then times out returning the stage home against a closed simulated microcontroller. ## Fix - **`tests/conftest.py` (new)**: suite-wide autouse fixture tracking every `Microscope`, `MultiPointController` (incl. `QtMultiPointController`) and `Microcontroller` built during a test, closing them at teardown in dependency order — MPC first (joins the acquisition thread and shuts down JobRunner children while the microcontroller is still alive), then Microscope (stops camera streaming), then stray microcontrollers. Replaces the Microcontroller-only fixture in `tests/control/conftest.py` and extends coverage to `tests/squid/` and root-level tests. Watchdog breadcrumb writes during cleanup are pinned to a tmp dir. - **`Microscope.close()` is now idempotent**, so tests that already close explicitly don't double-close every component when the fixture runs. - **Watchdog tests** call `mpc.close()` before `scope.close()`, fixing the standing `TimeoutError` thread leak. ## Incorporates #595 (by @hongquanli) as belt-and-braces - `SQUID_PYTEST_HARD_EXIT=1`: a `pytest_unconfigure` hook `os._exit()`s with pytest's real exit status after the session completes, skipping the crash-prone interpreter teardown entirely (same trick `main_hcs.py` uses). Real test failures still fail the step. - `test_HighContentScreeningGui.py` runs in its own pytest process pinned to PyQt5 instead of being skipped (its full-GUI Qt state crashes later tests in a shared process; mixed pytest-qt/qtpy bindings cause the QObject double-init error). The hooks moved to the new top-level `tests/conftest.py`, so if this merges, #595 is superseded. ## Verification - Watchdog tests pass locally with no `PytestUnhandledThreadExceptionWarning` (previously present in every CI run). - `test_MultiPointController.py` passes under the new fixture. - Hard-exit hook preserves exit status in both directions (0 on pass, 1 on fail, verified locally). - One local-only failure (`test_microscope_wraps_pi_focus_when_enabled`) reproduces identically on unmodified master — pre-existing local machine-config difference, unrelated. - CI on this PR is the authoritative check for the flake itself; the leaked-thread warning should be absent from the log. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2 participants
Summary
tests/control/test_HighContentScreeningGui.pyhas been--ignore'd in CI since #391 with the note "PySide6 QObject double-init bug". This PR stops skipping it:QT_API: pyqt5andPYTEST_QT_API: pyqt5as job-level env inmain.yml, forcing qtpy/pytest-qt onto PyQt5 (the known-good local recipe).--ignorefrom the pytest invocation.setup_22.04.shalready installs PyQt5 via apt (python3-pyqt5), so no dependency changes are needed.Verification
configurations/configuration_Squid+.ini, same file CI copies): 5 passed in 17.9s.Open question (documented, not fixed here)
The PySide6 QObject double-init failure itself is unchanged: under PySide6, GUI construction in the test fails (qtpy picks PySide6 when it's importable). Pinning CI to PyQt5 matches the production launch path (
main_hcs.pyandrun_acquisition.pyboth setQT_API=pyqt5), so this is an acceptable steady state; root-causing the PySide6 path (likely theQtMultiPointController(MultiPointController, QObject)multiple-inheritance init pattern) remains a separate issue if PySide6 support is ever wanted.🤖 Generated with Claude Code