Fix the component bugs surfaced by the full run

[rcannood]

Describe your changes

Component bugs found in the logs of run 1gVHaBsNbAHcfT. Each one is verified rather than reasoned about; where I could not close something out I left it out of this PR and listed it below.

correlation and mse no longer halt on non-finite predictions

ss_opm emitted NaN on pbmc_cite/normal, and sd() on that prediction returns NA, so correlation died on if (NA) -- missing value where TRUE/FALSE needed -- rather than scoring the method. mse did not crash but wrote a NaN score. Both now score non-finite entries as zero, which is the convention already used for the zero-variance case. A method emitting NaN should land at the bottom of the scale, not take the metric down with it.

I also dropped the two commented-out !is.finite(...) lines in correlation, which read like a disabled guard for exactly this.

Checked by poisoning a prediction with NaN and Inf and running both metrics: all six correlation metrics and both mse metrics come out finite, and viash test stays green on both.

exit 99 instead of ValueError

babel and scbutterfly are GEX<->ATAC only, but raised on the CITE datasets, so they failed with exit 1 and were retried three times each -- 16 of their 25 failures were nothing but this, and the whole run produced no exit-99 task at all. exit_non_applicable() already existed for guanlab_dengkw_pm.

senkin_tmp_train can finally use a GPU

It went 0/8, seven walltime kills and seven OOMs. The cause is a CUDA mismatch rather than a resource shortfall: inside the real image on a real T4, the container gets the GPU and PyTorch uses it (torch cuda: True 13.1), but the pip-installed TensorFlow 2.21 is built against CUDA 12.5 and fails with Error loading CUDA libraries. tensorflow[and-cuda] does not rescue it -- the twelve nvidia-*-cu12 wheels install cleanly and TF still reports no GPU.

openproblems/base_tensorflow_nvidia:1 works, verified in a real srun -p gpu allocation:

IMGCHECK tf 2.17.0 cuda 12.8 gpus [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
IMGCHECK matmul on /job:localhost/replica:0/task:0/device:GPU:0

nvcr have stopped building TensorFlow containers, but 25.02-tf2-py3 is still pullable and core has already published a base image from it, so nothing is blocked. tensorflow drops out of the pip list since the base carries it -- pinning it is what broke the CUDA pairing in the first place.

novel_predict truthiness

uns["removed_vars"] round-trips through h5ad as an array, so if ...uns.get("removed_vars"): raised the truth value of an array with more than one element is ambiguous. This has been failing since #33 made it record every all-zero feature rather than the first.

ss_opm no longer asks for 222 GiB

build_metadata() densified the whole normalized layer to compute six per-cell statistics, which is Unable to allocate 222. GiB for an array with shape (130095, 228942) on the multiome datasets -- a size that fits on no node. Now done one row block at a time; I checked all six statistics come out bit-identical.

Still open

Not fixed here, so the next person is not surprised:

• cellmapper_scvi -- 0/8, KeyError: 'mod2_pred' after training completes.
• ss_opm -- the NaN itself. It enters inside ss_opm's own IterativeSVDImputator.transform, upstream of the metric; the two safe wrappers already in ss_opm_train do not cover it.
• simple_mlp_train -- split() masks on site == 's1'/'s2'/'s3' parsed from obs['batch'][:2]. The 2022 pbmc batches do not use the NeurIPS-2021 s#d# naming, so the validation fold is empty, valid_RMSE is never logged, and ckpt_path='best' has no checkpoint. Needs the same kind of fallback novel got in Seed novel's internal validation split #46.
• guanlab_dengkw_pm -- kernel ridge builds an n x n kernel and dies in scipy.linalg.solve (10 segfaults, 2 OOMs). Probably an inherent limit on the 2022 datasets rather than a defect. It also runs ~58 threads against a 30-core allocation, which nothing enforces.

Checklist before requesting a review

• I have performed a self-review of my code

• Check the correct box. Does this PR contain:

  • Breaking changes
  • New functionality
  • Major changes
  • Minor changes
  • Bug fixes

• Proposed changes are described in the CHANGELOG.md

• CI Tests succeed and look good!