diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c3c64c..9885e92 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,8 @@ * `guanlab_dengkw_pm`: Only map the predictions back through the mod2 SVD when that SVD was actually fitted. When the target had fewer features than `n_mod2`, the method raised `NameError: embedder_mod2`. Unsupported modality pairs now exit 99 (non-applicable) instead of raising a `KeyError` (PR #32). +* `novel_train`: Seed the fallback that picks the internal validation batches, and log which ones were picked. Both `bmmc_multiome` datasets take this path, so the checkpoint `novel` kept differed from run to run (PR #46). + * `process_dataset`: Fall back to holding out a quarter of the batches when the dataset has no `obs["is_train"]`, rather than silently producing four empty h5ads. `obs["is_train"]` carries the NeurIPS 2021 competition split and stays optional; `obs["cell_type"]` is now declared and required (PR #28). * Fix the component paths, build paths and `rename_keys` separator in the helper scripts, which prevented `scripts/create_datasets/test_resources.sh` and both `run_test.sh` scripts from running at all (PR #22). diff --git a/src/methods/novel/novel_train/config.vsh.yaml b/src/methods/novel/novel_train/config.vsh.yaml index 7d646ab..bd584d6 100644 --- a/src/methods/novel/novel_train/config.vsh.yaml +++ b/src/methods/novel/novel_train/config.vsh.yaml @@ -7,6 +7,10 @@ arguments: description: Number of training epochs. info: test_default: 2 + - name: "--seed" + type: "integer" + default: 1 + description: "Seed for choosing the internal validation batches, when the dataset does not contain the batches used in the competition." resources: - path: script.py type: python_script diff --git a/src/methods/novel/novel_train/script.py b/src/methods/novel/novel_train/script.py index 64daf19..91e3588 100644 --- a/src/methods/novel/novel_train/script.py +++ b/src/methods/novel/novel_train/script.py @@ -24,7 +24,8 @@ 'input_train_mod1': 'resources_test/task_predict_modality/openproblems_neurips2021/bmmc_multiome/normal/train_mod1.h5ad', 'input_train_mod2': 'resources_test/task_predict_modality/openproblems_neurips2021/bmmc_multiome/normal/train_mod2.h5ad', 'output': 'resources_test/task_predict_modality/openproblems_neurips2021/bmmc_multiome/normal/models/novel', - 'n_epochs': 100 + 'n_epochs': 100, + 'seed': 1 } meta = { 'resources_dir': 'src/methods/novel', @@ -77,7 +78,9 @@ # if none of phase1_batch is in batch, sample 25% of batch categories rounded up if len(test_batches.intersection(set(batch))) == 0: all_batches = batch.cat.categories.tolist() - test_batches = set(np.random.choice(all_batches, math.ceil(len(all_batches) * 0.25), replace=False)) + rng = np.random.default_rng(par['seed']) + test_batches = set(map(str, rng.choice(all_batches, math.ceil(len(all_batches) * 0.25), replace=False))) +print(f"Using {sorted(test_batches)} as internal validation batches", flush=True) train_ix = [ k for k,v in enumerate(batch) if v not in test_batches ] test_ix = [ k for k,v in enumerate(batch) if v in test_batches ]