Skip to content

feat(preprocessing): generalize qc_stats into coverage_stats for reuse with tissue masks#6

Merged
vojtech-cifka merged 13 commits into
masterfrom
feature/qc-statistics
May 5, 2026
Merged

feat(preprocessing): generalize qc_stats into coverage_stats for reuse with tissue masks#6
vojtech-cifka merged 13 commits into
masterfrom
feature/qc-statistics

Conversation

@vojtech-cifka

@vojtech-cifka vojtech-cifka commented Apr 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Generalizes the per-tile QC coverage pipeline into a config-driven coverage_stats module that any mask-based coverage stats can reuse. Tissue stats (previously a near-duplicate script on feature/tile-statistics) now drops in as a single experiment config, so that branch can be closed.

What changed

  • New module preprocessing/coverage_stats.py — generic per-tile coverage computation. Driven by:
    • masks: dict[str, str]{name: filename_template} where {stem} expands to the WSI file stem
    • mask_source — either {local_path: ...} or {mlflow_run_id: ..., artifact_path: ...}
    • mpp — MPP at which the masks were rendered
  • New config configs/preprocessing/coverage_stats.yaml — defaults shared by all consumers
  • QC experiment configs/experiment/preprocessing/qc_stats_standard.yaml — updated to the new schema; preserves existing mlflow_artifact_path: qc_stats, output column names (tile_residual_coverage, etc.), and metric names. No behavior change.
  • Tissue experiment configs/experiment/preprocessing/tissue_stats_05mpp.yaml — single mask tissue: "{stem}.tiff", mask_source pulls from
    the tissue_masks_run_id MLflow artifact
  • Submit scripts — both point at preprocessing.coverage_stats
  • Dataset configtissue_masks_run_id added (was previously only on feature/tile-statistics)
  • Removed preprocessing/qc_stats.py, configs/preprocessing/qc_stats.yaml (rename detected by git)

Summary by CodeRabbit

  • New Features

    • Added a coverage-statistics preprocessing pipeline to compute per-tile and per-ROI foreground coverage.
    • Added standard QC and tissue-statistics experiment configurations, including a 0.5 MPP tissue config and a QC stats config.
    • Added top-level job submission scripts to run QC and tissue-statistics jobs.
  • Chores

    • Extended dataset configuration to reference two additional MLflow artifact IDs.
    • Added runtime dependencies for improved data handling (pyarrow, datasets).
  • Chores

    • Updated QC default resolution parameters (mask/sample MPP).

@vojtech-cifka
vojtech-cifka requested a review from a team April 26, 2026 19:11
@vojtech-cifka vojtech-cifka self-assigned this Apr 26, 2026
@vojtech-cifka
vojtech-cifka requested review from a team, JakubPekar and ejdam87 April 26, 2026 19:11
@coderabbitai

coderabbitai Bot commented Apr 26, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a new coverage-stats preprocessing module, related experiment and dataset configs, two job-submission scripts, and dependency additions; computes per-tile and per-ROI mask coverage using summed-area tables, parallelized with Ray, and logs artifacts/metrics to MLflow.

Changes

Cohort / File(s) Summary
Dataset Config
configs/data/dataset.yaml
Added two MLflow artifact identifiers: mlflow_artifacts.tiling_run_id and mlflow_artifacts.tissue_masks_run_id.
Experiment Configs
configs/experiment/preprocessing/qc_stats_standard.yaml, configs/experiment/preprocessing/tissue_stats_05mpp.yaml, configs/experiment/preprocessing/qc_standard.yaml
Added qc_stats_standard and tissue_stats_05mpp experiment configs (mpp, mask sources, filename templates, mlflow artifact paths, metadata). Updated qc_standard MPP values (mask_mpp and sample_mpp).
Preprocessing Config
configs/preprocessing/coverage_stats.yaml
New global preprocessing config declaring required parameters (mpp, masks, mask_source) and mlflow_artifact_path.
Coverage Statistics Implementation
preprocessing/coverage_stats.py
New Hydra/MLflow/Ray module implementing: tile/slide Parquet loading, mask resolution (local or MLflow), summed-area-table coverage queries, per-slide Ray tasks, streaming Parquet output, and MLflow metric/artifact logging. Contains multiple new functions and a main entrypoint.
Job Submission Scripts
scripts/submit_qc_stats.py, scripts/submit_tissue_stats.py
Added two top-level job submission scripts that configure and submit compute jobs (8 CPU, 64Gi memory), clone repo, sync deps, and run coverage stats preprocessing with experiment overrides.
Project Dependencies
pyproject.toml
Added pyarrow (>=19.0.1) and datasets (>=4.0.0) to project dependencies.

Sequence Diagram(s)

sequenceDiagram
    participant Submit as Submit Script
    participant Job as Compute Job
    participant Repo as Repository
    participant MLflow as MLflow Server
    participant Ray as Ray Worker Pool
    participant Storage as Parquet Storage

    Submit->>Job: submit job (8 CPU, 64Gi)
    Job->>Repo: git clone repo
    Job->>Repo: uv sync (install deps)
    Job->>Job: run preprocessing.coverage_stats

    Job->>MLflow: download `slides.parquet` / `tiles.parquet`
    MLflow-->>Job: return Parquet artifacts

    Job->>Job: resolve mask_source (local dir or MLflow)
    alt masks on MLflow
        Job->>MLflow: download masks artifact
        MLflow-->>Job: return mask images
    end

    Job->>Ray: dispatch slides for processing
    Ray->>Ray: for each slide: load masks -> build SAT -> compute tile & ROI coverage
    Ray-->>Job: return per-slide coverage results

    Job->>Storage: stream-write coverage Parquet
    Storage-->>Job: ack write

    Job->>MLflow: log metrics and upload output artifacts
    MLflow-->>Job: artifacts stored
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 I hop through masks and count each tile,

SATs sum fields across every mile,
Ray scatters tasks, Parquets stack neat,
MLflow hums while results take a seat,
I twirl a carrot — preprocessing complete. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'feat(preprocessing): generalize qc_stats into coverage_stats for reuse with tissue masks' accurately describes the main change: generalizing a QC statistics module into a reusable coverage_stats module for multiple mask types, which aligns with the PR's core objective of refactoring the pipeline for broader applicability.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/qc-statistics

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
Review rate limit: 0/1 reviews remaining, refill in 60 minutes.

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new preprocessing script and associated configurations to calculate quality control coverage statistics for whole-slide image tiles using Ray and Summed Area Tables. The implementation includes new Hydra configuration files and a Kubernetes job submission script. A review comment identifies a potential integer overflow risk when calculating the Summed Area Table for very large images and suggests using np.int64 to ensure numerical stability.

Comment thread preprocessing/coverage_stats.py Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (6)
scripts/submit_qc_stats.py (2)

4-17: Optional: add gpu=None for consistency with other submit scripts.

Other submit scripts in the repo (e.g., scripts/submit_qc.py) explicitly pass gpu=None. Omitting it here works only if submit_job's default already implies no GPU; making it explicit avoids accidentally inheriting a different default if kube_jobs ever changes.

Proposed alignment
     cpu=8,
     memory="64Gi",
+    gpu=None,
     public=False,
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/submit_qc_stats.py` around lines 4 - 17, The submit_job invocation
for "tissue-classification-qc-stats" should explicitly set gpu=None to match
other submit scripts and avoid relying on submit_job defaults; update the call
to include gpu=None alongside cpu, memory, public, script, and storage so the
submit_job(...) invocation (function name submit_job and parameter
job_name="tissue-classification-qc-stats") clearly declares no GPU.

14-14: Align with the existing pattern in submit_qc.py for consistency and robustness.

While uv run -m module_name is supported in uv v0.4.18+, the reference script submit_qc.py uses uv run python -m preprocessing.qc. For consistency and to ensure compatibility with any older uv versions that may be present in the cluster environment, use the same pattern here.

Proposed change
-        "uv run -m preprocessing.qc_stats +experiment=...",
+        "uv run python -m preprocessing.qc_stats +experiment=...",
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/submit_qc_stats.py` at line 14, Replace the uv invocation string to
match the pattern used in submit_qc.py: change the command "uv run -m
preprocessing.qc_stats +experiment=..." to use "uv run python -m
preprocessing.qc_stats +experiment=..." so it remains compatible with older uv
versions; update the string wherever it's defined in scripts/submit_qc_stats.py
(the command variable / argument construction that currently contains "-m
preprocessing.qc_stats") to the new form.
preprocessing/qc_stats.py (4)

195-214: Cancel pending Ray tasks on exception.

If ray.get(done[0]) raises (e.g., a corrupted QC mask on one slide), the finally block closes the writer but the remaining pending futures keep running on the cluster, wasting CPU/memory until the driver exits. Adding a cancel pass keeps shutdowns clean:

Proposed cancellation
     finally:
         if writer is not None:
             writer.close()
+        for fut in pending:
+            ray.cancel(fut, force=False)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@preprocessing/qc_stats.py` around lines 195 - 214, If ray.get(done[0])
raises, the remaining Ray tasks in pending keep running; modify the finally
block in the loop that uses pending/futures (the code that calls ray.wait,
ray.get and manages writer/output_path, cov_cols, cov_sums) to cancel all
remaining futures before closing the writer: iterate over pending and call
ray.cancel(...) (optionally with force=True) inside a safe try/except to swallow
cancel errors, then close writer; this ensures any running tasks are terminated
cleanly and resources freed when an exception occurs.

92-95: SAT dtype is brittle if qc_mpp is lowered.

sat is int32 with a max of ~2.1B. At the current qc_mpp: 8.0 a typical WSI mask is ≲10⁷ pixels, so this is safe today. However, if an experiment ever sets a finer qc_mpp (say 1.0), a 100k×100k mask easily exceeds int32 and silently wraps — coverages would become negative/garbage with no error. Using int64 (or np.uintp) costs only memory for the SAT and removes the foot-gun.

Proposed dtype hardening
-        sat = np.zeros((mask_h + 1, mask_w + 1), dtype=np.int32)
+        sat = np.zeros((mask_h + 1, mask_w + 1), dtype=np.int64)
         sat[1:, 1:] = mask > 0
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@preprocessing/qc_stats.py` around lines 92 - 95, The summed-area table `sat`
is created with dtype=np.int32 which can overflow for finer qc_mpp values;
change its dtype to a 64-bit integer (e.g. dtype=np.int64 or np.intp) when
creating `sat` so cumulative sums won't wrap: update the allocation of `sat`
where `sat = np.zeros((mask_h + 1, mask_w + 1), dtype=...)` to use a 64-bit
integer type (refer to the `sat` variable, and the lines that set `sat[1:, 1:] =
mask > 0` and call `np.cumsum`) so cumsum outputs remain correct for large
masks.

173-182: Use the configured logger instead of print for warnings.

logger: MLFlowLogger is plumbed in via @autolog and the rest of the module relies on MLflow for observability. print lines won't reach the run logs and are easy to miss in Kubernetes pod logs. Either switch to Python logging (consistent with library norms) or surface the missing-slide count as an MLflow tag/metric so it's visible alongside the run.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@preprocessing/qc_stats.py` around lines 173 - 182, Replace the print(...)
that reports the number of missing slides with the configured MLFlowLogger
instead of printing to stdout: call the provided logger (variable name logger:
MLFlowLogger) to emit a warning (e.g., logger.warning or logger.log) and also
record the missing_slides count as an MLflow artifact/tag/metric (e.g.,
logger.log_metric or logger.set_tag) so the value surfaces in run telemetry;
keep the subsequent RuntimeError behavior unchanged. Use the existing
missing_slides variable and the same conditional that currently wraps the print
to locate where to change this.

34-45: Switch from internal Dataset.data.table API to pyarrow.parquet.read_table()

Dataset.data.table is not part of the public datasets API and has changed shape between minor releases. With datasets>=4.0.0 pinned with >=, an upstream refactor could silently break this loader.

A more stable alternative is to read the parquet directly with pyarrow (already a dependency), which also avoids the redundant Arrow IPC cache that HF datasets writes:

-def load_tiles_columns(run_id: str, artifact_path: str, columns: list[str]) -> pa.Table:
-    """Load a tiles parquet column subset as a memory-mapped Arrow table.
-
-    HF datasets caches the parquet into Arrow IPC on first load. We return the
-    underlying pyarrow Table without materializing 80M rows into pandas — only
-    small per-slide slices are converted before dispatching Ray tasks.
-    """
-    local_path = mlflow.artifacts.download_artifacts(
-        run_id=run_id, artifact_path=artifact_path
-    )
-    dataset = load_dataset("parquet", data_files=local_path, split="train")
-    return dataset.select_columns(columns).data.table
+def load_tiles_columns(run_id: str, artifact_path: str, columns: list[str]) -> pa.Table:
+    """Load a tiles parquet column subset as an Arrow table (memory-mapped where possible)."""
+    local_path = mlflow.artifacts.download_artifacts(
+        run_id=run_id, artifact_path=artifact_path
+    )
+    return pq.read_table(local_path, columns=columns, memory_map=True)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@preprocessing/qc_stats.py` around lines 34 - 45, The loader uses the internal
Dataset.data.table access which is unstable; update load_tiles_columns to read
the parquet directly with pyarrow.parquet.read_table by opening the downloaded
local_path, pass columns to read_table to return a pyarrow Table, and remove
reliance on load_dataset/dataset.select_columns; ensure the function still
returns a pa.Table and that pyarrow.parquet is imported and used when reading
local_path and selecting columns (refer to load_tiles_columns, local_path, and
columns).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@preprocessing/qc_stats.py`:
- Around line 215-219: add a guard in the end of add_qc_coverage so you don't
divide by zero: check if count == 0 and, if so, return tile_count 0 with
mean_<col> set to a safe sentinel (e.g., float("nan") or None) instead of
computing cov_sums[col] / count; otherwise compute the means as before using
cov_sums, cov_cols and count. This uses the existing symbols cov_sums, cov_cols,
and count to locate where to add the conditional.

---

Nitpick comments:
In `@preprocessing/qc_stats.py`:
- Around line 195-214: If ray.get(done[0]) raises, the remaining Ray tasks in
pending keep running; modify the finally block in the loop that uses
pending/futures (the code that calls ray.wait, ray.get and manages
writer/output_path, cov_cols, cov_sums) to cancel all remaining futures before
closing the writer: iterate over pending and call ray.cancel(...) (optionally
with force=True) inside a safe try/except to swallow cancel errors, then close
writer; this ensures any running tasks are terminated cleanly and resources
freed when an exception occurs.
- Around line 92-95: The summed-area table `sat` is created with dtype=np.int32
which can overflow for finer qc_mpp values; change its dtype to a 64-bit integer
(e.g. dtype=np.int64 or np.intp) when creating `sat` so cumulative sums won't
wrap: update the allocation of `sat` where `sat = np.zeros((mask_h + 1, mask_w +
1), dtype=...)` to use a 64-bit integer type (refer to the `sat` variable, and
the lines that set `sat[1:, 1:] = mask > 0` and call `np.cumsum`) so cumsum
outputs remain correct for large masks.
- Around line 173-182: Replace the print(...) that reports the number of missing
slides with the configured MLFlowLogger instead of printing to stdout: call the
provided logger (variable name logger: MLFlowLogger) to emit a warning (e.g.,
logger.warning or logger.log) and also record the missing_slides count as an
MLflow artifact/tag/metric (e.g., logger.log_metric or logger.set_tag) so the
value surfaces in run telemetry; keep the subsequent RuntimeError behavior
unchanged. Use the existing missing_slides variable and the same conditional
that currently wraps the print to locate where to change this.
- Around line 34-45: The loader uses the internal Dataset.data.table access
which is unstable; update load_tiles_columns to read the parquet directly with
pyarrow.parquet.read_table by opening the downloaded local_path, pass columns to
read_table to return a pyarrow Table, and remove reliance on
load_dataset/dataset.select_columns; ensure the function still returns a
pa.Table and that pyarrow.parquet is imported and used when reading local_path
and selecting columns (refer to load_tiles_columns, local_path, and columns).

In `@scripts/submit_qc_stats.py`:
- Around line 4-17: The submit_job invocation for
"tissue-classification-qc-stats" should explicitly set gpu=None to match other
submit scripts and avoid relying on submit_job defaults; update the call to
include gpu=None alongside cpu, memory, public, script, and storage so the
submit_job(...) invocation (function name submit_job and parameter
job_name="tissue-classification-qc-stats") clearly declares no GPU.
- Line 14: Replace the uv invocation string to match the pattern used in
submit_qc.py: change the command "uv run -m preprocessing.qc_stats
+experiment=..." to use "uv run python -m preprocessing.qc_stats
+experiment=..." so it remains compatible with older uv versions; update the
string wherever it's defined in scripts/submit_qc_stats.py (the command variable
/ argument construction that currently contains "-m preprocessing.qc_stats") to
the new form.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 54d57456-5971-475c-91e0-0c5c0a80a802

📥 Commits

Reviewing files that changed from the base of the PR and between f624f23 and 2ca73fe.

⛔ Files ignored due to path filters (1)
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing (6)
  • configs/data/dataset.yaml
  • configs/experiment/preprocessing/qc_stats_standard.yaml
  • configs/preprocessing/qc_stats.yaml
  • preprocessing/qc_stats.py
  • pyproject.toml
  • scripts/submit_qc_stats.py

Comment thread preprocessing/coverage_stats.py
@vojtech-cifka
vojtech-cifka removed the request for review from a team April 26, 2026 19:24
…ue masks

Replaces qc_stats with a config-driven coverage_stats pipeline that takes
a masks dict and a mask_source (local path or mlflow artifact). QC keeps
its existing artifact path, column names, and metrics; tissue_stats reuses
the same module via a new experiment config.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@preprocessing/coverage_stats.py`:
- Around line 145-163: The loop assumes there are tiles but
indices/sorted_indices can be empty; before indexing sorted_indices[start]
inside the for i in range(...) loop (and before using
dictionary[int(sorted_indices[start])].as_py()), add a guard to skip empty
splits — e.g., check if sorted_indices.size == 0 or if boundaries length makes
start==end and continue; ensure this check is placed alongside the existing
futures/missing_slides logic so empty splits are skipped without causing
IndexError.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c76eadfd-d721-4911-b050-c91052a1174e

📥 Commits

Reviewing files that changed from the base of the PR and between 2ca73fe and c278adc.

📒 Files selected for processing (6)
  • configs/experiment/preprocessing/qc_stats_standard.yaml
  • configs/experiment/preprocessing/tissue_stats_05mpp.yaml
  • configs/preprocessing/coverage_stats.yaml
  • preprocessing/coverage_stats.py
  • scripts/submit_qc_stats.py
  • scripts/submit_tissue_stats.py
✅ Files skipped from review due to trivial changes (3)
  • configs/preprocessing/coverage_stats.yaml
  • configs/experiment/preprocessing/qc_stats_standard.yaml
  • scripts/submit_tissue_stats.py

Comment thread preprocessing/coverage_stats.py
@vojtech-cifka vojtech-cifka changed the title feat(preprocessing): compute per-tile QC artifact coverage feat(preprocessing): generalize qc_stats into coverage_stats for reuse with tissue masks Apr 29, 2026
@vojtech-cifka
vojtech-cifka requested review from vejtek and removed request for JakubPekar April 29, 2026 12:04
Comment thread preprocessing/coverage_stats.py Outdated
Comment thread preprocessing/coverage_stats.py
@vojtech-cifka
vojtech-cifka requested a review from ejdam87 May 1, 2026 14:52

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@preprocessing/coverage_stats.py`:
- Around line 95-98: The summed-area table 'sat' is using np.int32 and can
overflow for high-resolution masks; change its dtype to a 64-bit integer (e.g.,
np.int64) when creating sat in preprocessing/coverage_stats.py (the sat =
np.zeros((mask_h + 1, mask_w + 1), dtype=...) line) so that sat[1:, 1:] = mask >
0 and the subsequent np.cumsum calls operate on a 64-bit accumulator and avoid
silent overflow.
- Around line 21-32: load_tiles_columns currently calls load_dataset without a
columns projection and then uses dataset.select_columns, which defeats parquet
projection pushdown; change the load_dataset call in load_tiles_columns to pass
the columns argument (i.e., use load_dataset(..., data_files=local_path,
split="train", columns=columns)) so the parquet reader only reads requested
fields, and remove or keep select_columns as redundant after that change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 05903518-7015-4d9b-be60-06c09706507a

📥 Commits

Reviewing files that changed from the base of the PR and between c278adc and 10db1f0.

📒 Files selected for processing (2)
  • configs/data/dataset.yaml
  • preprocessing/coverage_stats.py
✅ Files skipped from review due to trivial changes (1)
  • configs/data/dataset.yaml

Comment thread preprocessing/coverage_stats.py Outdated
Comment thread preprocessing/coverage_stats.py Outdated
Comment thread configs/experiment/preprocessing/qc_stats_standard.yaml Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
preprocessing/coverage_stats.py (1)

21-32: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Push the parquet projection into load_dataset().

select_columns() trims the dataset after the parquet scan, so this still reads every column from tiles.parquet. On the 80M-row path mentioned in the docstring, that is a large avoidable I/O and cache hit. Pass columns=columns to load_dataset(...) and keep or drop select_columns() as a no-op safeguard.

Suggested change
-    dataset = load_dataset("parquet", data_files=local_path, split="train")
-    return dataset.select_columns(columns).data.table
+    dataset = load_dataset(
+        "parquet",
+        data_files=local_path,
+        split="train",
+        columns=columns,
+    )
+    return dataset.data.table
In Hugging Face Datasets, when loading Parquet with `load_dataset("parquet", ...)`, does passing `columns=[...]` enable projection pushdown while calling `Dataset.select_columns()` after load does not avoid reading the other columns?
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@preprocessing/coverage_stats.py` around lines 21 - 32, The function
load_tiles_columns currently calls load_dataset("parquet", ...) without a
columns projection so the parquet reader still scans all columns; update
load_tiles_columns to pass columns=columns into load_dataset(...) to enable
parquet projection pushdown (and you may keep the subsequent
dataset.select_columns(columns) as a no-op safeguard). Locate load_tiles_columns
and change the load_dataset invocation to include the columns argument while
retaining the existing return of dataset.select_columns(columns).data.table.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@preprocessing/coverage_stats.py`:
- Around line 35-48: The resolve_mask_dir function currently prefers local_path
when both keys exist and lets partial MLflow configs cause AttributeError;
update resolve_mask_dir to explicitly validate mask_source as an exclusive
union: if "local_path" is present, ensure neither "mlflow_run_id" nor
"artifact_path" are present and return Path(mask_source.local_path); else if
both "mlflow_run_id" and "artifact_path" are present, use
mlflow.artifacts.download_artifacts(...) and return its Path; otherwise raise a
clear ValueError explaining the required shapes ("either 'local_path' OR both
'mlflow_run_id' and 'artifact_path'") so mixed or partial configs fail fast
instead of silently misbehaving (refer to resolve_mask_dir and mask_source keys
to locate the change).

---

Duplicate comments:
In `@preprocessing/coverage_stats.py`:
- Around line 21-32: The function load_tiles_columns currently calls
load_dataset("parquet", ...) without a columns projection so the parquet reader
still scans all columns; update load_tiles_columns to pass columns=columns into
load_dataset(...) to enable parquet projection pushdown (and you may keep the
subsequent dataset.select_columns(columns) as a no-op safeguard). Locate
load_tiles_columns and change the load_dataset invocation to include the columns
argument while retaining the existing return of
dataset.select_columns(columns).data.table.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c1343dd9-965d-493c-815e-3da588e5048a

📥 Commits

Reviewing files that changed from the base of the PR and between 10db1f0 and c539e4a.

📒 Files selected for processing (1)
  • preprocessing/coverage_stats.py

Comment thread preprocessing/coverage_stats.py

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@preprocessing/coverage_stats.py`:
- Around line 84-85: The ROI offset is currently computed from tm_extent
(roi_offset = tm_extent // 4) which mis-centers when tm_extent is odd; instead
compute roi_offset from roi_extent so the ROI is centered: set roi_offset =
(tm_extent - roi_extent) // 2 after you compute roi_extent = max(1, tm_extent //
2). Update the assignment in preprocessing/coverage_stats.py where roi_extent
and roi_offset are defined so downstream uses (e.g., roi_*_coverage
calculations) use the correctly centered ROI.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: dc5ee7c4-ff40-476a-bb3d-89333ed4874c

📥 Commits

Reviewing files that changed from the base of the PR and between c539e4a and c32d42b.

📒 Files selected for processing (3)
  • configs/experiment/preprocessing/qc_standard.yaml
  • configs/experiment/preprocessing/qc_stats_standard.yaml
  • preprocessing/coverage_stats.py
✅ Files skipped from review due to trivial changes (2)
  • configs/experiment/preprocessing/qc_standard.yaml
  • configs/experiment/preprocessing/qc_stats_standard.yaml

Comment thread preprocessing/coverage_stats.py
@vojtech-cifka
vojtech-cifka merged commit d4d259f into master May 5, 2026
3 checks passed
@vojtech-cifka
vojtech-cifka deleted the feature/qc-statistics branch May 5, 2026 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants