feat: add IVIM workflow DAG with MinIO integration and platform docum… by Unique-Usman · Pull Request #112 · OSIPI/TF2.4_IVIM-MRI_CodeCollection

Unique-Usman · 2025-07-24T20:43:14Z

…entation

Created a custom Airflow DAG for Kaapana that:
- Retrieves input files (e.g., bvec, bval) from MinIO using MinioOperator
- Processes data via IVIM Operator using nifti_wrapper_kaapana.py
- Saves outputs back to MinIO using MinioOperator
- Utilizes OPERATOR_IN_DIR and OPERATOR_OUT_DIR volume mounts
- Cleans up intermediate files with CleanOperator
Added diagram (workflow_extension.png) showing DAG structure
Structured project under kaapana_ivim_osipi/ with clear separation of Docker, workflows, and processing code
Documented Otsus method extension installation steps
README: Added full Kaapana platform installation guide

…entation - Created a custom Airflow DAG for Kaapana that: - Retrieves input files (e.g., bvec, bval) from MinIO using MinioOperator - Processes data via IVIM Operator using nifti_wrapper_kaapana.py - Saves outputs back to MinIO using MinioOperator - Utilizes OPERATOR_IN_DIR and OPERATOR_OUT_DIR volume mounts - Cleans up intermediate files with CleanOperator - Added diagram (workflow_extension.png) showing DAG structure - Structured project under `kaapana_ivim_osipi/` with clear separation of Docker, workflows, and processing code - Documented Otsus method extension installation steps - README: Added full Kaapana platform installation guide

etpeterson

first glance - more to come

etpeterson · 2025-07-25T04:19:51Z

@@ -0,0 +1,8 @@
+FROM local-only/base-installer:latest


Is this a special tag? How does this work?

Yeah, it is. It is made available by the kaapana team

One thing we can do here to avoid requiring multiple dockerfiles is define the base image as a build-time argument. We can set reasonable defaults so that the image builds and runs for regular users, and if a user wants to build for kaapana they can pass in build arguments for that (which would be described in the readme).

etpeterson · 2025-07-25T04:27:56Z

This seems very similar to the current Dockerfile: https://github.com/OSIPI/TF2.4_IVIM-MRI_CodeCollection/blob/main/Docker/Dockerfile
Could they be unified in some way, perhaps with variables or sections?

We should be able to unified it.

etpeterson · 2025-07-25T04:31:27Z

This too seems like essentially a duplicate of something that already exists: https://github.com/OSIPI/TF2.4_IVIM-MRI_CodeCollection/blob/main/WrapImage/nifti_wrapper.py

Yeah, I think we should be able to unified this also.

Unique-Usman · 2025-07-25T09:47:53Z

Hi @etpeterson, thanks for the review. Sorry for lot of noise in the commit. I did not meant for this current implementation to be perfect. I will keep making push here. It will serve as a synchronization for what the current state of the implementation is and also to get earlier review for whatever implementation is being done and also to ensure everyone is on the same page.

- Updated Chart.yaml with accurate description, tags, and maintainers for IVIM task force - Removed unused UI form fields in DAG definition - Fixed tuple syntax for file extensions - make `nifti_wrapper_kaapana.py` to drop batch loop and use direct input/output paths since our workflow is not batch.

ltorres6

Leaving some questions, comments.

ltorres6 · 2025-08-06T15:25:18Z

@@ -0,0 +1,8 @@
+FROM local-only/base-installer:latest


One thing we can do here to avoid requiring multiple dockerfiles is define the base image as a build-time argument. We can set reasonable defaults so that the image builds and runs for regular users, and if a user wants to build for kaapana they can pass in build arguments for that (which would be described in the readme).

ltorres6 · 2025-08-06T21:38:39Z

Can you point me to where you got the original implementation of the fitting to compare?

https://github.com/OSIPI/TF2.4_IVIM-MRI_CodeCollection/blob/main/WrapImage/nifti_wrapper.py

… dynamic UI configuration This change makes the IVIM fitting pipeline fully configurable from the UI form instead of relying on hardcoded file paths and algorithm settings. Key updates: - UI form schema changes: - Added `source_files` for comma-separated NIfTI, bvec, bval paths (used for MinIO data retrieval) - Added optional `affine` matrix override - Added `algorithm` selection from a predefined list - Removed unused `single_execution` field and added `required` validation - MyIvimFittingOperator: - Increased default execution timeout to 5 hours - Added configurable `ram_mem_mb` and `ram_mem_mb_lmt` - nifti_wrapper_kaapana.py: - Read config from `$WORKFLOW_DIR/conf/conf.json` (no hardcoded paths) - Dynamically resolve input/output paths and algorithm from config - Support optional affine override - Added workaround to copy `.nii.gz` files to `WORKFLOW_DIR` to handle MinioOperator limitations Motivation: - Eliminate hardcoded values for input files and algorithm parameters - Allow workflows to be driven entirely by user-provided configuration - Improve maintainability and flexibility for different datasets and algorithms

etpeterson

I took a brief look. Let me know if you want something more thorough.

etpeterson · 2025-08-20T04:27:36Z

  - name: dag-installer-chart
    version: 0.0.0
-    repository: "file:///home/ivim/kaapana/services/utils/dag-installer-chart/"
+    repository: "file:///home/usman/kaapana/services/utils/dag-installer-chart/"


This seems to be hard coded for you only.

we can make the file portable so it that we would not have to hardcoded. We basically bring the dag-installer-chart into the workflow, but, we might have to figure out what to do if the kaapana teams make an update to it.

In their examples I see relative paths. https://codebase.helmholtz.cloud/odile.elias/kaapana/-/blob/0.1.2/platforms/kaapana-platform/kaapana-platform-chart/requirements.yaml?ref_type=tags

Yeah, it could be relative or absolute. It just have to point to the dependency folder dag-installer-chart which is inside the kaapana repository/directory. I will make sure to add it to the docs.

etpeterson · 2025-08-20T04:31:49Z

+    # Temporary solution to pass through the kaapana MinioOperator limitation
+    # where it uses the same source_files for both put and get actions
+
+    # Find all .nii.gz files in the input directory


Do we need all files here or just the output files?

This update extends the IVIM fitting pipeline to handle both NIfTI and raw DICOM uploads with a branching DAG structure. Key changes: - Added new DicomToNiftiOperator for converting DICOMs to NIfTI. - Introduced new `dicom_to_nifti` processing container with Dockerfile, conversion script, and requirements. - Updated DAG (dag_ivim-fitting_method.py): - Added `upload_type` field in UI form (`nifti` | `dicom`). - Implemented BranchPythonOperator to route workflow based on input type. - Defined parallel execution paths for DICOM and NIfTI. - Separated output MinIO upload tasks per path with cleanup. - Updated nifti_wrapper_kaapana.py: - Added support for resolving inputs from DICOM-converted batches. - Preallocated arrays for IVIM fitting outputs (f, Dp, D). - Improved logging and error handling for missing batch files. - Added clear section headers and structure for maintainability.

- Clone TF2.4_IVIM-MRI_CodeCollection from GitHub in Dockerfile - Moved nifti_wrapper_kaapana.py into WrapImage/ directory

etpeterson · 2025-08-29T06:17:08Z

+    save_nifti_file(D_image, affine, os.path.join(element_output_dir, "d.nii.gz"))
+
+    # Copy all .nii.gz from input directory to WORKFLOW_DIR (Kaapana workaround)
+    nii_files = glob.glob(os.path.join(element_input_dir, "*.nii.gz"))


How about just saving the output files to this folder directly?

Due to the sources file bug, the minio put operator is looking for the nifti input fil in the WORKFLOW_DIR that is why I am copying it there. The output file is expected to be saved in the element_output_dir, that is the convension.The above was a workaround due to the bug in kaapana.

I don't really understand. The input files are already in the input folder so it's just about moving the new files to the correct folder?

etpeterson · 2025-08-29T06:20:48Z

+                "description": "Select the algorithm to use.",
+                "type": "string",
+                "enum": [
+                    "ASD_MemorialSloanKettering_QAMPER_IVIM",


Is there a way to populate this from the algorithms itself rather than hard coding?

It should be possible, we will just need an interface(function or something) that returns all the algorithms, we can then convert it to json.

@etpeterson, do we currently have any function that return all the algorithms present in our repository ?

etpeterson · 2025-08-29T06:21:33Z

+# -----------------------------
+# Shared IVIM fitting
+# -----------------------------
+ivim_fitting_task_dicom = MyIvimFittingOperator(


No luck merging earlier?

if you look at the https://github.com/OSIPI/TF2.4_IVIM-MRI_CodeCollection/pull/112/files/8372c92a6bc251bd2bf7a77828bb373182769a54#diff-c9a68a5e72bad521effe2518b51fba76337e0e0d8c31dbfdf22d055ec4ecc7d3, we combine it by

trigger_rule="none_failed_or_skipped, we can also do that here but, we would not be able to use theelement_input_dir to get the input data which is an output from the previous dag, we will have figure out where the data from previous operator is stored which means, we will mostly hardcode where the data is coming from.

I'm not sure what you're trying to show. I don't see it in Kaapana docs, but in airflow I was suggesting conditional operations. Is that also what you're talking about? https://airflow.apache.org/docs/apache-airflow/1.10.1/concepts.html

Yeah above, I am using the conditional operation above.

# ----------------------------- # Branch function # ----------------------------- def choose_upload_type(**context): wf_form = context["dag_run"].conf.get("workflow_form", {}) upload_type = wf_form.get("upload_type", "nifti") log.info(f"Upload type selected: {upload_type}") if upload_type == "dicom": return "get_input" return "download-data"

…aapana - Removed otsus method descrition. - Add Docs for Kaapana and ivim-fitting-method workflow INSTALL.md - for installing Kaapana README.md - Description about the workflow and how to install the workflow on Kaapan

Add complete documentation covering NIfTI and DICOM workflows, parameter configuration, and step-by-step usage instructions

etpeterson

It's getting there but I'm worried about how many unresolved comments are still here. Are you able to resolve everything in a few days?

etpeterson · 2025-09-03T04:17:27Z

+# -----------------------------
+# Shared IVIM fitting
+# -----------------------------
+ivim_fitting_task_dicom = MyIvimFittingOperator(


I'm not sure what you're trying to show. I don't see it in Kaapana docs, but in airflow I was suggesting conditional operations. Is that also what you're talking about? https://airflow.apache.org/docs/apache-airflow/1.10.1/concepts.html

Unique-Usman · 2025-09-03T04:30:55Z

It's getting there but I'm worried about how many unresolved comments are still here. Are you able to resolve everything in a few days?

Yeah, it should, Since I have the baseline now, I will focus on addressing all the unresolved comments now.

Instead of parsing the sources_file based on order, we now parsed it based on the file extension.

- Install dcm2niix via curl from GitHub releases instead of Debian package to ensure JPEG-LS support - Remove dcmdjpls decode step and temporary raw_dicoms folder - Run dcm2niix directly on input directories - Simplifies workflow and ensures compatibility with JPEG-LS encoded DICOMs

- Ensures each NIfTI in dicom_to_nifti uses correct BVAL/BVEC - Handles filenames with multiple dots

etpeterson

Looks good. I think you fixed the major stuff.

etpeterson · 2025-09-05T01:20:59Z

@oliverchampion Ok to merge this? It should provide a web (kaapana) based instance of the fitting that can run uploaded data. It's independent of the other code so I don't think it impacts anything other than perhaps needing to update it if the standard wrapper changes.

oliverchampion · 2025-09-05T04:41:06Z

Cool! Is there any way I can see the web-based fitting?

Unique-Usman · 2025-09-05T05:52:25Z

Cool! Is there any way I can see the web-based fitting?

While it required setting up a kaapana instance, we can give you access to the instance we are using for development and testing.

oliverchampion · 2025-09-05T14:29:37Z

If easy, I'm interested in seeing it work. But happy to merge otherwise :). Maybe a short demo Tuesday?

Unique-Usman · 2025-09-05T14:58:09Z

If easy, I'm interested in seeing it work. But happy to merge otherwise :). Maybe a short demo Tuesday?

Yeah, I will love to have a demo with you, what time are you chanced on Tuesday ? Also, we can have it merged before then though :)

Unique-Usman · 2025-09-08T21:19:37Z

@oliverchampion, calling your attention to this, what time are you chanced on Tuesday ? Thank you.

oliverchampion · 2025-09-09T05:24:14Z

Hey! We have our regular TF meeting at 15:00 CEST. Maybe you can join and if there is time at the end, give a demo? Otherwise right after the meeting?

oliverchampion · 2025-09-09T05:24:57Z

Hey! We have our regular TF meeting at 15:00 CEST. Maybe you can join and if there is time at the end, give a demo? Otherwise right after the meeting?

Unique-Usman · 2025-09-09T06:31:03Z

@oliverchampion, that is perfectly okay by me, I will send you an email to get access to the meeting link. Thank you.

Unique-Usman · 2025-09-10T14:10:12Z

@oliverchampion bringing your attention to this. Thanks.

oliverchampion · 2025-09-10T14:11:27Z

@Unique-Usman , my compliments!

Unique-Usman mentioned this pull request Jul 24, 2025

feat: add IVIM workflow DAG with MinIO integration and platform documentation #111

Closed

etpeterson requested changes Jul 25, 2025

View reviewed changes

etpeterson reviewed Jul 25, 2025

View reviewed changes

Comment thread kaapana_ivim_osipi/ivim-fitting-method/extension/docker/files/dag_ivim-fitting_method.py Outdated

etpeterson reviewed Jul 25, 2025

View reviewed changes

Comment thread kaapana_ivim_osipi/ivim-fitting-method/extension/docker/files/dag_ivim-fitting_method.py Outdated

etpeterson reviewed Jul 25, 2025

View reviewed changes

Unique-Usman requested a review from etpeterson July 27, 2025 07:24

ltorres6 self-requested a review July 28, 2025 14:49

etpeterson reviewed Aug 1, 2025

View reviewed changes

Comment thread ...essing-containers/ivim-fitting-method/TF2.4_IVIM-MRI_CodeCollection/nifti_wrapper_kaapana.py Outdated

ltorres6 reviewed Aug 6, 2025

View reviewed changes

etpeterson requested changes Aug 20, 2025

View reviewed changes

Unique-Usman requested review from etpeterson and ltorres6 August 27, 2025 22:54

ivim-fitting-method: clone repo and move nifti_wrapper_kaapana.py

8372c92

- Clone TF2.4_IVIM-MRI_CodeCollection from GitHub in Dockerfile - Moved nifti_wrapper_kaapana.py into WrapImage/ directory

etpeterson requested changes Aug 29, 2025

View reviewed changes

etpeterson reviewed Aug 29, 2025

View reviewed changes

Comment thread kaapana_ivim_osipi/ivim-fitting-method/processing-containers/dicom_to_nifti/Dockerfile

Unique-Usman added 6 commits August 30, 2025 17:53

docs(workflow): add IVIM fitting method user guide

accb817

Add complete documentation covering NIfTI and DICOM workflows, parameter configuration, and step-by-step usage instructions

renamed the docs for clarity

04b86ed

Mount the DicomToNifti.py to the Kaapana directory

04396bf

Removed the unecessary README.md file

fedad5a

Added the TECHNICAL.md for technical docs of the workflow

c450fd1

etpeterson requested changes Sep 3, 2025

View reviewed changes

Unique-Usman added 3 commits September 3, 2025 10:31

Make the parsing of sources file more robust

c75496a

Instead of parsing the sources_file based on order, we now parsed it based on the file extension.

Pair NIfTI files with matching BVAL/BVEC in DICOM batch path

10f9c57

- Ensures each NIfTI in dicom_to_nifti uses correct BVAL/BVEC - Handles filenames with multiple dots

etpeterson approved these changes Sep 5, 2025

View reviewed changes

oliverchampion merged commit 8c632cf into OSIPI:main Sep 10, 2025
9 checks passed

Unique-Usman mentioned this pull request Oct 27, 2025

Ehancement to kaapana_ivim_osipi #115

Open

5 tasks

rohitx01 mentioned this pull request Nov 2, 2025

feat: consolidate Dockerfiles with build arguments (Issue #115 Task #4) #127

Open

Conversation

Unique-Usman commented Jul 24, 2025

Uh oh!

etpeterson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Unique-Usman commented Jul 25, 2025

Uh oh!

Uh oh!

ltorres6 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etpeterson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Unique-Usman Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Unique-Usman Aug 29, 2025 •

edited

Loading