feature/use-dask

.github/issue_template/bug_report.md

@@ -1,18 +1,35 @@
-**Description:**
-A clear and concise description of what the bug is.
+---
+name: Bug Report
+about: '"Something''s wrong..."'
+title: ''
+labels: ''
+assignees: ''
 
-**Expected Behavior:**
-A clear and concise description of what you expected to happen.
+---
 
-**System and Software:**
+## System and Software
 * aicsimageio Version:
 * Python Version:
 * Operating System:
 
-**Steps to Reproduce:**
-```
-Paste the command(s) you used and the output.
-```
 
- **Additional context**
-Add any other context about the problem here.
+## Description
+*A clear description of the bug*
+
+
+
+
+## Expected Behavior
+*What did you expect to happen instead?*
+
+
+
+
+## Reproduction
+*A minimal example that exhibits the behavior.*
+
+
+
+
+## Environment
+*Any additional information about your environment*

.github/issue_template/feature_request.md

@@ -1,5 +1,23 @@
-**Description:**
-A clear and concise description of what you want to happen.
+---
+name: Feature Request
+about: '"It would be really cool if x did y..."'
+title: ''
+labels: ''
+assignees: ''
 
-**Additional context**
-Add any other context or screenshots about the feature request here.
+---
+
+## Use Case
+*Please provide a use case to help us understand your request in context*
+
+
+
+
+## Solution
+*Please describe your ideal solution*
+
+
+
+
+## Alternatives
+*Please describe any alternatives you've considered, even if you've dismissed them*

.github/workflows/build-docs.yml

@@ -1,4 +1,4 @@
-name: Docs
+name: Documentation
 
 on:
   push:
@@ -9,12 +9,23 @@ jobs:
   docs:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
+    - name: Set up Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.7
+    - name: Install Dependencies
+      run: |
+        pip install --upgrade pip
+        pip install .[all]
     - name: Generate Docs
-      uses: JacksonMaxfield/github-pages-deploy-action-python@master
-      env:
+      run: |
+        make gen-docs
+        touch docs/_build/html/.nojekyll
+    - name: Publish Docs
+      uses: JamesIves/github-pages-deploy-action@releases/v3
+      with:
         ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
         BASE_BRANCH: master # The branch the action should deploy from.
         BRANCH: gh-pages # The branch the action should deploy to.
-        FOLDER: docs/_build/html # The folder the action should deploy. This example folder is generated by Sphinx
-        BUILD_SCRIPT: pip install .[dev] && make docs && touch docs/_build/html/.nojekyll # The build script the action should run prior to deploying.
+        FOLDER: docs/_build/html/ # The folder the action should deploy.

.github/workflows/build-master.yml

@@ -20,12 +20,12 @@ jobs:
         os: [ubuntu-latest, windows-latest, macOS-latest]
 
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@master
+      uses: actions/setup-python@v1
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install dependencies
+    - name: Install Dependencies
       run: |
         python -m pip install --upgrade pip
         pip install .[test]
@@ -38,12 +38,12 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
     - name: Set up Python 3.7
-      uses: actions/setup-python@master
+      uses: actions/setup-python@v1
       with:
         python-version: 3.7
-    - name: Install dependencies
+    - name: Install Dependencies
       run: |
         python -m pip install --upgrade pip
         pip install .[test]

.github/workflows/publish.yml

@@ -9,20 +9,20 @@ jobs:
   publish:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
     - name: Set up Python
-      uses: actions/setup-python@master
+      uses: actions/setup-python@v1
       with:
         python-version: 3.7
-    - name: Install dependencies
+    - name: Install Dependencies
       run: |
         python -m pip install --upgrade pip
         pip install setuptools wheel
-    - name: Build package
+    - name: Build Package
       run: |
         python setup.py sdist bdist_wheel
     - name: Publish to PyPI
-      uses: pypa/gh-action-pypi-publish@master
+      uses: pypa/gh-action-pypi-publish@v1
       with:
         user: aicspypi
         password: ${{ secrets.PYPI_TOKEN }}

.github/workflows/test-and-lint.yml

@@ -12,30 +12,29 @@ jobs:
         os: [ubuntu-latest, windows-latest, macOS-latest]
 
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@master
+      uses: actions/setup-python@v1
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install dependencies
+    - name: Install Dependencies
       run: |
         python -m pip install --upgrade pip
         pip install .[test]
     - name: Test with pytest
       run: |
         pytest --cov-report xml --cov=aicsimageio aicsimageio/tests/
-        codecov -t ${{ secrets.CODECOV_TOKEN }}
 
   lint:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
     - name: Set up Python 3.7
-      uses: actions/setup-python@master
+      uses: actions/setup-python@v1
       with:
         python-version: 3.7
-    - name: Install dependencies
+    - name: Install Dependencies
       run: |
         python -m pip install --upgrade pip
         pip install .[test]

.gitignore

@@ -69,6 +69,9 @@ instance/
 docs/_build/
 docs/aicsimageio.*rst
 
+# Dask workers
+dask-worker-space/
+
 # PyBuilder
 target/
 

README.md

@@ -9,69 +9,128 @@ A Python library for reading and writing image data with specific support for ha
 ---
 
 ## Features
-* Supports reading metadata and imaging data from file path or buffered bytes for:
+* Supports reading metadata and imaging data for:
     * `CZI`
     * `OME-TIFF`
     * `TIFF`
-    * Any additional format supported by `imageio`
+    * Any additional format supported by [`imageio`](https://github.com/imageio/imageio)
 * Supports writing metadata and imaging data for:
     * `OME-TIFF`
-    * `TIFF`
-    * Any additional format supported by `imageio`
 
-### Disclaimer:
-This package is under heavy revision in preparation for version 3.0.0 release. The quick start below is representative
-of how to interact with the package under 3.0.0 and not under the current stable release.
+## Installation
+**Stable Release:** `pip install aicsimageio`<br>
+**Development Head:** `pip install git+https://github.com/AllenCellModeling/aicsimageio.git`
+
+## Documentation
+For full package documentation please visit
+[allencellmodeling.github.io/aicsimageio](https://allencellmodeling.github.io/aicsimageio/index.html).
 
 ## Quick Start
+
+### Full Image Reading
 ```python
 from aicsimageio import AICSImage, imread
 
-# For numpy array of image data
-im = imread("/path/to/your/file_or_buffer.ome.tiff")
+# Get an AICSImage object
+img = AICSImage("my_file.tiff")
+img.data  # returns 6D STCZYX numpy array
+img.dims  # returns string "STCZYX"
+img.shape  # returns tuple of dimension sizes in STCZYX order
+img.size("STC")  # returns tuple of dimensions sizes for just STC
+img.get_image_data("CZYX", S=0, T=0)  # returns 4D CZYX numpy array
 
-# For AICSImage object that
-with AICSImage("/path/to/your/file_or_buffer.ome.tiff") as im:
-    # use im object
+# Get 6D STCZYX numpy array
+data = imread("my_file.tiff")
+```
 
-# To specify a known dimension order
-with AICSImage("/path/to/your/file_or_buffer.ome.tiff", known_dims="SYX") as im:
-    # use im object
+### Delayed Image Slice Reading
+```python
+from aicsimageio import AICSImage, imread_dask
+
+# Get an AICSImage object
+img = AICSImage("my_file.tiff")
+img.dask_data  # returns 6D STCZYX dask array
+img.dims  # returns string "STCZYX"
+img.shape  # returns tuple of dimension sizes in STCZYX order
+img.size("STC")  # returns tuple of dimensions sizes for just STC
+img.get_image_dask_data("CZYX", S=0, T=0)  # returns 4D CZYX dask array
+
+# Read specified portion of dask array
+lazy_s0t0 = img.get_image_dask_data("CZYX", S=0, T=0)  # returns 4D CZYX dask array
+s0t0 = lazy_s0t0.compute()  # returns 4D CZYX numpy array
+
+# Or use normal numpy array slicing
+lazy_data = imread_dask("my_file.tiff")
+lazy_s0t0 = lazy_data[0, 0, :]
+s0t0 = lazy_s0t0.compute()
+```
 
-# if you instantiate an AICSImage:
-im = AICSImage("/path/to/your/file_or_buffer.ome.tiff")
-# you should close it when done:
-im.close()
+When using the `dask_data` array, it is important to know when to `compute` or `persist` data and when to keep
+chaining computation. [Here is a good rundown on the trade offs.](https://stackoverflow.com/questions/41806850/dask-difference-between-client-persist-and-client-compute#answer-41807160)
 
-# Image data is stored in `data` attribute
-im.data  # returns the image data numpy array
 
-# Image dimension sizes can be obtained via properties:
-im.size_z  # returns the size of the Z dimension. X,Y,Z,C,T, and S supported.
+### Speed up IO and Processing with Dask Clients and Clusters
+```python
+from aicsimageio import AICSImage, dask_utils
+
+# Create a local dask cluster and client for the duration of the context manager
+with AICSImage("filename.ome.tiff") as img:
+    # do your work like normal
+    print(img.dask_data.shape)
+
+# Specify arguments for the local cluster initialization
+with AICSImage("filename.ome.tiff", dask_kwargs={"nworkers": 4}) as img:
+    # do your work like normal
+    print(img.dask_data.shape)
+
+# Connect to a dask client for the duration of the context manager
+with AICSImage("filename.ome.tiff", dask_kwargs={"address": "tcp://localhost:12345"}) as img:
+    # do your work like normal
+    print(img.dask_data.shape)
+
+# Or spawn a local cluster and / or connect to a client outside of a context manager
+# This uses the same "address" and dask kwargs as above
+# If you pass an address in, it will create and shutdown the client and no cluster will be created.
+# Similar to AICSImage, these objects will be connected and useable for the lifespan of the context manager.
+with dask_utils.cluster_and_client() as (cluster, client):
+
+    img1 = AICSImage("1.tiff")
+    img2 = AICSImage("2.tiff")
+    img3 = AICSImage("3.tiff")
+
+    # Do your image processing work
+```
 
-# Image dimensions can also be obtained as a tuple in two ways:
-im.size("ZYX")  # returns a tuple containing the Z, Y, and X sizes only
-im.get_image_data(out_orientation="ZYX").shape  # returns same as above
+### Metadata Reading
+```python
+from aicsimageio import AICSImage
 
-# Image metadata is stored in `metadata` attribute
-im.metadata  # returns whichever metadata parser best suits the file format
+# Get an AICSImage object
+img = AICSImage("my_file.tiff")
+img.metadata  # returns the metadata object for this image type
+img.get_channel_names()  # returns a list of string channel names if found in the metadata
+```
 
-# Subsets or transposes of the image data can be requested:
-im.get_image_data(out_orientation="ZYX")  # returns a 3d data block containing only the ZYX dimensions
+### Napari Interactive Viewer
+[napari](https://github.com/Napari/napari) is a fast, interactive, multi-dimensional image viewer for python and
+it is pretty useful for imaging data that this package tends to interact with.
+```python
+from aicsimageio import AICSImage
 
+# Get an AICSImage object
+img = AICSImage("my_file.tiff")
+img.view_napari()  # launches napari GUI and viewer
 ```
 
 ## Notes
-* Image data numpy arrays are always returned as six dimensional in dimension order `STCZYX`
+* Image `data` and `dask_data` are always returned as six dimensional in dimension order `STCZYX`
 or `Scene`, `Time`, `Channel`, `Z`, `Y`, and `X`.
 * Each file format may use a different metadata parser it is dependent on the reader's implementation.
-
-## Installation
-**Stable Release:** `pip install aicsimageio`<br>
-**Development Head:** `pip install git+https://github.com/AllenCellModeling/aicsimageio.git`
-
-## Documentation
-For full package documentation please visit [allencellmodeling.github.io/aicsimageio](https://allencellmodeling.github.io/aicsimageio/index.html).
+* The `AICSImage` object will only pull the `Scene`, `Time`, `Channel`, `Z`, `Y`, `X` dimensions from the reader.
+If your file has dimensions outside of those, use the base reader classes `CziReader`, `OmeTiffReader`, `TiffReader`,
+or `DefaultReader`.
+* We make some choices for the user based off the image data during `img.view_napari`.
+If you don't want this behavior, simply pass the `img.dask_data` into `napari.view_image` instead.
 
 ## Development
 See [CONTRIBUTING.md](CONTRIBUTING.md) for information related to developing the code.

aicsimageio/__init__.py

@@ -1,5 +1,9 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
 from .aics_image import AICSImage  # noqa: F401
 from .aics_image import imread  # noqa: F401
+from .aics_image import imread_dask  # noqa: F401
 
 # Do not edit this string manually, always use bumpversion
 # Details in CONTRIBUTING.md