Dask-image ndfilters not working with cupy version > 10.6

[JorgeGtz]

Describe the issue:
I originally posted this in Dask Discouse. I get an error (see example below) when I use dask_image.ndfilters with the versions 2023.03.0 of dask-image and 12.0.0 of cupy. I do not get the error when switching the version of cupy to 10.6.0. It looks to me like the structure of the package changed after that version.

Minimal Complete Verifiable Example (from https://genevievebuckley.github.io/dask-image-talk-2020/):

import cupy as cp
import dask.array as da
import dask_image.ndfilters

s = (10, 10)
a = da.from_array(cp.arange(int(cp.prod(cp.array(s)))).reshape(s), chunks=5)  # <- cupy dask array
w = cp.ones(a.ndim * (3,))  # <- cupy dask array
result = dask_image.ndfilters.convolve(a, w)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 4
      2 a = da.from_array(cp.arange(int(cp.prod(cp.array(s)))).reshape(s), chunks=5)  # <- cupy dask array
      3 w = cp.ones(a.ndim * (3,))  # <- cupy dask array
----> 4 result = dask_image.ndfilters.convolve(a, w)

File ~/miniconda3/envs/question/lib/python3.11/site-packages/dask_image/ndfilters/_conv.py:28, in convolve(image, weights, mode, cval, origin)
     24     boundary = "periodic"
     25     mode = "constant"
     27 result = image.map_overlap(
---> 28     dispatch_convolve(image),
     29     depth=depth,
     30     boundary=boundary,
     31     dtype=image.dtype,
     32     meta=image._meta,
     33     weights=weights,
     34     mode=mode,
     35     cval=cval,
     36     origin=origin,
     37 )
     39 return result

File ~/miniconda3/envs/question/lib/python3.11/site-packages/dask_image/dispatch/_dispatcher.py:24, in Dispatcher.__call__(self, arg, *args, **kwargs)
     22 datatype = get_type(arg)
     23 meth = self.dispatch(datatype)
---> 24 return meth(arg, *args, **kwargs)

File ~/miniconda3/envs/question/lib/python3.11/site-packages/dask_image/dispatch/_dispatch_ndfilters.py:59, in register_cupy_convolve.<locals>.cupy_convolve(*args, **kwargs)
     57 @dispatch_convolve.register(cupy.ndarray)
     58 def cupy_convolve(*args, **kwargs):
---> 59     return cupyx.scipy.ndimage.filters.convolve

AttributeError: module 'cupyx.scipy.ndimage' has no attribute 'filters'

Anything else we need to know?:

Environment that reproduces the error:

• Dask version: 2023.7.0
• Dask-image version: 2023.03.0
• Cupy version: 12.0.0
• Python version: 3.11.4
• Operating System: Ubuntu 20.04.3 LTS
• Install method (conda, pip, source): conda

Environment that runs without error:

• Dask version: 2023.7.0
• Dask-image version: 2023.03.0
• Cupy version: 10.6.0
• Python version: 3.10.12
• Operating System: Ubuntu 20.04.3 LTS
• Install method (conda, pip, source): conda

[m-albert]

Hi @JorgeGtz, I can reproduce the error using your minimal example.

Indeed it seems that from cupy version 11.0 on, cupyx.scipy.ndimage.filters is not available anymore (however all relevant functions are available in the cupyx.scipy.ndimage namespace directly. They seem to have made filters private in this commit.

I can't find a deprecation warning or a notice in the changelog, and from the commit message it seems there was a confusion. I'll open an issue at cupy. Update: scipy deprecates filters in v2 in favor of the ndimage namespace`

So this means on the dask-image side all is good in principle. I'm not sure whether in such a case it makes sense to adapt to cupyx, since all versions 11.0-13.0 are affected. @jakirkham @GenevieveBuckley what do you think?

@JorgeGtz Until this is fixed and if you depend on using cupy 11-13, there'd be the option to monkey patch cupyx. For this, you can include sth like these lines anywhere before the dask_image.ndfilters.convolve call:

import cupyx.scipy.ndimage
cupyx.scipy.ndimage.filters = cupyx.scipy.ndimage

(less dirty would be to only explicitely make available the needed functions)

[JorgeGtz]

Thank you @m-albert that will work for me for now.

[m-albert]

I'm not sure whether in such a case it makes sense to adapt to cupyx, since all versions 11.0-13.0 are affected

Probably dask-image should depend on cupy <= 10.6.

I wonder why this wasn't caught by the GPU CI.

[m-albert]

I was wrong: scipy.ndimage.filters is deprecated and the filter functions will disappear in scipy v2.0.0. Actually, as far as I understand all scipy.ndimage submodules will be deprecated in version 2.

cupy seems to have performed this change earlier than scipy, making it urgent especially for dask_image.dispatch to adapt to this.

[jakirkham]

Thanks for report Jorge! 🙏

Also thanks for digging in here Marvin! 🙏

cc @rgommers (in case you have additional advice here about the SciPy filter changes)

[rgommers]

cc @rgommers (in case you have additional advice here about the SciPy filter changes)

Nothing very insightful, just drop .filters everywhere and things will be fixed. It was never really a public namespace; all functions are also present in the ndimage namespace.

[GenevieveBuckley]

Closed by #325

There are plans to release a new dask-image version including this fix soon.
If you need to use this before that happens, you can install dask-image from the main github branch using pip install git+https://github.com/dask/dask-image.git