Skip to content

Fix rolling operation with dask and bottleneck#2942

Closed
fujiisoup wants to merge 2 commits into
pydata:masterfrom
fujiisoup:fix_2940
Closed

Fix rolling operation with dask and bottleneck#2942
fujiisoup wants to merge 2 commits into
pydata:masterfrom
fujiisoup:fix_2940

Conversation

@fujiisoup

Copy link
Copy Markdown
Member

Fix for #2940
It looks that there was a bug in the previous logic, but I am not sure why it was working...

@fujiisoup fujiisoup changed the title Fix: 2940 Fix rolling operation with dask and bottleneck May 6, 2019
@shoyer

shoyer commented May 7, 2019

Copy link
Copy Markdown
Member

@fujiisoup, thanks for looking into this! Unfortunately these tests still seem to still be failing on Travis.

@dcherian dcherian mentioned this pull request May 21, 2019
15 tasks
@max-sixty

max-sixty commented Jun 15, 2019

Copy link
Copy Markdown
Collaborator

@fujiisoup I mistakenly removed the dask-master tests (I moved the dask test from the test matrix to the ignored failures, rather than adding it to ignored failures)

Here's a tiny patch which restores it, if you want to add this to your PR:

EDIT: moved to separate PR

diff --git a/.travis.yml b/.travis.yml
index 913c5e1c..8249b134 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -19,12 +19,12 @@ matrix:
   - env: CONDA_ENV=py36-pandas-dev
   - env: CONDA_ENV=py36-rasterio
   - env: CONDA_ENV=py36-zarr-dev
+  - env: CONDA_ENV=py36-dask-dev
   - env: CONDA_ENV=docs
   - env: CONDA_ENV=lint
   - env: CONDA_ENV=py36-hypothesis
 
   allow_failures:
-  - env: CONDA_ENV=py36-dask-dev
   - env:
     - CONDA_ENV=py36
     - EXTRA_FLAGS="--run-flaky --run-network-tests"

@zbarry

zbarry commented Jun 18, 2019

Copy link
Copy Markdown

Hey just wanted to chime in and say it appears that commit fujiisoup@098daf3 is still losing chunking for me as far as I can tell when running it with dask distributed / dask jobqueue. I can do some extended testing & look further into this if people have some suggestions for how to go about it.

@fujiisoup

@shoyer

shoyer commented Jun 20, 2019

Copy link
Copy Markdown
Member

I did a little more looking into this, and I unfortunately I think the necessary fix is a little more involved. For now, I think we should disable rolling() with dask arrays to avoid computing incorrect results: #2940 (comment)

@fujiisoup

Copy link
Copy Markdown
Member Author

Hi. Sorry for my slow responce...

lf I understood correctly, it happens with dask and bottleneck.
So, probably we can skip using bottleneck if the backend array is dask array?

@shoyer

shoyer commented Jun 23, 2019

Copy link
Copy Markdown
Member

@fujiisoup yes, indeed, that's a much better temporary fix! See #3040.

@shoyer

shoyer commented Jun 30, 2019

Copy link
Copy Markdown
Member

This is subsumed by #3040

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_rolling_wrapped_dask is failing with dask-master

4 participants