ARROW-3368: [Integration/CI/Python] Add dask integration test to docker-compose setup

[kszucs]

Port dask integration testing, and refactored the hdfs one.
Multiple python hdfs tests cases are failing, nut sure why.

[codecov-io]

Codecov Report

Merging #3086 into master will decrease coverage by 0.02%.
The diff coverage is 59.45%.

@@            Coverage Diff            @@
##           master   #3086      +/-   ##
=========================================
- Coverage   86.42%   86.4%   -0.03%     
=========================================
  Files         504     504              
  Lines       69623   69655      +32     
=========================================
+ Hits        60175   60183       +8     
- Misses       9351    9371      +20     
- Partials       97     101       +4

Impacted Files	Coverage Δ
python/pyarrow/tests/test_hdfs.py	19.78% <55%> (+0.86%)	⬆️
python/pyarrow/tests/conftest.py	71.73% <64.7%> (-2.95%)	⬇️
go/arrow/math/int64_avx2_amd64.go	0% <0%> (-100%)	⬇️
go/arrow/memory/memory_avx2_amd64.go	0% <0%> (-100%)	⬇️
go/arrow/math/float64_avx2_amd64.go	0% <0%> (-100%)	⬇️
go/arrow/math/uint64_avx2_amd64.go	0% <0%> (-100%)	⬇️
go/arrow/memory/memory_amd64.go	28.57% <0%> (-14.29%)	⬇️
go/arrow/math/math_amd64.go	31.57% <0%> (-5.27%)	⬇️
go/arrow/math/float64_amd64.go	33.33% <0%> (ø)	⬆️
go/arrow/math/int64_amd64.go	33.33% <0%> (ø)	⬆️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1fd2a25...d6e98ec. Read the comment docs.

[kszucs]

AFAICS it doesn't test interoperability between dask and pyarrow, rather dask <-> pandas and pandas <-> pyarrow

[wesm]

A few comments; I am not sure anything has to be done this second about ARROW-3975 but let me know if there is another option for procuring the HDFS client dependencies

[wesm]

Oof, I missed that we were putting integration testing unrelated to the Arrow binary protocol in integration/. I don't think it makes sense to organize the tests like this.

https://issues.apache.org/jira/browse/ARROW-3975

Might make sense to put the protocol tests under format/

[kszucs]

I thought integration is the right place for 3rd party integration tests as well (I mean it's still better then dev).

[wesm]

I think we should find a better place for the cross-language protocol tests. integration/ seems find for 3rd-party integration tests, but having both in the same place seems too odd to me

[kszucs]

Running tests: https://github.com/kszucs/crossbow/branches/all?utf8=%E2%9C%93&query=build-371
Multiple Python HDFS tests will fail.

[wesm]

What's wrong with the Python tests?

[kszucs]

The hdfs filesystem opens files as closed..., I'm investigating it.

[wesm]

Needs rebase

[kszucs]

@wesm rebased, but the python tests are still failing. Resolve them in a follow-up PR to not block this PR?

[wesm]

Sure no problem. Can you open a follow up JIRA?

[kszucs]

JIRA ticket: ARROW-4053
Crossbow builds: build-384

[wesm]

+1. Thanks @kszucs! I haven't resolved the Docker problems on my machine yet but I'll take this for a spin when I can