Support Dataset cpu-mode in environment with GPUs that have not been detected

[oliverholworthy]

• Support Dataset cpu-mode in environment with GPUs that have not been detected.

When CUDA visible devices environment variable is empty CUDA_VISIBLE_DEVICES=""

Currently you'll get an error like the following in the case where cudf has imported sucessfully while HAS_GPU is False.

AttributeError: 'NoneType' object has no attribute 'DataFrame'

• Update TensorTable Tests to run in non-GPU environment
• Adds workflow to run tests in environment where GPUs are not visible

[karlhigley]

Everything outside dispatch.py LGTM, but I have concerns about using the GPU classes there

[karlhigley]

💯

[karlhigley]

Is there a way we could put the HAS_GPU in merlin.core.compat so we don't have to repeat it in so many places?

[oliverholworthy]

are you suggesting making cp return None if HAS_GPU=False or something else?

[karlhigley]

I think so? Can we import CuPy successfully if we don't have a GPU?

[oliverholworthy]

that depends on the extent of what "don't have a GPU" means

If CuPy or cudf fail to find relevant cuda drivers they raise an ImportError which we catch in compat. and cp will be None. However, if we set CUDA_VISIBLE_DEVICES= in an environment that has the relevant cuda drivers and libs available then the import is successful, cp will have the value of the cupy module, but if you try to use it e.g. cupy.array(...), then you get a CUDARuntimeError

[oliverholworthy]

Is it possible this is a misuse of CUDA_VISIBLE_DEVICES and not a scenario we would like to support.

[karlhigley]

In the latter case, is HAS_GPU false? If so, then seems like we might be able to use it in compat

[karlhigley]

This test environment looks like it breaks some of the Dask cluster tests