HAS_GPU does not reflect CUDA VISIBLE DEVICE Settings

[bschifferer]

We check, if GPUs are available in following line:

core/merlin/core/compat.py

Line 32 in 5dbafa6

HAS_GPU = nvml.device_get_count() > 0

If my machine has GPUs, but I want to run in CPU-only mode by setting CUDA_VISIBLE_DEVICE, the Merlin Library still has HAS_GPU=True

Example:
export CUDA_VISIBLE_DEVICES='';python -c 'from dask.distributed.diagnostics import nvml; print(nvml.device_get_count())'

Returns 2 for 2x GPUs. But I set CUDA_VISIBLE_DEVICES to empty

[bschifferer]

@radekosmulski FYI

[oliverholworthy]

At one point we were using cuda to set this global variable which did account for CUDA_VISIBLE_DEVICES. However, calling a cuda function in the global scope caused problems with dask and we switched to using NVML #145 which tells you how many devices are available on the machine, but doesn't know anything about CUDA_VISIBLE_DEVICES

[oliverholworthy]

I've opened a PR #221 that checks the CUDA_VISIBLE_DEVICES environment variable and handles this case where it is set to an empty string