Fix test of cudf spilling metrics

[madsbk]

The current implementation of test_cudf_metrics() sometimes fails because it doesn't wait for the worker's metrics to update. Fixed in this PR.

Is it possible to wait on a worker's metrics to update? In test_rmm_metrics() and now in test_cudf_metrics(), we use await asyncio.sleep(1), which isn't ideal.

• Tests added / passed
• Passes pre-commit run --all-files

cc. @charlesbluca

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ± 0      27 suites  ±0   9h 56m 16s ⏱️ + 12m 16s
 3 964 tests ± 0   3 851 ✅  -  2    109 💤 ±0  4 ❌ +2 
49 853 runs  +18  47 560 ✅ +22  2 289 💤  - 6  4 ❌ +2 

For more details on these failures, see this check.

Results for commit 2734b0f. ± Comparison against base commit e30a3e4.

[charlesbluca]

rerun tests

[charlesbluca]

Thanks @madsbk! Approving for now as I'd like to unblock GPU CI as soon as possible, but worth noting that it does seem like this method has issues in test_rmm_metrics that make it flaky:

https://gpuci.gpuopenanalytics.com/job/dask/job/distributed/job/prb/job/distributed-prb/8309/

[madsbk]

rerun tests

[rjzamora]

This is a good fix. CI failures are unrelated.

[quasiben]

Planning to merge later this afternoon unless there are concerns