Default to static linking of libcudart

[bdice]

Summary

• Enable static linking of libcudart by default (CUDA_STATIC_RUNTIME=ON)
• Remove cuda-cudart from conda recipe run requirements (no longer needed when statically linked)

This is part of a RAPIDS-wide effort to switch to static CUDA runtime linking. See rapidsai/build-planning#235 for tracking.

Changes

• cpp/CMakeLists.txt: Change CUDA_STATIC_RUNTIME default from OFF to ON
• conda/recipes/cudf/recipe.yaml: Remove cuda-cudart from run deps
• conda/recipes/cudf_kafka/recipe.yaml: Remove cuda-cudart from run deps

Note: Python builds already use CUDA_STATIC_RUNTIME=ON (set in python/libcudf/CMakeLists.txt).

[ttnghia]

We will also need to update spark-rapids-jni after this.

[robertmaynard]

@vyasr Can we even support stream related testing with a static cuda runtime?

[bdice]

CI is passing. Is that enough signal or do we need more? I changed this in bcfbbd1.

[robertmaynard]

If we can't even enable CUDF_BUILD_STREAMS_TEST_UTIL any more we need to remove it entirely.

@vyasr Do we still need these tests to enforce usage of stream only APIs? If so I think we will need to figure out new CI builds that use cudart=shared and stream tests but aren't part of our release

[bdice]

Marking as DO NOT MERGE. @vyasr will need to investigate what we can do to enable a separate CI job for stream testing that dynamically links to libcudart.

[jakirkham]

Previously Spark RAPIDS encountered performance issues using the static cudart

• [BUG] Pinned memory allocation within Parquet reader can be very slow #7600

Admittedly that was a long time ago and many things have changed since then

Do we know whether this is still relevant for Spark RAPIDS? cc @abellina (who debugged this issue previously)

[vyasr]

That issue looks like it was with Arrow statically linking cudart and not cudf.

[abellina]

That issue looks like it was with Arrow statically linking cudart and not cudf.

Yes at this rate, we are linking cudf with cudart statically in our spark-rapids-jni project, so we are already doing static linking ourselves. I don't see an issue here.