Bifurcate Dependency Lists [skip-gpuci]

[ajschmidt8]

Summary

This PR uses rapids-dependency-file-generator along with a new dependencies.yaml file to bifurcate the existing dependency lists in the conda/environments directory. rapids-dependency-file-generator will generate conda environment files based on the contents of the new dependencies.yaml file (see rapidsai/dependency-file-generator for more details). There is also a new GitHub Action that runs a shared workflow (which lives here) to ensure that the generated dependency files are up-to-date in each PR. This shared workflow can easily be implemented across other RAPIDS repositories.

Why is this PR Needed?

This PR is necessary to support some upcoming changes in CI. We've recently created some new, slimmer CI images (~450MB vs. the current 6.5GB) in anticipation of the move to GitHub Actions. These new images omit the large dependency packages (i.e. rapids-{build,notebook,doc}-env) from the rapidsai/integration repository, which will eventually be deprecated and deleted entirely. One of the consequences of removing the integration packages from our CI images is that commonly used test dependencies (i.e. pytest) are no longer included in the CI images by default. Therefore, we need to ensure that each repository has a list of the dependencies that are required to test its respective libraries. A similar list will also be needed in other repositories for JupyterLab Notebook dependencies (which will be utilized in our runtime end-user images), but that list does not apply to rmm.

Impacts of these Changes

The changes in this PR impact developers and CI as follows:

Developers

Developers should now update the dependencies in the dependencies.yaml file instead of the generated dependency files. If they accidentally update the generated dependency files, the new GitHub Action will fail. At this point, the developer will need to install rapids-dependency-file-generator and run it from the project's root directory. For example:

pip install rapids-dependency-file-generator
rapids-dependency-file-generator

This step can also be implemented as a pre-commit hook.

CI

The new GitHub Action CI process will use the dependencies in the test list in dependencies.yaml to create a new test environment for rmm/librmm in CI. For example:

ENV_NAME="rmm_test"

rapids-dependency-file-generator \
  --file_key "test" \
  --generate "conda" \
  --matrix "cuda=11.5;arch=$(arch)" > env.yaml
mamba env create --file env.yaml
mamba activate "$ENV_NAME"

# ... run `rmm`/`librmm` tests

Additional Information

The new GitHub Action will enforce that an all environment (under the files.all key) is included in the dependencies.yaml file and that it includes all of the dependency lists in the dependencies.conda and dependencies.conda_and_requirements sections. This is enforced in the shared workflow here. It exists to provide some standardization across the RAPIDS libraries and will eventually replace the rapids-{build,notebook,doc}-env packages in our devel end-user images.

[harrism]

Is this really a 22.08 PR?

[ajschmidt8]

Is this really a 22.08 PR?

Nope, I will re-target this PR for 22.10

[bdice]

All looks good to me. Thanks @ajschmidt8 for your hard work and many iterations!