Skip to content

Commit 8b7236b

Browse files
ajschmidt8ajschmidt8
authored
Bifurcate Dependency Lists (rapidsai#1065)
This PR uses the [`rapids-dependency-file-generator`](https://github.com/rapidsai/dependency-file-generator/) to handle sourcing dependencies. Similarly to rapidsai/rmm#1073 and rapidsai/cudf#11674, this PR introduces a GitHub Action that enforces consistency between the new `dependencies.yaml` file and the generated conda environment for developers. Authors: - AJ Schmidt (https://github.com/ajschmidt8) Approvers: - Bradley Dice (https://github.com/bdice) - Corey J. Nolet (https://github.com/cjnolet)
1 parent fd7058c commit 8b7236b

File tree

8 files changed

+162
-120
lines changed

8 files changed

+162
-120
lines changed

‎.github/workflows/dependency-files.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
name: dependency-files
2+
3+
on:
4+
pull_request:
5+
6+
jobs:
7+
checks:
8+
secrets: inherit
9+
uses: rapidsai/shared-action-workflows/.github/workflows/checks.yaml@main
10+
with:
11+
enable_check_size: false
12+
enable_check_style: false

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -262,9 +262,9 @@ Several CMake targets can be made available by adding components in the table be
262262
### Source
263263

264264
The easiest way to build RAFT from source is to use the `build.sh` script at the root of the repository:
265-
1. Create an environment with the needed dependencies:
265+
1. Create an environment with the needed dependencies:
266266
```
267-
mamba env create --name raft_dev_env -f conda/environments/raft_dev_cuda11.5.yml
267+
mamba env create --name raft_dev_env -f conda/environments/all_cuda-115_arch-x86_64.yaml
268268
mamba activate raft_dev_env
269269
```
270270
```

‎ci/release/update-version.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ sed_runner 's/'"branch-.*\/RAPIDS.cmake"'/'"branch-${NEXT_SHORT_TAG}\/RAPIDS.cma
4141
sed_runner 's/version = .*/version = '"'${NEXT_SHORT_TAG}'"'/g' docs/source/conf.py
4242
sed_runner 's/release = .*/release = '"'${NEXT_FULL_TAG}'"'/g' docs/source/conf.py
4343

44-
for FILE in conda/environments/*.yml; do
44+
for FILE in conda/environments/*.yaml dependencies.yaml; do
4545
sed_runner "s/dask-cuda=${CURRENT_SHORT_TAG}/dask-cuda=${NEXT_SHORT_TAG}/g" ${FILE};
4646
sed_runner "s/rapids-build-env=${CURRENT_SHORT_TAG}/rapids-build-env=${NEXT_SHORT_TAG}/g" ${FILE};
4747
sed_runner "s/rapids-doc-env=${CURRENT_SHORT_TAG}/rapids-doc-env=${NEXT_SHORT_TAG}/g" ${FILE};

conda/environments/raft_dev_cuda11.5.yml renamed to conda/environments/all_cuda-115_arch-x86_64.yaml

Lines changed: 22 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,38 @@
1-
name: raft_dev
1+
# This file is generated by `rapids-dependency-file-generator`.
2+
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
23
channels:
34
- rapidsai
45
- rapidsai-nightly
56
- dask/label/dev
67
- conda-forge
78
- nvidia
89
dependencies:
10+
- breathe
911
- c-compiler
10-
- cxx-compiler
11-
- cudatoolkit=11.5
12-
- cuda-python >=11.7.1,<12.0
13-
- ninja
14-
- clang=11.1.0
1512
- clang-tools=11.1.0
16-
- cython>=0.29,<0.30
13+
- clang=11.1.0
1714
- cmake>=3.23.1,!=3.25.0
15+
- cuda-python >=11.7.1,<12.0
16+
- cudatoolkit=11.5
17+
- cxx-compiler
18+
- cython>=0.29,<0.30
19+
- dask-cuda=23.02.*
1820
- dask==2022.11.1
1921
- distributed==2022.11.1
20-
- scikit-build>=0.13.1
22+
- doxygen>=1.8.20
23+
- faiss-proc=*=cuda
24+
- gcc_linux-64=9.*
25+
- libfaiss>=1.7.0
26+
- ninja
27+
- pytest
2128
- rapids-build-env=23.02.*
22-
- rapids-notebook-env=23.02.*
2329
- rapids-doc-env=23.02.*
30+
- rapids-notebook-env=23.02.*
2431
- rmm=23.02.*
25-
- dask-cuda=23.02.*
26-
- ucx>=1.13.0
27-
- ucx-py=0.30.*
32+
- scikit-build>=0.13.1
33+
- sphinx-markdown-tables
34+
- sysroot_linux-64==2.17
2835
- ucx-proc=*=gpu
29-
- doxygen>=1.8.20
30-
- libfaiss>=1.7.0
31-
- faiss-proc=*=cuda
32-
- ccache
33-
- pip
34-
- pip:
35-
- sphinx_markdown_tables
36-
- breathe
37-
38-
# rapids-build-env, notebook-env and doc-env are defined in
39-
# https://docs.rapids.ai/maintainers/depmgmt/
40-
41-
# To install different versions of packages contained in those meta packages,
42-
# it is recommended to remove those meta packages (without removing the actual
43-
# packages contained in the environment) first with:
44-
# conda remove --force rapids-build-env rapids-notebook-env rapids-doc-env
36+
- ucx-py=0.30.*
37+
- ucx>=1.13.0
38+
name: all_cuda-115_arch-x86_64

‎conda/environments/raft_dev_cuda11.2.yml‎

Lines changed: 0 additions & 44 deletions
This file was deleted.

‎conda/environments/raft_dev_cuda11.4.yml‎

Lines changed: 0 additions & 44 deletions
This file was deleted.

‎dependencies.yaml‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Dependency list for https://github.com/rapidsai/dependency-file-generator
2+
files:
3+
all:
4+
output: conda
5+
matrix:
6+
cuda: ["11.5"]
7+
arch: [x86_64]
8+
includes:
9+
- build
10+
- cudatoolkit
11+
- develop
12+
- doc
13+
- notebook
14+
- run
15+
- test_python
16+
channels:
17+
- rapidsai
18+
- rapidsai-nightly
19+
- dask/label/dev
20+
- conda-forge
21+
- nvidia
22+
dependencies:
23+
build:
24+
common:
25+
- output_types: [conda, requirements]
26+
packages:
27+
- cmake>=3.23.1,!=3.25.0
28+
- cuda-python >=11.7.1,<12.0
29+
- cython>=0.29,<0.30
30+
- ninja
31+
- scikit-build>=0.13.1
32+
- output_types: [conda]
33+
packages:
34+
- c-compiler
35+
- cxx-compiler
36+
- rapids-build-env=23.02.*
37+
specific:
38+
- output_types: conda
39+
matrices:
40+
- matrix:
41+
arch: x86_64
42+
packages:
43+
- gcc_linux-64=9.*
44+
- sysroot_linux-64==2.17
45+
- matrix:
46+
arch: aarch64
47+
packages:
48+
- gcc_linux-aarch64=9.*
49+
- sysroot_linux-aarch64==2.17
50+
develop:
51+
common:
52+
- output_types: [conda, requirements]
53+
packages:
54+
- clang=11.1.0
55+
- output_types: [conda]
56+
packages:
57+
- clang-tools=11.1.0
58+
cudatoolkit:
59+
specific:
60+
- output_types: conda
61+
matrices:
62+
- matrix:
63+
cuda: "11.2"
64+
packages:
65+
- cudatoolkit=11.2
66+
- matrix:
67+
cuda: "11.4"
68+
packages:
69+
- cudatoolkit=11.4
70+
- matrix:
71+
cuda: "11.5"
72+
packages:
73+
- cudatoolkit=11.5
74+
doc:
75+
common:
76+
- output_types: [conda, requirements]
77+
packages:
78+
- breathe
79+
- output_types: [requirements]
80+
packages:
81+
- sphinx_markdown_tables
82+
- output_types: [conda]
83+
packages:
84+
- doxygen>=1.8.20
85+
- rapids-doc-env=23.02.*
86+
- sphinx-markdown-tables
87+
notebook:
88+
common:
89+
- output_types: [conda]
90+
packages:
91+
- rapids-notebook-env=23.02.*
92+
py_version:
93+
specific:
94+
- output_types: conda
95+
matrices:
96+
- matrix:
97+
py: "3.8"
98+
packages:
99+
- python=3.8
100+
- matrix:
101+
py: "3.9"
102+
packages:
103+
- python=3.9
104+
- matrix:
105+
packages:
106+
- python>=3.8,<3.10
107+
run:
108+
common:
109+
- output_types: [conda]
110+
packages:
111+
- rmm=23.02.*
112+
- dask==2022.11.1
113+
- distributed==2022.11.1
114+
- ucx>=1.13.0
115+
- ucx-py=0.30.*
116+
- ucx-proc=*=gpu
117+
- libfaiss>=1.7.0
118+
- faiss-proc=*=cuda
119+
- dask-cuda=23.02.*
120+
test_python:
121+
common:
122+
- output_types: [conda, requirements]
123+
packages:
124+
- pytest

‎docs/source/build.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ Currently, shared libraries are provided for the `libraft-nn` and `libraft-dista
145145
Conda environment scripts are provided for installing the necessary dependencies for building and using the Python APIs. It is preferred to use `mamba`, as it provides significant speedup over `conda`. In addition you will have to manually install `nvcc` as it will not be installed as part of the conda environment. The following example will install create and install dependencies for a CUDA 11.5 conda environment:
146146

147147
```bash
148-
mamba env create --name raft_env_name -f conda/environments/raft_dev_cuda11.5.yml
148+
mamba env create --name raft_env_name -f conda/environments/all_cuda-115_arch-x86_64.yaml
149149
mamba activate raft_env_name
150150
```
151151

0 commit comments

Comments
 (0)