Skip to content

Commit 79f86ab

Browse files
BradReesWorkBradReesWork
authored
Adds setup.py files and conda recipes for cugraph-service (#2862)
Closes rapidsai/GaaS#20 Closes rapidsai/GaaS#19 For cugraph-service - Adding python setup - Adding build options to the build.sh script - creating the build to conda packages Authors: - Brad Rees (https://github.com/BradReesWork) - Rick Ratzel (https://github.com/rlratzel) Approvers: - Ray Douglass (https://github.com/raydouglass) - Rick Ratzel (https://github.com/rlratzel) - Joseph Nke (https://github.com/jnke2016) URL: #2862
1 parent a950bd9 commit 79f86ab

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+5194
-89
lines changed

‎build.sh‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,16 +20,17 @@ REPODIR=$(cd $(dirname $0); pwd)
2020
LIBCUGRAPH_BUILD_DIR=${LIBCUGRAPH_BUILD_DIR:=${REPODIR}/cpp/build}
2121
LIBCUGRAPH_ETL_BUILD_DIR=${LIBCUGRAPH_ETL_BUILD_DIR:=${REPODIR}/cpp/libcugraph_etl/build}
2222

23-
VALIDARGS="clean uninstall uninstall_cmake_deps libcugraph libcugraph_etl cugraph pylibcugraph cpp-mgtests docs -v -g -n --pydevelop --allgpuarch --skip_cpp_tests --cmake_default_generator -h --help"
23+
VALIDARGS="clean uninstall uninstall_cmake_deps libcugraph libcugraph_etl cugraph cugraph-service pylibcugraph cpp-mgtests docs -v -g -n --pydevelop --allgpuarch --skip_cpp_tests --cmake_default_generator -h --help"
2424
HELP="$0 [<target> ...] [<flag> ...]
2525
where <target> is:
2626
clean - remove all existing build artifacts and configuration (start over)
2727
uninstall - uninstall libcugraph and cugraph from a prior build/install (see also -n)
2828
uninstall_cmake_deps - uninstall headers from external dependencies installed by cmake (raft, rmm, cuco, etc.) (see also -n)
2929
libcugraph - build libcugraph.so and SG test binaries
3030
libcugraph_etl - build libcugraph_etl.so and SG test binaries
31-
cugraph - build the cugraph Python package
3231
pylibcugraph - build the pylibcugraph Python package
32+
cugraph - build the cugraph Python package
33+
cugraph-service - build the cugraph-service_client and cugraph-service_server Python package
3334
cpp-mgtests - build libcugraph and libcugraph_etl MG tests. Builds MPI communicator, adding MPI as a dependency.
3435
docs - build the docs
3536
and <flag> is:
@@ -42,7 +43,7 @@ HELP="$0 [<target> ...] [<flag> ...]
4243
--cmake_default_generator - use the default cmake generator instead of ninja
4344
-h - print this text
4445
45-
default action (no args) is to build and install 'libcugraph' then 'libcugraph_etl' then 'pylibcugraph' then 'cugraph' targets
46+
default action (no args) is to build and install 'libcugraph' then 'libcugraph_etl' then 'pylibcugraph' then 'cugraph' then 'cugraph-service' targets
4647
4748
libcugraph build dir is: ${LIBCUGRAPH_BUILD_DIR}
4849
@@ -74,7 +75,7 @@ function hasArg {
7475
}
7576

7677
function buildAll {
77-
(( ${NUMARGS} == 0 )) || !(echo " ${ARGS} " | grep -q " [^-]\+ ")
78+
(( ${NUMARGS} == 0 )) || !(echo " ${ARGS} " | grep -q " [^-][a-zA-Z0-9\_\-]\+ ")
7879
}
7980

8081
if hasArg -h || hasArg --help; then
@@ -265,6 +266,16 @@ if buildAll || hasArg cugraph; then
265266
fi
266267
fi
267268

269+
# Install the cugraph-service-client and cugraph-service-server Python packages
270+
if buildAll || hasArg cugraph-service; then
271+
if [[ ${INSTALL_TARGET} != "" ]]; then
272+
cd ${REPODIR}/python/cugraph-service/client
273+
python setup.py ${PYTHON_INSTALL}
274+
cd ${REPODIR}/python/cugraph-service/server
275+
python setup.py ${PYTHON_INSTALL}
276+
fi
277+
fi
278+
268279
# Build the docs
269280
if hasArg docs; then
270281
if [ ! -d ${LIBCUGRAPH_BUILD_DIR} ]; then

‎ci/checks/style.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ fi
6262

6363
# Check for copyright headers in the files modified currently
6464
#COPYRIGHT=`env PYTHONPATH=ci/utils python ci/checks/copyright.py cpp python benchmarks ci 2>&1`
65-
COPYRIGHT=`env PYTHONPATH=ci/utils python ci/checks/copyright.py --git-modified-only 2>&1`
65+
COPYRIGHT=`env PYTHONPATH=ci/utils python ci/checks/copyright.py --git-modified-only --exclude=".*/versioneer.py" 2>&1`
6666
CR_RETVAL=$?
6767
ERRORCODE=$((ERRORCODE | ${CR_RETVAL}))
6868

‎ci/cpu/build.sh‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,13 +107,21 @@ else
107107
fi
108108

109109
if [ "$BUILD_CUGRAPH" == "1" ]; then
110-
gpuci_logger "Building conda packages for pylibcugraph and cugraph"
110+
gpuci_logger "Building conda packages for pylibcugraph, cugraph, and cugraph-service"
111111
if [[ -z "$PROJECT_FLASH" || "$PROJECT_FLASH" == "0" ]]; then
112+
gpuci_logger "pylibcugraph"
112113
gpuci_conda_retry mambabuild --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/pylibcugraph --python=$PYTHON
114+
gpuci_logger "cugraph"
113115
gpuci_conda_retry mambabuild --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/cugraph --python=$PYTHON
116+
gpuci_logger "cugraph-service"
117+
gpuci_conda_retry mambabuild --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/cugraph-service --python=$PYTHON
114118
else
119+
gpuci_logger "pylibcugraph"
115120
gpuci_conda_retry mambabuild --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/pylibcugraph -c ${CONDA_LOCAL_CHANNEL} --dirty --no-remove-work-dir --python=$PYTHON
121+
gpuci_logger "cugraph"
116122
gpuci_conda_retry mambabuild --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/cugraph -c ${CONDA_LOCAL_CHANNEL} --dirty --no-remove-work-dir --python=$PYTHON
123+
gpuci_logger "cugraph-service"
124+
gpuci_conda_retry mambabuild --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/cugraph-service -c ${CONDA_LOCAL_CHANNEL} --dirty --no-remove-work-dir --python=$PYTHON
117125
mkdir -p ${CONDA_BLD_DIR}/cugraph
118126
mv ${CONDA_BLD_DIR}/work ${CONDA_BLD_DIR}/cugraph/work
119127
fi

‎ci/cpu/upload.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,4 +45,7 @@ if [[ "$BUILD_CUGRAPH" == "1" ]]; then
4545
test -e ${CUGRAPH_FILE}
4646
echo "Upload cugraph file: ${CUGRAPH_FILE}"
4747
gpuci_retry anaconda -t ${MY_UPLOAD_KEY} upload -u ${CONDA_USERNAME:-rapidsai} ${LABEL_OPTION} --skip-existing ${CUGRAPH_FILE} --no-progress
48+
CUGRAPH_SERVICE_FILES=$(conda build --croot ${CONDA_BLD_DIR} conda/recipes/cugraph-service --python=$PYTHON --output)
49+
echo "Upload cugraph-server files: ${CUGRAPH_SERVICE_FILES}"
50+
gpuci_retry anaconda -t ${MY_UPLOAD_KEY} upload -u ${CONDA_USERNAME:-rapidsai} ${LABEL_OPTION} --skip-existing --no-progress ${CUGRAPH_SERVICE_FILES}
4851
fi

‎ci/gpu/build.sh‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ conda list --show-channel-urls
114114

115115
if [[ -z "$PROJECT_FLASH" || "$PROJECT_FLASH" == "0" ]]; then
116116
gpuci_logger "Build from source"
117-
$WORKSPACE/build.sh -v clean libcugraph pylibcugraph cugraph
117+
$WORKSPACE/build.sh -v clean libcugraph pylibcugraph cugraph cugraph-service-server cugraph-service-client
118118
else
119119
gpuci_logger "Installing libcugraph-tests"
120120
gpuci_mamba_retry install -c ${CONDA_ARTIFACT_PATH} libcugraph libcugraph_etl libcugraph-tests
@@ -125,11 +125,15 @@ else
125125
gpuci_logger "Building and installing pylibcugraph and cugraph..."
126126
export CONDA_BLD_DIR="${WORKSPACE}/.conda-bld"
127127
export VERSION_SUFFIX=""
128+
gpuci_logger "pylibcugraph"
128129
gpuci_conda_retry mambabuild conda/recipes/pylibcugraph --no-build-id --croot ${CONDA_BLD_DIR} -c ${CONDA_ARTIFACT_PATH} --python=${PYTHON}
130+
gpuci_logger "cugraph"
129131
gpuci_conda_retry mambabuild conda/recipes/cugraph --no-build-id --croot ${CONDA_BLD_DIR} -c ${CONDA_ARTIFACT_PATH} --python=${PYTHON}
132+
gpuci_logger "cugraph-service"
133+
gpuci_conda_retry mambabuild conda/recipes/cugraph-service --no-build-id --croot ${CONDA_BLD_DIR} -c ${CONDA_ARTIFACT_PATH} --python=${PYTHON}
130134

131-
gpuci_logger "Installing pylibcugraph and cugraph from build / artifact dirs"
132-
gpuci_mamba_retry install -c ${CONDA_BLD_DIR} -c ${CONDA_ARTIFACT_PATH} --strict-channel-priority pylibcugraph cugraph
135+
gpuci_logger "Installing pylibcugraph, cugraph, and cugraph-service from build / artifact dirs"
136+
gpuci_mamba_retry install -c ${CONDA_BLD_DIR} -c ${CONDA_ARTIFACT_PATH} --strict-channel-priority pylibcugraph cugraph cugraph-service-server cugraph-service-client
133137
fi
134138

135139
################################################################################

‎ci/test.sh‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -87,20 +87,25 @@ fi
8787
if hasArg "--run-python-tests"; then
8888
echo "Python pytest for pylibcugraph..."
8989
cd ${CUGRAPH_ROOT}/python/pylibcugraph/pylibcugraph
90-
pytest --cache-clear --junitxml=${CUGRAPH_ROOT}/junit-pylibcugraph-pytests.xml -v --cov-config=.coveragerc --cov=pylibcugraph --cov-report=xml:${WORKSPACE}/python/pylibcugraph/pylibcugraph-coverage.xml --cov-report term --ignore=raft --benchmark-disable
90+
pytest -sv --cache-clear --junitxml=${CUGRAPH_ROOT}/junit-pylibcugraph-pytests.xml --cov-config=.coveragerc --cov=pylibcugraph --cov-report=xml:${WORKSPACE}/python/pylibcugraph/pylibcugraph-coverage.xml --cov-report term --ignore=raft --benchmark-disable
9191
echo "Ran Python pytest for pylibcugraph : return code was: $?, test script exit code is now: $EXITCODE"
9292

9393
echo "Python pytest for cuGraph (single-GPU only)..."
9494
conda list
9595
cd ${CUGRAPH_ROOT}/python/cugraph/cugraph
9696
# rmat is not tested because of MG testing
97-
pytest --cache-clear --junitxml=${CUGRAPH_ROOT}/junit-cugraph-pytests.xml -v --cov-config=.coveragerc --cov=cugraph --cov-report=xml:${WORKSPACE}/python/cugraph/cugraph-coverage.xml --cov-report term --ignore=raft --ignore=tests/mg --ignore=tests/generators --benchmark-disable
97+
pytest -sv --cache-clear --junitxml=${CUGRAPH_ROOT}/junit-cugraph-pytests.xml --cov-config=.coveragerc --cov=cugraph --cov-report=xml:${WORKSPACE}/python/cugraph/cugraph-coverage.xml --cov-report term --ignore=raft --ignore=tests/mg --ignore=tests/generators --benchmark-disable
9898
echo "Ran Python pytest for cugraph : return code was: $?, test script exit code is now: $EXITCODE"
9999

100100
echo "Python benchmarks for cuGraph (running as tests)..."
101101
cd ${CUGRAPH_ROOT}/benchmarks
102-
pytest -v -m "managedmem_on and poolallocator_on and tiny" --benchmark-disable
102+
pytest -sv -m "managedmem_on and poolallocator_on and tiny" --benchmark-disable
103103
echo "Ran Python benchmarks for cuGraph (running as tests) : return code was: $?, test script exit code is now: $EXITCODE"
104+
105+
echo "Python pytest for cugraph-service (single-GPU only)..."
106+
cd ${CUGRAPH_ROOT}/python/cugraph-service
107+
pytest -sv --cache-clear --junitxml=${CUGRAPH_ROOT}/junit-cugraph-service-pytests.xml --benchmark-disable -k "not mg" ./tests
108+
echo "Ran Python pytest for cugraph-service : return code was: $?, test script exit code is now: $EXITCODE"
104109
fi
105110

106111
echo "Test script exiting with value: $EXITCODE"

‎conda/recipes/cugraph-service/install_cugraph-service-client.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright (c) 2022, NVIDIA CORPORATION.
4+
5+
# This assumes the script is executed from the root of the repo directory
6+
7+
# The standard "./build.sh cugraph_service" cannot be used here since a custom
8+
# setup.py invocation must be done. This is required to work around the bug
9+
# with split packages described here:
10+
# https://github.com/conda/conda-build/issues/3993
11+
# This workaround was copied from this PR:
12+
# https://github.com/rapidsai/ucx-split-feedstock/pull/28
13+
14+
cd "${SRC_DIR}/python/cugraph-service/client"
15+
mkdir -p pip_cache
16+
$PYTHON -m pip install --no-index --no-deps --ignore-installed --cache-dir ./pip_cache . -vv
17+
18+
#./build.sh cugraph-service

‎conda/recipes/cugraph-service/install_cugraph-service-server.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright (c) 2022, NVIDIA CORPORATION.
4+
5+
# This assumes the script is executed from the root of the repo directory
6+
7+
# The standard "./build.sh cugraph_service" cannot be used here since a custom
8+
# setup.py invocation must be done. This is required to work around the bug
9+
# with split packages described here:
10+
# https://github.com/conda/conda-build/issues/3993
11+
# This workaround was copied from this PR:
12+
# https://github.com/rapidsai/ucx-split-feedstock/pull/28
13+
14+
cd "${SRC_DIR}/python/cugraph-service/server"
15+
mkdir -p pip_cache
16+
$PYTHON -m pip install --no-index --no-deps --ignore-installed --cache-dir ./pip_cache . -vv
17+
18+
#./build.sh cugraph-service

‎conda/recipes/cugraph-service/meta.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Copyright (c) 2018-2022, NVIDIA CORPORATION.
2+
3+
# Usage:
4+
# conda build -c nvidia -c rapidsai -c conda-forge .
5+
{% set version = environ.get('GIT_DESCRIBE_TAG', '0.0.0.dev').lstrip('v') + environ.get('VERSION_SUFFIX', '') %}
6+
{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %}
7+
8+
package:
9+
name: cugraph-service-split
10+
11+
source:
12+
git_url: ../../..
13+
14+
outputs:
15+
- name: cugraph-service-client
16+
version: {{ version }}
17+
about:
18+
home: http://rapids.ai/
19+
license: Apache-2.0
20+
license_family: APACHE
21+
license_file: ../../../LICENSE
22+
summary: cugraph-service client
23+
script: install_cugraph-service-client.sh
24+
requirements:
25+
host:
26+
- python x.x
27+
- pip
28+
# FIXME: this pin can be removed once we move to the GitHub Actions build process
29+
- setuptools<=65.2.0
30+
run:
31+
- python x.x
32+
- thriftpy2 >=0.4.15
33+
34+
- name: cugraph-service-server
35+
version: {{ version }}
36+
about:
37+
home: http://rapids.ai/
38+
license: Apache-2.0
39+
license_family: APACHE
40+
license_file: ../../../LICENSE
41+
summary: cugraph-service server
42+
script: install_cugraph-service-server.sh
43+
requirements:
44+
host:
45+
- python x.x
46+
- pip
47+
# FIXME: this pin can be removed once we move to the GitHub Actions build process
48+
- setuptools<=65.2.0
49+
run:
50+
- python x.x
51+
- thriftpy2 >=0.4.15
52+
- {{ pin_compatible('cugraph-service-client', exact=True) }}
53+
- {{ pin_compatible('cugraph', exact=True) }}
54+
- cupy >=9.5.0,<12.0.0a0
55+
- numpy
56+
- ucx-py 0.29.*
57+
- distributed >=2022.9.2
58+
- dask-cuda {{ minor_version }}.*
59+
- cudf {{ minor_version }}.*
60+
- dask-cudf {{ minor_version }}.*
61+
62+
tests: # [linux64]
63+
imports: # [linux64]
64+
- cugraph_service_server # [linux64]
65+
- cugraph_service_client # [linux64]

python/cugraph_service/README.md renamed to python/cugraph-service/README.md

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,14 @@
77

88
-----
99

10-
## Server
11-
(description)
12-
### Installing `cugraph_service_server` conda package
13-
14-
TBD
15-
16-
## Client
17-
(description)
18-
### Installing the `cugraph_service_client` conda package
19-
20-
TBD
10+
## Build & Install
11+
Build and install the client first, then the server. This is necessary because the server depends on shared modules provided by the client.
12+
```
13+
$> cd cugraph_repo/python/cugraph_service/client
14+
$> python setup.py install
15+
$> cd ../server
16+
$> python setup.py install
17+
```
2118

2219
### Example
2320
Starting a server for single-GPU-only cuGraph, using server extensions in `/my/cugraph_service/extensions`:

0 commit comments

Comments
 (0)