Update cugraph-pyg Recipe and CI Script by alexbarghi-nv · Pull Request #3288 · rapidsai/cugraph

alexbarghi-nv · 2023-02-16T18:11:42Z

Properly installs torch in test script; skips PyG tests on CUDA 11.8. Removes OGB from dependencies since it is no longer needed and created a version conflict. Properly adds PyG and PyTorch as dependencies of cugraph-pyg.

Closes #3083
Closes #3225

…cugraph into update-ci-for-pyg

VibhuJawa

LGTM . Thanks for working on this Alex. This resolves a important P0 for this release.

ajschmidt8

We might need to brainstorm on how to get this properly working in CI.

Although the tests are currently passing, I'm seeing both the cudatoolkit and cuda-toolkit packages being installed in our CI test environments now, which is not something that we want.

Additionally, the cuda-toolkit package version that is being installed doesn't match the cudatoolkit version. See the links below for an example on the 11.2.2 test job:

cudatoolkit 11.2.2 is installed (which is what we want) - https://github.com/rapidsai/cugraph/actions/runs/4200038972/jobs/7286563545#step:6:87
cuda-toolkit 11.7.1 is installed shortly after - https://github.com/rapidsai/cugraph/actions/runs/4200038972/jobs/7286563545#step:6:513

I'm not very familiar with the cugraph packages that are being changed here.

If they have dependencies that conflict with the other RAPIDS libraries, we might want to consider creating an entirely new conda environment in this test job to accommodate testing these packages.

Happy to hop on a call to chat about this further if needed.

ajschmidt8 · 2023-02-17T22:56:31Z

I want to mention that we could also move the cugraph-pyg tests to a new job that doesn't run against a full matrix if that makes more sense.

Instead, it would only test against a single CUDA version (e.g. 11.8).

To make this work, you could adapt the conda-notebook-tests job below.

cugraph/.github/workflows/pr.yaml

Lines 55 to 64 in 2cf487f

    
           conda-notebook-tests: 
        
             needs: conda-python-build 
        
             secrets: inherit 
        
             uses: rapidsai/shared-action-workflows/.github/workflows/custom-job.yaml@branch-23.04 
        
             with: 
        
               build_type: pull-request 
        
               node_type: "gpu-latest-1" 
        
               arch: "amd64" 
        
               container_image: "rapidsai/ci:latest" 
        
               run_script: "ci/test_notebooks.sh"

The only changes you would need to make are:

Adjusting the container to run in a container with an appropriate CUDA version (all of our CI image tags are here for reference)
Adjusting the script to something like ci/test_python_pyg.sh

…to update-ci-for-pyg

alexbarghi-nv · 2023-03-17T17:36:38Z

@rlratzel and @ajschmidt8 could you please re-review this PR?

VibhuJawa

LGTM

ajschmidt8

Look good!

Looks like the wrong cugraph-pyg version is getting installed though. We'll want to get that sorted out.

ajschmidt8 · 2023-03-22T17:36:45Z

ci/test_python.sh

+    rapids-print-env
+
+    rapids-mamba-retry install \
+      --channel "${CPP_CHANNEL}" \
+      --channel "${PYTHON_CHANNEL}" \
+      libcugraph \
+      pylibcugraph \
+      cugraph \
+      cugraph-pyg
+


Two quick notes:

rapids-print-env does a conda list (src). so it might be worth placing that command after all the mamba install commands.

I notice that CI is installing the wrong cugraph-pyg version (see here). This is probably from a dependency issue somewhere. It might take a bit of iterating to figure out why this is happening. I wrote some docs about reproducing CI locally here that might help. Happy to hop on a call to help or discuss this more if needed.

…cugraph into update-ci-for-pyg

rlratzel

looks good, just possibly one change request.

rlratzel · 2023-03-22T20:53:09Z

conda/recipes/cugraph-pyg/meta.yaml

+    - pytorch >=2.0
+    - cupy >=9.5.0,<12.0.0a0
+    - cugraph ={{ version }}
+    - pyg-nightly


just want to confirm that we're putting a nightly dependency in our release packages, or are we planning on changing this before code freeze? If not, can we put an upper-bound pin on this so users installing this package some time later don't pull in a possibly incompatible newer nightly version?

I just changed this now to resolve the issue of pulling the wrong version of cugraph-pyg. We will need to bind to pyg 2.3 once it is released, which is supposed to be next week.

rlratzel · 2023-03-27T14:48:46Z

/merge

Updates tests to support presence of PyTorch and PyG in the environment. Updates cugraph-pyg to include support for loading from samples on disk. Makes CuGraphStore serializable and allows creating CuGraphStore and BulkSampleLoader instances without the graph structure. Merge after #3288 Closes #3287 Closes #3176 Authors: - Alex Barghi (https://github.com/alexbarghi-nv) Approvers: - Vibhu Jawa (https://github.com/VibhuJawa) - Rick Ratzel (https://github.com/rlratzel) URL: #3289

- Adds new examples for cugraph-pyg. - Removes outdated examples. - Moves MG scripts to top-level directory. - Makes the input to `_get_vertex_groups_from_sample` a tensor instead of Series - Adds `is_sorted` arg to `_get_vertex_groups_from_sample` to skip sorting if tensor already sorted - Some fixes to `CuGraphStore` for running multi-GPU workflows Merge after #3288 - merged Merge after #3289 - merged Merge after #3382 - merged Closes #3316 Closes #3226 Closes #3072 Authors: - Alex Barghi (https://github.com/alexbarghi-nv) Approvers: - Vibhu Jawa (https://github.com/VibhuJawa) - Rick Ratzel (https://github.com/rlratzel) URL: #3326

alexbarghi-nv added 2 commits February 16, 2023 18:07

pull in files, add skip for all pyg tests

9a1ef56

skip tests on 11.8

f0ec571

alexbarghi-nv self-assigned this Feb 16, 2023

Merge branch 'branch-23.04' into update-ci-for-pyg

656f5d3

alexbarghi-nv added improvement Improvement / enhancement to an existing function breaking Breaking change labels Feb 16, 2023

alexbarghi-nv added this to the 23.04 milestone Feb 16, 2023

alexbarghi-nv mentioned this pull request Feb 16, 2023

Update cuGraph-PyG Tests and Support Loading Saved Bulk Samples #3289

Merged

alexbarghi-nv added 2 commits February 16, 2023 18:28

update skip reason

04e83c5

Merge branch 'update-ci-for-pyg' of https://github.com/alexbarghi-nv/…

3bae1ef

…cugraph into update-ci-for-pyg

VibhuJawa approved these changes Feb 16, 2023

View reviewed changes

skip mg test on sg

4510729

alexbarghi-nv marked this pull request as ready for review February 16, 2023 23:09

alexbarghi-nv requested review from a team as code owners February 16, 2023 23:09

update backend script

cd02854

ajschmidt8 requested changes Feb 17, 2023

View reviewed changes

VibhuJawa mentioned this pull request Feb 17, 2023

[IMP] Add CI to cugraph-dgl #3293

Closed

alexbarghi-nv added 3 commits March 8, 2023 16:21

remove backend

cdc7733

separate the pyg tests, use different env

2b36c64

minor formatting fix

447fdd3

alexbarghi-nv requested review from VibhuJawa, ajschmidt8 and rlratzel March 8, 2023 16:41

alexbarghi-nv and others added 5 commits March 8, 2023 18:24

Merge branch 'branch-23.04' into update-ci-for-pyg

610d482

update pytorch req to >=2.0

4637af9

fix pytorch only to version number

dd36a95

add comments to ci file

18d650f

skip testing pyg on arm

8f4e577

alexbarghi-nv and others added 2 commits March 9, 2023 17:35

skip tests on arm

cfa6331

Merge branch 'branch-23.04' into update-ci-for-pyg

731483c

alexbarghi-nv mentioned this pull request Mar 10, 2023

New PyG End-to-End Examples #3326

Merged

Merge branch 'branch-23.04' of https://github.com/rapidsai/cugraph in…

7903e36

…to update-ci-for-pyg

alexbarghi-nv added 2 commits March 21, 2023 06:55

Merge branch 'branch-23.04' into update-ci-for-pyg

d0978da

Merge branch 'branch-23.04' into update-ci-for-pyg

39a6b8d

VibhuJawa approved these changes Mar 22, 2023

View reviewed changes

ajschmidt8 reviewed Mar 22, 2023

View reviewed changes

alexbarghi-nv added 5 commits March 22, 2023 18:39

change install order

0dee9b5

remove dask logs

bacacd2

update test

a458b12

Merge branch 'update-ci-for-pyg' of https://github.com/alexbarghi-nv/…

49b97bc

…cugraph into update-ci-for-pyg

revert test order, change to depend on pyg-nightly

173e35f

rlratzel requested changes Mar 22, 2023

View reviewed changes

rlratzel approved these changes Mar 23, 2023

View reviewed changes

alexbarghi-nv added the DO NOT MERGE Hold off on merging; see PR for details label Mar 23, 2023

alexbarghi-nv and others added 5 commits March 23, 2023 16:15

bind version to pyg 2.3

c48e2e3

remove useless file

4ebc4dc

Merge branch 'branch-23.04' into update-ci-for-pyg

e557658

add deactivate

2d39e58

Merge branch 'branch-23.04' into update-ci-for-pyg

abcd225

alexbarghi-nv removed the DO NOT MERGE Hold off on merging; see PR for details label Mar 27, 2023

ajschmidt8 approved these changes Mar 27, 2023

View reviewed changes

rlratzel approved these changes Mar 27, 2023

View reviewed changes

rapids-bot bot merged commit 6f1bf0e into rapidsai:branch-23.04 Mar 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update cugraph-pyg Recipe and CI Script#3288

Update cugraph-pyg Recipe and CI Script#3288
rapids-bot[bot] merged 30 commits intorapidsai:branch-23.04from
alexbarghi-nv:update-ci-for-pyg

alexbarghi-nv commented Feb 16, 2023 •

edited

Loading

Uh oh!

VibhuJawa left a comment

Uh oh!

ajschmidt8 left a comment

Uh oh!

ajschmidt8 commented Feb 17, 2023

Uh oh!

alexbarghi-nv commented Mar 17, 2023

Uh oh!

VibhuJawa left a comment

Uh oh!

ajschmidt8 left a comment

Uh oh!

ajschmidt8 Mar 22, 2023

Uh oh!

rlratzel left a comment •

edited

Loading

Uh oh!

rlratzel Mar 22, 2023

Uh oh!

alexbarghi-nv Mar 22, 2023

Uh oh!

rlratzel commented Mar 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

alexbarghi-nv commented Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VibhuJawa left a comment

Choose a reason for hiding this comment

Uh oh!

ajschmidt8 left a comment

Choose a reason for hiding this comment

Uh oh!

ajschmidt8 commented Feb 17, 2023

Uh oh!

alexbarghi-nv commented Mar 17, 2023

Uh oh!

VibhuJawa left a comment

Choose a reason for hiding this comment

Uh oh!

ajschmidt8 left a comment

Choose a reason for hiding this comment

Uh oh!

ajschmidt8 Mar 22, 2023

Choose a reason for hiding this comment

Uh oh!

rlratzel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlratzel Mar 22, 2023

Choose a reason for hiding this comment

Uh oh!

alexbarghi-nv Mar 22, 2023

Choose a reason for hiding this comment

Uh oh!

rlratzel commented Mar 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alexbarghi-nv commented Feb 16, 2023 •

edited

Loading

rlratzel left a comment •

edited

Loading