Python API updates to enable explicit control of internal `graph_t` creation and deletion by rlratzel · Pull Request #2023 · rapidsai/cugraph

rlratzel · 2022-01-20T15:35:18Z

Python API updates to enable explicit control of internal graph_t creation and deletion. These changes will allow users who need more control over how the internal graph_t object is managed to have additional APIs to do so. Current cugraph algos construct the appropriate graph_t at the C++ boundary, run the algo, and destroy the object each time. This is more convenient and safer in some cases since it does not require the user to understand the necessary data format requirements for each algo (CSC, CSR, etc.), but it adds overhead to each algo call. If a user knows which algos will be called ahead of time and the relevant graph creation options to use, they can reuse the underlying graph_t object and eliminate the redundant creation/deletion expense. These APIs also allow for benchmarks to measure only the algo run time without the initial graph creation time.

These changes are grouped into a category of enhancements sometimes referred to as "expert mode"

Changes here include:

Updates to pylibcugraph to add the APIs from the libcugraph_c library for graph_t management and calling Pagerank and SSSP
Addition of the experimental namespace to pylibcugraph, which is where some of the new APIs will be until we finalize decisions on names, signatures, etc.
~~Updates to cugraph Graph classes, Pagerank, and SSSP algos to call in to pylibcugraph~~

…can be imported.

@experimental

…t probably is not necessary given the dir name, updated @experimental decorator to not error on extension types (from cython), removed unnecessary gpu_graph_data class and replaced with cython SGGraph class.

…e callable, removed __init__.py file contents from connectivity in favor of just exposing functions in the top-level __init__.py.

…d an experimental/__init__.py file.

…y copies), added additional types for C API structs, added tests.

…2-initialexpertmode

codecov-commenter · 2022-01-20T18:50:30Z

Codecov Report

Merging #2023 (22667c6) into branch-22.02 (63efaea) will increase coverage by 0.93%.
The diff coverage is 91.12%.

@@               Coverage Diff                @@
##           branch-22.02    #2023      +/-   ##
================================================
+ Coverage         71.93%   72.87%   +0.93%     
================================================
  Files               146      150       +4     
  Lines              9264     9669     +405     
================================================
+ Hits               6664     7046     +382     
- Misses             2600     2623      +23

Impacted Files	Coverage Δ
python/pylibcugraph/pylibcugraph/tests/utils.py	`34.61% <15.78%> (-52.89%)`	⬇️
...n/pylibcugraph/pylibcugraph/utilities/api_tools.py	`95.23% <95.23%> (ø)`
python/pylibcugraph/pylibcugraph/tests/conftest.py	`96.42% <96.36%> (-3.58%)`	⬇️
python/pylibcugraph/pylibcugraph/__init__.py	`100.00% <100.00%> (ø)`
...pylibcugraph/pylibcugraph/experimental/__init__.py	`100.00% <100.00%> (ø)`
...ph/pylibcugraph/tests/test_connected_components.py	`100.00% <100.00%> (ø)`
...n/pylibcugraph/pylibcugraph/tests/test_graph_sg.py	`100.00% <100.00%> (ø)`
...n/pylibcugraph/pylibcugraph/tests/test_pagerank.py	`100.00% <100.00%> (ø)`
...ython/pylibcugraph/pylibcugraph/tests/test_sssp.py	`100.00% <100.00%> (ø)`
python/cugraph/cugraph/tests/test_ecg.py	`100.00% <0.00%> (ø)`
... and 60 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 63efaea...22667c6. Read the comment docs.

… of cugraph_c with a leading underscore, moved test fixtures to conftest.py.

…2-initialexpertmode

…sociated expected outputs, more updates to pagerank function and test.

…2-initialexpertmode

…e set correctly, fixed flake8 error.

…to branch-22.02-initialexpertmode

ChuckHastings

A few minor observations. I think it's fine to go forward, but we probably should address these early in 22.04

python/pylibcugraph/pylibcugraph/graphs.pyx

python/pylibcugraph/pylibcugraph/pagerank.pyx

python/pylibcugraph/pylibcugraph/sssp.pyx

cpp/src/c_api/graph_mg.cpp

seunghwak · 2022-01-26T18:53:50Z

cpp/src/c_api/graph_sg.cpp

+                       c_api::cugraph_type_erased_device_array_view_t const* weights,
                       bool_t renumber,
                       bool_t check,
                       data_type_id_t edge_type)


So, if we have 'edge_type', shouldn't there be 'vertex_type' as well?

Vertex_type is embedded in the type erased array. Edge type cannot be deduced from the other parameters.

cpp/src/c_api/graph_sg.cpp

cpp/tests/c_api/bfs_test.c

python/pylibcugraph/pylibcugraph/_cugraph_c/graph.pxd

python/pylibcugraph/pylibcugraph/graphs.pyx

python/pylibcugraph/pylibcugraph/pagerank.pyx

python/pylibcugraph/pylibcugraph/tests/conftest.py

…st coverage.

seunghwak

Approving (assuming FIXMEs will be addressed quickly in the next release).

BradReesWork · 2022-01-26T23:17:05Z

@gpucibot merge

…reation and deletion (rapidsai#2023) Python API updates to enable explicit control of internal `graph_t` creation and deletion. These changes will allow users who need more control over how the internal `graph_t` object is managed to have additional APIs to do so. Current cugraph algos construct the appropriate `graph_t` at the C++ boundary, run the algo, and destroy the object each time. This is more convenient and safer in some cases since it does not require the user to understand the necessary data format requirements for each algo (CSC, CSR, etc.), but it adds overhead to each algo call. If a user knows which algos will be called ahead of time and the relevant graph creation options to use, they can reuse the underlying `graph_t` object and eliminate the redundant creation/deletion expense. These APIs also allow for benchmarks to measure only the algo run time without the initial graph creation time. These changes are grouped into a category of enhancements sometimes referred to as "expert mode" Changes here include: * Updates to `pylibcugraph` to add the APIs from the `libcugraph_c` library for `graph_t` management and calling Pagerank and SSSP * Addition of the `experimental` namespace to `pylibcugraph`, which is where some of the new APIs will be until we finalize decisions on names, signatures, etc. * <s>Updates to cugraph Graph classes, Pagerank, and SSSP algos to call in to `pylibcugraph`</s> Authors: - Rick Ratzel (https://github.com/rlratzel) - Chuck Hastings (https://github.com/ChuckHastings) - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Seunghwa Kang (https://github.com/seunghwak) - Brad Rees (https://github.com/BradReesWork) URL: rapidsai#2023

rlratzel added 9 commits January 17, 2022 00:46

Start of GPUGraphData class and test, added 'experimental' decorator.

55ecf5c

Initial pylibcugraph cython files for cugraph_c API, still WIP.

9814793

pylibcugraph C API cython code cythonizes and compiles, new bindings …

ca784da

…can be imported.

Added initial RAII-style cdef classes (not building yet).

56e189f

Renamed files under _cugraph_c to drop the leading underscore since i…

f3d2c49

…t probably is not necessary given the dir name, updated @experimental decorator to not error on extension types (from cython), removed unnecessary gpu_graph_data class and replaced with cython SGGraph class.

Refactored experimental() to use a separate function for returning th…

2ea5408

…e callable, removed __init__.py file contents from connectivity in favor of just exposing functions in the top-level __init__.py.

Removed experimental decorator in favor of simple wrapper function an…

7420e84

…d an experimental/__init__.py file.

More progress on SGGraph (just need C API to do device-to-device arra…

da468a4

…y copies), added additional types for C API structs, added tests.

Merge remote-tracking branch 'upstream/branch-22.02' into branch-22.0…

7eded4c

…2-initialexpertmode

rlratzel added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jan 20, 2022

rlratzel added this to the 22.02 milestone Jan 20, 2022

rlratzel self-assigned this Jan 20, 2022

rlratzel added the python label Jan 20, 2022

rlratzel added 3 commits January 20, 2022 09:38

flake8 fixes.

2c2f85b

flake8 fix, updated copyright dates.

ab387c1

Merge remote-tracking branch 'upstream/branch-22.02' into branch-22.0…

e7c3fc8

…2-initialexpertmode

rlratzel added 3 commits January 20, 2022 15:10

Test refactoring.

6ad71c8

Added start of pagerank bindings, renamed all .pxd files that are 1:1…

c35740e

… of cugraph_c with a leading underscore, moved test fixtures to conftest.py.

Merge remote-tracking branch 'upstream/branch-22.02' into branch-22.0…

4595b80

…2-initialexpertmode

BradReesWork added the DRAFT label Jan 21, 2022

rlratzel and others added 5 commits January 21, 2022 10:24

Test refactoring to better handle fixtures that provide inputs and as…

4acd8aa

…sociated expected outputs, more updates to pagerank function and test.

flake8 fixes.

b340d7a

Updated copyright year.

7d5806b

add view semantic for arrays. Add copy between device arrays

1c2fc02

Merge branch 'branch-22.02' into mods_for_pylibcugraph

5c3414e

ChuckHastings mentioned this pull request Jan 22, 2022

Mods for pylibcugraph #2030

Closed

ChuckHastings and others added 2 commits January 21, 2022 20:36

update copyright

d25eefe

Updated pagerank test to include expected results, added comments.

8f8c1c9

rlratzel and others added 4 commits January 25, 2022 18:04

Merge remote-tracking branch 'upstream/branch-22.02' into branch-22.0…

a5d5daf

…2-initialexpertmode

Added documentation.

96aa3cc

bug fix

0824dca

Added docstring content, added checks to ensure unsupported params ar…

48b5c51

…e set correctly, fixed flake8 error.

rlratzel marked this pull request as ready for review January 26, 2022 15:20

rlratzel requested review from a team as code owners January 26, 2022 15:20

rlratzel removed the DRAFT label Jan 26, 2022

ChuckHastings added 2 commits January 26, 2022 11:31

Merge branch 'bug_transpose_storage' into mods_for_pylibcugraph

7dc94de

update tests to reflect bug fix for transpose

cf5b25e

ChuckHastings added the 5 - Merge After Dependencies label Jan 26, 2022

ChuckHastings and others added 2 commits January 26, 2022 12:36

add FIXME to code that is current ifdeffed out

6748668

Merge remote-tracking branch 'ChuckHastings/mods_for_pylibcugraph' in…

357d2b8

…to branch-22.02-initialexpertmode

ChuckHastings approved these changes Jan 26, 2022

View reviewed changes

python/pylibcugraph/pylibcugraph/graphs.pyx Outdated Show resolved Hide resolved

python/pylibcugraph/pylibcugraph/graphs.pyx Show resolved Hide resolved

python/pylibcugraph/pylibcugraph/pagerank.pyx Show resolved Hide resolved

python/pylibcugraph/pylibcugraph/sssp.pyx Show resolved Hide resolved

ChuckHastings removed the 5 - Merge After Dependencies label Jan 26, 2022

seunghwak reviewed Jan 26, 2022

View reviewed changes

rlratzel added 5 commits January 26, 2022 14:01

Added FIXMEs based on review feedback, fixed typo in docstring.

d2b9176

Fixed inconsistency in param name/doc string, added FIXME for more te…

60549ba

…st coverage.

Added FIXMEs

404f3fc

Fixed param name after changing it in the API.

f761712

Minor update to docstring example to match param update.

22667c6

seunghwak approved these changes Jan 26, 2022

View reviewed changes

BradReesWork approved these changes Jan 26, 2022

View reviewed changes

rapids-bot bot merged commit aecd54a into rapidsai:branch-22.02 Jan 27, 2022

rlratzel deleted the branch-22.02-initialexpertmode branch June 17, 2022 00:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python API updates to enable explicit control of internal `graph_t` creation and deletion#2023

Python API updates to enable explicit control of internal `graph_t` creation and deletion#2023
rapids-bot[bot] merged 55 commits intorapidsai:branch-22.02from
rlratzel:branch-22.02-initialexpertmode

rlratzel commented Jan 20, 2022 •

edited

Loading

Uh oh!

codecov-commenter commented Jan 20, 2022 •

edited

Loading

Uh oh!

ChuckHastings left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seunghwak Jan 26, 2022

Uh oh!

ChuckHastings Jan 26, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seunghwak left a comment

Uh oh!

BradReesWork commented Jan 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

rlratzel commented Jan 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jan 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ChuckHastings left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seunghwak Jan 26, 2022

Choose a reason for hiding this comment

Uh oh!

ChuckHastings Jan 26, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seunghwak left a comment

Choose a reason for hiding this comment

Uh oh!

BradReesWork commented Jan 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rlratzel commented Jan 20, 2022 •

edited

Loading

codecov-commenter commented Jan 20, 2022 •

edited

Loading