* Add support for next cusparse release
* Fix lint
* Add switch and tune the performance
* Fix lint issue
* Fine tune the heuristics
* Fix lint issue
* Address comments
* Minor fix
* Address comments
* replace dgl PRNG with pcg32
* remove pcg submodule, add a simple implementation
* replace pcg32 with std::mt19937_64
* fix include order
* change RandomEngine to pcg32
* Remove custom pcg32 implementation, use the submodule provided by the original author.
* minor bug
* move include for linting
* include pcg for tests too
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
* Leverage hashmap to accelerate CSRSliceMatrix
* fix lint check
* use `min` in cuda_runtime.ch
* fix hash func
* add some comments and adjust the <grid,block> of the _SegmentMaskColKernel kernel
* set device and stream for thrust::for_each
* use thrust::cuda::par_nosync
Co-authored-by: Xin Yao <xiny@nvidia.com>
* adding LABOR sampling
* add ladies and pladies samplers
* fix compile error after rebase
* add reference for ladies sampler
* Improve ladies implementation.
* weighted labor sampling initial implementation draft
fix indentation and small bug in ladies script
* importance_sampling currently doesn't work with weights
* fix weighted importance sampling
* move labor example into its own folder
* lint fixes
* Improve documentation
* remove examples from the main PR
* fix linting by not using c++17 features
* fix documentation of labor_sampler.py
* update documentation for labor.py
* reformat the labor.py file with black
* fix linting errors
* replace exception use with if
* fix typo in error comment
* fixing win64 build for ci
* fixing weighted implementation, works now.
* fix bug in the weighted case and importance_sampling==0
* address part of the reviews
* remove unused code paths from cuda
* remove unused code path from cpu side
* remove extra features of labor making use of random seed.
* fix exclude_edges bug
* remove pcg and seed logic from cpu implementation, seed logic should still work for cuda.
* minor style change
* refactor CPU implementation, take out the importance_sampling probability computation into a function.
* improve CUDAWorkspaceAllocator
* refactor importance_sampling part out to a function
* minor optimization
* fix linting issue
* Revert "remove pcg and seed logic from cpu implementation, seed logic should still work for cuda."
This reverts commit c250e07ac6d7e13f57e79e8a2c2f098d777378c2.
* Revert "remove extra features of labor making use of random seed."
This reverts commit 7f99034353080308f4783f27d9a08bea343fb796.
* fix the documentation
* disable NIDs
* improve the documentation in the code
* use the stream argument in pcg32 instead of skipping ahead t times, can discard the use of hashmap now since it is faster this way.
* fix linting issue
* address another round of reviews
* further optimize CPU LABOR sampling implementation
* fix linting error
* update the comment
* reformat
* rename and rephrase comment
* fix formatting according to new linting specs
* fix compile error due to renaming, fix linting.
* lint
* rename DGLHeteroGraph to DGLGraph to match master
* replace other occurrences of DGLHeteroGraph to DGLGraph
Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>
Co-authored-by: Kaan Sancak <kaansnck@gmail.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
* add bf16 specializations
* remove SWITCH_BITS
* enable amp for bf16
* remove SWITCH_BITS for cpu kernels
* enbale bf16 based on CUDART
* fix compiling for sm<80
* fix cpu build
* enable unit tests
* update doc
* disable test for CUDA < 11.0
* address comments
* address comments
* Update from master (#4584)
* [Example][Refactor] Refactor graphsage multigpu and full-graph example (#4430)
* Add refactors for multi-gpu and full-graph example
* Fix format
* Update
* Update
* Update
* [Cleanup] Remove async_transferer (#4505)
* Remove async_transferer
* remove test
* Remove AsyncTransferer
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Xin Yao <yaox12@outlook.com>
* [Cleanup] Remove duplicate entries of CUB submodule (issue# 4395) (#4499)
* remove third_part/cub
* remove from third_party
Co-authored-by: Israt Nisa <nisisrat@amazon.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
* [Bug] Enable turn on/off libxsmm at runtime (#4455)
* enable turn on/off libxsmm at runtime by adding a global config and related API
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-194.ap-northeast-1.compute.internal>
* [Feature] Unify the cuda stream used in core library (#4480)
* Use an internal cuda stream for CopyDataFromTo
* small fix white space
* Fix to compile
* Make stream optional in copydata for compile
* fix lint issue
* Update cub functions to use internal stream
* Lint check
* Update CopyTo/CopyFrom/CopyFromTo to use internal stream
* Address comments
* Fix backward CUDA stream
* Avoid overloading CopyFromTo()
* Minor comment update
* Overload copydatafromto in cuda device api
Co-authored-by: xiny <xiny@nvidia.com>
* [Feature] Added exclude_self and output_batch to knn graph construction (Issues #4323#4316) (#4389)
* * Added "exclude_self" and "output_batch" options to knn_graph and segmented_knn_graph
* Updated out-of-date comments on remove_edges and remove_self_loop, since they now preserve batch information
* * Changed defaults on new knn_graph and segmented_knn_graph function parameters, for compatibility; pytorch/test_geometry.py was failing
* * Added test to ensure dgl.remove_self_loop function correctly updates batch information
* * Added new knn_graph and segmented_knn_graph parameters to dgl.nn.KNNGraph and dgl.nn.SegmentedKNNGraph
* * Formatting
* * Oops, I missed the one in segmented_knn_graph when I fixed the similar thing in knn_graph
* * Fixed edge case handling when invalid k specified, since it still needs to be handled consistently for tests to pass
* Fixed context of batch info, since it must match the context of the input position data for remove_self_loop to succeed
* * Fixed batch info resulting from knn_graph when output_batch is true, for case of 3D input tensor, representing multiple segments
* * Added testing of new exclude_self and output_batch parameters on knn_graph and segmented_knn_graph, and their wrappers, KNNGraph and SegmentedKNNGraph, into the test_knn_cuda test
* * Added doc comments for new parameters
* * Added correct handling for uncommon case of k or more coincident points when excluding self edges in knn_graph and segmented_knn_graph
* Added test cases for more than k coincident points
* * Updated doc comments for output_batch parameters for clarity
* * Linter formatting fixes
* * Extracted out common function for test_knn_cpu and test_knn_cuda, to add the new test cases to test_knn_cpu
* * Rewording in doc comments
* * Removed output_batch parameter from knn_graph and segmented_knn_graph, in favour of always setting the batch information, except in knn_graph if x is a 2D tensor
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* [CI] only known devs are authorized to trigger CI (#4518)
* [CI] only known devs are authorized to trigger CI
* fix if author is null
* add comments
* [Readability] Auto fix setup.py and update-version.py (#4446)
* Auto fix update-version
* Auto fix setup.py
* Auto fix update-version
* Auto fix setup.py
* [Doc] Change random.py to random_partition.py in guide on distributed partition pipeline (#4438)
* Update distributed-preprocessing.rst
* Update
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
* fix unpinning when tensoradaptor is not available (#4450)
* [Doc] fix print issue in tutorial (#4459)
* [Example][Refactor] Refactor RGCN example (#4327)
* Refactor full graph entity classification
* Refactor rgcn with sampling
* README update
* Update
* Results update
* Respect default setting of self_loop=false in entity.py
* Update
* Update README
* Update for multi-gpu
* Update
* [doc] fix invalid link in user guide (#4468)
* [Example] directional_GSN for ogbg-molpcba (#4405)
* version-1
* version-2
* version-3
* update examples/README
* Update .gitignore
* update performance in README, delete scripts
* 1st approving review
* 2nd approving review
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
* Clarify the message name, which is 'm'. (#4462)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
* [Refactor] Auto fix view.py. (#4461)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* [Example] SEAL for OGBL (#4291)
* [Example] SEAL for OGBL
* update index
* update
* fix readme typo
* add seal sampler
* modify set ops
* prefetch
* efficiency test
* update
* optimize
* fix ScatterAdd dtype issue
* update sampler style
* update
Co-authored-by: Quan Gan <coin2028@hotmail.com>
* [CI] use https instead of http (#4488)
* [BugFix] fix crash due to incorrect dtype in dgl.to_block() (#4487)
* [BugFix] fix crash due to incorrect dtype in dgl.to_block()
* fix test failure in TF
* [Feature] Make TensorAdapter Stream Aware (#4472)
* Allocate tensors in DGL's current stream
* make tensoradaptor stream-aware
* replace TAemtpy with cpu allocator
* fix typo
* try fix cpu allocation
* clean header
* redirect AllocDataSpace as well
* resolve comments
* [Build][Doc] Specify the sphinx version (#4465)
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* reformat
* reformat
* Auto fix update-version
* Auto fix setup.py
* reformat
* reformat
Co-authored-by: Ubuntu <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Chang Liu <chang.liu@utexas.edu>
Co-authored-by: Zhiteng Li <55398076+ZHITENGLI@users.noreply.github.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: rudongyu <ru_dongyu@outlook.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
* Move mock version of dgl_sparse library to DGL main repo (#4524)
* init
* Add api doc for sparse library
* support op btwn matrices with differnt sparsity
* Fixed docstring
* addresses comments
* lint check
* change keyword format to fmt
Co-authored-by: Israt Nisa <nisisrat@amazon.com>
* [DistPart] expose timeout config for process group (#4532)
* [DistPart] expose timeout config for process group
* refine code
* Update tools/distpartitioning/data_proc_pipeline.py
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* [Feature] Import PyTorch's CUDA stream management (#4503)
* add set_stream
* add .record_stream for NDArray and HeteroGraph
* refactor dgl stream Python APIs
* test record_stream
* add unit test for record stream
* use pytorch's stream
* fix lint
* fix cpu build
* address comments
* address comments
* add record stream tests for dgl.graph
* record frames and update dataloder
* add docstring
* update frame
* add backend check for record_stream
* remove CUDAThreadEntry::stream
* record stream for newly created formats
* fix bug
* fix cpp test
* fix None c_void_p to c_handle
* [examples]educe memory consumption (#4558)
* [examples]educe memory consumption
* reffine help message
* refine
* [Feature][REVIEW] Enable DGL cugaph nightly CI (#4525)
* Added cugraph nightly scripts
* Removed nvcr.io//nvidia/pytorch:22.04-py3 reference
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
* Revert "[Feature][REVIEW] Enable DGL cugaph nightly CI (#4525)" (#4563)
This reverts commit ec171c648a.
* [Misc] Add flake8 lint workflow. (#4566)
* Add pyproject.toml for autopep8.
* Add pyproject.toml for autopep8.
* Add flake8 annotation in workflow.
* remove
* add
* clean up
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* [Misc] Try use official pylint workflow. (#4568)
* polish update_version
* update pylint workflow.
* add
* revert.
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* [CI] refine stage logic (#4565)
* [CI] refine stage logic
* refine
* refine
* remove (#4570)
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* Add Pylint workflow for flake8. (#4571)
* remove
* Add pylint.
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* [Misc] Update the python version in Pylint workflow for flake8. (#4572)
* remove
* Add pylint.
* Change the python version for pylint.
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* Update pylint. (#4574)
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* [Misc] Use another workflow. (#4575)
* Update pylint.
* Use another workflow.
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* Update pylint. (#4576)
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* Update pylint.yml
* Update pylint.yml
* Delete pylint.yml
* [Misc]Add pyproject.toml for autopep8 & black. (#4543)
* Add pyproject.toml for autopep8.
* Add pyproject.toml for autopep8.
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
* [Feature] Bump DLPack to v0.7 and decouple DLPack from the core library (#4454)
* rename `DLContext` to `DGLContext`
* rename `kDLGPU` to `kDLCUDA`
* replace DLTensor with DGLArray
* fix linting
* Unify DGLType and DLDataType to DGLDataType
* Fix FFI
* rename DLDeviceType to DGLDeviceType
* decouple dlpack from the core library
* fix bug
* fix lint
* fix merge
* fix build
* address comments
* rename dl_converter to dlpack_convert
* remove redundant comments
Co-authored-by: Chang Liu <chang.liu@utexas.edu>
Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Xin Yao <yaox12@outlook.com>
Co-authored-by: Israt Nisa <neesha295@gmail.com>
Co-authored-by: Israt Nisa <nisisrat@amazon.com>
Co-authored-by: peizhou001 <110809584+peizhou001@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-194.ap-northeast-1.compute.internal>
Co-authored-by: ndickson-nvidia <99772994+ndickson-nvidia@users.noreply.github.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
Co-authored-by: Zhiteng Li <55398076+ZHITENGLI@users.noreply.github.com>
Co-authored-by: rudongyu <ru_dongyu@outlook.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
Co-authored-by: Vibhu Jawa <vibhujawa@gmail.com>
* [Deprecation] Dataset Attributes (#4546)
* Update
* CI
* CI
* Update
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
* [Example] Bug Fix (#4665)
* Update
* CI
* CI
* Update
* Update
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
* Update
Co-authored-by: Chang Liu <chang.liu@utexas.edu>
Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Xin Yao <yaox12@outlook.com>
Co-authored-by: Israt Nisa <neesha295@gmail.com>
Co-authored-by: Israt Nisa <nisisrat@amazon.com>
Co-authored-by: peizhou001 <110809584+peizhou001@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-194.ap-northeast-1.compute.internal>
Co-authored-by: ndickson-nvidia <99772994+ndickson-nvidia@users.noreply.github.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
Co-authored-by: Zhiteng Li <55398076+ZHITENGLI@users.noreply.github.com>
Co-authored-by: rudongyu <ru_dongyu@outlook.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
Co-authored-by: Vibhu Jawa <vibhujawa@gmail.com>
* add set_stream
* add .record_stream for NDArray and HeteroGraph
* refactor dgl stream Python APIs
* test record_stream
* add unit test for record stream
* use pytorch's stream
* fix lint
* fix cpu build
* address comments
* address comments
* add record stream tests for dgl.graph
* record frames and update dataloder
* add docstring
* update frame
* add backend check for record_stream
* remove CUDAThreadEntry::stream
* record stream for newly created formats
* fix bug
* fix cpp test
* fix None c_void_p to c_handle
* Use an internal cuda stream for CopyDataFromTo
* small fix white space
* Fix to compile
* Make stream optional in copydata for compile
* fix lint issue
* Update cub functions to use internal stream
* Lint check
* Update CopyTo/CopyFrom/CopyFromTo to use internal stream
* Address comments
* Fix backward CUDA stream
* Avoid overloading CopyFromTo()
* Minor comment update
* Overload copydatafromto in cuda device api
Co-authored-by: xiny <xiny@nvidia.com>
* Allocate tensors in DGL's current stream
* make tensoradaptor stream-aware
* replace TAemtpy with cpu allocator
* fix typo
* try fix cpu allocation
* clean header
* redirect AllocDataSpace as well
* resolve comments
* Add helper method for temporary affinitization of compute threads
* Rework DL affinitization as single helper
* Add example usage in benchmarks
* Fix python linter warnings
* Fix affinity helper params
* Use NUMA node 0 cores only by default
* Fix benchmarks
* Fix lint errors
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
nv-dlasalle