文件历史

提交图

951 次代码提交

作者 SHA1 备注 提交日期
Rhett Ying 408eba2479 Multiple dist dl sampler (#4704)
* [Dist] enable iterate multiple dist dataloaders simultaneously

* format file

* add support for any number of dataloaders

* fix lint

* refine code
2022-10-12 15:51:45 +08:00
Hongzhi (Steve), Chen ea48ce7a37 [Misc] Black auto fix. (#4697)
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
2022-10-11 13:31:13 +08:00
Hongzhi (Steve), Chen 89a4cc4db2 [Misc] Black auto fix. (#4694)
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
2022-10-11 00:12:30 +08:00
Hongzhi (Steve), Chen 98325b1097 [Misc] Black auto fix. (#4691)
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
2022-10-10 10:53:51 +08:00
Zhiteng Li 29434e6501 [Refactor] Refactor Laplacian positional encoding transform (#4628)
* refactor the function and module of laplacian_pe

* add blank lines between design doc

* 1st approving review

* update the unittest

* fix lint issues

* fix trailing space

* update by mufei's comments and fix backend bugs

* del test file

Co-authored-by: rudongyu <ru_dongyu@outlook.com>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2022-10-08 13:03:03 +08:00
kylasa eae6ce2a4f ParMETIS wrapper script to enable ParMETIS to process chunked dataset format (#4605)
* Creating ParMETIS wrapper script to run parmetis using one script from user perspective

* Addressed all the CI comments from PR https://github.com/dmlc/dgl/pull/4529

* Addressing CI comments.

* Isort, and black changes.

* Replaced python with python3

* Replaced single quote with double quotes per suggestion.

* Removed print statement

* Addressing CI Commets.

* Addressing CI review comments.

* Addressing CI comments as per chime discussion with Rui

* CI Comments, Black and isort changes

* Align with code refactoring, black, isort and code review comments.

* Addressing CI review comments, and fixing merge issues with the master branch

* Updated with proper unit test skip decorator
2022-10-03 12:23:57 -07:00
kylasa 1f471396d8 Edge Feature support for input graph datasets for dist. graph partitioning pipeline (#4623)
* Added support for edge features.

* Added comments and removing unnecessary print statements.

* updated data_shuffle.py to remove compile error.

* Repaled python3 with python to match CI test framework.

* Removed unrelated files from the pull request.

* Isort changes.

* black changes on this file.

* Addressing CI review comments.

* Addressing CI comments.

* Removed duplicated and resolved merge conflict code.

* Addressing CI Comments from Rui.

* Addressing CI comments, and fixing merge issues.

* Addressing CI comments, code refactoring, isort and black
2022-10-03 01:02:15 -07:00
Quan (Andy) Gan 5bf17547f7 [DistTensor] add bitwise or function (#4638)
* first commit

* add test

* fixes

* ah this is how you skip setup

* fix

* ugh

* address comments

* i like black
2022-09-30 19:38:37 +08:00
czkkkkkk bb7b3c6f25 [Sparse] Add SparseMatrix unittest and fix docstring problem (#4627)
* [Sparse] Add SparseMatrix unittest and fix docstring problem

* Minor fix

* Update

* check permission

* rm future annonations

* Skip create_from_csr and create_from_csc tests because Pytorch 1.9.0 does not have torch.sparse_csr_tensor

Co-authored-by: Israt Nisa <nisisrat@amazon.com>
2022-09-30 11:53:33 +08:00
Israt Nisa 3d01340d23 [Sparse] Add elementwise operators for diagonal matrix (#4644)
* Add elementwise operators for diagonal matrix

* minor

* resolve ambiguity of ops

* lint check

* lint check

* lint check

Co-authored-by: Israt Nisa <nisisrat@amazon.com>
2022-09-29 13:08:50 +08:00
LuckyLiuM eb729c5401 Add Deepwalk module (#4562)
* metapath2vec package

* fix bugs --metapath2vec package

* add unittest and fix bugs

* fix pyling messages

* del init.py

* fix bugs

* modify metapath2vec and add deepwalk

* deepwalk module

* del metapath2vec files

* Update

* Update

* Update

* Update

* Update

* Fix

Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2022-09-29 09:30:56 +08:00
Israt Nisa fa05ccb961 Add sddmm operator in mock_sparse library (#4579)
* sddmm init

* SDDMM with N-D nonzero values

* drop support for vector shaped non zero elements

* address comments

* skip cpu test

* skip GPU test too

Co-authored-by: Israt Nisa <nisisrat@amazon.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-09-28 11:50:30 -04:00
Rhett Ying cf19254a19 [Dist] enable to partition many chunks into less partitions via pipeline (#4620)
* [Dist] enable to partition many chunks into less partitions via pipeline

* refine

* add meta file for num_parts, add more tests, refine docstring

* remove args.num_parts

* create pydantic class for partition metadata

* refine

* rename json file
2022-09-28 16:52:39 +08:00
Rhett Ying 6c1500d49f [Dist] save original node/edge IDs into separate files (#4649)
* [Dist] save original node/edge IDs into separate files

* separate nids and eids
2022-09-28 14:30:24 +08:00
Mufei Li 977b1ba448 [Sparse] Migration of Unary, Softmax, Matmul Ops (#4616)
* Update

* lint

* Fix

* Update

* Update

* Update

* update

* Fix

* Update

* Fix

* Fix

* Fix

* CI

* Update

* Update

* Update

* update test

Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
2022-09-27 19:34:46 +08:00
Xin Yao 188bc2bf71 [Fix][Unittest] Fix test stream (#4635)
* fix test stream

* init cusparse handle
2022-09-27 12:41:26 +08:00
czkkkkkk 544a2cd192 [Sparse] Add reduction operators (#4600)
* [Sparse] Add reduction operators

* Fix linting problem

* Fix linting problem

* Fix linting problem

* Fix docstring

* Fix test cases

* Fix linting problem

* Skip reduction unittests
2022-09-26 16:06:05 +08:00
Israt Nisa 7dba19913e Sparse ops: Support elementwise ops between sparse and diagonal matrix (#4617)
* initial commit

* address comments

* lint check

Co-authored-by: Israt Nisa <nisisrat@amazon.com>
2022-09-23 22:22:51 -04:00
czkkkkkk db64cc3760 [Sparse] Add Transposition for SparseMatrix and DiagMatrix (#4597)
* [Sparse] Add Transposition

* Fix docstring

* Fix linting problem

* Minor fix

* Minor fix

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-09-23 15:17:55 +08:00
Vibhu Jawa d04534c940 [Bugfix][Hotfix] Fix cugraph Pytorch CI failures (#4622)
* get_more_logs

* Update cugraph_unit_test.sh

* Update test_basics.py

* Update cugraph_unit_test.sh

* Update tests/cugraph/test_basics.py

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-09-23 11:49:03 +08:00
Rhett Ying be11409312 [CI] Enable DGL cugaph nightly CI (#4573)
* [CI] Enable DGL cugaph nightly CI

* fix build issue

* remove runtime
2022-09-22 15:26:29 +08:00
Xin Yao 880b3b1f97 [Fix] Enable lint check for cuh files and fix compiler warnings (#4585)
* disable warning for tensorpipe

* fix warning

* enable lint check for cuh files

* resolve comments
2022-09-21 10:09:14 +08:00
peizhou001 166b273bec add ssh port config for dispatchdata (#4557) 2022-09-20 20:41:41 +08:00
Mufei Li 73b9f3a1da [Sparse] Migrate DiagMatrix class (#4567)
* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
2022-09-20 17:17:43 +08:00
Quan (Andy) Gan 8366774122 [Feature] Support canonical edge types in HeteroGraphConv (#4440)
* fix

* Update hetero.py

* why did i remove this

Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-09-20 14:15:26 +08:00
nv-dlasalle 2630d2eb02 [Performance][bugfix] Implement is_unibipartite in C++ with caching. (#4556)
* updates

* Enable caching C++ result

* Add missing docstring

* Remove unused function

* Add unit test

* Address comments
2022-09-19 16:46:27 -07:00
Xin Yao cded5b80fe [Feature] Bump DLPack to v0.7 and decouple DLPack from the core library (#4454)
* rename `DLContext` to `DGLContext`

* rename `kDLGPU` to `kDLCUDA`

* replace DLTensor with DGLArray

* fix linting

* Unify DGLType and DLDataType to DGLDataType

* Fix FFI

* rename DLDeviceType to DGLDeviceType

* decouple dlpack from the core library

* fix bug

* fix lint

* fix merge

* fix build

* address comments

* rename dl_converter to dlpack_convert

* remove redundant comments
2022-09-19 16:02:43 +08:00
Rhett Ying 7f984b64b2 Revert "[Feature][REVIEW] Enable DGL cugaph nightly CI (#4525)" (#4563)
This reverts commit ec171c648a.
2022-09-16 13:57:50 +08:00
Vibhu Jawa ec171c648a [Feature][REVIEW] Enable DGL cugaph nightly CI (#4525)
* Added cugraph nightly scripts

* Removed nvcr.io//nvidia/pytorch:22.04-py3 reference

Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2022-09-16 11:28:30 +08:00
Xin Yao 9a00cf194f [Feature] Import PyTorch's CUDA stream management (#4503)
* add set_stream

* add .record_stream for NDArray and HeteroGraph

* refactor dgl stream Python APIs

* test record_stream

* add unit test for record stream

* use pytorch's stream

* fix lint

* fix cpu build

* address comments

* address comments

* add record stream tests for dgl.graph

* record frames and update dataloder

* add docstring

* update frame

* add backend check for record_stream

* remove CUDAThreadEntry::stream

* record stream for newly created formats

* fix bug

* fix cpp test

* fix None c_void_p to c_handle
2022-09-15 16:54:52 +08:00
Rhett Ying 099b173f6f [DistPart] expose timeout config for process group (#4532)
* [DistPart] expose timeout config for process group

* refine code

* Update tools/distpartitioning/data_proc_pipeline.py

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-09-15 15:28:36 +08:00
Israt Nisa cf03592743 Move mock version of dgl_sparse library to DGL main repo (#4524)
* init

* Add api doc for sparse library

* support op btwn matrices with differnt sparsity

* Fixed docstring

* addresses comments

* lint check

* change keyword format to fmt

Co-authored-by: Israt Nisa <nisisrat@amazon.com>
2022-09-14 12:08:20 -04:00
ndickson-nvidia 684f66b7aa [Feature] Added exclude_self and output_batch to knn graph construction (Issues #4323 #4316) (#4389)
* * Added "exclude_self" and "output_batch" options to knn_graph and segmented_knn_graph
* Updated out-of-date comments on remove_edges and remove_self_loop, since they now preserve batch information

* * Changed defaults on new knn_graph and segmented_knn_graph function parameters, for compatibility; pytorch/test_geometry.py was failing

* * Added test to ensure dgl.remove_self_loop function correctly updates batch information

* * Added new knn_graph and segmented_knn_graph parameters to dgl.nn.KNNGraph and dgl.nn.SegmentedKNNGraph

* * Formatting

* * Oops, I missed the one in segmented_knn_graph when I fixed the similar thing in knn_graph

* * Fixed edge case handling when invalid k specified, since it still needs to be handled consistently for tests to pass
* Fixed context of batch info, since it must match the context of the input position data for remove_self_loop to succeed

* * Fixed batch info resulting from knn_graph when output_batch is true, for case of 3D input tensor, representing multiple segments

* * Added testing of new exclude_self and output_batch parameters on knn_graph and segmented_knn_graph, and their wrappers, KNNGraph and SegmentedKNNGraph, into the test_knn_cuda test

* * Added doc comments for new parameters

* * Added correct handling for uncommon case of k or more coincident points when excluding self edges in knn_graph and segmented_knn_graph
* Added test cases for more than k coincident points

* * Updated doc comments for output_batch parameters for clarity

* * Linter formatting fixes

* * Extracted out common function for test_knn_cpu and test_knn_cuda, to add the new test cases to test_knn_cpu

* * Rewording in doc comments

* * Removed output_batch parameter from knn_graph and segmented_knn_graph, in favour of always setting the batch information, except in knn_graph if x is a 2D tensor

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-09-07 08:35:26 +08:00
Chang Liu 1c9d2a0302 [Feature] Unify the cuda stream used in core library (#4480)
* Use an internal cuda stream for CopyDataFromTo

* small fix white space

* Fix to compile

* Make stream optional in copydata for compile

* fix lint issue

* Update cub functions to use internal stream

* Lint check

* Update CopyTo/CopyFrom/CopyFromTo to use internal stream

* Address comments

* Fix backward CUDA stream

* Avoid overloading CopyFromTo()

* Minor comment update

* Overload copydatafromto in cuda device api

Co-authored-by: xiny <xiny@nvidia.com>
2022-09-06 10:14:55 +08:00
peizhou001 62af41c245 [Bug] Enable turn on/off libxsmm at runtime (#4455)
* enable turn on/off libxsmm at runtime by adding a global config and related API


Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-194.ap-northeast-1.compute.internal>
2022-09-05 17:38:34 +08:00
nv-dlasalle 806def67a1 [Cleanup] Remove async_transferer (#4505)
* Remove async_transferer

* remove test

* Remove AsyncTransferer

Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Xin Yao <yaox12@outlook.com>
2022-09-05 11:46:06 +08:00
Rhett Ying 468c0ca4b4 [BugFix] fix crash due to incorrect dtype in dgl.to_block() (#4487)
* [BugFix] fix crash due to incorrect dtype in dgl.to_block()

* fix test failure in TF
2022-08-31 16:09:46 +08:00
Rhett Ying 86656a6b34 [CI] use https instead of http (#4488) 2022-08-31 14:59:40 +08:00
Mufei Li 7a41c126f6 [Doc] Change random.py to random_partition.py in guide on distributed partition pipeline (#4438)
* Update distributed-preprocessing.rst

* Update

Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
2022-08-22 18:12:55 +08:00
Minjie Wang 2cf4bd0acf Merge branch 'master' into dist_part 2022-08-20 13:42:07 +08:00
Mufei Li 2e8ae9f980 [Dist][CI] Unit test for the new distributed partitioning pipeline (#4394)
* chunked graph data format

* Update

* Update

* Update task_distributed_test.sh

* Update

* Update

* Revert "Update"

This reverts commit 03c461870f19375fb03125b061fc853ab555577f.

* Update

* Update

* ssh-keygen

* CI

* install openssh

* openssh

* Update

* CI

* Update

* Update

Co-authored-by: Ubuntu <ubuntu@ip-172-31-53-142.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-87.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-20-21.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
2022-08-19 14:20:56 +08:00
Rhett Ying 8d3c58202a [CI] upgrade python version to 3.7.0 (#4406)
* [CI] upgrade python version to 3.7.0

* do not upgrade for mxnet cpu due to seg fault

* fix test failure for mxnet
2022-08-17 08:02:44 +00:00
Rhett Ying cf4727a9d9 [CI] upgrade python version to 3.7.0 (#4406)
* [CI] upgrade python version to 3.7.0

* do not upgrade for mxnet cpu due to seg fault

* fix test failure for mxnet
2022-08-17 15:28:34 +08:00
Rhett Ying 39987bc599 [Feature] enable graph partition book support canonical etypes (#4343)
* [Feature] enable graph partition book support canonical etypes

* fix lint

* fix lint

* add todo

* refine according to review comments

* fix lint

* refine naming

* revert PartitionPolicy __init__

* refine docstring

* fix doc string
2022-08-16 11:52:02 +08:00
Xin Yao 3685000aa4 [Bugfix] Fix pinning empty tensors and graphs (#4393) 2022-08-15 16:18:04 +08:00
YJ-Zhao a88e7f7e28 [Example]rgcn-ogbn-mag (#4331)
* rgcn-ogbn-mag

* Add link in README.md

* correct code-format,add the reset_parameters function to the HeteroEmbedding module

* add the annotation in hetero.py

* add a unit test

* modify format

* Update

Co-authored-by: Mufei Li <mufeili1996@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-50-143.us-west-2.compute.internal>
2022-08-10 20:12:31 +08:00
Quan (Andy) Gan 62c827c81c [Bug] A bunch of fixes in edge_softmax_hetero (#4336)
* bunch of fixes

* Update test_edge_softmax_hetero.py

* Update test_edge_softmax_hetero.py

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-08-09 15:16:48 +08:00
Chang Liu 5ba5106aca [Bugfix] Fix the default value of num_bases in RelGraphConv module (#4321)
* Fix doc and default settings for RelGraphConv

* Add unit test

* Split msg in two lines to pass CI-lint
2022-08-07 16:33:16 +08:00
Rhett Ying 43ba94ee49 [BugFix] fix etype check in DistGraph.edge_subgraph (#4322) 2022-08-03 11:56:00 +08:00
Xin Yao 463650a772 [Unittest] Improve test_dataloader (#4301)
* test ddp dataloader

* add pure_gpu for edgedataloader

* resolve ddp issue
2022-08-02 10:31:09 +08:00