d1124b7bef
[Example][BugFix] Fix Lightning GraphSAGE examples (dmlc#4046) ( #4053 )
2022-05-26 12:40:40 +08:00
744896e2d8
fix examples ( #4016 )
...
Co-authored-by: Mufei Li <mufeili1996@gmail.com >
2022-05-23 23:41:12 +08:00
74f0140533
[Example] Rename NodeDataLoader to DataLoader in GraphSAGE example ( #3972 )
...
* rename
* Update node_classification.py
* more fixes...
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2022-05-11 09:10:54 +08:00
37be02a486
[Feature] enable socket net_type for rpc ( #3951 )
...
* [Feature] enable socket net_type for rpc
* fix lint
* fix lint
* fix build issue on windows
* fix test failure on windows
* fix test failure
* fix cpp unit test failure
* net_type blocking max_try_times
* fix other comments
* fix lint
* fix comment
* fix lint
* fix cpp
2022-04-28 00:10:06 +08:00
e06e63d5d5
[Feature] Enable UVA sampling with CPU indices ( #3892 )
...
* enable UVA sampling with CPU indices
* add docs
* add more docs
* lint
* fix
* fix
* better error message
* use mp.Barrier instead of queues
* revert
* revert
* oops
* revert dgl.multiprocessing.spawn
* Update pytorch.py
2022-04-12 20:56:51 +08:00
0d878ff8ef
[Example] Cleaned GraphSAGE node classification example with PyTorch Lightning ( #3863 )
...
* cleaned pl node classification example
* conform to PL's method of updating the dataloader
* update
* lint
* fix test
* fix
2022-04-12 17:26:23 +08:00
f931c6ba54
[Examples] Add pure gpu mode in the GraphSAGE node classification and link prediction ( #3856 )
...
Signed-off-by: Serge Panev <spanev@nvidia.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-04-12 13:03:03 +08:00
01e50626d3
[Example] Add TorchMetrics in README ( #3913 )
...
* Update README.md
* Update README.md
* Update README.md
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com >
2022-04-09 23:25:51 +08:00
27a6eb564c
[Examples] Update graphsage multi-gpu example to use mutliple GPUs for validation and testing. ( #3827 )
...
* Update graphsage multi-gpu example to use mutliple GPUs for validation and
testing.
* Remove argmax
* Fix rebase error
* Add more documentation to example and simplify
* Switch to name shared memory
* Add comment about how training is distributed
* Restore iteration count
* fix munmap error reporting for better error messages
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-04-06 00:32:37 +08:00
7d4160869c
[Bug] Fix multiple issues in distributed multi-GPU GraphSAGE example ( #3870 )
...
* fix distributed multi-GPU example device
* try Join
* update version requirement in README
* use model.join
* fix docs
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-03-25 22:04:02 +08:00
57d2f31f20
[Examples] Add pure gpu example of graphsage ( #3796 )
...
* Add pure_gpu example of graphsage
* move to advanced directory
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2022-03-10 17:33:23 -08:00
44638b9336
fix ddp dataloader in heterogeneous cases ( #3801 )
2022-03-07 16:56:43 +08:00
0528e90d7d
[Examples] fix path for load_graph ( #3797 )
2022-03-03 15:36:48 +08:00
0ec439247a
[Examples] re-locate load_graph for share ( #3784 )
2022-03-01 09:32:53 +08:00
d41d07d0f6
[Doc and bugfix] Add docs and user guide and update tutorial for sampling pipeline ( #3774 )
...
* huuuuge update
* remove
* lint
* lint
* fix
* what happened to nccl
* update multi-gpu unsupervised graphsage example
* replace most of the dgl.mp.process with torch.mp.spawn
* update if condition for use_uva case
* update user guide
* address comments
* incorporating suggestions from @jermainewang
* oops
* fix tutorial to pass CI
* oops
* fix again
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-02-28 02:21:51 +08:00
738e8318fd
[Feature] CUDA UVA sampling for MultiLayerNeighborSampler ( #3674 )
...
* implement pin_memory/unpin_memory/is_pinned for dgl.graph
* update python docstring
* update c++ docstring
* add test
* fix the broken UnifiedTensor
* XPU_SWITCH for kDLCPUPinned
* a rough version ready for testing
* eliminate extra context parameter for pin/unpin
* update train_sampling
* fix linting
* fix typo
* multi-gpu uva sampling case
* disable new format materialization for pinned graphs
* update python doc for pin_memory_
* fix unit test
* UVA sampling for link prediction
* dispatch most csr ops
* update graphsage example to combine uva sampling and UnifiedTensor
* update graphsage example to combine uva sampling and UnifiedTensor
* update graphsage example to combine uva sampling and UnifiedTensor
* update doc
* update examples
* change unitgraph and heterograph's PinMemory to in-place
* update examples for multi-gpu uva sampling
* update doc
* fix linting
* fix cpu build
* fix is_pinned for DistGraph
* fix is_pinned for DistGraph
* update graphsage unsupervised example
* update doc for gpu sampling
* update some check for sampling device switching
* fix linting
* adapt for new dataloader
* fix linting
* fix
* fix some name issue
* adjust device check
* add unit test for uva sampling & fix some zero_copy bug
* fix linting
* update num_threads in graphsage examples
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-02-09 14:06:17 +08:00
0767c5fcc9
Fix dist example padding problem ( #3687 )
2022-02-07 17:44:44 +08:00
701b4fccc2
[Sampling] New sampling pipeline plus asynchronous prefetching ( #3665 )
...
* initial update
* more
* more
* multi-gpu example
* cluster gcn, finalize homogeneous
* more explanation
* fix
* bunch of fixes
* fix
* RGAT example and more fixes
* shadow-gnn sampler and some changes in unit test
* fix
* wth
* more fixes
* remove shadow+node/edge dataloader tests for possible ux changes
* lints
* add legacy dataloading import just in case
* fix
* update pylint for f-strings
* fix
* lint
* lint
* lint again
* cherry-picking commit fa9f494
* oops
* fix
* add sample_neighbors in dist_graph
* fix
* lint
* fix
* fix
* fix
* fix tutorial
* fix
* fix
* fix
* fix warning
* remove debug
* add get_foo_storage apis
* lint
2022-01-30 16:13:00 +08:00
14ab462fd8
Update examples/pytorch/graphsage/experimental/README.md
...
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
2022-01-20 09:19:47 -08:00
3b1978a379
fix.
2022-01-19 12:49:55 -08:00
4889c57822
Add 'nccl' backend in train_dist.py and fix pad_data function cuda bug ( #3607 )
...
* Add nccl backend and fix pad_data function cuda bug
* Update train_dist.py
* Update train_dist.py
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-12-24 18:45:01 +08:00
421c362266
Update train_dist.py ( #3594 )
...
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-12-20 20:24:45 +08:00
78e0dae693
[DistGNN, Graph partitioning] Libra partition ( #3376 )
...
* added distgnn plus libra codebase
* Dist application codes
* added comments in partition code. changed the interface of partitioning call.
* updated readme
* create libra partitioning branch for the PR
* removed disgnn files for first PR
* updated kernel.cc
* added libra_partition.cc and moved libra code from kernel.cc to libra_partition.cc
* fixed lint error; merged libra2dgl.py and main_Libra.py to libra_partition.py; added graphsage/distgnn folder and partition script.
* removed libra2dgl.py
* fixed the lint error and cleaned the code.
* revisions due to PR comments. added distgnn/tools contains partitions routines
* update 2 PR revision I
* fixed errors; also improved the runtime by 10x.
* fixed minor lint error
* fixed some more lints
* PR revision II changed the interface of libra partition function
* rewrite docstring
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-12-15 10:02:31 +08:00
987db374fd
Fix for distributed training ( #3542 )
...
* tmp fix
* add description
2021-12-06 18:35:29 +08:00
a8c81018c5
[Sampling] Implement dgl.compact_graphs() for the GPU ( #3423 )
...
* gpu compact graph template
* cuda compact graph draft
* fix typo
* compact graphs
* pass unit test but fail in training
* example using EdgeDataLoader on the GPU
* refactor cuda_compact_graph and cuda_to_block
* update training scripts
* fix linting
* fix linting
* fix exclude_edges for the GPU
* add --data-cpu & fix copyright
2021-10-20 22:07:35 -07:00
367a3a34c4
Fix torch import in example ( #3372 )
2021-09-23 13:23:05 +08:00
01a2214430
Enable faster validation for pytorch graphsage example ( #3361 )
2021-09-19 17:25:46 -07:00
21a4027969
Fix distributed device mapping problem. ( #3313 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-09-02 13:52:00 +08:00
905c0aa578
[Feature][Performance][GPU] Introducing UnifiedTensor for efficient zero-copy host memory access from GPU ( #3086 )
...
* Add pytorch-direct version
* Initial commit of unified tensor
* Merge branch 'master' of https://github.com/davidmin7/dgl
* Remove unnecessary things
* Fix error message
* Fix/Add descriptions
* whitespace fix
* add unpin
* disable IndexSelectCPUFromGPU with no CUDA
* add a newline for unified_tensor.py
* Apply changes based on feedback
* add 'os' module
* skip unified tensor unit test for cpu only
* Update tests/pytorch/test_unified_tensor.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
* reflect feedback
Co-authored-by: shhssdm <shhssdm@gmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-07-17 00:06:20 +08:00
3f6f694159
[Bug fix] Various fix from bug bash ( #3133 )
...
* Update
* Update
* Update dependencies
* Update
* Update
* Fix ogbn-products gat
* Update
* Update
* Reformat
* Fix typo in node2vec_random_walk
* Specify file encoding
* Working for 6.7
* Update
* Fix subgraph
* Fix doc for sample_neighbors_biased
* Fix hyperlink
* Add example for udf cross reducer
* Fix
* Add example for slice_batch
* Replace dgl.bipartite
* Fix GATConv
* Fix math rendering
* Fix doc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-156.us-west-2.compute.internal >
2021-07-15 09:17:15 +08:00
d7390763f0
[Distributed] Deprecate old DistEmbedding impl, use synchronized embedding impl ( #3111 )
...
* fix.
* fix.
* fix.
* fix.
* Fix test
* Deprecate old DistEmbedding impl, use synchronized embedding impl
* update doc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-07-14 00:15:53 +08:00
2f43cdb3c8
deal with situation where num_layers equals 1 ( #3066 )
2021-06-28 10:17:16 +08:00
427a5a96e0
[Doc] Update NodeDataLoader and EdgeDataLoader for GPU-based neighbor sampling ( #3046 )
...
* update docstrings and tidy code
* add docs
* address comments
* Update __init__.py
* address comments
2021-06-25 10:46:21 +08:00
aaec3d8a0b
[Distributed] Support hierarchical partitioning ( #3000 )
...
* add.
* fix.
* fix.
* fix.
* fix.
* add tests.
* support node split and edge split.
* support 1 partition.
* add tests.
* fix.
* fix test.
* use hierarchical partition.
* add check.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-57.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-06-16 16:58:23 +08:00
8b64ae59b8
[Performance] Perform to_block on the GPU when the dataloader is created with a GPU device. ( #3016 )
...
* add output device for dataloading
* Update dataloader
* Get sampler device from dataloader
* Fix line length
* Update examples
* Fix to_block GPU for empty relation types
* Handle the case where the DistGraph has None for the underlying graph
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-06-13 18:51:00 +08:00
9497a9be85
[Bugfix][Examples] Fix graphsage multigpu training example training set size ( #3002 )
...
* Make multigpu graphsage use whole datset
* Specify queeze dimension
* Remove squeeze dimension
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-06-10 16:35:08 -07:00
18dbaebebc
[Distributed] Specify the graph format for distributed training ( #2948 )
...
* explicitly set the graph format.
* fix.
* fix.
* fix launch script.
* fix readme.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-05-26 18:09:30 +08:00
7c7113f660
add use_ddp to dataloaders ( #2911 )
2021-05-17 15:46:02 +08:00
caa6d6072c
[Feature] Replacing thread_wrapped_func with minimal mp.Process wrapper ( #2905 )
...
* standardizing thread_wrapped_func
* lints
* Update __init__.py
2021-05-14 11:34:49 +08:00
70695ff8f8
[Model] Training GraphSAGE with PyTorch Lightning ( #2878 )
...
* pytorch lightning initial examples
* revert most changes in dataloader to favor #2886 .
* address comments
2021-05-11 11:28:13 +08:00
975eb8fc5d
[Distributed] Distributed node embedding and sparse optimizer ( #2733 )
...
* Draft for sparse emb
* add some notes
* Fix
* Add sparse optim for dist pytorch
* Update test
* Fix
* upd
* upd
* Fix
* Fix
* Fix bug
* add transductive exmpale
* Fix example
* Some fix
* Upd
* Fix lint
* lint
* lint
* lint
* upd
* Fix lint
* lint
* upd
* remove dead import
* update
* lint
* update unitest
* update example
* Add adam optimizer
* Add unitest and update data
* upd
* upd
* upd
* Fix docstring and fix some bug in example code
* Update rgcn readme
Co-authored-by: Ubuntu <ubuntu@ip-172-31-57-25.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-05-03 08:37:40 +08:00
e70138bbe4
[Performance][GPU] Enable GPU uniform edge sampling ( #2716 )
...
* Start on uniform GPU sampling
* Save more work
* Get cu file compiling
* Update sampling
* More changes
* Get GPU sampling for uniform probabilities solved
* Fix batch tensor migration
* Fix
* update kernels
* expand blocking
* Undo testing change
* Cut down on sampling overhead
* Fix replacement
* Update unit tests
* Add option to gpu sample in graphsage
* Copy only csc to gpu
* Add ogbn support
* Fix linting
* Remove nvtx from sample
* Improve documentation and error checking
* Expand documentation
* Update assert checking
* delete extra space
* Use standard dataloader when dataset is a dictionary
* ogb -> ogbn
* Fix edge selection determinism
* Fix typos
* Remove nvtx
* Add comment for self.fanout_arrays and assert
* Fix linting
* Migrate to scalarbatcher
* Fix indentation
* Fix batcher
* Fix indexing
* Only use databatcher for GPU
* Convert to DGL NDArray to PyTorch Tensor
* Add optimization for PyTorch's F.tensor() for list of GPU tensors
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-04-14 18:55:09 -07:00
bfbbefa793
[Distributed] Fix a bug in multiprocessing sampling. ( #2826 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-73-81.ec2.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-04-08 01:29:33 -07:00
e36c5db614
[Distributed] Simplify distributed API ( #2775 )
...
* remove num_workers.
* remove num_workers.
* remove num_workers.
* remove num-servers.
* update error message.
* update docstring.
* fix docs.
* fix tests.
* fix test.
* fix.
* print messages in test.
* fix.
* fix test.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
2021-03-29 19:19:23 -07:00
e6f6c2eb31
[Bugfix] Update deprecated method name in load_graph.py ( #2769 )
...
Method `dataset.num_labels` has been deprecated and replaced by `dataset.num_classes`.
Updating the method name to avoid runtime warning.
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-03-22 20:07:32 +08:00
bb54206620
remove pyinstrument. ( #2772 )
...
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-03-21 20:27:12 -07:00
28b0a76618
[Example] Provide instructions of setting up passwordless SSH login. ( #2707 )
2021-02-27 18:34:46 -08:00
6568d0aaf6
[Example] Recommend NFS over copy files ( #2689 )
...
* Update README to use NFS, rather than copying files
* Update README.md
* Update README.md
* Update README.md
* Add ParMetis
* Update README.md
* fix directory.
* fix
* redo.
* fix.
* update.
* add more descriptions
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2021-02-25 20:20:52 +08:00
ff64bd0de8
add warning for a bug in README ( #2665 )
2021-02-16 00:16:40 -08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
彭齐荣