694ef6518c
[Dist] import dgl.distributed for examples ( #7761 )
2024-09-02 10:55:48 +08:00
9782c021de
[Dist] move DistNode/EdgeDataLoader to distributed ( #7755 )
2024-08-29 16:10:05 +08:00
5008af2210
[Misc] Rename number_of_edges and number_of_nodes to num_edges and num_nodes in examples. ( #5492 )
...
* pytorch_example
* fix
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-03-29 11:01:20 +08:00
704bcaf6dd
examples ( #5323 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-02-19 08:35:15 +08:00
3132da2826
Deprecate ( #4864 )
...
rename DGLHeteroGraph to DGLGraph
2022-11-15 10:01:01 +08:00
e14860d9cc
[Example][Bug] fix device index for dist train on GPUs ( #4403 )
2022-08-18 13:00:21 +08:00
701b4fccc2
[Sampling] New sampling pipeline plus asynchronous prefetching ( #3665 )
...
* initial update
* more
* more
* multi-gpu example
* cluster gcn, finalize homogeneous
* more explanation
* fix
* bunch of fixes
* fix
* RGAT example and more fixes
* shadow-gnn sampler and some changes in unit test
* fix
* wth
* more fixes
* remove shadow+node/edge dataloader tests for possible ux changes
* lints
* add legacy dataloading import just in case
* fix
* update pylint for f-strings
* fix
* lint
* lint
* lint again
* cherry-picking commit fa9f494
* oops
* fix
* add sample_neighbors in dist_graph
* fix
* lint
* fix
* fix
* fix
* fix tutorial
* fix
* fix
* fix
* fix warning
* remove debug
* add get_foo_storage apis
* lint
2022-01-30 16:13:00 +08:00
8d14a739bc
fix. ( #3652 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal >
2022-01-15 09:12:13 +08:00
2a757d4aab
Remove self-loops and duplicate edges before ParMETIS and restore when converting to DGLGraph ( #3472 )
...
* save self-loops and duplicated edges separately.
* [BugFix] sort graph by dgl.ETYPE
* fix bugs in verify script
* fix verify logic
* refine README
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-11-08 11:17:19 +08:00
21a4027969
Fix distributed device mapping problem. ( #3313 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-09-02 13:52:00 +08:00
b131920066
[Feature] Add removed edges in distributed graph partitioning to handle heterogeneous graph ( #3137 )
...
* Added code for Rectifying (TypeError: unhashable type: 'slice') when copying file
* 1) added distributed preprocessing code to create ParMetis Input from CSV files
2) add code to run pm_dglpart on multiple machines
3) added support for recreating heteregenous graph from homo geneous graph based on dropped edges, as ParMetis currently only supports homogeneous graphs
* move to pandas
* Added comments and remove drop_duplicates as it was redundant
* Addressed Pr Comments
* Rename variable
* Added comment
* Added comment
* updated ReadMe
Co-authored-by: Ankit Garg <gaank@amazon.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-08-02 11:41:13 -07:00
34426a98e9
[Distributed] Distributed heterograph training ( #3069 )
...
* support hetero RGCN.
* fix.
* simplify code.
* sample_neighbors return heterograph directly.
* avoid using to_heterogeneous.
* compute canonical etypes in advance.
* fix tests.
* fix.
* fix distributed data loader for heterograph.
* use NodeDataLoader.
* fix bugs in partitioning on heterogeneous graphs.
* fix lint.
* fix tests.
* fix.
* fix.
* fix bugs.
* fix tests.
* fix.
* enable coo for distributed.
* fix.
* fix.
* fix.
* fix.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-17 10:57:00 +08:00
d7390763f0
[Distributed] Deprecate old DistEmbedding impl, use synchronized embedding impl ( #3111 )
...
* fix.
* fix.
* fix.
* fix.
* Fix test
* Deprecate old DistEmbedding impl, use synchronized embedding impl
* update doc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-07-14 00:15:53 +08:00
0884d02465
[Distributed] Fix bugs in partitioning on heterogeneous graphs. ( #3085 )
...
* fix bugs in partitioning on heterogeneous graphs.
* fix.
* fix.
* fix example.
* fix.
* fix test.
* fix.
* fix.
* fix.
* fix tests.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-02 21:16:14 +08:00
975eb8fc5d
[Distributed] Distributed node embedding and sparse optimizer ( #2733 )
...
* Draft for sparse emb
* add some notes
* Fix
* Add sparse optim for dist pytorch
* Update test
* Fix
* upd
* upd
* Fix
* Fix
* Fix bug
* add transductive exmpale
* Fix example
* Some fix
* Upd
* Fix lint
* lint
* lint
* lint
* upd
* Fix lint
* lint
* upd
* remove dead import
* update
* lint
* update unitest
* update example
* Add adam optimizer
* Add unitest and update data
* upd
* upd
* upd
* Fix docstring and fix some bug in example code
* Update rgcn readme
Co-authored-by: Ubuntu <ubuntu@ip-172-31-57-25.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-05-03 08:37:40 +08:00
bfbbefa793
[Distributed] Fix a bug in multiprocessing sampling. ( #2826 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-73-81.ec2.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-04-08 01:29:33 -07:00
e36c5db614
[Distributed] Simplify distributed API ( #2775 )
...
* remove num_workers.
* remove num_workers.
* remove num_workers.
* remove num-servers.
* update error message.
* update docstring.
* fix docs.
* fix tests.
* fix test.
* fix.
* print messages in test.
* fix.
* fix test.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
2021-03-29 19:19:23 -07:00
bb54206620
remove pyinstrument. ( #2772 )
...
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-03-21 20:27:12 -07:00
d0638b1ea9
[Example] Update the example of distributed GNN training of RGCN ( #2709 )
...
* support gpu training.
* remove unnecessary arguments.
* update README.
* update time measurement.
* add zero_grad.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-03-01 12:00:15 -08:00
28b0a76618
[Example] Provide instructions of setting up passwordless SSH login. ( #2707 )
2021-02-27 18:34:46 -08:00
6568d0aaf6
[Example] Recommend NFS over copy files ( #2689 )
...
* Update README to use NFS, rather than copying files
* Update README.md
* Update README.md
* Update README.md
* Add ParMetis
* Update README.md
* fix directory.
* fix
* redo.
* fix.
* update.
* add more descriptions
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2021-02-25 20:20:52 +08:00
0526b885ee
[Distributed] Fix the path for distributed partitioning. ( #2700 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
2021-02-24 20:41:41 -08:00
ff64bd0de8
add warning for a bug in README ( #2665 )
2021-02-16 00:16:40 -08:00
e4ff484494
[Distributed] Distributed METIS partition ( #2576 )
...
* add convert.
* fix.
* add write_mag.
* fix convert_partition.py
* write data.
* use pyarrow to read.
* update write_mag.py
* fix convert_partition.py.
* load node/edge features when necessary.
* reshuffle nodes.
* write mag correctly.
* fix a bug: inner nodes in a partition might be empty.
* fix bugs.
* add verify code.
* insert reverse edges.
* fix a bug.
* add get node/edge data.
* add instructions.
* remove unnecessary argument.
* update distributed preprocessing.
* fix readme.
* fix.
* fix.
* fix.
* fix readme.
* fix doc.
* fix.
* update readme
* update doc.
* update readme.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
2021-02-08 17:19:18 -08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
782527d481
Fix partition for 0.5.1 ( #2187 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
2020-09-13 16:25:51 +08:00
e8273a8510
update README.md ( #2138 )
2020-09-02 15:42:28 +08:00
0052f1212b
[Example] Rgcn distributed training ( #1999 )
...
* add entity_classify_dist
* upd
* update
* Fix
* Fix
* upd
* upd
* upd
* upd
* global eval
* Fix
* Fix
* Fix
* Fix
* FIx
* upd
* upd
* update
* support pytorch sparse embedding
* Fix
* Fix
* update Readme
* update with new API
* Fix
* update Readme
* add fanout for validation neighbor sampling
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-08-18 12:12:21 -07:00
Rhett Ying