3bcb268a36
[Doc] Update distributed chapter according to new pipeline ( #4275 )
...
* dist index chapter
* preproc chapter
* rst
* tools page
* partition chapter
* rst
* hetero chapter
* 7.1 step1
* add parmetis back
* changed based on feedback
* address comments
2022-08-17 13:30:04 +08:00
69226588a5
[Dist] defer to load node/edge feats ( #4143 )
...
* [Dist] defer to load node/edge feats
* fix lint
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* fix lint
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
2022-06-20 19:44:37 +08:00
fbbca994b7
[Feature] METIS Partition with Communication Volume Minimization ( #3821 )
...
* upd
* upd
* upd
* upd
* upd
* fix OpenMP compatibility issues
* typo
* partition
* misc
* fix typo
* num_parts=1
* import torch
* long
* print info
* print info
* print info
* upd
* remove debug code
* revert partition.py
* fix cut count
* fix cut count
* Revert "fix cut count"
This reverts commit 10926b4fd48f45c8f1ddb58be7db6c22e653effd.
* Revert "fix cut count"
This reverts commit 76465283bef093a2b4209ad70dd15d2437b2ec8a.
* type of deprecate
* typo in deprecate info
* fix typo
* use cv for partitioning
* CE
* no message
* revert
* typo
* add objtype
* no message
* fix bug
* fix bug
* fix bug
* ?
* semicolon
* drop tensors
* no message
* backward
* backward
* max op
* store X.shape
* th
* test
* Revert "test"
This reverts commit 92b3b2f64a3a1128590098fa03ce429c5466e6ce.
* test
* tolist
* debug
* to cuda
* tuple
* fix bug
* remove X
* no message
* fix bug
* workload balance
* Revert "workload balance"
This reverts commit d7f8e4a16ba2a7eabb4a9bb945523bfe6623e723.
* reverse
* Revert "reverse"
This reverts commit 8a71cf25685aa7d889b9b8881b46f7a16b7d6e6d.
* Revert "Revert "reverse""
This reverts commit 196b143932d5cf9813576ece7c990b63d322d063.
* Revert "Revert "Revert "reverse"""
This reverts commit cf9e89a07013582056e7cde235e51331aca7fa9c.
* no message
* Merge commit '5498cf051e9cf450dcb798ca09920bbde008a661'
# Conflicts:
# python/dgl/distributed/partition.py
* Revert "Merge commit '5498cf051e9cf450dcb798ca09920bbde008a661'"
This reverts commit f79be2ad777897c7025b28308454cad81ad6bb27.
* fix bug
* third party
* no message
* try to avoid memory leak
* try to avoid memory leak
* avoid memory leak with no hope
* Revert "avoid memory leak with no hope"
This reverts commit c77befe9479f46758e744642f66dd209b50eef7d.
* no message
* Revert "no message"
This reverts commit 478cb28fe25fb1002b2f1dc202bb9bdaad8b2a56.
* del
* Revert "del"
This reverts commit 1b468e45ce646b400ff3ffa61a0b2da058b3bdfd.
* no message
* no message
* Revert "no message"
This reverts commit 92e4f5561ed42da0606618b2fff9f1ad5ed439d9.
* third party
* document
* Update metis_partition.cc
* Update metis_partition_hetero.cc
* Update metis_partition_hetero.cc
* Update partition.py
* Update partition.py
* Update partition.py
Co-authored-by: yzh119 <expye@outlook.com >
Co-authored-by: chwan-rice <54331508+chwan-rice@users.noreply.github.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2022-03-27 14:22:01 -07:00
1d80d91a32
[BugFix] call g.is_homogenous instead of len(g.etypes) ( #3793 )
2022-03-02 14:27:52 +08:00
2dd114ed24
[Distributed] Measure peak memory size in graph partitioning ( #3633 )
...
* measure peak memory size.
* fix lint.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal >
2022-01-13 15:41:53 -08:00
0884d02465
[Distributed] Fix bugs in partitioning on heterogeneous graphs. ( #3085 )
...
* fix bugs in partitioning on heterogeneous graphs.
* fix.
* fix.
* fix example.
* fix.
* fix test.
* fix.
* fix.
* fix.
* fix tests.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-02 21:16:14 +08:00
aaec3d8a0b
[Distributed] Support hierarchical partitioning ( #3000 )
...
* add.
* fix.
* fix.
* fix.
* fix.
* add tests.
* support node split and edge split.
* support 1 partition.
* add tests.
* fix.
* fix test.
* use hierarchical partition.
* add check.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-57.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-06-16 16:58:23 +08:00
e156331b0e
return a list of ntypes/etypes ( #2742 )
...
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-05-13 10:42:50 +08:00
d76af4d4c1
[Distributed] Return the ID mapping in graph partitioning. ( #2857 )
...
* return mapping.
* support heterogeneous graph.
* more test.
* fix lint.
* fix for diff backends.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-22 17:51:42 +08:00
7ab3d993a4
Change the absolute path to relative one in part_config.json ( #2690 )
...
* Update partition.py
* fix lint.
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-02-24 16:12:19 -08:00
8a07ab7737
[Doc] Tutorials re-organization ( #2683 )
...
* reorg
* change titles
* rm some stale API doc; minor fix
* fix docs
* add warning
* rm new-tutorial run in ci
* lint
2021-02-20 17:42:45 +08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
9da1629c9d
[Distributed] Some small fixes on doc ( #2049 )
...
* quick fix.
* update sparse optimizer.
* fix.
* fix
2020-08-18 21:13:19 -07:00
75ffc31f76
[Doc] Update the docstring of distributed APIs. ( #2025 )
...
* add doc.
* update DistGraph.
* add DistTensor.
* update DistEmbedding.
* add partition.py
* add sampling.
* fix.
* add graph partition book and create a base class.
* fix test.
* add rst.
* update doc rst.
* update.
* fix.
* fix docs
* update distributed tensor and embeddings.
* add checks.
* update DistGraph.
* update initialization.
* fix graph partition book.
* update graph partition book.
* update partition.
* update partition.
* fix.
* add example code.
* update DistGraph
* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* update initialize.
* update dataloader.
* update distgraph.
* update DistGraph.
* update DistTensor.
* update.
* more updates.
* fix lint.
* add num_nodes and num_edges
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-08-18 11:53:45 -07:00
71f4230a8b
[Fix] OpenMP compatibility issues ( #1992 )
...
* fix OpenMP compatibility issues
* typo
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-08-11 14:26:34 +08:00
a4c931a9bf
[Distributed] Support multiple servers ( #1886 )
...
* client init graph on the backup servers.
* fix.
* test multi-server.
* fix anonymous dist tensors.
* check #parts.
* fix init_data
* add multi-server multi-client tests.
* update tests in kvstore.
* fix.
* verify the loaded partition.
* fix a bug.
* fix lint.
* fix.
* fix example.
* fix rpc.
* fix pull/push handler for backup kvstore
* fix example readme.
* change ip.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-02 19:50:34 -07:00
015acfd2d8
[Distributed] Small fixes ( #1859 )
...
* fix node/edge_split.
* fix partition.
* support heterograph interface.
* fix test.
* fix
* fix docstring.
2020-07-27 01:39:48 -07:00
ea420c0abd
[Distributed] Reduce memory consumption in graph partitioning ( #1823 )
...
* save mem.
* save mem.
* reduce mem
* fix test
* fix lint
* fix test
* fix.
* fix.
* fix.
* fix.
* fix lint.
* fix backend operator.
* fix tensorflow operators.
* fix.
* revert change in mxnet operator.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-18 06:53:48 -07:00
20734637f0
[Distributed] Fix partition ( #1821 )
...
* fix partition and print timing.
* fix lint.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-17 16:03:37 -07:00
a1472bcff6
[Distributed] Run distributed graph server inside DGL ( #1801 )
...
* run dist server in dgl.
* fix bugs.
* fix example.
* check environment variables and fix lint.
* fix lint
2020-07-14 07:41:44 -07:00
04d4680d5b
[Distributed] add constraints for metis partitioning. ( #1708 )
...
* add constraint.
* fix bugs.
* temp fix.
* fix lint
* add balance_ntypes and balance_edges
* add comments.
* fix.
* fix
2020-06-27 21:27:18 -07:00
45e1333e13
[Distributed] Optimize distributed sampling ( #1700 )
...
* optimize sampling neighbors from the local partition.
* fix minior bugs.
* fix lint
* overlap local sampling and remote sampling.
* fix.
* fix lint.
* fix a potential bug.
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-06-26 17:05:29 -07:00
4b96addca0
[BUGFIX] Fix bugs in range partitioning. ( #1703 )
...
* Fix bugs in partitioning.
* add comment.
* fix lint.
* fix test.
* add more tests.
* simplify the fix
* fix.
2020-06-26 15:59:45 -07:00
ff5b5a4a10
[Distributed] support no shared memory. ( #1663 )
...
* support no shared memory.
* add test.
* add CAPI to check existence of shared memory.
* revert the change in src/runtime/ndarray.cc
* update docstring.
* fix compile.
2020-06-18 09:42:11 -07:00
dc86bd421e
[Doc] fix a doc for partition_graph. ( #1657 )
...
* fix a doc for partition_graph.
* Update partition.py
* fix.
2020-06-17 22:45:05 -07:00
b9ec3e36ec
[BUGFIX] reshuffle node features in range partitioning. ( #1659 )
...
* fix a bug.
* fix test.
* simplify.
* fix test.
* test edge features.
* fix.
2020-06-17 22:20:17 -07:00
41349dcef5
[Feature] Range partition ( #1522 )
...
* add reorder immutable graph.
* add python API.
* add reorder for csr.
* remove gk version.
* fix
* add cpp test.
* bug fixes
* fix tests.
* fix bugs and add check
* fix test.
* add omp.
* add comments.
* add coo reorder.
* fix a bug.
* handle reorder for different graph structues.
* fix lint.
* fix.
* add original ids.
* reshuffle nodes before metis partition.
* inner nodes are in contiguous Id range.
* reshuffle nodes/edges when partitioning.
* load partition return graph partition book.
* use inner_node/inner_edges
* add and test range partition book.
* count inner_edge correctly.
* fix lint.
* fix lint.
* fix lint.
* fix errors.
* fix errors.
* fix for TF.
* fix.
* fix.
* change docstring.
* support logical and.
* add comments.
* avoid copy.
* fix
* update docstring.
* fix a bug.
* add range search.
* fix
* fix a bug.
* add more tests.
* load graph partition book.
* support shared memory for range partition book.
* fix a bug.
* fix.
* fix lint.
* remove check
* fix test.
* remove num_nodes and num_edges
* fix lint.
* fix graph partition book.
* address comments.
* use makedirs.
* fix compile
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-06-15 11:43:04 -07:00
484bbcc8f1
[Test] Fix partition ci fail ( #1614 )
2020-06-10 15:44:38 +08:00
ef7e475051
[Distributed] Add GraphPartitionBook ( #1496 )
...
* Add graph-partition-book
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* udpate
* fix lint
* update
* update
* update
* update
* update
* update
* update
* skip test on GPU
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* update
* update
* update
* fix unittest
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-05-08 14:52:12 +08:00
16561a2e6b
[Test] Add tests for TensorFlow ( #1501 )
...
* add test.
* move test code.
* remvoe unnecessary test.
* fix.
* turn on tests for TF.
* Revert "move test code."
This reverts commit e7b4f36395b2121a7be030bd4364a704d0e357bf.
* fix.
* fix.
* skip test for tensorflow.
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-05-06 00:33:53 -07:00
2190c39d67
[Feature] Distributed graph store ( #1383 )
...
* initial version from distributed training.
This is copied from multiprocessing training.
* modify for distributed training.
* it's runnable now.
* measure time in neighbor sampling.
* simplify neighbor sampling.
* fix a bug in distributed neighbor sampling.
* allow single-machine training.
* fix a bug.
* fix a bug.
* fix openmp.
* make some improvement.
* fix.
* add prepare in the sampler.
* prepare nodeflow async.
* fix a bug.
* get id.
* simplify the code.
* improve.
* fix partition.py
* fix the example.
* add more features.
* fix the example.
* allow one partition
* use distributed kvstore.
* do g2l map manually.
* fix commandline.
* a temp script to save reddit.
* fix pull_handler.
* add pytorch version.
* estimate the time for copying data.
* delete unused code.
* fix a bug.
* print id.
* fix a bug
* fix a bug
* fix a bug.
* remove redundent code.
* revert modify in sampler.
* fix temp script.
* remove pytorch version.
* fix.
* distributed training with pytorch.
* add distributed graph store.
* fix.
* add metis_partition_assignment.
* fix a few bugs in distributed graph store.
* fix test.
* fix bugs in distributed graph store.
* fix tests.
* remove code of defining DistGraphStore.
* fix partition.
* fix example.
* update run.sh.
* only read necessary node data.
* batching data fetch of multiple NodeFlows.
* simplify gcn.
* remove unnecessary code.
* use the new copy_from_kvstore.
* update training script.
* print time in graphsage.
* make distributed training runnable.
* use val_nid.
* fix train_sampling.
* add distributed training.
* add run.sh
* add more timing.
* fix a bug.
* save graph metadata when partition.
* create ndata and edata in distributed graph store.
* add timing in minibatch training of GraphSage.
* use pytorch distributed.
* add checks.
* fix a bug in global vs. local ids.
* remove fast pull
* fix a compile error.
* update and add new APIs.
* implement more methods in DistGraphStore.
* update more APIs.
* rename it to DistGraph.
* rename to DistTensor
* remove some unnecessary API.
* remove unnecessary files.
* revert changes in sampler.
* Revert "simplify gcn."
This reverts commit 0ed3a34ca714203a5b45240af71555d4227ce452.
* Revert "simplify neighbor sampling."
This reverts commit 551c72d20f05a029360ba97f312c7a7a578aacec.
* Revert "measure time in neighbor sampling."
This reverts commit 63ae80c7b402bb626e24acbbc8fdfe9fffd0bc64.
* Revert "add timing in minibatch training of GraphSage."
This reverts commit e59dc8957a414c7df5c316f51d78bce822bdef5e.
* Revert "fix train_sampling."
This reverts commit ea6aea9a4aabb8ba0ff63070aa51e7ca81536ad9.
* fix lint.
* add comments and small update.
* add more comments.
* add more unit tests and fix bugs.
* check the existence of shared-mem graph index.
* use new partitioned graph storage.
* fix bugs.
* print error in fast pull.
* fix lint
* fix a compile error.
* save absolute path after partitioning.
* small fixes in the example
* Revert "[kvstore] support any data type for init_data() (#1465 )"
This reverts commit 87b6997bf2#1465 )""
This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.
* support set and init data.
* support set and init data.
* Revert "Revert "[kvstore] support any data type for init_data() (#1465 )""
This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.
* fix bugs.
* fix unit test.
* move to dgl.distributed.
* fix lint.
* fix lint.
* remove local_nids.
* fix lint.
* fix test.
* remove train_dist.
* revert train_sampling.
* rename funcs.
* address comments.
* address comments.
Use NodeDataView/EdgeDataView to keep track of data.
* address comments.
* address comments.
* revert.
* save data with DGL serializer.
* use the right way of getting shape.
* fix lint.
* address comments.
* address comments.
* fix an error in mxnet.
* address comments.
* add edge_map.
* add more test and fix bugs.
Co-authored-by: Zheng <dzzhen@186590dc80ff.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-6-131.us-east-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-26-167.us-east-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-150.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-250.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-135.us-west-2.compute.internal >
2020-05-02 19:30:51 -07:00
Minjie Wang