9c5250829f
[Misc] remove unnecessary print() from test_partition.py
2023-06-15 13:55:42 +08:00
1e886cb9cf
[DistDGL] sort node/edge_map to obtain expected id ranges ( #5872 )
2023-06-15 13:53:53 +08:00
00972deecf
[GraphBolt] convert dgl partition to CSCSamplingGraph ( #5736 )
2023-05-29 16:49:39 +08:00
5b409bf7f5
Rename_test ( #5487 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-03-24 18:44:24 +08:00
f0759a96ee
[Misc] Auto-reformat test/. ( #5324 )
...
* auto-format-test
* more
* remove
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-02-19 22:50:37 +08:00
3a2a5031e9
[API Deprecation]Deprecate candidates in dgl.distributed ( #5116 )
2023-01-06 16:17:44 +08:00
611393021a
[API Deprecation] Remove candidates in DGLGraph ( #4946 )
2022-12-01 16:41:53 +08:00
6e1cc7dac2
[Dist] fix is_node check according to policy ( #4911 )
...
* [Dist] fix is_node check according to policy
* add more tests
2022-11-17 09:27:50 +08:00
ed8e9c4440
[Dist] deprecate etype and always use canonical etype for partition and load ( #4777 )
...
* [Dist] deprecate etype and always use canonical etype for partition and load
* enable canonical etypes in dist part pipeline
* resolve rebase conflicts
* fix lint
* fix test failure
* throw exception if outdated part config is loaded
* refine
* refine
* revert unnecessary change
* fix typo
2022-11-04 10:08:18 +08:00
1990e797e1
[Dist] Reduce startup overhead: sort etypes and save in specified formats ( #4735 )
...
* [Dist] reduce startup overhead: enable to save in specified formats
* [Dist] reduce startup overhead: sort partitions when generating
* sort csc/csr only whenmultiple etypes
* refine
2022-10-26 19:07:10 +08:00
b13092172d
[Dist] Reduce peak memory in DistDGL ( #4687 )
...
* [Dist] Reduce peak memory in DistDGL: avoid validation, release memory once loaded
* remove orig_id from ndata/edata for partition_graph()
* delete orig_id from ndata/edata in dist part pipeline
* reduce dtype size and format before saving graphs
* fix lint
* ETYPE requires to be int32/64 for CSRSortByTag
* fix test failure
* refine
2022-10-17 13:35:55 +08:00
39987bc599
[Feature] enable graph partition book support canonical etypes ( #4343 )
...
* [Feature] enable graph partition book support canonical etypes
* fix lint
* fix lint
* add todo
* refine according to review comments
* fix lint
* refine naming
* revert PartitionPolicy __init__
* refine docstring
* fix doc string
2022-08-16 11:52:02 +08:00
69226588a5
[Dist] defer to load node/edge feats ( #4143 )
...
* [Dist] defer to load node/edge feats
* fix lint
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* fix lint
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
2022-06-20 19:44:37 +08:00
1d80d91a32
[BugFix] call g.is_homogenous instead of len(g.etypes) ( #3793 )
2022-03-02 14:27:52 +08:00
0884d02465
[Distributed] Fix bugs in partitioning on heterogeneous graphs. ( #3085 )
...
* fix bugs in partitioning on heterogeneous graphs.
* fix.
* fix.
* fix example.
* fix.
* fix test.
* fix.
* fix.
* fix.
* fix tests.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-02 21:16:14 +08:00
6e7f19f27b
[Distributed] add distributed in-degree and out-degree. ( #2918 )
...
* add distributed in-degree and out-degree.
* update comments.
* fix a bug.
* add tests.
* add tests.
* fix a bug.
* fix docstring.
* update doc.
* fix
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-05-18 19:46:09 +08:00
e7046f1e0e
[Distributed] Fix a bug in graph partition. ( #2869 )
...
* update distributed training doc.
* explain data split.
* fix message passing.
* id mapping.
* fix.
* test data reshuffling.
* fix a bug.
* fix test.
* Revert "fix."
This reverts commit 2d025e9e1a5c05c3da9b803a035a788ced59bd77.
* Revert "id mapping."
This reverts commit 2a6a93ceb81fbdff86e6e9e5a58e1ace1e9d9882.
* Revert "fix message passing."
This reverts commit ed8a86bf2b015e5e4f64ba160e81b207ad2a1d65.
* Revert "explain data split."
This reverts commit 4338ddf8a336014cf92d4cb9a1db02b9badc0e55.
* Revert "update distributed training doc."
This reverts commit dda1c35c44536934c19715534f01f832afda6ad2.
* add more tests.
* fix.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-26 18:17:16 +08:00
d76af4d4c1
[Distributed] Return the ID mapping in graph partitioning. ( #2857 )
...
* return mapping.
* support heterogeneous graph.
* more test.
* fix lint.
* fix for diff backends.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-22 17:51:42 +08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
be444e52d9
[Doc/Feature] Refactor, doc update and behavior fix for graphs ( #1983 )
...
* Update graph
* Fix for dgl.graph
* from_scipy
* Replace canonical_etypes with relations
* from_networkx
* Update for hetero_from_relations
* Roll back the change of canonical_etypes to relations
* heterograph
* bipartite
* Update doc
* Fix lint
* Fix lint
* Fix test cases
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* Update
* Fix test
* Fix
* Update
* Use DGLError
* Update
* Update
* Update
* Update
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* Update
* Fix
* Update
* Fix
* Fix
* Fix
* Update
* Fix
* Update
* Fix
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Fix
* Fix
* Update
* Update
* Update
* Update
* Update
* Update
* rewrite sanity checks
* delete unnecessary checks
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Fix
* Update
* Update
* Update
* Fix
* Fix
* Fix
* Update
* Fix
* Update
* Fix
* Fix
* Update
* Fix
* Update
* Fix
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2020-08-18 04:26:29 +08:00
729ff2ef38
Fix the performance issue of graph partitioning in new DGLGraph ( #1934 )
...
* fix perf.
* fix.
* accelerate metis.
* fix lint.
* use gklib.
* fix perf.
* fix.
* update metis.
* update launch script
* handle synchronized API.
* fix.
* fix example.
* fix dataloader.
* temp fix.
* temp fix omp.
* distinguish roles.
* initialize iterator of DistDataloader correctly.
* check the correctness of launch script.
* move feature copy to sampler.
* measure mem/network copy time.
* remove
* Revert "measure mem/network copy time."
This reverts commit 86cefdc14b7815fcf5aad6496af912dba48e4aa6.
* fix.
* fix
* fix.
* fix cmake.
* disable metis in windows.
* disable metis tests in windows.
* remove test for multigraph.
* fix test.
* fix.
* fix cmake.
* fix.
* revert.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-09 18:33:05 -07:00
3e2f94ed20
[Distributed] Refactor distributed training to new DGLGraph. ( #1874 )
...
* fix tests in partition.
* fix DistGraph.
* fix without shared memory.
* fix sampling.
* enable distributed test.
* fix tests.
* fix a bug in shared-mem heterograph.
* print better error messages.
* fix.
* don't specify formats.
* fix.
* fix
* small fix.
2020-07-28 22:07:24 -07:00
33abd2757b
[HeteroGraph] Metis partition for hetero ( #1843 )
...
* metis for hetero
* fix partition
2020-07-22 16:31:10 +08:00
a1472bcff6
[Distributed] Run distributed graph server inside DGL ( #1801 )
...
* run dist server in dgl.
* fix bugs.
* fix example.
* check environment variables and fix lint.
* fix lint
2020-07-14 07:41:44 -07:00
4b96addca0
[BUGFIX] Fix bugs in range partitioning. ( #1703 )
...
* Fix bugs in partitioning.
* add comment.
* fix lint.
* fix test.
* add more tests.
* simplify the fix
* fix.
2020-06-26 15:59:45 -07:00
fa1478a4bd
[BUGFIX] fix a bug in partition book. ( #1698 )
...
* fix a bug.
* add msg for assert
2020-06-24 12:38:58 -07:00
fb248b678a
[RPC] Sampling over RPC ( #1616 )
...
* fix
* test1111
* 111
* 111
* fff
* lint
* 111
* lint
* lint
* 111
* fijx
* 111
* fix
* 111
* commit
* 111
* 111
* lint
* fix typo
* fix
* lint
* fix
* 111
* support mxnet
* support mxnet
* lint
* remove print
* fix
* fix test.
* fix test.
* fix test.
* try to fix an undetermistic error.
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-18 12:05:23 -07:00
b9ec3e36ec
[BUGFIX] reshuffle node features in range partitioning. ( #1659 )
...
* fix a bug.
* fix test.
* simplify.
* fix test.
* test edge features.
* fix.
2020-06-17 22:20:17 -07:00
41349dcef5
[Feature] Range partition ( #1522 )
...
* add reorder immutable graph.
* add python API.
* add reorder for csr.
* remove gk version.
* fix
* add cpp test.
* bug fixes
* fix tests.
* fix bugs and add check
* fix test.
* add omp.
* add comments.
* add coo reorder.
* fix a bug.
* handle reorder for different graph structues.
* fix lint.
* fix.
* add original ids.
* reshuffle nodes before metis partition.
* inner nodes are in contiguous Id range.
* reshuffle nodes/edges when partitioning.
* load partition return graph partition book.
* use inner_node/inner_edges
* add and test range partition book.
* count inner_edge correctly.
* fix lint.
* fix lint.
* fix lint.
* fix errors.
* fix errors.
* fix for TF.
* fix.
* fix.
* change docstring.
* support logical and.
* add comments.
* avoid copy.
* fix
* update docstring.
* fix a bug.
* add range search.
* fix
* fix a bug.
* add more tests.
* load graph partition book.
* support shared memory for range partition book.
* fix a bug.
* fix.
* fix lint.
* remove check
* fix test.
* remove num_nodes and num_edges
* fix lint.
* fix graph partition book.
* address comments.
* use makedirs.
* fix compile
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-06-15 11:43:04 -07:00
77006db209
[CI] Use spawn in unittest instead of fork ( #1627 )
...
* update
* update
* update
* update
* update
* update
* fix ci using spawn
Co-authored-by: aksnzhy <mctt90@gmail.com >
2020-06-11 18:15:40 +08:00
64f49703af
[KVStore] Re-write kvstore using DGL RPC infrastructure ( #1569 )
...
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update init_data
* update server_state
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* debug init_data
* update
* update
* update
* update
* update
* update
* test get_meta_data
* update
* update
* update
* update
* update
* debug push
* update
* update
* update
* update
* update
* update
* update
* update
* update
* use F.reverse_data_type_dict
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* fix lint
* update
* fix lint
* update
* update
* update
* update
* fix test
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* set random seed
* update
2020-06-04 16:48:53 +08:00
ef7e475051
[Distributed] Add GraphPartitionBook ( #1496 )
...
* Add graph-partition-book
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* udpate
* fix lint
* update
* update
* update
* update
* update
* update
* update
* skip test on GPU
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* update
* update
* update
* fix unittest
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-05-08 14:52:12 +08:00
16561a2e6b
[Test] Add tests for TensorFlow ( #1501 )
...
* add test.
* move test code.
* remvoe unnecessary test.
* fix.
* turn on tests for TF.
* Revert "move test code."
This reverts commit e7b4f36395b2121a7be030bd4364a704d0e357bf.
* fix.
* fix.
* skip test for tensorflow.
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-05-06 00:33:53 -07:00
2190c39d67
[Feature] Distributed graph store ( #1383 )
...
* initial version from distributed training.
This is copied from multiprocessing training.
* modify for distributed training.
* it's runnable now.
* measure time in neighbor sampling.
* simplify neighbor sampling.
* fix a bug in distributed neighbor sampling.
* allow single-machine training.
* fix a bug.
* fix a bug.
* fix openmp.
* make some improvement.
* fix.
* add prepare in the sampler.
* prepare nodeflow async.
* fix a bug.
* get id.
* simplify the code.
* improve.
* fix partition.py
* fix the example.
* add more features.
* fix the example.
* allow one partition
* use distributed kvstore.
* do g2l map manually.
* fix commandline.
* a temp script to save reddit.
* fix pull_handler.
* add pytorch version.
* estimate the time for copying data.
* delete unused code.
* fix a bug.
* print id.
* fix a bug
* fix a bug
* fix a bug.
* remove redundent code.
* revert modify in sampler.
* fix temp script.
* remove pytorch version.
* fix.
* distributed training with pytorch.
* add distributed graph store.
* fix.
* add metis_partition_assignment.
* fix a few bugs in distributed graph store.
* fix test.
* fix bugs in distributed graph store.
* fix tests.
* remove code of defining DistGraphStore.
* fix partition.
* fix example.
* update run.sh.
* only read necessary node data.
* batching data fetch of multiple NodeFlows.
* simplify gcn.
* remove unnecessary code.
* use the new copy_from_kvstore.
* update training script.
* print time in graphsage.
* make distributed training runnable.
* use val_nid.
* fix train_sampling.
* add distributed training.
* add run.sh
* add more timing.
* fix a bug.
* save graph metadata when partition.
* create ndata and edata in distributed graph store.
* add timing in minibatch training of GraphSage.
* use pytorch distributed.
* add checks.
* fix a bug in global vs. local ids.
* remove fast pull
* fix a compile error.
* update and add new APIs.
* implement more methods in DistGraphStore.
* update more APIs.
* rename it to DistGraph.
* rename to DistTensor
* remove some unnecessary API.
* remove unnecessary files.
* revert changes in sampler.
* Revert "simplify gcn."
This reverts commit 0ed3a34ca714203a5b45240af71555d4227ce452.
* Revert "simplify neighbor sampling."
This reverts commit 551c72d20f05a029360ba97f312c7a7a578aacec.
* Revert "measure time in neighbor sampling."
This reverts commit 63ae80c7b402bb626e24acbbc8fdfe9fffd0bc64.
* Revert "add timing in minibatch training of GraphSage."
This reverts commit e59dc8957a414c7df5c316f51d78bce822bdef5e.
* Revert "fix train_sampling."
This reverts commit ea6aea9a4aabb8ba0ff63070aa51e7ca81536ad9.
* fix lint.
* add comments and small update.
* add more comments.
* add more unit tests and fix bugs.
* check the existence of shared-mem graph index.
* use new partitioned graph storage.
* fix bugs.
* print error in fast pull.
* fix lint
* fix a compile error.
* save absolute path after partitioning.
* small fixes in the example
* Revert "[kvstore] support any data type for init_data() (#1465 )"
This reverts commit 87b6997bf2#1465 )""
This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.
* support set and init data.
* support set and init data.
* Revert "Revert "[kvstore] support any data type for init_data() (#1465 )""
This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.
* fix bugs.
* fix unit test.
* move to dgl.distributed.
* fix lint.
* fix lint.
* remove local_nids.
* fix lint.
* fix test.
* remove train_dist.
* revert train_sampling.
* rename funcs.
* address comments.
* address comments.
Use NodeDataView/EdgeDataView to keep track of data.
* address comments.
* address comments.
* revert.
* save data with DGL serializer.
* use the right way of getting shape.
* fix lint.
* address comments.
* address comments.
* fix an error in mxnet.
* address comments.
* add edge_map.
* add more test and fix bugs.
Co-authored-by: Zheng <dzzhen@186590dc80ff.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-6-131.us-east-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-26-167.us-east-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-150.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-250.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-135.us-west-2.compute.internal >
2020-05-02 19:30:51 -07:00
Rhett Ying