d92c98d927
[distGB] change test_mp_dataloader ( #7819 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-52-174.us-west-2.compute.internal >
2024-10-16 12:52:59 +08:00
bf125d8015
[Dist] fix node/edge map for few nodes/edges ( #7785 )
2024-09-10 08:10:39 +08:00
0734e33e0e
[DistGB] save as graphbolt graph directly after partition test case ( #7724 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-8-126.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-52-174.us-west-2.compute.internal >
2024-09-05 14:08:28 +08:00
ceb6672e63
[GraphBolt] Allow using multiple processes for GraphBolt partition conversion ( #7497 )
2024-07-30 13:28:29 +08:00
4729f7ba7b
[DistGB] add testcase for etype sorted graph ( #7429 )
2024-05-27 15:08:49 +08:00
470769c877
[DistGB] enable find_edges() for GraphBolt partitions ( #7416 )
2024-05-27 14:18:29 +08:00
d8d87243c8
[DistGB] restrict NID/EID as int64_t ( #7177 )
2024-02-29 16:03:34 +08:00
d1bea9e87f
[DistGB] format dtype when converting partition to GraphBolt format ( #7150 )
2024-02-28 09:24:39 +08:00
63541c88f1
[DistGB] enable int32 id ranges in IdMap ( #7144 )
2024-02-26 10:28:59 +08:00
571340dac6
[DistGB] update partition_graph to support graphbolt ( #7053 )
2024-02-01 11:37:43 +08:00
366fb02fbd
[DistGB] enable to load GB partitions ( #7029 )
2024-01-26 12:38:25 +08:00
0bfe34d96b
[DistGB] refine convertion from dgl to graphbolt ( #7007 )
2024-01-25 14:55:25 +08:00
1a8e0c00a9
[GraphBolt] remove GraphMeta and update API accordingly ( #6709 )
2023-12-12 14:49:50 +08:00
c213444268
[GraphBolt] use torch.load/save instead of load/save_fused_xxx() ( #6707 )
2023-12-07 17:17:31 +08:00
3cdc37ccd7
[GraphBolt] add ntype/etype_to_id into graph and save/load() ( #6687 )
2023-12-05 15:56:10 +08:00
19c6491b66
[DistGB] recover dist unit tests while skip unstable ones ( #6582 )
2023-11-21 16:44:10 +08:00
91fe0c9069
[GraphBolt] Refactor CSCSamplingGraph and update the corresponding method. ( #6515 )
2023-11-04 03:09:14 +08:00
9c5250829f
[Misc] remove unnecessary print() from test_partition.py
2023-06-15 13:55:42 +08:00
1e886cb9cf
[DistDGL] sort node/edge_map to obtain expected id ranges ( #5872 )
2023-06-15 13:53:53 +08:00
00972deecf
[GraphBolt] convert dgl partition to CSCSamplingGraph ( #5736 )
2023-05-29 16:49:39 +08:00
5b409bf7f5
Rename_test ( #5487 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-03-24 18:44:24 +08:00
f0759a96ee
[Misc] Auto-reformat test/. ( #5324 )
...
* auto-format-test
* more
* remove
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-02-19 22:50:37 +08:00
3a2a5031e9
[API Deprecation]Deprecate candidates in dgl.distributed ( #5116 )
2023-01-06 16:17:44 +08:00
611393021a
[API Deprecation] Remove candidates in DGLGraph ( #4946 )
2022-12-01 16:41:53 +08:00
6e1cc7dac2
[Dist] fix is_node check according to policy ( #4911 )
...
* [Dist] fix is_node check according to policy
* add more tests
2022-11-17 09:27:50 +08:00
ed8e9c4440
[Dist] deprecate etype and always use canonical etype for partition and load ( #4777 )
...
* [Dist] deprecate etype and always use canonical etype for partition and load
* enable canonical etypes in dist part pipeline
* resolve rebase conflicts
* fix lint
* fix test failure
* throw exception if outdated part config is loaded
* refine
* refine
* revert unnecessary change
* fix typo
2022-11-04 10:08:18 +08:00
1990e797e1
[Dist] Reduce startup overhead: sort etypes and save in specified formats ( #4735 )
...
* [Dist] reduce startup overhead: enable to save in specified formats
* [Dist] reduce startup overhead: sort partitions when generating
* sort csc/csr only whenmultiple etypes
* refine
2022-10-26 19:07:10 +08:00
b13092172d
[Dist] Reduce peak memory in DistDGL ( #4687 )
...
* [Dist] Reduce peak memory in DistDGL: avoid validation, release memory once loaded
* remove orig_id from ndata/edata for partition_graph()
* delete orig_id from ndata/edata in dist part pipeline
* reduce dtype size and format before saving graphs
* fix lint
* ETYPE requires to be int32/64 for CSRSortByTag
* fix test failure
* refine
2022-10-17 13:35:55 +08:00
39987bc599
[Feature] enable graph partition book support canonical etypes ( #4343 )
...
* [Feature] enable graph partition book support canonical etypes
* fix lint
* fix lint
* add todo
* refine according to review comments
* fix lint
* refine naming
* revert PartitionPolicy __init__
* refine docstring
* fix doc string
2022-08-16 11:52:02 +08:00
69226588a5
[Dist] defer to load node/edge feats ( #4143 )
...
* [Dist] defer to load node/edge feats
* fix lint
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* fix lint
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
2022-06-20 19:44:37 +08:00
1d80d91a32
[BugFix] call g.is_homogenous instead of len(g.etypes) ( #3793 )
2022-03-02 14:27:52 +08:00
0884d02465
[Distributed] Fix bugs in partitioning on heterogeneous graphs. ( #3085 )
...
* fix bugs in partitioning on heterogeneous graphs.
* fix.
* fix.
* fix example.
* fix.
* fix test.
* fix.
* fix.
* fix.
* fix tests.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-02 21:16:14 +08:00
6e7f19f27b
[Distributed] add distributed in-degree and out-degree. ( #2918 )
...
* add distributed in-degree and out-degree.
* update comments.
* fix a bug.
* add tests.
* add tests.
* fix a bug.
* fix docstring.
* update doc.
* fix
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-05-18 19:46:09 +08:00
e7046f1e0e
[Distributed] Fix a bug in graph partition. ( #2869 )
...
* update distributed training doc.
* explain data split.
* fix message passing.
* id mapping.
* fix.
* test data reshuffling.
* fix a bug.
* fix test.
* Revert "fix."
This reverts commit 2d025e9e1a5c05c3da9b803a035a788ced59bd77.
* Revert "id mapping."
This reverts commit 2a6a93ceb81fbdff86e6e9e5a58e1ace1e9d9882.
* Revert "fix message passing."
This reverts commit ed8a86bf2b015e5e4f64ba160e81b207ad2a1d65.
* Revert "explain data split."
This reverts commit 4338ddf8a336014cf92d4cb9a1db02b9badc0e55.
* Revert "update distributed training doc."
This reverts commit dda1c35c44536934c19715534f01f832afda6ad2.
* add more tests.
* fix.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-26 18:17:16 +08:00
d76af4d4c1
[Distributed] Return the ID mapping in graph partitioning. ( #2857 )
...
* return mapping.
* support heterogeneous graph.
* more test.
* fix lint.
* fix for diff backends.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-22 17:51:42 +08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
be444e52d9
[Doc/Feature] Refactor, doc update and behavior fix for graphs ( #1983 )
...
* Update graph
* Fix for dgl.graph
* from_scipy
* Replace canonical_etypes with relations
* from_networkx
* Update for hetero_from_relations
* Roll back the change of canonical_etypes to relations
* heterograph
* bipartite
* Update doc
* Fix lint
* Fix lint
* Fix test cases
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* Update
* Fix test
* Fix
* Update
* Use DGLError
* Update
* Update
* Update
* Update
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* Update
* Fix
* Update
* Fix
* Fix
* Fix
* Update
* Fix
* Update
* Fix
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Fix
* Fix
* Update
* Update
* Update
* Update
* Update
* Update
* rewrite sanity checks
* delete unnecessary checks
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Fix
* Update
* Update
* Update
* Fix
* Fix
* Fix
* Update
* Fix
* Update
* Fix
* Fix
* Update
* Fix
* Update
* Fix
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2020-08-18 04:26:29 +08:00
729ff2ef38
Fix the performance issue of graph partitioning in new DGLGraph ( #1934 )
...
* fix perf.
* fix.
* accelerate metis.
* fix lint.
* use gklib.
* fix perf.
* fix.
* update metis.
* update launch script
* handle synchronized API.
* fix.
* fix example.
* fix dataloader.
* temp fix.
* temp fix omp.
* distinguish roles.
* initialize iterator of DistDataloader correctly.
* check the correctness of launch script.
* move feature copy to sampler.
* measure mem/network copy time.
* remove
* Revert "measure mem/network copy time."
This reverts commit 86cefdc14b7815fcf5aad6496af912dba48e4aa6.
* fix.
* fix
* fix.
* fix cmake.
* disable metis in windows.
* disable metis tests in windows.
* remove test for multigraph.
* fix test.
* fix.
* fix cmake.
* fix.
* revert.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-09 18:33:05 -07:00
3e2f94ed20
[Distributed] Refactor distributed training to new DGLGraph. ( #1874 )
...
* fix tests in partition.
* fix DistGraph.
* fix without shared memory.
* fix sampling.
* enable distributed test.
* fix tests.
* fix a bug in shared-mem heterograph.
* print better error messages.
* fix.
* don't specify formats.
* fix.
* fix
* small fix.
2020-07-28 22:07:24 -07:00
33abd2757b
[HeteroGraph] Metis partition for hetero ( #1843 )
...
* metis for hetero
* fix partition
2020-07-22 16:31:10 +08:00
a1472bcff6
[Distributed] Run distributed graph server inside DGL ( #1801 )
...
* run dist server in dgl.
* fix bugs.
* fix example.
* check environment variables and fix lint.
* fix lint
2020-07-14 07:41:44 -07:00
4b96addca0
[BUGFIX] Fix bugs in range partitioning. ( #1703 )
...
* Fix bugs in partitioning.
* add comment.
* fix lint.
* fix test.
* add more tests.
* simplify the fix
* fix.
2020-06-26 15:59:45 -07:00
fa1478a4bd
[BUGFIX] fix a bug in partition book. ( #1698 )
...
* fix a bug.
* add msg for assert
2020-06-24 12:38:58 -07:00
fb248b678a
[RPC] Sampling over RPC ( #1616 )
...
* fix
* test1111
* 111
* 111
* fff
* lint
* 111
* lint
* lint
* 111
* fijx
* 111
* fix
* 111
* commit
* 111
* 111
* lint
* fix typo
* fix
* lint
* fix
* 111
* support mxnet
* support mxnet
* lint
* remove print
* fix
* fix test.
* fix test.
* fix test.
* try to fix an undetermistic error.
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-18 12:05:23 -07:00
b9ec3e36ec
[BUGFIX] reshuffle node features in range partitioning. ( #1659 )
...
* fix a bug.
* fix test.
* simplify.
* fix test.
* test edge features.
* fix.
2020-06-17 22:20:17 -07:00
41349dcef5
[Feature] Range partition ( #1522 )
...
* add reorder immutable graph.
* add python API.
* add reorder for csr.
* remove gk version.
* fix
* add cpp test.
* bug fixes
* fix tests.
* fix bugs and add check
* fix test.
* add omp.
* add comments.
* add coo reorder.
* fix a bug.
* handle reorder for different graph structues.
* fix lint.
* fix.
* add original ids.
* reshuffle nodes before metis partition.
* inner nodes are in contiguous Id range.
* reshuffle nodes/edges when partitioning.
* load partition return graph partition book.
* use inner_node/inner_edges
* add and test range partition book.
* count inner_edge correctly.
* fix lint.
* fix lint.
* fix lint.
* fix errors.
* fix errors.
* fix for TF.
* fix.
* fix.
* change docstring.
* support logical and.
* add comments.
* avoid copy.
* fix
* update docstring.
* fix a bug.
* add range search.
* fix
* fix a bug.
* add more tests.
* load graph partition book.
* support shared memory for range partition book.
* fix a bug.
* fix.
* fix lint.
* remove check
* fix test.
* remove num_nodes and num_edges
* fix lint.
* fix graph partition book.
* address comments.
* use makedirs.
* fix compile
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-06-15 11:43:04 -07:00
77006db209
[CI] Use spawn in unittest instead of fork ( #1627 )
...
* update
* update
* update
* update
* update
* update
* fix ci using spawn
Co-authored-by: aksnzhy <mctt90@gmail.com >
2020-06-11 18:15:40 +08:00
64f49703af
[KVStore] Re-write kvstore using DGL RPC infrastructure ( #1569 )
...
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update init_data
* update server_state
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* debug init_data
* update
* update
* update
* update
* update
* update
* test get_meta_data
* update
* update
* update
* update
* update
* debug push
* update
* update
* update
* update
* update
* update
* update
* update
* update
* use F.reverse_data_type_dict
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* fix lint
* update
* fix lint
* update
* update
* update
* update
* fix test
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* set random seed
* update
2020-06-04 16:48:53 +08:00
ef7e475051
[Distributed] Add GraphPartitionBook ( #1496 )
...
* Add graph-partition-book
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* udpate
* fix lint
* update
* update
* update
* update
* update
* update
* update
* skip test on GPU
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* update
* update
* update
* fix unittest
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-05-08 14:52:12 +08:00
16561a2e6b
[Test] Add tests for TensorFlow ( #1501 )
...
* add test.
* move test code.
* remvoe unnecessary test.
* fix.
* turn on tests for TF.
* Revert "move test code."
This reverts commit e7b4f36395b2121a7be030bd4364a704d0e357bf.
* fix.
* fix.
* skip test for tensorflow.
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-05-06 00:33:53 -07:00
Wenxuan Cao