d8d87243c8
[DistGB] restrict NID/EID as int64_t ( #7177 )
2024-02-29 16:03:34 +08:00
d1bea9e87f
[DistGB] format dtype when converting partition to GraphBolt format ( #7150 )
2024-02-28 09:24:39 +08:00
63541c88f1
[DistGB] enable int32 id ranges in IdMap ( #7144 )
2024-02-26 10:28:59 +08:00
0504bc2c11
[DistGB] use expand_indptr for csc to coo ( #7090 )
2024-02-07 10:08:56 +08:00
571340dac6
[DistGB] update partition_graph to support graphbolt ( #7053 )
2024-02-01 11:37:43 +08:00
af870387fc
[DistGB] add verify logic for GraphBolt partitions ( #7031 )
2024-01-30 16:30:00 +08:00
366fb02fbd
[DistGB] enable to load GB partitions ( #7029 )
2024-01-26 12:38:25 +08:00
0bfe34d96b
[DistGB] refine convertion from dgl to graphbolt ( #7007 )
2024-01-25 14:55:25 +08:00
e9d5156e3c
[GraphBolt] rename from_fused_csc as fused_csc_sampling_graph ( #6732 )
2023-12-12 16:19:41 +08:00
1a8e0c00a9
[GraphBolt] remove GraphMeta and update API accordingly ( #6709 )
2023-12-12 14:49:50 +08:00
c213444268
[GraphBolt] use torch.load/save instead of load/save_fused_xxx() ( #6707 )
2023-12-07 17:17:31 +08:00
3cdc37ccd7
[GraphBolt] add ntype/etype_to_id into graph and save/load() ( #6687 )
2023-12-05 15:56:10 +08:00
91fe0c9069
[GraphBolt] Refactor CSCSamplingGraph and update the corresponding method. ( #6515 )
2023-11-04 03:09:14 +08:00
6519ec2722
[BugFix] Fix API convert_dgl_partition_to_csc_sampling_graph ( #5992 )
2023-07-13 12:32:57 -04:00
1e886cb9cf
[DistDGL] sort node/edge_map to obtain expected id ranges ( #5872 )
2023-06-15 13:53:53 +08:00
d0fa2909ef
[GraphBolt] import when necessary for DistDGL ( #5755 )
2023-06-01 11:22:12 +08:00
00972deecf
[GraphBolt] convert dgl partition to CSCSamplingGraph ( #5736 )
2023-05-29 16:49:39 +08:00
1ab0170a10
[Distributed] Ensure round-robin edge file downloads, reduce logging, other improvements. ( #5578 )
...
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com >
2023-04-27 11:17:55 -07:00
7f5da69709
[Misc] Rename number_of_edges and number_of_nodes to num_edges and num_nodes in dist related python files. ( #5489 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-03-27 15:55:37 +08:00
d2ed57e94b
[DistDGL] add log when load graph and data ( #5350 )
...
* [DistDGL] add log when load graph and data
* fix lint
* refine log
---------
Co-authored-by: Israt Nisa <neesha295@gmail.com >
2023-03-10 14:21:32 +08:00
0b3a447ba4
auto format distributed ( #5317 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-02-19 08:33:09 +08:00
3a2a5031e9
[API Deprecation]Deprecate candidates in dgl.distributed ( #5116 )
2023-01-06 16:17:44 +08:00
ed8e9c4440
[Dist] deprecate etype and always use canonical etype for partition and load ( #4777 )
...
* [Dist] deprecate etype and always use canonical etype for partition and load
* enable canonical etypes in dist part pipeline
* resolve rebase conflicts
* fix lint
* fix test failure
* throw exception if outdated part config is loaded
* refine
* refine
* revert unnecessary change
* fix typo
2022-11-04 10:08:18 +08:00
1990e797e1
[Dist] Reduce startup overhead: sort etypes and save in specified formats ( #4735 )
...
* [Dist] reduce startup overhead: enable to save in specified formats
* [Dist] reduce startup overhead: sort partitions when generating
* sort csc/csr only whenmultiple etypes
* refine
2022-10-26 19:07:10 +08:00
83ada2964d
[Doc] Fix the doc for the partition_graph function ( #4708 )
...
* Fix the doc for the partition_graph func
* Lint for a long line
* Fix a lint
Co-authored-by: Triston Cao <tristonc@nvidia.com >
2022-10-18 12:06:05 +08:00
b13092172d
[Dist] Reduce peak memory in DistDGL ( #4687 )
...
* [Dist] Reduce peak memory in DistDGL: avoid validation, release memory once loaded
* remove orig_id from ndata/edata for partition_graph()
* delete orig_id from ndata/edata in dist part pipeline
* reduce dtype size and format before saving graphs
* fix lint
* ETYPE requires to be int32/64 for CSRSortByTag
* fix test failure
* refine
2022-10-17 13:35:55 +08:00
3bcb268a36
[Doc] Update distributed chapter according to new pipeline ( #4275 )
...
* dist index chapter
* preproc chapter
* rst
* tools page
* partition chapter
* rst
* hetero chapter
* 7.1 step1
* add parmetis back
* changed based on feedback
* address comments
2022-08-17 13:30:04 +08:00
69226588a5
[Dist] defer to load node/edge feats ( #4143 )
...
* [Dist] defer to load node/edge feats
* fix lint
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* fix lint
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
2022-06-20 19:44:37 +08:00
fbbca994b7
[Feature] METIS Partition with Communication Volume Minimization ( #3821 )
...
* upd
* upd
* upd
* upd
* upd
* fix OpenMP compatibility issues
* typo
* partition
* misc
* fix typo
* num_parts=1
* import torch
* long
* print info
* print info
* print info
* upd
* remove debug code
* revert partition.py
* fix cut count
* fix cut count
* Revert "fix cut count"
This reverts commit 10926b4fd48f45c8f1ddb58be7db6c22e653effd.
* Revert "fix cut count"
This reverts commit 76465283bef093a2b4209ad70dd15d2437b2ec8a.
* type of deprecate
* typo in deprecate info
* fix typo
* use cv for partitioning
* CE
* no message
* revert
* typo
* add objtype
* no message
* fix bug
* fix bug
* fix bug
* ?
* semicolon
* drop tensors
* no message
* backward
* backward
* max op
* store X.shape
* th
* test
* Revert "test"
This reverts commit 92b3b2f64a3a1128590098fa03ce429c5466e6ce.
* test
* tolist
* debug
* to cuda
* tuple
* fix bug
* remove X
* no message
* fix bug
* workload balance
* Revert "workload balance"
This reverts commit d7f8e4a16ba2a7eabb4a9bb945523bfe6623e723.
* reverse
* Revert "reverse"
This reverts commit 8a71cf25685aa7d889b9b8881b46f7a16b7d6e6d.
* Revert "Revert "reverse""
This reverts commit 196b143932d5cf9813576ece7c990b63d322d063.
* Revert "Revert "Revert "reverse"""
This reverts commit cf9e89a07013582056e7cde235e51331aca7fa9c.
* no message
* Merge commit '5498cf051e9cf450dcb798ca09920bbde008a661'
# Conflicts:
# python/dgl/distributed/partition.py
* Revert "Merge commit '5498cf051e9cf450dcb798ca09920bbde008a661'"
This reverts commit f79be2ad777897c7025b28308454cad81ad6bb27.
* fix bug
* third party
* no message
* try to avoid memory leak
* try to avoid memory leak
* avoid memory leak with no hope
* Revert "avoid memory leak with no hope"
This reverts commit c77befe9479f46758e744642f66dd209b50eef7d.
* no message
* Revert "no message"
This reverts commit 478cb28fe25fb1002b2f1dc202bb9bdaad8b2a56.
* del
* Revert "del"
This reverts commit 1b468e45ce646b400ff3ffa61a0b2da058b3bdfd.
* no message
* no message
* Revert "no message"
This reverts commit 92e4f5561ed42da0606618b2fff9f1ad5ed439d9.
* third party
* document
* Update metis_partition.cc
* Update metis_partition_hetero.cc
* Update metis_partition_hetero.cc
* Update partition.py
* Update partition.py
* Update partition.py
Co-authored-by: yzh119 <expye@outlook.com >
Co-authored-by: chwan-rice <54331508+chwan-rice@users.noreply.github.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2022-03-27 14:22:01 -07:00
1d80d91a32
[BugFix] call g.is_homogenous instead of len(g.etypes) ( #3793 )
2022-03-02 14:27:52 +08:00
2dd114ed24
[Distributed] Measure peak memory size in graph partitioning ( #3633 )
...
* measure peak memory size.
* fix lint.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal >
2022-01-13 15:41:53 -08:00
0884d02465
[Distributed] Fix bugs in partitioning on heterogeneous graphs. ( #3085 )
...
* fix bugs in partitioning on heterogeneous graphs.
* fix.
* fix.
* fix example.
* fix.
* fix test.
* fix.
* fix.
* fix.
* fix tests.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-02 21:16:14 +08:00
aaec3d8a0b
[Distributed] Support hierarchical partitioning ( #3000 )
...
* add.
* fix.
* fix.
* fix.
* fix.
* add tests.
* support node split and edge split.
* support 1 partition.
* add tests.
* fix.
* fix test.
* use hierarchical partition.
* add check.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-57.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-06-16 16:58:23 +08:00
e156331b0e
return a list of ntypes/etypes ( #2742 )
...
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-05-13 10:42:50 +08:00
d76af4d4c1
[Distributed] Return the ID mapping in graph partitioning. ( #2857 )
...
* return mapping.
* support heterogeneous graph.
* more test.
* fix lint.
* fix for diff backends.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-22 17:51:42 +08:00
7ab3d993a4
Change the absolute path to relative one in part_config.json ( #2690 )
...
* Update partition.py
* fix lint.
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-02-24 16:12:19 -08:00
8a07ab7737
[Doc] Tutorials re-organization ( #2683 )
...
* reorg
* change titles
* rm some stale API doc; minor fix
* fix docs
* add warning
* rm new-tutorial run in ci
* lint
2021-02-20 17:42:45 +08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
9da1629c9d
[Distributed] Some small fixes on doc ( #2049 )
...
* quick fix.
* update sparse optimizer.
* fix.
* fix
2020-08-18 21:13:19 -07:00
75ffc31f76
[Doc] Update the docstring of distributed APIs. ( #2025 )
...
* add doc.
* update DistGraph.
* add DistTensor.
* update DistEmbedding.
* add partition.py
* add sampling.
* fix.
* add graph partition book and create a base class.
* fix test.
* add rst.
* update doc rst.
* update.
* fix.
* fix docs
* update distributed tensor and embeddings.
* add checks.
* update DistGraph.
* update initialization.
* fix graph partition book.
* update graph partition book.
* update partition.
* update partition.
* fix.
* add example code.
* update DistGraph
* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* update initialize.
* update dataloader.
* update distgraph.
* update DistGraph.
* update DistTensor.
* update.
* more updates.
* fix lint.
* add num_nodes and num_edges
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-08-18 11:53:45 -07:00
71f4230a8b
[Fix] OpenMP compatibility issues ( #1992 )
...
* fix OpenMP compatibility issues
* typo
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-08-11 14:26:34 +08:00
a4c931a9bf
[Distributed] Support multiple servers ( #1886 )
...
* client init graph on the backup servers.
* fix.
* test multi-server.
* fix anonymous dist tensors.
* check #parts.
* fix init_data
* add multi-server multi-client tests.
* update tests in kvstore.
* fix.
* verify the loaded partition.
* fix a bug.
* fix lint.
* fix.
* fix example.
* fix rpc.
* fix pull/push handler for backup kvstore
* fix example readme.
* change ip.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-02 19:50:34 -07:00
015acfd2d8
[Distributed] Small fixes ( #1859 )
...
* fix node/edge_split.
* fix partition.
* support heterograph interface.
* fix test.
* fix
* fix docstring.
2020-07-27 01:39:48 -07:00
ea420c0abd
[Distributed] Reduce memory consumption in graph partitioning ( #1823 )
...
* save mem.
* save mem.
* reduce mem
* fix test
* fix lint
* fix test
* fix.
* fix.
* fix.
* fix.
* fix lint.
* fix backend operator.
* fix tensorflow operators.
* fix.
* revert change in mxnet operator.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-18 06:53:48 -07:00
20734637f0
[Distributed] Fix partition ( #1821 )
...
* fix partition and print timing.
* fix lint.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-17 16:03:37 -07:00
a1472bcff6
[Distributed] Run distributed graph server inside DGL ( #1801 )
...
* run dist server in dgl.
* fix bugs.
* fix example.
* check environment variables and fix lint.
* fix lint
2020-07-14 07:41:44 -07:00
04d4680d5b
[Distributed] add constraints for metis partitioning. ( #1708 )
...
* add constraint.
* fix bugs.
* temp fix.
* fix lint
* add balance_ntypes and balance_edges
* add comments.
* fix.
* fix
2020-06-27 21:27:18 -07:00
45e1333e13
[Distributed] Optimize distributed sampling ( #1700 )
...
* optimize sampling neighbors from the local partition.
* fix minior bugs.
* fix lint
* overlap local sampling and remote sampling.
* fix.
* fix lint.
* fix a potential bug.
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-06-26 17:05:29 -07:00
4b96addca0
[BUGFIX] Fix bugs in range partitioning. ( #1703 )
...
* Fix bugs in partitioning.
* add comment.
* fix lint.
* fix test.
* add more tests.
* simplify the fix
* fix.
2020-06-26 15:59:45 -07:00
ff5b5a4a10
[Distributed] support no shared memory. ( #1663 )
...
* support no shared memory.
* add test.
* add CAPI to check existence of shared memory.
* revert the change in src/runtime/ndarray.cc
* update docstring.
* fix compile.
2020-06-18 09:42:11 -07:00
Rhett Ying