a6505e86a4
[DistGB] revert toindex() but refine tests ( #7197 )
2024-03-06 09:56:40 +08:00
045beeba14
[DistGB] remove toindex() as torch tensor is always be expected ( #7146 )
2024-02-26 14:04:35 +08:00
7f5da69709
[Misc] Rename number_of_edges and number_of_nodes to num_edges and num_nodes in dist related python files. ( #5489 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-03-27 15:55:37 +08:00
0b3a447ba4
auto format distributed ( #5317 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-02-19 08:33:09 +08:00
2bca4759c1
[Sampling] Enable sampling with edge masks in sample_etype_neighbors ( #4749 )
...
* sample neighbors with masks
* oops
* refactor again
* remove
* remove debug code
* rename macro
* address comments
* more stuff
* remove
* fix
* try fix unit test
* oops
* fix test
* oops
* change name
* rename a lot of stuff
* oops
* ugh
* misc fixes
* lint
* address a lot of comments
* lint
* lint
* fix
* that was silly
* fix
* fix
* fix
* oops
2022-10-29 15:49:44 +08:00
5bf17547f7
[DistTensor] add bitwise or function ( #4638 )
...
* first commit
* add test
* fixes
* ah this is how you skip setup
* fix
* ugh
* address comments
* i like black
2022-09-30 19:38:37 +08:00
02e4cd8b56
[Feature] long live server for multiple client groups ( #3645 )
...
* [Feature] long live server for multiple client groups
* generate globally unique name for DistTensor within DGL automatically
2022-01-26 15:21:14 +08:00
45858d6803
[feature] add count_nonzero function for DistTensor ( #3203 )
...
* add count_nonzero function for DistTensor
* change the load method of local data
Co-authored-by: liuzichang04 <liuzichang04@meituan.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-08-05 10:47:50 -07:00
485c04cff6
[Distributed] Fix a few bugs in distributed API ( #3094 )
...
* fix.
* fix.
* fix.
* fix.
* Fix test
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-07-05 15:12:02 +08:00
975eb8fc5d
[Distributed] Distributed node embedding and sparse optimizer ( #2733 )
...
* Draft for sparse emb
* add some notes
* Fix
* Add sparse optim for dist pytorch
* Update test
* Fix
* upd
* upd
* Fix
* Fix
* Fix bug
* add transductive exmpale
* Fix example
* Some fix
* Upd
* Fix lint
* lint
* lint
* lint
* upd
* Fix lint
* lint
* upd
* remove dead import
* update
* lint
* update unitest
* update example
* Add adam optimizer
* Add unitest and update data
* upd
* upd
* upd
* Fix docstring and fix some bug in example code
* Update rgcn readme
Co-authored-by: Ubuntu <ubuntu@ip-172-31-57-25.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-05-03 08:37:40 +08:00
0fc649523b
[Doc] Distributed heterogeneous graph training ( #2654 )
...
* update DistGraph docstrings.
* add user guide.
* add doc string.
* fix.
* fix.
* fix.
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-02-19 10:48:05 -08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
75ffc31f76
[Doc] Update the docstring of distributed APIs. ( #2025 )
...
* add doc.
* update DistGraph.
* add DistTensor.
* update DistEmbedding.
* add partition.py
* add sampling.
* fix.
* add graph partition book and create a base class.
* fix test.
* add rst.
* update doc rst.
* update.
* fix.
* fix docs
* update distributed tensor and embeddings.
* add checks.
* update DistGraph.
* update initialization.
* fix graph partition book.
* update graph partition book.
* update partition.
* update partition.
* fix.
* add example code.
* update DistGraph
* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* update initialize.
* update dataloader.
* update distgraph.
* update DistGraph.
* update DistTensor.
* update.
* more updates.
* fix lint.
* add num_nodes and num_edges
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-08-18 11:53:45 -07:00
efa95d8673
[Distributed] Fix bugs ( #2009 )
...
* fix.
* add tests.
* fix.
* fix.
2020-08-13 23:51:11 -07:00
1ad46fd015
[Distributed] support distributed sampling in NodeDataLoader ( #2012 )
...
* fix.
* doesn't support EdgeDataLoader
* add test.
* fix.
* fix test.
* catch errors in the test.
* fix.
* fix error message
* add print.
* join the server processes.
* update test.
* fix test.
2020-08-13 22:49:21 -07:00
d1cf5c383e
[Distributed] adjust various APIs. ( #1993 )
...
* rename get_data_size.
* remove g from DistTensor.
* remove g from DistEmbedding.
* clean up API of graph partition book.
* fix DistGraph
* fix lint.
* collect all part policies.
* fix.
* fix.
* support distributed sampler.
* remove partition.py
2020-08-11 20:22:00 -07:00
4f499c7ffb
[Distributed] DistDataloader ( #1901 )
...
* 111
* 111
* fix
* 111
* fix
* 11
* fix
* lint
* Update __init__.py
* lint
* fix
* lint
* fix
* fix
* fix
* fix
* fix
* try fix
* try fix
* fix
* Revert "fix"
This reverts commit a0b954fd4e99b7df92b53db8334dcb583d6e1551.
* fixes.
* fix.
* fix test.
* fix exit.
* fix.
* fix
* fix
* lint
* lint
* lint
* fix
* Update .gitignore
* 111
* fix
* 111
* 111
* fff
* 1111
* 111
* 1325315
* ffff
* f???
* fff
* 1111
* 111
* fix
* 111
* asda
* 1111
* 11
* 123
* 啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊
* spawn
* 1231231
* up
* 111
* fix
* fix
* Revert "fix"
This reverts commit 7373f95312fdcaa36d2fc330bf242339e89c045d.
* fix
* fix
* 1111
* fix
* fix tests
* start kvclient as early as possible.
* lint
* fix test
* lint
* 1111
* fix
* fix
* 111
* fix
* fix
* 1
* fix
* fix
* lint
* fix
* lint
* lint
* remove quit
* fix
* lint
* fix
* fix several
* lint
* fix minor
* fix
* lint
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-08-05 17:43:17 +08:00
a4c931a9bf
[Distributed] Support multiple servers ( #1886 )
...
* client init graph on the backup servers.
* fix.
* test multi-server.
* fix anonymous dist tensors.
* check #parts.
* fix init_data
* add multi-server multi-client tests.
* update tests in kvstore.
* fix.
* verify the loaded partition.
* fix a bug.
* fix lint.
* fix.
* fix example.
* fix rpc.
* fix pull/push handler for backup kvstore
* fix example readme.
* change ip.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-02 19:50:34 -07:00
6963d796c5
[Distributed] update Embedding API ( #1847 )
...
* update.
* update Embedding.
* add comments.
* fix lint
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-07-26 11:59:41 -07:00
562871e76b
[Distributed] add distributed evaluation. ( #1810 )
...
* add eval.
* extend DistTensor.
* fix.
* add barrier.
* add more print.
* add more checks in kvstore.
* fix lint.
* get all neighbors for eval.
* reorganize.
* fix.
* fix.
* fix.
* fix test.
* add reuse_if_exist.
* add test for reuse_if_exist.
* fix lint.
* fix bugs.
* fix.
* print errors of tcp socket.
* support delete tensors.
* fix lint.
* fix
* fix example
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-22 00:40:14 -07:00
Rhett Ying