0d9b6bfd5a
[TensorpipeDeprecation] remove long live server support from DistDGL ( #5931 )
2023-07-03 09:26:27 +08:00
5b409bf7f5
Rename_test ( #5487 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-03-24 18:44:24 +08:00
f0759a96ee
[Misc] Auto-reformat test/. ( #5324 )
...
* auto-format-test
* more
* remove
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-02-19 22:50:37 +08:00
3a2a5031e9
[API Deprecation]Deprecate candidates in dgl.distributed ( #5116 )
2023-01-06 16:17:44 +08:00
f40db9b743
[Dist][Test] Add exit_code asserts in some distributed tests ( #5053 )
...
Signed-off-by: Serge Panev <spanev@nvidia.com >
Signed-off-by: Serge Panev <spanev@nvidia.com >
2022-12-21 09:03:13 +08:00
611393021a
[API Deprecation] Remove candidates in DGLGraph ( #4946 )
2022-12-01 16:41:53 +08:00
5f0b61b858
[Dist] enable access DistGraph.edges via canonical etype ( #4814 )
...
* [Dist] enable access DistGraph.edges via canonical etype
* refine code
* refine test
* refine code
2022-11-07 21:35:41 +08:00
ed8e9c4440
[Dist] deprecate etype and always use canonical etype for partition and load ( #4777 )
...
* [Dist] deprecate etype and always use canonical etype for partition and load
* enable canonical etypes in dist part pipeline
* resolve rebase conflicts
* fix lint
* fix test failure
* throw exception if outdated part config is loaded
* refine
* refine
* revert unnecessary change
* fix typo
2022-11-04 10:08:18 +08:00
2bca4759c1
[Sampling] Enable sampling with edge masks in sample_etype_neighbors ( #4749 )
...
* sample neighbors with masks
* oops
* refactor again
* remove
* remove debug code
* rename macro
* address comments
* more stuff
* remove
* fix
* try fix unit test
* oops
* fix test
* oops
* change name
* rename a lot of stuff
* oops
* ugh
* misc fixes
* lint
* address a lot of comments
* lint
* lint
* fix
* that was silly
* fix
* fix
* fix
* oops
2022-10-29 15:49:44 +08:00
1990e797e1
[Dist] Reduce startup overhead: sort etypes and save in specified formats ( #4735 )
...
* [Dist] reduce startup overhead: enable to save in specified formats
* [Dist] reduce startup overhead: sort partitions when generating
* sort csc/csr only whenmultiple etypes
* refine
2022-10-26 19:07:10 +08:00
b13092172d
[Dist] Reduce peak memory in DistDGL ( #4687 )
...
* [Dist] Reduce peak memory in DistDGL: avoid validation, release memory once loaded
* remove orig_id from ndata/edata for partition_graph()
* delete orig_id from ndata/edata in dist part pipeline
* reduce dtype size and format before saving graphs
* fix lint
* ETYPE requires to be int32/64 for CSRSortByTag
* fix test failure
* refine
2022-10-17 13:35:55 +08:00
4dd16f5dd8
[BugFix] enable DistGraph.find_edge() works with str or tuple of str ( #4319 )
2022-08-01 20:47:15 +08:00
17f1432ab2
[DistTest] fix incorrect shell if statement ( #4304 )
...
* [DistTest] fix incorrect shell if statement
* fix incorrect use of dist.initialize()
2022-07-28 20:14:47 +08:00
14e4e1b0f7
[Dist] enable to specify sort_etype for sample_etype_neighbours ( #4212 )
...
* [Dist] enable to specify sort_etype for sample_etype_neighbours
* fix lint
* pass argument instead of env
* fix lint and doc string
* refine args
* remove unnecessary lines
* debug only
* debug add sort time log
* change interface
* fix typo
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-07-11 18:37:25 +08:00
abcc9cce83
disable multiple groups tests due to random failure in CI ( #4101 )
2022-06-09 17:40:53 +08:00
53835bdb75
[Dist][BugFix] enable sampling on bipartite ( #4014 )
...
* [Dist][BugFix] enable sampling on bipartite
* add comments for tests
2022-05-18 10:14:54 +08:00
9c8c162a00
[Fix] sleep for a while when launching clients which will connect to … ( #3704 )
...
* [Fix] sleep for a while when launching clients which will connect to multiple servers
* pre-allocate more ports
* no multiple partitions on single machine
2022-01-30 16:59:26 +08:00
02e4cd8b56
[Feature] long live server for multiple client groups ( #3645 )
...
* [Feature] long live server for multiple client groups
* generate globally unique name for DistTensor within DGL automatically
2022-01-26 15:21:14 +08:00
e4cb4a3779
[Fix] reduce error msg, refine fetch logic of available ports ( #3658 )
...
* [Fix] reduce error msg, refine fetch logic of available ports
* un-initialize client before sending shutdown request
* fix import error
* print connect failure log only in debug mode
* enable DMLC_LOG_DEBUG=1 in CI
2022-01-19 09:41:58 +08:00
37467e2589
[Feature][Dist] change TP::Receiver/TP::Sender for multiple connections ( #3574 )
...
* [Feature] enable TP::Receiver wait for any numbers of senders
* fix random unit test failure
* avoid endless future wait
* fix unit test failure
* fix seg fault when finalize wait in receiver
* [Feature] refactor sender connect logic and remove unnecessary sleeps in unit tests
* fix lint
* release RPCContext resources before process exits
* [Debug] TPReceiver wait start log
* [Debug] add log in get port
* [Debug] add log
* [ReDebug] revert time sleep in unit tests
* [Debug] remove sleep for test_distri,test_mp
* [debug] add more log
* [debug] add listen_booted_ flag
* [debug] restore commented code for queue
* [debug] sleep more in rpc_client
* restore change in tests
* Revert "restore change in tests"
This reverts commit 41a18926d181ec2517069389bfc41de2cc949280.
* Revert "[debug] sleep more in rpc_client"
This reverts commit a908e758eabca0a6ce62eb2e59baea02a840ac67.
* Revert "[debug] restore commented code for queue"
This reverts commit d3f993b3746e6bb6e2cc2f90204dd7e9461c6301.
* Revert "[debug] add listen_booted_ flag"
This reverts commit 244b2167d94942ff2a0acec8823b974975e52580.
* Revert "[debug] add more log"
This reverts commit 4b78447b0a575a824821dc7e25cca2246e6e30e2.
* Revert "[Debug] remove sleep for test_distri,test_mp"
This reverts commit e1df1aadcc8b1c2a0013ed77322ac391a8807612.
* remove debug code
* revert unnecessary change
* revert unnecessary changes
* always reset RPCContext when get started and reset all data
* remove time.sleep in dist tests
* fix lint
* reset envs before each dist test
* reset env properly
* add time sleep when start each server
* sleep for a while when boot server
* replace wait_thread with callback
* fix lint
* add dglconnect handshake check
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-01-11 08:55:36 +08:00
f4fe518ff7
[Feature] Add a HINT for the per edge type sampler of heterogeneous DistGraph that highlighting the etypes are sorted already. ( #3260 )
...
* pass cpp test
* distgraph use sorted edge flag.
* lint
* triger
* update test
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-09-01 15:36:01 +08:00
e5ed7adc12
[Distributed] Fix a bug in sampling an empty frontier ( #3298 )
...
* handle empty frontiers.
* fix lint.
* fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-08-29 10:54:46 -07:00
cb4a83a233
[Distributed] Enable distributed EdgeDataLoader ( #3192 )
...
* make heterogeneous find_edges
* add distributed EdgeDataLoader.
* fix.
* fix a bug.
* fix bugs.
* add tests on distributed heterogeneous graph sampling.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-28 15:28:27 +08:00
ba7e7cf931
[New Feature] Per edge type sampler for to_homogeneous graphs. ( #3131 )
...
* fix.
* fix.
* fix.
* fix.
* Fix test
* Deprecate old DistEmbedding impl, use synchronized embedding impl
* Basic imple of heterogeneous on homogenenous sampling
* make pass
* Pass C++ test
* Add python test code
* lint
* lint
* Add MultiLayerEtypeNeighborSampler
* Add unitest for single machine dataloader
* Add dist dataloader test for edge type sampler
* Fix lint
* fix
* support for per etype sample
* Fix some bug and enable distributed training with per edge sample
* fix
* Now distributed training works
* turn off some mxnet
* turn off mxnet for some dist test
* fix
* upd
* upd according to the comments
* Fix
* Fix test and now distributed works.
* upd
* upd
* Fix
* Fix bug
* remove dead code.
* upd
* Fix
* upd
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-07-28 11:51:32 +08:00
34426a98e9
[Distributed] Distributed heterograph training ( #3069 )
...
* support hetero RGCN.
* fix.
* simplify code.
* sample_neighbors return heterograph directly.
* avoid using to_heterogeneous.
* compute canonical etypes in advance.
* fix tests.
* fix.
* fix distributed data loader for heterograph.
* use NodeDataLoader.
* fix bugs in partitioning on heterogeneous graphs.
* fix lint.
* fix tests.
* fix.
* fix.
* fix bugs.
* fix tests.
* fix.
* enable coo for distributed.
* fix.
* fix.
* fix.
* fix.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-17 10:57:00 +08:00
18dbaebebc
[Distributed] Specify the graph format for distributed training ( #2948 )
...
* explicitly set the graph format.
* fix.
* fix.
* fix launch script.
* fix readme.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-05-26 18:09:30 +08:00
6e7f19f27b
[Distributed] add distributed in-degree and out-degree. ( #2918 )
...
* add distributed in-degree and out-degree.
* update comments.
* fix a bug.
* add tests.
* add tests.
* fix a bug.
* fix docstring.
* update doc.
* fix
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-05-18 19:46:09 +08:00
e36c5db614
[Distributed] Simplify distributed API ( #2775 )
...
* remove num_workers.
* remove num_workers.
* remove num_workers.
* remove num-servers.
* update error message.
* update docstring.
* fix docs.
* fix tests.
* fix test.
* fix.
* print messages in test.
* fix.
* fix test.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
2021-03-29 19:19:23 -07:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
1ad46fd015
[Distributed] support distributed sampling in NodeDataLoader ( #2012 )
...
* fix.
* doesn't support EdgeDataLoader
* add test.
* fix.
* fix test.
* catch errors in the test.
* fix.
* fix error message
* add print.
* join the server processes.
* update test.
* fix test.
2020-08-13 22:49:21 -07:00
d1cf5c383e
[Distributed] adjust various APIs. ( #1993 )
...
* rename get_data_size.
* remove g from DistTensor.
* remove g from DistEmbedding.
* clean up API of graph partition book.
* fix DistGraph
* fix lint.
* collect all part policies.
* fix.
* fix.
* support distributed sampler.
* remove partition.py
2020-08-11 20:22:00 -07:00
d340ea3aa8
[Distributed] Remove server_count from ip_config.txt ( #1985 )
...
* remove server_count from ip_config.txt
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* lint
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* Update dist_context.py
* fix lint.
* make it work for multiple spaces.
* update ip_config.txt.
* fix examples.
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* udpate
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-11 17:41:48 +08:00
ee30b2aadd
[Distributed] Fix standalone ( #1974 )
...
* fix tests.
* fix.
* remove a test.
* make code work in the standalone mode.
* fix example.
* more fix.
* make DistDataloader work with num_workers=0
* fix DistDataloader tests.
* fix.
* fix lint.
* fix cleanup.
* fix test
* remove unnecessary code.
* remove tests.
* fix.
* fix.
* fix.
* fix example
* fix.
* fix.
* fix launch script.
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-10 15:49:28 -07:00
4f499c7ffb
[Distributed] DistDataloader ( #1901 )
...
* 111
* 111
* fix
* 111
* fix
* 11
* fix
* lint
* Update __init__.py
* lint
* fix
* lint
* fix
* fix
* fix
* fix
* fix
* try fix
* try fix
* fix
* Revert "fix"
This reverts commit a0b954fd4e99b7df92b53db8334dcb583d6e1551.
* fixes.
* fix.
* fix test.
* fix exit.
* fix.
* fix
* fix
* lint
* lint
* lint
* fix
* Update .gitignore
* 111
* fix
* 111
* 111
* fff
* 1111
* 111
* 1325315
* ffff
* f???
* fff
* 1111
* 111
* fix
* 111
* asda
* 1111
* 11
* 123
* 啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊
* spawn
* 1231231
* up
* 111
* fix
* fix
* Revert "fix"
This reverts commit 7373f95312fdcaa36d2fc330bf242339e89c045d.
* fix
* fix
* 1111
* fix
* fix tests
* start kvclient as early as possible.
* lint
* fix test
* lint
* 1111
* fix
* fix
* 111
* fix
* fix
* 1
* fix
* fix
* lint
* fix
* lint
* lint
* remove quit
* fix
* lint
* fix
* fix several
* lint
* fix minor
* fix
* lint
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-08-05 17:43:17 +08:00
56bbf9cb6d
[Distributed][Example] Distributed training for unsupervised graphsage ( #1853 )
...
* Standalone can run
* fix
* Add save
* Fix
* Fix
* Fix
* Fix
* debug
* test
* test
* Fix
* Fix
* log
* Fix
* fix
* Profile
* auto sync grad
* update
* add test for unsupervised dist training
* upd
* Fix lr
* Fix update
* sync
* fix
* Revert "fix"
This reverts commit d5caa7398b36125f6d6e2c742a95c6ff4298c9e9.
* Fix
* unsupervised
* Fix
* remove debug
* Add test case for dist_graph find_edges()
* Fix
* skip tensorflow test for find_edges
* Update readme
* remove some test
* upd
* Update partition_graph.py
Co-authored-by: Ubuntu <ubuntu@ip-172-31-68-185.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-08-01 12:24:55 -07:00
4be4b13424
[Distributed] add copy_partitions.py ( #1866 )
...
* fix bugs.
* eval on both vaidation and testing.
* add script.
* update.
* update launch.
* make train_dist.py independent.
* update readme.
* update readme.
* update readme.
* update readme.
* generate undirected graph.
* rename conf_file to part_config
* use rsync
* make train_dist independent.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-07-31 11:11:56 -07:00
3e2f94ed20
[Distributed] Refactor distributed training to new DGLGraph. ( #1874 )
...
* fix tests in partition.
* fix DistGraph.
* fix without shared memory.
* fix sampling.
* enable distributed test.
* fix tests.
* fix a bug in shared-mem heterograph.
* print better error messages.
* fix.
* don't specify formats.
* fix.
* fix
* small fix.
2020-07-28 22:07:24 -07:00
5c92f6c218
[RPC] Rpc exit with explicit invocation ( #1825 )
...
* exit client
* update
* update
* update
* update
* update
* update
* update
* update test
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-07-20 14:56:26 +08:00
cda0abf75a
[Distributed] add the standalone mode in DistGraph ( #1800 )
...
* add standalone mode
* add comments.
* add tests for sampling.
* fix.
* make the code to run the standalone mode
* fix
* fix
* fix readme.
* fix.
* fix test
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-07-14 23:46:25 -07:00
a1472bcff6
[Distributed] Run distributed graph server inside DGL ( #1801 )
...
* run dist server in dgl.
* fix bugs.
* fix example.
* check environment variables and fix lint.
* fix lint
2020-07-14 07:41:44 -07:00
167216afc8
[Distributed] add in_subgraph on DistGraph. ( #1755 )
...
* add in_subgraph on DistGraph.
* check in more.
* fix test.
* add comments.
* fix test.
* update test.
* update.
* rename.
* update comment
* fix test
2020-07-08 14:39:21 -07:00
45e1333e13
[Distributed] Optimize distributed sampling ( #1700 )
...
* optimize sampling neighbors from the local partition.
* fix minior bugs.
* fix lint
* overlap local sampling and remote sampling.
* fix.
* fix lint.
* fix a potential bug.
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-06-26 17:05:29 -07:00
fb248b678a
[RPC] Sampling over RPC ( #1616 )
...
* fix
* test1111
* 111
* 111
* fff
* lint
* 111
* lint
* lint
* 111
* fijx
* 111
* fix
* 111
* commit
* 111
* 111
* lint
* fix typo
* fix
* lint
* fix
* 111
* support mxnet
* support mxnet
* lint
* remove print
* fix
* fix test.
* fix test.
* fix test.
* try to fix an undetermistic error.
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-18 12:05:23 -07:00
Rhett Ying