0d9b6bfd5a
[TensorpipeDeprecation] remove long live server support from DistDGL ( #5931 )
2023-07-03 09:26:27 +08:00
9c5250829f
[Misc] remove unnecessary print() from test_partition.py
2023-06-15 13:55:42 +08:00
1e886cb9cf
[DistDGL] sort node/edge_map to obtain expected id ranges ( #5872 )
2023-06-15 13:53:53 +08:00
0f3223310d
[DistDGL] remove tensorpipe from python APIs ( #5848 )
2023-06-09 16:35:51 +08:00
00972deecf
[GraphBolt] convert dgl partition to CSCSamplingGraph ( #5736 )
2023-05-29 16:49:39 +08:00
fbb26ee5f6
[Dist] fix crash issue if empty test set ( #5653 )
2023-05-11 17:21:31 +08:00
5b409bf7f5
Rename_test ( #5487 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-03-24 18:44:24 +08:00
f0759a96ee
[Misc] Auto-reformat test/. ( #5324 )
...
* auto-format-test
* more
* remove
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-02-19 22:50:37 +08:00
56ffb650fe
[API Deprecation]Deprecate contrib module ( #5114 )
2023-01-06 20:16:01 +08:00
3a2a5031e9
[API Deprecation]Deprecate candidates in dgl.distributed ( #5116 )
2023-01-06 16:17:44 +08:00
f40db9b743
[Dist][Test] Add exit_code asserts in some distributed tests ( #5053 )
...
Signed-off-by: Serge Panev <spanev@nvidia.com >
Signed-off-by: Serge Panev <spanev@nvidia.com >
2022-12-21 09:03:13 +08:00
611393021a
[API Deprecation] Remove candidates in DGLGraph ( #4946 )
2022-12-01 16:41:53 +08:00
cd3fa03035
[Dist] init node/edge data store for Node/EdgeDataView in appropriate place ( #4906 )
...
* [Dist] instantiate NodeDataView in lazy mode
* fix test failure
* init node/edge data store at the very beginning
* fix test failures
* refine comment
* add more tests
2022-11-21 09:52:36 +08:00
6e1cc7dac2
[Dist] fix is_node check according to policy ( #4911 )
...
* [Dist] fix is_node check according to policy
* add more tests
2022-11-17 09:27:50 +08:00
5f0b61b858
[Dist] enable access DistGraph.edges via canonical etype ( #4814 )
...
* [Dist] enable access DistGraph.edges via canonical etype
* refine code
* refine test
* refine code
2022-11-07 21:35:41 +08:00
ed8e9c4440
[Dist] deprecate etype and always use canonical etype for partition and load ( #4777 )
...
* [Dist] deprecate etype and always use canonical etype for partition and load
* enable canonical etypes in dist part pipeline
* resolve rebase conflicts
* fix lint
* fix test failure
* throw exception if outdated part config is loaded
* refine
* refine
* revert unnecessary change
* fix typo
2022-11-04 10:08:18 +08:00
2caa6bd02d
[Dist] Enable save and load for Distributed Optimizer ( #4752 )
...
* add save/load for distributed optimizer
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-19.ap-northeast-1.compute.internal >
2022-11-01 14:30:16 +08:00
2bca4759c1
[Sampling] Enable sampling with edge masks in sample_etype_neighbors ( #4749 )
...
* sample neighbors with masks
* oops
* refactor again
* remove
* remove debug code
* rename macro
* address comments
* more stuff
* remove
* fix
* try fix unit test
* oops
* fix test
* oops
* change name
* rename a lot of stuff
* oops
* ugh
* misc fixes
* lint
* address a lot of comments
* lint
* lint
* fix
* that was silly
* fix
* fix
* fix
* oops
2022-10-29 15:49:44 +08:00
1990e797e1
[Dist] Reduce startup overhead: sort etypes and save in specified formats ( #4735 )
...
* [Dist] reduce startup overhead: enable to save in specified formats
* [Dist] reduce startup overhead: sort partitions when generating
* sort csc/csr only whenmultiple etypes
* refine
2022-10-26 19:07:10 +08:00
b13092172d
[Dist] Reduce peak memory in DistDGL ( #4687 )
...
* [Dist] Reduce peak memory in DistDGL: avoid validation, release memory once loaded
* remove orig_id from ndata/edata for partition_graph()
* delete orig_id from ndata/edata in dist part pipeline
* reduce dtype size and format before saving graphs
* fix lint
* ETYPE requires to be int32/64 for CSRSortByTag
* fix test failure
* refine
2022-10-17 13:35:55 +08:00
408eba2479
Multiple dist dl sampler ( #4704 )
...
* [Dist] enable iterate multiple dist dataloaders simultaneously
* format file
* add support for any number of dataloaders
* fix lint
* refine code
2022-10-12 15:51:45 +08:00
89a4cc4db2
[Misc] Black auto fix. ( #4694 )
...
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal >
2022-10-11 00:12:30 +08:00
5bf17547f7
[DistTensor] add bitwise or function ( #4638 )
...
* first commit
* add test
* fixes
* ah this is how you skip setup
* fix
* ugh
* address comments
* i like black
2022-09-30 19:38:37 +08:00
39987bc599
[Feature] enable graph partition book support canonical etypes ( #4343 )
...
* [Feature] enable graph partition book support canonical etypes
* fix lint
* fix lint
* add todo
* refine according to review comments
* fix lint
* refine naming
* revert PartitionPolicy __init__
* refine docstring
* fix doc string
2022-08-16 11:52:02 +08:00
43ba94ee49
[BugFix] fix etype check in DistGraph.edge_subgraph ( #4322 )
2022-08-03 11:56:00 +08:00
4dd16f5dd8
[BugFix] enable DistGraph.find_edge() works with str or tuple of str ( #4319 )
2022-08-01 20:47:15 +08:00
17f1432ab2
[DistTest] fix incorrect shell if statement ( #4304 )
...
* [DistTest] fix incorrect shell if statement
* fix incorrect use of dist.initialize()
2022-07-28 20:14:47 +08:00
14e4e1b0f7
[Dist] enable to specify sort_etype for sample_etype_neighbours ( #4212 )
...
* [Dist] enable to specify sort_etype for sample_etype_neighbours
* fix lint
* pass argument instead of env
* fix lint and doc string
* refine args
* remove unnecessary lines
* debug only
* debug add sort time log
* change interface
* fix typo
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-07-11 18:37:25 +08:00
c65d6fa55f
[Dist] format dtypes when loading graph in server ( #4228 )
...
* [Dist] format dtypes when loading graph in server
* add test
* refine
* add comments
2022-07-11 14:08:17 +08:00
69226588a5
[Dist] defer to load node/edge feats ( #4143 )
...
* [Dist] defer to load node/edge feats
* fix lint
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* Update python/dgl/distributed/partition.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
* fix lint
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
2022-06-20 19:44:37 +08:00
b258729b3f
[Dist] set socket as default backend for RPC ( #4120 )
...
* [Dist] set socket as default backend for RPC
* add tests both for socket and tensorpipe
2022-06-16 13:08:19 +08:00
abcc9cce83
disable multiple groups tests due to random failure in CI ( #4101 )
2022-06-09 17:40:53 +08:00
cac3720b48
[Dist] enable time out when fetching msg ( #4043 )
...
* [ist] enable time out when fetching msg
* fix lint error
* minor refinements
* improve minor log
* fix dist test
* fix timeout issue in tensorpipe
2022-06-08 20:20:03 +08:00
53835bdb75
[Dist][BugFix] enable sampling on bipartite ( #4014 )
...
* [Dist][BugFix] enable sampling on bipartite
* add comments for tests
2022-05-18 10:14:54 +08:00
22e218d335
[Dist] Enable maximum try times for socket backend via DGL_DIST_MAX_T… ( #3977 )
...
* [Dist] Enable maximum try times for socket backend via DGL_DIST_MAX_TRY_TIMES
* reset env before/after test
* print log for info when trying to connect
* fix
* print log in python instead of cpp
2022-05-11 13:03:06 +08:00
37be02a486
[Feature] enable socket net_type for rpc ( #3951 )
...
* [Feature] enable socket net_type for rpc
* fix lint
* fix lint
* fix build issue on windows
* fix test failure on windows
* fix test failure
* fix cpp unit test failure
* net_type blocking max_try_times
* fix other comments
* fix lint
* fix comment
* fix lint
* fix cpp
2022-04-28 00:10:06 +08:00
e9fd65e9c9
[BugFix] send rpc messages blockingly in case of congestion ( #3867 )
...
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-03-24 09:51:45 +08:00
480a4ae35c
[BugFix] pass ntype/etype into partition book when node/edge_split ( #3828 )
...
* [BugFix] pass ntype/etype into partition book when node/edge_split
* fix test failure
* fix test failue on mxnet
* fix test failure
2022-03-14 15:11:01 -07:00
1d80d91a32
[BugFix] call g.is_homogenous instead of len(g.etypes) ( #3793 )
2022-03-02 14:27:52 +08:00
9c8c162a00
[Fix] sleep for a while when launching clients which will connect to … ( #3704 )
...
* [Fix] sleep for a while when launching clients which will connect to multiple servers
* pre-allocate more ports
* no multiple partitions on single machine
2022-01-30 16:59:26 +08:00
701b4fccc2
[Sampling] New sampling pipeline plus asynchronous prefetching ( #3665 )
...
* initial update
* more
* more
* multi-gpu example
* cluster gcn, finalize homogeneous
* more explanation
* fix
* bunch of fixes
* fix
* RGAT example and more fixes
* shadow-gnn sampler and some changes in unit test
* fix
* wth
* more fixes
* remove shadow+node/edge dataloader tests for possible ux changes
* lints
* add legacy dataloading import just in case
* fix
* update pylint for f-strings
* fix
* lint
* lint
* lint again
* cherry-picking commit fa9f494
* oops
* fix
* add sample_neighbors in dist_graph
* fix
* lint
* fix
* fix
* fix
* fix tutorial
* fix
* fix
* fix
* fix warning
* remove debug
* add get_foo_storage apis
* lint
2022-01-30 16:13:00 +08:00
da877df238
[CI] Migrate to pylint 2.6.0 ( #3698 )
...
* migrate to pylint 2.6.0
* fix
* fix?
* ???
* oops
2022-01-28 18:32:18 +08:00
bf5497a25d
[Fix] reduce concurrent client/server processes to avoid port already in use ( #3697 )
2022-01-28 14:19:15 +08:00
02e4cd8b56
[Feature] long live server for multiple client groups ( #3645 )
...
* [Feature] long live server for multiple client groups
* generate globally unique name for DistTensor within DGL automatically
2022-01-26 15:21:14 +08:00
51651ecadc
Fix test failure in multithread rpc test ( #3660 )
2022-01-19 14:06:57 +08:00
e4cb4a3779
[Fix] reduce error msg, refine fetch logic of available ports ( #3658 )
...
* [Fix] reduce error msg, refine fetch logic of available ports
* un-initialize client before sending shutdown request
* fix import error
* print connect failure log only in debug mode
* enable DMLC_LOG_DEBUG=1 in CI
2022-01-19 09:41:58 +08:00
37467e2589
[Feature][Dist] change TP::Receiver/TP::Sender for multiple connections ( #3574 )
...
* [Feature] enable TP::Receiver wait for any numbers of senders
* fix random unit test failure
* avoid endless future wait
* fix unit test failure
* fix seg fault when finalize wait in receiver
* [Feature] refactor sender connect logic and remove unnecessary sleeps in unit tests
* fix lint
* release RPCContext resources before process exits
* [Debug] TPReceiver wait start log
* [Debug] add log in get port
* [Debug] add log
* [ReDebug] revert time sleep in unit tests
* [Debug] remove sleep for test_distri,test_mp
* [debug] add more log
* [debug] add listen_booted_ flag
* [debug] restore commented code for queue
* [debug] sleep more in rpc_client
* restore change in tests
* Revert "restore change in tests"
This reverts commit 41a18926d181ec2517069389bfc41de2cc949280.
* Revert "[debug] sleep more in rpc_client"
This reverts commit a908e758eabca0a6ce62eb2e59baea02a840ac67.
* Revert "[debug] restore commented code for queue"
This reverts commit d3f993b3746e6bb6e2cc2f90204dd7e9461c6301.
* Revert "[debug] add listen_booted_ flag"
This reverts commit 244b2167d94942ff2a0acec8823b974975e52580.
* Revert "[debug] add more log"
This reverts commit 4b78447b0a575a824821dc7e25cca2246e6e30e2.
* Revert "[Debug] remove sleep for test_distri,test_mp"
This reverts commit e1df1aadcc8b1c2a0013ed77322ac391a8807612.
* remove debug code
* revert unnecessary change
* revert unnecessary changes
* always reset RPCContext when get started and reset all data
* remove time.sleep in dist tests
* fix lint
* reset envs before each dist test
* reset env properly
* add time sleep when start each server
* sleep for a while when boot server
* replace wait_thread with callback
* fix lint
* add dglconnect handshake check
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-01-11 08:55:36 +08:00
a3ce780d29
[RPC] Use tensorpipe for rpc communication ( #3335 )
...
* doesn't know whether works
* add change
* fix
* fix
* fix
* remove
* revert
* lint
* lint
* fix
* revert
* lint
* fix
* only build rpc on linux
* lint
* lint
* fix build on windows
* fix windows
* remove old test
* fix cmake
* Revert "remove old test"
This reverts commit f1ea75c777c34cdc1f08c0589676ba6aee1feb29.
* fix windows
* fix
* fix
* fix indent
* fix indent
* address comment
* fix
* fix
* fix
* fix
* fix
* lint
* fix indent
* fix lint
* add introduction
* fix
* lint
* lint
* add more logs
* fix
* update xbyak for C++14 with gcc5
* Remove channels
* fix
* add test script
* fix
* remove unused file
* fix lint
* add timeout
2021-12-06 20:03:33 +08:00
eb08ef3831
[Distributed] Edge-type-specific fanouts for heterogeneous graphs ( #3558 )
...
* first commit
* second commit
* spaghetti unit tests
* rewrite test
2021-12-06 13:59:29 +08:00
7c7b60be18
[BugFix] add count_nonzero() into SA_Client ( #3417 )
2021-10-12 06:17:30 -07:00
Rhett Ying