b258729b3f
[Dist] set socket as default backend for RPC ( #4120 )
...
* [Dist] set socket as default backend for RPC
* add tests both for socket and tensorpipe
2022-06-16 13:08:19 +08:00
abcc9cce83
disable multiple groups tests due to random failure in CI ( #4101 )
2022-06-09 17:40:53 +08:00
cac3720b48
[Dist] enable time out when fetching msg ( #4043 )
...
* [ist] enable time out when fetching msg
* fix lint error
* minor refinements
* improve minor log
* fix dist test
* fix timeout issue in tensorpipe
2022-06-08 20:20:03 +08:00
22e218d335
[Dist] Enable maximum try times for socket backend via DGL_DIST_MAX_T… ( #3977 )
...
* [Dist] Enable maximum try times for socket backend via DGL_DIST_MAX_TRY_TIMES
* reset env before/after test
* print log for info when trying to connect
* fix
* print log in python instead of cpp
2022-05-11 13:03:06 +08:00
37be02a486
[Feature] enable socket net_type for rpc ( #3951 )
...
* [Feature] enable socket net_type for rpc
* fix lint
* fix lint
* fix build issue on windows
* fix test failure on windows
* fix test failure
* fix cpp unit test failure
* net_type blocking max_try_times
* fix other comments
* fix lint
* fix comment
* fix lint
* fix cpp
2022-04-28 00:10:06 +08:00
da877df238
[CI] Migrate to pylint 2.6.0 ( #3698 )
...
* migrate to pylint 2.6.0
* fix
* fix?
* ???
* oops
2022-01-28 18:32:18 +08:00
02e4cd8b56
[Feature] long live server for multiple client groups ( #3645 )
...
* [Feature] long live server for multiple client groups
* generate globally unique name for DistTensor within DGL automatically
2022-01-26 15:21:14 +08:00
e4cb4a3779
[Fix] reduce error msg, refine fetch logic of available ports ( #3658 )
...
* [Fix] reduce error msg, refine fetch logic of available ports
* un-initialize client before sending shutdown request
* fix import error
* print connect failure log only in debug mode
* enable DMLC_LOG_DEBUG=1 in CI
2022-01-19 09:41:58 +08:00
37467e2589
[Feature][Dist] change TP::Receiver/TP::Sender for multiple connections ( #3574 )
...
* [Feature] enable TP::Receiver wait for any numbers of senders
* fix random unit test failure
* avoid endless future wait
* fix unit test failure
* fix seg fault when finalize wait in receiver
* [Feature] refactor sender connect logic and remove unnecessary sleeps in unit tests
* fix lint
* release RPCContext resources before process exits
* [Debug] TPReceiver wait start log
* [Debug] add log in get port
* [Debug] add log
* [ReDebug] revert time sleep in unit tests
* [Debug] remove sleep for test_distri,test_mp
* [debug] add more log
* [debug] add listen_booted_ flag
* [debug] restore commented code for queue
* [debug] sleep more in rpc_client
* restore change in tests
* Revert "restore change in tests"
This reverts commit 41a18926d181ec2517069389bfc41de2cc949280.
* Revert "[debug] sleep more in rpc_client"
This reverts commit a908e758eabca0a6ce62eb2e59baea02a840ac67.
* Revert "[debug] restore commented code for queue"
This reverts commit d3f993b3746e6bb6e2cc2f90204dd7e9461c6301.
* Revert "[debug] add listen_booted_ flag"
This reverts commit 244b2167d94942ff2a0acec8823b974975e52580.
* Revert "[debug] add more log"
This reverts commit 4b78447b0a575a824821dc7e25cca2246e6e30e2.
* Revert "[Debug] remove sleep for test_distri,test_mp"
This reverts commit e1df1aadcc8b1c2a0013ed77322ac391a8807612.
* remove debug code
* revert unnecessary change
* revert unnecessary changes
* always reset RPCContext when get started and reset all data
* remove time.sleep in dist tests
* fix lint
* reset envs before each dist test
* reset env properly
* add time sleep when start each server
* sleep for a while when boot server
* replace wait_thread with callback
* fix lint
* add dglconnect handshake check
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-01-11 08:55:36 +08:00
a3ce780d29
[RPC] Use tensorpipe for rpc communication ( #3335 )
...
* doesn't know whether works
* add change
* fix
* fix
* fix
* remove
* revert
* lint
* lint
* fix
* revert
* lint
* fix
* only build rpc on linux
* lint
* lint
* fix build on windows
* fix windows
* remove old test
* fix cmake
* Revert "remove old test"
This reverts commit f1ea75c777c34cdc1f08c0589676ba6aee1feb29.
* fix windows
* fix
* fix
* fix indent
* fix indent
* address comment
* fix
* fix
* fix
* fix
* fix
* lint
* fix indent
* fix lint
* add introduction
* fix
* lint
* lint
* add more logs
* fix
* update xbyak for C++14 with gcc5
* Remove channels
* fix
* add test script
* fix
* remove unused file
* fix lint
* add timeout
2021-12-06 20:03:33 +08:00
3d1f2e870a
delete shared memory when receiving signals. ( #2419 )
...
* delete shared memory when receive signal.
* rename.
* fix lint.
* fix lint.
* fix compile.
* Fix.
* we need to report error if the shared memory exist.
* disable tensorflow test for shared memory.
* revert.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-12-26 10:41:10 -08:00
d340ea3aa8
[Distributed] Remove server_count from ip_config.txt ( #1985 )
...
* remove server_count from ip_config.txt
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* lint
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* Update dist_context.py
* fix lint.
* make it work for multiple spaces.
* update ip_config.txt.
* fix examples.
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* udpate
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-11 17:41:48 +08:00
5b515cf667
[KVStore] Support group barrier ( #1880 )
...
* udpate
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* update
* update
* update
* udpate
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-07-30 19:21:20 +08:00
dd8d5289ef
[rpc] Add num_clients and sort client ID by machine-id ( #1713 )
...
* add num_clients to kvstore
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* update
* update
* add test
* update
* update
* update
* update
* fix test
* update
* update
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-28 18:40:38 -07:00
fb248b678a
[RPC] Sampling over RPC ( #1616 )
...
* fix
* test1111
* 111
* 111
* fff
* lint
* 111
* lint
* lint
* 111
* fijx
* 111
* fix
* 111
* commit
* 111
* 111
* lint
* fix typo
* fix
* lint
* fix
* 111
* support mxnet
* support mxnet
* lint
* remove print
* fix
* fix test.
* fix test.
* fix test.
* try to fix an undetermistic error.
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-18 12:05:23 -07:00
8fccdaf60d
[RPC] HandleCtrlC Register for handling Ctrl+C event. ( #1652 )
...
* update
* update
* update
* update
* fix lint
2020-06-17 14:54:13 +08:00
b372b3c796
[KVStore] Add fast-pull for kvstore ( #1647 )
...
* add fast-pull
* update
* add fast-pull
* update
* update
* update
* update test
* update
* update
* update
* update
* update
* update
* update
* update
* add omp
* update
* update
2020-06-17 13:50:25 +08:00
64f49703af
[KVStore] Re-write kvstore using DGL RPC infrastructure ( #1569 )
...
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update init_data
* update server_state
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* debug init_data
* update
* update
* update
* update
* update
* update
* test get_meta_data
* update
* update
* update
* update
* update
* debug push
* update
* update
* update
* update
* update
* update
* update
* update
* update
* use F.reverse_data_type_dict
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* fix lint
* update
* fix lint
* update
* update
* update
* update
* fix test
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* set random seed
* update
2020-06-04 16:48:53 +08:00
6a363378d5
[RPC] Init server_state during get_server_state() ( #1559 )
...
* init server_state
* update
* update
2020-05-25 15:41:45 +08:00
3e69692233
[RPC] New RPC infrastructure. ( #1549 )
...
* WIP: rpc components
* client & server
* move network package to rpc
* fix include
* fix compile
* c api
* wip: test
* add basic tests
* missing file
* [RPC] Zero copy serializer (#1517 )
* zerocopy serialization
* add test for HeteroGraph
* fix lint
* remove unnecessary codes
* add comment
* lint
* lint
* disable pylint for now
* add include for win
* windows guard
* lint
* lint
* skip test on windows
* refactor
* add comment
* fix
* comment
* 1111
* fix
* Update Jenkinsfile
* [RPC] Implementation of RPC infra (#1544 )
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* remove client.cc and server.cc
* fix lint
* update
* update
* fix linr
* update
* fix lint
* update
* update
* update
* update
* update
* update
* update test
* update
* update test
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update comment
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* fix lint
* fix lint
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-05-22 18:48:37 +08:00
Rhett Ying