a49ec9e516
[Tools] use torchrun instead of torch.distributed.launch ( #6304 )
2023-09-11 16:47:03 +08:00
381421b7fb
[TensorpipeDeprecation] remove long live server from launch ( #5920 )
2023-06-29 13:59:59 +08:00
ea48ce7a37
[Misc] Black auto fix. ( #4697 )
...
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal >
2022-10-11 13:31:13 +08:00
ee672c0b4a
Let distributed training launch script report error when any trainer or kvserver fails. ( #4437 )
...
* Collect error reports
* update
* fix
Co-authored-by: root <root@ip-10-0-80-128.ec2.internal >
2022-08-21 00:44:16 -07:00
9501ed6a07
[Dist] master port should be fixed for all trainers ( #4108 )
...
* [Dist] master port should be fixed for all trainers
* add tests for tools/launch.py
2022-06-14 11:22:22 +08:00
966d1aa8b7
[Dist] avoid busy ssh connection ( #4096 )
2022-06-09 19:35:42 +08:00
fcd8ed9acc
[Feature] Launch Long Live Servers and Multiple Client Groups ( #3688 )
...
* enable to launch multiple client groups sequentially
* launch simultaneously is enabled
* refine docstring
* revert unnecessary change
* [DOC] add doc for long live server
* refine
* refine doc
* refine doc
2022-02-09 15:56:38 +08:00
179d6aab1e
[Distributed] Allow user to pass-in extra env parameters when launching a distributed training task. ( #3375 )
...
* Allow user to pass-in extra env parameters when launching a distributed training task.
* Update
* upd
Co-authored-by: xiangsx <xiangsx@ip-10-3-59-214.eu-west-1.compute.internal >
2021-09-23 19:03:33 +08:00
ad61a9a550
[Bugfix] And PYTHONPATH in server launch. ( #3352 )
...
* put PYTHONPATH in server launch
* remove prints
Co-authored-by: xiangsx <xiangsx@ip-10-3-59-214.eu-west-1.compute.internal >
2021-09-14 17:59:16 +08:00
ac01e88082
[Tools] In tools/launch.py, correctly pass all DGL client/server env vars if udf is a multi-command ( #3245 )
...
* Correctly pass all DGL client/server env vars if udf is a multi-command
* Refactor to use wrap_cmd_with_local_envvars() helper fn
2021-08-16 18:54:48 -07:00
c40bbf4f33
[Tools] Refactor tools/launch.py to handle more python binary names ( #3205 )
...
* Refactors torch dist launcher udf-wrap code to handle more python versions
* minor changes
2021-08-02 17:31:02 +08:00
efd0fc6cde
[Tools] Adds --ssh_username to the dist launcher (tools/launch.py) ( #3202 )
2021-07-30 23:34:26 +08:00
18dbaebebc
[Distributed] Specify the graph format for distributed training ( #2948 )
...
* explicitly set the graph format.
* fix.
* fix.
* fix launch script.
* fix readme.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-05-26 18:09:30 +08:00
2d372e35e2
[Distributed] Kill training jobs in distributed training ( #2881 )
...
* kill training jobs.
* update.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-73-81.ec2.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-05-01 12:21:29 +08:00
bfbbefa793
[Distributed] Fix a bug in multiprocessing sampling. ( #2826 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-73-81.ec2.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-04-08 01:29:33 -07:00
86229d426f
[Distributed] Automatically setting the number of OMP threads for trainers ( #2812 )
...
* set omp thread.
* add comment.
* fix.
2021-04-04 11:50:28 -07:00
e36c5db614
[Distributed] Simplify distributed API ( #2775 )
...
* remove num_workers.
* remove num_workers.
* remove num_workers.
* remove num-servers.
* update error message.
* update docstring.
* fix docs.
* fix tests.
* fix test.
* fix.
* print messages in test.
* fix.
* fix test.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
2021-03-29 19:19:23 -07:00
4327d7125d
[Distributed] Check num_workers and num_samplers ( #2108 )
...
* check num_workers
* update
* update
* update
* update
* update
* update
2020-08-27 12:42:25 +08:00
f5d8fa8476
[Distributed] Fix all arguments to the format of xx_xxx ( #2005 )
...
* update
* update
2020-08-12 14:45:30 +08:00
4efa332044
[Distributed] Move server start code to initialize. ( #2002 )
...
* move server start code to initialize.
* fix.
* fix lint.
* fix examples.
* add more checks.
2020-08-11 16:58:18 -07:00
d340ea3aa8
[Distributed] Remove server_count from ip_config.txt ( #1985 )
...
* remove server_count from ip_config.txt
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* lint
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* Update dist_context.py
* fix lint.
* make it work for multiple spaces.
* update ip_config.txt.
* fix examples.
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* udpate
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-11 17:41:48 +08:00
ee30b2aadd
[Distributed] Fix standalone ( #1974 )
...
* fix tests.
* fix.
* remove a test.
* make code work in the standalone mode.
* fix example.
* more fix.
* make DistDataloader work with num_workers=0
* fix DistDataloader tests.
* fix.
* fix lint.
* fix cleanup.
* fix test
* remove unnecessary code.
* remove tests.
* fix.
* fix.
* fix.
* fix example
* fix.
* fix.
* fix launch script.
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-10 15:49:28 -07:00
b9ef70e5ac
[Distributed] Set the number of threads correctly to speed up ( #1976 )
...
* temp fix omp.
* set server threads.
* add CAPI to set up OMP threads.
* fix.
* fix.
* update namesapce.
* set cpi properly.
* allow to config num worker threads.
* set #threads.
* fix.
2020-08-08 19:30:53 -07:00
f0fbbc1632
[Distributed] Fix the launch script. ( #1977 )
...
* update launch script
* check the correctness of launch script.
* fix.
2020-08-08 09:56:25 -07:00
4be4b13424
[Distributed] add copy_partitions.py ( #1866 )
...
* fix bugs.
* eval on both vaidation and testing.
* add script.
* update.
* update launch.
* make train_dist.py independent.
* update readme.
* update readme.
* update readme.
* update readme.
* generate undirected graph.
* rename conf_file to part_config
* use rsync
* make train_dist independent.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-07-31 11:11:56 -07:00
bcb988bdea
[Distributed] Small fix on launch script ( #1867 )
...
* update
* update
* update
* update
2020-07-27 13:44:14 +08:00
7645c6609a
fix launch. ( #1822 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-17 16:25:26 -07:00
ca9d3216d0
[Distributed] Distributed launching script ( #1772 )
...
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* fix launch script.
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-07-16 10:20:57 -07:00
9rum