* [Fix] sleep for a while when launching clients which will connect to multiple servers
* pre-allocate more ports
* no multiple partitions on single machine
* [Feature] enable TP::Receiver wait for any numbers of senders
* fix random unit test failure
* avoid endless future wait
* fix unit test failure
* fix seg fault when finalize wait in receiver
* [Feature] refactor sender connect logic and remove unnecessary sleeps in unit tests
* fix lint
* release RPCContext resources before process exits
* [Debug] TPReceiver wait start log
* [Debug] add log in get port
* [Debug] add log
* [ReDebug] revert time sleep in unit tests
* [Debug] remove sleep for test_distri,test_mp
* [debug] add more log
* [debug] add listen_booted_ flag
* [debug] restore commented code for queue
* [debug] sleep more in rpc_client
* restore change in tests
* Revert "restore change in tests"
This reverts commit 41a18926d181ec2517069389bfc41de2cc949280.
* Revert "[debug] sleep more in rpc_client"
This reverts commit a908e758eabca0a6ce62eb2e59baea02a840ac67.
* Revert "[debug] restore commented code for queue"
This reverts commit d3f993b3746e6bb6e2cc2f90204dd7e9461c6301.
* Revert "[debug] add listen_booted_ flag"
This reverts commit 244b2167d94942ff2a0acec8823b974975e52580.
* Revert "[debug] add more log"
This reverts commit 4b78447b0a575a824821dc7e25cca2246e6e30e2.
* Revert "[Debug] remove sleep for test_distri,test_mp"
This reverts commit e1df1aadcc8b1c2a0013ed77322ac391a8807612.
* remove debug code
* revert unnecessary change
* revert unnecessary changes
* always reset RPCContext when get started and reset all data
* remove time.sleep in dist tests
* fix lint
* reset envs before each dist test
* reset env properly
* add time sleep when start each server
* sleep for a while when boot server
* replace wait_thread with callback
* fix lint
* add dglconnect handshake check
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
* fix.
* fix.
* fix.
* fix.
* Fix test
* Deprecate old DistEmbedding impl, use synchronized embedding impl
* Basic imple of heterogeneous on homogenenous sampling
* make pass
* Pass C++ test
* Add python test code
* lint
* lint
* Add MultiLayerEtypeNeighborSampler
* Add unitest for single machine dataloader
* Add dist dataloader test for edge type sampler
* Fix lint
* fix
* support for per etype sample
* Fix some bug and enable distributed training with per edge sample
* fix
* Now distributed training works
* turn off some mxnet
* turn off mxnet for some dist test
* fix
* upd
* upd according to the comments
* Fix
* Fix test and now distributed works.
* upd
* upd
* Fix
* Fix bug
* remove dead code.
* upd
* Fix
* upd
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal>
Co-authored-by: Da Zheng <zhengda1936@gmail.com>
* rename get_data_size.
* remove g from DistTensor.
* remove g from DistEmbedding.
* clean up API of graph partition book.
* fix DistGraph
* fix lint.
* collect all part policies.
* fix.
* fix.
* support distributed sampler.
* remove partition.py
* add standalone mode
* add comments.
* add tests for sampling.
* fix.
* make the code to run the standalone mode
* fix
* fix
* fix readme.
* fix.
* fix test
Co-authored-by: Chao Ma <mctt90@gmail.com>
* support no shared memory.
* add test.
* add CAPI to check existence of shared memory.
* revert the change in src/runtime/ndarray.cc
* update docstring.
* fix compile.
* fix tests in graph partition book.
* implement node_split and edge_split.
* fix a bug.
* add tests.
* remove unnecessary code.
* avoid storing graph structure in partition book.
* add partition book in DistGraph.
* Revert "avoid storing graph structure in partition book."
This reverts commit 55b11fbf9293efcae5f8c97e93f1e121399dadae.
* small fixes.
* add a test for boolean mask vector.
* fix test.
* fix bugs.
* make it work for all different cases.
* fix tests.
* fix boolean mask
* fix for TF
* fix for tensorflow.
* fix test for TF
* only support boolean mask for now.
* fix tests.
* make the code more readable.
* fix test.
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
* initial version from distributed training.
This is copied from multiprocessing training.
* modify for distributed training.
* it's runnable now.
* measure time in neighbor sampling.
* simplify neighbor sampling.
* fix a bug in distributed neighbor sampling.
* allow single-machine training.
* fix a bug.
* fix a bug.
* fix openmp.
* make some improvement.
* fix.
* add prepare in the sampler.
* prepare nodeflow async.
* fix a bug.
* get id.
* simplify the code.
* improve.
* fix partition.py
* fix the example.
* add more features.
* fix the example.
* allow one partition
* use distributed kvstore.
* do g2l map manually.
* fix commandline.
* a temp script to save reddit.
* fix pull_handler.
* add pytorch version.
* estimate the time for copying data.
* delete unused code.
* fix a bug.
* print id.
* fix a bug
* fix a bug
* fix a bug.
* remove redundent code.
* revert modify in sampler.
* fix temp script.
* remove pytorch version.
* fix.
* distributed training with pytorch.
* add distributed graph store.
* fix.
* add metis_partition_assignment.
* fix a few bugs in distributed graph store.
* fix test.
* fix bugs in distributed graph store.
* fix tests.
* remove code of defining DistGraphStore.
* fix partition.
* fix example.
* update run.sh.
* only read necessary node data.
* batching data fetch of multiple NodeFlows.
* simplify gcn.
* remove unnecessary code.
* use the new copy_from_kvstore.
* update training script.
* print time in graphsage.
* make distributed training runnable.
* use val_nid.
* fix train_sampling.
* add distributed training.
* add run.sh
* add more timing.
* fix a bug.
* save graph metadata when partition.
* create ndata and edata in distributed graph store.
* add timing in minibatch training of GraphSage.
* use pytorch distributed.
* add checks.
* fix a bug in global vs. local ids.
* remove fast pull
* fix a compile error.
* update and add new APIs.
* implement more methods in DistGraphStore.
* update more APIs.
* rename it to DistGraph.
* rename to DistTensor
* remove some unnecessary API.
* remove unnecessary files.
* revert changes in sampler.
* Revert "simplify gcn."
This reverts commit 0ed3a34ca714203a5b45240af71555d4227ce452.
* Revert "simplify neighbor sampling."
This reverts commit 551c72d20f05a029360ba97f312c7a7a578aacec.
* Revert "measure time in neighbor sampling."
This reverts commit 63ae80c7b402bb626e24acbbc8fdfe9fffd0bc64.
* Revert "add timing in minibatch training of GraphSage."
This reverts commit e59dc8957a414c7df5c316f51d78bce822bdef5e.
* Revert "fix train_sampling."
This reverts commit ea6aea9a4aabb8ba0ff63070aa51e7ca81536ad9.
* fix lint.
* add comments and small update.
* add more comments.
* add more unit tests and fix bugs.
* check the existence of shared-mem graph index.
* use new partitioned graph storage.
* fix bugs.
* print error in fast pull.
* fix lint
* fix a compile error.
* save absolute path after partitioning.
* small fixes in the example
* Revert "[kvstore] support any data type for init_data() (#1465)"
This reverts commit 87b6997bf2.
* fix a bug.
* disable evaluation.
* Revert "Revert "[kvstore] support any data type for init_data() (#1465)""
This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.
* support set and init data.
* support set and init data.
* Revert "Revert "[kvstore] support any data type for init_data() (#1465)""
This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.
* fix bugs.
* fix unit test.
* move to dgl.distributed.
* fix lint.
* fix lint.
* remove local_nids.
* fix lint.
* fix test.
* remove train_dist.
* revert train_sampling.
* rename funcs.
* address comments.
* address comments.
Use NodeDataView/EdgeDataView to keep track of data.
* address comments.
* address comments.
* revert.
* save data with DGL serializer.
* use the right way of getting shape.
* fix lint.
* address comments.
* address comments.
* fix an error in mxnet.
* address comments.
* add edge_map.
* add more test and fix bugs.
Co-authored-by: Zheng <dzzhen@186590dc80ff.ant.amazon.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-6-131.us-east-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-26-167.us-east-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-150.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-250.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-135.us-west-2.compute.internal>
Rhett Ying