2823c61f37
[Distributed] Fix synchronized APIs in kvstore ( #1948 )
...
* handle synchronized API.
* fix.
* fix.
* fix.
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-08-07 19:01:52 -07:00
bab32d5ba1
[Distributed] Fix dataloader ( #1970 )
...
* fix dataloader.
* initialize iterator of DistDataloader correctly.
* update test.
2020-08-07 18:09:21 -07:00
22167f7203
[Refactor] Enable new kernel in all message passing APIs ( #1953 )
...
* WIP: frame refactor
* new frame
* simple update_all builtin
* move all subgraph routines into the same file
* sddmm & spmm schedule; node & edge udf
* degree bucketing
* some tricky 0deg corner cases
* bug in frame append
* merge test_hetero_basics and test_basics
* some code rearange
* fix test_heterograph
* add mean spmm
* enable all builtin combinations
* pass gpu test
* pass pytorch tests
* wip
* fix some pt debugging codes
* fix bug in mxnet backward
* pass all mxnet utests
* passed tf tests
* docstring
* lint
* lint
* fix broadcasting bugs
* add warning and clamp for mean reducer
* add test for zero-degree mean
* address comments
* lint
* small fix
2020-08-07 15:40:25 +08:00
4f499c7ffb
[Distributed] DistDataloader ( #1901 )
...
* 111
* 111
* fix
* 111
* fix
* 11
* fix
* lint
* Update __init__.py
* lint
* fix
* lint
* fix
* fix
* fix
* fix
* fix
* try fix
* try fix
* fix
* Revert "fix"
This reverts commit a0b954fd4e99b7df92b53db8334dcb583d6e1551.
* fixes.
* fix.
* fix test.
* fix exit.
* fix.
* fix
* fix
* lint
* lint
* lint
* fix
* Update .gitignore
* 111
* fix
* 111
* 111
* fff
* 1111
* 111
* 1325315
* ffff
* f???
* fff
* 1111
* 111
* fix
* 111
* asda
* 1111
* 11
* 123
* 啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊
* spawn
* 1231231
* up
* 111
* fix
* fix
* Revert "fix"
This reverts commit 7373f95312fdcaa36d2fc330bf242339e89c045d.
* fix
* fix
* 1111
* fix
* fix tests
* start kvclient as early as possible.
* lint
* fix test
* lint
* 1111
* fix
* fix
* 111
* fix
* fix
* 1
* fix
* fix
* lint
* fix
* lint
* lint
* remove quit
* fix
* lint
* fix
* fix several
* lint
* fix minor
* fix
* lint
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-08-05 17:43:17 +08:00
a4c931a9bf
[Distributed] Support multiple servers ( #1886 )
...
* client init graph on the backup servers.
* fix.
* test multi-server.
* fix anonymous dist tensors.
* check #parts.
* fix init_data
* add multi-server multi-client tests.
* update tests in kvstore.
* fix.
* verify the loaded partition.
* fix a bug.
* fix lint.
* fix.
* fix example.
* fix rpc.
* fix pull/push handler for backup kvstore
* fix example readme.
* change ip.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-02 19:50:34 -07:00
56bbf9cb6d
[Distributed][Example] Distributed training for unsupervised graphsage ( #1853 )
...
* Standalone can run
* fix
* Add save
* Fix
* Fix
* Fix
* Fix
* debug
* test
* test
* Fix
* Fix
* log
* Fix
* fix
* Profile
* auto sync grad
* update
* add test for unsupervised dist training
* upd
* Fix lr
* Fix update
* sync
* fix
* Revert "fix"
This reverts commit d5caa7398b36125f6d6e2c742a95c6ff4298c9e9.
* Fix
* unsupervised
* Fix
* remove debug
* Add test case for dist_graph find_edges()
* Fix
* skip tensorflow test for find_edges
* Update readme
* remove some test
* upd
* Update partition_graph.py
Co-authored-by: Ubuntu <ubuntu@ip-172-31-68-185.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-08-01 12:24:55 -07:00
4be4b13424
[Distributed] add copy_partitions.py ( #1866 )
...
* fix bugs.
* eval on both vaidation and testing.
* add script.
* update.
* update launch.
* make train_dist.py independent.
* update readme.
* update readme.
* update readme.
* update readme.
* generate undirected graph.
* rename conf_file to part_config
* use rsync
* make train_dist independent.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-07-31 11:11:56 -07:00
e5e1a99d46
[KVStore] Fix bug in barrier. ( #1904 )
...
* update
* update
* fix lint
* update
* update
2020-07-31 22:01:03 +08:00
5b515cf667
[KVStore] Support group barrier ( #1880 )
...
* udpate
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* update
* update
* update
* udpate
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-07-30 19:21:20 +08:00
3e2f94ed20
[Distributed] Refactor distributed training to new DGLGraph. ( #1874 )
...
* fix tests in partition.
* fix DistGraph.
* fix without shared memory.
* fix sampling.
* enable distributed test.
* fix tests.
* fix a bug in shared-mem heterograph.
* print better error messages.
* fix.
* don't specify formats.
* fix.
* fix
* small fix.
2020-07-28 22:07:24 -07:00
71cf78652d
Revert "[Distributed] DistDataloader ( #1870 )" ( #1876 )
...
This reverts commit 6557291fdc
2020-07-28 17:57:57 +08:00
6557291fdc
[Distributed] DistDataloader ( #1870 )
...
* 111
* 111
* fix
* 111
* fix
* 11
* fix
* lint
* Update __init__.py
* lint
* fix
* lint
* fix
* fix
* fix
* fix
* fix
* try fix
* try fix
* fix
* Revert "fix"
This reverts commit a0b954fd4e99b7df92b53db8334dcb583d6e1551.
* fixes.
* fix.
* fix test.
* fix exit.
* fix.
* fix
* fix
* lint
* lint
* lint
* fix
* Update .gitignore
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-07-28 16:36:37 +08:00
015acfd2d8
[Distributed] Small fixes ( #1859 )
...
* fix node/edge_split.
* fix partition.
* support heterograph interface.
* fix test.
* fix
* fix docstring.
2020-07-27 01:39:48 -07:00
ff8f7082a3
[Distributed] turn off recording on embeddings in the inference. ( #1861 )
...
* turn on/off recording in sparse embedding.
* add test.
2020-07-27 00:42:40 -07:00
6963d796c5
[Distributed] update Embedding API ( #1847 )
...
* update.
* update Embedding.
* add comments.
* fix lint
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-07-26 11:59:41 -07:00
562871e76b
[Distributed] add distributed evaluation. ( #1810 )
...
* add eval.
* extend DistTensor.
* fix.
* add barrier.
* add more print.
* add more checks in kvstore.
* fix lint.
* get all neighbors for eval.
* reorganize.
* fix.
* fix.
* fix.
* fix test.
* add reuse_if_exist.
* add test for reuse_if_exist.
* fix lint.
* fix bugs.
* fix.
* print errors of tcp socket.
* support delete tensors.
* fix lint.
* fix
* fix example
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-22 00:40:14 -07:00
5c92f6c218
[RPC] Rpc exit with explicit invocation ( #1825 )
...
* exit client
* update
* update
* update
* update
* update
* update
* update
* update test
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-07-20 14:56:26 +08:00
65e7805e22
[KVStore] Support delete_data() api on client ( #1824 )
...
* delete tensor
* update
* update
* update
* update
* update
* update
* udpate
* update
* update
* update
2020-07-20 12:24:23 +08:00
ea420c0abd
[Distributed] Reduce memory consumption in graph partitioning ( #1823 )
...
* save mem.
* save mem.
* reduce mem
* fix test
* fix lint
* fix test
* fix.
* fix.
* fix.
* fix.
* fix lint.
* fix backend operator.
* fix tensorflow operators.
* fix.
* revert change in mxnet operator.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-18 06:53:48 -07:00
20734637f0
[Distributed] Fix partition ( #1821 )
...
* fix partition and print timing.
* fix lint.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-17 16:03:37 -07:00
cda0abf75a
[Distributed] add the standalone mode in DistGraph ( #1800 )
...
* add standalone mode
* add comments.
* add tests for sampling.
* fix.
* make the code to run the standalone mode
* fix
* fix
* fix readme.
* fix.
* fix test
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-07-14 23:46:25 -07:00
a1472bcff6
[Distributed] Run distributed graph server inside DGL ( #1801 )
...
* run dist server in dgl.
* fix bugs.
* fix example.
* check environment variables and fix lint.
* fix lint
2020-07-14 07:41:44 -07:00
6a3685bea0
[BUGFIX] fix a bug in graph partitioning. ( #1769 )
...
* fix
* use utils.toindex in the right place.
* fix.
* update tensor for mxnet backend.
* fix
* fix
2020-07-09 01:16:42 -07:00
167216afc8
[Distributed] add in_subgraph on DistGraph. ( #1755 )
...
* add in_subgraph on DistGraph.
* check in more.
* fix test.
* add comments.
* fix test.
* update test.
* update.
* rename.
* update comment
* fix test
2020-07-08 14:39:21 -07:00
afb7f5b839
[Distributed] fix init_data in kvstore. ( #1750 )
...
* fix
* fix.
* update.
* fix
* add assert
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-07-06 23:02:00 +08:00
cadcc1c2a6
[Feature] add sparse embedding. ( #1497 )
...
* add sparse embedding.
* fix
* add test.
* man fixes.
* many fixes
* fix sparse emb.
* fix.
* fix lint.
* fix lint.
* fix kvstore.
* expose DistTensor.
* test sparse embeddings.
* add attach_grad to the backends.
* remove part_id
* fix.
* move backward computation.
* move more computation to backend.
* fix a bug when applying learning rate.
* fix a few things.
* fix a few things.
* add docstring
* fix.
* apply no_grad.
* fix tests.
* fix for other frameworks.
* add examples in docstring.
2020-07-03 10:42:09 -07:00
781b9cec4c
[Fix] Add RangePartitionBook to __init__ ( #1738 )
...
* update
* update
2020-07-02 16:05:32 +08:00
9eb0efcf2a
[KVStore] Test multi-server and multi-client ( #1732 )
...
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-07-01 20:26:40 -07:00
2c88f7c0b0
[Distributed] Split data to ensure data locality for multiple clients ( #1710 )
...
* fix.
* fix tests.
* fix
* add tests.
* fix.
* have default rank.
* add comment.
* fix test.
* remove check
* simplify code.
* add test.
* split data evenly.
* simplify the distributed training code.
* add comments.
* add comments.
2020-07-01 13:03:40 -07:00
69e6afdf82
Fix crashing on SIOCGIFADDR ( #1733 )
...
During scanning the list of interfaces,
some of them can be configured invalidly:
lack of ip, link down etc.
This patch handles the error
and inform about problem with concrete interface.
2020-07-01 22:47:24 +08:00
18a26fcfb1
[Data] Support HeteroGraph save/load ( #1526 )
...
* 111
* add history version test
* fix
* 111
* save
* ``
* fix1
* 111
* add save heterograph
* lint
* lint
* add tests
* minor fix
* fix
* docs
* add format tets
* use unique_ptr
* fix
* fix interface
* 111
* 111
* fix
* lint
* fix
* add support to s3
* fix
* fix
* fix leak
* fix
* fix docs
* fix
* linlt
* fix
* fix
* fix
* address comment
* address comment
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-06-30 13:34:50 +08:00
dd8d5289ef
[rpc] Add num_clients and sort client ID by machine-id ( #1713 )
...
* add num_clients to kvstore
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* update
* update
* add test
* update
* update
* update
* update
* fix test
* update
* update
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-28 18:40:38 -07:00
02d3197407
[Distributed] Pytorch example of distributed GraphSage. ( #1495 )
...
* add train_dist.
* Fix sampling example.
* use distributed sampler.
* fix a bug in DistTensor.
* fix distributed training example.
* add graph partition.
* add command
* disable pytorch parallel.
* shutdown correctly.
* load diff graphs.
* add ip_config.txt.
* record timing for each step.
* use ogb
* add profiler.
* fix a bug.
* add train_dist.
* Fix sampling example.
* use distributed sampler.
* fix a bug in DistTensor.
* fix distributed training example.
* add graph partition.
* add command
* disable pytorch parallel.
* shutdown correctly.
* load diff graphs.
* add ip_config.txt.
* record timing for each step.
* use ogb
* add profiler.
* add Ips of the cluster.
* fix exit.
* support multiple clients.
* balance node types and edges.
* move code.
* remove run.sh
* Revert "support multiple clients."
* fix.
* update train_sampling.
* fix.
* fix
* remove run.sh
* update readme.
* update readme.
* use pytorch distributed.
* ensure all trainers run the same number of steps.
* Update README.md
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-250.us-west-2.compute.internal >
2020-06-28 13:09:40 +08:00
04d4680d5b
[Distributed] add constraints for metis partitioning. ( #1708 )
...
* add constraint.
* fix bugs.
* temp fix.
* fix lint
* add balance_ntypes and balance_edges
* add comments.
* fix.
* fix
2020-06-27 21:27:18 -07:00
45e1333e13
[Distributed] Optimize distributed sampling ( #1700 )
...
* optimize sampling neighbors from the local partition.
* fix minior bugs.
* fix lint
* overlap local sampling and remote sampling.
* fix.
* fix lint.
* fix a potential bug.
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-06-26 17:05:29 -07:00
4b96addca0
[BUGFIX] Fix bugs in range partitioning. ( #1703 )
...
* Fix bugs in partitioning.
* add comment.
* fix lint.
* fix test.
* add more tests.
* simplify the fix
* fix.
2020-06-26 15:59:45 -07:00
fa1478a4bd
[BUGFIX] fix a bug in partition book. ( #1698 )
...
* fix a bug.
* add msg for assert
2020-06-24 12:38:58 -07:00
7afd1d0d79
[KVStore] Fix a small bug ( #1686 )
...
* update
* update
* update
* update
* update
2020-06-23 19:20:42 +08:00
fc7cd27598
[RPC] accelerate sampling RPC ( #1669 )
...
* support null ndarray
* speed up
* placeholder
* fix
* lint
* lint
* fix
* done
* comment
* Update sampling.py
* Update sampling.py
* Update sampling.py
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-22 11:05:06 -07:00
fb248b678a
[RPC] Sampling over RPC ( #1616 )
...
* fix
* test1111
* 111
* 111
* fff
* lint
* 111
* lint
* lint
* 111
* fijx
* 111
* fix
* 111
* commit
* 111
* 111
* lint
* fix typo
* fix
* lint
* fix
* 111
* support mxnet
* support mxnet
* lint
* remove print
* fix
* fix test.
* fix test.
* fix test.
* try to fix an undetermistic error.
Co-authored-by: Chao Ma <mctt90@gmail.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-18 12:05:23 -07:00
ff5b5a4a10
[Distributed] support no shared memory. ( #1663 )
...
* support no shared memory.
* add test.
* add CAPI to check existence of shared memory.
* revert the change in src/runtime/ndarray.cc
* update docstring.
* fix compile.
2020-06-18 09:42:11 -07:00
dc86bd421e
[Doc] fix a doc for partition_graph. ( #1657 )
...
* fix a doc for partition_graph.
* Update partition.py
* fix.
2020-06-17 22:45:05 -07:00
b9ec3e36ec
[BUGFIX] reshuffle node features in range partitioning. ( #1659 )
...
* fix a bug.
* fix test.
* simplify.
* fix test.
* test edge features.
* fix.
2020-06-17 22:20:17 -07:00
8fccdaf60d
[RPC] HandleCtrlC Register for handling Ctrl+C event. ( #1652 )
...
* update
* update
* update
* update
* fix lint
2020-06-17 14:54:13 +08:00
b372b3c796
[KVStore] Add fast-pull for kvstore ( #1647 )
...
* add fast-pull
* update
* add fast-pull
* update
* update
* update
* update test
* update
* update
* update
* update
* update
* update
* update
* update
* add omp
* update
* update
2020-06-17 13:50:25 +08:00
e8a56dc1b6
[KVStore] make pull/push handler per tensor. ( #1646 )
...
* make pull/push handler per tensor.
* update.
2020-06-15 23:22:49 -07:00
41349dcef5
[Feature] Range partition ( #1522 )
...
* add reorder immutable graph.
* add python API.
* add reorder for csr.
* remove gk version.
* fix
* add cpp test.
* bug fixes
* fix tests.
* fix bugs and add check
* fix test.
* add omp.
* add comments.
* add coo reorder.
* fix a bug.
* handle reorder for different graph structues.
* fix lint.
* fix.
* add original ids.
* reshuffle nodes before metis partition.
* inner nodes are in contiguous Id range.
* reshuffle nodes/edges when partitioning.
* load partition return graph partition book.
* use inner_node/inner_edges
* add and test range partition book.
* count inner_edge correctly.
* fix lint.
* fix lint.
* fix lint.
* fix errors.
* fix errors.
* fix for TF.
* fix.
* fix.
* change docstring.
* support logical and.
* add comments.
* avoid copy.
* fix
* update docstring.
* fix a bug.
* add range search.
* fix
* fix a bug.
* add more tests.
* load graph partition book.
* support shared memory for range partition book.
* fix a bug.
* fix.
* fix lint.
* remove check
* fix test.
* remove num_nodes and num_edges
* fix lint.
* fix graph partition book.
* address comments.
* use makedirs.
* fix compile
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-06-15 11:43:04 -07:00
a67ba946bc
[RPC] Add send_request_to_machine() and remote_call_to_machine() ( #1619 )
...
* add send_request_to_machine()
* update
* update
* update
* update
* update
* update
* fix lint
* update
2020-06-11 13:04:06 +08:00
50962c433f
Refactor distributed graph store ( #1597 )
...
* refactor graph store for new kvstore.
* fix kvstore.
* fix.
* fix lint complains.
* add docstring.
* small fix in graph partition book.
* fix.
* remove tests
* disable sampling.
* Revert "disable sampling."
This reverts commit 70451008f61ff1481d5dadbf8bd199470aee559d.
* Revert "remove tests"
This reverts commit 1394364243bdd73b669abde6193a74e2cda5521d.
2020-06-10 18:51:32 -07:00
484bbcc8f1
[Test] Fix partition ci fail ( #1614 )
2020-06-10 15:44:38 +08:00
Da Zheng