6557291fdc
[Distributed] DistDataloader ( #1870 )
...
* 111
* 111
* fix
* 111
* fix
* 11
* fix
* lint
* Update __init__.py
* lint
* fix
* lint
* fix
* fix
* fix
* fix
* fix
* try fix
* try fix
* fix
* Revert "fix"
This reverts commit a0b954fd4e99b7df92b53db8334dcb583d6e1551.
* fixes.
* fix.
* fix test.
* fix exit.
* fix.
* fix
* fix
* lint
* lint
* lint
* fix
* Update .gitignore
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-07-28 16:36:37 +08:00
40950629b2
[Feature] Shared memory utilities ( #1807 )
...
* update
* update
* update
* update
* fix
* update
* fix
* update
* update
* win32
* update
* fix
* update
* update
* update
* updat
* update
* update
* fix
* update
* update
* update
* update
* update
* fix
* TODO
* 111
* fix
* minor fix
* minor fix
* fox
* Update shared_mem_manager.cc
* update
* update
* update
* update metis
* update metis
* update
Co-authored-by: VoVAllen <jz1749@nyu.edu >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-07-28 15:26:15 +08:00
1c1bb77231
hot fix ( #1871 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-68-185.ec2.internal >
2020-07-28 14:46:27 +08:00
44089c8b4d
[Refactor][Graph] Merge DGLGraph and DGLHeteroGraph ( #1862 )
...
* Merge
* [Graph][CUDA] Graph on GPU and many refactoring (#1791 )
* change edge_ids behavior and C++ impl
* fix unittests; remove utils.Index in edge_id
* pass mx and th tests
* pass tf test
* add aten::Scatter_
* Add nonzero; impl CSRGetDataAndIndices/CSRSliceMatrix
* CSRGetData and CSRGetDataAndIndices passed tests
* CSRSliceMatrix basic tests
* fix bug in empty slice
* CUDA CSRHasDuplicate
* has_node; has_edge_between
* predecessors, successors
* deprecate send/recv; fix send_and_recv
* deprecate send/recv; fix send_and_recv
* in_edges; out_edges; all_edges; apply_edges
* in deg/out deg
* subgraph/edge_subgraph
* adj
* in_subgraph/out_subgraph
* sample neighbors
* set/get_n/e_repr
* wip: working on refactoring all idtypes
* pass ndata/edata tests on gpu
* fix
* stash
* workaround nonzero issue
* stash
* nx conversion
* test_hetero_basics except update routines
* test_update_routines
* test_hetero_basics for pytorch
* more fixes
* WIP: flatten graph
* wip: flatten
* test_flatten
* test_to_device
* fix bug in to_homo
* fix bug in CSRSliceMatrix
* pass subgraph test
* fix send_and_recv
* fix filter
* test_heterograph
* passed all pytorch tests
* fix mx unittest
* fix pytorch test_nn
* fix all unittests for PyTorch
* passed all mxnet tests
* lint
* fix tf nn test
* pass all tf tests
* lint
* lint
* change deprecation
* try fix compile
* lint
* update METIDS
* fix utest
* fix
* fix utests
* try debug
* revert
* small fix
* fix utests
* upd
* upd
* upd
* fix
* upd
* upd
* upd
* upd
* upd
* trigger
* +1s
* [kernel] Use heterograph index instead of unitgraph index (#1813 )
* upd
* upd
* upd
* fix
* upd
* upd
* upd
* upd
* upd
* trigger
* +1s
* [Graph] Mutation for Heterograph (#1818 )
* mutation add_nodes and add_edges
* Add support for remove_edges, remove_nodes, add_selfloop, remove_selfloop
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
* upd
* upd
* upd
* fix
* [Transfom] Mutable transform (#1833 )
* add nodesy
* All three
* Fix
* lint
* Add some test case
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* fix
* triger
* Fix
* fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
* [Graph] Migrate Batch & Readout module to heterograph (#1836 )
* dgl.batch
* unbatch
* fix to device
* reduce readout; segment reduce
* change batch_num_nodes|edges to function
* reduce readout/ softmax
* broadcast
* topk
* fix
* fix tf and mx
* fix some ci
* fix batch but unbatch differently
* new checkk
* upd
* upd
* upd
* idtype behavior; code reorg
* idtype behavior; code reorg
* wip: test_basics
* pass test_basics
* WIP: from nx/ to nx
* missing files
* upd
* pass test_basics:test_nx_conversion
* Fix test
* Fix inplace update
* WIP: fixing tests
* upd
* pass test_transform cpu
* pass gpu test_transform
* pass test_batched_graph
* GPU graph auto cast to int32
* missing file
* stash
* WIP: rgcn-hetero
* Fix two datasety
* upd
* weird
* Fix capsuley
* fuck you
* fuck matthias
* Fix dgmg
* fix bug in block degrees; pass rgcn-hetero
* rgcn
* gat and diffpool fix
also fix ppi and tu dataset
* Tree LSTM
* pointcloud
* rrn; wip: sgc
* resolve conflicts
* upd
* sgc and reddit dataset
* upd
* Fix deepwalk, gindt and gcn
* fix datasets and sign
* optimization
* optimization
* upd
* upd
* Fix GIN
* fix bug in add_nodes add_edges; tagcn
* adaptive sampling and gcmc
* upd
* upd
* fix geometric
* fix
* metapath2vec
* fix agnn
* fix pickling problem of block
* fix utests
* miss file
* linegraph
* upd
* upd
* upd
* graphsage
* stgcn_wave
* fix hgt
* on unittests
* Fix transformer
* Fix HAN
* passed pytorch unittests
* lint
* fix
* Fix cluster gcn
* cluster-gcn is ready
* on fixing block related codes
* 2nd order derivative
* Revert "2nd order derivative"
This reverts commit 523bf6c249bee61b51b1ad1babf42aad4167f206.
* passed torch utests again
* fix all mxnet unittests
* delete some useless tests
* pass all tf cpu tests
* disable
* disable distributed unittest
* fix
* fix
* lint
* fix
* fix
* fix script
* fix tutorial
* fix apply edges bug
* fix 2 basics
* fix tutorial
Co-authored-by: yzh119 <expye@outlook.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-42.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-1-5.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-68-185.ec2.internal >
2020-07-28 14:30:41 +08:00
015acfd2d8
[Distributed] Small fixes ( #1859 )
...
* fix node/edge_split.
* fix partition.
* support heterograph interface.
* fix test.
* fix
* fix docstring.
2020-07-27 01:39:48 -07:00
ff8f7082a3
[Distributed] turn off recording on embeddings in the inference. ( #1861 )
...
* turn on/off recording in sparse embedding.
* add test.
2020-07-27 00:42:40 -07:00
6963d796c5
[Distributed] update Embedding API ( #1847 )
...
* update.
* update Embedding.
* add comments.
* fix lint
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-07-26 11:59:41 -07:00
ec2e24be6d
[Feature] Slice dstnode features in NN modules ( #1838 )
...
* slice dstdata from srcdata within nn module
* a bunch of fixes
* add comment
* fix gcmc layer
* repr for blocks
* fix
* fix context
* fix
* do not copy internal columns
* docstring
2020-07-25 21:57:49 +08:00
0e896a921e
update. ( #1857 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal >
2020-07-24 10:47:57 -07:00
53fc639201
[Torch] Fix inter-process communication for PyTorch ( #1858 )
...
* fix
* add comment
2020-07-24 18:15:24 +08:00
62077ef33b
[Distributed] update Metis ( #1856 )
...
* fix bugs.
* improve.
* more updates.
* remove print.
* change the heterograph version.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal >
2020-07-23 21:06:22 -07:00
33abd2757b
[HeteroGraph] Metis partition for hetero ( #1843 )
...
* metis for hetero
* fix partition
2020-07-22 16:31:10 +08:00
562871e76b
[Distributed] add distributed evaluation. ( #1810 )
...
* add eval.
* extend DistTensor.
* fix.
* add barrier.
* add more print.
* add more checks in kvstore.
* fix lint.
* get all neighbors for eval.
* reorganize.
* fix.
* fix.
* fix.
* fix test.
* add reuse_if_exist.
* add test for reuse_if_exist.
* fix lint.
* fix bugs.
* fix.
* print errors of tcp socket.
* support delete tensors.
* fix lint.
* fix
* fix example
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-22 00:40:14 -07:00
9d853977d2
Deprecate old random walk implementations ( #1846 )
2020-07-22 15:08:54 +08:00
41b905da71
Fix #1826 ( #1835 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-07-21 16:32:54 +08:00
5c92f6c218
[RPC] Rpc exit with explicit invocation ( #1825 )
...
* exit client
* update
* update
* update
* update
* update
* update
* update
* update test
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-07-20 14:56:26 +08:00
64557819d9
Basic class for dgl datasetyy ( #1809 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-07-20 13:12:36 +08:00
65e7805e22
[KVStore] Support delete_data() api on client ( #1824 )
...
* delete tensor
* update
* update
* update
* update
* update
* update
* udpate
* update
* update
* update
2020-07-20 12:24:23 +08:00
ea420c0abd
[Distributed] Reduce memory consumption in graph partitioning ( #1823 )
...
* save mem.
* save mem.
* reduce mem
* fix test
* fix lint
* fix test
* fix.
* fix.
* fix.
* fix.
* fix lint.
* fix backend operator.
* fix tensorflow operators.
* fix.
* revert change in mxnet operator.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-18 06:53:48 -07:00
20734637f0
[Distributed] Fix partition ( #1821 )
...
* fix partition and print timing.
* fix lint.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-07-17 16:03:37 -07:00
0e92dad69a
[Transfom] Add support for to_bidirected of heterogeneous graphs ( #1793 )
...
* code
* joint_union
* joint union pass all test
* test case for test_transform
* upd
* lint
* lint
* Fix
* list
* Fix
* Fix
* update to_bidirected impl
* Fix
* Fix
* upd
* remove joint_union
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-07-15 18:01:02 +08:00
cda0abf75a
[Distributed] add the standalone mode in DistGraph ( #1800 )
...
* add standalone mode
* add comments.
* add tests for sampling.
* fix.
* make the code to run the standalone mode
* fix
* fix
* fix readme.
* fix.
* fix test
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-07-14 23:46:25 -07:00
a5722d02a4
[Transform] New ToSimple API ( #1789 )
...
* To simple
* lint
* upd
* more test
* fix
* Fix
* Fix
* add share edata
* doc
* Fix
* Fix
* Fix
* Fix
* lint
* lint
* Fix
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
Co-authored-by: Mufei Li <mufeili1996@gmail.com >
2020-07-15 13:45:32 +08:00
a1472bcff6
[Distributed] Run distributed graph server inside DGL ( #1801 )
...
* run dist server in dgl.
* fix bugs.
* fix example.
* check environment variables and fix lint.
* fix lint
2020-07-14 07:41:44 -07:00
bc209bff94
[bugfix] Fix the deterministic of kernel test and add documentation. ( #1799 )
...
* upd
* upd
* upd
* relax
* relax too
* upd
2020-07-14 04:22:31 +08:00
f25b1a064d
[Feature] Autograd of gspmm and gsddmm on PyTorch/MXNet/Tensorflow ( #1680 )
...
* init
* reverse(by minjie
* upd
* upd
* upd
* upd
* upd
* upd
* upd
* gpu
* upd
* upd
* upd
* upd
* udp
* upd
* upd
* imidiot
* fix
* upd
* upd
* upd
* udp
* upd
* upd
* fix
* udp
* upd
* upd
* upd
* upd
* upd
* fix
* remove redundency
* upd
* upd
* upd
* cache
* upd
* upd
* upd
* upd
* upd
* udp
* upd
* trigger
* upd
* fix
* upd
* unused import
* upd
* upd
2020-07-13 23:21:22 +08:00
c13903bf64
fix batched heterograph serializations ( #1794 )
2020-07-13 19:43:56 +08:00
200340ab38
[Transform] Reverse for heterogenous graph ( #1784 )
...
* reverse
* Add more test
* Fix lint
* Fix
* move to transform
* upd
* upd
* upd
* Add more test
* lint
* Fix
* Fix doc
* Fix
* test
* doc
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-07-13 18:05:44 +08:00
fb9d213848
Fix SAGEConv ( #1792 )
2020-07-12 13:32:14 +08:00
8a183e3fcb
[BugFix] Fix #1779 ( #1781 )
...
* Fix bug #1779
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-07-10 17:35:23 +08:00
33330fcb2d
[Kernel] Line graph with new Kernel ( #1744 )
...
* line graph
* update docstring
* Add test and compile OK
* Add python API
* Fix linty
* Fix
* Fix
* upd
* upd
* Fix
* merge
* Fix
* fix
* upd
* lint
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2020-07-10 12:24:01 +08:00
6a3685bea0
[BUGFIX] fix a bug in graph partitioning. ( #1769 )
...
* fix
* use utils.toindex in the right place.
* fix.
* update tensor for mxnet backend.
* fix
* fix
2020-07-09 01:16:42 -07:00
29e6c93fba
[Kernel] Matrix toSimple ( #1756 )
...
* Matrix to simple
* Pass test
* new impl
* Fix test
* Fix lint
* trigger
* upd
* Fix comments
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-07-09 13:50:41 +08:00
167216afc8
[Distributed] add in_subgraph on DistGraph. ( #1755 )
...
* add in_subgraph on DistGraph.
* check in more.
* fix test.
* add comments.
* fix test.
* update test.
* update.
* rename.
* update comment
* fix test
2020-07-08 14:39:21 -07:00
d6b4e286b6
RGCN minibatch using homogeneous graph. ( #1745 )
...
* minibatch rgcn
* minibatch based rgcn
* remove ogb
* lint
* Add sparse emb
* Fix
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-68-185.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-87-240.ec2.internal >
2020-07-07 00:26:10 +08:00
afb7f5b839
[Distributed] fix init_data in kvstore. ( #1750 )
...
* fix
* fix.
* update.
* fix
* add assert
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-07-06 23:02:00 +08:00
cc73c60cbc
[Bug Fix] Fix Heterogenous graph save crash when some edge type names are identical. ( #1751 )
...
* Fix hetero serialize
* Add test case
y
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-07-04 21:44:53 +08:00
feabcabd97
[Kernel]GraphOp::Reverse ( #1730 )
...
* Compile OK
* Fix compile
* Explore reverse to heterograph_indx and add python test code
* lint
* Fix some bug
* Fix bug
* upd
* Fix
* fix lint
* Fix
* Fix
* test
* triger
* Fix
* upd
* add TODO
* upd
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
Co-authored-by: Zihao Ye <expye@outlook.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-07-04 13:52:47 +08:00
cadcc1c2a6
[Feature] add sparse embedding. ( #1497 )
...
* add sparse embedding.
* fix
* add test.
* man fixes.
* many fixes
* fix sparse emb.
* fix.
* fix lint.
* fix lint.
* fix kvstore.
* expose DistTensor.
* test sparse embeddings.
* add attach_grad to the backends.
* remove part_id
* fix.
* move backward computation.
* move more computation to backend.
* fix a bug when applying learning rate.
* fix a few things.
* fix a few things.
* add docstring
* fix.
* apply no_grad.
* fix tests.
* fix for other frameworks.
* add examples in docstring.
2020-07-03 10:42:09 -07:00
168a88e570
[Sampling] NodeDataLoader for node classification ( #1635 )
...
* neighbor sampler data loader first commit
* more commit
* nodedataloader
* fix
* update RGCN example
* update OGB
* fixes
* fix minibatch RGCN crashing with self loop
* reverting gatconv test code
* fix
* change to new solution that doesn't require tf dataloader
* fix
* lint
* fix
* fixes
* change doc
* fix docstring
* docstring fixes
* return seeds and input nodes from data loader
* fixes
* fix test
* fix windows build problem
* add pytorch wrapper
* fixes
* add pytorch wrapper
* add unit test
* add -1 support to sample_neighbors & fix docstrings
* docstring fix
* lint
* add minibatch rgcn evaluations
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Tong He <hetong007@gmail.com >
2020-07-02 16:31:05 +08:00
781b9cec4c
[Fix] Add RangePartitionBook to __init__ ( #1738 )
...
* update
* update
2020-07-02 16:05:32 +08:00
9eb0efcf2a
[KVStore] Test multi-server and multi-client ( #1732 )
...
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-07-01 20:26:40 -07:00
2c88f7c0b0
[Distributed] Split data to ensure data locality for multiple clients ( #1710 )
...
* fix.
* fix tests.
* fix
* add tests.
* fix.
* have default rank.
* add comment.
* fix test.
* remove check
* simplify code.
* add test.
* split data evenly.
* simplify the distributed training code.
* add comments.
* add comments.
2020-07-01 13:03:40 -07:00
69e6afdf82
Fix crashing on SIOCGIFADDR ( #1733 )
...
During scanning the list of interfaces,
some of them can be configured invalidly:
lack of ip, link down etc.
This patch handles the error
and inform about problem with concrete interface.
2020-07-01 22:47:24 +08:00
18a26fcfb1
[Data] Support HeteroGraph save/load ( #1526 )
...
* 111
* add history version test
* fix
* 111
* save
* ``
* fix1
* 111
* add save heterograph
* lint
* lint
* add tests
* minor fix
* fix
* docs
* add format tets
* use unique_ptr
* fix
* fix interface
* 111
* 111
* fix
* lint
* fix
* add support to s3
* fix
* fix
* fix leak
* fix
* fix docs
* fix
* linlt
* fix
* fix
* fix
* address comment
* address comment
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-06-30 13:34:50 +08:00
dd8d5289ef
[rpc] Add num_clients and sort client ID by machine-id ( #1713 )
...
* add num_clients to kvstore
* update
* update
* update
* update
* update
* update
* update
* fix lint
* update
* update
* update
* add test
* update
* update
* update
* update
* fix test
* update
* update
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2020-06-28 18:40:38 -07:00
5747542f35
[Kernel] Migrate batching/unbatching on adjlist to CSR/COO ( #1687 )
...
* start
* coo csr union partition
* lint
* lint
* lint
* Add matrix->data transform
* update
* Fix window compile
* Add CSR support for DisjointPartition
* lint
* Fix
* Use IdArray Op
* Concat ready
* Fix and all pass
* resolve comments
* Add union COO C++ test
* Add C++ test for csr
* lint
* triger
* Update include
* Fix merge
* test
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-06-28 23:33:55 +08:00
55072b4e83
[Feature] dgl.subgraph support Bool mask tensor as index. ( #1709 )
...
* dgl.subgraph support using Bool mask tensor
* support numpy boo tensor and add some test
* Update
* Fix docstring
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-06-28 22:08:04 +08:00
2f983abe64
fix knn graph bugs ( #1716 )
2020-06-28 17:31:38 +08:00
02d3197407
[Distributed] Pytorch example of distributed GraphSage. ( #1495 )
...
* add train_dist.
* Fix sampling example.
* use distributed sampler.
* fix a bug in DistTensor.
* fix distributed training example.
* add graph partition.
* add command
* disable pytorch parallel.
* shutdown correctly.
* load diff graphs.
* add ip_config.txt.
* record timing for each step.
* use ogb
* add profiler.
* fix a bug.
* add train_dist.
* Fix sampling example.
* use distributed sampler.
* fix a bug in DistTensor.
* fix distributed training example.
* add graph partition.
* add command
* disable pytorch parallel.
* shutdown correctly.
* load diff graphs.
* add ip_config.txt.
* record timing for each step.
* use ogb
* add profiler.
* add Ips of the cluster.
* fix exit.
* support multiple clients.
* balance node types and edges.
* move code.
* remove run.sh
* Revert "support multiple clients."
* fix.
* update train_sampling.
* fix.
* fix
* remove run.sh
* update readme.
* update readme.
* use pytorch distributed.
* ensure all trainers run the same number of steps.
* Update README.md
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-250.us-west-2.compute.internal >
2020-06-28 13:09:40 +08:00
Jinjing Zhou