fbbca994b7
[Feature] METIS Partition with Communication Volume Minimization ( #3821 )
...
* upd
* upd
* upd
* upd
* upd
* fix OpenMP compatibility issues
* typo
* partition
* misc
* fix typo
* num_parts=1
* import torch
* long
* print info
* print info
* print info
* upd
* remove debug code
* revert partition.py
* fix cut count
* fix cut count
* Revert "fix cut count"
This reverts commit 10926b4fd48f45c8f1ddb58be7db6c22e653effd.
* Revert "fix cut count"
This reverts commit 76465283bef093a2b4209ad70dd15d2437b2ec8a.
* type of deprecate
* typo in deprecate info
* fix typo
* use cv for partitioning
* CE
* no message
* revert
* typo
* add objtype
* no message
* fix bug
* fix bug
* fix bug
* ?
* semicolon
* drop tensors
* no message
* backward
* backward
* max op
* store X.shape
* th
* test
* Revert "test"
This reverts commit 92b3b2f64a3a1128590098fa03ce429c5466e6ce.
* test
* tolist
* debug
* to cuda
* tuple
* fix bug
* remove X
* no message
* fix bug
* workload balance
* Revert "workload balance"
This reverts commit d7f8e4a16ba2a7eabb4a9bb945523bfe6623e723.
* reverse
* Revert "reverse"
This reverts commit 8a71cf25685aa7d889b9b8881b46f7a16b7d6e6d.
* Revert "Revert "reverse""
This reverts commit 196b143932d5cf9813576ece7c990b63d322d063.
* Revert "Revert "Revert "reverse"""
This reverts commit cf9e89a07013582056e7cde235e51331aca7fa9c.
* no message
* Merge commit '5498cf051e9cf450dcb798ca09920bbde008a661'
# Conflicts:
# python/dgl/distributed/partition.py
* Revert "Merge commit '5498cf051e9cf450dcb798ca09920bbde008a661'"
This reverts commit f79be2ad777897c7025b28308454cad81ad6bb27.
* fix bug
* third party
* no message
* try to avoid memory leak
* try to avoid memory leak
* avoid memory leak with no hope
* Revert "avoid memory leak with no hope"
This reverts commit c77befe9479f46758e744642f66dd209b50eef7d.
* no message
* Revert "no message"
This reverts commit 478cb28fe25fb1002b2f1dc202bb9bdaad8b2a56.
* del
* Revert "del"
This reverts commit 1b468e45ce646b400ff3ffa61a0b2da058b3bdfd.
* no message
* no message
* Revert "no message"
This reverts commit 92e4f5561ed42da0606618b2fff9f1ad5ed439d9.
* third party
* document
* Update metis_partition.cc
* Update metis_partition_hetero.cc
* Update metis_partition_hetero.cc
* Update partition.py
* Update partition.py
* Update partition.py
Co-authored-by: yzh119 <expye@outlook.com >
Co-authored-by: chwan-rice <54331508+chwan-rice@users.noreply.github.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2022-03-27 14:22:01 -07:00
2dd114ed24
[Distributed] Measure peak memory size in graph partitioning ( #3633 )
...
* measure peak memory size.
* fix lint.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal >
2022-01-13 15:41:53 -08:00
2d88db5a3c
[Bug] check dtype before convert to gk ( #3414 )
2021-10-12 14:34:40 +08:00
7f9279397a
[Feature][DistDGL] Add NCCL support for range based partitions ( #3213 )
...
* Implement range based NDArrayPartition
* Finish implement range based partition support
* Add unit test
* Fix whitepace
* Add Kernel suffix
* Fix argument passing
* Add doxygen docs and improve variable naming
* Add unit test
* Add function for converting a partition book
* Add example to partition_op docs
* Fix dtype conversion for mxnet and tensorflow
2021-08-19 21:17:55 -07:00
7359481497
[Feature][GPU] Add function for setting weights of a sparse embedding on multiple GPUs. ( #3047 )
...
* add unit test
* Extend NDArrayPartition object
* Add method for setting embedding, and improve documentation
* Sync before returning
* Use name unique to sparse embedding class to avoid delete
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-06-22 08:48:12 -07:00
ff519f98c3
[API] Standardize Subgraph APIs ( #2929 )
...
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Fix
* Update
* Fix subgraph tests
* Capture stdout for distributed test
* Capture stdout for distributed test
* Update
* Update
* Update
* Update subgraph.cc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-06-21 19:53:37 +08:00
aaec3d8a0b
[Distributed] Support hierarchical partitioning ( #3000 )
...
* add.
* fix.
* fix.
* fix.
* fix.
* add tests.
* support node split and edge split.
* support 1 partition.
* add tests.
* fix.
* fix test.
* use hierarchical partition.
* add check.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-57.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-06-16 16:58:23 +08:00
17d604b5c7
[Feature] Allow using NCCL for communication in dgl.NodeEmbedding and dgl.SparseOptimizer ( #2824 )
...
* Split from NCCL PR
* Fix type in comment
* Expand documentation for sparse_all_to_all_push
* Restore previous behavior in example
* Re-work optimizer to use NCCL based on gradient location
* Allow for running with embedding on CPU but using NCCL for gradient exchange
* Optimize single partition case
* Fix pylint errors
* Add missing include
* fix gradient indexing
* Fix line continuation
* Migrate 'first_step'
* Skip tests without enough GPUs to run NCCL
* Improve empty tensor handling for pytorch 1.5
* Fix indentation
* Allow multiple NCCL communicator to coexist
* Improve handling of empty message
* Update python/dgl/nn/pytorch/sparse_emb.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
* Update python/dgl/nn/pytorch/sparse_emb.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
* Keepy empty tensor dimensionaless
* th.empty -> th.tensor
* Preserve shape for empty non-zero dimension tensors
* Use shared state, when embedding is shared
* Add support for gathering an embedding
* Fix typo
* Fix more typos
* Fix backend call
* Use NodeDataLoader to take advantage of ddp
* Update training script to share memory
* Only squeeze last dimension
* Better handle empty message
* Keep embedding on the target device GPU if dgl_sparse if false in RGCN example
* Fix typo in comment
* Add asserts
* Improve documentation in example
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-06-10 21:19:00 -07:00
ae8dbe6d3c
[Feature][Performance] Implement NCCL wrapper for communicating NodeEmbeddings and sparse gradients. ( #2825 )
...
* Split NCCL wrapper from sparse optimizer and sparse embedding
* Add more unit tests for single node nccl
* Fix unit test for tf
* Switch to device histogram
* Fix histgram issues
* Finish migration to histogram
* Handle cases with zero send/recieve data
* Start on partition object
* Get compiling
* Updates
* Add unit tests
* Switch to partition object
* Fix linting issues
* Rename partition file
* Add python doc
* Fix python assert and finish doxygen comments
* Remove stubs for range based partition to satisfy pylint
* Wrap unit test in GPU only
* Wrap explicit cuda call in ifdef
* Merge with partition.py
* update docstrings
* Cleanup partition_op
* Add Workspace object
* Switch to using workspace object
* Move last remainder based function out of nccl_api
* Add error messages
* Update docs with examples
* Fix linting erros
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-05-20 10:58:17 -07:00
6e7f19f27b
[Distributed] add distributed in-degree and out-degree. ( #2918 )
...
* add distributed in-degree and out-degree.
* update comments.
* fix a bug.
* add tests.
* add tests.
* fix a bug.
* fix docstring.
* update doc.
* fix
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-05-18 19:46:09 +08:00
e7046f1e0e
[Distributed] Fix a bug in graph partition. ( #2869 )
...
* update distributed training doc.
* explain data split.
* fix message passing.
* id mapping.
* fix.
* test data reshuffling.
* fix a bug.
* fix test.
* Revert "fix."
This reverts commit 2d025e9e1a5c05c3da9b803a035a788ced59bd77.
* Revert "id mapping."
This reverts commit 2a6a93ceb81fbdff86e6e9e5a58e1ace1e9d9882.
* Revert "fix message passing."
This reverts commit ed8a86bf2b015e5e4f64ba160e81b207ad2a1d65.
* Revert "explain data split."
This reverts commit 4338ddf8a336014cf92d4cb9a1db02b9badc0e55.
* Revert "update distributed training doc."
This reverts commit dda1c35c44536934c19715534f01f832afda6ad2.
* add more tests.
* fix.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-26 18:17:16 +08:00
d76af4d4c1
[Distributed] Return the ID mapping in graph partitioning. ( #2857 )
...
* return mapping.
* support heterogeneous graph.
* more test.
* fix lint.
* fix for diff backends.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-22 17:51:42 +08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
cd48435206
[Transform] Docstring and subframes ( #1962 )
...
* update knn graph docs
* more docs
* [Doc] transform module docstrings
* remove copy_ndata and copy_edata
* fix
* lint
* fix
* fix
* fix
* clean up docstrings
* fix docstring
* dtype specifications
* addresses comments
* fix
Co-authored-by: Mufei Li <mufeili1996@gmail.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-08-13 10:41:43 +08:00
729ff2ef38
Fix the performance issue of graph partitioning in new DGLGraph ( #1934 )
...
* fix perf.
* fix.
* accelerate metis.
* fix lint.
* use gklib.
* fix perf.
* fix.
* update metis.
* update launch script
* handle synchronized API.
* fix.
* fix example.
* fix dataloader.
* temp fix.
* temp fix omp.
* distinguish roles.
* initialize iterator of DistDataloader correctly.
* check the correctness of launch script.
* move feature copy to sampler.
* measure mem/network copy time.
* remove
* Revert "measure mem/network copy time."
This reverts commit 86cefdc14b7815fcf5aad6496af912dba48e4aa6.
* fix.
* fix
* fix.
* fix cmake.
* disable metis in windows.
* disable metis tests in windows.
* remove test for multigraph.
* fix test.
* fix.
* fix cmake.
* fix.
* revert.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-09 18:33:05 -07:00
879e4ae5f2
[Transform] Add add_reverse_edges ( #1936 )
...
* Add add_reverse_edges and is_simple_graph
Update to_bidirected
* remove is_simple_graph
* Fix lint
* to bidirected only support cpu
* support copy ndata
* upd
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-08-05 15:09:55 +08:00
44089c8b4d
[Refactor][Graph] Merge DGLGraph and DGLHeteroGraph ( #1862 )
...
* Merge
* [Graph][CUDA] Graph on GPU and many refactoring (#1791 )
* change edge_ids behavior and C++ impl
* fix unittests; remove utils.Index in edge_id
* pass mx and th tests
* pass tf test
* add aten::Scatter_
* Add nonzero; impl CSRGetDataAndIndices/CSRSliceMatrix
* CSRGetData and CSRGetDataAndIndices passed tests
* CSRSliceMatrix basic tests
* fix bug in empty slice
* CUDA CSRHasDuplicate
* has_node; has_edge_between
* predecessors, successors
* deprecate send/recv; fix send_and_recv
* deprecate send/recv; fix send_and_recv
* in_edges; out_edges; all_edges; apply_edges
* in deg/out deg
* subgraph/edge_subgraph
* adj
* in_subgraph/out_subgraph
* sample neighbors
* set/get_n/e_repr
* wip: working on refactoring all idtypes
* pass ndata/edata tests on gpu
* fix
* stash
* workaround nonzero issue
* stash
* nx conversion
* test_hetero_basics except update routines
* test_update_routines
* test_hetero_basics for pytorch
* more fixes
* WIP: flatten graph
* wip: flatten
* test_flatten
* test_to_device
* fix bug in to_homo
* fix bug in CSRSliceMatrix
* pass subgraph test
* fix send_and_recv
* fix filter
* test_heterograph
* passed all pytorch tests
* fix mx unittest
* fix pytorch test_nn
* fix all unittests for PyTorch
* passed all mxnet tests
* lint
* fix tf nn test
* pass all tf tests
* lint
* lint
* change deprecation
* try fix compile
* lint
* update METIDS
* fix utest
* fix
* fix utests
* try debug
* revert
* small fix
* fix utests
* upd
* upd
* upd
* fix
* upd
* upd
* upd
* upd
* upd
* trigger
* +1s
* [kernel] Use heterograph index instead of unitgraph index (#1813 )
* upd
* upd
* upd
* fix
* upd
* upd
* upd
* upd
* upd
* trigger
* +1s
* [Graph] Mutation for Heterograph (#1818 )
* mutation add_nodes and add_edges
* Add support for remove_edges, remove_nodes, add_selfloop, remove_selfloop
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
* upd
* upd
* upd
* fix
* [Transfom] Mutable transform (#1833 )
* add nodesy
* All three
* Fix
* lint
* Add some test case
* Fix
* Fix
* Fix
* Fix
* Fix
* Fix
* fix
* triger
* Fix
* fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
* [Graph] Migrate Batch & Readout module to heterograph (#1836 )
* dgl.batch
* unbatch
* fix to device
* reduce readout; segment reduce
* change batch_num_nodes|edges to function
* reduce readout/ softmax
* broadcast
* topk
* fix
* fix tf and mx
* fix some ci
* fix batch but unbatch differently
* new checkk
* upd
* upd
* upd
* idtype behavior; code reorg
* idtype behavior; code reorg
* wip: test_basics
* pass test_basics
* WIP: from nx/ to nx
* missing files
* upd
* pass test_basics:test_nx_conversion
* Fix test
* Fix inplace update
* WIP: fixing tests
* upd
* pass test_transform cpu
* pass gpu test_transform
* pass test_batched_graph
* GPU graph auto cast to int32
* missing file
* stash
* WIP: rgcn-hetero
* Fix two datasety
* upd
* weird
* Fix capsuley
* fuck you
* fuck matthias
* Fix dgmg
* fix bug in block degrees; pass rgcn-hetero
* rgcn
* gat and diffpool fix
also fix ppi and tu dataset
* Tree LSTM
* pointcloud
* rrn; wip: sgc
* resolve conflicts
* upd
* sgc and reddit dataset
* upd
* Fix deepwalk, gindt and gcn
* fix datasets and sign
* optimization
* optimization
* upd
* upd
* Fix GIN
* fix bug in add_nodes add_edges; tagcn
* adaptive sampling and gcmc
* upd
* upd
* fix geometric
* fix
* metapath2vec
* fix agnn
* fix pickling problem of block
* fix utests
* miss file
* linegraph
* upd
* upd
* upd
* graphsage
* stgcn_wave
* fix hgt
* on unittests
* Fix transformer
* Fix HAN
* passed pytorch unittests
* lint
* fix
* Fix cluster gcn
* cluster-gcn is ready
* on fixing block related codes
* 2nd order derivative
* Revert "2nd order derivative"
This reverts commit 523bf6c249bee61b51b1ad1babf42aad4167f206.
* passed torch utests again
* fix all mxnet unittests
* delete some useless tests
* pass all tf cpu tests
* disable
* disable distributed unittest
* fix
* fix
* lint
* fix
* fix
* fix script
* fix tutorial
* fix apply edges bug
* fix 2 basics
* fix tutorial
Co-authored-by: yzh119 <expye@outlook.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-42.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-1-5.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-68-185.ec2.internal >
2020-07-28 14:30:41 +08:00
0e896a921e
update. ( #1857 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal >
2020-07-24 10:47:57 -07:00
62077ef33b
[Distributed] update Metis ( #1856 )
...
* fix bugs.
* improve.
* more updates.
* remove print.
* change the heterograph version.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal >
2020-07-23 21:06:22 -07:00
33abd2757b
[HeteroGraph] Metis partition for hetero ( #1843 )
...
* metis for hetero
* fix partition
2020-07-22 16:31:10 +08:00
Cheng Wan