d9a2d74167
[Performance][Feature] Implement edge excluding in EdgeDataLoader on GPU ( #3226 )
...
* Update filter code
* Add unit tests
* Fixes
* Switch to indices
* Rename functions
* Fix linting
* Fix whitespace
* Add doc
* Fix heterograph
* Change workspace allocation
* Fix linting
* Fix docs in filter.py
* Add todo
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-08-20 02:47:48 +00:00
3ccd4a29e8
fix cuda 11.1 crashing bug ( #3265 )
2021-08-20 02:44:56 +00:00
10beb7b2e0
[Tools] In tools/launch.py, correctly pass all DGL client/server env vars if udf is a multi-command ( #3245 )
...
* Correctly pass all DGL client/server env vars if udf is a multi-command
* Refactor to use wrap_cmd_with_local_envvars() helper fn
2021-08-20 02:42:54 +00:00
a172b48465
[Tools] Refactor tools/launch.py to handle more python binary names ( #3205 )
...
* Refactors torch dist launcher udf-wrap code to handle more python versions
* minor changes
2021-08-20 02:27:13 +00:00
392a2e587d
[bugfix] fix default ntypes/etypes consistency between dgl.DGLGraph and dgl.graph ( #3198 )
2021-08-20 02:18:59 +00:00
d7390763f0
[Distributed] Deprecate old DistEmbedding impl, use synchronized embedding impl ( #3111 )
...
* fix.
* fix.
* fix.
* fix.
* Fix test
* Deprecate old DistEmbedding impl, use synchronized embedding impl
* update doc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-07-14 00:15:53 +08:00
f0fafa2062
[Bug fix] Fix batch information with remove_nodes/edges applied ( #3119 )
...
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Fix lint
* Fix
* Update
* Fix test cases
* Fix
* add docstrings
* Update
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-07-13 16:27:31 +08:00
2e19ba8025
add exclude self option for EdgeDataLoader ( #3122 )
...
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-07-13 13:30:16 +08:00
b576e617ad
[Feature] Add left normalizer for GCN ( #3114 )
...
* add left normalizer for gcn
* fix
* fixes and some bug stuff
2021-07-13 12:49:35 +08:00
186ef59283
[Feature] apply dgl.reorder() onto several node classification datase… ( #3102 )
...
* [Feature] apply dgl.reorder() onto several node classification datasets in DGL
* rebase on latest dgl.reorder_graph()
2021-07-13 08:52:17 +08:00
175f53decf
[Feature] enable edge reorder in dgl.reorder_graph() ( #3113 )
...
* [Feature] enable edge reorder in dgl.reorder_graph()
* refine doc string
* refine doc string for dgl.reorder_graph
* refine doc string further
2021-07-12 14:11:46 +08:00
32d1f3ac97
[BugFix] skip frames whose num_rows is zero in dgl.heterograph.combine_frames() ( #3110 )
...
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-07-07 14:15:06 +08:00
188152b853
[Feature] Add Heterograph support on Python for builtin unary msg functions (copy_u, copy_e) ( #2989 )
...
* heterograph for binary func
* Added SDDMM support
* Added unittest
* added binary test cases
* unary mfuncs works
* Fixed lint err
* lint check and others
* link check
* fixed import *_hetero issue
* lint check
* replace torch with dgl backend
* lint cehck
* removed torch from test
* skip mxnet unittest
* skip gpu test
* Remove unused/duplicated code
* minor
* changed data structure of ndata and edata
* link check
* reorganized
* minor lint
* minor lint
* raise error for udf func
* lint check
* fix for CUDA 10.1
* add a note for future cross-type max/min reducing
* Add support CUDA < 11
* lint check
* tidied C code
* remove dummy GSDDMM_hetero backward implementation
Co-authored-by: Israt Nisa <nisisrat@amazon.com >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2021-07-06 20:41:58 +08:00
d3e4460b98
[Doc] Add docstring for missing APIs ( #3088 )
...
* add docstring for missing API; fix some docstring
* rename apis; address comments
2021-07-05 18:40:07 +08:00
485c04cff6
[Distributed] Fix a few bugs in distributed API ( #3094 )
...
* fix.
* fix.
* fix.
* fix.
* Fix test
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-07-05 15:12:02 +08:00
0884d02465
[Distributed] Fix bugs in partitioning on heterogeneous graphs. ( #3085 )
...
* fix bugs in partitioning on heterogeneous graphs.
* fix.
* fix.
* fix example.
* fix.
* fix test.
* fix.
* fix.
* fix.
* fix tests.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-02 21:16:14 +08:00
0b3a6216f5
[Test] Enable kvstore test ( #3079 )
...
* try enable kvstore test
* fix
* fix
* seperate out kvstore test
* add comment
2021-07-02 14:54:38 +08:00
a0390dde93
[Feature] Add dgl.utils.is_sorted_srcdst() ( #2685 )
...
* Add dgl.utils.is_sorted_srcdst
* Fix linting issues
* delete blank line
* Specify datatype to index tensor in test
* Force integer conversion
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-07-02 11:29:24 +08:00
4307fe8895
[Feature] Add dgl.reorder() to re-order graph according to specified … ( #3063 )
...
* [Feature] Add dgl.reorder() to re-order graph according to specified strategy
* fix unit test failure for metis reorder
* fix unit test failure on mxnet_cpu
* refine unit test for dgl.reorder
* fix unit test failure on mxnet
* fix array_equal error for mxnet unit test
* fix unit test failure for mxnet
* convert metis output to numpy array explicitly
Co-authored-by: Tong He <hetong007@gmail.com >
2021-06-30 13:29:08 +08:00
9664cdffd3
[Build] Make nccl optional ( #3056 )
...
* fix
* remove nvidiasmi
* fix
* fix docs
* fix
* fix
* 1
* fix
* remove
* skip deprecated kernel
* fix
* Revert "skip deprecated kernel"
This reverts commit c5ceb7f60dbbaf065b81cc3680757fd611d90ad3.
* fix
2021-06-27 22:36:01 +08:00
acd21a6d60
[Feature] Support direct creation from CSR and CSC ( #3045 )
...
* csr and csc creation
* fix
* fix
* fixes to adj transpose
* fine
* raise error if indptr did not match number of nodes
* fix
* huh?
* oh
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-06-25 10:05:47 +08:00
2f7ca41459
[Bug fix] Use shared memory for grad sync when NCCL is not avaliable as PyTorch distributed backend. ( #3034 )
...
* Use shared memory for grad sync when NCCL is not avaliable as PyTorch distributed backend.
Fix small bugs and update unitests
* Fix bug
* update test
* update test
* Fix unitest
* Fix unitest
* Fix test
* Fix
* simple update
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-212.ec2.internal >
2021-06-24 12:06:54 +08:00
e56bbafd25
[Feature] Biased Neighbor Sampling ( #2987 )
...
* update
* update
* update
* update
* lint
* lint
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* lint
* update
* clone
* update
* update
* update
* update
* replace idarray with ndarray
* refactor cpp part
* refactor python part
* debug
* refactor interface
* test and doc
* lint and test
* lint
* fix
* fix
* fix
* const
* doc
* fix
* fix
* fix
* fix
* fix & doc
* fix
* fix
* update
* update
* update
* merge
* doc
* doc
* lint
* fix
* more tests
* doc
* fix
* fix
* update
* update
* update
* fix
* fix
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-06-23 17:44:51 +08:00
e667545da5
[Feature] Node2vec ( #2992 )
...
* add seal example
* 1. add paper infomation in examples/README
2. adjust codes
3. option test
* use latest `to_simple` to replace coalesce graph function
* remove outdated codes
* remove useless comment
* Node2vec
1.implement node2vec random walk c++ op
2.implement node2vec model
3.implement node2vec example
* add CMakeLists file modify
* refine c++ codes
* refine c++ codes
* add missing whitespace
* refine python codes
* add codes
* add node2vec_impl.h
* fix codes
* fix code style problem
* fixes
* remove
* lots of changes
* add benchmark
* fixes
Co-authored-by: smilexuhc <smile.xuhc@gmail.com >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-06-23 14:47:56 +08:00
7359481497
[Feature][GPU] Add function for setting weights of a sparse embedding on multiple GPUs. ( #3047 )
...
* add unit test
* Extend NDArrayPartition object
* Add method for setting embedding, and improve documentation
* Sync before returning
* Use name unique to sparse embedding class to avoid delete
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-06-22 08:48:12 -07:00
ff519f98c3
[API] Standardize Subgraph APIs ( #2929 )
...
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Fix
* Update
* Fix subgraph tests
* Capture stdout for distributed test
* Capture stdout for distributed test
* Update
* Update
* Update
* Update subgraph.cc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-06-21 19:53:37 +08:00
9706eaa895
[Feature] add permission information and fix import problems ( #3036 )
...
* [Feature] add positive negative statistics
* [Feature] add permission information and fix import problem
* fix backend incompatible problem
* modify random split to remove sklearn usage
* modify file read to remove pandas usage
* add datasets into doc
* add random seed in data splitting
* add dataset unit test
* usage permission information update
Co-authored-by: zhjwy9343 <6593865@qq.com >
2021-06-21 11:55:15 +08:00
aaec3d8a0b
[Distributed] Support hierarchical partitioning ( #3000 )
...
* add.
* fix.
* fix.
* fix.
* fix.
* add tests.
* support node split and edge split.
* support 1 partition.
* add tests.
* fix.
* fix test.
* use hierarchical partition.
* add check.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-57.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-06-16 16:58:23 +08:00
a303f07819
fix #2952 ( #3010 )
2021-06-14 15:09:14 +08:00
17141dd372
[Fix] Fix to_homo for graph with zero nodes ntype ( #3011 )
...
* fix #2870
* lint
* fix
2021-06-14 14:34:13 +08:00
2570d412f9
[NN] Add fast path for GateGCNConv when it has only one edge type ( #2994 )
...
* fix gatedgcn
* fix lint
2021-06-12 01:13:18 +08:00
17d604b5c7
[Feature] Allow using NCCL for communication in dgl.NodeEmbedding and dgl.SparseOptimizer ( #2824 )
...
* Split from NCCL PR
* Fix type in comment
* Expand documentation for sparse_all_to_all_push
* Restore previous behavior in example
* Re-work optimizer to use NCCL based on gradient location
* Allow for running with embedding on CPU but using NCCL for gradient exchange
* Optimize single partition case
* Fix pylint errors
* Add missing include
* fix gradient indexing
* Fix line continuation
* Migrate 'first_step'
* Skip tests without enough GPUs to run NCCL
* Improve empty tensor handling for pytorch 1.5
* Fix indentation
* Allow multiple NCCL communicator to coexist
* Improve handling of empty message
* Update python/dgl/nn/pytorch/sparse_emb.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
* Update python/dgl/nn/pytorch/sparse_emb.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
* Keepy empty tensor dimensionaless
* th.empty -> th.tensor
* Preserve shape for empty non-zero dimension tensors
* Use shared state, when embedding is shared
* Add support for gathering an embedding
* Fix typo
* Fix more typos
* Fix backend call
* Use NodeDataLoader to take advantage of ddp
* Update training script to share memory
* Only squeeze last dimension
* Better handle empty message
* Keep embedding on the target device GPU if dgl_sparse if false in RGCN example
* Fix typo in comment
* Add asserts
* Improve documentation in example
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-06-10 21:19:00 -07:00
5be937a7fb
[Kernel] Slicing Batched Graphs ( #2965 )
...
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Update
* Update
* Add files via upload
* Add files via upload
* Update
* Lint
* Add files via upload
* Lint
* Update
* Update
* Update
* Update
* Update
* Lint Fix
* Lint
Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-161.us-west-2.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-06-10 12:07:44 +08:00
972a9f1323
[Doc] Re-organize the code for dgl.geometry, and expose it in the doc ( #2982 )
...
* reorg and expose dgl.geometry
* fix lint
* fix test
* fix
2021-06-07 11:19:39 +08:00
b1628f2398
[Tutorial] Distributed node classification. ( #2969 )
...
* add init version.
* fix build.
* fix format.
* fix.
* fix.
* fix format.
* update README.
* avoid running CI on distributed training tutorials.
* Update tutorials/dist/1_node_classification.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
* fix.
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-06-04 21:50:16 +08:00
6a56562a7c
[CI] Use k8s cluster ( #2957 )
...
* add
* fix
* set default
* fix
* try master
* try fix
* try
* fix
* 111
* fix
* fix
* update
* ccc
* try
* fix
* fix
* try new machine
* fix
* fix
* fix
* Revert "fix"
This reverts commit e716d66b046f92fe7ae368947a51a036a7a3188a.
* try
* more parrallel
* use k8s for all
* fix name
* try not specify instance type
* ci
* use one yaml
* Revert "use one yaml"
This reverts commit 717d8d852be39fbf2e2e45f9f224aa97907c372c.
* add timeout
* fix permission
* mount efs
* print
* fix pvc
* fix
* restrict num of gpu instances
* check
* fix
* fix
2021-06-04 18:31:11 +08:00
2df4a95f3e
[Feature] Make to_heterogeneous(to_homogeneous(hg)) return hg ( #2958 )
...
* make to_heterogeneous and to_homogeneous invertible
* docstring
* oops
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-06-03 18:50:18 +08:00
b8fe2b4837
[Feature][Sampler] Sort CSR by tag ( #1664 )
...
* update
* update
* update
* update
* lint
* lint
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* lint
* update
* clone
* update
* update
* update
* update
* replace idarray with ndarray
* refactor cpp part
* refactor python part
* debug
* refactor interface
* test and doc
* lint and test
* lint
* fix
* fix
* fix
* const
* doc
* fix
* fix
* fix
* fix
* fix & doc
* fix
* fix
* fix
* fix
* fix
* fix
* update
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-06-01 23:33:56 +08:00
fcfe52aee5
fix residual ( #2962 )
2021-06-01 13:50:39 +08:00
2ad7a9e9dd
[Fix] Fix #2808 ( #2917 )
...
* Fix #2808
* fix
* fix
* fix
* fix
* fix
* fix
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-05-31 19:05:58 +08:00
18dbaebebc
[Distributed] Specify the graph format for distributed training ( #2948 )
...
* explicitly set the graph format.
* fix.
* fix.
* fix launch script.
* fix readme.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-05-26 18:09:30 +08:00
ae8dbe6d3c
[Feature][Performance] Implement NCCL wrapper for communicating NodeEmbeddings and sparse gradients. ( #2825 )
...
* Split NCCL wrapper from sparse optimizer and sparse embedding
* Add more unit tests for single node nccl
* Fix unit test for tf
* Switch to device histogram
* Fix histgram issues
* Finish migration to histogram
* Handle cases with zero send/recieve data
* Start on partition object
* Get compiling
* Updates
* Add unit tests
* Switch to partition object
* Fix linting issues
* Rename partition file
* Add python doc
* Fix python assert and finish doxygen comments
* Remove stubs for range based partition to satisfy pylint
* Wrap unit test in GPU only
* Wrap explicit cuda call in ifdef
* Merge with partition.py
* update docstrings
* Cleanup partition_op
* Add Workspace object
* Switch to using workspace object
* Move last remainder based function out of nccl_api
* Add error messages
* Update docs with examples
* Fix linting erros
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-05-20 10:58:17 -07:00
5d7e80f413
[Feature] Add bruteforce implementation for KNN with O(Nk) space complexity ( #2892 )
...
* add bruteforce impl
* add support for bruteforce-sharemem
* modify python API
* add tests
* change file path
* change python API
* fix lint
* fix test
* also check worst_dist in the last few dim
* use heap and early-stop on CPU
* fix lint
* fix lint
* add device check
* use cuda function to determine max shared mem
* use cuda to determine block info
* add memory free for tmp var
* update doc-string and add dist option
* fix lint
* add more tests
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-05-19 18:27:43 +08:00
6e7f19f27b
[Distributed] add distributed in-degree and out-degree. ( #2918 )
...
* add distributed in-degree and out-degree.
* update comments.
* fix a bug.
* add tests.
* add tests.
* fix a bug.
* fix docstring.
* update doc.
* fix
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-05-18 19:46:09 +08:00
c018436588
[Example] add latent dirichlet allocation ( #2883 )
...
* add lda model
* tweak latent dirichlet allocation
* Update README.md
* Update README.md
* update example index
* update header
* minor tweak
* add example test
* update doc
* Update README.md
* Update README.md
* add partial_fit for free
* Update examples/pytorch/lda/lda_model.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update examples/pytorch/lda/example_20newsgroups.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
* Update lda_model.py
* bugfix torch Gamma uses rate parameter
Co-authored-by: Yifei Ma <yifeim@amazon.com >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-05-17 20:25:36 +08:00
657c220de7
[Feature] Python interface for adjacency matrix summation and multiplication ( #2893 )
...
* test commit
* fixes
* oops
* add docs
* lint
* why does it say I have a trailing whitespace
* oh ok
* fixes
* why there's an invalid argument error
* address comments
* fix
* address comments
2021-05-17 19:47:02 +08:00
103444c58f
Remove __len__ method from DGLGraph ( #2902 )
...
* Update heterograph.py
* remove unit tests
* replace tutorial
2021-05-11 14:06:18 +08:00
bfef789ea1
[Dataloading] Make loader iters iterator ( #2886 )
...
* Make loader items iterator
* Update test_dataloader.py
* Update __init__.py
* Update test_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-05-07 10:15:49 +08:00
975eb8fc5d
[Distributed] Distributed node embedding and sparse optimizer ( #2733 )
...
* Draft for sparse emb
* add some notes
* Fix
* Add sparse optim for dist pytorch
* Update test
* Fix
* upd
* upd
* Fix
* Fix
* Fix bug
* add transductive exmpale
* Fix example
* Some fix
* Upd
* Fix lint
* lint
* lint
* lint
* upd
* Fix lint
* lint
* upd
* remove dead import
* update
* lint
* update unitest
* update example
* Add adam optimizer
* Add unitest and update data
* upd
* upd
* upd
* Fix docstring and fix some bug in example code
* Update rgcn readme
Co-authored-by: Ubuntu <ubuntu@ip-172-31-57-25.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-05-03 08:37:40 +08:00
ab2bd1f13d
[Feature] Add cuda support for Sparse Matrix multiplication, summation and masking ( #2782 )
...
* init cuda support
* cuSPARSE err
* passed unittest for csr_mm/SpGEMM. int64 not supported
* Debugging cuSPARSE error 3
* csrgeam only supports int32?
* disabling int64 for cuda
* refactor and add CSRMask
* lint
* oops
* remove todo
* rewrite CSRMask with CSRGetData
* lint
* fix test
* address comments
* lint
* fix
* addresses comments and rename BUG_ON
Co-authored-by: Israt Nisa <nisisrat@amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-71.ec2.internal >
Co-authored-by: Quan Gan <coin2028@hotmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-04-27 16:06:35 +08:00
nv-dlasalle