* csr and csc creation
* fix
* fix
* fixes to adj transpose
* fine
* raise error if indptr did not match number of nodes
* fix
* huh?
* oh
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* Use shared memory for grad sync when NCCL is not avaliable as PyTorch distributed backend.
Fix small bugs and update unitests
* Fix bug
* update test
* update test
* Fix unitest
* Fix unitest
* Fix test
* Fix
* simple update
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-212.ec2.internal>
* add unit test
* Extend NDArrayPartition object
* Add method for setting embedding, and improve documentation
* Sync before returning
* Use name unique to sparse embedding class to avoid delete
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
* [Feature] add positive negative statistics
* [Feature] add permission information and fix import problem
* fix backend incompatible problem
* modify random split to remove sklearn usage
* modify file read to remove pandas usage
* add datasets into doc
* add random seed in data splitting
* add dataset unit test
* usage permission information update
Co-authored-by: zhjwy9343 <6593865@qq.com>
* Split from NCCL PR
* Fix type in comment
* Expand documentation for sparse_all_to_all_push
* Restore previous behavior in example
* Re-work optimizer to use NCCL based on gradient location
* Allow for running with embedding on CPU but using NCCL for gradient exchange
* Optimize single partition case
* Fix pylint errors
* Add missing include
* fix gradient indexing
* Fix line continuation
* Migrate 'first_step'
* Skip tests without enough GPUs to run NCCL
* Improve empty tensor handling for pytorch 1.5
* Fix indentation
* Allow multiple NCCL communicator to coexist
* Improve handling of empty message
* Update python/dgl/nn/pytorch/sparse_emb.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
* Update python/dgl/nn/pytorch/sparse_emb.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
* Keepy empty tensor dimensionaless
* th.empty -> th.tensor
* Preserve shape for empty non-zero dimension tensors
* Use shared state, when embedding is shared
* Add support for gathering an embedding
* Fix typo
* Fix more typos
* Fix backend call
* Use NodeDataLoader to take advantage of ddp
* Update training script to share memory
* Only squeeze last dimension
* Better handle empty message
* Keep embedding on the target device GPU if dgl_sparse if false in RGCN example
* Fix typo in comment
* Add asserts
* Improve documentation in example
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
* Split NCCL wrapper from sparse optimizer and sparse embedding
* Add more unit tests for single node nccl
* Fix unit test for tf
* Switch to device histogram
* Fix histgram issues
* Finish migration to histogram
* Handle cases with zero send/recieve data
* Start on partition object
* Get compiling
* Updates
* Add unit tests
* Switch to partition object
* Fix linting issues
* Rename partition file
* Add python doc
* Fix python assert and finish doxygen comments
* Remove stubs for range based partition to satisfy pylint
* Wrap unit test in GPU only
* Wrap explicit cuda call in ifdef
* Merge with partition.py
* update docstrings
* Cleanup partition_op
* Add Workspace object
* Switch to using workspace object
* Move last remainder based function out of nccl_api
* Add error messages
* Update docs with examples
* Fix linting erros
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
* add bruteforce impl
* add support for bruteforce-sharemem
* modify python API
* add tests
* change file path
* change python API
* fix lint
* fix test
* also check worst_dist in the last few dim
* use heap and early-stop on CPU
* fix lint
* fix lint
* add device check
* use cuda function to determine max shared mem
* use cuda to determine block info
* add memory free for tmp var
* update doc-string and add dist option
* fix lint
* add more tests
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* test commit
* fixes
* oops
* add docs
* lint
* why does it say I have a trailing whitespace
* oh ok
* fixes
* why there's an invalid argument error
* address comments
* fix
* address comments
* Add row/col sorted flags
* improve sorting paths
* Remove print statement
* Keep track of sorted matrices
* Remove sort check in to_block
* Improve CPU sorted COO->CSR
* Handle the zero edge case
* Remove omp default clause to work with MSVC
* Update comments on sorted COO->CSR cpu implementatoin
* Expose sorted to python interface
* Make check_sorted default to false for dgl.graph()
* remove check sorted; add utests
* remove check_sorted flag
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* Start on uniform GPU sampling
* Save more work
* Get cu file compiling
* Update sampling
* More changes
* Get GPU sampling for uniform probabilities solved
* Fix batch tensor migration
* Fix
* update kernels
* expand blocking
* Undo testing change
* Cut down on sampling overhead
* Fix replacement
* Update unit tests
* Add option to gpu sample in graphsage
* Copy only csc to gpu
* Add ogbn support
* Fix linting
* Remove nvtx from sample
* Improve documentation and error checking
* Expand documentation
* Update assert checking
* delete extra space
* Use standard dataloader when dataset is a dictionary
* ogb -> ogbn
* Fix edge selection determinism
* Fix typos
* Remove nvtx
* Add comment for self.fanout_arrays and assert
* Fix linting
* Migrate to scalarbatcher
* Fix indentation
* Fix batcher
* Fix indexing
* Only use databatcher for GPU
* Convert to DGL NDArray to PyTorch Tensor
* Add optimization for PyTorch's F.tensor() for list of GPU tensors
Co-authored-by: Da Zheng <zhengda1936@gmail.com>
* tf nn for edgeConv
* Auto stash before merge of "tf_working" and "origin/tf_working"
* clean up
* added test for edge_conv
* fix
* fix
* fix
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
* finish graph matching gpu version
* use C++ shuffle
* finish graph matching
* fix bug
* fix bug
* change name and use swap
* upt
* fix format problem
* fix format problem
* stronger test
* upt
* upt
* change python api
* upt
* upt
* format check
* upt
* upt
* fix bug
Co-authored-by: Tong He <hetong007@gmail.com>
Jinjing Zhou