e7046f1e0e
[Distributed] Fix a bug in graph partition. ( #2869 )
...
* update distributed training doc.
* explain data split.
* fix message passing.
* id mapping.
* fix.
* test data reshuffling.
* fix a bug.
* fix test.
* Revert "fix."
This reverts commit 2d025e9e1a5c05c3da9b803a035a788ced59bd77.
* Revert "id mapping."
This reverts commit 2a6a93ceb81fbdff86e6e9e5a58e1ace1e9d9882.
* Revert "fix message passing."
This reverts commit ed8a86bf2b015e5e4f64ba160e81b207ad2a1d65.
* Revert "explain data split."
This reverts commit 4338ddf8a336014cf92d4cb9a1db02b9badc0e55.
* Revert "update distributed training doc."
This reverts commit dda1c35c44536934c19715534f01f832afda6ad2.
* add more tests.
* fix.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-26 18:17:16 +08:00
c37e036432
[Bug Fix] Fix sparse opt bug ( #2859 )
...
* Fix #2856
* upd
* Fix unitest
* upd
* upd
* upd
* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-57-25.ec2.internal >
2021-04-25 13:40:21 +08:00
d76af4d4c1
[Distributed] Return the ID mapping in graph partitioning. ( #2857 )
...
* return mapping.
* support heterogeneous graph.
* more test.
* fix lint.
* fix for diff backends.
* fix.
* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-04-22 17:51:42 +08:00
6b022d2fe2
[Sampler] BiasedChoice sampler ( #1665 )
...
* update
* update
* update
* update
* update
* update
* update
* fix
* fix
* update
* doc
* doc
* fix
* fix
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-04-22 15:12:52 +08:00
bbebde4660
[Performance] Track sorted status of COO from creation ( #2645 )
...
* Add row/col sorted flags
* improve sorting paths
* Remove print statement
* Keep track of sorted matrices
* Remove sort check in to_block
* Improve CPU sorted COO->CSR
* Handle the zero edge case
* Remove omp default clause to work with MSVC
* Update comments on sorted COO->CSR cpu implementatoin
* Expose sorted to python interface
* Make check_sorted default to false for dgl.graph()
* remove check sorted; add utests
* remove check_sorted flag
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-04-16 10:30:07 +08:00
e70138bbe4
[Performance][GPU] Enable GPU uniform edge sampling ( #2716 )
...
* Start on uniform GPU sampling
* Save more work
* Get cu file compiling
* Update sampling
* More changes
* Get GPU sampling for uniform probabilities solved
* Fix batch tensor migration
* Fix
* update kernels
* expand blocking
* Undo testing change
* Cut down on sampling overhead
* Fix replacement
* Update unit tests
* Add option to gpu sample in graphsage
* Copy only csc to gpu
* Add ogbn support
* Fix linting
* Remove nvtx from sample
* Improve documentation and error checking
* Expand documentation
* Update assert checking
* delete extra space
* Use standard dataloader when dataset is a dictionary
* ogb -> ogbn
* Fix edge selection determinism
* Fix typos
* Remove nvtx
* Add comment for self.fanout_arrays and assert
* Fix linting
* Migrate to scalarbatcher
* Fix indentation
* Fix batcher
* Fix indexing
* Only use databatcher for GPU
* Convert to DGL NDArray to PyTorch Tensor
* Add optimization for PyTorch's F.tensor() for list of GPU tensors
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-04-14 18:55:09 -07:00
de5e8e2392
[Distributed] Fix a bug for graphs without node/edge data. ( #2838 )
...
* fix.
* test distributed graph without node/edge data.
* remove some tests.
* fix lint
2021-04-12 22:35:01 -07:00
e83d0a8007
[Feature] Add kd-tree implementation (CPU) for kNN ( #2767 )
...
* add submodule nanoflann
* finish python API for knn
* finish ndarray adaptor
* finish cpu-kdtree version of knn
* use openmp
* add endline
* upt
* upt
* fix format and code style
* upt
* add warning for gpu-cpu copy
* avoid contiguous copy
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
Co-authored-by: Tong He <hetong007@gmail.com >
2021-04-09 17:39:34 +08:00
54c74803b2
[CI] Disable distributed KVStore UT ( #2805 )
2021-04-01 16:00:56 +08:00
e36c5db614
[Distributed] Simplify distributed API ( #2775 )
...
* remove num_workers.
* remove num_workers.
* remove num_workers.
* remove num-servers.
* update error message.
* update docstring.
* fix docs.
* fix tests.
* fix test.
* fix.
* print messages in test.
* fix.
* fix test.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
2021-03-29 19:19:23 -07:00
03482f0a4c
[NN] tf nn for edgeConv ( #2741 )
...
* tf nn for edgeConv
* Auto stash before merge of "tf_working" and "origin/tf_working"
* clean up
* added test for edge_conv
* fix
* fix
* fix
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2021-03-25 13:44:07 +08:00
929d863447
[Feature] Sparse-sparse matrix multiplication, addition, and masking ( #2753 )
...
* test
* more stuff
* add test
* fixes
* optimize algo
* replace unordered_map with arrays
* lint
* lint x2
* oops
* disable gpu csrmm tests
* remove gpu invocation
* optimize with openmp
* remove python functions
* add back with docstrings
* lint
* lint
* update python interface
* functionize
* functionize
* lint
* lint
2021-03-24 17:24:41 +08:00
edf644633d
[performance] Optimize the association order of AXW in GraphSAGE. ( #2747 )
...
* upd
* lint
* upd
* upd
* compatibility
* upd
* upd
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-03-19 01:41:33 +08:00
366cc7ebf3
[Pickle] Fix HeteroGraphConv pickle problem ( #2761 )
...
* fix pickle problem
* lint
* add pickle tests
* fix
* fix
* fix
* fix
* fix for windows
2021-03-18 17:56:28 +08:00
c88fca5055
[Feature] Add edge coarsening for homogeneous undirected graphs ( #2691 )
...
* finish graph matching gpu version
* use C++ shuffle
* finish graph matching
* fix bug
* fix bug
* change name and use swap
* upt
* fix format problem
* fix format problem
* stronger test
* upt
* upt
* change python api
* upt
* upt
* format check
* upt
* upt
* fix bug
Co-authored-by: Tong He <hetong007@gmail.com >
2021-03-09 16:35:27 +08:00
7380d61eb9
[NN] Support Unidirectional Bipartite Graphs in CFConv ( #2674 )
...
* Update
* update
* Update
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-03-03 16:35:07 +08:00
e6bf54cdcb
[Feature] Support aggregate multiple edge features in to_simple. ( #2623 )
...
* upd
* fix
* upd
* upd
* upd
* upd
* upd
* upd
* upd
* upd
* upd
* fix
* refactor
* upd test
* large feat_len or n in segment reduce
* lint
2021-02-21 10:05:42 +08:00
8a07ab7737
[Doc] Tutorials re-organization ( #2683 )
...
* reorg
* change titles
* rm some stale API doc; minor fix
* fix docs
* add warning
* rm new-tutorial run in ci
* lint
2021-02-20 17:42:45 +08:00
d93a975997
revert ( #2672 )
2021-02-19 12:05:42 +08:00
9e630101a1
fix and lots of tests ( #2650 )
2021-02-12 11:40:05 +08:00
117dd25204
[Doc] Fix inconsistencies and GPU code ( #2642 )
...
* fix inconsistencies and GPu
* bug fixes
* fix
* trigger new tutorials
2021-02-08 16:33:39 +08:00
bc3a532f5e
[Sampling] Implement dgl.to_block() for the GPU ( #2339 )
...
* Add start of to_block gpu implementation
* Pull in more changes from 0.4.2 cuda_to_block
* Move more code to IdArray
* Refactor DeviceNodeMapMaker
* Updates
* get compiling
* Integrate to_block
* Fix ID allocation
* Minor fixes
* Cleanup cuda calls to use cuda_common
* Reduce kernel calls
* Lint cleanup
* Expand documentation
* Remove unused function
* Rename variables for consistency
* Add doxygen comments
* Fix file extension
* Remove raw asynccopy for deviceapi
* Remove unused function
* Fix block/tile configuration
* Add cuda_device_common.cuh
* Add basic hashtable
* Migrate part of hashtable
* Refactor to use external hashtable
* Make functions members
* Format hash table functions
* Migrate duplicate filling
* Move last function over
* Refactor with cu file
* lint c++ code
* Move context check to C++ code
* Use macro switch
* Add missing files
* Update docstring
* update docs
* Move atomic functions
* Refactor hashtable
* Fix linting
* Expand docs
* Fix mismatched argument names
* Switch doxygen comments from using @param to \param
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-02-08 13:00:43 +08:00
0346b0aa80
[Test] Add test for Cython and force rebuild when run setup.py ( #2635 )
2021-02-05 22:38:46 +08:00
fb4a0508c6
[bugfix] Solve the boundary issue in backward function of segment sum ( #2610 )
...
* upd
* trigger
* upd
2021-02-03 18:50:07 +08:00
1f6eba9ed3
[Bug] Heterogeneous graph convolution bugfix ( #2578 )
...
* fix heterograph conv
* remove test cases
* fix test
* fix test
* fix test
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-01-29 09:30:19 +08:00
e4ddafe951
[NN] add multihead in DotGatConv ( #2549 )
...
* add multihead in DotGatConv
* Fix spacing issue
* Add Unit test for dotgat
* Modified Unit test for dotgat
* Add transformer like divisor
* Update dotgatconv.py
Co-authored-by: Chen <chesirui@3c22fbe5458c.ant.amazon.com >
Co-authored-by: Zihao Ye <expye@outlook.com >
2021-01-28 21:05:10 +08:00
a7e941c379
[Feature] Add support for sparse embedding ( #2451 )
...
* Add sparse embedding for dgl and update rgcn example
* upd
* Fix
* Revert "Fix"
This reverts commit 4da87cdfb8b8c3506b7fc7376cd2385ba8045c2a.
* Fix
* upd
* upd
* Fix
* Add unitest and update impl
* fix
* Clean up rgcn example code
* upd
* upd
* update
* Fix
* update score
* sparse for sage
* remove model sparse
* upd
* upd
* remove global norm
* revert delete model_sparse.py
* update according to comments
* Fix doc
* upd
* Fix test
* upd
* lint
* lint
* lint
* upd
* upd
* clean up
Co-authored-by: Ubuntu <ubuntu@ip-172-31-56-220.ec2.internal >
2021-01-28 00:26:49 +08:00
2576647c1f
[Performance] Improve COO to CSR, and sort columns of CSR only when necessary. ( #2391 )
...
* Remove double-checking sorted
* Remove sorting of CSR by default
* Update unit test to use unsorted matix
* delete whitespace
* Expand unit tests
* Replace cusparse sort
* Fix row column sorting
* Explicitly don't sort columns
* Fix linting errors
* Fix bit-width calculation
* Fix sorting assertion and unit test
* Fix linting
* Improve CPU COO2CSR
* Remove references
* Rename and add documentation to edge encoding/decoding funcionts
* Fix sorting keys as 64 bit
* Revert cosmetic changes to unit tests
* Update documentation
* Update complexity documentation for coo to csr conversion
* Remove COOIsSorted check in CPU implementation too
2021-01-27 15:33:34 +08:00
0855d25501
[NN] Support scalar edge weight for GraphConv, SAGEConv and GINConv ( #2557 )
...
* add edge weight in forward
* fix lint
* fix
* fix
* address comments
* add utils
* add util to normalize in gcn way
* fix lint
* add unittest
* fix lint
* fix docstring
* fix docstring
* address comments
* improve notation consistence
* use preferred fn
2021-01-26 23:34:12 +08:00
0f9056edb9
[feature] Implement missing CUDA operators for COO format (part 1). ( #2565 )
...
* upd
* upd
* upd
* upd
* fix
* upd
* upd
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-01-25 16:15:22 +08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
2288098ba6
[Feature] Converting a block to a graph and allow block construction ( #2555 )
...
* first commit
* more commit
* fix and docstring
* add to api reference
* address comments
* Update convert.py
2021-01-24 12:44:54 +08:00
8c8c5b1dc1
[Bug fix] Stop erasing schemas of frames with no data rows ( #2529 )
...
* added option for batching empty data, fixes #2526
* added option for batching empty data, fixes #2526
* decreased line lengths
* removed trailing whitespace
* fixed wrong feature name
* [Bugfix] stop erasing schema of frames w/ no cols (fixes #2528 )
* removed arg doc
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-01-17 10:32:47 +08:00
5da3439c16
[Feature] Add Device Flag in Data Loaders ( #2450 )
...
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-01-15 07:39:26 +08:00
0778766411
[Bug fix] [Feature] added option for batching empty data ( #2527 )
...
* added option for batching empty data, fixes #2526
* added option for batching empty data, fixes #2526
* decreased line lengths
* removed trailing whitespace
* fixed wrong feature name
* now default behavior when all graphs are empty
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-01-14 15:25:13 +08:00
54f186bba0
Fix ( #2524 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-59-204.ec2.internal >
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-01-13 15:09:21 +08:00
bdb88df576
[Performance] Allow DataLoaders to return indices of the seed array ( #2495 )
...
* return indices from dataloader
* fixes
* fix
* fix distgraph and added some todos
* Update dataloader.py
* Update dataloader.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-01-08 14:30:52 +08:00
e2e524df41
[Feature] Add GraphDataLoader implementation ( #2496 )
...
* add graph dataloader
* add to doc
* fix
* fix
* fix docstring
* update according to torch default_collate
* add unittest
* fix
* fix lint
* fix
2021-01-07 13:38:21 +08:00
2caac086c2
[Bug] send_and_recv and pull may write to wrong places ( #2497 )
...
* fix
* fix test
2021-01-06 17:18:37 +08:00
0c2a2ea1e2
[Bugfix] Handle a Corner Case of Batching after Removing Nodes/Edges ( #2465 )
...
* Update
* Update
* Update
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-01-05 22:22:03 +08:00
f960468f50
[feature] APIs for manually set batch_num_nodes and batch_num_edges ( #2430 )
...
* wip
* udp
* upd
* fix typo
* lint
* lint
* upd
* upd
* lint
* lint
* upd
* upd
* tftest
* fix
* fallback numpy
* fix tf
* docstring
Co-authored-by: Mufei Li <mufeili1996@gmail.com >
2021-01-04 23:43:58 +08:00
d3c1645550
[Performance][Test] New low memory implementation for RGCN and related regression tests ( #2468 )
...
* WIP
* finish lowmem impl; benchmarking
* wip
* wip
* fix benchmarks
* fix bug in searchsorted
* update readme
* update
* lint and ogb dependency
* add flags to to_homogeneous
* fix docstring
* address comments
2021-01-04 22:23:59 +08:00
4208ce2b9e
[Feature] Tvm integration ( #2367 )
...
Co-authored-by: Zihao Ye <expye@outlook.com >
2020-12-31 17:40:25 +08:00
3d1f2e870a
delete shared memory when receiving signals. ( #2419 )
...
* delete shared memory when receive signal.
* rename.
* fix lint.
* fix lint.
* fix compile.
* Fix.
* we need to report error if the shared memory exist.
* disable tensorflow test for shared memory.
* revert.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-12-26 10:41:10 -08:00
9a7235faf2
[Performance] Use allocator from PyTorch if possible ( #2328 )
...
* first commit
* some thoughts
* move around
* more commit
* more fixes
* now it uses torch allocator
* fix symbol export error
* fix
* fixes
* test fix
* add script
* building separate library per version
* fix for vs2019
* more fixes
* fix on windows build
* update jenkinsfile
* auto copy built dlls for windows
* lint and installation guide update
* fix
* specify conda environment
* set environment for ci
* fix
* fix
* fix
* fix again
* revert
* fix cmake
* fix
* switch to using python interpreter path
* remove scripts
* debug
* oops sorry
* Update index.rst
* Update index.rst
* copies automatically, no need for this
* do not print message if library not found
* tiny fixes
* debug on nightly
* replace add_compile_definitions to make CMake 3.5 happy
* fix linking to wrong lib for multiple pytorch envs
* changed building strategy
* fix nightly
* fix windows
* fix windows again
* setup bugfix
* address comments
* change README
2020-12-25 13:57:51 +08:00
e379e52585
[hotfix] Make USE_AVX a flag in cmake to avoid compilation error for arm user ( #2428 )
...
* upd cmake
* upd
* format
2020-12-17 17:29:15 +08:00
6634b984f4
[Test] Basic regression test setup. ( #2415 )
...
* add machine name
* update scripts
* update script
* test commit
* change run.sh
* model acc bench for gcn and sage
* get basic pipeline setup for local benchmarking
* try to bridge pytest with asv
* fix deps
* move asv to other folders
* move dir
* update script
* new setup
* delete useless file
* delete outputs
* remove dependency on pytest
* update script
* test commit
* stuck by torch version in dgl-ci-gpu
* update readme
* update asv conf
* missing files
* remove the old regression folder
* api bench
* add batch api bench
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2020-12-15 14:35:15 +08:00
013d145663
[NN] Attention Retrieval for NN Modules ( #2397 )
...
* Update
* Update
2020-12-07 23:42:25 +08:00
3adbfa1812
[Performance] Use segment operators for graph readout. ( #2361 )
...
* upd
* upd
* update
* upd
* upd
* upd
* fix
* lint
* lint
* pylint
* doc
2020-11-23 01:45:33 +08:00
f8ebcd7f90
[Performance] Dynamic cpu kernel V3 for SpMMSumCsr all Ops ( #2309 )
...
* support AVX512
* env DGL_CPU_INTEL_KERNEL_ENABLED=1
* env DGL_CPU_INTEL_KERNEL_LOG=1
* Add unittest test_spmm.cc
Co-authored-by: Izabela Mazur <izabela.mazur@intel.com >
Co-authored-by: Michal Szarmach <michal.szarmach@intel.com >
Review patch
2020-11-17 15:12:23 +08:00
Da Zheng