9a00cf194f
[Feature] Import PyTorch's CUDA stream management ( #4503 )
...
* add set_stream
* add .record_stream for NDArray and HeteroGraph
* refactor dgl stream Python APIs
* test record_stream
* add unit test for record stream
* use pytorch's stream
* fix lint
* fix cpu build
* address comments
* address comments
* add record stream tests for dgl.graph
* record frames and update dataloder
* add docstring
* update frame
* add backend check for record_stream
* remove CUDAThreadEntry::stream
* record stream for newly created formats
* fix bug
* fix cpp test
* fix None c_void_p to c_handle
2022-09-15 16:54:52 +08:00
1c9d2a0302
[Feature] Unify the cuda stream used in core library ( #4480 )
...
* Use an internal cuda stream for CopyDataFromTo
* small fix white space
* Fix to compile
* Make stream optional in copydata for compile
* fix lint issue
* Update cub functions to use internal stream
* Lint check
* Update CopyTo/CopyFrom/CopyFromTo to use internal stream
* Address comments
* Fix backward CUDA stream
* Avoid overloading CopyFromTo()
* Minor comment update
* Overload copydatafromto in cuda device api
Co-authored-by: xiny <xiny@nvidia.com >
2022-09-06 10:14:55 +08:00
3f138ebaaf
[Bugfix] Bug fixes in new dataloader ( #3727 )
...
* fixes
* fix
* more fixes
* update
* oops
* lint?
* temporarily revert - will fix in another PR
* more fixes
* skipping mxnet test
* address comments
* fix DDP
* fix edge dataloader exclusion problems
* stupid bug
* fix
* use_uvm option
* fix
* fixes
* fixes
* fixes
* fixes
* add evaluation for cluster gcn and ddp
* stupid bug again
* fixes
* move sanity checks to only support DGLGraphs
* pytorch lightning compatibility fixes
* remove
* poke
* more fixes
* fix
* fix
* disable test
* docstrings
* why is it getting a memory leak?
* fix
* update
* updates and temporarily disable forkingpickler
* update
* fix?
* fix?
* oops
* oops
* fix
* lint
* huh
* uh
* update
* fix
* made it memory efficient
* refine exclude interface
* fix tutorial
* fix tutorial
* fix graph duplication in CPU dataloader workers
* lint
* lint
* Revert "lint"
This reverts commit 805484dd553695111b5fb37f2125214a6b7276e9.
* Revert "lint"
This reverts commit 0bce411b2b415c2ab770343949404498436dc8b2.
* Revert "fix graph duplication in CPU dataloader workers"
This reverts commit 9e3a8cf34c175d3093c773f6bb023b155f2bd27f.
Co-authored-by: xiny <xiny@nvidia.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-02-22 01:30:31 +08:00
905c0aa578
[Feature][Performance][GPU] Introducing UnifiedTensor for efficient zero-copy host memory access from GPU ( #3086 )
...
* Add pytorch-direct version
* Initial commit of unified tensor
* Merge branch 'master' of https://github.com/davidmin7/dgl
* Remove unnecessary things
* Fix error message
* Fix/Add descriptions
* whitespace fix
* add unpin
* disable IndexSelectCPUFromGPU with no CUDA
* add a newline for unified_tensor.py
* Apply changes based on feedback
* add 'os' module
* skip unified tensor unit test for cpu only
* Update tests/pytorch/test_unified_tensor.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
* reflect feedback
Co-authored-by: shhssdm <shhssdm@gmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-07-17 00:06:20 +08:00
1c4bfb62bb
[Performance] Replace np.array with np.asarray ( #1301 )
...
* replace np.array with np.asarray
* fix
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu >
2020-03-03 20:45:41 +08:00
5dd35580f7
[Feature] Improve sampling speed; Better pickle/unpickle; other fixes ( #1299 )
...
* improve performance of sample_neighbors
* some more improve
* test script
* benchmarks
* multi process
* update more tests
* WIP
* adding two API for state saving
* add create from state
* upd test
* missing file
* wip: pickle/unpickle
* more c apis
* find the problem of empty data array
* add null array; pickling speed is bad
* still bad perf
* still bad perf
* wip
* fix the pickle speed test; now everything looks good
* minor fix
* bugfix
* some lint fix
* address comments
* more fix
* fix lint
* add utest for random.choice
* add utest for dgl.rand_graph
* fix cpp utests
* try fix ci
* fix bug in TF backend
* upd choice docstring
* address comments
* upd
* try fix compile
* add comment
2020-03-02 23:16:05 +08:00
bfdd1eaa44
[Feature] Create shared memory graph store. ( #468 )
...
* accelerate gcn_ns.
* add timing.
* run infer with whole graph.
* distributed gcn_ns.
* reconstruct gcn_ns.
* minor fix.
* change graphsage_cv for numa.
* fix #OMP threads.
* accelerate graphsage_cv.
* fix a weird bug.
* add profiler in graphsage_cv.
* accelerate graphsage_cv.
manually aggregate neighbors' embeddings with pull.
* load csr directly in gcn_ns_sc.
* parallel sort for graph index.
* Revert "parallel sort for graph index."
This reverts commit 86fe2c7117fe5e56b0d481b39849c258b166945b.
* run gcn_ns_sc on GPUs.
* acc gcn_cv_sc.
* change gcn_cv for numa.
* fix gcn_cv to use numa and gpu.
* improve graphsage_cv to use numa and gpu.
* improve gcn_ns.
* improve graphsage_cv.
* init shared memory graph store.
* fix.
* enable init ndata.
* improve tests.
* add bidirectional communication.
* link to rt.
* fix compilation error.
* fix shared memory init.
* use MessageQueue for inter-process communication.
* reconstruct immutable graph csr.
* fix gcn.
* load csr to shared memory.
* fix minor bugs.
* add comments.
* refactor SharedMemory.
* fix bugs in ImmutableGraph.
* create CSR graph from shared memory.
* add more test for loading a csr graph.
* terminate graph store properly.
* allow initializing ndata in the graph store server.
* use RPC for inter-process communication.
* a script for loading a graph.
* allow customizing port.
* list all ndata and edata.
* support dtype.
* reorganize SharedMemoryGraphStore.
* fix ndata shape.
* reconstruct gcn_ns.
* print info.
* set omp in gcn_ns.
* reset sampling examples.
* fix lint.
* fix lint.
* reset gcn.
* disable shared memory in windows.
* fix.
* fix.
* reset changes.
* revert nodeflow changes.
* fix cmake.
* fix test.
* fix test.
* fix test.
* fix test.
* add comments.
* fix test.
* move vector out.
* fix lint.
* fix lint.
* move SharedMemory.
* update cmake.
* update comment.
* fix comments.
* Revert "update cmake."
This reverts commit 592445e37077f70a6e3f2e5245f9a3d086b04f3b.
* update cmake.
* add comments.
* rename.
* change the comment.
* fix a bug.
* rename.
* add comments.
* add comments.
* add init_edata.
* rewrite memory alloc.
* move vector to CSR.
* fix.
* init data.
* Revert "init data."
This reverts commit 2b217b9553911b7dd84a9f1d9b68430b5aa18e23.
* init data.
* init new columns correctly.
2019-04-08 12:09:02 -07:00
707334ce85
minor spelling tweaks ( #349 )
...
* minor spelling tweaks
* Update CONTRIBUTORS.md
2019-01-10 12:02:31 -05:00
a1d50f0f53
[Refactor] Rename before release ( #261 )
...
* include/dgl/runtime
* include
* src/runtime
* src/graph
* src/scheduler
* src
* clean up CMakeLists
* further clean up in cmake
* install commands
* python/dgl/_ffi/_cython
* python/dgl/_ffi/_ctypes
* python/dgl/_ffi
* python/dgl
* some fix
* copy right
2018-12-05 16:45:36 -05:00
2ecd2b23ec
[Frame] Support slice type indexing; optimize dgl.batch ( #110 )
...
* cherry picking optimization from jtnn
* unbatch by slicing frames
* reduce pack
* oops
* support frame read/write with slices
* reverting to unbatch by splitting; slicing is unfriendly to backward
* replacing lru cache with static object factory
* replacing Scheme object with namedtuple
* remove comment
* forgot the find edges interface
* subclassing namedtuple
2018-11-01 00:03:06 -04:00
2694b12725
import ffi solution from TVM
2018-09-05 10:51:31 -04:00
Xin Yao