9a00cf194f
[Feature] Import PyTorch's CUDA stream management ( #4503 )
...
* add set_stream
* add .record_stream for NDArray and HeteroGraph
* refactor dgl stream Python APIs
* test record_stream
* add unit test for record stream
* use pytorch's stream
* fix lint
* fix cpu build
* address comments
* address comments
* add record stream tests for dgl.graph
* record frames and update dataloder
* add docstring
* update frame
* add backend check for record_stream
* remove CUDAThreadEntry::stream
* record stream for newly created formats
* fix bug
* fix cpp test
* fix None c_void_p to c_handle
2022-09-15 16:54:52 +08:00
47993776df
[Feature] Rework Dataloader cpu affinitization as helper method ( #4126 )
...
* Add helper method for temporary affinitization of compute threads
* Rework DL affinitization as single helper
* Add example usage in benchmarks
* Fix python linter warnings
* Fix affinity helper params
* Use NUMA node 0 cores only by default
* Fix benchmarks
* Fix lint errors
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-08-18 22:34:12 +08:00
4f7972957b
[Bugfix] Fix dataloader pytorch cuda indexing ( #4297 )
...
* Modify to repro crash
* Revert to orig. scenario and add fix
* Update
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-07-26 16:59:58 +08:00
82ca781911
[Bug] Record stream when using another CUDA stream for data transfer ( #4250 )
...
* record stream when using another cuda stream for data transfer
* fix linting
* fix None stream
2022-07-14 13:17:37 +08:00
2f322a94b9
[BugFix] fetch device from graph ( #4245 )
...
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-07-14 10:25:07 +08:00
c56e27a807
[Bugfix] Accessing data from the indexes stored in same device ( #4242 )
...
* First update to fix two examples
* Update to fix RGCN/graphsage example and dataloader
* Update
2022-07-13 09:02:07 +08:00
b76d0ed1db
[Example][Refactor] Regolden graphsage example for future guide ( #4186 )
...
* Regolden graphsage example to guide others
* update golden
* update
* Update example and propagate to original folder
* Update to remove ^M (windows DOS) character
* update
* Merge file changes and update README
* Minor comment update
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: Mufei Li <mufeili1996@gmail.com >
2022-06-30 21:56:13 +08:00
7735473b44
[Doc] fix typo ( #4193 )
...
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-06-30 15:00:43 +08:00
32f12ee19e
[Doc] Unify the minimal versions required for PyTorch/TensorFlow/MXNet ( #4180 )
2022-06-29 18:37:19 +08:00
5bef48dfda
[Performance] Optimize the use of alternative streams in dataloader ( #4177 )
...
* fix using alternative streams
* use a alternative stream for subgraph transferring
* fix StreamContext when stream is None
2022-06-29 16:02:25 +08:00
b8f905f18b
[Bugfix] Fix that pin_prefetcher is not actually enabled ( #4169 )
2022-06-27 16:07:22 +08:00
794ec4a440
[BugFix] fix unstable sort when using dataloader with HeteroGraph ( #4147 )
...
* fix unstable sort
* add torch version check
* reformat
* split too long comments
* Update dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-06-22 18:05:42 +08:00
4c14781427
[Optimization] Memory consumption optimization on index shuffling in dataloader ( #3980 )
...
* fix
* revert
* Update dataloader.py
2022-05-14 02:11:01 +08:00
5561af4517
[Feature] Dataloader worker affinitization ( #3723 )
...
* PR3355 + CSR conversion workaround
* Remove debug code
* Fix convention errors
* Remove wrongly added code section during merge
* Update to reflect dataloading changes
* Fix missing changes
* Remove comment
* Fix linter errors
* Fix trailing whitespace
* Add wrapper around worker init function
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-04-28 13:00:04 +08:00
c3baf4332b
[Bug] Fix multi-GPU edge classification crashing with pure GPU sampling ( #3946 )
...
* fix
* fix
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-04-27 22:59:21 +08:00
e06e63d5d5
[Feature] Enable UVA sampling with CPU indices ( #3892 )
...
* enable UVA sampling with CPU indices
* add docs
* add more docs
* lint
* fix
* fix
* better error message
* use mp.Barrier instead of queues
* revert
* revert
* oops
* revert dgl.multiprocessing.spawn
* Update pytorch.py
2022-04-12 20:56:51 +08:00
0d878ff8ef
[Example] Cleaned GraphSAGE node classification example with PyTorch Lightning ( #3863 )
...
* cleaned pl node classification example
* conform to PL's method of updating the dataloader
* update
* lint
* fix test
* fix
2022-04-12 17:26:23 +08:00
23a5e674e7
[fix] device cannot be Nonetype object. ( #3822 )
...
* fix device = none to cpu.
* fix singleton-comparison
* modify device is none
* fix use uva
* add docs
* add edge dataloader
* add doc
* fix trailing-whitespace
* modify default
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-04-11 14:58:27 +08:00
d8375c10ec
[BugFix] involve indices_devices for other dataset object. ( #3810 )
...
* involve indices_devices for other dataset object.
* modify raise error when device not found.
* remove empty line
* fix line too long
2022-03-30 11:50:03 +08:00
ae3316c846
[Doc] Update doc of dataloading package. ( #3886 )
...
* wip: dataloading doc
* update dataloading package doc and many others
* lint
2022-03-26 21:59:02 +08:00
f758db386f
[Bug] Fix dtype mismatch in heterogeneous DataLoader ( #3878 )
...
* fix
* unit test
2022-03-26 13:56:21 +08:00
a0e8cf0d3e
[Bugfix] Fix UVA sampling with lazy feature slicing ( #3862 )
...
* fix uva sampling with features
* fix
* add is_listlike function to distinguish strings from sequence
* fix
Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com >
2022-03-24 20:13:01 +08:00
44638b9336
fix ddp dataloader in heterogeneous cases ( #3801 )
2022-03-07 16:56:43 +08:00
d41d07d0f6
[Doc and bugfix] Add docs and user guide and update tutorial for sampling pipeline ( #3774 )
...
* huuuuge update
* remove
* lint
* lint
* fix
* what happened to nccl
* update multi-gpu unsupervised graphsage example
* replace most of the dgl.mp.process with torch.mp.spawn
* update if condition for use_uva case
* update user guide
* address comments
* incorporating suggestions from @jermainewang
* oops
* fix tutorial to pass CI
* oops
* fix again
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-02-28 02:21:51 +08:00
4f00d5ac6f
[Bugfix] Fix graph being duplicated in multi-GPU and CPU dataloader workers ( #3760 )
...
* fix shared memory issue
* oops
* add explanation
* add explanation
2022-02-22 18:00:47 +08:00
3f138ebaaf
[Bugfix] Bug fixes in new dataloader ( #3727 )
...
* fixes
* fix
* more fixes
* update
* oops
* lint?
* temporarily revert - will fix in another PR
* more fixes
* skipping mxnet test
* address comments
* fix DDP
* fix edge dataloader exclusion problems
* stupid bug
* fix
* use_uvm option
* fix
* fixes
* fixes
* fixes
* fixes
* add evaluation for cluster gcn and ddp
* stupid bug again
* fixes
* move sanity checks to only support DGLGraphs
* pytorch lightning compatibility fixes
* remove
* poke
* more fixes
* fix
* fix
* disable test
* docstrings
* why is it getting a memory leak?
* fix
* update
* updates and temporarily disable forkingpickler
* update
* fix?
* fix?
* oops
* oops
* fix
* lint
* huh
* uh
* update
* fix
* made it memory efficient
* refine exclude interface
* fix tutorial
* fix tutorial
* fix graph duplication in CPU dataloader workers
* lint
* lint
* Revert "lint"
This reverts commit 805484dd553695111b5fb37f2125214a6b7276e9.
* Revert "lint"
This reverts commit 0bce411b2b415c2ab770343949404498436dc8b2.
* Revert "fix graph duplication in CPU dataloader workers"
This reverts commit 9e3a8cf34c175d3093c773f6bb023b155f2bd27f.
Co-authored-by: xiny <xiny@nvidia.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-02-22 01:30:31 +08:00
701b4fccc2
[Sampling] New sampling pipeline plus asynchronous prefetching ( #3665 )
...
* initial update
* more
* more
* multi-gpu example
* cluster gcn, finalize homogeneous
* more explanation
* fix
* bunch of fixes
* fix
* RGAT example and more fixes
* shadow-gnn sampler and some changes in unit test
* fix
* wth
* more fixes
* remove shadow+node/edge dataloader tests for possible ux changes
* lints
* add legacy dataloading import just in case
* fix
* update pylint for f-strings
* fix
* lint
* lint
* lint again
* cherry-picking commit fa9f494
* oops
* fix
* add sample_neighbors in dist_graph
* fix
* lint
* fix
* fix
* fix
* fix tutorial
* fix
* fix
* fix
* fix warning
* remove debug
* add get_foo_storage apis
* lint
2022-01-30 16:13:00 +08:00
b8ce0f41a3
[Sampling] Cluster-GCN and ShaDow-GNN DataLoader ( #3487 )
...
* first commit
* next commit
* third commit
* add ShaDow-GNN sampler and unit tests
* fixes
* lint
* cr*p
* lint
* fix lint
* fixes and more unit tests
* more tests
* fix docs
* lint
* fix
* fix
* fix
* fixes
* fix doc
2021-11-16 13:16:25 +08:00
b717c8bf0d
[BugFix] Fix bugs in GPU sampling and enable unit tests for dataloaders on the GPU ( #3474 )
...
* enable unit tests for dataloader on the GPU
* fix compatibility
* copyright
* fix linting
Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com >
2021-11-04 10:34:56 -07:00
a8c81018c5
[Sampling] Implement dgl.compact_graphs() for the GPU ( #3423 )
...
* gpu compact graph template
* cuda compact graph draft
* fix typo
* compact graphs
* pass unit test but fail in training
* example using EdgeDataLoader on the GPU
* refactor cuda_compact_graph and cuda_to_block
* update training scripts
* fix linting
* fix linting
* fix exclude_edges for the GPU
* add --data-cpu & fix copyright
2021-10-20 22:07:35 -07:00
b81bb91465
[Bug] Fix edge exclusion still not working for full neighbor sampling ( #3424 )
2021-10-15 17:51:56 +08:00
5d4f6bca2a
[Fix] Fix edge ID exclusion not working in EdgeDataLoader ( #3412 )
2021-10-14 14:37:54 +08:00
bc14829fb3
[Feature] Exclude edges in sample_neighbors ( #2971 )
...
* [Feature] Exclude edges in sample_neighbors
Extending sample_neighbors and sample_frontier
API to support exclude_edges parameter.
exclude_edges support tensor and dict data
Feature enable excluding certain edges
during neighborhood sampling
Exclude_edges contains EID's of edges
which will be excluded
during neighbor picking for seed nodes.
Added test case for heterograph and homograph
RFC issue id: 2944
* compatibility
* fix
* fix
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2021-09-22 02:15:56 +08:00
75d793a1bc
Update dataloader.py ( #3322 )
2021-09-06 12:16:03 +08:00
f634950825
[Performance][Feature] Implement edge excluding in EdgeDataLoader on GPU ( #3226 )
...
* Update filter code
* Add unit tests
* Fixes
* Switch to indices
* Rename functions
* Fix linting
* Fix whitespace
* Add doc
* Fix heterograph
* Change workspace allocation
* Fix linting
* Fix docs in filter.py
* Add todo
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-08-19 12:24:15 -07:00
34426a98e9
[Distributed] Distributed heterograph training ( #3069 )
...
* support hetero RGCN.
* fix.
* simplify code.
* sample_neighbors return heterograph directly.
* avoid using to_heterogeneous.
* compute canonical etypes in advance.
* fix tests.
* fix.
* fix distributed data loader for heterograph.
* use NodeDataLoader.
* fix bugs in partitioning on heterogeneous graphs.
* fix lint.
* fix tests.
* fix.
* fix.
* fix bugs.
* fix tests.
* fix.
* enable coo for distributed.
* fix.
* fix.
* fix.
* fix.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
2021-07-17 10:57:00 +08:00
2e19ba8025
add exclude self option for EdgeDataLoader ( #3122 )
...
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-07-13 13:30:16 +08:00
427a5a96e0
[Doc] Update NodeDataLoader and EdgeDataLoader for GPU-based neighbor sampling ( #3046 )
...
* update docstrings and tidy code
* add docs
* address comments
* Update __init__.py
* address comments
2021-06-25 10:46:21 +08:00
ff519f98c3
[API] Standardize Subgraph APIs ( #2929 )
...
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Fix
* Update
* Fix subgraph tests
* Capture stdout for distributed test
* Capture stdout for distributed test
* Update
* Update
* Update
* Update subgraph.cc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-06-21 19:53:37 +08:00
8b64ae59b8
[Performance] Perform to_block on the GPU when the dataloader is created with a GPU device. ( #3016 )
...
* add output device for dataloading
* Update dataloader
* Get sampler device from dataloader
* Fix line length
* Update examples
* Fix to_block GPU for empty relation types
* Handle the case where the DistGraph has None for the underlying graph
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-06-13 18:51:00 +08:00
48b9ecd201
Fix discrepancy in dgl.dataloading.EdgeDataLoader ( #2819 )
...
* fix discrepancy
* address comments
2021-04-07 16:08:14 +08:00
1789972c4b
[Bugfix] add default value for BlockSampler ( #2771 )
...
* [Bugfix] add default value for BlockSampler
* [Doc] modify user_guide description about MultiLayerDropoutSample
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-03-22 14:02:27 +08:00
99751d4960
[Doc] Rename block to message flow graph ( #2702 )
...
* rename block to mfg
* revert
* rename
2021-02-25 18:26:19 +08:00
d93a975997
revert ( #2672 )
2021-02-19 12:05:42 +08:00
a6abffe303
[Fix] Don't create extra formats in dataloader ( #2551 )
...
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-01-24 20:59:24 +08:00
bdb88df576
[Performance] Allow DataLoaders to return indices of the seed array ( #2495 )
...
* return indices from dataloader
* fixes
* fix
* fix distgraph and added some todos
* Update dataloader.py
* Update dataloader.py
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-01-08 14:30:52 +08:00
e2e524df41
[Feature] Add GraphDataLoader implementation ( #2496 )
...
* add graph dataloader
* add to doc
* fix
* fix
* fix docstring
* update according to torch default_collate
* add unittest
* fix
* fix lint
* fix
2021-01-07 13:38:21 +08:00
0c2a2ea1e2
[Bugfix] Handle a Corner Case of Batching after Removing Nodes/Edges ( #2465 )
...
* Update
* Update
* Update
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-01-05 22:22:03 +08:00
76d66fd35d
[Bug] Fix dtype mismatch in EdgeDataLoader on Windows ( #2188 )
...
* fix node and edge dataloader on windows
* fix distributed
2020-09-13 18:25:05 +08:00
b10b541c11
fix multiple docstring inconsistencies ( #2167 )
2020-09-10 11:03:34 +08:00
Xin Yao