0dc0d57faa
[DistDGL] back-compatible with dgl.dataloading ( #7817 )
2024-10-06 08:38:09 +08:00
12841c675b
[Dist] backward compatible with dgl.dataloading.DistDataLoader ( #7782 )
2024-09-06 11:21:52 +08:00
9782c021de
[Dist] move DistNode/EdgeDataLoader to distributed ( #7755 )
2024-08-29 16:10:05 +08:00
963f855920
Ralaxing the limit on the possible number of num_workers used. ( #6440 )
2023-10-19 10:25:43 +08:00
ede0558c20
[Feature] Dataloader gpu cache ( #6033 )
...
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com >
2023-07-27 17:23:22 +08:00
68275b9f3e
[Feature] Decompose dataset to each process to add multi-node support for dataloader ( #5617 )
2023-05-24 16:30:56 -07:00
8e73c75f4a
[Fix] Fix tensor.storage() deprecation warning ( #5656 )
2023-05-09 13:41:53 +08:00
e6226e826d
[LOG] check whether cpu_affinity is avaible on the platform ( #5572 )
...
Co-authored-by: Mufei Li <mufeili1996@gmail.com >
2023-04-20 15:44:52 +08:00
851d66fa65
[Bugfix] Fix duplicate worker_init_fn argument when provided in DataLoader ( #5420 )
...
* fix duplicate worker_init_fn
* lint
* lint again
* uugh
2023-03-06 23:41:53 +08:00
6e58f5f1a5
[Misc] Auto format with manual fix of 3 files and add pylint: disable= too-many-lines for functional.py. ( #5330 )
...
* blabal
* 2more
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal >
2023-02-20 17:14:10 +08:00
580c702417
[DataLoader] Allow batch_size=None for GraphDataLoader ( #4483 )
...
* overwrite default_collate_fn
* Update dataloader.py
* Update dataloader.py
* Update dataloader.py
* Update dataloader.py
* Update test_dataloader.py
* revert the test code being reverted in #4956
2023-01-09 00:44:46 +08:00
c55ab2d147
deprecate candidates in dataloader ( #5117 )
2023-01-06 16:18:32 +08:00
f4eef72628
[Fix core lib warning] Remove LooseVersion ( #5026 )
...
* replace LooseVersion with packaging.version
* import packaging from setuptools
* import packaging.version in dgl.utils
* format
* revert changes to apps/kg
* revert changes to _dataloading
2022-12-19 09:02:14 +08:00
f85da5f533
[Bugfix] Turn off use_alternate_stream when TensorAdaptor is not enabled ( #4987 )
...
* turn off use_alternate_stream when tensoradaptor is not enabled
* add docstring
2022-12-07 14:41:28 +08:00
3132da2826
Deprecate ( #4864 )
...
rename DGLHeteroGraph to DGLGraph
2022-11-15 10:01:01 +08:00
5ebd3bf05a
[DataLoader] Disable the usage of shared memory when persistent_workers=False on single GPU ( #4497 )
...
* toggle shared memory usage
* Update dataloader.py
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-09-20 11:40:33 +08:00
9a00cf194f
[Feature] Import PyTorch's CUDA stream management ( #4503 )
...
* add set_stream
* add .record_stream for NDArray and HeteroGraph
* refactor dgl stream Python APIs
* test record_stream
* add unit test for record stream
* use pytorch's stream
* fix lint
* fix cpu build
* address comments
* address comments
* add record stream tests for dgl.graph
* record frames and update dataloder
* add docstring
* update frame
* add backend check for record_stream
* remove CUDAThreadEntry::stream
* record stream for newly created formats
* fix bug
* fix cpp test
* fix None c_void_p to c_handle
2022-09-15 16:54:52 +08:00
47993776df
[Feature] Rework Dataloader cpu affinitization as helper method ( #4126 )
...
* Add helper method for temporary affinitization of compute threads
* Rework DL affinitization as single helper
* Add example usage in benchmarks
* Fix python linter warnings
* Fix affinity helper params
* Use NUMA node 0 cores only by default
* Fix benchmarks
* Fix lint errors
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-08-18 22:34:12 +08:00
4f7972957b
[Bugfix] Fix dataloader pytorch cuda indexing ( #4297 )
...
* Modify to repro crash
* Revert to orig. scenario and add fix
* Update
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-07-26 16:59:58 +08:00
82ca781911
[Bug] Record stream when using another CUDA stream for data transfer ( #4250 )
...
* record stream when using another cuda stream for data transfer
* fix linting
* fix None stream
2022-07-14 13:17:37 +08:00
2f322a94b9
[BugFix] fetch device from graph ( #4245 )
...
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-07-14 10:25:07 +08:00
c56e27a807
[Bugfix] Accessing data from the indexes stored in same device ( #4242 )
...
* First update to fix two examples
* Update to fix RGCN/graphsage example and dataloader
* Update
2022-07-13 09:02:07 +08:00
b76d0ed1db
[Example][Refactor] Regolden graphsage example for future guide ( #4186 )
...
* Regolden graphsage example to guide others
* update golden
* update
* Update example and propagate to original folder
* Update to remove ^M (windows DOS) character
* update
* Merge file changes and update README
* Minor comment update
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
Co-authored-by: Mufei Li <mufeili1996@gmail.com >
2022-06-30 21:56:13 +08:00
7735473b44
[Doc] fix typo ( #4193 )
...
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-06-30 15:00:43 +08:00
32f12ee19e
[Doc] Unify the minimal versions required for PyTorch/TensorFlow/MXNet ( #4180 )
2022-06-29 18:37:19 +08:00
5bef48dfda
[Performance] Optimize the use of alternative streams in dataloader ( #4177 )
...
* fix using alternative streams
* use a alternative stream for subgraph transferring
* fix StreamContext when stream is None
2022-06-29 16:02:25 +08:00
b8f905f18b
[Bugfix] Fix that pin_prefetcher is not actually enabled ( #4169 )
2022-06-27 16:07:22 +08:00
794ec4a440
[BugFix] fix unstable sort when using dataloader with HeteroGraph ( #4147 )
...
* fix unstable sort
* add torch version check
* reformat
* split too long comments
* Update dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-06-22 18:05:42 +08:00
4c14781427
[Optimization] Memory consumption optimization on index shuffling in dataloader ( #3980 )
...
* fix
* revert
* Update dataloader.py
2022-05-14 02:11:01 +08:00
5561af4517
[Feature] Dataloader worker affinitization ( #3723 )
...
* PR3355 + CSR conversion workaround
* Remove debug code
* Fix convention errors
* Remove wrongly added code section during merge
* Update to reflect dataloading changes
* Fix missing changes
* Remove comment
* Fix linter errors
* Fix trailing whitespace
* Add wrapper around worker init function
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-04-28 13:00:04 +08:00
c3baf4332b
[Bug] Fix multi-GPU edge classification crashing with pure GPU sampling ( #3946 )
...
* fix
* fix
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-04-27 22:59:21 +08:00
e06e63d5d5
[Feature] Enable UVA sampling with CPU indices ( #3892 )
...
* enable UVA sampling with CPU indices
* add docs
* add more docs
* lint
* fix
* fix
* better error message
* use mp.Barrier instead of queues
* revert
* revert
* oops
* revert dgl.multiprocessing.spawn
* Update pytorch.py
2022-04-12 20:56:51 +08:00
0d878ff8ef
[Example] Cleaned GraphSAGE node classification example with PyTorch Lightning ( #3863 )
...
* cleaned pl node classification example
* conform to PL's method of updating the dataloader
* update
* lint
* fix test
* fix
2022-04-12 17:26:23 +08:00
23a5e674e7
[fix] device cannot be Nonetype object. ( #3822 )
...
* fix device = none to cpu.
* fix singleton-comparison
* modify device is none
* fix use uva
* add docs
* add edge dataloader
* add doc
* fix trailing-whitespace
* modify default
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2022-04-11 14:58:27 +08:00
d8375c10ec
[BugFix] involve indices_devices for other dataset object. ( #3810 )
...
* involve indices_devices for other dataset object.
* modify raise error when device not found.
* remove empty line
* fix line too long
2022-03-30 11:50:03 +08:00
ae3316c846
[Doc] Update doc of dataloading package. ( #3886 )
...
* wip: dataloading doc
* update dataloading package doc and many others
* lint
2022-03-26 21:59:02 +08:00
f758db386f
[Bug] Fix dtype mismatch in heterogeneous DataLoader ( #3878 )
...
* fix
* unit test
2022-03-26 13:56:21 +08:00
a0e8cf0d3e
[Bugfix] Fix UVA sampling with lazy feature slicing ( #3862 )
...
* fix uva sampling with features
* fix
* add is_listlike function to distinguish strings from sequence
* fix
Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com >
2022-03-24 20:13:01 +08:00
44638b9336
fix ddp dataloader in heterogeneous cases ( #3801 )
2022-03-07 16:56:43 +08:00
d41d07d0f6
[Doc and bugfix] Add docs and user guide and update tutorial for sampling pipeline ( #3774 )
...
* huuuuge update
* remove
* lint
* lint
* fix
* what happened to nccl
* update multi-gpu unsupervised graphsage example
* replace most of the dgl.mp.process with torch.mp.spawn
* update if condition for use_uva case
* update user guide
* address comments
* incorporating suggestions from @jermainewang
* oops
* fix tutorial to pass CI
* oops
* fix again
Co-authored-by: Xin Yao <xiny@nvidia.com >
2022-02-28 02:21:51 +08:00
4f00d5ac6f
[Bugfix] Fix graph being duplicated in multi-GPU and CPU dataloader workers ( #3760 )
...
* fix shared memory issue
* oops
* add explanation
* add explanation
2022-02-22 18:00:47 +08:00
3f138ebaaf
[Bugfix] Bug fixes in new dataloader ( #3727 )
...
* fixes
* fix
* more fixes
* update
* oops
* lint?
* temporarily revert - will fix in another PR
* more fixes
* skipping mxnet test
* address comments
* fix DDP
* fix edge dataloader exclusion problems
* stupid bug
* fix
* use_uvm option
* fix
* fixes
* fixes
* fixes
* fixes
* add evaluation for cluster gcn and ddp
* stupid bug again
* fixes
* move sanity checks to only support DGLGraphs
* pytorch lightning compatibility fixes
* remove
* poke
* more fixes
* fix
* fix
* disable test
* docstrings
* why is it getting a memory leak?
* fix
* update
* updates and temporarily disable forkingpickler
* update
* fix?
* fix?
* oops
* oops
* fix
* lint
* huh
* uh
* update
* fix
* made it memory efficient
* refine exclude interface
* fix tutorial
* fix tutorial
* fix graph duplication in CPU dataloader workers
* lint
* lint
* Revert "lint"
This reverts commit 805484dd553695111b5fb37f2125214a6b7276e9.
* Revert "lint"
This reverts commit 0bce411b2b415c2ab770343949404498436dc8b2.
* Revert "fix graph duplication in CPU dataloader workers"
This reverts commit 9e3a8cf34c175d3093c773f6bb023b155f2bd27f.
Co-authored-by: xiny <xiny@nvidia.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2022-02-22 01:30:31 +08:00
701b4fccc2
[Sampling] New sampling pipeline plus asynchronous prefetching ( #3665 )
...
* initial update
* more
* more
* multi-gpu example
* cluster gcn, finalize homogeneous
* more explanation
* fix
* bunch of fixes
* fix
* RGAT example and more fixes
* shadow-gnn sampler and some changes in unit test
* fix
* wth
* more fixes
* remove shadow+node/edge dataloader tests for possible ux changes
* lints
* add legacy dataloading import just in case
* fix
* update pylint for f-strings
* fix
* lint
* lint
* lint again
* cherry-picking commit fa9f494
* oops
* fix
* add sample_neighbors in dist_graph
* fix
* lint
* fix
* fix
* fix
* fix tutorial
* fix
* fix
* fix
* fix warning
* remove debug
* add get_foo_storage apis
* lint
2022-01-30 16:13:00 +08:00
b8ce0f41a3
[Sampling] Cluster-GCN and ShaDow-GNN DataLoader ( #3487 )
...
* first commit
* next commit
* third commit
* add ShaDow-GNN sampler and unit tests
* fixes
* lint
* cr*p
* lint
* fix lint
* fixes and more unit tests
* more tests
* fix docs
* lint
* fix
* fix
* fix
* fixes
* fix doc
2021-11-16 13:16:25 +08:00
b717c8bf0d
[BugFix] Fix bugs in GPU sampling and enable unit tests for dataloaders on the GPU ( #3474 )
...
* enable unit tests for dataloader on the GPU
* fix compatibility
* copyright
* fix linting
Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com >
2021-11-04 10:34:56 -07:00
a8c81018c5
[Sampling] Implement dgl.compact_graphs() for the GPU ( #3423 )
...
* gpu compact graph template
* cuda compact graph draft
* fix typo
* compact graphs
* pass unit test but fail in training
* example using EdgeDataLoader on the GPU
* refactor cuda_compact_graph and cuda_to_block
* update training scripts
* fix linting
* fix linting
* fix exclude_edges for the GPU
* add --data-cpu & fix copyright
2021-10-20 22:07:35 -07:00
b81bb91465
[Bug] Fix edge exclusion still not working for full neighbor sampling ( #3424 )
2021-10-15 17:51:56 +08:00
5d4f6bca2a
[Fix] Fix edge ID exclusion not working in EdgeDataLoader ( #3412 )
2021-10-14 14:37:54 +08:00
bc14829fb3
[Feature] Exclude edges in sample_neighbors ( #2971 )
...
* [Feature] Exclude edges in sample_neighbors
Extending sample_neighbors and sample_frontier
API to support exclude_edges parameter.
exclude_edges support tensor and dict data
Feature enable excluding certain edges
during neighborhood sampling
Exclude_edges contains EID's of edges
which will be excluded
during neighbor picking for seed nodes.
Added test case for heterograph and homograph
RFC issue id: 2944
* compatibility
* fix
* fix
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2021-09-22 02:15:56 +08:00
75d793a1bc
Update dataloader.py ( #3322 )
2021-09-06 12:16:03 +08:00
Rhett Ying