文件历史

132 次代码提交

作者 SHA1 备注 提交日期
Rhett Ying 0dc0d57faa [DistDGL] back-compatible with dgl.dataloading (#7817) 2024-10-06 08:38:09 +08:00
Rhett Ying 12841c675b [Dist] backward compatible with dgl.dataloading.DistDataLoader (#7782) 2024-09-06 11:21:52 +08:00
Rhett Ying 9782c021de [Dist] move DistNode/EdgeDataLoader to distributed (#7755) 2024-08-29 16:10:05 +08:00
Ayush Noori 025767c204 [Feature] Fixed sampler with limit on sampled nodes/edges in batch subgraph (#6668)
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
2024-07-09 14:00:36 +08:00
Rhett Ying 7f1d164392 [DistGB]enable prob/mask sampling on graphbolt partitions (#7458) 2024-06-13 17:25:22 +08:00
Rhett Ying 1eaa9c266f [DistGB] enable graphbolt sampling for DistEdgeDataLoader (#7432) 2024-05-29 09:22:59 +08:00
Rhett Ying d2a2bf0f8d [DistDGL] enable exclude_edges for sample_etype_neighbors() (#7427) 2024-05-24 13:42:46 +08:00
Rhett Ying 3ebdee7768 [DistGB] sample with graphbolt on homograph via DistNodeDataLoader (#7108) 2024-02-09 09:58:32 +08:00
Ayush Noori 3af61c6b76 [Doc] Fix typo in ShaDowKHopSampler (#6587) 2023-11-22 07:50:08 +08:00
Andrei Ivanov 963f855920 Ralaxing the limit on the possible number of num_workers used. (#6440) 2023-10-19 10:25:43 +08:00
Andrzej Kotłowski c94d2a5f04 [Performance] Do not fuse neighbor sampler for 1 thread (#6421) 2023-10-12 14:31:07 +08:00
Muhammed Fatih BALIN 7e49ccef39 Update Labor sampler docs, add NeurIPS acceptance (#6369) 2023-09-22 12:01:56 +08:00
Jing Zhu a1a3ce89ed [SpotTarget]Edge sampler with excluding edges adjacent to low-degree nodes (#5893)
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: ZYH <31127200+Tonyzhou98@users.noreply.github.com>
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
2023-08-04 16:35:27 +08:00
Muhammed Fatih BALIN ede0558c20 [Feature] Dataloader gpu cache (#6033)
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
2023-07-27 17:23:22 +08:00
AdamGrabowski 4135b1bd5b [Performance] Fused sampling with compaction (#5924)
Co-authored-by: Hesham Mostafa <hesham.mostafa@intel.com>
2023-07-20 09:38:22 +08:00
Muhammed Fatih BALIN d3bd4c6112 [Feature] Adding kappa feature for labor (Cooperative Minibatching) (#6006)
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
2023-07-19 10:34:28 +08:00
Chang Liu 68275b9f3e [Feature] Decompose dataset to each process to add multi-node support for dataloader (#5617) 2023-05-24 16:30:56 -07:00
Xin Yao 8e73c75f4a [Fix] Fix tensor.storage() deprecation warning (#5656) 2023-05-09 13:41:53 +08:00
Hongyi ZHANG e6226e826d [LOG] check whether cpu_affinity is avaible on the platform (#5572)
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2023-04-20 15:44:52 +08:00
peizhou001 4864a9f903 [Bug] Merge rather replace when find reverse eids (#5532)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-19.ap-northeast-1.compute.internal>
2023-04-12 10:42:42 +08:00
Hongzhi (Steve), Chen 8f9f2e2a4d [Misc] Rename number_of_edges and number_of_nodes to num_edges and num_nodes. (#5472)
* core

* revert

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-03-29 08:57:47 +08:00
Mufei Li f5ddb11448 Update neighbor_sampler.py (#5450) 2023-03-14 15:46:40 +08:00
kkranen fe3d29ac4d [Dataloading] Ignore edge types without reverse types in edge dataloader (#5411)
* Bugfix for reverse edge issue.

* Resolved copy/paste transcription error

* lint

* Added unit tests for graph with not all reverses.

* linting + added backend device test

---------

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2023-03-08 22:04:51 +08:00
Quan (Andy) Gan 851d66fa65 [Bugfix] Fix duplicate worker_init_fn argument when provided in DataLoader (#5420)
* fix duplicate worker_init_fn

* lint

* lint again

* uugh
2023-03-06 23:41:53 +08:00
Hongzhi (Steve), Chen d18274885d autofix2 (#5333)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-20 23:27:27 +08:00
Hongzhi (Steve), Chen 6e58f5f1a5 [Misc] Auto format with manual fix of 3 files and add pylint: disable= too-many-lines for functional.py. (#5330)
* blabal

* 2more

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-20 17:14:10 +08:00
Rhett Ying cf67829998 [Doc] fix docstring issue (#5260) 2023-02-01 16:13:43 +08:00
Muhammed Fatih BALIN 5888d9d1b6 [Doc] Labor sampling improvement (#5221)
* [Doc] Labor sampling improvement

When importance sampling option is used, the returned blocks include edge weights that need to be used during the message passing operation. So, I am including this in the documentation.

* Add documentation to weighted sampling case also
2023-01-29 21:32:09 +08:00
Quan (Andy) Gan 580c702417 [DataLoader] Allow batch_size=None for GraphDataLoader (#4483)
* overwrite default_collate_fn

* Update dataloader.py

* Update dataloader.py

* Update dataloader.py

* Update dataloader.py

* Update test_dataloader.py

* revert the test code being reverted in #4956
2023-01-09 00:44:46 +08:00
peizhou001 c55ab2d147 deprecate candidates in dataloader (#5117) 2023-01-06 16:18:32 +08:00
Xin Yao f4eef72628 [Fix core lib warning] Remove LooseVersion (#5026)
* replace LooseVersion with packaging.version

* import packaging from setuptools

* import packaging.version in dgl.utils

* format

* revert changes to apps/kg

* revert changes to _dataloading
2022-12-19 09:02:14 +08:00
Xin Yao f85da5f533 [Bugfix] Turn off use_alternate_stream when TensorAdaptor is not enabled (#4987)
* turn off use_alternate_stream when tensoradaptor is not enabled

* add docstring
2022-12-07 14:41:28 +08:00
Rhett Ying edb14f9358 [Dist] remove dependency on deprecated _dataloading (#4979)
* [Dist] remove dependency on deprecated _dataloading

* fix lint

* refine comment
2022-12-05 18:23:09 +08:00
Muhammed Fatih BALIN bf264d00fe [Feature] (La)yer-Neigh(bor) sampling implementation (#4668)
* adding LABOR sampling

* add ladies and pladies samplers

* fix compile error after rebase

* add reference for ladies sampler

* Improve ladies implementation.

* weighted labor sampling initial implementation draft
fix indentation and small bug in ladies script

* importance_sampling currently doesn't work with weights

* fix weighted importance sampling

* move labor example into its own folder

* lint fixes

* Improve documentation

* remove examples from the main PR

* fix linting by not using c++17 features

* fix documentation of labor_sampler.py

* update documentation for labor.py

* reformat the labor.py file with black

* fix linting errors

* replace exception use with if

* fix typo in error comment

* fixing win64 build for ci

* fixing weighted implementation, works now.

* fix bug in the weighted case and importance_sampling==0

* address part of the reviews

* remove unused code paths from cuda

* remove unused code path from cpu side

* remove extra features of labor making use of random seed.

* fix exclude_edges bug

* remove pcg and seed logic from cpu implementation, seed logic should still work for cuda.

* minor style change

* refactor CPU implementation, take out the importance_sampling probability computation into a function.

* improve CUDAWorkspaceAllocator

* refactor importance_sampling part out to a function

* minor optimization

* fix linting issue

* Revert "remove pcg and seed logic from cpu implementation, seed logic should still work for cuda."

This reverts commit c250e07ac6d7e13f57e79e8a2c2f098d777378c2.

* Revert "remove extra features of labor making use of random seed."

This reverts commit 7f99034353080308f4783f27d9a08bea343fb796.

* fix the documentation

* disable NIDs

* improve the documentation in the code

* use the stream argument in pcg32 instead of skipping ahead t times, can discard the use of hashmap now since it is faster this way.

* fix linting issue

* address another round of reviews

* further optimize CPU LABOR sampling implementation

* fix linting error

* update the comment

* reformat

* rename and rephrase comment

* fix formatting according to new linting specs

* fix compile error due to renaming, fix linting.

* lint

* rename DGLHeteroGraph to DGLGraph to match master

* replace other occurrences of DGLHeteroGraph to DGLGraph

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>
Co-authored-by: Kaan Sancak <kaansnck@gmail.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
2022-11-22 09:03:02 +08:00
peizhou001 3132da2826 Deprecate (#4864)
rename DGLHeteroGraph to DGLGraph
2022-11-15 10:01:01 +08:00
Quan (Andy) Gan 2bca4759c1 [Sampling] Enable sampling with edge masks in sample_etype_neighbors (#4749)
* sample neighbors with masks

* oops

* refactor again

* remove

* remove debug code

* rename macro

* address comments

* more stuff

* remove

* fix

* try fix unit test

* oops

* fix test

* oops

* change name

* rename a lot of stuff

* oops

* ugh

* misc fixes

* lint

* address a lot of comments

* lint

* lint

* fix

* that was silly

* fix

* fix

* fix

* oops
2022-10-29 15:49:44 +08:00
Hongzhi (Steve), Chen a208e8868b [Misc] Black auto fix. (#4680)
* [Misc] Black auto fix.

* fix pylint disable

Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
2022-10-08 14:15:56 +08:00
Quan (Andy) Gan 5ebd3bf05a [DataLoader] Disable the usage of shared memory when persistent_workers=False on single GPU (#4497)
* toggle shared memory usage

* Update dataloader.py

Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-09-20 11:40:33 +08:00
Xin Yao 9a00cf194f [Feature] Import PyTorch's CUDA stream management (#4503)
* add set_stream

* add .record_stream for NDArray and HeteroGraph

* refactor dgl stream Python APIs

* test record_stream

* add unit test for record stream

* use pytorch's stream

* fix lint

* fix cpu build

* address comments

* address comments

* add record stream tests for dgl.graph

* record frames and update dataloder

* add docstring

* update frame

* add backend check for record_stream

* remove CUDAThreadEntry::stream

* record stream for newly created formats

* fix bug

* fix cpp test

* fix None c_void_p to c_handle
2022-09-15 16:54:52 +08:00
Daniil Sizov 47993776df [Feature] Rework Dataloader cpu affinitization as helper method (#4126)
* Add helper method for temporary affinitization of compute threads

* Rework DL affinitization as single helper

* Add example usage in benchmarks

* Fix python linter warnings

* Fix affinity helper params

* Use NUMA node 0 cores only by default

* Fix benchmarks

* Fix lint errors

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-08-18 22:34:12 +08:00
Chang Liu 4f7972957b [Bugfix] Fix dataloader pytorch cuda indexing (#4297)
* Modify to repro crash

* Revert to orig. scenario and add fix

* Update

Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-07-26 16:59:58 +08:00
Xin Yao 82ca781911 [Bug] Record stream when using another CUDA stream for data transfer (#4250)
* record stream when using another cuda stream for data transfer

* fix linting

* fix None stream
2022-07-14 13:17:37 +08:00
Rhett Ying 2f322a94b9 [BugFix] fetch device from graph (#4245)
Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-07-14 10:25:07 +08:00
Chang Liu c56e27a807 [Bugfix] Accessing data from the indexes stored in same device (#4242)
* First update to fix two examples

* Update to fix RGCN/graphsage example and dataloader

* Update
2022-07-13 09:02:07 +08:00
Chang Liu b76d0ed1db [Example][Refactor] Regolden graphsage example for future guide (#4186)
* Regolden graphsage example to guide others

* update golden

* update

* Update example and propagate to original folder

* Update to remove ^M (windows DOS) character

* update

* Merge file changes and update README

* Minor comment update

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2022-06-30 21:56:13 +08:00
Quan (Andy) Gan 7735473b44 [Doc] fix typo (#4193)
Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-06-30 15:00:43 +08:00
Xin Yao 32f12ee19e [Doc] Unify the minimal versions required for PyTorch/TensorFlow/MXNet (#4180) 2022-06-29 18:37:19 +08:00
Xin Yao 5bef48dfda [Performance] Optimize the use of alternative streams in dataloader (#4177)
* fix using alternative streams

* use a alternative stream for subgraph transferring

* fix StreamContext when stream is None
2022-06-29 16:02:25 +08:00
Xin Yao b8f905f18b [Bugfix] Fix that pin_prefetcher is not actually enabled (#4169) 2022-06-27 16:07:22 +08:00
maqy 794ec4a440 [BugFix] fix unstable sort when using dataloader with HeteroGraph (#4147)
* fix unstable sort

* add torch version check

* reformat

* split too long comments

* Update dataloader.py

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-06-22 18:05:42 +08:00