文件历史

提交图

111 次代码提交

作者 SHA1 备注 提交日期
Mufei Li f5ddb11448 Update neighbor_sampler.py (#5450) 2023-03-14 15:46:40 +08:00
kkranen fe3d29ac4d [Dataloading] Ignore edge types without reverse types in edge dataloader (#5411)
* Bugfix for reverse edge issue.

* Resolved copy/paste transcription error

* lint

* Added unit tests for graph with not all reverses.

* linting + added backend device test

---------

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2023-03-08 22:04:51 +08:00
Quan (Andy) Gan 851d66fa65 [Bugfix] Fix duplicate worker_init_fn argument when provided in DataLoader (#5420)
* fix duplicate worker_init_fn

* lint

* lint again

* uugh
2023-03-06 23:41:53 +08:00
Hongzhi (Steve), Chen d18274885d autofix2 (#5333)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-20 23:27:27 +08:00
Hongzhi (Steve), Chen 6e58f5f1a5 [Misc] Auto format with manual fix of 3 files and add pylint: disable= too-many-lines for functional.py. (#5330)
* blabal

* 2more

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-20 17:14:10 +08:00
Rhett Ying cf67829998 [Doc] fix docstring issue (#5260) 2023-02-01 16:13:43 +08:00
Muhammed Fatih BALIN 5888d9d1b6 [Doc] Labor sampling improvement (#5221)
* [Doc] Labor sampling improvement

When importance sampling option is used, the returned blocks include edge weights that need to be used during the message passing operation. So, I am including this in the documentation.

* Add documentation to weighted sampling case also
2023-01-29 21:32:09 +08:00
Quan (Andy) Gan 580c702417 [DataLoader] Allow batch_size=None for GraphDataLoader (#4483)
* overwrite default_collate_fn

* Update dataloader.py

* Update dataloader.py

* Update dataloader.py

* Update dataloader.py

* Update test_dataloader.py

* revert the test code being reverted in #4956
2023-01-09 00:44:46 +08:00
peizhou001 c55ab2d147 deprecate candidates in dataloader (#5117) 2023-01-06 16:18:32 +08:00
Xin Yao f4eef72628 [Fix core lib warning] Remove LooseVersion (#5026)
* replace LooseVersion with packaging.version

* import packaging from setuptools

* import packaging.version in dgl.utils

* format

* revert changes to apps/kg

* revert changes to _dataloading
2022-12-19 09:02:14 +08:00
Xin Yao f85da5f533 [Bugfix] Turn off use_alternate_stream when TensorAdaptor is not enabled (#4987)
* turn off use_alternate_stream when tensoradaptor is not enabled

* add docstring
2022-12-07 14:41:28 +08:00
Rhett Ying edb14f9358 [Dist] remove dependency on deprecated _dataloading (#4979)
* [Dist] remove dependency on deprecated _dataloading

* fix lint

* refine comment
2022-12-05 18:23:09 +08:00
Muhammed Fatih BALIN bf264d00fe [Feature] (La)yer-Neigh(bor) sampling implementation (#4668)
* adding LABOR sampling

* add ladies and pladies samplers

* fix compile error after rebase

* add reference for ladies sampler

* Improve ladies implementation.

* weighted labor sampling initial implementation draft
fix indentation and small bug in ladies script

* importance_sampling currently doesn't work with weights

* fix weighted importance sampling

* move labor example into its own folder

* lint fixes

* Improve documentation

* remove examples from the main PR

* fix linting by not using c++17 features

* fix documentation of labor_sampler.py

* update documentation for labor.py

* reformat the labor.py file with black

* fix linting errors

* replace exception use with if

* fix typo in error comment

* fixing win64 build for ci

* fixing weighted implementation, works now.

* fix bug in the weighted case and importance_sampling==0

* address part of the reviews

* remove unused code paths from cuda

* remove unused code path from cpu side

* remove extra features of labor making use of random seed.

* fix exclude_edges bug

* remove pcg and seed logic from cpu implementation, seed logic should still work for cuda.

* minor style change

* refactor CPU implementation, take out the importance_sampling probability computation into a function.

* improve CUDAWorkspaceAllocator

* refactor importance_sampling part out to a function

* minor optimization

* fix linting issue

* Revert "remove pcg and seed logic from cpu implementation, seed logic should still work for cuda."

This reverts commit c250e07ac6d7e13f57e79e8a2c2f098d777378c2.

* Revert "remove extra features of labor making use of random seed."

This reverts commit 7f99034353080308f4783f27d9a08bea343fb796.

* fix the documentation

* disable NIDs

* improve the documentation in the code

* use the stream argument in pcg32 instead of skipping ahead t times, can discard the use of hashmap now since it is faster this way.

* fix linting issue

* address another round of reviews

* further optimize CPU LABOR sampling implementation

* fix linting error

* update the comment

* reformat

* rename and rephrase comment

* fix formatting according to new linting specs

* fix compile error due to renaming, fix linting.

* lint

* rename DGLHeteroGraph to DGLGraph to match master

* replace other occurrences of DGLHeteroGraph to DGLGraph

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>
Co-authored-by: Kaan Sancak <kaansnck@gmail.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
2022-11-22 09:03:02 +08:00
peizhou001 3132da2826 Deprecate (#4864)
rename DGLHeteroGraph to DGLGraph
2022-11-15 10:01:01 +08:00
Quan (Andy) Gan 2bca4759c1 [Sampling] Enable sampling with edge masks in sample_etype_neighbors (#4749)
* sample neighbors with masks

* oops

* refactor again

* remove

* remove debug code

* rename macro

* address comments

* more stuff

* remove

* fix

* try fix unit test

* oops

* fix test

* oops

* change name

* rename a lot of stuff

* oops

* ugh

* misc fixes

* lint

* address a lot of comments

* lint

* lint

* fix

* that was silly

* fix

* fix

* fix

* oops
2022-10-29 15:49:44 +08:00
Hongzhi (Steve), Chen a208e8868b [Misc] Black auto fix. (#4680)
* [Misc] Black auto fix.

* fix pylint disable

Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
2022-10-08 14:15:56 +08:00
Quan (Andy) Gan 5ebd3bf05a [DataLoader] Disable the usage of shared memory when persistent_workers=False on single GPU (#4497)
* toggle shared memory usage

* Update dataloader.py

Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-09-20 11:40:33 +08:00
Xin Yao 9a00cf194f [Feature] Import PyTorch's CUDA stream management (#4503)
* add set_stream

* add .record_stream for NDArray and HeteroGraph

* refactor dgl stream Python APIs

* test record_stream

* add unit test for record stream

* use pytorch's stream

* fix lint

* fix cpu build

* address comments

* address comments

* add record stream tests for dgl.graph

* record frames and update dataloder

* add docstring

* update frame

* add backend check for record_stream

* remove CUDAThreadEntry::stream

* record stream for newly created formats

* fix bug

* fix cpp test

* fix None c_void_p to c_handle
2022-09-15 16:54:52 +08:00
Daniil Sizov 47993776df [Feature] Rework Dataloader cpu affinitization as helper method (#4126)
* Add helper method for temporary affinitization of compute threads

* Rework DL affinitization as single helper

* Add example usage in benchmarks

* Fix python linter warnings

* Fix affinity helper params

* Use NUMA node 0 cores only by default

* Fix benchmarks

* Fix lint errors

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-08-18 22:34:12 +08:00
Chang Liu 4f7972957b [Bugfix] Fix dataloader pytorch cuda indexing (#4297)
* Modify to repro crash

* Revert to orig. scenario and add fix

* Update

Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-07-26 16:59:58 +08:00
Xin Yao 82ca781911 [Bug] Record stream when using another CUDA stream for data transfer (#4250)
* record stream when using another cuda stream for data transfer

* fix linting

* fix None stream
2022-07-14 13:17:37 +08:00
Rhett Ying 2f322a94b9 [BugFix] fetch device from graph (#4245)
Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-07-14 10:25:07 +08:00
Chang Liu c56e27a807 [Bugfix] Accessing data from the indexes stored in same device (#4242)
* First update to fix two examples

* Update to fix RGCN/graphsage example and dataloader

* Update
2022-07-13 09:02:07 +08:00
Chang Liu b76d0ed1db [Example][Refactor] Regolden graphsage example for future guide (#4186)
* Regolden graphsage example to guide others

* update golden

* update

* Update example and propagate to original folder

* Update to remove ^M (windows DOS) character

* update

* Merge file changes and update README

* Minor comment update

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2022-06-30 21:56:13 +08:00
Quan (Andy) Gan 7735473b44 [Doc] fix typo (#4193)
Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-06-30 15:00:43 +08:00
Xin Yao 32f12ee19e [Doc] Unify the minimal versions required for PyTorch/TensorFlow/MXNet (#4180) 2022-06-29 18:37:19 +08:00
Xin Yao 5bef48dfda [Performance] Optimize the use of alternative streams in dataloader (#4177)
* fix using alternative streams

* use a alternative stream for subgraph transferring

* fix StreamContext when stream is None
2022-06-29 16:02:25 +08:00
Xin Yao b8f905f18b [Bugfix] Fix that pin_prefetcher is not actually enabled (#4169) 2022-06-27 16:07:22 +08:00
maqy 794ec4a440 [BugFix] fix unstable sort when using dataloader with HeteroGraph (#4147)
* fix unstable sort

* add torch version check

* reformat

* split too long comments

* Update dataloader.py

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-06-22 18:05:42 +08:00
Sai Kandregula 7ec165c2c5 default value typo fix in GlobalUniform negative sampler (#4060)
Co-authored-by: decoherencer <decoherencer@users.noreply.github.com>
2022-05-28 13:22:41 +08:00
Feiyang(Vance) Chen d70a362dba Update shadow.py (#4034)
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2022-05-23 19:06:18 +08:00
Quan (Andy) Gan 4c14781427 [Optimization] Memory consumption optimization on index shuffling in dataloader (#3980)
* fix

* revert

* Update dataloader.py
2022-05-14 02:11:01 +08:00
Daniil Sizov 5561af4517 [Feature] Dataloader worker affinitization (#3723)
* PR3355 + CSR conversion workaround

* Remove debug code

* Fix convention errors

* Remove wrongly added code section during merge

* Update to reflect dataloading changes

* Fix missing changes

* Remove comment

* Fix linter errors

* Fix trailing whitespace

* Add wrapper around worker init function

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-04-28 13:00:04 +08:00
Quan (Andy) Gan c3baf4332b [Bug] Fix multi-GPU edge classification crashing with pure GPU sampling (#3946)
* fix

* fix

Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-04-27 22:59:21 +08:00
Quan (Andy) Gan e06e63d5d5 [Feature] Enable UVA sampling with CPU indices (#3892)
* enable UVA sampling with CPU indices

* add docs

* add more docs

* lint

* fix

* fix

* better error message

* use mp.Barrier instead of queues

* revert

* revert

* oops

* revert dgl.multiprocessing.spawn

* Update pytorch.py
2022-04-12 20:56:51 +08:00
Quan (Andy) Gan 0d878ff8ef [Example] Cleaned GraphSAGE node classification example with PyTorch Lightning (#3863)
* cleaned pl node classification example

* conform to PL's method of updating the dataloader

* update

* lint

* fix test

* fix
2022-04-12 17:26:23 +08:00
nv-dlasalle baa9292899 [bugfix] Fix EdgePredictionSampler for UVA sampling (#3904)
* Add failing unit test

* Fix negative sampler edge types

* fix test

* oops

* revert

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
2022-04-12 15:25:21 +08:00
Peiqi Yin 23a5e674e7 [fix] device cannot be Nonetype object. (#3822)
* fix device = none to cpu.

* fix singleton-comparison

* modify device is none

* fix use uva

* add docs

* add edge dataloader

* add doc

* fix trailing-whitespace

* modify default

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-04-11 14:58:27 +08:00
Mufei Li 11e910b6f8 [Doc] Hotfix for GraphSAINT Doc (#3898)
* Update graphsaint.py

* Update graphsaint.py
2022-03-31 19:22:21 +08:00
Mufei Li 9fee20b91d [Sampler] [Example] SAINTSampler and Simplify GraphSAINT Example (#3879)
* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-03-30 13:51:37 +08:00
Peiqi Yin d8375c10ec [BugFix] involve indices_devices for other dataset object. (#3810)
* involve indices_devices for other dataset object.

* modify raise error when device not found.

* remove empty line

* fix line too long
2022-03-30 11:50:03 +08:00
Minjie Wang ae3316c846 [Doc] Update doc of dataloading package. (#3886)
* wip: dataloading doc

* update dataloading package doc and many others

* lint
2022-03-26 21:59:02 +08:00
Quan (Andy) Gan f758db386f [Bug] Fix dtype mismatch in heterogeneous DataLoader (#3878)
* fix

* unit test
2022-03-26 13:56:21 +08:00
Quan (Andy) Gan a0e8cf0d3e [Bugfix] Fix UVA sampling with lazy feature slicing (#3862)
* fix uva sampling with features

* fix

* add is_listlike function to distinguish strings from sequence

* fix

Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com>
2022-03-24 20:13:01 +08:00
Quan (Andy) Gan 44638b9336 fix ddp dataloader in heterogeneous cases (#3801) 2022-03-07 16:56:43 +08:00
Quan (Andy) Gan d41d07d0f6 [Doc and bugfix] Add docs and user guide and update tutorial for sampling pipeline (#3774)
* huuuuge update

* remove

* lint

* lint

* fix

* what happened to nccl

* update multi-gpu unsupervised graphsage example

* replace most of the dgl.mp.process with torch.mp.spawn

* update if condition for use_uva case

* update user guide

* address comments

* incorporating suggestions from @jermainewang

* oops

* fix tutorial to pass CI

* oops

* fix again

Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-02-28 02:21:51 +08:00
Mufei Li 6d9433b04f [Transform] [Doc] Rename transform to transforms and update doc (#3765)
* Update

* Update

* Update

* Fix

* Update

* Update

* Update

* Fix
2022-02-24 21:28:15 +08:00
Quan (Andy) Gan 4f00d5ac6f [Bugfix] Fix graph being duplicated in multi-GPU and CPU dataloader workers (#3760)
* fix shared memory issue

* oops

* add explanation

* add explanation
2022-02-22 18:00:47 +08:00
Quan (Andy) Gan 3f138ebaaf [Bugfix] Bug fixes in new dataloader (#3727)
* fixes

* fix

* more fixes

* update

* oops

* lint?

* temporarily revert - will fix in another PR

* more fixes

* skipping mxnet test

* address comments

* fix DDP

* fix edge dataloader exclusion problems

* stupid bug

* fix

* use_uvm option

* fix

* fixes

* fixes

* fixes

* fixes

* add evaluation for cluster gcn and ddp

* stupid bug again

* fixes

* move sanity checks to only support DGLGraphs

* pytorch lightning compatibility fixes

* remove

* poke

* more fixes

* fix

* fix

* disable test

* docstrings

* why is it getting a memory leak?

* fix

* update

* updates and temporarily disable forkingpickler

* update

* fix?

* fix?

* oops

* oops

* fix

* lint

* huh

* uh

* update

* fix

* made it memory efficient

* refine exclude interface

* fix tutorial

* fix tutorial

* fix graph duplication in CPU dataloader workers

* lint

* lint

* Revert "lint"

This reverts commit 805484dd553695111b5fb37f2125214a6b7276e9.

* Revert "lint"

This reverts commit 0bce411b2b415c2ab770343949404498436dc8b2.

* Revert "fix graph duplication in CPU dataloader workers"

This reverts commit 9e3a8cf34c175d3093c773f6bb023b155f2bd27f.

Co-authored-by: xiny <xiny@nvidia.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
2022-02-22 01:30:31 +08:00
Quan (Andy) Gan 701b4fccc2 [Sampling] New sampling pipeline plus asynchronous prefetching (#3665)
* initial update

* more

* more

* multi-gpu example

* cluster gcn, finalize homogeneous

* more explanation

* fix

* bunch of fixes

* fix

* RGAT example and more fixes

* shadow-gnn sampler and some changes in unit test

* fix

* wth

* more fixes

* remove shadow+node/edge dataloader tests for possible ux changes

* lints

* add legacy dataloading import just in case

* fix

* update pylint for f-strings

* fix

* lint

* lint

* lint again

* cherry-picking commit fa9f494

* oops

* fix

* add sample_neighbors in dist_graph

* fix

* lint

* fix

* fix

* fix

* fix tutorial

* fix

* fix

* fix

* fix warning

* remove debug

* add get_foo_storage apis

* lint
2022-01-30 16:13:00 +08:00