文件历史

24 次代码提交

作者 SHA1 备注 提交日期
nv-dlasalle 1425150459 Fix test naming (#4023)
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2022-05-23 17:21:06 +08:00
Quan (Andy) Gan e06e63d5d5 [Feature] Enable UVA sampling with CPU indices (#3892)
* enable UVA sampling with CPU indices

* add docs

* add more docs

* lint

* fix

* fix

* better error message

* use mp.Barrier instead of queues

* revert

* revert

* oops

* revert dgl.multiprocessing.spawn

* Update pytorch.py
2022-04-12 20:56:51 +08:00
Mufei Li 9fee20b91d [Sampler] [Example] SAINTSampler and Simplify GraphSAINT Example (#3879)
* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-03-30 13:51:37 +08:00
Quan (Andy) Gan f758db386f [Bug] Fix dtype mismatch in heterogeneous DataLoader (#3878)
* fix

* unit test
2022-03-26 13:56:21 +08:00
Quan (Andy) Gan a0e8cf0d3e [Bugfix] Fix UVA sampling with lazy feature slicing (#3862)
* fix uva sampling with features

* fix

* add is_listlike function to distinguish strings from sequence

* fix

Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com>
2022-03-24 20:13:01 +08:00
Quan (Andy) Gan 3f138ebaaf [Bugfix] Bug fixes in new dataloader (#3727)
* fixes

* fix

* more fixes

* update

* oops

* lint?

* temporarily revert - will fix in another PR

* more fixes

* skipping mxnet test

* address comments

* fix DDP

* fix edge dataloader exclusion problems

* stupid bug

* fix

* use_uvm option

* fix

* fixes

* fixes

* fixes

* fixes

* add evaluation for cluster gcn and ddp

* stupid bug again

* fixes

* move sanity checks to only support DGLGraphs

* pytorch lightning compatibility fixes

* remove

* poke

* more fixes

* fix

* fix

* disable test

* docstrings

* why is it getting a memory leak?

* fix

* update

* updates and temporarily disable forkingpickler

* update

* fix?

* fix?

* oops

* oops

* fix

* lint

* huh

* uh

* update

* fix

* made it memory efficient

* refine exclude interface

* fix tutorial

* fix tutorial

* fix graph duplication in CPU dataloader workers

* lint

* lint

* Revert "lint"

This reverts commit 805484dd553695111b5fb37f2125214a6b7276e9.

* Revert "lint"

This reverts commit 0bce411b2b415c2ab770343949404498436dc8b2.

* Revert "fix graph duplication in CPU dataloader workers"

This reverts commit 9e3a8cf34c175d3093c773f6bb023b155f2bd27f.

Co-authored-by: xiny <xiny@nvidia.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
2022-02-22 01:30:31 +08:00
Xin Yao 738e8318fd [Feature] CUDA UVA sampling for MultiLayerNeighborSampler (#3674)
* implement pin_memory/unpin_memory/is_pinned for dgl.graph

* update python docstring

* update c++ docstring

* add test

* fix the broken UnifiedTensor

* XPU_SWITCH for kDLCPUPinned

* a rough version ready for testing

* eliminate extra context parameter for pin/unpin

* update train_sampling

* fix linting

* fix typo

* multi-gpu uva sampling case

* disable new format materialization for pinned graphs

* update python doc for pin_memory_

* fix unit test

* UVA sampling for link prediction

* dispatch most csr ops

* update graphsage example to combine uva sampling and UnifiedTensor

* update graphsage example to combine uva sampling and UnifiedTensor

* update graphsage example to combine uva sampling and UnifiedTensor

* update doc

* update examples

* change unitgraph and heterograph's PinMemory to in-place

* update examples for multi-gpu uva sampling

* update doc

* fix linting

* fix cpu build

* fix is_pinned for DistGraph

* fix is_pinned for DistGraph

* update graphsage unsupervised example

* update doc for gpu sampling

* update some check for sampling device switching

* fix linting

* adapt for new dataloader

* fix linting

* fix

* fix some name issue

* adjust device check

* add unit test for uva sampling & fix some zero_copy bug

* fix linting

* update num_threads in graphsage examples

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
2022-02-09 14:06:17 +08:00
Quan (Andy) Gan 701b4fccc2 [Sampling] New sampling pipeline plus asynchronous prefetching (#3665)
* initial update

* more

* more

* multi-gpu example

* cluster gcn, finalize homogeneous

* more explanation

* fix

* bunch of fixes

* fix

* RGAT example and more fixes

* shadow-gnn sampler and some changes in unit test

* fix

* wth

* more fixes

* remove shadow+node/edge dataloader tests for possible ux changes

* lints

* add legacy dataloading import just in case

* fix

* update pylint for f-strings

* fix

* lint

* lint

* lint again

* cherry-picking commit fa9f494

* oops

* fix

* add sample_neighbors in dist_graph

* fix

* lint

* fix

* fix

* fix

* fix tutorial

* fix

* fix

* fix

* fix warning

* remove debug

* add get_foo_storage apis

* lint
2022-01-30 16:13:00 +08:00
Quan (Andy) Gan 90f10b31cb [Feature] Negative sampling (#3599)
* first commit

* a bunch of fixes

* add unique

* lint

* lint

* lint

* address comments

* Update negative_sampler.py

* fix

* description

* address comments and fix

* fix

* replace unique with replace

* test pylint

* Update negative_sampler.py
2022-01-07 18:26:14 +08:00
Rhett Ying e053df7965 [Feature] enable async transfer in NodeDataLoader for homograph (#3407)
* [Feature] enable async transfer in NodeDataLoader for homograph

* fix lint issues

* fix device choose when creating stream

* fix test on cpu only machine

* fix pin_memory config

* support homo only

* avoid creating stream in each step and sync via event

* fix lint

* enable graph copy on non-default stream

* fix lint

* refine arg description

* fix conflicts
2021-11-17 10:06:48 +08:00
Quan (Andy) Gan b8ce0f41a3 [Sampling] Cluster-GCN and ShaDow-GNN DataLoader (#3487)
* first commit

* next commit

* third commit

* add ShaDow-GNN sampler and unit tests

* fixes

* lint

* cr*p

* lint

* fix lint

* fixes and more unit tests

* more tests

* fix docs

* lint

* fix

* fix

* fix

* fixes

* fix doc
2021-11-16 13:16:25 +08:00
Xin Yao efe0b06162 fix test dataloader (#3482)
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2021-11-05 15:13:56 +08:00
Xin Yao b717c8bf0d [BugFix] Fix bugs in GPU sampling and enable unit tests for dataloaders on the GPU (#3474)
* enable unit tests for dataloader on the GPU

* fix compatibility

* copyright

* fix linting

Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com>
2021-11-04 10:34:56 -07:00
xiang song(charlie.song) ba7e7cf931 [New Feature] Per edge type sampler for to_homogeneous graphs. (#3131)
* fix.

* fix.

* fix.

* fix.

* Fix test

* Deprecate old DistEmbedding impl, use synchronized embedding impl

* Basic imple of heterogeneous on homogenenous sampling

* make pass

* Pass C++ test

* Add python test code

* lint

* lint

* Add MultiLayerEtypeNeighborSampler

* Add unitest for single machine dataloader

* Add dist dataloader test for edge type sampler

* Fix lint

* fix

* support for per etype sample

* Fix some bug and enable distributed training with per edge sample

* fix

* Now distributed training works

* turn off some mxnet

* turn off mxnet for some dist test

* fix

* upd

* upd according to the comments

* Fix

* Fix test and now distributed works.

* upd

* upd

* Fix

* Fix bug

* remove dead code.

* upd

* Fix

* upd

* Fix

Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal>
Co-authored-by: Da Zheng <zhengda1936@gmail.com>
2021-07-28 11:51:32 +08:00
Quan (Andy) Gan 2e19ba8025 add exclude self option for EdgeDataLoader (#3122)
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2021-07-13 13:30:16 +08:00
Justus Schock bfef789ea1 [Dataloading] Make loader iters iterator (#2886)
* Make loader items iterator

* Update test_dataloader.py

* Update __init__.py

* Update test_dataloader.py

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2021-05-07 10:15:49 +08:00
Quan (Andy) Gan d93a975997 revert (#2672) 2021-02-19 12:05:42 +08:00
Mufei Li 5da3439c16 [Feature] Add Device Flag in Data Loaders (#2450)
* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2021-01-15 07:39:26 +08:00
Quan (Andy) Gan bdb88df576 [Performance] Allow DataLoaders to return indices of the seed array (#2495)
* return indices from dataloader

* fixes

* fix

* fix distgraph and added some todos

* Update dataloader.py

* Update dataloader.py

Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
2021-01-08 14:30:52 +08:00
Tong He e2e524df41 [Feature] Add GraphDataLoader implementation (#2496)
* add graph dataloader

* add to doc

* fix

* fix

* fix docstring

* update according to torch default_collate

* add unittest

* fix

* fix lint

* fix
2021-01-07 13:38:21 +08:00
Quan (Andy) Gan ac5c79cdb8 [Bug] Fixes issues in feature pickling and transmission for subgraphs and blocks (#2139)
* bug fixes

* remove __deepcopy__; not sure what the behavior should be

* lint

* skip gpu test

* fix

* fix dist dataloader

* add comment

* remove assert
2020-09-03 12:48:59 +08:00
Mufei Li be444e52d9 [Doc/Feature] Refactor, doc update and behavior fix for graphs (#1983)
* Update graph

* Fix for dgl.graph

* from_scipy

* Replace canonical_etypes with relations

* from_networkx

* Update for hetero_from_relations

* Roll back the change of canonical_etypes to relations

* heterograph

* bipartite

* Update doc

* Fix lint

* Fix lint

* Fix test cases

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Update

* Fix test

* Fix

* Update

* Use DGLError

* Update

* Update

* Update

* Update

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Update

* Fix

* Update

* Fix

* Fix

* Fix

* Update

* Fix

* Update

* Fix

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix

* Fix

* Update

* Update

* Update

* Update

* Update

* Update

* rewrite sanity checks

* delete unnecessary checks

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix

* Update

* Update

* Update

* Fix

* Fix

* Fix

* Update

* Fix

* Update

* Fix

* Fix

* Update

* Fix

* Update

* Fix

Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
2020-08-18 04:26:29 +08:00
Quan (Andy) Gan f5eb80d221 [Feature] Edge DataLoader for edge classification & link prediction (#1828)
* clean commit

* oops forgot the most important files

* use einsum

* copy feature from frontier to block

* Revert "copy feature from frontier to block"

This reverts commit 5224ec963eb6a3ef1b6ab74d8ecbd44e4e42f285.

* temp fix

* unit test

* fix

* revert jtnn

* lint

* fix win64

* docstring fixes and doc indexing

* revert einsum in sparse bidecoder

* fix some examples

* lint

* fix due to some tediousness in remove_edges

* addresses comments

* fix

* more jtnn fixes

* fix
2020-08-11 18:00:58 +08:00
Quan (Andy) Gan 168a88e570 [Sampling] NodeDataLoader for node classification (#1635)
* neighbor sampler data loader first commit

* more commit

* nodedataloader

* fix

* update RGCN example

* update OGB

* fixes

* fix minibatch RGCN crashing with self loop

* reverting gatconv test code

* fix

* change to new solution that doesn't require tf dataloader

* fix

* lint

* fix

* fixes

* change doc

* fix docstring

* docstring fixes

* return seeds and input nodes from data loader

* fixes

* fix test

* fix windows build problem

* add pytorch wrapper

* fixes

* add pytorch wrapper

* add unit test

* add -1 support to sample_neighbors & fix docstrings

* docstring fix

* lint

* add minibatch rgcn evaluations

Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Tong He <hetong007@gmail.com>
2020-07-02 16:31:05 +08:00