50dd8517c8
Enable faster validation for pytorch graphsage example ( #3361 )
2021-11-05 03:11:50 +00:00
3d90edda3e
Fix torch import in example ( #3372 )
2021-11-04 06:10:53 +00:00
e3eb3ee511
Fix distributed device mapping problem. ( #3313 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-11-04 05:47:56 +00:00
3f6f694159
[Bug fix] Various fix from bug bash ( #3133 )
...
* Update
* Update
* Update dependencies
* Update
* Update
* Fix ogbn-products gat
* Update
* Update
* Reformat
* Fix typo in node2vec_random_walk
* Specify file encoding
* Working for 6.7
* Update
* Fix subgraph
* Fix doc for sample_neighbors_biased
* Fix hyperlink
* Add example for udf cross reducer
* Fix
* Add example for slice_batch
* Replace dgl.bipartite
* Fix GATConv
* Fix math rendering
* Fix doc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-156.us-west-2.compute.internal >
2021-07-15 09:17:15 +08:00
d7390763f0
[Distributed] Deprecate old DistEmbedding impl, use synchronized embedding impl ( #3111 )
...
* fix.
* fix.
* fix.
* fix.
* Fix test
* Deprecate old DistEmbedding impl, use synchronized embedding impl
* update doc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-07-14 00:15:53 +08:00
2f43cdb3c8
deal with situation where num_layers equals 1 ( #3066 )
2021-06-28 10:17:16 +08:00
427a5a96e0
[Doc] Update NodeDataLoader and EdgeDataLoader for GPU-based neighbor sampling ( #3046 )
...
* update docstrings and tidy code
* add docs
* address comments
* Update __init__.py
* address comments
2021-06-25 10:46:21 +08:00
aaec3d8a0b
[Distributed] Support hierarchical partitioning ( #3000 )
...
* add.
* fix.
* fix.
* fix.
* fix.
* add tests.
* support node split and edge split.
* support 1 partition.
* add tests.
* fix.
* fix test.
* use hierarchical partition.
* add check.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-57.us-west-2.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-06-16 16:58:23 +08:00
8b64ae59b8
[Performance] Perform to_block on the GPU when the dataloader is created with a GPU device. ( #3016 )
...
* add output device for dataloading
* Update dataloader
* Get sampler device from dataloader
* Fix line length
* Update examples
* Fix to_block GPU for empty relation types
* Handle the case where the DistGraph has None for the underlying graph
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-06-13 18:51:00 +08:00
9497a9be85
[Bugfix][Examples] Fix graphsage multigpu training example training set size ( #3002 )
...
* Make multigpu graphsage use whole datset
* Specify queeze dimension
* Remove squeeze dimension
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com >
2021-06-10 16:35:08 -07:00
18dbaebebc
[Distributed] Specify the graph format for distributed training ( #2948 )
...
* explicitly set the graph format.
* fix.
* fix.
* fix launch script.
* fix readme.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal >
2021-05-26 18:09:30 +08:00
7c7113f660
add use_ddp to dataloaders ( #2911 )
2021-05-17 15:46:02 +08:00
caa6d6072c
[Feature] Replacing thread_wrapped_func with minimal mp.Process wrapper ( #2905 )
...
* standardizing thread_wrapped_func
* lints
* Update __init__.py
2021-05-14 11:34:49 +08:00
70695ff8f8
[Model] Training GraphSAGE with PyTorch Lightning ( #2878 )
...
* pytorch lightning initial examples
* revert most changes in dataloader to favor #2886 .
* address comments
2021-05-11 11:28:13 +08:00
975eb8fc5d
[Distributed] Distributed node embedding and sparse optimizer ( #2733 )
...
* Draft for sparse emb
* add some notes
* Fix
* Add sparse optim for dist pytorch
* Update test
* Fix
* upd
* upd
* Fix
* Fix
* Fix bug
* add transductive exmpale
* Fix example
* Some fix
* Upd
* Fix lint
* lint
* lint
* lint
* upd
* Fix lint
* lint
* upd
* remove dead import
* update
* lint
* update unitest
* update example
* Add adam optimizer
* Add unitest and update data
* upd
* upd
* upd
* Fix docstring and fix some bug in example code
* Update rgcn readme
Co-authored-by: Ubuntu <ubuntu@ip-172-31-57-25.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal >
2021-05-03 08:37:40 +08:00
e70138bbe4
[Performance][GPU] Enable GPU uniform edge sampling ( #2716 )
...
* Start on uniform GPU sampling
* Save more work
* Get cu file compiling
* Update sampling
* More changes
* Get GPU sampling for uniform probabilities solved
* Fix batch tensor migration
* Fix
* update kernels
* expand blocking
* Undo testing change
* Cut down on sampling overhead
* Fix replacement
* Update unit tests
* Add option to gpu sample in graphsage
* Copy only csc to gpu
* Add ogbn support
* Fix linting
* Remove nvtx from sample
* Improve documentation and error checking
* Expand documentation
* Update assert checking
* delete extra space
* Use standard dataloader when dataset is a dictionary
* ogb -> ogbn
* Fix edge selection determinism
* Fix typos
* Remove nvtx
* Add comment for self.fanout_arrays and assert
* Fix linting
* Migrate to scalarbatcher
* Fix indentation
* Fix batcher
* Fix indexing
* Only use databatcher for GPU
* Convert to DGL NDArray to PyTorch Tensor
* Add optimization for PyTorch's F.tensor() for list of GPU tensors
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
2021-04-14 18:55:09 -07:00
bfbbefa793
[Distributed] Fix a bug in multiprocessing sampling. ( #2826 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-73-81.ec2.internal >
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
2021-04-08 01:29:33 -07:00
e36c5db614
[Distributed] Simplify distributed API ( #2775 )
...
* remove num_workers.
* remove num_workers.
* remove num_workers.
* remove num-servers.
* update error message.
* update docstring.
* fix docs.
* fix tests.
* fix test.
* fix.
* print messages in test.
* fix.
* fix test.
* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
2021-03-29 19:19:23 -07:00
e6f6c2eb31
[Bugfix] Update deprecated method name in load_graph.py ( #2769 )
...
Method `dataset.num_labels` has been deprecated and replaced by `dataset.num_classes`.
Updating the method name to avoid runtime warning.
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com >
2021-03-22 20:07:32 +08:00
bb54206620
remove pyinstrument. ( #2772 )
...
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2021-03-21 20:27:12 -07:00
28b0a76618
[Example] Provide instructions of setting up passwordless SSH login. ( #2707 )
2021-02-27 18:34:46 -08:00
6568d0aaf6
[Example] Recommend NFS over copy files ( #2689 )
...
* Update README to use NFS, rather than copying files
* Update README.md
* Update README.md
* Update README.md
* Add ParMetis
* Update README.md
* fix directory.
* fix
* redo.
* fix.
* update.
* add more descriptions
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Quan Gan <coin2028@hotmail.com >
2021-02-25 20:20:52 +08:00
ff64bd0de8
add warning for a bug in README ( #2665 )
2021-02-16 00:16:40 -08:00
25ac334403
[Distributed] Heterogeneous graph support ( #2457 )
...
* Distributed heterograph (#3 )
* heterogeneous graph partition.
* fix graph partition book for heterograph.
* load heterograph partitions.
* update DistGraphServer to support heterograph.
* make DistGraph runnable for heterograph.
* partition a graph and store parts with homogeneous graph structure.
* update DistGraph server&client to use homogeneous graph.
* shuffle node Ids based on node types.
* load mag in heterograph.
* fix per-node-type mapping.
* balance node types.
* fix for homogeneous graph
* store etype for now.
* fix data name.
* fix a bug in example.
* add profiler in rgcn.
* heterogeneous RGCN.
* map homogeneous node ids to hetero node ids.
* fix graph partition book.
* fix DistGraph.
* shuffle eids.
* verify eids and their mappings when loading a partition.
* Id map from homogneous Ids to per-type Ids.
* verify partitioned results.
* add test for distributed sampler.
* add mapping from per-type Ids to homogeneous Ids.
* update example.
* fix DistGraph.
* Revert "add profiler in rgcn."
This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.
* add tests for homogeneous graphs.
* fix a bug.
* fix test.
* fix for one partition.
* fix for standalone training and evaluation.
* small fix.
* fix two bugs.
* initialize projection matrix.
* small fix on RGCN.
* Fix rgcn performance (#17 )
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix lint.
* fix.
* fix test.
* fix lint.
* test partitions.
* remove redundant test for partitioning.
* remove commented code.
* fix partition.
* fix tests.
* fix RGCN.
* fix test.
* fix test.
* fix test.
* fix.
* fix a bug.
* update dmlc-core.
* fix.
* fix rgcn.
* update readme.
* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
* fix.
* fix.
* add div_int.
* fix.
* fix.
* fix lint.
* fix.
* fix.
* fix.
* adjust.
* move code.
* handle heterograph.
* return pytorch tensor in GPB.
* remove some tests in example.
* add to_block for distributed training.
* use distributed to_block.
* remove unnecessary function in DistGraph.
* remove distributed to_block.
* use pytorch tensor.
* fix a bug in ntypes and etypes.
* enable norm.
* make the data loader compatible with the old format.
* fix.
* add comments.
* fix a bug.
* add test for heterograph.
* support partition without reshuffle.
* add test.
* support partition without reshuffle.
* fix.
* add test.
* fix bugs.
* fix lint.
* fix dataset.
* fix for mxnet.
* update docstring.
* rename to floor_div
* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.
* fix docstring.
* fix error.
* fixes.
* fix comments.
* rename.
* rename.
* explain IdMap.
* fix docstring.
* fix docstring.
* update docstring.
* remove the code of returning heterograph.
* remove argument.
* fix example.
* make GraphPartitionBook an abstract class.
* fix.
* fix.
* fix a bug.
* fix a bug in example
* fix a bug
* reverse heterograph sampling.
* temp fix.
* fix lint.
* Revert "temp fix."
This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.
* compute norm.
* Revert "reverse heterograph sampling."
This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.
* fix.
* move id_map.py
* remove check
* add more comments.
* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal >
2021-01-24 19:18:00 -08:00
927d2b319b
[Example] Move Data to GPU before Minibatch Training ( #2453 )
...
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
* Update
2020-12-28 21:26:32 +08:00
88a042d34a
fix reddit self loop ( #2330 )
2020-11-09 14:40:21 +08:00
d06c540336
[Example] fix print info ( #2305 )
2020-10-29 09:20:27 +08:00
39260242a2
[Distributed Example] Check args.num_servers with DGL_NUM_SERVER ( #2203 )
...
* update
* update
* update
2020-09-16 14:31:47 +08:00
c53bf77452
[Distributed Example] Add --close_profiler option to train_dist.py ( #2199 )
...
* update
* update
2020-09-15 14:37:28 +08:00
0ced22c237
fix ( #2142 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-10-127.us-west-2.compute.internal >
Co-authored-by: Chao Ma <mctt90@gmail.com >
2020-09-15 14:19:16 +08:00
1b2658b926
update ( #2192 )
2020-09-14 15:35:39 +08:00
1aa25cb618
update ( #2191 )
2020-09-14 15:13:39 +08:00
55f3e92631
[Fix] use DistDataLoader instead of Pytorch’s DataLoader. ( #2182 )
...
Co-authored-by: hzliuchw <hzliuchw@corp.netease.com >
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com >
2020-09-12 20:08:39 +08:00
7e7376eb4a
[Hotfix] Fix unsupervised graphsage ( #2143 )
...
* hotfix
* Hotfix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-87-240.ec2.internal >
2020-09-03 11:11:15 +08:00
e8273a8510
update README.md ( #2138 )
2020-09-02 15:42:28 +08:00
1e3fcc7c53
update comments ( #2132 )
...
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal >
2020-08-31 13:23:43 +08:00
9eaace9216
rename create_format_ to create_formats_ ( #2126 )
2020-08-29 22:49:27 +08:00
4327d7125d
[Distributed] Check num_workers and num_samplers ( #2108 )
...
* check num_workers
* update
* update
* update
* update
* update
* update
2020-08-27 12:42:25 +08:00
283b2cfb96
[Distributed] Fix issues of GraphSage to support running on GPU ( #2084 )
...
* fix issues on GPU
* update
* update
* update
* update
* update
* update
* update
* update
* update
Co-authored-by: Ma <manchao@38f9d3587685.ant.amazon.com >
2020-08-25 13:20:46 +08:00
edfbee2ca8
[Feature] Make sample_neighbors copy features on demand ( #2042 )
...
* fix
* fix
* lint
* fix
* test
* fix
* fix
2020-08-18 22:25:19 +08:00
eb9c067b08
[Distributed] Copy training scripts in copy_partitions.py ( #2010 )
...
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
2020-08-13 12:20:39 +08:00
98c1117d00
update ( #2007 )
2020-08-12 16:07:15 +08:00
f5d8fa8476
[Distributed] Fix all arguments to the format of xx_xxx ( #2005 )
...
* update
* update
2020-08-12 14:45:30 +08:00
d1cf5c383e
[Distributed] adjust various APIs. ( #1993 )
...
* rename get_data_size.
* remove g from DistTensor.
* remove g from DistEmbedding.
* clean up API of graph partition book.
* fix DistGraph
* fix lint.
* collect all part policies.
* fix.
* fix.
* support distributed sampler.
* remove partition.py
2020-08-11 20:22:00 -07:00
4efa332044
[Distributed] Move server start code to initialize. ( #2002 )
...
* move server start code to initialize.
* fix.
* fix lint.
* fix examples.
* add more checks.
2020-08-11 16:58:18 -07:00
f5eb80d221
[Feature] Edge DataLoader for edge classification & link prediction ( #1828 )
...
* clean commit
* oops forgot the most important files
* use einsum
* copy feature from frontier to block
* Revert "copy feature from frontier to block"
This reverts commit 5224ec963eb6a3ef1b6ab74d8ecbd44e4e42f285.
* temp fix
* unit test
* fix
* revert jtnn
* lint
* fix win64
* docstring fixes and doc indexing
* revert einsum in sparse bidecoder
* fix some examples
* lint
* fix due to some tediousness in remove_edges
* addresses comments
* fix
* more jtnn fixes
* fix
2020-08-11 18:00:58 +08:00
d340ea3aa8
[Distributed] Remove server_count from ip_config.txt ( #1985 )
...
* remove server_count from ip_config.txt
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* lint
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* Update dist_context.py
* fix lint.
* make it work for multiple spaces.
* update ip_config.txt.
* fix examples.
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* udpate
* update
* update
* update
* update
* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-11 17:41:48 +08:00
a089b91fdf
Remove stale partition code and fix #1949 ( #1984 )
...
* remove unused code
* fix litn
* fix
2020-08-11 10:51:07 +08:00
ee30b2aadd
[Distributed] Fix standalone ( #1974 )
...
* fix tests.
* fix.
* remove a test.
* make code work in the standalone mode.
* fix example.
* more fix.
* make DistDataloader work with num_workers=0
* fix DistDataloader tests.
* fix.
* fix lint.
* fix cleanup.
* fix test
* remove unnecessary code.
* remove tests.
* fix.
* fix.
* fix.
* fix example
* fix.
* fix.
* fix launch script.
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com >
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-10 15:49:28 -07:00
4b8eaf205b
[Distributed] move feature copy to samplers in distributed GraphSage. ( #1975 )
...
* fix example.
* move feature copy to sampler.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal >
2020-08-08 09:19:09 -07:00
nv-dlasalle