文件历史

88 次代码提交

作者 SHA1 备注 提交日期
Quan (Andy) Gan 3b0c0cec46 enable sparse on windows and mac (#5277)
* enable sparse on windows and mac

* that was stupid

* let's see what's going on..

* [Sparse] Fix the import error on Mac OS.

When using template functions that are defined in source files from DGL,
the loader of MacOS somehow cannot find their definitions. This fix simply
avoids depending on template functions from DGL headers.

With this fix, the sparse tests all pass on the MAC environment.

* ok this is the problem

* make errors clearer

* uh

* test

* Update __init__.py

* disabling ddp on windows

---------

Co-authored-by: czkkkkkk <zekucai@gmail.com>
2023-02-15 16:08:20 +08:00
Rhett Ying 0c5d6f1b00 [BUILD] enable dgl sparse build in default (#5175)
* enable dgl sparse build in default

* disable sparse build on cugraph

* update
2023-01-16 14:01:39 +08:00
peizhou001 56ffb650fe [API Deprecation]Deprecate contrib module (#5114) 2023-01-06 20:16:01 +08:00
Minjie Wang 025e43210f [Sparse] Add tests for sparse examples; Migrate from mock_sparse to mock_sparse2 (#5073)
* pass gcn

* example tests

* pass gcnii, hgnn, sgc, sign

* black

* test file

* add torchmetrics

* migrate appnp, c_and_s, twirls

* fix style

* minor fix

* fix hypergraphatt and twirls

* pass gat

* fix torchmetric

* lintrunner

Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2022-12-29 08:55:25 +08:00
Rhett Ying 9731e02325 [Dist] enable to chunk node/edge data into arbitrary number of chunks (#4930)
* [Dist] enable to chunk node/edge data into arbitrary number of chunks

* [Dist] enable to split node/edge data into arbitrary parts

* refine code

* Format boolean to uint8 forcely to avoid dist.scatter failure

* convert boolean to int8 before scatter and revert it after scatter

* refine code

* fix test

* refine code

* move test utilities into utils.py

* update comment

* fix empty data

* update

* update

* fix empty data issue

* release unnecessary mem

* release unnecessary mem

* release unnecessary mem

* release unnecessary mem

* release unnecessary mem

* remove unnecessary shuffle data

* separate array_split into standalone utility

* add example

Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
2022-12-15 10:01:39 +08:00
czkkkkkk 7c059e86c7 [Sparse] Add sparse matrix C++ implementation (#4773)
* [Sparse] Add sparse matrix C++ implementation

* Add documentation

* Update

* Minor fix

* Move Python code to dgl/mock_sparse2

* Move headers to include

* lint

* Update

* Add dgl_sparse directory

* Move src code to dgl_sparse

* Add __init__.py in tests to avoid naming conflict

* Add dgl sparse so in Jenkinsfile

* Complete docstring & SparseMatrix basic op

* lint

* Disable win tests
2022-11-07 18:01:19 +08:00
Xin Yao 96297fb8fd [Feature] Add bfloat16 (bf16) support (#4648)
* add bf16 specializations

* remove SWITCH_BITS

* enable amp for bf16

* remove SWITCH_BITS for cpu kernels

* enbale bf16 based on CUDART

* fix compiling for sm<80

* fix cpu build

* enable unit tests

* update doc

* disable test for CUDA < 11.0

* address comments

* address comments
2022-11-06 19:48:36 +08:00
Hongzhi (Steve), Chen ea48ce7a37 [Misc] Black auto fix. (#4697)
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
2022-10-11 13:31:13 +08:00
Rhett Ying be11409312 [CI] Enable DGL cugaph nightly CI (#4573)
* [CI] Enable DGL cugaph nightly CI

* fix build issue

* remove runtime
2022-09-22 15:26:29 +08:00
Xin Yao 880b3b1f97 [Fix] Enable lint check for cuh files and fix compiler warnings (#4585)
* disable warning for tensorpipe

* fix warning

* enable lint check for cuh files

* resolve comments
2022-09-21 10:09:14 +08:00
Rhett Ying 7f984b64b2 Revert "[Feature][REVIEW] Enable DGL cugaph nightly CI (#4525)" (#4563)
This reverts commit ec171c648a.
2022-09-16 13:57:50 +08:00
Vibhu Jawa ec171c648a [Feature][REVIEW] Enable DGL cugaph nightly CI (#4525)
* Added cugraph nightly scripts

* Removed nvcr.io//nvidia/pytorch:22.04-py3 reference

Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2022-09-16 11:28:30 +08:00
Rhett Ying 86656a6b34 [CI] use https instead of http (#4488) 2022-08-31 14:59:40 +08:00
Mufei Li 2e8ae9f980 [Dist][CI] Unit test for the new distributed partitioning pipeline (#4394)
* chunked graph data format

* Update

* Update

* Update task_distributed_test.sh

* Update

* Update

* Revert "Update"

This reverts commit 03c461870f19375fb03125b061fc853ab555577f.

* Update

* Update

* ssh-keygen

* CI

* install openssh

* openssh

* Update

* CI

* Update

* Update

Co-authored-by: Ubuntu <ubuntu@ip-172-31-53-142.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-87.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-20-21.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-26.ap-northeast-1.compute.internal>
2022-08-19 14:20:56 +08:00
Rhett Ying 8d3c58202a [CI] upgrade python version to 3.7.0 (#4406)
* [CI] upgrade python version to 3.7.0

* do not upgrade for mxnet cpu due to seg fault

* fix test failure for mxnet
2022-08-17 08:02:44 +00:00
Rhett Ying b3242e90b8 [CI] separate distributed tests from torch cpu tests (#4313)
* [CI] separate distributed tests from torch cpu tests

* remove TF related env
2022-07-30 16:32:35 +08:00
Rhett Ying 17f1432ab2 [DistTest] fix incorrect shell if statement (#4304)
* [DistTest] fix incorrect shell if statement

* fix incorrect use of dist.initialize()
2022-07-28 20:14:47 +08:00
Xin Yao 79b0a50afc [Unittest][Fix] Several unit tests fixes for Ampere+ and PyTorch 1.12+ (#4213)
* Fix test_csrmm for tensor core

* unset allow tf32 flag

* update test unified tensor

* skip fp16 for CPU
2022-07-14 19:59:57 +08:00
Chang Liu 885be1784b [Example][Refactor] Refactor GCN example (#4160)
* Refactor GCN example

* Refactor GCN based on graphsage

* Readme update

* Minor update

* update

* Remove user-defined GCN implementation

* README update

* Update

* Update CONTRIBUTORS.md

* update task_example_test

Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-07-05 15:58:31 +08:00
Rhett Ying 85f281170f [CI] Add new CI stage for testing cugraph (#4171)
* [CI] add new stage specific forcuda related features based on nvidia+pytorch

* build and test for gpu_nv

* fix build failure

* fix unit tests

* make -j

* install cython beforehand

* copy cython lib

* test cugraph tests only

* fix typo

* separate test script for cugraph

* refactor build dgl shell
2022-07-05 10:57:07 +08:00
Minjie Wang f7dae4533a [CI] Reduce CI workload (#4196)
* try optimize CI

* fix go test; adjust timing report

* disable certain tests for mx/tf backends

* fix ut

* add pydantic
2022-06-30 20:01:16 +08:00
Serge Panev 652f4c0743 [Dist] Add env var for non-default SSH configs in tests (#4098)
Signed-off-by: Serge Panev <spanev@nvidia.com>

Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2022-06-15 12:38:50 +08:00
Rhett Ying 9501ed6a07 [Dist] master port should be fixed for all trainers (#4108)
* [Dist] master port should be fixed for all trainers

* add tests for tools/launch.py
2022-06-14 11:22:22 +08:00
Rhett Ying c1ff4c9b41 [DistTest] add basic pipeline for dist test across machines (#3984)
* [DistTest] add basic pipeline for dist test across machines

* move launch remote cmd to separate file

* add test for rpc

* fix function naming rule

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-06-08 12:37:02 +08:00
nv-dlasalle 7a065a9c56 [Build][Tests] Enable FP16 for GPU builds in CI (#4030)
* Enable FP16 for GPU builds in CI

* Limit default GPU archs to pascal and above

* Disable FP16 dispatching for cuda architectures less than 60

* Fix linting

* Fix typos
2022-05-26 09:48:28 -07:00
Rhett Ying 65e6b04d8b [Build] enable USE_AVX in CI and regression tests (#4004) 2022-05-12 21:29:11 -07:00
Vikram Sharma cb39eae1c4 Supporting DGL build with arch specific optimization (#3996)
Based on the pull request: https://github.com/dmlc/dgl/pull/3983
2022-05-11 12:38:42 -07:00
Rhett Ying 22d7f92407 [CI] print commit/build ID into comment (#3989) 2022-05-10 16:04:21 +08:00
Mufei Li 334e6434d2 [DGL-Go] CI for DGL-Go (#3959)
* Update

* Update

* Fix

* Update

* CI

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update
2022-05-06 12:26:50 +08:00
Rhett Ying 3f792071f6 [BugFix] fix CI status (#3963) 2022-04-29 14:47:19 +08:00
Rhett Ying 56614f320f [BugFix] fix job status in master CI (#3962)
* [BugFix] fix job status in master CI

* finalize
2022-04-29 12:54:13 +08:00
Daniil Sizov 5561af4517 [Feature] Dataloader worker affinitization (#3723)
* PR3355 + CSR conversion workaround

* Remove debug code

* Fix convention errors

* Remove wrongly added code section during merge

* Update to reflect dataloading changes

* Fix missing changes

* Remove comment

* Fix linter errors

* Fix trailing whitespace

* Add wrapper around worker init function

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2022-04-28 13:00:04 +08:00
Rhett Ying 2eaa58e047 [CI] refine status check (#3958) 2022-04-27 15:28:26 +08:00
Jinjing Zhou 338db32d21 [Feature] Try upload report to s3 (#3891)
refactor CI report and log
2022-04-22 16:02:12 +08:00
Jinjing Zhou ab50eb9b4e Add unbuffered to distributed test (#3732)
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2022-02-15 10:23:45 +08:00
Jinjing Zhou dc78e11c3f [Dataset] Adapter to convert a dataset for link prediction task (#3699)
* add ut

* add doc link

* install dep

* add

* Revert "install dep"

This reverts commit e574a8377144749056c6849b655004df2771e179.

* add

* merge fix

* rm files

* fix

* fix

* fix

* fix

* fix typo

* fix tf

* fix

* fix

* fix

* fix

* fix

* fix dependency

* fix test

* fix

* fix

* add doc

* fix

* fix

* fix test

* fix test

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-02-11 16:59:14 +08:00
Minjie Wang 5152a87982 [Data] Utility function and class for converting a dataset for node prediction (#3695)
* add ut

* add doc link

* install dep

* fix  ci

* fix ut; more comments

* remove deprecated attributes in rdf datasets; fix label feature name

* address comments

* fix ut for other frameworks
2022-01-30 14:31:37 +08:00
Rhett Ying e4cb4a3779 [Fix] reduce error msg, refine fetch logic of available ports (#3658)
* [Fix] reduce error msg, refine fetch logic of available ports

* un-initialize client before sending shutdown request

* fix import error

* print connect failure log only in debug mode

* enable DMLC_LOG_DEBUG=1 in CI
2022-01-19 09:41:58 +08:00
Rhett Ying 95c0ff6302 [Feature] add CSVDataset to load data from csv files (#3547)
* [Feature] add CSVDataset to load data from csv files

* add CSVDataset class file

* install pyyaml when running unit tests

* install pandas for unit tests

* utilize pydantic for YAML config check

* generate yaml and csv files during test

* make more keys as optional

* remove/rename several keys in yaml config and more tets though looks a bit clumsy

* fix test failure on mxnet

* pass /path/to/dataset instead of yaml path

* code refinement

* code refine

* change several yaml field such as feat and graph_id

* merge graph generation logic

* refine code

* Refactored_first_version

* DGLCSVDataset works for single heterograph

* add more tests

* fix test failure in mxnet

* add docstring

* use list comprehension for dict

* fix version in YAML

* refine data length assert

* use dict.pop directly

* remove ambiguous variable names

* refine graph id missing logic

* refine graph create call

* separate node/edge/graph data parser

* remove separator in DefaultDataParser

* refine validation error log for yaml field

* minor check

* refine code via dict.get()

* move load_from_csv into Node/Edge/GraphData

* move _parse_node/edge/graph_data into Node/Edge/GraphData

* refine id-related fields check

* check duplicate ntypes/etypes when load yaml

* refine docstring
2022-01-10 17:21:57 +08:00
Jinjing Zhou 0b3a6216f5 [Test] Enable kvstore test (#3079)
* try enable kvstore test

* fix

* fix

* seperate out kvstore test

* add comment
2021-07-02 14:54:38 +08:00
Jinjing Zhou 9664cdffd3 [Build] Make nccl optional (#3056)
* fix

* remove nvidiasmi

* fix

* fix docs

* fix

* fix

* 1

* fix

* remove

* skip deprecated kernel

* fix

* Revert "skip deprecated kernel"

This reverts commit c5ceb7f60dbbaf065b81cc3680757fd611d90ad3.

* fix
2021-06-27 22:36:01 +08:00
Mufei Li ff519f98c3 [API] Standardize Subgraph APIs (#2929)
* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix

* Update

* Fix subgraph tests

* Capture stdout for distributed test

* Capture stdout for distributed test

* Update

* Update

* Update

* Update subgraph.cc

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
2021-06-21 19:53:37 +08:00
Da Zheng b1628f2398 [Tutorial] Distributed node classification. (#2969)
* add init version.

* fix build.

* fix format.

* fix.

* fix.

* fix format.

* update README.

* avoid running CI on distributed training tutorials.

* Update tutorials/dist/1_node_classification.py

Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

* fix.

Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
2021-06-04 21:50:16 +08:00
Jinjing Zhou 6a56562a7c [CI] Use k8s cluster (#2957)
* add

* fix

* set default

* fix

* try master

* try fix

* try

* fix

* 111

* fix

* fix

* update

* ccc

* try

* fix

* fix

* try new machine

* fix

* fix

* fix

* Revert "fix"

This reverts commit e716d66b046f92fe7ae368947a51a036a7a3188a.

* try

* more parrallel

* use k8s for all

* fix name

* try not specify instance type

* ci

* use one yaml

* Revert "use one yaml"

This reverts commit 717d8d852be39fbf2e2e45f9f224aa97907c372c.

* add timeout

* fix permission

* mount efs

* print

* fix pvc

* fix

* restrict num of gpu instances

* check

* fix

* fix
2021-06-04 18:31:11 +08:00
yifeim c018436588 [Example] add latent dirichlet allocation (#2883)
* add lda model

* tweak latent dirichlet allocation

* Update README.md

* Update README.md

* update example index

* update header

* minor tweak

* add example test

* update doc

* Update README.md

* Update README.md

* add partial_fit for free

* Update examples/pytorch/lda/lda_model.py

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update examples/pytorch/lda/example_20newsgroups.py

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update lda_model.py

* bugfix torch Gamma uses rate parameter

Co-authored-by: Yifei Ma <yifeim@amazon.com>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2021-05-17 20:25:36 +08:00
Minjie Wang 8a07ab7737 [Doc] Tutorials re-organization (#2683)
* reorg

* change titles

* rm some stale API doc; minor fix

* fix docs

* add warning

* rm new-tutorial run in ci

* lint
2021-02-20 17:42:45 +08:00
Quan (Andy) Gan 117dd25204 [Doc] Fix inconsistencies and GPU code (#2642)
* fix inconsistencies and GPu

* bug fixes

* fix

* trigger new tutorials
2021-02-08 16:33:39 +08:00
Quan (Andy) Gan 9a7235faf2 [Performance] Use allocator from PyTorch if possible (#2328)
* first commit

* some thoughts

* move around

* more commit

* more fixes

* now it uses torch allocator

* fix symbol export error

* fix

* fixes

* test fix

* add script

* building separate library per version

* fix for vs2019

* more fixes

* fix on windows build

* update jenkinsfile

* auto copy built dlls for windows

* lint and installation guide update

* fix

* specify conda environment

* set environment for ci

* fix

* fix

* fix

* fix again

* revert

* fix cmake

* fix

* switch to using python interpreter path

* remove scripts

* debug

* oops sorry

* Update index.rst

* Update index.rst

* copies automatically, no need for this

* do not print message if library not found

* tiny fixes

* debug on nightly

* replace add_compile_definitions to make CMake 3.5 happy

* fix linking to wrong lib for multiple pytorch envs

* changed building strategy

* fix nightly

* fix windows

* fix windows again

* setup bugfix

* address comments

* change README
2020-12-25 13:57:51 +08:00
xiang song(charlie.song) 09ec6020ac [Example] rm gcn spmv (#2044)
* rm gcn spmv

* remove useless test

* remove useless test

Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal>
2020-08-18 17:48:38 +08:00
Minjie Wang 444becf00a [Misc] Move many deprecated codes to the deprecate folder (#1893)
* some file movements

* move some codes to deprecated

* more deprecation

* lint

* remove useless test
2020-07-30 18:17:33 +08:00