文件历史

提交图

951 次代码提交

作者 SHA1 备注 提交日期
Daniil Sizov 87fb7ed05b [Config] Enable libxsmm by default for AVX cpu (#5165)
* Enable AVX by default

* Fix linting errors

* Fix win64 build (libxsmm not linked)

Libxsmm on Win64 is not linked, should be disabled by default

* Fix clang format issues

* Change lower supported cpu version to LIBXSMM_X86_AVX2

Change lower supported cpu version to LIBXSMM_X86_AVX2 to address https://github.com/dmlc/dgl/issues/3459 issue

* Fix unit test

Remove assumption that libxsmm is enabled in the config by default (only true for intel CPUs with AVX2 instructions)

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-15-137.us-west-2.compute.internal>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2023-03-16 02:05:22 +08:00
czkkkkkk 92e383c3b8 [Sparse] Remove the Python DiagMatrix class (#5444)
* [Sparse] Remove the Python DiagMatrix class

* Update

* Update

* update

* Update
2023-03-15 18:17:52 +08:00
kylasa 92e2299593 [DistDGL][Robustness]Replacing numpy's unique with custom implementation (#5391)
* Replacing numpy's unique with custom implementation

* Added docstring to the new function.

* Adding unit tests

* Numpy's version issues with the 'kind' argument.

* Addressing CI Test Failure.

* Addressing CI review comments.

* revised implementation, optimized for time.

* added missing arguments for fallback case.

* Addressing CI test failures.

* Resolving issues with PYTHONPATH

* Fix CI Test Failure issues.

* fix CI test failures.

---------

Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2023-03-10 13:38:27 -08:00
kylasa e74b3d3dd2 [DistDGL][TestCoverage]Added testcase for testing distributed lookup service. (#5365)
* Added testcase for testing distributed lookup service.

* Applying lintrunner patch.

* Fixing CI Test environment failures.

* lintrunner patch.

* lintrunner patch

* Fix CI Failure.

* Fixing CI Test failure cases.

* lintrunner patch.

* lintrunner patch and CI test failure.

* Restore no. of test cases.

* Resolving pythonpath issues.

* lintrunner patch.

* updating PYTHONPATH to resolve lib path

* Resolve merge conflicts

* Resolving issues with PYTHONPATH env variable.

* fix module path

* rename utils script under test to avoid ambiguity

* remove unnecessary pythonpath

* fix lint error

* fix lint error

---------

Co-authored-by: RhettYing <rhett_ying@qq.com>
2023-03-10 12:19:41 +08:00
Gleb Bazhenov 54b4bd0acd [Data] new functional for creating data splits in graph (#5418)
* new functional for creating data splits in graph

* minor fix in data split implementation

* apply suggestions from code review

Co-authored-by: Mufei Li <mufeili1996@gmail.com>

* refactoring + unit tests

* fix test file name

* move imports to the top

* Revert "fix test file name"

This reverts commit 126323e38c8de6fdc73c826dccc9048aed8f9634.

* remove nccl submodule

* address linter issues

---------

Co-authored-by: Mufei Li <mufeili1996@gmail.com>
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
2023-03-09 18:55:25 +08:00
Hongzhi (Steve), Chen e250ba3d99 [Misc] Rename test_utils -> pytests_utils. (#5438)
* rename

* sort

* rename

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-03-09 09:42:29 +08:00
kkranen fe3d29ac4d [Dataloading] Ignore edge types without reverse types in edge dataloader (#5411)
* Bugfix for reverse edge issue.

* Resolved copy/paste transcription error

* lint

* Added unit tests for graph with not all reverses.

* linting + added backend device test

---------

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2023-03-08 22:04:51 +08:00
Xin Yao ae4a5b7394 [Feature] Add state_dict, load_state_dict, param_groups to dgl.optim.SparseGradOptimizer (#5311)
* init update

* all get/set optm_state

* add unit tests

* add docstring

* fix for multiple embeddings

* move embedding methods to private

* fix lint

* fix unit tests

* resolve comments

* merge master
2023-03-08 16:46:34 +08:00
Guangyu Zhou 178290249e [Dataset] Add PATTERN dataset (#5422)
* add PATTERN dataset

* fix bug

* fix bugs

* fix issues

* refine according to dongyu's comments

---------

Co-authored-by: BuptTab <gyzhou2000@gmail.com>
Co-authored-by: rudongyu <ru_dongyu@outlook.com>
2023-03-08 14:08:12 +08:00
Xin Yao 8d5d8962ad [Refactor] Replace third_party/nccl with PyTorch's NCCL backend (#4989)
* expose GeneratePermutation

* add sparse_all_to_all_push

* add sparse_all_to_all_pull

* add unit test

* handle world_size=1

* remove python nccl wrapper

* remove the nccl dependency

* use pinned memory to speedup D2H copy

* fix lint

* resolve comments

* fix lint

* fix ut

* resolve comments
2023-03-08 12:59:10 +08:00
kylasa 7b766393f8 [DistDGL][UserEx]Sync parmetis_wrapper with changes in metadata.json (#5385)
* Sync parmetis_wrapper with changes in metadata.json

1. In the preprocess.py, make sure that num_partitions is defined as input argument. Also, align 'input_dir' with the input dataset. schema_file is assumed to be located inside the input_dir. Also, graph_stats.txt file is assumed to be present in the input_dir.

2. Use DGL_HOME environment variable so that parmetis_wrapper command can be run anywhere.

* Fix CI test failure cases.

* Addressing CI review comments.

* Addressing CI test failures.

* Applying lintrunner patch
2023-03-06 15:50:43 -08:00
kylasa 894ad1e3ea Support for no. of chunks smaller than no. of partitions. (#5390)
* Support for no. of chunks smaller than no. of partitions and Adding appropriate test cases.

Following changes are made with this PR.
1. Code changes for handling no. of chunks smaller than no. of partitions
2. Adding new test cases, which were previously deleted, for no. of chunks smaller than no. of partitions.
3. Also adding test cases, where multiple partitions are handled by a single process.

* Committing the missing files in this commit.

* lintrunner patch.

* lintrunner check

* lintrunner patch here.

* CI review comments.
2023-03-06 14:10:51 -08:00
Quan (Andy) Gan 851d66fa65 [Bugfix] Fix duplicate worker_init_fn argument when provided in DataLoader (#5420)
* fix duplicate worker_init_fn

* lint

* lint again

* uugh
2023-03-06 23:41:53 +08:00
Tingyu Wang bfd411d0d5 [Model] Add dgl.nn.CuGraphGATConv model (#5168)
* add CuGraphGATConv model

* lintrunner

* update model to reflect changes in make_mfg_csr(), move max_in_degree to forward()

* simplify pytest markers

* fall back to FG option for large fanout

* update error msg

* add feat_drop and activation options

* add residual option

* Update python/dgl/nn/pytorch/conv/cugraph_gatconv.py

Co-authored-by: Mufei Li <mufeili1996@gmail.com>

* Update python/dgl/nn/pytorch/conv/cugraph_gatconv.py

Co-authored-by: Mufei Li <mufeili1996@gmail.com>

* reset res_fc

---------

Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2023-03-04 14:10:06 +08:00
Mufei Li f00cd6efbd [Utils] Edge and LINKX homophily measure (#5382)
* Update

* lint

* lint

* r prefix

* CI

* lint

* skip TF

* Update

* edge homophily

* linkx homophily

* format

* skip TF

* fix test

* update

* lint

* lint

* review

* lint

* update

* lint

* update

* CI

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-36-188.ap-northeast-1.compute.internal>
2023-03-03 16:58:20 +08:00
Hongzhi (Steve), Chen 325e795a18 Revert "Set USE_LIBXSMM default to OFF. (#5287)" (#5392)
This reverts commit a5e3139155.
2023-03-01 16:47:34 +08:00
Zhiteng Li a53ecd229b [Dataset] Add CLUSTER dataset (#5389)
* add CLUSTER dataset

* refine according to dongyu's comments

---------

Co-authored-by: rudongyu <ru_dongyu@outlook.com>
2023-03-01 16:36:12 +08:00
czkkkkkk 999c624582 [Sparse] Support converson to/from torch sparse tensor. (#5388)
* [Sparse] Support converson to/from torch sparse tensor.

* Update
2023-02-28 17:27:23 +08:00
kylasa a14f69c97d [DistDGL][Feature_Request]Changes in the metadata.json file for input graph dataset. (#5310)
* Implemented the following changes.

* Remove NUM_NODES_PER_CHUNK
* Remove NUM_EDGES_PER_CHUNK
* Remove the dependency between no. of edge files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files and no. of edge files per edge type.
* Remove the dependency between no. of node feature files and no. of partitions
* Add “node_type_counts”. This will be a list of integers. Each integer will represent total count of a node-type. The index in this list and the index in the “node_type” will be the same for a given node-type.
* Add “edge_type_counts”. This will be a list of integers. Each integer will represent total count of an edge-type. The index in this list and the index in the “edge_type” list will be the same for a given edge-type.

* Applying lintrunner patch.

* Adding missing keys to the metadata in the unit test framework.

* lintrunner patch.

* Resolving CI test failures due to merge conflicts.

* Applying lintrunner patch

* applying lintrunner patch

* Replacing tabspace with spaces - to satisfy lintrunner

* Fixing the CI Test Failure cases.

* Applying lintrunner patch

* lintrunner complaining about a blank line.

* Resolving issues with print statement for NoneType

* Removed tests for the arbitrary chunks tests. Since this functionality is not supported anymore.

* Addressing CI review comments.

* addressing CI review comments

* lintrunner patch

* lintrunner patch.

* Addressing CI review comments.

* lintrunner patch.
2023-02-24 17:01:04 -08:00
Mufei Li fcf5ad5f49 [Utils] Node homophily measure (#5376)
* Update

* lint

* lint

* r prefix

* CI

* lint

* skip TF

* Update

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-36-188.ap-northeast-1.compute.internal>
2023-02-24 14:26:02 +08:00
czkkkkkk 5ffd2a022c [Sparse] Support column-wise softmax (#5377)
* [Sparse] Support column-wise softmax

* Update python/dgl/sparse/softmax.py

Co-authored-by: Mufei Li <mufeili1996@gmail.com>

---------

Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2023-02-24 10:03:43 +08:00
kylasa c42fa8a55c New script for customers to validate partitioned graph objects (#5340)
* A new script to validate graph partitioning pipeline

* Addressing CI review comments.

* lintrunner patch.
2023-02-23 14:30:39 -08:00
Kunal Mukherjee 45153fc078 [Model] Implemented SubgraphX Explainer for Homogeneous graph (#5315)
* subgraphx commit

* nits

* newline eof added

* lint fix

* test script updated to use default values

* lint fix

* graphs that are used for test cases are updated to a small graph

* lint formatted

* test paramter adj to complete the test under 20s

* lint fixes

---------

Co-authored-by: kxm180046 <kxm180046@utdallas.edu>
2023-02-23 12:50:31 +08:00
peizhou001 7ff041528c [Refactor] Add default ffi namespce capi (#5359) 2023-02-22 21:33:30 +08:00
Hongzhi (Steve), Chen 30b89e6a8a [Sparse] Lower the accuracy requirement of test_twirls in test. (#5364)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-22 14:37:51 +08:00
Tingyu Wang bcf9923b32 [Model] Add dgl.nn.CuGraphSAGEConv model (#5137)
* add CuGraphSAGEConv model

* fix lint issues

* update model to reflect changes in make_mfg_csr(), move max_in_degree to forward()

* lintrunner

* allow reset_parameters()

* remove norm option, simplify test

* allow full graph fallback option, add example

* address comments

* address reviews

---------

Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2023-02-22 13:43:28 +08:00
Hongzhi (Steve), Chen 02f2526b85 [Misc] Fix typo in status.py (#5362)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 22:37:22 +08:00
Hongzhi (Steve), Chen 085b19d7c2 [Misc] Treat aborted as failure in Jenkins. (#5361)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 22:19:26 +08:00
Hongzhi (Steve), Chen e41ce0c66a [Misc] Update Jenkins status. (#5356)
* test

* blabla

* add

* reformat

* balbla

* rollback

* remove

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 18:45:00 +08:00
peizhou001 ed2e540911 [Enhancement] Change id hash map (#5304)
* change concurrent id hash map
2023-02-21 16:47:03 +08:00
Hongzhi (Steve), Chen 71f7e96aeb [Misc] Abort previous CI. (#5346)
* Abort previous CI

* script

* newtry

* more

* fix

* comment

* add notes

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 15:09:30 +08:00
Hongzhi (Steve), Chen 27b3cd87e2 fix (#5349)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 14:02:25 +08:00
Hongzhi (Steve), Chen b8ed4d9ba3 [Misc] Revert status.py (#5348)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 14:00:08 +08:00
Hongzhi (Steve), Chen e4e0132972 [Misc] Add debug output to avoid silent failure. (#5347)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 13:42:00 +08:00
Hongzhi (Steve), Chen 0127a2ea4d [Misc] Update Jenkins status. (#5345)
* update

* polish

* polish

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 12:29:00 +08:00
Hongzhi (Steve), Chen e330954ca6 [Misc] Print jenkins response for test. (#5344)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 11:32:29 +08:00
Hongzhi (Steve), Chen deed945625 [Misc] Revert jenkins update change and add build url in log. (#5343)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 11:18:05 +08:00
Hongzhi (Steve), Chen f87e98e4b5 Update Jenkins response status (#5342)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-21 11:06:56 +08:00
Hongzhi (Steve), Chen f0759a96ee [Misc] Auto-reformat test/. (#5324)
* auto-format-test

* more

* remove

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-19 22:50:37 +08:00
Hongzhi (Steve), Chen 74c9d27d16 [Misc] Auto-format tests. (#5313)
* [Misc] Auto-format tests.

* more

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-17 23:33:44 +08:00
Hongzhi (Steve), Chen 86193c26f7 [Misc] Reorg python test (with backend) according to the guideline. (#5312)
* rename

* remove test transpose

* udpate

* rename

* cleanup

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-17 16:11:39 +08:00
Hongzhi (Steve), Chen d9e550369f [Misc] Reorg python test according to the guideline. (#5307)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-17 16:08:20 +08:00
Hongzhi (Steve), Chen 2bf2566fed remove (#5309)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-17 12:14:02 +08:00
Songqing Zhang 1329be9606 [Misc] Fix build warnings (#5303)
Co-authored-by: songqing.zhang <songqing.zhang@shopee.com>
2023-02-16 16:44:50 +08:00
nv-dlasalle 2bbca12ad5 [bugfix] Fix assertions in /src/runtime/workspace.h and expand unit tests for sparse optimizer (#5299)
* Fix assertions for size 0 workspaces

* Expand unit test to cover case of communication

* Fixes

* Format

* Fix c++ formatting
2023-02-16 13:43:03 +08:00
Hongzhi (Steve), Chen a5e3139155 Set USE_LIBXSMM default to OFF. (#5287)
* Set USE_LIBXSMM default to OFF.

* update

---------

Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
2023-02-16 13:08:11 +08:00
Tingyu Wang 19b3cea166 [Model] Update CuGraphRelGraphConv to use pylibcugraphops=23.02 (#5217)
* update cugraph_relgraphconv

* update equality test

* update cugraph rgcn example

* update RelGraphConvAgg based on latest API changes

* enable fallback option to fg when fanout is large

---------

Co-authored-by: Mufei Li <mufeili1996@gmail.com>
2023-02-15 20:11:20 +08:00
Quan (Andy) Gan fb8677b419 [Test] Move example test from unit test to examples (#5290)
* move example test from unit test to examples

* oops

* oops

* ok
2023-02-14 16:25:05 +08:00
kylasa 432c71ef25 Code changes to fix order sensitivity of the pipeline (#5288)
Following changes are made in this PR.
1. In dataset_utils.py, when reading edges from disk we follow the order defined by the STR_EDGE_TYPE key in the metadata.json file. This order is implicitly used to assign edgeid to edge types. This same order is used to read edges from the disk as well.
2. Now the unit test framework will also randomize the order of edges read from the disk. This is done for the edges when reading from the disk for the unit tests.

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2023-02-13 10:56:26 -08:00
Quan (Andy) Gan f62669b05c enable sparse on windows and mac (#5277)
* enable sparse on windows and mac

* that was stupid

* let's see what's going on..

* [Sparse] Fix the import error on Mac OS.

When using template functions that are defined in source files from DGL,
the loader of MacOS somehow cannot find their definitions. This fix simply
avoids depending on template functions from DGL headers.

With this fix, the sparse tests all pass on the MAC environment.

* ok this is the problem

* make errors clearer

* uh

* test

* Update __init__.py

* disabling ddp on windows

---------

Co-authored-by: czkkkkkk <zekucai@gmail.com>
2023-02-13 13:16:17 +08:00