mirrors / dmlc--dgl

提交

2,937 提交 34 分支 42 Git 标签
test_sparse_compile
提交图

2937 次代码提交

此分支
此分支
所有分支
作者 SHA1 备注 提交日期
Kacper Pietkun 999374224f [Bugfix] fixed leak in SpMMCreateBlocks (#5210) 
* fixed leak in SpMMCreateBlocks

* clang format
 2023-02-23 16:30:44 +08:00
Kunal Mukherjee 45153fc078 [Model] Implemented SubgraphX Explainer for Homogeneous graph (#5315) 
* subgraphx commit

* nits

* newline eof added

* lint fix

* test script updated to use default values

* lint fix

* graphs that are used for test cases are updated to a small graph

* lint formatted

* test paramter adj to complete the test under 20s

* lint fixes

---------

Co-authored-by: kxm180046 <kxm180046@utdallas.edu>
 2023-02-23 12:50:31 +08:00
czkkkkkk 73a508e156 [Sparse] Stack SparseMatrix COO row and column coordinates into one tensor. (#5314) 2023-02-23 09:53:19 +08:00
kylasa 5ea04713d2 [DistDGL] Memory optimization to reduce memory footprint of the Dist Graph partitioning pipeline. (#5130) 
* Wrap np.argsort() in a function. This

Use a python wrapper for the np.argsort() function for better usage of systems memory.

* lintrunner patch.

* lintrunner patch.

* Changes to address code review comments.
 2023-02-22 11:34:03 -08:00
peizhou001 7ff041528c [Refactor] Add default ffi namespce capi (#5359) 2023-02-22 21:33:30 +08:00
Hongzhi (Steve), Chen 30b89e6a8a [Sparse] Lower the accuracy requirement of test_twirls in test. (#5364) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-22 14:37:51 +08:00
Tingyu Wang bcf9923b32 [Model] Add dgl.nn.CuGraphSAGEConv model (#5137) 
* add CuGraphSAGEConv model

* fix lint issues

* update model to reflect changes in make_mfg_csr(), move max_in_degree to forward()

* lintrunner

* allow reset_parameters()

* remove norm option, simplify test

* allow full graph fallback option, add example

* address comments

* address reviews

---------

Co-authored-by: Mufei Li <mufeili1996@gmail.com>
 2023-02-22 13:43:28 +08:00
Quan (Andy) Gan d8ca6317a3 [Release] Bump nightly version (#5357) 
* bump version

* Update update_version.py
 2023-02-22 09:13:38 +08:00
Hongzhi (Steve), Chen 9ce80e858b [Misc] Autoformat python dgl. (#5335) 
* autofix

* sort

* sort

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 23:38:30 +08:00
Hongzhi (Steve), Chen 02f2526b85 [Misc] Fix typo in status.py (#5362) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 22:37:22 +08:00
Hongzhi (Steve), Chen 085b19d7c2 [Misc] Treat aborted as failure in Jenkins. (#5361) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 22:19:26 +08:00
Hongzhi (Steve), Chen 529b26627b autofix (#5337) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 20:12:43 +08:00
Hongzhi (Steve), Chen 5bfa813761 [Misc] All overrun on master CI. (#5360) 
* exclude_master

* fix

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 20:08:01 +08:00
Hongzhi (Steve), Chen e41ce0c66a [Misc] Update Jenkins status. (#5356) 
* test

* blabla

* add

* reformat

* balbla

* rollback

* remove

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 18:45:00 +08:00
Rhett Ying 197f1d25ed [CI] use more light-weight node for lint check (#5358) 2023-02-21 18:44:30 +08:00
peizhou001 ed2e540911 [Enhancement] Change id hash map (#5304) 
* change concurrent id hash map
 2023-02-21 16:47:03 +08:00
Hongzhi (Steve), Chen 4706b6bf72 [Misc] Autoformat python dgl. (#5338) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 15:24:06 +08:00
Hongzhi (Steve), Chen 71f7e96aeb [Misc] Abort previous CI. (#5346) 
* Abort previous CI

* script

* newtry

* more

* fix

* comment

* add notes

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 15:09:30 +08:00
Hongzhi (Steve), Chen 27b3cd87e2 fix (#5349) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 14:02:25 +08:00
Hongzhi (Steve), Chen b8ed4d9ba3 [Misc] Revert status.py (#5348) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 14:00:08 +08:00
Hongzhi (Steve), Chen e4e0132972 [Misc] Add debug output to avoid silent failure. (#5347) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 13:42:00 +08:00
Hongzhi (Steve), Chen 0127a2ea4d [Misc] Update Jenkins status. (#5345) 
* update

* polish

* polish

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 12:29:00 +08:00
Hongzhi (Steve), Chen e330954ca6 [Misc] Print jenkins response for test. (#5344) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 11:32:29 +08:00
Hongzhi (Steve), Chen deed945625 [Misc] Revert jenkins update change and add build url in log. (#5343) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 11:18:05 +08:00
Hongzhi (Steve), Chen f87e98e4b5 Update Jenkins response status (#5342) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 11:06:56 +08:00
Hongzhi (Steve), Chen 36065bfd4e autofix (#5336) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 09:39:35 +08:00
Hongzhi (Steve), Chen a566b60be4 auto-fix (#5331) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-21 01:25:16 +08:00
Hongzhi (Steve), Chen d18274885d autofix2 (#5333) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-20 23:27:27 +08:00
Hongzhi (Steve), Chen 3e5137feaf [Misc] Autoformat python dgl (Part 2). (#5332) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-20 20:56:25 +08:00
Hongzhi (Steve), Chen 6e58f5f1a5 [Misc] Auto format with manual fix of 3 files and add pylint: disable= too-many-lines for functional.py. (#5330) 
* blabal

* 2more

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-20 17:14:10 +08:00
Rhett Ying f17c0d6224 [CI] add boto3 to interactive with AWS services like batch (#5329) 
* [CI] add boto3 to interactive with AWS services like batch

* Update dgl_dev.yml.template
 2023-02-20 11:37:33 +08:00
Hongzhi (Steve), Chen dce899190e [Misc] Auto-reformat multiple python folders. (#5325) 
* auto-reformat

* lintrunner

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-20 10:09:22 +08:00
Hongzhi (Steve), Chen ab812179a3 [Misc] Auto reformat data/. (#5318) 
* data

* lintrunner

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-19 22:50:55 +08:00
Hongzhi (Steve), Chen f0759a96ee [Misc] Auto-reformat test/. (#5324) 
* auto-format-test

* more

* remove

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-19 22:50:37 +08:00
Hongzhi (Steve), Chen 9fe5092c17 lintrunner (#5326) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-19 22:50:07 +08:00
Hongzhi (Steve), Chen 9836f78e77 autoformat (#5322) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-19 11:24:16 +08:00
Hongzhi (Steve), Chen 704bcaf6dd examples (#5323) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-19 08:35:15 +08:00
Hongzhi (Steve), Chen 6bc8216118 [Misc] auto-format tools. (#5321) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-19 08:34:23 +08:00
Hongzhi (Steve), Chen 63ac788ff4 auto-reformat-nn (#5319) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-19 08:33:30 +08:00
Hongzhi (Steve), Chen 0b3a447ba4 auto format distributed (#5317) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-19 08:33:09 +08:00
Hongzhi (Steve), Chen 74c9d27d16 [Misc] Auto-format tests. (#5313) 
* [Misc] Auto-format tests.

* more

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-17 23:33:44 +08:00
Hongzhi (Steve), Chen 86193c26f7 [Misc] Reorg python test (with backend) according to the guideline. (#5312) 
* rename

* remove test transpose

* udpate

* rename

* cleanup

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-17 16:11:39 +08:00
Hongzhi (Steve), Chen d9e550369f [Misc] Reorg python test according to the guideline. (#5307) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-17 16:08:20 +08:00
Hongzhi (Steve), Chen 2bf2566fed remove (#5309) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-17 12:14:02 +08:00
Hongzhi (Steve), Chen 03dbe75808 blabal (#5306) 
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>
 2023-02-17 11:17:08 +08:00
kylasa 9ce800d2f3 [DistDGL][Optimizations]Rehash code to optimize for loop (#5224) 
* Rehash code to optimize for loop

Reduced number of instructions in for loop, which exchanging edge features. This will reduce the number of times numpy's intersect1d is invoked (saving the runtime and memory overhead needs of numpy).

* Applying lintrunner patch to data_shuffle.py
 2023-02-16 09:34:07 -08:00
kylasa e25f47dea2 [DistDGL][Mem_Optimizations]Edge Ownership processes are computed on the fly when required. (#5225) 
* Edge Ownership processes are computed on the fly when required.

Earlier we were storing Edge ownership processes after the dataset was retrieved from the disk. For massively large datasets, each node can handle upto 5 Billion edges, this means storing owner process-ids will consume 5 * 8 = 40GB. This memory will be hanging around until the edges are exchanged.

To reduce the memory footprint of the pipeline, we no longer store the ownership process-ids in the 'edge_data' dictionary after reading the dataset from the disk. Instead, we compute them on the fly at the time of exchanging edges.

Another optimization is not to send/receive all the messages in a one single large message. Instead we now split the total number edges into chunks, limited by 8 GB per node. And we iterate until all the chunks are exchanged.

Once all the edges are exchanged, as a sanity check, we compute the total number of edges in the system and compare it with the original value before edge shuffling, in a final assert statement before return the result to the caller.

* Applying lintrunner patch.
 2023-02-16 09:33:40 -08:00
Songqing Zhang 1329be9606 [Misc] Fix build warnings (#5303) 
Co-authored-by: songqing.zhang <songqing.zhang@shopee.com>
 2023-02-16 16:44:50 +08:00
Hongzhi (Steve), Chen 85dfcea0bc [Dev] Create a script for kick off python test. (#5298) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-16 14:57:46 +08:00
Hongzhi (Steve), Chen ccbc30fdd2 [Dev] Create a script for building dgl from source. (#5294) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
 2023-02-16 13:46:05 +08:00