文件历史

提交图

13 次代码提交

作者 SHA1 备注 提交日期
Theodore Vasiloudis 1ab0170a10 [Distributed] Ensure round-robin edge file downloads, reduce logging, other improvements. (#5578)
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2023-04-27 11:17:55 -07:00
kylasa 4085ec8a46 Use correct delimiter when reading edge files during parmetis processing step (#5481)
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
2023-04-13 11:07:08 -07:00
kylasa 7b766393f8 [DistDGL][UserEx]Sync parmetis_wrapper with changes in metadata.json (#5385)
* Sync parmetis_wrapper with changes in metadata.json

1. In the preprocess.py, make sure that num_partitions is defined as input argument. Also, align 'input_dir' with the input dataset. schema_file is assumed to be located inside the input_dir. Also, graph_stats.txt file is assumed to be present in the input_dir.

2. Use DGL_HOME environment variable so that parmetis_wrapper command can be run anywhere.

* Fix CI test failure cases.

* Addressing CI review comments.

* Addressing CI test failures.

* Applying lintrunner patch
2023-03-06 15:50:43 -08:00
kylasa a14f69c97d [DistDGL][Feature_Request]Changes in the metadata.json file for input graph dataset. (#5310)
* Implemented the following changes.

* Remove NUM_NODES_PER_CHUNK
* Remove NUM_EDGES_PER_CHUNK
* Remove the dependency between no. of edge files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files and no. of edge files per edge type.
* Remove the dependency between no. of node feature files and no. of partitions
* Add “node_type_counts”. This will be a list of integers. Each integer will represent total count of a node-type. The index in this list and the index in the “node_type” will be the same for a given node-type.
* Add “edge_type_counts”. This will be a list of integers. Each integer will represent total count of an edge-type. The index in this list and the index in the “edge_type” list will be the same for a given edge-type.

* Applying lintrunner patch.

* Adding missing keys to the metadata in the unit test framework.

* lintrunner patch.

* Resolving CI test failures due to merge conflicts.

* Applying lintrunner patch

* applying lintrunner patch

* Replacing tabspace with spaces - to satisfy lintrunner

* Fixing the CI Test Failure cases.

* Applying lintrunner patch

* lintrunner complaining about a blank line.

* Resolving issues with print statement for NoneType

* Removed tests for the arbitrary chunks tests. Since this functionality is not supported anymore.

* Addressing CI review comments.

* addressing CI review comments

* lintrunner patch

* lintrunner patch.

* Addressing CI review comments.

* lintrunner patch.
2023-02-24 17:01:04 -08:00
Hongzhi (Steve), Chen 6bc8216118 [Misc] auto-format tools. (#5321)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-02-19 08:34:23 +08:00
kylasa aa42aaeb9f [DistDGL][Lintrunner]Lintrunner for tools directory (#5261)
* lintrunner patch for gloo_wrapper.py

* lintrunner changes to the tools directory.
2023-02-03 09:56:47 -08:00
Rhett Ying eff16b6116 [Dist] add input_dir for parmetis preprocess (#5232)
* [Dist] add input_dir for parmetis preprocess

* add support for parquet

* update parmetis_wrapper accordingly
2023-02-02 09:44:35 +08:00
Theodore Vasiloudis 774709d399 [Dist] Add support for Parquet-formatted edges files, remove some assumptions on edge file number. (#5051)
* [Dist] Add support for Parquet-formatted edges files, remove some assumptions on edge file number.

* [Dist] Add parquet edges option to unit tests.

Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
2023-01-03 16:01:50 +08:00
xiang song(charlie.song) adb07d1835 [Bugfix] Make preprocess compatible with openmpi (#4971)
* Make preprocess compatible with openmpi

* update docstr

Co-authored-by: Xiang Song <xiangsx@amazon.com>
2022-11-30 16:55:10 +08:00
xiang song(charlie.song) 344be1ef27 [Dist] Fix typo of metis preprocess in dist partitin pipeline 2022-11-09 13:31:16 +08:00
kylasa 98b9e0faf6 [Dist] Create <graph_name>_stats.txt file if it does not exist before ParMETIS execution (#4791)
* check if stats file exists, if not create one before parmetis run

* correct the typo error and correctly use constants.GRAPH_NAME
2022-11-07 14:06:16 -08:00
kylasa 9a72b78bc2 Updated the key to retrieve correct rank of a process (#4756)
Merging this PR to the master branch
2022-10-31 13:24:21 -07:00
kylasa eae6ce2a4f ParMETIS wrapper script to enable ParMETIS to process chunked dataset format (#4605)
* Creating ParMETIS wrapper script to run parmetis using one script from user perspective

* Addressed all the CI comments from PR https://github.com/dmlc/dgl/pull/4529

* Addressing CI comments.

* Isort, and black changes.

* Replaced python with python3

* Replaced single quote with double quotes per suggestion.

* Removed print statement

* Addressing CI Commets.

* Addressing CI review comments.

* Addressing CI comments as per chime discussion with Rui

* CI Comments, Black and isort changes

* Align with code refactoring, black, isort and code review comments.

* Addressing CI review comments, and fixing merge issues with the master branch

* Updated with proper unit test skip decorator
2022-10-03 12:23:57 -07:00