* Sync parmetis_wrapper with changes in metadata.json
1. In the preprocess.py, make sure that num_partitions is defined as input argument. Also, align 'input_dir' with the input dataset. schema_file is assumed to be located inside the input_dir. Also, graph_stats.txt file is assumed to be present in the input_dir.
2. Use DGL_HOME environment variable so that parmetis_wrapper command can be run anywhere.
* Fix CI test failure cases.
* Addressing CI review comments.
* Addressing CI test failures.
* Applying lintrunner patch
* Support for no. of chunks smaller than no. of partitions and Adding appropriate test cases.
Following changes are made with this PR.
1. Code changes for handling no. of chunks smaller than no. of partitions
2. Adding new test cases, which were previously deleted, for no. of chunks smaller than no. of partitions.
3. Also adding test cases, where multiple partitions are handled by a single process.
* Committing the missing files in this commit.
* lintrunner patch.
* lintrunner check
* lintrunner patch here.
* CI review comments.
* add CuGraphGATConv model
* lintrunner
* update model to reflect changes in make_mfg_csr(), move max_in_degree to forward()
* simplify pytest markers
* fall back to FG option for large fanout
* update error msg
* add feat_drop and activation options
* add residual option
* Update python/dgl/nn/pytorch/conv/cugraph_gatconv.py
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
* Update python/dgl/nn/pytorch/conv/cugraph_gatconv.py
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
* reset res_fc
---------
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
Handling corner cases in the distributed lookup service. When the get partition ids function is invoked with empty request. This is needed because we are using alltoall function in the get_partition_ids function.
* Implemented the following changes.
* Remove NUM_NODES_PER_CHUNK
* Remove NUM_EDGES_PER_CHUNK
* Remove the dependency between no. of edge files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files and no. of edge files per edge type.
* Remove the dependency between no. of node feature files and no. of partitions
* Add “node_type_counts”. This will be a list of integers. Each integer will represent total count of a node-type. The index in this list and the index in the “node_type” will be the same for a given node-type.
* Add “edge_type_counts”. This will be a list of integers. Each integer will represent total count of an edge-type. The index in this list and the index in the “edge_type” list will be the same for a given edge-type.
* Applying lintrunner patch.
* Adding missing keys to the metadata in the unit test framework.
* lintrunner patch.
* Resolving CI test failures due to merge conflicts.
* Applying lintrunner patch
* applying lintrunner patch
* Replacing tabspace with spaces - to satisfy lintrunner
* Fixing the CI Test Failure cases.
* Applying lintrunner patch
* lintrunner complaining about a blank line.
* Resolving issues with print statement for NoneType
* Removed tests for the arbitrary chunks tests. Since this functionality is not supported anymore.
* Addressing CI review comments.
* addressing CI review comments
* lintrunner patch
* lintrunner patch.
* Addressing CI review comments.
* lintrunner patch.
* Uneven distribution of nodes/edges/features
To handle unevenly sized files for nodes/edges and feature files for nodes and edges, we have to synchronize before starting large no. of messages (either one large message or a burst of messages).
* Applying lintrunner patch.
* Removing tabspaces for lintrunner.
* lintrunner patch.
* removed issues introduced by the merge conflicts. Lots of code was repeated
* get_partition_ids, service provided by the distributed lookup service has high memory footprint
'get_partitionid' function, which is used to retrieve owner processes of the given list of global node ids, has high memory footprint. Currently this is of the order of 8x compared to the size of the input list.
For massively large datasets, this memory needs are very unrealistic and may result in OOM. In the case of CoreGraph, when retrieving owner of an edge list of size 6 Billion edges, the memory needs can be as high as 8*8*8 = 256 GB.
To limit the amount of memory used by this function, we split the size of the message sent to the distributed lookup service, so that each message is limited by the number of global node ids, which is 200 million. This reduced the memory footprint of this entire function to be no more than 0.2 * 8 * 8 = 13 GB. which is within reasonable limits.
Now since we send multiple small messages compared to one large message to the distributed lookup service, this may consume more wall-clock-time compared to earlier implementation.
* lintrunner patch.
* using np.ceil() per suggestion.
* converting the output of np.ceil() as ints.
* subgraphx commit
* nits
* newline eof added
* lint fix
* test script updated to use default values
* lint fix
* graphs that are used for test cases are updated to a small graph
* lint formatted
* test paramter adj to complete the test under 20s
* lint fixes
---------
Co-authored-by: kxm180046 <kxm180046@utdallas.edu>
* Wrap np.argsort() in a function. This
Use a python wrapper for the np.argsort() function for better usage of systems memory.
* lintrunner patch.
* lintrunner patch.
* Changes to address code review comments.
Rhett Ying