Vasimuddin Md
78e0dae693
* added distgnn plus libra codebase * Dist application codes * added comments in partition code. changed the interface of partitioning call. * updated readme * create libra partitioning branch for the PR * removed disgnn files for first PR * updated kernel.cc * added libra_partition.cc and moved libra code from kernel.cc to libra_partition.cc * fixed lint error; merged libra2dgl.py and main_Libra.py to libra_partition.py; added graphsage/distgnn folder and partition script. * removed libra2dgl.py * fixed the lint error and cleaned the code. * revisions due to PR comments. added distgnn/tools contains partitions routines * update 2 PR revision I * fixed errors; also improved the runtime by 10x. * fixed minor lint error * fixed some more lints * PR revision II changed the interface of libra partition function * rewrite docstring Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
1.4 KiB
DistGNN vertex-cut based graph partitioning (using Libra)
How to run graph partitioning
python partition_graph.py --dataset <dataset> --num-parts <num_parts> --out-dir <output_location>
Example: The following command-line creates 4 partitions of pubmed graph
python partition_graph.py --dataset pubmed --num-parts 4 --out-dir ./
The ouptut partitions are created in the current directory in Libra_result_<dataset>/ folder.
The upcoming DistGNN application can directly use these partitions for distributed training.
How Libra partitioning works
Libra is a vertex-cut based graph partitioning method. It applies greedy heuristics to uniquely distribute the input graph edges among the partitions. It generates the partitions as a list of edges. Script libra_partition.py after generates the Libra partitions and converts the Libra output to DGL/DistGNN input format.
Note: Current Libra implementation is sequential. Extra overhead is paid due to the additional work of format conversion of the partitioned graph.
Expected partitioning timinigs
Cora, Pubmed, Citeseer: < 10 sec (<10GB)
Reddit: 150 sec ( 25GB)
OGBN-Products: ~200 sec (~30GB)
Proteins: 1800 sec (Format conversion from public data takes time) (~100GB)
OGBN-Paper100M: 2500 sec (~200GB)
Settings
Tested with: Cent OS 7.6 gcc v8.3.0 PyTorch 1.7.1 Python 3.7.10