* Enable AVX by default
* Fix linting errors
* Fix win64 build (libxsmm not linked)
Libxsmm on Win64 is not linked, should be disabled by default
* Fix clang format issues
* Change lower supported cpu version to LIBXSMM_X86_AVX2
Change lower supported cpu version to LIBXSMM_X86_AVX2 to address https://github.com/dmlc/dgl/issues/3459 issue
* Fix unit test
Remove assumption that libxsmm is enabled in the config by default (only true for intel CPUs with AVX2 instructions)
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-15-137.us-west-2.compute.internal>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
* Replacing numpy's unique with custom implementation
* Added docstring to the new function.
* Adding unit tests
* Numpy's version issues with the 'kind' argument.
* Addressing CI Test Failure.
* Addressing CI review comments.
* revised implementation, optimized for time.
* added missing arguments for fallback case.
* Addressing CI test failures.
* Resolving issues with PYTHONPATH
* Fix CI Test Failure issues.
* fix CI test failures.
---------
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
* new functional for creating data splits in graph
* minor fix in data split implementation
* apply suggestions from code review
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
* refactoring + unit tests
* fix test file name
* move imports to the top
* Revert "fix test file name"
This reverts commit 126323e38c8de6fdc73c826dccc9048aed8f9634.
* remove nccl submodule
* address linter issues
---------
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
* Bugfix for reverse edge issue.
* Resolved copy/paste transcription error
* lint
* Added unit tests for graph with not all reverses.
* linting + added backend device test
---------
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
* Sync parmetis_wrapper with changes in metadata.json
1. In the preprocess.py, make sure that num_partitions is defined as input argument. Also, align 'input_dir' with the input dataset. schema_file is assumed to be located inside the input_dir. Also, graph_stats.txt file is assumed to be present in the input_dir.
2. Use DGL_HOME environment variable so that parmetis_wrapper command can be run anywhere.
* Fix CI test failure cases.
* Addressing CI review comments.
* Addressing CI test failures.
* Applying lintrunner patch
* Support for no. of chunks smaller than no. of partitions and Adding appropriate test cases.
Following changes are made with this PR.
1. Code changes for handling no. of chunks smaller than no. of partitions
2. Adding new test cases, which were previously deleted, for no. of chunks smaller than no. of partitions.
3. Also adding test cases, where multiple partitions are handled by a single process.
* Committing the missing files in this commit.
* lintrunner patch.
* lintrunner check
* lintrunner patch here.
* CI review comments.
* add CuGraphGATConv model
* lintrunner
* update model to reflect changes in make_mfg_csr(), move max_in_degree to forward()
* simplify pytest markers
* fall back to FG option for large fanout
* update error msg
* add feat_drop and activation options
* add residual option
* Update python/dgl/nn/pytorch/conv/cugraph_gatconv.py
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
* Update python/dgl/nn/pytorch/conv/cugraph_gatconv.py
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
* reset res_fc
---------
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
Handling corner cases in the distributed lookup service. When the get partition ids function is invoked with empty request. This is needed because we are using alltoall function in the get_partition_ids function.
* Implemented the following changes.
* Remove NUM_NODES_PER_CHUNK
* Remove NUM_EDGES_PER_CHUNK
* Remove the dependency between no. of edge files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files and no. of edge files per edge type.
* Remove the dependency between no. of node feature files and no. of partitions
* Add “node_type_counts”. This will be a list of integers. Each integer will represent total count of a node-type. The index in this list and the index in the “node_type” will be the same for a given node-type.
* Add “edge_type_counts”. This will be a list of integers. Each integer will represent total count of an edge-type. The index in this list and the index in the “edge_type” list will be the same for a given edge-type.
* Applying lintrunner patch.
* Adding missing keys to the metadata in the unit test framework.
* lintrunner patch.
* Resolving CI test failures due to merge conflicts.
* Applying lintrunner patch
* applying lintrunner patch
* Replacing tabspace with spaces - to satisfy lintrunner
* Fixing the CI Test Failure cases.
* Applying lintrunner patch
* lintrunner complaining about a blank line.
* Resolving issues with print statement for NoneType
* Removed tests for the arbitrary chunks tests. Since this functionality is not supported anymore.
* Addressing CI review comments.
* addressing CI review comments
* lintrunner patch
* lintrunner patch.
* Addressing CI review comments.
* lintrunner patch.
* Uneven distribution of nodes/edges/features
To handle unevenly sized files for nodes/edges and feature files for nodes and edges, we have to synchronize before starting large no. of messages (either one large message or a burst of messages).
* Applying lintrunner patch.
* Removing tabspaces for lintrunner.
* lintrunner patch.
* removed issues introduced by the merge conflicts. Lots of code was repeated
* get_partition_ids, service provided by the distributed lookup service has high memory footprint
'get_partitionid' function, which is used to retrieve owner processes of the given list of global node ids, has high memory footprint. Currently this is of the order of 8x compared to the size of the input list.
For massively large datasets, this memory needs are very unrealistic and may result in OOM. In the case of CoreGraph, when retrieving owner of an edge list of size 6 Billion edges, the memory needs can be as high as 8*8*8 = 256 GB.
To limit the amount of memory used by this function, we split the size of the message sent to the distributed lookup service, so that each message is limited by the number of global node ids, which is 200 million. This reduced the memory footprint of this entire function to be no more than 0.2 * 8 * 8 = 13 GB. which is within reasonable limits.
Now since we send multiple small messages compared to one large message to the distributed lookup service, this may consume more wall-clock-time compared to earlier implementation.
* lintrunner patch.
* using np.ceil() per suggestion.
* converting the output of np.ceil() as ints.
czkkkkkk