* Rehash code to optimize for loop
Reduced number of instructions in for loop, which exchanging edge features. This will reduce the number of times numpy's intersect1d is invoked (saving the runtime and memory overhead needs of numpy).
* Applying lintrunner patch to data_shuffle.py
* Edge Ownership processes are computed on the fly when required.
Earlier we were storing Edge ownership processes after the dataset was retrieved from the disk. For massively large datasets, each node can handle upto 5 Billion edges, this means storing owner process-ids will consume 5 * 8 = 40GB. This memory will be hanging around until the edges are exchanged.
To reduce the memory footprint of the pipeline, we no longer store the ownership process-ids in the 'edge_data' dictionary after reading the dataset from the disk. Instead, we compute them on the fly at the time of exchanging edges.
Another optimization is not to send/receive all the messages in a one single large message. Instead we now split the total number edges into chunks, limited by 8 GB per node. And we iterate until all the chunks are exchanged.
Once all the edges are exchanged, as a sanity check, we compute the total number of edges in the system and compare it with the original value before edge shuffling, in a final assert statement before return the result to the caller.
* Applying lintrunner patch.
* update cugraph_relgraphconv
* update equality test
* update cugraph rgcn example
* update RelGraphConvAgg based on latest API changes
* enable fallback option to fg when fanout is large
---------
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
Following changes are made in this PR.
1. In dataset_utils.py, when reading edges from disk we follow the order defined by the STR_EDGE_TYPE key in the metadata.json file. This order is implicitly used to assign edgeid to edge types. This same order is used to read edges from the disk as well.
2. Now the unit test framework will also randomize the order of edges read from the disk. This is done for the edges when reading from the disk for the unit tests.
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Some weird formatting occurred in the documentation for EGATConv, so this quick commit should fix that.
Co-authored-by: Mufei Li <mufeili1996@gmail.com>
* enable sparse on windows and mac
* that was stupid
* let's see what's going on..
* [Sparse] Fix the import error on Mac OS.
When using template functions that are defined in source files from DGL,
the loader of MacOS somehow cannot find their definitions. This fix simply
avoids depending on template functions from DGL headers.
With this fix, the sparse tests all pass on the MAC environment.
* ok this is the problem
* make errors clearer
* uh
* test
* Update __init__.py
* disabling ddp on windows
---------
Co-authored-by: czkkkkkk <zekucai@gmail.com>
* Fix Column.record_stream(...) for unmaterialized tensors
* lint
* Record streams for indices on non-materialized columns
* add docstring
* fix for cpu index
* Record also stream on storage
* Always call self.storage.record_stream when storage is on GPU
* fix lint
---------
Co-authored-by: Xin Yao <xiny@nvidia.com>
* add SVD positional encoding
* modify importing module
* Fixed certain problems
* Change the test unit to a nonsigular one
* Fixed typo and make accord with lintrunner
* added svd_pe into dgl.rst
* Modified dgl.rst
Hongzhi (Steve), Chen