* [DistTest] add basic pipeline for dist test across machines
* move launch remote cmd to separate file
* add test for rpc
* fix function naming rule
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* * Added specialization of cublasGemm function for `__half` type, to try to address https://github.com/dmlc/dgl/issues/3988
* * Added USE_FP16 guard
* * Added test cases to test_segment_mm, to test newly-added FP16 specialization of cublasGemm
* * Replaced for loop in test_segment_mm with pytest.mark.parametrize, as recommended
Co-authored-by: Xin Yao <xiny@nvidia.com>
* * Added support for common operations on FP16 (`half` or `__half`) for older GPU architectures
* Fixed an issue with previous check for FP16 support
* * Removing FP16 type checks, since they should no longer be needed
* * Fixed AtomicAdd to be atomic for `float` and `double` for old GPU architectures. Unfortunately, it seems that atomicCAS for unsigned short seems to be unavailable until architecture 70, so half will have to stay non-atomic on old GPUs.
* * Fixed non-atomic version of `AtomicAdd<half>` for older GPUs to return old value instead value of new
* Enable FP16 for GPU builds in CI
* Limit default GPU archs to pascal and above
* Disable FP16 dispatching for cuda architectures less than 60
* Fix linting
* Fix typos
* Initial implementation of Distributed data processing step in the Distributed Training pipeline
Implemented the following:
1) Read the output of parmetis (node-id to partition-id mappings)
2) Read the original graph files
3) Shuffle the node/edge metadata and features
4) output the partition specific files in DGL format using convert_partition.py functionality
5) Graph meta data is serialized in json format on rank-0 machine.
* Bug Fixes identified during verification of the dataset
1. When sending out global-id lookups for non-local nodes, in the msg_alltoall.py, conditional filter was used to identify the indices in node_data which is incorrect. Replaced the conditional filter with intersect1d to find out the common node ids and appropriate indices which are later used to identify the needed information to communicate.
2. When writing the graph level json file in distributed processing, the edge_offset on non-rank-0 machines was starting from 0 instead of the appropriate offset. Now added code to start the edge(s) from correct starting offset instead of 0 always.
* Restructuring and consolidation of code
1) Fixed issue when running verify_mag_dataset.py, Now we read xxx_removed_edges.txt and add these edges to `edge_data`. This will ensure that the self-loops and duplicate edges are handling appropriately when compared to the original dataset.
2) Consolidated code into a fewer files and changed code to following the python naming convention.
* Code changes addressing code review comments
Following changes are made in this commit.
1) Naming convention is defined and code is changed accordingly. Definition of various global_ids are defined and how to read them is mentioned.
2) All the code review comments are addressed
3)Files are moved to a new directory with dgl/tools directory as per suggestion
4) README.md file is include and it contains detailed information about the Naming convention adopted by the code, high level overview of the algorithm used in data-shuffling, example command-line to use on a single machine.
* addressing github review comments
Made code changes addressing all the review comments from GitHub.
* Addressing latest code review comments
Addressed all the latest code reviewing comments. One of the major changes is treating the node and edge metadata as dictionary objects and removing all the python lists with numpy arrays.
* Update README.md
Text rendering corrections
* Addressed code review comments
Addressed code review comments for the latest code review
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
* Change the curand_init parameter
* Change the curand_init parameter
* commit
* commit
* change the curandState and launch dim of CSRRowwiseSample kernel
* commit
* keep _CSRRowWiseSampleReplaceKernel in sync
Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com>
* * Added half_(), float_(), and double_() functions to DGLHeteroGraph, HeteroNodeDataView, and HeteroEdgeDataView, for converting floating-point tensor data to float16, float32, or float64 precision
* * Extracted out private functions for floating-point type conversion, to reduce code duplication
* * Added test for floating-point data conversion functions, half_(), float_(), and double_()
* * Moved half_(), float_(), and double_() functions from HeteroNodeDataView and HeteroEdgeDataView to Frame class
* * Updated test_float_cast() to use dgl.heterograph instead of dgl.graph
* Added to CONTRIBUTORS.md
* * Changed data type conversion to be deferred until the data is accessed, to avoid redundant conversions of data that isn't used.
* * Addressed issues flagged by linter
* * Worked around a bug in the old version of mxnet that's currently used for DGL testing
* * Only defer Column data type conversion if there is a pending device transfer or index sampling to be done. This is expected to be the desired behaviour based on discussions of a few use cases, as described in the comments.
* * Moved floating-point feature data conversion functions to dgl.transforms.functional
* Changed them from in-place behaviour to shallow copy (out-of-place) behaviour
* * Fixed linter issues
* * Removed lines that unintentionally added to_half, to_float, and to_double to DGLHeteroGraph
* Moved _init_api line to the end of the file again
* * Removed one of the two leading underscores from Frame.__astype_float, making it not fully private
Co-authored-by: nv-dlasalle <63612878+nv-dlasalle@users.noreply.github.com>
* Disable pinning non-contiguous memory
* Prevent views from being converted for write
* Fix linting
* Add unit tests
* Improve error message for users
* Switch to pytest function
* exclude mxnet and tensorflow from inplace pinning
* Add skip
* Restrict to pytorch backend
* Use backend to retrieve device
* Fix capitalization in decorator
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
With the emergence of new ISA (like ARM and RISCV) retaining USE_AVX ON default makes the default build instructions fail. Fundamentally DGL does not require the use of AVX for functional working. AVX is mainly needed when to enable optimization. So proposal is to default turn it off and then later during build instructions, folks with AVX capabilities can enable with
`cmake .. -DUSE_AVX=ON`
Co-authored-by: Zihao Ye <expye@outlook.com>
Rhett Ying