mirrors / karpathy--llm.c

项目文件夹

文件

T

文件历史

Andrej Karpathy 31be5e790d add rmsnorm backward in dev/cuda, it seems to work surprisingly, and is probably ready to be integrated into llmc. we are still using 2X too much shared memory because I didn't want to change way too many things at the same time. I copy pasted our kernel10 of layernorm backward and made tweaks to it removing the bias and mean cool

2024-09-26 03:17:58 +00:00

..

README.md

add RoPE PyTorch and C reference code

2024-09-24 23:52:16 +00:00

rmsnorm.py

add rmsnorm backward in dev/cuda, it seems to work surprisingly, and is probably ready to be integrated into llmc. we are still using 2X too much shared memory because I didn't want to change way too many things at the same time. I copy pasted our kernel10 of layernorm backward and made tweaks to it removing the bias and mean cool

2024-09-26 03:17:58 +00:00

rope.c

add RoPE PyTorch and C reference code

2024-09-24 23:52:16 +00:00

rope.py

add RoPE PyTorch and C reference code

2024-09-24 23:52:16 +00:00

README.md

cbridge

We'll use this directory for the PyTorch -> C bridge. So we have some PyTorch code and we'd like to have the equivalent C implementation (usually that one in turn serves as reference for the CUDA kernels later).

For starters we have RoPE. E.g. generate the reference with PyTorch and then match it in C:

python rope.py
gcc -o rope rope.c -lm
./rope

The .py file writes a robe.bin file with the intermediate tensors.