文件历史

作者	SHA1	备注	提交日期
Andrej Karpathy	bf4ac57adc	backward the gelu and the matmul after	2024-04-16 20:29:08 +00:00
Andrej Karpathy	d753b1872e	clean up checks a bit in test script	2024-04-16 20:16:26 +00:00
Andrej Karpathy	0da60460e9	backprop through the first residual and final matmul of the final residual block	2024-04-16 20:12:17 +00:00
Andrej Karpathy	6be8080501	add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far	2024-04-16 02:10:45 +00:00
Andrej	8822e7803e	use cublaslt, which fuses bias, and also use tf32 when your GPU supports it (#98 ) Fusing the bias into matmul and moving to tf32 speed up the code a lot, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.	2024-04-12 15:32:07 -07:00
Andrej Karpathy	e1e0a08035	change fabs to fabsf	2024-04-10 19:23:18 +00:00
Andrej Karpathy	80f52e5290	the full forward pass of GPT-2 in one file of pure CUDA	2024-04-10 18:15:55 +00:00

1 2