8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it ( #98 )
...
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
2024-04-12 15:32:07 -07:00
47b2f9312c
fixed homebrew path for intel based macs
2024-04-10 23:01:07 -04:00
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
1df7bceff9
using the compiler at hand
2024-04-09 06:22:03 +02:00
b2228d0e83
Minor correction for openmp check
2024-04-08 22:33:47 +02:00
e8e1628632
first commit of just the reference cpu fp32 gpt2 training
2024-04-08 12:41:21 -07:00
Andrej