9d70d9a1b2
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
2024-04-28 16:17:22 +00:00
4f7d8d9f44
allow user to make different precisions, add prints and error handling around precisions
2024-04-28 16:08:27 +00:00
a00a3a88ef
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
2024-04-26 18:11:09 +00:00
2e17140d33
more defensive check for openmpi install, and an install command
2024-04-26 16:27:57 +00:00
d49e8301eb
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
2024-04-24 22:11:00 +00:00
ed6387a695
updates to readme, and introduce the test fp32 cuda file too
2024-04-23 17:41:42 +00:00
095d27662b
checkpoint the fp32 CUDA implementation to separate file. our mainline iteration will now continue in the new (mixed precision) file
2024-04-23 17:21:44 +00:00
c7aad65a04
Adding a workflow that builds with and without CUDA and OMP and on CPU also tests on ubuntu and macos
2024-04-22 02:14:09 +02:00
fced6d180f
small adjustments to Makefile and gitignore
2024-04-21 17:40:26 +00:00
3cd6f83e32
removing some warnings
2024-04-20 22:03:35 +02:00
be106f09a4
Merge branch 'master' into master
2024-04-20 22:00:02 +02:00
f534e4bdfe
Fixes -Ofast optimizations breaking model by skipping them for gelu_backward
2024-04-20 21:33:00 +02:00
280023bf8e
Support for vanilla build without CUDA tools
2024-04-20 21:32:35 +02:00
9c1c1f0cec
helper target that gets compiled with lineinfo and runs a single layer forward-backward pass
2024-04-20 03:19:23 +03:00
66c30fb55f
-fno-finite-math-only for almost 2x speed up
2024-04-16 12:49:54 +02:00
752cbe068d
Update Makefile with -march=native
...
-march=native Results in a 30% speedup on all the platforms I've tried, for train_gpt2 (Admittedly, all older Intel and AMD ones)
```
model name : Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz
step 0: train loss 5.356172 (took 15197.876634 ms)
vs
step 0: train loss 5.356185 (took 10418.548668 ms)
model name : AMD Ryzen 5 3600 6-Core Processor
step 0: train loss 5.356173 (took 3599.359234 ms)
vs
step 0: train loss 5.356185 (took 2708.045790 ms)
```
2024-04-15 19:52:19 +02:00
8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it ( #98 )
...
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
2024-04-12 15:32:07 -07:00
47b2f9312c
fixed homebrew path for intel based macs
2024-04-10 23:01:07 -04:00
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
1df7bceff9
using the compiler at hand
2024-04-09 06:22:03 +02:00
b2228d0e83
Minor correction for openmp check
2024-04-08 22:33:47 +02:00
e8e1628632
first commit of just the reference cpu fp32 gpt2 training
2024-04-08 12:41:21 -07:00
Andrej Karpathy