45e17e3ced
don't generate atomic functions for (b)float16 if those are not enabled
2024-04-25 01:39:08 +03:00
d49e8301eb
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
2024-04-24 22:11:00 +00:00
0ee2d4a965
Add llm.go in description
2024-04-24 17:48:49 -04:00
e44ed15639
small updates on intent for fp32 and non-fp32 CUDA versions
2024-04-24 19:44:12 +00:00
98567a4509
turn on benchmarking in mixed precision. example command: python train_gpt2.py --write_tensors 0 --inference_only 0 --dtype bfloat16 --compile 1 --tensorcores 1 --num_iterations 50 --sequence_length 1024
2024-04-24 19:26:40 +00:00
b2c779d40e
Merge pull request #231 from Saimirbaci/zig-port-ref
...
Added ref to the zig port
2024-04-24 10:00:08 -07:00
cd898b79d3
.add to .gitignore: fp32 binaries, *.exe, *.obj, .*
2024-04-24 17:58:49 +02:00
250159e140
encoder backward add, we want to get rid of atomics, hmm
2024-04-23 19:06:15 +00:00
fd7da62564
minor comment fixes, more to come
2024-04-23 18:20:50 +00:00
249140287f
rename positional_forward to encoder_forward as it should have been
2024-04-23 18:19:20 +00:00
0b59d80880
not proud of this but i have to increase the threshold for now to make test pass. later we should come back around and tighten thresholds
2024-04-23 18:13:57 +00:00
754627c0a2
update ci to pick up only the fp32 version for now, todo add mixed precision
2024-04-23 17:56:46 +00:00
c5f1db6020
fix important typo on precision
2024-04-23 17:43:19 +00:00
ed6387a695
updates to readme, and introduce the test fp32 cuda file too
2024-04-23 17:41:42 +00:00
7f279cd6cb
Adding Zig port ref
2024-04-23 13:37:53 -04:00
095d27662b
checkpoint the fp32 CUDA implementation to separate file. our mainline iteration will now continue in the new (mixed precision) file
2024-04-23 17:21:44 +00:00
6b6ad35f29
Merge pull request #218 from ademeure/linear16
...
Support for FP16/BF16 in train_gpt2.cu (1.86x Perf)
2024-04-23 10:17:01 -07:00
0333e36981
tiny fixes to make fp8 work in the future (missing conversions)
2024-04-23 14:38:41 +01:00
91ec92f4d5
fix typo
2024-04-23 06:43:40 +01:00
a876485151
Fill in param_sizeof in a slightly less terrible way
2024-04-23 06:39:58 +01:00
53dc40e804
Fixes for merge with latest, now ~86% faster!
2024-04-23 06:28:42 +01:00
c1992a19d5
Merge remote-tracking branch 'karpathy/master' into linear16
2024-04-23 06:18:23 +01:00
8775856b1d
Added floatN to choose FP32 vs FP16 for layernorm (parameters/gradients only)
2024-04-23 06:13:23 +01:00
7c193bd4cc
Good news: it works. Bad news: 16-bit atomics are *incredibly slow* :( (is it doing CAS?)
2024-04-23 05:00:47 +01:00
bdc661a609
BF16/FP16 Attention + bug fixes
2024-04-23 02:26:57 +01:00
85290a5228
Updated FP16/BF16 mix-and-match ncluding stochastic rounding
2024-04-23 01:39:33 +01:00
896f83f422
Merge pull request #225 from dagelf/master
...
Make msvc compile by adding the ifdefs back in gelu_backward, fixes #200
2024-04-22 14:45:29 -07:00
2efac6670a
Make msvc compile by adding the ifdefs back in gelu_backward, fixes #200
2024-04-22 23:17:06 +02:00
451c1a4c99
update readme to point to SoU
2024-04-22 19:18:03 +00:00
d3c5025a3e
Merge branch 'ChrisDryden-sharedmem_layernormback'
2024-04-22 18:45:01 +00:00
e3bcae6b2b
incorporate layernorm backward kernel 2, we are now down to 73.1ms/iter on average
2024-04-22 18:44:43 +00:00
86b3401dd5
Merge branch 'sharedmem_layernormback' of https://github.com/ChrisDryden/llm.c into ChrisDryden-sharedmem_layernormback
2024-04-22 18:33:42 +00:00
5f545ca6cc
Merge branch 'al0vya-matmul-backward-bias'
2024-04-22 17:51:26 +00:00
988489519c
nice new kernel for bias reduce, we are down by ~1ms/iter, to 76.13ms/iter on average
2024-04-22 17:51:06 +00:00
d605b993da
Fully working BF16 training! (very hacky Adam, and converges much slower than FP32 for now)
2024-04-22 18:38:51 +01:00
58df2629ae
Gradients now working in BF16 mode! (still need adam etc...)
2024-04-22 18:09:18 +01:00
a42f739675
Merge branch 'matmul-backward-bias' of https://github.com/al0vya/llm.c into al0vya-matmul-backward-bias
2024-04-22 16:52:31 +00:00
f813d635b2
Merge pull request #222 from regrettable-username/patch-1
...
Added llm.metal to notable forks
2024-04-22 09:33:27 -07:00
b1e55951c6
rename to _forward for the kernel, and small fixes to docs, and avoid auto
2024-04-22 16:32:03 +00:00
35393b4442
add more comments to explain the philosophy behind the kernel
2024-04-22 17:29:11 +01:00
b82ec201be
add comment
2024-04-22 17:05:05 +01:00
7830cf6a12
Merge pull request #214 from ngc92/trimul
...
added triangular matrix multiplication kernel
2024-04-22 09:03:53 -07:00
38857ff1d0
Added llm.metal to notable forks
...
This is a fork of llm.c designed to take full advantage of Apple Silicon via the Metal framework. Just got the forward pass working and will start on the backward pass. I wrote a minimal C wrapper API over the Objective-C Metal API that was needed to set up and run compute shaders. The rest of the code is pure-C and Metal Shading Language. I'm trying to mirror the original repo as closely as possible both in code and project philosophy.
2024-04-22 09:01:21 -07:00
55fbd7e6ad
add a faster matmul backward bias kernel that uses coalesced reads and shared memory
2024-04-22 16:35:21 +01:00
f35adbec77
WIP support for FP16/BF16 in train_gpt2.cu (compiles, not correct results)
2024-04-22 16:21:28 +01:00
732a8b4806
added NaN-based makes for reference checks
2024-04-22 14:53:24 +03:00
caa69a5e52
added triangular matrix multiplication kernel
2024-04-22 14:45:51 +03:00
6984e83c6e
Merge branch 'patricxu-master'
2024-04-22 00:28:00 +00:00
9ed0f10ee0
Merge branch 'master' of https://github.com/patricxu/llm.c into patricxu-master
2024-04-22 00:26:08 +00:00
fe85da4f3e
Merge pull request #209 from Ricardicus/master
...
Adding a github workflow
2024-04-21 17:22:53 -07:00
Erik Schultheis