 Andrej Karpathy
|
d7581fc542
|
make recompute be an int instead of bool, so we can strengthen it over time just like ZeRO stages, as we recompute more and more of the model in the future possibly. and make it default on because it is awesome
|
2024-05-16 19:07:39 +00:00 |
|
|
Andrej Karpathy
|
3277ccc35d
|
Merge branch 'act-recomp' of https://github.com/ngc92/llm.c into ngc92-act-recomp
|
2024-05-16 10:26:23 +00:00 |
|
|
Andrej
|
fbc5cba1f9
|
Merge pull request #415 from azret/feature/eot
train_gpt.c: Removing the hardcoded GPT2_EOT
|
2024-05-16 10:45:22 +01:00 |
|
|
Erik Schultheis
|
d48c3a4945
|
(optionally) recompute gelu activations to reduce activation memory
|
2024-05-16 12:39:28 +03:00 |
|
|
Andrej
|
0f48a28f12
|
Merge pull request #412 from karpathy/feature/grad_accum3
[wip] gradient accumulation, another attempt
|
2024-05-15 22:53:16 +01:00 |
|
|
Andrej Karpathy
|
8eb3a432d9
|
revert all streams and synchronization events, we'll bring them back but only one at a time and very very carefully, because they cause really subtle bugs and issues
|
2024-05-15 21:09:46 +00:00 |
|
|
Azret Botash
|
16f9dad301
|
Update train_gpt2.c
|
2024-05-14 17:45:08 -07:00 |
|
|
Azret Botash
|
7de50af511
|
train_gpt.c: Removing the hardcoded GPT2_EOT
|
2024-05-14 17:38:09 -07:00 |
|
|
Andrej Karpathy
|
a4567ae940
|
delete parallels, still not fixed
|
2024-05-14 22:13:54 +00:00 |
|
|
Andrej Karpathy
|
92fc26eba4
|
the nuts and bolts of gradient accumulation again, merged to master, but there is a bug and it doesn't work, debugging...
|
2024-05-14 21:32:47 +00:00 |
|
|
Andrej Karpathy
|
2d43e5bc97
|
remove legacy comment
|
2024-05-14 19:14:07 +00:00 |
|
|
Andrej
|
222d59fa2f
|
Merge pull request #408 from ngc92/layernorm-bw-dev-cuda
Layernorm backward updates
|
2024-05-14 20:13:09 +01:00 |
|
|
Erik Schultheis
|
e553e2f084
|
update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results
|
2024-05-14 20:44:50 +03:00 |
|
|
Erik Schultheis
|
dd8c9f5ec9
|
fix layernorm backward: accumulate weight gradient
|
2024-05-14 20:43:51 +03:00 |
|
|
Andrej Karpathy
|
2346cdac93
|
Merge branch 'ngc92-backward-bias'
|
2024-05-13 20:19:09 +00:00 |
|
|
Andrej Karpathy
|
37a96debbf
|
Merge branch 'backward-bias' of https://github.com/ngc92/llm.c into ngc92-backward-bias
|
2024-05-13 20:18:33 +00:00 |
|
|
Andrej
|
750c5fd7e8
|
Merge pull request #309 from ahrefs/zero-stage1
Zero Redundancy Optimizer - Stage1
|
2024-05-13 21:08:53 +01:00 |
|
|
Erik Schultheis
|
c66e48c06c
|
fixup comment
|
2024-05-13 20:58:07 +03:00 |
|
|
Erik Schultheis
|
65727d5a4d
|
fix CI compile by disabling kernel 5
|
2024-05-13 19:20:03 +03:00 |
|
|
Erik Schultheis
|
49ee3c8307
|
fix non-atomic version:
* accumulate instead of assign
* need dedicated argument to correctly handle the floatX == float case
|
2024-05-13 18:27:56 +03:00 |
|
|
Erik Schultheis
|
aa41b32624
|
update main file
|
2024-05-13 17:48:25 +03:00 |
|
|
Erik Schultheis
|
081d224b21
|
automatically switch to buffer-less version if that can fill up the GPU
|
2024-05-13 17:39:32 +03:00 |
|
|
Erik Schultheis
|
c0329ebdba
|
new kernel version with fewer atomics
|
2024-05-13 17:18:27 +03:00 |
|
|
chinthysl
|
f613ce895b
|
Fix copy and cast params to master weights
|
2024-05-13 09:13:02 +00:00 |
|
|
chinthysl
|
086ce2ff4d
|
Remove unsused template specializations and refactor
|
2024-05-13 06:48:22 +00:00 |
|
|
chinthysl
|
b3a5d1da15
|
shard master_weights
|
2024-05-13 05:55:46 +00:00 |
|
|
Chinthaka Gamanayake
|
0d9ce74b74
|
Merge branch 'karpathy:master' into zero-stage1
|
2024-05-13 13:34:52 +08:00 |
|
|
Erik Schultheis
|
2287da0120
|
enable bf16
|
2024-05-12 19:42:32 +03:00 |
|
|
Andrej
|
c1814d5d2f
|
Merge pull request #401 from rosslwheeler/ci-add-win
Add Windows to CI
|
2024-05-12 18:32:13 +02:00 |
|
|
Andrej
|
a2bffb5a8a
|
Merge pull request #403 from nietras/nietras-llm-cs
Add Llm.cs by nietras to README
|
2024-05-12 18:29:20 +02:00 |
|
|
Erik Schultheis
|
ec92368f8b
|
added current backward bias kernel to dev cuda
|
2024-05-12 19:24:43 +03:00 |
|
|
ntr
|
e64df91191
|
Add Llm.cs by nietras to README
|
2024-05-12 16:02:36 +02:00 |
|
|
Ross Wheeler
|
b88f683569
|
Add Windows to CI
Change notes:
• Add Windows build/test to CI matrix build
• Add Cuda Windows build
• Replace v3 checkout actions to v4 per GitHub recommendations
|
2024-05-11 17:49:22 -07:00 |
|
|
Andrej Karpathy
|
1cfe899121
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-05-11 17:21:18 +00:00 |
|
|
Andrej Karpathy
|
fb86c121a1
|
Merge branch 'Anerudhan-feature/simplify_cudnn_usage'
|
2024-05-11 17:21:04 +00:00 |
|
|
Andrej Karpathy
|
3dbb0bb89c
|
bump the threshold for qkvw because flashattention expands the error here a tiny bit
|
2024-05-11 17:19:41 +00:00 |
|
|
Andrej Karpathy
|
58d9dda497
|
Merge branch 'feature/simplify_cudnn_usage' of https://github.com/Anerudhan/llm.c into Anerudhan-feature/simplify_cudnn_usage
|
2024-05-11 17:08:31 +00:00 |
|
|
Andrej
|
5dbda7e115
|
Merge pull request #397 from lancerts/comment-fix
Update the -lcublas -lcublasLt flag in the comment
|
2024-05-11 19:03:40 +02:00 |
|
|
Andrej
|
923b2b0a8f
|
Merge pull request #398 from zwets/fix-cudnn-frontend-check
Fix detection of cudnn-frontend in '.' on Linux
|
2024-05-10 12:13:53 +02:00 |
|
|
Marco van Zwetselaar
|
6da5e63e2c
|
Fix detection of cudnn-frontend in '.' on Linux
Plus minor legibility fixes in the cudnn-frontend explanation.
|
2024-05-10 10:57:39 +03:00 |
|
|
lancer
|
5c90845f7a
|
update the -lcublas -lcublasLt flag in the comment
|
2024-05-09 22:04:02 -07:00 |
|
|
Andrej
|
fd007d6508
|
Merge pull request #395 from yijunyu/patch-1
Update README.md
|
2024-05-10 01:18:25 +02:00 |
|
|
Andrej Karpathy
|
83ad9ac87f
|
Fused Residual+LayerNorm layers, ~1% faster
|
2024-05-09 23:16:41 +00:00 |
|
|
Andrej Karpathy
|
43c48707a7
|
Merge branch 'fuse-ln-res-2' of https://github.com/ngc92/llm.c into ngc92-fuse-ln-res-2
|
2024-05-09 23:09:53 +00:00 |
|
|
Yijun Yu
|
8ccf2f97f8
|
Update README.md
|
2024-05-10 06:49:24 +08:00 |
|
|
ngc92
|
75ec629f52
|
remove duplicate function
|
2024-05-10 00:03:30 +02:00 |
|
|
ngc92
|
5920143d47
|
remove warning noise
|
2024-05-10 00:02:37 +02:00 |
|
|
ngc92
|
691c1df969
|
fused layernorm+residual
|
2024-05-09 23:48:11 +02:00 |
|
|
Anerudhan
|
01c7a331bb
|
- Simplify graph cache and usage of cudnn.
- Fix failures in H100
|
2024-05-09 19:07:00 +00:00 |
|
|
Andrej
|
077b649ee1
|
Merge pull request #389 from vedaanta/cudnn/install
Adds cuDNN install instructions for a smaller but sufficient package
|
2024-05-09 20:30:48 +02:00 |
|