 Andrej Karpathy
|
8d55c4a6ac
|
fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracy
|
2024-05-19 15:45:28 +00:00 |
|
|
Andrej Karpathy
|
4e14b52285
|
report both acc and acc_norm
|
2024-05-19 15:18:04 +00:00 |
|
|
Andrej Karpathy
|
7bd2389c1f
|
add mmlu as well and refine both a bit
|
2024-05-19 14:23:51 +00:00 |
|
|
Andrej Karpathy
|
aec9ce5d0a
|
move hellaswag file to dev
|
2024-05-19 12:18:02 +00:00 |
|
|
Andrej Karpathy
|
77b991281f
|
add hellaswag reference eval. it looks pretty bad honestly, the examples themselves, but i'm told it is predictive... hmm
|
2024-05-19 11:51:01 +00:00 |
|
|
Andrej Karpathy
|
3de05baaf4
|
Merge branch 'ngc92-errchk'
|
2024-05-18 18:35:28 +00:00 |
|
|
Andrej Karpathy
|
4374360015
|
adjust wte upper bound a bit, and print always because this part is really tricky and i don't trust anything other than manual inspection, even if we pass, allegedly
|
2024-05-18 18:34:24 +00:00 |
|
|
Erik Schultheis
|
bf36a4b6b5
|
improved numerical error checking:
tighter tolarances
relative tolerance based of bf16 epsilon
less verbose output if all is OK
|
2024-05-18 13:27:35 +03:00 |
|
|
Andrej Karpathy
|
f226a2c9d9
|
Merge branch 'ngc92-alloc'
|
2024-05-17 15:13:48 +00:00 |
|
|
Andrej Karpathy
|
c8fa7a8c63
|
revert the adamw allocation to previous. minor
|
2024-05-17 15:13:36 +00:00 |
|
|
Erik Schultheis
|
b24279c4db
|
remove duplicate workspace allocation
|
2024-05-17 17:37:47 +03:00 |
|
|
Erik Schultheis
|
88c3bea890
|
print message before actual allocation for more informative OOM behaviour
|
2024-05-17 17:37:45 +03:00 |
|
|
Erik Schultheis
|
0f23723ae4
|
joined optimizer state allocation
|
2024-05-17 17:37:16 +03:00 |
|
|
Andrej
|
b554d4d093
|
Merge pull request #423 from anthonix/readme_link
Add link to AMD fork
|
2024-05-16 21:35:18 +01:00 |
|
|
Anthony Blake
|
3113656e3f
|
Add link to AMD fork
|
2024-05-16 13:33:34 -07:00 |
|
|
Andrej
|
109f516367
|
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs
Free cooperative groups implementation of online softmax forward
|
2024-05-16 21:23:56 +01:00 |
|
|
Andrej
|
9e04deabe3
|
Merge pull request #416 from azret/feature/cpu_dev
Setting up dev/cpu area with the first matmul_forward.c
|
2024-05-16 21:12:55 +01:00 |
|
|
Andrej Karpathy
|
130712215d
|
Merge branch 'backward-bias' of https://github.com/ngc92/llm.c
|
2024-05-16 20:00:56 +00:00 |
|
|
Andrej Karpathy
|
68eea0bd5f
|
Fix ZeRO-1, we were communicating way too much
|
2024-05-16 19:48:24 +00:00 |
|
|
Andrej Karpathy
|
6cfc7c5ce3
|
Merge branch 'zero-update' of https://github.com/ngc92/llm.c into ngc92-zero-update
|
2024-05-16 19:30:40 +00:00 |
|
|
Andrej
|
bd7dc7af96
|
Merge pull request #422 from karpathy/feature/recompute
feature/recompute
|
2024-05-16 20:25:31 +01:00 |
|
|
Andrej Karpathy
|
d7581fc542
|
make recompute be an int instead of bool, so we can strengthen it over time just like ZeRO stages, as we recompute more and more of the model in the future possibly. and make it default on because it is awesome
|
2024-05-16 19:07:39 +00:00 |
|
|
Erik Schultheis
|
fbd8f03eea
|
fixup profiling
|
2024-05-16 14:25:06 +03:00 |
|
|
Erik Schultheis
|
8b57cf6535
|
reduce communication overhead for ZERO stage 1
|
2024-05-16 14:12:29 +03:00 |
|
|
Erik Schultheis
|
57f70ea66b
|
simplify multi-gpu logic by reducing #ifdefs
|
2024-05-16 14:12:14 +03:00 |
|
|
Andrej Karpathy
|
3277ccc35d
|
Merge branch 'act-recomp' of https://github.com/ngc92/llm.c into ngc92-act-recomp
|
2024-05-16 10:26:23 +00:00 |
|
|
Andrej
|
fbc5cba1f9
|
Merge pull request #415 from azret/feature/eot
train_gpt.c: Removing the hardcoded GPT2_EOT
|
2024-05-16 10:45:22 +01:00 |
|
|
Erik Schultheis
|
d48c3a4945
|
(optionally) recompute gelu activations to reduce activation memory
|
2024-05-16 12:39:28 +03:00 |
|
|
Erik Schultheis
|
7b810c1a3b
|
update main training script
|
2024-05-16 01:04:58 +03:00 |
|
|
Erik Schultheis
|
858c6e6dae
|
deterministic kernel
|
2024-05-16 01:04:10 +03:00 |
|
|
Erik Schultheis
|
2ccdfb70e0
|
general cleanup
|
2024-05-16 01:04:10 +03:00 |
|
|
Andrej
|
0f48a28f12
|
Merge pull request #412 from karpathy/feature/grad_accum3
[wip] gradient accumulation, another attempt
|
2024-05-15 22:53:16 +01:00 |
|
|
Andrej Karpathy
|
8eb3a432d9
|
revert all streams and synchronization events, we'll bring them back but only one at a time and very very carefully, because they cause really subtle bugs and issues
|
2024-05-15 21:09:46 +00:00 |
|
|
Azret Botash
|
160b3bd007
|
Setting up dev/CPU area with the first matmul_forward.c
|
2024-05-14 18:22:39 -07:00 |
|
|
Azret Botash
|
16f9dad301
|
Update train_gpt2.c
|
2024-05-14 17:45:08 -07:00 |
|
|
Azret Botash
|
7de50af511
|
train_gpt.c: Removing the hardcoded GPT2_EOT
|
2024-05-14 17:38:09 -07:00 |
|
|
Andrej Karpathy
|
a4567ae940
|
delete parallels, still not fixed
|
2024-05-14 22:13:54 +00:00 |
|
|
Andrej Karpathy
|
92fc26eba4
|
the nuts and bolts of gradient accumulation again, merged to master, but there is a bug and it doesn't work, debugging...
|
2024-05-14 21:32:47 +00:00 |
|
|
Andrej Karpathy
|
2d43e5bc97
|
remove legacy comment
|
2024-05-14 19:14:07 +00:00 |
|
|
Andrej
|
222d59fa2f
|
Merge pull request #408 from ngc92/layernorm-bw-dev-cuda
Layernorm backward updates
|
2024-05-14 20:13:09 +01:00 |
|
|
Erik Schultheis
|
e553e2f084
|
update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results
|
2024-05-14 20:44:50 +03:00 |
|
|
Erik Schultheis
|
dd8c9f5ec9
|
fix layernorm backward: accumulate weight gradient
|
2024-05-14 20:43:51 +03:00 |
|
|
Andrej Karpathy
|
2346cdac93
|
Merge branch 'ngc92-backward-bias'
|
2024-05-13 20:19:09 +00:00 |
|
|
Andrej Karpathy
|
37a96debbf
|
Merge branch 'backward-bias' of https://github.com/ngc92/llm.c into ngc92-backward-bias
|
2024-05-13 20:18:33 +00:00 |
|
|
Andrej
|
750c5fd7e8
|
Merge pull request #309 from ahrefs/zero-stage1
Zero Redundancy Optimizer - Stage1
|
2024-05-13 21:08:53 +01:00 |
|
|
Erik Schultheis
|
c66e48c06c
|
fixup comment
|
2024-05-13 20:58:07 +03:00 |
|
|
Erik Schultheis
|
65727d5a4d
|
fix CI compile by disabling kernel 5
|
2024-05-13 19:20:03 +03:00 |
|
|
Erik Schultheis
|
49ee3c8307
|
fix non-atomic version:
* accumulate instead of assign
* need dedicated argument to correctly handle the floatX == float case
|
2024-05-13 18:27:56 +03:00 |
|
|
Erik Schultheis
|
aa41b32624
|
update main file
|
2024-05-13 17:48:25 +03:00 |
|
|
Erik Schultheis
|
081d224b21
|
automatically switch to buffer-less version if that can fill up the GPU
|
2024-05-13 17:39:32 +03:00 |
|