 Andrej Karpathy
|
1f91bfc442
|
fix small bug on eval logging
|
2024-05-23 18:19:25 +00:00 |
|
|
Andrej Karpathy
|
86682af9a9
|
llm.c matches pytorch init from scratch exactly now
|
2024-05-23 17:50:33 +00:00 |
|
|
Andrej Karpathy
|
70a9c75348
|
use pytorch rand and fix dumb bug lol
|
2024-05-23 15:16:49 +00:00 |
|
|
Andrej Karpathy
|
d09631807a
|
first draft of random init, crashes with some cuBLAS error, debugging
|
2024-05-23 14:35:46 +00:00 |
|
|
Andrej Karpathy
|
d3cf9e211c
|
fix bug free only if initted
|
2024-05-23 00:49:32 +00:00 |
|
|
Andrej Karpathy
|
da59861d38
|
make hellaswag optional eval yay
|
2024-05-22 22:42:37 +00:00 |
|
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|
|
Andrej Karpathy
|
f7cbb30316
|
step 2 of dataloader refactor: separate out the dataloader to its own file, change its signature a little bit, and (notably) change from cudaMallocHost to a simple malloc, so that this file doesn't have to be cuda aware
|
2024-05-21 15:32:46 +00:00 |
|
|
Andrej Karpathy
|
f671cf9288
|
more changes, trying to help people out because when this merges to master it will brick everyone's code...
|
2024-05-20 23:02:41 +00:00 |
|
|
Andrej Karpathy
|
722e5b2fe5
|
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
|
2024-05-20 22:43:40 +00:00 |
|
|
Andrej
|
6c8bc172cc
|
Merge pull request #436 from ChrisDryden/boundscheck
Moved bounds checks to outside of the kernel
|
2024-05-20 00:41:41 +01:00 |
|
|
Andrej
|
2751fa0880
|
Merge pull request #435 from ChrisDryden/warpsize
Added warpsize as a constant for better compile time optimization and standardization
|
2024-05-20 00:39:15 +01:00 |
|
|
Christopher
|
6de1137e0e
|
Moved bounds checks outside of kernel into assertions
|
2024-05-19 23:17:45 +00:00 |
|
|
Christopher
|
ead5d35973
|
Added warpsize as a constant for better compile time optimization and standardization
|
2024-05-19 22:54:47 +00:00 |
|
|
Andrej Karpathy
|
bc58cd1dc1
|
fix small bugs in grad clip, introduce a GPU CPU synch point to communicate the float grad_clip to print it, and small printing changes
|
2024-05-19 17:51:29 +00:00 |
|
|
Andrej Karpathy
|
c2d12f725e
|
small touchups to grad clip
|
2024-05-19 17:07:55 +00:00 |
|
|
ngc92
|
589ead1e4f
|
updated code to adapt to latest changes
|
2024-05-18 22:47:42 +03:00 |
|
|
Erik Schultheis
|
a9947a8315
|
added a flag and the missing sqrt;
testing now has clipping enabled
|
2024-05-18 22:47:15 +03:00 |
|
|
Erik Schultheis
|
d7a81ef26f
|
added a useful mixed precision utility for dev/cuda
|
2024-05-18 22:45:44 +03:00 |
|
|
Erik Schultheis
|
44d45bdd6a
|
first draft for gradient clipping by global norm
|
2024-05-18 22:45:21 +03:00 |
|
|
Andrej Karpathy
|
c8fa7a8c63
|
revert the adamw allocation to previous. minor
|
2024-05-17 15:13:36 +00:00 |
|
|
Erik Schultheis
|
b24279c4db
|
remove duplicate workspace allocation
|
2024-05-17 17:37:47 +03:00 |
|
|
Erik Schultheis
|
88c3bea890
|
print message before actual allocation for more informative OOM behaviour
|
2024-05-17 17:37:45 +03:00 |
|
|
Erik Schultheis
|
0f23723ae4
|
joined optimizer state allocation
|
2024-05-17 17:37:16 +03:00 |
|
|
Andrej Karpathy
|
130712215d
|
Merge branch 'backward-bias' of https://github.com/ngc92/llm.c
|
2024-05-16 20:00:56 +00:00 |
|
|
Andrej Karpathy
|
6cfc7c5ce3
|
Merge branch 'zero-update' of https://github.com/ngc92/llm.c into ngc92-zero-update
|
2024-05-16 19:30:40 +00:00 |
|
|
Andrej Karpathy
|
d7581fc542
|
make recompute be an int instead of bool, so we can strengthen it over time just like ZeRO stages, as we recompute more and more of the model in the future possibly. and make it default on because it is awesome
|
2024-05-16 19:07:39 +00:00 |
|
|
Erik Schultheis
|
8b57cf6535
|
reduce communication overhead for ZERO stage 1
|
2024-05-16 14:12:29 +03:00 |
|
|
Erik Schultheis
|
57f70ea66b
|
simplify multi-gpu logic by reducing #ifdefs
|
2024-05-16 14:12:14 +03:00 |
|
|
Erik Schultheis
|
d48c3a4945
|
(optionally) recompute gelu activations to reduce activation memory
|
2024-05-16 12:39:28 +03:00 |
|
|
Erik Schultheis
|
7b810c1a3b
|
update main training script
|
2024-05-16 01:04:58 +03:00 |
|
|
Andrej Karpathy
|
8eb3a432d9
|
revert all streams and synchronization events, we'll bring them back but only one at a time and very very carefully, because they cause really subtle bugs and issues
|
2024-05-15 21:09:46 +00:00 |
|
|
Andrej Karpathy
|
a4567ae940
|
delete parallels, still not fixed
|
2024-05-14 22:13:54 +00:00 |
|
|
Andrej Karpathy
|
92fc26eba4
|
the nuts and bolts of gradient accumulation again, merged to master, but there is a bug and it doesn't work, debugging...
|
2024-05-14 21:32:47 +00:00 |
|
|
Andrej
|
222d59fa2f
|
Merge pull request #408 from ngc92/layernorm-bw-dev-cuda
Layernorm backward updates
|
2024-05-14 20:13:09 +01:00 |
|
|
Erik Schultheis
|
dd8c9f5ec9
|
fix layernorm backward: accumulate weight gradient
|
2024-05-14 20:43:51 +03:00 |
|
|
Andrej Karpathy
|
37a96debbf
|
Merge branch 'backward-bias' of https://github.com/ngc92/llm.c into ngc92-backward-bias
|
2024-05-13 20:18:33 +00:00 |
|
|
Erik Schultheis
|
c66e48c06c
|
fixup comment
|
2024-05-13 20:58:07 +03:00 |
|
|
Erik Schultheis
|
49ee3c8307
|
fix non-atomic version:
* accumulate instead of assign
* need dedicated argument to correctly handle the floatX == float case
|
2024-05-13 18:27:56 +03:00 |
|
|
Erik Schultheis
|
aa41b32624
|
update main file
|
2024-05-13 17:48:25 +03:00 |
|
|
chinthysl
|
f613ce895b
|
Fix copy and cast params to master weights
|
2024-05-13 09:13:02 +00:00 |
|
|
chinthysl
|
086ce2ff4d
|
Remove unsused template specializations and refactor
|
2024-05-13 06:48:22 +00:00 |
|
|
chinthysl
|
b3a5d1da15
|
shard master_weights
|
2024-05-13 05:55:46 +00:00 |
|
|
Chinthaka Gamanayake
|
0d9ce74b74
|
Merge branch 'karpathy:master' into zero-stage1
|
2024-05-13 13:34:52 +08:00 |
|
|
Erik Schultheis
|
ec92368f8b
|
added current backward bias kernel to dev cuda
|
2024-05-12 19:24:43 +03:00 |
|
|
Andrej Karpathy
|
43c48707a7
|
Merge branch 'fuse-ln-res-2' of https://github.com/ngc92/llm.c into ngc92-fuse-ln-res-2
|
2024-05-09 23:09:53 +00:00 |
|
|
ngc92
|
691c1df969
|
fused layernorm+residual
|
2024-05-09 23:48:11 +02:00 |
|