 Andrej Karpathy
|
35aa110244
|
more refactor
|
2024-05-30 19:39:29 +00:00 |
|
|
Andrej Karpathy
|
f8f9e49ba3
|
Merge branch 'master' into feature/libdir
|
2024-05-30 19:07:51 +00:00 |
|
|
Chris Dryden
|
3bca31fde1
|
Nit: duplicated line in header of project
|
2024-05-29 20:03:13 -07:00 |
|
|
Andrej Karpathy
|
4cc8b1570b
|
hardcode grad_clip to 1.0, delete confusion around legacy code that used -c flag for it, but became inactive in the if else block
|
2024-05-29 19:01:12 +00:00 |
|
|
QIAOQIAN
|
27cf5223e0
|
softmax_autoregressive_backward_kernel does not use share memory in the kernel
we do not need to launch the kernel with 256 bytes share memory, so remove it
|
2024-05-29 09:23:46 +08:00 |
|
|
Andrej Karpathy
|
037ec67a5d
|
small tweak to launch
|
2024-05-28 15:07:07 +00:00 |
|
|
Andrej Karpathy
|
eed242702a
|
experiment with adding the llmc lib directory
|
2024-05-27 21:10:28 +00:00 |
|
|
Andrej Karpathy
|
69d0583ac8
|
conditionally include dirent on not windows
|
2024-05-27 20:28:52 +00:00 |
|
|
Andrej Karpathy
|
b75738c4eb
|
resume optimization, seems to be working
|
2024-05-27 18:02:51 +00:00 |
|
|
Andrej Karpathy
|
f93a30fbbf
|
more careful with conditional MPI use
|
2024-05-27 15:54:55 +00:00 |
|
|
Andrej Karpathy
|
d295cb8d81
|
part 1 of v1 of resume training functionality, writes the files but doesn't load them yet, coming up in a bit
|
2024-05-27 15:49:03 +00:00 |
|
|
Erik Schultheis
|
7dc3b7d7dc
|
bugfix
|
2024-05-27 16:36:44 +03:00 |
|
|
Erik Schultheis
|
b913905f84
|
fix out-of-bounds access for encoder_backward: we were allocating too little memory (/ -> DIV_CEIL)
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
841c4dec03
|
update main training file
|
2024-05-27 13:15:36 +03:00 |
|
|
Andrej Karpathy
|
c6680a25ae
|
fix a bug where the max_steps was not accounting for the gradient accumulation, meaning that e.g. with grad accum 2, we'd accidentally train for 2 epochs instead of just 1 epoch. also simplify the interface a bit and the code make more readable etc
|
2024-05-26 16:49:30 +00:00 |
|
|
Andrej Karpathy
|
05fb642ae1
|
attempt fix zero 1
|
2024-05-25 23:49:16 +00:00 |
|
|
Erik Schultheis
|
7ba3ef6bac
|
make sure to run Adam only for the range of parameters that is available on the local device
|
2024-05-25 23:20:42 +03:00 |
|
|
Andrej Karpathy
|
f5d041ab0c
|
careful with NULL and checkpoint correctly
|
2024-05-25 16:32:18 +00:00 |
|
|
Andrej Karpathy
|
273925d10d
|
save checkpoint files to log directory, and small readability changes
|
2024-05-25 16:05:29 +00:00 |
|
|
Andrej Karpathy
|
90a745c40c
|
add MFU (model flops utilization) calculation and print it. also refactor the printing a bit to make it nice. this calculation only works for A100 fp16 we should think through how to generalize it a bit
|
2024-05-25 14:30:59 +00:00 |
|
|
Andrej Karpathy
|
a13b2bd10b
|
fix bug check null
|
2024-05-25 13:50:56 +00:00 |
|
|
Andrej Karpathy
|
f2ee356a06
|
resolve merge conflict
|
2024-05-25 00:33:12 +00:00 |
|
|
Andrej Karpathy
|
9f08882051
|
add weight decay, but only for 2D tensors, as done in GPT series and in general too. this forces us to break up our adamw kernel again into one call per tensor, so there is a small throughput hit, of about 0.5% for me. but we have to break up this kernel in near future anyway
|
2024-05-25 00:14:10 +00:00 |
|
|
Andrej Karpathy
|
8b951056b5
|
resolve merge conflict
|
2024-05-24 22:04:02 +00:00 |
|
|
Andrej Karpathy
|
25f17e6748
|
small formatting fix before merge
|
2024-05-24 21:53:00 +00:00 |
|
|
Andrej Karpathy
|
dbacaf84cf
|
Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm
|
2024-05-24 21:46:25 +00:00 |
|
|
Andrej Karpathy
|
16b364d256
|
change -o to be not file but directory, and create a main.log inside it for the log. this is to prepare for writing of checkpoints, where we want to re-use this directory for those as well
|
2024-05-24 17:57:50 +00:00 |
|
|
Erik Schultheis
|
df2e0dadd2
|
int64_t
|
2024-05-24 19:59:53 +03:00 |
|
|
Andrej Karpathy
|
1d10e788af
|
fix bug pretty sure this should just be a void* pointer
|
2024-05-24 16:51:26 +00:00 |
|
|
Andrej Karpathy
|
613a125216
|
add checkpoint function write to file
|
2024-05-24 16:49:20 +00:00 |
|
|
Erik Schultheis
|
2a736cb9e2
|
fix for large batch sizes
|
2024-05-24 19:45:41 +03:00 |
|
|
Andrej Karpathy
|
dee4e42548
|
add option to not run hellaswag, interferes with a bunch of testing, e.g. if T is low
|
2024-05-24 03:47:07 +00:00 |
|
|
Andrej Karpathy
|
032e76c259
|
start putting llm.c and pytorch right next to each other, identical training runs with identical results and prints. almost
|
2024-05-24 01:02:21 +00:00 |
|
|
Andrej Karpathy
|
64b6a14679
|
add learning rate decay schedule, now we have the full scheduler implemented
|
2024-05-23 21:19:00 +00:00 |
|
|
Andrej Karpathy
|
661975cc3c
|
add learning rate warmup option
|
2024-05-23 20:57:04 +00:00 |
|
|
Andrej Karpathy
|
645869b6f7
|
add weight decay -c option and be more careful in our tests of correctness, our weight decay didn't match to pytorch. also modify the betas in AdamW to be consistent with those used in GPT-3 training
|
2024-05-23 20:45:59 +00:00 |
|
|
Andrej Karpathy
|
949d71a3d2
|
only rank 0 logs
|
2024-05-23 20:32:56 +00:00 |
|
|
Andrej Karpathy
|
3cb2812774
|
skip hellaswag eval on step 0 i think... not sure but ok for now
|
2024-05-23 19:27:04 +00:00 |
|
|
Andrej Karpathy
|
1f91bfc442
|
fix small bug on eval logging
|
2024-05-23 18:19:25 +00:00 |
|
|
Andrej Karpathy
|
86682af9a9
|
llm.c matches pytorch init from scratch exactly now
|
2024-05-23 17:50:33 +00:00 |
|
|
Andrej Karpathy
|
70a9c75348
|
use pytorch rand and fix dumb bug lol
|
2024-05-23 15:16:49 +00:00 |
|
|
Andrej Karpathy
|
d09631807a
|
first draft of random init, crashes with some cuBLAS error, debugging
|
2024-05-23 14:35:46 +00:00 |
|
|
Andrej Karpathy
|
d3cf9e211c
|
fix bug free only if initted
|
2024-05-23 00:49:32 +00:00 |
|
|
Andrej Karpathy
|
da59861d38
|
make hellaswag optional eval yay
|
2024-05-22 22:42:37 +00:00 |
|
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
|
ademeure
|
7cbeefc7f3
|
added new layernorm backward to /dev/cuda/
|
2024-05-21 23:26:54 +01:00 |
|
|
ademeure
|
7d0891f6dd
|
Fully deterministic layernorm (slight perf loss)
|
2024-05-21 22:37:09 +01:00 |
|
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|