 Andrej Karpathy
|
a13b2bd10b
|
fix bug check null
|
2024-05-25 13:50:56 +00:00 |
|
|
Andrej Karpathy
|
f2ee356a06
|
resolve merge conflict
|
2024-05-25 00:33:12 +00:00 |
|
|
Andrej Karpathy
|
9f08882051
|
add weight decay, but only for 2D tensors, as done in GPT series and in general too. this forces us to break up our adamw kernel again into one call per tensor, so there is a small throughput hit, of about 0.5% for me. but we have to break up this kernel in near future anyway
|
2024-05-25 00:14:10 +00:00 |
|
|
Andrej Karpathy
|
8b951056b5
|
resolve merge conflict
|
2024-05-24 22:04:02 +00:00 |
|
|
Andrej Karpathy
|
25f17e6748
|
small formatting fix before merge
|
2024-05-24 21:53:00 +00:00 |
|
|
Andrej Karpathy
|
dbacaf84cf
|
Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm
|
2024-05-24 21:46:25 +00:00 |
|
|
Andrej Karpathy
|
16b364d256
|
change -o to be not file but directory, and create a main.log inside it for the log. this is to prepare for writing of checkpoints, where we want to re-use this directory for those as well
|
2024-05-24 17:57:50 +00:00 |
|
|
Erik Schultheis
|
df2e0dadd2
|
int64_t
|
2024-05-24 19:59:53 +03:00 |
|
|
Andrej Karpathy
|
1d10e788af
|
fix bug pretty sure this should just be a void* pointer
|
2024-05-24 16:51:26 +00:00 |
|
|
Andrej Karpathy
|
613a125216
|
add checkpoint function write to file
|
2024-05-24 16:49:20 +00:00 |
|
|
Erik Schultheis
|
2a736cb9e2
|
fix for large batch sizes
|
2024-05-24 19:45:41 +03:00 |
|
|
Andrej Karpathy
|
dee4e42548
|
add option to not run hellaswag, interferes with a bunch of testing, e.g. if T is low
|
2024-05-24 03:47:07 +00:00 |
|
|
Andrej Karpathy
|
032e76c259
|
start putting llm.c and pytorch right next to each other, identical training runs with identical results and prints. almost
|
2024-05-24 01:02:21 +00:00 |
|
|
Andrej Karpathy
|
64b6a14679
|
add learning rate decay schedule, now we have the full scheduler implemented
|
2024-05-23 21:19:00 +00:00 |
|
|
Andrej Karpathy
|
661975cc3c
|
add learning rate warmup option
|
2024-05-23 20:57:04 +00:00 |
|
|
Andrej Karpathy
|
645869b6f7
|
add weight decay -c option and be more careful in our tests of correctness, our weight decay didn't match to pytorch. also modify the betas in AdamW to be consistent with those used in GPT-3 training
|
2024-05-23 20:45:59 +00:00 |
|
|
Andrej Karpathy
|
949d71a3d2
|
only rank 0 logs
|
2024-05-23 20:32:56 +00:00 |
|
|
Andrej Karpathy
|
3cb2812774
|
skip hellaswag eval on step 0 i think... not sure but ok for now
|
2024-05-23 19:27:04 +00:00 |
|
|
Andrej Karpathy
|
1f91bfc442
|
fix small bug on eval logging
|
2024-05-23 18:19:25 +00:00 |
|
|
Andrej Karpathy
|
86682af9a9
|
llm.c matches pytorch init from scratch exactly now
|
2024-05-23 17:50:33 +00:00 |
|
|
Andrej Karpathy
|
70a9c75348
|
use pytorch rand and fix dumb bug lol
|
2024-05-23 15:16:49 +00:00 |
|
|
Andrej Karpathy
|
d09631807a
|
first draft of random init, crashes with some cuBLAS error, debugging
|
2024-05-23 14:35:46 +00:00 |
|
|
Andrej Karpathy
|
d3cf9e211c
|
fix bug free only if initted
|
2024-05-23 00:49:32 +00:00 |
|
|
Andrej Karpathy
|
da59861d38
|
make hellaswag optional eval yay
|
2024-05-22 22:42:37 +00:00 |
|
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
|
ademeure
|
7cbeefc7f3
|
added new layernorm backward to /dev/cuda/
|
2024-05-21 23:26:54 +01:00 |
|
|
ademeure
|
7d0891f6dd
|
Fully deterministic layernorm (slight perf loss)
|
2024-05-21 22:37:09 +01:00 |
|
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|
|
ademeure
|
a3801f01ef
|
added algorithm header for std::sort on windows (not sure about compile time impact...)
|
2024-05-21 16:53:11 +01:00 |
|
|
Andrej Karpathy
|
f7cbb30316
|
step 2 of dataloader refactor: separate out the dataloader to its own file, change its signature a little bit, and (notably) change from cudaMallocHost to a simple malloc, so that this file doesn't have to be cuda aware
|
2024-05-21 15:32:46 +00:00 |
|
|
ademeure
|
b5e75dde8e
|
Fully deterministic encoder backward kernels for train_gpt2.cu
|
2024-05-21 15:57:07 +01:00 |
|
|
Andrej Karpathy
|
f671cf9288
|
more changes, trying to help people out because when this merges to master it will brick everyone's code...
|
2024-05-20 23:02:41 +00:00 |
|
|
Andrej Karpathy
|
722e5b2fe5
|
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
|
2024-05-20 22:43:40 +00:00 |
|
|
Andrej
|
6c8bc172cc
|
Merge pull request #436 from ChrisDryden/boundscheck
Moved bounds checks to outside of the kernel
|
2024-05-20 00:41:41 +01:00 |
|
|
Andrej
|
2751fa0880
|
Merge pull request #435 from ChrisDryden/warpsize
Added warpsize as a constant for better compile time optimization and standardization
|
2024-05-20 00:39:15 +01:00 |
|
|
Christopher
|
6de1137e0e
|
Moved bounds checks outside of kernel into assertions
|
2024-05-19 23:17:45 +00:00 |
|
|
Christopher
|
ead5d35973
|
Added warpsize as a constant for better compile time optimization and standardization
|
2024-05-19 22:54:47 +00:00 |
|
|
Andrej Karpathy
|
bc58cd1dc1
|
fix small bugs in grad clip, introduce a GPU CPU synch point to communicate the float grad_clip to print it, and small printing changes
|
2024-05-19 17:51:29 +00:00 |
|
|
Andrej Karpathy
|
c2d12f725e
|
small touchups to grad clip
|
2024-05-19 17:07:55 +00:00 |
|
|
ngc92
|
589ead1e4f
|
updated code to adapt to latest changes
|
2024-05-18 22:47:42 +03:00 |
|
|
Erik Schultheis
|
a9947a8315
|
added a flag and the missing sqrt;
testing now has clipping enabled
|
2024-05-18 22:47:15 +03:00 |
|
|
Erik Schultheis
|
d7a81ef26f
|
added a useful mixed precision utility for dev/cuda
|
2024-05-18 22:45:44 +03:00 |
|
|
Erik Schultheis
|
44d45bdd6a
|
first draft for gradient clipping by global norm
|
2024-05-18 22:45:21 +03:00 |
|
|
Andrej Karpathy
|
c8fa7a8c63
|
revert the adamw allocation to previous. minor
|
2024-05-17 15:13:36 +00:00 |
|
|
Erik Schultheis
|
b24279c4db
|
remove duplicate workspace allocation
|
2024-05-17 17:37:47 +03:00 |
|
|
Erik Schultheis
|
88c3bea890
|
print message before actual allocation for more informative OOM behaviour
|
2024-05-17 17:37:45 +03:00 |
|
|
Erik Schultheis
|
0f23723ae4
|
joined optimizer state allocation
|
2024-05-17 17:37:16 +03:00 |
|
|
Andrej Karpathy
|
130712215d
|
Merge branch 'backward-bias' of https://github.com/ngc92/llm.c
|
2024-05-16 20:00:56 +00:00 |
|