 Andrej Karpathy
|
05fb642ae1
|
attempt fix zero 1
|
2024-05-25 23:49:16 +00:00 |
|
|
Erik Schultheis
|
7ba3ef6bac
|
make sure to run Adam only for the range of parameters that is available on the local device
|
2024-05-25 23:20:42 +03:00 |
|
|
Andrej Karpathy
|
d0c014372c
|
add ability to export fineweb100B as well
|
2024-05-25 17:21:51 +00:00 |
|
|
Andrej Karpathy
|
f5d041ab0c
|
careful with NULL and checkpoint correctly
|
2024-05-25 16:32:18 +00:00 |
|
|
Andrej Karpathy
|
273925d10d
|
save checkpoint files to log directory, and small readability changes
|
2024-05-25 16:05:29 +00:00 |
|
|
Andrej Karpathy
|
90a745c40c
|
add MFU (model flops utilization) calculation and print it. also refactor the printing a bit to make it nice. this calculation only works for A100 fp16 we should think through how to generalize it a bit
|
2024-05-25 14:30:59 +00:00 |
|
|
Andrej
|
ab090a80c5
|
Merge pull request #461 from karpathy/feature/write_checkpoints
fix bug check null
|
2024-05-25 07:01:26 -07:00 |
|
|
Andrej Karpathy
|
a13b2bd10b
|
fix bug check null
|
2024-05-25 13:50:56 +00:00 |
|
|
Andrej
|
fe698b32f1
|
Merge pull request #457 from karpathy/feature/write_checkpoints
add checkpoint function write to file
|
2024-05-24 17:56:53 -07:00 |
|
|
Andrej Karpathy
|
2a0f78d617
|
attempt to fix windows episode 2
|
2024-05-25 00:44:16 +00:00 |
|
|
Andrej Karpathy
|
5b96d91893
|
attempt to fix windows?
|
2024-05-25 00:39:40 +00:00 |
|
|
Andrej Karpathy
|
f2ee356a06
|
resolve merge conflict
|
2024-05-25 00:33:12 +00:00 |
|
|
Andrej
|
4ff0412751
|
Merge pull request #450 from rosslwheeler/dataloader_win_fixes
Add glob() for windows
|
2024-05-24 17:30:35 -07:00 |
|
|
Andrej
|
6a7fd56d7c
|
Merge pull request #439 from lancerts/matmul-fix
Fix the unsupported block_size in matmul_backward_bias kernel 1
|
2024-05-24 17:26:37 -07:00 |
|
|
Andrej
|
5ef2c04a39
|
Merge pull request #458 from karpathy/feature/weight_decay_2d_only
add weight decay
|
2024-05-24 17:22:49 -07:00 |
|
|
Andrej Karpathy
|
9f08882051
|
add weight decay, but only for 2D tensors, as done in GPT series and in general too. this forces us to break up our adamw kernel again into one call per tensor, so there is a small throughput hit, of about 0.5% for me. but we have to break up this kernel in near future anyway
|
2024-05-25 00:14:10 +00:00 |
|
|
Ross Wheeler
|
79738d2ca4
|
fixed mkdir change
|
2024-05-24 17:05:59 -07:00 |
|
|
Ross Wheeler
|
e5083be900
|
Moved windows glob() over to dev/unistd.h
Added header guard and changed long->int64_t in dataloader.h
|
2024-05-24 16:56:52 -07:00 |
|
|
rosslwheeler
|
1ec081e64e
|
Merge branch 'karpathy:master' into dataloader_win_fixes
|
2024-05-24 16:19:45 -07:00 |
|
|
Andrej Karpathy
|
e60c4843a3
|
Merge branch 'ngc92-cpu-matmul'
|
2024-05-24 23:10:34 +00:00 |
|
|
Andrej Karpathy
|
3221e4b2d2
|
small cosmetic changes
|
2024-05-24 23:10:17 +00:00 |
|
|
Andrej Karpathy
|
6e4296fb9a
|
Merge branch 'cpu-matmul' of https://github.com/ngc92/llm.c into ngc92-cpu-matmul
|
2024-05-24 22:59:24 +00:00 |
|
|
rosslwheeler
|
5e0fa452e0
|
Merge branch 'karpathy:master' into dataloader_win_fixes
|
2024-05-24 15:48:45 -07:00 |
|
|
Andrej Karpathy
|
b8674cedf8
|
Merge branch 'ademeure-deterministic_encoder_v2'
|
2024-05-24 22:18:26 +00:00 |
|
|
Andrej Karpathy
|
8b951056b5
|
resolve merge conflict
|
2024-05-24 22:04:02 +00:00 |
|
|
Andrej Karpathy
|
5ddb06182c
|
Merge branch 'ademeure-deterministic_layernorm'
|
2024-05-24 21:53:08 +00:00 |
|
|
Andrej Karpathy
|
25f17e6748
|
small formatting fix before merge
|
2024-05-24 21:53:00 +00:00 |
|
|
Andrej Karpathy
|
dbacaf84cf
|
Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm
|
2024-05-24 21:46:25 +00:00 |
|
|
Andrej
|
bf03e7fd4c
|
Merge pull request #456 from ngc92/large-batch-fix
fix for large batch sizes
|
2024-05-24 12:00:41 -07:00 |
|
|
Andrej Karpathy
|
16b364d256
|
change -o to be not file but directory, and create a main.log inside it for the log. this is to prepare for writing of checkpoints, where we want to re-use this directory for those as well
|
2024-05-24 17:57:50 +00:00 |
|
|
Erik Schultheis
|
1b98637960
|
int -> int64_t
|
2024-05-24 20:11:34 +03:00 |
|
|
Erik Schultheis
|
df2e0dadd2
|
int64_t
|
2024-05-24 19:59:53 +03:00 |
|
|
Andrej Karpathy
|
1d10e788af
|
fix bug pretty sure this should just be a void* pointer
|
2024-05-24 16:51:26 +00:00 |
|
|
Andrej Karpathy
|
613a125216
|
add checkpoint function write to file
|
2024-05-24 16:49:20 +00:00 |
|
|
Erik Schultheis
|
2a736cb9e2
|
fix for large batch sizes
|
2024-05-24 19:45:41 +03:00 |
|
|
Andrej Karpathy
|
08fe3eef43
|
change default shard size for fineweb10B
|
2024-05-24 15:56:05 +00:00 |
|
|
Andrej
|
b0f065fcf8
|
Merge pull request #455 from ngc92/fineweb
update fine-web preprocessing: faster, and much less RAM consumption
|
2024-05-24 08:54:55 -07:00 |
|
|
Erik Schultheis
|
b66eb66ac7
|
update fine-web preprocessing: faster, and much less RAM consumption
|
2024-05-24 17:45:26 +03:00 |
|
|
rosslwheeler
|
a241a00ad4
|
Merge branch 'karpathy:master' into dataloader_win_fixes
|
2024-05-23 23:12:15 -07:00 |
|
|
Andrej Karpathy
|
dee4e42548
|
add option to not run hellaswag, interferes with a bunch of testing, e.g. if T is low
|
2024-05-24 03:47:07 +00:00 |
|
|
Andrej Karpathy
|
032e76c259
|
start putting llm.c and pytorch right next to each other, identical training runs with identical results and prints. almost
|
2024-05-24 01:02:21 +00:00 |
|
|
Andrej
|
4b88d2a976
|
Merge pull request #453 from otabuzzman/master
A Swift port of this project
|
2024-05-23 14:26:28 -07:00 |
|
|
Andrej Karpathy
|
64b6a14679
|
add learning rate decay schedule, now we have the full scheduler implemented
|
2024-05-23 21:19:00 +00:00 |
|
|
Andrej Karpathy
|
661975cc3c
|
add learning rate warmup option
|
2024-05-23 20:57:04 +00:00 |
|
|
Andrej Karpathy
|
645869b6f7
|
add weight decay -c option and be more careful in our tests of correctness, our weight decay didn't match to pytorch. also modify the betas in AdamW to be consistent with those used in GPT-3 training
|
2024-05-23 20:45:59 +00:00 |
|
|
Andrej Karpathy
|
949d71a3d2
|
only rank 0 logs
|
2024-05-23 20:32:56 +00:00 |
|
|
Andrej Karpathy
|
3cb2812774
|
skip hellaswag eval on step 0 i think... not sure but ok for now
|
2024-05-23 19:27:04 +00:00 |
|
|
Jürgen Schuck
|
8af1818cb1
|
Merge branch 'karpathy:master' into master
|
2024-05-23 21:18:01 +02:00 |
|
|
otabuzzman
|
5f87b13f34
|
Update documentation with Swift port reference
|
2024-05-23 21:15:01 +02:00 |
|
|
Andrej
|
518805b7f4
|
Init from random weights, matching PyTorch
Merge pull request #451 from karpathy/feature/random_init
|
2024-05-23 12:11:05 -07:00 |
|