9f08882051
add weight decay, but only for 2D tensors, as done in GPT series and in general too. this forces us to break up our adamw kernel again into one call per tensor, so there is a small throughput hit, of about 0.5% for me. but we have to break up this kernel in near future anyway
2024-05-25 00:14:10 +00:00
e60c4843a3
Merge branch 'ngc92-cpu-matmul'
2024-05-24 23:10:34 +00:00
3221e4b2d2
small cosmetic changes
2024-05-24 23:10:17 +00:00
6e4296fb9a
Merge branch 'cpu-matmul' of https://github.com/ngc92/llm.c into ngc92-cpu-matmul
2024-05-24 22:59:24 +00:00
b8674cedf8
Merge branch 'ademeure-deterministic_encoder_v2'
2024-05-24 22:18:26 +00:00
8b951056b5
resolve merge conflict
2024-05-24 22:04:02 +00:00
5ddb06182c
Merge branch 'ademeure-deterministic_layernorm'
2024-05-24 21:53:08 +00:00
25f17e6748
small formatting fix before merge
2024-05-24 21:53:00 +00:00
dbacaf84cf
Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm
2024-05-24 21:46:25 +00:00
bf03e7fd4c
Merge pull request #456 from ngc92/large-batch-fix
...
fix for large batch sizes
2024-05-24 12:00:41 -07:00
1b98637960
int -> int64_t
2024-05-24 20:11:34 +03:00
df2e0dadd2
int64_t
2024-05-24 19:59:53 +03:00
2a736cb9e2
fix for large batch sizes
2024-05-24 19:45:41 +03:00
08fe3eef43
change default shard size for fineweb10B
2024-05-24 15:56:05 +00:00
b0f065fcf8
Merge pull request #455 from ngc92/fineweb
...
update fine-web preprocessing: faster, and much less RAM consumption
2024-05-24 08:54:55 -07:00
b66eb66ac7
update fine-web preprocessing: faster, and much less RAM consumption
2024-05-24 17:45:26 +03:00
dee4e42548
add option to not run hellaswag, interferes with a bunch of testing, e.g. if T is low
2024-05-24 03:47:07 +00:00
032e76c259
start putting llm.c and pytorch right next to each other, identical training runs with identical results and prints. almost
2024-05-24 01:02:21 +00:00
4b88d2a976
Merge pull request #453 from otabuzzman/master
...
A Swift port of this project
2024-05-23 14:26:28 -07:00
64b6a14679
add learning rate decay schedule, now we have the full scheduler implemented
2024-05-23 21:19:00 +00:00
661975cc3c
add learning rate warmup option
2024-05-23 20:57:04 +00:00
645869b6f7
add weight decay -c option and be more careful in our tests of correctness, our weight decay didn't match to pytorch. also modify the betas in AdamW to be consistent with those used in GPT-3 training
2024-05-23 20:45:59 +00:00
949d71a3d2
only rank 0 logs
2024-05-23 20:32:56 +00:00
3cb2812774
skip hellaswag eval on step 0 i think... not sure but ok for now
2024-05-23 19:27:04 +00:00
8af1818cb1
Merge branch 'karpathy:master' into master
2024-05-23 21:18:01 +02:00
5f87b13f34
Update documentation with Swift port reference
2024-05-23 21:15:01 +02:00
518805b7f4
Init from random weights, matching PyTorch
...
Merge pull request #451 from karpathy/feature/random_init
2024-05-23 12:11:05 -07:00
1f91bfc442
fix small bug on eval logging
2024-05-23 18:19:25 +00:00
86682af9a9
llm.c matches pytorch init from scratch exactly now
2024-05-23 17:50:33 +00:00
70a9c75348
use pytorch rand and fix dumb bug lol
2024-05-23 15:16:49 +00:00
bc1ebc110a
Merge branch 'feature/rand' of https://github.com/azret/llm.c into feature/random_init
2024-05-23 14:59:44 +00:00
e6a7d1d3e9
allow the python script to also init from random and save those weights, so it's a good reference for our C implementation
2024-05-23 14:59:25 +00:00
d09631807a
first draft of random init, crashes with some cuBLAS error, debugging
2024-05-23 14:35:46 +00:00
d3cf9e211c
fix bug free only if initted
2024-05-23 00:49:32 +00:00
eda0c2f591
Merge pull request #447 from karpathy/feature/hellaswagc
...
HellaSwag eval in C
2024-05-22 15:50:12 -07:00
da59861d38
make hellaswag optional eval yay
2024-05-22 22:42:37 +00:00
67239d9b8f
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
2024-05-22 22:21:53 +00:00
051f3ca53c
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
2024-05-22 19:24:20 +00:00
69f12214bf
Merge pull request #444 from karpathy/feature/sharded_data
...
extend dataloader to be sharded
2024-05-22 06:21:48 -07:00
099d30f814
add a super small crappy glob for windows that only matches a single unique file. this will make CI happy but we can't train on sharded data on windows until this is improved
2024-05-22 13:08:18 +00:00
05be4f6d82
readme changes
2024-05-22 12:32:25 +00:00
edb0df967a
continued changes for sharded dataloader
2024-05-22 11:59:09 +00:00
7cbeefc7f3
added new layernorm backward to /dev/cuda/
2024-05-21 23:26:54 +01:00
7d0891f6dd
Fully deterministic layernorm (slight perf loss)
2024-05-21 22:37:09 +01:00
31310282e1
extend dataloader to be sharded
2024-05-21 21:33:24 +00:00
967420d1d1
fix print format warning for size_t vs int
2024-05-21 19:30:02 +00:00
587506d096
torch tensor can't handle uint16 so let's convert to int32, which is silly because we'll convert to .long right after but ok
2024-05-21 19:22:09 +00:00
170c92b771
Merge pull request #440 from karpathy/feature/organize_data
...
refactor datasets
you'll have to re-generate the dataset .bin files, for example, `python dev/data/tinyshakespeare.py`.
see main README section "major changes log" for more details.
2024-05-21 11:32:38 -07:00
54ccbd300c
docs on master-breaking changes around dataset file representation
2024-05-21 18:31:19 +00:00
7d58fd2abb
adjust py file as well and make the errors better
2024-05-21 18:29:55 +00:00
Andrej Karpathy