dbacaf84cf
Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm
2024-05-24 21:46:25 +00:00
bf03e7fd4c
Merge pull request #456 from ngc92/large-batch-fix
...
fix for large batch sizes
2024-05-24 12:00:41 -07:00
1b98637960
int -> int64_t
2024-05-24 20:11:34 +03:00
08fe3eef43
change default shard size for fineweb10B
2024-05-24 15:56:05 +00:00
b66eb66ac7
update fine-web preprocessing: faster, and much less RAM consumption
2024-05-24 17:45:26 +03:00
da59861d38
make hellaswag optional eval yay
2024-05-22 22:42:37 +00:00
67239d9b8f
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
2024-05-22 22:21:53 +00:00
051f3ca53c
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
2024-05-22 19:24:20 +00:00
edb0df967a
continued changes for sharded dataloader
2024-05-22 11:59:09 +00:00
7cbeefc7f3
added new layernorm backward to /dev/cuda/
2024-05-21 23:26:54 +01:00
31310282e1
extend dataloader to be sharded
2024-05-21 21:33:24 +00:00
d53608820a
the write_datafile function accepts python list, which i think is faster but i didn't check
2024-05-21 17:55:56 +00:00
1defbd4a19
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
2024-05-21 17:05:27 +00:00
4bbd01fe12
add fineweb, and add the first version of a new write_shard function that contains a header properly
2024-05-21 00:16:07 +00:00
7d11b7996c
i also forgot to include the readme file for the new dev/data dir
2024-05-20 23:13:42 +00:00
71774b3c3d
oops forgot to include data_common.py
2024-05-20 23:10:49 +00:00
722e5b2fe5
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
2024-05-20 22:43:40 +00:00
c2d12f725e
small touchups to grad clip
2024-05-19 17:07:55 +00:00
da9f4e21b0
Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clipping
2024-05-19 16:03:53 +00:00
8d55c4a6ac
fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracy
2024-05-19 15:45:28 +00:00
4e14b52285
report both acc and acc_norm
2024-05-19 15:18:04 +00:00
7bd2389c1f
add mmlu as well and refine both a bit
2024-05-19 14:23:51 +00:00
aec9ce5d0a
move hellaswag file to dev
2024-05-19 12:18:02 +00:00
66ce5766e0
fixed up dev/cuda
2024-05-18 23:06:44 +03:00
d7a81ef26f
added a useful mixed precision utility for dev/cuda
2024-05-18 22:45:44 +03:00
109f516367
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs
...
Free cooperative groups implementation of online softmax forward
2024-05-16 21:23:56 +01:00
9e04deabe3
Merge pull request #416 from azret/feature/cpu_dev
...
Setting up dev/cpu area with the first matmul_forward.c
2024-05-16 21:12:55 +01:00
858c6e6dae
deterministic kernel
2024-05-16 01:04:10 +03:00
2ccdfb70e0
general cleanup
2024-05-16 01:04:10 +03:00
160b3bd007
Setting up dev/CPU area with the first matmul_forward.c
2024-05-14 18:22:39 -07:00
2d43e5bc97
remove legacy comment
2024-05-14 19:14:07 +00:00
222d59fa2f
Merge pull request #408 from ngc92/layernorm-bw-dev-cuda
...
Layernorm backward updates
2024-05-14 20:13:09 +01:00
e553e2f084
update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results
2024-05-14 20:44:50 +03:00
dd8c9f5ec9
fix layernorm backward: accumulate weight gradient
2024-05-14 20:43:51 +03:00
c66e48c06c
fixup comment
2024-05-13 20:58:07 +03:00
65727d5a4d
fix CI compile by disabling kernel 5
2024-05-13 19:20:03 +03:00
49ee3c8307
fix non-atomic version:
...
* accumulate instead of assign
* need dedicated argument to correctly handle the floatX == float case
2024-05-13 18:27:56 +03:00
081d224b21
automatically switch to buffer-less version if that can fill up the GPU
2024-05-13 17:39:32 +03:00
c0329ebdba
new kernel version with fewer atomics
2024-05-13 17:18:27 +03:00
2287da0120
enable bf16
2024-05-12 19:42:32 +03:00
ec92368f8b
added current backward bias kernel to dev cuda
2024-05-12 19:24:43 +03:00
5c90845f7a
update the -lcublas -lcublasLt flag in the comment
2024-05-09 22:04:02 -07:00
43c48707a7
Merge branch 'fuse-ln-res-2' of https://github.com/ngc92/llm.c into ngc92-fuse-ln-res-2
2024-05-09 23:09:53 +00:00
75ec629f52
remove duplicate function
2024-05-10 00:03:30 +02:00
5920143d47
remove warning noise
2024-05-10 00:02:37 +02:00
691c1df969
fused layernorm+residual
2024-05-09 23:48:11 +02:00
99e765de8d
moved bf16 boilerplate to common.h
2024-05-09 11:19:24 +02:00
69aa64cc80
Adds cuDNN install instructions for a smaller but sufficient package
...
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).
- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)
- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions
Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com >
2024-05-08 14:53:31 -07:00
6d7a99cb4e
Adjust comments
2024-05-08 17:45:14 +08:00
6a52d8619e
Optimize codes and comments
2024-05-08 16:12:39 +08:00
Andrej Karpathy