 Ross Wheeler
|
bb80f93618
|
Minor cleanup
Added static inline to the functions
|
2024-05-27 13:16:34 -07:00 |
|
|
Ross Wheeler
|
758acdffbf
|
Adding dirent.h support to unistd.h
|
2024-05-27 13:08:07 -07:00 |
|
|
Erik Schultheis
|
7dc3b7d7dc
|
bugfix
|
2024-05-27 16:36:44 +03:00 |
|
|
Erik Schultheis
|
5f73ecfcf1
|
fix C % 256 != 0
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
d35daf19c9
|
fail fast and hard; don't go into the deadlock
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
3b6808269a
|
some comments and optimized shared memory amount
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
9eba86f0a5
|
fully vectorized smem access
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
cb8cc25e3a
|
utilities for Packed and more vectorization
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
53ee3297dd
|
use vectorized access to shared memory
|
2024-05-27 13:15:36 +03:00 |
|
|
Andrej Karpathy
|
ee6fa5ec9f
|
fill in the other model sizes
|
2024-05-26 18:04:43 +00:00 |
|
|
Andrej Karpathy
|
5ec4b774f2
|
simple jupyter notebook example on how to parse and vis the logfiles
|
2024-05-26 17:40:49 +00:00 |
|
|
Andrej Karpathy
|
d0c014372c
|
add ability to export fineweb100B as well
|
2024-05-25 17:21:51 +00:00 |
|
|
Andrej Karpathy
|
2a0f78d617
|
attempt to fix windows episode 2
|
2024-05-25 00:44:16 +00:00 |
|
|
Andrej Karpathy
|
5b96d91893
|
attempt to fix windows?
|
2024-05-25 00:39:40 +00:00 |
|
|
Andrej
|
4ff0412751
|
Merge pull request #450 from rosslwheeler/dataloader_win_fixes
Add glob() for windows
|
2024-05-24 17:30:35 -07:00 |
|
|
Andrej
|
6a7fd56d7c
|
Merge pull request #439 from lancerts/matmul-fix
Fix the unsupported block_size in matmul_backward_bias kernel 1
|
2024-05-24 17:26:37 -07:00 |
|
|
Ross Wheeler
|
79738d2ca4
|
fixed mkdir change
|
2024-05-24 17:05:59 -07:00 |
|
|
Ross Wheeler
|
e5083be900
|
Moved windows glob() over to dev/unistd.h
Added header guard and changed long->int64_t in dataloader.h
|
2024-05-24 16:56:52 -07:00 |
|
|
Andrej Karpathy
|
dbacaf84cf
|
Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm
|
2024-05-24 21:46:25 +00:00 |
|
|
Andrej
|
bf03e7fd4c
|
Merge pull request #456 from ngc92/large-batch-fix
fix for large batch sizes
|
2024-05-24 12:00:41 -07:00 |
|
|
Erik Schultheis
|
1b98637960
|
int -> int64_t
|
2024-05-24 20:11:34 +03:00 |
|
|
Andrej Karpathy
|
08fe3eef43
|
change default shard size for fineweb10B
|
2024-05-24 15:56:05 +00:00 |
|
|
Erik Schultheis
|
b66eb66ac7
|
update fine-web preprocessing: faster, and much less RAM consumption
|
2024-05-24 17:45:26 +03:00 |
|
|
Andrej Karpathy
|
da59861d38
|
make hellaswag optional eval yay
|
2024-05-22 22:42:37 +00:00 |
|
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
|
ademeure
|
7cbeefc7f3
|
added new layernorm backward to /dev/cuda/
|
2024-05-21 23:26:54 +01:00 |
|
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|
|
Andrej Karpathy
|
d53608820a
|
the write_datafile function accepts python list, which i think is faster but i didn't check
|
2024-05-21 17:55:56 +00:00 |
|
|
Andrej Karpathy
|
1defbd4a19
|
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
|
2024-05-21 17:05:27 +00:00 |
|
|
Andrej Karpathy
|
4bbd01fe12
|
add fineweb, and add the first version of a new write_shard function that contains a header properly
|
2024-05-21 00:16:07 +00:00 |
|
|
Andrej Karpathy
|
7d11b7996c
|
i also forgot to include the readme file for the new dev/data dir
|
2024-05-20 23:13:42 +00:00 |
|
|
Andrej Karpathy
|
71774b3c3d
|
oops forgot to include data_common.py
|
2024-05-20 23:10:49 +00:00 |
|
|
Andrej Karpathy
|
722e5b2fe5
|
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
|
2024-05-20 22:43:40 +00:00 |
|
|
lancer
|
2b0667aee1
|
update the utils function and assert
|
2024-05-20 08:00:39 -07:00 |
|
|
lancer
|
6348d4196d
|
fix the unsupported block_size
|
2024-05-19 17:39:25 -07:00 |
|
|
Andrej Karpathy
|
c2d12f725e
|
small touchups to grad clip
|
2024-05-19 17:07:55 +00:00 |
|
|
Andrej Karpathy
|
da9f4e21b0
|
Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clipping
|
2024-05-19 16:03:53 +00:00 |
|
|
Andrej Karpathy
|
8d55c4a6ac
|
fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracy
|
2024-05-19 15:45:28 +00:00 |
|
|
Andrej Karpathy
|
4e14b52285
|
report both acc and acc_norm
|
2024-05-19 15:18:04 +00:00 |
|
|
Andrej Karpathy
|
7bd2389c1f
|
add mmlu as well and refine both a bit
|
2024-05-19 14:23:51 +00:00 |
|
|
Andrej Karpathy
|
aec9ce5d0a
|
move hellaswag file to dev
|
2024-05-19 12:18:02 +00:00 |
|
|
Erik Schultheis
|
66ce5766e0
|
fixed up dev/cuda
|
2024-05-18 23:06:44 +03:00 |
|
|
Erik Schultheis
|
d7a81ef26f
|
added a useful mixed precision utility for dev/cuda
|
2024-05-18 22:45:44 +03:00 |
|
|
Andrej
|
109f516367
|
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs
Free cooperative groups implementation of online softmax forward
|
2024-05-16 21:23:56 +01:00 |
|
|
Andrej
|
9e04deabe3
|
Merge pull request #416 from azret/feature/cpu_dev
Setting up dev/cpu area with the first matmul_forward.c
|
2024-05-16 21:12:55 +01:00 |
|
|
Erik Schultheis
|
858c6e6dae
|
deterministic kernel
|
2024-05-16 01:04:10 +03:00 |
|
|
Erik Schultheis
|
2ccdfb70e0
|
general cleanup
|
2024-05-16 01:04:10 +03:00 |
|
|
Azret Botash
|
160b3bd007
|
Setting up dev/CPU area with the first matmul_forward.c
|
2024-05-14 18:22:39 -07:00 |
|