 Andrej Karpathy
|
54ccbd300c
|
docs on master-breaking changes around dataset file representation
|
2024-05-21 18:31:19 +00:00 |
|
|
Andrej Karpathy
|
7d58fd2abb
|
adjust py file as well and make the errors better
|
2024-05-21 18:29:55 +00:00 |
|
|
Andrej Karpathy
|
f7cb77f3d9
|
docs on master-breaking change around how we store data .bin files
|
2024-05-21 18:20:51 +00:00 |
|
|
Andrej Karpathy
|
ccc240caab
|
make comment more helpful
|
2024-05-21 18:01:35 +00:00 |
|
|
Andrej Karpathy
|
d53608820a
|
the write_datafile function accepts python list, which i think is faster but i didn't check
|
2024-05-21 17:55:56 +00:00 |
|
|
Andrej Karpathy
|
9bad49a99a
|
also fix the python file. that should be it now, w.r.t. the new token format .bin files
|
2024-05-21 17:46:14 +00:00 |
|
|
Andrej Karpathy
|
666145e7b8
|
also fix the .c file and fp32 file
|
2024-05-21 17:30:56 +00:00 |
|
|
Andrej Karpathy
|
1defbd4a19
|
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
|
2024-05-21 17:05:27 +00:00 |
|
|
Andrej Karpathy
|
f7cbb30316
|
step 2 of dataloader refactor: separate out the dataloader to its own file, change its signature a little bit, and (notably) change from cudaMallocHost to a simple malloc, so that this file doesn't have to be cuda aware
|
2024-05-21 15:32:46 +00:00 |
|
|
Andrej Karpathy
|
4bbd01fe12
|
add fineweb, and add the first version of a new write_shard function that contains a header properly
|
2024-05-21 00:16:07 +00:00 |
|
|
Andrej Karpathy
|
c51cd70cfb
|
fix a slip in gitignore, i think i am getting tired today
|
2024-05-20 23:22:31 +00:00 |
|
|
Andrej Karpathy
|
bced34d71e
|
adjust gitignore
|
2024-05-20 23:15:44 +00:00 |
|
|
Andrej Karpathy
|
7d11b7996c
|
i also forgot to include the readme file for the new dev/data dir
|
2024-05-20 23:13:42 +00:00 |
|
|
Andrej Karpathy
|
71774b3c3d
|
oops forgot to include data_common.py
|
2024-05-20 23:10:49 +00:00 |
|
|
Andrej Karpathy
|
f671cf9288
|
more changes, trying to help people out because when this merges to master it will brick everyone's code...
|
2024-05-20 23:02:41 +00:00 |
|
|
Andrej Karpathy
|
722e5b2fe5
|
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
|
2024-05-20 22:43:40 +00:00 |
|
|
Andrej
|
6c8bc172cc
|
Merge pull request #436 from ChrisDryden/boundscheck
Moved bounds checks to outside of the kernel
|
2024-05-20 00:41:41 +01:00 |
|
|
Andrej
|
2751fa0880
|
Merge pull request #435 from ChrisDryden/warpsize
Added warpsize as a constant for better compile time optimization and standardization
|
2024-05-20 00:39:15 +01:00 |
|
|
Christopher
|
6de1137e0e
|
Moved bounds checks outside of kernel into assertions
|
2024-05-19 23:17:45 +00:00 |
|
|
Christopher
|
ead5d35973
|
Added warpsize as a constant for better compile time optimization and standardization
|
2024-05-19 22:54:47 +00:00 |
|
|
Andrej
|
1809740a1c
|
Merge pull request #433 from junaire/jun/some_fix
Check if file exists using platform specific commands & Add *.o to gi…
|
2024-05-19 21:02:04 +01:00 |
|
|
Andrej
|
00c291c679
|
Merge pull request #434 from ngc92/profile-fix
initialize multi_gpu_config so profile doesn't crash
|
2024-05-19 20:57:11 +01:00 |
|
|
Erik Schultheis
|
299ce659ec
|
initialize multi_gpu_config so profile doesn't crash
|
2024-05-19 22:25:00 +03:00 |
|
|
Andrej Karpathy
|
5a26799efb
|
Merge branch 'ngc92-gradient-clipping'
|
2024-05-19 17:51:44 +00:00 |
|
|
Andrej Karpathy
|
bc58cd1dc1
|
fix small bugs in grad clip, introduce a GPU CPU synch point to communicate the float grad_clip to print it, and small printing changes
|
2024-05-19 17:51:29 +00:00 |
|
|
Andrej Karpathy
|
c2d12f725e
|
small touchups to grad clip
|
2024-05-19 17:07:55 +00:00 |
|
|
Andrej Karpathy
|
da9f4e21b0
|
Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clipping
|
2024-05-19 16:03:53 +00:00 |
|
|
Jun Zhang
|
9e645314e6
|
Check if file exists using platform specific commands & Add *.o to gitignore
Signed-off-by: Jun Zhang <jun@junz.org>
|
2024-05-20 00:00:21 +08:00 |
|
|
Andrej
|
4a899f5685
|
Merge pull request #431 from karpathy/feature/hellaswag
add hellaswag and mmlu PyTorch reference eval
|
2024-05-19 16:53:38 +01:00 |
|
|
Andrej Karpathy
|
8d55c4a6ac
|
fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracy
|
2024-05-19 15:45:28 +00:00 |
|
|
Andrej Karpathy
|
4e14b52285
|
report both acc and acc_norm
|
2024-05-19 15:18:04 +00:00 |
|
|
Andrej Karpathy
|
7bd2389c1f
|
add mmlu as well and refine both a bit
|
2024-05-19 14:23:51 +00:00 |
|
|
Andrej Karpathy
|
aec9ce5d0a
|
move hellaswag file to dev
|
2024-05-19 12:18:02 +00:00 |
|
|
Andrej Karpathy
|
77b991281f
|
add hellaswag reference eval. it looks pretty bad honestly, the examples themselves, but i'm told it is predictive... hmm
|
2024-05-19 11:51:01 +00:00 |
|
|
Erik Schultheis
|
66ce5766e0
|
fixed up dev/cuda
|
2024-05-18 23:06:44 +03:00 |
|
|
ngc92
|
589ead1e4f
|
updated code to adapt to latest changes
|
2024-05-18 22:47:42 +03:00 |
|
|
Erik Schultheis
|
c3a3b9daa5
|
fixed profile target
|
2024-05-18 22:47:40 +03:00 |
|
|
Erik Schultheis
|
a9947a8315
|
added a flag and the missing sqrt;
testing now has clipping enabled
|
2024-05-18 22:47:15 +03:00 |
|
|
Erik Schultheis
|
d7a81ef26f
|
added a useful mixed precision utility for dev/cuda
|
2024-05-18 22:45:44 +03:00 |
|
|
Erik Schultheis
|
44d45bdd6a
|
first draft for gradient clipping by global norm
|
2024-05-18 22:45:21 +03:00 |
|
|
Andrej Karpathy
|
3de05baaf4
|
Merge branch 'ngc92-errchk'
|
2024-05-18 18:35:28 +00:00 |
|
|
Andrej Karpathy
|
4374360015
|
adjust wte upper bound a bit, and print always because this part is really tricky and i don't trust anything other than manual inspection, even if we pass, allegedly
|
2024-05-18 18:34:24 +00:00 |
|
|
Erik Schultheis
|
bf36a4b6b5
|
improved numerical error checking:
tighter tolarances
relative tolerance based of bf16 epsilon
less verbose output if all is OK
|
2024-05-18 13:27:35 +03:00 |
|
|
Andrej Karpathy
|
f226a2c9d9
|
Merge branch 'ngc92-alloc'
|
2024-05-17 15:13:48 +00:00 |
|
|
Andrej Karpathy
|
c8fa7a8c63
|
revert the adamw allocation to previous. minor
|
2024-05-17 15:13:36 +00:00 |
|
|
Erik Schultheis
|
b24279c4db
|
remove duplicate workspace allocation
|
2024-05-17 17:37:47 +03:00 |
|
|
Erik Schultheis
|
88c3bea890
|
print message before actual allocation for more informative OOM behaviour
|
2024-05-17 17:37:45 +03:00 |
|
|
Erik Schultheis
|
0f23723ae4
|
joined optimizer state allocation
|
2024-05-17 17:37:16 +03:00 |
|
|
Andrej
|
b554d4d093
|
Merge pull request #423 from anthonix/readme_link
Add link to AMD fork
|
2024-05-16 21:35:18 +01:00 |
|
|
Anthony Blake
|
3113656e3f
|
Add link to AMD fork
|
2024-05-16 13:33:34 -07:00 |
|