 Andrej Karpathy
|
755458d0e1
|
fix tokenizer omg
|
2024-08-13 17:24:15 +00:00 |
|
|
Andrej Karpathy
|
16635d41a2
|
attempt to fix PR
|
2024-08-13 02:13:14 +00:00 |
|
|
Aleksa Gordic
|
68624cc6ac
|
Refactor tokenize, delegate instead
|
2024-08-12 20:26:21 +02:00 |
|
|
Aleksa Gordic
|
e8739c89a0
|
Refactor magic/version header dict
|
2024-08-12 20:17:53 +02:00 |
|
|
Aleksa Gordic
|
eb1b359dae
|
Replace fire with argparse
|
2024-08-10 22:45:46 +02:00 |
|
|
Aleksa Gordic
|
50765638c0
|
Add fineweb llama 3 support
|
2024-08-10 22:35:46 +02:00 |
|
|
Aleksa Gordic
|
f8c1ff590f
|
Use fire module, update readme
|
2024-08-10 22:29:19 +02:00 |
|
|
Aleksa Gordic
|
15b76bf0fb
|
Add tiny shakespeare
|
2024-08-10 22:23:33 +02:00 |
|
|
Aleksa Gordic
|
71419cbd61
|
Update header description
|
2024-08-10 21:47:09 +02:00 |
|
|
Aleksa Gordic
|
d3151c4ace
|
Add llama 3 tokenization and data loader
|
2024-08-10 21:25:57 +02:00 |
|
|
Andrej Karpathy
|
48fee049cb
|
add edu fineweb sh download script, change paths to my HF repo, add hellaswag to starter pack
|
2024-07-01 21:41:15 +00:00 |
|
|
Andrej Karpathy
|
ea36d04e39
|
small cosmetic fixes to fineweb data file
|
2024-06-30 15:52:51 +00:00 |
|
|
Andrej Karpathy
|
ff7a90bd8d
|
Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_support
|
2024-06-28 19:35:44 +00:00 |
|
|
Andrej Karpathy
|
2677d82dcb
|
add shell script that downloads the .bin files of FineWeb100B directly
|
2024-06-20 22:39:09 +00:00 |
|
|
“eliebak”
|
67379a28cc
|
add edu fineweb support, with 10B and 1000B version
|
2024-06-02 14:18:44 +00:00 |
|
|
Andrej Karpathy
|
d0c014372c
|
add ability to export fineweb100B as well
|
2024-05-25 17:21:51 +00:00 |
|
|
Andrej Karpathy
|
08fe3eef43
|
change default shard size for fineweb10B
|
2024-05-24 15:56:05 +00:00 |
|
|
Erik Schultheis
|
b66eb66ac7
|
update fine-web preprocessing: faster, and much less RAM consumption
|
2024-05-24 17:45:26 +03:00 |
|
|
Andrej Karpathy
|
da59861d38
|
make hellaswag optional eval yay
|
2024-05-22 22:42:37 +00:00 |
|
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|
|
Andrej Karpathy
|
d53608820a
|
the write_datafile function accepts python list, which i think is faster but i didn't check
|
2024-05-21 17:55:56 +00:00 |
|
|
Andrej Karpathy
|
1defbd4a19
|
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
|
2024-05-21 17:05:27 +00:00 |
|
|
Andrej Karpathy
|
4bbd01fe12
|
add fineweb, and add the first version of a new write_shard function that contains a header properly
|
2024-05-21 00:16:07 +00:00 |
|
|
Andrej Karpathy
|
7d11b7996c
|
i also forgot to include the readme file for the new dev/data dir
|
2024-05-20 23:13:42 +00:00 |
|
|
Andrej Karpathy
|
71774b3c3d
|
oops forgot to include data_common.py
|
2024-05-20 23:10:49 +00:00 |
|
|
Andrej Karpathy
|
722e5b2fe5
|
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
|
2024-05-20 22:43:40 +00:00 |
|