 Andrej Karpathy
|
755458d0e1
|
fix tokenizer omg
|
2024-08-13 17:24:15 +00:00 |
|
|
Andrej Karpathy
|
16635d41a2
|
attempt to fix PR
|
2024-08-13 02:13:14 +00:00 |
|
|
Aleksa Gordic
|
68624cc6ac
|
Refactor tokenize, delegate instead
|
2024-08-12 20:26:21 +02:00 |
|
|
Aleksa Gordic
|
50765638c0
|
Add fineweb llama 3 support
|
2024-08-10 22:35:46 +02:00 |
|
|
Aleksa Gordic
|
d3151c4ace
|
Add llama 3 tokenization and data loader
|
2024-08-10 21:25:57 +02:00 |
|
|
Andrej Karpathy
|
ea36d04e39
|
small cosmetic fixes to fineweb data file
|
2024-06-30 15:52:51 +00:00 |
|
|
“eliebak”
|
67379a28cc
|
add edu fineweb support, with 10B and 1000B version
|
2024-06-02 14:18:44 +00:00 |
|
|
Andrej Karpathy
|
d0c014372c
|
add ability to export fineweb100B as well
|
2024-05-25 17:21:51 +00:00 |
|
|
Andrej Karpathy
|
08fe3eef43
|
change default shard size for fineweb10B
|
2024-05-24 15:56:05 +00:00 |
|
|
Erik Schultheis
|
b66eb66ac7
|
update fine-web preprocessing: faster, and much less RAM consumption
|
2024-05-24 17:45:26 +03:00 |
|
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|
|
Andrej Karpathy
|
1defbd4a19
|
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
|
2024-05-21 17:05:27 +00:00 |
|
|
Andrej Karpathy
|
4bbd01fe12
|
add fineweb, and add the first version of a new write_shard function that contains a header properly
|
2024-05-21 00:16:07 +00:00 |
|