 Andrej Karpathy
|
2677d82dcb
|
add shell script that downloads the .bin files of FineWeb100B directly
|
2024-06-20 22:39:09 +00:00 |
|
|
Andrej Karpathy
|
d0c014372c
|
add ability to export fineweb100B as well
|
2024-05-25 17:21:51 +00:00 |
|
|
Andrej Karpathy
|
08fe3eef43
|
change default shard size for fineweb10B
|
2024-05-24 15:56:05 +00:00 |
|
|
Erik Schultheis
|
b66eb66ac7
|
update fine-web preprocessing: faster, and much less RAM consumption
|
2024-05-24 17:45:26 +03:00 |
|
|
Andrej Karpathy
|
da59861d38
|
make hellaswag optional eval yay
|
2024-05-22 22:42:37 +00:00 |
|
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
|
Andrej Karpathy
|
31310282e1
|
extend dataloader to be sharded
|
2024-05-21 21:33:24 +00:00 |
|
|
Andrej Karpathy
|
d53608820a
|
the write_datafile function accepts python list, which i think is faster but i didn't check
|
2024-05-21 17:55:56 +00:00 |
|
|
Andrej Karpathy
|
1defbd4a19
|
adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead
|
2024-05-21 17:05:27 +00:00 |
|
|
Andrej Karpathy
|
4bbd01fe12
|
add fineweb, and add the first version of a new write_shard function that contains a header properly
|
2024-05-21 00:16:07 +00:00 |
|
|
Andrej Karpathy
|
7d11b7996c
|
i also forgot to include the readme file for the new dev/data dir
|
2024-05-20 23:13:42 +00:00 |
|
|
Andrej Karpathy
|
71774b3c3d
|
oops forgot to include data_common.py
|
2024-05-20 23:10:49 +00:00 |
|
|
Andrej Karpathy
|
722e5b2fe5
|
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
|
2024-05-20 22:43:40 +00:00 |
|