Andrej Karpathy
|
938f8f78bb
|
more defensive
|
2024-04-29 20:25:29 +00:00 |
|
Andrej Karpathy
|
b4c346ac2a
|
don't hardcode the EOT token and store it in the Tokenizer file
|
2024-04-29 20:23:41 +00:00 |
|
Andrej Karpathy
|
699c25469a
|
separate out our Tokenizer
|
2024-04-29 19:54:18 +00:00 |
|