 Andrej Karpathy
|
9d70d9a1b2
|
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
|
2024-04-28 16:17:22 +00:00 |
|
|
Andrej Karpathy
|
a3f5ad9095
|
reshuffle the ifdefs to make bf16 the default if no PRECISION is requested via defines
|
2024-04-28 16:12:39 +00:00 |
|
|
Andrej Karpathy
|
4f7d8d9f44
|
allow user to make different precisions, add prints and error handling around precisions
|
2024-04-28 16:08:27 +00:00 |
|
|
Andrej Karpathy
|
82d7907214
|
adjust comment
|
2024-04-28 00:05:03 +00:00 |
|
|
Andrej Karpathy
|
9a91b40207
|
bring back original ordering. i also had to bump the thresholds by 3X for some tensors and i don't exactly know why sad
|
2024-04-28 00:02:51 +00:00 |
|
|
Andrej Karpathy
|
00627070aa
|
fix a really bad bug in how i was checking the gradients, where i loaded them in the old order, so yeah...
|
2024-04-27 23:17:22 +00:00 |
|
|
Andrej Karpathy
|
a58b8d5dc7
|
print more in the comparison
|
2024-04-27 16:04:55 +00:00 |
|
|
Andrej Karpathy
|
9d6fd30e3c
|
tweak the tolerances until we pass lol
|
2024-04-27 15:49:54 +00:00 |
|
|
Andrej Karpathy
|
e067a27395
|
fix dumb bug. i'll blame github copilot but i can't remember
|
2024-04-27 15:40:32 +00:00 |
|
|
Andrej Karpathy
|
d4a642ba19
|
i think github copilot betrayed me on this index here, i cant remember
|
2024-04-27 15:38:20 +00:00 |
|
|
Andrej Karpathy
|
09d935c82b
|
i think i am making things cleaner, but i am not fixing the problem
|
2024-04-27 15:27:05 +00:00 |
|
|
Andrej Karpathy
|
09cd67e868
|
code to load bf16 weights directly, and also re-wire the position of tensors to put the layernorms at the end. the training loop seems to work ok, and the tests pass and the loss and optimization looks ok, but the gradients don't match. which can't be right. so there is a bug, but it's a bit too late in the day for me to debug right now, creating a PR and going to sleep, will fix tomorrow
|
2024-04-27 00:54:06 +00:00 |
|
|
Andrej
|
311d6d8441
|
Merge pull request #260 from ChrisDryden/patch-2
Remove unused helper functions
|
2024-04-26 15:32:59 -07:00 |
|
|
Andrej
|
7804a971cd
|
Merge pull request #259 from karpathy/feature/fix_int_overflow
candidate fix for int overflow
|
2024-04-26 15:32:43 -07:00 |
|
|
Andrej Karpathy
|
9c2209254b
|
also improve situation in the c code
|
2024-04-26 22:27:22 +00:00 |
|
|
Andrej Karpathy
|
b019927100
|
change more ints to size_t for a bit more safety, still not perfect most likely
|
2024-04-26 22:20:15 +00:00 |
|
|
Chris Dryden
|
1e2df12f14
|
Remove unused helper functions
Removing two helper functions that are no longer used
|
2024-04-26 14:21:35 -07:00 |
|
|
Andrej Karpathy
|
689ccb8ec8
|
candidate fix for int overflow
|
2024-04-26 20:20:53 +00:00 |
|
|
Andrej
|
ad602c59ff
|
Merge pull request #239 from dagelf/master
Add to .gitignore: fp32 binaries, *.exe, *.obj, .venv
|
2024-04-26 13:07:17 -07:00 |
|
|
Andrej Karpathy
|
085270618d
|
upgrade our .py script so it can also do torchrun with many processes, as our mixed precision mpi nccl code can do now
|
2024-04-26 19:50:37 +00:00 |
|
|
Coenraad Loubser
|
8e8db3d817
|
Merge branch 'master' into master
|
2024-04-26 20:44:44 +02:00 |
|
|
Andrej Karpathy
|
0c3e3e37f4
|
Merge branch 'PeterZhizhin-multi_gpu_support'
|
2024-04-26 18:23:43 +00:00 |
|
|
Andrej Karpathy
|
aedf3a5cdd
|
changes to readme for multigpu. i think this is ready now for v0 merge to master nice
|
2024-04-26 18:20:15 +00:00 |
|
|
Andrej Karpathy
|
29add03b64
|
also adjust gitignore
|
2024-04-26 18:14:55 +00:00 |
|
|
Andrej Karpathy
|
e5fd817de8
|
revert fp32 version to no multigpu support
|
2024-04-26 18:12:44 +00:00 |
|
|
Andrej Karpathy
|
a00a3a88ef
|
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
|
2024-04-26 18:11:09 +00:00 |
|
|
Andrej Karpathy
|
2e17140d33
|
more defensive check for openmpi install, and an install command
|
2024-04-26 16:27:57 +00:00 |
|
|
Andrej Karpathy
|
8389fbaa0c
|
Merge branch 'multi_gpu_support' of https://github.com/PeterZhizhin/llm.c into PeterZhizhin-multi_gpu_support
|
2024-04-26 15:55:15 +00:00 |
|
|
Andrej Karpathy
|
8ac4b47696
|
for now save both fp32 and bf16. todo let's think this through a bit more maybe
|
2024-04-26 04:36:53 +00:00 |
|
|
Coenraad Loubser
|
d195e1c3e0
|
Merge branch 'karpathy:master' into master
|
2024-04-25 21:07:28 +02:00 |
|
|
Andrej Karpathy
|
bb56144384
|
we are an epsilon away from writing our model in bf16 as well, in addition to fp32, with the re-ordered layernorms
|
2024-04-25 18:32:08 +00:00 |
|
|
Andrej Karpathy
|
3fb7252924
|
round 1 of some changes. we will now always write in fp32, even if dtype is set to float16 or bfloat16. next up, we actually want to write in lower precision, when the dtype is set so
|
2024-04-25 17:38:01 +00:00 |
|
|
Andrej Karpathy
|
7a52a21fcb
|
Merge branch 'master' of https://github.com/karpathy/llm.c
|
2024-04-25 16:31:44 +00:00 |
|
|
Andrej Karpathy
|
7fe1a4cd49
|
add device argparse
|
2024-04-25 16:31:01 +00:00 |
|
|
dagelf
|
f36891675e
|
Allow data directory to be a softlink and still ignore it
|
2024-04-25 11:43:03 +02:00 |
|
|
dagelf
|
bcddcd6587
|
if / specified git doesn't ignore softlinks to subdirectories
|
2024-04-25 11:40:47 +02:00 |
|
|
Coenraad Loubser
|
819df1cd0f
|
Delete .venv
|
2024-04-25 11:37:01 +02:00 |
|
|
dagelf
|
7476b9ec60
|
more specific gitignore
|
2024-04-25 11:33:16 +02:00 |
|
|
Andrej
|
716a2eddac
|
Merge pull request #247 from joshcarp/patch-1
Add reference to llm.go in description
|
2024-04-24 20:38:05 -07:00 |
|
|
Andrej
|
1555c080aa
|
Merge pull request #249 from ngc92/small-fixes
Small fixes
|
2024-04-24 20:35:35 -07:00 |
|
|
Erik Schultheis
|
533054b393
|
corrected calculation of num_activations
|
2024-04-25 01:39:24 +03:00 |
|
|
Erik Schultheis
|
45e17e3ced
|
don't generate atomic functions for (b)float16 if those are not enabled
|
2024-04-25 01:39:08 +03:00 |
|
|
Petr Zhizhin
|
d49e8301eb
|
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
|
2024-04-24 22:11:00 +00:00 |
|
|
Joshua Carpeggiani
|
0ee2d4a965
|
Add llm.go in description
|
2024-04-24 17:48:49 -04:00 |
|
|
Andrej Karpathy
|
e44ed15639
|
small updates on intent for fp32 and non-fp32 CUDA versions
|
2024-04-24 19:44:12 +00:00 |
|
|
Andrej Karpathy
|
98567a4509
|
turn on benchmarking in mixed precision. example command: python train_gpt2.py --write_tensors 0 --inference_only 0 --dtype bfloat16 --compile 1 --tensorcores 1 --num_iterations 50 --sequence_length 1024
|
2024-04-24 19:26:40 +00:00 |
|
|
Andrej
|
b2c779d40e
|
Merge pull request #231 from Saimirbaci/zig-port-ref
Added ref to the zig port
|
2024-04-24 10:00:08 -07:00 |
|
|
dagelf
|
cd898b79d3
|
.add to .gitignore: fp32 binaries, *.exe, *.obj, .*
|
2024-04-24 17:58:49 +02:00 |
|
|
Andrej Karpathy
|
250159e140
|
encoder backward add, we want to get rid of atomics, hmm
|
2024-04-23 19:06:15 +00:00 |
|
|
Andrej Karpathy
|
fd7da62564
|
minor comment fixes, more to come
|
2024-04-23 18:20:50 +00:00 |
|