 Andrej Karpathy
|
9c2209254b
|
also improve situation in the c code
|
2024-04-26 22:27:22 +00:00 |
|
|
Andrej Karpathy
|
b019927100
|
change more ints to size_t for a bit more safety, still not perfect most likely
|
2024-04-26 22:20:15 +00:00 |
|
|
Andrej Karpathy
|
689ccb8ec8
|
candidate fix for int overflow
|
2024-04-26 20:20:53 +00:00 |
|
|
Andrej
|
ad602c59ff
|
Merge pull request #239 from dagelf/master
Add to .gitignore: fp32 binaries, *.exe, *.obj, .venv
|
2024-04-26 13:07:17 -07:00 |
|
|
Andrej Karpathy
|
085270618d
|
upgrade our .py script so it can also do torchrun with many processes, as our mixed precision mpi nccl code can do now
|
2024-04-26 19:50:37 +00:00 |
|
|
Coenraad Loubser
|
8e8db3d817
|
Merge branch 'master' into master
|
2024-04-26 20:44:44 +02:00 |
|
|
Andrej Karpathy
|
0c3e3e37f4
|
Merge branch 'PeterZhizhin-multi_gpu_support'
|
2024-04-26 18:23:43 +00:00 |
|
|
Andrej Karpathy
|
aedf3a5cdd
|
changes to readme for multigpu. i think this is ready now for v0 merge to master nice
|
2024-04-26 18:20:15 +00:00 |
|
|
Andrej Karpathy
|
29add03b64
|
also adjust gitignore
|
2024-04-26 18:14:55 +00:00 |
|
|
Andrej Karpathy
|
e5fd817de8
|
revert fp32 version to no multigpu support
|
2024-04-26 18:12:44 +00:00 |
|
|
Andrej Karpathy
|
a00a3a88ef
|
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
|
2024-04-26 18:11:09 +00:00 |
|
|
Andrej Karpathy
|
2e17140d33
|
more defensive check for openmpi install, and an install command
|
2024-04-26 16:27:57 +00:00 |
|
|
Andrej Karpathy
|
8389fbaa0c
|
Merge branch 'multi_gpu_support' of https://github.com/PeterZhizhin/llm.c into PeterZhizhin-multi_gpu_support
|
2024-04-26 15:55:15 +00:00 |
|
|
Andrej Karpathy
|
8ac4b47696
|
for now save both fp32 and bf16. todo let's think this through a bit more maybe
|
2024-04-26 04:36:53 +00:00 |
|
|
Coenraad Loubser
|
d195e1c3e0
|
Merge branch 'karpathy:master' into master
|
2024-04-25 21:07:28 +02:00 |
|
|
Andrej Karpathy
|
bb56144384
|
we are an epsilon away from writing our model in bf16 as well, in addition to fp32, with the re-ordered layernorms
|
2024-04-25 18:32:08 +00:00 |
|
|
Andrej Karpathy
|
3fb7252924
|
round 1 of some changes. we will now always write in fp32, even if dtype is set to float16 or bfloat16. next up, we actually want to write in lower precision, when the dtype is set so
|
2024-04-25 17:38:01 +00:00 |
|
|
Andrej Karpathy
|
7a52a21fcb
|
Merge branch 'master' of https://github.com/karpathy/llm.c
|
2024-04-25 16:31:44 +00:00 |
|
|
Andrej Karpathy
|
7fe1a4cd49
|
add device argparse
|
2024-04-25 16:31:01 +00:00 |
|
|
dagelf
|
f36891675e
|
Allow data directory to be a softlink and still ignore it
|
2024-04-25 11:43:03 +02:00 |
|
|
dagelf
|
bcddcd6587
|
if / specified git doesn't ignore softlinks to subdirectories
|
2024-04-25 11:40:47 +02:00 |
|
|
Coenraad Loubser
|
819df1cd0f
|
Delete .venv
|
2024-04-25 11:37:01 +02:00 |
|
|
dagelf
|
7476b9ec60
|
more specific gitignore
|
2024-04-25 11:33:16 +02:00 |
|
|
Andrej
|
716a2eddac
|
Merge pull request #247 from joshcarp/patch-1
Add reference to llm.go in description
|
2024-04-24 20:38:05 -07:00 |
|
|
Andrej
|
1555c080aa
|
Merge pull request #249 from ngc92/small-fixes
Small fixes
|
2024-04-24 20:35:35 -07:00 |
|
|
Erik Schultheis
|
533054b393
|
corrected calculation of num_activations
|
2024-04-25 01:39:24 +03:00 |
|
|
Erik Schultheis
|
45e17e3ced
|
don't generate atomic functions for (b)float16 if those are not enabled
|
2024-04-25 01:39:08 +03:00 |
|
|
Petr Zhizhin
|
d49e8301eb
|
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
|
2024-04-24 22:11:00 +00:00 |
|
|
Joshua Carpeggiani
|
0ee2d4a965
|
Add llm.go in description
|
2024-04-24 17:48:49 -04:00 |
|
|
Andrej Karpathy
|
e44ed15639
|
small updates on intent for fp32 and non-fp32 CUDA versions
|
2024-04-24 19:44:12 +00:00 |
|
|
Andrej Karpathy
|
98567a4509
|
turn on benchmarking in mixed precision. example command: python train_gpt2.py --write_tensors 0 --inference_only 0 --dtype bfloat16 --compile 1 --tensorcores 1 --num_iterations 50 --sequence_length 1024
|
2024-04-24 19:26:40 +00:00 |
|
|
Andrej
|
b2c779d40e
|
Merge pull request #231 from Saimirbaci/zig-port-ref
Added ref to the zig port
|
2024-04-24 10:00:08 -07:00 |
|
|
dagelf
|
cd898b79d3
|
.add to .gitignore: fp32 binaries, *.exe, *.obj, .*
|
2024-04-24 17:58:49 +02:00 |
|
|
Andrej Karpathy
|
250159e140
|
encoder backward add, we want to get rid of atomics, hmm
|
2024-04-23 19:06:15 +00:00 |
|
|
Andrej Karpathy
|
fd7da62564
|
minor comment fixes, more to come
|
2024-04-23 18:20:50 +00:00 |
|
|
Andrej Karpathy
|
249140287f
|
rename positional_forward to encoder_forward as it should have been
|
2024-04-23 18:19:20 +00:00 |
|
|
Andrej Karpathy
|
0b59d80880
|
not proud of this but i have to increase the threshold for now to make test pass. later we should come back around and tighten thresholds
|
2024-04-23 18:13:57 +00:00 |
|
|
Andrej Karpathy
|
754627c0a2
|
update ci to pick up only the fp32 version for now, todo add mixed precision
|
2024-04-23 17:56:46 +00:00 |
|
|
Andrej Karpathy
|
c5f1db6020
|
fix important typo on precision
|
2024-04-23 17:43:19 +00:00 |
|
|
Andrej Karpathy
|
ed6387a695
|
updates to readme, and introduce the test fp32 cuda file too
|
2024-04-23 17:41:42 +00:00 |
|
|
Saimir Baci
|
7f279cd6cb
|
Adding Zig port ref
|
2024-04-23 13:37:53 -04:00 |
|
|
Andrej Karpathy
|
095d27662b
|
checkpoint the fp32 CUDA implementation to separate file. our mainline iteration will now continue in the new (mixed precision) file
|
2024-04-23 17:21:44 +00:00 |
|
|
Andrej
|
6b6ad35f29
|
Merge pull request #218 from ademeure/linear16
Support for FP16/BF16 in train_gpt2.cu (1.86x Perf)
|
2024-04-23 10:17:01 -07:00 |
|
|
ademeure
|
0333e36981
|
tiny fixes to make fp8 work in the future (missing conversions)
|
2024-04-23 14:38:41 +01:00 |
|
|
ademeure
|
91ec92f4d5
|
fix typo
|
2024-04-23 06:43:40 +01:00 |
|
|
ademeure
|
a876485151
|
Fill in param_sizeof in a slightly less terrible way
|
2024-04-23 06:39:58 +01:00 |
|
|
ademeure
|
53dc40e804
|
Fixes for merge with latest, now ~86% faster!
|
2024-04-23 06:28:42 +01:00 |
|
|
ademeure
|
c1992a19d5
|
Merge remote-tracking branch 'karpathy/master' into linear16
|
2024-04-23 06:18:23 +01:00 |
|
|
ademeure
|
8775856b1d
|
Added floatN to choose FP32 vs FP16 for layernorm (parameters/gradients only)
|
2024-04-23 06:13:23 +01:00 |
|
|
ademeure
|
7c193bd4cc
|
Good news: it works. Bad news: 16-bit atomics are *incredibly slow* :( (is it doing CAS?)
|
2024-04-23 05:00:47 +01:00 |
|