a3f5ad9095
reshuffle the ifdefs to make bf16 the default if no PRECISION is requested via defines
2024-04-28 16:12:39 +00:00
4f7d8d9f44
allow user to make different precisions, add prints and error handling around precisions
2024-04-28 16:08:27 +00:00
1c7d23ab3c
amend the float4 kernel
2024-04-27 20:26:19 -07:00
61c5c05ea8
amend the float4 kernel
2024-04-27 20:23:39 -07:00
cfccd821e6
Include the float4 kernel
2024-04-27 19:58:18 -07:00
82d7907214
adjust comment
2024-04-28 00:05:03 +00:00
9a91b40207
bring back original ordering. i also had to bump the thresholds by 3X for some tensors and i don't exactly know why sad
2024-04-28 00:02:51 +00:00
1d93edf010
turn on full BF16 on by default
2024-04-28 00:32:54 +01:00
8768f6bd74
Full BF16 including layernorm at good perf by minimising BF16 atomics
2024-04-28 00:31:42 +01:00
00627070aa
fix a really bad bug in how i was checking the gradients, where i loaded them in the old order, so yeah...
2024-04-27 23:17:22 +00:00
2954d90f30
Enable multithreading in nvcc
...
Tested locally and reduced compilation time by 200ms, unfortunately for me upgrading to 12.4 made my compilations times slow by 2x but at least this can make it a bit faster
2024-04-27 14:14:02 -07:00
a58b8d5dc7
print more in the comparison
2024-04-27 16:04:55 +00:00
9d6fd30e3c
tweak the tolerances until we pass lol
2024-04-27 15:49:54 +00:00
e067a27395
fix dumb bug. i'll blame github copilot but i can't remember
2024-04-27 15:40:32 +00:00
d4a642ba19
i think github copilot betrayed me on this index here, i cant remember
2024-04-27 15:38:20 +00:00
09d935c82b
i think i am making things cleaner, but i am not fixing the problem
2024-04-27 15:27:05 +00:00
12da2c16cc
slight docs tweaks to dev/cuda
2024-04-27 14:40:57 +00:00
a5d23e7f97
Merge pull request #266 from AnswerDotAI/master
...
Add a local convenience Makefile for dev/cuda/
2024-04-27 07:18:22 -07:00
25d703fbec
Use a make pattern rule to simplify Makefile, group kernels into forward/backward/updates/nccls, add some comments and update the README.md
2024-04-27 00:13:01 -04:00
d937f93aaa
Add a local convenience Makefile for dev/cuda/, fix 2 minor warnings observed when building all targets.
2024-04-26 21:19:44 -04:00
09cd67e868
code to load bf16 weights directly, and also re-wire the position of tensors to put the layernorms at the end. the training loop seems to work ok, and the tests pass and the loss and optimization looks ok, but the gradients don't match. which can't be right. so there is a bug, but it's a bit too late in the day for me to debug right now, creating a PR and going to sleep, will fix tomorrow
2024-04-27 00:54:06 +00:00
311d6d8441
Merge pull request #260 from ChrisDryden/patch-2
...
Remove unused helper functions
2024-04-26 15:32:59 -07:00
7804a971cd
Merge pull request #259 from karpathy/feature/fix_int_overflow
...
candidate fix for int overflow
2024-04-26 15:32:43 -07:00
9c2209254b
also improve situation in the c code
2024-04-26 22:27:22 +00:00
b019927100
change more ints to size_t for a bit more safety, still not perfect most likely
2024-04-26 22:20:15 +00:00
1e2df12f14
Remove unused helper functions
...
Removing two helper functions that are no longer used
2024-04-26 14:21:35 -07:00
689ccb8ec8
candidate fix for int overflow
2024-04-26 20:20:53 +00:00
ad602c59ff
Merge pull request #239 from dagelf/master
...
Add to .gitignore: fp32 binaries, *.exe, *.obj, .venv
2024-04-26 13:07:17 -07:00
085270618d
upgrade our .py script so it can also do torchrun with many processes, as our mixed precision mpi nccl code can do now
2024-04-26 19:50:37 +00:00
8e8db3d817
Merge branch 'master' into master
2024-04-26 20:44:44 +02:00
0c3e3e37f4
Merge branch 'PeterZhizhin-multi_gpu_support'
2024-04-26 18:23:43 +00:00
aedf3a5cdd
changes to readme for multigpu. i think this is ready now for v0 merge to master nice
2024-04-26 18:20:15 +00:00
29add03b64
also adjust gitignore
2024-04-26 18:14:55 +00:00
e5fd817de8
revert fp32 version to no multigpu support
2024-04-26 18:12:44 +00:00
a00a3a88ef
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
2024-04-26 18:11:09 +00:00
2e17140d33
more defensive check for openmpi install, and an install command
2024-04-26 16:27:57 +00:00
8389fbaa0c
Merge branch 'multi_gpu_support' of https://github.com/PeterZhizhin/llm.c into PeterZhizhin-multi_gpu_support
2024-04-26 15:55:15 +00:00
8ac4b47696
for now save both fp32 and bf16. todo let's think this through a bit more maybe
2024-04-26 04:36:53 +00:00
d195e1c3e0
Merge branch 'karpathy:master' into master
2024-04-25 21:07:28 +02:00
bb56144384
we are an epsilon away from writing our model in bf16 as well, in addition to fp32, with the re-ordered layernorms
2024-04-25 18:32:08 +00:00
3fb7252924
round 1 of some changes. we will now always write in fp32, even if dtype is set to float16 or bfloat16. next up, we actually want to write in lower precision, when the dtype is set so
2024-04-25 17:38:01 +00:00
7a52a21fcb
Merge branch 'master' of https://github.com/karpathy/llm.c
2024-04-25 16:31:44 +00:00
7fe1a4cd49
add device argparse
2024-04-25 16:31:01 +00:00
f36891675e
Allow data directory to be a softlink and still ignore it
2024-04-25 11:43:03 +02:00
bcddcd6587
if / specified git doesn't ignore softlinks to subdirectories
2024-04-25 11:40:47 +02:00
819df1cd0f
Delete .venv
2024-04-25 11:37:01 +02:00
7476b9ec60
more specific gitignore
2024-04-25 11:33:16 +02:00
716a2eddac
Merge pull request #247 from joshcarp/patch-1
...
Add reference to llm.go in description
2024-04-24 20:38:05 -07:00
1555c080aa
Merge pull request #249 from ngc92/small-fixes
...
Small fixes
2024-04-24 20:35:35 -07:00
533054b393
corrected calculation of num_activations
2024-04-25 01:39:24 +03:00
Andrej Karpathy