mirrors / karpathy--llm.c

提交

提交图

  • d95b8d8320 Merge pull request #265 from karpathy/feature/load_bf16 Andrej 2024-04-28 09:24:56 -07:00
  • 9d70d9a1b2 profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though feature/load_bf16 Andrej Karpathy 2024-04-28 16:17:22 +00:00
  • a3f5ad9095 reshuffle the ifdefs to make bf16 the default if no PRECISION is requested via defines Andrej Karpathy 2024-04-28 16:12:39 +00:00
  • 4f7d8d9f44 allow user to make different precisions, add prints and error handling around precisions Andrej Karpathy 2024-04-28 16:08:27 +00:00
  • 1c7d23ab3c amend the float4 kernel lancer 2024-04-27 20:26:19 -07:00
  • 61c5c05ea8 amend the float4 kernel lancer 2024-04-27 20:23:39 -07:00
  • cfccd821e6 Include the float4 kernel lancer 2024-04-27 19:58:18 -07:00
  • 82d7907214 adjust comment Andrej Karpathy 2024-04-28 00:05:03 +00:00
  • 9a91b40207 bring back original ordering. i also had to bump the thresholds by 3X for some tensors and i don't exactly know why sad Andrej Karpathy 2024-04-28 00:02:51 +00:00
  • 1d93edf010 turn on full BF16 on by default ademeure 2024-04-28 00:32:54 +01:00
  • 8768f6bd74 Full BF16 including layernorm at good perf by minimising BF16 atomics ademeure 2024-04-28 00:31:42 +01:00
  • 00627070aa fix a really bad bug in how i was checking the gradients, where i loaded them in the old order, so yeah... Andrej Karpathy 2024-04-27 23:17:22 +00:00
  • 2954d90f30 Enable multithreading in nvcc Chris Dryden 2024-04-27 14:14:02 -07:00
  • a58b8d5dc7 print more in the comparison Andrej Karpathy 2024-04-27 16:04:55 +00:00
  • 9d6fd30e3c tweak the tolerances until we pass lol Andrej Karpathy 2024-04-27 15:49:54 +00:00
  • e067a27395 fix dumb bug. i'll blame github copilot but i can't remember Andrej Karpathy 2024-04-27 15:40:32 +00:00
  • d4a642ba19 i think github copilot betrayed me on this index here, i cant remember Andrej Karpathy 2024-04-27 15:38:20 +00:00
  • 09d935c82b i think i am making things cleaner, but i am not fixing the problem Andrej Karpathy 2024-04-27 15:27:05 +00:00
  • 12da2c16cc slight docs tweaks to dev/cuda Andrej Karpathy 2024-04-27 14:40:57 +00:00
  • a5d23e7f97 Merge pull request #266 from AnswerDotAI/master Andrej 2024-04-27 07:18:22 -07:00
  • 25d703fbec Use a make pattern rule to simplify Makefile, group kernels into forward/backward/updates/nccls, add some comments and update the README.md austinvhuang 2024-04-27 00:13:01 -04:00
  • d937f93aaa Add a local convenience Makefile for dev/cuda/, fix 2 minor warnings observed when building all targets. austinvhuang 2024-04-26 21:19:44 -04:00
  • 09cd67e868 code to load bf16 weights directly, and also re-wire the position of tensors to put the layernorms at the end. the training loop seems to work ok, and the tests pass and the loss and optimization looks ok, but the gradients don't match. which can't be right. so there is a bug, but it's a bit too late in the day for me to debug right now, creating a PR and going to sleep, will fix tomorrow Andrej Karpathy 2024-04-27 00:54:06 +00:00
  • 311d6d8441 Merge pull request #260 from ChrisDryden/patch-2 Andrej 2024-04-26 15:32:59 -07:00
  • 7804a971cd Merge pull request #259 from karpathy/feature/fix_int_overflow Andrej 2024-04-26 15:32:43 -07:00
  • 9c2209254b also improve situation in the c code feature/fix_int_overflow Andrej Karpathy 2024-04-26 22:27:22 +00:00
  • b019927100 change more ints to size_t for a bit more safety, still not perfect most likely Andrej Karpathy 2024-04-26 22:20:15 +00:00
  • 1e2df12f14 Remove unused helper functions Chris Dryden 2024-04-26 14:21:35 -07:00
  • 689ccb8ec8 candidate fix for int overflow Andrej Karpathy 2024-04-26 20:20:53 +00:00
  • ad602c59ff Merge pull request #239 from dagelf/master Andrej 2024-04-26 13:07:17 -07:00
  • 085270618d upgrade our .py script so it can also do torchrun with many processes, as our mixed precision mpi nccl code can do now Andrej Karpathy 2024-04-26 19:50:37 +00:00
  • 8e8db3d817 Merge branch 'master' into master Coenraad Loubser 2024-04-26 20:44:44 +02:00
  • 0c3e3e37f4 Merge branch 'PeterZhizhin-multi_gpu_support' Andrej Karpathy 2024-04-26 18:23:43 +00:00
  • aedf3a5cdd changes to readme for multigpu. i think this is ready now for v0 merge to master nice Andrej Karpathy 2024-04-26 18:20:15 +00:00
  • 29add03b64 also adjust gitignore Andrej Karpathy 2024-04-26 18:14:55 +00:00
  • e5fd817de8 revert fp32 version to no multigpu support Andrej Karpathy 2024-04-26 18:12:44 +00:00
  • a00a3a88ef fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0 Andrej Karpathy 2024-04-26 18:11:09 +00:00
  • 2e17140d33 more defensive check for openmpi install, and an install command Andrej Karpathy 2024-04-26 16:27:57 +00:00
  • 8389fbaa0c Merge branch 'multi_gpu_support' of https://github.com/PeterZhizhin/llm.c into PeterZhizhin-multi_gpu_support Andrej Karpathy 2024-04-26 15:55:15 +00:00
  • 8ac4b47696 for now save both fp32 and bf16. todo let's think this through a bit more maybe Andrej Karpathy 2024-04-26 04:36:53 +00:00
  • d195e1c3e0 Merge branch 'karpathy:master' into master Coenraad Loubser 2024-04-25 21:07:28 +02:00
  • bb56144384 we are an epsilon away from writing our model in bf16 as well, in addition to fp32, with the re-ordered layernorms Andrej Karpathy 2024-04-25 18:32:08 +00:00
  • 3fb7252924 round 1 of some changes. we will now always write in fp32, even if dtype is set to float16 or bfloat16. next up, we actually want to write in lower precision, when the dtype is set so Andrej Karpathy 2024-04-25 17:38:01 +00:00
  • 7a52a21fcb Merge branch 'master' of https://github.com/karpathy/llm.c Andrej Karpathy 2024-04-25 16:31:44 +00:00
  • 7fe1a4cd49 add device argparse Andrej Karpathy 2024-04-25 16:31:01 +00:00
  • f36891675e Allow data directory to be a softlink and still ignore it dagelf 2024-04-25 11:43:03 +02:00
  • bcddcd6587 if / specified git doesn't ignore softlinks to subdirectories dagelf 2024-04-25 11:40:47 +02:00
  • 819df1cd0f Delete .venv Coenraad Loubser 2024-04-25 11:37:01 +02:00
  • 7476b9ec60 more specific gitignore dagelf 2024-04-25 11:33:16 +02:00
  • 716a2eddac Merge pull request #247 from joshcarp/patch-1 Andrej 2024-04-24 20:38:05 -07:00
  • 1555c080aa Merge pull request #249 from ngc92/small-fixes Andrej 2024-04-24 20:35:35 -07:00
  • 533054b393 corrected calculation of num_activations Erik Schultheis 2024-04-25 01:39:24 +03:00
  • 45e17e3ced don't generate atomic functions for (b)float16 if those are not enabled Erik Schultheis 2024-04-25 01:39:08 +03:00
  • d49e8301eb [Multi-GPU] llm.c now runs on multiple GPUs with NCCL Petr Zhizhin 2024-04-22 22:12:33 +02:00
  • 0ee2d4a965 Add llm.go in description Joshua Carpeggiani 2024-04-24 17:48:49 -04:00
  • e44ed15639 small updates on intent for fp32 and non-fp32 CUDA versions Andrej Karpathy 2024-04-24 19:44:12 +00:00
  • 98567a4509 turn on benchmarking in mixed precision. example command: python train_gpt2.py --write_tensors 0 --inference_only 0 --dtype bfloat16 --compile 1 --tensorcores 1 --num_iterations 50 --sequence_length 1024 Andrej Karpathy 2024-04-24 19:26:40 +00:00
  • b2c779d40e Merge pull request #231 from Saimirbaci/zig-port-ref Andrej 2024-04-24 10:00:08 -07:00
  • cd898b79d3 .add to .gitignore: fp32 binaries, *.exe, *.obj, .* dagelf 2024-04-24 17:58:49 +02:00
  • 250159e140 encoder backward add, we want to get rid of atomics, hmm Andrej Karpathy 2024-04-23 19:06:15 +00:00
  • fd7da62564 minor comment fixes, more to come Andrej Karpathy 2024-04-23 18:20:50 +00:00
  • 249140287f rename positional_forward to encoder_forward as it should have been Andrej Karpathy 2024-04-23 18:19:20 +00:00
  • 0b59d80880 not proud of this but i have to increase the threshold for now to make test pass. later we should come back around and tighten thresholds Andrej Karpathy 2024-04-23 18:13:57 +00:00
  • 754627c0a2 update ci to pick up only the fp32 version for now, todo add mixed precision Andrej Karpathy 2024-04-23 17:56:46 +00:00
  • c5f1db6020 fix important typo on precision Andrej Karpathy 2024-04-23 17:43:19 +00:00
  • ed6387a695 updates to readme, and introduce the test fp32 cuda file too Andrej Karpathy 2024-04-23 17:41:42 +00:00
  • 7f279cd6cb Adding Zig port ref Saimir Baci 2024-04-23 13:37:53 -04:00
  • 095d27662b checkpoint the fp32 CUDA implementation to separate file. our mainline iteration will now continue in the new (mixed precision) file Andrej Karpathy 2024-04-23 17:21:44 +00:00
  • 6b6ad35f29 Merge pull request #218 from ademeure/linear16 Andrej 2024-04-23 10:17:01 -07:00
  • 0333e36981 tiny fixes to make fp8 work in the future (missing conversions) ademeure 2024-04-23 14:38:41 +01:00
  • 91ec92f4d5 fix typo ademeure 2024-04-23 06:43:40 +01:00
  • a876485151 Fill in param_sizeof in a slightly less terrible way ademeure 2024-04-23 06:39:58 +01:00
  • 53dc40e804 Fixes for merge with latest, now ~86% faster! ademeure 2024-04-23 06:28:42 +01:00
  • c1992a19d5 Merge remote-tracking branch 'karpathy/master' into linear16 ademeure 2024-04-23 06:18:23 +01:00
  • 8775856b1d Added floatN to choose FP32 vs FP16 for layernorm (parameters/gradients only) ademeure 2024-04-23 06:13:23 +01:00
  • 7c193bd4cc Good news: it works. Bad news: 16-bit atomics are *incredibly slow* :( (is it doing CAS?) ademeure 2024-04-23 05:00:47 +01:00
  • bdc661a609 BF16/FP16 Attention + bug fixes ademeure 2024-04-23 02:26:57 +01:00
  • 85290a5228 Updated FP16/BF16 mix-and-match ncluding stochastic rounding ademeure 2024-04-23 01:39:33 +01:00
  • 896f83f422 Merge pull request #225 from dagelf/master Andrej 2024-04-22 14:45:29 -07:00
  • 2efac6670a Make msvc compile by adding the ifdefs back in gelu_backward, fixes #200 Coenraad 2024-04-22 23:15:14 +02:00
  • 451c1a4c99 update readme to point to SoU Andrej Karpathy 2024-04-22 19:18:03 +00:00
  • d3c5025a3e Merge branch 'ChrisDryden-sharedmem_layernormback' Andrej Karpathy 2024-04-22 18:45:01 +00:00
  • e3bcae6b2b incorporate layernorm backward kernel 2, we are now down to 73.1ms/iter on average Andrej Karpathy 2024-04-22 18:44:43 +00:00
  • 86b3401dd5 Merge branch 'sharedmem_layernormback' of https://github.com/ChrisDryden/llm.c into ChrisDryden-sharedmem_layernormback Andrej Karpathy 2024-04-22 18:33:42 +00:00
  • 5f545ca6cc Merge branch 'al0vya-matmul-backward-bias' Andrej Karpathy 2024-04-22 17:51:26 +00:00
  • 988489519c nice new kernel for bias reduce, we are down by ~1ms/iter, to 76.13ms/iter on average Andrej Karpathy 2024-04-22 17:51:06 +00:00
  • d605b993da Fully working BF16 training! (very hacky Adam, and converges much slower than FP32 for now) ademeure 2024-04-22 18:38:51 +01:00
  • 58df2629ae Gradients now working in BF16 mode! (still need adam etc...) ademeure 2024-04-22 18:09:18 +01:00
  • a42f739675 Merge branch 'matmul-backward-bias' of https://github.com/al0vya/llm.c into al0vya-matmul-backward-bias Andrej Karpathy 2024-04-22 16:52:31 +00:00
  • f813d635b2 Merge pull request #222 from regrettable-username/patch-1 Andrej 2024-04-22 09:33:27 -07:00
  • b1e55951c6 rename to _forward for the kernel, and small fixes to docs, and avoid auto Andrej Karpathy 2024-04-22 16:32:03 +00:00
  • 35393b4442 add more comments to explain the philosophy behind the kernel cip19aac 2024-04-22 17:29:11 +01:00
  • b82ec201be add comment cip19aac 2024-04-22 17:05:05 +01:00
  • 7830cf6a12 Merge pull request #214 from ngc92/trimul Andrej 2024-04-22 09:03:53 -07:00
  • 38857ff1d0 Added llm.metal to notable forks James Thompson 2024-04-22 09:01:21 -07:00
  • 55fbd7e6ad add a faster matmul backward bias kernel that uses coalesced reads and shared memory cip19aac 2024-04-22 16:35:21 +01:00
  • f35adbec77 WIP support for FP16/BF16 in train_gpt2.cu (compiles, not correct results) ademeure 2024-04-22 16:21:28 +01:00
  • 732a8b4806 added NaN-based makes for reference checks Erik Schultheis 2024-04-22 14:53:24 +03:00
  • caa69a5e52 added triangular matrix multiplication kernel Erik Schultheis 2024-04-22 14:45:51 +03:00
  • 6984e83c6e Merge branch 'patricxu-master' Andrej Karpathy 2024-04-22 00:28:00 +00:00