mirrors / karpathy--llm.c

提交

提交图

  • 229d70b2d5 uint and designated initializers non-standard for C++ Ross Wheeler 2024-04-30 18:46:54 -07:00
  • 40da8cf233 Merge pull request #314 from jrhemstad/patch-3 Andrej 2024-04-30 17:03:25 -07:00
  • 78eba5fff3 Add llm.cpp fork to README Jake Hemstad 2024-04-30 18:52:42 -05:00
  • 769c911ab6 small tweaks Andrej Karpathy 2024-04-30 23:48:06 +00:00
  • bad76f408a Merge pull request #313 from ngc92/small-improvements Andrej 2024-04-30 16:18:08 -07:00
  • 74d5bcd03a Adding a java port of the project harry 2024-04-30 14:51:06 -07:00
  • 603d862830 fix up bandwidth calculation Erik Schultheis 2024-04-30 23:52:08 +03:00
  • c3515cfe47 fixed potential error and generalized gelu forward Erik Schultheis 2024-04-30 23:45:34 +03:00
  • 7ba1ed8a1a delete packed128 .fp32, use (float) ok Andrej Karpathy 2024-04-30 19:39:20 +00:00
  • 7fd0659ba8 Pack gelu_forward kernel using the new Packed128 Andrej Karpathy 2024-04-30 19:35:23 +00:00
  • 9141e04693 fixes to the PR, careful with float/floatX etc Andrej Karpathy 2024-04-30 19:35:05 +00:00
  • 242981eecf Merge branch 'gelu_forwards_packing' of https://github.com/ChrisDryden/llm.c into ChrisDryden-gelu_forwards_packing Andrej Karpathy 2024-04-30 19:06:39 +00:00
  • 70b4de8700 add comment documenting how to reproduce python reference exactly Andrej Karpathy 2024-04-30 18:51:45 +00:00
  • 050cbfa42c override train split to val split if we are debugging and trying to overfit a single batch of data, following the python script reference behavior Andrej Karpathy 2024-04-30 18:48:12 +00:00
  • 44656c3850 allow fp32 precision in the test script as well Andrej Karpathy 2024-04-30 18:38:38 +00:00
  • b84571f745 also add argparse to force tf32 to zero Andrej Karpathy 2024-04-30 17:54:15 +00:00
  • 654d6f55c5 add arg to cap the number of steps, and offset all prints to start steps at 1, which i think looks better to the eye Andrej Karpathy 2024-04-30 17:46:36 +00:00
  • 52e2ca8378 Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-04-30 16:36:00 +00:00
  • 179b159a1d Merge pull request #311 from leloykun/fc--modal-script Andrej 2024-04-30 09:35:51 -07:00
  • b239b67ae1 add modal script Franz Louis Cesista 2024-05-01 00:15:08 +08:00
  • 9d8a6d13f0 add flag overfit_single_batch useful debugging into train_gpt2.cu Andrej Karpathy 2024-04-30 16:03:33 +00:00
  • 435ac928b4 cuDNN Forward Flash Attention is working in both /dev/cuda/ and train_gpt2.cu! (backwards pass is broken for now as a conquence) ademeure 2024-04-30 17:00:09 +01:00
  • 51face88d9 fix bug where backward/step must be outside of amp context Andrej Karpathy 2024-04-30 15:48:40 +00:00
  • 7e8f14d2c0 Merge pull request #310 from ngc92/flush-l2-for-benchmark Andrej 2024-04-30 07:23:41 -07:00
  • 1c516c732d re-commit of my old cudnn forward attention changes ademeure 2024-04-30 14:54:35 +01:00
  • d7813d281d clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results Erik Schultheis 2024-04-30 12:45:37 +03:00
  • 1f442fdf7f fix gpt2_update call chinthysl 2024-04-30 09:10:50 +00:00
  • b3e8abdd54 optimizer update per shard and nccl all gather chinthysl 2024-04-30 09:09:52 +00:00
  • 2d26ec10b2 setting the zero opt configs chinthysl 2024-04-30 09:08:25 +00:00
  • 3254a51522 Zero Optimizations configs chinthysl 2024-04-30 09:07:01 +00:00
  • 568615fef1 Added packing for gelu forwards kernel Christopher Dryden 2024-04-30 03:21:42 +00:00
  • 9464f4272e Merge pull request #283 from ChrisDryden/configurationRewrite Andrej 2024-04-29 18:58:03 -07:00
  • 997864920b Merge pull request #290 from Ricardicus/ci-dev-cuda Andrej 2024-04-29 18:53:03 -07:00
  • 2490f7896d Merge pull request #298 from karpathy/feature/packed128 Andrej 2024-04-29 16:31:02 -07:00
  • 906d22f5b6 fix profile gpt2cu but note it currently needs the NO_MULTI_GPU=1 switch i think when calling it with make, todo fix later how we add the mpi flags Andrej Karpathy 2024-04-29 23:27:55 +00:00
  • af2bc47d22 remove forceinline. blame @arund42 if this ever does not get inlined by the compiler feature/packed128 Andrej Karpathy 2024-04-29 23:10:51 +00:00
  • f02ba4b436 import updated tokenizer properly Andrej Karpathy 2024-04-29 23:02:26 +00:00
  • 7634f0863c add packed128 to the mainline file Andrej Karpathy 2024-04-29 22:57:59 +00:00
  • 5e12c66f5a move packed128 to common Andrej Karpathy 2024-04-29 22:55:28 +00:00
  • f43592954f reset back to precious hyperparameters Andrej Karpathy 2024-04-29 22:44:25 +00:00
  • ee125cffc4 make it imo clearer that the grid size is one quarter as large as it would otherwise be. because each thread now does 4X work Andrej Karpathy 2024-04-29 22:40:26 +00:00
  • 083b210dd4 small changes Andrej Karpathy 2024-04-29 22:38:03 +00:00
  • d9506a7a0a minor changes to packed128 Andrej Karpathy 2024-04-29 22:31:19 +00:00
  • 7eefefb511 Merge branch 'yet-another-gelu' of https://github.com/ngc92/llm.c into ngc92-yet-another-gelu Andrej Karpathy 2024-04-29 22:19:35 +00:00
  • 39ce9be516 fix missing \n on a print Andrej Karpathy 2024-04-29 21:05:44 +00:00
  • f66825faba allow using flash attention via argparse and print tok/s Andrej Karpathy 2024-04-29 21:05:30 +00:00
  • 5ad0079fca Reverting changes to Makefile, adding -j4 to make Rickard Hallerbäck 2024-04-29 22:44:21 +02:00
  • 0d6e096b9b Merge pull request #296 from karpathy/separate_tokenizer Andrej 2024-04-29 13:44:05 -07:00
  • 1443edc746 Merge branch 'master' into separate_tokenizer separate_tokenizer Andrej 2024-04-29 13:38:58 -07:00
  • 70f777617d Single call to NCCL AllReduce and AdamW for bf16 Andrej Karpathy 2024-04-29 20:36:29 +00:00
  • dd60fc3991 fix prints Andrej Karpathy 2024-04-29 20:36:07 +00:00
  • be4aeb3d2d fix accumulated loss Andrej Karpathy 2024-04-29 20:32:10 +00:00
  • 938f8f78bb more defensive Andrej Karpathy 2024-04-29 20:25:29 +00:00
  • b4c346ac2a don't hardcode the EOT token and store it in the Tokenizer file Andrej Karpathy 2024-04-29 20:23:41 +00:00
  • c24bb88bce Add back FP16 and disable multi_gpu all reduce when there is only 1 GPU ademeure 2024-04-29 21:15:52 +01:00
  • 699c25469a separate out our Tokenizer Andrej Karpathy 2024-04-29 19:54:18 +00:00
  • 4872c57164 comments Erik Schultheis 2024-04-29 22:46:33 +03:00
  • 9f07a176e0 simplify to only support cs Erik Schultheis 2024-04-29 22:39:49 +03:00
  • 66c454874f avoid warning for cublas_compute_type in BF16 mode ademeure 2024-04-29 20:25:07 +01:00
  • 1d4effd5a4 Remove FloatN and associated code for adam & allreduce ademeure 2024-04-29 20:21:34 +01:00
  • 896b644cb9 slight simplification Erik Schultheis 2024-04-29 22:19:12 +03:00
  • 1310e2f77d Merge pull request #286 from rosslwheeler/Makefile-windows-support Andrej 2024-04-29 12:11:12 -07:00
  • 977c688b9d Merge pull request #294 from karpathy/ademeure-layernormbf16-3 Andrej 2024-04-29 12:10:07 -07:00
  • 0e4506ac30 unify these two ademeure-layernormbf16-3 Andrej Karpathy 2024-04-29 19:03:49 +00:00
  • 8be7370a55 gpt2_build_from_checkpoint ademeure 2024-04-29 19:54:32 +01:00
  • 45252d434c yet another gelu Erik Schultheis 2024-04-29 21:53:08 +03:00
  • ebedadfee8 Fix train_gpt2.py to write all weights as BF16 (+merge fixes) ademeure 2024-04-29 19:49:57 +01:00
  • d23bb1fd13 Merge remote-tracking branch 'karpathy/master' into layernormbf16 ademeure 2024-04-29 19:41:54 +01:00
  • e92ec3ffd9 Merge pull request #281 from ChrisDryden/templateCleanup Andrej 2024-04-29 10:05:29 -07:00
  • 237593f47b Merge pull request #287 from tairov/fix-zig-link Andrej 2024-04-29 09:04:18 -07:00
  • fcef97d6a1 Adding back PFLAGS Ross Wheeler 2024-04-29 08:53:50 -07:00
  • 8915d5ab0d Adding the example kernels to CI Rickard Hallerbäck 2024-04-29 16:58:03 +02:00
  • 49ffd68317 Fix sizeof(T) -> sizeof(D) (was overallocating memory) ademeure 2024-04-29 15:47:59 +01:00
  • d0b943ba04 Fix link to zig port Aydyn Tairov 2024-04-29 11:18:03 +01:00
  • 81b689ed01 Merge branch 'Makefile-windows-support' of https://github.com/rosslwheeler/llm.c into Makefile-windows-support Ross Wheeler 2024-04-29 03:00:56 -07:00
  • c9ab8ee123 Updating Makefile with latest changes Ross Wheeler 2024-04-29 03:00:46 -07:00
  • 4b80126913 Add check with Windows OS Ross Wheeler 2024-04-29 02:46:31 -07:00
  • 0e38ac7b8c Refactoring parameter size filling Eymen Ünay 2024-04-29 12:35:21 +03:00
  • f3b1bb70df Updating Makefile with latest changes Ross Wheeler 2024-04-29 02:28:03 -07:00
  • cf3e6efc79 Changed ordering of type configuration to easily see unchanged values Christopher Dryden 2024-04-29 02:28:23 +00:00
  • 109c62c6c6 Removed templates from kernels to clarify data types Christopher Dryden 2024-04-29 02:06:59 +00:00
  • 1e50f3b976 Only enable kernel4 with BF16 ademeure 2024-04-29 01:55:06 +01:00
  • 82fef03acd Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes) ademeure 2024-04-29 01:51:10 +01:00
  • 50acc125f3 Merge branch 'ngc92-split-file' Separates out common error-checking wrapper utils, that are broadly useful across all file Andrej Karpathy 2024-04-28 23:27:08 +00:00
  • c20497cfc1 add comments pointing to the definition of the utils functions Andrej Karpathy 2024-04-28 23:26:58 +00:00
  • b1c80e9d7b Merge branch 'split-file' of https://github.com/ngc92/llm.c into ngc92-split-file Andrej Karpathy 2024-04-28 23:23:00 +00:00
  • 4a3c278f3a moved checked helper functions into a separate file Erik Schultheis 2024-04-28 20:38:03 +03:00
  • 18b41b4df6 Merge pull request #279 from Ricardicus/prec-ci Andrej 2024-04-28 13:47:48 -07:00
  • 66a92c6791 Merge branch 'ngc92-cleanup' Andrej Karpathy 2024-04-28 20:45:04 +00:00
  • 518565654e Merge branch 'cleanup' of https://github.com/ngc92/llm.c into ngc92-cleanup Andrej Karpathy 2024-04-28 20:43:37 +00:00
  • 49228b091a add small comment on -t=0 Andrej Karpathy 2024-04-28 20:20:51 +00:00
  • 4c295c7e97 and even more cleanup Erik Schultheis 2024-04-28 23:12:32 +03:00
  • ca48791522 as promised, cleanup enabled by padding :) Erik Schultheis 2024-04-28 23:04:59 +03:00
  • b522333851 Updating the CI to build different precisions Rickard Hallerbäck 2024-04-28 20:15:37 +02:00
  • 10aa24e2cc Merge pull request #269 from ChrisDryden/patch-3 Andrej 2024-04-28 13:16:47 -07:00
  • 4b6f68a9a9 Merge branch 'lancerts-encoder_forward-float4' Andrej Karpathy 2024-04-28 20:10:48 +00:00
  • 327eef3f22 incorporate faster encoder_forward kernel to fp32 CUDA version Andrej Karpathy 2024-04-28 20:10:33 +00:00
  • 4b6a532b9b Merge branch 'encoder_forward-float4' of https://github.com/lancerts/llm.c into lancerts-encoder_forward-float4 Andrej Karpathy 2024-04-28 19:49:16 +00:00
  • b7972ff928 make padded vocab fixes in the .c code as well, i missed it in the previous PR, should satisfy the CI now Andrej Karpathy 2024-04-28 19:33:21 +00:00
  • 835060e18d padded vocab change. touched a lot of code. very stressful and error prone, but i think it is done. had to bump versions on all .bin files, invalidating the previous files. re-run the python training script to re-export the new version files. let's not do too much of things like this in the future lol. actually, fun fact i had a chance to do the padded vocab really really early in the history of llm.c development, and chose not do it, thinking i'll just do it later. i should have done it. such is life, you make mistakes, you accumulate scar tissue, and you learn, and you become better, faster, stronger. this is the mindset one must have to lead a happy and fulfilling life. it's not important that you are perfect at any point in time, it's only important that you keep improving, every day. Andrej Karpathy 2024-04-28 18:47:03 +00:00