mirrors / karpathy--llm.c

提交

提交图

  • 963aa6828e add gpt-2 774M repro script Andrej Karpathy 2024-06-12 15:33:19 +00:00
  • 8818a75157 fixup to profiling script Erik Schultheis 2024-06-10 11:15:26 +03:00
  • 368644959a fix non-multi-gpu case Erik Schultheis 2024-06-09 18:14:56 +03:00
  • 8cba9ba0c0 cleanup Erik Schultheis 2024-06-09 18:07:33 +03:00
  • ecca976132 re-enable reduce-scatter operation Erik Schultheis 2024-06-09 15:51:24 +03:00
  • ecdb0ca86e refactored code Erik Schultheis 2024-06-09 12:55:17 +03:00
  • fdcb30256d first working ZeRO-1 Erik Schultheis 2024-06-09 02:16:14 +03:00
  • 6d2c72cb4b rearranged adam code Erik Schultheis 2024-06-09 01:17:34 +03:00
  • 50a9411114 overlap backward with nccl for non-ZeRO runs Erik Schultheis 2024-06-09 00:45:16 +03:00
  • d6793645f5 replace cublaslt with custom matmul Erik Schultheis 2024-06-12 09:35:42 +03:00
  • 7f3f3acf16 shared memory matmul Erik Schultheis 2024-06-11 08:23:54 +03:00
  • 071c977c79 vectorized memory access Erik Schultheis 2024-06-11 00:46:51 +03:00
  • bb89efdcb1 register reuse Erik Schultheis 2024-06-11 00:41:08 +03:00
  • 8a3a29ecb7 Merge branch 'karpathy:master' into master Vyom Sharma 2024-06-10 13:54:29 +05:30
  • 615ec0becd Merge pull request #523 from ngc92/testability Andrej 2024-06-09 11:02:39 -07:00
  • 02e30faef2 bugfix for LN recomputation Erik Schultheis 2024-06-08 21:20:59 +03:00
  • 8fbfba2864 allow setting master weights and recompute in tests Erik Schultheis 2024-06-02 23:06:35 +03:00
  • 1bf33cc66d make sure failed testing returns failure status Erik Schultheis 2024-06-02 22:33:21 +03:00
  • d2482ff1ad Merge pull request #570 from rosslwheeler/feature/ubicloud-gpu-ci Andrej 2024-06-08 11:01:03 -07:00
  • 637c1b69c4 Merge pull request #552 from karpathy/feature/streams Andrej 2024-06-08 09:25:56 -07:00
  • ee6b3c9325 take out Async copies and memsets feature/streams Andrej Karpathy 2024-06-08 16:17:02 +00:00
  • 077451927f Merge pull request #546 from kmyusk/master Andrej 2024-06-07 21:37:51 -07:00
  • d396cd18b7 Merge pull request #567 from gordicaleksa/improve_logdir_check_logic Andrej 2024-06-07 09:22:32 -07:00
  • 58415736a4 Merge pull request #568 from gordicaleksa/improve_logger Andrej 2024-06-07 09:20:29 -07:00
  • 40e7289916 Make it cross-platform, support Windows Aleksa Gordic 2024-06-07 17:49:49 +02:00
  • d3790c64a1 Change default path Aleksa Gordic 2024-06-07 17:38:01 +02:00
  • 4090ca8fa1 Add learning rate and grad norm logging Aleksa Gordic 2024-06-07 17:34:42 +02:00
  • aa3d8aca86 Minor fix - better handling of out log dir Aleksa Gordic 2024-06-07 17:06:53 +02:00
  • 089fdd7f23 Merge branch 'master' into fix-warning lancerts 2024-06-07 07:17:52 -07:00
  • ecb13ea74d missing parameter in comments added vyom1611 2024-06-07 16:14:59 +05:30
  • 009be4dcab Merge remote-tracking branch 'origin/master' vyom1611 2024-06-07 14:21:35 +05:30
  • 58c59064ec Most updated working modal script vyom1611 2024-06-07 14:21:13 +05:30
  • c24ef4a4b4 Merge branch 'karpathy:master' into master Vyom Sharma 2024-06-07 14:20:55 +05:30
  • 73506dff2d Adding GPU CI workflow file Ross Wheeler 2024-06-06 23:27:45 -07:00
  • 8a0809b39c Merge pull request #560 from gordicaleksa/refactor_cuda_kernels Andrej 2024-06-06 20:10:52 -07:00
  • 4b41087c64 fix the compiler warnings and errors lancerts 2024-06-06 11:46:40 -07:00
  • 7b99ac96a5 Fix mem leaks, reduce memory, refactor Aleksa Gordic 2024-06-06 19:58:50 +02:00
  • d931fc0507 -std=c++17 default, merge PR #559 from ngc92/ubuntu20.04 Andrej 2024-06-06 09:23:07 -07:00
  • 875196b672 add c++17 flag also to dev/cuda Erik Schultheis 2024-06-06 19:02:45 +03:00
  • 067aadc226 Merge pull request #553 from gordicaleksa/refactor_trimat Andrej 2024-06-06 08:56:12 -07:00
  • bb9c895400 fix Erik Schultheis 2024-06-06 12:02:33 +03:00
  • f4dbd88151 enable cudnn Erik Schultheis 2024-06-06 11:59:45 +03:00
  • 52ea988860 fixup image name Erik Schultheis 2024-06-06 11:01:10 +03:00
  • 43f013631b set compile flag and add ci check Erik Schultheis 2024-06-06 10:55:33 +03:00
  • c116fbfa29 Fix q,k bug, move all kernels to q@k^t form Aleksa Gordic 2024-06-05 22:22:55 +02:00
  • 875d362381 Refactor trimat Aleksa Gordic 2024-06-05 17:33:26 +02:00
  • 1e8efe94c0 Update Makefile - revert back Vyom Sharma 2024-06-05 12:02:09 +05:30
  • 6d14c29df5 streams, everything right now is on main stream, almost Andrej Karpathy 2024-06-05 01:01:24 +00:00
  • d5f7458e03 Merge branch 'stream-2' of https://github.com/ngc92/llm.c into ngc92-stream-2 Andrej Karpathy 2024-06-05 00:23:19 +00:00
  • b1171051dd Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-06-05 00:19:47 +00:00
  • 9a07817380 fix minor bugs in MFU calculation Andrej Karpathy 2024-06-05 00:15:08 +00:00
  • 9af7628063 moved remaining memcpy/memset in the training loop to main_stream Erik Schultheis 2024-06-05 02:06:32 +03:00
  • 8940dcdcb1 Merge branch 'stream-2' of https://github.com/ngc92/llm.c into ngc92-stream-2 Andrej Karpathy 2024-06-04 22:52:26 +00:00
  • bc35d3438e naming the stream Erik Schultheis 2024-06-05 01:50:16 +03:00
  • 0ee2152106 Merge pull request #549 from rosslwheeler/feature/new-branch-ci-trigger Andrej 2024-06-04 15:42:08 -07:00
  • 0de54d10da moved last remaining kernel to main stream Erik Schultheis 2024-06-05 01:32:25 +03:00
  • 04b59c87be tune comments Andrej Karpathy 2024-06-04 22:16:23 +00:00
  • 894ada5041 Merge pull request #551 from gordicaleksa/fix_loss_bug Andrej 2024-06-04 15:10:10 -07:00
  • 6f20ee9f8e Remove * 2 for total batch size Aleksa Gordic 2024-06-04 23:42:54 +02:00
  • 34766253cf Move all reduce outside of the gradacc loop for efficiency Aleksa Gordic 2024-06-04 23:39:52 +02:00
  • 724085ecd6 tmp Aleksa Gordic 2024-06-04 23:23:11 +02:00
  • 4c369eea0f overlap validity check and data movement Erik Schultheis 2024-06-04 22:47:54 +03:00
  • ca7caac5c5 move all kernels into a main_stream in preparation for future parallelism Erik Schultheis 2024-06-02 22:29:04 +03:00
  • ab8b040fdd Merge remote-tracking branch 'origin/master' vyom1611 2024-06-04 23:39:50 +05:30
  • 4564a24c69 Modal script updated for cudnn and gcc version match vyom1611 2024-06-04 23:39:36 +05:30
  • 453fde77fd Merge branch 'karpathy:master' into master Vyom Sharma 2024-06-04 23:39:10 +05:30
  • d51c8db395 Trigger CI on new branch creation Ross Wheeler 2024-06-04 10:14:42 -07:00
  • 5379b1d9e9 Merge pull request #548 from gordicaleksa/fix_zerograds_bug Andrej 2024-06-04 09:51:44 -07:00
  • 3e44d275d8 Update comment Aleksa Gordic 2024-06-04 18:48:29 +02:00
  • ec92ff64a2 Improve the design and comment Aleksa Gordic 2024-06-04 18:47:23 +02:00
  • 54ec686649 Fix zero grads bug Aleksa Gordic 2024-06-04 18:40:54 +02:00
  • c73b4668f3 Merge pull request #547 from ngc92/includes Andrej 2024-06-04 08:56:41 -07:00
  • b02aab1f16 made implicit includes of standard headers explicit Erik Schultheis 2024-06-04 13:20:15 +03:00
  • 1ee0b437ca Fix periodic inference during training Yusik Kim 2024-06-04 10:23:42 +02:00
  • cb40f458a3 Merge pull request #543 from rosslwheeler/bugfix/make-clean Andrej 2024-06-03 21:06:35 -07:00
  • 4bfb23948d Make clean fix and Windows cuDNN build fix Ross Wheeler 2024-06-03 20:26:52 -07:00
  • 45b0d1b378 copy and cast kernel move to cuda_utils, good general util Andrej Karpathy 2024-06-03 23:14:59 +00:00
  • d697eaecfd move the global norm kernel Andrej Karpathy 2024-06-03 23:12:20 +00:00
  • 8c52a652b4 move adamw kernel to llmc Andrej Karpathy 2024-06-03 23:09:42 +00:00
  • e484c650b5 layernorm and residual, yolo push to master Andrej Karpathy 2024-06-03 23:02:46 +00:00
  • 7d50d3318a Merge pull request #541 from karpathy/move/matmul Andrej 2024-06-03 15:57:09 -07:00
  • 094988738b adjust the comment on extern deviceProp to be more descriptive move/matmul Andrej Karpathy 2024-06-03 22:57:00 +00:00
  • ec3921047c updated the script to run the same number of threads as files Christopher 2024-06-03 22:54:25 +00:00
  • 645feac1e7 move matmul Andrej Karpathy 2024-06-03 22:54:07 +00:00
  • 3936492620 Forgot the folder name in the readme Christopher 2024-06-03 22:51:57 +00:00
  • 027a43e0f5 Adds simplified way to download the already tokenized tinyshakespeare dataset and gpt2 weights Christopher 2024-06-03 22:49:50 +00:00
  • 299d375fbc Merge pull request #540 from karpathy/move/fused_classifier Andrej 2024-06-03 15:38:24 -07:00
  • 90a7bfecf1 move fused classifier move/fused_classifier Andrej Karpathy 2024-06-03 22:35:20 +00:00
  • dd4191fbd5 Merge pull request #539 from karpathy/move/attention Andrej 2024-06-03 15:23:27 -07:00
  • 2186cd6fe7 move attention move/attention Andrej Karpathy 2024-06-03 22:19:47 +00:00
  • 3427983615 Merge pull request #537 from karpathy/move/gelu Andrej 2024-06-03 15:08:22 -07:00
  • d8d55e0a08 move gelu to llmc move/gelu Andrej Karpathy 2024-06-03 21:57:02 +00:00
  • 49b9776e50 also move the launchers move/encoder_kernel Andrej Karpathy 2024-06-03 21:37:18 +00:00
  • 93546aed01 move encoder, kernel 1/N Andrej Karpathy 2024-06-03 21:21:28 +00:00
  • 3392671826 Merge pull request #535 from ngc92/speed-up-compile Andrej 2024-06-03 14:11:10 -07:00
  • 67502c7f8e moved device utilities into a separate file, and follow-up cleanup Erik Schultheis 2024-06-03 23:58:38 +03:00
  • 8cc39a794b Merge pull request #528 from gordicaleksa/refactor_evalloader Andrej 2024-06-03 13:54:52 -07:00
  • 2a6a5b2c6c Merge pull request #529 from gordicaleksa/improve_start_training_print_info Andrej 2024-06-03 13:53:32 -07:00
  • 0cd9f1ac21 Merge pull request #530 from gordicaleksa/minor_refactor Andrej 2024-06-03 13:52:42 -07:00
  • d17893b754 Merge pull request #532 from ngc92/ln-buffers Andrej 2024-06-03 13:51:22 -07:00