mirrors / karpathy--llm.c

提交

提交图

  • 2287da0120 enable bf16 Erik Schultheis 2024-05-12 19:42:32 +03:00
  • c1814d5d2f Merge pull request #401 from rosslwheeler/ci-add-win Andrej 2024-05-12 18:32:13 +02:00
  • a2bffb5a8a Merge pull request #403 from nietras/nietras-llm-cs Andrej 2024-05-12 18:29:20 +02:00
  • ec92368f8b added current backward bias kernel to dev cuda Erik Schultheis 2024-05-12 19:24:43 +03:00
  • e64df91191 Add Llm.cs by nietras to README ntr 2024-05-12 16:02:36 +02:00
  • b88f683569 Add Windows to CI Ross Wheeler 2024-05-11 17:49:22 -07:00
  • 1cfe899121 Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-05-11 17:21:18 +00:00
  • fb86c121a1 Merge branch 'Anerudhan-feature/simplify_cudnn_usage' Andrej Karpathy 2024-05-11 17:21:04 +00:00
  • 3dbb0bb89c bump the threshold for qkvw because flashattention expands the error here a tiny bit Andrej Karpathy 2024-05-11 17:19:41 +00:00
  • 58d9dda497 Merge branch 'feature/simplify_cudnn_usage' of https://github.com/Anerudhan/llm.c into Anerudhan-feature/simplify_cudnn_usage Andrej Karpathy 2024-05-11 17:08:31 +00:00
  • 5dbda7e115 Merge pull request #397 from lancerts/comment-fix Andrej 2024-05-11 19:03:40 +02:00
  • 923b2b0a8f Merge pull request #398 from zwets/fix-cudnn-frontend-check Andrej 2024-05-10 12:13:53 +02:00
  • 6da5e63e2c Fix detection of cudnn-frontend in '.' on Linux Marco van Zwetselaar 2024-05-10 10:57:39 +03:00
  • 5c90845f7a update the -lcublas -lcublasLt flag in the comment lancer 2024-05-09 22:04:02 -07:00
  • fd007d6508 Merge pull request #395 from yijunyu/patch-1 Andrej 2024-05-10 01:18:25 +02:00
  • 83ad9ac87f Fused Residual+LayerNorm layers, ~1% faster Andrej Karpathy 2024-05-09 23:16:41 +00:00
  • 43c48707a7 Merge branch 'fuse-ln-res-2' of https://github.com/ngc92/llm.c into ngc92-fuse-ln-res-2 Andrej Karpathy 2024-05-09 23:09:53 +00:00
  • 8ccf2f97f8 Update README.md Yijun Yu 2024-05-10 06:49:24 +08:00
  • 75ec629f52 remove duplicate function ngc92 2024-05-10 00:03:30 +02:00
  • 5920143d47 remove warning noise ngc92 2024-05-10 00:02:37 +02:00
  • 691c1df969 fused layernorm+residual ngc92 2024-05-09 23:48:11 +02:00
  • 01c7a331bb - Simplify graph cache and usage of cudnn. - Fix failures in H100 Anerudhan 2024-05-09 19:07:00 +00:00
  • 077b649ee1 Merge pull request #389 from vedaanta/cudnn/install Andrej 2024-05-09 20:30:48 +02:00
  • 99e765de8d moved bf16 boilerplate to common.h ngc92 2024-05-09 11:19:24 +02:00
  • 69aa64cc80 Adds cuDNN install instructions for a smaller but sufficient package Joe Halabi 2024-05-07 17:18:03 -07:00
  • b8eaafdf9b Merge pull request #386 from Anerudhan/feature/rename_cudnn_attn Andrej 2024-05-08 23:00:57 +02:00
  • 68509c89a2 Merge branch 'pjj-fix-gpt2-xl' Andrej Karpathy 2024-05-08 20:20:27 +00:00
  • 0c73ba6ca5 Merge branch 'fix-gpt2-xl' of https://github.com/pjj/llm.c into pjj-fix-gpt2-xl Andrej Karpathy 2024-05-08 20:19:15 +00:00
  • 26dbbc75c2 unbreak the tests oops Andrej Karpathy 2024-05-08 20:18:33 +00:00
  • 84a2aa0dd0 allow exporting all GPT-2 model sizes now, using --model flag in python script. also allow loading all the models from C, using the new flag -e, to point directly to the file to load. added some error handling for potentially common mistakes. note that loadin GPT-2 XL does not work yet and crashes, but I am imminently merging a fix as the next commit that resolves this Andrej Karpathy 2024-05-08 20:13:14 +00:00
  • 0ce5fcf7e8 Rename cudnn_att.cu to cudnn_att.cpp to speed up compilation. Anerudhan 2024-05-08 19:53:48 +00:00
  • 99e5c5d283 Allow the code to train gpt2-xl Paul Maragakis 2024-05-08 15:17:25 -04:00
  • 1c5ef8cea3 Merge pull request #382 from ahrefs/fix-multigpu-hang Andrej 2024-05-08 19:50:00 +02:00
  • 6d7a99cb4e Adjust comments KarhouTam 2024-05-08 17:45:14 +08:00
  • 6a52d8619e Optimize codes and comments KarhouTam 2024-05-08 16:12:39 +08:00
  • 2356be7333 set stream to main_stream in ncclAllReduce chinthysl 2024-05-08 05:02:23 +00:00
  • dc901d420b set correct gpu using multigpu config chinthysl 2024-05-08 04:50:24 +00:00
  • b3e8a9fe67 Implementation of online softmax forward kernel without cgs. KarhouTam 2024-05-08 09:22:14 +08:00
  • 2f6c545131 Merge branch 'ademeure-more_kernel_opt' Andrej Karpathy 2024-05-07 09:36:37 +00:00
  • 5b070903ce Merge branch 'more_kernel_opt' of https://github.com/ademeure/llm.c into ademeure-more_kernel_opt Andrej Karpathy 2024-05-07 09:31:12 +00:00
  • c261240852 3 x 512 threads max for layernorm_backward to avoid cache thrashing (hacky -> better way?) ademeure 2024-05-07 03:43:27 +01:00
  • 0141408ca5 Merge pull request #357 from rosslwheeler/cudnn_makefile_changes Andrej 2024-05-07 02:50:11 +02:00
  • ce72242c46 Merge pull request #376 from rosslwheeler/train_gp2cu-porting-fixes Andrej 2024-05-07 02:47:47 +02:00
  • 3a6f89cebc Merge branch 'train_gp2cu-porting-fixes' of https://github.com/rosslwheeler/llm.c into train_gp2cu-porting-fixes Ross Wheeler 2024-05-06 17:41:47 -07:00
  • 2550754215 Change FLT_MAX to flt_max Ross Wheeler 2024-05-06 17:41:19 -07:00
  • 3cc16f1354 fix logits bug Andrej Karpathy 2024-05-07 00:39:25 +00:00
  • 60b819630d Merge branch 'karpathy:master' into train_gp2cu-porting-fixes rosslwheeler 2024-05-06 17:25:59 -07:00
  • e1f89b3043 remove BF16 default from classified_fused before PR ademeure 2024-05-07 00:57:01 +01:00
  • 1ea7f9bf25 tiny irrelevant optimisation to final unaligned fused_classifier loop + add missing common.h changes ademeure 2024-05-07 00:54:50 +01:00
  • ec9fd51d5c Merge pull request #375 from Chillee/patch-1 Andrej 2024-05-07 01:49:02 +02:00
  • aa54a0a808 Merge pull request #377 from ademeure/profiler_efficiency Andrej 2024-05-07 01:45:33 +02:00
  • 9b55ea8cf5 More crazy optimisations to layernorm_backward, fused_classifier, and matmul_backward_bias. ademeure 2024-05-07 00:11:42 +01:00
  • f7d7760084 Non-standard C syntax replace with standard C Ross Wheeler 2024-05-06 13:56:17 -07:00
  • 5adb6ef2c1 Add tensor core and overall efficiency stats to profiler script. ademeure 2024-05-06 21:24:42 +01:00
  • 69f6c4f765 Don't return logits during training for PyTorch baseline Horace He 2024-05-06 13:10:17 -07:00
  • 134f4c7fc8 Adding two directory search for cuDNN frontend files Ross Wheeler 2024-05-06 12:38:40 -07:00
  • dedd86c1b3 Merge branch 'karpathy:master' into cudnn_makefile_changes rosslwheeler 2024-05-06 11:20:22 -07:00
  • 3048a3a780 oops have to use labels= feature/wikitext103 Andrej Karpathy 2024-05-06 18:04:35 +00:00
  • b4278f094a v1 of what was supposed to be a repro of wikitext ppl, but isn't, what Andrej Karpathy 2024-05-06 18:03:10 +00:00
  • 4274d95b4a Merge pull request #365 from lancerts/classifier-fix Andrej 2024-05-06 12:04:16 +02:00
  • 6798dd4b91 Merge branch 'rosslwheeler-Makefile-detect-gpu' Andrej Karpathy 2024-05-06 10:01:57 +00:00
  • 7a8f471bc7 further reorganization chinthysl 2024-05-06 17:49:58 +08:00
  • 5344cc30b8 Merge branch 'Makefile-detect-gpu' of https://github.com/rosslwheeler/llm.c into Makefile-detect-gpu Ross Wheeler 2024-05-06 01:11:37 -07:00
  • 0c4908d815 Adding CI check to disable auto-detect Ross Wheeler 2024-05-06 01:11:27 -07:00
  • 98aa83b282 Merge branch 'karpathy:master' into Makefile-detect-gpu rosslwheeler 2024-05-06 00:58:09 -07:00
  • 2bfd2b6a7e Auto-detect GPU capability Ross Wheeler 2024-05-06 00:46:28 -07:00
  • aae7145495 Merge branch 'master' into zero-stage1 Chinthaka 2024-05-06 10:36:24 +08:00
  • bbfe8c989c Minor update on the code lancer 2024-05-05 16:59:58 -07:00
  • 9e4f9df9a2 Merge pull request #337 from lancerts/make-fix Andrej 2024-05-05 15:37:02 -07:00
  • a7c1e4437d Merge pull request #364 from ngc92/cleanup_may4-nvcc11 Andrej 2024-05-05 14:42:49 -07:00
  • d6e8f5e026 Merge branch 'ademeure-cleanup_may4' Andrej Karpathy 2024-05-05 21:29:41 +00:00
  • 804a9af1dc make things compile with nvcc11 Erik Schultheis 2024-05-05 23:43:02 +03:00
  • 876ab93c0a Add FP16 path for atomicStochasticAdd (+remove __bfloat1622float2 to work on older CUDAs) + fixes ademeure 2024-05-05 15:05:59 +01:00
  • 9910a40863 Removed makefile change so we can integrate #339 instead which feels slightly cleaner ademeure 2024-05-05 14:42:32 +01:00
  • 42176ad4d5 Merge branch 'cudnn_makefile_changes' of https://github.com/rosslwheeler/llm.c into cudnn_makefile_changes Ross Wheeler 2024-05-04 23:27:21 -07:00
  • 2d4e5fd840 Minor fixes for Makefile for cudnn and windows support Ross Wheeler 2024-05-04 23:24:41 -07:00
  • 9663719c62 Minor fixes for Make for cudnn and windows support Ross Wheeler 2024-05-04 23:24:41 -07:00
  • c15ca1f4cf Fix makefile ademeure 2024-05-05 02:49:16 +01:00
  • 7789738879 Add PTX back to binary + fix whitespaces ademeure 2024-05-05 02:44:24 +01:00
  • 8675104b73 Compile for the user's GPU architecture using nvidia-smi query on Linux ademeure 2024-05-05 02:24:31 +01:00
  • 83ec4b8e70 Slightly reduce lines of code in cudnn_att ademeure 2024-05-05 02:09:52 +01:00
  • c261eecda8 Merge remote-tracking branch 'karpathy/master' into cleanup_may4 ademeure 2024-05-05 01:49:25 +01:00
  • 64b6c2a77f Merge branch 'ngc92-separate-compilation' Andrej Karpathy 2024-05-05 00:41:18 +00:00
  • f2224f2eb2 Merge branch 'separate-compilation' of https://github.com/ngc92/llm.c into ngc92-separate-compilation Andrej Karpathy 2024-05-05 00:32:31 +00:00
  • 8168b78549 Merge branch 'ademeure-less_idle_more_brrr-3' Andrej Karpathy 2024-05-05 00:27:15 +00:00
  • ce333de156 add ema to tok/s Andrej Karpathy 2024-05-05 00:18:39 +00:00
  • 2642ffd19c Merge branch 'less_idle_more_brrr' of https://github.com/ademeure/llm.c into ademeure-less_idle_more_brrr-3 Andrej Karpathy 2024-05-05 00:05:38 +00:00
  • a0b79358bd Fix loss not being calculated for forward-only paths ademeure 2024-05-05 01:01:59 +01:00
  • 6c179faaeb Merge pull request #349 from lancerts/fix-include Andrej 2024-05-04 17:01:32 -07:00
  • b087b9c819 don't compile/link cudnn if not asked for it Erik Schultheis 2024-05-05 02:44:15 +03:00
  • 34304361ba fix merge conflict ademeure 2024-05-05 00:16:49 +01:00
  • ec0ab2d2d0 Remove arch=native as it only available on recent CUDA versions ademeure 2024-05-05 00:00:58 +01:00
  • 18d7ed9242 revert profile.py changes for now ademeure 2024-05-04 23:46:39 +01:00
  • abaaceb801 Added makefile gencode changes ademeure 2024-05-04 23:43:13 +01:00
  • bfb9c51446 refactoring & remove unused functions to reduce LOC (+wip profile.py improvements) ademeure 2024-05-04 23:40:15 +01:00
  • 66b9755ce9 Use template for blockReduce's warp reduction + add out_of_bounds parameter ademeure 2024-05-04 18:48:54 +01:00
  • 8d593b916e fix cuda stream bug for test_gpt2cu... by removing most of the memcpy/memset parallelism for now (keeping loss_event to avoid sync between forward and backward). ademeure 2024-05-04 18:18:51 +01:00
  • 19c290d7e6 improved debugging for cudnn Erik Schultheis 2024-05-04 12:31:15 +03:00
  • aa5bb258b6 fixed up test and profile targets Erik Schultheis 2024-05-04 11:59:59 +03:00
  • 58ac83f5ec Merge branch 'master' into zero-stage1 Chinthaka 2024-05-04 16:24:22 +08:00