mirrors / karpathy--llm.c

提交

提交图

  • 312b043697 Merge pull request #122 from ngc92/fix-illegal-access Andrej 2024-04-14 12:19:03 -07:00
  • 5c9d61fd35 Prevent out-of-bounds token-ids in input padding for test generation Erik Schultheis 2024-04-14 13:05:57 +03:00
  • b9ad95e74c attempt to fix, but it's not working huh fix/illegalaccess Andrej Karpathy 2024-04-14 04:40:35 +00:00
  • 780d8bd012 initial version Erik Schultheis 2024-04-14 03:48:56 +03:00
  • 534b93a732 erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7 Andrej Karpathy 2024-04-13 22:02:31 +00:00
  • 0554c7e414 Merge pull request #115 from ademeure/softmax_largeC Andrej 2024-04-13 14:49:26 -07:00
  • fc3dce37ce Add __ldcs to a few reads for which the memory will never be read again in the forward pass ademeure 2024-04-13 22:38:12 +01:00
  • 321b042657 New softmax for non-self-attention softmax case with large Cs ademeure 2024-04-13 22:31:23 +01:00
  • c02bae2ebc average iterates in code Andrej Karpathy 2024-04-13 21:09:37 +00:00
  • c70fbd059b refine comment on time Andrej Karpathy 2024-04-13 21:09:25 +00:00
  • ca242a3d0b get rid of spurious handle variable creation, remnant from before, and update README Andrej Karpathy 2024-04-13 21:04:49 +00:00
  • c65d1d305b Merge pull request #113 from ademeure/cublasTF32 Andrej 2024-04-13 13:56:13 -07:00
  • 5ab44973b0 Fix cuBLAS calls to use pre-initialised handle with TF32 enabled (big perf gain) ademeure 2024-04-13 20:35:53 +01:00
  • dd3dc179a4 git push origin masterMerge branch 'ngc92-common' Andrej Karpathy 2024-04-13 17:57:33 +00:00
  • c5ee371564 make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common Andrej Karpathy 2024-04-13 17:56:18 +00:00
  • 82623518a4 fix comment block of gelu forward kernel Andrej Karpathy 2024-04-13 17:48:53 +00:00
  • 02b5b33c5f remove unused variable C Andrej Karpathy 2024-04-13 17:48:11 +00:00
  • 84145411d3 fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nans Andrej Karpathy 2024-04-13 17:47:19 +00:00
  • 69f0385237 add comments and handling of kernels and their correctness Andrej Karpathy 2024-04-13 17:44:31 +00:00
  • 035b9e76fc Merge branch 'common' of https://github.com/ngc92/llm.c into ngc92-common Andrej Karpathy 2024-04-13 17:25:06 +00:00
  • 552b972e52 Merge branch 'zocterminal-warnings' Andrej Karpathy 2024-04-13 16:22:10 +00:00
  • f347957edf change comment style Andrej Karpathy 2024-04-13 16:21:57 +00:00
  • f459bfa833 fixed re-initialization of device memory after test Erik Schultheis 2024-04-13 18:23:12 +03:00
  • d035725d22 mark problematic tests Erik Schultheis 2024-04-13 18:09:28 +03:00
  • 59bada94d9 improved testing Erik Schultheis 2024-04-13 17:06:52 +03:00
  • ae63f6d008 warnings in train_gpt2.cu Markus Schmidt 2024-04-13 12:52:54 +00:00
  • 7e7460cfbd const'ness fixes Erik Schultheis 2024-04-13 13:37:54 +03:00
  • 7e8e038de8 extracted common testing functions into their own header Erik Schultheis 2024-04-13 13:00:05 +03:00
  • 6b49ed1c0b incorporate the fastest kernels and adjust readme Andrej Karpathy 2024-04-13 02:58:23 +00:00
  • 4f75e64573 Merge branch 'ngc92-attention-softmax-2' Andrej Karpathy 2024-04-13 02:39:10 +00:00
  • 6f9e28a1b6 few comments Andrej Karpathy 2024-04-13 02:38:57 +00:00
  • 2f99b3e608 Merge branch 'attention-softmax' of https://github.com/ngc92/llm.c into ngc92-attention-softmax-2 Andrej Karpathy 2024-04-13 02:34:06 +00:00
  • bd743f806a Merge branch 'lancerts-online_softmax' Andrej Karpathy 2024-04-13 02:11:50 +00:00
  • 40b78be722 small changes to comments, and very minor code modifications and movements Andrej Karpathy 2024-04-13 02:11:36 +00:00
  • 6250336c09 Merge branch 'online_softmax' of https://github.com/lancerts/llm.c into lancerts-online_softmax Andrej Karpathy 2024-04-13 01:54:16 +00:00
  • a18e517c1a alright let's start the backward pass. crossentropy_softmax_backward first, ez one Andrej Karpathy 2024-04-13 01:52:55 +00:00
  • 492a4b151a oops we missed a comment here, remove as we got rid of the inline mallocs now, so we're all ok Andrej Karpathy 2024-04-13 01:21:31 +00:00
  • e9f095208a Merge pull request #100 from ademeure/cudahostmalloc Andrej 2024-04-12 18:19:38 -07:00
  • 2616a6461d Use cudaHostMalloc for inputs/targets and cpu_losses (+moved its malloc/free away from main loop). ademeure 2024-04-13 01:31:16 +01:00
  • 3ab2039e32 update timings Andrej Karpathy 2024-04-12 23:47:15 +00:00
  • 59b9e8b96a fix handles in cublas in attention forward as well Andrej Karpathy 2024-04-12 23:15:47 +00:00
  • 8822e7803e use cublaslt, which fuses bias, and also use tf32 when your GPU supports it (#98) Andrej 2024-04-12 15:32:07 -07:00
  • e048825f5a div_ceil Erik Schultheis 2024-04-13 01:20:09 +03:00
  • 44cd50cb25 fix all the handles and things feature/cublaslt Andrej Karpathy 2024-04-12 22:18:18 +00:00
  • 5364e0b853 change main train/test code to use new cublaslt functions. inference now causes segfault though, very strange have to fix before merge Andrej Karpathy 2024-04-12 21:23:30 +00:00
  • 8cf66fbb84 change workspace from 4mib to 32mib so we don't have to detect hopper. seems ok for now Andrej Karpathy 2024-04-12 20:41:08 +00:00
  • bfd11ce5b6 use cublaslt and optionally tf32, which fuses bias and deprecates the need for add_bias kernel Andrej Karpathy 2024-04-12 19:38:36 +00:00
  • 8e87aac341 online softmax Erik Schultheis 2024-04-12 12:47:48 +03:00
  • de98a9a44a vector loads Erik Schultheis 2024-04-12 11:24:56 +03:00
  • 6396e393e3 add links to discussions Andrej Karpathy 2024-04-12 01:11:29 +00:00
  • baba3cfa01 recalculate instead of memory round-trip Erik Schultheis 2024-04-12 02:20:20 +03:00
  • 69d0e5bac7 Include a parallel kernal of softmax_online lancer 2024-04-11 15:30:36 -07:00
  • ca1e5ee04d Include a parallel kernal of softmax_online lancer 2024-04-11 15:24:41 -07:00
  • dec54904b4 one more simple clarifying comment Andrej Karpathy 2024-04-11 22:24:38 +00:00
  • d680fbad25 one more comment on attention Andrej Karpathy 2024-04-11 21:49:14 +00:00
  • 9710163a60 did one pass of additional comments, potentially helpful, as i think a lot of people may be looking at this for the first time Andrej Karpathy 2024-04-11 21:45:46 +00:00
  • 5f99fb30d2 cooperative groups and fused scale kernel Erik Schultheis 2024-04-11 22:11:28 +03:00
  • d6bcc129b9 Improve the efficiency lancer 2024-04-11 12:16:20 -07:00
  • 00010c7188 Improve the efficiency lancer 2024-04-11 12:13:34 -07:00
  • 2c81198101 make the new cooperative groups layernorm kernel the default. shaves off aonly bout 1ms of the total running time though Andrej Karpathy 2024-04-11 18:23:46 +00:00
  • 6f5ec063f0 Merge branch 'ngc92-layer-norm' Andrej Karpathy 2024-04-11 18:22:40 +00:00
  • 00b9fda4a1 ceildiv in layernorm Andrej Karpathy 2024-04-11 18:21:34 +00:00
  • 089cd5f1dd make sure we also test correctness of mean and rstd and do it all block sizes Andrej Karpathy 2024-04-11 18:12:26 +00:00
  • f28613e0b9 small edits to comments and variables Andrej Karpathy 2024-04-11 18:09:54 +00:00
  • 2553518c97 Merge branch 'layer-norm' of https://github.com/ngc92/llm.c into ngc92-layer-norm Andrej Karpathy 2024-04-11 18:07:00 +00:00
  • 4e91362fa2 optionally generate mean and rstd Erik Schultheis 2024-04-11 20:48:14 +03:00
  • f524a90de3 Include the online softmax CPU code (from the paper "Online normalizer calculation for softmax") and native port to GPU kernel. Both the CPU and GPU kernels' correctness are verified by executing the script. lancer 2024-04-11 09:47:50 -07:00
  • 79b8af4a9b Merge pull request #56 from scotthaleen/omp-osx-intel Andrej 2024-04-11 09:13:10 -07:00
  • 21a250d372 Merge pull request #64 from krrishnarraj/master Andrej 2024-04-11 09:12:07 -07:00
  • 1751af1607 inline docs for sizes Andrej Karpathy 2024-04-11 16:04:14 +00:00
  • bcd0dad6c8 Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-04-11 15:50:10 +00:00
  • f26cf00a61 make gelu constant be a #define, speeds up the kernel by 1% or so Andrej Karpathy 2024-04-11 15:49:01 +00:00
  • f46d5b2e2a Merge pull request #72 from Soldy/master Andrej 2024-04-11 08:38:57 -07:00
  • 3aaed922a3 -O3 cannot go with -Ofast soldy 2024-04-11 16:27:59 +01:00
  • 4542f8913a [train_gpt2.py] synchronize based on device Krishnaraj Bhat 2024-04-11 17:06:51 +05:30
  • 47b2f9312c fixed homebrew path for intel based macs Scott Haleen 2024-04-10 23:01:07 -04:00
  • afddb1af5b fixed block size and improved load op hint Erik Schultheis 2024-04-11 03:47:38 +03:00
  • 466723bb06 fully fused layer-norm kernel Erik Schultheis 2024-04-11 00:36:48 +03:00
  • a08c11b60e Merge pull request #48 from lancerts/fix_matmul Andrej 2024-04-10 13:44:22 -07:00
  • 68d2ce0601 fix error in small typos in matmul_forward.cu lancer 2024-04-10 13:35:20 -07:00
  • ef7a4867b1 Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-04-10 19:25:13 +00:00
  • e1e0a08035 change fabs to fabsf Andrej Karpathy 2024-04-10 19:23:18 +00:00
  • 919b2200a0 Merge pull request #20 from varunlakkur/master Andrej 2024-04-10 12:21:07 -07:00
  • 9f5069d3b2 Merge pull request #38 from lancerts/fix_matmul Andrej 2024-04-10 12:15:53 -07:00
  • c0d57b1c45 Merge pull request #47 from eltociear/patch-1 Andrej 2024-04-10 12:14:56 -07:00
  • add9e77daa mention compilation issue that seems to be common. we may want to resolve this inside the Makefile though Andrej Karpathy 2024-04-10 19:11:44 +00:00
  • fd8459d858 small gitignore for now. probably iterate on Andrej Karpathy 2024-04-10 18:54:23 +00:00
  • 15fad9bc96 Create LICENSE MIT Andrej 2024-04-10 11:47:40 -07:00
  • 80f52e5290 the full forward pass of GPT-2 in one file of pure CUDA Andrej Karpathy 2024-04-10 18:15:55 +00:00
  • ca137b02d5 update layernorm.md Ikko Eltociear Ashimine 2024-04-11 00:07:22 +09:00
  • d8e2a36f15 ok crossentropy is fairly ez, just have to be careful with logs of negative numbers in debug tests. so this should be all the layers now Andrej Karpathy 2024-04-10 01:13:20 +00:00
  • 46ee0a3944 sorry there should be no mallocs inside these functions, have to pass in buffer memory from outside. getting tired Andrej Karpathy 2024-04-10 00:55:14 +00:00
  • 919ae1f27b ok attention is now optimized too, by 20X, nice. actually this one was really painful, i had to make sure all the cuBLAS calls take all the right arguments and deal with the fact that cuBLAS is column major instead of row major. i go for nap now Andrej Karpathy 2024-04-10 00:50:30 +00:00
  • fbd679adec fix the consistency of the transpose notation lancer 2024-04-09 15:19:55 -07:00
  • 8386e5393c intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice Andrej Karpathy 2024-04-09 20:08:11 +00:00
  • c29d70a9b3 add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions Andrej Karpathy 2024-04-09 19:22:47 +00:00
  • 327705d03f mention ongoing cuda work in main readme Andrej Karpathy 2024-04-09 18:29:24 +00:00
  • ea1fad0337 Merge pull request #33 from lancerts/fix_M_PI_undefined Andrej 2024-04-09 11:25:23 -07:00
  • e3e0d73fbc Merge pull request #34 from VinciGit00/patch-1 Andrej 2024-04-09 11:24:32 -07:00
  • bb4f04bdf8 add flash attention, but a naive implementation there of Andrej Karpathy 2024-04-09 18:22:49 +00:00