mirrors / karpathy--llm.c

提交

提交图

  • 9ed0f10ee0 Merge branch 'master' of https://github.com/patricxu/llm.c into patricxu-master Andrej Karpathy 2024-04-22 00:26:08 +00:00
  • fe85da4f3e Merge pull request #209 from Ricardicus/master Andrej 2024-04-21 17:22:53 -07:00
  • 3d2605bf8a Merge branch 'master' into master Rickard Hallerbäck 2024-04-22 02:21:49 +02:00
  • 4e4eba0441 simplify bool Andrej Karpathy 2024-04-22 00:19:00 +00:00
  • a332ea5d92 Merge pull request #208 from Ricardicus/train_gpt2.py-fix Andrej 2024-04-21 17:18:13 -07:00
  • c7aad65a04 Adding a workflow that builds with and without CUDA and OMP and on CPU also tests on ubuntu and macos Rickard Hallerbäck 2024-04-21 23:35:30 +02:00
  • 2d97be8315 Accidental last minute change Chris Dryden 2024-04-21 16:44:07 -07:00
  • 2752212151 Added shared memory for the atomic additions for the layernorm_backward.cu weights and biases Chris Dryden 2024-04-21 16:37:00 -07:00
  • 6c163d3391 train_gpt2.py: fix for conditional use of fused and for computing gradients Rickard Hallerbäck 2024-04-22 00:19:13 +02:00
  • 4f8f5fa833 Merge pull request #207 from ToJen/patch-1 Andrej 2024-04-21 15:06:08 -07:00
  • 43ffd2af8a add Rust port to notable forks Tomisin Jenrola 2024-04-21 19:23:50 -02:30
  • 5ca1b939e9 small type reshuffles Andrej Karpathy 2024-04-21 19:28:17 +00:00
  • 48f2170e7b pad step so that we don't change horizontal size in early parts of the script Andrej Karpathy 2024-04-21 18:53:08 +00:00
  • 8d4f93ccbd experiment with pretty printing the run config, model config, device config, and data config Andrej Karpathy 2024-04-21 18:47:36 +00:00
  • e55ed0a4e6 change default lr to 3e-4, exp -> expf, and print tok/s Andrej Karpathy 2024-04-21 18:22:45 +00:00
  • e93438efea Merge pull request #201 from ngc92/memory Andrej 2024-04-21 11:02:50 -07:00
  • 734f65f33e Merge pull request #192 from rosslwheeler/const_fixes Andrej 2024-04-21 10:42:15 -07:00
  • 14d96e7c0d Merge branch 'Ricardicus-master' Andrej Karpathy 2024-04-21 17:41:24 +00:00
  • fced6d180f small adjustments to Makefile and gitignore Andrej Karpathy 2024-04-21 17:40:26 +00:00
  • e70bc19af5 Merge branch 'master' of https://github.com/Ricardicus/llm.c into Ricardicus-master Andrej Karpathy 2024-04-21 17:34:05 +00:00
  • fb3bea9e2a more aggressive reuse of no longer needed memory from the forward activations Erik Schultheis 2024-04-21 19:48:21 +03:00
  • 8f6d1caefd merged output and scratch Erik Schultheis 2024-04-21 19:31:46 +03:00
  • 118f3dde12 removed yet more tensors from backward state Erik Schultheis 2024-04-21 19:23:25 +03:00
  • da5e8f7f56 split ActtivationTensors into separate classes for forward and backward to improve readability Erik Schultheis 2024-04-21 19:10:47 +03:00
  • e3cbe7d886 saved another 60 MiB Erik Schultheis 2024-04-21 18:53:25 +03:00
  • 2ed8eddb3e slight optimization of the python script Erik Schultheis 2024-04-21 18:43:45 +03:00
  • 33e81ba55f print peak memory consumption in python Erik Schultheis 2024-04-21 18:25:04 +03:00
  • bfdbf50189 more use of scratch buffer Erik Schultheis 2024-04-21 18:16:12 +03:00
  • 10fc806900 replace vaccum with generic scratch buffer Erik Schultheis 2024-04-21 17:10:29 +03:00
  • 0636bc4ee3 removed the union and moved the logit->probability conversion into the sampling function Erik Schultheis 2024-04-21 16:45:46 +03:00
  • 41c3cc1fab Merge branch 'karpathy:master' into master patricxu 2024-04-21 10:31:59 +08:00
  • 7f3272c38f unionize output buffer Erik Schultheis 2024-04-21 02:03:35 +03:00
  • 9fb9c9185a change baseline kernel to float, for now, for fair comparison too Andrej Karpathy 2024-04-20 22:57:18 +00:00
  • 2f81c6cbda qkv is only needed once, not per-layer Erik Schultheis 2024-04-21 01:50:10 +03:00
  • 8488669d25 speed up the backward bias kernel by 45% and speed up the full running time by 1% Andrej Karpathy 2024-04-20 22:46:19 +00:00
  • 3cd6f83e32 removing some warnings Rickard Hallerbäck 2024-04-20 22:03:35 +02:00
  • 9b722ce5c0 Merge branch 'rosslwheeler-unistd_h_in_dev' Andrej Karpathy 2024-04-20 20:00:39 +00:00
  • 19e337b99e a comment and some breathing room Andrej Karpathy 2024-04-20 20:00:24 +00:00
  • be106f09a4 Merge branch 'master' into master Rickard Hallerbäck 2024-04-20 22:00:02 +02:00
  • 7e42c8b356 Merge branch 'unistd_h_in_dev' of https://github.com/rosslwheeler/llm.c into rosslwheeler-unistd_h_in_dev Andrej Karpathy 2024-04-20 19:59:11 +00:00
  • aa1184d9ca Merge pull request #199 from ngc92/fused-buffer Andrej 2024-04-20 12:54:37 -07:00
  • 0e79ac7d8b Merge branch 'dagelf-master' Andrej Karpathy 2024-04-20 19:45:49 +00:00
  • adac21e97b add a small comment as to what this is Andrej Karpathy 2024-04-20 19:45:34 +00:00
  • 48a9f2ec69 Merge branch 'master' of https://github.com/dagelf/llm.c into dagelf-master Andrej Karpathy 2024-04-20 19:43:17 +00:00
  • 7fa9f7b140 document one way to do experiments / sweeps as just an example Andrej Karpathy 2024-04-20 19:39:12 +00:00
  • f534e4bdfe Fixes -Ofast optimizations breaking model by skipping them for gelu_backward Coenraad 2024-04-20 21:33:00 +02:00
  • 280023bf8e Support for vanilla build without CUDA tools Rickard Hallerbäck 2024-04-20 18:14:53 +02:00
  • 4e3b7ea0f3 add a first, super simple logger Andrej Karpathy 2024-04-20 19:26:25 +00:00
  • acdcfeaab6 introduce a poor mans argparse because i want to start training on tinystories, and messing with some of the options of the runs, now that things are more efficient Andrej Karpathy 2024-04-20 18:37:18 +00:00
  • 5b0d52619f Merge pull request #195 from ngc92/prof-script Andrej 2024-04-20 10:48:10 -07:00
  • af1ee01768 Merge pull request #197 from ngc92/fused-attention-slow Andrej 2024-04-20 10:40:04 -07:00
  • 24dd728953 only need one V-sized buffer that can switch from logits to dlogits in the fused backward pass Erik Schultheis 2024-04-20 01:20:36 +03:00
  • af9ac14b2f don't track preatt and vaccum across layers Erik Schultheis 2024-04-20 20:18:41 +03:00
  • 8ffc3e388b slow attention kernel that demonstrates a direct translation of the CPU code Erik Schultheis 2024-04-20 16:51:30 +03:00
  • d12b923c00 profile and print out some nice stats Erik Schultheis 2024-04-20 13:09:52 +03:00
  • d24562932f Adding unistd.h for Windows in Dev directory Ross Wheeler 2024-04-20 02:37:16 -07:00
  • e564222113 const changes to support OpenMP 5.2 MSVC work-around Ross Wheeler 2024-04-20 02:07:12 -07:00
  • d48b0d2b2b fix indexing in matmul_forward_kernel1 patricxu 2024-04-20 15:38:12 +08:00
  • 2d3a0fbe50 Merge pull request #181 from lancerts/const-fix-adamw Andrej 2024-04-19 22:06:11 -07:00
  • d34556cf5e more profile docs Andrej Karpathy 2024-04-20 01:29:14 +00:00
  • 67fd37ace4 add crappy instructions for profiling basics Andrej Karpathy 2024-04-20 01:06:53 +00:00
  • 7b79fbb230 Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-04-20 00:59:39 +00:00
  • 7df2d4d92b also requirements for quick start Andrej Karpathy 2024-04-20 00:59:32 +00:00
  • 694cb571ec Merge pull request #189 from ngc92/profile-target Andrej 2024-04-19 17:55:35 -07:00
  • 91aab62aad fix kernel 5 of layernorm, bug from previous copy paste with loop bounds Andrej Karpathy 2024-04-20 00:51:06 +00:00
  • 5bdbc45806 only do 5 validation batches by default, we really don't need much precision here for this little script with current defaults Andrej Karpathy 2024-04-20 00:45:13 +00:00
  • 38fd42d3e3 also add a quick start CPU section Andrej Karpathy 2024-04-20 00:41:13 +00:00
  • da4c5c79ca very quick start section add Andrej Karpathy 2024-04-20 00:35:35 +00:00
  • 9c1c1f0cec helper target that gets compiled with lineinfo and runs a single layer forward-backward pass Erik Schultheis 2024-04-20 03:09:42 +03:00
  • 49d41ae296 add one more kernel, allocating a block per row. bad idea if C is too low, as we have it right now Andrej Karpathy 2024-04-19 23:24:51 +00:00
  • cb791c4ef5 new kernel that does a single pass over x on load, using a more clever variance formula. only very slightly faster on my A100 sadly Andrej Karpathy 2024-04-19 22:55:30 +00:00
  • 816254e454 Merge pull request #187 from azret/csharp-port-ref Andrej 2024-04-19 14:17:59 -07:00
  • c6bd590b47 typo Azret Botash 2024-04-19 14:17:13 -07:00
  • 6cd1354b3e C# port ref Azret Botash 2024-04-19 13:59:36 -07:00
  • 792812985b Merge pull request #185 from dorjeduck/update-readme-notable-forks Andrej 2024-04-19 13:22:32 -07:00
  • b173e0237c Merge branch 'ngc92-and-one-more' Andrej Karpathy 2024-04-19 20:21:50 +00:00
  • 1f08134fb9 Added link to Mojo port in Notable Forks section Martin Dudek 2024-04-20 01:44:50 +05:30
  • 44b5bb028c one more version of attention backward for another few percent perf gains Erik Schultheis 2024-04-19 22:33:05 +03:00
  • a360f104ae some low-level optimizations Erik Schultheis 2024-04-19 21:46:14 +03:00
  • 7005e09450 added missing checks Erik Schultheis 2024-04-19 22:37:35 +03:00
  • bba794f156 Merge pull request #180 from ngc92/yet-more-attention Andrej 2024-04-19 11:05:55 -07:00
  • 3cb993b5f6 fix the constness and the warning C4305: 'initializing': truncation from 'double' to 'float' lancer 2024-04-19 11:04:58 -07:00
  • b2e6b6867f separating the loops leads to yet more improvements Erik Schultheis 2024-04-19 20:47:19 +03:00
  • 810b0dd90f adjust header Andrej Karpathy 2024-04-19 17:19:26 +00:00
  • f7fb3e67d5 update readme Andrej Karpathy 2024-04-19 17:16:48 +00:00
  • 1dbf241fbe cleanups of the main script Andrej Karpathy 2024-04-19 17:02:46 +00:00
  • 09a117a664 add a compile option for torch compile suggested by Horace Andrej Karpathy 2024-04-19 16:32:24 +00:00
  • b556ad971c Merge pull request #179 from ngc92/even-better-attention Andrej 2024-04-19 09:16:22 -07:00
  • 1d3b6738a7 Merge pull request #169 from lancerts/fix-crossentropy-bk Andrej 2024-04-19 09:15:23 -07:00
  • c8f3981d6b Merge pull request #175 from lancerts/fix-double-f Andrej 2024-04-19 09:12:58 -07:00
  • 50e105a250 further cleanup and ability to handle arbitrary sequence lengths Erik Schultheis 2024-04-19 19:03:40 +03:00
  • 4294820a8a don't do stupid redundant work; the inner loop is just a block-level sum!!!! Erik Schultheis 2024-04-19 18:31:26 +03:00
  • 02aa8e47dc further simplification of the loop Erik Schultheis 2024-04-19 17:36:18 +03:00
  • 16123c3af8 towards an even better backward attention kernel Erik Schultheis 2024-04-19 10:56:34 +03:00
  • 0c92ea9b90 fix the 0.0f initialization for double lancer 2024-04-18 19:35:34 -07:00
  • e991fe1efa fix the 0.0f initialization for double lancer 2024-04-18 19:33:30 -07:00
  • 0dbfa32fd2 sorry this is a double so i shouldn't use 0.0f. i have to dedicate a whole commit to this because of my OCD Andrej Karpathy 2024-04-19 02:23:21 +00:00
  • 315af5f5dd micro-optimization on the order, maximizing cache re-use ty @YichengDWu for original PR, which I adapted a bit Andrej Karpathy 2024-04-19 02:21:28 +00:00
  • c393d828dc size_t instead of int. everywhere. this will come to bite us at one point... Andrej Karpathy 2024-04-19 02:11:30 +00:00
  • 213ed60afd fused classifier! we don't materialize the probs. we go from 200ms to 190ms Andrej Karpathy 2024-04-19 01:33:35 +00:00