mirrors / karpathy--llm.c

提交

提交图

  • 12c7410bab docs: make Chinese README the default main wehub-resource-sync 2026-07-13 10:30:59 +00:00
  • 3fe054ae00 docs: preserve upstream English README wehub-resource-sync 2026-07-13 10:28:40 +00:00
  • 1612ff5875 chore: import upstream snapshot with attribution wehub-resource-sync 2026-07-13 12:37:59 +08:00
  • da88cb11b0 Merge pull request #803 from ngc92/ngc92/llama3-tied-weights llama3 Andrej 2025-06-26 10:03:39 -07:00
  • 5c17e4e256 Merge pull request #821 from ngc92/out-of-bounds-bugfix Andrej 2025-06-26 10:01:10 -07:00
  • 9caeceb7ae command-line overwrite to forcibly untie embeddings for llama3.2 models Erik Schultheis 2025-04-14 13:22:39 +02:00
  • 9688eef519 enable tied embeddings Erik Schultheis 2025-04-14 13:07:37 +02:00
  • ffcfe99955 fix out-of-bounds access in rmsnorm kernel Erik Schultheis 2025-06-26 17:52:20 +02:00
  • 9c60616276 fix out-of-bounds access in rmsnorm kernel Erik Schultheis 2025-06-26 17:52:20 +02:00
  • d36f0e6756 Merge pull request #811 from ngc92/llama-fixes Andrej 2025-05-10 16:27:57 -07:00
  • f1e2ace651 Merge pull request #801 from ngc92/ngc92/fix-test master Andrej 2025-05-10 16:24:10 -07:00
  • 76a7cce3db replace offload with smaller model Erik Schultheis 2025-05-04 23:29:10 +02:00
  • 35e1ad6fcc enable storing the expected loss values in the state file, so we can run testing with different model configurations Erik Schultheis 2025-05-04 23:28:44 +02:00
  • f38eadce35 allow reducing number of transformer blocks to make smaller models that can be tested on commodity GPUs Erik Schultheis 2025-05-04 22:49:41 +02:00
  • a860922827 set stream for attention softmax Erik Schultheis 2025-05-04 22:10:38 +02:00
  • 082d9fa78e added missing stream argument for repkv_backward Erik Schultheis 2025-05-02 23:45:08 +02:00
  • d4347a7115 ensure 32-bit master params in python training Erik Schultheis 2025-05-02 22:16:05 +02:00
  • 7b7d39c007 include grad norm in logging Erik Schultheis 2025-05-02 22:11:59 +02:00
  • 1c02d54716 cudnn does not support fp32 -> remove this pointless test Erik Schultheis 2025-05-01 14:59:57 +02:00
  • 49cef1d59a Merge pull request #802 from ngc92/ngc92/llama3-dev Andrej 2025-05-01 04:21:07 -07:00
  • 9c52a9557e fix: CPUOffloadOptimizer + gradient clipping is broken; we use an inefficient workaround to make it correct Erik Schultheis 2025-04-14 16:49:35 +02:00
  • 5b928298d9 !! DROP THIS COMMIT !! hard-code a hf token to make the tests run Erik Schultheis 2025-04-13 20:31:43 +02:00
  • f8a43cea84 fix: writing checkpoint Erik Schultheis 2025-04-14 10:52:21 +02:00
  • 24d91298cf fix: ensure freqs_cis are not broken when calling model.to(dtype) (was discarding imaginary part) Erik Schultheis 2025-04-14 10:35:04 +02:00
  • 2c3fecced2 fix: gradient checking Erik Schultheis 2025-04-13 18:36:50 +02:00
  • 6866623259 fix: match pytorch learning rate in test file Erik Schultheis 2025-04-13 18:34:19 +02:00
  • 4983c462a7 fix: fully ignore biases Erik Schultheis 2025-04-13 18:32:06 +02:00
  • a94471c627 use optimizer offloading when running in CI Erik Schultheis 2025-04-14 11:15:26 +02:00
  • 090341e8dc enable llama3 CI Erik Schultheis 2025-04-13 17:55:15 +02:00
  • 4d7980c26f renaming gpt2 -> llama3 Erik Schultheis 2025-04-13 17:25:57 +02:00
  • 3745dac6de define llama3.2 1B and 3B for export from python (will untie embeddings and lm-head for now) Erik Schultheis 2025-04-13 17:23:57 +02:00
  • 42aa89184f adjusted error thresholds Erik Schultheis 2025-04-13 10:18:17 +02:00
  • 166f5e6c60 reorder checks to match actual tensor order Erik Schultheis 2025-04-13 10:01:18 +02:00
  • 7ecd8906af Merge pull request #766 from msaroufim/patch-1 Andrej 2024-10-02 10:25:39 +09:00
  • 2c5ced6a77 fix bug due to bf16 adamw mv Andrej Karpathy 2024-10-01 21:41:17 +00:00
  • d808d78030 Merge pull request #772 from ademeure/llama3_arun_new Andrej 2024-10-02 04:17:46 +09:00
  • 2602b46bb3 BF16 opt state (m/v) with stochastic rounding, seems to work really well (OPTIMIZER_LOW_PRECISION=1) ademeure 2024-10-01 18:42:38 +00:00
  • c746e06f49 take out debugging stuff. we can now run training loop for both models. they don't match yet Andrej Karpathy 2024-10-01 17:19:39 +00:00
  • 9099a0ae9c ok the full backward now shows max abs diff of 3e-3, except for the encoder backward (that's coming next). i think 3e-3 seems ok just inspecting the differences manually. probably this is correct. encoder backward next Andrej Karpathy 2024-10-01 17:03:29 +00:00
  • e6481b679c fix bug with qkvr sizing, has to be 3*C. Credit to @ademeure for finding this bug and bringing light to darkness and order to chaos. A true warrior in the fight against entropy. Andrej Karpathy 2024-10-01 16:36:35 +00:00
  • 7d945e994c reshuffle repkv a bit, i wrote it from scratch. the kernel is still correct. repkv backward looks correct. rope backward is trivial so i don't see how it's not correct, and i also checked it. basically i'm really confused right now Andrej Karpathy 2024-09-27 19:25:09 +00:00
  • 8d49062293 add backward kernel to dev/cuda for rope, to ensure correctness. but i mean, it's trivial. this can't possibly be the issue. it must be the repkv Andrej Karpathy 2024-09-27 18:07:10 +00:00
  • 075e430d23 just pushing what i have. it's epsilon away from working sigh. basically at this point of where prints happen, gradients match. but once we backward attention, rope and repkv, gradients don't match. attention hasn't changed so that can't be wrong (?), so it's either repkv or rope. i have to go slower and double check the backward pass of both of these in detail. also had to introduce one more additional buffer for backward Andrej Karpathy 2024-09-27 03:37:53 +00:00
  • 28e4a7f83e small fixes, but still not too happy with this kernel, it wastes thread and more efficient implementation kernel2 is desireable and desired Andrej Karpathy 2024-09-27 01:13:32 +00:00
  • c8b348ee39 Merge pull request #764 from insop/insop/llama3 Andrej 2024-09-27 10:03:39 +09:00
  • 1b54612c46 i can backward through MLP block. Attention block is next Andrej Karpathy 2024-09-27 00:21:43 +00:00
  • 01c2895e17 Update RoPE naming Insop Song 2024-09-26 16:34:50 -07:00
  • cbf53e3f61 Merge remote-tracking branch 'upstream/llama3' into insop/llama3 Insop Song 2024-09-26 16:33:01 -07:00
  • 2c4b3cc8bb integrate our rmsnorm backward and move the other rmsnorm functions into rmsnorm.cuh that is a new file Andrej Karpathy 2024-09-26 22:27:58 +00:00
  • 102067fbf5 oops i think i accidentally forgot to include swiglu.cuh Andrej Karpathy 2024-09-26 22:08:19 +00:00
  • a2b66f153a Merge remote-tracking branch 'upstream/llama3' into insop/llama3 Insop Song 2024-09-26 11:42:31 -07:00
  • 31be5e790d add rmsnorm backward in dev/cuda, it seems to work surprisingly, and is probably ready to be integrated into llmc. we are still using 2X too much shared memory because I didn't want to change way too many things at the same time. I copy pasted our kernel10 of layernorm backward and made tweaks to it removing the bias and mean cool Andrej Karpathy 2024-09-26 03:17:58 +00:00
  • d1f2f64541 Updated repkv_backward cuda kernel Insop Song 2024-09-25 17:40:38 -07:00
  • 3e5134d94f Merge branch 'insop/llama3_wip' into insop/llama3 Insop Song 2024-09-25 15:53:17 -07:00
  • 8874c2c3f6 Merge remote-tracking branch 'upstream/llama3' into insop/llama3 Insop Song 2024-09-25 15:50:51 -07:00
  • 0731b39a73 forward pass matchesgit add train_llama3.cu train_llama3.py ! losses are the same. now comes the backward pass Andrej Karpathy 2024-09-25 22:31:31 +00:00
  • 1826752ae1 add swigul yaygit add -u! Andrej Karpathy 2024-09-25 22:16:45 +00:00
  • bb3c92da55 integrate the fused rmsnorm forward Andrej Karpathy 2024-09-25 19:12:18 +00:00
  • 6538df64cd Merge pull request #769 from gordicaleksa/fused_rmsnorm Andrej 2024-09-26 04:03:38 +09:00
  • 52c7254267 add the finished RoPE forward pass Andrej Karpathy 2024-09-25 19:00:27 +00:00
  • 2ebf8f6b8a Add rmsnorm fused kernel Aleksa Gordic 2024-09-25 10:37:53 -07:00
  • 8336d2a055 Merge remote-tracking branch 'upstream/llama3' into insop/llama3 Insop Song 2024-09-24 18:31:03 -07:00
  • 026e4ed323 add RoPE PyTorch and C reference code Andrej Karpathy 2024-09-24 23:52:16 +00:00
  • 315b8d1f62 Update README.md Mark Saroufim 2024-09-24 13:08:11 -07:00
  • 76b40e43ba integrate the repkv kernel with minor changes. use the bt4c buffer for the replication. rope is next Andrej Karpathy 2024-09-24 17:30:51 +00:00
  • de9c8170e5 wip Insop Song 2024-09-23 20:52:50 -07:00
  • 5b2e3180fb cuda mode -> gpu mode Mark Saroufim 2024-09-23 17:09:45 -07:00
  • fcc3466b2f Add minor change Insop Song 2024-09-22 09:05:56 -07:00
  • 42d09e8732 minor clean up Insop Song 2024-09-22 00:26:29 -07:00
  • ad46043aaf WIP updating cuda kernel Insop Song 2024-09-22 00:22:26 -07:00
  • 6c68657c6b Put cuda kernel caller placeholder Insop Song 2024-09-21 21:30:45 -07:00
  • 080e57fd46 CPU version tested - [ ] WIP cuda version Insop Song 2024-09-21 21:21:13 -07:00
  • 45401b42eb DRAFT: Adding backward kernel for repkv Insop Song 2024-09-21 20:40:57 -07:00
  • 56f956cc4a add repkv kernel to replicate K,V heads after the QKV projection Andrej Karpathy 2024-09-21 01:53:08 +00:00
  • 685617f164 make fp32 path in .py code work correctly Andrej Karpathy 2024-09-17 21:31:18 +00:00
  • 508c474bf9 move debugging into fp32, so python has to write the fp32 version, and then we are focusing on the non-cudnn path at first. we're currently right after the first rmsnorm. the encoding right before this matched EXACTLY. but right now, after the first rmsnorm there is already an error of 1e-3 or so, which is highly suspicious so we are looking into it. Andrej Karpathy 2024-09-17 21:19:46 +00:00
  • 234de31fdf introduce rmsnorm, unfused, forward Andrej Karpathy 2024-09-16 21:43:15 +00:00
  • 72e6f1ab0b add new Encoder that does not use positional embeddings, like in llama 3. The activations match after encoding. onwards Andrej Karpathy 2024-09-16 21:03:01 +00:00
  • 77e1d7afda add support for dataloader to serve uint32_t tokens, as necessary in Llama 3 Andrej Karpathy 2024-09-16 19:44:33 +00:00
  • 45026f6ead make llama3cu phony Andrej Karpathy 2024-09-16 19:43:44 +00:00
  • 88663086fb adapt the sizes of all the parameter tensors and load them from file. so now we are loading all the Llama 3 weights. I verified that the sizes of all the tensors agree with python, and the total number of parameters Andrej Karpathy 2024-09-16 17:45:28 +00:00
  • b883560d26 change the export code of Llama 3 to be very GPT-2 friendly, using a combination of 3 hacks. this will make it so that we have to change very little code on the C side Andrej Karpathy 2024-09-13 22:47:30 +00:00
  • 01bc4c685a first set of changes to match up the .py and the .cu version. default hyperparameters, introduce int+float section of header, read the header and EXIT for now Andrej Karpathy 2024-09-13 20:44:10 +00:00
  • 09b47a747d llama3 starting point is at gpt-2 exact copy paste for both train/test files Andrej Karpathy 2024-09-13 19:29:29 +00:00
  • bd8c6045be change default params: use tinyshakespeare and decrease LR Andrej Karpathy 2024-09-13 19:04:08 +00:00
  • bd457aa19b Merge pull request #724 from GaoYusong/master Andrej 2024-08-27 04:40:06 +09:00
  • a2bdae248a Merge pull request #733 from zhangpiu/feature/llm.cpp Andrej 2024-08-27 04:37:33 +09:00
  • ebc28b9563 Merge pull request #735 from gordicaleksa/minor_refactor3 Andrej 2024-08-27 04:24:22 +09:00
  • 2c9213731e Merge pull request #744 from dengl11/pr Andrej 2024-08-16 12:49:55 -07:00
  • 0ddedf940d Merge pull request #745 from karpathy/feature/managed2 Andrej 2024-08-16 12:11:11 -07:00
  • 18298f3a40 i misspelled reduced feature/managed2 Andrej Karpathy 2024-08-16 18:16:46 +00:00
  • 8c586f9158 reduce across GPUs nicer Andrej Karpathy 2024-08-16 18:08:34 +00:00
  • e6856bc566 fallback to memory allocation of m,v,master_weights on host automatically in case of OOM. will run slower but won't OOM Andrej Karpathy 2024-08-16 17:23:42 +00:00
  • 2882ec6b9e fix makefile for multigpu setups Andrej Karpathy 2024-08-16 17:22:30 +00:00
  • f72c1f2c64 hint to host Erik Schultheis 2024-07-25 01:16:28 +02:00
  • c8457572fa just try to allocate on device; fallback if that fails Erik Schultheis 2024-07-24 16:55:09 +02:00
  • 0d52d2a3d7 fall back to cudaMallocManaged for optimizer states if we're out of memory Erik Schultheis 2024-07-24 15:46:30 +02:00
  • ff356e4bc2 fix a typo Li Deng 2024-08-14 09:22:44 -07:00
  • 4c84bc743c Merge pull request #740 from karpathy/gordicaleksa-fix_dataloader2 Andrej 2024-08-13 11:18:57 -07:00
  • 755458d0e1 fix tokenizer omg gordicaleksa-fix_dataloader2 Andrej Karpathy 2024-08-13 17:24:15 +00:00