mirrors / karpathy--llm.c

提交

提交图

  • 251421102d cleanup; these mallocs are no longer needed Erik Schultheis 2024-07-18 20:00:51 +02:00
  • bfce9bdbcf bugfixes Erik Schultheis 2024-07-18 19:35:37 +02:00
  • 0bcb3e5840 allocate_weights utility function reverted some earlier changes + more error checking Erik Schultheis 2024-07-18 19:29:49 +02:00
  • cde79ffb49 allocate_state utility function Erik Schultheis 2024-07-18 19:19:06 +02:00
  • 5c89416adf also allocate gradient eagerly moved helper buffer allocation to forward, so backward now no longer does any memory allocation Erik Schultheis 2024-07-18 18:54:51 +02:00
  • 72698a5ba3 update gpt3 125M script to be better. should exactly reproduce gpt-3 125M, except we don't use banded attention and we use fineweb Andrej Karpathy 2024-07-18 16:43:14 +00:00
  • b20def92d4 consolidate model init Erik Schultheis 2024-07-18 17:23:06 +02:00
  • cbe04c3f92 moved dtype utilities to cuda_utils Erik Schultheis 2024-07-18 17:13:38 +02:00
  • 85d17f4aa2 Merge pull request #668 from abhilash1910/tpc Andrej 2024-07-15 18:22:15 -07:00
  • 835bb2e6d7 delete segmentation fault when B < 4 atm Andrej Karpathy 2024-07-16 01:20:16 +00:00
  • 88514df231 Merge pull request #689 from karpathy/refactor/code-to-zerocuh Andrej 2024-07-15 12:51:01 -07:00
  • 5afc57064c document the imports from zero.cuh to be explicit refactor/code-to-zerocuh Andrej Karpathy 2024-07-15 19:09:52 +00:00
  • 882a214ad4 Merge branch 'move-to-zero' of https://github.com/ngc92/llm.c into ngc92-move-to-zero Andrej Karpathy 2024-07-15 18:59:22 +00:00
  • a6dc64533a Merge pull request #688 from karpathy/feature/gpt3v1 Andrej 2024-07-15 11:53:05 -07:00
  • 220125f39c ok let's allow T < max_seq_len because our tests use it. make warning very prominent feature/gpt3v1 Andrej Karpathy 2024-07-15 18:21:34 +00:00
  • faf8d159a3 use int instead of bool Andrej Karpathy 2024-07-15 18:01:01 +00:00
  • 528b1b0e31 fix fat fingers Andrej Karpathy 2024-07-15 17:56:41 +00:00
  • 039904ee9f revert legacy behavior in .sh scripts for -e option Andrej Karpathy 2024-07-15 17:55:26 +00:00
  • 64db3ded85 refactor gpt2 gpt3 descriptor to keep legacy behavior of -e, be strict everywhere, remove interpolation for now Andrej Karpathy 2024-07-15 17:51:27 +00:00
  • 4e629e22a1 Merge branch 'gpt3-series' of https://github.com/ngc92/llm.c into ngc92-gpt3-series Andrej Karpathy 2024-07-15 16:46:30 +00:00
  • f45c2191b6 Merge pull request #684 from rosslwheeler/feature/loss_checker_ci Andrej 2024-07-15 09:30:59 -07:00
  • e71c1565c9 Merge pull request #686 from indianspeedster/cuda Andrej 2024-07-15 09:28:02 -07:00
  • 6bb562bd88 added cudaCheck where missing for proper error checking. indianspeedster 2024-07-13 13:20:39 -04:00
  • a6f6bc74a8 Merge pull request #682 from AnswerDotAI/master Andrej 2024-07-13 10:12:59 -07:00
  • 0e2ff48440 Merge pull request #685 from planetis-m/patch-1 Andrej 2024-07-13 10:08:26 -07:00
  • a86a69c135 Add nim port Antonis Geralis 2024-07-13 18:57:51 +03:00
  • 56621bba0e Adding CI check for exceeding loss tolerance Ross Wheeler 2024-07-13 01:10:46 -07:00
  • ddd8aa0f34 add gpu.cpp link to related projects in readme austinvhuang 2024-07-12 14:42:24 -04:00
  • 37b6e328c1 enable model interpolation Erik Schultheis 2024-07-12 15:07:05 +02:00
  • b5d5ed7f68 updated example scripts Erik Schultheis 2024-07-12 14:46:09 +02:00
  • d19a6267a8 add gpt3 model series Erik Schultheis 2024-07-12 14:43:22 +02:00
  • 39a41fa701 move cpu float reduction to zero Erik Schultheis 2024-07-12 11:12:58 +02:00
  • 4d1ec6832a rename parameters to avoid name clashes with global variable Erik Schultheis 2024-07-11 19:04:02 +02:00
  • 09b78bc902 move set_zero_configs into zero.cuh Erik Schultheis 2024-07-09 17:12:37 +02:00
  • bdb0fb5599 Merge pull request #671 from ademeure/less_slow_inference Andrej 2024-07-10 17:47:04 -07:00
  • db2454f88f Merge pull request #620 from ngc92/small-fixes Andrej 2024-07-09 12:53:34 -07:00
  • 2b76155976 small fixes based on clang-tidy Erik Schultheis 2024-06-19 20:35:04 +03:00
  • 2fd6fee971 fix the corner case where T < 256 ademeure 2024-07-07 12:35:29 +00:00
  • efa6767a8a Make inference sanity checks much faster by changing (B,T) to (1,256) ademeure 2024-07-07 12:23:26 +00:00
  • a2737d366a add gaudi2 tpc kernel link Abhilash Majumder 2024-07-04 00:34:13 +05:30
  • f0abd09ffa Merge pull request #641 from gordicaleksa/add_close_check_socket Andrej 2024-07-02 14:26:03 -07:00
  • 6b62f3b9ae Fix linker issue Aleksa Gordic 2024-07-02 21:34:21 +02:00
  • d27331cced Move socket header to utils.h Aleksa Gordic 2024-06-26 10:43:39 +00:00
  • 5645bd18b1 CI fix - add sockets header Aleksa Gordic 2024-06-26 07:56:32 +00:00
  • 1ae3243185 Add check versions of socket close Aleksa Gordic 2024-06-25 14:20:27 +00:00
  • c6efd90a86 upload good 1558M sh script Andrej Karpathy 2024-07-02 16:49:09 +00:00
  • 39270cc69a Merge pull request #665 from ngc92/zero-grad Andrej 2024-07-01 16:08:07 -07:00
  • 486f98e3cc zero-grad is async and part of backward call Erik Schultheis 2024-07-02 00:47:41 +02:00
  • 79223b950b Merge branch 'ademeure-optimize_attproj_fcproj' Andrej Karpathy 2024-07-01 22:07:24 +00:00
  • cee396ab5a resolve merge conflict Andrej Karpathy 2024-07-01 22:07:14 +00:00
  • 48fee049cb add edu fineweb sh download script, change paths to my HF repo, add hellaswag to starter pack Andrej Karpathy 2024-07-01 21:41:15 +00:00
  • a876282eb8 Merge pull request #653 from ademeure/cublaslt_refactor Andrej 2024-07-01 11:53:02 -07:00
  • 00247dca9b Disable GELU fusion by deffault & fix potential argv[] out of bounds access ademeure 2024-07-01 18:40:21 +00:00
  • 2e6a397d5f Merge pull request #663 from karpathy/fix/reshuffle_zero_grad Andrej 2024-07-01 10:39:23 -07:00
  • 7b92930021 move things around a bit, a few comments, and also update python script to correspond to C more fix/reshuffle_zero_grad Andrej Karpathy 2024-07-01 17:20:46 +00:00
  • 8253823432 Merge branch 'fix_periodic_spikes' of https://github.com/gordicaleksa/llm.c into gordicaleksa-fix_periodic_spikes-2 Andrej Karpathy 2024-07-01 17:14:22 +00:00
  • 942fed59fe Merge pull request #662 from karpathy/overfit-batch-actually Andrej 2024-07-01 10:12:20 -07:00
  • 7fd7717816 fix the overfit single batch behavior to actually overfit batch, not microbatch, we do this more cleanly by simply resetting the dataloader every step overfit-batch-actually Andrej Karpathy 2024-07-01 17:00:40 +00:00
  • 74e5ba7260 Add micro step guard Aleksa Gordic 2024-07-01 12:11:02 +02:00
  • 78e7627859 Remove local args for testing Aleksa Gordic 2024-07-01 11:31:55 +02:00
  • de8454d2f4 Make the grad cleaning more robust Aleksa Gordic 2024-07-01 11:30:36 +02:00
  • d2f06aed01 Fix periodic spikes - clean loss after val Aleksa Gordic 2024-07-01 11:24:16 +02:00
  • 61625d168c only rank 0 creates log dir Andrej Karpathy 2024-07-01 06:31:37 +00:00
  • 20e450ca70 merge fix ademeure 2024-06-30 19:48:53 +00:00
  • 6e29dd000d fix default for test_gpt2cu ademeure 2024-06-27 14:11:13 +00:00
  • 0f11a4ec49 Make GELU fusion default 2 for H100+ and 0 on older GPUs for now ademeure 2024-06-27 12:32:41 +00:00
  • fa5a196be0 Refactor matmul to only use cuBLASLt and optionally use fused GELU ademeure 2024-06-27 09:21:32 +00:00
  • ea36d04e39 small cosmetic fixes to fineweb data file Andrej Karpathy 2024-06-30 15:52:51 +00:00
  • ba4e2c05ad Remove per-layer attproj and fcproj activation tensors (reuse output as scratch buffer instead). Significant memory saving. ademeure 2024-06-30 15:28:05 +00:00
  • 2daaf6ab83 Merge pull request #629 from rosslwheeler/feature/tests Andrej 2024-06-29 15:17:31 -07:00
  • e6e365a1a8 Merge branch 'eliebak-add_edu_fineweb_support' Andrej Karpathy 2024-06-28 19:35:58 +00:00
  • ff7a90bd8d Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_support Andrej Karpathy 2024-06-28 19:35:44 +00:00
  • 2cf7c5b7d3 Merge branch 'ngc92-mixed-dtypes-2' Andrej Karpathy 2024-06-27 17:47:22 +00:00
  • 012915c06c handle unknown datatype to squash compiler warning Andrej Karpathy 2024-06-27 17:46:46 +00:00
  • 06cd83eacc Merge pull request #652 from ademeure/cudnn_determinism Andrej 2024-06-27 10:22:04 -07:00
  • 29a067f916 Make cuDNN deterministic for Flash Attention backward (requires frontend >=1.5) ademeure 2024-06-27 13:33:02 +00:00
  • d6d08be3c0 Addressed review comments Ross Wheeler 2024-06-27 00:58:19 -07:00
  • eb0b74e559 Add H100 and A100 ptx/sass files Ross Wheeler 2024-06-27 00:30:38 -07:00
  • 267efc9a0e Merge branch 'karpathy:master' into feature/tests rosslwheeler 2024-06-26 22:38:21 -07:00
  • 98875dd022 macro-based alternative ngc92 2024-06-27 02:14:45 +02:00
  • d38c614eb9 add some comments ngc92 2024-06-26 23:58:31 +03:00
  • 7beaa1ed96 keep layernorm stats in fp32 Erik Schultheis 2024-06-25 21:40:09 +03:00
  • f0c3e35de8 switch losses to fp32 Erik Schultheis 2024-06-25 21:33:00 +03:00
  • ca47165d29 allocation function for mixed dtype activation Erik Schultheis 2024-06-25 21:26:11 +03:00
  • 68535cb6ce Merge pull request #649 from karpathy/feature/openmpi_flexible_path Andrej 2024-06-26 09:11:22 -07:00
  • 7c283df9c4 use ?= for OPENMPI_DIR so it can be overriden feature/openmpi_flexible_path Andrej Karpathy 2024-06-26 16:06:54 +00:00
  • c1ac121c77 Merge pull request #646 from Ricardicus/makefile-fix Andrej 2024-06-26 08:42:59 -07:00
  • a08e32e474 Merge pull request #648 from krrishnarraj/mention_fork Andrej 2024-06-26 08:41:15 -07:00
  • 2a8f82f14f tune openmpi make a bit more convenient Andrej Karpathy 2024-06-26 15:39:10 +00:00
  • 0aa0eb09a6 [readme] mention opencl fork Krishnaraj Bhat 2024-06-26 20:32:21 +05:30
  • 66cd8c3f1f Makefile: respect the NO_MULTI_GPU Rickard Hallerbäck 2024-06-26 14:14:03 +02:00
  • 44fde98344 Merge pull request #637 from karpathy/feature/outliers Andrej 2024-06-25 21:16:29 -07:00
  • e12d090483 resolve merge conflicts feature/outliers Andrej Karpathy 2024-06-26 04:05:46 +00:00
  • ac018c3302 Merge branch 'ngc92-more-streams' Andrej Karpathy 2024-06-25 23:41:55 +00:00
  • 9c7f1f9701 minor comments Andrej Karpathy 2024-06-25 23:41:40 +00:00
  • 4f84e3daa7 Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams Andrej Karpathy 2024-06-25 23:25:57 +00:00
  • 7171f78f04 fix check call Erik Schultheis 2024-06-26 02:21:44 +03:00
  • c0e84eb859 Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams Andrej Karpathy 2024-06-25 23:16:50 +00:00
  • 4aae1e529b fix up profile Erik Schultheis 2024-06-26 02:11:55 +03:00
  • 6c24f13b55 check token validity Erik Schultheis 2024-06-26 02:01:24 +03:00