mirrors / karpathy--llm.c

提交

提交图

  • 16635d41a2 attempt to fix PR Andrej Karpathy 2024-08-13 02:13:14 +00:00
  • 9740a652cd Merge branch 'fix_dataloader2' of https://github.com/gordicaleksa/llm.c into gordicaleksa-fix_dataloader2 Andrej Karpathy 2024-08-12 23:50:10 +00:00
  • 1787210306 Merge pull request #738 from ademeure/faster_compile Andrej 2024-08-12 16:45:15 -07:00
  • 68624cc6ac Refactor tokenize, delegate instead Aleksa Gordic 2024-08-12 20:26:21 +02:00
  • e8739c89a0 Refactor magic/version header dict Aleksa Gordic 2024-08-12 20:17:53 +02:00
  • fc8834409b Improve compile time via nvidia-smi instead of __nvcc_device_query (& add "--threads=0"). ademeure 2024-08-12 16:51:39 +00:00
  • 9b1c14bd93 Improve compile time via nvidia-smi instead of __nvcc_device_query (& add "--threads=0"). ademeure 2024-08-12 16:51:39 +00:00
  • eb1b359dae Replace fire with argparse Aleksa Gordic 2024-08-10 22:45:46 +02:00
  • 311c149a2a Add fire to reqs Aleksa Gordic 2024-08-10 22:39:07 +02:00
  • 50765638c0 Add fineweb llama 3 support Aleksa Gordic 2024-08-10 22:35:46 +02:00
  • f8c1ff590f Use fire module, update readme Aleksa Gordic 2024-08-10 22:29:19 +02:00
  • 15b76bf0fb Add tiny shakespeare Aleksa Gordic 2024-08-10 22:23:33 +02:00
  • b018b0c85c Change default dataset path to tiny stories Aleksa Gordic 2024-08-10 21:50:39 +02:00
  • 71419cbd61 Update header description Aleksa Gordic 2024-08-10 21:47:09 +02:00
  • d3151c4ace Add llama 3 tokenization and data loader Aleksa Gordic 2024-08-10 21:25:57 +02:00
  • b0bc864320 Refactor flash logic Aleksa Gordic 2024-08-10 18:51:56 +02:00
  • a0c666bfbb Remove logprobs - like andrej in nano llama 3 Aleksa Gordic 2024-08-10 18:24:51 +02:00
  • 92cc4ebf36 Add msg to asserts Aleksa Gordic 2024-08-10 18:04:59 +02:00
  • a1278731a3 Minor refactor Aleksa Gordic 2024-08-10 17:43:50 +02:00
  • 25a302fde6 Update README.md Biao Zhang 2024-08-10 17:53:12 +08:00
  • ed4c50dd6e Merge branch 'master' into feature/llm.cpp Biao Zhang 2024-08-10 17:51:49 +08:00
  • ee0a92f915 Add llm.cpp fork zhangpiu 2024-08-10 17:45:21 +08:00
  • 6e6a528111 Merge pull request #725 from gordicaleksa/llama Andrej 2024-08-08 14:08:20 -07:00
  • d773c88e49 Remove unnecessary comment Aleksa Gordic 2024-08-08 21:06:01 +02:00
  • c5c87fc4e7 Add comments, fix stop tokens Aleksa Gordic 2024-08-08 20:54:17 +02:00
  • 8b672ffcb9 Add data loader not implemented exception Aleksa Gordic 2024-08-08 18:55:17 +02:00
  • f1c91f8ae3 Address Andrej's PR comments Aleksa Gordic 2024-08-08 18:45:04 +02:00
  • 89addd3e3e Remove pycache Aleksa Gordic 2024-08-08 18:20:51 +02:00
  • ac01536b99 Remove the whole llmc_py Aleksa Gordic 2024-08-08 18:20:18 +02:00
  • dfd459bfc4 Remove prompts.json Aleksa Gordic 2024-08-08 18:19:40 +02:00
  • 624ed3ce30 Rename the new file to train llama3 Aleksa Gordic 2024-08-08 18:15:37 +02:00
  • b7c98c9365 Add llama config error check Aleksa Gordic 2024-08-08 18:12:35 +02:00
  • b25e325c69 Add explicit external mask Aleksa Gordic 2024-08-08 17:55:01 +02:00
  • d4ef9c5afd Remove llmc_py, single file Aleksa Gordic 2024-08-08 17:09:41 +02:00
  • 0aa9175569 still wip just putting things up for comment feature/finetune_llama31py Andrej Karpathy 2024-08-04 20:26:41 +00:00
  • 72dcfeb404 Remove float(-inf) Aleksa Gordic 2024-08-04 22:20:46 +02:00
  • 8e55d16863 Support HF & Meta models Aleksa Gordic 2024-08-04 16:27:12 +02:00
  • 0749a4af42 Minor refactor - remove wpe pos array from fwd Aleksa Gordic 2024-08-04 09:28:17 +00:00
  • 4b386a2a89 Test training loop works Aleksa Gordic 2024-08-04 09:13:29 +00:00
  • cd902735b2 Fix saving model / state logic Aleksa Gordic 2024-08-04 08:54:21 +00:00
  • 7768a36f37 Add prompts file Aleksa Gordic 2024-08-04 10:07:48 +02:00
  • 879cc5f46c Remove init weights as it's gpt-2 specific Aleksa Gordic 2024-08-04 08:00:44 +00:00
  • bad7857dff Done - need to test train loop and saving model Aleksa Gordic 2024-08-03 22:37:54 +02:00
  • b1acb5988f Got to llama 3 inference (end) Aleksa Gordic 2024-08-03 22:22:38 +02:00
  • d855c9695a Refactoring - got to main Aleksa Gordic 2024-08-03 22:16:13 +02:00
  • c1d2b7fec1 Removed from pretrained for GPT-2 Aleksa Gordic 2024-08-03 22:02:02 +02:00
  • 8a48df7b16 MLP GPT2->LLaMA3 Aleksa Gordic 2024-08-03 21:53:26 +02:00
  • ccdbdfd4ce Going towards pure llama 3 file - fixed attn Aleksa Gordic 2024-08-03 21:49:45 +02:00
  • 8919b66c37 Fix generate for gpt-2 Aleksa Gordic 2024-08-03 14:14:05 +02:00
  • 180215fd60 Unify generate and generate_llama Aleksa Gordic 2024-08-03 12:38:49 +02:00
  • fa7bcc3f29 One more minor attn fix Aleksa Gordic 2024-08-03 11:54:10 +02:00
  • fdd5345931 Refactor attn mechanism Aleksa Gordic 2024-08-03 11:49:26 +02:00
  • 0199e51a26 Refactor MLP Aleksa Gordic 2024-08-03 10:46:15 +02:00
  • de879d129b Refactor checkpoint state dict map func Aleksa Gordic 2024-08-03 10:28:32 +02:00
  • c0c08ba53d Have prompts in a file instead of inline, prompt 4 is different Aleksa Gordic 2024-08-03 10:21:56 +02:00
  • f50f2de8a2 Refactor attn, change numerics but equivalent Aleksa Gordic 2024-08-03 10:11:26 +02:00
  • 465aac4aa5 Equivalent to nano llama 3 reference code Aleksa Gordic 2024-08-03 10:03:28 +02:00
  • c414d0284a Minor refactor Aleksa Gordic 2024-08-02 23:12:49 +02:00
  • 838cd13c4f Refactor Aleksa Gordic 2024-08-02 22:38:11 +02:00
  • 41bf8e09d7 Equivalent with nano llama 3 Aleksa Gordic 2024-08-02 21:27:05 +02:00
  • 74c0a59524 add llm.cpp link to notable forks in readme Yusong Gao 2024-08-02 23:32:52 +08:00
  • 29aacba1f5 Merge pull request #705 from gordicaleksa/refactor_c Andrej 2024-07-30 13:25:04 -07:00
  • ef12d1b80e Merge pull request #717 from ngc92/nvml Andrej 2024-07-30 13:10:23 -07:00
  • 3cefe09f7e Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-07-30 20:08:13 +00:00
  • e48ebc0138 Merge branch 'mspronesti-master' Andrej Karpathy 2024-07-30 20:08:04 +00:00
  • 1566d6b9f6 revert makefile change Andrej Karpathy 2024-07-30 20:07:54 +00:00
  • 9cc357e27d Merge pull request #715 from karpathy/feature/restore_from_master Andrej 2024-07-30 13:04:38 -07:00
  • 51dd102328 remove confusing comment feature/restore_from_master Andrej Karpathy 2024-07-30 20:01:43 +00:00
  • e4b5e5775f disable if header is not found Erik Schultheis 2024-07-29 10:00:44 +02:00
  • bb9467dcca make gpu log configurable Erik Schultheis 2024-07-28 12:35:19 +02:00
  • bdff672741 add temperature Erik Schultheis 2024-07-28 12:10:57 +02:00
  • 294a6bca55 throttle reason Erik Schultheis 2024-07-28 12:07:22 +02:00
  • 743062142c units + more frequent reporting Erik Schultheis 2024-07-28 11:07:37 +02:00
  • 7d7954caa7 nvml for more detailed gpu status info Erik Schultheis 2024-07-28 10:51:27 +02:00
  • 2b827f1659 bring back state allocation into build_from_checkpoint Andrej Karpathy 2024-07-29 02:39:50 +00:00
  • 574ef9b87d chore(permute): use common.h utils for validation and benchmarking mspronesti 2024-07-29 00:19:31 +01:00
  • a49da7d917 make: generalize TARGETS definition mspronesti 2024-07-29 00:19:03 +01:00
  • a794bcb395 small fixes Andrej Karpathy 2024-07-28 17:10:20 +00:00
  • b2ae84745d Merge branch 'restore_from_master_weights' of https://github.com/ademeure/llm.c into ademeure-restore_from_master_weights Andrej Karpathy 2024-07-28 16:23:16 +00:00
  • b4623bc5e7 Merge pull request #712 from indianspeedster/dev-cuda Andrej 2024-07-28 06:08:20 -10:00
  • 7379cd10b4 Merge pull request #713 from mspronesti/master Andrej 2024-07-28 06:05:44 -10:00
  • 362c6a8d7f Merge pull request #700 from YuchenJin/integer-overflow Andrej 2024-07-27 07:26:58 -10:00
  • c6e74ea322 fix(dev/cuda): memory leaks mspronesti 2024-07-26 00:32:47 +01:00
  • a80bce65e5 Added compilation steps in makefile for permute.cu indianspeedster 2024-07-25 18:02:06 -04:00
  • 8952c524af added kernel to permute in dev/cuda indianspeedster 2024-07-25 17:35:29 -04:00
  • 52e6e0f80b fix bug from merge (init_state set to false too late) ademeure 2024-07-25 21:32:01 +00:00
  • 4d77ece6b4 simplified further (don't need non-functional error checking...) ademeure 2024-07-20 16:45:24 +00:00
  • 2eabc223cc simplify a little bit ademeure 2024-07-20 16:31:27 +00:00
  • 9781627a8a allow restoring from checkpoint of different precision ademeure 2024-07-20 16:06:24 +00:00
  • 5cae10f907 make restoring from master weights actually work ademeure 2024-07-20 16:04:03 +00:00
  • f470fbd56d Allow restoring weights from the master weights of a checkpoint (deterministically by also saving RNG state of last update) ademeure 2024-07-20 15:45:50 +00:00
  • cb4451137b Merge pull request #703 from rosslwheeler/bugfix/msvc_latest_fix Andrej 2024-07-25 13:31:41 -07:00
  • e33402f7f3 Merge pull request #694 from ngc92/model-init-cleanup Andrej 2024-07-25 13:14:11 -07:00
  • 16c990fdc0 Refactor C code Aleksa Gordic 2024-07-21 11:20:04 +02:00
  • 9dc920a66f Move WIN32 code into unistd.h Ross Wheeler 2024-07-20 12:05:20 -07:00
  • b203820232 Including unistd.h in cudnn_att.cpp for latest MSVC compiler Ross Wheeler 2024-07-20 11:57:10 -07:00
  • 30f4f223be Fix integer overflow by using size_t for parameter sizes. YuchenJin 2024-07-19 08:55:44 -07:00
  • 9bb7a36730 tiny cleanup Erik Schultheis 2024-07-18 21:47:26 +02:00
  • f2ac85ea53 estimate maximum batch size Erik Schultheis 2024-07-18 20:12:29 +02:00
  • 794ed07f3c print memory report after allocating model state Erik Schultheis 2024-07-18 20:06:13 +02:00