mirrors / karpathy--llm.c

提交

提交图

  • ac931455a4 Merge pull request #533 from karpathy/feature/refactor_part2 Andrej 2024-06-03 13:44:59 -07:00
  • 001ad78fe1 resolve deviceProp and cudaCheck dupes feature/refactor_part2 Andrej Karpathy 2024-06-03 19:54:32 +00:00
  • 020b15ad77 small more fixes Andrej Karpathy 2024-06-03 19:32:32 +00:00
  • 5d9403479c some fixes, still not done Andrej Karpathy 2024-06-03 18:50:41 +00:00
  • 804d0693c8 refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well Andrej Karpathy 2024-06-03 18:10:33 +00:00
  • e40268457f Merge pull request #526 from karpathy/try/move_cudnn_to_llmc Andrej 2024-06-03 09:32:31 -07:00
  • 0d16b51626 more efficient use of memory buffers for LN recomputation Erik Schultheis 2024-06-03 18:19:52 +03:00
  • 6434009298 attempt to remove PATH_SEP from Makefile try/move_cudnn_to_llmc Andrej Karpathy 2024-06-03 15:30:45 +00:00
  • 42ca620306 Minor refactor: remove hardcoded val, delete unused vars... Aleksa Gordic 2024-06-03 16:24:22 +02:00
  • 9b540fa859 Makefile updated to match script vyom1611 2024-06-03 18:59:57 +05:30
  • fbc1e4983c Benchmark modal script revamped - profiling and external libraries vyom1611 2024-06-03 17:00:10 +05:30
  • 0867653aaf Merge remote-tracking branch 'origin/master' vyom1611 2024-06-03 16:48:58 +05:30
  • 7413a1a315 Remove unnecessary load filename line Aleksa Gordic 2024-06-03 12:33:33 +02:00
  • 728762ceed Improve start of training print info Aleksa Gordic 2024-06-03 12:31:33 +02:00
  • d9e867678e Remove file size - not used Aleksa Gordic 2024-06-03 12:14:58 +02:00
  • e86bd62374 Refactor eval loader - remove 2 unnecessary malloc calls per iteration Aleksa Gordic 2024-06-03 11:43:33 +02:00
  • 41dc65a0ef attempt to move cudnn attention to the llmc directory Andrej Karpathy 2024-06-03 03:23:04 +00:00
  • 08fc45b492 delete copy paste code for gpt2 model init Andrej Karpathy 2024-06-03 02:08:29 +00:00
  • c827dd2f8e Merge branch 'ngc92-mfu' Andrej Karpathy 2024-06-03 01:41:31 +00:00
  • 1ac30591f2 add new file llmc/mfu.h for MFU calculations for different GPUs Andrej Karpathy 2024-06-03 01:41:18 +00:00
  • 80e0c3b8d9 Merge pull request #512 from gordicaleksa/refactor_encoder_bwd_kernel Andrej 2024-06-02 16:43:10 -07:00
  • cb5eff0578 Merge pull request #514 from ngc92/compile-fix Andrej 2024-06-02 16:42:31 -07:00
  • ff98d8bda7 Merge pull request #515 from gordicaleksa/fix_adamw_kernel3_num_threads Andrej 2024-06-02 16:41:59 -07:00
  • 1ce30fdc4a Merge pull request #516 from gordicaleksa/update_random_doc Andrej 2024-06-02 16:37:37 -07:00
  • 252584954e Merge pull request #519 from gordicaleksa/mem_leak Andrej 2024-06-02 16:34:54 -07:00
  • 1e893f4e11 Merge branch 'gordicaleksa-fix_zero_stage1' Andrej Karpathy 2024-06-02 23:28:22 +00:00
  • 4944ac5743 adjust comments Andrej Karpathy 2024-06-02 23:28:13 +00:00
  • a8a5a444bc Add ; - ah c my old friend :) Aleksa Gordic 2024-06-02 23:04:47 +02:00
  • 9a455edc25 Improve comments Aleksa Gordic 2024-06-02 23:03:41 +02:00
  • b4e8bd7f66 Fix zero stage 1 norm computation - WIP Aleksa Gordic 2024-06-02 22:45:26 +02:00
  • df0a826ef3 Close file Aleksa Gordic 2024-06-02 21:37:52 +02:00
  • 30d81cce15 Fix mem leak Aleksa Gordic 2024-06-02 21:28:42 +02:00
  • 67379a28cc add edu fineweb support, with 10B and 1000B version “eliebak” 2024-06-02 14:18:44 +00:00
  • 79cbc03ccf Use local params for num blocks Aleksa Gordic 2024-06-02 15:06:06 +02:00
  • ae9c957273 Add clarification on Box-Muller Aleksa Gordic 2024-06-02 15:04:45 +02:00
  • a902323f36 fix compilation with older nvcc Erik Schultheis 2024-06-02 12:40:28 +03:00
  • 290c00a362 Remove redundant CPU computation Aleksa Gordic 2024-06-01 18:27:24 +02:00
  • a26041af12 Merge pull request #495 from ChrisDryden/shared_memory Andrej 2024-06-01 08:12:21 -07:00
  • 9cf8c2f135 Merge pull request #509 from ngc92/inplace Andrej 2024-06-01 08:10:56 -07:00
  • bd0f036ee5 Merge pull request #511 from gordicaleksa/gordicaleksa_fix_docs Andrej 2024-06-01 08:09:31 -07:00
  • 8eafd40615 Update compile cmd in the dev/cuda README Aleksa Gordic 2024-06-01 16:46:32 +02:00
  • e0abfae2bc Benchmark modal script revamped - profiling and external libraries vyom1611 2024-06-01 19:06:42 +05:30
  • 761fb2cf2b rename gelu_backward to gelu_backward_inplace, and made it obvious in the code that this is an inplace operation Erik Schultheis 2024-06-01 14:31:26 +03:00
  • cc4bed0806 added missing checks Erik Schultheis 2024-06-01 14:30:40 +03:00
  • 9c8edaffa7 fix recompute bug Andrej Karpathy 2024-05-31 22:38:03 +00:00
  • 0bc24c36e6 fix bug in resumption of dataloader, have to reload shard Andrej Karpathy 2024-05-31 22:19:37 +00:00
  • 1de3bef0ac fix argparse comment too Andrej Karpathy 2024-05-31 22:06:48 +00:00
  • 60a1037db9 Merge branch 'ChrisDryden-layernorm_backwards_mem_saving' Andrej Karpathy 2024-05-31 22:04:44 +00:00
  • f4874a9f63 make things a bit more consistent, keep recompute 1 as default, and add a comment Andrej Karpathy 2024-05-31 22:04:34 +00:00
  • 629a24020d Merge branch 'layernorm_backwards_mem_saving' of https://github.com/ChrisDryden/llm.c into ChrisDryden-layernorm_backwards_mem_saving Andrej Karpathy 2024-05-31 21:47:28 +00:00
  • ed37d9261b fix bug in readme Andrej Karpathy 2024-05-30 21:39:47 +00:00
  • b48224cb0c refactor the README file because it was getting crazy Andrej Karpathy 2024-05-30 21:39:06 +00:00
  • 0fddf06043 more docs on scripts Andrej Karpathy 2024-05-30 21:06:36 +00:00
  • fe36e40f2c also move pyrun script to scripts/ Andrej Karpathy 2024-05-30 20:17:16 +00:00
  • d98d3ed82b code refactor, introduce llmc dir and scripts dir Andrej 2024-05-30 13:13:04 -07:00
  • 22d1607f6d delete these files, they are now in scripts/ folder and better feature/libdir Andrej Karpathy 2024-05-30 20:11:59 +00:00
  • fc40ffc4aa adjust readme to new scripts/ dir Andrej Karpathy 2024-05-30 20:10:04 +00:00
  • a0d5dbbda6 scripts folder Andrej Karpathy 2024-05-30 20:00:07 +00:00
  • 35aa110244 more refactor Andrej Karpathy 2024-05-30 19:39:29 +00:00
  • f8f9e49ba3 Merge branch 'master' into feature/libdir Andrej Karpathy 2024-05-30 19:07:51 +00:00
  • 5f547601a0 further hide the c pytorch bridge logic Andrej Karpathy 2024-05-30 18:20:29 +00:00
  • ca27eb8fa5 bugfix don't train on val batch Andrej Karpathy 2024-05-30 18:18:05 +00:00
  • 59432fea19 modified the recalculation parameter to be based off of 2 instead of 1 Christopher 2024-05-30 17:23:47 +00:00
  • e5719e18b3 Rearranged layernorm forward for 1 and 2 and used layer variables instead. Validated with test file Christopher 2024-05-30 02:19:52 +00:00
  • 95b1fad088 First step in creating option to recalculate layernorm activations in backwards pass Christopher 2024-05-28 02:04:23 +00:00
  • 0e2298247a Merge pull request #496 from ChrisDryden/patch-7 Andrej 2024-05-30 09:13:30 -07:00
  • 862c957d3a fix default number of gpus from 4 to 8 for consistency with run124M.sh Andrej Karpathy 2024-05-30 14:50:09 +00:00
  • ed805f60be Merge pull request #494 from karpathy/feature/pytorch_parity Andrej 2024-05-30 07:47:58 -07:00
  • 5959851d2d small comments for pyrun124M.sh feature/pytorch_parity Andrej Karpathy 2024-05-30 14:47:43 +00:00
  • 9b935a8f04 fix bug with advancing position in dataloader Andrej Karpathy 2024-05-30 13:16:07 +00:00
  • 3bca31fde1 Nit: duplicated line in header of project Chris Dryden 2024-05-29 20:03:13 -07:00
  • 5450632237 Removed unnecesary shared memory due to blockreduce using static defined shared memory Christopher 2024-05-30 03:01:52 +00:00
  • 555d07502a more feature parity work, i think it's quite close now Andrej Karpathy 2024-05-30 02:29:40 +00:00
  • ea88a343b5 also updated pyrun124M.sh Andrej Karpathy 2024-05-30 02:05:41 +00:00
  • 03c2e85601 sharded dataloader for PyTorch Andrej Karpathy 2024-05-30 02:05:06 +00:00
  • 3fdf88e4d4 add logfile similar to C Andrej Karpathy 2024-05-29 23:59:35 +00:00
  • 083c50a52e small stylistic touchups that match the c code Andrej Karpathy 2024-05-29 23:50:13 +00:00
  • 1f6fdfbd9a add pyrun124M.sh Andrej Karpathy 2024-05-29 23:30:27 +00:00
  • c413bfe0ca pytorch parity v1 Andrej Karpathy 2024-05-29 23:25:55 +00:00
  • 4cc8b1570b hardcode grad_clip to 1.0, delete confusion around legacy code that used -c flag for it, but became inactive in the if else block Andrej Karpathy 2024-05-29 19:01:12 +00:00
  • 55f3665b64 Merge pull request #493 from vyom1611/master-2 Andrej 2024-05-29 07:50:10 -07:00
  • 062f096da1 Modal benchmarking script updated to replace deprecated calls vyom1611 2024-05-29 20:01:41 +05:30
  • 2db7716d68 Merge pull request #476 from lancerts/layernorm-bk-fix Andrej 2024-05-29 07:30:52 -07:00
  • c3a65a556f Merge pull request #487 from huoyushequ/patch-1 Andrej 2024-05-28 18:52:38 -07:00
  • 27cf5223e0 softmax_autoregressive_backward_kernel does not use share memory in the kernel QIAOQIAN 2024-05-29 09:23:46 +08:00
  • daa95e04ae ampere-consumer Erik Schultheis 2024-05-29 02:27:11 +03:00
  • de7a9b83cd more GPUs Erik Schultheis 2024-05-29 02:09:20 +03:00
  • 184cc9d10a added H100 SMX version Erik Schultheis 2024-05-29 01:31:16 +03:00
  • e2b411362e include string view Erik Schultheis 2024-05-29 00:54:23 +03:00
  • 9374d751de MFU for other GPUs Erik Schultheis 2024-05-29 00:48:37 +03:00
  • a664aa3094 include datasets as req Andrej Karpathy 2024-05-28 21:43:11 +00:00
  • afe40be994 Merge pull request #479 from rosslwheeler/feature/ci-manual-trigger Andrej 2024-05-28 12:46:27 -07:00
  • 88dd670c42 link to recent post from readme Andrej Karpathy 2024-05-28 17:31:31 +00:00
  • a6a2f042f8 keep -y 1 flag Andrej Karpathy 2024-05-28 15:24:30 +00:00
  • a7f01088dc repro scripts for 124M and 350M models Andrej Karpathy 2024-05-28 15:08:04 +00:00
  • 037ec67a5d small tweak to launch Andrej Karpathy 2024-05-28 15:07:07 +00:00
  • 0c83a23188 tweak the logfile parser and visualizer Andrej Karpathy 2024-05-28 01:38:15 +00:00
  • c60d7de466 Adding manual trigger for CI Ross Wheeler 2024-05-27 16:50:06 -07:00
  • 47c670b885 amend lancerts 2024-05-27 15:13:11 -07:00
  • c4c985cb87 spell explicitly uint to unsigned int lancerts 2024-05-27 14:44:47 -07:00