mirrors / karpathy--llm.c

提交

提交图

  • f0ffb648cc Represent loss threeshold using scientific notation Aleksa Gordic 2024-06-18 16:00:00 +02:00
  • 8f7f20506e Stricter FP32 tests Aleksa Gordic 2024-06-18 15:41:38 +02:00
  • 1120fa4465 Remove -1, point at current sample Aleksa Gordic 2024-06-18 13:35:17 +02:00
  • 37c3815ede fix path Erik Schultheis 2024-06-18 14:35:05 +03:00
  • 077f9bdd59 Fix dataloader & determinism testing Aleksa Gordic 2024-06-18 13:23:13 +02:00
  • 33136e0b08 fix compilation with clang Erik Schultheis 2024-06-18 14:16:59 +03:00
  • dbeb8fc551 added missing checks Erik Schultheis 2024-06-18 13:29:12 +03:00
  • 188e7274e6 add unit test to CI Erik Schultheis 2024-06-18 01:22:31 +03:00
  • 593a71c374 use new functions for checkpointing Erik Schultheis 2024-06-06 00:58:15 +03:00
  • 04234d0c24 utility functions for device <-> disk IO Erik Schultheis 2024-06-06 00:49:11 +03:00
  • 100bba0329 multigpu training without mpi chinthysl 2024-05-17 14:16:46 +00:00
  • 30095bcb0b Fixes on accessing past memory buffer Ross Wheeler 2024-06-17 21:37:35 -07:00
  • efbbdc8f46 Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-06-18 03:03:37 +00:00
  • 4fd2df8870 Merge branch 'ngc92-check-determinism' Andrej Karpathy 2024-06-18 03:03:06 +00:00
  • 5b4fb0d457 delete intermediate files produced by test code Andrej Karpathy 2024-06-18 03:02:53 +00:00
  • 75b844898d delete warning using an explicit cast to size_t Andrej Karpathy 2024-06-18 03:01:06 +00:00
  • 4e0221de0d Merge branch 'karpathy:master' into feature/dataloader_test rosslwheeler 2024-06-17 18:18:23 -07:00
  • 37023dd9a1 Cleanup for data loader Ross Wheeler 2024-06-17 18:15:29 -07:00
  • 19f2c609eb Merge branch 'check-determinism' of https://github.com/ngc92/llm.c into ngc92-check-determinism Andrej Karpathy 2024-06-17 22:16:21 +00:00
  • a33a49a7b6 Merge pull request #606 from gordicaleksa/minor_refactor_dataloader Andrej 2024-06-17 14:45:36 -07:00
  • 19a34d8cdd fix dataloader Erik Schultheis 2024-06-17 23:59:54 +03:00
  • c06ef28a0d also test dataloader as part of test script Erik Schultheis 2024-06-17 23:16:48 +03:00
  • f104009bc7 Merge pull request #600 from gordicaleksa/add_faster_layernorm_kernel Andrej 2024-06-17 14:19:22 -07:00
  • 88d4eff946 Swap kernel 5 for 3 for fallback Aleksa Gordic 2024-06-17 23:05:01 +02:00
  • 674462adfe added missing check Erik Schultheis 2024-06-17 23:16:36 +03:00
  • 4d35bac1ae added a determinism check Erik Schultheis 2024-06-17 13:23:12 +03:00
  • 7b5ee07e18 Minor refactor of dataloader Aleksa Gordic 2024-06-17 21:53:18 +02:00
  • d05e848010 fix pytorch code bug forward needs to know about backward_grad_sync Andrej Karpathy 2024-06-17 19:49:36 +00:00
  • be0e10b518 Permuted DataLoader Andrej 2024-06-17 12:28:03 -07:00
  • 14d9cd671e resolve merge conflict in staate file between use_master_weights and DataLoader should_shuffle feature/permuted_dataloader Andrej Karpathy 2024-06-17 19:16:42 +00:00
  • 93d1450bdd malloc -> mallocCheck in dataloader.h Andrej Karpathy 2024-06-17 18:56:24 +00:00
  • de53e71729 small touchups Andrej Karpathy 2024-06-17 18:55:16 +00:00
  • d57f112e16 Merge pull request #604 from gordicaleksa/minor_refactor_and_fix_guards Andrej 2024-06-17 10:17:24 -07:00
  • 9b5ec8f719 Set v memory to random floats >=0 Aleksa Gordic 2024-06-17 17:44:00 +02:00
  • a0a8795be1 Fix guarding logic, refactor Aleksa Gordic 2024-06-17 14:12:05 +02:00
  • f845efd49a Add nvtx range to layernorm forward Aleksa Gordic 2024-06-16 18:01:52 +02:00
  • c6040eef5f Even better kernel - modification of what we use for fused residual fwd Aleksa Gordic 2024-06-16 16:36:50 +02:00
  • 0f36cdd15b Use faster kernel 5 for layernorm forward Aleksa Gordic 2024-06-16 10:37:39 +02:00
  • fa7f2b6763 Merge pull request #602 from gordicaleksa/hotfix_global_norm_assert Andrej 2024-06-16 15:14:47 -07:00
  • 0eec8cee4d Patch numpy version Aleksa Gordic 2024-06-16 23:59:29 +02:00
  • 8f724dbb72 Add a minor note that code is coupled Aleksa Gordic 2024-06-16 23:54:09 +02:00
  • 9caf2c7c2f Proper handling of max num of block sums Aleksa Gordic 2024-06-16 23:49:34 +02:00
  • 13c893ffec Merge pull request #561 from lancerts/fix-warning Andrej 2024-06-16 09:25:51 -07:00
  • 1973631c6b Merge pull request #583 from vyom1611/FLT_MAX-fix Andrej 2024-06-16 09:25:03 -07:00
  • 3e7d89b81e Merge pull request #522 from gordicaleksa/add_master_weights_to_state Andrej 2024-06-16 09:22:10 -07:00
  • 8f40ef17e3 Merge pull request #597 from gordicaleksa/fix_stochastic_rounding Andrej 2024-06-16 09:13:37 -07:00
  • 89e5d51fa4 Merge pull request #596 from gordicaleksa/fix_copy_casting Andrej 2024-06-16 09:02:14 -07:00
  • a4852cbafb Merge pull request #598 from gordicaleksa/add_new_dev_kernel Andrej 2024-06-16 09:00:01 -07:00
  • d5601d16da Fix stuff in layernorm forward Aleksa Gordic 2024-06-16 09:34:01 +02:00
  • d003274b7c Remove warp size - moved to common.h Aleksa Gordic 2024-06-15 20:23:59 +02:00
  • 09910acc0b Remove blockreduce from classifier fused, move to common.h Aleksa Gordic 2024-06-15 19:55:11 +02:00
  • 0f112f1b62 Fix CI complaining about default args Aleksa Gordic 2024-06-15 19:39:05 +02:00
  • 1e7c31eef0 add test for shuffled, permuted, multi-process dataloader Andrej Karpathy 2024-06-15 16:36:43 +00:00
  • f089f90bcf add multiprocess test simple Andrej Karpathy 2024-06-15 16:21:05 +00:00
  • d4255f1d19 first draft of a dataloader test Andrej Karpathy 2024-06-15 15:56:56 +00:00
  • f3195d19af Expose deviceProp as a global var Aleksa Gordic 2024-06-15 17:40:14 +02:00
  • 0f6853d399 Add global norm prod kernels to dev Aleksa Gordic 2024-06-15 17:29:14 +02:00
  • ff14b0819f Make sure each parameter was rounded with a unique seed Aleksa Gordic 2024-06-15 12:02:39 +00:00
  • 20f0e7f4e8 Fix copy and casting from spawning excessive threads Aleksa Gordic 2024-06-15 08:15:38 +00:00
  • 60ed946641 Merge branch 'dataloader_rand' of https://github.com/gordicaleksa/llm.c into gordicaleksa-dataloader_rand Andrej Karpathy 2024-06-14 22:10:55 +00:00
  • c8cd023063 Full determinism Aleksa Gordic 2024-06-14 21:44:20 +00:00
  • d7ea4da298 add scripts to export to HF and run Eleuther evals Andrej Karpathy 2024-06-14 19:49:05 +00:00
  • 0e69e3ae46 Merge pull request #592 from gordicaleksa/add_to_mfu_table Andrej 2024-06-14 09:29:00 -07:00
  • ee4b056ddf Merge branch 'karpathy:master' into FLT_MAX-fix Vyom Sharma 2024-06-14 16:32:37 +05:30
  • 0f144e3052 Support 100% determinism, fix race condition, update global norm Aleksa Gordic 2024-06-13 10:36:16 +00:00
  • 269b54ae45 Allocate memory in load state Aleksa Gordic 2024-06-07 16:34:59 +02:00
  • 2d23d1f08e Save master weights Aleksa Gordic 2024-06-07 16:24:44 +02:00
  • 328641f728 Minor refactor Aleksa Gordic 2024-06-14 09:57:19 +02:00
  • ef3afcb1d6 Fix num samples bug Aleksa Gordic 2024-06-14 09:37:38 +02:00
  • 2cd4d3985d Add H100 PCIe Aleksa Gordic 2024-06-14 09:23:55 +02:00
  • 2bc0b474c5 Merge pull request #590 from karpathy/feature/delete_backward_act Andrej 2024-06-13 18:21:25 -07:00
  • cbe82e783a consolidate memory by deleting the grad activation struct, move some of the last pieces of memory to the forward pass feature/delete_backward_act Andrej Karpathy 2024-06-14 00:54:58 +00:00
  • 747a5792b7 Merge pull request #582 from vyom1611/master Andrej 2024-06-13 16:47:20 -07:00
  • 3bedc9853c Merge pull request #589 from karpathy/feature/overlap-backward-reduce Andrej 2024-06-13 16:31:48 -07:00
  • 43a903e1b4 revert old printf0, minor touchups and comments, add assert feature/overlap-backward-reduce Andrej Karpathy 2024-06-13 23:18:22 +00:00
  • 6515f838d9 Merge branch 'comp-and-comms' of https://github.com/ngc92/llm.c into ngc92-comp-and-comms2 Andrej Karpathy 2024-06-13 22:34:50 +00:00
  • 3fc8d6192f strided all-gather Erik Schultheis 2024-06-10 18:49:16 +03:00
  • 6c10cf2791 Remove local paths Aleksa Gordic 2024-06-13 23:51:07 +02:00
  • efd9df4c1b Remove bool, add permutation on every shard advance Aleksa Gordic 2024-06-13 23:36:24 +02:00
  • 7e9328f724 Remove redundant headers, refactor Aleksa Gordic 2024-06-13 23:15:39 +02:00
  • 1329a2c80c Change type from size_t to int to save space Aleksa Gordic 2024-06-13 23:04:15 +02:00
  • 814b83bbf1 Refactor Aleksa Gordic 2024-06-13 22:43:57 +02:00
  • 1dafa60ad9 Hardcode shuffle info in all train scripts Aleksa Gordic 2024-06-13 21:27:42 +02:00
  • 8367a56c03 Merge branch 'comp-and-comms' of https://github.com/ngc92/llm.c into ngc92-comp-and-comms2 Andrej Karpathy 2024-06-13 19:25:45 +00:00
  • b7933ef62c Modify state save/load Aleksa Gordic 2024-06-13 21:22:16 +02:00
  • afcc0a4f71 Merge pull request #586 from gordicaleksa/readme_debug_note Andrej 2024-06-13 11:30:20 -07:00
  • 8f61f48cba Add random inter/intra shard data shuffling Aleksa Gordic 2024-06-13 20:17:14 +02:00
  • 2faa5e2b6c Update other calls with null Aleksa Gordic 2024-06-07 21:36:08 +00:00
  • d57a52408e Make C compiler happy - no default args Aleksa Gordic 2024-06-07 21:31:43 +00:00
  • c3db1bb112 Set default seed value to null Aleksa Gordic 2024-06-07 20:27:28 +00:00
  • af0b2f51c0 Start from a random position in a random shard Aleksa Gordic 2024-06-07 20:02:00 +00:00
  • 2a6797b1cd Add debugging tip to README Aleksa Gordic 2024-06-13 13:58:14 +02:00
  • f071efd94f stride-based adam and copy-and-cast Erik Schultheis 2024-06-10 18:49:16 +03:00
  • 99965a1e3e fix typo Erik Schultheis 2024-06-10 18:49:16 +03:00
  • e2a18d9385 Fixed missing semi-colon vyom1611 2024-06-13 00:59:15 +05:30
  • dc8d8de06a Replaced hard-coded max float with FLT_MAX vyom1611 2024-06-13 00:56:53 +05:30
  • 9d9da3b209 fewer global norm kernel calls Erik Schultheis 2024-06-09 22:19:07 +03:00
  • afa7978227 Merge branch 'karpathy:master' into master Vyom Sharma 2024-06-13 00:19:16 +05:30
  • 95cef79c17 Merge pull request #578 from ngc92/matmul Andrej 2024-06-12 09:05:04 -07:00
  • 42dff87939 clarification we trained with 150B tokens Andrej Karpathy 2024-06-12 15:51:14 +00:00