mirrors / karpathy--llm.c

提交

提交图

  • 967420d1d1 fix print format warning for size_t vs int Andrej Karpathy 2024-05-21 19:30:02 +00:00
  • 587506d096 torch tensor can't handle uint16 so let's convert to int32, which is silly because we'll convert to .long right after but ok Andrej Karpathy 2024-05-21 19:22:09 +00:00
  • 170c92b771 Merge pull request #440 from karpathy/feature/organize_data Andrej 2024-05-21 11:32:38 -07:00
  • 54ccbd300c docs on master-breaking changes around dataset file representation feature/organize_data Andrej Karpathy 2024-05-21 18:31:19 +00:00
  • 7d58fd2abb adjust py file as well and make the errors better Andrej Karpathy 2024-05-21 18:29:55 +00:00
  • f7cb77f3d9 docs on master-breaking change around how we store data .bin files Andrej Karpathy 2024-05-21 18:20:51 +00:00
  • ccc240caab make comment more helpful Andrej Karpathy 2024-05-21 18:01:35 +00:00
  • d53608820a the write_datafile function accepts python list, which i think is faster but i didn't check Andrej Karpathy 2024-05-21 17:55:56 +00:00
  • 9bad49a99a also fix the python file. that should be it now, w.r.t. the new token format .bin files Andrej Karpathy 2024-05-21 17:46:14 +00:00
  • 666145e7b8 also fix the .c file and fp32 file Andrej Karpathy 2024-05-21 17:30:56 +00:00
  • 1defbd4a19 adjust the dataloader to load the new .bin data files, and both prod datasets to use it instead Andrej Karpathy 2024-05-21 17:05:27 +00:00
  • a3801f01ef added algorithm header for std::sort on windows (not sure about compile time impact...) ademeure 2024-05-21 16:53:11 +01:00
  • f7cbb30316 step 2 of dataloader refactor: separate out the dataloader to its own file, change its signature a little bit, and (notably) change from cudaMallocHost to a simple malloc, so that this file doesn't have to be cuda aware Andrej Karpathy 2024-05-21 15:32:46 +00:00
  • b5e75dde8e Fully deterministic encoder backward kernels for train_gpt2.cu ademeure 2024-05-21 15:57:07 +01:00
  • 4bbd01fe12 add fineweb, and add the first version of a new write_shard function that contains a header properly Andrej Karpathy 2024-05-21 00:16:07 +00:00
  • c51cd70cfb fix a slip in gitignore, i think i am getting tired today Andrej Karpathy 2024-05-20 23:22:31 +00:00
  • bced34d71e adjust gitignore Andrej Karpathy 2024-05-20 23:15:44 +00:00
  • 7d11b7996c i also forgot to include the readme file for the new dev/data dir Andrej Karpathy 2024-05-20 23:13:42 +00:00
  • 71774b3c3d oops forgot to include data_common.py Andrej Karpathy 2024-05-20 23:10:49 +00:00
  • f671cf9288 more changes, trying to help people out because when this merges to master it will brick everyone's code... Andrej Karpathy 2024-05-20 23:02:41 +00:00
  • 722e5b2fe5 refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale Andrej Karpathy 2024-05-20 22:43:40 +00:00
  • 2b0667aee1 update the utils function and assert lancer 2024-05-20 08:00:39 -07:00
  • 6348d4196d fix the unsupported block_size lancer 2024-05-19 17:39:25 -07:00
  • 6c8bc172cc Merge pull request #436 from ChrisDryden/boundscheck Andrej 2024-05-20 00:41:41 +01:00
  • 2751fa0880 Merge pull request #435 from ChrisDryden/warpsize Andrej 2024-05-20 00:39:15 +01:00
  • 6de1137e0e Moved bounds checks outside of kernel into assertions Christopher 2024-05-19 23:11:57 +00:00
  • ead5d35973 Added warpsize as a constant for better compile time optimization and standardization Christopher 2024-05-19 22:54:47 +00:00
  • 1809740a1c Merge pull request #433 from junaire/jun/some_fix Andrej 2024-05-19 21:02:04 +01:00
  • 00c291c679 Merge pull request #434 from ngc92/profile-fix Andrej 2024-05-19 20:57:11 +01:00
  • 299ce659ec initialize multi_gpu_config so profile doesn't crash Erik Schultheis 2024-05-19 22:25:00 +03:00
  • 5a26799efb Merge branch 'ngc92-gradient-clipping' Andrej Karpathy 2024-05-19 17:51:44 +00:00
  • bc58cd1dc1 fix small bugs in grad clip, introduce a GPU CPU synch point to communicate the float grad_clip to print it, and small printing changes Andrej Karpathy 2024-05-19 17:51:29 +00:00
  • c2d12f725e small touchups to grad clip Andrej Karpathy 2024-05-19 17:07:55 +00:00
  • da9f4e21b0 Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clipping Andrej Karpathy 2024-05-19 16:03:53 +00:00
  • 9e645314e6 Check if file exists using platform specific commands & Add *.o to gitignore Jun Zhang 2024-05-20 00:00:21 +08:00
  • 4a899f5685 Merge pull request #431 from karpathy/feature/hellaswag Andrej 2024-05-19 16:53:38 +01:00
  • 8d55c4a6ac fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracy feature/hellaswag Andrej Karpathy 2024-05-19 15:45:28 +00:00
  • 4e14b52285 report both acc and acc_norm Andrej Karpathy 2024-05-19 15:18:04 +00:00
  • 7bd2389c1f add mmlu as well and refine both a bit Andrej Karpathy 2024-05-19 14:23:51 +00:00
  • aec9ce5d0a move hellaswag file to dev Andrej Karpathy 2024-05-19 12:18:02 +00:00
  • 77b991281f add hellaswag reference eval. it looks pretty bad honestly, the examples themselves, but i'm told it is predictive... hmm Andrej Karpathy 2024-05-19 11:51:01 +00:00
  • 66ce5766e0 fixed up dev/cuda Erik Schultheis 2024-05-18 23:06:26 +03:00
  • 589ead1e4f updated code to adapt to latest changes ngc92 2024-05-08 00:13:09 +02:00
  • c3a3b9daa5 fixed profile target Erik Schultheis 2024-05-03 01:08:04 +03:00
  • a9947a8315 added a flag and the missing sqrt; testing now has clipping enabled Erik Schultheis 2024-05-03 00:39:06 +03:00
  • d7a81ef26f added a useful mixed precision utility for dev/cuda Erik Schultheis 2024-05-01 13:35:54 +03:00
  • 44d45bdd6a first draft for gradient clipping by global norm Erik Schultheis 2024-05-01 04:24:54 +03:00
  • 3de05baaf4 Merge branch 'ngc92-errchk' Andrej Karpathy 2024-05-18 18:35:28 +00:00
  • 4374360015 adjust wte upper bound a bit, and print always because this part is really tricky and i don't trust anything other than manual inspection, even if we pass, allegedly Andrej Karpathy 2024-05-18 18:34:24 +00:00
  • bf36a4b6b5 improved numerical error checking: tighter tolarances relative tolerance based of bf16 epsilon less verbose output if all is OK Erik Schultheis 2024-05-18 13:27:35 +03:00
  • f226a2c9d9 Merge branch 'ngc92-alloc' Andrej Karpathy 2024-05-17 15:13:48 +00:00
  • c8fa7a8c63 revert the adamw allocation to previous. minor Andrej Karpathy 2024-05-17 15:13:36 +00:00
  • b24279c4db remove duplicate workspace allocation Erik Schultheis 2024-05-17 01:15:47 +03:00
  • 88c3bea890 print message before actual allocation for more informative OOM behaviour Erik Schultheis 2024-05-17 00:57:09 +03:00
  • 0f23723ae4 joined optimizer state allocation Erik Schultheis 2024-05-17 00:01:38 +03:00
  • b554d4d093 Merge pull request #423 from anthonix/readme_link Andrej 2024-05-16 21:35:18 +01:00
  • 3113656e3f Add link to AMD fork Anthony Blake 2024-05-16 13:30:45 -07:00
  • 109f516367 Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs Andrej 2024-05-16 21:23:56 +01:00
  • 9e04deabe3 Merge pull request #416 from azret/feature/cpu_dev Andrej 2024-05-16 21:12:55 +01:00
  • 130712215d Merge branch 'backward-bias' of https://github.com/ngc92/llm.c Andrej Karpathy 2024-05-16 20:00:56 +00:00
  • 68eea0bd5f Fix ZeRO-1, we were communicating way too much Andrej Karpathy 2024-05-16 19:48:24 +00:00
  • 6cfc7c5ce3 Merge branch 'zero-update' of https://github.com/ngc92/llm.c into ngc92-zero-update Andrej Karpathy 2024-05-16 19:30:40 +00:00
  • bd7dc7af96 Merge pull request #422 from karpathy/feature/recompute Andrej 2024-05-16 20:25:31 +01:00
  • d7581fc542 make recompute be an int instead of bool, so we can strengthen it over time just like ZeRO stages, as we recompute more and more of the model in the future possibly. and make it default on because it is awesome feature/recompute Andrej Karpathy 2024-05-16 19:07:39 +00:00
  • fbd8f03eea fixup profiling Erik Schultheis 2024-05-16 14:25:06 +03:00
  • 8b57cf6535 reduce communication overhead for ZERO stage 1 Erik Schultheis 2024-05-16 14:12:29 +03:00
  • 57f70ea66b simplify multi-gpu logic by reducing #ifdefs Erik Schultheis 2024-05-16 11:17:55 +03:00
  • 3277ccc35d Merge branch 'act-recomp' of https://github.com/ngc92/llm.c into ngc92-act-recomp Andrej Karpathy 2024-05-16 10:26:23 +00:00
  • fbc5cba1f9 Merge pull request #415 from azret/feature/eot Andrej 2024-05-16 10:45:22 +01:00
  • d48c3a4945 (optionally) recompute gelu activations to reduce activation memory Erik Schultheis 2024-05-16 12:39:28 +03:00
  • 7b810c1a3b update main training script Erik Schultheis 2024-05-16 00:20:52 +03:00
  • 858c6e6dae deterministic kernel Erik Schultheis 2024-05-14 18:40:57 +03:00
  • 2ccdfb70e0 general cleanup Erik Schultheis 2024-05-14 10:44:12 +03:00
  • 0f48a28f12 Merge pull request #412 from karpathy/feature/grad_accum3 Andrej 2024-05-15 22:53:16 +01:00
  • 8eb3a432d9 revert all streams and synchronization events, we'll bring them back but only one at a time and very very carefully, because they cause really subtle bugs and issues feature/grad_accum3 Andrej Karpathy 2024-05-15 21:09:46 +00:00
  • 160b3bd007 Setting up dev/CPU area with the first matmul_forward.c Azret Botash 2024-05-14 18:22:39 -07:00
  • 16f9dad301 Update train_gpt2.c Azret Botash 2024-05-14 17:45:08 -07:00
  • 7de50af511 train_gpt.c: Removing the hardcoded GPT2_EOT Azret Botash 2024-05-14 17:38:09 -07:00
  • 3c3c965840 Adding Mersenne Twisters C Azret Botash 2024-05-14 17:16:24 -07:00
  • a4567ae940 delete parallels, still not fixed Andrej Karpathy 2024-05-14 22:13:54 +00:00
  • 92fc26eba4 the nuts and bolts of gradient accumulation again, merged to master, but there is a bug and it doesn't work, debugging... Andrej Karpathy 2024-05-14 21:32:47 +00:00
  • b2a5508b84 constness fixes Erik Schultheis 2024-05-14 22:48:59 +03:00
  • 3b5933ecfb considerably speed up CPU matmul while still keeping it relatively readable Erik Schultheis 2024-05-14 22:32:24 +03:00
  • 2d43e5bc97 remove legacy comment Andrej Karpathy 2024-05-14 19:14:07 +00:00
  • 222d59fa2f Merge pull request #408 from ngc92/layernorm-bw-dev-cuda Andrej 2024-05-14 20:13:09 +01:00
  • e553e2f084 update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results Erik Schultheis 2024-05-14 20:44:50 +03:00
  • dd8c9f5ec9 fix layernorm backward: accumulate weight gradient Erik Schultheis 2024-05-14 20:43:51 +03:00
  • 2346cdac93 Merge branch 'ngc92-backward-bias' Andrej Karpathy 2024-05-13 20:19:09 +00:00
  • 37a96debbf Merge branch 'backward-bias' of https://github.com/ngc92/llm.c into ngc92-backward-bias Andrej Karpathy 2024-05-13 20:18:33 +00:00
  • 750c5fd7e8 Merge pull request #309 from ahrefs/zero-stage1 Andrej 2024-05-13 21:08:53 +01:00
  • c66e48c06c fixup comment Erik Schultheis 2024-05-13 20:58:07 +03:00
  • 65727d5a4d fix CI compile by disabling kernel 5 Erik Schultheis 2024-05-13 19:20:03 +03:00
  • 49ee3c8307 fix non-atomic version: * accumulate instead of assign * need dedicated argument to correctly handle the floatX == float case Erik Schultheis 2024-05-13 18:27:56 +03:00
  • aa41b32624 update main file Erik Schultheis 2024-05-13 17:48:25 +03:00
  • 081d224b21 automatically switch to buffer-less version if that can fill up the GPU Erik Schultheis 2024-05-13 17:39:32 +03:00
  • c0329ebdba new kernel version with fewer atomics Erik Schultheis 2024-05-13 17:15:48 +03:00
  • f613ce895b Fix copy and cast params to master weights chinthysl 2024-05-13 09:13:02 +00:00
  • 086ce2ff4d Remove unsused template specializations and refactor chinthysl 2024-05-13 06:48:22 +00:00
  • b3a5d1da15 shard master_weights chinthysl 2024-05-13 05:55:46 +00:00
  • 0d9ce74b74 Merge branch 'karpathy:master' into zero-stage1 Chinthaka Gamanayake 2024-05-13 13:34:52 +08:00