mirrors / karpathy--llm.c

提交

提交图

  • f27ca4df41 fix the issue Mismatch of dweight at layernorm_backward.cu lancerts 2024-05-27 14:39:57 -07:00
  • 888c0d4be3 also update the other files Andrej Karpathy 2024-05-27 21:15:09 +00:00
  • eed242702a experiment with adding the llmc lib directory Andrej Karpathy 2024-05-27 21:10:28 +00:00
  • b67880a879 Merge pull request #471 from karpathy/feature/resume Andrej 2024-05-27 13:38:52 -07:00
  • 24a8c03cb6 Merge branch 'master' into feature/resume feature/resume Andrej Karpathy 2024-05-27 20:30:30 +00:00
  • 69d0583ac8 conditionally include dirent on not windows Andrej Karpathy 2024-05-27 20:28:52 +00:00
  • 3b4d8bece6 Merge pull request #474 from rosslwheeler/Windows-dirent-fix Andrej 2024-05-27 13:27:58 -07:00
  • bb80f93618 Minor cleanup Ross Wheeler 2024-05-27 13:16:34 -07:00
  • 758acdffbf Adding dirent.h support to unistd.h Ross Wheeler 2024-05-27 13:08:07 -07:00
  • 63f0e25f5a make compiler happy Andrej Karpathy 2024-05-27 19:32:31 +00:00
  • b75738c4eb resume optimization, seems to be working Andrej Karpathy 2024-05-27 18:02:51 +00:00
  • f93a30fbbf more careful with conditional MPI use Andrej Karpathy 2024-05-27 15:54:55 +00:00
  • d295cb8d81 part 1 of v1 of resume training functionality, writes the files but doesn't load them yet, coming up in a bit Andrej Karpathy 2024-05-27 15:49:03 +00:00
  • 12999f7082 Merge pull request #470 from ngc92/bugfix Andrej 2024-05-27 06:39:56 -07:00
  • 7dc3b7d7dc bugfix Erik Schultheis 2024-05-27 16:36:44 +03:00
  • 4d6f80e02b Merge pull request #469 from ahrefs/zero1-torch Andrej 2024-05-27 06:04:00 -07:00
  • c3097ba308 Merge pull request #468 from ngc92/layernorm-bw-deadlock Andrej 2024-05-27 05:48:42 -07:00
  • b913905f84 fix out-of-bounds access for encoder_backward: we were allocating too little memory (/ -> DIV_CEIL) Erik Schultheis 2024-05-27 12:50:54 +03:00
  • 841c4dec03 update main training file Erik Schultheis 2024-05-27 03:58:45 +03:00
  • 5f73ecfcf1 fix C % 256 != 0 Erik Schultheis 2024-05-27 03:36:16 +03:00
  • d35daf19c9 fail fast and hard; don't go into the deadlock Erik Schultheis 2024-05-27 02:43:11 +03:00
  • 3b6808269a some comments and optimized shared memory amount Erik Schultheis 2024-05-27 02:34:07 +03:00
  • 9eba86f0a5 fully vectorized smem access Erik Schultheis 2024-05-27 02:17:38 +03:00
  • cb8cc25e3a utilities for Packed and more vectorization Erik Schultheis 2024-05-26 22:55:18 +03:00
  • 53ee3297dd use vectorized access to shared memory Erik Schultheis 2024-05-26 22:17:50 +03:00
  • f2809a6bdb zero changes for torch Chinthaka Gamanayakege 2024-05-27 06:44:54 +00:00
  • ee6fa5ec9f fill in the other model sizes Andrej Karpathy 2024-05-26 18:04:43 +00:00
  • 5ec4b774f2 simple jupyter notebook example on how to parse and vis the logfiles Andrej Karpathy 2024-05-26 17:40:49 +00:00
  • f145c1b101 fix the other files due to dataloader change Andrej Karpathy 2024-05-26 17:01:32 +00:00
  • c6680a25ae fix a bug where the max_steps was not accounting for the gradient accumulation, meaning that e.g. with grad accum 2, we'd accidentally train for 2 epochs instead of just 1 epoch. also simplify the interface a bit and the code make more readable etc Andrej Karpathy 2024-05-26 16:49:30 +00:00
  • 5d65c5f52e Merge pull request #464 from karpathy/attempt_fix_zero1 Andrej 2024-05-25 17:21:04 -07:00
  • 05fb642ae1 attempt fix zero 1 attempt_fix_zero1 Andrej Karpathy 2024-05-25 23:49:16 +00:00
  • 7ba3ef6bac make sure to run Adam only for the range of parameters that is available on the local device Erik Schultheis 2024-05-25 23:20:42 +03:00
  • d0c014372c add ability to export fineweb100B as well Andrej Karpathy 2024-05-25 17:21:51 +00:00
  • f5d041ab0c careful with NULL and checkpoint correctly Andrej Karpathy 2024-05-25 16:32:18 +00:00
  • 273925d10d save checkpoint files to log directory, and small readability changes Andrej Karpathy 2024-05-25 16:05:29 +00:00
  • 90a745c40c add MFU (model flops utilization) calculation and print it. also refactor the printing a bit to make it nice. this calculation only works for A100 fp16 we should think through how to generalize it a bit Andrej Karpathy 2024-05-25 14:30:59 +00:00
  • ab090a80c5 Merge pull request #461 from karpathy/feature/write_checkpoints Andrej 2024-05-25 07:01:26 -07:00
  • a13b2bd10b fix bug check null feature/write_checkpoints Andrej Karpathy 2024-05-25 13:50:56 +00:00
  • fe698b32f1 Merge pull request #457 from karpathy/feature/write_checkpoints Andrej 2024-05-24 17:56:53 -07:00
  • 2a0f78d617 attempt to fix windows episode 2 Andrej Karpathy 2024-05-25 00:44:16 +00:00
  • 5b96d91893 attempt to fix windows? Andrej Karpathy 2024-05-25 00:39:40 +00:00
  • f2ee356a06 resolve merge conflict Andrej Karpathy 2024-05-25 00:33:12 +00:00
  • 4ff0412751 Merge pull request #450 from rosslwheeler/dataloader_win_fixes Andrej 2024-05-24 17:30:35 -07:00
  • 6a7fd56d7c Merge pull request #439 from lancerts/matmul-fix Andrej 2024-05-24 17:26:37 -07:00
  • 5ef2c04a39 Merge pull request #458 from karpathy/feature/weight_decay_2d_only Andrej 2024-05-24 17:22:49 -07:00
  • 9f08882051 add weight decay, but only for 2D tensors, as done in GPT series and in general too. this forces us to break up our adamw kernel again into one call per tensor, so there is a small throughput hit, of about 0.5% for me. but we have to break up this kernel in near future anyway feature/weight_decay_2d_only Andrej Karpathy 2024-05-25 00:14:10 +00:00
  • 79738d2ca4 fixed mkdir change Ross Wheeler 2024-05-24 17:05:59 -07:00
  • e5083be900 Moved windows glob() over to dev/unistd.h Ross Wheeler 2024-05-24 16:56:52 -07:00
  • 1ec081e64e Merge branch 'karpathy:master' into dataloader_win_fixes rosslwheeler 2024-05-24 16:19:45 -07:00
  • e60c4843a3 Merge branch 'ngc92-cpu-matmul' Andrej Karpathy 2024-05-24 23:10:34 +00:00
  • 3221e4b2d2 small cosmetic changes Andrej Karpathy 2024-05-24 23:10:17 +00:00
  • 6e4296fb9a Merge branch 'cpu-matmul' of https://github.com/ngc92/llm.c into ngc92-cpu-matmul Andrej Karpathy 2024-05-24 22:59:24 +00:00
  • 5e0fa452e0 Merge branch 'karpathy:master' into dataloader_win_fixes rosslwheeler 2024-05-24 15:48:45 -07:00
  • b8674cedf8 Merge branch 'ademeure-deterministic_encoder_v2' Andrej Karpathy 2024-05-24 22:18:26 +00:00
  • 8b951056b5 resolve merge conflict Andrej Karpathy 2024-05-24 22:04:02 +00:00
  • 5ddb06182c Merge branch 'ademeure-deterministic_layernorm' Andrej Karpathy 2024-05-24 21:53:08 +00:00
  • 25f17e6748 small formatting fix before merge Andrej Karpathy 2024-05-24 21:53:00 +00:00
  • dbacaf84cf Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm Andrej Karpathy 2024-05-24 21:46:25 +00:00
  • bf03e7fd4c Merge pull request #456 from ngc92/large-batch-fix Andrej 2024-05-24 12:00:41 -07:00
  • 16b364d256 change -o to be not file but directory, and create a main.log inside it for the log. this is to prepare for writing of checkpoints, where we want to re-use this directory for those as well Andrej Karpathy 2024-05-24 17:57:50 +00:00
  • 1b98637960 int -> int64_t Erik Schultheis 2024-05-24 20:11:34 +03:00
  • df2e0dadd2 int64_t Erik Schultheis 2024-05-24 19:59:53 +03:00
  • 1d10e788af fix bug pretty sure this should just be a void* pointer Andrej Karpathy 2024-05-24 16:51:26 +00:00
  • 613a125216 add checkpoint function write to file Andrej Karpathy 2024-05-24 16:49:20 +00:00
  • 2a736cb9e2 fix for large batch sizes Erik Schultheis 2024-05-24 19:45:41 +03:00
  • 08fe3eef43 change default shard size for fineweb10B Andrej Karpathy 2024-05-24 15:56:05 +00:00
  • b0f065fcf8 Merge pull request #455 from ngc92/fineweb Andrej 2024-05-24 08:54:55 -07:00
  • b66eb66ac7 update fine-web preprocessing: faster, and much less RAM consumption Erik Schultheis 2024-05-24 17:21:53 +03:00
  • a241a00ad4 Merge branch 'karpathy:master' into dataloader_win_fixes rosslwheeler 2024-05-23 23:12:15 -07:00
  • dee4e42548 add option to not run hellaswag, interferes with a bunch of testing, e.g. if T is low Andrej Karpathy 2024-05-24 03:47:07 +00:00
  • 032e76c259 start putting llm.c and pytorch right next to each other, identical training runs with identical results and prints. almost Andrej Karpathy 2024-05-24 01:02:21 +00:00
  • 4b88d2a976 Merge pull request #453 from otabuzzman/master Andrej 2024-05-23 14:26:28 -07:00
  • 64b6a14679 add learning rate decay schedule, now we have the full scheduler implemented Andrej Karpathy 2024-05-23 21:19:00 +00:00
  • 661975cc3c add learning rate warmup option Andrej Karpathy 2024-05-23 20:57:04 +00:00
  • 645869b6f7 add weight decay -c option and be more careful in our tests of correctness, our weight decay didn't match to pytorch. also modify the betas in AdamW to be consistent with those used in GPT-3 training Andrej Karpathy 2024-05-23 20:45:59 +00:00
  • 949d71a3d2 only rank 0 logs Andrej Karpathy 2024-05-23 20:32:56 +00:00
  • 3cb2812774 skip hellaswag eval on step 0 i think... not sure but ok for now Andrej Karpathy 2024-05-23 19:27:04 +00:00
  • 8af1818cb1 Merge branch 'karpathy:master' into master Jürgen Schuck 2024-05-23 21:18:01 +02:00
  • 5f87b13f34 Update documentation with Swift port reference otabuzzman 2024-05-23 21:15:01 +02:00
  • 518805b7f4 Init from random weights, matching PyTorch Andrej 2024-05-23 12:11:05 -07:00
  • 1f91bfc442 fix small bug on eval logging feature/random_init Andrej Karpathy 2024-05-23 18:19:25 +00:00
  • 86682af9a9 llm.c matches pytorch init from scratch exactly now Andrej Karpathy 2024-05-23 17:50:33 +00:00
  • 70a9c75348 use pytorch rand and fix dumb bug lol Andrej Karpathy 2024-05-23 15:16:49 +00:00
  • bc1ebc110a Merge branch 'feature/rand' of https://github.com/azret/llm.c into feature/random_init Andrej Karpathy 2024-05-23 14:59:44 +00:00
  • e6a7d1d3e9 allow the python script to also init from random and save those weights, so it's a good reference for our C implementation Andrej Karpathy 2024-05-23 14:59:25 +00:00
  • d09631807a first draft of random init, crashes with some cuBLAS error, debugging Andrej Karpathy 2024-05-23 14:35:46 +00:00
  • d3f26951ef Add glob() for windows Ross Wheeler 2024-05-23 00:06:18 -07:00
  • d3cf9e211c fix bug free only if initted Andrej Karpathy 2024-05-23 00:49:32 +00:00
  • eda0c2f591 Merge pull request #447 from karpathy/feature/hellaswagc Andrej 2024-05-22 15:50:12 -07:00
  • da59861d38 make hellaswag optional eval yay feature/hellaswagc Andrej Karpathy 2024-05-22 22:42:37 +00:00
  • 67239d9b8f fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready Andrej Karpathy 2024-05-22 22:21:53 +00:00
  • 051f3ca53c first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch Andrej Karpathy 2024-05-22 19:24:20 +00:00
  • 69f12214bf Merge pull request #444 from karpathy/feature/sharded_data Andrej 2024-05-22 06:21:48 -07:00
  • 099d30f814 add a super small crappy glob for windows that only matches a single unique file. this will make CI happy but we can't train on sharded data on windows until this is improved feature/sharded_data Andrej Karpathy 2024-05-22 13:08:18 +00:00
  • 05be4f6d82 readme changes Andrej Karpathy 2024-05-22 12:32:25 +00:00
  • edb0df967a continued changes for sharded dataloader Andrej Karpathy 2024-05-22 11:59:09 +00:00
  • 7cbeefc7f3 added new layernorm backward to /dev/cuda/ ademeure 2024-05-21 23:26:54 +01:00
  • 7d0891f6dd Fully deterministic layernorm (slight perf loss) ademeure 2024-05-21 22:37:09 +01:00
  • 31310282e1 extend dataloader to be sharded Andrej Karpathy 2024-05-21 21:33:24 +00:00