mirrors / karpathy--llm.c

提交

提交图

  • 7b13f20343 moved targets to backward/validate Erik Schultheis 2024-06-26 01:56:23 +03:00
  • 496d8bf449 Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams Andrej Karpathy 2024-06-25 22:44:00 +00:00
  • 04e1c183ec set mean_loss to -1 if not calculated Erik Schultheis 2024-06-26 01:18:19 +03:00
  • 16b5bd53fb Merge pull request #636 from karpathy/feature/rolling_checkpoints Andrej 2024-06-25 14:43:21 -07:00
  • 11e2fd1cf2 separate out write_checkpoint and delete_checkpoint feature/rolling_checkpoints Andrej Karpathy 2024-06-25 20:46:03 +00:00
  • 7018280448 fix zero-1 Erik Schultheis 2024-06-25 02:05:17 +03:00
  • 09503dea4f make windows happy Erik Schultheis 2024-06-25 01:43:40 +03:00
  • 2413b0a6d0 fix profile Erik Schultheis 2024-06-25 01:35:15 +03:00
  • 39f976f013 added missing free Erik Schultheis 2024-06-25 00:31:08 +03:00
  • 8b36c2f60b fix loss accumulation Erik Schultheis 2024-06-25 00:25:43 +03:00
  • 75d4610442 update norm computation: multi-process reduce on GPU Erik Schultheis 2024-06-24 23:57:42 +03:00
  • 8d994782cf moved loss calculation to backward Erik Schultheis 2024-06-24 23:49:35 +03:00
  • 4745f53589 boolean_constant utilities Erik Schultheis 2024-06-24 23:48:39 +03:00
  • 1f49cc1c03 Merge pull request #639 from rosslwheeler/bug_fixes/windows_clean_up Andrej 2024-06-25 11:23:31 -07:00
  • be7a278351 Merge pull request #640 from ahrefs/multigpu-fs-build-fix Andrej 2024-06-25 11:22:51 -07:00
  • 53985a77f4 CI for Windows needs cuDNN installed - revert Ross Wheeler 2024-06-25 10:43:06 -07:00
  • 2e6fd9b150 Add unistd.h for Windows cuDNN build Ross Wheeler 2024-06-25 10:36:16 -07:00
  • 934e6f22c5 Fix missing compiler flag chinthysl 2024-06-25 09:35:24 +00:00
  • 2e32e25e37 Merge branch 'karpathy:master' into feature/tests rosslwheeler 2024-06-24 23:05:55 -07:00
  • b51fe0124d oops forgot to update defaults comment in the argparse, smallfix Andrej Karpathy 2024-06-25 05:07:42 +00:00
  • 665ff144fb skip update if lossz or gradz are above thresholds determined by new args -sl and -sg Andrej Karpathy 2024-06-25 04:50:38 +00:00
  • 329aee333e fix training loop in other parts Andrej Karpathy 2024-06-25 04:30:34 +00:00
  • 59830367f2 refactor grad norm and add tracking of grad norm outliers as well Andrej Karpathy 2024-06-25 03:38:21 +00:00
  • 7a0d864f37 oops i was only using window size 16 for testing let's make it 128 or something Andrej Karpathy 2024-06-25 01:56:04 +00:00
  • 569f9f16f9 add outlier detector, test for it, and start tracking z score of the loss, for now only print it Andrej Karpathy 2024-06-25 01:37:31 +00:00
  • 8952f0235b rolling checkpoints - checkpoints are either MINOR or MAJOR and minor checkpoints get deleted with a rolling window. This is an optimization that will allow us to save state more often, but preserve disk space overall. And if we ever see loss spikes and such, it is easier to reset the state to an earlier checkpoint that is more recent Andrej Karpathy 2024-06-24 23:39:18 +00:00
  • 3a986bc81b Merge pull request #619 from gordicaleksa/cast_2d_noise Andrej 2024-06-24 14:43:20 -07:00
  • 76b133d3f4 Merge pull request #631 from gordicaleksa/minor_refactor2 Andrej 2024-06-24 14:38:27 -07:00
  • 69b50adcf5 Merge pull request #632 from gordicaleksa/multi_node_my Andrej 2024-06-24 14:36:30 -07:00
  • 4af6a6ab3d Remove /tmp as default init value for fs_path Aleksa Gordic 2024-06-24 21:26:55 +00:00
  • 2667aa1e3e Fix windows include issue Aleksa Gordic 2024-06-24 19:12:50 +00:00
  • ac1e37b659 Remove linux socket funcs when windows Aleksa Gordic 2024-06-24 18:56:48 +00:00
  • 60c95c82aa Add a multi node readme section Aleksa Gordic 2024-06-24 18:50:56 +00:00
  • f9d0e8b67a Add socket logic for Windows Aleksa Gordic 2024-06-24 18:26:53 +00:00
  • 05ef1e7dc0 Fix makefile Aleksa Gordic 2024-06-24 14:53:58 +00:00
  • c420a522b2 Fix arg permutation Aleksa Gordic 2024-06-24 14:38:49 +00:00
  • fbf4f59c8f Add MPI under multi gpu guard Aleksa Gordic 2024-06-24 14:25:01 +00:00
  • a259eeabe5 Fix CI errors Aleksa Gordic 2024-06-24 14:09:42 +00:00
  • 3abdf9a3e4 Add gpus per node arg Aleksa Gordic 2024-06-24 12:07:35 +00:00
  • 75a2c8b10f Add usage for new args Aleksa Gordic 2024-06-24 11:37:18 +00:00
  • bfdea887eb Add launch scripts - tested Aleksa Gordic 2024-06-24 11:28:53 +00:00
  • 1bc3a5ef6e Add all 3 init methods Aleksa Gordic 2024-06-23 21:42:05 +00:00
  • 1c6c5ff0b7 TCP working Aleksa Gordic 2024-06-23 20:30:09 +00:00
  • 5500b61da5 fs working Aleksa Gordic 2024-06-23 16:29:54 +00:00
  • 08c055cd72 Free up unified buffer mem Aleksa Gordic 2024-06-23 10:46:54 +00:00
  • 1bd737546c Mini refactor Aleksa Gordic 2024-06-23 10:42:19 +00:00
  • ac72875c24 Use NCCL stream Aleksa Gordic 2024-06-23 10:39:55 +00:00
  • 52db158367 Can not call set device after NCCL init Aleksa Gordic 2024-06-22 20:22:06 +00:00
  • 0addb503ef Use NCCL where possible Aleksa Gordic 2024-06-22 17:16:35 +02:00
  • 2a4be7f349 Merge branch 'ngc92-streams-io' Andrej Karpathy 2024-06-23 01:14:50 +00:00
  • 98e928f1d8 minor fixes for disk device io Andrej Karpathy 2024-06-23 01:14:42 +00:00
  • 2543b62b88 Merge branch 'streams-io' of https://github.com/ngc92/llm.c into ngc92-streams-io Andrej Karpathy 2024-06-23 00:48:05 +00:00
  • 71f10e7960 Rename gpt2 backward Aleksa Gordic 2024-06-22 14:32:51 +02:00
  • 5b95bff118 Dataloader test and ptx/sass file generator Ross Wheeler 2024-06-21 17:15:52 -07:00
  • 72a2158812 remove a spurious print that is making the table a bit uglier Andrej Karpathy 2024-06-21 15:08:30 +00:00
  • c6e21c78b5 Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-06-21 02:40:19 +00:00
  • 61ed938dfb delete spurious check it's fine Andrej Karpathy 2024-06-21 02:40:11 +00:00
  • dbe06082ca delete spurious check its fine feature/lr_schedulers Andrej Karpathy 2024-06-21 02:39:40 +00:00
  • 5b82822462 Merge pull request #627 from karpathy/feature/lr_schedulers Andrej 2024-06-20 19:31:42 -07:00
  • 4b5fecc8b2 small touchups and add the wsd schedule Andrej Karpathy 2024-06-21 00:49:01 +00:00
  • 96ebb16298 simplify lr schedulers Andrej Karpathy 2024-06-21 00:29:47 +00:00
  • 9abd9ec8a8 Merge branch 'add_lr_schedules' of https://github.com/gordicaleksa/llm.c into gordicaleksa-add_lr_schedules Andrej Karpathy 2024-06-21 00:12:15 +00:00
  • dad58b6800 Merge pull request #625 from gordicaleksa/add_nccl_to_readme Andrej 2024-06-20 16:06:31 -07:00
  • 2677d82dcb add shell script that downloads the .bin files of FineWeb100B directly Andrej Karpathy 2024-06-20 22:39:09 +00:00
  • 5b4d11ac1d Add NCCL instruction Aleksa Gordic 2024-06-21 00:16:17 +02:00
  • f19202c858 small fixes on top of @chinthysl PR to kick off the feature branch feature/nccl-only Andrej Karpathy 2024-06-20 20:13:27 +00:00
  • 10e4dac2db Merge branch 'multinode-slurm' of https://github.com/ahrefs/llm.c into ahrefs-multinode-slurm Andrej Karpathy 2024-06-20 19:11:14 +00:00
  • 716808d9b9 Use C not C++ string header Aleksa Gordic 2024-06-20 16:09:28 +02:00
  • 3e9fb6cb36 Fix compile issues - add comments Aleksa Gordic 2024-06-19 20:51:27 +02:00
  • 7416ee425f Simplify logic since they all share similar args Aleksa Gordic 2024-06-19 20:39:17 +02:00
  • 9d141ddab9 Add ugly switching between schedulers Aleksa Gordic 2024-06-19 18:36:45 +02:00
  • 7308c2831c Add linear LR scheduler Aleksa Gordic 2024-06-17 17:27:24 +02:00
  • 6f3d099e88 Add constant lr scheduler Aleksa Gordic 2024-06-17 17:16:59 +02:00
  • 4a1c5a9c0f Add cyclic triangular LR scheduler Aleksa Gordic 2024-06-17 17:11:27 +02:00
  • 2eaea3b397 Add schedulers header Aleksa Gordic 2024-06-17 16:40:07 +02:00
  • 8249736233 Add learning rate schedulers Aleksa Gordic 2024-06-17 16:36:17 +02:00
  • bb9eab8736 Change positionX to uint Aleksa Gordic 2024-06-19 21:05:28 +02:00
  • 2ddf128365 Merge pull request #615 from gordicaleksa/relax_grad_tests Andrej 2024-06-19 09:37:42 -07:00
  • 66950f3790 Cast computation to uint Aleksa Gordic 2024-06-19 17:45:34 +02:00
  • 8079bd9dbb Relax grad tensor thresholds a bit Aleksa Gordic 2024-06-19 10:21:59 +02:00
  • fd564f7b8a add the gpt2 1558M repro script too Andrej Karpathy 2024-06-19 02:26:39 +00:00
  • 0bc8753bc0 update vislog to be a bit more fancy, has GPT-2/3 perf for hellaswag Andrej Karpathy 2024-06-19 01:51:09 +00:00
  • 3bce68b495 Merge branch 'ChrisDryden-script_to_download_tokenized_dataset' Andrej Karpathy 2024-06-19 01:45:08 +00:00
  • b29478e20b add starter pack .sh script for faster quickstart Andrej Karpathy 2024-06-19 01:44:36 +00:00
  • 48b7885f72 Merge branch 'script_to_download_tokenized_dataset' of https://github.com/ChrisDryden/llm.c into ChrisDryden-script_to_download_tokenized_dataset Andrej Karpathy 2024-06-19 01:29:54 +00:00
  • 483f675e92 Merge branch 'master' of github.com:karpathy/llm.c Andrej Karpathy 2024-06-19 01:29:23 +00:00
  • 7534c3fdf4 minor simplification inside gpt2_update of copy paste code Andrej Karpathy 2024-06-19 01:29:11 +00:00
  • 4577d58cba Merge pull request #611 from rosslwheeler/feature/dataloader_test Andrej 2024-06-18 18:08:05 -07:00
  • 88f0543c38 Merge pull request #594 from karpathy/feature/hf_eval Andrej 2024-06-18 18:04:14 -07:00
  • 1a66151a80 small fixes and readme feature/hf_eval Andrej Karpathy 2024-06-19 01:03:36 +00:00
  • 24923829c4 Adding suggested changes Ross Wheeler 2024-06-18 10:38:33 -07:00
  • b3b4dabab0 small touchups Erik Schultheis 2024-06-18 20:03:28 +03:00
  • d95313d0a9 explicit sync Erik Schultheis 2024-06-18 20:02:07 +03:00
  • ec4424013a made buffer size more easily configurable Erik Schultheis 2024-06-18 20:00:25 +03:00
  • 3ebd5abe82 fixup tests Erik Schultheis 2024-06-18 19:58:03 +03:00
  • 6ecc52e3fc Merge pull request #614 from gordicaleksa/more_strict_tests Andrej 2024-06-18 09:29:13 -07:00
  • 03cdca1b0d Merge pull request #601 from gordicaleksa/fix_stochastic_rounding_2 Andrej 2024-06-18 09:28:39 -07:00
  • e12b172610 Merge pull request #613 from gordicaleksa/fix_dataloader Andrej 2024-06-18 08:31:01 -07:00
  • e5d51c09e2 Add a comment for why pass a unique seed per param Aleksa Gordic 2024-06-18 17:01:03 +02:00
  • 6b2b3dc33f Pass unique seeds in stochastic rounding encoder bwd Aleksa Gordic 2024-06-16 23:01:24 +02:00