 Andrej Karpathy
|
7fd7717816
|
fix the overfit single batch behavior to actually overfit batch, not microbatch, we do this more cleanly by simply resetting the dataloader every step
|
2024-07-01 17:00:40 +00:00 |
|
|
Andrej Karpathy
|
61625d168c
|
only rank 0 creates log dir
|
2024-07-01 06:31:37 +00:00 |
|
|
Andrej Karpathy
|
ea36d04e39
|
small cosmetic fixes to fineweb data file
|
2024-06-30 15:52:51 +00:00 |
|
|
Andrej
|
2daaf6ab83
|
Merge pull request #629 from rosslwheeler/feature/tests
CI Dataloader test and ptx/sass file generator
|
2024-06-29 15:17:31 -07:00 |
|
|
Andrej Karpathy
|
e6e365a1a8
|
Merge branch 'eliebak-add_edu_fineweb_support'
|
2024-06-28 19:35:58 +00:00 |
|
|
Andrej Karpathy
|
ff7a90bd8d
|
Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_support
|
2024-06-28 19:35:44 +00:00 |
|
|
Andrej Karpathy
|
2cf7c5b7d3
|
Merge branch 'ngc92-mixed-dtypes-2'
|
2024-06-27 17:47:22 +00:00 |
|
|
Andrej Karpathy
|
012915c06c
|
handle unknown datatype to squash compiler warning
|
2024-06-27 17:46:46 +00:00 |
|
|
Andrej
|
06cd83eacc
|
Merge pull request #652 from ademeure/cudnn_determinism
Make cuDNN deterministic for Flash Attention backward
|
2024-06-27 10:22:04 -07:00 |
|
|
ademeure
|
29a067f916
|
Make cuDNN deterministic for Flash Attention backward (requires frontend >=1.5)
|
2024-06-27 13:33:02 +00:00 |
|
|
Ross Wheeler
|
d6d08be3c0
|
Addressed review comments
Explicit precision set for tests
|
2024-06-27 00:58:19 -07:00 |
|
|
Ross Wheeler
|
eb0b74e559
|
Add H100 and A100 ptx/sass files
|
2024-06-27 00:30:38 -07:00 |
|
|
rosslwheeler
|
267efc9a0e
|
Merge branch 'karpathy:master' into feature/tests
|
2024-06-26 22:38:21 -07:00 |
|
|
ngc92
|
98875dd022
|
macro-based alternative
|
2024-06-27 02:31:02 +02:00 |
|
|
ngc92
|
d38c614eb9
|
add some comments
|
2024-06-26 23:58:31 +03:00 |
|
|
Erik Schultheis
|
7beaa1ed96
|
keep layernorm stats in fp32
|
2024-06-26 20:48:11 +03:00 |
|
|
Erik Schultheis
|
f0c3e35de8
|
switch losses to fp32
|
2024-06-26 20:48:08 +03:00 |
|
|
Erik Schultheis
|
ca47165d29
|
allocation function for mixed dtype activation
|
2024-06-26 20:48:03 +03:00 |
|
|
Andrej
|
68535cb6ce
|
Merge pull request #649 from karpathy/feature/openmpi_flexible_path
tune openmpi make a bit more convenient
|
2024-06-26 09:11:22 -07:00 |
|
|
Andrej Karpathy
|
7c283df9c4
|
use ?= for OPENMPI_DIR so it can be overriden
|
2024-06-26 16:06:54 +00:00 |
|
|
Andrej
|
c1ac121c77
|
Merge pull request #646 from Ricardicus/makefile-fix
Makefile: Respect the NO_MULTI_GPU
|
2024-06-26 08:42:59 -07:00 |
|
|
Andrej
|
a08e32e474
|
Merge pull request #648 from krrishnarraj/mention_fork
[readme] mention opencl fork
|
2024-06-26 08:41:15 -07:00 |
|
|
Andrej Karpathy
|
2a8f82f14f
|
tune openmpi make a bit more convenient
|
2024-06-26 15:39:10 +00:00 |
|
|
Krishnaraj Bhat
|
0aa0eb09a6
|
[readme] mention opencl fork
|
2024-06-26 20:32:21 +05:30 |
|
|
Rickard Hallerbäck
|
66cd8c3f1f
|
Makefile: respect the NO_MULTI_GPU
|
2024-06-26 14:55:02 +02:00 |
|
|
Andrej
|
44fde98344
|
Merge pull request #637 from karpathy/feature/outliers
add outlier detector, test for it, and start tracking z score of loss
|
2024-06-25 21:16:29 -07:00 |
|
|
Andrej Karpathy
|
e12d090483
|
resolve merge conflicts
|
2024-06-26 04:05:46 +00:00 |
|
|
Andrej Karpathy
|
ac018c3302
|
Merge branch 'ngc92-more-streams'
|
2024-06-25 23:41:55 +00:00 |
|
|
Andrej Karpathy
|
9c7f1f9701
|
minor comments
|
2024-06-25 23:41:40 +00:00 |
|
|
Andrej Karpathy
|
4f84e3daa7
|
Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams
|
2024-06-25 23:25:57 +00:00 |
|
|
Erik Schultheis
|
7171f78f04
|
fix check call
|
2024-06-26 02:21:44 +03:00 |
|
|
Andrej Karpathy
|
c0e84eb859
|
Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams
|
2024-06-25 23:16:50 +00:00 |
|
|
Erik Schultheis
|
4aae1e529b
|
fix up profile
|
2024-06-26 02:11:55 +03:00 |
|
|
Erik Schultheis
|
6c24f13b55
|
check token validity
|
2024-06-26 02:01:24 +03:00 |
|
|
Erik Schultheis
|
7b13f20343
|
moved targets to backward/validate
|
2024-06-26 01:56:23 +03:00 |
|
|
Andrej Karpathy
|
496d8bf449
|
Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams
|
2024-06-25 22:44:00 +00:00 |
|
|
Erik Schultheis
|
04e1c183ec
|
set mean_loss to -1 if not calculated
|
2024-06-26 01:18:19 +03:00 |
|
|
Andrej
|
16b5bd53fb
|
Merge pull request #636 from karpathy/feature/rolling_checkpoints
rolling checkpoints
|
2024-06-25 14:43:21 -07:00 |
|
|
Andrej Karpathy
|
11e2fd1cf2
|
separate out write_checkpoint and delete_checkpoint
|
2024-06-25 20:46:03 +00:00 |
|
|
Erik Schultheis
|
7018280448
|
fix zero-1
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
09503dea4f
|
make windows happy
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
2413b0a6d0
|
fix profile
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
39f976f013
|
added missing free
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
8b36c2f60b
|
fix loss accumulation
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
75d4610442
|
update norm computation: multi-process reduce on GPU
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
8d994782cf
|
moved loss calculation to backward
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
4745f53589
|
boolean_constant utilities
|
2024-06-25 21:28:52 +03:00 |
|
|
Andrej
|
1f49cc1c03
|
Merge pull request #639 from rosslwheeler/bug_fixes/windows_clean_up
Add unistd.h to fix Windows cuDNN build
|
2024-06-25 11:23:31 -07:00 |
|
|
Andrej
|
be7a278351
|
Merge pull request #640 from ahrefs/multigpu-fs-build-fix
Add missing MULTI_GPU compiler flag
|
2024-06-25 11:22:51 -07:00 |
|
|
Ross Wheeler
|
53985a77f4
|
CI for Windows needs cuDNN installed - revert
|
2024-06-25 10:43:06 -07:00 |
|