 Andrej Karpathy
|
7b92930021
|
move things around a bit, a few comments, and also update python script to correspond to C more
|
2024-07-01 17:20:46 +00:00 |
|
|
Andrej Karpathy
|
8253823432
|
Merge branch 'fix_periodic_spikes' of https://github.com/gordicaleksa/llm.c into gordicaleksa-fix_periodic_spikes-2
|
2024-07-01 17:14:22 +00:00 |
|
|
Andrej
|
942fed59fe
|
Merge pull request #662 from karpathy/overfit-batch-actually
fix the overfit single batch behavior to actually overfit batch, not …
|
2024-07-01 10:12:20 -07:00 |
|
|
Andrej Karpathy
|
7fd7717816
|
fix the overfit single batch behavior to actually overfit batch, not microbatch, we do this more cleanly by simply resetting the dataloader every step
|
2024-07-01 17:00:40 +00:00 |
|
|
Aleksa Gordic
|
74e5ba7260
|
Add micro step guard
|
2024-07-01 12:11:02 +02:00 |
|
|
Aleksa Gordic
|
78e7627859
|
Remove local args for testing
|
2024-07-01 11:31:55 +02:00 |
|
|
Aleksa Gordic
|
de8454d2f4
|
Make the grad cleaning more robust
|
2024-07-01 11:30:36 +02:00 |
|
|
Aleksa Gordic
|
d2f06aed01
|
Fix periodic spikes - clean loss after val
|
2024-07-01 11:24:16 +02:00 |
|
|
Andrej Karpathy
|
61625d168c
|
only rank 0 creates log dir
|
2024-07-01 06:31:37 +00:00 |
|
|
Andrej Karpathy
|
ea36d04e39
|
small cosmetic fixes to fineweb data file
|
2024-06-30 15:52:51 +00:00 |
|
|
Andrej
|
2daaf6ab83
|
Merge pull request #629 from rosslwheeler/feature/tests
CI Dataloader test and ptx/sass file generator
|
2024-06-29 15:17:31 -07:00 |
|
|
Andrej Karpathy
|
e6e365a1a8
|
Merge branch 'eliebak-add_edu_fineweb_support'
|
2024-06-28 19:35:58 +00:00 |
|
|
Andrej Karpathy
|
ff7a90bd8d
|
Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_support
|
2024-06-28 19:35:44 +00:00 |
|
|
Andrej Karpathy
|
2cf7c5b7d3
|
Merge branch 'ngc92-mixed-dtypes-2'
|
2024-06-27 17:47:22 +00:00 |
|
|
Andrej Karpathy
|
012915c06c
|
handle unknown datatype to squash compiler warning
|
2024-06-27 17:46:46 +00:00 |
|
|
Andrej
|
06cd83eacc
|
Merge pull request #652 from ademeure/cudnn_determinism
Make cuDNN deterministic for Flash Attention backward
|
2024-06-27 10:22:04 -07:00 |
|
|
ademeure
|
29a067f916
|
Make cuDNN deterministic for Flash Attention backward (requires frontend >=1.5)
|
2024-06-27 13:33:02 +00:00 |
|
|
Ross Wheeler
|
d6d08be3c0
|
Addressed review comments
Explicit precision set for tests
|
2024-06-27 00:58:19 -07:00 |
|
|
Ross Wheeler
|
eb0b74e559
|
Add H100 and A100 ptx/sass files
|
2024-06-27 00:30:38 -07:00 |
|
|
rosslwheeler
|
267efc9a0e
|
Merge branch 'karpathy:master' into feature/tests
|
2024-06-26 22:38:21 -07:00 |
|
|
ngc92
|
98875dd022
|
macro-based alternative
|
2024-06-27 02:31:02 +02:00 |
|
|
ngc92
|
d38c614eb9
|
add some comments
|
2024-06-26 23:58:31 +03:00 |
|
|
Erik Schultheis
|
7beaa1ed96
|
keep layernorm stats in fp32
|
2024-06-26 20:48:11 +03:00 |
|
|
Erik Schultheis
|
f0c3e35de8
|
switch losses to fp32
|
2024-06-26 20:48:08 +03:00 |
|
|
Erik Schultheis
|
ca47165d29
|
allocation function for mixed dtype activation
|
2024-06-26 20:48:03 +03:00 |
|
|
Andrej
|
68535cb6ce
|
Merge pull request #649 from karpathy/feature/openmpi_flexible_path
tune openmpi make a bit more convenient
|
2024-06-26 09:11:22 -07:00 |
|
|
Andrej Karpathy
|
7c283df9c4
|
use ?= for OPENMPI_DIR so it can be overriden
|
2024-06-26 16:06:54 +00:00 |
|
|
Andrej
|
c1ac121c77
|
Merge pull request #646 from Ricardicus/makefile-fix
Makefile: Respect the NO_MULTI_GPU
|
2024-06-26 08:42:59 -07:00 |
|
|
Andrej
|
a08e32e474
|
Merge pull request #648 from krrishnarraj/mention_fork
[readme] mention opencl fork
|
2024-06-26 08:41:15 -07:00 |
|
|
Andrej Karpathy
|
2a8f82f14f
|
tune openmpi make a bit more convenient
|
2024-06-26 15:39:10 +00:00 |
|
|
Krishnaraj Bhat
|
0aa0eb09a6
|
[readme] mention opencl fork
|
2024-06-26 20:32:21 +05:30 |
|
|
Rickard Hallerbäck
|
66cd8c3f1f
|
Makefile: respect the NO_MULTI_GPU
|
2024-06-26 14:55:02 +02:00 |
|
|
Andrej
|
44fde98344
|
Merge pull request #637 from karpathy/feature/outliers
add outlier detector, test for it, and start tracking z score of loss
|
2024-06-25 21:16:29 -07:00 |
|
|
Andrej Karpathy
|
e12d090483
|
resolve merge conflicts
|
2024-06-26 04:05:46 +00:00 |
|
|
Andrej Karpathy
|
ac018c3302
|
Merge branch 'ngc92-more-streams'
|
2024-06-25 23:41:55 +00:00 |
|
|
Andrej Karpathy
|
9c7f1f9701
|
minor comments
|
2024-06-25 23:41:40 +00:00 |
|
|
Andrej Karpathy
|
4f84e3daa7
|
Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams
|
2024-06-25 23:25:57 +00:00 |
|
|
Erik Schultheis
|
7171f78f04
|
fix check call
|
2024-06-26 02:21:44 +03:00 |
|
|
Andrej Karpathy
|
c0e84eb859
|
Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams
|
2024-06-25 23:16:50 +00:00 |
|
|
Erik Schultheis
|
4aae1e529b
|
fix up profile
|
2024-06-26 02:11:55 +03:00 |
|
|
Erik Schultheis
|
6c24f13b55
|
check token validity
|
2024-06-26 02:01:24 +03:00 |
|
|
Erik Schultheis
|
7b13f20343
|
moved targets to backward/validate
|
2024-06-26 01:56:23 +03:00 |
|
|
Andrej Karpathy
|
496d8bf449
|
Merge branch 'more-streams' of https://github.com/ngc92/llm.c into ngc92-more-streams
|
2024-06-25 22:44:00 +00:00 |
|
|
Erik Schultheis
|
04e1c183ec
|
set mean_loss to -1 if not calculated
|
2024-06-26 01:18:19 +03:00 |
|
|
Andrej
|
16b5bd53fb
|
Merge pull request #636 from karpathy/feature/rolling_checkpoints
rolling checkpoints
|
2024-06-25 14:43:21 -07:00 |
|
|
Andrej Karpathy
|
11e2fd1cf2
|
separate out write_checkpoint and delete_checkpoint
|
2024-06-25 20:46:03 +00:00 |
|
|
Erik Schultheis
|
7018280448
|
fix zero-1
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
09503dea4f
|
make windows happy
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
2413b0a6d0
|
fix profile
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
39f976f013
|
added missing free
|
2024-06-25 21:28:52 +03:00 |
|