 Andrej Karpathy
|
af2bc47d22
|
remove forceinline. blame @arund42 if this ever does not get inlined by the compiler
|
2024-04-29 23:10:51 +00:00 |
|
|
Andrej Karpathy
|
7634f0863c
|
add packed128 to the mainline file
|
2024-04-29 22:57:59 +00:00 |
|
|
Andrej Karpathy
|
5e12c66f5a
|
move packed128 to common
|
2024-04-29 22:55:28 +00:00 |
|
|
Andrej Karpathy
|
f43592954f
|
reset back to precious hyperparameters
|
2024-04-29 22:44:25 +00:00 |
|
|
Andrej Karpathy
|
ee125cffc4
|
make it imo clearer that the grid size is one quarter as large as it would otherwise be. because each thread now does 4X work
|
2024-04-29 22:40:26 +00:00 |
|
|
Andrej Karpathy
|
083b210dd4
|
small changes
|
2024-04-29 22:38:03 +00:00 |
|
|
Andrej Karpathy
|
d9506a7a0a
|
minor changes to packed128
|
2024-04-29 22:31:19 +00:00 |
|
|
Andrej Karpathy
|
7eefefb511
|
Merge branch 'yet-another-gelu' of https://github.com/ngc92/llm.c into ngc92-yet-another-gelu
|
2024-04-29 22:19:35 +00:00 |
|
|
Andrej Karpathy
|
39ce9be516
|
fix missing \n on a print
|
2024-04-29 21:05:44 +00:00 |
|
|
Andrej Karpathy
|
f66825faba
|
allow using flash attention via argparse and print tok/s
|
2024-04-29 21:05:30 +00:00 |
|
|
Andrej
|
0d6e096b9b
|
Merge pull request #296 from karpathy/separate_tokenizer
separate out our Tokenizer
|
2024-04-29 13:44:05 -07:00 |
|
|
Andrej
|
1443edc746
|
Merge branch 'master' into separate_tokenizer
|
2024-04-29 13:38:58 -07:00 |
|
|
Andrej Karpathy
|
70f777617d
|
Single call to NCCL AllReduce and AdamW for bf16
|
2024-04-29 20:36:29 +00:00 |
|
|
Andrej Karpathy
|
dd60fc3991
|
fix prints
|
2024-04-29 20:36:07 +00:00 |
|
|
Andrej Karpathy
|
be4aeb3d2d
|
fix accumulated loss
|
2024-04-29 20:32:10 +00:00 |
|
|
Andrej Karpathy
|
938f8f78bb
|
more defensive
|
2024-04-29 20:25:29 +00:00 |
|
|
Andrej Karpathy
|
b4c346ac2a
|
don't hardcode the EOT token and store it in the Tokenizer file
|
2024-04-29 20:23:41 +00:00 |
|
|
ademeure
|
c24bb88bce
|
Add back FP16 and disable multi_gpu all reduce when there is only 1 GPU
|
2024-04-29 21:15:52 +01:00 |
|
|
Andrej Karpathy
|
699c25469a
|
separate out our Tokenizer
|
2024-04-29 19:54:18 +00:00 |
|
|
Erik Schultheis
|
4872c57164
|
comments
|
2024-04-29 22:46:33 +03:00 |
|
|
Erik Schultheis
|
9f07a176e0
|
simplify to only support cs
|
2024-04-29 22:43:52 +03:00 |
|
|
ademeure
|
66c454874f
|
avoid warning for cublas_compute_type in BF16 mode
|
2024-04-29 20:25:07 +01:00 |
|
|
ademeure
|
1d4effd5a4
|
Remove FloatN and associated code for adam & allreduce
|
2024-04-29 20:21:34 +01:00 |
|
|
Erik Schultheis
|
896b644cb9
|
slight simplification
|
2024-04-29 22:19:12 +03:00 |
|
|
Andrej
|
1310e2f77d
|
Merge pull request #286 from rosslwheeler/Makefile-windows-support
Makefile windows support
|
2024-04-29 12:11:12 -07:00 |
|
|
Andrej
|
977c688b9d
|
Merge pull request #294 from karpathy/ademeure-layernormbf16-3
Ademeure layernormbf16 3
|
2024-04-29 12:10:07 -07:00 |
|
|
Andrej Karpathy
|
0e4506ac30
|
unify these two
|
2024-04-29 19:03:49 +00:00 |
|
|
ademeure
|
8be7370a55
|
gpt2_build_from_checkpoint
|
2024-04-29 19:54:32 +01:00 |
|
|
Erik Schultheis
|
45252d434c
|
yet another gelu
|
2024-04-29 21:53:08 +03:00 |
|
|
ademeure
|
ebedadfee8
|
Fix train_gpt2.py to write all weights as BF16 (+merge fixes)
|
2024-04-29 19:49:57 +01:00 |
|
|
ademeure
|
d23bb1fd13
|
Merge remote-tracking branch 'karpathy/master' into layernormbf16
|
2024-04-29 19:41:54 +01:00 |
|
|
Andrej
|
e92ec3ffd9
|
Merge pull request #281 from ChrisDryden/templateCleanup
Removed templates from kernels to clarify data types and simplify code
|
2024-04-29 10:05:29 -07:00 |
|
|
Andrej
|
237593f47b
|
Merge pull request #287 from tairov/fix-zig-link
Fix link to zig port in README.md
|
2024-04-29 09:04:18 -07:00 |
|
|
Ross Wheeler
|
fcef97d6a1
|
Adding back PFLAGS
Got missed during the merge
Missed the profiler PFLAGS
Move PFLAGS ahead to match original change
|
2024-04-29 08:53:50 -07:00 |
|
|
ademeure
|
49ffd68317
|
Fix sizeof(T) -> sizeof(D) (was overallocating memory)
|
2024-04-29 15:47:59 +01:00 |
|
|
Aydyn Tairov
|
d0b943ba04
|
Fix link to zig port
|
2024-04-29 11:18:03 +01:00 |
|
|
Ross Wheeler
|
81b689ed01
|
Merge branch 'Makefile-windows-support' of https://github.com/rosslwheeler/llm.c into Makefile-windows-support
|
2024-04-29 03:00:56 -07:00 |
|
|
Ross Wheeler
|
c9ab8ee123
|
Updating Makefile with latest changes
Add check with Windows OS
|
2024-04-29 03:00:46 -07:00 |
|
|
Ross Wheeler
|
4b80126913
|
Add check with Windows OS
|
2024-04-29 02:46:31 -07:00 |
|
|
Ross Wheeler
|
f3b1bb70df
|
Updating Makefile with latest changes
|
2024-04-29 02:28:03 -07:00 |
|
|
Christopher Dryden
|
109c62c6c6
|
Removed templates from kernels to clarify data types
|
2024-04-29 02:06:59 +00:00 |
|
|
ademeure
|
1e50f3b976
|
Only enable kernel4 with BF16
|
2024-04-29 01:55:06 +01:00 |
|
|
ademeure
|
82fef03acd
|
Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)
|
2024-04-29 01:51:10 +01:00 |
|
|
Andrej Karpathy
|
50acc125f3
|
Merge branch 'ngc92-split-file'
Separates out common error-checking wrapper utils, that are broadly useful across all file
|
2024-04-28 23:27:08 +00:00 |
|
|
Andrej Karpathy
|
c20497cfc1
|
add comments pointing to the definition of the utils functions
|
2024-04-28 23:26:58 +00:00 |
|
|
Andrej Karpathy
|
b1c80e9d7b
|
Merge branch 'split-file' of https://github.com/ngc92/llm.c into ngc92-split-file
|
2024-04-28 23:23:00 +00:00 |
|
|
Erik Schultheis
|
4a3c278f3a
|
moved checked helper functions into a separate file
|
2024-04-29 00:00:37 +03:00 |
|
|
Andrej
|
18b41b4df6
|
Merge pull request #279 from Ricardicus/prec-ci
Updating the CI to build different precisions
|
2024-04-28 13:47:48 -07:00 |
|
|
Andrej Karpathy
|
66a92c6791
|
Merge branch 'ngc92-cleanup'
|
2024-04-28 20:45:04 +00:00 |
|
|
Andrej Karpathy
|
518565654e
|
Merge branch 'cleanup' of https://github.com/ngc92/llm.c into ngc92-cleanup
|
2024-04-28 20:43:37 +00:00 |
|