 Andrej
|
1443edc746
|
Merge branch 'master' into separate_tokenizer
|
2024-04-29 13:38:58 -07:00 |
|
|
Andrej Karpathy
|
70f777617d
|
Single call to NCCL AllReduce and AdamW for bf16
|
2024-04-29 20:36:29 +00:00 |
|
|
Andrej Karpathy
|
dd60fc3991
|
fix prints
|
2024-04-29 20:36:07 +00:00 |
|
|
Andrej Karpathy
|
be4aeb3d2d
|
fix accumulated loss
|
2024-04-29 20:32:10 +00:00 |
|
|
Andrej Karpathy
|
938f8f78bb
|
more defensive
|
2024-04-29 20:25:29 +00:00 |
|
|
Andrej Karpathy
|
b4c346ac2a
|
don't hardcode the EOT token and store it in the Tokenizer file
|
2024-04-29 20:23:41 +00:00 |
|
|
ademeure
|
c24bb88bce
|
Add back FP16 and disable multi_gpu all reduce when there is only 1 GPU
|
2024-04-29 21:15:52 +01:00 |
|
|
Andrej Karpathy
|
699c25469a
|
separate out our Tokenizer
|
2024-04-29 19:54:18 +00:00 |
|
|
ademeure
|
66c454874f
|
avoid warning for cublas_compute_type in BF16 mode
|
2024-04-29 20:25:07 +01:00 |
|
|
ademeure
|
1d4effd5a4
|
Remove FloatN and associated code for adam & allreduce
|
2024-04-29 20:21:34 +01:00 |
|
|
Andrej
|
1310e2f77d
|
Merge pull request #286 from rosslwheeler/Makefile-windows-support
Makefile windows support
|
2024-04-29 12:11:12 -07:00 |
|
|
Andrej
|
977c688b9d
|
Merge pull request #294 from karpathy/ademeure-layernormbf16-3
Ademeure layernormbf16 3
|
2024-04-29 12:10:07 -07:00 |
|
|
Andrej Karpathy
|
0e4506ac30
|
unify these two
|
2024-04-29 19:03:49 +00:00 |
|
|
ademeure
|
8be7370a55
|
gpt2_build_from_checkpoint
|
2024-04-29 19:54:32 +01:00 |
|
|
ademeure
|
ebedadfee8
|
Fix train_gpt2.py to write all weights as BF16 (+merge fixes)
|
2024-04-29 19:49:57 +01:00 |
|
|
ademeure
|
d23bb1fd13
|
Merge remote-tracking branch 'karpathy/master' into layernormbf16
|
2024-04-29 19:41:54 +01:00 |
|
|
Andrej
|
e92ec3ffd9
|
Merge pull request #281 from ChrisDryden/templateCleanup
Removed templates from kernels to clarify data types and simplify code
|
2024-04-29 10:05:29 -07:00 |
|
|
Andrej
|
237593f47b
|
Merge pull request #287 from tairov/fix-zig-link
Fix link to zig port in README.md
|
2024-04-29 09:04:18 -07:00 |
|
|
Ross Wheeler
|
fcef97d6a1
|
Adding back PFLAGS
Got missed during the merge
Missed the profiler PFLAGS
Move PFLAGS ahead to match original change
|
2024-04-29 08:53:50 -07:00 |
|
|
ademeure
|
49ffd68317
|
Fix sizeof(T) -> sizeof(D) (was overallocating memory)
|
2024-04-29 15:47:59 +01:00 |
|
|
Aydyn Tairov
|
d0b943ba04
|
Fix link to zig port
|
2024-04-29 11:18:03 +01:00 |
|
|
Ross Wheeler
|
81b689ed01
|
Merge branch 'Makefile-windows-support' of https://github.com/rosslwheeler/llm.c into Makefile-windows-support
|
2024-04-29 03:00:56 -07:00 |
|
|
Ross Wheeler
|
c9ab8ee123
|
Updating Makefile with latest changes
Add check with Windows OS
|
2024-04-29 03:00:46 -07:00 |
|
|
Ross Wheeler
|
4b80126913
|
Add check with Windows OS
|
2024-04-29 02:46:31 -07:00 |
|
|
Ross Wheeler
|
f3b1bb70df
|
Updating Makefile with latest changes
|
2024-04-29 02:28:03 -07:00 |
|
|
Christopher Dryden
|
109c62c6c6
|
Removed templates from kernels to clarify data types
|
2024-04-29 02:06:59 +00:00 |
|
|
ademeure
|
1e50f3b976
|
Only enable kernel4 with BF16
|
2024-04-29 01:55:06 +01:00 |
|
|
ademeure
|
82fef03acd
|
Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)
|
2024-04-29 01:51:10 +01:00 |
|
|
Andrej Karpathy
|
50acc125f3
|
Merge branch 'ngc92-split-file'
Separates out common error-checking wrapper utils, that are broadly useful across all file
|
2024-04-28 23:27:08 +00:00 |
|
|
Andrej Karpathy
|
c20497cfc1
|
add comments pointing to the definition of the utils functions
|
2024-04-28 23:26:58 +00:00 |
|
|
Andrej Karpathy
|
b1c80e9d7b
|
Merge branch 'split-file' of https://github.com/ngc92/llm.c into ngc92-split-file
|
2024-04-28 23:23:00 +00:00 |
|
|
Erik Schultheis
|
4a3c278f3a
|
moved checked helper functions into a separate file
|
2024-04-29 00:00:37 +03:00 |
|
|
Andrej
|
18b41b4df6
|
Merge pull request #279 from Ricardicus/prec-ci
Updating the CI to build different precisions
|
2024-04-28 13:47:48 -07:00 |
|
|
Andrej Karpathy
|
66a92c6791
|
Merge branch 'ngc92-cleanup'
|
2024-04-28 20:45:04 +00:00 |
|
|
Andrej Karpathy
|
518565654e
|
Merge branch 'cleanup' of https://github.com/ngc92/llm.c into ngc92-cleanup
|
2024-04-28 20:43:37 +00:00 |
|
|
Andrej Karpathy
|
49228b091a
|
add small comment on -t=0
|
2024-04-28 20:20:51 +00:00 |
|
|
Erik Schultheis
|
4c295c7e97
|
and even more cleanup
|
2024-04-28 23:19:36 +03:00 |
|
|
Erik Schultheis
|
ca48791522
|
as promised, cleanup enabled by padding :)
|
2024-04-28 23:19:36 +03:00 |
|
|
Rickard Hallerbäck
|
b522333851
|
Updating the CI to build different precisions
|
2024-04-28 22:17:47 +02:00 |
|
|
Andrej
|
10aa24e2cc
|
Merge pull request #269 from ChrisDryden/patch-3
Enable multithreading in nvcc
|
2024-04-28 13:16:47 -07:00 |
|
|
Andrej Karpathy
|
4b6f68a9a9
|
Merge branch 'lancerts-encoder_forward-float4'
|
2024-04-28 20:10:48 +00:00 |
|
|
Andrej Karpathy
|
327eef3f22
|
incorporate faster encoder_forward kernel to fp32 CUDA version
|
2024-04-28 20:10:33 +00:00 |
|
|
Andrej Karpathy
|
4b6a532b9b
|
Merge branch 'encoder_forward-float4' of https://github.com/lancerts/llm.c into lancerts-encoder_forward-float4
|
2024-04-28 19:49:16 +00:00 |
|
|
Andrej Karpathy
|
b7972ff928
|
make padded vocab fixes in the .c code as well, i missed it in the previous PR, should satisfy the CI now
|
2024-04-28 19:33:21 +00:00 |
|
|
Andrej Karpathy
|
835060e18d
|
padded vocab change. touched a lot of code. very stressful and error prone, but i think it is done. had to bump versions on all .bin files, invalidating the previous files. re-run the python training script to re-export the new version files. let's not do too much of things like this in the future lol. actually, fun fact i had a chance to do the padded vocab really really early in the history of llm.c development, and chose not do it, thinking i'll just do it later. i should have done it. such is life, you make mistakes, you accumulate scar tissue, and you learn, and you become better, faster, stronger. this is the mindset one must have to lead a happy and fulfilling life. it's not important that you are perfect at any point in time, it's only important that you keep improving, every day.
|
2024-04-28 18:47:03 +00:00 |
|
|
Andrej
|
d95b8d8320
|
Merge pull request #265 from karpathy/feature/load_bf16
load bf16 directly, and some "quality of life" handling of fp32/fp16/bf16 precisions
|
2024-04-28 09:24:56 -07:00 |
|
|
Andrej Karpathy
|
9d70d9a1b2
|
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
|
2024-04-28 16:17:22 +00:00 |
|
|
Andrej Karpathy
|
a3f5ad9095
|
reshuffle the ifdefs to make bf16 the default if no PRECISION is requested via defines
|
2024-04-28 16:12:39 +00:00 |
|
|
Andrej Karpathy
|
4f7d8d9f44
|
allow user to make different precisions, add prints and error handling around precisions
|
2024-04-28 16:08:27 +00:00 |
|
|
lancer
|
1c7d23ab3c
|
amend the float4 kernel
|
2024-04-27 20:26:19 -07:00 |
|