938f8f78bb
more defensive
2024-04-29 20:25:29 +00:00
b4c346ac2a
don't hardcode the EOT token and store it in the Tokenizer file
2024-04-29 20:23:41 +00:00
c24bb88bce
Add back FP16 and disable multi_gpu all reduce when there is only 1 GPU
2024-04-29 21:15:52 +01:00
699c25469a
separate out our Tokenizer
2024-04-29 19:54:18 +00:00
4872c57164
comments
2024-04-29 22:46:33 +03:00
9f07a176e0
simplify to only support cs
2024-04-29 22:43:52 +03:00
66c454874f
avoid warning for cublas_compute_type in BF16 mode
2024-04-29 20:25:07 +01:00
1d4effd5a4
Remove FloatN and associated code for adam & allreduce
2024-04-29 20:21:34 +01:00
896b644cb9
slight simplification
2024-04-29 22:19:12 +03:00
1310e2f77d
Merge pull request #286 from rosslwheeler/Makefile-windows-support
...
Makefile windows support
2024-04-29 12:11:12 -07:00
977c688b9d
Merge pull request #294 from karpathy/ademeure-layernormbf16-3
...
Ademeure layernormbf16 3
2024-04-29 12:10:07 -07:00
0e4506ac30
unify these two
2024-04-29 19:03:49 +00:00
8be7370a55
gpt2_build_from_checkpoint
2024-04-29 19:54:32 +01:00
45252d434c
yet another gelu
2024-04-29 21:53:08 +03:00
ebedadfee8
Fix train_gpt2.py to write all weights as BF16 (+merge fixes)
2024-04-29 19:49:57 +01:00
d23bb1fd13
Merge remote-tracking branch 'karpathy/master' into layernormbf16
2024-04-29 19:41:54 +01:00
e92ec3ffd9
Merge pull request #281 from ChrisDryden/templateCleanup
...
Removed templates from kernels to clarify data types and simplify code
2024-04-29 10:05:29 -07:00
237593f47b
Merge pull request #287 from tairov/fix-zig-link
...
Fix link to zig port in README.md
2024-04-29 09:04:18 -07:00
fcef97d6a1
Adding back PFLAGS
...
Got missed during the merge
Missed the profiler PFLAGS
Move PFLAGS ahead to match original change
2024-04-29 08:53:50 -07:00
8915d5ab0d
Adding the example kernels to CI
2024-04-29 17:02:05 +02:00
49ffd68317
Fix sizeof(T) -> sizeof(D) (was overallocating memory)
2024-04-29 15:47:59 +01:00
d0b943ba04
Fix link to zig port
2024-04-29 11:18:03 +01:00
81b689ed01
Merge branch 'Makefile-windows-support' of https://github.com/rosslwheeler/llm.c into Makefile-windows-support
2024-04-29 03:00:56 -07:00
c9ab8ee123
Updating Makefile with latest changes
...
Add check with Windows OS
2024-04-29 03:00:46 -07:00
4b80126913
Add check with Windows OS
2024-04-29 02:46:31 -07:00
0e38ac7b8c
Refactoring parameter size filling
...
Similar to a new function is defined for parameter size table. This will be useful for reuse with e.g. initializing weights from scratch instead of a checkpoint.
2024-04-29 12:35:21 +03:00
f3b1bb70df
Updating Makefile with latest changes
2024-04-29 02:28:03 -07:00
cf3e6efc79
Changed ordering of type configuration to easily see unchanged values
2024-04-29 02:28:23 +00:00
109c62c6c6
Removed templates from kernels to clarify data types
2024-04-29 02:06:59 +00:00
1e50f3b976
Only enable kernel4 with BF16
2024-04-29 01:55:06 +01:00
82fef03acd
Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)
2024-04-29 01:51:10 +01:00
50acc125f3
Merge branch 'ngc92-split-file'
...
Separates out common error-checking wrapper utils, that are broadly useful across all file
2024-04-28 23:27:08 +00:00
c20497cfc1
add comments pointing to the definition of the utils functions
2024-04-28 23:26:58 +00:00
b1c80e9d7b
Merge branch 'split-file' of https://github.com/ngc92/llm.c into ngc92-split-file
2024-04-28 23:23:00 +00:00
4a3c278f3a
moved checked helper functions into a separate file
2024-04-29 00:00:37 +03:00
18b41b4df6
Merge pull request #279 from Ricardicus/prec-ci
...
Updating the CI to build different precisions
2024-04-28 13:47:48 -07:00
66a92c6791
Merge branch 'ngc92-cleanup'
2024-04-28 20:45:04 +00:00
518565654e
Merge branch 'cleanup' of https://github.com/ngc92/llm.c into ngc92-cleanup
2024-04-28 20:43:37 +00:00
49228b091a
add small comment on -t=0
2024-04-28 20:20:51 +00:00
4c295c7e97
and even more cleanup
2024-04-28 23:19:36 +03:00
ca48791522
as promised, cleanup enabled by padding :)
2024-04-28 23:19:36 +03:00
b522333851
Updating the CI to build different precisions
2024-04-28 22:17:47 +02:00
10aa24e2cc
Merge pull request #269 from ChrisDryden/patch-3
...
Enable multithreading in nvcc
2024-04-28 13:16:47 -07:00
4b6f68a9a9
Merge branch 'lancerts-encoder_forward-float4'
2024-04-28 20:10:48 +00:00
327eef3f22
incorporate faster encoder_forward kernel to fp32 CUDA version
2024-04-28 20:10:33 +00:00
4b6a532b9b
Merge branch 'encoder_forward-float4' of https://github.com/lancerts/llm.c into lancerts-encoder_forward-float4
2024-04-28 19:49:16 +00:00
b7972ff928
make padded vocab fixes in the .c code as well, i missed it in the previous PR, should satisfy the CI now
2024-04-28 19:33:21 +00:00
835060e18d
padded vocab change. touched a lot of code. very stressful and error prone, but i think it is done. had to bump versions on all .bin files, invalidating the previous files. re-run the python training script to re-export the new version files. let's not do too much of things like this in the future lol. actually, fun fact i had a chance to do the padded vocab really really early in the history of llm.c development, and chose not do it, thinking i'll just do it later. i should have done it. such is life, you make mistakes, you accumulate scar tissue, and you learn, and you become better, faster, stronger. this is the mindset one must have to lead a happy and fulfilling life. it's not important that you are perfect at any point in time, it's only important that you keep improving, every day.
2024-04-28 18:47:03 +00:00
d95b8d8320
Merge pull request #265 from karpathy/feature/load_bf16
...
load bf16 directly, and some "quality of life" handling of fp32/fp16/bf16 precisions
2024-04-28 09:24:56 -07:00
9d70d9a1b2
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
2024-04-28 16:17:22 +00:00
Andrej Karpathy