43c48707a7
Merge branch 'fuse-ln-res-2' of https://github.com/ngc92/llm.c into ngc92-fuse-ln-res-2
2024-05-09 23:09:53 +00:00
75ec629f52
remove duplicate function
2024-05-10 00:03:30 +02:00
5920143d47
remove warning noise
2024-05-10 00:02:37 +02:00
691c1df969
fused layernorm+residual
2024-05-09 23:48:11 +02:00
99e765de8d
moved bf16 boilerplate to common.h
2024-05-09 11:19:24 +02:00
69aa64cc80
Adds cuDNN install instructions for a smaller but sufficient package
...
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).
- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)
- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions
Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com >
2024-05-08 14:53:31 -07:00
6d7a99cb4e
Adjust comments
2024-05-08 17:45:14 +08:00
6a52d8619e
Optimize codes and comments
2024-05-08 16:12:39 +08:00
b3e8a9fe67
Implementation of online softmax forward kernel without cgs.
2024-05-08 09:22:14 +08:00
e1f89b3043
remove BF16 default from classified_fused before PR
2024-05-07 00:57:01 +01:00
1ea7f9bf25
tiny irrelevant optimisation to final unaligned fused_classifier loop + add missing common.h changes
2024-05-07 00:54:50 +01:00
9b55ea8cf5
More crazy optimisations to layernorm_backward, fused_classifier, and matmul_backward_bias.
2024-05-07 00:11:42 +01:00
bbfe8c989c
Minor update on the code
2024-05-05 16:59:58 -07:00
c81adeb6b2
move gelu_backward to backward block
2024-05-02 17:36:12 -07:00
937fbd8e36
small doc fixes
2024-05-02 22:17:22 +00:00
6e48501d60
bias backward kernel that will use all available threads
2024-05-03 01:05:47 +03:00
50714d2254
pragma unroll quick fix
2024-05-02 21:06:11 +00:00
9da2729c5a
update encoder_forward with the latest and greatest Packed128 and incorporate into train_gpt2cu
2024-05-02 20:40:05 +00:00
3d1761b370
Merge branch 'patch-1' of https://github.com/JaneIllario/llm.c into JaneIllario-patch-1
2024-05-02 19:43:14 +00:00
99f51bae1c
Merge branch 'gelu_backwards' of https://github.com/JaneIllario/llm.c into JaneIllario-gelu_backwards-3
2024-05-02 19:15:06 +00:00
ec73086a83
Merge branch 'removing_cg_example' of https://github.com/ChrisDryden/llm.c into ChrisDryden-removing_cg_example
2024-05-02 18:14:14 +00:00
2202c9a51b
add kernel 4 to docs. have to improve these docs more and document them better
2024-05-02 18:09:27 +00:00
398cdaf4cd
Removed cooperative groups and added packed128 to fused classifier and prepare softmax blockwide
2024-05-02 02:33:51 +00:00
665d0a4f89
Added example of removing cooperative groups
2024-05-02 02:09:57 +00:00
d6d0d50627
fix typo
2024-05-02 00:18:55 +00:00
d0e5fea2dd
update residual_forward to use util functions
2024-05-02 00:17:47 +00:00
52b4dd02a1
update residual forward to allow kernels to use both types
2024-05-02 00:17:47 +00:00
4ffcf5b90a
Update residual_forward.cu
2024-05-02 00:17:47 +00:00
3de3c53ba9
update kernel with util functions
2024-05-01 23:58:49 +00:00
7746217433
update gelu backward rto allow all kernels to use both types
2024-05-01 23:36:59 +00:00
80072802d7
Create gelu_backward.cu
2024-05-01 23:36:59 +00:00
b42db706b2
this kernel will be released later, sorry for the spoiler :)
2024-05-01 22:47:24 +03:00
91bc72d0c0
fixup
2024-05-01 22:40:26 +03:00
06980b6b2f
mixed precision utilities for dev/cuda
2024-05-01 22:35:55 +03:00
1147983b8c
Add missing cudnnDestroy() calls
2024-05-01 17:19:51 +01:00
2b540b61da
attention_forward.cu ~matches train_gpt2cu & allocate less memory for stats in "att"
2024-05-01 16:53:01 +01:00
be54842622
Rename lowp_float to floatX in /dev/cuda/attention_forward.cu
2024-05-01 15:43:28 +01:00
c4ecc04dc3
Merge remote-tracking branch 'karpathy/master' into cudnn_try2
2024-05-01 15:30:00 +01:00
769c911ab6
small tweaks
2024-04-30 23:48:06 +00:00
603d862830
fix up bandwidth calculation
2024-04-30 23:52:08 +03:00
c3515cfe47
fixed potential error and generalized gelu forward
2024-04-30 23:45:34 +03:00
7ba1ed8a1a
delete packed128 .fp32, use (float) ok
2024-04-30 19:39:20 +00:00
9141e04693
fixes to the PR, careful with float/floatX etc
2024-04-30 19:35:05 +00:00
242981eecf
Merge branch 'gelu_forwards_packing' of https://github.com/ChrisDryden/llm.c into ChrisDryden-gelu_forwards_packing
2024-04-30 19:06:39 +00:00
b239b67ae1
add modal script
2024-05-01 00:15:08 +08:00
1c516c732d
re-commit of my old cudnn forward attention changes
2024-04-30 14:54:35 +01:00
d7813d281d
clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results
2024-04-30 12:45:37 +03:00
568615fef1
Added packing for gelu forwards kernel
2024-04-30 03:21:42 +00:00
af2bc47d22
remove forceinline. blame @arund42 if this ever does not get inlined by the compiler
2024-04-29 23:10:51 +00:00
5e12c66f5a
move packed128 to common
2024-04-29 22:55:28 +00:00
Andrej Karpathy