 lancer
|
c81adeb6b2
|
move gelu_backward to backward block
|
2024-05-02 17:36:12 -07:00 |
|
|
Andrej Karpathy
|
937fbd8e36
|
small doc fixes
|
2024-05-02 22:17:22 +00:00 |
|
|
Erik Schultheis
|
6e48501d60
|
bias backward kernel that will use all available threads
|
2024-05-03 01:05:47 +03:00 |
|
|
Andrej Karpathy
|
50714d2254
|
pragma unroll quick fix
|
2024-05-02 21:06:11 +00:00 |
|
|
Andrej Karpathy
|
9da2729c5a
|
update encoder_forward with the latest and greatest Packed128 and incorporate into train_gpt2cu
|
2024-05-02 20:40:05 +00:00 |
|
|
Andrej Karpathy
|
3d1761b370
|
Merge branch 'patch-1' of https://github.com/JaneIllario/llm.c into JaneIllario-patch-1
|
2024-05-02 19:43:14 +00:00 |
|
|
Andrej Karpathy
|
99f51bae1c
|
Merge branch 'gelu_backwards' of https://github.com/JaneIllario/llm.c into JaneIllario-gelu_backwards-3
|
2024-05-02 19:15:06 +00:00 |
|
|
Andrej Karpathy
|
ec73086a83
|
Merge branch 'removing_cg_example' of https://github.com/ChrisDryden/llm.c into ChrisDryden-removing_cg_example
|
2024-05-02 18:14:14 +00:00 |
|
|
Andrej Karpathy
|
2202c9a51b
|
add kernel 4 to docs. have to improve these docs more and document them better
|
2024-05-02 18:09:27 +00:00 |
|
|
“Christopher
|
398cdaf4cd
|
Removed cooperative groups and added packed128 to fused classifier and prepare softmax blockwide
|
2024-05-02 02:33:51 +00:00 |
|
|
“Christopher
|
665d0a4f89
|
Added example of removing cooperative groups
|
2024-05-02 02:09:57 +00:00 |
|
|
Jane Illarionova
|
d6d0d50627
|
fix typo
|
2024-05-02 00:18:55 +00:00 |
|
|
Jane Illarionova
|
d0e5fea2dd
|
update residual_forward to use util functions
|
2024-05-02 00:17:47 +00:00 |
|
|
Jane Illarionova
|
52b4dd02a1
|
update residual forward to allow kernels to use both types
|
2024-05-02 00:17:47 +00:00 |
|
|
Jane Illarionova
|
4ffcf5b90a
|
Update residual_forward.cu
|
2024-05-02 00:17:47 +00:00 |
|
|
Jane Illarionova
|
3de3c53ba9
|
update kernel with util functions
|
2024-05-01 23:58:49 +00:00 |
|
|
Jane Illarionova
|
7746217433
|
update gelu backward rto allow all kernels to use both types
|
2024-05-01 23:36:59 +00:00 |
|
|
Jane Illarionova
|
80072802d7
|
Create gelu_backward.cu
|
2024-05-01 23:36:59 +00:00 |
|
|
Erik Schultheis
|
b42db706b2
|
this kernel will be released later, sorry for the spoiler :)
|
2024-05-01 22:47:24 +03:00 |
|
|
Erik Schultheis
|
91bc72d0c0
|
fixup
|
2024-05-01 22:40:26 +03:00 |
|
|
Erik Schultheis
|
06980b6b2f
|
mixed precision utilities for dev/cuda
|
2024-05-01 22:35:55 +03:00 |
|
|
ademeure
|
1147983b8c
|
Add missing cudnnDestroy() calls
|
2024-05-01 17:19:51 +01:00 |
|
|
ademeure
|
2b540b61da
|
attention_forward.cu ~matches train_gpt2cu & allocate less memory for stats in "att"
|
2024-05-01 16:53:01 +01:00 |
|
|
ademeure
|
be54842622
|
Rename lowp_float to floatX in /dev/cuda/attention_forward.cu
|
2024-05-01 15:43:28 +01:00 |
|
|
ademeure
|
c4ecc04dc3
|
Merge remote-tracking branch 'karpathy/master' into cudnn_try2
|
2024-05-01 15:30:00 +01:00 |
|
|
Andrej Karpathy
|
769c911ab6
|
small tweaks
|
2024-04-30 23:48:06 +00:00 |
|
|
Erik Schultheis
|
603d862830
|
fix up bandwidth calculation
|
2024-04-30 23:52:08 +03:00 |
|
|
Erik Schultheis
|
c3515cfe47
|
fixed potential error and generalized gelu forward
|
2024-04-30 23:45:34 +03:00 |
|
|
Andrej Karpathy
|
7ba1ed8a1a
|
delete packed128 .fp32, use (float) ok
|
2024-04-30 19:39:20 +00:00 |
|
|
Andrej Karpathy
|
9141e04693
|
fixes to the PR, careful with float/floatX etc
|
2024-04-30 19:35:05 +00:00 |
|
|
Andrej Karpathy
|
242981eecf
|
Merge branch 'gelu_forwards_packing' of https://github.com/ChrisDryden/llm.c into ChrisDryden-gelu_forwards_packing
|
2024-04-30 19:06:39 +00:00 |
|
|
Franz Louis Cesista
|
b239b67ae1
|
add modal script
|
2024-05-01 00:15:08 +08:00 |
|
|
ademeure
|
1c516c732d
|
re-commit of my old cudnn forward attention changes
|
2024-04-30 14:54:35 +01:00 |
|
|
Erik Schultheis
|
d7813d281d
|
clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results
|
2024-04-30 12:45:37 +03:00 |
|
|
Christopher Dryden
|
568615fef1
|
Added packing for gelu forwards kernel
|
2024-04-30 03:21:42 +00:00 |
|
|
Andrej Karpathy
|
af2bc47d22
|
remove forceinline. blame @arund42 if this ever does not get inlined by the compiler
|
2024-04-29 23:10:51 +00:00 |
|
|
Andrej Karpathy
|
5e12c66f5a
|
move packed128 to common
|
2024-04-29 22:55:28 +00:00 |
|
|
Andrej Karpathy
|
f43592954f
|
reset back to precious hyperparameters
|
2024-04-29 22:44:25 +00:00 |
|
|
Andrej Karpathy
|
ee125cffc4
|
make it imo clearer that the grid size is one quarter as large as it would otherwise be. because each thread now does 4X work
|
2024-04-29 22:40:26 +00:00 |
|
|
Andrej Karpathy
|
083b210dd4
|
small changes
|
2024-04-29 22:38:03 +00:00 |
|
|
Andrej Karpathy
|
d9506a7a0a
|
minor changes to packed128
|
2024-04-29 22:31:19 +00:00 |
|
|
Andrej Karpathy
|
7eefefb511
|
Merge branch 'yet-another-gelu' of https://github.com/ngc92/llm.c into ngc92-yet-another-gelu
|
2024-04-29 22:19:35 +00:00 |
|
|
Erik Schultheis
|
4872c57164
|
comments
|
2024-04-29 22:46:33 +03:00 |
|
|
Erik Schultheis
|
9f07a176e0
|
simplify to only support cs
|
2024-04-29 22:43:52 +03:00 |
|
|
Erik Schultheis
|
896b644cb9
|
slight simplification
|
2024-04-29 22:19:12 +03:00 |
|
|
Erik Schultheis
|
45252d434c
|
yet another gelu
|
2024-04-29 21:53:08 +03:00 |
|
|
ademeure
|
d23bb1fd13
|
Merge remote-tracking branch 'karpathy/master' into layernormbf16
|
2024-04-29 19:41:54 +01:00 |
|
|
ademeure
|
49ffd68317
|
Fix sizeof(T) -> sizeof(D) (was overallocating memory)
|
2024-04-29 15:47:59 +01:00 |
|
|
ademeure
|
1e50f3b976
|
Only enable kernel4 with BF16
|
2024-04-29 01:55:06 +01:00 |
|
|
ademeure
|
82fef03acd
|
Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)
|
2024-04-29 01:51:10 +01:00 |
|