4ee98f5135
Merge branch 'master' into optimised_fused_classifier
2024-04-17 22:58:51 +01:00
c327923df9
Constness fix layernorm_backward.cu
2024-04-17 14:56:03 -07:00
81aa596251
Constness fix layernorm_backward.cu
2024-04-17 14:54:49 -07:00
c1bfe4a639
warp-wide reduction for layernorm backward
2024-04-17 21:52:40 +03:00
fa4ebfa663
kernel2 for softmax backward
2024-04-17 08:14:22 +00:00
d74d7ae652
small adamw fixes, needs more work
2024-04-17 04:58:16 +00:00
66a0b5242e
Merge branch 'master' of github.com:karpathy/llm.c
2024-04-17 04:51:26 +00:00
8a673edcc3
push my attention_backward file
2024-04-17 04:42:42 +00:00
97e61e2258
add slightly more optimized adamw kernel
2024-04-17 03:31:07 +02:00
5684ff7e77
add adamw cuda kernel
2024-04-17 03:30:59 +02:00
43bf9eb66a
ensure valid test cases and incorporate fixes by @ademeure
2024-04-16 20:53:50 +03:00
b225501b7b
optimised version of the cool new fused classifier + hopefully bugfixes?
2024-04-16 14:54:23 +01:00
6ff4ad646d
fix the backward dbias precision issue
2024-04-15 19:58:16 -07:00
6be8080501
add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far
2024-04-16 02:10:45 +00:00
3de507739c
minor fixes to matmul backward
2024-04-16 01:23:41 +00:00
9560b1a92b
add matmul backward kernel, problem is the bias backward, ironically. which i thought i could cublas at first
2024-04-16 00:40:11 +00:00
3c6dafeb53
Merge pull request #117 from ngc92/fused-classifier
...
WIP: Fully fused classification layer
2024-04-15 12:13:48 -07:00
c73fa8842a
fixed numerics for online kernel, and an existing race condition in kernel 2
2024-04-15 03:41:09 +03:00
8fb966bc31
kernel 1 should also accumulate in double to ensure stability
2024-04-15 02:52:38 +03:00
60cf281e46
use V dimension in softmax tests
...
adjusted tests to be more challenging
2024-04-15 02:14:48 +03:00
99b42c1a94
nicer printing
2024-04-15 02:03:48 +03:00
fe1d8cfb47
fix timing output
2024-04-15 02:03:42 +03:00
780d8bd012
initial version
2024-04-14 03:48:56 +03:00
534b93a732
erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7
2024-04-13 22:02:31 +00:00
c5ee371564
make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common
2024-04-13 17:56:18 +00:00
82623518a4
fix comment block of gelu forward kernel
2024-04-13 17:48:53 +00:00
02b5b33c5f
remove unused variable C
2024-04-13 17:48:11 +00:00
84145411d3
fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nans
2024-04-13 17:47:19 +00:00
69f0385237
add comments and handling of kernels and their correctness
2024-04-13 17:44:31 +00:00
f459bfa833
fixed re-initialization of device memory after test
2024-04-13 18:23:12 +03:00
d035725d22
mark problematic tests
2024-04-13 18:09:28 +03:00
59bada94d9
improved testing
2024-04-13 17:06:52 +03:00
7e7460cfbd
const'ness fixes
2024-04-13 13:38:11 +03:00
7e8e038de8
extracted common testing functions into their own header
2024-04-13 13:00:05 +03:00
6b49ed1c0b
incorporate the fastest kernels and adjust readme
2024-04-13 02:58:23 +00:00
6f9e28a1b6
few comments
2024-04-13 02:38:57 +00:00
2f99b3e608
Merge branch 'attention-softmax' of https://github.com/ngc92/llm.c into ngc92-attention-softmax-2
2024-04-13 02:34:06 +00:00
40b78be722
small changes to comments, and very minor code modifications and movements
2024-04-13 02:11:36 +00:00
6250336c09
Merge branch 'online_softmax' of https://github.com/lancerts/llm.c into lancerts-online_softmax
2024-04-13 01:54:16 +00:00
a18e517c1a
alright let's start the backward pass. crossentropy_softmax_backward first, ez one
2024-04-13 01:52:55 +00:00
59b9e8b96a
fix handles in cublas in attention forward as well
2024-04-12 23:15:47 +00:00
8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it ( #98 )
...
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
2024-04-12 15:32:07 -07:00
e048825f5a
div_ceil
2024-04-13 01:20:09 +03:00
8e87aac341
online softmax
2024-04-12 12:47:48 +03:00
de98a9a44a
vector loads
2024-04-12 12:47:43 +03:00
baba3cfa01
recalculate instead of memory round-trip
2024-04-12 02:20:20 +03:00
69d0e5bac7
Include a parallel kernal of softmax_online
2024-04-11 15:30:36 -07:00
ca1e5ee04d
Include a parallel kernal of softmax_online
2024-04-11 15:24:41 -07:00
5f99fb30d2
cooperative groups and fused scale kernel
2024-04-11 23:06:46 +03:00
d6bcc129b9
Improve the efficiency
2024-04-11 12:16:20 -07:00
ademeure