7e8e038de8
extracted common testing functions into their own header
2024-04-13 13:00:05 +03:00
6b49ed1c0b
incorporate the fastest kernels and adjust readme
2024-04-13 02:58:23 +00:00
6f9e28a1b6
few comments
2024-04-13 02:38:57 +00:00
2f99b3e608
Merge branch 'attention-softmax' of https://github.com/ngc92/llm.c into ngc92-attention-softmax-2
2024-04-13 02:34:06 +00:00
40b78be722
small changes to comments, and very minor code modifications and movements
2024-04-13 02:11:36 +00:00
6250336c09
Merge branch 'online_softmax' of https://github.com/lancerts/llm.c into lancerts-online_softmax
2024-04-13 01:54:16 +00:00
a18e517c1a
alright let's start the backward pass. crossentropy_softmax_backward first, ez one
2024-04-13 01:52:55 +00:00
59b9e8b96a
fix handles in cublas in attention forward as well
2024-04-12 23:15:47 +00:00
8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it ( #98 )
...
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
2024-04-12 15:32:07 -07:00
e048825f5a
div_ceil
2024-04-13 01:20:09 +03:00
8e87aac341
online softmax
2024-04-12 12:47:48 +03:00
de98a9a44a
vector loads
2024-04-12 12:47:43 +03:00
baba3cfa01
recalculate instead of memory round-trip
2024-04-12 02:20:20 +03:00
69d0e5bac7
Include a parallel kernal of softmax_online
2024-04-11 15:30:36 -07:00
ca1e5ee04d
Include a parallel kernal of softmax_online
2024-04-11 15:24:41 -07:00
5f99fb30d2
cooperative groups and fused scale kernel
2024-04-11 23:06:46 +03:00
d6bcc129b9
Improve the efficiency
2024-04-11 12:16:20 -07:00
00010c7188
Improve the efficiency
2024-04-11 12:13:34 -07:00
00b9fda4a1
ceildiv in layernorm
2024-04-11 18:21:34 +00:00
089cd5f1dd
make sure we also test correctness of mean and rstd and do it all block sizes
2024-04-11 18:12:26 +00:00
f28613e0b9
small edits to comments and variables
2024-04-11 18:09:54 +00:00
2553518c97
Merge branch 'layer-norm' of https://github.com/ngc92/llm.c into ngc92-layer-norm
2024-04-11 18:07:00 +00:00
4e91362fa2
optionally generate mean and rstd
2024-04-11 20:48:14 +03:00
f524a90de3
Include the online softmax CPU code (from the paper "Online normalizer calculation for softmax") and native port to GPU kernel.
...
Both the CPU and GPU kernels' correctness are verified by executing the script.
Performance: the online kernel (native CPU port) is faster than the kernel1 (native CPU port) but worse than kernel 2-4.
2024-04-11 09:47:50 -07:00
f26cf00a61
make gelu constant be a #define, speeds up the kernel by 1% or so
2024-04-11 15:49:01 +00:00
afddb1af5b
fixed block size and improved load op hint
2024-04-11 03:47:38 +03:00
466723bb06
fully fused layer-norm kernel
2024-04-11 02:01:41 +03:00
68d2ce0601
fix error in small typos in matmul_forward.cu
2024-04-10 13:35:20 -07:00
9f5069d3b2
Merge pull request #38 from lancerts/fix_matmul
...
fix the consistency of the transpose notation in matmul_foward.cu
2024-04-10 12:15:53 -07:00
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
d8e2a36f15
ok crossentropy is fairly ez, just have to be careful with logs of negative numbers in debug tests. so this should be all the layers now
2024-04-10 01:13:20 +00:00
46ee0a3944
sorry there should be no mallocs inside these functions, have to pass in buffer memory from outside. getting tired
2024-04-10 00:55:14 +00:00
919ae1f27b
ok attention is now optimized too, by 20X, nice. actually this one was really painful, i had to make sure all the cuBLAS calls take all the right arguments and deal with the fact that cuBLAS is column major instead of row major. i go for nap now
2024-04-10 00:50:30 +00:00
fbd679adec
fix the consistency of the transpose notation
2024-04-09 15:19:55 -07:00
8386e5393c
intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice
2024-04-09 20:08:11 +00:00
c29d70a9b3
add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions
2024-04-09 19:22:47 +00:00
ea1fad0337
Merge pull request #33 from lancerts/fix_M_PI_undefined
...
fix a potential error: identifier M_PI is undefined in the gelu kernal
2024-04-09 11:25:23 -07:00
bb4f04bdf8
add flash attention, but a naive implementation there of
2024-04-09 18:22:49 +00:00
3a4d783633
amend
2024-04-09 09:59:03 -07:00
73c92912b0
fix error: identifier M_PI is undefined
2024-04-09 09:51:28 -07:00
03f37cfb6b
naive attention kernel only parallelizing over batch,time,heads. have to speed this up a lot
2024-04-09 14:53:18 +00:00
41d5f56278
might as well push the few kernels that i feel ok about so far
2024-04-09 14:34:13 +00:00
Erik Schultheis