f524a90de3
Include the online softmax CPU code (from the paper "Online normalizer calculation for softmax") and native port to GPU kernel.
...
Both the CPU and GPU kernels' correctness are verified by executing the script.
Performance: the online kernel (native CPU port) is faster than the kernel1 (native CPU port) but worse than kernel 2-4.
2024-04-11 09:47:50 -07:00
f26cf00a61
make gelu constant be a #define, speeds up the kernel by 1% or so
2024-04-11 15:49:01 +00:00
afddb1af5b
fixed block size and improved load op hint
2024-04-11 03:47:38 +03:00
466723bb06
fully fused layer-norm kernel
2024-04-11 02:01:41 +03:00
68d2ce0601
fix error in small typos in matmul_forward.cu
2024-04-10 13:35:20 -07:00
9f5069d3b2
Merge pull request #38 from lancerts/fix_matmul
...
fix the consistency of the transpose notation in matmul_foward.cu
2024-04-10 12:15:53 -07:00
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
d8e2a36f15
ok crossentropy is fairly ez, just have to be careful with logs of negative numbers in debug tests. so this should be all the layers now
2024-04-10 01:13:20 +00:00
46ee0a3944
sorry there should be no mallocs inside these functions, have to pass in buffer memory from outside. getting tired
2024-04-10 00:55:14 +00:00
919ae1f27b
ok attention is now optimized too, by 20X, nice. actually this one was really painful, i had to make sure all the cuBLAS calls take all the right arguments and deal with the fact that cuBLAS is column major instead of row major. i go for nap now
2024-04-10 00:50:30 +00:00
fbd679adec
fix the consistency of the transpose notation
2024-04-09 15:19:55 -07:00
8386e5393c
intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice
2024-04-09 20:08:11 +00:00
c29d70a9b3
add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions
2024-04-09 19:22:47 +00:00
ea1fad0337
Merge pull request #33 from lancerts/fix_M_PI_undefined
...
fix a potential error: identifier M_PI is undefined in the gelu kernal
2024-04-09 11:25:23 -07:00
bb4f04bdf8
add flash attention, but a naive implementation there of
2024-04-09 18:22:49 +00:00
3a4d783633
amend
2024-04-09 09:59:03 -07:00
73c92912b0
fix error: identifier M_PI is undefined
2024-04-09 09:51:28 -07:00
03f37cfb6b
naive attention kernel only parallelizing over batch,time,heads. have to speed this up a lot
2024-04-09 14:53:18 +00:00
41d5f56278
might as well push the few kernels that i feel ok about so far
2024-04-09 14:34:13 +00:00
lancer