7b99ac96a5
Fix mem leaks, reduce memory, refactor
2024-06-06 19:58:50 +02:00
109f516367
Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgs
...
Free cooperative groups implementation of online softmax forward
2024-05-16 21:23:56 +01:00
5c90845f7a
update the -lcublas -lcublasLt flag in the comment
2024-05-09 22:04:02 -07:00
691c1df969
fused layernorm+residual
2024-05-09 23:48:11 +02:00
6d7a99cb4e
Adjust comments
2024-05-08 17:45:14 +08:00
6a52d8619e
Optimize codes and comments
2024-05-08 16:12:39 +08:00
b3e8a9fe67
Implementation of online softmax forward kernel without cgs.
2024-05-08 09:22:14 +08:00
e991fe1efa
fix the 0.0f initialization for double
2024-04-18 19:33:30 -07:00
c73fa8842a
fixed numerics for online kernel, and an existing race condition in kernel 2
2024-04-15 03:41:09 +03:00
8fb966bc31
kernel 1 should also accumulate in double to ensure stability
2024-04-15 02:52:38 +03:00
60cf281e46
use V dimension in softmax tests
...
adjusted tests to be more challenging
2024-04-15 02:14:48 +03:00
99b42c1a94
nicer printing
2024-04-15 02:03:48 +03:00
534b93a732
erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7
2024-04-13 22:02:31 +00:00
59bada94d9
improved testing
2024-04-13 17:06:52 +03:00
7e7460cfbd
const'ness fixes
2024-04-13 13:38:11 +03:00
7e8e038de8
extracted common testing functions into their own header
2024-04-13 13:00:05 +03:00
40b78be722
small changes to comments, and very minor code modifications and movements
2024-04-13 02:11:36 +00:00
69d0e5bac7
Include a parallel kernal of softmax_online
2024-04-11 15:30:36 -07:00
ca1e5ee04d
Include a parallel kernal of softmax_online
2024-04-11 15:24:41 -07:00
d6bcc129b9
Improve the efficiency
2024-04-11 12:16:20 -07:00
00010c7188
Improve the efficiency
2024-04-11 12:13:34 -07:00
f524a90de3
Include the online softmax CPU code (from the paper "Online normalizer calculation for softmax") and native port to GPU kernel.
...
Both the CPU and GPU kernels' correctness are verified by executing the script.
Performance: the online kernel (native CPU port) is faster than the kernel1 (native CPU port) but worse than kernel 2-4.
2024-04-11 09:47:50 -07:00
8386e5393c
intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice
2024-04-09 20:08:11 +00:00
c29d70a9b3
add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions
2024-04-09 19:22:47 +00:00
Aleksa Gordic