 lancer
|
5c90845f7a
|
update the -lcublas -lcublasLt flag in the comment
|
2024-05-09 22:04:02 -07:00 |
|
|
Erik Schultheis
|
06980b6b2f
|
mixed precision utilities for dev/cuda
|
2024-05-01 22:35:55 +03:00 |
|
|
Erik Schultheis
|
44b5bb028c
|
one more version of attention backward for another few percent perf gains
|
2024-04-19 22:46:40 +03:00 |
|
|
Erik Schultheis
|
a360f104ae
|
some low-level optimizations
|
2024-04-19 22:38:13 +03:00 |
|
|
Erik Schultheis
|
7005e09450
|
added missing checks
|
2024-04-19 22:37:35 +03:00 |
|
|
Erik Schultheis
|
b2e6b6867f
|
separating the loops leads to yet more improvements
|
2024-04-19 20:47:19 +03:00 |
|
|
Erik Schultheis
|
50e105a250
|
further cleanup and ability to handle arbitrary sequence lengths
|
2024-04-19 19:12:37 +03:00 |
|
|
Erik Schultheis
|
4294820a8a
|
don't do stupid redundant work; the inner loop is just a block-level sum!!!!
|
2024-04-19 19:12:32 +03:00 |
|
|
Erik Schultheis
|
02aa8e47dc
|
further simplification of the loop
|
2024-04-19 19:12:32 +03:00 |
|
|
Erik Schultheis
|
16123c3af8
|
towards an even better backward attention kernel
|
2024-04-19 19:12:30 +03:00 |
|
|
Erik Schultheis
|
09d35a5e4d
|
splitting the loop to get rid of the indicator
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
6d98af1d32
|
another 15% by getting rid of ifs
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
314661f516
|
significantly faster kernel by reusing data in registers
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
58a7b57f96
|
enable benchmarking
|
2024-04-18 01:39:47 +03:00 |
|
|
Erik Schultheis
|
e278112c19
|
coalesced reading in attention backward pass
|
2024-04-18 01:39:43 +03:00 |
|
|
Andrej Karpathy
|
fa4ebfa663
|
kernel2 for softmax backward
|
2024-04-17 08:14:22 +00:00 |
|
|
Andrej Karpathy
|
8a673edcc3
|
push my attention_backward file
|
2024-04-17 04:42:42 +00:00 |
|