 Erik Schultheis
|
8ffc3e388b
|
slow attention kernel that demonstrates a direct translation of the CPU code
|
2024-04-20 19:03:34 +03:00 |
|
|
FeSens
|
0c09a37568
|
fix(attention_forward.cu): Fix the number of elements to check when verifying attention implementations.
This was causing me problems when I was trying to debug a new implementation on fewer tokens.
|
2024-04-18 01:23:05 -03:00 |
|
|
Andrej Karpathy
|
c5ee371564
|
make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common
|
2024-04-13 17:56:18 +00:00 |
|
|
Andrej Karpathy
|
69f0385237
|
add comments and handling of kernels and their correctness
|
2024-04-13 17:44:31 +00:00 |
|
|
Erik Schultheis
|
d035725d22
|
mark problematic tests
|
2024-04-13 18:09:28 +03:00 |
|
|
Erik Schultheis
|
59bada94d9
|
improved testing
|
2024-04-13 17:06:52 +03:00 |
|
|
Erik Schultheis
|
7e7460cfbd
|
const'ness fixes
|
2024-04-13 13:38:11 +03:00 |
|
|
Erik Schultheis
|
7e8e038de8
|
extracted common testing functions into their own header
|
2024-04-13 13:00:05 +03:00 |
|
|
Andrej Karpathy
|
6b49ed1c0b
|
incorporate the fastest kernels and adjust readme
|
2024-04-13 02:58:23 +00:00 |
|
|
Andrej Karpathy
|
6f9e28a1b6
|
few comments
|
2024-04-13 02:38:57 +00:00 |
|
|
Andrej Karpathy
|
2f99b3e608
|
Merge branch 'attention-softmax' of https://github.com/ngc92/llm.c into ngc92-attention-softmax-2
|
2024-04-13 02:34:06 +00:00 |
|
|
Andrej Karpathy
|
59b9e8b96a
|
fix handles in cublas in attention forward as well
|
2024-04-12 23:15:47 +00:00 |
|
|
Erik Schultheis
|
e048825f5a
|
div_ceil
|
2024-04-13 01:20:09 +03:00 |
|
|
Erik Schultheis
|
8e87aac341
|
online softmax
|
2024-04-12 12:47:48 +03:00 |
|
|
Erik Schultheis
|
de98a9a44a
|
vector loads
|
2024-04-12 12:47:43 +03:00 |
|
|
Erik Schultheis
|
baba3cfa01
|
recalculate instead of memory round-trip
|
2024-04-12 02:20:20 +03:00 |
|
|
Erik Schultheis
|
5f99fb30d2
|
cooperative groups and fused scale kernel
|
2024-04-11 23:06:46 +03:00 |
|
|
Andrej Karpathy
|
46ee0a3944
|
sorry there should be no mallocs inside these functions, have to pass in buffer memory from outside. getting tired
|
2024-04-10 00:55:14 +00:00 |
|
|
Andrej Karpathy
|
919ae1f27b
|
ok attention is now optimized too, by 20X, nice. actually this one was really painful, i had to make sure all the cuBLAS calls take all the right arguments and deal with the fact that cuBLAS is column major instead of row major. i go for nap now
|
2024-04-10 00:50:30 +00:00 |
|
|
Andrej Karpathy
|
bb4f04bdf8
|
add flash attention, but a naive implementation there of
|
2024-04-09 18:22:49 +00:00 |
|
|
Andrej Karpathy
|
03f37cfb6b
|
naive attention kernel only parallelizing over batch,time,heads. have to speed this up a lot
|
2024-04-09 14:53:18 +00:00 |
|