 Andrej Karpathy
|
44cd50cb25
|
fix all the handles and things
|
2024-04-12 22:18:18 +00:00 |
|
|
Andrej Karpathy
|
8cf66fbb84
|
change workspace from 4mib to 32mib so we don't have to detect hopper. seems ok for now
|
2024-04-12 20:41:08 +00:00 |
|
|
Andrej Karpathy
|
bfd11ce5b6
|
use cublaslt and optionally tf32, which fuses bias and deprecates the need for add_bias kernel
|
2024-04-12 19:38:36 +00:00 |
|
|
Andrej Karpathy
|
00b9fda4a1
|
ceildiv in layernorm
|
2024-04-11 18:21:34 +00:00 |
|
|
Andrej Karpathy
|
089cd5f1dd
|
make sure we also test correctness of mean and rstd and do it all block sizes
|
2024-04-11 18:12:26 +00:00 |
|
|
Andrej Karpathy
|
f28613e0b9
|
small edits to comments and variables
|
2024-04-11 18:09:54 +00:00 |
|
|
Andrej Karpathy
|
2553518c97
|
Merge branch 'layer-norm' of https://github.com/ngc92/llm.c into ngc92-layer-norm
|
2024-04-11 18:07:00 +00:00 |
|
|
Erik Schultheis
|
4e91362fa2
|
optionally generate mean and rstd
|
2024-04-11 20:48:14 +03:00 |
|
|
Andrej Karpathy
|
f26cf00a61
|
make gelu constant be a #define, speeds up the kernel by 1% or so
|
2024-04-11 15:49:01 +00:00 |
|
|
Erik Schultheis
|
afddb1af5b
|
fixed block size and improved load op hint
|
2024-04-11 03:47:38 +03:00 |
|
|
Erik Schultheis
|
466723bb06
|
fully fused layer-norm kernel
|
2024-04-11 02:01:41 +03:00 |
|
|
lancer
|
68d2ce0601
|
fix error in small typos in matmul_forward.cu
|
2024-04-10 13:35:20 -07:00 |
|
|
Andrej
|
9f5069d3b2
|
Merge pull request #38 from lancerts/fix_matmul
fix the consistency of the transpose notation in matmul_foward.cu
|
2024-04-10 12:15:53 -07:00 |
|
|
Andrej Karpathy
|
80f52e5290
|
the full forward pass of GPT-2 in one file of pure CUDA
|
2024-04-10 18:15:55 +00:00 |
|
|
Andrej Karpathy
|
d8e2a36f15
|
ok crossentropy is fairly ez, just have to be careful with logs of negative numbers in debug tests. so this should be all the layers now
|
2024-04-10 01:13:20 +00:00 |
|
|
Andrej Karpathy
|
46ee0a3944
|
sorry there should be no mallocs inside these functions, have to pass in buffer memory from outside. getting tired
|
2024-04-10 00:55:14 +00:00 |
|
|
Andrej Karpathy
|
919ae1f27b
|
ok attention is now optimized too, by 20X, nice. actually this one was really painful, i had to make sure all the cuBLAS calls take all the right arguments and deal with the fact that cuBLAS is column major instead of row major. i go for nap now
|
2024-04-10 00:50:30 +00:00 |
|
|
lancer
|
fbd679adec
|
fix the consistency of the transpose notation
|
2024-04-09 15:19:55 -07:00 |
|
|
Andrej Karpathy
|
8386e5393c
|
intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice
|
2024-04-09 20:08:11 +00:00 |
|
|
Andrej Karpathy
|
c29d70a9b3
|
add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions
|
2024-04-09 19:22:47 +00:00 |
|
|
Andrej
|
ea1fad0337
|
Merge pull request #33 from lancerts/fix_M_PI_undefined
fix a potential error: identifier M_PI is undefined in the gelu kernal
|
2024-04-09 11:25:23 -07:00 |
|
|
Andrej Karpathy
|
bb4f04bdf8
|
add flash attention, but a naive implementation there of
|
2024-04-09 18:22:49 +00:00 |
|
|
lancer
|
3a4d783633
|
amend
|
2024-04-09 09:59:03 -07:00 |
|
|
lancer
|
73c92912b0
|
fix error: identifier M_PI is undefined
|
2024-04-09 09:51:28 -07:00 |
|
|
Andrej Karpathy
|
03f37cfb6b
|
naive attention kernel only parallelizing over batch,time,heads. have to speed this up a lot
|
2024-04-09 14:53:18 +00:00 |
|
|
Andrej Karpathy
|
41d5f56278
|
might as well push the few kernels that i feel ok about so far
|
2024-04-09 14:34:13 +00:00 |
|