d6bcc129b9
Improve the efficiency
2024-04-11 12:16:20 -07:00
00010c7188
Improve the efficiency
2024-04-11 12:13:34 -07:00
2c81198101
make the new cooperative groups layernorm kernel the default. shaves off aonly bout 1ms of the total running time though
2024-04-11 18:23:46 +00:00
6f5ec063f0
Merge branch 'ngc92-layer-norm'
2024-04-11 18:22:40 +00:00
00b9fda4a1
ceildiv in layernorm
2024-04-11 18:21:34 +00:00
089cd5f1dd
make sure we also test correctness of mean and rstd and do it all block sizes
2024-04-11 18:12:26 +00:00
f28613e0b9
small edits to comments and variables
2024-04-11 18:09:54 +00:00
2553518c97
Merge branch 'layer-norm' of https://github.com/ngc92/llm.c into ngc92-layer-norm
2024-04-11 18:07:00 +00:00
4e91362fa2
optionally generate mean and rstd
2024-04-11 20:48:14 +03:00
f524a90de3
Include the online softmax CPU code (from the paper "Online normalizer calculation for softmax") and native port to GPU kernel.
...
Both the CPU and GPU kernels' correctness are verified by executing the script.
Performance: the online kernel (native CPU port) is faster than the kernel1 (native CPU port) but worse than kernel 2-4.
2024-04-11 09:47:50 -07:00
79b8af4a9b
Merge pull request #56 from scotthaleen/omp-osx-intel
...
Detect OpenMP support - macOS Intel
2024-04-11 09:13:10 -07:00
21a250d372
Merge pull request #64 from krrishnarraj/master
...
[train_gpt2.py] synchronize based on device
2024-04-11 09:12:07 -07:00
1751af1607
inline docs for sizes
2024-04-11 16:04:14 +00:00
bcd0dad6c8
Merge branch 'master' of github.com:karpathy/llm.c
2024-04-11 15:50:10 +00:00
f26cf00a61
make gelu constant be a #define, speeds up the kernel by 1% or so
2024-04-11 15:49:01 +00:00
f46d5b2e2a
Merge pull request #72 from Soldy/master
...
-O3 cannot go with -Ofast
2024-04-11 08:38:57 -07:00
3aaed922a3
-O3 cannot go with -Ofast
...
Signed-off-by: soldy <4786022+soldy@users.noreply.github.com >
2024-04-11 16:27:59 +01:00
4542f8913a
[train_gpt2.py] synchronize based on device
2024-04-11 17:06:51 +05:30
47b2f9312c
fixed homebrew path for intel based macs
2024-04-10 23:01:07 -04:00
afddb1af5b
fixed block size and improved load op hint
2024-04-11 03:47:38 +03:00
466723bb06
fully fused layer-norm kernel
2024-04-11 02:01:41 +03:00
a08c11b60e
Merge pull request #48 from lancerts/fix_matmul
...
fix error in small typos in matmul_forward.cu
2024-04-10 13:44:22 -07:00
68d2ce0601
fix error in small typos in matmul_forward.cu
2024-04-10 13:35:20 -07:00
ef7a4867b1
Merge branch 'master' of github.com:karpathy/llm.c
2024-04-10 19:25:13 +00:00
e1e0a08035
change fabs to fabsf
2024-04-10 19:23:18 +00:00
919b2200a0
Merge pull request #20 from varunlakkur/master
...
Fix a typo
2024-04-10 12:21:07 -07:00
9f5069d3b2
Merge pull request #38 from lancerts/fix_matmul
...
fix the consistency of the transpose notation in matmul_foward.cu
2024-04-10 12:15:53 -07:00
c0d57b1c45
Merge pull request #47 from eltociear/patch-1
...
update layernorm.md
2024-04-10 12:14:56 -07:00
add9e77daa
mention compilation issue that seems to be common. we may want to resolve this inside the Makefile though
2024-04-10 19:11:44 +00:00
fd8459d858
small gitignore for now. probably iterate on
2024-04-10 18:54:23 +00:00
15fad9bc96
Create LICENSE MIT
2024-04-10 11:47:40 -07:00
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
ca137b02d5
update layernorm.md
...
burried -> buried
2024-04-11 00:07:22 +09:00
d8e2a36f15
ok crossentropy is fairly ez, just have to be careful with logs of negative numbers in debug tests. so this should be all the layers now
2024-04-10 01:13:20 +00:00
46ee0a3944
sorry there should be no mallocs inside these functions, have to pass in buffer memory from outside. getting tired
2024-04-10 00:55:14 +00:00
919ae1f27b
ok attention is now optimized too, by 20X, nice. actually this one was really painful, i had to make sure all the cuBLAS calls take all the right arguments and deal with the fact that cuBLAS is column major instead of row major. i go for nap now
2024-04-10 00:50:30 +00:00
fbd679adec
fix the consistency of the transpose notation
2024-04-09 15:19:55 -07:00
8386e5393c
intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice
2024-04-09 20:08:11 +00:00
c29d70a9b3
add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions
2024-04-09 19:22:47 +00:00
327705d03f
mention ongoing cuda work in main readme
2024-04-09 18:29:24 +00:00
ea1fad0337
Merge pull request #33 from lancerts/fix_M_PI_undefined
...
fix a potential error: identifier M_PI is undefined in the gelu kernal
2024-04-09 11:25:23 -07:00
e3e0d73fbc
Merge pull request #34 from VinciGit00/patch-1
...
Free the memory in layernorm.c
2024-04-09 11:24:32 -07:00
bb4f04bdf8
add flash attention, but a naive implementation there of
2024-04-09 18:22:49 +00:00
8397fb1512
Free the memory in layernorm.c
2024-04-09 20:14:18 +02:00
3a4d783633
amend
2024-04-09 09:59:03 -07:00
73c92912b0
fix error: identifier M_PI is undefined
2024-04-09 09:51:28 -07:00
03f37cfb6b
naive attention kernel only parallelizing over batch,time,heads. have to speed this up a lot
2024-04-09 14:53:18 +00:00
41d5f56278
might as well push the few kernels that i feel ok about so far
2024-04-09 14:34:13 +00:00
4d98c44904
Merge pull request #23 from DominguesAddem1974/fix/torch-warning
...
fix: torch warning of python demo
2024-04-09 05:45:18 -07:00
daf1dd81b2
fix: torch warning of python demo
2024-04-09 18:17:04 +08:00
lancer