 Andrej Karpathy
|
44cd50cb25
|
fix all the handles and things
|
2024-04-12 22:18:18 +00:00 |
|
|
Andrej Karpathy
|
5364e0b853
|
change main train/test code to use new cublaslt functions. inference now causes segfault though, very strange have to fix before merge
|
2024-04-12 21:23:30 +00:00 |
|
|
Andrej Karpathy
|
8cf66fbb84
|
change workspace from 4mib to 32mib so we don't have to detect hopper. seems ok for now
|
2024-04-12 20:41:08 +00:00 |
|
|
Andrej Karpathy
|
bfd11ce5b6
|
use cublaslt and optionally tf32, which fuses bias and deprecates the need for add_bias kernel
|
2024-04-12 19:38:36 +00:00 |
|
|
Andrej Karpathy
|
6396e393e3
|
add links to discussions
|
2024-04-12 01:11:29 +00:00 |
|
|
Andrej Karpathy
|
dec54904b4
|
one more simple clarifying comment
|
2024-04-11 22:24:38 +00:00 |
|
|
Andrej Karpathy
|
d680fbad25
|
one more comment on attention
|
2024-04-11 21:49:14 +00:00 |
|
|
Andrej Karpathy
|
9710163a60
|
did one pass of additional comments, potentially helpful, as i think a lot of people may be looking at this for the first time
|
2024-04-11 21:45:46 +00:00 |
|
|
Andrej Karpathy
|
2c81198101
|
make the new cooperative groups layernorm kernel the default. shaves off aonly bout 1ms of the total running time though
|
2024-04-11 18:23:46 +00:00 |
|
|
Andrej Karpathy
|
6f5ec063f0
|
Merge branch 'ngc92-layer-norm'
|
2024-04-11 18:22:40 +00:00 |
|
|
Andrej Karpathy
|
00b9fda4a1
|
ceildiv in layernorm
|
2024-04-11 18:21:34 +00:00 |
|
|
Andrej Karpathy
|
089cd5f1dd
|
make sure we also test correctness of mean and rstd and do it all block sizes
|
2024-04-11 18:12:26 +00:00 |
|
|
Andrej Karpathy
|
f28613e0b9
|
small edits to comments and variables
|
2024-04-11 18:09:54 +00:00 |
|
|
Andrej Karpathy
|
2553518c97
|
Merge branch 'layer-norm' of https://github.com/ngc92/llm.c into ngc92-layer-norm
|
2024-04-11 18:07:00 +00:00 |
|
|
Erik Schultheis
|
4e91362fa2
|
optionally generate mean and rstd
|
2024-04-11 20:48:14 +03:00 |
|
|
Andrej
|
79b8af4a9b
|
Merge pull request #56 from scotthaleen/omp-osx-intel
Detect OpenMP support - macOS Intel
|
2024-04-11 09:13:10 -07:00 |
|
|
Andrej
|
21a250d372
|
Merge pull request #64 from krrishnarraj/master
[train_gpt2.py] synchronize based on device
|
2024-04-11 09:12:07 -07:00 |
|
|
Andrej Karpathy
|
1751af1607
|
inline docs for sizes
|
2024-04-11 16:04:14 +00:00 |
|
|
Andrej Karpathy
|
bcd0dad6c8
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-11 15:50:10 +00:00 |
|
|
Andrej Karpathy
|
f26cf00a61
|
make gelu constant be a #define, speeds up the kernel by 1% or so
|
2024-04-11 15:49:01 +00:00 |
|
|
Andrej
|
f46d5b2e2a
|
Merge pull request #72 from Soldy/master
-O3 cannot go with -Ofast
|
2024-04-11 08:38:57 -07:00 |
|
|
soldy
|
3aaed922a3
|
-O3 cannot go with -Ofast
Signed-off-by: soldy <4786022+soldy@users.noreply.github.com>
|
2024-04-11 16:27:59 +01:00 |
|
|
Krishnaraj Bhat
|
4542f8913a
|
[train_gpt2.py] synchronize based on device
|
2024-04-11 17:06:51 +05:30 |
|
|
Scott Haleen
|
47b2f9312c
|
fixed homebrew path for intel based macs
|
2024-04-10 23:01:07 -04:00 |
|
|
Erik Schultheis
|
afddb1af5b
|
fixed block size and improved load op hint
|
2024-04-11 03:47:38 +03:00 |
|
|
Erik Schultheis
|
466723bb06
|
fully fused layer-norm kernel
|
2024-04-11 02:01:41 +03:00 |
|
|
Andrej
|
a08c11b60e
|
Merge pull request #48 from lancerts/fix_matmul
fix error in small typos in matmul_forward.cu
|
2024-04-10 13:44:22 -07:00 |
|
|
lancer
|
68d2ce0601
|
fix error in small typos in matmul_forward.cu
|
2024-04-10 13:35:20 -07:00 |
|
|
Andrej Karpathy
|
ef7a4867b1
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-10 19:25:13 +00:00 |
|
|
Andrej Karpathy
|
e1e0a08035
|
change fabs to fabsf
|
2024-04-10 19:23:18 +00:00 |
|
|
Andrej
|
919b2200a0
|
Merge pull request #20 from varunlakkur/master
Fix a typo
|
2024-04-10 12:21:07 -07:00 |
|
|
Andrej
|
9f5069d3b2
|
Merge pull request #38 from lancerts/fix_matmul
fix the consistency of the transpose notation in matmul_foward.cu
|
2024-04-10 12:15:53 -07:00 |
|
|
Andrej
|
c0d57b1c45
|
Merge pull request #47 from eltociear/patch-1
update layernorm.md
|
2024-04-10 12:14:56 -07:00 |
|
|
Andrej Karpathy
|
add9e77daa
|
mention compilation issue that seems to be common. we may want to resolve this inside the Makefile though
|
2024-04-10 19:11:44 +00:00 |
|
|
Andrej Karpathy
|
fd8459d858
|
small gitignore for now. probably iterate on
|
2024-04-10 18:54:23 +00:00 |
|
|
Andrej
|
15fad9bc96
|
Create LICENSE MIT
|
2024-04-10 11:47:40 -07:00 |
|
|
Andrej Karpathy
|
80f52e5290
|
the full forward pass of GPT-2 in one file of pure CUDA
|
2024-04-10 18:15:55 +00:00 |
|
|
Ikko Eltociear Ashimine
|
ca137b02d5
|
update layernorm.md
burried -> buried
|
2024-04-11 00:07:22 +09:00 |
|
|
Andrej Karpathy
|
d8e2a36f15
|
ok crossentropy is fairly ez, just have to be careful with logs of negative numbers in debug tests. so this should be all the layers now
|
2024-04-10 01:13:20 +00:00 |
|
|
Andrej Karpathy
|
46ee0a3944
|
sorry there should be no mallocs inside these functions, have to pass in buffer memory from outside. getting tired
|
2024-04-10 00:55:14 +00:00 |
|
|
Andrej Karpathy
|
919ae1f27b
|
ok attention is now optimized too, by 20X, nice. actually this one was really painful, i had to make sure all the cuBLAS calls take all the right arguments and deal with the fact that cuBLAS is column major instead of row major. i go for nap now
|
2024-04-10 00:50:30 +00:00 |
|
|
lancer
|
fbd679adec
|
fix the consistency of the transpose notation
|
2024-04-09 15:19:55 -07:00 |
|
|
Andrej Karpathy
|
8386e5393c
|
intra-warp reductions and inter-warp shared memory reductions for B,T,C parallelism of softmax kernel nice
|
2024-04-09 20:08:11 +00:00 |
|
|
Andrej Karpathy
|
c29d70a9b3
|
add softmax forward, both a naive parallel over B,T version, and a more optimized parallel over B,T,C version using shared memory for reductions
|
2024-04-09 19:22:47 +00:00 |
|
|
Andrej Karpathy
|
327705d03f
|
mention ongoing cuda work in main readme
|
2024-04-09 18:29:24 +00:00 |
|
|
Andrej
|
ea1fad0337
|
Merge pull request #33 from lancerts/fix_M_PI_undefined
fix a potential error: identifier M_PI is undefined in the gelu kernal
|
2024-04-09 11:25:23 -07:00 |
|
|
Andrej
|
e3e0d73fbc
|
Merge pull request #34 from VinciGit00/patch-1
Free the memory in layernorm.c
|
2024-04-09 11:24:32 -07:00 |
|
|
Andrej Karpathy
|
bb4f04bdf8
|
add flash attention, but a naive implementation there of
|
2024-04-09 18:22:49 +00:00 |
|
|
Marco Vinciguerra
|
8397fb1512
|
Free the memory in layernorm.c
|
2024-04-09 20:14:18 +02:00 |
|
|
lancer
|
3a4d783633
|
amend
|
2024-04-09 09:59:03 -07:00 |
|