 Andrej Karpathy
|
ea3e2abdcf
|
small fixes and more comments very cool kernels
|
2024-04-18 23:48:39 +00:00 |
|
|
Erik Schultheis
|
09d35a5e4d
|
splitting the loop to get rid of the indicator
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
6d98af1d32
|
another 15% by getting rid of ifs
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
314661f516
|
significantly faster kernel by reusing data in registers
|
2024-04-19 01:01:47 +03:00 |
|
|
Andrej Karpathy
|
de1e87158c
|
add comments to classifier_fused
|
2024-04-18 16:59:45 +00:00 |
|
|
Andrej
|
2d2f1df8f1
|
Merge pull request #150 from ademeure/optimised_fused_classifier
Optimised version of fused classifier + bugfixes(?)
|
2024-04-18 08:39:05 -07:00 |
|
|
FeSens
|
0c09a37568
|
fix(attention_forward.cu): Fix the number of elements to check when verifying attention implementations.
This was causing me problems when I was trying to debug a new implementation on fewer tokens.
|
2024-04-18 01:23:05 -03:00 |
|
|
lancer
|
a1abcbc32d
|
fix the bug that yields cpu, gpu results mismatch
|
2024-04-17 21:04:45 -07:00 |
|
|
ademeure
|
557fecd6e9
|
bugfix for bounds checking
|
2024-04-18 04:45:07 +01:00 |
|
|
ademeure
|
2fd8ada4a6
|
Merge remote-tracking branch 'karpathy/master' into optimised_fused_classifier
|
2024-04-18 03:51:01 +01:00 |
|
|
ademeure
|
ea77a26473
|
mini fix
|
2024-04-18 03:42:34 +01:00 |
|
|
ademeure
|
3ba2cdc723
|
fixes + bounds checking
|
2024-04-18 03:41:27 +01:00 |
|
|
lancer
|
ac055f730c
|
fix the bug that yields cpu, gpu results mismatch
|
2024-04-17 17:47:11 -07:00 |
|
|
Erik Schultheis
|
58a7b57f96
|
enable benchmarking
|
2024-04-18 01:39:47 +03:00 |
|
|
Erik Schultheis
|
e278112c19
|
coalesced reading in attention backward pass
|
2024-04-18 01:39:43 +03:00 |
|
|
ademeure
|
4ee98f5135
|
Merge branch 'master' into optimised_fused_classifier
|
2024-04-17 22:58:51 +01:00 |
|
|
lancer
|
c327923df9
|
Constness fix layernorm_backward.cu
|
2024-04-17 14:56:03 -07:00 |
|
|
lancer
|
81aa596251
|
Constness fix layernorm_backward.cu
|
2024-04-17 14:54:49 -07:00 |
|
|
Erik Schultheis
|
c1bfe4a639
|
warp-wide reduction for layernorm backward
|
2024-04-17 21:52:40 +03:00 |
|
|
Andrej Karpathy
|
fa4ebfa663
|
kernel2 for softmax backward
|
2024-04-17 08:14:22 +00:00 |
|
|
Andrej Karpathy
|
d74d7ae652
|
small adamw fixes, needs more work
|
2024-04-17 04:58:16 +00:00 |
|
|
Andrej Karpathy
|
66a0b5242e
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-17 04:51:26 +00:00 |
|
|
Andrej Karpathy
|
8a673edcc3
|
push my attention_backward file
|
2024-04-17 04:42:42 +00:00 |
|
|
nopperl
|
97e61e2258
|
add slightly more optimized adamw kernel
|
2024-04-17 03:31:07 +02:00 |
|
|
nopperl
|
5684ff7e77
|
add adamw cuda kernel
|
2024-04-17 03:30:59 +02:00 |
|
|
Erik Schultheis
|
43bf9eb66a
|
ensure valid test cases and incorporate fixes by @ademeure
|
2024-04-16 20:53:50 +03:00 |
|
|
ademeure
|
b225501b7b
|
optimised version of the cool new fused classifier + hopefully bugfixes?
|
2024-04-16 14:54:23 +01:00 |
|
|
lancer
|
6ff4ad646d
|
fix the backward dbias precision issue
|
2024-04-15 19:58:16 -07:00 |
|
|
Andrej Karpathy
|
6be8080501
|
add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far
|
2024-04-16 02:10:45 +00:00 |
|
|
Andrej Karpathy
|
3de507739c
|
minor fixes to matmul backward
|
2024-04-16 01:23:41 +00:00 |
|
|
Andrej Karpathy
|
9560b1a92b
|
add matmul backward kernel, problem is the bias backward, ironically. which i thought i could cublas at first
|
2024-04-16 00:40:11 +00:00 |
|
|
Andrej
|
3c6dafeb53
|
Merge pull request #117 from ngc92/fused-classifier
WIP: Fully fused classification layer
|
2024-04-15 12:13:48 -07:00 |
|
|
Erik Schultheis
|
c73fa8842a
|
fixed numerics for online kernel, and an existing race condition in kernel 2
|
2024-04-15 03:41:09 +03:00 |
|
|
Erik Schultheis
|
8fb966bc31
|
kernel 1 should also accumulate in double to ensure stability
|
2024-04-15 02:52:38 +03:00 |
|
|
Erik Schultheis
|
60cf281e46
|
use V dimension in softmax tests
adjusted tests to be more challenging
|
2024-04-15 02:14:48 +03:00 |
|
|
Erik Schultheis
|
99b42c1a94
|
nicer printing
|
2024-04-15 02:03:48 +03:00 |
|
|
Erik Schultheis
|
fe1d8cfb47
|
fix timing output
|
2024-04-15 02:03:42 +03:00 |
|
|
Erik Schultheis
|
780d8bd012
|
initial version
|
2024-04-14 03:48:56 +03:00 |
|
|
Andrej Karpathy
|
534b93a732
|
erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7
|
2024-04-13 22:02:31 +00:00 |
|
|
Andrej Karpathy
|
c5ee371564
|
make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common
|
2024-04-13 17:56:18 +00:00 |
|
|
Andrej Karpathy
|
82623518a4
|
fix comment block of gelu forward kernel
|
2024-04-13 17:48:53 +00:00 |
|
|
Andrej Karpathy
|
02b5b33c5f
|
remove unused variable C
|
2024-04-13 17:48:11 +00:00 |
|
|
Andrej Karpathy
|
84145411d3
|
fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nans
|
2024-04-13 17:47:19 +00:00 |
|
|
Andrej Karpathy
|
69f0385237
|
add comments and handling of kernels and their correctness
|
2024-04-13 17:44:31 +00:00 |
|
|
Erik Schultheis
|
f459bfa833
|
fixed re-initialization of device memory after test
|
2024-04-13 18:23:12 +03:00 |
|
|
Erik Schultheis
|
d035725d22
|
mark problematic tests
|
2024-04-13 18:09:28 +03:00 |
|
|
Erik Schultheis
|
59bada94d9
|
improved testing
|
2024-04-13 17:06:52 +03:00 |
|
|
Erik Schultheis
|
7e7460cfbd
|
const'ness fixes
|
2024-04-13 13:38:11 +03:00 |
|
|
Erik Schultheis
|
7e8e038de8
|
extracted common testing functions into their own header
|
2024-04-13 13:00:05 +03:00 |
|
|
Andrej Karpathy
|
6b49ed1c0b
|
incorporate the fastest kernels and adjust readme
|
2024-04-13 02:58:23 +00:00 |
|