 Erik Schultheis
|
7005e09450
|
added missing checks
|
2024-04-19 22:37:35 +03:00 |
|
|
lancer
|
3cb993b5f6
|
fix the constness and the warning C4305: 'initializing': truncation from 'double' to 'float'
|
2024-04-19 11:04:58 -07:00 |
|
|
Erik Schultheis
|
b2e6b6867f
|
separating the loops leads to yet more improvements
|
2024-04-19 20:47:19 +03:00 |
|
|
Andrej
|
b556ad971c
|
Merge pull request #179 from ngc92/even-better-attention
towards an even better backward attention kernel
|
2024-04-19 09:16:22 -07:00 |
|
|
Andrej
|
1d3b6738a7
|
Merge pull request #169 from lancerts/fix-crossentropy-bk
Fix the bug that yields cpu, gpu results mismatch in crossentropy_softmax_backward.cu
|
2024-04-19 09:15:23 -07:00 |
|
|
Erik Schultheis
|
50e105a250
|
further cleanup and ability to handle arbitrary sequence lengths
|
2024-04-19 19:12:37 +03:00 |
|
|
Erik Schultheis
|
4294820a8a
|
don't do stupid redundant work; the inner loop is just a block-level sum!!!!
|
2024-04-19 19:12:32 +03:00 |
|
|
Erik Schultheis
|
02aa8e47dc
|
further simplification of the loop
|
2024-04-19 19:12:32 +03:00 |
|
|
Erik Schultheis
|
16123c3af8
|
towards an even better backward attention kernel
|
2024-04-19 19:12:30 +03:00 |
|
|
lancer
|
0c92ea9b90
|
fix the 0.0f initialization for double
|
2024-04-18 19:35:34 -07:00 |
|
|
lancer
|
e991fe1efa
|
fix the 0.0f initialization for double
|
2024-04-18 19:33:30 -07:00 |
|
|
Andrej Karpathy
|
c393d828dc
|
size_t instead of int. everywhere. this will come to bite us at one point...
|
2024-04-19 02:11:30 +00:00 |
|
|
Andrej Karpathy
|
dcff5613c6
|
add kernel3, which is like kernel 2 but no float4
|
2024-04-19 01:19:20 +00:00 |
|
|
Andrej Karpathy
|
8a2cab3431
|
make default dloss be 1/BT
|
2024-04-19 00:27:28 +00:00 |
|
|
Andrej Karpathy
|
ea3e2abdcf
|
small fixes and more comments very cool kernels
|
2024-04-18 23:48:39 +00:00 |
|
|
Erik Schultheis
|
09d35a5e4d
|
splitting the loop to get rid of the indicator
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
6d98af1d32
|
another 15% by getting rid of ifs
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
314661f516
|
significantly faster kernel by reusing data in registers
|
2024-04-19 01:01:47 +03:00 |
|
|
Andrej Karpathy
|
de1e87158c
|
add comments to classifier_fused
|
2024-04-18 16:59:45 +00:00 |
|
|
Andrej
|
2d2f1df8f1
|
Merge pull request #150 from ademeure/optimised_fused_classifier
Optimised version of fused classifier + bugfixes(?)
|
2024-04-18 08:39:05 -07:00 |
|
|
FeSens
|
0c09a37568
|
fix(attention_forward.cu): Fix the number of elements to check when verifying attention implementations.
This was causing me problems when I was trying to debug a new implementation on fewer tokens.
|
2024-04-18 01:23:05 -03:00 |
|
|
lancer
|
a1abcbc32d
|
fix the bug that yields cpu, gpu results mismatch
|
2024-04-17 21:04:45 -07:00 |
|
|
ademeure
|
557fecd6e9
|
bugfix for bounds checking
|
2024-04-18 04:45:07 +01:00 |
|
|
ademeure
|
2fd8ada4a6
|
Merge remote-tracking branch 'karpathy/master' into optimised_fused_classifier
|
2024-04-18 03:51:01 +01:00 |
|
|
ademeure
|
ea77a26473
|
mini fix
|
2024-04-18 03:42:34 +01:00 |
|
|
ademeure
|
3ba2cdc723
|
fixes + bounds checking
|
2024-04-18 03:41:27 +01:00 |
|
|
lancer
|
ac055f730c
|
fix the bug that yields cpu, gpu results mismatch
|
2024-04-17 17:47:11 -07:00 |
|
|
Erik Schultheis
|
58a7b57f96
|
enable benchmarking
|
2024-04-18 01:39:47 +03:00 |
|
|
Erik Schultheis
|
e278112c19
|
coalesced reading in attention backward pass
|
2024-04-18 01:39:43 +03:00 |
|
|
ademeure
|
4ee98f5135
|
Merge branch 'master' into optimised_fused_classifier
|
2024-04-17 22:58:51 +01:00 |
|
|
lancer
|
c327923df9
|
Constness fix layernorm_backward.cu
|
2024-04-17 14:56:03 -07:00 |
|
|
lancer
|
81aa596251
|
Constness fix layernorm_backward.cu
|
2024-04-17 14:54:49 -07:00 |
|
|
Erik Schultheis
|
c1bfe4a639
|
warp-wide reduction for layernorm backward
|
2024-04-17 21:52:40 +03:00 |
|
|
Andrej Karpathy
|
fa4ebfa663
|
kernel2 for softmax backward
|
2024-04-17 08:14:22 +00:00 |
|
|
Andrej Karpathy
|
d74d7ae652
|
small adamw fixes, needs more work
|
2024-04-17 04:58:16 +00:00 |
|
|
Andrej Karpathy
|
66a0b5242e
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-17 04:51:26 +00:00 |
|
|
Andrej Karpathy
|
8a673edcc3
|
push my attention_backward file
|
2024-04-17 04:42:42 +00:00 |
|
|
nopperl
|
97e61e2258
|
add slightly more optimized adamw kernel
|
2024-04-17 03:31:07 +02:00 |
|
|
nopperl
|
5684ff7e77
|
add adamw cuda kernel
|
2024-04-17 03:30:59 +02:00 |
|
|
Erik Schultheis
|
43bf9eb66a
|
ensure valid test cases and incorporate fixes by @ademeure
|
2024-04-16 20:53:50 +03:00 |
|
|
ademeure
|
b225501b7b
|
optimised version of the cool new fused classifier + hopefully bugfixes?
|
2024-04-16 14:54:23 +01:00 |
|
|
lancer
|
6ff4ad646d
|
fix the backward dbias precision issue
|
2024-04-15 19:58:16 -07:00 |
|
|
Andrej Karpathy
|
6be8080501
|
add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far
|
2024-04-16 02:10:45 +00:00 |
|
|
Andrej Karpathy
|
3de507739c
|
minor fixes to matmul backward
|
2024-04-16 01:23:41 +00:00 |
|
|
Andrej Karpathy
|
9560b1a92b
|
add matmul backward kernel, problem is the bias backward, ironically. which i thought i could cublas at first
|
2024-04-16 00:40:11 +00:00 |
|
|
Andrej
|
3c6dafeb53
|
Merge pull request #117 from ngc92/fused-classifier
WIP: Fully fused classification layer
|
2024-04-15 12:13:48 -07:00 |
|
|
Erik Schultheis
|
c73fa8842a
|
fixed numerics for online kernel, and an existing race condition in kernel 2
|
2024-04-15 03:41:09 +03:00 |
|
|
Erik Schultheis
|
8fb966bc31
|
kernel 1 should also accumulate in double to ensure stability
|
2024-04-15 02:52:38 +03:00 |
|
|
Erik Schultheis
|
60cf281e46
|
use V dimension in softmax tests
adjusted tests to be more challenging
|
2024-04-15 02:14:48 +03:00 |
|
|
Erik Schultheis
|
99b42c1a94
|
nicer printing
|
2024-04-15 02:03:48 +03:00 |
|