 Erik Schultheis
|
af9ac14b2f
|
don't track preatt and vaccum across layers
|
2024-04-20 20:18:41 +03:00 |
|
|
Erik Schultheis
|
8ffc3e388b
|
slow attention kernel that demonstrates a direct translation of the CPU code
|
2024-04-20 19:03:34 +03:00 |
|
|
Erik Schultheis
|
d12b923c00
|
profile and print out some nice stats
|
2024-04-20 13:30:31 +03:00 |
|
|
Ross Wheeler
|
d24562932f
|
Adding unistd.h for Windows in Dev directory
Adding unistd.h for Windows in Dev directory
|
2024-04-20 02:37:16 -07:00 |
|
|
Ross Wheeler
|
e564222113
|
const changes to support OpenMP 5.2 MSVC work-around
const changes to support OpenMP MSVC work-around
|
2024-04-20 02:07:12 -07:00 |
|
|
patricxu
|
d48b0d2b2b
|
fix indexing in matmul_forward_kernel1
|
2024-04-20 15:38:12 +08:00 |
|
|
Andrej
|
2d3a0fbe50
|
Merge pull request #181 from lancerts/const-fix-adamw
const-fix adamw.cu and fix the warning C4305: 'initializing': truncation from 'double' to 'float'
|
2024-04-19 22:06:11 -07:00 |
|
|
Andrej Karpathy
|
d34556cf5e
|
more profile docs
|
2024-04-20 01:29:14 +00:00 |
|
|
Andrej Karpathy
|
67fd37ace4
|
add crappy instructions for profiling basics
|
2024-04-20 01:06:53 +00:00 |
|
|
Andrej Karpathy
|
7b79fbb230
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-20 00:59:39 +00:00 |
|
|
Andrej Karpathy
|
7df2d4d92b
|
also requirements for quick start
|
2024-04-20 00:59:32 +00:00 |
|
|
Andrej
|
694cb571ec
|
Merge pull request #189 from ngc92/profile-target
profiling script
|
2024-04-19 17:55:35 -07:00 |
|
|
Andrej Karpathy
|
91aab62aad
|
fix kernel 5 of layernorm, bug from previous copy paste with loop bounds
|
2024-04-20 00:51:06 +00:00 |
|
|
Andrej Karpathy
|
5bdbc45806
|
only do 5 validation batches by default, we really don't need much precision here for this little script with current defaults
|
2024-04-20 00:45:13 +00:00 |
|
|
Andrej Karpathy
|
38fd42d3e3
|
also add a quick start CPU section
|
2024-04-20 00:41:13 +00:00 |
|
|
Andrej Karpathy
|
da4c5c79ca
|
very quick start section add
|
2024-04-20 00:35:35 +00:00 |
|
|
Erik Schultheis
|
9c1c1f0cec
|
helper target that gets compiled with lineinfo and runs a single layer forward-backward pass
|
2024-04-20 03:19:23 +03:00 |
|
|
Andrej Karpathy
|
49d41ae296
|
add one more kernel, allocating a block per row. bad idea if C is too low, as we have it right now
|
2024-04-19 23:24:51 +00:00 |
|
|
Andrej Karpathy
|
cb791c4ef5
|
new kernel that does a single pass over x on load, using a more clever variance formula. only very slightly faster on my A100 sadly
|
2024-04-19 22:55:30 +00:00 |
|
|
Andrej
|
816254e454
|
Merge pull request #187 from azret/csharp-port-ref
README: Adding C# port ref
|
2024-04-19 14:17:59 -07:00 |
|
|
Azret Botash
|
c6bd590b47
|
typo
|
2024-04-19 14:17:13 -07:00 |
|
|
Azret Botash
|
6cd1354b3e
|
C# port ref
|
2024-04-19 13:59:36 -07:00 |
|
|
Andrej
|
792812985b
|
Merge pull request #185 from dorjeduck/update-readme-notable-forks
Added link to Mojo port in Notable Forks section
|
2024-04-19 13:22:32 -07:00 |
|
|
Andrej Karpathy
|
b173e0237c
|
Merge branch 'ngc92-and-one-more'
|
2024-04-19 20:21:50 +00:00 |
|
|
Martin Dudek
|
1f08134fb9
|
Added link to Mojo port in Notable Forks section
|
2024-04-20 01:44:50 +05:30 |
|
|
Erik Schultheis
|
44b5bb028c
|
one more version of attention backward for another few percent perf gains
|
2024-04-19 22:46:40 +03:00 |
|
|
Erik Schultheis
|
a360f104ae
|
some low-level optimizations
|
2024-04-19 22:38:13 +03:00 |
|
|
Erik Schultheis
|
7005e09450
|
added missing checks
|
2024-04-19 22:37:35 +03:00 |
|
|
Andrej
|
bba794f156
|
Merge pull request #180 from ngc92/yet-more-attention
separating the loops leads to yet more improvements
|
2024-04-19 11:05:55 -07:00 |
|
|
lancer
|
3cb993b5f6
|
fix the constness and the warning C4305: 'initializing': truncation from 'double' to 'float'
|
2024-04-19 11:04:58 -07:00 |
|
|
Erik Schultheis
|
b2e6b6867f
|
separating the loops leads to yet more improvements
|
2024-04-19 20:47:19 +03:00 |
|
|
Andrej Karpathy
|
810b0dd90f
|
adjust header
|
2024-04-19 17:19:26 +00:00 |
|
|
Andrej Karpathy
|
f7fb3e67d5
|
update readme
|
2024-04-19 17:16:48 +00:00 |
|
|
Andrej Karpathy
|
1dbf241fbe
|
cleanups of the main script
|
2024-04-19 17:02:46 +00:00 |
|
|
Andrej Karpathy
|
09a117a664
|
add a compile option for torch compile suggested by Horace
|
2024-04-19 16:32:24 +00:00 |
|
|
Andrej
|
b556ad971c
|
Merge pull request #179 from ngc92/even-better-attention
towards an even better backward attention kernel
|
2024-04-19 09:16:22 -07:00 |
|
|
Andrej
|
1d3b6738a7
|
Merge pull request #169 from lancerts/fix-crossentropy-bk
Fix the bug that yields cpu, gpu results mismatch in crossentropy_softmax_backward.cu
|
2024-04-19 09:15:23 -07:00 |
|
|
Andrej
|
c8f3981d6b
|
Merge pull request #175 from lancerts/fix-double-f
Fix the 0.0f initialization for double
|
2024-04-19 09:12:58 -07:00 |
|
|
Erik Schultheis
|
50e105a250
|
further cleanup and ability to handle arbitrary sequence lengths
|
2024-04-19 19:12:37 +03:00 |
|
|
Erik Schultheis
|
4294820a8a
|
don't do stupid redundant work; the inner loop is just a block-level sum!!!!
|
2024-04-19 19:12:32 +03:00 |
|
|
Erik Schultheis
|
02aa8e47dc
|
further simplification of the loop
|
2024-04-19 19:12:32 +03:00 |
|
|
Erik Schultheis
|
16123c3af8
|
towards an even better backward attention kernel
|
2024-04-19 19:12:30 +03:00 |
|
|
lancer
|
0c92ea9b90
|
fix the 0.0f initialization for double
|
2024-04-18 19:35:34 -07:00 |
|
|
lancer
|
e991fe1efa
|
fix the 0.0f initialization for double
|
2024-04-18 19:33:30 -07:00 |
|
|
Andrej Karpathy
|
0dbfa32fd2
|
sorry this is a double so i shouldn't use 0.0f. i have to dedicate a whole commit to this because of my OCD
|
2024-04-19 02:23:21 +00:00 |
|
|
Andrej Karpathy
|
315af5f5dd
|
micro-optimization on the order, maximizing cache re-use ty @YichengDWu for original PR, which I adapted a bit
|
2024-04-19 02:21:28 +00:00 |
|
|
Andrej Karpathy
|
c393d828dc
|
size_t instead of int. everywhere. this will come to bite us at one point...
|
2024-04-19 02:11:30 +00:00 |
|
|
Andrej Karpathy
|
213ed60afd
|
fused classifier! we don't materialize the probs. we go from 200ms to 190ms
|
2024-04-19 01:33:35 +00:00 |
|
|
Andrej Karpathy
|
dcff5613c6
|
add kernel3, which is like kernel 2 but no float4
|
2024-04-19 01:19:20 +00:00 |
|
|
Andrej Karpathy
|
8a2cab3431
|
make default dloss be 1/BT
|
2024-04-19 00:27:28 +00:00 |
|