 Andrej Karpathy
|
a00a3a88ef
|
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
|
2024-04-26 18:11:09 +00:00 |
|
|
Andrej Karpathy
|
8389fbaa0c
|
Merge branch 'multi_gpu_support' of https://github.com/PeterZhizhin/llm.c into PeterZhizhin-multi_gpu_support
|
2024-04-26 15:55:15 +00:00 |
|
|
Petr Zhizhin
|
d49e8301eb
|
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
|
2024-04-24 22:11:00 +00:00 |
|
|
Andrej Karpathy
|
250159e140
|
encoder backward add, we want to get rid of atomics, hmm
|
2024-04-23 19:06:15 +00:00 |
|
|
Andrej Karpathy
|
249140287f
|
rename positional_forward to encoder_forward as it should have been
|
2024-04-23 18:19:20 +00:00 |
|
|
Andrej Karpathy
|
e3bcae6b2b
|
incorporate layernorm backward kernel 2, we are now down to 73.1ms/iter on average
|
2024-04-22 18:44:43 +00:00 |
|
|
Andrej Karpathy
|
86b3401dd5
|
Merge branch 'sharedmem_layernormback' of https://github.com/ChrisDryden/llm.c into ChrisDryden-sharedmem_layernormback
|
2024-04-22 18:33:42 +00:00 |
|
|
Andrej Karpathy
|
988489519c
|
nice new kernel for bias reduce, we are down by ~1ms/iter, to 76.13ms/iter on average
|
2024-04-22 17:51:06 +00:00 |
|
|
Andrej Karpathy
|
a42f739675
|
Merge branch 'matmul-backward-bias' of https://github.com/al0vya/llm.c into al0vya-matmul-backward-bias
|
2024-04-22 16:52:31 +00:00 |
|
|
Andrej Karpathy
|
b1e55951c6
|
rename to _forward for the kernel, and small fixes to docs, and avoid auto
|
2024-04-22 16:32:03 +00:00 |
|
|
cip19aac
|
35393b4442
|
add more comments to explain the philosophy behind the kernel
|
2024-04-22 17:29:11 +01:00 |
|
|
cip19aac
|
b82ec201be
|
add comment
|
2024-04-22 17:05:05 +01:00 |
|
|
cip19aac
|
55fbd7e6ad
|
add a faster matmul backward bias kernel that uses coalesced reads and shared memory
|
2024-04-22 16:35:21 +01:00 |
|
|
Erik Schultheis
|
732a8b4806
|
added NaN-based makes for reference checks
|
2024-04-22 14:53:24 +03:00 |
|
|
Erik Schultheis
|
caa69a5e52
|
added triangular matrix multiplication kernel
|
2024-04-22 14:45:51 +03:00 |
|
|
Chris Dryden
|
2752212151
|
Added shared memory for the atomic additions for the layernorm_backward.cu weights and biases
|
2024-04-21 16:37:08 -07:00 |
|
|
patricxu
|
41c3cc1fab
|
Merge branch 'karpathy:master' into master
|
2024-04-21 10:31:59 +08:00 |
|
|
Andrej Karpathy
|
9fb9c9185a
|
change baseline kernel to float, for now, for fair comparison too
|
2024-04-20 22:57:18 +00:00 |
|
|
Andrej Karpathy
|
8488669d25
|
speed up the backward bias kernel by 45% and speed up the full running time by 1%
|
2024-04-20 22:46:19 +00:00 |
|
|
Erik Schultheis
|
8ffc3e388b
|
slow attention kernel that demonstrates a direct translation of the CPU code
|
2024-04-20 19:03:34 +03:00 |
|
|
patricxu
|
d48b0d2b2b
|
fix indexing in matmul_forward_kernel1
|
2024-04-20 15:38:12 +08:00 |
|
|
Andrej
|
2d3a0fbe50
|
Merge pull request #181 from lancerts/const-fix-adamw
const-fix adamw.cu and fix the warning C4305: 'initializing': truncation from 'double' to 'float'
|
2024-04-19 22:06:11 -07:00 |
|
|
Andrej Karpathy
|
91aab62aad
|
fix kernel 5 of layernorm, bug from previous copy paste with loop bounds
|
2024-04-20 00:51:06 +00:00 |
|
|
Andrej Karpathy
|
49d41ae296
|
add one more kernel, allocating a block per row. bad idea if C is too low, as we have it right now
|
2024-04-19 23:24:51 +00:00 |
|
|
Andrej Karpathy
|
cb791c4ef5
|
new kernel that does a single pass over x on load, using a more clever variance formula. only very slightly faster on my A100 sadly
|
2024-04-19 22:55:30 +00:00 |
|
|
Erik Schultheis
|
44b5bb028c
|
one more version of attention backward for another few percent perf gains
|
2024-04-19 22:46:40 +03:00 |
|
|
Erik Schultheis
|
a360f104ae
|
some low-level optimizations
|
2024-04-19 22:38:13 +03:00 |
|
|
Erik Schultheis
|
7005e09450
|
added missing checks
|
2024-04-19 22:37:35 +03:00 |
|
|
lancer
|
3cb993b5f6
|
fix the constness and the warning C4305: 'initializing': truncation from 'double' to 'float'
|
2024-04-19 11:04:58 -07:00 |
|
|
Erik Schultheis
|
b2e6b6867f
|
separating the loops leads to yet more improvements
|
2024-04-19 20:47:19 +03:00 |
|
|
Andrej
|
b556ad971c
|
Merge pull request #179 from ngc92/even-better-attention
towards an even better backward attention kernel
|
2024-04-19 09:16:22 -07:00 |
|
|
Andrej
|
1d3b6738a7
|
Merge pull request #169 from lancerts/fix-crossentropy-bk
Fix the bug that yields cpu, gpu results mismatch in crossentropy_softmax_backward.cu
|
2024-04-19 09:15:23 -07:00 |
|
|
Erik Schultheis
|
50e105a250
|
further cleanup and ability to handle arbitrary sequence lengths
|
2024-04-19 19:12:37 +03:00 |
|
|
Erik Schultheis
|
4294820a8a
|
don't do stupid redundant work; the inner loop is just a block-level sum!!!!
|
2024-04-19 19:12:32 +03:00 |
|
|
Erik Schultheis
|
02aa8e47dc
|
further simplification of the loop
|
2024-04-19 19:12:32 +03:00 |
|
|
Erik Schultheis
|
16123c3af8
|
towards an even better backward attention kernel
|
2024-04-19 19:12:30 +03:00 |
|
|
lancer
|
0c92ea9b90
|
fix the 0.0f initialization for double
|
2024-04-18 19:35:34 -07:00 |
|
|
lancer
|
e991fe1efa
|
fix the 0.0f initialization for double
|
2024-04-18 19:33:30 -07:00 |
|
|
Andrej Karpathy
|
c393d828dc
|
size_t instead of int. everywhere. this will come to bite us at one point...
|
2024-04-19 02:11:30 +00:00 |
|
|
Andrej Karpathy
|
dcff5613c6
|
add kernel3, which is like kernel 2 but no float4
|
2024-04-19 01:19:20 +00:00 |
|
|
Andrej Karpathy
|
8a2cab3431
|
make default dloss be 1/BT
|
2024-04-19 00:27:28 +00:00 |
|
|
Andrej Karpathy
|
ea3e2abdcf
|
small fixes and more comments very cool kernels
|
2024-04-18 23:48:39 +00:00 |
|
|
Erik Schultheis
|
09d35a5e4d
|
splitting the loop to get rid of the indicator
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
6d98af1d32
|
another 15% by getting rid of ifs
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
314661f516
|
significantly faster kernel by reusing data in registers
|
2024-04-19 01:01:47 +03:00 |
|
|
Andrej Karpathy
|
de1e87158c
|
add comments to classifier_fused
|
2024-04-18 16:59:45 +00:00 |
|
|
Andrej
|
2d2f1df8f1
|
Merge pull request #150 from ademeure/optimised_fused_classifier
Optimised version of fused classifier + bugfixes(?)
|
2024-04-18 08:39:05 -07:00 |
|
|
FeSens
|
0c09a37568
|
fix(attention_forward.cu): Fix the number of elements to check when verifying attention implementations.
This was causing me problems when I was trying to debug a new implementation on fewer tokens.
|
2024-04-18 01:23:05 -03:00 |
|
|
lancer
|
a1abcbc32d
|
fix the bug that yields cpu, gpu results mismatch
|
2024-04-17 21:04:45 -07:00 |
|
|
ademeure
|
557fecd6e9
|
bugfix for bounds checking
|
2024-04-18 04:45:07 +01:00 |
|