 Andrej Karpathy
|
769c911ab6
|
small tweaks
|
2024-04-30 23:48:06 +00:00 |
|
|
Erik Schultheis
|
603d862830
|
fix up bandwidth calculation
|
2024-04-30 23:52:08 +03:00 |
|
|
Erik Schultheis
|
c3515cfe47
|
fixed potential error and generalized gelu forward
|
2024-04-30 23:45:34 +03:00 |
|
|
Andrej Karpathy
|
7ba1ed8a1a
|
delete packed128 .fp32, use (float) ok
|
2024-04-30 19:39:20 +00:00 |
|
|
Andrej Karpathy
|
9141e04693
|
fixes to the PR, careful with float/floatX etc
|
2024-04-30 19:35:05 +00:00 |
|
|
Andrej Karpathy
|
242981eecf
|
Merge branch 'gelu_forwards_packing' of https://github.com/ChrisDryden/llm.c into ChrisDryden-gelu_forwards_packing
|
2024-04-30 19:06:39 +00:00 |
|
|
Franz Louis Cesista
|
b239b67ae1
|
add modal script
|
2024-05-01 00:15:08 +08:00 |
|
|
Erik Schultheis
|
d7813d281d
|
clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results
|
2024-04-30 12:45:37 +03:00 |
|
|
Christopher Dryden
|
568615fef1
|
Added packing for gelu forwards kernel
|
2024-04-30 03:21:42 +00:00 |
|
|
Andrej Karpathy
|
af2bc47d22
|
remove forceinline. blame @arund42 if this ever does not get inlined by the compiler
|
2024-04-29 23:10:51 +00:00 |
|
|
Andrej Karpathy
|
5e12c66f5a
|
move packed128 to common
|
2024-04-29 22:55:28 +00:00 |
|
|
Andrej Karpathy
|
f43592954f
|
reset back to precious hyperparameters
|
2024-04-29 22:44:25 +00:00 |
|
|
Andrej Karpathy
|
ee125cffc4
|
make it imo clearer that the grid size is one quarter as large as it would otherwise be. because each thread now does 4X work
|
2024-04-29 22:40:26 +00:00 |
|
|
Andrej Karpathy
|
083b210dd4
|
small changes
|
2024-04-29 22:38:03 +00:00 |
|
|
Andrej Karpathy
|
d9506a7a0a
|
minor changes to packed128
|
2024-04-29 22:31:19 +00:00 |
|
|
Andrej Karpathy
|
7eefefb511
|
Merge branch 'yet-another-gelu' of https://github.com/ngc92/llm.c into ngc92-yet-another-gelu
|
2024-04-29 22:19:35 +00:00 |
|
|
Erik Schultheis
|
4872c57164
|
comments
|
2024-04-29 22:46:33 +03:00 |
|
|
Erik Schultheis
|
9f07a176e0
|
simplify to only support cs
|
2024-04-29 22:43:52 +03:00 |
|
|
Erik Schultheis
|
896b644cb9
|
slight simplification
|
2024-04-29 22:19:12 +03:00 |
|
|
Erik Schultheis
|
45252d434c
|
yet another gelu
|
2024-04-29 21:53:08 +03:00 |
|
|
ademeure
|
d23bb1fd13
|
Merge remote-tracking branch 'karpathy/master' into layernormbf16
|
2024-04-29 19:41:54 +01:00 |
|
|
ademeure
|
49ffd68317
|
Fix sizeof(T) -> sizeof(D) (was overallocating memory)
|
2024-04-29 15:47:59 +01:00 |
|
|
ademeure
|
1e50f3b976
|
Only enable kernel4 with BF16
|
2024-04-29 01:55:06 +01:00 |
|
|
ademeure
|
82fef03acd
|
Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)
|
2024-04-29 01:51:10 +01:00 |
|
|
Andrej Karpathy
|
327eef3f22
|
incorporate faster encoder_forward kernel to fp32 CUDA version
|
2024-04-28 20:10:33 +00:00 |
|
|
lancer
|
1c7d23ab3c
|
amend the float4 kernel
|
2024-04-27 20:26:19 -07:00 |
|
|
lancer
|
61c5c05ea8
|
amend the float4 kernel
|
2024-04-27 20:23:39 -07:00 |
|
|
lancer
|
cfccd821e6
|
Include the float4 kernel
|
2024-04-27 19:58:18 -07:00 |
|
|
Andrej Karpathy
|
12da2c16cc
|
slight docs tweaks to dev/cuda
|
2024-04-27 14:40:57 +00:00 |
|
|
austinvhuang
|
25d703fbec
|
Use a make pattern rule to simplify Makefile, group kernels into forward/backward/updates/nccls, add some comments and update the README.md
|
2024-04-27 00:13:01 -04:00 |
|
|
austinvhuang
|
d937f93aaa
|
Add a local convenience Makefile for dev/cuda/, fix 2 minor warnings observed when building all targets.
|
2024-04-26 21:19:44 -04:00 |
|
|
Andrej Karpathy
|
a00a3a88ef
|
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
|
2024-04-26 18:11:09 +00:00 |
|
|
Andrej Karpathy
|
8389fbaa0c
|
Merge branch 'multi_gpu_support' of https://github.com/PeterZhizhin/llm.c into PeterZhizhin-multi_gpu_support
|
2024-04-26 15:55:15 +00:00 |
|
|
Petr Zhizhin
|
d49e8301eb
|
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
|
2024-04-24 22:11:00 +00:00 |
|
|
Andrej Karpathy
|
250159e140
|
encoder backward add, we want to get rid of atomics, hmm
|
2024-04-23 19:06:15 +00:00 |
|
|
Andrej Karpathy
|
249140287f
|
rename positional_forward to encoder_forward as it should have been
|
2024-04-23 18:19:20 +00:00 |
|
|
Andrej Karpathy
|
e3bcae6b2b
|
incorporate layernorm backward kernel 2, we are now down to 73.1ms/iter on average
|
2024-04-22 18:44:43 +00:00 |
|
|
Andrej Karpathy
|
86b3401dd5
|
Merge branch 'sharedmem_layernormback' of https://github.com/ChrisDryden/llm.c into ChrisDryden-sharedmem_layernormback
|
2024-04-22 18:33:42 +00:00 |
|
|
Andrej Karpathy
|
988489519c
|
nice new kernel for bias reduce, we are down by ~1ms/iter, to 76.13ms/iter on average
|
2024-04-22 17:51:06 +00:00 |
|
|
Andrej Karpathy
|
a42f739675
|
Merge branch 'matmul-backward-bias' of https://github.com/al0vya/llm.c into al0vya-matmul-backward-bias
|
2024-04-22 16:52:31 +00:00 |
|
|
Andrej Karpathy
|
b1e55951c6
|
rename to _forward for the kernel, and small fixes to docs, and avoid auto
|
2024-04-22 16:32:03 +00:00 |
|
|
cip19aac
|
35393b4442
|
add more comments to explain the philosophy behind the kernel
|
2024-04-22 17:29:11 +01:00 |
|
|
cip19aac
|
b82ec201be
|
add comment
|
2024-04-22 17:05:05 +01:00 |
|
|
cip19aac
|
55fbd7e6ad
|
add a faster matmul backward bias kernel that uses coalesced reads and shared memory
|
2024-04-22 16:35:21 +01:00 |
|
|
Erik Schultheis
|
732a8b4806
|
added NaN-based makes for reference checks
|
2024-04-22 14:53:24 +03:00 |
|
|
Erik Schultheis
|
caa69a5e52
|
added triangular matrix multiplication kernel
|
2024-04-22 14:45:51 +03:00 |
|
|
Chris Dryden
|
2752212151
|
Added shared memory for the atomic additions for the layernorm_backward.cu weights and biases
|
2024-04-21 16:37:08 -07:00 |
|
|
patricxu
|
41c3cc1fab
|
Merge branch 'karpathy:master' into master
|
2024-04-21 10:31:59 +08:00 |
|
|
Andrej Karpathy
|
9fb9c9185a
|
change baseline kernel to float, for now, for fair comparison too
|
2024-04-20 22:57:18 +00:00 |
|
|
Andrej Karpathy
|
8488669d25
|
speed up the backward bias kernel by 45% and speed up the full running time by 1%
|
2024-04-20 22:46:19 +00:00 |
|