780d8bd012
initial version
2024-04-14 03:48:56 +03:00
534b93a732
erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7
2024-04-13 22:02:31 +00:00
0554c7e414
Merge pull request #115 from ademeure/softmax_largeC
...
New softmax for non-self-attention case optimised for large Cs + add more __ldcs
2024-04-13 14:49:26 -07:00
fc3dce37ce
Add __ldcs to a few reads for which the memory will never be read again in the forward pass
2024-04-13 22:38:12 +01:00
321b042657
New softmax for non-self-attention softmax case with large Cs
2024-04-13 22:31:23 +01:00
c02bae2ebc
average iterates in code
2024-04-13 21:09:37 +00:00
c70fbd059b
refine comment on time
2024-04-13 21:09:25 +00:00
ca242a3d0b
get rid of spurious handle variable creation, remnant from before, and update README
2024-04-13 21:04:49 +00:00
c65d1d305b
Merge pull request #113 from ademeure/cublasTF32
...
Fix cuBLAS to use TF32 with pre-initialised handle
2024-04-13 13:56:13 -07:00
5ab44973b0
Fix cuBLAS calls to use pre-initialised handle with TF32 enabled (big perf gain)
2024-04-13 20:35:53 +01:00
dd3dc179a4
git push origin masterMerge branch 'ngc92-common'
2024-04-13 17:57:33 +00:00
c5ee371564
make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common
2024-04-13 17:56:18 +00:00
82623518a4
fix comment block of gelu forward kernel
2024-04-13 17:48:53 +00:00
02b5b33c5f
remove unused variable C
2024-04-13 17:48:11 +00:00
84145411d3
fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nans
2024-04-13 17:47:19 +00:00
69f0385237
add comments and handling of kernels and their correctness
2024-04-13 17:44:31 +00:00
035b9e76fc
Merge branch 'common' of https://github.com/ngc92/llm.c into ngc92-common
2024-04-13 17:25:06 +00:00
552b972e52
Merge branch 'zocterminal-warnings'
2024-04-13 16:22:10 +00:00
f347957edf
change comment style
2024-04-13 16:21:57 +00:00
f459bfa833
fixed re-initialization of device memory after test
2024-04-13 18:23:12 +03:00
d035725d22
mark problematic tests
2024-04-13 18:09:28 +03:00
59bada94d9
improved testing
2024-04-13 17:06:52 +03:00
ae63f6d008
warnings in train_gpt2.cu
2024-04-13 12:52:54 +00:00
7e7460cfbd
const'ness fixes
2024-04-13 13:38:11 +03:00
7e8e038de8
extracted common testing functions into their own header
2024-04-13 13:00:05 +03:00
6b49ed1c0b
incorporate the fastest kernels and adjust readme
2024-04-13 02:58:23 +00:00
4f75e64573
Merge branch 'ngc92-attention-softmax-2'
2024-04-13 02:39:10 +00:00
6f9e28a1b6
few comments
2024-04-13 02:38:57 +00:00
2f99b3e608
Merge branch 'attention-softmax' of https://github.com/ngc92/llm.c into ngc92-attention-softmax-2
2024-04-13 02:34:06 +00:00
bd743f806a
Merge branch 'lancerts-online_softmax'
2024-04-13 02:11:50 +00:00
40b78be722
small changes to comments, and very minor code modifications and movements
2024-04-13 02:11:36 +00:00
6250336c09
Merge branch 'online_softmax' of https://github.com/lancerts/llm.c into lancerts-online_softmax
2024-04-13 01:54:16 +00:00
a18e517c1a
alright let's start the backward pass. crossentropy_softmax_backward first, ez one
2024-04-13 01:52:55 +00:00
492a4b151a
oops we missed a comment here, remove as we got rid of the inline mallocs now, so we're all ok
2024-04-13 01:21:31 +00:00
e9f095208a
Merge pull request #100 from ademeure/cudahostmalloc
...
Use cudaHostMalloc for inputs/targets and cpu_losses
2024-04-12 18:19:38 -07:00
2616a6461d
Use cudaHostMalloc for inputs/targets and cpu_losses (+moved its malloc/free away from main loop).
2024-04-13 01:31:16 +01:00
3ab2039e32
update timings
2024-04-12 23:47:15 +00:00
59b9e8b96a
fix handles in cublas in attention forward as well
2024-04-12 23:15:47 +00:00
8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it ( #98 )
...
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
2024-04-12 15:32:07 -07:00
e048825f5a
div_ceil
2024-04-13 01:20:09 +03:00
8e87aac341
online softmax
2024-04-12 12:47:48 +03:00
de98a9a44a
vector loads
2024-04-12 12:47:43 +03:00
6396e393e3
add links to discussions
2024-04-12 01:11:29 +00:00
baba3cfa01
recalculate instead of memory round-trip
2024-04-12 02:20:20 +03:00
69d0e5bac7
Include a parallel kernal of softmax_online
2024-04-11 15:30:36 -07:00
ca1e5ee04d
Include a parallel kernal of softmax_online
2024-04-11 15:24:41 -07:00
dec54904b4
one more simple clarifying comment
2024-04-11 22:24:38 +00:00
d680fbad25
one more comment on attention
2024-04-11 21:49:14 +00:00
9710163a60
did one pass of additional comments, potentially helpful, as i think a lot of people may be looking at this for the first time
2024-04-11 21:45:46 +00:00
5f99fb30d2
cooperative groups and fused scale kernel
2024-04-11 23:06:46 +03:00
Erik Schultheis