 Insop Song
|
d1f2f64541
|
Updated repkv_backward cuda kernel
- kernel 1 is tested
- build
```
make repkv_backward
/usr/local/cuda/bin/nvcc -O3 --use_fast_math --generate-code arch=compute_80,code=[compute_80,sm_80] -lcublas -lcublasLt -std=c++17 repkv_backward.cu -o repkv_backward
```
- test run on A30
```
Using kernel 1
Checking block size 32.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 64.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 128.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 256.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 512.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 1024.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
All results match. Starting benchmarks.
block_size 32 time 3.2461 ms
block_size 64 time 1.7509 ms
block_size 128 time 1.7374 ms
block_size 256 time 1.7441 ms
block_size 512 time 1.8092 ms
block_size 1024 time 2.0443 ms
```
|
2024-09-25 17:40:38 -07:00 |
|
|
Insop Song
|
3e5134d94f
|
Merge branch 'insop/llama3_wip' into insop/llama3
|
2024-09-25 15:53:17 -07:00 |
|
|
Insop Song
|
8874c2c3f6
|
Merge remote-tracking branch 'upstream/llama3' into insop/llama3
|
2024-09-25 15:50:51 -07:00 |
|
|
Andrej Karpathy
|
52c7254267
|
add the finished RoPE forward pass
|
2024-09-25 19:00:27 +00:00 |
|
|
Insop Song
|
de9c8170e5
|
wip
|
2024-09-23 20:52:50 -07:00 |
|
|
Insop Song
|
45401b42eb
|
DRAFT: Adding backward kernel for repkv
- [ ] WIP: CPU kernel
- [ ] Cuda kernel
|
2024-09-21 20:40:57 -07:00 |
|
|
Andrej Karpathy
|
56f956cc4a
|
add repkv kernel to replicate K,V heads after the QKV projection
|
2024-09-21 01:53:08 +00:00 |
|
|
Andrej Karpathy
|
1566d6b9f6
|
revert makefile change
|
2024-07-30 20:07:54 +00:00 |
|
|
mspronesti
|
a49da7d917
|
make: generalize TARGETS definition
|
2024-07-29 00:19:03 +01:00 |
|
|
indianspeedster
|
a80bce65e5
|
Added compilation steps in makefile for permute.cu
|
2024-07-25 18:02:06 -04:00 |
|
|
Ross Wheeler
|
5b95bff118
|
Dataloader test and ptx/sass file generator
Setup new test infrastructure
|
2024-06-21 17:23:03 -07:00 |
|
|
Erik Schultheis
|
875196b672
|
add c++17 flag also to dev/cuda
|
2024-06-06 19:02:45 +03:00 |
|
|
Erik Schultheis
|
d7a81ef26f
|
added a useful mixed precision utility for dev/cuda
|
2024-05-18 22:45:44 +03:00 |
|
|
ngc92
|
691c1df969
|
fused layernorm+residual
|
2024-05-09 23:48:11 +02:00 |
|
|
lancer
|
c81adeb6b2
|
move gelu_backward to backward block
|
2024-05-02 17:36:12 -07:00 |
|
|
Jane Illarionova
|
7746217433
|
update gelu backward rto allow all kernels to use both types
|
2024-05-01 23:36:59 +00:00 |
|
|
Erik Schultheis
|
b42db706b2
|
this kernel will be released later, sorry for the spoiler :)
|
2024-05-01 22:47:24 +03:00 |
|
|
Erik Schultheis
|
06980b6b2f
|
mixed precision utilities for dev/cuda
|
2024-05-01 22:35:55 +03:00 |
|
|
Andrej Karpathy
|
12da2c16cc
|
slight docs tweaks to dev/cuda
|
2024-04-27 14:40:57 +00:00 |
|
|
austinvhuang
|
25d703fbec
|
Use a make pattern rule to simplify Makefile, group kernels into forward/backward/updates/nccls, add some comments and update the README.md
|
2024-04-27 00:13:01 -04:00 |
|
|
austinvhuang
|
d937f93aaa
|
Add a local convenience Makefile for dev/cuda/, fix 2 minor warnings observed when building all targets.
|
2024-04-26 21:19:44 -04:00 |
|