 Andrej Karpathy
|
7d945e994c
|
reshuffle repkv a bit, i wrote it from scratch. the kernel is still correct. repkv backward looks correct. rope backward is trivial so i don't see how it's not correct, and i also checked it. basically i'm really confused right now
|
2024-09-27 19:25:09 +00:00 |
|
|
Andrej Karpathy
|
8d49062293
|
add backward kernel to dev/cuda for rope, to ensure correctness. but i mean, it's trivial. this can't possibly be the issue. it must be the repkv
|
2024-09-27 18:07:10 +00:00 |
|
|
Andrej Karpathy
|
28e4a7f83e
|
small fixes, but still not too happy with this kernel, it wastes thread and more efficient implementation kernel2 is desireable and desired
|
2024-09-27 01:13:32 +00:00 |
|
|
Insop Song
|
a2b66f153a
|
Merge remote-tracking branch 'upstream/llama3' into insop/llama3
|
2024-09-26 11:42:31 -07:00 |
|
|
Andrej Karpathy
|
31be5e790d
|
add rmsnorm backward in dev/cuda, it seems to work surprisingly, and is probably ready to be integrated into llmc. we are still using 2X too much shared memory because I didn't want to change way too many things at the same time. I copy pasted our kernel10 of layernorm backward and made tweaks to it removing the bias and mean cool
|
2024-09-26 03:17:58 +00:00 |
|
|
Insop Song
|
d1f2f64541
|
Updated repkv_backward cuda kernel
- kernel 1 is tested
- build
```
make repkv_backward
/usr/local/cuda/bin/nvcc -O3 --use_fast_math --generate-code arch=compute_80,code=[compute_80,sm_80] -lcublas -lcublasLt -std=c++17 repkv_backward.cu -o repkv_backward
```
- test run on A30
```
Using kernel 1
Checking block size 32.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 64.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 128.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 256.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 512.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 1024.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
All results match. Starting benchmarks.
block_size 32 time 3.2461 ms
block_size 64 time 1.7509 ms
block_size 128 time 1.7374 ms
block_size 256 time 1.7441 ms
block_size 512 time 1.8092 ms
block_size 1024 time 2.0443 ms
```
|
2024-09-25 17:40:38 -07:00 |
|
|
Insop Song
|
3e5134d94f
|
Merge branch 'insop/llama3_wip' into insop/llama3
|
2024-09-25 15:53:17 -07:00 |
|
|
Insop Song
|
8874c2c3f6
|
Merge remote-tracking branch 'upstream/llama3' into insop/llama3
|
2024-09-25 15:50:51 -07:00 |
|
|
Andrej Karpathy
|
52c7254267
|
add the finished RoPE forward pass
|
2024-09-25 19:00:27 +00:00 |
|
|
Insop Song
|
de9c8170e5
|
wip
|
2024-09-23 20:52:50 -07:00 |
|
|
Insop Song
|
fcc3466b2f
|
Add minor change
|
2024-09-22 09:05:56 -07:00 |
|
|
Insop Song
|
42d09e8732
|
minor clean up
|
2024-09-22 00:26:29 -07:00 |
|
|
Insop Song
|
ad46043aaf
|
WIP updating cuda kernel
|
2024-09-22 00:22:26 -07:00 |
|
|
Insop Song
|
6c68657c6b
|
Put cuda kernel caller placeholder
|
2024-09-21 21:30:45 -07:00 |
|
|
Insop Song
|
080e57fd46
|
CPU version tested
- [ ] WIP cuda version
|
2024-09-21 21:21:13 -07:00 |
|
|
Insop Song
|
45401b42eb
|
DRAFT: Adding backward kernel for repkv
- [ ] WIP: CPU kernel
- [ ] Cuda kernel
|
2024-09-21 20:40:57 -07:00 |
|
|
Andrej Karpathy
|
56f956cc4a
|
add repkv kernel to replicate K,V heads after the QKV projection
|
2024-09-21 01:53:08 +00:00 |
|
|
Andrej Karpathy
|
1566d6b9f6
|
revert makefile change
|
2024-07-30 20:07:54 +00:00 |
|
|
mspronesti
|
574ef9b87d
|
chore(permute): use common.h utils for validation and benchmarking
|
2024-07-29 00:19:31 +01:00 |
|
|
mspronesti
|
a49da7d917
|
make: generalize TARGETS definition
|
2024-07-29 00:19:03 +01:00 |
|
|
Andrej
|
b4623bc5e7
|
Merge pull request #712 from indianspeedster/dev-cuda
Added permute kernel in dev/cuda
|
2024-07-28 06:08:20 -10:00 |
|
|
mspronesti
|
c6e74ea322
|
fix(dev/cuda): memory leaks
|
2024-07-26 00:33:51 +01:00 |
|
|
indianspeedster
|
a80bce65e5
|
Added compilation steps in makefile for permute.cu
|
2024-07-25 18:02:06 -04:00 |
|
|
indianspeedster
|
8952c524af
|
added kernel to permute in dev/cuda
|
2024-07-25 17:35:29 -04:00 |
|
|
Ross Wheeler
|
5b95bff118
|
Dataloader test and ptx/sass file generator
Setup new test infrastructure
|
2024-06-21 17:23:03 -07:00 |
|
|
Andrej
|
f104009bc7
|
Merge pull request #600 from gordicaleksa/add_faster_layernorm_kernel
Use faster kernel for LayerNorm forward
|
2024-06-17 14:19:22 -07:00 |
|
|
Aleksa Gordic
|
9b5ec8f719
|
Set v memory to random floats >=0
|
2024-06-17 17:44:00 +02:00 |
|
|
Aleksa Gordic
|
a0a8795be1
|
Fix guarding logic, refactor
|
2024-06-17 14:12:05 +02:00 |
|
|
Aleksa Gordic
|
c6040eef5f
|
Even better kernel - modification of what we use for fused residual fwd
|
2024-06-17 09:53:58 +02:00 |
|
|
Andrej
|
13c893ffec
|
Merge pull request #561 from lancerts/fix-warning
Fix the compiler warnings and errors
|
2024-06-16 09:25:51 -07:00 |
|
|
Andrej
|
1973631c6b
|
Merge pull request #583 from vyom1611/FLT_MAX-fix
Replaced hard-coded max float with FLT_MAX
|
2024-06-16 09:25:03 -07:00 |
|
|
Aleksa Gordic
|
d5601d16da
|
Fix stuff in layernorm forward
|
2024-06-16 09:34:01 +02:00 |
|
|
Aleksa Gordic
|
d003274b7c
|
Remove warp size - moved to common.h
|
2024-06-15 20:23:59 +02:00 |
|
|
Aleksa Gordic
|
09910acc0b
|
Remove blockreduce from classifier fused, move to common.h
|
2024-06-15 19:55:11 +02:00 |
|
|
Aleksa Gordic
|
0f112f1b62
|
Fix CI complaining about default args
|
2024-06-15 19:39:05 +02:00 |
|
|
Aleksa Gordic
|
f3195d19af
|
Expose deviceProp as a global var
|
2024-06-15 17:40:14 +02:00 |
|
|
Aleksa Gordic
|
0f6853d399
|
Add global norm prod kernels to dev
|
2024-06-15 17:29:14 +02:00 |
|
|
Vyom Sharma
|
ee4b056ddf
|
Merge branch 'karpathy:master' into FLT_MAX-fix
|
2024-06-14 16:32:37 +05:30 |
|
|
vyom1611
|
e2a18d9385
|
Fixed missing semi-colon
|
2024-06-13 00:59:15 +05:30 |
|
|
vyom1611
|
dc8d8de06a
|
Replaced hard-coded max float with FLT_MAX
|
2024-06-13 00:56:53 +05:30 |
|
|
Vyom Sharma
|
afa7978227
|
Merge branch 'karpathy:master' into master
|
2024-06-13 00:19:16 +05:30 |
|
|
Erik Schultheis
|
d6793645f5
|
replace cublaslt with custom matmul
|
2024-06-12 09:42:19 +03:00 |
|
|
Erik Schultheis
|
7f3f3acf16
|
shared memory matmul
|
2024-06-11 21:03:43 +03:00 |
|
|
Erik Schultheis
|
071c977c79
|
vectorized memory access
|
2024-06-11 00:46:51 +03:00 |
|
|
Erik Schultheis
|
bb89efdcb1
|
register reuse
|
2024-06-11 00:41:08 +03:00 |
|
|
Vyom Sharma
|
8a3a29ecb7
|
Merge branch 'karpathy:master' into master
|
2024-06-10 13:54:29 +05:30 |
|
|
Andrej
|
637c1b69c4
|
Merge pull request #552 from karpathy/feature/streams
Feature/streams
|
2024-06-08 09:25:56 -07:00 |
|
|
Andrej Karpathy
|
ee6b3c9325
|
take out Async copies and memsets
|
2024-06-08 16:17:02 +00:00 |
|
|
lancerts
|
089fdd7f23
|
Merge branch 'master' into fix-warning
|
2024-06-07 07:17:52 -07:00 |
|
|
vyom1611
|
ecb13ea74d
|
missing parameter in comments added
|
2024-06-07 16:14:59 +05:30 |
|