 Aleksa Gordic
|
0f112f1b62
|
Fix CI complaining about default args
|
2024-06-15 19:39:05 +02:00 |
|
|
Aleksa Gordic
|
f3195d19af
|
Expose deviceProp as a global var
|
2024-06-15 17:40:14 +02:00 |
|
|
Aleksa Gordic
|
0f6853d399
|
Add global norm prod kernels to dev
|
2024-06-15 17:29:14 +02:00 |
|
|
Erik Schultheis
|
cb8cc25e3a
|
utilities for Packed and more vectorization
|
2024-05-27 13:15:36 +03:00 |
|
|
Andrej Karpathy
|
2d43e5bc97
|
remove legacy comment
|
2024-05-14 19:14:07 +00:00 |
|
|
Erik Schultheis
|
e553e2f084
|
update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results
|
2024-05-14 20:44:50 +03:00 |
|
|
ngc92
|
5920143d47
|
remove warning noise
|
2024-05-10 00:02:37 +02:00 |
|
|
ngc92
|
691c1df969
|
fused layernorm+residual
|
2024-05-09 23:48:11 +02:00 |
|
|
ngc92
|
99e765de8d
|
moved bf16 boilerplate to common.h
|
2024-05-09 11:19:24 +02:00 |
|
|
ademeure
|
1ea7f9bf25
|
tiny irrelevant optimisation to final unaligned fused_classifier loop + add missing common.h changes
|
2024-05-07 00:54:50 +01:00 |
|
|
Erik Schultheis
|
06980b6b2f
|
mixed precision utilities for dev/cuda
|
2024-05-01 22:35:55 +03:00 |
|
|
Andrej Karpathy
|
769c911ab6
|
small tweaks
|
2024-04-30 23:48:06 +00:00 |
|
|
Erik Schultheis
|
c3515cfe47
|
fixed potential error and generalized gelu forward
|
2024-04-30 23:45:34 +03:00 |
|
|
Andrej Karpathy
|
7ba1ed8a1a
|
delete packed128 .fp32, use (float) ok
|
2024-04-30 19:39:20 +00:00 |
|
|
Erik Schultheis
|
d7813d281d
|
clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results
|
2024-04-30 12:45:37 +03:00 |
|
|
Andrej Karpathy
|
af2bc47d22
|
remove forceinline. blame @arund42 if this ever does not get inlined by the compiler
|
2024-04-29 23:10:51 +00:00 |
|
|
Andrej Karpathy
|
5e12c66f5a
|
move packed128 to common
|
2024-04-29 22:55:28 +00:00 |
|
|
ademeure
|
49ffd68317
|
Fix sizeof(T) -> sizeof(D) (was overallocating memory)
|
2024-04-29 15:47:59 +01:00 |
|
|
ademeure
|
82fef03acd
|
Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)
|
2024-04-29 01:51:10 +01:00 |
|
|
Erik Schultheis
|
732a8b4806
|
added NaN-based makes for reference checks
|
2024-04-22 14:53:24 +03:00 |
|
|
Erik Schultheis
|
16123c3af8
|
towards an even better backward attention kernel
|
2024-04-19 19:12:30 +03:00 |
|
|
Andrej Karpathy
|
c393d828dc
|
size_t instead of int. everywhere. this will come to bite us at one point...
|
2024-04-19 02:11:30 +00:00 |
|
|
Andrej Karpathy
|
8a673edcc3
|
push my attention_backward file
|
2024-04-17 04:42:42 +00:00 |
|
|
Andrej Karpathy
|
6be8080501
|
add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far
|
2024-04-16 02:10:45 +00:00 |
|
|
Andrej Karpathy
|
c5ee371564
|
make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in common
|
2024-04-13 17:56:18 +00:00 |
|
|
Andrej Karpathy
|
84145411d3
|
fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nans
|
2024-04-13 17:47:19 +00:00 |
|
|
Erik Schultheis
|
f459bfa833
|
fixed re-initialization of device memory after test
|
2024-04-13 18:23:12 +03:00 |
|
|
Erik Schultheis
|
59bada94d9
|
improved testing
|
2024-04-13 17:06:52 +03:00 |
|
|
Erik Schultheis
|
7e8e038de8
|
extracted common testing functions into their own header
|
2024-04-13 13:00:05 +03:00 |
|