 ademeure
|
7cbeefc7f3
|
added new layernorm backward to /dev/cuda/
|
2024-05-21 23:26:54 +01:00 |
|
|
Erik Schultheis
|
e553e2f084
|
update dev/cuda/layernorm_backward and improve validate_result to take into account fp epsilon when comparing results
|
2024-05-14 20:44:50 +03:00 |
|
|
Erik Schultheis
|
dd8c9f5ec9
|
fix layernorm backward: accumulate weight gradient
|
2024-05-14 20:43:51 +03:00 |
|
|
lancer
|
5c90845f7a
|
update the -lcublas -lcublasLt flag in the comment
|
2024-05-09 22:04:02 -07:00 |
|
|
ngc92
|
75ec629f52
|
remove duplicate function
|
2024-05-10 00:03:30 +02:00 |
|
|
ngc92
|
99e765de8d
|
moved bf16 boilerplate to common.h
|
2024-05-09 11:19:24 +02:00 |
|
|
ademeure
|
9b55ea8cf5
|
More crazy optimisations to layernorm_backward, fused_classifier, and matmul_backward_bias.
|
2024-05-07 00:11:42 +01:00 |
|
|
Andrej Karpathy
|
ec73086a83
|
Merge branch 'removing_cg_example' of https://github.com/ChrisDryden/llm.c into ChrisDryden-removing_cg_example
|
2024-05-02 18:14:14 +00:00 |
|
|
“Christopher
|
665d0a4f89
|
Added example of removing cooperative groups
|
2024-05-02 02:09:57 +00:00 |
|
|
Erik Schultheis
|
06980b6b2f
|
mixed precision utilities for dev/cuda
|
2024-05-01 22:35:55 +03:00 |
|
|
ademeure
|
1e50f3b976
|
Only enable kernel4 with BF16
|
2024-04-29 01:55:06 +01:00 |
|
|
ademeure
|
82fef03acd
|
Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)
|
2024-04-29 01:51:10 +01:00 |
|
|
austinvhuang
|
d937f93aaa
|
Add a local convenience Makefile for dev/cuda/, fix 2 minor warnings observed when building all targets.
|
2024-04-26 21:19:44 -04:00 |
|
|
Andrej Karpathy
|
e3bcae6b2b
|
incorporate layernorm backward kernel 2, we are now down to 73.1ms/iter on average
|
2024-04-22 18:44:43 +00:00 |
|
|
Chris Dryden
|
2752212151
|
Added shared memory for the atomic additions for the layernorm_backward.cu weights and biases
|
2024-04-21 16:37:08 -07:00 |
|
|
lancer
|
c327923df9
|
Constness fix layernorm_backward.cu
|
2024-04-17 14:56:03 -07:00 |
|
|
lancer
|
81aa596251
|
Constness fix layernorm_backward.cu
|
2024-04-17 14:54:49 -07:00 |
|
|
Erik Schultheis
|
c1bfe4a639
|
warp-wide reduction for layernorm backward
|
2024-04-17 21:52:40 +03:00 |
|
|
Andrej Karpathy
|
6be8080501
|
add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far
|
2024-04-16 02:10:45 +00:00 |
|