 Andrej Karpathy
|
ea3e2abdcf
|
small fixes and more comments very cool kernels
|
2024-04-18 23:48:39 +00:00 |
|
|
Andrej Karpathy
|
70170006ca
|
adjust readme with latest speed
|
2024-04-18 22:50:52 +00:00 |
|
|
Andrej Karpathy
|
6b603fa6c2
|
Merge branch 'ngc92-bwd-att-coarsened'
|
2024-04-18 22:48:24 +00:00 |
|
|
Andrej Karpathy
|
d95e62436b
|
Merge branch 'bwd-att-coarsened' of https://github.com/ngc92/llm.c into ngc92-bwd-att-coarsened
|
2024-04-18 22:47:24 +00:00 |
|
|
Erik Schultheis
|
0d55421f32
|
updated main training script
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
09d35a5e4d
|
splitting the loop to get rid of the indicator
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
6d98af1d32
|
another 15% by getting rid of ifs
|
2024-04-19 01:01:47 +03:00 |
|
|
Erik Schultheis
|
314661f516
|
significantly faster kernel by reusing data in registers
|
2024-04-19 01:01:47 +03:00 |
|
|
Andrej Karpathy
|
601e371e78
|
change MB to MiB when reporting capacity
|
2024-04-18 21:59:41 +00:00 |
|
|
Andrej Karpathy
|
5c8eaec359
|
report all big mallocs
|
2024-04-18 21:56:29 +00:00 |
|
|
Andrej Karpathy
|
6c0958eb79
|
Merge branch 'ngc92-memory-savings-2'
|
2024-04-18 21:43:02 +00:00 |
|
|
Andrej Karpathy
|
2981467415
|
delete a bunch of buffers that are not necessary in the backward pass, and also delete residual_backward
|
2024-04-18 21:42:21 +00:00 |
|
|
Erik Schultheis
|
ecf072f332
|
remove duplication for residuals. requires memset so we remain correct
|
2024-04-18 23:23:57 +03:00 |
|
|
Erik Schultheis
|
fab549bacb
|
removed further storage
|
2024-04-18 23:23:57 +03:00 |
|
|
Erik Schultheis
|
93db09577e
|
first step towards reduced memory usage in backwards
|
2024-04-18 23:23:57 +03:00 |
|
|
Erik Schultheis
|
60090dcf2e
|
update activation backward kernels that are not accumulated into the residual stream to overwrite instead of update.
|
2024-04-18 23:23:55 +03:00 |
|
|
Andrej Karpathy
|
de1e87158c
|
add comments to classifier_fused
|
2024-04-18 16:59:45 +00:00 |
|
|
Andrej
|
2d2f1df8f1
|
Merge pull request #150 from ademeure/optimised_fused_classifier
Optimised version of fused classifier + bugfixes(?)
|
2024-04-18 08:39:05 -07:00 |
|
|
Andrej Karpathy
|
eef92c3fbe
|
make comment on comparison to PyTorch runtime with kernels as they are right now
|
2024-04-18 04:53:09 +00:00 |
|
|
Andrej Karpathy
|
959ea543dc
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-18 04:40:09 +00:00 |
|
|
Andrej Karpathy
|
33a4985d19
|
change readme again to keep it up to date
|
2024-04-18 04:39:59 +00:00 |
|
|
Andrej
|
aed262d773
|
Merge pull request #172 from FeSens/fix/validate_result_attention
fix(attention_forward.cu): Fix the number of elements to check
|
2024-04-17 21:28:31 -07:00 |
|
|
Andrej Karpathy
|
ac183b3ba7
|
small tweaks for the test script. also i noticed that i had to bump 1e-2 -> 2e-2 for tolerance, i think we changed something very minor and tripped this threshold, but the tensors look ok
|
2024-04-18 04:27:13 +00:00 |
|
|
FeSens
|
0c09a37568
|
fix(attention_forward.cu): Fix the number of elements to check when verifying attention implementations.
This was causing me problems when I was trying to debug a new implementation on fewer tokens.
|
2024-04-18 01:23:05 -03:00 |
|
|
Andrej Karpathy
|
fdd6608281
|
turn on the full training by default, as things are fast enough. we're doing exactly 1 epoch of finetuning on shakespeare
|
2024-04-18 04:12:04 +00:00 |
|
|
lancer
|
a1abcbc32d
|
fix the bug that yields cpu, gpu results mismatch
|
2024-04-17 21:04:45 -07:00 |
|
|
Andrej
|
20187634ec
|
Merge pull request #167 from ngc92/bwd-att-coalesced
coleased memory reads for faster backward pass in attention
|
2024-04-17 20:50:33 -07:00 |
|
|
ademeure
|
557fecd6e9
|
bugfix for bounds checking
|
2024-04-18 04:45:07 +01:00 |
|
|
ademeure
|
2fd8ada4a6
|
Merge remote-tracking branch 'karpathy/master' into optimised_fused_classifier
|
2024-04-18 03:51:01 +01:00 |
|
|
ademeure
|
ea77a26473
|
mini fix
|
2024-04-18 03:42:34 +01:00 |
|
|
ademeure
|
3ba2cdc723
|
fixes + bounds checking
|
2024-04-18 03:41:27 +01:00 |
|
|
lancer
|
ac055f730c
|
fix the bug that yields cpu, gpu results mismatch
|
2024-04-17 17:47:11 -07:00 |
|
|
Erik Schultheis
|
58a7b57f96
|
enable benchmarking
|
2024-04-18 01:39:47 +03:00 |
|
|
Erik Schultheis
|
e278112c19
|
coalesced reading in attention backward pass
|
2024-04-18 01:39:43 +03:00 |
|
|
Andrej
|
4c27dd923b
|
Merge pull request #164 from lancerts/const-fix-ln_back
Constness fix layernorm_backward.cu
|
2024-04-17 14:59:04 -07:00 |
|
|
ademeure
|
4ee98f5135
|
Merge branch 'master' into optimised_fused_classifier
|
2024-04-17 22:58:51 +01:00 |
|
|
lancer
|
c327923df9
|
Constness fix layernorm_backward.cu
|
2024-04-17 14:56:03 -07:00 |
|
|
lancer
|
81aa596251
|
Constness fix layernorm_backward.cu
|
2024-04-17 14:54:49 -07:00 |
|
|
Andrej
|
e5d519c62b
|
Merge pull request #162 from ngc92/layernorm-bw
warp-wide reduction for layernorm backward
|
2024-04-17 13:34:36 -07:00 |
|
|
Erik Schultheis
|
c1bfe4a639
|
warp-wide reduction for layernorm backward
|
2024-04-17 21:52:40 +03:00 |
|
|
Andrej Karpathy
|
79586880b7
|
adjust readme now that the full training loop works in cud
|
2024-04-17 17:15:59 +00:00 |
|
|
Andrej Karpathy
|
fa4ebfa663
|
kernel2 for softmax backward
|
2024-04-17 08:14:22 +00:00 |
|
|
Andrej
|
2ecb4ec3ea
|
Merge pull request #161 from ngc92/att-bw-fixup
fix accumulation and load balancing
|
2024-04-17 01:12:35 -07:00 |
|
|
Erik Schultheis
|
0c16c2ce57
|
fix accumulation and load balancing
|
2024-04-17 10:59:20 +03:00 |
|
|
Andrej Karpathy
|
954077fb88
|
TRAINING WORKSgit add train_gpt2.cu! ITS SLOW BUT IT WORKS WOOT
|
2024-04-17 04:58:37 +00:00 |
|
|
Andrej Karpathy
|
d74d7ae652
|
small adamw fixes, needs more work
|
2024-04-17 04:58:16 +00:00 |
|
|
Andrej Karpathy
|
66a0b5242e
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-17 04:51:26 +00:00 |
|
|
Andrej
|
bfa69805f6
|
Merge pull request #159 from nopperl/adam
AdamW CUDA kernel
|
2024-04-16 21:48:14 -07:00 |
|
|
Andrej Karpathy
|
8a673edcc3
|
push my attention_backward file
|
2024-04-17 04:42:42 +00:00 |
|
|
Andrej Karpathy
|
3f24ac961d
|
check the entire gradient tensor all at once nice
|
2024-04-17 04:41:58 +00:00 |
|