0d55421f32
updated main training script
Erik Schultheis
2024-04-18 12:38:44 +03:00
09d35a5e4d
splitting the loop to get rid of the indicator
Erik Schultheis
2024-04-18 12:31:54 +03:00
6d98af1d32
another 15% by getting rid of ifs
Erik Schultheis
2024-04-18 12:14:58 +03:00
314661f516
significantly faster kernel by reusing data in registers
Erik Schultheis
2024-04-18 03:19:12 +03:00
601e371e78
change MB to MiB when reporting capacity
Andrej Karpathy
2024-04-18 21:59:41 +00:00
5c8eaec359
report all big mallocs
Andrej Karpathy
2024-04-18 21:56:29 +00:00
6c0958eb79
Merge branch 'ngc92-memory-savings-2'
Andrej Karpathy
2024-04-18 21:43:02 +00:00
2981467415
delete a bunch of buffers that are not necessary in the backward pass, and also delete residual_backward
Andrej Karpathy
2024-04-18 21:42:21 +00:00
ecf072f332
remove duplication for residuals. requires memset so we remain correct
Erik Schultheis
2024-04-18 00:12:56 +03:00
fab549bacb
removed further storage
Erik Schultheis
2024-04-18 00:02:02 +03:00
93db09577e
first step towards reduced memory usage in backwards
Erik Schultheis
2024-04-17 23:21:26 +03:00
60090dcf2e
update activation backward kernels that are not accumulated into the residual stream to overwrite instead of update.
Erik Schultheis
2024-04-17 23:05:02 +03:00
de1e87158c
add comments to classifier_fused
Andrej Karpathy
2024-04-18 16:59:45 +00:00
2d2f1df8f1
Merge pull request #150 from ademeure/optimised_fused_classifier
Andrej
2024-04-18 08:39:05 -07:00
eef92c3fbe
make comment on comparison to PyTorch runtime with kernels as they are right now
Andrej Karpathy
2024-04-18 04:53:09 +00:00
959ea543dc
Merge branch 'master' of github.com:karpathy/llm.c
Andrej Karpathy
2024-04-18 04:40:09 +00:00
33a4985d19
change readme again to keep it up to date
Andrej Karpathy
2024-04-18 04:39:59 +00:00
aed262d773
Merge pull request #172 from FeSens/fix/validate_result_attention
Andrej
2024-04-17 21:28:31 -07:00
ac183b3ba7
small tweaks for the test script. also i noticed that i had to bump 1e-2 -> 2e-2 for tolerance, i think we changed something very minor and tripped this threshold, but the tensors look ok
Andrej Karpathy
2024-04-18 04:27:13 +00:00
0c09a37568
fix(attention_forward.cu): Fix the number of elements to check when verifying attention implementations.
FeSens
2024-04-18 01:23:05 -03:00
fdd6608281
turn on the full training by default, as things are fast enough. we're doing exactly 1 epoch of finetuning on shakespeare
Andrej Karpathy
2024-04-18 04:12:04 +00:00
a1abcbc32d
fix the bug that yields cpu, gpu results mismatch
lancer
2024-04-17 21:04:45 -07:00
20187634ec
Merge pull request #167 from ngc92/bwd-att-coalesced
Andrej
2024-04-17 20:50:33 -07:00
557fecd6e9
bugfix for bounds checking
ademeure
2024-04-18 04:45:07 +01:00
b225501b7b
optimised version of the cool new fused classifier + hopefully bugfixes?
ademeure
2024-04-16 14:54:23 +01:00
66c30fb55f
-fno-finite-math-only for almost 2x speed up
ent0n29
2024-04-16 12:49:54 +02:00
67e0e6d6d4
diligently cudaCheck last error after every kernel launch nice
Andrej Karpathy
2024-04-16 06:21:35 +00:00
2517fe7492
exit(1) -> exit(EXIT_FAILURE), more portable and readable
Andrej Karpathy
2024-04-16 06:12:23 +00:00
de0af15576
add mallocCheck as well. is this excessive? hmm. seem ok
Andrej Karpathy
2024-04-16 06:04:51 +00:00
062ba5a30e
experiment with nice macros that do error handling, fopenCheck, freadCheck, fcloseCheck, to get rid of warning errors during compile. this functionality is yet another example of something we want to make common to both the .c file and the .cu file. we have to do that refactor soon. for now just keeping it in the .cu file
Andrej Karpathy
2024-04-16 05:40:37 +00:00
df010fe5b5
Merge pull request #145 from lancerts/matmul_backward
Andrej
2024-04-15 20:32:16 -07:00
6be8080501
add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far
Andrej Karpathy
2024-04-16 02:10:45 +00:00
3de507739c
minor fixes to matmul backward
Andrej Karpathy
2024-04-16 01:23:41 +00:00
9560b1a92b
add matmul backward kernel, problem is the bias backward, ironically. which i thought i could cublas at first
Andrej Karpathy
2024-04-16 00:40:11 +00:00
27ccf0b595
fix bugs, must use size_t instead of int, this was bad. and also init cpu_losses to NULL if we want to free it later
Andrej Karpathy
2024-04-15 22:20:44 +00:00
3c6dafeb53
Merge pull request #117 from ngc92/fused-classifier
Andrej
2024-04-15 12:13:48 -07:00
a44b0ce0f0
Merge pull request #142 from dagelf/master
Andrej
2024-04-15 12:01:25 -07:00
480b638606
few more words on what i want the repo to be atm
Andrej Karpathy
2024-04-15 18:55:50 +00:00
752cbe068d
Update Makefile with -march=native
Coenraad Loubser
2024-04-15 19:52:19 +02:00
cc5931bfdb
add clarification on notable forks to readme
Andrej Karpathy
2024-04-15 03:43:43 +00:00
82761ff29b
confirm that linux and macos disagree, putting in the macos version as before
Andrej Karpathy
2024-04-15 02:38:23 +00:00
bcb5228ab9
also update the .c file in addition to .cu with recent changes, e.g. the tokenizer and the B*T sampling, and update the README to match
Andrej Karpathy
2024-04-15 02:32:26 +00:00
3652832073
update the .c code, not just .cu code. oh copy paste, starting to bite us haha. also fflush when printing tokens
Andrej Karpathy
2024-04-15 02:13:49 +00:00
be8ace4be6
Merge pull request #128 from ngc92/test-updates
Andrej
2024-04-14 19:00:26 -07:00
c165855cd3
add optional support for the tokenizer. people have to re-run train_gpt2.py to get this feature. later, we will change the code to demand this and error if it is not found, for now i just don't want to brick people who do a pull. also i only modified the train_gpt2.cu for now, not the .c. it might be time to separate out the dataloader, the tokenizer, and any other common utilities to their own files (?). otherwise it feels a bit silly to copy paste stuff around? not sure
Andrej Karpathy
2024-04-15 01:57:04 +00:00
c73fa8842a
fixed numerics for online kernel, and an existing race condition in kernel 2
Erik Schultheis
2024-04-15 03:41:09 +03:00
f4cfd78245
only call gpt2_forward with the exact same sizing configuration B,T right now for safety. We can later bring back the <= B, <=T forward pass, but we need to do it carefully and have tests that make sure that a strictly smaller configuration produces the exact same results for that chunk of b,t. In other words we'd want ot make sure that the entire forward pass is range invariant. Currently it is not only because of the attention kernel. I think this is fixable, not too difficult, but it requires careful thought and associated tests for ensuring the range invariance. When those tests pass we can bring back old and more efficient behavior. For now it's just causing bugs, so I am putting in this highly defensive, but a lot more likely correct code
Andrej Karpathy
2024-04-14 23:56:52 +00:00
8fb966bc31
kernel 1 should also accumulate in double to ensure stability
Erik Schultheis
2024-04-15 02:52:38 +03:00
60cf281e46
use V dimension in softmax tests adjusted tests to be more challenging
Erik Schultheis
2024-04-14 14:56:51 +03:00
99b42c1a94
nicer printing
Erik Schultheis
2024-04-14 14:56:31 +03:00
fe1d8cfb47
fix timing output
Erik Schultheis
2024-04-14 13:15:32 +03:00
9807c041ee
reshuffle the validation code a bit and veirfy all inputs/targets are in [0, V)
Andrej Karpathy
2024-04-14 19:30:56 +00:00
Andrej Karpathy