fd6882a1e3
encoder backward really naive, just to get this finished full
2024-04-17 04:39:02 +00:00
a3a3b82d82
also backward the attention, that was not easy lol
2024-04-17 04:23:08 +00:00
97e61e2258
add slightly more optimized adamw kernel
2024-04-17 03:31:07 +02:00
5684ff7e77
add adamw cuda kernel
2024-04-17 03:30:59 +02:00
9e71af9678
Merge branch 'master' of github.com:karpathy/llm.c
2024-04-16 20:36:42 +00:00
45b0df0d3a
backward now up to and not including attention
2024-04-16 20:34:13 +00:00
e96f0ee2c8
backward the layernorm
2024-04-16 20:31:21 +00:00
bf4ac57adc
backward the gelu and the matmul after
2024-04-16 20:29:08 +00:00
d753b1872e
clean up checks a bit in test script
2024-04-16 20:16:26 +00:00
0da60460e9
backprop through the first residual and final matmul of the final residual block
2024-04-16 20:12:17 +00:00
136875fa5c
Merge pull request #149 from ent0n29/master
...
-fno-finite-math-only for almost 2x speed up, fix for [#19 ]
2024-04-16 12:42:21 -07:00
e15691ad16
Merge pull request #151 from msharmavikram/fix-out-of-resource
...
simple fix for bug [#106 ]
2024-04-16 12:35:52 -07:00
862852fe9d
Merge pull request #152 from ngc92/fused-fixes
...
ensure valid test cases and incorporate fixes by @ademeure
2024-04-16 12:31:54 -07:00
43bf9eb66a
ensure valid test cases and incorporate fixes by @ademeure
2024-04-16 20:53:50 +03:00
c62b50015f
simple fix for bug https://github.com/karpathy/llm.c/issues/106
2024-04-16 10:23:48 -05:00
b225501b7b
optimised version of the cool new fused classifier + hopefully bugfixes?
2024-04-16 14:54:23 +01:00
66c30fb55f
-fno-finite-math-only for almost 2x speed up
2024-04-16 12:49:54 +02:00
67e0e6d6d4
diligently cudaCheck last error after every kernel launch nice
2024-04-16 06:21:35 +00:00
2517fe7492
exit(1) -> exit(EXIT_FAILURE), more portable and readable
2024-04-16 06:12:23 +00:00
de0af15576
add mallocCheck as well. is this excessive? hmm. seem ok
2024-04-16 06:04:51 +00:00
062ba5a30e
experiment with nice macros that do error handling, fopenCheck, freadCheck, fcloseCheck, to get rid of warning errors during compile. this functionality is yet another example of something we want to make common to both the .c file and the .cu file. we have to do that refactor soon. for now just keeping it in the .cu file
2024-04-16 05:40:37 +00:00
df010fe5b5
Merge pull request #145 from lancerts/matmul_backward
...
Fix the backward dbias precision issue in matmul_backward
2024-04-15 20:32:16 -07:00
6ff4ad646d
fix the backward dbias precision issue
2024-04-15 19:58:16 -07:00
6be8080501
add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so far
2024-04-16 02:10:45 +00:00
3de507739c
minor fixes to matmul backward
2024-04-16 01:23:41 +00:00
9560b1a92b
add matmul backward kernel, problem is the bias backward, ironically. which i thought i could cublas at first
2024-04-16 00:40:11 +00:00
27ccf0b595
fix bugs, must use size_t instead of int, this was bad. and also init cpu_losses to NULL if we want to free it later
2024-04-15 22:20:44 +00:00
3c6dafeb53
Merge pull request #117 from ngc92/fused-classifier
...
WIP: Fully fused classification layer
2024-04-15 12:13:48 -07:00
a44b0ce0f0
Merge pull request #142 from dagelf/master
...
Update Makefile with -march=native for 25%+ CPU speedup
2024-04-15 12:01:25 -07:00
480b638606
few more words on what i want the repo to be atm
2024-04-15 18:55:50 +00:00
9aa4efdcbe
Merge remote-tracking branch 'refs/remotes/origin/master'
2024-04-15 20:33:03 +02:00
ecde56c1c8
Add -march=native to README
2024-04-15 20:32:27 +02:00
7064e36472
more original typos fixed
2024-04-15 20:22:33 +02:00
c75183d318
typo
2024-04-15 20:04:55 +02:00
752cbe068d
Update Makefile with -march=native
...
-march=native Results in a 30% speedup on all the platforms I've tried, for train_gpt2 (Admittedly, all older Intel and AMD ones)
```
model name : Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz
step 0: train loss 5.356172 (took 15197.876634 ms)
vs
step 0: train loss 5.356185 (took 10418.548668 ms)
model name : AMD Ryzen 5 3600 6-Core Processor
step 0: train loss 5.356173 (took 3599.359234 ms)
vs
step 0: train loss 5.356185 (took 2708.045790 ms)
```
2024-04-15 19:52:19 +02:00
cc5931bfdb
add clarification on notable forks to readme
2024-04-15 03:43:43 +00:00
82761ff29b
confirm that linux and macos disagree, putting in the macos version as before
2024-04-15 02:38:23 +00:00
bcb5228ab9
also update the .c file in addition to .cu with recent changes, e.g. the tokenizer and the B*T sampling, and update the README to match
2024-04-15 02:32:26 +00:00
3652832073
update the .c code, not just .cu code. oh copy paste, starting to bite us haha. also fflush when printing tokens
2024-04-15 02:13:49 +00:00
be8ace4be6
Merge pull request #128 from ngc92/test-updates
...
Test updates
2024-04-14 19:00:26 -07:00
c165855cd3
add optional support for the tokenizer. people have to re-run train_gpt2.py to get this feature. later, we will change the code to demand this and error if it is not found, for now i just don't want to brick people who do a pull. also i only modified the train_gpt2.cu for now, not the .c. it might be time to separate out the dataloader, the tokenizer, and any other common utilities to their own files (?). otherwise it feels a bit silly to copy paste stuff around? not sure
2024-04-15 01:57:04 +00:00
c73fa8842a
fixed numerics for online kernel, and an existing race condition in kernel 2
2024-04-15 03:41:09 +03:00
f4cfd78245
only call gpt2_forward with the exact same sizing configuration B,T right now for safety. We can later bring back the <= B, <=T forward pass, but we need to do it carefully and have tests that make sure that a strictly smaller configuration produces the exact same results for that chunk of b,t. In other words we'd want ot make sure that the entire forward pass is range invariant. Currently it is not only because of the attention kernel. I think this is fixable, not too difficult, but it requires careful thought and associated tests for ensuring the range invariance. When those tests pass we can bring back old and more efficient behavior. For now it's just causing bugs, so I am putting in this highly defensive, but a lot more likely correct code
2024-04-14 23:56:52 +00:00
8fb966bc31
kernel 1 should also accumulate in double to ensure stability
2024-04-15 02:52:38 +03:00
60cf281e46
use V dimension in softmax tests
...
adjusted tests to be more challenging
2024-04-15 02:14:48 +03:00
99b42c1a94
nicer printing
2024-04-15 02:03:48 +03:00
fe1d8cfb47
fix timing output
2024-04-15 02:03:42 +03:00
9807c041ee
reshuffle the validation code a bit and veirfy all inputs/targets are in [0, V)
2024-04-14 19:30:56 +00:00
312b043697
Merge pull request #122 from ngc92/fix-illegal-access
...
fix #114
2024-04-14 12:19:03 -07:00
5c9d61fd35
Prevent out-of-bounds token-ids in input padding for test generation
2024-04-14 13:05:57 +03:00
Andrej Karpathy