3221e4b2d2
small cosmetic changes
2024-05-24 23:10:17 +00:00
6e4296fb9a
Merge branch 'cpu-matmul' of https://github.com/ngc92/llm.c into ngc92-cpu-matmul
2024-05-24 22:59:24 +00:00
967420d1d1
fix print format warning for size_t vs int
2024-05-21 19:30:02 +00:00
666145e7b8
also fix the .c file and fp32 file
2024-05-21 17:30:56 +00:00
f671cf9288
more changes, trying to help people out because when this merges to master it will brick everyone's code...
2024-05-20 23:02:41 +00:00
722e5b2fe5
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scale
2024-05-20 22:43:40 +00:00
16f9dad301
Update train_gpt2.c
2024-05-14 17:45:08 -07:00
7de50af511
train_gpt.c: Removing the hardcoded GPT2_EOT
2024-05-14 17:38:09 -07:00
b2a5508b84
constness fixes
2024-05-14 22:48:59 +03:00
3b5933ecfb
considerably speed up CPU matmul while still keeping it relatively readable
2024-05-14 22:32:24 +03:00
f4f7a987e7
Merge pull request #324 from eymay/init_weights_from_scratch
...
Refactoring parameter size filling
2024-05-01 14:50:39 -07:00
f02ba4b436
import updated tokenizer properly
2024-04-29 23:02:26 +00:00
0e38ac7b8c
Refactoring parameter size filling
...
Similar to a new function is defined for parameter size table. This will be useful for reuse with e.g. initializing weights from scratch instead of a checkpoint.
2024-04-29 12:35:21 +03:00
c20497cfc1
add comments pointing to the definition of the utils functions
2024-04-28 23:26:58 +00:00
4a3c278f3a
moved checked helper functions into a separate file
2024-04-29 00:00:37 +03:00
b7972ff928
make padded vocab fixes in the .c code as well, i missed it in the previous PR, should satisfy the CI now
2024-04-28 19:33:21 +00:00
9c2209254b
also improve situation in the c code
2024-04-26 22:27:22 +00:00
2efac6670a
Make msvc compile by adding the ifdefs back in gelu_backward, fixes #200
2024-04-22 23:17:06 +02:00
734f65f33e
Merge pull request #192 from rosslwheeler/const_fixes
...
const changes to help with Microsoft compiler OpenMP 5.2 support
2024-04-21 10:42:15 -07:00
adac21e97b
add a small comment as to what this is
2024-04-20 19:45:34 +00:00
f534e4bdfe
Fixes -Ofast optimizations breaking model by skipping them for gelu_backward
2024-04-20 21:33:00 +02:00
e564222113
const changes to support OpenMP 5.2 MSVC work-around
...
const changes to support OpenMP MSVC work-around
2024-04-20 02:07:12 -07:00
5bdbc45806
only do 5 validation batches by default, we really don't need much precision here for this little script with current defaults
2024-04-20 00:45:13 +00:00
27ccf0b595
fix bugs, must use size_t instead of int, this was bad. and also init cpu_losses to NULL if we want to free it later
2024-04-15 22:20:44 +00:00
7064e36472
more original typos fixed
2024-04-15 20:22:33 +02:00
bcb5228ab9
also update the .c file in addition to .cu with recent changes, e.g. the tokenizer and the B*T sampling, and update the README to match
2024-04-15 02:32:26 +00:00
3652832073
update the .c code, not just .cu code. oh copy paste, starting to bite us haha. also fflush when printing tokens
2024-04-15 02:13:49 +00:00
dec54904b4
one more simple clarifying comment
2024-04-11 22:24:38 +00:00
d680fbad25
one more comment on attention
2024-04-11 21:49:14 +00:00
9710163a60
did one pass of additional comments, potentially helpful, as i think a lot of people may be looking at this for the first time
2024-04-11 21:45:46 +00:00
1751af1607
inline docs for sizes
2024-04-11 16:04:14 +00:00
f26cf00a61
make gelu constant be a #define, speeds up the kernel by 1% or so
2024-04-11 15:49:01 +00:00
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
3bcb9ba7d2
remove unused variables
2024-04-08 15:10:51 -07:00
b2228d0e83
Minor correction for openmp check
2024-04-08 22:33:47 +02:00
e8e1628632
first commit of just the reference cpu fp32 gpt2 training
2024-04-08 12:41:21 -07:00
Andrej Karpathy