 Andrej Karpathy
|
001ad78fe1
|
resolve deviceProp and cudaCheck dupes
|
2024-06-03 19:54:32 +00:00 |
|
|
Andrej Karpathy
|
020b15ad77
|
small more fixes
|
2024-06-03 19:32:32 +00:00 |
|
|
Andrej Karpathy
|
5d9403479c
|
some fixes, still not done
|
2024-06-03 18:50:41 +00:00 |
|
|
Andrej Karpathy
|
804d0693c8
|
refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well
|
2024-06-03 18:10:33 +00:00 |
|
|
Andrej
|
e40268457f
|
Merge pull request #526 from karpathy/try/move_cudnn_to_llmc
attempt to move cudnn attention to the llmc directory
|
2024-06-03 09:32:31 -07:00 |
|
|
Andrej Karpathy
|
6434009298
|
attempt to remove PATH_SEP from Makefile
|
2024-06-03 15:30:45 +00:00 |
|
|
Andrej Karpathy
|
41dc65a0ef
|
attempt to move cudnn attention to the llmc directory
|
2024-06-03 03:23:04 +00:00 |
|
|
Andrej Karpathy
|
08fc45b492
|
delete copy paste code for gpt2 model init
|
2024-06-03 02:08:29 +00:00 |
|
|
Andrej Karpathy
|
c827dd2f8e
|
Merge branch 'ngc92-mfu'
|
2024-06-03 01:41:31 +00:00 |
|
|
Andrej Karpathy
|
1ac30591f2
|
add new file llmc/mfu.h for MFU calculations for different GPUs
|
2024-06-03 01:41:18 +00:00 |
|
|
Andrej
|
80e0c3b8d9
|
Merge pull request #512 from gordicaleksa/refactor_encoder_bwd_kernel
Remove redundant CPU computation in encoder bwd
|
2024-06-02 16:43:10 -07:00 |
|
|
Andrej
|
cb5eff0578
|
Merge pull request #514 from ngc92/compile-fix
fix compilation with older nvcc
|
2024-06-02 16:42:31 -07:00 |
|
|
Andrej
|
ff98d8bda7
|
Merge pull request #515 from gordicaleksa/fix_adamw_kernel3_num_threads
Use local params for num blocks in adamw_kernel3
|
2024-06-02 16:41:59 -07:00 |
|
|
Andrej
|
1ce30fdc4a
|
Merge pull request #516 from gordicaleksa/update_random_doc
Add clarification on Box-Muller
|
2024-06-02 16:37:37 -07:00 |
|
|
Andrej
|
252584954e
|
Merge pull request #519 from gordicaleksa/mem_leak
Fix mem leak
|
2024-06-02 16:34:54 -07:00 |
|
|
Andrej Karpathy
|
1e893f4e11
|
Merge branch 'gordicaleksa-fix_zero_stage1'
|
2024-06-02 23:28:22 +00:00 |
|
|
Andrej Karpathy
|
4944ac5743
|
adjust comments
|
2024-06-02 23:28:13 +00:00 |
|
|
Aleksa Gordic
|
a8a5a444bc
|
Add ; - ah c my old friend :)
|
2024-06-02 23:04:47 +02:00 |
|
|
Aleksa Gordic
|
9a455edc25
|
Improve comments
|
2024-06-02 23:03:41 +02:00 |
|
|
Aleksa Gordic
|
b4e8bd7f66
|
Fix zero stage 1 norm computation - WIP
|
2024-06-02 22:45:26 +02:00 |
|
|
Aleksa Gordic
|
df0a826ef3
|
Close file
|
2024-06-02 21:37:52 +02:00 |
|
|
Aleksa Gordic
|
30d81cce15
|
Fix mem leak
|
2024-06-02 21:28:42 +02:00 |
|
|
Aleksa Gordic
|
79cbc03ccf
|
Use local params for num blocks
|
2024-06-02 15:06:06 +02:00 |
|
|
Aleksa Gordic
|
ae9c957273
|
Add clarification on Box-Muller
|
2024-06-02 15:04:45 +02:00 |
|
|
Erik Schultheis
|
a902323f36
|
fix compilation with older nvcc
|
2024-06-02 12:40:28 +03:00 |
|
|
Aleksa Gordic
|
290c00a362
|
Remove redundant CPU computation
|
2024-06-01 18:27:24 +02:00 |
|
|
Andrej
|
a26041af12
|
Merge pull request #495 from ChrisDryden/shared_memory
Removed unnecesary shared memory due to blockreduce using static defined shared memory
|
2024-06-01 08:12:21 -07:00 |
|
|
Andrej
|
9cf8c2f135
|
Merge pull request #509 from ngc92/inplace
Inplace gelu backward
|
2024-06-01 08:10:56 -07:00 |
|
|
Andrej
|
bd0f036ee5
|
Merge pull request #511 from gordicaleksa/gordicaleksa_fix_docs
Update compile cmd in the dev/cuda README
|
2024-06-01 08:09:31 -07:00 |
|
|
Aleksa Gordic
|
8eafd40615
|
Update compile cmd in the dev/cuda README
|
2024-06-01 16:46:32 +02:00 |
|
|
Erik Schultheis
|
761fb2cf2b
|
rename gelu_backward to gelu_backward_inplace, and made it obvious in the code that this is an inplace operation
|
2024-06-01 14:31:26 +03:00 |
|
|
Erik Schultheis
|
cc4bed0806
|
added missing checks
|
2024-06-01 14:30:40 +03:00 |
|
|
Andrej Karpathy
|
9c8edaffa7
|
fix recompute bug
|
2024-05-31 22:38:03 +00:00 |
|
|
Andrej Karpathy
|
0bc24c36e6
|
fix bug in resumption of dataloader, have to reload shard
|
2024-05-31 22:19:37 +00:00 |
|
|
Andrej Karpathy
|
1de3bef0ac
|
fix argparse comment too
|
2024-05-31 22:06:48 +00:00 |
|
|
Andrej Karpathy
|
60a1037db9
|
Merge branch 'ChrisDryden-layernorm_backwards_mem_saving'
|
2024-05-31 22:04:44 +00:00 |
|
|
Andrej Karpathy
|
f4874a9f63
|
make things a bit more consistent, keep recompute 1 as default, and add a comment
|
2024-05-31 22:04:34 +00:00 |
|
|
Andrej Karpathy
|
629a24020d
|
Merge branch 'layernorm_backwards_mem_saving' of https://github.com/ChrisDryden/llm.c into ChrisDryden-layernorm_backwards_mem_saving
|
2024-05-31 21:47:28 +00:00 |
|
|
Andrej Karpathy
|
ed37d9261b
|
fix bug in readme
|
2024-05-30 21:39:47 +00:00 |
|
|
Andrej Karpathy
|
b48224cb0c
|
refactor the README file because it was getting crazy
|
2024-05-30 21:39:06 +00:00 |
|
|
Andrej Karpathy
|
0fddf06043
|
more docs on scripts
|
2024-05-30 21:06:36 +00:00 |
|
|
Andrej Karpathy
|
fe36e40f2c
|
also move pyrun script to scripts/
|
2024-05-30 20:17:16 +00:00 |
|
|
Andrej
|
d98d3ed82b
|
code refactor, introduce llmc dir and scripts dir
The main file train_gpt2.cu is getting out of control. So here we are.
|
2024-05-30 13:13:04 -07:00 |
|
|
Andrej Karpathy
|
22d1607f6d
|
delete these files, they are now in scripts/ folder and better
|
2024-05-30 20:11:59 +00:00 |
|
|
Andrej Karpathy
|
fc40ffc4aa
|
adjust readme to new scripts/ dir
|
2024-05-30 20:10:04 +00:00 |
|
|
Andrej Karpathy
|
a0d5dbbda6
|
scripts folder
|
2024-05-30 20:00:07 +00:00 |
|
|
Andrej Karpathy
|
35aa110244
|
more refactor
|
2024-05-30 19:39:29 +00:00 |
|
|
Andrej Karpathy
|
f8f9e49ba3
|
Merge branch 'master' into feature/libdir
|
2024-05-30 19:07:51 +00:00 |
|
|
Andrej Karpathy
|
5f547601a0
|
further hide the c pytorch bridge logic
|
2024-05-30 18:20:29 +00:00 |
|
|
Andrej Karpathy
|
ca27eb8fa5
|
bugfix don't train on val batch
|
2024-05-30 18:18:05 +00:00 |
|