 Andrej Karpathy
|
ee6b3c9325
|
take out Async copies and memsets
|
2024-06-08 16:17:02 +00:00 |
|
|
Andrej Karpathy
|
6d14c29df5
|
streams, everything right now is on main stream, almost
|
2024-06-05 01:01:24 +00:00 |
|
|
Andrej Karpathy
|
d5f7458e03
|
Merge branch 'stream-2' of https://github.com/ngc92/llm.c into ngc92-stream-2
|
2024-06-05 00:23:19 +00:00 |
|
|
Erik Schultheis
|
9af7628063
|
moved remaining memcpy/memset in the training loop to main_stream
|
2024-06-05 02:06:32 +03:00 |
|
|
Andrej Karpathy
|
8940dcdcb1
|
Merge branch 'stream-2' of https://github.com/ngc92/llm.c into ngc92-stream-2
|
2024-06-04 22:52:26 +00:00 |
|
|
Erik Schultheis
|
bc35d3438e
|
naming the stream
|
2024-06-05 01:50:16 +03:00 |
|
|
Erik Schultheis
|
0de54d10da
|
moved last remaining kernel to main stream
|
2024-06-05 01:32:25 +03:00 |
|
|
Andrej Karpathy
|
04b59c87be
|
tune comments
|
2024-06-04 22:16:23 +00:00 |
|
|
Andrej
|
894ada5041
|
Merge pull request #551 from gordicaleksa/fix_loss_bug
Fix PyTorch DDP loss bug
|
2024-06-04 15:10:10 -07:00 |
|
|
Aleksa Gordic
|
6f20ee9f8e
|
Remove * 2 for total batch size
|
2024-06-04 23:42:54 +02:00 |
|
|
Aleksa Gordic
|
34766253cf
|
Move all reduce outside of the gradacc loop for efficiency
|
2024-06-04 23:39:52 +02:00 |
|
|
Aleksa Gordic
|
724085ecd6
|
tmp
|
2024-06-04 23:23:11 +02:00 |
|
|
Erik Schultheis
|
4c369eea0f
|
overlap validity check and data movement
|
2024-06-04 22:47:54 +03:00 |
|
|
Erik Schultheis
|
ca7caac5c5
|
move all kernels into a main_stream in preparation for future parallelism
|
2024-06-04 22:35:12 +03:00 |
|
|
Andrej
|
5379b1d9e9
|
Merge pull request #548 from gordicaleksa/fix_zerograds_bug
Fix zero grads bug
|
2024-06-04 09:51:44 -07:00 |
|
|
Aleksa Gordic
|
3e44d275d8
|
Update comment
|
2024-06-04 18:48:29 +02:00 |
|
|
Aleksa Gordic
|
ec92ff64a2
|
Improve the design and comment
|
2024-06-04 18:47:23 +02:00 |
|
|
Aleksa Gordic
|
54ec686649
|
Fix zero grads bug
|
2024-06-04 18:40:54 +02:00 |
|
|
Andrej
|
c73b4668f3
|
Merge pull request #547 from ngc92/includes
made implicit includes of standard headers explicit
|
2024-06-04 08:56:41 -07:00 |
|
|
Erik Schultheis
|
b02aab1f16
|
made implicit includes of standard headers explicit
|
2024-06-04 13:22:16 +03:00 |
|
|
Andrej
|
cb40f458a3
|
Merge pull request #543 from rosslwheeler/bugfix/make-clean
Make clean fix and Windows cuDNN build fix
|
2024-06-03 21:06:35 -07:00 |
|
|
Ross Wheeler
|
4bfb23948d
|
Make clean fix and Windows cuDNN build fix
|
2024-06-03 20:26:52 -07:00 |
|
|
Andrej Karpathy
|
45b0d1b378
|
copy and cast kernel move to cuda_utils, good general util
|
2024-06-03 23:14:59 +00:00 |
|
|
Andrej Karpathy
|
d697eaecfd
|
move the global norm kernel
|
2024-06-03 23:12:20 +00:00 |
|
|
Andrej Karpathy
|
8c52a652b4
|
move adamw kernel to llmc
|
2024-06-03 23:09:42 +00:00 |
|
|
Andrej Karpathy
|
e484c650b5
|
layernorm and residual, yolo push to master
|
2024-06-03 23:02:46 +00:00 |
|
|
Andrej
|
7d50d3318a
|
Merge pull request #541 from karpathy/move/matmul
move matmul
|
2024-06-03 15:57:09 -07:00 |
|
|
Andrej Karpathy
|
094988738b
|
adjust the comment on extern deviceProp to be more descriptive
|
2024-06-03 22:57:00 +00:00 |
|
|
Andrej Karpathy
|
645feac1e7
|
move matmul
|
2024-06-03 22:54:07 +00:00 |
|
|
Andrej
|
299d375fbc
|
Merge pull request #540 from karpathy/move/fused_classifier
move fused classifier
|
2024-06-03 15:38:24 -07:00 |
|
|
Andrej Karpathy
|
90a7bfecf1
|
move fused classifier
|
2024-06-03 22:35:20 +00:00 |
|
|
Andrej
|
dd4191fbd5
|
Merge pull request #539 from karpathy/move/attention
move attention
|
2024-06-03 15:23:27 -07:00 |
|
|
Andrej Karpathy
|
2186cd6fe7
|
move attention
|
2024-06-03 22:19:47 +00:00 |
|
|
Andrej
|
3427983615
|
Merge pull request #537 from karpathy/move/gelu
move gelu to llmc
|
2024-06-03 15:08:22 -07:00 |
|
|
Andrej Karpathy
|
d8d55e0a08
|
move gelu to llmc
|
2024-06-03 21:57:02 +00:00 |
|
|
Andrej Karpathy
|
49b9776e50
|
also move the launchers
|
2024-06-03 21:37:18 +00:00 |
|
|
Andrej Karpathy
|
93546aed01
|
move encoder, kernel 1/N
|
2024-06-03 21:21:28 +00:00 |
|
|
Andrej
|
3392671826
|
Merge pull request #535 from ngc92/speed-up-compile
moved device utilities into a separate file, and follow-up cleanup
|
2024-06-03 14:11:10 -07:00 |
|
|
Erik Schultheis
|
67502c7f8e
|
moved device utilities into a separate file, and follow-up cleanup
|
2024-06-03 23:58:47 +03:00 |
|
|
Andrej
|
8cc39a794b
|
Merge pull request #528 from gordicaleksa/refactor_evalloader
Refactor eval loader - remove 2 unnecessary malloc calls per iteration
|
2024-06-03 13:54:52 -07:00 |
|
|
Andrej
|
2a6a5b2c6c
|
Merge pull request #529 from gordicaleksa/improve_start_training_print_info
Improve start training print info
|
2024-06-03 13:53:32 -07:00 |
|
|
Andrej
|
0cd9f1ac21
|
Merge pull request #530 from gordicaleksa/minor_refactor
Minor refactor: remove hardcoded val, delete unused vars...
|
2024-06-03 13:52:42 -07:00 |
|
|
Andrej
|
d17893b754
|
Merge pull request #532 from ngc92/ln-buffers
more efficient use of memory buffers for LN recomputation
|
2024-06-03 13:51:22 -07:00 |
|
|
Andrej
|
ac931455a4
|
Merge pull request #533 from karpathy/feature/refactor_part2
refactor part 2
|
2024-06-03 13:44:59 -07:00 |
|
|
Andrej Karpathy
|
001ad78fe1
|
resolve deviceProp and cudaCheck dupes
|
2024-06-03 19:54:32 +00:00 |
|
|
Andrej Karpathy
|
020b15ad77
|
small more fixes
|
2024-06-03 19:32:32 +00:00 |
|
|
Andrej Karpathy
|
5d9403479c
|
some fixes, still not done
|
2024-06-03 18:50:41 +00:00 |
|
|
Andrej Karpathy
|
804d0693c8
|
refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well
|
2024-06-03 18:10:33 +00:00 |
|
|
Andrej
|
e40268457f
|
Merge pull request #526 from karpathy/try/move_cudnn_to_llmc
attempt to move cudnn attention to the llmc directory
|
2024-06-03 09:32:31 -07:00 |
|
|
Erik Schultheis
|
0d16b51626
|
more efficient use of memory buffers for LN recomputation
|
2024-06-03 18:32:51 +03:00 |
|