 Andrej Karpathy
|
cbe82e783a
|
consolidate memory by deleting the grad activation struct, move some of the last pieces of memory to the forward pass
|
2024-06-14 00:54:58 +00:00 |
|
|
Andrej Karpathy
|
43a903e1b4
|
revert old printf0, minor touchups and comments, add assert
|
2024-06-13 23:18:22 +00:00 |
|
|
Erik Schultheis
|
f071efd94f
|
stride-based adam and copy-and-cast
|
2024-06-13 02:01:26 +03:00 |
|
|
Erik Schultheis
|
9d9da3b209
|
fewer global norm kernel calls
|
2024-06-12 21:55:01 +03:00 |
|
|
Erik Schultheis
|
8cba9ba0c0
|
cleanup
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
ecca976132
|
re-enable reduce-scatter operation
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
ecdb0ca86e
|
refactored code
|
2024-06-12 13:16:19 +03:00 |
|
|
Andrej
|
637c1b69c4
|
Merge pull request #552 from karpathy/feature/streams
Feature/streams
|
2024-06-08 09:25:56 -07:00 |
|
|
Aleksa Gordic
|
4090ca8fa1
|
Add learning rate and grad norm logging
|
2024-06-07 17:34:42 +02:00 |
|
|
Andrej Karpathy
|
d5f7458e03
|
Merge branch 'stream-2' of https://github.com/ngc92/llm.c into ngc92-stream-2
|
2024-06-05 00:23:19 +00:00 |
|
|
Erik Schultheis
|
9af7628063
|
moved remaining memcpy/memset in the training loop to main_stream
|
2024-06-05 02:06:32 +03:00 |
|
|
Andrej Karpathy
|
8940dcdcb1
|
Merge branch 'stream-2' of https://github.com/ngc92/llm.c into ngc92-stream-2
|
2024-06-04 22:52:26 +00:00 |
|
|
Erik Schultheis
|
bc35d3438e
|
naming the stream
|
2024-06-05 01:50:16 +03:00 |
|
|
Erik Schultheis
|
0de54d10da
|
moved last remaining kernel to main stream
|
2024-06-05 01:32:25 +03:00 |
|
|
Erik Schultheis
|
ca7caac5c5
|
move all kernels into a main_stream in preparation for future parallelism
|
2024-06-04 22:35:12 +03:00 |
|
|
Erik Schultheis
|
b02aab1f16
|
made implicit includes of standard headers explicit
|
2024-06-04 13:22:16 +03:00 |
|
|
Ross Wheeler
|
4bfb23948d
|
Make clean fix and Windows cuDNN build fix
|
2024-06-03 20:26:52 -07:00 |
|
|
Andrej Karpathy
|
45b0d1b378
|
copy and cast kernel move to cuda_utils, good general util
|
2024-06-03 23:14:59 +00:00 |
|
|
Andrej Karpathy
|
d697eaecfd
|
move the global norm kernel
|
2024-06-03 23:12:20 +00:00 |
|
|
Andrej Karpathy
|
8c52a652b4
|
move adamw kernel to llmc
|
2024-06-03 23:09:42 +00:00 |
|
|
Andrej Karpathy
|
e484c650b5
|
layernorm and residual, yolo push to master
|
2024-06-03 23:02:46 +00:00 |
|
|
Andrej Karpathy
|
094988738b
|
adjust the comment on extern deviceProp to be more descriptive
|
2024-06-03 22:57:00 +00:00 |
|
|
Andrej Karpathy
|
645feac1e7
|
move matmul
|
2024-06-03 22:54:07 +00:00 |
|
|
Andrej Karpathy
|
90a7bfecf1
|
move fused classifier
|
2024-06-03 22:35:20 +00:00 |
|
|
Andrej Karpathy
|
2186cd6fe7
|
move attention
|
2024-06-03 22:19:47 +00:00 |
|
|
Andrej Karpathy
|
d8d55e0a08
|
move gelu to llmc
|
2024-06-03 21:57:02 +00:00 |
|
|
Andrej Karpathy
|
49b9776e50
|
also move the launchers
|
2024-06-03 21:37:18 +00:00 |
|
|
Andrej Karpathy
|
93546aed01
|
move encoder, kernel 1/N
|
2024-06-03 21:21:28 +00:00 |
|
|
Andrej
|
3392671826
|
Merge pull request #535 from ngc92/speed-up-compile
moved device utilities into a separate file, and follow-up cleanup
|
2024-06-03 14:11:10 -07:00 |
|
|
Erik Schultheis
|
67502c7f8e
|
moved device utilities into a separate file, and follow-up cleanup
|
2024-06-03 23:58:47 +03:00 |
|
|
Andrej
|
8cc39a794b
|
Merge pull request #528 from gordicaleksa/refactor_evalloader
Refactor eval loader - remove 2 unnecessary malloc calls per iteration
|
2024-06-03 13:54:52 -07:00 |
|
|
Andrej Karpathy
|
001ad78fe1
|
resolve deviceProp and cudaCheck dupes
|
2024-06-03 19:54:32 +00:00 |
|
|
Andrej Karpathy
|
020b15ad77
|
small more fixes
|
2024-06-03 19:32:32 +00:00 |
|
|
Andrej Karpathy
|
5d9403479c
|
some fixes, still not done
|
2024-06-03 18:50:41 +00:00 |
|
|
Andrej Karpathy
|
804d0693c8
|
refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well
|
2024-06-03 18:10:33 +00:00 |
|
|
Aleksa Gordic
|
d9e867678e
|
Remove file size - not used
|
2024-06-03 12:14:58 +02:00 |
|
|
Aleksa Gordic
|
e86bd62374
|
Refactor eval loader - remove 2 unnecessary malloc calls per iteration
|
2024-06-03 11:43:33 +02:00 |
|
|
Andrej Karpathy
|
41dc65a0ef
|
attempt to move cudnn attention to the llmc directory
|
2024-06-03 03:23:04 +00:00 |
|
|
Andrej Karpathy
|
1ac30591f2
|
add new file llmc/mfu.h for MFU calculations for different GPUs
|
2024-06-03 01:41:18 +00:00 |
|
|
Aleksa Gordic
|
ae9c957273
|
Add clarification on Box-Muller
|
2024-06-02 15:04:45 +02:00 |
|
|
Andrej Karpathy
|
0bc24c36e6
|
fix bug in resumption of dataloader, have to reload shard
|
2024-05-31 22:19:37 +00:00 |
|
|
Andrej Karpathy
|
35aa110244
|
more refactor
|
2024-05-30 19:39:29 +00:00 |
|
|
Andrej Karpathy
|
eed242702a
|
experiment with adding the llmc lib directory
|
2024-05-27 21:10:28 +00:00 |
|