 Chinthaka
|
58ac83f5ec
|
Merge branch 'master' into zero-stage1
|
2024-05-04 16:24:22 +08:00 |
|
|
chinthysl
|
59b66f6c07
|
refactor and fix CI issue
|
2024-05-04 16:22:44 +08:00 |
|
|
lancer
|
03b7323d1e
|
minor fix of the include
|
2024-05-03 17:38:39 -07:00 |
|
|
Andrej
|
ff2fbdc2a1
|
Merge pull request #332 from harryjackson/llm.java
Adding a java port of the project
|
2024-05-03 13:58:59 -07:00 |
|
|
Erik Schultheis
|
67a82a611e
|
first attempt at moving cudnn out of the main file for faster compiles
|
2024-05-03 23:08:44 +03:00 |
|
|
Andrej
|
aa5dd2eeab
|
Update README.md
|
2024-05-03 12:43:08 -07:00 |
|
|
Andrej
|
6260a1844b
|
Merge pull request #345 from ngc92/profile
updated the profile script to be more robust, and adapted to recent k…
|
2024-05-03 12:17:22 -07:00 |
|
|
Erik Schultheis
|
977d6a6c6f
|
fix × symbol
|
2024-05-03 21:57:31 +03:00 |
|
|
Erik Schultheis
|
7fabb12e2e
|
fixed instruction counts
|
2024-05-03 21:54:06 +03:00 |
|
|
Erik Schultheis
|
13eeeb20c8
|
try to detect if we need sudo
|
2024-05-03 21:31:54 +03:00 |
|
|
Erik Schultheis
|
325b45647f
|
updated the profile script to be more robust, and adapted to recent kernel changes
make it compile with cudnn
default batchsize is 24 now
|
2024-05-03 21:20:06 +03:00 |
|
|
ademeure
|
a0b80920f1
|
Reducing profiler to only 3 steps instead of 10 given slight perf penalty (todo: make it configurable)
|
2024-05-03 19:03:06 +01:00 |
|
|
ademeure
|
a19fcecb79
|
Set cuBLAS(Lt) to use the new main_stream
|
2024-05-03 18:35:46 +01:00 |
|
|
Andrej Karpathy
|
6c629ac571
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-05-03 15:54:36 +00:00 |
|
|
Andrej Karpathy
|
e780b56974
|
small fixes for profiling
|
2024-05-03 15:54:29 +00:00 |
|
|
ademeure
|
b58181079a
|
super optimised matmul_bias + cuda streams for better parallelism + fix fused_classifier cache hint + remove remaining cooperative groups
|
2024-05-03 16:38:46 +01:00 |
|
|
Chinthaka
|
0fd384bb21
|
Merge branch 'master' into zero-stage1
|
2024-05-03 11:08:41 +08:00 |
|
|
ademeure
|
f456a75e3f
|
add CUDA streams (no parallelism yet) + fix profiling init + new "blockReduce" function
|
2024-05-03 03:14:16 +01:00 |
|
|
lancer
|
c81adeb6b2
|
move gelu_backward to backward block
|
2024-05-02 17:36:12 -07:00 |
|
|
Andrej
|
c04c133b25
|
Merge pull request #335 from karpathy/feature/bias_backward2
v1 of the new matmul backward bias kernel
|
2024-05-02 16:52:34 -07:00 |
|
|
Andrej Karpathy
|
776541dac7
|
v1 of the new matmul backward bias kernel
|
2024-05-02 23:32:20 +00:00 |
|
|
Andrej Karpathy
|
6ebef46f83
|
ugh didn't notice this tiny rebasing mistake, introduced a bug. good candidate for a CI that we can overfit a single batch in the train_gpt2.cu script and get the exact same numbers as we expect in the test_gpt2.cu file
|
2024-05-02 23:23:28 +00:00 |
|
|
Andrej Karpathy
|
2c7960040d
|
Merge branch 'ngc92-bias-backward'
|
2024-05-02 22:17:31 +00:00 |
|
|
Andrej Karpathy
|
937fbd8e36
|
small doc fixes
|
2024-05-02 22:17:22 +00:00 |
|
|
Erik Schultheis
|
6e48501d60
|
bias backward kernel that will use all available threads
|
2024-05-03 01:05:47 +03:00 |
|
|
Andrej Karpathy
|
6a3d63fb1d
|
Merge branch 'PeterZhizhin-add_nvtx_ranges_and_timings'
|
2024-05-02 21:13:26 +00:00 |
|
|
Andrej Karpathy
|
79505bc6b3
|
resolve merge and small fixes
|
2024-05-02 21:13:08 +00:00 |
|
|
Andrej Karpathy
|
50714d2254
|
pragma unroll quick fix
|
2024-05-02 21:06:11 +00:00 |
|
|
Andrej Karpathy
|
9da2729c5a
|
update encoder_forward with the latest and greatest Packed128 and incorporate into train_gpt2cu
|
2024-05-02 20:40:05 +00:00 |
|
|
Peter Zhizhin
|
41a0789a44
|
Added NVTX ranges for FlashAttention
|
2024-05-02 20:30:08 +00:00 |
|
|
Andrej Karpathy
|
8a9510f32b
|
very minor stylistic preference let's keep the code style consistent
|
2024-05-02 19:48:56 +00:00 |
|
|
Andrej Karpathy
|
84dbc70160
|
Merge branch 'JaneIllario-patch-1'
|
2024-05-02 19:45:18 +00:00 |
|
|
Andrej Karpathy
|
3d1761b370
|
Merge branch 'patch-1' of https://github.com/JaneIllario/llm.c into JaneIllario-patch-1
|
2024-05-02 19:43:14 +00:00 |
|
|
Andrej Karpathy
|
01df41fbba
|
careful to guard the overloaded warpReduceSum and small fixes
|
2024-05-02 19:24:04 +00:00 |
|
|
Andrej Karpathy
|
6879c36cf3
|
Merge branch 'JaneIllario-gelu_backwards-3'
|
2024-05-02 19:16:38 +00:00 |
|
|
Andrej Karpathy
|
99f51bae1c
|
Merge branch 'gelu_backwards' of https://github.com/JaneIllario/llm.c into JaneIllario-gelu_backwards-3
|
2024-05-02 19:15:06 +00:00 |
|
|
Andrej Karpathy
|
8ccd05ec29
|
group warp reduce ops together
|
2024-05-02 19:09:53 +00:00 |
|
|
Andrej Karpathy
|
d05c51c7d1
|
Merge branch 'ChrisDryden-removing_cg_example'
|
2024-05-02 19:08:26 +00:00 |
|
|
Andrej
|
2583feb7a2
|
Merge pull request #327 from yushengsu-thu/master
[ModuleNotFoundError]: Solution - add required package (requests) into requirements.txt
|
2024-05-02 12:03:46 -07:00 |
|
|
Andrej Karpathy
|
d1771a7b59
|
quick fix master. there is some weirdness here in the adam update that is todo to understand better, for now i just want the master to be ok
|
2024-05-02 19:01:26 +00:00 |
|
|
Harry Jackson
|
c13a730149
|
Merge branch 'master' into llm.java
|
2024-05-02 11:44:44 -07:00 |
|
|
Andrej Karpathy
|
ec73086a83
|
Merge branch 'removing_cg_example' of https://github.com/ChrisDryden/llm.c into ChrisDryden-removing_cg_example
|
2024-05-02 18:14:14 +00:00 |
|
|
Andrej Karpathy
|
2202c9a51b
|
add kernel 4 to docs. have to improve these docs more and document them better
|
2024-05-02 18:09:27 +00:00 |
|
|
Andrej Karpathy
|
2feb9ffcfb
|
Fused classifier: use Packed128 and delete cg dep
|
2024-05-02 18:08:27 +00:00 |
|
|
Peter Zhizhin
|
bcf7d4fe93
|
Add NSight Compute ranges, use CUDA events for timings
|
2024-05-02 17:23:07 +00:00 |
|
|
Yusheng (Ethan) Su
|
31e59273e1
|
Merge branch 'karpathy:master' into master
|
2024-05-02 14:04:47 +04:00 |
|
|
chinthysl
|
632caf1ce6
|
Refactored zero_active var
|
2024-05-02 07:38:49 +00:00 |
|
|
chinthysl
|
d35d2e606b
|
Generalized copy_and_cast_kernel and changes to cater model->master_weights
|
2024-05-02 07:31:34 +00:00 |
|
|
Chinthaka
|
07b8102e79
|
Merge branch 'master' into zero-stage1
|
2024-05-02 11:48:56 +08:00 |
|
|
“Christopher
|
398cdaf4cd
|
Removed cooperative groups and added packed128 to fused classifier and prepare softmax blockwide
|
2024-05-02 02:33:51 +00:00 |
|