 Andrej Karpathy
|
3048a3a780
|
oops have to use labels=
|
2024-05-06 18:04:35 +00:00 |
|
|
Andrej Karpathy
|
b4278f094a
|
v1 of what was supposed to be a repro of wikitext ppl, but isn't, what
|
2024-05-06 18:03:10 +00:00 |
|
|
Andrej Karpathy
|
6798dd4b91
|
Merge branch 'rosslwheeler-Makefile-detect-gpu'
|
2024-05-06 10:01:57 +00:00 |
|
|
Ross Wheeler
|
5344cc30b8
|
Merge branch 'Makefile-detect-gpu' of https://github.com/rosslwheeler/llm.c into Makefile-detect-gpu
|
2024-05-06 01:11:37 -07:00 |
|
|
Ross Wheeler
|
0c4908d815
|
Adding CI check to disable auto-detect
|
2024-05-06 01:11:27 -07:00 |
|
|
rosslwheeler
|
98aa83b282
|
Merge branch 'karpathy:master' into Makefile-detect-gpu
|
2024-05-06 00:58:09 -07:00 |
|
|
Ross Wheeler
|
2bfd2b6a7e
|
Auto-detect GPU capability
3 cases tested on Windows 11 and Ubuntu 22.04
|
2024-05-06 00:46:28 -07:00 |
|
|
Andrej
|
9e4f9df9a2
|
Merge pull request #337 from lancerts/make-fix
Makefile type fix: move gelu_backward to backward block
|
2024-05-05 15:37:02 -07:00 |
|
|
Andrej
|
a7c1e4437d
|
Merge pull request #364 from ngc92/cleanup_may4-nvcc11
make things compile with nvcc11
|
2024-05-05 14:42:49 -07:00 |
|
|
Andrej Karpathy
|
d6e8f5e026
|
Merge branch 'ademeure-cleanup_may4'
|
2024-05-05 21:29:41 +00:00 |
|
|
Erik Schultheis
|
804a9af1dc
|
make things compile with nvcc11
|
2024-05-05 23:43:02 +03:00 |
|
|
ademeure
|
876ab93c0a
|
Add FP16 path for atomicStochasticAdd (+remove __bfloat1622float2 to work on older CUDAs) + fixes
|
2024-05-05 15:05:59 +01:00 |
|
|
ademeure
|
9910a40863
|
Removed makefile change so we can integrate #339 instead which feels slightly cleaner
|
2024-05-05 14:42:32 +01:00 |
|
|
ademeure
|
c15ca1f4cf
|
Fix makefile
|
2024-05-05 02:49:16 +01:00 |
|
|
ademeure
|
7789738879
|
Add PTX back to binary + fix whitespaces
|
2024-05-05 02:44:24 +01:00 |
|
|
ademeure
|
8675104b73
|
Compile for the user's GPU architecture using nvidia-smi query on Linux
|
2024-05-05 02:24:31 +01:00 |
|
|
ademeure
|
83ec4b8e70
|
Slightly reduce lines of code in cudnn_att
|
2024-05-05 02:09:52 +01:00 |
|
|
ademeure
|
c261eecda8
|
Merge remote-tracking branch 'karpathy/master' into cleanup_may4
|
2024-05-05 01:49:25 +01:00 |
|
|
Andrej Karpathy
|
64b6c2a77f
|
Merge branch 'ngc92-separate-compilation'
|
2024-05-05 00:41:18 +00:00 |
|
|
Andrej Karpathy
|
f2224f2eb2
|
Merge branch 'separate-compilation' of https://github.com/ngc92/llm.c into ngc92-separate-compilation
|
2024-05-05 00:32:31 +00:00 |
|
|
Andrej Karpathy
|
8168b78549
|
Merge branch 'ademeure-less_idle_more_brrr-3'
|
2024-05-05 00:27:15 +00:00 |
|
|
Andrej Karpathy
|
ce333de156
|
add ema to tok/s
|
2024-05-05 00:18:39 +00:00 |
|
|
Andrej Karpathy
|
2642ffd19c
|
Merge branch 'less_idle_more_brrr' of https://github.com/ademeure/llm.c into ademeure-less_idle_more_brrr-3
|
2024-05-05 00:05:38 +00:00 |
|
|
ademeure
|
a0b79358bd
|
Fix loss not being calculated for forward-only paths
|
2024-05-05 01:01:59 +01:00 |
|
|
Andrej
|
6c179faaeb
|
Merge pull request #349 from lancerts/fix-include
Remove the duplicated #include <assert.h>
|
2024-05-04 17:01:32 -07:00 |
|
|
Erik Schultheis
|
b087b9c819
|
don't compile/link cudnn if not asked for it
|
2024-05-05 02:54:42 +03:00 |
|
|
ademeure
|
34304361ba
|
fix merge conflict
|
2024-05-05 00:16:49 +01:00 |
|
|
ademeure
|
ec0ab2d2d0
|
Remove arch=native as it only available on recent CUDA versions
|
2024-05-05 00:00:58 +01:00 |
|
|
ademeure
|
18d7ed9242
|
revert profile.py changes for now
|
2024-05-04 23:46:39 +01:00 |
|
|
ademeure
|
abaaceb801
|
Added makefile gencode changes
|
2024-05-04 23:43:13 +01:00 |
|
|
ademeure
|
bfb9c51446
|
refactoring & remove unused functions to reduce LOC (+wip profile.py improvements)
|
2024-05-04 23:40:15 +01:00 |
|
|
ademeure
|
66b9755ce9
|
Use template for blockReduce's warp reduction + add out_of_bounds parameter
|
2024-05-04 18:48:54 +01:00 |
|
|
ademeure
|
8d593b916e
|
fix cuda stream bug for test_gpt2cu... by removing most of the memcpy/memset parallelism for now (keeping loss_event to avoid sync between forward and backward).
|
2024-05-04 18:18:51 +01:00 |
|
|
Erik Schultheis
|
19c290d7e6
|
improved debugging for cudnn
|
2024-05-04 12:31:15 +03:00 |
|
|
Erik Schultheis
|
aa5bb258b6
|
fixed up test and profile targets
|
2024-05-04 11:59:59 +03:00 |
|
|
lancer
|
03b7323d1e
|
minor fix of the include
|
2024-05-03 17:38:39 -07:00 |
|
|
Andrej
|
ff2fbdc2a1
|
Merge pull request #332 from harryjackson/llm.java
Adding a java port of the project
|
2024-05-03 13:58:59 -07:00 |
|
|
Erik Schultheis
|
67a82a611e
|
first attempt at moving cudnn out of the main file for faster compiles
|
2024-05-03 23:08:44 +03:00 |
|
|
Andrej
|
aa5dd2eeab
|
Update README.md
|
2024-05-03 12:43:08 -07:00 |
|
|
Andrej
|
6260a1844b
|
Merge pull request #345 from ngc92/profile
updated the profile script to be more robust, and adapted to recent k…
|
2024-05-03 12:17:22 -07:00 |
|
|
Erik Schultheis
|
977d6a6c6f
|
fix × symbol
|
2024-05-03 21:57:31 +03:00 |
|
|
Erik Schultheis
|
7fabb12e2e
|
fixed instruction counts
|
2024-05-03 21:54:06 +03:00 |
|
|
Erik Schultheis
|
13eeeb20c8
|
try to detect if we need sudo
|
2024-05-03 21:31:54 +03:00 |
|
|
Erik Schultheis
|
325b45647f
|
updated the profile script to be more robust, and adapted to recent kernel changes
make it compile with cudnn
default batchsize is 24 now
|
2024-05-03 21:20:06 +03:00 |
|
|
ademeure
|
a0b80920f1
|
Reducing profiler to only 3 steps instead of 10 given slight perf penalty (todo: make it configurable)
|
2024-05-03 19:03:06 +01:00 |
|
|
ademeure
|
a19fcecb79
|
Set cuBLAS(Lt) to use the new main_stream
|
2024-05-03 18:35:46 +01:00 |
|
|
Andrej Karpathy
|
6c629ac571
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-05-03 15:54:36 +00:00 |
|
|
Andrej Karpathy
|
e780b56974
|
small fixes for profiling
|
2024-05-03 15:54:29 +00:00 |
|
|
ademeure
|
b58181079a
|
super optimised matmul_bias + cuda streams for better parallelism + fix fused_classifier cache hint + remove remaining cooperative groups
|
2024-05-03 16:38:46 +01:00 |
|
|
ademeure
|
f456a75e3f
|
add CUDA streams (no parallelism yet) + fix profiling init + new "blockReduce" function
|
2024-05-03 03:14:16 +01:00 |
|