 Andrej Karpathy
|
804d0693c8
|
refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well
|
2024-06-03 18:10:33 +00:00 |
|
|
ademeure
|
b5e75dde8e
|
Fully deterministic encoder backward kernels for train_gpt2.cu
|
2024-05-21 15:57:07 +01:00 |
|
|
Andrej
|
00c291c679
|
Merge pull request #434 from ngc92/profile-fix
initialize multi_gpu_config so profile doesn't crash
|
2024-05-19 20:57:11 +01:00 |
|
|
Erik Schultheis
|
299ce659ec
|
initialize multi_gpu_config so profile doesn't crash
|
2024-05-19 22:25:00 +03:00 |
|
|
Erik Schultheis
|
c3a3b9daa5
|
fixed profile target
|
2024-05-18 22:47:40 +03:00 |
|
|
Erik Schultheis
|
fbd8f03eea
|
fixup profiling
|
2024-05-16 14:25:06 +03:00 |
|
|
ademeure
|
876ab93c0a
|
Add FP16 path for atomicStochasticAdd (+remove __bfloat1622float2 to work on older CUDAs) + fixes
|
2024-05-05 15:05:59 +01:00 |
|
|
ademeure
|
c261eecda8
|
Merge remote-tracking branch 'karpathy/master' into cleanup_may4
|
2024-05-05 01:49:25 +01:00 |
|
|
Andrej Karpathy
|
f2224f2eb2
|
Merge branch 'separate-compilation' of https://github.com/ngc92/llm.c into ngc92-separate-compilation
|
2024-05-05 00:32:31 +00:00 |
|
|
Andrej Karpathy
|
2642ffd19c
|
Merge branch 'less_idle_more_brrr' of https://github.com/ademeure/llm.c into ademeure-less_idle_more_brrr-3
|
2024-05-05 00:05:38 +00:00 |
|
|
ademeure
|
34304361ba
|
fix merge conflict
|
2024-05-05 00:16:49 +01:00 |
|
|
ademeure
|
bfb9c51446
|
refactoring & remove unused functions to reduce LOC (+wip profile.py improvements)
|
2024-05-04 23:40:15 +01:00 |
|
|
Erik Schultheis
|
aa5bb258b6
|
fixed up test and profile targets
|
2024-05-04 11:59:59 +03:00 |
|
|
ademeure
|
a19fcecb79
|
Set cuBLAS(Lt) to use the new main_stream
|
2024-05-03 18:35:46 +01:00 |
|
|
Andrej Karpathy
|
e780b56974
|
small fixes for profiling
|
2024-05-03 15:54:29 +00:00 |
|
|
ademeure
|
b58181079a
|
super optimised matmul_bias + cuda streams for better parallelism + fix fused_classifier cache hint + remove remaining cooperative groups
|
2024-05-03 16:38:46 +01:00 |
|
|
ademeure
|
f456a75e3f
|
add CUDA streams (no parallelism yet) + fix profiling init + new "blockReduce" function
|
2024-05-03 03:14:16 +01:00 |
|
|
Andrej Karpathy
|
8aa9a9ed53
|
and also profile lol
|
2024-05-01 17:56:01 +00:00 |
|
|
ademeure
|
1147983b8c
|
Add missing cudnnDestroy() calls
|
2024-05-01 17:19:51 +01:00 |
|
|
ademeure
|
d9e7a0a378
|
Fixed profile_gpt2.cu/test_gpt2.cu for cuDNN
|
2024-05-01 17:03:28 +01:00 |
|
|
Andrej Karpathy
|
906d22f5b6
|
fix profile gpt2cu but note it currently needs the NO_MULTI_GPU=1 switch i think when calling it with make, todo fix later how we add the mpi flags
|
2024-04-29 23:27:55 +00:00 |
|
|
Rickard Hallerbäck
|
b522333851
|
Updating the CI to build different precisions
|
2024-04-28 22:17:47 +02:00 |
|
|
Andrej Karpathy
|
9d70d9a1b2
|
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
|
2024-04-28 16:17:22 +00:00 |
|
|
Andrej Karpathy
|
d34556cf5e
|
more profile docs
|
2024-04-20 01:29:14 +00:00 |
|
|
Andrej Karpathy
|
67fd37ace4
|
add crappy instructions for profiling basics
|
2024-04-20 01:06:53 +00:00 |
|
|
Erik Schultheis
|
9c1c1f0cec
|
helper target that gets compiled with lineinfo and runs a single layer forward-backward pass
|
2024-04-20 03:19:23 +03:00 |
|