 ademeure
|
e1f89b3043
|
remove BF16 default from classified_fused before PR
|
2024-05-07 00:57:01 +01:00 |
|
|
ademeure
|
1ea7f9bf25
|
tiny irrelevant optimisation to final unaligned fused_classifier loop + add missing common.h changes
|
2024-05-07 00:54:50 +01:00 |
|
|
Andrej
|
ec9fd51d5c
|
Merge pull request #375 from Chillee/patch-1
Don't return logits during training for PyTorch baseline
|
2024-05-07 01:49:02 +02:00 |
|
|
Andrej
|
aa54a0a808
|
Merge pull request #377 from ademeure/profiler_efficiency
Add DRAM BW, Tensor Core Utilisation, & Efficiency Stats
|
2024-05-07 01:45:33 +02:00 |
|
|
ademeure
|
9b55ea8cf5
|
More crazy optimisations to layernorm_backward, fused_classifier, and matmul_backward_bias.
|
2024-05-07 00:11:42 +01:00 |
|
|
Ross Wheeler
|
f7d7760084
|
Non-standard C syntax replace with standard C
Also, adding unistd.h back because that has some changes required to find M_PI.
|
2024-05-06 13:56:17 -07:00 |
|
|
ademeure
|
5adb6ef2c1
|
Add tensor core and overall efficiency stats to profiler script.
|
2024-05-06 21:24:42 +01:00 |
|
|
Horace He
|
69f6c4f765
|
Don't return logits during training for PyTorch baseline
This improves perf somewhat, since currently it's always returning logits (which thus need to be materialized).
|
2024-05-06 13:10:17 -07:00 |
|
|
Ross Wheeler
|
134f4c7fc8
|
Adding two directory search for cuDNN frontend files
Search in $(HOME) and in the current directory.
|
2024-05-06 12:38:40 -07:00 |
|
|
rosslwheeler
|
dedd86c1b3
|
Merge branch 'karpathy:master' into cudnn_makefile_changes
|
2024-05-06 11:20:22 -07:00 |
|
|
Andrej
|
4274d95b4a
|
Merge pull request #365 from lancerts/classifier-fix
Minor update on the classifier_fused.cu
|
2024-05-06 12:04:16 +02:00 |
|
|
Andrej Karpathy
|
6798dd4b91
|
Merge branch 'rosslwheeler-Makefile-detect-gpu'
|
2024-05-06 10:01:57 +00:00 |
|
|
chinthysl
|
7a8f471bc7
|
further reorganization
|
2024-05-06 17:49:58 +08:00 |
|
|
Ross Wheeler
|
5344cc30b8
|
Merge branch 'Makefile-detect-gpu' of https://github.com/rosslwheeler/llm.c into Makefile-detect-gpu
|
2024-05-06 01:11:37 -07:00 |
|
|
Ross Wheeler
|
0c4908d815
|
Adding CI check to disable auto-detect
|
2024-05-06 01:11:27 -07:00 |
|
|
rosslwheeler
|
98aa83b282
|
Merge branch 'karpathy:master' into Makefile-detect-gpu
|
2024-05-06 00:58:09 -07:00 |
|
|
Ross Wheeler
|
2bfd2b6a7e
|
Auto-detect GPU capability
3 cases tested on Windows 11 and Ubuntu 22.04
|
2024-05-06 00:46:28 -07:00 |
|
|
Chinthaka
|
aae7145495
|
Merge branch 'master' into zero-stage1
|
2024-05-06 10:36:24 +08:00 |
|
|
lancer
|
bbfe8c989c
|
Minor update on the code
|
2024-05-05 16:59:58 -07:00 |
|
|
Andrej
|
9e4f9df9a2
|
Merge pull request #337 from lancerts/make-fix
Makefile type fix: move gelu_backward to backward block
|
2024-05-05 15:37:02 -07:00 |
|
|
Andrej
|
a7c1e4437d
|
Merge pull request #364 from ngc92/cleanup_may4-nvcc11
make things compile with nvcc11
|
2024-05-05 14:42:49 -07:00 |
|
|
Andrej Karpathy
|
d6e8f5e026
|
Merge branch 'ademeure-cleanup_may4'
|
2024-05-05 21:29:41 +00:00 |
|
|
Erik Schultheis
|
804a9af1dc
|
make things compile with nvcc11
|
2024-05-05 23:43:02 +03:00 |
|
|
ademeure
|
876ab93c0a
|
Add FP16 path for atomicStochasticAdd (+remove __bfloat1622float2 to work on older CUDAs) + fixes
|
2024-05-05 15:05:59 +01:00 |
|
|
ademeure
|
9910a40863
|
Removed makefile change so we can integrate #339 instead which feels slightly cleaner
|
2024-05-05 14:42:32 +01:00 |
|
|
Ross Wheeler
|
42176ad4d5
|
Merge branch 'cudnn_makefile_changes' of https://github.com/rosslwheeler/llm.c into cudnn_makefile_changes
|
2024-05-04 23:27:21 -07:00 |
|
|
Ross Wheeler
|
2d4e5fd840
|
Minor fixes for Makefile for cudnn and windows support
Tested on Ubuntu Linux 22.04 and Windows 11
|
2024-05-04 23:26:57 -07:00 |
|
|
Ross Wheeler
|
9663719c62
|
Minor fixes for Make for cudnn and windows support
Tested on Ubuntu Linux 22.04 and Windows 11
|
2024-05-04 23:24:41 -07:00 |
|
|
ademeure
|
c15ca1f4cf
|
Fix makefile
|
2024-05-05 02:49:16 +01:00 |
|
|
ademeure
|
7789738879
|
Add PTX back to binary + fix whitespaces
|
2024-05-05 02:44:24 +01:00 |
|
|
ademeure
|
8675104b73
|
Compile for the user's GPU architecture using nvidia-smi query on Linux
|
2024-05-05 02:24:31 +01:00 |
|
|
ademeure
|
83ec4b8e70
|
Slightly reduce lines of code in cudnn_att
|
2024-05-05 02:09:52 +01:00 |
|
|
ademeure
|
c261eecda8
|
Merge remote-tracking branch 'karpathy/master' into cleanup_may4
|
2024-05-05 01:49:25 +01:00 |
|
|
Andrej Karpathy
|
64b6c2a77f
|
Merge branch 'ngc92-separate-compilation'
|
2024-05-05 00:41:18 +00:00 |
|
|
Andrej Karpathy
|
f2224f2eb2
|
Merge branch 'separate-compilation' of https://github.com/ngc92/llm.c into ngc92-separate-compilation
|
2024-05-05 00:32:31 +00:00 |
|
|
Andrej Karpathy
|
8168b78549
|
Merge branch 'ademeure-less_idle_more_brrr-3'
|
2024-05-05 00:27:15 +00:00 |
|
|
Andrej Karpathy
|
ce333de156
|
add ema to tok/s
|
2024-05-05 00:18:39 +00:00 |
|
|
Andrej Karpathy
|
2642ffd19c
|
Merge branch 'less_idle_more_brrr' of https://github.com/ademeure/llm.c into ademeure-less_idle_more_brrr-3
|
2024-05-05 00:05:38 +00:00 |
|
|
ademeure
|
a0b79358bd
|
Fix loss not being calculated for forward-only paths
|
2024-05-05 01:01:59 +01:00 |
|
|
Andrej
|
6c179faaeb
|
Merge pull request #349 from lancerts/fix-include
Remove the duplicated #include <assert.h>
|
2024-05-04 17:01:32 -07:00 |
|
|
Erik Schultheis
|
b087b9c819
|
don't compile/link cudnn if not asked for it
|
2024-05-05 02:54:42 +03:00 |
|
|
ademeure
|
34304361ba
|
fix merge conflict
|
2024-05-05 00:16:49 +01:00 |
|
|
ademeure
|
ec0ab2d2d0
|
Remove arch=native as it only available on recent CUDA versions
|
2024-05-05 00:00:58 +01:00 |
|
|
ademeure
|
18d7ed9242
|
revert profile.py changes for now
|
2024-05-04 23:46:39 +01:00 |
|
|
ademeure
|
abaaceb801
|
Added makefile gencode changes
|
2024-05-04 23:43:13 +01:00 |
|
|
ademeure
|
bfb9c51446
|
refactoring & remove unused functions to reduce LOC (+wip profile.py improvements)
|
2024-05-04 23:40:15 +01:00 |
|
|
ademeure
|
66b9755ce9
|
Use template for blockReduce's warp reduction + add out_of_bounds parameter
|
2024-05-04 18:48:54 +01:00 |
|
|
ademeure
|
8d593b916e
|
fix cuda stream bug for test_gpt2cu... by removing most of the memcpy/memset parallelism for now (keeping loss_event to avoid sync between forward and backward).
|
2024-05-04 18:18:51 +01:00 |
|
|
Erik Schultheis
|
19c290d7e6
|
improved debugging for cudnn
|
2024-05-04 12:31:15 +03:00 |
|
|
Erik Schultheis
|
aa5bb258b6
|
fixed up test and profile targets
|
2024-05-04 11:59:59 +03:00 |
|