4bfb23948d
Make clean fix and Windows cuDNN build fix
2024-06-03 20:26:52 -07:00
67502c7f8e
moved device utilities into a separate file, and follow-up cleanup
2024-06-03 23:58:47 +03:00
020b15ad77
small more fixes
2024-06-03 19:32:32 +00:00
6434009298
attempt to remove PATH_SEP from Makefile
2024-06-03 15:30:45 +00:00
41dc65a0ef
attempt to move cudnn attention to the llmc directory
2024-06-03 03:23:04 +00:00
9e645314e6
Check if file exists using platform specific commands & Add *.o to gitignore
...
Signed-off-by: Jun Zhang <jun@junz.org >
2024-05-20 00:00:21 +08:00
6da5e63e2c
Fix detection of cudnn-frontend in '.' on Linux
...
Plus minor legibility fixes in the cudnn-frontend explanation.
2024-05-10 10:57:39 +03:00
69aa64cc80
Adds cuDNN install instructions for a smaller but sufficient package
...
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).
- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)
- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions
Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com >
2024-05-08 14:53:31 -07:00
0ce5fcf7e8
Rename cudnn_att.cu to cudnn_att.cpp to speed up compilation.
2024-05-08 19:53:48 +00:00
134f4c7fc8
Adding two directory search for cuDNN frontend files
...
Search in $(HOME) and in the current directory.
2024-05-06 12:38:40 -07:00
dedd86c1b3
Merge branch 'karpathy:master' into cudnn_makefile_changes
2024-05-06 11:20:22 -07:00
0c4908d815
Adding CI check to disable auto-detect
2024-05-06 01:11:27 -07:00
2bfd2b6a7e
Auto-detect GPU capability
...
3 cases tested on Windows 11 and Ubuntu 22.04
2024-05-06 00:46:28 -07:00
2d4e5fd840
Minor fixes for Makefile for cudnn and windows support
...
Tested on Ubuntu Linux 22.04 and Windows 11
2024-05-04 23:26:57 -07:00
b087b9c819
don't compile/link cudnn if not asked for it
2024-05-05 02:54:42 +03:00
aa5bb258b6
fixed up test and profile targets
2024-05-04 11:59:59 +03:00
67a82a611e
first attempt at moving cudnn out of the main file for faster compiles
2024-05-03 23:08:44 +03:00
e780b56974
small fixes for profiling
2024-05-03 15:54:29 +00:00
69d40ebfff
add readme and search for cudd in home dir
2024-05-01 17:43:50 +00:00
35640651b3
updates to makefile: docs, override, etc
2024-05-01 17:32:52 +00:00
04718fbbd1
round 1 of small fixes for cudnn support
2024-05-01 17:07:10 +00:00
7beff72f5f
Add makefile changes for cuDNN
2024-05-01 15:31:30 +01:00
906d22f5b6
fix profile gpt2cu but note it currently needs the NO_MULTI_GPU=1 switch i think when calling it with make, todo fix later how we add the mpi flags
2024-04-29 23:27:55 +00:00
fcef97d6a1
Adding back PFLAGS
...
Got missed during the merge
Missed the profiler PFLAGS
Move PFLAGS ahead to match original change
2024-04-29 08:53:50 -07:00
c9ab8ee123
Updating Makefile with latest changes
...
Add check with Windows OS
2024-04-29 03:00:46 -07:00
18b41b4df6
Merge pull request #279 from Ricardicus/prec-ci
...
Updating the CI to build different precisions
2024-04-28 13:47:48 -07:00
49228b091a
add small comment on -t=0
2024-04-28 20:20:51 +00:00
b522333851
Updating the CI to build different precisions
2024-04-28 22:17:47 +02:00
10aa24e2cc
Merge pull request #269 from ChrisDryden/patch-3
...
Enable multithreading in nvcc
2024-04-28 13:16:47 -07:00
9d70d9a1b2
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
2024-04-28 16:17:22 +00:00
4f7d8d9f44
allow user to make different precisions, add prints and error handling around precisions
2024-04-28 16:08:27 +00:00
2954d90f30
Enable multithreading in nvcc
...
Tested locally and reduced compilation time by 200ms, unfortunately for me upgrading to 12.4 made my compilations times slow by 2x but at least this can make it a bit faster
2024-04-27 14:14:02 -07:00
a00a3a88ef
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
2024-04-26 18:11:09 +00:00
2e17140d33
more defensive check for openmpi install, and an install command
2024-04-26 16:27:57 +00:00
d49e8301eb
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
2024-04-24 22:11:00 +00:00
ed6387a695
updates to readme, and introduce the test fp32 cuda file too
2024-04-23 17:41:42 +00:00
095d27662b
checkpoint the fp32 CUDA implementation to separate file. our mainline iteration will now continue in the new (mixed precision) file
2024-04-23 17:21:44 +00:00
c7aad65a04
Adding a workflow that builds with and without CUDA and OMP and on CPU also tests on ubuntu and macos
2024-04-22 02:14:09 +02:00
fced6d180f
small adjustments to Makefile and gitignore
2024-04-21 17:40:26 +00:00
3cd6f83e32
removing some warnings
2024-04-20 22:03:35 +02:00
be106f09a4
Merge branch 'master' into master
2024-04-20 22:00:02 +02:00
f534e4bdfe
Fixes -Ofast optimizations breaking model by skipping them for gelu_backward
2024-04-20 21:33:00 +02:00
280023bf8e
Support for vanilla build without CUDA tools
2024-04-20 21:32:35 +02:00
9c1c1f0cec
helper target that gets compiled with lineinfo and runs a single layer forward-backward pass
2024-04-20 03:19:23 +03:00
66c30fb55f
-fno-finite-math-only for almost 2x speed up
2024-04-16 12:49:54 +02:00
752cbe068d
Update Makefile with -march=native
...
-march=native Results in a 30% speedup on all the platforms I've tried, for train_gpt2 (Admittedly, all older Intel and AMD ones)
```
model name : Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz
step 0: train loss 5.356172 (took 15197.876634 ms)
vs
step 0: train loss 5.356185 (took 10418.548668 ms)
model name : AMD Ryzen 5 3600 6-Core Processor
step 0: train loss 5.356173 (took 3599.359234 ms)
vs
step 0: train loss 5.356185 (took 2708.045790 ms)
```
2024-04-15 19:52:19 +02:00
8822e7803e
use cublaslt, which fuses bias, and also use tf32 when your GPU supports it ( #98 )
...
Fusing the bias into matmul and moving to tf32 speed up the code *a lot*, we're now down to 59ms/iter compared to 25ms/iter for pytorch compiled tf32. i.e. still off by 2.36X.
2024-04-12 15:32:07 -07:00
47b2f9312c
fixed homebrew path for intel based macs
2024-04-10 23:01:07 -04:00
80f52e5290
the full forward pass of GPT-2 in one file of pure CUDA
2024-04-10 18:15:55 +00:00
1df7bceff9
using the compiler at hand
2024-04-09 06:22:03 +02:00
Ross Wheeler