2667aa1e3e
Fix windows include issue
2024-06-24 23:18:38 +02:00
f9d0e8b67a
Add socket logic for Windows
2024-06-24 18:26:53 +00:00
05ef1e7dc0
Fix makefile
2024-06-24 14:53:58 +00:00
c420a522b2
Fix arg permutation
2024-06-24 14:38:49 +00:00
a259eeabe5
Fix CI errors
2024-06-24 16:26:12 +02:00
bfdea887eb
Add launch scripts - tested
2024-06-24 16:26:12 +02:00
c24ef4a4b4
Merge branch 'karpathy:master' into master
2024-06-07 14:20:55 +05:30
43f013631b
set compile flag and add ci check
2024-06-06 10:55:33 +03:00
1e8efe94c0
Update Makefile - revert back
2024-06-05 12:02:09 +05:30
453fde77fd
Merge branch 'karpathy:master' into master
2024-06-04 23:39:10 +05:30
4bfb23948d
Make clean fix and Windows cuDNN build fix
2024-06-03 20:26:52 -07:00
67502c7f8e
moved device utilities into a separate file, and follow-up cleanup
2024-06-03 23:58:47 +03:00
020b15ad77
small more fixes
2024-06-03 19:32:32 +00:00
6434009298
attempt to remove PATH_SEP from Makefile
2024-06-03 15:30:45 +00:00
9b540fa859
Makefile updated to match script
2024-06-03 18:59:57 +05:30
41dc65a0ef
attempt to move cudnn attention to the llmc directory
2024-06-03 03:23:04 +00:00
9e645314e6
Check if file exists using platform specific commands & Add *.o to gitignore
...
Signed-off-by: Jun Zhang <jun@junz.org >
2024-05-20 00:00:21 +08:00
6da5e63e2c
Fix detection of cudnn-frontend in '.' on Linux
...
Plus minor legibility fixes in the cudnn-frontend explanation.
2024-05-10 10:57:39 +03:00
69aa64cc80
Adds cuDNN install instructions for a smaller but sufficient package
...
- Modifies README.md to provide example apt-get cuDNN install instructions that install the cuDNN dev package. "sudo apt-get install -y cudnn" will install the default cuDNN packages, but for a minimal setup, installing the dev package will see a 50% reduction in both, download size (~850MB to 425MB now) and local storage size (~2GB to ~1GB now).
- Modifies the Makefile to point users to the README for cuDNN install instructions (through comments and the cuDNN install error message)
- Modifies attention_forward.cu comments to point users to the README for cuDNN install instructions
Signed-off-by: Vedaanta Agarwalla <vagarwalla@ipp2-1949.nvidia.com >
2024-05-08 14:53:31 -07:00
0ce5fcf7e8
Rename cudnn_att.cu to cudnn_att.cpp to speed up compilation.
2024-05-08 19:53:48 +00:00
134f4c7fc8
Adding two directory search for cuDNN frontend files
...
Search in $(HOME) and in the current directory.
2024-05-06 12:38:40 -07:00
dedd86c1b3
Merge branch 'karpathy:master' into cudnn_makefile_changes
2024-05-06 11:20:22 -07:00
0c4908d815
Adding CI check to disable auto-detect
2024-05-06 01:11:27 -07:00
2bfd2b6a7e
Auto-detect GPU capability
...
3 cases tested on Windows 11 and Ubuntu 22.04
2024-05-06 00:46:28 -07:00
2d4e5fd840
Minor fixes for Makefile for cudnn and windows support
...
Tested on Ubuntu Linux 22.04 and Windows 11
2024-05-04 23:26:57 -07:00
b087b9c819
don't compile/link cudnn if not asked for it
2024-05-05 02:54:42 +03:00
aa5bb258b6
fixed up test and profile targets
2024-05-04 11:59:59 +03:00
67a82a611e
first attempt at moving cudnn out of the main file for faster compiles
2024-05-03 23:08:44 +03:00
e780b56974
small fixes for profiling
2024-05-03 15:54:29 +00:00
69d40ebfff
add readme and search for cudd in home dir
2024-05-01 17:43:50 +00:00
35640651b3
updates to makefile: docs, override, etc
2024-05-01 17:32:52 +00:00
04718fbbd1
round 1 of small fixes for cudnn support
2024-05-01 17:07:10 +00:00
7beff72f5f
Add makefile changes for cuDNN
2024-05-01 15:31:30 +01:00
906d22f5b6
fix profile gpt2cu but note it currently needs the NO_MULTI_GPU=1 switch i think when calling it with make, todo fix later how we add the mpi flags
2024-04-29 23:27:55 +00:00
fcef97d6a1
Adding back PFLAGS
...
Got missed during the merge
Missed the profiler PFLAGS
Move PFLAGS ahead to match original change
2024-04-29 08:53:50 -07:00
c9ab8ee123
Updating Makefile with latest changes
...
Add check with Windows OS
2024-04-29 03:00:46 -07:00
18b41b4df6
Merge pull request #279 from Ricardicus/prec-ci
...
Updating the CI to build different precisions
2024-04-28 13:47:48 -07:00
49228b091a
add small comment on -t=0
2024-04-28 20:20:51 +00:00
b522333851
Updating the CI to build different precisions
2024-04-28 22:17:47 +02:00
10aa24e2cc
Merge pull request #269 from ChrisDryden/patch-3
...
Enable multithreading in nvcc
2024-04-28 13:16:47 -07:00
9d70d9a1b2
profile and test only use bf16. but the train script can be run with fp32 or bf16 or fp16. fp16 will error, though
2024-04-28 16:17:22 +00:00
4f7d8d9f44
allow user to make different precisions, add prints and error handling around precisions
2024-04-28 16:08:27 +00:00
2954d90f30
Enable multithreading in nvcc
...
Tested locally and reduced compilation time by 200ms, unfortunately for me upgrading to 12.4 made my compilations times slow by 2x but at least this can make it a bit faster
2024-04-27 14:14:02 -07:00
a00a3a88ef
fix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0
2024-04-26 18:11:09 +00:00
2e17140d33
more defensive check for openmpi install, and an install command
2024-04-26 16:27:57 +00:00
d49e8301eb
[Multi-GPU] llm.c now runs on multiple GPUs with NCCL
2024-04-24 22:11:00 +00:00
ed6387a695
updates to readme, and introduce the test fp32 cuda file too
2024-04-23 17:41:42 +00:00
095d27662b
checkpoint the fp32 CUDA implementation to separate file. our mainline iteration will now continue in the new (mixed precision) file
2024-04-23 17:21:44 +00:00
c7aad65a04
Adding a workflow that builds with and without CUDA and OMP and on CPU also tests on ubuntu and macos
2024-04-22 02:14:09 +02:00
fced6d180f
small adjustments to Makefile and gitignore
2024-04-21 17:40:26 +00:00
Aleksa Gordic