 Andrej Karpathy
|
e6856bc566
|
fallback to memory allocation of m,v,master_weights on host automatically in case of OOM. will run slower but won't OOM
|
2024-08-16 17:23:42 +00:00 |
|
|
Erik Schultheis
|
f72c1f2c64
|
hint to host
|
2024-08-16 01:44:08 +03:00 |
|
|
Erik Schultheis
|
c8457572fa
|
just try to allocate on device; fallback if that fails
|
2024-08-16 01:43:56 +03:00 |
|
|
Erik Schultheis
|
0d52d2a3d7
|
fall back to cudaMallocManaged for optimizer states if we're out of memory
|
2024-08-16 01:42:30 +03:00 |
|
|
Andrej
|
ef12d1b80e
|
Merge pull request #717 from ngc92/nvml
Nvidia management library for more detailed GPU state printing
|
2024-07-30 13:10:23 -07:00 |
|
|
Erik Schultheis
|
e4b5e5775f
|
disable if header is not found
|
2024-07-29 10:07:37 +02:00 |
|
|
Erik Schultheis
|
bdff672741
|
add temperature
|
2024-07-29 09:40:50 +02:00 |
|
|
Erik Schultheis
|
294a6bca55
|
throttle reason
|
2024-07-29 09:40:50 +02:00 |
|
|
Erik Schultheis
|
743062142c
|
units + more frequent reporting
|
2024-07-29 09:40:50 +02:00 |
|
|
Erik Schultheis
|
7d7954caa7
|
nvml for more detailed gpu status info
|
2024-07-29 09:40:50 +02:00 |
|
|
Andrej Karpathy
|
a794bcb395
|
small fixes
|
2024-07-28 17:10:20 +00:00 |
|
|
ademeure
|
4d77ece6b4
|
simplified further (don't need non-functional error checking...)
|
2024-07-25 22:04:14 +01:00 |
|
|
ademeure
|
2eabc223cc
|
simplify a little bit
|
2024-07-25 22:04:14 +01:00 |
|
|
ademeure
|
5cae10f907
|
make restoring from master weights actually work
|
2024-07-25 22:04:14 +01:00 |
|
|
ademeure
|
f470fbd56d
|
Allow restoring weights from the master weights of a checkpoint (deterministically by also saving RNG state of last update)
|
2024-07-25 22:04:14 +01:00 |
|
|
Andrej
|
cb4451137b
|
Merge pull request #703 from rosslwheeler/bugfix/msvc_latest_fix
Fix for upgraded Cuda 12.5.1 and Microsoft latest compiler
|
2024-07-25 13:31:41 -07:00 |
|
|
Ross Wheeler
|
9dc920a66f
|
Move WIN32 code into unistd.h
|
2024-07-20 12:05:20 -07:00 |
|
|
Ross Wheeler
|
b203820232
|
Including unistd.h in cudnn_att.cpp for latest MSVC compiler
|
2024-07-20 11:57:10 -07:00 |
|
|
Erik Schultheis
|
cbe04c3f92
|
moved dtype utilities to cuda_utils
|
2024-07-18 17:13:38 +02:00 |
|
|
Andrej Karpathy
|
835bb2e6d7
|
delete segmentation fault when B < 4 atm
|
2024-07-16 01:20:16 +00:00 |
|
|
Andrej Karpathy
|
882a214ad4
|
Merge branch 'move-to-zero' of https://github.com/ngc92/llm.c into ngc92-move-to-zero
|
2024-07-15 18:59:22 +00:00 |
|
|
Andrej Karpathy
|
faf8d159a3
|
use int instead of bool
|
2024-07-15 18:01:01 +00:00 |
|
|
Andrej Karpathy
|
64db3ded85
|
refactor gpt2 gpt3 descriptor to keep legacy behavior of -e, be strict everywhere, remove interpolation for now
|
2024-07-15 17:51:27 +00:00 |
|
|
indianspeedster
|
6bb562bd88
|
added cudaCheck where missing for proper error checking.
|
2024-07-13 13:20:39 -04:00 |
|
|
Erik Schultheis
|
39a41fa701
|
move cpu float reduction to zero
|
2024-07-12 11:12:58 +02:00 |
|
|
Erik Schultheis
|
4d1ec6832a
|
rename parameters to avoid name clashes with global variable
|
2024-07-11 19:04:02 +02:00 |
|
|
Erik Schultheis
|
09b78bc902
|
move set_zero_configs into zero.cuh
|
2024-07-11 09:55:31 +02:00 |
|
|
Aleksa Gordic
|
6b62f3b9ae
|
Fix linker issue
|
2024-07-02 21:54:50 +02:00 |
|
|
Aleksa Gordic
|
d27331cced
|
Move socket header to utils.h
|
2024-07-02 21:17:28 +02:00 |
|
|
Aleksa Gordic
|
5645bd18b1
|
CI fix - add sockets header
|
2024-07-02 21:17:28 +02:00 |
|
|
Aleksa Gordic
|
1ae3243185
|
Add check versions of socket close
|
2024-07-02 21:17:28 +02:00 |
|
|
ademeure
|
6e29dd000d
|
fix default for test_gpt2cu
|
2024-06-30 20:45:37 +01:00 |
|
|
ademeure
|
0f11a4ec49
|
Make GELU fusion default 2 for H100+ and 0 on older GPUs for now
|
2024-06-30 20:45:37 +01:00 |
|
|
ademeure
|
fa5a196be0
|
Refactor matmul to only use cuBLASLt and optionally use fused GELU
|
2024-06-30 20:45:37 +01:00 |
|
|
Andrej
|
2daaf6ab83
|
Merge pull request #629 from rosslwheeler/feature/tests
CI Dataloader test and ptx/sass file generator
|
2024-06-29 15:17:31 -07:00 |
|
|
Andrej Karpathy
|
2cf7c5b7d3
|
Merge branch 'ngc92-mixed-dtypes-2'
|
2024-06-27 17:47:22 +00:00 |
|
|
ademeure
|
29a067f916
|
Make cuDNN deterministic for Flash Attention backward (requires frontend >=1.5)
|
2024-06-27 13:33:02 +00:00 |
|
|
rosslwheeler
|
267efc9a0e
|
Merge branch 'karpathy:master' into feature/tests
|
2024-06-26 22:38:21 -07:00 |
|
|
Erik Schultheis
|
7beaa1ed96
|
keep layernorm stats in fp32
|
2024-06-26 20:48:11 +03:00 |
|
|
Erik Schultheis
|
f0c3e35de8
|
switch losses to fp32
|
2024-06-26 20:48:08 +03:00 |
|
|
Andrej Karpathy
|
e12d090483
|
resolve merge conflicts
|
2024-06-26 04:05:46 +00:00 |
|
|
Erik Schultheis
|
6c24f13b55
|
check token validity
|
2024-06-26 02:01:24 +03:00 |
|
|
Erik Schultheis
|
09503dea4f
|
make windows happy
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
8b36c2f60b
|
fix loss accumulation
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
75d4610442
|
update norm computation: multi-process reduce on GPU
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
8d994782cf
|
moved loss calculation to backward
|
2024-06-25 21:28:52 +03:00 |
|
|
Erik Schultheis
|
4745f53589
|
boolean_constant utilities
|
2024-06-25 21:28:52 +03:00 |
|
|
Ross Wheeler
|
2e6fd9b150
|
Add unistd.h for Windows cuDNN build
Small fix - move WIN32 change into dev.unistd.h
Add USE_CUDNN=1 build for Windows for CI
|
2024-06-25 10:36:59 -07:00 |
|
|
rosslwheeler
|
2e32e25e37
|
Merge branch 'karpathy:master' into feature/tests
|
2024-06-24 23:05:55 -07:00 |
|
|
Andrej Karpathy
|
7a0d864f37
|
oops i was only using window size 16 for testing let's make it 128 or something
|
2024-06-25 01:56:04 +00:00 |
|