 Andrej Karpathy
|
51dd102328
|
remove confusing comment
|
2024-07-30 20:01:43 +00:00 |
|
|
Andrej Karpathy
|
2b827f1659
|
bring back state allocation into build_from_checkpoint
|
2024-07-29 02:39:50 +00:00 |
|
|
Andrej Karpathy
|
a794bcb395
|
small fixes
|
2024-07-28 17:10:20 +00:00 |
|
|
Andrej Karpathy
|
b2ae84745d
|
Merge branch 'restore_from_master_weights' of https://github.com/ademeure/llm.c into ademeure-restore_from_master_weights
|
2024-07-28 16:23:16 +00:00 |
|
|
Andrej
|
b4623bc5e7
|
Merge pull request #712 from indianspeedster/dev-cuda
Added permute kernel in dev/cuda
|
2024-07-28 06:08:20 -10:00 |
|
|
Andrej
|
7379cd10b4
|
Merge pull request #713 from mspronesti/master
fix(dev/cuda): memory leaks
|
2024-07-28 06:05:44 -10:00 |
|
|
Andrej
|
362c6a8d7f
|
Merge pull request #700 from YuchenJin/integer-overflow
Fix integer overflow by using `size_t` for parameter sizes.
|
2024-07-27 07:26:58 -10:00 |
|
|
mspronesti
|
c6e74ea322
|
fix(dev/cuda): memory leaks
|
2024-07-26 00:33:51 +01:00 |
|
|
indianspeedster
|
a80bce65e5
|
Added compilation steps in makefile for permute.cu
|
2024-07-25 18:02:06 -04:00 |
|
|
indianspeedster
|
8952c524af
|
added kernel to permute in dev/cuda
|
2024-07-25 17:35:29 -04:00 |
|
|
ademeure
|
52e6e0f80b
|
fix bug from merge (init_state set to false too late)
|
2024-07-25 21:32:01 +00:00 |
|
|
ademeure
|
4d77ece6b4
|
simplified further (don't need non-functional error checking...)
|
2024-07-25 22:04:14 +01:00 |
|
|
ademeure
|
2eabc223cc
|
simplify a little bit
|
2024-07-25 22:04:14 +01:00 |
|
|
ademeure
|
9781627a8a
|
allow restoring from checkpoint of different precision
|
2024-07-25 22:04:14 +01:00 |
|
|
ademeure
|
5cae10f907
|
make restoring from master weights actually work
|
2024-07-25 22:04:14 +01:00 |
|
|
ademeure
|
f470fbd56d
|
Allow restoring weights from the master weights of a checkpoint (deterministically by also saving RNG state of last update)
|
2024-07-25 22:04:14 +01:00 |
|
|
Andrej
|
cb4451137b
|
Merge pull request #703 from rosslwheeler/bugfix/msvc_latest_fix
Fix for upgraded Cuda 12.5.1 and Microsoft latest compiler
|
2024-07-25 13:31:41 -07:00 |
|
|
Andrej
|
e33402f7f3
|
Merge pull request #694 from ngc92/model-init-cleanup
Model init cleanup
|
2024-07-25 13:14:11 -07:00 |
|
|
Ross Wheeler
|
9dc920a66f
|
Move WIN32 code into unistd.h
|
2024-07-20 12:05:20 -07:00 |
|
|
Ross Wheeler
|
b203820232
|
Including unistd.h in cudnn_att.cpp for latest MSVC compiler
|
2024-07-20 11:57:10 -07:00 |
|
|
YuchenJin
|
30f4f223be
|
Fix integer overflow by using size_t for parameter sizes.
|
2024-07-19 08:55:44 -07:00 |
|
|
Erik Schultheis
|
9bb7a36730
|
tiny cleanup
|
2024-07-18 21:47:26 +02:00 |
|
|
Erik Schultheis
|
f2ac85ea53
|
estimate maximum batch size
|
2024-07-18 20:12:29 +02:00 |
|
|
Erik Schultheis
|
794ed07f3c
|
print memory report after allocating model state
|
2024-07-18 20:06:13 +02:00 |
|
|
Erik Schultheis
|
251421102d
|
cleanup; these mallocs are no longer needed
|
2024-07-18 20:00:51 +02:00 |
|
|
Erik Schultheis
|
bfce9bdbcf
|
bugfixes
|
2024-07-18 19:43:47 +02:00 |
|
|
Erik Schultheis
|
0bcb3e5840
|
allocate_weights utility function
reverted some earlier changes + more error checking
|
2024-07-18 19:29:49 +02:00 |
|
|
Erik Schultheis
|
cde79ffb49
|
allocate_state utility function
|
2024-07-18 19:19:06 +02:00 |
|
|
Erik Schultheis
|
5c89416adf
|
also allocate gradient eagerly
moved helper buffer allocation to forward, so backward now no longer does any memory allocation
|
2024-07-18 18:54:51 +02:00 |
|
|
Andrej Karpathy
|
72698a5ba3
|
update gpt3 125M script to be better. should exactly reproduce gpt-3 125M, except we don't use banded attention and we use fineweb
|
2024-07-18 16:43:14 +00:00 |
|
|
Erik Schultheis
|
b20def92d4
|
consolidate model init
|
2024-07-18 17:23:06 +02:00 |
|
|
Erik Schultheis
|
cbe04c3f92
|
moved dtype utilities to cuda_utils
|
2024-07-18 17:13:38 +02:00 |
|
|
Andrej
|
85d17f4aa2
|
Merge pull request #668 from abhilash1910/tpc
Add Habana gaudi2 tpc kernel link
|
2024-07-15 18:22:15 -07:00 |
|
|
Andrej Karpathy
|
835bb2e6d7
|
delete segmentation fault when B < 4 atm
|
2024-07-16 01:20:16 +00:00 |
|
|
Andrej
|
88514df231
|
Merge pull request #689 from karpathy/refactor/code-to-zerocuh
Refactor/code to zerocuh
|
2024-07-15 12:51:01 -07:00 |
|
|
Andrej Karpathy
|
5afc57064c
|
document the imports from zero.cuh to be explicit
|
2024-07-15 19:09:52 +00:00 |
|
|
Andrej Karpathy
|
882a214ad4
|
Merge branch 'move-to-zero' of https://github.com/ngc92/llm.c into ngc92-move-to-zero
|
2024-07-15 18:59:22 +00:00 |
|
|
Andrej
|
a6dc64533a
|
Merge pull request #688 from karpathy/feature/gpt3v1
feature/gpt3v1
|
2024-07-15 11:53:05 -07:00 |
|
|
Andrej Karpathy
|
220125f39c
|
ok let's allow T < max_seq_len because our tests use it. make warning very prominent
|
2024-07-15 18:21:34 +00:00 |
|
|
Andrej Karpathy
|
faf8d159a3
|
use int instead of bool
|
2024-07-15 18:01:01 +00:00 |
|
|
Andrej Karpathy
|
528b1b0e31
|
fix fat fingers
|
2024-07-15 17:56:41 +00:00 |
|
|
Andrej Karpathy
|
039904ee9f
|
revert legacy behavior in .sh scripts for -e option
|
2024-07-15 17:55:26 +00:00 |
|
|
Andrej Karpathy
|
64db3ded85
|
refactor gpt2 gpt3 descriptor to keep legacy behavior of -e, be strict everywhere, remove interpolation for now
|
2024-07-15 17:51:27 +00:00 |
|
|
Andrej Karpathy
|
4e629e22a1
|
Merge branch 'gpt3-series' of https://github.com/ngc92/llm.c into ngc92-gpt3-series
|
2024-07-15 16:46:30 +00:00 |
|
|
Andrej
|
f45c2191b6
|
Merge pull request #684 from rosslwheeler/feature/loss_checker_ci
Adding CI check for exceeding loss tolerance
|
2024-07-15 09:30:59 -07:00 |
|
|
Andrej
|
e71c1565c9
|
Merge pull request #686 from indianspeedster/cuda
Added cudaCheck wherever missing.
|
2024-07-15 09:28:02 -07:00 |
|
|
indianspeedster
|
6bb562bd88
|
added cudaCheck where missing for proper error checking.
|
2024-07-13 13:20:39 -04:00 |
|
|
Andrej
|
a6f6bc74a8
|
Merge pull request #682 from AnswerDotAI/master
Add a README link under related related projects for gpu.cpp under WebGPU C++
|
2024-07-13 10:12:59 -07:00 |
|
|
Andrej
|
0e2ff48440
|
Merge pull request #685 from planetis-m/patch-1
Add nim port
|
2024-07-13 10:08:26 -07:00 |
|
|
Antonis Geralis
|
a86a69c135
|
Add nim port
|
2024-07-13 18:57:51 +03:00 |
|