 Andrej Karpathy
|
5959851d2d
|
small comments for pyrun124M.sh
|
2024-05-30 14:47:43 +00:00 |
|
|
Andrej Karpathy
|
9b935a8f04
|
fix bug with advancing position in dataloader
|
2024-05-30 13:16:07 +00:00 |
|
|
Andrej Karpathy
|
555d07502a
|
more feature parity work, i think it's quite close now
|
2024-05-30 02:29:40 +00:00 |
|
|
Andrej Karpathy
|
ea88a343b5
|
also updated pyrun124M.sh
|
2024-05-30 02:05:41 +00:00 |
|
|
Andrej Karpathy
|
03c2e85601
|
sharded dataloader for PyTorch
|
2024-05-30 02:05:06 +00:00 |
|
|
Andrej Karpathy
|
3fdf88e4d4
|
add logfile similar to C
|
2024-05-29 23:59:35 +00:00 |
|
|
Andrej Karpathy
|
083c50a52e
|
small stylistic touchups that match the c code
|
2024-05-29 23:50:13 +00:00 |
|
|
Andrej Karpathy
|
1f6fdfbd9a
|
add pyrun124M.sh
|
2024-05-29 23:30:27 +00:00 |
|
|
Andrej Karpathy
|
c413bfe0ca
|
pytorch parity v1
|
2024-05-29 23:25:55 +00:00 |
|
|
Andrej Karpathy
|
4cc8b1570b
|
hardcode grad_clip to 1.0, delete confusion around legacy code that used -c flag for it, but became inactive in the if else block
|
2024-05-29 19:01:12 +00:00 |
|
|
Andrej
|
55f3665b64
|
Merge pull request #493 from vyom1611/master-2
Modal benchmarking script updated to replace deprecated calls
|
2024-05-29 07:50:10 -07:00 |
|
|
vyom1611
|
062f096da1
|
Modal benchmarking script updated to replace deprecated calls
|
2024-05-29 20:01:41 +05:30 |
|
|
Andrej
|
2db7716d68
|
Merge pull request #476 from lancerts/layernorm-bk-fix
Fix "Mismatch of dweight at layernorm_backward.cu #428"
|
2024-05-29 07:30:52 -07:00 |
|
|
Andrej
|
c3a65a556f
|
Merge pull request #487 from huoyushequ/patch-1
`softmax_autoregressive_backward_kernel` does not use share memory in the kernel
|
2024-05-28 18:52:38 -07:00 |
|
|
QIAOQIAN
|
27cf5223e0
|
softmax_autoregressive_backward_kernel does not use share memory in the kernel
we do not need to launch the kernel with 256 bytes share memory, so remove it
|
2024-05-29 09:23:46 +08:00 |
|
|
Andrej Karpathy
|
a664aa3094
|
include datasets as req
|
2024-05-28 21:43:11 +00:00 |
|
|
Andrej
|
afe40be994
|
Merge pull request #479 from rosslwheeler/feature/ci-manual-trigger
Adding manual trigger for CI
|
2024-05-28 12:46:27 -07:00 |
|
|
Andrej Karpathy
|
88dd670c42
|
link to recent post from readme
|
2024-05-28 17:31:31 +00:00 |
|
|
Andrej Karpathy
|
a6a2f042f8
|
keep -y 1 flag
|
2024-05-28 15:24:30 +00:00 |
|
|
Andrej Karpathy
|
a7f01088dc
|
repro scripts for 124M and 350M models
|
2024-05-28 15:08:04 +00:00 |
|
|
Ross Wheeler
|
c60d7de466
|
Adding manual trigger for CI
|
2024-05-27 16:50:06 -07:00 |
|
|
lancerts
|
47c670b885
|
amend
|
2024-05-27 15:13:11 -07:00 |
|
|
lancerts
|
c4c985cb87
|
spell explicitly uint to unsigned int
|
2024-05-27 14:44:47 -07:00 |
|
|
lancerts
|
f27ca4df41
|
fix the issue Mismatch of dweight at layernorm_backward.cu
|
2024-05-27 14:39:57 -07:00 |
|
|
Andrej
|
b67880a879
|
Merge pull request #471 from karpathy/feature/resume
resume training v1
|
2024-05-27 13:38:52 -07:00 |
|
|
Andrej Karpathy
|
24a8c03cb6
|
Merge branch 'master' into feature/resume
|
2024-05-27 20:30:30 +00:00 |
|
|
Andrej Karpathy
|
69d0583ac8
|
conditionally include dirent on not windows
|
2024-05-27 20:28:52 +00:00 |
|
|
Andrej
|
3b4d8bece6
|
Merge pull request #474 from rosslwheeler/Windows-dirent-fix
Adding dirent.h support to unistd.h
|
2024-05-27 13:27:58 -07:00 |
|
|
Ross Wheeler
|
bb80f93618
|
Minor cleanup
Added static inline to the functions
|
2024-05-27 13:16:34 -07:00 |
|
|
Ross Wheeler
|
758acdffbf
|
Adding dirent.h support to unistd.h
|
2024-05-27 13:08:07 -07:00 |
|
|
Andrej Karpathy
|
63f0e25f5a
|
make compiler happy
|
2024-05-27 19:32:31 +00:00 |
|
|
Andrej Karpathy
|
b75738c4eb
|
resume optimization, seems to be working
|
2024-05-27 18:02:51 +00:00 |
|
|
Andrej Karpathy
|
f93a30fbbf
|
more careful with conditional MPI use
|
2024-05-27 15:54:55 +00:00 |
|
|
Andrej Karpathy
|
d295cb8d81
|
part 1 of v1 of resume training functionality, writes the files but doesn't load them yet, coming up in a bit
|
2024-05-27 15:49:03 +00:00 |
|
|
Andrej
|
12999f7082
|
Merge pull request #470 from ngc92/bugfix
bugfix for layernorm correctness (dev/cuda did not trip, but end to end test `make test_gpt2cu PRECISION=FP32 && ./test_gpt2cu
` did)
|
2024-05-27 06:39:56 -07:00 |
|
|
Erik Schultheis
|
7dc3b7d7dc
|
bugfix
|
2024-05-27 16:36:44 +03:00 |
|
|
Andrej
|
4d6f80e02b
|
Merge pull request #469 from ahrefs/zero1-torch
Optional Zero1 changes for pytorch training
|
2024-05-27 06:04:00 -07:00 |
|
|
Andrej
|
c3097ba308
|
Merge pull request #468 from ngc92/layernorm-bw-deadlock
Layernorm deadlock
|
2024-05-27 05:48:42 -07:00 |
|
|
Erik Schultheis
|
b913905f84
|
fix out-of-bounds access for encoder_backward: we were allocating too little memory (/ -> DIV_CEIL)
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
841c4dec03
|
update main training file
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
5f73ecfcf1
|
fix C % 256 != 0
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
d35daf19c9
|
fail fast and hard; don't go into the deadlock
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
3b6808269a
|
some comments and optimized shared memory amount
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
9eba86f0a5
|
fully vectorized smem access
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
cb8cc25e3a
|
utilities for Packed and more vectorization
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
53ee3297dd
|
use vectorized access to shared memory
|
2024-05-27 13:15:36 +03:00 |
|
|
Chinthaka Gamanayakege
|
f2809a6bdb
|
zero changes for torch
|
2024-05-27 06:44:54 +00:00 |
|
|
Andrej Karpathy
|
ee6fa5ec9f
|
fill in the other model sizes
|
2024-05-26 18:04:43 +00:00 |
|
|
Andrej Karpathy
|
5ec4b774f2
|
simple jupyter notebook example on how to parse and vis the logfiles
|
2024-05-26 17:40:49 +00:00 |
|
|
Andrej Karpathy
|
f145c1b101
|
fix the other files due to dataloader change
|
2024-05-26 17:01:32 +00:00 |
|