 Andrej Karpathy
|
11e2fd1cf2
|
separate out write_checkpoint and delete_checkpoint
|
2024-06-25 20:46:03 +00:00 |
|
|
Andrej Karpathy
|
8952f0235b
|
rolling checkpoints - checkpoints are either MINOR or MAJOR and minor checkpoints get deleted with a rolling window. This is an optimization that will allow us to save state more often, but preserve disk space overall. And if we ever see loss spikes and such, it is easier to reset the state to an earlier checkpoint that is more recent
|
2024-06-24 23:39:18 +00:00 |
|
|
Andrej
|
76b133d3f4
|
Merge pull request #631 from gordicaleksa/minor_refactor2
Minor refactor
|
2024-06-24 14:38:27 -07:00 |
|
|
Andrej
|
69b50adcf5
|
Merge pull request #632 from gordicaleksa/multi_node_my
MPI/TCP/FS for NCCL-init
|
2024-06-24 14:36:30 -07:00 |
|
|
Aleksa Gordic
|
4af6a6ab3d
|
Remove /tmp as default init value for fs_path
|
2024-06-24 21:26:55 +00:00 |
|
|
Aleksa Gordic
|
2667aa1e3e
|
Fix windows include issue
|
2024-06-24 23:18:38 +02:00 |
|
|
Aleksa Gordic
|
ac1e37b659
|
Remove linux socket funcs when windows
|
2024-06-24 18:56:48 +00:00 |
|
|
Aleksa Gordic
|
60c95c82aa
|
Add a multi node readme section
|
2024-06-24 18:50:56 +00:00 |
|
|
Aleksa Gordic
|
f9d0e8b67a
|
Add socket logic for Windows
|
2024-06-24 18:26:53 +00:00 |
|
|
Aleksa Gordic
|
05ef1e7dc0
|
Fix makefile
|
2024-06-24 14:53:58 +00:00 |
|
|
Aleksa Gordic
|
c420a522b2
|
Fix arg permutation
|
2024-06-24 14:38:49 +00:00 |
|
|
Aleksa Gordic
|
fbf4f59c8f
|
Add MPI under multi gpu guard
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
a259eeabe5
|
Fix CI errors
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
3abdf9a3e4
|
Add gpus per node arg
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
75a2c8b10f
|
Add usage for new args
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
bfdea887eb
|
Add launch scripts - tested
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
1bc3a5ef6e
|
Add all 3 init methods
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
1c6c5ff0b7
|
TCP working
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
5500b61da5
|
fs working
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
08c055cd72
|
Free up unified buffer mem
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
1bd737546c
|
Mini refactor
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
ac72875c24
|
Use NCCL stream
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
52db158367
|
Can not call set device after NCCL init
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
0addb503ef
|
Use NCCL where possible
|
2024-06-24 16:26:12 +02:00 |
|
|
Andrej Karpathy
|
2a4be7f349
|
Merge branch 'ngc92-streams-io'
|
2024-06-23 01:14:50 +00:00 |
|
|
Andrej Karpathy
|
98e928f1d8
|
minor fixes for disk device io
|
2024-06-23 01:14:42 +00:00 |
|
|
Andrej Karpathy
|
2543b62b88
|
Merge branch 'streams-io' of https://github.com/ngc92/llm.c into ngc92-streams-io
|
2024-06-23 00:48:05 +00:00 |
|
|
Aleksa Gordic
|
71f10e7960
|
Rename gpt2 backward
|
2024-06-22 14:32:51 +02:00 |
|
|
Andrej Karpathy
|
72a2158812
|
remove a spurious print that is making the table a bit uglier
|
2024-06-21 15:08:30 +00:00 |
|
|
Andrej Karpathy
|
c6e21c78b5
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-06-21 02:40:19 +00:00 |
|
|
Andrej Karpathy
|
61ed938dfb
|
delete spurious check it's fine
|
2024-06-21 02:40:11 +00:00 |
|
|
Andrej
|
5b82822462
|
Merge pull request #627 from karpathy/feature/lr_schedulers
feature/lr_schedulers
|
2024-06-20 19:31:42 -07:00 |
|
|
Andrej Karpathy
|
4b5fecc8b2
|
small touchups and add the wsd schedule
|
2024-06-21 00:49:01 +00:00 |
|
|
Andrej Karpathy
|
96ebb16298
|
simplify lr schedulers
|
2024-06-21 00:29:47 +00:00 |
|
|
Andrej Karpathy
|
9abd9ec8a8
|
Merge branch 'add_lr_schedules' of https://github.com/gordicaleksa/llm.c into gordicaleksa-add_lr_schedules
|
2024-06-21 00:12:15 +00:00 |
|
|
Andrej
|
dad58b6800
|
Merge pull request #625 from gordicaleksa/add_nccl_to_readme
Add NCCL instruction to README
|
2024-06-20 16:06:31 -07:00 |
|
|
Andrej Karpathy
|
2677d82dcb
|
add shell script that downloads the .bin files of FineWeb100B directly
|
2024-06-20 22:39:09 +00:00 |
|
|
Aleksa Gordic
|
5b4d11ac1d
|
Add NCCL instruction
|
2024-06-21 00:16:17 +02:00 |
|
|
Aleksa Gordic
|
716808d9b9
|
Use C not C++ string header
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
3e9fb6cb36
|
Fix compile issues - add comments
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
7416ee425f
|
Simplify logic since they all share similar args
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
9d141ddab9
|
Add ugly switching between schedulers
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
7308c2831c
|
Add linear LR scheduler
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
6f3d099e88
|
Add constant lr scheduler
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
4a1c5a9c0f
|
Add cyclic triangular LR scheduler
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
2eaea3b397
|
Add schedulers header
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
8249736233
|
Add learning rate schedulers
|
2024-06-20 16:09:57 +02:00 |
|
|
Andrej
|
2ddf128365
|
Merge pull request #615 from gordicaleksa/relax_grad_tests
Relax grad tensor thresholds in tests
|
2024-06-19 09:37:42 -07:00 |
|
|
Aleksa Gordic
|
8079bd9dbb
|
Relax grad tensor thresholds a bit
|
2024-06-19 10:21:59 +02:00 |
|
|
Andrej Karpathy
|
fd564f7b8a
|
add the gpt2 1558M repro script too
|
2024-06-19 02:26:39 +00:00 |
|