 Andrej Karpathy
|
cbe82e783a
|
consolidate memory by deleting the grad activation struct, move some of the last pieces of memory to the forward pass
|
2024-06-14 00:54:58 +00:00 |
|
|
Andrej
|
3bedc9853c
|
Merge pull request #589 from karpathy/feature/overlap-backward-reduce
Feature/overlap backward reduce
|
2024-06-13 16:31:48 -07:00 |
|
|
Andrej Karpathy
|
43a903e1b4
|
revert old printf0, minor touchups and comments, add assert
|
2024-06-13 23:18:22 +00:00 |
|
|
Andrej Karpathy
|
6515f838d9
|
Merge branch 'comp-and-comms' of https://github.com/ngc92/llm.c into ngc92-comp-and-comms2
|
2024-06-13 22:34:50 +00:00 |
|
|
Erik Schultheis
|
3fc8d6192f
|
strided all-gather
|
2024-06-14 01:33:43 +03:00 |
|
|
Andrej Karpathy
|
8367a56c03
|
Merge branch 'comp-and-comms' of https://github.com/ngc92/llm.c into ngc92-comp-and-comms2
|
2024-06-13 19:25:45 +00:00 |
|
|
Andrej
|
afcc0a4f71
|
Merge pull request #586 from gordicaleksa/readme_debug_note
Add a debugging tip to README
|
2024-06-13 11:30:20 -07:00 |
|
|
Aleksa Gordic
|
2a6797b1cd
|
Add debugging tip to README
|
2024-06-13 13:58:14 +02:00 |
|
|
Erik Schultheis
|
f071efd94f
|
stride-based adam and copy-and-cast
|
2024-06-13 02:01:26 +03:00 |
|
|
Erik Schultheis
|
99965a1e3e
|
fix typo
|
2024-06-13 00:36:34 +03:00 |
|
|
Erik Schultheis
|
9d9da3b209
|
fewer global norm kernel calls
|
2024-06-12 21:55:01 +03:00 |
|
|
Andrej
|
95cef79c17
|
Merge pull request #578 from ngc92/matmul
Remove cublaslt from fp32cu versions
|
2024-06-12 09:05:04 -07:00 |
|
|
Andrej Karpathy
|
42dff87939
|
clarification we trained with 150B tokens
|
2024-06-12 15:51:14 +00:00 |
|
|
Andrej Karpathy
|
963aa6828e
|
add gpt-2 774M repro script
|
2024-06-12 15:33:19 +00:00 |
|
|
Erik Schultheis
|
8818a75157
|
fixup to profiling script
|
2024-06-12 13:16:25 +03:00 |
|
|
Erik Schultheis
|
368644959a
|
fix non-multi-gpu case
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
8cba9ba0c0
|
cleanup
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
ecca976132
|
re-enable reduce-scatter operation
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
ecdb0ca86e
|
refactored code
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
fdcb30256d
|
first working ZeRO-1
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
6d2c72cb4b
|
rearranged adam code
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
50a9411114
|
overlap backward with nccl for non-ZeRO runs
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
d6793645f5
|
replace cublaslt with custom matmul
|
2024-06-12 09:42:19 +03:00 |
|
|
Erik Schultheis
|
7f3f3acf16
|
shared memory matmul
|
2024-06-11 21:03:43 +03:00 |
|
|
Erik Schultheis
|
071c977c79
|
vectorized memory access
|
2024-06-11 00:46:51 +03:00 |
|
|
Erik Schultheis
|
bb89efdcb1
|
register reuse
|
2024-06-11 00:41:08 +03:00 |
|
|
Andrej
|
615ec0becd
|
Merge pull request #523 from ngc92/testability
Testability
|
2024-06-09 11:02:39 -07:00 |
|
|
Erik Schultheis
|
02e30faef2
|
bugfix for LN recomputation
|
2024-06-08 21:25:27 +03:00 |
|
|
Erik Schultheis
|
8fbfba2864
|
allow setting master weights and recompute in tests
|
2024-06-08 21:03:07 +03:00 |
|
|
Erik Schultheis
|
1bf33cc66d
|
make sure failed testing returns failure status
|
2024-06-08 21:03:07 +03:00 |
|
|
Andrej
|
d2482ff1ad
|
Merge pull request #570 from rosslwheeler/feature/ubicloud-gpu-ci
Adding GPU CI workflow file
|
2024-06-08 11:01:03 -07:00 |
|
|
Andrej
|
637c1b69c4
|
Merge pull request #552 from karpathy/feature/streams
Feature/streams
|
2024-06-08 09:25:56 -07:00 |
|
|
Andrej Karpathy
|
ee6b3c9325
|
take out Async copies and memsets
|
2024-06-08 16:17:02 +00:00 |
|
|
Andrej
|
077451927f
|
Merge pull request #546 from kmyusk/master
Fix periodic inference during training
|
2024-06-07 21:37:51 -07:00 |
|
|
Andrej
|
d396cd18b7
|
Merge pull request #567 from gordicaleksa/improve_logdir_check_logic
Minor fix - better handling of out log dir
|
2024-06-07 09:22:32 -07:00 |
|
|
Andrej
|
58415736a4
|
Merge pull request #568 from gordicaleksa/improve_logger
Improve logger, add grad norm + learning rate
|
2024-06-07 09:20:29 -07:00 |
|
|
Aleksa Gordic
|
40e7289916
|
Make it cross-platform, support Windows
|
2024-06-07 17:49:49 +02:00 |
|
|
Aleksa Gordic
|
d3790c64a1
|
Change default path
|
2024-06-07 17:38:01 +02:00 |
|
|
Aleksa Gordic
|
4090ca8fa1
|
Add learning rate and grad norm logging
|
2024-06-07 17:34:42 +02:00 |
|
|
Aleksa Gordic
|
aa3d8aca86
|
Minor fix - better handling of out log dir
|
2024-06-07 17:06:53 +02:00 |
|
|
Ross Wheeler
|
73506dff2d
|
Adding GPU CI workflow file
|
2024-06-06 23:43:01 -07:00 |
|
|
Andrej
|
8a0809b39c
|
Merge pull request #560 from gordicaleksa/refactor_cuda_kernels
Fix mem leaks, reduce memory, refactor
|
2024-06-06 20:10:52 -07:00 |
|
|
Aleksa Gordic
|
7b99ac96a5
|
Fix mem leaks, reduce memory, refactor
|
2024-06-06 19:58:50 +02:00 |
|
|
Andrej
|
d931fc0507
|
-std=c++17 default, merge PR #559 from ngc92/ubuntu20.04
set compile flag and add ci check
|
2024-06-06 09:23:07 -07:00 |
|
|
Erik Schultheis
|
875196b672
|
add c++17 flag also to dev/cuda
|
2024-06-06 19:02:45 +03:00 |
|
|
Andrej
|
067aadc226
|
Merge pull request #553 from gordicaleksa/refactor_trimat
Refactor trimat
|
2024-06-06 08:56:12 -07:00 |
|
|
Erik Schultheis
|
bb9c895400
|
fix
|
2024-06-06 12:02:33 +03:00 |
|
|
Erik Schultheis
|
f4dbd88151
|
enable cudnn
|
2024-06-06 11:59:45 +03:00 |
|
|
Erik Schultheis
|
52ea988860
|
fixup image name
|
2024-06-06 11:01:10 +03:00 |
|
|
Erik Schultheis
|
43f013631b
|
set compile flag and add ci check
|
2024-06-06 10:55:33 +03:00 |
|