 Andrej Karpathy
|
1a66151a80
|
small fixes and readme
|
2024-06-19 01:03:36 +00:00 |
|
|
Andrej Karpathy
|
d7ea4da298
|
add scripts to export to HF and run Eleuther evals
|
2024-06-14 19:49:05 +00:00 |
|
|
Andrej
|
0e69e3ae46
|
Merge pull request #592 from gordicaleksa/add_to_mfu_table
Add H100 PCIe to MFU table
|
2024-06-14 09:29:00 -07:00 |
|
|
Aleksa Gordic
|
2cd4d3985d
|
Add H100 PCIe
|
2024-06-14 09:23:55 +02:00 |
|
|
Andrej
|
2bc0b474c5
|
Merge pull request #590 from karpathy/feature/delete_backward_act
consolidate memory
|
2024-06-13 18:21:25 -07:00 |
|
|
Andrej Karpathy
|
cbe82e783a
|
consolidate memory by deleting the grad activation struct, move some of the last pieces of memory to the forward pass
|
2024-06-14 00:54:58 +00:00 |
|
|
Andrej
|
747a5792b7
|
Merge pull request #582 from vyom1611/master
Benchmark modal script fixed - profiling and cuDNN (Issue #504 and PR #510 fixes)
|
2024-06-13 16:47:20 -07:00 |
|
|
Andrej
|
3bedc9853c
|
Merge pull request #589 from karpathy/feature/overlap-backward-reduce
Feature/overlap backward reduce
|
2024-06-13 16:31:48 -07:00 |
|
|
Andrej Karpathy
|
43a903e1b4
|
revert old printf0, minor touchups and comments, add assert
|
2024-06-13 23:18:22 +00:00 |
|
|
Andrej Karpathy
|
6515f838d9
|
Merge branch 'comp-and-comms' of https://github.com/ngc92/llm.c into ngc92-comp-and-comms2
|
2024-06-13 22:34:50 +00:00 |
|
|
Erik Schultheis
|
3fc8d6192f
|
strided all-gather
|
2024-06-14 01:33:43 +03:00 |
|
|
Andrej Karpathy
|
8367a56c03
|
Merge branch 'comp-and-comms' of https://github.com/ngc92/llm.c into ngc92-comp-and-comms2
|
2024-06-13 19:25:45 +00:00 |
|
|
Andrej
|
afcc0a4f71
|
Merge pull request #586 from gordicaleksa/readme_debug_note
Add a debugging tip to README
|
2024-06-13 11:30:20 -07:00 |
|
|
Aleksa Gordic
|
2a6797b1cd
|
Add debugging tip to README
|
2024-06-13 13:58:14 +02:00 |
|
|
Erik Schultheis
|
f071efd94f
|
stride-based adam and copy-and-cast
|
2024-06-13 02:01:26 +03:00 |
|
|
Erik Schultheis
|
99965a1e3e
|
fix typo
|
2024-06-13 00:36:34 +03:00 |
|
|
Erik Schultheis
|
9d9da3b209
|
fewer global norm kernel calls
|
2024-06-12 21:55:01 +03:00 |
|
|
Vyom Sharma
|
afa7978227
|
Merge branch 'karpathy:master' into master
|
2024-06-13 00:19:16 +05:30 |
|
|
Andrej
|
95cef79c17
|
Merge pull request #578 from ngc92/matmul
Remove cublaslt from fp32cu versions
|
2024-06-12 09:05:04 -07:00 |
|
|
Andrej Karpathy
|
42dff87939
|
clarification we trained with 150B tokens
|
2024-06-12 15:51:14 +00:00 |
|
|
Andrej Karpathy
|
963aa6828e
|
add gpt-2 774M repro script
|
2024-06-12 15:33:19 +00:00 |
|
|
Erik Schultheis
|
8818a75157
|
fixup to profiling script
|
2024-06-12 13:16:25 +03:00 |
|
|
Erik Schultheis
|
368644959a
|
fix non-multi-gpu case
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
8cba9ba0c0
|
cleanup
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
ecca976132
|
re-enable reduce-scatter operation
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
ecdb0ca86e
|
refactored code
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
fdcb30256d
|
first working ZeRO-1
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
6d2c72cb4b
|
rearranged adam code
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
50a9411114
|
overlap backward with nccl for non-ZeRO runs
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
d6793645f5
|
replace cublaslt with custom matmul
|
2024-06-12 09:42:19 +03:00 |
|
|
Erik Schultheis
|
7f3f3acf16
|
shared memory matmul
|
2024-06-11 21:03:43 +03:00 |
|
|
Erik Schultheis
|
071c977c79
|
vectorized memory access
|
2024-06-11 00:46:51 +03:00 |
|
|
Erik Schultheis
|
bb89efdcb1
|
register reuse
|
2024-06-11 00:41:08 +03:00 |
|
|
Vyom Sharma
|
8a3a29ecb7
|
Merge branch 'karpathy:master' into master
|
2024-06-10 13:54:29 +05:30 |
|
|
Andrej
|
615ec0becd
|
Merge pull request #523 from ngc92/testability
Testability
|
2024-06-09 11:02:39 -07:00 |
|
|
Erik Schultheis
|
02e30faef2
|
bugfix for LN recomputation
|
2024-06-08 21:25:27 +03:00 |
|
|
Erik Schultheis
|
8fbfba2864
|
allow setting master weights and recompute in tests
|
2024-06-08 21:03:07 +03:00 |
|
|
Erik Schultheis
|
1bf33cc66d
|
make sure failed testing returns failure status
|
2024-06-08 21:03:07 +03:00 |
|
|
Andrej
|
d2482ff1ad
|
Merge pull request #570 from rosslwheeler/feature/ubicloud-gpu-ci
Adding GPU CI workflow file
|
2024-06-08 11:01:03 -07:00 |
|
|
Andrej
|
637c1b69c4
|
Merge pull request #552 from karpathy/feature/streams
Feature/streams
|
2024-06-08 09:25:56 -07:00 |
|
|
Andrej Karpathy
|
ee6b3c9325
|
take out Async copies and memsets
|
2024-06-08 16:17:02 +00:00 |
|
|
Andrej
|
077451927f
|
Merge pull request #546 from kmyusk/master
Fix periodic inference during training
|
2024-06-07 21:37:51 -07:00 |
|
|
Andrej
|
d396cd18b7
|
Merge pull request #567 from gordicaleksa/improve_logdir_check_logic
Minor fix - better handling of out log dir
|
2024-06-07 09:22:32 -07:00 |
|
|
Andrej
|
58415736a4
|
Merge pull request #568 from gordicaleksa/improve_logger
Improve logger, add grad norm + learning rate
|
2024-06-07 09:20:29 -07:00 |
|
|
Aleksa Gordic
|
40e7289916
|
Make it cross-platform, support Windows
|
2024-06-07 17:49:49 +02:00 |
|
|
Aleksa Gordic
|
d3790c64a1
|
Change default path
|
2024-06-07 17:38:01 +02:00 |
|
|
Aleksa Gordic
|
4090ca8fa1
|
Add learning rate and grad norm logging
|
2024-06-07 17:34:42 +02:00 |
|
|
Aleksa Gordic
|
aa3d8aca86
|
Minor fix - better handling of out log dir
|
2024-06-07 17:06:53 +02:00 |
|
|
vyom1611
|
ecb13ea74d
|
missing parameter in comments added
|
2024-06-07 16:14:59 +05:30 |
|
|
vyom1611
|
009be4dcab
|
Merge remote-tracking branch 'origin/master'
|
2024-06-07 14:21:35 +05:30 |
|