 Aleksa Gordic
|
2667aa1e3e
|
Fix windows include issue
|
2024-06-24 23:18:38 +02:00 |
|
|
Aleksa Gordic
|
ac1e37b659
|
Remove linux socket funcs when windows
|
2024-06-24 18:56:48 +00:00 |
|
|
Aleksa Gordic
|
f9d0e8b67a
|
Add socket logic for Windows
|
2024-06-24 18:26:53 +00:00 |
|
|
Aleksa Gordic
|
fbf4f59c8f
|
Add MPI under multi gpu guard
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
3abdf9a3e4
|
Add gpus per node arg
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
bfdea887eb
|
Add launch scripts - tested
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
1bc3a5ef6e
|
Add all 3 init methods
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
1c6c5ff0b7
|
TCP working
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
5500b61da5
|
fs working
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
08c055cd72
|
Free up unified buffer mem
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
1bd737546c
|
Mini refactor
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
52db158367
|
Can not call set device after NCCL init
|
2024-06-24 16:26:12 +02:00 |
|
|
Aleksa Gordic
|
0addb503ef
|
Use NCCL where possible
|
2024-06-24 16:26:12 +02:00 |
|
|
Andrej Karpathy
|
98e928f1d8
|
minor fixes for disk device io
|
2024-06-23 01:14:42 +00:00 |
|
|
Andrej Karpathy
|
2543b62b88
|
Merge branch 'streams-io' of https://github.com/ngc92/llm.c into ngc92-streams-io
|
2024-06-23 00:48:05 +00:00 |
|
|
Andrej Karpathy
|
4b5fecc8b2
|
small touchups and add the wsd schedule
|
2024-06-21 00:49:01 +00:00 |
|
|
Andrej Karpathy
|
96ebb16298
|
simplify lr schedulers
|
2024-06-21 00:29:47 +00:00 |
|
|
Aleksa Gordic
|
716808d9b9
|
Use C not C++ string header
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
3e9fb6cb36
|
Fix compile issues - add comments
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
7416ee425f
|
Simplify logic since they all share similar args
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
9d141ddab9
|
Add ugly switching between schedulers
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
7308c2831c
|
Add linear LR scheduler
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
6f3d099e88
|
Add constant lr scheduler
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
4a1c5a9c0f
|
Add cyclic triangular LR scheduler
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
2eaea3b397
|
Add schedulers header
|
2024-06-20 16:09:57 +02:00 |
|
|
Aleksa Gordic
|
8249736233
|
Add learning rate schedulers
|
2024-06-20 16:09:57 +02:00 |
|
|
Erik Schultheis
|
d95313d0a9
|
explicit sync
|
2024-06-18 20:02:07 +03:00 |
|
|
Andrej
|
03cdca1b0d
|
Merge pull request #601 from gordicaleksa/fix_stochastic_rounding_2
Fix stochastic rounding in encoder backward kernel
|
2024-06-18 09:28:39 -07:00 |
|
|
Aleksa Gordic
|
e5d51c09e2
|
Add a comment for why pass a unique seed per param
|
2024-06-18 17:01:03 +02:00 |
|
|
Aleksa Gordic
|
6b2b3dc33f
|
Pass unique seeds in stochastic rounding encoder bwd
|
2024-06-18 16:40:23 +02:00 |
|
|
Aleksa Gordic
|
1120fa4465
|
Remove -1, point at current sample
|
2024-06-18 13:35:17 +02:00 |
|
|
Aleksa Gordic
|
077f9bdd59
|
Fix dataloader & determinism testing
|
2024-06-18 13:23:13 +02:00 |
|
|
Erik Schultheis
|
33136e0b08
|
fix compilation with clang
|
2024-06-18 14:16:59 +03:00 |
|
|
Erik Schultheis
|
04234d0c24
|
utility functions for device <-> disk IO
|
2024-06-18 13:23:56 +03:00 |
|
|
Andrej Karpathy
|
efbbdc8f46
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-06-18 03:03:37 +00:00 |
|
|
Andrej Karpathy
|
75b844898d
|
delete warning using an explicit cast to size_t
|
2024-06-18 03:01:06 +00:00 |
|
|
Andrej Karpathy
|
19f2c609eb
|
Merge branch 'check-determinism' of https://github.com/ngc92/llm.c into ngc92-check-determinism
|
2024-06-17 22:16:21 +00:00 |
|
|
Andrej
|
a33a49a7b6
|
Merge pull request #606 from gordicaleksa/minor_refactor_dataloader
Minor refactor of dataloader
|
2024-06-17 14:45:36 -07:00 |
|
|
Erik Schultheis
|
19a34d8cdd
|
fix dataloader
|
2024-06-18 00:25:41 +03:00 |
|
|
Andrej
|
f104009bc7
|
Merge pull request #600 from gordicaleksa/add_faster_layernorm_kernel
Use faster kernel for LayerNorm forward
|
2024-06-17 14:19:22 -07:00 |
|
|
Aleksa Gordic
|
88d4eff946
|
Swap kernel 5 for 3 for fallback
|
2024-06-17 23:05:01 +02:00 |
|
|
Erik Schultheis
|
4d35bac1ae
|
added a determinism check
|
2024-06-17 23:47:29 +03:00 |
|
|
Aleksa Gordic
|
7b5ee07e18
|
Minor refactor of dataloader
|
2024-06-17 21:53:18 +02:00 |
|
|
Andrej Karpathy
|
14d9cd671e
|
resolve merge conflict in staate file between use_master_weights and DataLoader should_shuffle
|
2024-06-17 19:16:42 +00:00 |
|
|
Andrej Karpathy
|
93d1450bdd
|
malloc -> mallocCheck in dataloader.h
|
2024-06-17 18:56:24 +00:00 |
|
|
Aleksa Gordic
|
a0a8795be1
|
Fix guarding logic, refactor
|
2024-06-17 14:12:05 +02:00 |
|
|
Aleksa Gordic
|
f845efd49a
|
Add nvtx range to layernorm forward
|
2024-06-17 09:53:58 +02:00 |
|
|
Aleksa Gordic
|
c6040eef5f
|
Even better kernel - modification of what we use for fused residual fwd
|
2024-06-17 09:53:58 +02:00 |
|
|
Aleksa Gordic
|
0f36cdd15b
|
Use faster kernel 5 for layernorm forward
|
2024-06-17 09:53:58 +02:00 |
|
|
Aleksa Gordic
|
8f724dbb72
|
Add a minor note that code is coupled
|
2024-06-16 23:54:09 +02:00 |
|