 Andrej Karpathy
|
14d9cd671e
|
resolve merge conflict in staate file between use_master_weights and DataLoader should_shuffle
|
2024-06-17 19:16:42 +00:00 |
|
|
Andrej Karpathy
|
93d1450bdd
|
malloc -> mallocCheck in dataloader.h
|
2024-06-17 18:56:24 +00:00 |
|
|
Aleksa Gordic
|
a0a8795be1
|
Fix guarding logic, refactor
|
2024-06-17 14:12:05 +02:00 |
|
|
Aleksa Gordic
|
8f724dbb72
|
Add a minor note that code is coupled
|
2024-06-16 23:54:09 +02:00 |
|
|
Aleksa Gordic
|
9caf2c7c2f
|
Proper handling of max num of block sums
|
2024-06-16 23:49:34 +02:00 |
|
|
Andrej
|
3e7d89b81e
|
Merge pull request #522 from gordicaleksa/add_master_weights_to_state
Add master weights to resume state
|
2024-06-16 09:22:10 -07:00 |
|
|
Aleksa Gordic
|
ff14b0819f
|
Make sure each parameter was rounded with a unique seed
|
2024-06-15 12:02:39 +00:00 |
|
|
Andrej Karpathy
|
60ed946641
|
Merge branch 'dataloader_rand' of https://github.com/gordicaleksa/llm.c into gordicaleksa-dataloader_rand
|
2024-06-14 22:10:55 +00:00 |
|
|
Aleksa Gordic
|
c8cd023063
|
Full determinism
|
2024-06-14 21:44:20 +00:00 |
|
|
Aleksa Gordic
|
0f144e3052
|
Support 100% determinism, fix race condition, update global norm
|
2024-06-14 11:57:07 +02:00 |
|
|
Aleksa Gordic
|
328641f728
|
Minor refactor
|
2024-06-14 09:57:19 +02:00 |
|
|
Aleksa Gordic
|
ef3afcb1d6
|
Fix num samples bug
|
2024-06-14 09:37:38 +02:00 |
|
|
Aleksa Gordic
|
2cd4d3985d
|
Add H100 PCIe
|
2024-06-14 09:23:55 +02:00 |
|
|
Andrej Karpathy
|
cbe82e783a
|
consolidate memory by deleting the grad activation struct, move some of the last pieces of memory to the forward pass
|
2024-06-14 00:54:58 +00:00 |
|
|
Andrej Karpathy
|
43a903e1b4
|
revert old printf0, minor touchups and comments, add assert
|
2024-06-13 23:18:22 +00:00 |
|
|
Aleksa Gordic
|
efd9df4c1b
|
Remove bool, add permutation on every shard advance
|
2024-06-13 23:36:24 +02:00 |
|
|
Aleksa Gordic
|
7e9328f724
|
Remove redundant headers, refactor
|
2024-06-13 23:15:39 +02:00 |
|
|
Aleksa Gordic
|
1329a2c80c
|
Change type from size_t to int to save space
|
2024-06-13 23:04:15 +02:00 |
|
|
Aleksa Gordic
|
814b83bbf1
|
Refactor
|
2024-06-13 22:43:57 +02:00 |
|
|
Aleksa Gordic
|
b7933ef62c
|
Modify state save/load
|
2024-06-13 21:22:16 +02:00 |
|
|
Aleksa Gordic
|
8f61f48cba
|
Add random inter/intra shard data shuffling
|
2024-06-13 20:21:49 +02:00 |
|
|
Aleksa Gordic
|
d57a52408e
|
Make C compiler happy - no default args
|
2024-06-13 20:21:49 +02:00 |
|
|
Aleksa Gordic
|
c3db1bb112
|
Set default seed value to null
|
2024-06-13 20:21:49 +02:00 |
|
|
Aleksa Gordic
|
af0b2f51c0
|
Start from a random position in a random shard
|
2024-06-13 20:21:49 +02:00 |
|
|
Erik Schultheis
|
f071efd94f
|
stride-based adam and copy-and-cast
|
2024-06-13 02:01:26 +03:00 |
|
|
Erik Schultheis
|
9d9da3b209
|
fewer global norm kernel calls
|
2024-06-12 21:55:01 +03:00 |
|
|
Erik Schultheis
|
8cba9ba0c0
|
cleanup
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
ecca976132
|
re-enable reduce-scatter operation
|
2024-06-12 13:16:19 +03:00 |
|
|
Erik Schultheis
|
ecdb0ca86e
|
refactored code
|
2024-06-12 13:16:19 +03:00 |
|
|
Andrej
|
637c1b69c4
|
Merge pull request #552 from karpathy/feature/streams
Feature/streams
|
2024-06-08 09:25:56 -07:00 |
|
|
Aleksa Gordic
|
4090ca8fa1
|
Add learning rate and grad norm logging
|
2024-06-07 17:34:42 +02:00 |
|
|
Andrej Karpathy
|
d5f7458e03
|
Merge branch 'stream-2' of https://github.com/ngc92/llm.c into ngc92-stream-2
|
2024-06-05 00:23:19 +00:00 |
|
|
Erik Schultheis
|
9af7628063
|
moved remaining memcpy/memset in the training loop to main_stream
|
2024-06-05 02:06:32 +03:00 |
|
|
Andrej Karpathy
|
8940dcdcb1
|
Merge branch 'stream-2' of https://github.com/ngc92/llm.c into ngc92-stream-2
|
2024-06-04 22:52:26 +00:00 |
|
|
Erik Schultheis
|
bc35d3438e
|
naming the stream
|
2024-06-05 01:50:16 +03:00 |
|
|
Erik Schultheis
|
0de54d10da
|
moved last remaining kernel to main stream
|
2024-06-05 01:32:25 +03:00 |
|
|
Erik Schultheis
|
ca7caac5c5
|
move all kernels into a main_stream in preparation for future parallelism
|
2024-06-04 22:35:12 +03:00 |
|
|
Erik Schultheis
|
b02aab1f16
|
made implicit includes of standard headers explicit
|
2024-06-04 13:22:16 +03:00 |
|
|
Ross Wheeler
|
4bfb23948d
|
Make clean fix and Windows cuDNN build fix
|
2024-06-03 20:26:52 -07:00 |
|
|
Andrej Karpathy
|
45b0d1b378
|
copy and cast kernel move to cuda_utils, good general util
|
2024-06-03 23:14:59 +00:00 |
|
|
Andrej Karpathy
|
d697eaecfd
|
move the global norm kernel
|
2024-06-03 23:12:20 +00:00 |
|
|
Andrej Karpathy
|
8c52a652b4
|
move adamw kernel to llmc
|
2024-06-03 23:09:42 +00:00 |
|
|
Andrej Karpathy
|
e484c650b5
|
layernorm and residual, yolo push to master
|
2024-06-03 23:02:46 +00:00 |
|
|
Andrej Karpathy
|
094988738b
|
adjust the comment on extern deviceProp to be more descriptive
|
2024-06-03 22:57:00 +00:00 |
|
|
Andrej Karpathy
|
645feac1e7
|
move matmul
|
2024-06-03 22:54:07 +00:00 |
|
|
Andrej Karpathy
|
90a7bfecf1
|
move fused classifier
|
2024-06-03 22:35:20 +00:00 |
|
|
Andrej Karpathy
|
2186cd6fe7
|
move attention
|
2024-06-03 22:19:47 +00:00 |
|
|
Andrej Karpathy
|
d8d55e0a08
|
move gelu to llmc
|
2024-06-03 21:57:02 +00:00 |
|
|
Andrej Karpathy
|
49b9776e50
|
also move the launchers
|
2024-06-03 21:37:18 +00:00 |
|
|
Andrej Karpathy
|
93546aed01
|
move encoder, kernel 1/N
|
2024-06-03 21:21:28 +00:00 |
|