 Andrej Karpathy
|
14d9cd671e
|
resolve merge conflict in staate file between use_master_weights and DataLoader should_shuffle
|
2024-06-17 19:16:42 +00:00 |
|
|
Andrej Karpathy
|
93d1450bdd
|
malloc -> mallocCheck in dataloader.h
|
2024-06-17 18:56:24 +00:00 |
|
|
Andrej Karpathy
|
de53e71729
|
small touchups
|
2024-06-17 18:55:16 +00:00 |
|
|
Andrej
|
d57f112e16
|
Merge pull request #604 from gordicaleksa/minor_refactor_and_fix_guards
Fix guarding logic, refactor
|
2024-06-17 10:17:24 -07:00 |
|
|
Aleksa Gordic
|
9b5ec8f719
|
Set v memory to random floats >=0
|
2024-06-17 17:44:00 +02:00 |
|
|
Aleksa Gordic
|
a0a8795be1
|
Fix guarding logic, refactor
|
2024-06-17 14:12:05 +02:00 |
|
|
Andrej
|
fa7f2b6763
|
Merge pull request #602 from gordicaleksa/hotfix_global_norm_assert
Hotfix - proper handling of max num of block sums
|
2024-06-16 15:14:47 -07:00 |
|
|
Aleksa Gordic
|
0eec8cee4d
|
Patch numpy version
|
2024-06-16 23:59:29 +02:00 |
|
|
Aleksa Gordic
|
8f724dbb72
|
Add a minor note that code is coupled
|
2024-06-16 23:54:09 +02:00 |
|
|
Aleksa Gordic
|
9caf2c7c2f
|
Proper handling of max num of block sums
|
2024-06-16 23:49:34 +02:00 |
|
|
Andrej
|
13c893ffec
|
Merge pull request #561 from lancerts/fix-warning
Fix the compiler warnings and errors
|
2024-06-16 09:25:51 -07:00 |
|
|
Andrej
|
1973631c6b
|
Merge pull request #583 from vyom1611/FLT_MAX-fix
Replaced hard-coded max float with FLT_MAX
|
2024-06-16 09:25:03 -07:00 |
|
|
Andrej
|
3e7d89b81e
|
Merge pull request #522 from gordicaleksa/add_master_weights_to_state
Add master weights to resume state
|
2024-06-16 09:22:10 -07:00 |
|
|
Andrej
|
8f40ef17e3
|
Merge pull request #597 from gordicaleksa/fix_stochastic_rounding
Fix stochastic rounding
|
2024-06-16 09:13:37 -07:00 |
|
|
Andrej
|
89e5d51fa4
|
Merge pull request #596 from gordicaleksa/fix_copy_casting
Fix copy and casting from spawning excessive threads
|
2024-06-16 09:02:14 -07:00 |
|
|
Andrej
|
a4852cbafb
|
Merge pull request #598 from gordicaleksa/add_new_dev_kernel
Add global norm prod kernels to dev
|
2024-06-16 09:00:01 -07:00 |
|
|
Aleksa Gordic
|
d5601d16da
|
Fix stuff in layernorm forward
|
2024-06-16 09:34:01 +02:00 |
|
|
Aleksa Gordic
|
d003274b7c
|
Remove warp size - moved to common.h
|
2024-06-15 20:23:59 +02:00 |
|
|
Aleksa Gordic
|
09910acc0b
|
Remove blockreduce from classifier fused, move to common.h
|
2024-06-15 19:55:11 +02:00 |
|
|
Aleksa Gordic
|
0f112f1b62
|
Fix CI complaining about default args
|
2024-06-15 19:39:05 +02:00 |
|
|
Andrej Karpathy
|
1e7c31eef0
|
add test for shuffled, permuted, multi-process dataloader
|
2024-06-15 16:36:43 +00:00 |
|
|
Andrej Karpathy
|
f089f90bcf
|
add multiprocess test simple
|
2024-06-15 16:21:05 +00:00 |
|
|
Andrej Karpathy
|
d4255f1d19
|
first draft of a dataloader test
|
2024-06-15 15:56:56 +00:00 |
|
|
Aleksa Gordic
|
f3195d19af
|
Expose deviceProp as a global var
|
2024-06-15 17:40:14 +02:00 |
|
|
Aleksa Gordic
|
0f6853d399
|
Add global norm prod kernels to dev
|
2024-06-15 17:29:14 +02:00 |
|
|
Aleksa Gordic
|
ff14b0819f
|
Make sure each parameter was rounded with a unique seed
|
2024-06-15 12:02:39 +00:00 |
|
|
Aleksa Gordic
|
20f0e7f4e8
|
Fix copy and casting from spawning excessive threads
|
2024-06-15 08:15:38 +00:00 |
|
|
Andrej Karpathy
|
60ed946641
|
Merge branch 'dataloader_rand' of https://github.com/gordicaleksa/llm.c into gordicaleksa-dataloader_rand
|
2024-06-14 22:10:55 +00:00 |
|
|
Aleksa Gordic
|
c8cd023063
|
Full determinism
|
2024-06-14 21:44:20 +00:00 |
|
|
Andrej
|
0e69e3ae46
|
Merge pull request #592 from gordicaleksa/add_to_mfu_table
Add H100 PCIe to MFU table
|
2024-06-14 09:29:00 -07:00 |
|
|
Vyom Sharma
|
ee4b056ddf
|
Merge branch 'karpathy:master' into FLT_MAX-fix
|
2024-06-14 16:32:37 +05:30 |
|
|
Aleksa Gordic
|
0f144e3052
|
Support 100% determinism, fix race condition, update global norm
|
2024-06-14 11:57:07 +02:00 |
|
|
Aleksa Gordic
|
269b54ae45
|
Allocate memory in load state
|
2024-06-14 10:58:17 +02:00 |
|
|
Aleksa Gordic
|
2d23d1f08e
|
Save master weights
|
2024-06-14 10:58:17 +02:00 |
|
|
Aleksa Gordic
|
328641f728
|
Minor refactor
|
2024-06-14 09:57:19 +02:00 |
|
|
Aleksa Gordic
|
ef3afcb1d6
|
Fix num samples bug
|
2024-06-14 09:37:38 +02:00 |
|
|
Aleksa Gordic
|
2cd4d3985d
|
Add H100 PCIe
|
2024-06-14 09:23:55 +02:00 |
|
|
Andrej
|
2bc0b474c5
|
Merge pull request #590 from karpathy/feature/delete_backward_act
consolidate memory
|
2024-06-13 18:21:25 -07:00 |
|
|
Andrej Karpathy
|
cbe82e783a
|
consolidate memory by deleting the grad activation struct, move some of the last pieces of memory to the forward pass
|
2024-06-14 00:54:58 +00:00 |
|
|
Andrej
|
747a5792b7
|
Merge pull request #582 from vyom1611/master
Benchmark modal script fixed - profiling and cuDNN (Issue #504 and PR #510 fixes)
|
2024-06-13 16:47:20 -07:00 |
|
|
Andrej
|
3bedc9853c
|
Merge pull request #589 from karpathy/feature/overlap-backward-reduce
Feature/overlap backward reduce
|
2024-06-13 16:31:48 -07:00 |
|
|
Andrej Karpathy
|
43a903e1b4
|
revert old printf0, minor touchups and comments, add assert
|
2024-06-13 23:18:22 +00:00 |
|
|
Andrej Karpathy
|
6515f838d9
|
Merge branch 'comp-and-comms' of https://github.com/ngc92/llm.c into ngc92-comp-and-comms2
|
2024-06-13 22:34:50 +00:00 |
|
|
Erik Schultheis
|
3fc8d6192f
|
strided all-gather
|
2024-06-14 01:33:43 +03:00 |
|
|
Aleksa Gordic
|
6c10cf2791
|
Remove local paths
|
2024-06-13 23:51:07 +02:00 |
|
|
Aleksa Gordic
|
efd9df4c1b
|
Remove bool, add permutation on every shard advance
|
2024-06-13 23:36:24 +02:00 |
|
|
Aleksa Gordic
|
7e9328f724
|
Remove redundant headers, refactor
|
2024-06-13 23:15:39 +02:00 |
|
|
Aleksa Gordic
|
1329a2c80c
|
Change type from size_t to int to save space
|
2024-06-13 23:04:15 +02:00 |
|
|
Aleksa Gordic
|
814b83bbf1
|
Refactor
|
2024-06-13 22:43:57 +02:00 |
|
|
Aleksa Gordic
|
1dafa60ad9
|
Hardcode shuffle info in all train scripts
|
2024-06-13 21:27:42 +02:00 |
|