 Aleksa Gordic
|
7b99ac96a5
|
Fix mem leaks, reduce memory, refactor
|
2024-06-06 19:58:50 +02:00 |
|
|
Andrej
|
d931fc0507
|
-std=c++17 default, merge PR #559 from ngc92/ubuntu20.04
set compile flag and add ci check
|
2024-06-06 09:23:07 -07:00 |
|
|
Erik Schultheis
|
875196b672
|
add c++17 flag also to dev/cuda
|
2024-06-06 19:02:45 +03:00 |
|
|
Aleksa Gordic
|
c116fbfa29
|
Fix q,k bug, move all kernels to q@k^t form
|
2024-06-05 22:22:55 +02:00 |
|
|
Aleksa Gordic
|
875d362381
|
Refactor trimat
|
2024-06-05 17:33:26 +02:00 |
|
|
vyom1611
|
4564a24c69
|
Modal script updated for cudnn and gcc version match
|
2024-06-04 23:39:36 +05:30 |
|
|
vyom1611
|
fbc1e4983c
|
Benchmark modal script revamped - profiling and external libraries
|
2024-06-03 17:00:10 +05:30 |
|
|
vyom1611
|
0867653aaf
|
Merge remote-tracking branch 'origin/master'
|
2024-06-03 16:48:58 +05:30 |
|
|
Andrej
|
80e0c3b8d9
|
Merge pull request #512 from gordicaleksa/refactor_encoder_bwd_kernel
Remove redundant CPU computation in encoder bwd
|
2024-06-02 16:43:10 -07:00 |
|
|
Aleksa Gordic
|
290c00a362
|
Remove redundant CPU computation
|
2024-06-01 18:27:24 +02:00 |
|
|
Andrej
|
a26041af12
|
Merge pull request #495 from ChrisDryden/shared_memory
Removed unnecesary shared memory due to blockreduce using static defined shared memory
|
2024-06-01 08:12:21 -07:00 |
|
|
Aleksa Gordic
|
8eafd40615
|
Update compile cmd in the dev/cuda README
|
2024-06-01 16:46:32 +02:00 |
|
|
vyom1611
|
e0abfae2bc
|
Benchmark modal script revamped - profiling and external libraries
|
2024-06-01 19:06:42 +05:30 |
|
|
Andrej Karpathy
|
f8f9e49ba3
|
Merge branch 'master' into feature/libdir
|
2024-05-30 19:07:51 +00:00 |
|
|
Christopher
|
5450632237
|
Removed unnecesary shared memory due to blockreduce using static defined shared memory
|
2024-05-30 03:01:52 +00:00 |
|
|
Andrej
|
55f3665b64
|
Merge pull request #493 from vyom1611/master-2
Modal benchmarking script updated to replace deprecated calls
|
2024-05-29 07:50:10 -07:00 |
|
|
vyom1611
|
062f096da1
|
Modal benchmarking script updated to replace deprecated calls
|
2024-05-29 20:01:41 +05:30 |
|
|
Andrej
|
2db7716d68
|
Merge pull request #476 from lancerts/layernorm-bk-fix
Fix "Mismatch of dweight at layernorm_backward.cu #428"
|
2024-05-29 07:30:52 -07:00 |
|
|
Andrej Karpathy
|
0c83a23188
|
tweak the logfile parser and visualizer
|
2024-05-28 01:38:15 +00:00 |
|
|
lancerts
|
47c670b885
|
amend
|
2024-05-27 15:13:11 -07:00 |
|
|
lancerts
|
c4c985cb87
|
spell explicitly uint to unsigned int
|
2024-05-27 14:44:47 -07:00 |
|
|
lancerts
|
f27ca4df41
|
fix the issue Mismatch of dweight at layernorm_backward.cu
|
2024-05-27 14:39:57 -07:00 |
|
|
Ross Wheeler
|
bb80f93618
|
Minor cleanup
Added static inline to the functions
|
2024-05-27 13:16:34 -07:00 |
|
|
Ross Wheeler
|
758acdffbf
|
Adding dirent.h support to unistd.h
|
2024-05-27 13:08:07 -07:00 |
|
|
Erik Schultheis
|
7dc3b7d7dc
|
bugfix
|
2024-05-27 16:36:44 +03:00 |
|
|
Erik Schultheis
|
5f73ecfcf1
|
fix C % 256 != 0
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
d35daf19c9
|
fail fast and hard; don't go into the deadlock
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
3b6808269a
|
some comments and optimized shared memory amount
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
9eba86f0a5
|
fully vectorized smem access
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
cb8cc25e3a
|
utilities for Packed and more vectorization
|
2024-05-27 13:15:36 +03:00 |
|
|
Erik Schultheis
|
53ee3297dd
|
use vectorized access to shared memory
|
2024-05-27 13:15:36 +03:00 |
|
|
Andrej Karpathy
|
ee6fa5ec9f
|
fill in the other model sizes
|
2024-05-26 18:04:43 +00:00 |
|
|
Andrej Karpathy
|
5ec4b774f2
|
simple jupyter notebook example on how to parse and vis the logfiles
|
2024-05-26 17:40:49 +00:00 |
|
|
Andrej Karpathy
|
d0c014372c
|
add ability to export fineweb100B as well
|
2024-05-25 17:21:51 +00:00 |
|
|
Andrej Karpathy
|
2a0f78d617
|
attempt to fix windows episode 2
|
2024-05-25 00:44:16 +00:00 |
|
|
Andrej Karpathy
|
5b96d91893
|
attempt to fix windows?
|
2024-05-25 00:39:40 +00:00 |
|
|
Andrej
|
4ff0412751
|
Merge pull request #450 from rosslwheeler/dataloader_win_fixes
Add glob() for windows
|
2024-05-24 17:30:35 -07:00 |
|
|
Andrej
|
6a7fd56d7c
|
Merge pull request #439 from lancerts/matmul-fix
Fix the unsupported block_size in matmul_backward_bias kernel 1
|
2024-05-24 17:26:37 -07:00 |
|
|
Ross Wheeler
|
79738d2ca4
|
fixed mkdir change
|
2024-05-24 17:05:59 -07:00 |
|
|
Ross Wheeler
|
e5083be900
|
Moved windows glob() over to dev/unistd.h
Added header guard and changed long->int64_t in dataloader.h
|
2024-05-24 16:56:52 -07:00 |
|
|
Andrej Karpathy
|
dbacaf84cf
|
Merge branch 'deterministic_layernorm' of https://github.com/ademeure/llm.c into ademeure-deterministic_layernorm
|
2024-05-24 21:46:25 +00:00 |
|
|
Andrej
|
bf03e7fd4c
|
Merge pull request #456 from ngc92/large-batch-fix
fix for large batch sizes
|
2024-05-24 12:00:41 -07:00 |
|
|
Erik Schultheis
|
1b98637960
|
int -> int64_t
|
2024-05-24 20:11:34 +03:00 |
|
|
Andrej Karpathy
|
08fe3eef43
|
change default shard size for fineweb10B
|
2024-05-24 15:56:05 +00:00 |
|
|
Erik Schultheis
|
b66eb66ac7
|
update fine-web preprocessing: faster, and much less RAM consumption
|
2024-05-24 17:45:26 +03:00 |
|
|
Andrej Karpathy
|
da59861d38
|
make hellaswag optional eval yay
|
2024-05-22 22:42:37 +00:00 |
|
|
Andrej Karpathy
|
67239d9b8f
|
fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be ready
|
2024-05-22 22:21:53 +00:00 |
|
|
Andrej Karpathy
|
051f3ca53c
|
first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batch
|
2024-05-22 19:24:20 +00:00 |
|
|
Andrej Karpathy
|
edb0df967a
|
continued changes for sharded dataloader
|
2024-05-22 11:59:09 +00:00 |
|
|
ademeure
|
7cbeefc7f3
|
added new layernorm backward to /dev/cuda/
|
2024-05-21 23:26:54 +01:00 |
|