 Rickard Hallerbäck
|
3d2605bf8a
|
Merge branch 'master' into master
|
2024-04-22 02:21:49 +02:00 |
|
|
Andrej Karpathy
|
4e4eba0441
|
simplify bool
|
2024-04-22 00:19:00 +00:00 |
|
|
Andrej
|
a332ea5d92
|
Merge pull request #208 from Ricardicus/train_gpt2.py-fix
train_gpt2.py: fix for conditional use of fused and for computing gradients
|
2024-04-21 17:18:13 -07:00 |
|
|
Rickard Hallerbäck
|
c7aad65a04
|
Adding a workflow that builds with and without CUDA and OMP and on CPU also tests on ubuntu and macos
|
2024-04-22 02:14:09 +02:00 |
|
|
Chris Dryden
|
2d97be8315
|
Accidental last minute change
|
2024-04-21 16:44:07 -07:00 |
|
|
Chris Dryden
|
2752212151
|
Added shared memory for the atomic additions for the layernorm_backward.cu weights and biases
|
2024-04-21 16:37:08 -07:00 |
|
|
Rickard Hallerbäck
|
6c163d3391
|
train_gpt2.py: fix for conditional use of fused and for computing gradients
|
2024-04-22 00:27:36 +02:00 |
|
|
Andrej
|
4f8f5fa833
|
Merge pull request #207 from ToJen/patch-1
add Rust port to notable forks
|
2024-04-21 15:06:08 -07:00 |
|
|
Tomisin Jenrola
|
43ffd2af8a
|
add Rust port to notable forks
|
2024-04-21 19:23:50 -02:30 |
|
|
Andrej Karpathy
|
5ca1b939e9
|
small type reshuffles
|
2024-04-21 19:28:17 +00:00 |
|
|
Andrej Karpathy
|
48f2170e7b
|
pad step so that we don't change horizontal size in early parts of the script
|
2024-04-21 18:53:08 +00:00 |
|
|
Andrej Karpathy
|
8d4f93ccbd
|
experiment with pretty printing the run config, model config, device config, and data config
|
2024-04-21 18:47:36 +00:00 |
|
|
Andrej Karpathy
|
e55ed0a4e6
|
change default lr to 3e-4, exp -> expf, and print tok/s
|
2024-04-21 18:22:45 +00:00 |
|
|
Andrej
|
e93438efea
|
Merge pull request #201 from ngc92/memory
Yet more memory reductions
|
2024-04-21 11:02:50 -07:00 |
|
|
Andrej
|
734f65f33e
|
Merge pull request #192 from rosslwheeler/const_fixes
const changes to help with Microsoft compiler OpenMP 5.2 support
|
2024-04-21 10:42:15 -07:00 |
|
|
Andrej Karpathy
|
14d96e7c0d
|
Merge branch 'Ricardicus-master'
|
2024-04-21 17:41:24 +00:00 |
|
|
Andrej Karpathy
|
fced6d180f
|
small adjustments to Makefile and gitignore
|
2024-04-21 17:40:26 +00:00 |
|
|
Andrej Karpathy
|
e70bc19af5
|
Merge branch 'master' of https://github.com/Ricardicus/llm.c into Ricardicus-master
|
2024-04-21 17:34:05 +00:00 |
|
|
Erik Schultheis
|
fb3bea9e2a
|
more aggressive reuse of no longer needed memory from the forward activations
|
2024-04-21 19:48:21 +03:00 |
|
|
Erik Schultheis
|
8f6d1caefd
|
merged output and scratch
|
2024-04-21 19:31:46 +03:00 |
|
|
Erik Schultheis
|
118f3dde12
|
removed yet more tensors from backward state
|
2024-04-21 19:23:25 +03:00 |
|
|
Erik Schultheis
|
da5e8f7f56
|
split ActtivationTensors into separate classes for forward and backward to improve readability
|
2024-04-21 19:10:47 +03:00 |
|
|
Erik Schultheis
|
e3cbe7d886
|
saved another 60 MiB
|
2024-04-21 18:53:25 +03:00 |
|
|
Erik Schultheis
|
2ed8eddb3e
|
slight optimization of the python script
|
2024-04-21 18:43:45 +03:00 |
|
|
Erik Schultheis
|
33e81ba55f
|
print peak memory consumption in python
|
2024-04-21 18:25:04 +03:00 |
|
|
Erik Schultheis
|
bfdbf50189
|
more use of scratch buffer
|
2024-04-21 18:16:12 +03:00 |
|
|
Erik Schultheis
|
10fc806900
|
replace vaccum with generic scratch buffer
|
2024-04-21 17:10:29 +03:00 |
|
|
Erik Schultheis
|
0636bc4ee3
|
removed the union and moved the logit->probability conversion into the sampling function
|
2024-04-21 16:45:46 +03:00 |
|
|
patricxu
|
41c3cc1fab
|
Merge branch 'karpathy:master' into master
|
2024-04-21 10:31:59 +08:00 |
|
|
Erik Schultheis
|
7f3272c38f
|
unionize output buffer
|
2024-04-21 02:11:41 +03:00 |
|
|
Andrej Karpathy
|
9fb9c9185a
|
change baseline kernel to float, for now, for fair comparison too
|
2024-04-20 22:57:18 +00:00 |
|
|
Erik Schultheis
|
2f81c6cbda
|
qkv is only needed once, not per-layer
|
2024-04-21 01:50:10 +03:00 |
|
|
Andrej Karpathy
|
8488669d25
|
speed up the backward bias kernel by 45% and speed up the full running time by 1%
|
2024-04-20 22:46:19 +00:00 |
|
|
Rickard Hallerbäck
|
3cd6f83e32
|
removing some warnings
|
2024-04-20 22:03:35 +02:00 |
|
|
Andrej Karpathy
|
9b722ce5c0
|
Merge branch 'rosslwheeler-unistd_h_in_dev'
|
2024-04-20 20:00:39 +00:00 |
|
|
Andrej Karpathy
|
19e337b99e
|
a comment and some breathing room
|
2024-04-20 20:00:24 +00:00 |
|
|
Rickard Hallerbäck
|
be106f09a4
|
Merge branch 'master' into master
|
2024-04-20 22:00:02 +02:00 |
|
|
Andrej Karpathy
|
7e42c8b356
|
Merge branch 'unistd_h_in_dev' of https://github.com/rosslwheeler/llm.c into rosslwheeler-unistd_h_in_dev
|
2024-04-20 19:59:11 +00:00 |
|
|
Andrej
|
aa1184d9ca
|
Merge pull request #199 from ngc92/fused-buffer
Further memory reductions
|
2024-04-20 12:54:37 -07:00 |
|
|
Andrej Karpathy
|
0e79ac7d8b
|
Merge branch 'dagelf-master'
|
2024-04-20 19:45:49 +00:00 |
|
|
Andrej Karpathy
|
adac21e97b
|
add a small comment as to what this is
|
2024-04-20 19:45:34 +00:00 |
|
|
Andrej Karpathy
|
48a9f2ec69
|
Merge branch 'master' of https://github.com/dagelf/llm.c into dagelf-master
|
2024-04-20 19:43:17 +00:00 |
|
|
Andrej Karpathy
|
7fa9f7b140
|
document one way to do experiments / sweeps as just an example
|
2024-04-20 19:39:12 +00:00 |
|
|
Coenraad
|
f534e4bdfe
|
Fixes -Ofast optimizations breaking model by skipping them for gelu_backward
|
2024-04-20 21:33:00 +02:00 |
|
|
Rickard Hallerbäck
|
280023bf8e
|
Support for vanilla build without CUDA tools
|
2024-04-20 21:32:35 +02:00 |
|
|
Andrej Karpathy
|
4e3b7ea0f3
|
add a first, super simple logger
|
2024-04-20 19:26:25 +00:00 |
|
|
Andrej Karpathy
|
acdcfeaab6
|
introduce a poor mans argparse because i want to start training on tinystories, and messing with some of the options of the runs, now that things are more efficient
|
2024-04-20 18:37:18 +00:00 |
|
|
Andrej
|
5b0d52619f
|
Merge pull request #195 from ngc92/prof-script
profile and print out some nice stats
|
2024-04-20 10:48:10 -07:00 |
|
|
Andrej
|
af1ee01768
|
Merge pull request #197 from ngc92/fused-attention-slow
slow attention kernel that demonstrates a direct translation of the CPU code
|
2024-04-20 10:40:04 -07:00 |
|
|
Erik Schultheis
|
24dd728953
|
only need one V-sized buffer that can switch from logits to dlogits in the fused backward pass
|
2024-04-20 20:19:16 +03:00 |
|