 Andrej
|
da88cb11b0
|
Merge pull request #803 from ngc92/ngc92/llama3-tied-weights
Tied and untied weights for LLama3
|
2025-06-26 10:03:39 -07:00 |
|
|
Andrej
|
5c17e4e256
|
Merge pull request #821 from ngc92/out-of-bounds-bugfix
fix out-of-bounds access in rmsnorm kernel
|
2025-06-26 10:01:10 -07:00 |
|
|
Erik Schultheis
|
9caeceb7ae
|
command-line overwrite to forcibly untie embeddings for llama3.2 models
|
2025-06-26 18:10:50 +02:00 |
|
|
Erik Schultheis
|
9688eef519
|
enable tied embeddings
|
2025-06-26 18:07:38 +02:00 |
|
|
Erik Schultheis
|
ffcfe99955
|
fix out-of-bounds access in rmsnorm kernel
|
2025-06-26 18:07:38 +02:00 |
|
|
Erik Schultheis
|
9c60616276
|
fix out-of-bounds access in rmsnorm kernel
|
2025-06-26 17:52:20 +02:00 |
|
|
Andrej
|
d36f0e6756
|
Merge pull request #811 from ngc92/llama-fixes
Llama fixes
|
2025-05-10 16:27:57 -07:00 |
|
|
Erik Schultheis
|
76a7cce3db
|
replace offload with smaller model
|
2025-05-04 23:47:58 +02:00 |
|
|
Erik Schultheis
|
35e1ad6fcc
|
enable storing the expected loss values in the state file, so we can run testing with different model configurations
fup
|
2025-05-04 23:47:58 +02:00 |
|
|
Erik Schultheis
|
f38eadce35
|
allow reducing number of transformer blocks to make smaller models that can be tested on commodity GPUs
|
2025-05-04 22:49:41 +02:00 |
|
|
Erik Schultheis
|
a860922827
|
set stream for attention softmax
|
2025-05-04 22:10:38 +02:00 |
|
|
Erik Schultheis
|
082d9fa78e
|
added missing stream argument for repkv_backward
|
2025-05-04 20:15:30 +02:00 |
|
|
Erik Schultheis
|
d4347a7115
|
ensure 32-bit master params in python training
|
2025-05-04 20:14:43 +02:00 |
|
|
Erik Schultheis
|
7b7d39c007
|
include grad norm in logging
|
2025-05-04 20:14:33 +02:00 |
|
|
Erik Schultheis
|
1c02d54716
|
cudnn does not support fp32 -> remove this pointless test
|
2025-05-04 20:14:11 +02:00 |
|
|
Andrej
|
49cef1d59a
|
Merge pull request #802 from ngc92/ngc92/llama3-dev
LLama3 MVP
|
2025-05-01 04:21:07 -07:00 |
|
|
Erik Schultheis
|
9c52a9557e
|
fix: CPUOffloadOptimizer + gradient clipping is broken; we use an inefficient workaround to make it correct
|
2025-04-14 17:06:07 +02:00 |
|
|
Erik Schultheis
|
5b928298d9
|
!! DROP THIS COMMIT !!
hard-code a hf token to make the tests run
|
2025-04-14 11:48:30 +02:00 |
|
|
Erik Schultheis
|
f8a43cea84
|
fix: writing checkpoint
|
2025-04-14 11:48:30 +02:00 |
|
|
Erik Schultheis
|
24d91298cf
|
fix: ensure freqs_cis are not broken when calling model.to(dtype) (was discarding imaginary part)
|
2025-04-14 11:48:30 +02:00 |
|
|
Erik Schultheis
|
2c3fecced2
|
fix: gradient checking
|
2025-04-14 11:48:30 +02:00 |
|
|
Erik Schultheis
|
6866623259
|
fix: match pytorch learning rate in test file
|
2025-04-14 11:33:34 +02:00 |
|
|
Erik Schultheis
|
4983c462a7
|
fix: fully ignore biases
|
2025-04-14 11:33:34 +02:00 |
|
|
Erik Schultheis
|
a94471c627
|
use optimizer offloading when running in CI
|
2025-04-14 11:33:34 +02:00 |
|
|
Erik Schultheis
|
090341e8dc
|
enable llama3 CI
|
2025-04-14 11:33:34 +02:00 |
|
|
Erik Schultheis
|
4d7980c26f
|
renaming gpt2 -> llama3
|
2025-04-14 11:27:52 +02:00 |
|
|
Erik Schultheis
|
3745dac6de
|
define llama3.2 1B and 3B for export from python (will untie embeddings and lm-head for now)
|
2025-04-13 21:56:37 +02:00 |
|
|
Andrej Karpathy
|
2c5ced6a77
|
fix bug due to bf16 adamw mv
|
2024-10-01 21:41:17 +00:00 |
|
|
Andrej
|
d808d78030
|
Merge pull request #772 from ademeure/llama3_arun_new
BF16 opt state (m/v) with stochastic rounding (Llama3 branch)
|
2024-10-02 04:17:46 +09:00 |
|
|
ademeure
|
2602b46bb3
|
BF16 opt state (m/v) with stochastic rounding, seems to work really well (OPTIMIZER_LOW_PRECISION=1)
|
2024-10-01 18:42:38 +00:00 |
|
|
Andrej Karpathy
|
c746e06f49
|
take out debugging stuff. we can now run training loop for both models. they don't match yet
|
2024-10-01 17:19:39 +00:00 |
|
|
Andrej Karpathy
|
9099a0ae9c
|
ok the full backward now shows max abs diff of 3e-3, except for the encoder backward (that's coming next). i think 3e-3 seems ok just inspecting the differences manually. probably this is correct. encoder backward next
|
2024-10-01 17:03:29 +00:00 |
|
|
Andrej Karpathy
|
e6481b679c
|
fix bug with qkvr sizing, has to be 3*C. Credit to @ademeure for finding this bug and bringing light to darkness and order to chaos. A true warrior in the fight against entropy.
|
2024-10-01 16:36:35 +00:00 |
|
|
Andrej Karpathy
|
7d945e994c
|
reshuffle repkv a bit, i wrote it from scratch. the kernel is still correct. repkv backward looks correct. rope backward is trivial so i don't see how it's not correct, and i also checked it. basically i'm really confused right now
|
2024-09-27 19:25:09 +00:00 |
|
|
Andrej Karpathy
|
8d49062293
|
add backward kernel to dev/cuda for rope, to ensure correctness. but i mean, it's trivial. this can't possibly be the issue. it must be the repkv
|
2024-09-27 18:07:10 +00:00 |
|
|
Andrej Karpathy
|
075e430d23
|
just pushing what i have. it's epsilon away from working sigh. basically at this point of where prints happen, gradients match. but once we backward attention, rope and repkv, gradients don't match. attention hasn't changed so that can't be wrong (?), so it's either repkv or rope. i have to go slower and double check the backward pass of both of these in detail. also had to introduce one more additional buffer for backward
|
2024-09-27 03:37:53 +00:00 |
|
|
Andrej Karpathy
|
28e4a7f83e
|
small fixes, but still not too happy with this kernel, it wastes thread and more efficient implementation kernel2 is desireable and desired
|
2024-09-27 01:13:32 +00:00 |
|
|
Andrej
|
c8b348ee39
|
Merge pull request #764 from insop/insop/llama3
Adding backward kernel for repkv on `llama3` branch (cudamode-irl)
|
2024-09-27 10:03:39 +09:00 |
|
|
Andrej Karpathy
|
1b54612c46
|
i can backward through MLP block. Attention block is next
|
2024-09-27 00:21:43 +00:00 |
|
|
Insop Song
|
01c2895e17
|
Update RoPE naming
|
2024-09-26 16:34:50 -07:00 |
|
|
Insop Song
|
cbf53e3f61
|
Merge remote-tracking branch 'upstream/llama3' into insop/llama3
|
2024-09-26 16:33:01 -07:00 |
|
|
Andrej Karpathy
|
2c4b3cc8bb
|
integrate our rmsnorm backward and move the other rmsnorm functions into rmsnorm.cuh that is a new file
|
2024-09-26 22:27:58 +00:00 |
|
|
Andrej Karpathy
|
102067fbf5
|
oops i think i accidentally forgot to include swiglu.cuh
|
2024-09-26 22:08:19 +00:00 |
|
|
Insop Song
|
a2b66f153a
|
Merge remote-tracking branch 'upstream/llama3' into insop/llama3
|
2024-09-26 11:42:31 -07:00 |
|
|
Andrej Karpathy
|
31be5e790d
|
add rmsnorm backward in dev/cuda, it seems to work surprisingly, and is probably ready to be integrated into llmc. we are still using 2X too much shared memory because I didn't want to change way too many things at the same time. I copy pasted our kernel10 of layernorm backward and made tweaks to it removing the bias and mean cool
|
2024-09-26 03:17:58 +00:00 |
|
|
Insop Song
|
d1f2f64541
|
Updated repkv_backward cuda kernel
- kernel 1 is tested
- build
```
make repkv_backward
/usr/local/cuda/bin/nvcc -O3 --use_fast_math --generate-code arch=compute_80,code=[compute_80,sm_80] -lcublas -lcublasLt -std=c++17 repkv_backward.cu -o repkv_backward
```
- test run on A30
```
Using kernel 1
Checking block size 32.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 64.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 128.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 256.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 512.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
Checking block size 1024.
0.531524 0.531524
0.600285 0.600285
0.458787 0.458787
0.296680 0.296680
-0.911627 -0.911627
All results match. Starting benchmarks.
block_size 32 time 3.2461 ms
block_size 64 time 1.7509 ms
block_size 128 time 1.7374 ms
block_size 256 time 1.7441 ms
block_size 512 time 1.8092 ms
block_size 1024 time 2.0443 ms
```
|
2024-09-25 17:40:38 -07:00 |
|
|
Insop Song
|
3e5134d94f
|
Merge branch 'insop/llama3_wip' into insop/llama3
|
2024-09-25 15:53:17 -07:00 |
|
|
Insop Song
|
8874c2c3f6
|
Merge remote-tracking branch 'upstream/llama3' into insop/llama3
|
2024-09-25 15:50:51 -07:00 |
|
|
Andrej Karpathy
|
0731b39a73
|
forward pass matchesgit add train_llama3.cu train_llama3.py ! losses are the same. now comes the backward pass
|
2024-09-25 22:31:31 +00:00 |
|
|
Andrej Karpathy
|
1826752ae1
|
add swigul yaygit add -u!
|
2024-09-25 22:16:45 +00:00 |
|