 Andrej Karpathy
|
c177c2694c
|
make nice and print
|
2024-05-01 23:06:23 +00:00 |
|
|
Andrej Karpathy
|
795f8b690c
|
fixes to keep master copy in fp32 of weights optionally
|
2024-05-01 22:50:22 +00:00 |
|
|
Andrej Karpathy
|
fd474feec8
|
Merge branch 'fp32-weights' of https://github.com/ngc92/llm.c
|
2024-05-01 22:02:55 +00:00 |
|
|
Erik Schultheis
|
df9f0efde6
|
updated letter and properly free buffer
|
2024-05-02 01:00:36 +03:00 |
|
|
Andrej
|
f4f7a987e7
|
Merge pull request #324 from eymay/init_weights_from_scratch
Refactoring parameter size filling
|
2024-05-01 14:50:39 -07:00 |
|
|
Erik Schultheis
|
e9c76a1753
|
option to keep weights as fp32
|
2024-05-02 00:11:07 +03:00 |
|
|
Andrej
|
ab95a11190
|
Merge pull request #325 from ngc92/dev-cuda-utils
mixed precision utilities for dev/cuda
|
2024-05-01 12:54:54 -07:00 |
|
|
Erik Schultheis
|
b42db706b2
|
this kernel will be released later, sorry for the spoiler :)
|
2024-05-01 22:47:24 +03:00 |
|
|
Erik Schultheis
|
91bc72d0c0
|
fixup
|
2024-05-01 22:40:26 +03:00 |
|
|
Erik Schultheis
|
06980b6b2f
|
mixed precision utilities for dev/cuda
|
2024-05-01 22:35:55 +03:00 |
|
|
Andrej
|
d37639a8cb
|
Merge pull request #316 from rosslwheeler/designated_init_fix
uint and designated initializers non-standard for C++ / Cuda
|
2024-05-01 12:13:12 -07:00 |
|
|
Andrej
|
be5cee21e6
|
Merge pull request #317 from ChrisDryden/patch-5
Removed detail about layernorm inputs being f32, no longer the case
|
2024-05-01 12:00:01 -07:00 |
|
|
Andrej
|
38b478d8f9
|
Merge pull request #323 from karpathy/feature/cudnn
feature/cudnn for flash-attention
|
2024-05-01 11:34:16 -07:00 |
|
|
Andrej Karpathy
|
7d0042d574
|
make training data configurable and also dont move the engire dataset to device that is not a good idea
|
2024-05-01 18:15:10 +00:00 |
|
|
Andrej Karpathy
|
8aa9a9ed53
|
and also profile lol
|
2024-05-01 17:56:01 +00:00 |
|
|
Andrej Karpathy
|
1ed9381fd0
|
fix cudnnDestroy outside of ifdef
|
2024-05-01 17:55:36 +00:00 |
|
|
Andrej Karpathy
|
69d40ebfff
|
add readme and search for cudd in home dir
|
2024-05-01 17:43:50 +00:00 |
|
|
Andrej Karpathy
|
35640651b3
|
updates to makefile: docs, override, etc
|
2024-05-01 17:32:52 +00:00 |
|
|
Andrej Karpathy
|
04718fbbd1
|
round 1 of small fixes for cudnn support
|
2024-05-01 17:07:10 +00:00 |
|
|
ademeure
|
1147983b8c
|
Add missing cudnnDestroy() calls
|
2024-05-01 17:19:51 +01:00 |
|
|
ademeure
|
d9e7a0a378
|
Fixed profile_gpt2.cu/test_gpt2.cu for cuDNN
|
2024-05-01 17:03:28 +01:00 |
|
|
ademeure
|
2b540b61da
|
attention_forward.cu ~matches train_gpt2cu & allocate less memory for stats in "att"
|
2024-05-01 16:53:01 +01:00 |
|
|
ademeure
|
be54842622
|
Rename lowp_float to floatX in /dev/cuda/attention_forward.cu
|
2024-05-01 15:43:28 +01:00 |
|
|
ademeure
|
c778dc0098
|
slightly increase wte threshold to make test_gpt2.cu pass (?)
|
2024-05-01 15:40:30 +01:00 |
|
|
ademeure
|
57c09529d9
|
Merge branch 'cudnn_try2' of https://github.com/ademeure/llm.c into cudnn_try2
|
2024-05-01 15:32:13 +01:00 |
|
|
ademeure
|
7beff72f5f
|
Add makefile changes for cuDNN
|
2024-05-01 15:31:30 +01:00 |
|
|
ademeure
|
c4ecc04dc3
|
Merge remote-tracking branch 'karpathy/master' into cudnn_try2
|
2024-05-01 15:30:00 +01:00 |
|
|
ademeure
|
d508463ad7
|
Merge remote-tracking branch 'origin/cudnn_try2' into cudnn_try2
|
2024-05-01 15:27:07 +01:00 |
|
|
ademeure
|
d42ee0d925
|
Cleanup + do not allocate gradient preatt memory with cuDNN + allocate as small a cuDNN workspace as possible
|
2024-05-01 15:26:25 +01:00 |
|
|
ademeure
|
61ad4b15ab
|
Fully working forward+backward cuDNN BF16 Flash Attention in train_gpt2.cu!
|
2024-05-01 15:02:47 +01:00 |
|
|
Chris Dryden
|
1806062a62
|
Removed detail about layernorm inputs being f32, no longer the case
|
2024-04-30 22:01:14 -07:00 |
|
|
Ross Wheeler
|
229d70b2d5
|
uint and designated initializers non-standard for C++
uint isn't defined by the MS compiler and designated initializers aren't part of C++ until C++ 20.
|
2024-04-30 18:46:54 -07:00 |
|
|
Andrej
|
40da8cf233
|
Merge pull request #314 from jrhemstad/patch-3
Add llm.cpp fork to README
|
2024-04-30 17:03:25 -07:00 |
|
|
Jake Hemstad
|
78eba5fff3
|
Add llm.cpp fork to README
|
2024-04-30 18:52:42 -05:00 |
|
|
Andrej Karpathy
|
769c911ab6
|
small tweaks
|
2024-04-30 23:48:06 +00:00 |
|
|
Andrej
|
bad76f408a
|
Merge pull request #313 from ngc92/small-improvements
fixed potential error and generalized gelu forward
|
2024-04-30 16:18:08 -07:00 |
|
|
Erik Schultheis
|
603d862830
|
fix up bandwidth calculation
|
2024-04-30 23:52:08 +03:00 |
|
|
Erik Schultheis
|
c3515cfe47
|
fixed potential error and generalized gelu forward
|
2024-04-30 23:45:34 +03:00 |
|
|
Andrej Karpathy
|
7ba1ed8a1a
|
delete packed128 .fp32, use (float) ok
|
2024-04-30 19:39:20 +00:00 |
|
|
Andrej Karpathy
|
7fd0659ba8
|
Pack gelu_forward kernel using the new Packed128
|
2024-04-30 19:35:23 +00:00 |
|
|
Andrej Karpathy
|
9141e04693
|
fixes to the PR, careful with float/floatX etc
|
2024-04-30 19:35:05 +00:00 |
|
|
Andrej Karpathy
|
242981eecf
|
Merge branch 'gelu_forwards_packing' of https://github.com/ChrisDryden/llm.c into ChrisDryden-gelu_forwards_packing
|
2024-04-30 19:06:39 +00:00 |
|
|
Andrej Karpathy
|
70b4de8700
|
add comment documenting how to reproduce python reference exactly
|
2024-04-30 18:51:45 +00:00 |
|
|
Andrej Karpathy
|
050cbfa42c
|
override train split to val split if we are debugging and trying to overfit a single batch of data, following the python script reference behavior
|
2024-04-30 18:48:12 +00:00 |
|
|
Andrej Karpathy
|
44656c3850
|
allow fp32 precision in the test script as well
|
2024-04-30 18:38:38 +00:00 |
|
|
Andrej Karpathy
|
b84571f745
|
also add argparse to force tf32 to zero
|
2024-04-30 17:54:15 +00:00 |
|
|
Andrej Karpathy
|
654d6f55c5
|
add arg to cap the number of steps, and offset all prints to start steps at 1, which i think looks better to the eye
|
2024-04-30 17:46:36 +00:00 |
|
|
Andrej Karpathy
|
52e2ca8378
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-30 16:36:00 +00:00 |
|
|
Andrej
|
179b159a1d
|
Merge pull request #311 from leloykun/fc--modal-script
Add script to run benchmarks on Modal
|
2024-04-30 09:35:51 -07:00 |
|
|
Franz Louis Cesista
|
b239b67ae1
|
add modal script
|
2024-05-01 00:15:08 +08:00 |
|