 Andrej Karpathy
|
7d0042d574
|
make training data configurable and also dont move the engire dataset to device that is not a good idea
|
2024-05-01 18:15:10 +00:00 |
|
|
Andrej Karpathy
|
8aa9a9ed53
|
and also profile lol
|
2024-05-01 17:56:01 +00:00 |
|
|
Andrej Karpathy
|
1ed9381fd0
|
fix cudnnDestroy outside of ifdef
|
2024-05-01 17:55:36 +00:00 |
|
|
Andrej Karpathy
|
69d40ebfff
|
add readme and search for cudd in home dir
|
2024-05-01 17:43:50 +00:00 |
|
|
Andrej Karpathy
|
35640651b3
|
updates to makefile: docs, override, etc
|
2024-05-01 17:32:52 +00:00 |
|
|
Andrej Karpathy
|
04718fbbd1
|
round 1 of small fixes for cudnn support
|
2024-05-01 17:07:10 +00:00 |
|
|
ademeure
|
1147983b8c
|
Add missing cudnnDestroy() calls
|
2024-05-01 17:19:51 +01:00 |
|
|
ademeure
|
d9e7a0a378
|
Fixed profile_gpt2.cu/test_gpt2.cu for cuDNN
|
2024-05-01 17:03:28 +01:00 |
|
|
ademeure
|
2b540b61da
|
attention_forward.cu ~matches train_gpt2cu & allocate less memory for stats in "att"
|
2024-05-01 16:53:01 +01:00 |
|
|
ademeure
|
be54842622
|
Rename lowp_float to floatX in /dev/cuda/attention_forward.cu
|
2024-05-01 15:43:28 +01:00 |
|
|
ademeure
|
c778dc0098
|
slightly increase wte threshold to make test_gpt2.cu pass (?)
|
2024-05-01 15:40:30 +01:00 |
|
|
ademeure
|
57c09529d9
|
Merge branch 'cudnn_try2' of https://github.com/ademeure/llm.c into cudnn_try2
|
2024-05-01 15:32:13 +01:00 |
|
|
ademeure
|
7beff72f5f
|
Add makefile changes for cuDNN
|
2024-05-01 15:31:30 +01:00 |
|
|
ademeure
|
c4ecc04dc3
|
Merge remote-tracking branch 'karpathy/master' into cudnn_try2
|
2024-05-01 15:30:00 +01:00 |
|
|
ademeure
|
d508463ad7
|
Merge remote-tracking branch 'origin/cudnn_try2' into cudnn_try2
|
2024-05-01 15:27:07 +01:00 |
|
|
ademeure
|
d42ee0d925
|
Cleanup + do not allocate gradient preatt memory with cuDNN + allocate as small a cuDNN workspace as possible
|
2024-05-01 15:26:25 +01:00 |
|
|
ademeure
|
61ad4b15ab
|
Fully working forward+backward cuDNN BF16 Flash Attention in train_gpt2.cu!
|
2024-05-01 15:02:47 +01:00 |
|
|
Andrej
|
40da8cf233
|
Merge pull request #314 from jrhemstad/patch-3
Add llm.cpp fork to README
|
2024-04-30 17:03:25 -07:00 |
|
|
Jake Hemstad
|
78eba5fff3
|
Add llm.cpp fork to README
|
2024-04-30 18:52:42 -05:00 |
|
|
Andrej Karpathy
|
769c911ab6
|
small tweaks
|
2024-04-30 23:48:06 +00:00 |
|
|
Andrej
|
bad76f408a
|
Merge pull request #313 from ngc92/small-improvements
fixed potential error and generalized gelu forward
|
2024-04-30 16:18:08 -07:00 |
|
|
Erik Schultheis
|
603d862830
|
fix up bandwidth calculation
|
2024-04-30 23:52:08 +03:00 |
|
|
Erik Schultheis
|
c3515cfe47
|
fixed potential error and generalized gelu forward
|
2024-04-30 23:45:34 +03:00 |
|
|
Andrej Karpathy
|
7ba1ed8a1a
|
delete packed128 .fp32, use (float) ok
|
2024-04-30 19:39:20 +00:00 |
|
|
Andrej Karpathy
|
7fd0659ba8
|
Pack gelu_forward kernel using the new Packed128
|
2024-04-30 19:35:23 +00:00 |
|
|
Andrej Karpathy
|
9141e04693
|
fixes to the PR, careful with float/floatX etc
|
2024-04-30 19:35:05 +00:00 |
|
|
Andrej Karpathy
|
242981eecf
|
Merge branch 'gelu_forwards_packing' of https://github.com/ChrisDryden/llm.c into ChrisDryden-gelu_forwards_packing
|
2024-04-30 19:06:39 +00:00 |
|
|
Andrej Karpathy
|
70b4de8700
|
add comment documenting how to reproduce python reference exactly
|
2024-04-30 18:51:45 +00:00 |
|
|
Andrej Karpathy
|
050cbfa42c
|
override train split to val split if we are debugging and trying to overfit a single batch of data, following the python script reference behavior
|
2024-04-30 18:48:12 +00:00 |
|
|
Andrej Karpathy
|
44656c3850
|
allow fp32 precision in the test script as well
|
2024-04-30 18:38:38 +00:00 |
|
|
Andrej Karpathy
|
b84571f745
|
also add argparse to force tf32 to zero
|
2024-04-30 17:54:15 +00:00 |
|
|
Andrej Karpathy
|
654d6f55c5
|
add arg to cap the number of steps, and offset all prints to start steps at 1, which i think looks better to the eye
|
2024-04-30 17:46:36 +00:00 |
|
|
Andrej Karpathy
|
52e2ca8378
|
Merge branch 'master' of github.com:karpathy/llm.c
|
2024-04-30 16:36:00 +00:00 |
|
|
Andrej
|
179b159a1d
|
Merge pull request #311 from leloykun/fc--modal-script
Add script to run benchmarks on Modal
|
2024-04-30 09:35:51 -07:00 |
|
|
Franz Louis Cesista
|
b239b67ae1
|
add modal script
|
2024-05-01 00:15:08 +08:00 |
|
|
Andrej Karpathy
|
9d8a6d13f0
|
add flag overfit_single_batch useful debugging into train_gpt2.cu
|
2024-04-30 16:03:33 +00:00 |
|
|
ademeure
|
435ac928b4
|
cuDNN Forward Flash Attention is working in both /dev/cuda/ and train_gpt2.cu! (backwards pass is broken for now as a conquence)
|
2024-04-30 17:00:09 +01:00 |
|
|
Andrej Karpathy
|
51face88d9
|
fix bug where backward/step must be outside of amp context
|
2024-04-30 15:48:40 +00:00 |
|
|
Andrej
|
7e8f14d2c0
|
Merge pull request #310 from ngc92/flush-l2-for-benchmark
clear the L2 cache between benchmarks
|
2024-04-30 07:23:41 -07:00 |
|
|
ademeure
|
1c516c732d
|
re-commit of my old cudnn forward attention changes
|
2024-04-30 14:54:35 +01:00 |
|
|
Erik Schultheis
|
d7813d281d
|
clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable results
|
2024-04-30 12:45:37 +03:00 |
|
|
Christopher Dryden
|
568615fef1
|
Added packing for gelu forwards kernel
|
2024-04-30 03:21:42 +00:00 |
|
|
Andrej
|
9464f4272e
|
Merge pull request #283 from ChrisDryden/configurationRewrite
Changed ordering of type configuration to easily see unchanged values
|
2024-04-29 18:58:03 -07:00 |
|
|
Andrej
|
997864920b
|
Merge pull request #290 from Ricardicus/ci-dev-cuda
Adding the example kernels to CI
|
2024-04-29 18:53:03 -07:00 |
|
|
Andrej
|
2490f7896d
|
Merge pull request #298 from karpathy/feature/packed128
Feature/packed128
|
2024-04-29 16:31:02 -07:00 |
|
|
Andrej Karpathy
|
906d22f5b6
|
fix profile gpt2cu but note it currently needs the NO_MULTI_GPU=1 switch i think when calling it with make, todo fix later how we add the mpi flags
|
2024-04-29 23:27:55 +00:00 |
|
|
Andrej Karpathy
|
af2bc47d22
|
remove forceinline. blame @arund42 if this ever does not get inlined by the compiler
|
2024-04-29 23:10:51 +00:00 |
|
|
Andrej Karpathy
|
f02ba4b436
|
import updated tokenizer properly
|
2024-04-29 23:02:26 +00:00 |
|
|
Andrej Karpathy
|
7634f0863c
|
add packed128 to the mainline file
|
2024-04-29 22:57:59 +00:00 |
|
|
Andrej Karpathy
|
5e12c66f5a
|
move packed128 to common
|
2024-04-29 22:55:28 +00:00 |
|