项目文件夹

文件

T

文件历史

Andrej 5c17e4e256 Merge pull request #821 from ngc92/out-of-bounds-bugfix

fix out-of-bounds access in rmsnorm kernel

2025-06-26 10:01:10 -07:00

adamw.cuh

fix bug due to bf16 adamw mv

2024-10-01 21:41:17 +00:00

attention.cuh

set stream for attention softmax

2025-05-04 22:10:38 +02:00

cublas_common.h

Refactor matmul to only use cuBLASLt and optionally use fused GELU

2024-06-30 20:45:37 +01:00

cuda_common.h

BF16 opt state (m/v) with stochastic rounding, seems to work really well (OPTIMIZER_LOW_PRECISION=1)

2024-10-01 18:42:38 +00:00

cuda_utils.cuh

BF16 opt state (m/v) with stochastic rounding, seems to work really well (OPTIMIZER_LOW_PRECISION=1)

2024-10-01 18:42:38 +00:00

cudnn_att.cpp

Including unistd.h in cudnn_att.cpp for latest MSVC compiler

2024-07-20 11:57:10 -07:00

cudnn_att.h

move all kernels into a main_stream in preparation for future parallelism

2024-06-04 22:35:12 +03:00

dataloader.h

add support for dataloader to serve uint32_t tokens, as necessary in Llama 3

2024-09-16 19:44:33 +00:00

encoder.cuh

take out debugging stuff. we can now run training loop for both models. they don't match yet

2024-10-01 17:19:39 +00:00

fused_classifier.cuh

switch losses to fp32

2024-06-26 20:48:08 +03:00

gelu.cuh

Fix guarding logic, refactor

2024-06-17 14:12:05 +02:00

global_norm.cuh

update norm computation: multi-process reduce on GPU

2024-06-25 21:28:52 +03:00

layernorm.cuh

integrate our rmsnorm backward and move the other rmsnorm functions into rmsnorm.cuh that is a new file

2024-09-26 22:27:58 +00:00

logger.h

Add learning rate and grad norm logging

2024-06-07 17:34:42 +02:00

matmul.cuh

fix default for test_gpt2cu

2024-06-30 20:45:37 +01:00

mfu.h

disable if header is not found

2024-07-29 10:07:37 +02:00

outlier_detector.h

oops i was only using window size 16 for testing let's make it 128 or something

2024-06-25 01:56:04 +00:00

rand.h

Dataloader test and ptx/sass file generator

2024-06-21 17:23:03 -07:00

repkv.cuh

added missing stream argument for repkv_backward

2025-05-04 20:15:30 +02:00

rmsnorm.cuh

fix out-of-bounds access in rmsnorm kernel

2025-06-26 17:52:20 +02:00

rope.cuh

just pushing what i have. it's epsilon away from working sigh. basically at this point of where prints happen, gradients match. but once we backward attention, rope and repkv, gradients don't match. attention hasn't changed so that can't be wrong (?), so it's either repkv or rope. i have to go slower and double check the backward pass of both of these in detail. also had to introduce one more additional buffer for backward

2024-09-27 03:37:53 +00:00

sampler.h

refactor part 2, moving stuff into common files so that we can later nicely separate out all the kernels as well

2024-06-03 18:10:33 +00:00

schedulers.h

small touchups and add the wsd schedule

2024-06-21 00:49:01 +00:00

swiglu.cuh

i can backward through MLP block. Attention block is next

2024-09-27 00:21:43 +00:00

tokenizer.h

made implicit includes of standard headers explicit

2024-06-04 13:22:16 +03:00

utils.h

Move WIN32 code into unistd.h

2024-07-20 12:05:20 -07:00

zero.cuh

fix: fully ignore biases

2025-04-14 11:33:34 +02:00