- https://huggingface.co/docs/accelerate/concept_guides/big_model_inference
- https://huggingface.co/docs/transformers/big_models




- Efficient and Economic Large Language Model Inference with Attention Offloading
- https://arxiv.org/pdf/2405.01814


- DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
- https://arxiv.org/pdf/2207.00032


FlexFlow
- https://github.com/flexflow/FlexFlow

FlexGen
- https://github.com/FMInference/FlexGen




kv cache offload:

- https://github.com/NVIDIA/TensorRT-LLM/blob/a96cccafcf6365c128f004f779160951f8c0801c/docs/source/kv_cache_reuse.md