lightning-ai--litgpt
wehub-resource-sync
816e440ed3
Check hyperlinks / test (push) Has been cancelled
Deploy MkDocs / deploy (push) Has been cancelled
CPU tests / testing-imports (macOS-14, 3.10) (push) Has been cancelled
CPU tests / testing-imports (ubuntu-22.04, 3.10) (push) Has been cancelled
CPU tests / testing-imports (ubuntu-24.04, 3.10) (push) Has been cancelled
CPU tests / testing-imports (windows-2022, 3.10) (push) Has been cancelled
CPU tests / pytester (macOS-14, 3.10, latest) (push) Has been cancelled
CPU tests / pytester (ubuntu-22.04, 3.10, latest) (push) Has been cancelled
CPU tests / pytester (ubuntu-22.04, 3.10, oldest) (push) Has been cancelled
CPU tests / pytester (ubuntu-22.04, 3.11, latest) (push) Has been cancelled
CPU tests / pytester (ubuntu-22.04, 3.12, latest) (push) Has been cancelled
CPU tests / pytester (ubuntu-22.04, 3.13, latest) (push) Has been cancelled
CPU tests / pytester (windows-2022, 3.10, latest) (push) Has been cancelled
CPU tests / testing-guardian (push) Has been cancelled
740 行
31 KiB
Markdown
740 行
31 KiB
Markdown
<!-- WEHUB_ZH_README -->
|
||
> [!NOTE]
|
||
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
|
||
> [English](./README.en.md) · [原始项目](https://github.com/Lightning-AI/litgpt) · [上游 README](https://github.com/Lightning-AI/litgpt/blob/HEAD/README.md)
|
||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||
|
||
<div align="center">
|
||
|
||
|
||
# ⚡ LitGPT
|
||
|
||
**20+ 款高性能 LLM，配套配方支持大规模预训练、微调与部署。**
|
||
|
||
<pre>
|
||
✅ 从零实现（from scratch） ✅ 无抽象层 ✅ 对初学者友好
|
||
✅ Flash attention ✅ FSDP ✅ LoRA、QLoRA、Adapter
|
||
✅ 降低 GPU 显存占用（fp4/8/16/32） ✅ 1-1000+ GPU/TPU ✅ 20+ LLM
|
||
</pre>
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
 [](https://github.com/Lightning-AI/litgpt/blob/main/LICENSE.md) [](https://discord.gg/VptPCZkGNa)
|
||
|
||
<p align="center">
|
||
<a href="#quick-start">快速开始</a> •
|
||
<a href="#choose-from-20-llms">模型</a> •
|
||
<a href="#finetune-an-llm">微调</a> •
|
||
<a href="#deploy-an-llm">部署</a> •
|
||
<a href="#all-workflows">全部工作流</a> •
|
||
<a href="#state-of-the-art-features">特性</a> •
|
||
<a href="#training-recipes">配方（YAML）</a> •
|
||
<a href="https://lightning.ai/">Lightning AI</a> •
|
||
<a href="#tutorials">教程</a>
|
||
</p>
|
||
|
||
|
||
|
||
<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-quick-start">
|
||
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/get-started-badge.svg" height="36px" alt="Get started"/>
|
||
</a>
|
||
|
||
|
||
|
||
</div>
|
||
|
||
# 需要 GPU？
|
||
超过 340,000 名开发者使用 [Lightning Cloud](https://lightning.ai/?utm_source=litgpt_readme&utm_medium=referral&utm_campaign=litgpt_readme) ——专为 PyTorch 与 PyTorch Lightning 打造。
|
||
- [GPU](https://lightning.ai/pricing?utm_source=litgpt_readme&utm_medium=referral&utm_campaign=litgpt_readme) 起价 $0.19。
|
||
- [集群](https://lightning.ai/clusters?utm_source=litgpt_readme&utm_medium=referral&utm_campaign=litgpt_readme): 前沿级训练/推理集群。
|
||
- [AI Studio（vibe train）](https://lightning.ai/studios?utm_source=litgpt_readme&utm_medium=referral&utm_campaign=litgpt_readme): AI 辅助你调试、调优并进行 vibe train 的工作空间。
|
||
- [AI Studio（vibe deploy）](https://lightning.ai/studios?utm_source=litgpt_readme&utm_medium=referral&utm_campaign=litgpt_readme): AI 辅助你优化并部署模型的工作空间。
|
||
- [Notebooks](https://lightning.ai/notebooks?utm_source=litgpt_readme&utm_medium=referral&utm_campaign=litgpt_readme): 持久化 GPU 工作空间，AI 辅助你编码与分析。
|
||
- [推理](https://lightning.ai/deploy?utm_source=litgpt_readme&utm_medium=referral&utm_campaign=litgpt_readme): 将模型部署为推理 API。
|
||
|
||
# 闪电般快速微调、预训练与推理 LLM ⚡⚡
|
||
每个 LLM 均从零实现，**无抽象层**且**完全可控**，在企业级规模下实现极速、精简且高性能。
|
||
|
||
✅ **企业级就绪 -** Apache 2.0 许可，可无限制用于企业场景。</br>
|
||
✅ **开发者友好 -** 无抽象层、单文件实现，便于调试。</br>
|
||
✅ **性能优化 -** 模型设计旨在最大化性能、降低成本并加速训练。</br>
|
||
✅ **经过验证的配方 -** 在企业级规模下测试过的高度优化训练/微调配方。</br>
|
||
|
||
|
||
|
||
# 快速开始
|
||
安装 LitGPT
|
||
```
|
||
pip install 'litgpt[extra]'
|
||
```
|
||
|
||
加载并使用任意一款 [20+ LLM](#choose-from-20-llms)：
|
||
```python
|
||
from litgpt import LLM
|
||
|
||
llm = LLM.load("microsoft/phi-2")
|
||
text = llm.generate("Fix the spelling: Every fall, the family goes to the mountains.")
|
||
print(text)
|
||
# Corrected Sentence: Every fall, the family goes to the mountains.
|
||
```
|
||
|
||
|
||
|
||
✅ 针对快速推理优化</br>
|
||
✅ 量化（Quantization）</br>
|
||
✅ 可在低显存 GPU 上运行</br>
|
||
✅ 无内部抽象层</br>
|
||
✅ 针对生产规模优化</br>
|
||
|
||
<details>
|
||
<summary>高级安装选项</summary>
|
||
|
||
从源码安装：
|
||
|
||
```bash
|
||
git clone https://github.com/Lightning-AI/litgpt
|
||
cd litgpt
|
||
# if using uv
|
||
uv sync --all-extras
|
||
# if using pip
|
||
pip install -e ".[extra,compiler,test]"
|
||
```
|
||
</details>
|
||
|
||
[浏览完整 Python API 文档](tutorials/python-api.md)。
|
||
|
||
|
||
|
||
---
|
||
# 从 20+ LLM 中选择
|
||
每个模型均从零编写，以最大化性能并去除抽象层：
|
||
|
||
| 模型 | 模型规模 | 作者 | 参考文献 |
|
||
|----|----|----|----|
|
||
| Llama 3, 3.1, 3.2, 3.3 | 1B, 3B, 8B, 70B, 405B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) |
|
||
| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) |
|
||
| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) |
|
||
| Gemma 2 | 2B, 9B, 27B | Google | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf) |
|
||
| Phi 4 | 14B | Microsoft Research | [Abdin et al. 2024](https://arxiv.org/abs/2412.08905) |
|
||
| Qwen2.5 | 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B | Alibaba Group | [Qwen Team 2024](https://qwenlm.github.io/blog/qwen2.5/) |
|
||
| Qwen2.5 Coder | 0.5B, 1.5B, 3B, 7B, 14B, 32B | Alibaba Group | [Hui, Binyuan et al. 2024](https://arxiv.org/abs/2409.12186) |
|
||
| R1 Distill Llama | 8B, 70B | DeepSeek AI | [DeepSeek AI 2025](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf) |
|
||
| ... | ... | ... | ... |
|
||
|
||
<details>
|
||
<summary>查看 20+ LLM 完整列表</summary>
|
||
|
||
|
||
|
||
#### 全部模型
|
||
|
||
| 模型 | 模型规模 | 作者 | 参考文献 |
|
||
|----|----|----|----|
|
||
| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) |
|
||
| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) |
|
||
| Falcon | 7B, 40B, 180B | TII UAE | [TII 2023](https://falconllm.tii.ae) |
|
||
| Falcon 3 | 1B, 3B, 7B, 10B | TII UAE | [TII 2024](https://huggingface.co/blog/falcon3) |
|
||
| FreeWilly2 (Stable Beluga 2) | 70B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models) |
|
||
| Function Calling Llama 2 | 7B | Trelis | [Trelis et al. 2023](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2) |
|
||
| Gemma | 2B, 7B | Google | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf) |
|
||
| Gemma 2 | 9B, 27B | Google | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf) |
|
||
| Gemma 3 | 1B, 4B, 12B, 27B | Google | [Google Team, Google Deepmind](https://arxiv.org/pdf/2503.19786) |
|
||
| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) |
|
||
| Llama 3.1 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) |
|
||
| Llama 3.2 | 1B, 3B | Meta AI | [Meta AI 2024](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/) |
|
||
| Llama 3.3 | 70B | Meta AI | [Meta AI 2024](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) |
|
||
| Mathstral | 7B | Mistral AI | [Mistral AI 2024](https://mistral.ai/news/mathstral/) |
|
||
| MicroLlama | 300M | Ken Wang | [MicroLlama repo](https://github.com/keeeeenw/MicroLlama) |
|
||
| Mixtral MoE | 8x7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/mixtral-of-experts/) |
|
||
| Mistral | 7B, 123B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/announcing-mistral-7b/) |
|
||
| Mixtral MoE | 8x22B | Mistral AI | [Mistral AI 2024](https://mistral.ai/news/mixtral-8x22b/) |
|
||
| OLMo | 1B, 7B | Allen Institute for AI (AI2) | [Groeneveld et al. 2024](https://aclanthology.org/2024.acl-long.841/) |
|
||
| OpenLLaMA | 3B, 7B, 13B | OpenLM Research | [Geng & Liu 2023](https://github.com/openlm-research/open_llama) |
|
||
| Phi 1.5 & 2 | 1.3B, 2.7B | Microsoft Research | [Li et al. 2023](https://arxiv.org/abs/2309.05463) |
|
||
| Phi 3 | 3.8B | Microsoft Research | [Abdin et al. 2024](https://arxiv.org/abs/2404.14219) |
|
||
| Phi 4 | 14B | Microsoft Research | [Abdin et al. 2024](https://arxiv.org/abs/2412.08905) |
|
||
| Phi 4 Mini Instruct | 3.8B | Microsoft Research | [Microsoft 2025](https://arxiv.org/abs/2503.01743) |
|
||
| Phi 4 Mini Reasoning | 3.8B | Microsoft Research | [Xu, Peng et al. 2025](https://arxiv.org/abs/2504.21233) |
|
||
| Phi 4 Reasoning | 3.8B | Microsoft Research | [Abdin et al. 2025](https://arxiv.org/abs/2504.21318) |
|
||
| Phi 4 Reasoning Plus | 3.8B | Microsoft Research | [Abdin et al. 2025](https://arxiv.org/abs/2504.21318) |
|
||
| Platypus | 7B, 13B, 70B | Lee et al. | [Lee, Hunter, and Ruiz 2023](https://arxiv.org/abs/2308.07317) |
|
||
| Pythia | {14,31,70,160,410}M, {1,1.4,2.8,6.9,12}B | EleutherAI | [Biderman et al. 2023](https://arxiv.org/abs/2304.01373) |
|
||
| Qwen2.5 | 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B | Alibaba Group | [Qwen Team 2024](https://qwenlm.github.io/blog/qwen2.5/) |
|
||
| Qwen2.5 Coder | 0.5B, 1.5B, 3B, 7B, 14B, 32B | Alibaba Group | [Hui, Binyuan et al. 2024](https://arxiv.org/abs/2409.12186) |
|
||
| Qwen2.5 1M (Long Context) | 7B, 14B | Alibaba Group | [Qwen Team 2025](https://qwenlm.github.io/blog/qwen2.5-1m/) |
|
||
| Qwen2.5 Math | 1.5B, 7B, 72B | Alibaba Group | [An, Yang et al. 2024](https://arxiv.org/abs/2409.12122) |
|
||
| QwQ | 32B | Alibaba Group | [Qwen Team 2025](https://qwenlm.github.io/blog/qwq-32b/) |
|
||
| QwQ-Preview | 32B | Alibaba Group | [Qwen Team 2024](https://qwenlm.github.io/blog/qwq-32b-preview/) |
|
||
| Qwen3 | 0.6B, 1.7B, 4B{Hybrid, Thinking-2507, Instruct-2507}, 8B, 14B, 32B | Alibaba Group | [Qwen Team 2025](https://arxiv.org/abs/2505.09388/) |
|
||
| Qwen3 MoE | 30B{Hybrid, Thinking-2507, Instruct-2507}, 235B{Hybrid, Thinking-2507, Instruct-2507} | Alibaba Group | [Qwen Team 2025](https://arxiv.org/abs/2505.09388/) |
|
||
| R1 Distill Llama | 8B, 70B | DeepSeek AI | [DeepSeek AI 2025](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf) |
|
||
| SmolLM2 | 135M, 360M, 1.7B | Hugging Face | [Hugging Face 2024](https://github.com/huggingface/smollm) |
|
||
| Salamandra | 2B, 7B | Barcelona Supercomputing Centre | [BSC-LTC 2024](https://github.com/BSC-LTC/salamandra) |
|
||
| StableCode | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) |
|
||
| StableLM | 3B, 7B | Stability AI | [Stability AI 2023](https://github.com/Stability-AI/StableLM) |
|
||
| StableLM Zephyr | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) |
|
||
| TinyLlama | 1.1B | Zhang et al. | [Zhang et al. 2023](https://github.com/jzhang38/TinyLlama) |
|
||
|
||
**提示**：运行 `litgpt download list` 命令即可列出所有可用模型。
|
||
|
||
|
||
</details>
|
||
|
||
|
||
|
||
---
|
||
|
||
# 工作流
|
||
|
||
<p align="center">
|
||
<a href="#finetune-an-llm">微调</a> •
|
||
<a href="#pretrain-an-llm">预训练</a> •
|
||
<a href="#continue-pretraining-an-llm">继续预训练</a> •
|
||
<a href="#evaluate-an-llm">评估</a> •
|
||
<a href="#deploy-an-llm">部署</a> •
|
||
<a href="#test-an-llm">测试</a>
|
||
</p>
|
||
|
||
|
||
|
||
使用命令行界面（CLI）在你自己的数据上运行预训练、微调等高级工作流。
|
||
|
||
|
||
## 所有工作流
|
||
安装 LitGPT 后，选择要运行的模型和工作流（微调、预训练、评估、部署等）：
|
||
|
||
```bash
|
||
# litgpt [action] [model]
|
||
litgpt serve meta-llama/Llama-3.2-3B-Instruct
|
||
litgpt finetune meta-llama/Llama-3.2-3B-Instruct
|
||
litgpt pretrain meta-llama/Llama-3.2-3B-Instruct
|
||
litgpt chat meta-llama/Llama-3.2-3B-Instruct
|
||
litgpt evaluate meta-llama/Llama-3.2-3B-Instruct
|
||
```
|
||
|
||
|
||
|
||
----
|
||
|
||
## 微调 LLM
|
||
|
||
<div align="center">
|
||
<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-finetune">
|
||
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/run-on-studio.svg" height="36px" alt="在 Studios 上运行"/>
|
||
</a>
|
||
</div>
|
||
|
||
|
||
|
||
微调（finetuning）是指在预训练 AI 模型基础上，使用针对特定任务或应用定制的小规模专用数据集进一步训练的过程。
|
||
|
||
|
||
|
||
|
||
```bash
|
||
# 0) setup your dataset
|
||
curl -L https://huggingface.co/datasets/ksaw008/finance_alpaca/resolve/main/finance_alpaca.json -o my_custom_dataset.json
|
||
|
||
# 1) Finetune a model (auto downloads weights)
|
||
litgpt finetune microsoft/phi-2 \
|
||
--data JSON \
|
||
--data.json_path my_custom_dataset.json \
|
||
--data.val_split_fraction 0.1 \
|
||
--out_dir out/custom-model
|
||
|
||
# 2) Test the model
|
||
litgpt chat out/custom-model/final
|
||
|
||
# 3) Deploy the model
|
||
litgpt serve out/custom-model/final
|
||
```
|
||
|
||
[阅读完整的微调文档](tutorials/finetune.md)
|
||
|
||
|
||
|
||
----
|
||
|
||
## 部署 LLM
|
||
|
||
<div align="center">
|
||
<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-serve">
|
||
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/deploy-on-studios.svg" height="36px" alt="在 Studios 上部署"/>
|
||
</a>
|
||
</div>
|
||
|
||
|
||
|
||
部署预训练或微调后的 LLM，以便在实际应用中使用。部署会自动设置可通过网站或应用访问的 Web 服务器。
|
||
|
||
```bash
|
||
# deploy an out-of-the-box LLM
|
||
litgpt serve microsoft/phi-2
|
||
|
||
# deploy your own trained model
|
||
litgpt serve path/to/microsoft/phi-2/checkpoint
|
||
```
|
||
|
||
<details>
|
||
<summary>显示查询服务器的代码：</summary>
|
||
|
||
|
||
|
||
在单独的终端中测试服务器，并将模型 API 集成到你的 AI 产品中：
|
||
```python
|
||
# 3) Use the server (in a separate Python session)
|
||
import requests, json
|
||
response = requests.post(
|
||
"http://127.0.0.1:8000/predict",
|
||
json={"prompt": "Fix typos in the following sentence: Example input"}
|
||
)
|
||
print(response.json()["output"])
|
||
```
|
||
</details>
|
||
|
||
[阅读完整的部署文档](tutorials/deploy.md)。
|
||
|
||
|
||
|
||
----
|
||
|
||
## 评估 LLM
|
||
评估 LLM 以测试其在各类任务上的表现，了解其理解与生成文本的能力。简单来说，我们可以评估它在大学水平化学、编程等方面的表现等（MMLU、Truthful QA 等）。
|
||
|
||
```bash
|
||
litgpt evaluate microsoft/phi-2 --tasks 'truthfulqa_mc2,mmlu'
|
||
```
|
||
|
||
[阅读完整的评估文档](tutorials/evaluation.md)。
|
||
|
||
|
||
|
||
----
|
||
|
||
## 测试 LLM
|
||
|
||
<div align="center">
|
||
<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-chat">
|
||
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/run-on-studio.svg" height="36px" alt="在 Studios 上运行"/>
|
||
</a>
|
||
</div>
|
||
|
||
|
||
|
||
通过交互式聊天测试模型效果。使用 `chat` 命令进行聊天、提取嵌入（embeddings）等。
|
||
|
||
以下示例展示如何使用 Phi-2 LLM：
|
||
```bash
|
||
litgpt chat microsoft/phi-2
|
||
|
||
>> Prompt: What do Llamas eat?
|
||
```
|
||
|
||
<details>
|
||
<summary>完整代码：</summary>
|
||
|
||
|
||
|
||
```bash
|
||
# 1) List all supported LLMs
|
||
litgpt download list
|
||
|
||
# 2) Use a model (auto downloads weights)
|
||
litgpt chat microsoft/phi-2
|
||
|
||
>> Prompt: What do Llamas eat?
|
||
```
|
||
|
||
某些模型的下载需要额外的访问令牌（access token）。详情请参阅[下载](tutorials/download_model_weights.md#specific-models-and-access-tokens)文档。
|
||
|
||
</details>
|
||
|
||
[阅读完整的聊天文档](tutorials/inference.md)。
|
||
|
||
|
||
|
||
----
|
||
|
||
## 预训练 LLM
|
||
|
||
<div align="center">
|
||
<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-pretrain">
|
||
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/run-on-studio.svg" height="36px" alt="在 Studios 上运行"/>
|
||
</a>
|
||
</div>
|
||
|
||
|
||
|
||
预训练（pretraining）是指在针对特定任务进行微调之前，通过让 AI 模型接触大量数据来训练模型的过程。
|
||
|
||
<details>
|
||
<summary>显示代码：</summary>
|
||
|
||
|
||
|
||
```bash
|
||
mkdir -p custom_texts
|
||
curl https://www.gutenberg.org/cache/epub/24440/pg24440.txt --output custom_texts/book1.txt
|
||
curl https://www.gutenberg.org/cache/epub/26393/pg26393.txt --output custom_texts/book2.txt
|
||
|
||
# 1) Download a tokenizer
|
||
litgpt download EleutherAI/pythia-160m \
|
||
--tokenizer_only True
|
||
|
||
# 2) Pretrain the model
|
||
litgpt pretrain EleutherAI/pythia-160m \
|
||
--tokenizer_dir EleutherAI/pythia-160m \
|
||
--data TextFiles \
|
||
--data.train_data_path "custom_texts/" \
|
||
--train.max_tokens 10_000_000 \
|
||
--out_dir out/custom-model
|
||
|
||
# 3) Test the model
|
||
litgpt chat out/custom-model/final
|
||
```
|
||
</details>
|
||
|
||
[阅读完整的预训练文档](tutorials/pretrain.md)
|
||
|
||
|
||
|
||
----
|
||
|
||
## 继续预训练 LLM
|
||
|
||
<div align="center">
|
||
<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-continue-pretraining">
|
||
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/run-on-studio.svg" height="36px" alt="在 Studios 上运行"/>
|
||
</a>
|
||
</div>
|
||
|
||
|
||
|
||
继续预训练（continued pretraining）是另一种微调方式，通过在自定义数据上训练，使已预训练的模型更加专业化：
|
||
|
||
<details>
|
||
<summary>显示代码：</summary>
|
||
|
||
|
||
|
||
```bash
|
||
mkdir -p custom_texts
|
||
curl https://www.gutenberg.org/cache/epub/24440/pg24440.txt --output custom_texts/book1.txt
|
||
curl https://www.gutenberg.org/cache/epub/26393/pg26393.txt --output custom_texts/book2.txt
|
||
|
||
# 1) Continue pretraining a model (auto downloads weights)
|
||
litgpt pretrain EleutherAI/pythia-160m \
|
||
--tokenizer_dir EleutherAI/pythia-160m \
|
||
--initial_checkpoint_dir EleutherAI/pythia-160m \
|
||
--data TextFiles \
|
||
--data.train_data_path "custom_texts/" \
|
||
--train.max_tokens 10_000_000 \
|
||
--out_dir out/custom-model
|
||
|
||
# 2) Test the model
|
||
litgpt chat out/custom-model/final
|
||
```
|
||
|
||
</details>
|
||
|
||
[阅读完整的继续预训练文档](tutorials/pretrain.md#continued-pretraining-on-custom-data)
|
||
|
||
|
||
|
||
----
|
||
|
||
# 前沿特性
|
||
|
||
✅ 前沿优化：Flash Attention v2、通过全分片数据并行（fully-sharded data parallelism）实现的多 GPU 支持、[可选 CPU 卸载](tutorials/oom.md#do-sharding-across-multiple-gpus)，以及 [TPU 和 XLA 支持](extensions/xla)。</br>
|
||
✅ [预训练](tutorials/pretrain.md)、[微调](tutorials/finetune.md)和[部署](tutorials/inference.md)</br>
|
||
✅ 通过低精度设置降低算力需求：FP16、BF16 以及 FP16/FP32 混合精度。</br>
|
||
✅ 通过[量化](tutorials/quantize.md)降低内存需求：4 位浮点数、8 位整数和双重量化（double quantization）。</br>
|
||
✅ [配置文件](config_hub)带来出色的开箱即用性能。</br>
|
||
✅ 参数高效微调（parameter-efficient finetuning）：[LoRA](tutorials/finetune_lora.md)、[QLoRA](tutorials/finetune_lora.md)、[Adapter](tutorials/finetune_adapter.md) 和 [Adapter v2](tutorials/finetune_adapter.md)。</br>
|
||
✅ [导出](tutorials/convert_lit_models.md)为其他流行的模型权重格式。</br>
|
||
✅ 丰富的热门数据集，适用于[预训练](tutorials/pretrain.md)和[微调](tutorials/prepare_dataset.md)，并[支持自定义数据集](tutorials/prepare_dataset.md#preparing-custom-datasets-for-instruction-finetuning)。</br>
|
||
✅ 代码清晰易改，便于试验最新研究思路。</br>
|
||
|
||
|
||
|
||
---
|
||
|
||
# 训练配方
|
||
|
||
LitGPT 提供经过验证的训练配方（YAML 配置），可在不同条件下训练模型。我们根据在不同训练条件下表现最佳的参数生成了这些配方。
|
||
|
||
在[此处](config_hub)浏览所有训练配方。
|
||
|
||
### 示例
|
||
|
||
```bash
|
||
litgpt finetune \
|
||
--config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml
|
||
```
|
||
<details>
|
||
<summary>✅ 使用配置自定义训练</summary>
|
||
|
||
配置允许你为所有细粒度参数自定义训练，例如：
|
||
|
||
```yaml
|
||
# The path to the base model's checkpoint directory to load for finetuning. (type: <class 'Path'>, default: checkpoints/stabilityai/stablelm-base-alpha-3b)
|
||
checkpoint_dir: checkpoints/meta-llama/Llama-2-7b-hf
|
||
|
||
# Directory in which to save checkpoints and logs. (type: <class 'Path'>, default: out/lora)
|
||
out_dir: out/finetune/qlora-llama2-7b
|
||
|
||
# The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null)
|
||
precision: bf16-true
|
||
|
||
...
|
||
```
|
||
</details>
|
||
|
||
<details>
|
||
<summary>✅ 示例：LoRA 微调配置</summary>
|
||
|
||
|
||
|
||
```yaml
|
||
# The path to the base model's checkpoint directory to load for finetuning. (type: <class 'Path'>, default: checkpoints/stabilityai/stablelm-base-alpha-3b)
|
||
checkpoint_dir: checkpoints/meta-llama/Llama-2-7b-hf
|
||
|
||
# Directory in which to save checkpoints and logs. (type: <class 'Path'>, default: out/lora)
|
||
out_dir: out/finetune/qlora-llama2-7b
|
||
|
||
# The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null)
|
||
precision: bf16-true
|
||
|
||
# If set, quantize the model with this algorithm. See ``tutorials/quantize.md`` for more information. (type: Optional[Literal['nf4', 'nf4-dq', 'fp4', 'fp4-dq', 'int8-training']], default: null)
|
||
quantize: bnb.nf4
|
||
|
||
# How many devices/GPUs to use. (type: Union[int, str], default: 1)
|
||
devices: 1
|
||
|
||
# How many nodes to use. (type: int, default: 1)
|
||
num_nodes: 1
|
||
|
||
# The LoRA rank. (type: int, default: 8)
|
||
lora_r: 32
|
||
|
||
# The LoRA alpha. (type: int, default: 16)
|
||
lora_alpha: 16
|
||
|
||
# The LoRA dropout value. (type: float, default: 0.05)
|
||
lora_dropout: 0.05
|
||
|
||
# Whether to apply LoRA to the query weights in attention. (type: bool, default: True)
|
||
lora_query: true
|
||
|
||
# Whether to apply LoRA to the key weights in attention. (type: bool, default: False)
|
||
lora_key: false
|
||
|
||
# Whether to apply LoRA to the value weights in attention. (type: bool, default: True)
|
||
lora_value: true
|
||
|
||
# Whether to apply LoRA to the output projection in the attention block. (type: bool, default: False)
|
||
lora_projection: false
|
||
|
||
# Whether to apply LoRA to the weights of the MLP in the attention block. (type: bool, default: False)
|
||
lora_mlp: false
|
||
|
||
# Whether to apply LoRA to output head in GPT. (type: bool, default: False)
|
||
lora_head: false
|
||
|
||
# Data-related arguments. If not provided, the default is ``litgpt.data.Alpaca``.
|
||
data:
|
||
class_path: litgpt.data.Alpaca2k
|
||
init_args:
|
||
mask_prompt: false
|
||
val_split_fraction: 0.05
|
||
prompt_style: alpaca
|
||
ignore_index: -100
|
||
seed: 42
|
||
num_workers: 4
|
||
download_dir: data/alpaca2k
|
||
|
||
# Training-related arguments. See ``litgpt.args.TrainArgs`` for details
|
||
train:
|
||
|
||
# Number of optimizer steps between saving checkpoints (type: Optional[int], default: 1000)
|
||
save_interval: 200
|
||
|
||
# Number of iterations between logging calls (type: int, default: 1)
|
||
log_interval: 1
|
||
|
||
# Number of samples between optimizer steps across data-parallel ranks (type: int, default: 128)
|
||
global_batch_size: 8
|
||
|
||
# Number of samples per data-parallel rank (type: int, default: 4)
|
||
micro_batch_size: 2
|
||
|
||
# Number of iterations with learning rate warmup active (type: int, default: 100)
|
||
lr_warmup_steps: 10
|
||
|
||
# Number of epochs to train on (type: Optional[int], default: 5)
|
||
epochs: 4
|
||
|
||
# Total number of tokens to train on (type: Optional[int], default: null)
|
||
max_tokens:
|
||
|
||
# Limits the number of optimizer steps to run (type: Optional[int], default: null)
|
||
max_steps:
|
||
|
||
# Limits the length of samples (type: Optional[int], default: null)
|
||
max_seq_length: 512
|
||
|
||
# Whether to tie the embedding weights with the language modeling head weights (type: Optional[bool], default: null)
|
||
tie_embeddings:
|
||
|
||
# (type: float, default: 0.0003)
|
||
learning_rate: 0.0002
|
||
|
||
# (type: float, default: 0.02)
|
||
weight_decay: 0.0
|
||
|
||
# (type: float, default: 0.9)
|
||
beta1: 0.9
|
||
|
||
# (type: float, default: 0.95)
|
||
beta2: 0.95
|
||
|
||
# (type: Optional[float], default: null)
|
||
max_norm:
|
||
|
||
# (type: float, default: 6e-05)
|
||
min_lr: 6.0e-05
|
||
|
||
# Evaluation-related arguments. See ``litgpt.args.EvalArgs`` for details
|
||
eval:
|
||
|
||
# Number of optimizer steps between evaluation calls (type: int, default: 100)
|
||
interval: 100
|
||
|
||
# Number of tokens to generate (type: Optional[int], default: 100)
|
||
max_new_tokens: 100
|
||
|
||
# Number of iterations (type: int, default: 100)
|
||
max_iters: 100
|
||
|
||
# The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv)
|
||
logger_name: csv
|
||
|
||
# The random seed to use for reproducibility. (type: int, default: 1337)
|
||
seed: 1337
|
||
```
|
||
</details>
|
||
|
||
<details>
|
||
<summary>✅ 在 CLI 中覆盖任意参数：</summary>
|
||
|
||
```bash
|
||
litgpt finetune \
|
||
--config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml \
|
||
--lora_r 4
|
||
```
|
||
</details>
|
||
|
||
|
||
|
||
----
|
||
|
||
# 项目亮点
|
||
|
||
LitGPT 为许多出色的 AI 项目、倡议、挑战赛以及企业级应用提供支持。欢迎提交 pull request 申请展示。
|
||
|
||
<details>
|
||
<summary>📊 SAMBA: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling</summary>
|
||
|
||
微软研究人员开展的 [Samba](https://github.com/microsoft/Samba) 项目基于 LitGPT 代码库构建，将 state space models（状态空间模型）与 sliding window attention（滑动窗口注意力）相结合，性能优于纯状态空间模型。
|
||
|
||
</details>
|
||
|
||
<details>
|
||
<summary>🏆 NeurIPS 2023 Large Language Model Efficiency Challenge: 1 LLM + 1 GPU + 1 Day</summary>
|
||
|
||
LitGPT 仓库是 [NeurIPS 2023 LLM Efficiency Challenge](https://llm-efficiency-challenge.github.io), 的官方入门套件，该竞赛专注于在单块 GPU 上对现有的非指令微调 LLM 进行 24 小时微调。
|
||
|
||
</details>
|
||
|
||
<details>
|
||
<summary>🦙 TinyLlama: An Open-Source Small Language Model</summary>
|
||
|
||
|
||
LitGPT 为 [TinyLlama 项目](https://github.com/jzhang38/TinyLlama) 以及研究论文 [TinyLlama: An Open-Source Small Language Model](https://arxiv.org/abs/2401.02385) 提供支持。
|
||
|
||
</details>
|
||
|
||
<details>
|
||
<summary>🍪 MicroLlama: MicroLlama-300M</summary>
|
||
|
||
[MicroLlama](https://github.com/keeeeenw/MicroLlama) 是一个 300M 参数的 Llama 模型，在 50B token 上预训练，由 TinyLlama 和 LitGPT 提供支持。
|
||
</details>
|
||
|
||
<details>
|
||
<summary>🔬 Pre-training Small Base LMs with Fewer Tokens</summary>
|
||
|
||
研究论文 ["Pre-training Small Base LMs with Fewer Tokens"](https://arxiv.org/abs/2404.08634), 使用 LitGPT，通过从更大模型继承少量 transformer 块并在远小于大模型所用数据量的数据上训练，来开发更小的 base language model（基础语言模型）。研究表明，这些更小的模型尽管使用的训练数据和资源显著更少，却能与更大模型达到相当性能。
|
||
|
||
</details>
|
||
|
||
|
||
|
||
----
|
||
|
||
# 社区
|
||
|
||
我们欢迎所有个人贡献者，无论经验水平或硬件条件如何。你的贡献十分宝贵，我们期待看到你能在这个协作且互助的环境中取得成就。
|
||
|
||
- [申请功能](https://github.com/Lightning-AI/litgpt/issues)
|
||
- [提交你的首次贡献](https://lightning.ai/pages/community/tutorial/how-to-contribute-to-litgpt/)
|
||
- [加入我们的 Discord](https://discord.gg/VptPCZkGNa)
|
||
|
||
|
||
|
||
# 教程
|
||
|
||
🚀 [入门指南](tutorials/0_to_litgpt.md)</br>
|
||
⚡️ [微调，包括 LoRA、QLoRA 和 Adapters](tutorials/finetune.md)</br>
|
||
🤖 [预训练](tutorials/pretrain.md)</br>
|
||
💬 [模型评估](tutorials/evaluation.md)</br>
|
||
📘 [支持的与自定义数据集](tutorials/prepare_dataset.md)</br>
|
||
🧹 [量化](tutorials/quantize.md)</br>
|
||
🤯 [处理显存不足（OOM）错误的技巧](tutorials/oom.md)</br>
|
||
🧑🏽💻 [使用云端 TPU](extensions/xla)</br>
|
||
|
||
|
||
|
||
----
|
||
|
||
### 致谢
|
||
|
||
本实现在 [Lit-LLaMA](https://github.com/lightning-AI/lit-llama) and [nanoGPT](https://github.com/karpathy/nanoGPT), 的基础上进行扩展，并由 **[Lightning Fabric](https://lightning.ai/docs/fabric/stable/) ⚡** 提供支持。
|
||
|
||
- [@karpathy](https://github.com/karpathy) 贡献了 [nanoGPT](https://github.com/karpathy/nanoGPT)
|
||
- [@EleutherAI](https://github.com/EleutherAI) 贡献了 [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) 以及 [Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness)
|
||
- [@TimDettmers](https://github.com/TimDettmers) 贡献了 [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
|
||
- [@Microsoft](https://github.com/microsoft) 贡献了 [LoRA](https://github.com/microsoft/LoRA)
|
||
- [@tridao](https://github.com/tridao) 贡献了 [Flash Attention 2](https://github.com/Dao-AILab/flash-attention)
|
||
|
||
### 许可证
|
||
|
||
LitGPT 在 [Apache 2.0](https://github.com/Lightning-AI/litgpt/blob/main/LICENSE.md) 许可证下发布。
|
||
|
||
### 引用
|
||
|
||
如果你在研究中使用 LitGPT，请引用以下工作：
|
||
|
||
```bibtex
|
||
@misc{litgpt-2023,
|
||
author = {Lightning AI},
|
||
title = {LitGPT},
|
||
howpublished = {\url{https://github.com/Lightning-AI/litgpt}},
|
||
year = {2023},
|
||
}
|
||
```
|
||
|
||
|