/llm-train/megatron-deepspeed

微软

• Megatron-DeepSpeed
• 目前，已支持llama/llama2

模型转换：

• https://github.com/microsoft/Megatron-DeepSpeed/tree/main/tools/convert_checkpoint
• https://github.com/huggingface/transformers/tree/main/src/transformers/models/megatron_gpt2

bigscience

• Megatron-DeepSpeed
• 训练过程：https://github.com/bigscience-workshop/bigscience
• https://huggingface.co/blog/zh/bloom-megatron-deepspeed
• 支持bloom
• 未支持序列并行