分布式训练 on vllbc02's blogs https://blog.vllbc.top/tags/%E5%88%86%E5%B8%83%E5%BC%8F%E8%AE%AD%E7%BB%83/ Recent content in 分布式训练 on vllbc02's blogs Hugo -- gohugo.io zh-cn Fri, 06 Nov 2026 00:00:00 +0000 Ultra-Scale 训练的容错与 Checkpoint https://blog.vllbc.top/visual-reading/ultrasale-training-fault-tolerance/ Fri, 06 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/ultrasale-training-fault-tolerance/ FSDP1 vs FSDP2 https://blog.vllbc.top/visual-reading/fsdp1-vs-fsdp2/ Thu, 05 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/fsdp1-vs-fsdp2/ ZeRO / DeepSpeed https://blog.vllbc.top/visual-reading/zero-deepspeed/ Thu, 05 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/zero-deepspeed/ Tensor Parallel（Megatron） https://blog.vllbc.top/visual-reading/tensor-parallel-megatron/ Thu, 05 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/tensor-parallel-megatron/ Expert Parallel（MoE 训练） https://blog.vllbc.top/visual-reading/expert-parallel-moe-training/ Fri, 06 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/expert-parallel-moe-training/ Pipeline Parallel https://blog.vllbc.top/visual-reading/pipeline-parallel/ Thu, 05 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/pipeline-parallel/ DDP 与 Ring All-Reduce https://blog.vllbc.top/visual-reading/ddp-ring-allreduce/ Fri, 06 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/ddp-ring-allreduce/ Device Mesh 与 3D/4D 并行 https://blog.vllbc.top/visual-reading/device-mesh-3d-parallel/ Fri, 06 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/device-mesh-3d-parallel/ Megatron-LM 与 PAI-Megatron：训练框架怎么组织一次训练 https://blog.vllbc.top/visual-reading/megatron-lm-pai-megatron/ Fri, 06 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/megatron-lm-pai-megatron/ 梯度累积与 Global Batch https://blog.vllbc.top/visual-reading/gradient-accumulation-global-batch/ Fri, 06 Nov 2026 00:00:00 +0000 https://blog.vllbc.top/visual-reading/gradient-accumulation-global-batch/