用 NVIDIA Megatron Core 扩展位级确定性预训练
介绍在万亿参数预训练中降低位级确定性开销的方法。
为什么重要
可帮助研究者和工程团队更可靠地调试、验证和恢复大规模预训练。
阅读原文developer.nvidia.com
AI 解读（根据原文生成）
NVIDIA Technical Blog 文章讨论如何用 NVIDIA Megatron Core 支持大规模位级确定性预训练。文章说明位级确定性要求在固定数据顺序、架构、配方、并行方式、软件、运行时设置和硬件时，独立运行和从检查点恢复的运行保持相同数值轨迹。Megatron Core 目标包括独立可复现和位级检查点恢复，并建议用每步数值指纹验证训练损失、语言建模损失、负载均衡损失、MTP 损失和梯度范数。文中还提到万亿参数 Nemotron 案例用于降低确定性预训练开销。
要点（AI 据原文整理，数字均已与原文核对）
- Bitwise determinism requires independent and checkpoint-resumed runs to follow the same numerical trajectory when data order, architecture, recipe, parallelism, software, runtime settings, and hardwar
- Megatron Core targets independent reproducibility and bitwise checkpoint resume.
- Bitwise determinism provides benefits including replay failures, preserve checkpoint continuity, detect unexpected differences, and compare changes.
- Printed loss values are not sensitive enough to establish bitwise identical numerics.
- The validation procedure fingerprints selected numerical metrics at every step.
- The trillion-parameter Nemotron model checkpoint path preserves a full-precision source of truth.
