January 2026 performance summary for NVIDIA/Megatron-LM and NVIDIA-NeMo/Megatron-Bridge. Delivered transformer, MoE, and scalability enhancements focused on improving model configurability, training efficiency, and inference performance for large-scale deployments (Qwen3-Next). Key outcomes include a new attention output gate for transformer attention, a shared expert gate for MoE, Gated Delta Net (GDN) attention enabling linear attention variants, weight decay support for QK LayerNorm with a test flag, and scalable tensor-parallel weight conversion for GDN and Mamba 1D convolutions. In addition, resolved a tensor-parallel conversion issue for TP > 1 to stabilize Qwen3NextBridge when configuring larger models. These changes enable larger models, more flexible configurations, and better regularization, contributing to improved accuracy and reduced training costs at scale.

September 2025 monthly summary for NVIDIA/TransformerEngine focused on memory efficiency and reliability improvements in sequence-parallel deployment paths. Delivered a critical bug fix that eliminates memory overhead and potential leaks during tensor deallocation in all-gather scenarios across linear layers and FP8 tensors, improving stability for large-scale training.

July 2025 monthly summary for NVIDIA/TransformerEngine: Implemented a focused FP8 Attention Backend Selection Condition Fix, strengthening the FP8 MLA attention path and backend routing under context parallelism. The patch ensures fused attention is disabled when appropriate and that the correct backend is selected for attention with differing head dimensions, reducing misrouting and potential correctness issues.

June 2025 — NVIDIA/TransformerEngine: Delivered Multi Latent Attention (MLA) support within the Context Parallel (CP) fused attention framework, enabling AttnFuncWithCPAndKVP2P2P to handle cases where query/key dimensions differ from value dimensions. Included data handling, communication buffer updates, and gradient calculation changes, plus new tests. Also delivered targeted fixes addressing MLA-CP correctness, notably FP8 handling (disabling FP8 CP for MLA due to correctness concerns) and ensuring proper handling when head dimensions differ under FP8. Commits: faee0e8bb046bfe9a481158e7ac9796d10e8640f; 9d173c93e67213bb87c7c4286a5543867bd22bdf.

April 2025 monthly summary: NVIDIA/Megatron-LM delivered precise resource estimation improvements for MLA, MoE, and MTP configurations, enhancing forecasting accuracy for complex model architectures. This supported better capacity planning, smoother deployment, and cost optimization for scalable AI workloads.