Worked on improving distributed training stability in the NVIDIA-NeMo/Automodel repository by addressing a critical issue with HSDP replica gradient synchronization for Mixture of Experts (MoE) models. Updated the parallelizer to utilize a root-preserving mesh for Fully Sharded Data Parallel (FSDP) when employing flattened data parallel axes, ensuring consistent gradient updates across replica groups. Developed and integrated a regression test to verify that replicated shards remain synchronized, which reduces training divergence and enhances reproducibility for large-scale MoE deployments. The work leveraged expertise in distributed systems, PyTorch, and Python, with a focus on robust unit testing and reliable model training workflows.