Contributed to NVIDIA/Megatron-LM by developing a layer-wise distributed optimizer and a muon optimizer to enhance scalable training for large models. The work focused on improving distributed training efficiency by distributing weights across ranks and optimizing parameter updates, which increased training throughput and scalability in multi-node, multi-GPU environments. Integration with the existing distributed training pipeline was validated to ensure compatibility and readiness for broader deployment. The implementation leveraged Python, PyTorch, and deep learning techniques, with an emphasis on distributed computing and optimizer design. These enhancements laid the foundation for scaling Megatron-LM to support larger architectures and more complex workloads.