Contributed to the NVIDIA/Megatron-LM repository by developing a distributed-training feature that enables overlap parameter gathering for layer-wise optimizers, supporting asynchronous multi-GPU parameter synchronization and improved memory management. This work leveraged Python, PyTorch, and GPU programming to enhance the scalability and efficiency of large-scale deep learning experiments. Comprehensive unit tests and integration checks were implemented to validate the new feature and ensure reliability in production training pipelines. The contribution strengthened distributed computing workflows, allowing for more efficient training of larger models and faster research iteration cycles, while also reinforcing collaborative development practices through co-authored pull requests and cross-team coordination.