Worked on stabilizing distributed training in the NVIDIA/Megatron-LM repository by addressing a subtle bug in the TransformerLayer’s attention mechanism. Focused on correcting the QK layer indexing logic under pipeline parallelism, ensuring that QK scaling calculations remain accurate when PP is greater than one. This fix targeted a critical source of instability in large-scale deep learning models, reducing the risk of divergence during training. The solution involved careful updates to the self_attention and cross_attention modules, enhancing both correctness and diagnostic clarity. Leveraged expertise in Python, distributed systems, and transformer architecture to improve maintainability and reliability for future model development.