Worked on enhancing the Gemma4 model within the NVIDIA-NeMo/Automodel repository, focusing on improving mixture of experts (MoE) attention mask handling and input packing. Leveraged Python and deep learning techniques to refine position ID propagation and implement top-k expert routing consistent with Gemma4Router standards. Addressed edge cases in attention mask logic to prevent sequence boundary leakage, ensuring more reliable model training and flexible configuration for future experiments. Expanded test coverage and performed lint cleanups to improve code maintainability. The work emphasized robust model training, NLP capabilities, and maintainable code, laying groundwork for future compatibility with EP_SHARD mesh state dicts.