Worked on the jeejeelee/vllm repository to introduce the Param2MoE model, a mixture-of-experts architecture designed for scalable causal language modeling. Focused on enabling reliable large-scale inference by aligning model architecture for parallel processing across devices. Addressed a critical tensor-parallel head alignment issue, ensuring accurate distribution of local and global attention heads in tensor-parallel mode. Utilized deep learning and model development expertise with PyTorch and Python to deliver both the new model integration and the bug fix. The work laid the foundation for robust, scalable deployments of Param2MoE, emphasizing code quality and architectural consistency for future development.