Developed a configurable head_dim parameter for the Attention and Rope modules in the mosaicml/llm-foundry repository, enabling more flexible customization of attention architectures. The implementation, written in Python using PyTorch, ensured that head_dim was correctly integrated into weight and bias calculations for attention projections, reducing the risk of misconfiguration. This addition supports scalable experimentation and easier tuning of transformer models by allowing varied head_dim settings across different architectures. The work focused on deep learning model architecture, facilitating more efficient and accurate model development while providing a foundation for improved performance through targeted adjustments to attention mechanism parameters.