Worked on the volcengine/verl repository to deliver the Self-Play Fine-Tuning (SPIN) algorithm, adapting the existing PPO framework to use a DPO-based objective. This involved enforcing a reference model requirement, removing the critic component, and shifting the update signal from advantage estimates to log-probability differences. The data pipeline was reworked to support preference pairs, enabling stable self-play fine-tuning for large language models. Leveraging Python, PyTorch, and Ray, the implementation laid foundational groundwork for improved sample efficiency and policy alignment in Verl, supporting faster experimentation and enhancing the platform’s capabilities in distributed deep learning and reinforcement learning workflows.