Worked on optimizing Parquet metadata handling in the pinterest/ray repository, focusing on improving memory efficiency and stability for large, wide-column datasets. Developed a feature that merges and simplifies dataset metadata within each _fetch_metadata task before transmitting it to the driver, which reduced peak memory usage and mitigated out-of-memory risks during data discovery workflows. This approach alleviated driver resource pressure and enhanced the scalability of metadata-heavy operations. Leveraged Python and applied expertise in data engineering, distributed systems, and memory management to deliver a solution that improved reliability and performance for large-scale data processing without introducing new bugs during the development period.