May 2025 monthly summary for huggingface/text-generation-inference: Delivered Chunked Prefill for Vision-Language Models (VLMs), including refactoring to isolate image embeddings and integrate them into text input embeddings. Implemented performance optimizations across VLM architectures and addressed image token handling issues. This work advances multimodal input efficiency and model throughput, enabling faster, more scalable VLM inference.

April 2025 monthly summary for huggingface/text-generation-inference, focusing on delivered features, fixes, and impact.

In March 2025, the team delivered strategic platform enhancements for HuggingFace text-generation-inference, expanding model support and reinforcing robustness. Gemma3 model integration now supports text and multimodal workflows with new configurations, integration tests, and updated chat templates, image processing, and model loading for seamless operation. Concurrently, attention and compatibility fixes for Gemma3 and Qwen2 addressed sliding-window attention issues, improved cross-model robustness, and updated dependencies. These efforts broaden client capabilities, reduce integration risk, and improve inference reliability across configurations.

Delivered end-to-end ROCm FP8-accelerated inference stack for text generation in Hugging Face, including FP8 per-tensor scales, FP8 KV cache for paged attention, FP8-aware MoE computations, and integration of Marlin/MoE kernels. Implemented Flash decoding kernel integration and Dockerfile stages to build and deploy FP8-optimized components on ROCm devices. Maintained the ROCm AMD environment by upgrading moe-kernels to v0.8.2 in Dockerfile_amd. Added PyTorch FA backend compatibility guard for AMD GPUs to disable the FA backend when PyTorch is below 2.4.1 to prevent potential performance issues. These efforts improved inference throughput and reliability on ROCm/AMD hardware and ensured compatibility with current PyTorch releases, enabling cost-effective 8-bit inference for large models and easier deployment across ROCm platforms.

December 2024: Delivered ROCm support and performance optimization for the text-generation-inference server in the huggingface/text-generation-inference repository. Key work included updating vLLM kernels for ROCm compatibility and performance improvements; Dockerfile enhancements to build and install ROCm dependencies; kernel configuration changes to improve partitioning and efficiency; ROCm-specific implementations for attention and normalization layers refactored to boost performance and stability on ROCm-enabled hardware. This work broadens hardware compatibility, improves inference throughput and stability, and lays the groundwork for broader GPU-accelerated deployments. Commit reference: 8f66d323d038dcac93d5f73f47cb44ab1da2ce17.