June 2026 performance month for ROCm/aiter focused on MoE workloads on gfx1250. Delivered a set of end-to-end MoE improvements, expanded hardware coverage, and strengthened reliability and testing, with clear business value in serving scale, latency, and maintainability. Key features delivered: - MoE kernel optimization and fused kernel for gfx1250: introduced 2-stage GEMM kernels with dispatch, numerical correctness fixes, and end-to-end support for non-WMMA-aligned shapes; added high-performance fused MoE kernel that combines routing, quantization, and scattering; improved AOT compilation and error handling; TDM store optimizations; comprehensive unit tests for MoE workloads on gfx1250. (Commit: 76ea4ac325ddde6c895b4e7ade4d42b592033828) - Grouped MoE kernel enhancements: added fused MoE route+psum+quant+scatter kernel for grouped MoE, enabling end-to-end grouped MoE execution on gfx1250 FlyDSL path; improvements to AOT cache handling and test harness that align with the main MoE path. (Commit: 3b161e226e8b8e76382a196edef7adc493b58b5c) - End-to-end gfx1250 MoE on non-WMMA shapes: syncs and updates across FlyDSL MoE kernels to support non-WMMA shapes, including per_1x32 quantization variants (fp4, a8w4, fp8) and forward/backward path alignment; strengthened numerical correctness and test coverage. - Reliability and test coverage: broad unit-test improvements, patching numerical correctness gaps in gfx1250 FP4/FP8 paths, and robust testing around end-to-end MoE flow, including GPU-accelerated routes and AOT cache checks. - Performance and scalability improvements: introduced TDM gather caching, LDS caching, and carry-safe tensor descriptors in the gfx1250 MoE kernels, enabling higher throughput and lower memory traffic for large-scale MoE models. Major bugs fixed: - Numerical correctness fixes for gfx1250 FP4/FP8 paths and per-1x32 quant branches, ensuring bit-exact results or within tight tolerances across diverse shapes. - Stabilized end-to-end gfx1250 MoE paths on non-WMMA shapes, including zero-initialization and quantization path alignment to avoid leaking uninitialized memory or mis-ordered tiles. - Resolved issues in bool reductions that could deadlock or hang in end-to-end MoE tests; replaced with safe reduction checks (e.g., sum-based checks) to avoid deadlocks. - Addressed data-format and layout edge-cases in grouped MoE paths (grouped GEMM, MS/DS/TD paths) to ensure stable routing and accumulation under diverse activation/data formats. Overall impact and accomplishments: - Business value: unlocks scalable MoE serving on gfx1250 hardware with end-to-end, reliable, high-performance paths, enabling production-grade inference for larger models and more diverse quant formats, with lower latency and higher throughput. - Technical achievements: robust end-to-end MoE feature set, first-class support for non-WMMA shapes on gfx1250, fused kernel design bringing routing/quantization/scattering closer, stronger AOT/test infrastructure, and solid performance improvements through TDM/LDS optimizations. - Collaboration and maintainability: increased test coverage, updated AOT cache handling, and alignment with FlyDSL updates to keep MoE kernels current with upstream changes. Technologies and skills demonstrated: - Kernel-level MoE optimization and FlyDSL integration, including 2-stage GEMM, WMMA/non-WMMA path handling, FP4/FP8/ A8W4 quantization formats, and per-1x32 quantization. - AOT compilation improvements and test harness enhancements for grouped MoE, including AOT cache checks and shared no-regression testing across main and grouped MoE paths. - Performance engineering with TDM gather, LDS caching, carry-safe TDM APIs, and barrier/coherence fixes for gfx1250. - Cross-repo collaboration and change management, validated with extensive end-to-end tests and stability improvements.