EvolvingLMMs-Lab/lmms-eval
Oct 2024 – Feb 2026
16 Months active
Languages Used
PythonMarkdownYAMLShellTOMLBashNone
Technical Skills
API IntegrationBackend DevelopmentCloud ServicesConfiguration ManagementDocumentationMultimodal Evaluation
Bo Li
PROFILE
Bo Li
Luodian Liu developed and maintained the lmms-eval repository, building a robust multimodal evaluation platform for large language and vision models. Over 16 months, he engineered features such as unified benchmarking workflows, distributed evaluation, and support for audio, video, and vision tasks. Using Python, YAML, and shell scripting, he integrated APIs for OpenAI, Azure, and VLLM, and implemented automated code review and CI/CD pipelines. His work included model integration, dataset management, and performance optimizations, with careful attention to documentation, error handling, and internationalization. The resulting system improved evaluation reliability, scalability, and reproducibility for research and product teams.
Overall Statistics
Feature vs Bugs
77%Features
Repository Contributions
99Total
Bugs
15
Commits
99
Features
49
Lines of code
51,340
Activity Months16
Your Network
76 people
Work History
February 2026
2 Commits • 1 Features
Feb 1, 2026February 2026 monthly summary for EvolvingLMMs-Lab/lmms-eval: delivered multimodal evaluation enhancements including new benchmarks, an HTTP evaluation server, and VLMEvalKit-compatible task variants, improving benchmarking accuracy and reproducibility for Qwen models. Updated documentation to reflect enhancements, enabling faster adoption and reproducible results. These changes enhance business value by accelerating model validation and iteration across teams.
2 Commits • 1 Features
Feb 1, 2026February 2026 monthly summary for EvolvingLMMs-Lab/lmms-eval: delivered multimodal evaluation enhancements including new benchmarks, an HTTP evaluation server, and VLMEvalKit-compatible task variants, improving benchmarking accuracy and reproducibility for Qwen models. Updated documentation to reflect enhancements, enabling faster adoption and reproducible results. These changes enhance business value by accelerating model validation and iteration across teams.
February 2026
January 2026
25 Commits • 12 Features
Jan 1, 2026January 2026 focused on expanding evaluation capabilities, enhancing global accessibility, and tightening stability for lmms-eval. We added eight new benchmarks (BabyVision, MMVP with GT corrections, RealUnify, Spatial457, AuxSolidMath, IllusionBench, Uni-MMMU, Geometry3K), strengthened multi-language documentation across 18 languages, and implemented key stability fixes (memory-leak prevention in video loaders, device-agnostic GPU handling) while streamlining CI with gitignore housekeeping and removal of automated Claude reviews. These efforts deliver broader, more reliable evaluation pipelines, reduced global friction, and faster, stable experimentation for product teams.
January 2026
25 Commits • 12 Features
Jan 1, 2026January 2026 focused on expanding evaluation capabilities, enhancing global accessibility, and tightening stability for lmms-eval. We added eight new benchmarks (BabyVision, MMVP with GT corrections, RealUnify, Spatial457, AuxSolidMath, IllusionBench, Uni-MMMU, Geometry3K), strengthened multi-language documentation across 18 languages, and implemented key stability fixes (memory-leak prevention in video loaders, device-agnostic GPU handling) while streamlining CI with gitignore housekeeping and removal of automated Claude reviews. These efforts deliver broader, more reliable evaluation pipelines, reduced global friction, and faster, stable experimentation for product teams.
December 2025
5 Commits • 3 Features
Dec 1, 2025December 2025: Delivered three primary outcomes for EvolvingLMMs-Lab/lmms-eval across the codebase: 1) Automated PR Code Review System built on Claude Actions with multi-agent scoring to provide fast, structured feedback on PRs and issues; 2) Logging Pipeline Enhancement that filters multimodal content to preserve scalar metadata, improving dataset traceability and preventing serialization issues; 3) Documentation and Visualization Enhancements including a comprehensive tasks/models overview, summary statistics, and robust spatial visualization utilities with improved exception handling, logging consistency, and type hints.
5 Commits • 3 Features
Dec 1, 2025December 2025: Delivered three primary outcomes for EvolvingLMMs-Lab/lmms-eval across the codebase: 1) Automated PR Code Review System built on Claude Actions with multi-agent scoring to provide fast, structured feedback on PRs and issues; 2) Logging Pipeline Enhancement that filters multimodal content to preserve scalar metadata, improving dataset traceability and preventing serialization issues; 3) Documentation and Visualization Enhancements including a comprehensive tasks/models overview, summary statistics, and robust spatial visualization utilities with improved exception handling, logging consistency, and type hints.
December 2025
October 2025
4 Commits • 1 Features
Oct 1, 2025In 2025-10, LMMS-Eval reached a major milestone with v0.5 Release: Multimodal Expansion. The release introduces audio evaluation capabilities, response caching for efficiency, and expanded support for five new multimodal models, with 50+ benchmarks across audio, vision, coding, and STEM. It integrates with the Model Context Protocol (MCP) and improves async OpenAI integration. Documentation updates accompany the release, including Qwen3-VL evaluation scripts for SGLang and vLLM backends, and a version bump to 0.5.0 with refined dependencies. These changes deliver faster, more scalable model evaluation and richer benchmarking data, enabling better research decisions and product decisions.
October 2025
4 Commits • 1 Features
Oct 1, 2025In 2025-10, LMMS-Eval reached a major milestone with v0.5 Release: Multimodal Expansion. The release introduces audio evaluation capabilities, response caching for efficiency, and expanded support for five new multimodal models, with 50+ benchmarks across audio, vision, coding, and STEM. It integrates with the Model Context Protocol (MCP) and improves async OpenAI integration. Documentation updates accompany the release, including Qwen3-VL evaluation scripts for SGLang and vLLM backends, and a version bump to 0.5.0 with refined dependencies. These changes deliver faster, more scalable model evaluation and richer benchmarking data, enabling better research decisions and product decisions.
September 2025
6 Commits • 4 Features
Sep 1, 2025September 2025: The lmms-eval team delivered reliability and usability enhancements across Thyme, Gemma3, and VQA components, reinforcing production readiness and developer experience. Key updates include hardening thyme.sh (shebang, strict mode, adjustable HF_HOME), enhanced Thyme image handling with robust multimodal processing and QA fallbacks, Gemma3 loading improvements ensuring .generate() availability, and VQA prompt type hints/docs to reduce integration errors. Dev tooling improvements and bug fixes included robust write_out handling with deprecation guidance. These changes reduce runtime errors, improve end-to-end workflows, and create a stronger foundation for upcoming features.
6 Commits • 4 Features
Sep 1, 2025September 2025: The lmms-eval team delivered reliability and usability enhancements across Thyme, Gemma3, and VQA components, reinforcing production readiness and developer experience. Key updates include hardening thyme.sh (shebang, strict mode, adjustable HF_HOME), enhanced Thyme image handling with robust multimodal processing and QA fallbacks, Gemma3 loading improvements ensuring .generate() availability, and VQA prompt type hints/docs to reduce integration errors. Dev tooling improvements and bug fixes included robust write_out handling with deprecation guidance. These changes reduce runtime errors, improve end-to-end workflows, and create a stronger foundation for upcoming features.
September 2025
August 2025
8 Commits • 4 Features
Aug 1, 2025August 2025 monthly summary for EvolvingLMMs-Lab/lmms-eval: Delivered robust video sampling controls, broadened API support for OpenAI and Azure, enhanced audio input handling and encoding, and updated documentation to improve onboarding and maintainability. Fixed a critical local cache race condition, delivering more reliable continual processing. These efforts reduce risk, expand deployment options, and accelerate evaluation workflows, underscoring the team's ability to ship reliable features with strong test coverage and clear docs.
August 2025
8 Commits • 4 Features
Aug 1, 2025August 2025 monthly summary for EvolvingLMMs-Lab/lmms-eval: Delivered robust video sampling controls, broadened API support for OpenAI and Azure, enhanced audio input handling and encoding, and updated documentation to improve onboarding and maintainability. Fixed a critical local cache race condition, delivering more reliable continual processing. These efforts reduce risk, expand deployment options, and accelerate evaluation workflows, underscoring the team's ability to ship reliable features with strong test coverage and clear docs.
July 2025
18 Commits • 7 Features
Jul 1, 2025July 2025 performance summary for EvolvingLMMs-Lab/lmms-eval focused on delivering a scalable, reliable evaluation platform, strengthening model integration, and improving collaboration and documentation. Key outcomes include a major LMMS-Eval 0.4 release with unified multimodal evaluation, multi-node distributed evaluation, and a standardized judge interface; enabling reproducible benchmarks and faster decision-making for product and research teams.
18 Commits • 7 Features
Jul 1, 2025July 2025 performance summary for EvolvingLMMs-Lab/lmms-eval focused on delivering a scalable, reliable evaluation platform, strengthening model integration, and improving collaboration and documentation. Key outcomes include a major LMMS-Eval 0.4 release with unified multimodal evaluation, multi-node distributed evaluation, and a standardized judge interface; enabling reproducible benchmarks and faster decision-making for product and research teams.
July 2025
June 2025
3 Commits • 2 Features
Jun 1, 2025June 2025 monthly summary for EvolvingLMMs-Lab/lmms-eval. Delivered enhancements to VideoMathQA evaluation task configuration and code organization, hardened distributed context handling, and expanded project documentation. These changes improve evaluation reliability, configurability, and developer onboarding.
June 2025
3 Commits • 2 Features
Jun 1, 2025June 2025 monthly summary for EvolvingLMMs-Lab/lmms-eval. Delivered enhancements to VideoMathQA evaluation task configuration and code organization, hardened distributed context handling, and expanded project documentation. These changes improve evaluation reliability, configurability, and developer onboarding.
May 2025
4 Commits • 3 Features
May 1, 2025This monthly summary covers May 2025 for the EvolvingLMMs-Lab lmms-eval workstream, emphasizing business value from reliability improvements, benchmarking expansion, and tooling enhancements. Key workflow improvements to the lmms-eval evaluation process were delivered, alongside a broader benchmarking slate and stricter dependency management to support newer datasets and better developer tooling. A CLI reliability fix ensures accurate task visibility, and improvements to model initialization and configuration enable flexible attention implementations. Overall, the month delivered tangible gains in evaluation reliability, reproducibility, and extensibility, helping teams ship faster with fewer integration issues.
4 Commits • 3 Features
May 1, 2025This monthly summary covers May 2025 for the EvolvingLMMs-Lab lmms-eval workstream, emphasizing business value from reliability improvements, benchmarking expansion, and tooling enhancements. Key workflow improvements to the lmms-eval evaluation process were delivered, alongside a broader benchmarking slate and stricter dependency management to support newer datasets and better developer tooling. A CLI reliability fix ensures accurate task visibility, and improvements to model initialization and configuration enable flexible attention implementations. Overall, the month delivered tangible gains in evaluation reliability, reproducibility, and extensibility, helping teams ship faster with fewer integration issues.
May 2025
April 2025
4 Commits • 3 Features
Apr 1, 2025April 2025 monthly summary for EvolvingLMMs-Lab/lmms-eval focused on delivering a more flexible multimodal generation workflow, widening model compatibility, and expanding documentation and evaluation tooling. Key features implemented include enhanced generation parameters and defaults for multimodal models (alignment with VoRA defaults, system prompts, interleaved visuals, and maximum sequence length) and broader compatibility across models, plus a comprehensive suite of evaluation scripts and improved visual data handling.
April 2025
4 Commits • 3 Features
Apr 1, 2025April 2025 monthly summary for EvolvingLMMs-Lab/lmms-eval focused on delivering a more flexible multimodal generation workflow, widening model compatibility, and expanding documentation and evaluation tooling. Key features implemented include enhanced generation parameters and defaults for multimodal models (alignment with VoRA defaults, system prompts, interleaved visuals, and maximum sequence length) and broader compatibility across models, plus a comprehensive suite of evaluation scripts and improved visual data handling.
March 2025
4 Commits • 3 Features
Mar 1, 2025March 2025 performance summary for EvolvingLMMs-Lab/lmms-eval focused on expanding multimodal reasoning evaluation, streamlining data collection, and strengthening automated judging metrics. Delivered: MME-COT Multimodal Reasoning Task Integration with YAML configurations supporting direct and reasoning modes; a document processing utility for visual/text processing and prompt generation with mode-specific postfixes; enabled integration of a multimodal reasoning evaluation task. Also launched Visual Reasoning Collection tasks (K12, OlympiadBench) and implemented prompt construction/logging improvements, including refactoring GPT model version retrieval to use environment variables for deployment flexibility and enhanced file tracking. Introduced LLM-based Evaluation Metric llm_as_judge_eval for MME-CoT and MME COT, integrating GPT-4o reasoning for judging solutions, updating configs and adding prompt/API utilities; simplified aggregation to mean where applicable. These changes broaden evaluation coverage, improve reliability and reproducibility, and enable faster iteration and business insights.
4 Commits • 3 Features
Mar 1, 2025March 2025 performance summary for EvolvingLMMs-Lab/lmms-eval focused on expanding multimodal reasoning evaluation, streamlining data collection, and strengthening automated judging metrics. Delivered: MME-COT Multimodal Reasoning Task Integration with YAML configurations supporting direct and reasoning modes; a document processing utility for visual/text processing and prompt generation with mode-specific postfixes; enabled integration of a multimodal reasoning evaluation task. Also launched Visual Reasoning Collection tasks (K12, OlympiadBench) and implemented prompt construction/logging improvements, including refactoring GPT model version retrieval to use environment variables for deployment flexibility and enhanced file tracking. Introduced LLM-based Evaluation Metric llm_as_judge_eval for MME-CoT and MME COT, integrating GPT-4o reasoning for judging solutions, updating configs and adding prompt/API utilities; simplified aggregation to mean where applicable. These changes broaden evaluation coverage, improve reliability and reproducibility, and enable faster iteration and business insights.
March 2025
February 2025
6 Commits • 2 Features
Feb 1, 2025February 2025: Delivered substantial enhancements to lmms-eval focused on evaluation, model integration, and documentation. Key features include multi-sampling and filtering during evaluation, a loguru-based logging overhaul, multimodal task handling improvements, MathVision dataset utilities, VLLM-compatible model integration, and an OpenAI-compatible API interface, with related metric/config updates. Documentation and release notes were refreshed to reflect accelerated evaluation paths and external integrations. Impact: faster, more scalable evaluation; broader model interoperability; clearer release history; and stronger business value through improved iteration speed and interoperability. Technologies: Python, loguru, VLLM, OpenAI-compatible interfaces, MathVision, multimodal data handling.
February 2025
6 Commits • 2 Features
Feb 1, 2025February 2025: Delivered substantial enhancements to lmms-eval focused on evaluation, model integration, and documentation. Key features include multi-sampling and filtering during evaluation, a loguru-based logging overhaul, multimodal task handling improvements, MathVision dataset utilities, VLLM-compatible model integration, and an OpenAI-compatible API interface, with related metric/config updates. Documentation and release notes were refreshed to reflect accelerated evaluation paths and external integrations. Impact: faster, more scalable evaluation; broader model interoperability; clearer release history; and stronger business value through improved iteration speed and interoperability. Technologies: Python, loguru, VLLM, OpenAI-compatible interfaces, MathVision, multimodal data handling.
January 2025
1 Commits • 1 Features
Jan 1, 2025January 2025 — EvolvingLMMs-Lab/lmms-eval: Delivered a Megabench Evaluation Pipeline Refactor and Performance Enhancements. This work improves readability and runtime performance of the evaluation pipeline, enhances traceability, and strengthens maintainability for future scaling. Key changes include reordering imports for consistency, optimizing loops and conditionals to reduce evaluation time, and adding timestamps to submission file names to improve traceability. Ensured Python 3.9 compatibility and reinforced the pipeline’s overall structure to support reliable, repeatable benchmarks. Commit reference: 50ed3ce68b08154108a17d1459db4bf282302107 ([WIP] style(megabench): improve code formatting and import ordering (#497)).
1 Commits • 1 Features
Jan 1, 2025January 2025 — EvolvingLMMs-Lab/lmms-eval: Delivered a Megabench Evaluation Pipeline Refactor and Performance Enhancements. This work improves readability and runtime performance of the evaluation pipeline, enhances traceability, and strengthens maintainability for future scaling. Key changes include reordering imports for consistency, optimizing loops and conditionals to reduce evaluation time, and adding timestamps to submission file names to improve traceability. Ensured Python 3.9 compatibility and reinforced the pipeline’s overall structure to support reliable, repeatable benchmarks. Commit reference: 50ed3ce68b08154108a17d1459db4bf282302107 ([WIP] style(megabench): improve code formatting and import ordering (#497)).
January 2025
December 2024
5 Commits • 1 Features
Dec 1, 2024Month: 2024-12. Focused on improving documentation clarity and reliability of the lmms-eval workflow. Delivered consolidated documentation updates for lmms-eval 0.3, refreshed README visuals, and announced the MME-Survey paper to raise awareness of features and research contributions. Implemented a robust fix for the score calculation utility to gracefully handle empty or insufficient data, reducing runtime errors and ensuring stable results. These changes improve user onboarding, maintainability, and trust in evaluation results, enabling smoother adoption by researchers and teams.
December 2024
5 Commits • 1 Features
Dec 1, 2024Month: 2024-12. Focused on improving documentation clarity and reliability of the lmms-eval workflow. Delivered consolidated documentation updates for lmms-eval 0.3, refreshed README visuals, and announced the MME-Survey paper to raise awareness of features and research contributions. Implemented a robust fix for the score calculation utility to gracefully handle empty or insufficient data, reducing runtime errors and ensuring stable results. These changes improve user onboarding, maintainability, and trust in evaluation results, enabling smoother adoption by researchers and teams.
November 2024
3 Commits • 1 Features
Nov 1, 2024November 2024 monthly summary for EvolvingLMMs-Lab/lmms-eval: Delivered a new multimodal evaluation task integration via MIA-Bench, enhanced configuration, and improved documentation. Strengthened evaluation capabilities and contributor visibility, driving reproducibility and onboarding.
3 Commits • 1 Features
Nov 1, 2024November 2024 monthly summary for EvolvingLMMs-Lab/lmms-eval: Delivered a new multimodal evaluation task integration via MIA-Bench, enhanced configuration, and improved documentation. Strengthened evaluation capabilities and contributor visibility, driving reproducibility and onboarding.
November 2024
October 2024
1 Commits • 1 Features
Oct 1, 2024For Oct 2024, lmms-eval delivered Azure OpenAI API support and backend flexibility, enabling evaluation with either Azure or OpenAI LLM backends. Dataset loading was updated to support local disk sources, and conditional logic was added to handle Azure and OpenAI endpoints and payload structures across multiple evaluation utilities, providing a seamless switch between backends. This work enhances deployment flexibility, reduces vendor lock-in, and improves evaluation throughput and reproducibility across environments.
October 2024
1 Commits • 1 Features
Oct 1, 2024For Oct 2024, lmms-eval delivered Azure OpenAI API support and backend flexibility, enabling evaluation with either Azure or OpenAI LLM backends. Dataset loading was updated to support local disk sources, and conditional logic was added to handle Azure and OpenAI endpoints and payload structures across multiple evaluation utilities, providing a seamless switch between backends. This work enhances deployment flexibility, reduces vendor lock-in, and improves evaluation throughput and reproducibility across environments.
Activity
Loading activity data...
Quality Metrics
Correctness93.4%
Maintainability90.0%
Architecture90.4%
Performance85.0%
AI Usage38.8%
Skills & Technologies
Programming Languages
BashMarkdownNonePythonShellTOMLYAML
Technical Skills
AI DevelopmentAI EvaluationAI IntegrationAI evaluationAPI DevelopmentAPI IntegrationArgument ParsingAudio ProcessingAutomationBackend DevelopmentBenchmark DevelopmentBenchmarkingBug FixBug FixingBuild System Configuration
Repositories Contributed To
Overview of all repositories you've contributed to across your timeline