EvolvingLMMs-Lab/lmms-eval
Jun 2026 – Jun 2026
1 Month active
Languages Used
Python
Technical Skills
AI Safety EvaluationData ProcessingMachine LearningPython Development
Youngwan Lee
PROFILE
Youngwan Lee
Developed a comprehensive safety evaluation benchmark for vision-language models in the EvolvingLMMs-Lab/lmms-eval repository, focusing on the HoliSafe benchmark suite. Built an end-to-end evaluation pipeline in Python, integrating dataset splits and supporting both refusal-phrase string matching and multimodal LLM-as-judge tasks. Designed statistical metrics such as rejection rates and attack success rates, validating results against official baselines to ensure measurement fidelity. Enabled environment-driven configuration for LLM evaluation, with robust fallback to string matching when needed. Emphasized code quality and reproducibility through pre-commit formatting and detailed documentation, providing a scalable, auditable framework for AI safety evaluation and risk assessment.
Overall Statistics
Feature vs Bugs
100%Features
Repository Contributions
1Total
Bugs
0
Commits
1
Features
1
Lines of code
565
Activity Months1
Your Network
95 peopleWork History
June 2026
1 Commits • 1 Features
Jun 1, 2026June 2026 monthly summary for EvolvingLMMs-Lab/lmms-eval. Focused on delivering a comprehensive safety-evaluation benchmark for vision-language models, establishing a robust evaluation pipeline and validating results to enable safer, more reliable deployments. Key features delivered: - HoliSafe Benchmark for Vision-Language Safety Evaluation added with two tasks: refusal-phrase string matching and multimodal LLM-as-judge. Implemented end-to-end evaluation, including metrics for rejection rates and per-type attack success rates, with a dedicated 4031-sample test split from etri-vilab/holisafe-bench. The task suite is designed to assess model behavior across safe/unsafe image-query pairs. - Tight integration with the lmms_eval pipeline, enabling environment-driven LLM judge configuration via API_TYPE, MODEL_VERSION, OPENAI_API_KEY, and OPENAI_API_URL. Implemented a robust fallback to string matching when LLM evaluation is unavailable or fails. - Validation and parity checks against established baselines: reproduced string-matching results exactly with the official standalone script and achieved near parity for the LLM-judge path on a stratified subset (approximately 99% of GPT-4o-like calls). - Documentation and reproducibility: commit 87125070fe458ff7484d0b0ee9d8a52aa0cf7af5; pre-commit formatting (black, isort) ensured; test splits and evaluation protocol clearly logged for auditability. Major bugs fixed: - None reported this month; primary focus on feature development and measurement fidelity. Minor quality improvements captured by pre-commit checks and formatting gates. Overall impact and accomplishments: - Provides a scalable, auditable safety evaluation framework for vision-language models, enabling product and risk teams to quantify safety behavior across a broad spectrum of inputs. This directly supports safer deployments and faster iteration cycles for safe AI systems. Technologies/skills demonstrated: - Python-based evaluation framework, multimodal evaluation, dataset integration, and environment-configurable LLM interactions (including OpenAI-compatible endpoints). - Statistical metrics design (RR, per-type ASR, avg_ASR), result validation against official baselines, and robust exception handling with fallback paths. - Code quality practices (pre-commit), repository hygiene, and traceable commits for auditability.
1 Commits • 1 Features
Jun 1, 2026June 2026 monthly summary for EvolvingLMMs-Lab/lmms-eval. Focused on delivering a comprehensive safety-evaluation benchmark for vision-language models, establishing a robust evaluation pipeline and validating results to enable safer, more reliable deployments. Key features delivered: - HoliSafe Benchmark for Vision-Language Safety Evaluation added with two tasks: refusal-phrase string matching and multimodal LLM-as-judge. Implemented end-to-end evaluation, including metrics for rejection rates and per-type attack success rates, with a dedicated 4031-sample test split from etri-vilab/holisafe-bench. The task suite is designed to assess model behavior across safe/unsafe image-query pairs. - Tight integration with the lmms_eval pipeline, enabling environment-driven LLM judge configuration via API_TYPE, MODEL_VERSION, OPENAI_API_KEY, and OPENAI_API_URL. Implemented a robust fallback to string matching when LLM evaluation is unavailable or fails. - Validation and parity checks against established baselines: reproduced string-matching results exactly with the official standalone script and achieved near parity for the LLM-judge path on a stratified subset (approximately 99% of GPT-4o-like calls). - Documentation and reproducibility: commit 87125070fe458ff7484d0b0ee9d8a52aa0cf7af5; pre-commit formatting (black, isort) ensured; test splits and evaluation protocol clearly logged for auditability. Major bugs fixed: - None reported this month; primary focus on feature development and measurement fidelity. Minor quality improvements captured by pre-commit checks and formatting gates. Overall impact and accomplishments: - Provides a scalable, auditable safety evaluation framework for vision-language models, enabling product and risk teams to quantify safety behavior across a broad spectrum of inputs. This directly supports safer deployments and faster iteration cycles for safe AI systems. Technologies/skills demonstrated: - Python-based evaluation framework, multimodal evaluation, dataset integration, and environment-configurable LLM interactions (including OpenAI-compatible endpoints). - Statistical metrics design (RR, per-type ASR, avg_ASR), result validation against official baselines, and robust exception handling with fallback paths. - Code quality practices (pre-commit), repository hygiene, and traceable commits for auditability.
June 2026
Activity
Loading activity data...
Quality Metrics
Correctness100.0%
Maintainability80.0%
Architecture100.0%
Performance80.0%
AI Usage80.0%
Skills & Technologies
Programming Languages
Python
Technical Skills
AI Safety EvaluationData ProcessingMachine LearningPython Development
Repositories Contributed To
Overview of all repositories you've contributed to across your timeline