이진욱님의 블로그

https://arxiv.org/abs/2410.04343

https://arxiv.org/pdf/2410.10934

https://arxiv.org/pdf/2410.12381

https://arxiv.org/abs/2410.07112Current benchmarks for assessing vision-language models (VLMs) often focus on their perception or problem-solving capabilities and neglect other critical aspects such as fairness, multilinguality, or toxicity.Furthermore, they differ in their evaluation procedures and the scope of the evaluation, making it diff i cult to compare models. To address these issues, we..

https://arxiv.org/pdf/2406.11069Recent breakthroughs in vision-language models (VLMs) emphasize the necessity of benchmarking human preferences in real-world multimodal interactions. To address this gap, we launched WILDVISION-ARENA (WV-ARENA), an online platform that collects human preferences to evaluate VLMs. We curated WVBENCH by selecting 500 high-quality samples from 8,000 user submissions..

https://arxiv.org/pdf/2408.00765https://huggingface.co/spaces/whyu/MM-Vet-v2_Evaluator MM-Vet v2 Evaluator - a Hugging Face Space by whyu huggingface.co MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluation. MM-Vet assesses six core vision-language (VL) capabilities: r..

https://arxiv.org/abs/2408.11791

https://arxiv.org/abs/2402.04788 MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language BenchmarkMultimodal Large Language Models (MLLMs) have gained significant attention recently, showing remarkable potential in artificial general intelligence. However, assessing the utility of MLLMs presents considerable challenges, primarily due to the absence ofarxiv.orgMultimodal Large L..

https://arxiv.org/abs/2311.01361Automatically evaluating vision-language tasks is challenging, especially when it comes to reflecting human judgments due to limitations in accounting for finegrained details. Although GPT-4V has shown promising results in various multimodal tasks, leveraging GPT-4V as a generalist evaluator for these tasks has not yet been systematically explored. We comprehensiv..

https://arxiv.org/abs/2312.10665 Silkie: Preference Distillation for Large Visual Language ModelsThis paper explores preference distillation for large vision language models (LVLMs), improving their ability to generate helpful and faithful responses anchoring the visual context. We first build a vision-language feedback (VLFeedback) dataset utilizingarxiv.orgThis paper explores preference distil..