이진욱님의 블로그

https://arxiv.org/pdf/2410.02740

https://arxiv.org/pdf/2312.10240Recent Text-to-Image (T2I) generation models such as Stable Diffusion and Imagen have made significant progress in generating high-resolution images based on text descriptions. However, many generated images still suffer from issues such as artifacts/implausibility, misalignment with text descriptions, and low aesthetic quality. Inspired by the success of Reinforc..

https://llava-vl.github.io/blog/2024-09-30-llava-video/ LLaVA-Video: Video Instruction Tuning with Synthetic DataWe fine-tune LLaVA-OneVision (SI) on the joint dataset of video and image data. Specifically, we added video data from the LLaVA-Video-178K dataset and four public datasets: ActivityNet-QA, NExT-QA, PerceptionTest, and LLaVA-Hound-255K, focusing on videosllava-vl.github.iohttps://arxi..

https://arxiv.org/pdf/2410.02712We introduce LLaVA-Critic, the first open-source large multimodal model (LMM) designed as a generalist evaluator to assess performance across a wide range of multimodal tasks. LLaVA-Critic is trained using a high-quality critic instructionfollowing dataset that incorporates diverse evaluation criteria and scenarios. Our experiments demonstrate the model’s effectiv..

https://arxiv.org/pdf/2308.08998요약 : Reward model + SFT 핵심 Grow stage에서 데이터셋을 샘플링 ~ current policy modelreward model을 통해 filtered improve 파트 NLL loss  Figure 1 모식도 pseudo - 알고리즘 Reward 부여 방식 : 토큰 뒤에 scalar reward로

https://arxiv.org/pdf/2401.06591Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For ..

https://arxiv.org/pdf/2405.14622요약reward 부여 방식: self-generated instruction-following score( calculated using the language decoder of the LVLM , 이거 하나로만 안되는 이유 : modality misalignment, potentially overlooking visual input information ), + the image-response relevance score, R^I (s).( We leverage CLIP-score [17] for this calculation ) 3 Calibrated Self-Rewarding Vision Language Models To address t..

https://arxiv.org/pdf/2404.01258v2 Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM) However, in tasks involving video instruction following, providing informative feedback, especially for detecting hallucinations in generated responses, remains a significant challenge Previous..

https://arxiv.org/pdf/2401.00789

https://arxiv.org/pdf/2304.08485Instruction tuning large language models (LLMs) using machine-generated instruction-following data has been shown to improve zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on suc..