이진욱님의 블로그

https://arxiv.org/abs/2404.12253

https://arxiv.org/pdf/2404.19733

https://arxiv.org/pdf/2404.10346

https://arxiv.org/pdf/2312.06585Generate (E-step):  모델을 사용해 샘플링후 필터링(binary feedback) Improve (M-step): 이 샘플을 통해 finetuning이 과정을 몇번 반복 We make some modifications to ReST (detailed in Section 3), and call our approach ReST𝐸𝑀. We show that ReST𝐸𝑀 can be viewed as applying expectation-maximization for reinforcement learning models fine-tuned on model-generated synthetic data exhibit remarkably ..

https://arxiv.org/pdf/2407.02446

https://arxiv.org/pdf/2408.02666Model-based evaluation is at the heart of successful model development – as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an..

https://arxiv.org/pdf/2408.07199기존의 PRM을 각 step의 correctness를 확인하게 위해 쓰인것과 달리, critic 모델을 통해 process 감독을 하고 가능한 에이젼트 action에 순위를 매김자세히policy(LLM actor)이 K개의 action을 제시 policy(LLM critic , 동일한 Base LLM)이 제안된 action에 순위를 매김순위는 expansion(MCTS) 후 노드 선(MCTS)을 가이드 하는데 사용되고, DPO pair를 구성하는데 사용됨  We combine a planning and reasoning agent with MCTS inference-time search and AI self-critique for self-supe..

https://arxiv.org/pdf/2405.16265

https://www.arxiv.org/pdf/2408.08152

수학 기출문제집을 푼다(STaR), 모르는 것을 친구들과 (gain confidence) 선생님게 지도받는다(evaluator) 수학공식과 같은 hint가 될수있는 것을 스스로 찾는다(search) 채점하고 풀이과정을 본다(self-critique, reward model via answer matching)  challenge verify the correctness for each intermediate step and the final answersc-CoT는 majority votingRAP self-rewarding본문에서는 near-random self-rewarding M*(mindSTaR, 2024)에서는 이를 https://arxiv.org/abs/2405.16265 MindStar: E..