https://arxiv.org/pdf/2411.16594Assessment and evaluation have long been critical challenges in artificial intelligence (AI) and natural language processing (NLP).
However, traditional methods, whether matching-based or embedding-based, often fall short of judging subtle attributes and delivering satisfactory results.
Recent advancements in Large Language Models (LLMs) inspire the "LLM-as-ajudge" paradigm, where LLMs are leveraged to perform scoring, ranking, or selection across various tasks and applications.
This paper provides a comprehensive survey of LLM-based judgment and assessment, offering an in-depth overview to advance this emerging field.
We begin by giving detailed definitions from both input and output perspectives.
Then we introduce a comprehensive taxonomy to explore LLM-as-a-judge from three dimensions: what to judge, how to judge and where to judge.
Finally, we compile benchmarks for evaluating LLM-as-a-judge and highlight key challenges and promising directions, aiming to provide valuable insights and inspire future research in this promising research area12.
3 Attribute In this section, we categorize current research in LLM-as-a-judge from attribute perspectives.
Figure 3 gives an overview summarization of what aspects can be assessed by the judge LLMs.
3.1 Helpfulness Modern state-of-the-art LLMs undergo instruction tuning and alignment processes, enabling them to follow user instructions and respond effectively.
This alignment step relies on large quantities of helpful and harmless data, typically gathered as human preference data, which is then used in reinforcement learning for alignment training.
Given the high cost of obtaining such alignment data, recent efforts have explored using LLMs to label helpfulness, as well as to generate or evaluate alignment data (Bai et al., 2022).
Authors in (Guo et al., 2024) use an LLM in an online manner to obtain preferences for direct alignment of another LLM.
Some recent work shows that helpfulness feedback from AI, i.e., LLMs are comparable to human feedback (Lee et al., 2023).
There have also been successful LLMs (Zhu et al., 2024a) with superior performance that have been fine-tuned with AI feedback data thus demonstrating the feasibility and usefulness of this method.
Alongside these works in alignment, general purposed frameworks leveraging LLMs as evaluators are also crucial in evaluating the helpfulness of candidate responses (Zheng et al., 2023; Lin et al., 2023; Li et al., 2024e).
3.2 Harmlessness Evaluating harmlessness in text data is important for both content moderation and for creating or curating synthetic datasets.
Given how expensive and time-consuming human labeling efforts are, and inspired by prior work that looks into moral beliefs embedded in LLMs (Scherrer et al., 2023), many recent works have investigated the use of LLMs to evaluate harmlessness.
State-of-the-art LLMs are capable of being used effectively for content moderation, either off the shelf when guided with some policy guidelines3 , or when fine-tuned on safe/unsafe data (Inan et al., 2023; Zhang et al., 2024e) 4.
Ye et al. (2024b) explore the feasibility of using LLMs to evaluate harmlessness in a finegrained manner, among other attributes, and find that proprietary models perform much better than open source ones.
Wang et al. (2024i) use OpenAI's GPT-4 to evaluate harmlessness and further compare the performance with that of a smaller pretrained language model fine-tuned for this specific task.
Additionally, Bai et al. (2022) use principles to guide LLMs to make harmlessness evaluations for alignment purposes, a paradigm they call Constitutional AI.
(Phute et al., 2023) use the same LLM to evaluate its responses as harmful, and provide insights and best practices for GPT-3.5 and Llama-2.
Xie et al. (2024a) perform a comprehensive comparison of several LLMs on a benchmark of LLM safety refusals and find small LLMs are effective safety judges when used in fine-tuned settings.
During inference time, (Li et al.) propose Rewindable Auto-regressive INference (RAIN), which allows LLMs to conduct self-evaluation and rewind for AI safety.
3.3 Reliability Reliability is a crucial attribute for LLMs, enabling them to generate factual and faithful content while also expressing uncertainty or acknowledging gaps in knowledge about certain topics.
Regarding factuality, Wang et al. (2024a) introduce HALU-J, a critique-based hallucination judge that enhances factuality assessment by selecting relevant evidence and providing detailed critiques.
Cheng et al. (2023) design an automated evaluation method using GPT-4 to judge whether a model's output is hallucinated.
Additionally, several works adopt judge LLMs for long-form factuality evaluation.
In the context of dialogues, Luo et al. (2024) collect a large-scale benchmark for automatic dialogue-level hallucination evaluation.
Based on this dataset, they introduce HalluJudge, a specialized judge language model for evaluating hallucinations at the dialogue level.
Min et al. (2023) develop FactScore, a fine-grained method for evaluating the factuality of long-form generation by first splitting content into atomic-level sentences and retrieving relevant corpus from Wikipedia to assess their factuality.
Building on this approach, Wei et al. (2024b) propose equipping judge LLMs with a Google search API to enable more flexible and efficient factuality assessments.
Jing et al. (2024) expand this finegrained reliability evaluation to the multimodal area and introduce FaithScore.
Zhang et al. (2024f) adopt a similar strategy in LLM alignment by creating a synthetic alignment dataset, which involves evaluating and filtering each generated sample using claim extraction and self-judging techniques.
Xie et al. (2024b) train an external critique-based LLM as the judge to provide claim-level claim level factuality feedback in the generation stage to improve response factuality.
For uncertainty judgment, Xu et al. (2024b) present SaySelf, a novel training framework that teaches LLMs to express more fine-grained confidence estimates with selfconsistency prompting and group-based calibration training.
3.4 Relevance
Relevance measures the extent to which generated or retrieved content aligns with the original query.
Traditional relevance assessment methods often rely on keyword matching (Robertson et al., 2009) or semantic similarity (Gao et al., 2021), which struggle to capture subtle differences or nuances in context.
Relevance assessment with judge LLMs has been explored and validated to be a more finegrained and effective manner across various applications (Chiang and Lee, 2023).
In conversation evaluation, Lin and Chen (2023a) first propose to replace the expensive and time-consuming human annotation with LLM judgment in relevance assessment, providing conversation context and generated response for the judge LLM to evaluate.
Similarly, Abbasiantaeb et al. (2024) apply LLM-as-ajudge in conversation search, collaborating with human annotators to address issues related to incomplete relevance judgments.
In retrieval-augmented generation (RAG) scenarios, Li and Qiu (2023) utilize LLMs to determine which historical memory is most relevant for solving the current problem.
Following this concept, Li et al. (2024c) also propose to adopt LLM as a re-ranker to judge and filter out noisy and irrelevant knowledge in a sub-knowledge graph.
Recently, LLM-as-a-judge has also been used in multimodal applications for relevance judgment.
Both Yang and Lin (2024) and Chen et al. (2024a) adopt a multimodal LLM to build the automatic assessment benchmark.
In contrast, Chen et al. (2024e) employ the multimodal reward model to assess the relevance in text-to-image generation.
Besides, LLM-as-a-judge has also been explored in many traditional retrieval applications for relevance assessment, such as search (Thomas et al., 2024), retrieval (Ma et al., 2024) and recommendation (Hou et al., 2024).
3.5 Feasibility The potential of LLMs can be further released by complicated and well-designed reasoning pipelines.
In these agentic LLMs, assessing the feasibility of candidate actions or steps is crucial for the success of planning, reasoning, and decision-making.
While some works leverage metrics or external tools for this feasibility assessment (Huang et al., 2023a; Yuan et al.), many others leverage LLMs themselves to select the most appropriate and reasonable actions to perform.
Hao et al. (2023) first propose to prompt the LLM to do self-evaluation and generate feasibility judgment as a reward signal to perform Monte Carlo Tree Search (MCTS).
Similarly, Yao et al. (2023a) suggest adopting the LLM as the state evaluator for potential step searching in their proposed tree-of-thought (ToT) framework.
Besta et al. (2024) replace the tree structures used in previous studies with graph structures and employ the LLM to assign a score to each thought based on its feasibility or correctness.
In multi-agent collaboration systems, both Liang et al. (2023) and Li et al. (2024b) propose to leverage the judge LLM to select the most feasible and reasonable solutions among multiple candidates' responses.
Besides, there are also works adopt the judge LLMs to perform feasibility assessment in API selection (Zhao et al., 2024b), tool using (Yang et al., 2023) and LLM routing (Ong et al., 2024).
3.6 Overall Quality As previously mentioned, LLM-as-a-judge can be employed to perform multi-aspect and fine-grained assessments across various tasks and applications.
However, in many cases, a general assessment is still required to represent the candidates' overall quality for comparison or ranking purposes.
One straightforward approach to obtain this overall score is to calculate the average of the aspect specific scores (Lin et al., 2023; Lin and Chen, 2023b).
Additionally, several other studies present assessment results for each attribute and prompt the LLM judge to generate an overall quality judgement (Yu et al., 2024; Guo et al., 2024; Zhang et al., 2024a).
Beyond summarizing overall quality from multiple attributes, other studies focus on directly producing an overall judgment.
For instance, in traditional NLP tasks like summarization (Gao et al., 2023; Jain et al., 2023a) and machine translation (Huang et al., 2024; Kocmi and Federmann, 2023), the evaluation dimensions are less diverse compared to more open-ended, long-form generation tasks.
Consequently, LLM-as-a-judge is typically used in these cases to directly prompt the LLM to generate an overall judgment.
4 Methodology In this section, we present commonly adopted methods and tricks for LLM-as-a-judging, splitting them into tuning approaches (Section 4.1) and prompting strategies (Section 4.2).
4.1 Tuning To enhance the judging capabilities of a general LLM, various tuning techniques have been employed in different studies.
In this section, we discuss these tuning approaches for LLM-as-a-judge from two perspectives: data sources (Section 4.1.1) and training methods (Section 4.1.2).
Table 1 collects all the research papers focused on tuning the judge LLMs.
4.1.1 Data Source Manually-labeled Data: To train a judge LLM with human-like criteria, one intuitive method is to collect manually-labeled samples and corresponding judgments.
Many previous works have leveraged and integrated existing sources to build comprehensive datasets for tuning judge LLMs.
Vu et al. (2024) construct a large and diverse collection of over 100 quality assessment tasks, comprising more than 5 million human judgments, curated and standardized using publicly available human evaluations from prior research.
Similarly, Wang et al. (2024h) propose PandaLM and collect a diverse set of human-annotated test data, where all contexts are human-generated and labels align with human preferences.
To enhance the policy's judging ability in alignment data synthesis, Lee et al. (2024) augment the supervised fine-tuning (SFT) dataset with a pairwise judgment task, where the instruction is to select the chosen response from a set of options.
There are also some works that collect datasets for fine-grained judgment feedback.
Xu et al. (2023) introduce InstructScore, an explainable text generation evaluation metric, and curate the MetricInstruct dataset, which covers six text generation tasks and 23 datasets.
Liu et al. (2024a) collect ASPECTINSTRUCT, the first instructiontuning dataset tailored for multi-aspect NLG evaluation, spanning 27 diverse evaluation aspects across 65 tasks.
Yue et al. (2023) first propose the evaluation of attribution and fine-tune judge LLMs with data from related tasks such as question answering, fact-checking, natural language inference, and summarization.
One distinct approach is from Ke et al. (2024), which first prompts GPT-4 to produce the feedback and manually check its generated texts for each user query, revising them if necessary to improve the quality.
Synthetic Feedback: While manually-labeled feedback is high-quality and accurately reflects human judgment preferences, it has limitations in terms of quantity and coverage.
As a result, some researchers have turned to synthetic feedback as a data source for tuning judge LLMs.
One approach in this direction relies on the judge LLMs themselves to generate the synthetic feedback.
For example, Wu et al. (2024) construct pairwise feedback for judgment enhancement by prompting the policy LLMs to evaluate their own judgments.
Wang et al. (2024f) prompt the LLM to generate a "noisy" version of the original instruction and use the corresponding response to this corrupted instruction as the inferior response.
Wang et al. (2024a) prompt GPT-4-Turbo to generate multiple pieces of evidence based on the original evidence for each instance, categorizing them into completely irrelevant evidence, partially irrelevant evidence and highly related evidence to train a hallucination judgment LLMs.
Park et al. (2024) build OFFSETBIAS, a pairwise preference dataset that leverages GPT-4 to generate bad, off-topic and erroneous responses and perform difficulty filtering.
For safety judging, Xie et al. (2024a) adopt GPT-4 as the classifier to map each data point to a predefined safety category to train an automated evaluator.
Different from previous works, Li et al. (2024e) adopt GPT-4 to synthesize both pairwise and pointwise data to train a generative judge LLM.
For pointwise data, they adopt a "divide-andconquer" strategy, where two critiques are collected from GPT-4 for a single response, combined into a more comprehensive critique, and a final rating is provided.
Following them, Kim et al. (2024) use GPT-4 to augment the preference learning dataset with detailed human evaluation criteria and verbal feedback.
In the multi-modal domain, Xiong et al. (2024b) propose LLaVA-Critic and adopt GPT-4o to generate reasons behind given scores or preference judgments for training data construction.
Moreover, By harnessing both explicit human instruction and the implicit knowledge of GPT4, Xu et al. (2023) fine-tune a judge LLM based on LLaMA, producing both a score for generated text and a human-readable diagnostic report.
Zhu et al. (2023) introduce JudgeLM and propose a comprehensive, large-scale, high-quality dataset containing task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges.
4.1.2 Tuning Techniques Supervised Fine-tuning: Supervised fine-tuning (SFT) is the most commonly used approach to facilitate the judge LLMs to learn from pairwise (Wang et al., 2024h; Li et al., 2024e; Wang et al., 2023b; Zhu et al., 2023) or pointwise (Xie et al., 2024a; Wang et al., 2023b; Yue et al., 2023) judgment data.
Among many works that adopted SFT, Vu et al. (2024) propose a supervised multitask training to tune their Foundational Large Autorater Models (FLAMe) across multiple mixed datasets of various tasks.
To equip the judge LLM with both pairwise and pointwise judging capabilities, Kim et al. (2024) novelly propose joint training and weight merging approaches during the tuning stage and find the latter does not improve evaluation performances in the majority of cases.
To obtain ajudge model that can not only generate responses but also compare pairwise preferences, Lee et al. (2024) devise Judge-augmented Supervised Finetuning (JSFT) with an augmented preference learning dataset.
During the training phase, Ke et al. (2024) enhance their model by adding simplified prompts to distinguish different parts of inputs and augment pairwise training data by swapping the order of two generated texts and exchanging the corresponding content in critiques.
Xu et al. (2023) further fine-tune their INSTRUCTSCORE model on self-generated outputs to optimize feedback scores, resulting in diagnostic reports that are better aligned with human judgment.
Liu et al. (2024a) also propose a two-stage supervised finetuning approach, first applying vanilla instruction tuning to equip the model with the ability to follow instructions for diverse evaluations.
They then perform further tuning with auxiliary aspects to enrich the training process, incorporating an additional instruction-tuning stage to leverage potential connections to the target evaluation aspect.
Preference Learning: Preference learning is closely aligned with judgment and evaluation tasks, especially comparative and ranking judgment.
In addition to works that directly adopt or augment preference learning datasets for supervised finetuning judge LLMs, several studies apply preference learning techniques to enhance LLMs' judging capabilities.
To enhance the quality of judgment provided by HALU-J, Wang et al. (2024a) further tune it with Directed Preference Optimization (DPO) (Rafailov et al., 2023) after the SFT stage under the multiple-evidence setting.
Similarly, Park et al. (2024) apply DPO with synthetic "bad" responses that contain critical errors but exhibit stylistic qualities favored by judge models, helping to mitigate bias in the judge LLMs.
Wu et al. (2024) novelly propose meta-rewarding, which leverages the policy LLMs to judge the quality of their own judgment and produce pairwise signals for enhancing the LLMs' judging capability.
This concept is also adopted by Wang et al. (2024f), who propose self-taught evaluators that use corrupted instructions to generate suboptimal responses as inferior examples for preference learning.
Recently, Hu et al. (2024) propose Themis, an LLM dedicated to NLG evaluation, which has been trained with designed multi-perspective consistency verification and rating-oriented preference alignment methods.
Li et al. (2024j) propose PORTIA, an alignmentbased approach designed to mimic human comparison behavior to calibrate position bias in an effective manner.
4.2 Prompting Designing appropriate prompting strategies and pipelines at the inference stage could improve judgment accuracy and mitigate bias.
In this section, we summarize and categorize existing prompting strategies for LLM-as-a-judge (Figure 4).
4.2.1 Swapping Operation Previous studies have demonstrated that LLMbased judges are sensitive to the positions of candidates, and the quality ranking of candidate responses can be easily manipulated by merely altering their order in the context (Wang et al., 2023c; Raina et al., 2024; Zhu et al., 2023).
To mitigate this positional bias and establish a more fair LLM judging system, the swapping operation (Zheng et al., 2023) has been introduced and widely adopted.
This technique involves invoking the judge LLM twice, swapping the order of the two candidates in each instance.
In evaluations, if the results are inconsistent after the swap, it is labeled a "tie," indicating that the LLM is unable to confidently distinguish the quality of the candidates (Zheng et al., 2023).
Several studies have also incorporated swapping operations in selfalignment (Lee et al., 2023; Sun et al., 2024; Lee et al., 2024) to obtain more accurate pairwise feedback from the judge LLM.
Zhu et al. (2024a) proposed a CoT-like prompting technique to mitigate the positional bias by asking the model to first provide all pairwise ranking, then summarize with a ranking list.
4.2.2 Rule Augmentation Rule-augmented prompting involves embedding a set of principles, references, and evaluation rubrics directly within the prompt for the judge LLM.
This approach is commonly employed in LLM-based evaluations, where judge LLMs are guided to assess specific aspects (Li et al., 2024e; Bai et al., 2023a; Yu et al., 2024; Qian et al., 2024) and provided with detailed rubrics (Gao et al., 2023; Kim et al.; Wang et al., 2024e; Murugadoss et al., 2024) to ensure a fair comparison.
A distinct approach is seen in Liu et al. (2024b), where the judge LLM is prompted to generate its own scoring criteria through in-context learning on a set of few-shot examples.
In alignment with LLM-as-a-judge, Bai et al. (2022) first propose to introduce a list of principles (e.g., helpfulness, harmlessness, honesty) for the judge LLM to compare the two candidates more precisely and directionally.
Following them, subsequent works (Lee et al., 2023, 2024; Guo et al., 2024; Sun et al., 2024; Beigi et al., 2024a) enhance this principle-driven prompting by incorporating more detailed explanations for each aspect of the principle or rubric.
Besides, both Li and Qiu (2023) and Li et al. (2024c) propose to prompt LLMs to retrieve appropriate demonstrations/ knowledge triples based on the candidates' helpfulness in solving specific problems.
To obtain diverse responses from LLMs, Lahoti et al. (2023) prompt multiple LLMs to judge the diversity of each candidate and select the most diverse one for further polishing.
Zhang et al. (2024e) propose RevisEval, which leverages the self-correction capabilities of LLMs to adaptively revise the response, then treat the revised text as the principles for the subsequent evaluation.
Recently, Dong et al. (2024) investigate the reliability of LLM-as-a-personalized-judge, providing persona as a part of principles for LLMs to make personalized judgments.
5 Application
While LLM-as-a-judge was initially proposed for the evaluation application, its usage scope has been largely expanded to many other scenarios like alignment, retrieval and reasoning.
Therefore, as Figure 5 shows, we provide a comprehensive introduction to how LLM-as-a-judge can be applied in various applications.
5.1 Evaluation
Traditional evaluation in NLP relies on predefined criteria, typically through metrics, to assess the quality of machine-generated text.
Some prominent metrics such as BLEU, ROUGH, and BERTScore have been widely used in the area.
However, metricbased evaluation overemphasizes lexical overlap and similarity, which may fall short when many valid responses and more nuanced semantic attributes are required to be considered (Post, 2018; Sai et al., 2022).
To address these limitations, LLM-as-a-judge has been used to serve as an automated judge for enhancing evaluations in many tasks (Lin and Chen, 2023b; Mondorf and Plank, 2024).
LLM-as-a-judge enables human-like qualitative evaluations rather than simple quantitative comparisons of how well machine-generated outputs match the ground truth.
This section discusses how LLM-as-a-judge has been utilized to evaluate open-ended generation, reasoning, and more emerging NLP tasks.
5.1.1
Open-ended Generation Tasks
Open-ended generation refers to tasks where the generated content is expected to be safe, accurate, and contextually relevant, though there isn't a single "correct" answer.
Such tasks include dialog response generation, summarization, story generation, and creative writing (Badshah and Sajjad, 2024; Bai et al., 2023a; Kumar et al., 2024).
Unlike conventional metrics-based evaluation methods, LLM-as-a-judge provides a more nuanced, adaptable, and customized evaluation.
As noted by Zheng et al. (2023), LLMs like GPT-4 perform comparably to humans when judging openended text generation.
In practice, LLM-as-a-judge has been applied to evaluate outputs from a single model to compare outputs from multiple models in a competitive setting.
For example, Gao et al. (2023) use ChatGPT to perform human-like summarization evaluation.
Similarly, Wu et al. (2023) propose a comparison-based framework to let LLMs act as judges with multiple role-playing to evaluate the summarization quality in a specific dimension and generate its evaluations.
Modern LLMs are good at generating detailed and long-form responses.
However, as the output length increases, so does the likelihood of hallucinations.
To better understand this phenomenon, Cheng et al. (2023) and Zhang et al. (2024d) introduce an evaluation method that uses GPT-4 to judge whether generated outputs include logically structured yet nonsensical statements.
Wang et al. (2024a) propose a critique-based judging system that evaluates hallucinations by selecting pertinent evidence and providing in-depth critiques.
Beyond hallucinations, a significant concern is the generation of harmful (e.g., encouragement of self-suicide) and unsafe (e.g., guidance of illegal activity) responses by LLMs.
Addressing this, Li et al. (2024f) introduce MD-Judge and MCQ-Judge for evaluating safety-related QA pairs, particularly focusing on queries crafted to provoke unsafe responses.
This approach supports a seamless and reliable evaluation.
However, an overly cautious stance toward unsafe queries can lead to excessive refusal responses, hindering normal functionality and negatively affecting user experience.
To explore this issue, Xie et al. (2024a) conduct a meta-evaluation across various LLM-as-a-judge frameworks, assessing the refusal tendencies of current LLMs in response to potentially unsafe queries.
Recent research has also leveraged LLM-as-ajudge to evaluate the general capabilities of generative models.
This approach often adopts a debatebased framework, where multiple LLMs generate responses that are subsequently evaluated by a separate judging LLM.
For example, Chan et al. (2023) introduce a multi-agent debate framework designed to facilitate autonomous discussions and assess the quality of generated responses from different LLMs in open-ended text generation tasks.
Similarly, Moniri et al. (2024) propose an automated debate framework that evaluates LLMs not only on domain knowledge but also on their abilities in problem definition and inconsistency recognition.
Let me separate each sentence with a line break:
5.1.2 Reasoning Tasks The reasoning capability of LLMs can be evaluated through their intermediate thinking process and final answers on specific reasoning tasks (Mondorf and Plank, 2024).
Recently, LLM-as-a-judge has been used to evaluate the logical progression, depth, and coherence of the model's intermediate reasoning paths.
For mathematical reasoning tasks, Xia et al. (2024) introduce an automatic evaluation framework using a judge LLM specifically designed to assess the quality of reasoning steps in problem-solving processes.
LLM-as-a-judge can also be applied to more complex reasoning tasks like temporal reasoning, where models need to understand the relationships of different events over time.
Fatemi et al. (2024) build synthetic datasets specifically tailored to evaluate LLMs' temporal reasoning abilities across varied scenarios, testing their proficiency in reasoning with sequences, causality, and dependencies among temporally ordered events.
The vast amount of training data presents a challenge in determining whether models are reasoning through deep logical understanding or merely leveraging memorized patterns (Parmar et al., 2024).
Wang et al. (2023a) design a debate-style framework to evaluate LLMs' reasoning capability.
Given a specific question, the LLM and the user adopt opposing positions and discuss the topic to reach a correct decision.
Nan et al. (2024) develop a multi-agent evaluation framework that simulates the academic peer-review process.
This framework engages LLMs-as-Judges in a collaborative review, offering a more nuanced understanding of LLMs' reasoning capabilities in data-driven tasks.
5.1.3 Emerging Tasks As the capabilities of LLMs evolve rapidly, machines are increasingly being employed for tasks previously considered exclusive to humans, especially in context-specific areas.
A prominent task is in social intelligence, where models are presented with complex social scenarios requiring the understanding of cultural values, ethical principles, and potential social impacts.
For example, Xu et al. (2024a) evaluate the social intelligence of LLMs, highlighting that while these models have made strides, they still lag significantly behind in social intelligence compared to their academic problemsolving abilities.
Similarly, Zhou et al. (2023) introduce SOTOPIA and SOTOPIA-EVAL to simulate complex social interactions between LLM agents and evaluate their social intelligence.
In their work, GPT-4 is used as a proxy for human judgment in assessing goal completion, financial management, and relationship preservation within simulated interactions.
Another line of research has been directed towards evaluating Large Multimodal Models (LMMs) and Large Vision-Language Models (LVLMs).
For example, Xiong et al. (2024b) explore LMM-as-a-judge to evaluate the performance of multimodal models, providing both a final score and the underlying rationale for evaluations, promoting transparency and consistency.
Chen et al. (2024c) propose the first benchmark for the automatic evaluation of LVLMs, specifically for selfdriving corner cases.
They find that evaluations performed by LLMs-as-judges align more closely with human preferences than those conducted by LVLM-as-judges.
Recently, we have seen more customized utilization of LLM-as-a-judge to evaluate emerging tasks such as code understanding (Zhao et al., 2024a), legal knowledge (Fei et al., 2023), game development (Isaza-Giraldo et al., 2024), ocean science (Bi et al., 2023), healthcare conversations (Wang et al., 2024j), debating judgment (Liang et al., 2024a) and more.
This trend reflects the growing adaptability of LLMs-as-judges in evaluating diverse and specialized domains.
5.2 Alignment Alignment tuning (Wei et al., 2022a; Ouyang et al., 2022) is a vital technique to align LLMs with human preferences and values.
A key component of this process is the collection of high-quality, pairwise feedback from humans, which is essential for reward modeling (Schulman et al., 2017) or direct preference learning (Rafailov et al., 2023).
Recently, there have been increasing research interests focused on automating this pairwise feedback mechanism by adopting LLM-as-a-judge in alignment tuning.
5.2.1 Larger Models as Judges One intuitive idea for adopting LLM-as-a-judge in alignment tuning is to leverage the feedback from larger, more powerful LLMs to guide smaller, less capable models.
(Bai et al., 2022) first propose employing the AI's feedback to build a harmless AI assistant.
They train the reward model using synthetic preference data based on the preference of a pre-trained language model.
Building on this, Lee et al. (2023) discover that the RLAIF method can achieve comparable performance with RLHF even when the LLM judge is not strong enough.
They also introduced DIRECT-RLAIF, which directly employs an off-the-shelf LLM as the judge model to mitigate reward staleness in reward models.
To avoid reward hacking in alignment, Sun et al. (2024) devise an instructable reward model trained on synthetic preference data.
It enables humans to perform RL-time interventions to better align the target policy with human values.
Apart from the abovementioned studies, Guo et al. (2024) introduce online AI feedback (OAIF), directly utilizing the preference signals from an annotation model to train the target model.
There are also works that utilize multi-agent cooperation for better judgment in alignment tuning.
Arif et al. (2024) construct a synthetic preference optimization dataset using multi-agent workflows and adopt LLMs as judges with diverse prompting strategies and pipelines.
Similarly, (Li et al., 2024g) employ multiple LLMs to debate with each other, iteratively improving response quality, while creating a judge LLM to select preferred responses for enhanced instruction tuning.
To align the generated code with human preference, Weyssow et al. (2024) introduce CodeUltraFeedback, a preference coding dataset constructed using the LLM-as-a-Judge methodology.
This synthetic dataset is later used to fine-tune and align small code LLMs using SFT and DPO.
Recently, Wang et al. (2024d) propose BPO, using GPT-4 as the judge and constructing synthetic pairwise feedback for knowledge depth and breadth balance in the alignment process.
5.2.2 Self-Judging Another line of work aims to utilize the preference signal from the same LLM for self-improving.
Yuan et al. (2024c) first propose the concept of a self-rewarding LLM, where pairwise data is constructed by having the LLM itself serve as the judge.
Following them, Wu et al. (2024) introduce metarewarding that judges the LLMs' judge and uses the feedback to refine their judging skills.
Their LLMas-a-meta-judge approach significantly enhances the model's capacity to evaluate and follow instructions.
To improve synthetic data quality, Pace et al. (2024) combine the Best-of-N and Worst-of-N sampling strategies and introduce the West-of-N approach.
Lee et al. (2024) devise Judge augmented Supervised Fine-Tuning (JSFT) to train a single model to act as both a policy and a judge.
To fully utilize this judge model, they also propose the SelfRejection by Tournament method to select the best response in the inference time.
Unlike the above approaches, which use the LLM as a judge for pairwise data construction, Tong et al. (2024) apply LLM-as-a-judge as a self-filtering method to ensure the quality of synthetic data pairs in alignment tasks for reasoning.
To reduce the computation overhead in pairwise judgment, Zhai et al. (2024) propose a ranked pairing method for self-preferring language models, which accelerates the comparison process by measuring the strength of each response against a baseline.
Liu et al. (2024c) introduce meta-ranking, enabling weaker LLMs to serve as reliable judges and provide trustworthy feedback.
They also apply their meta-ranking method in post-SFT training, combining it with KahnemanTversky Optimization (KTO) for improved alignment.
To enhance the quality of the synthetic instruction tuning data, Liang et al. (2024c) introduce an iterative self-enhancement paradigm (I-SHEEP).
During training, they adopt LLM-as-a-judge to score the synthetic responses and set a threshold to collect high-quality query-response pairs for the subsequent training iteration.
Several works have also employed LLM-as-ajudge in specific domains or for particular attributes.
Zhang et al. (2024f) propose a self-evaluation mechanism, judging responses' factuality by generating question-answer pairs.
They then utilize these self-annotated responses to fine-tune the model via the DPO algorithm for better factuality.
In robotics, Zeng et al. (2024) iteratively update the reward function with the self-ranking responses from LLMs, boosting learning efficiency without human supervision.
In the multimodal domain, Ahn et al. (2024) propose iterative self-retrospective judgment (i-SRT), which employs self-reflection to improve both response generation and preference modeling.
5.3 Retrieval The role of LLM-as-a-judge in retrieval encompasses both traditional document ranking and the more dynamic, context-adaptive RetrievalAugmented Generation (RAG) approaches.
In traditional retrieval, LLMs enhance ranking accuracy through advanced prompting techniques, enabling them to order documents by relevance with minimal labeled data.
Complementarily, RAG frameworks capitalize on LLMs' capacity to generate content guided by retrieved information, supporting applications where complex or evolving knowledge integration is essential.
Together, these techniques underscore LLMs' adaptability as judges in retrieval tasks, extending from foundational ranking to domain-specific, knowledge-augmented applications.
5.3.1 Traditional Retrieval Recent research has explored the role of LLMs as judges for ranking documents in information retrieval, aiming to boost ranking precision and reduce reliance on extensive training data.
For example, Sun et al. (2023) explore the potential of generative LLMs like GPT-4 for relevance ranking in information retrieval.
They propose a permutationbased approach to rank passages by relevance, instructing LLMs to output ordered permutations of passages, thus enhancing ranking precision.
Complementing this, Zhuang et al. (2024a) introduce a method that embeds fine-grained relevance labels within LLM prompts, enabling the models to distinguish subtle relevance variations and produce a more refined document ordering.
Further innovations in listwise ranking are illustrated by Ma et al. (2023), who present the Listwise Reranker with a Large Language Model (LRL), a tool that directly reorders document identifiers without relying on task-specific training data.
Additionally, Zhuang et al. (2024b) contribute a Setwise prompting strategy tailored for zero-shot ranking, which streamlines ranking operations by decreasing LLM inference frequency and token usage—enhancing efficiency without sacrificing performance.
To address positional biases, a common challenge in listwise ranking tasks, Tang et al. (2024b) introduce a permutation self-consistency technique that averages across multiple list orders to yield order-independent rankings.
This approach effectively reduces positional bias, an issue particularly problematic in LLM-driven listwise ranking.
Finally, Qin et al. (2024) critique the limitations of pointwise and listwise ranking prompts in existing methods, noting that typical LLMs often lack the depth to grasp complex ranking tasks.
To mitigate this, they propose Pairwise Ranking Prompting (PRP) with medium-sized, open-source LLMs as an effective, cost-efficient alternative to larger proprietary models.
Beyond general retrieval tasks, LLMs have demonstrated their utility as judges in specialized applications.
For example, Ma et al. (2024) outline a few-shot workflow that employs a generalpurpose LLM for relevance judgments in legal information retrieval.
This model achieves high consistency with expert annotations by breaking down tasks into stages, facilitating the integration of expert reasoning to improve relevance assessment accuracy in legal contexts.
In recommender systems, Hou et al. (2024) examine LLMs' potential to rank items by framing recommendations as conditional ranking tasks.
This framework factors in user interaction histories alongside candidate items, addressing LLMs' known biases, such as tendencies to favor popular or top-positioned items.
Specialized prompting and bootstrapping techniques are employed to correct these biases and improve interpretive accuracy.
Finally, in the realm of search systems, Thomas et al. (2023) find that LLMs perform comparably to human labelers in predicting searcher preferences, making them valuable for identifying high-performing systems and flagging challenging queries.
This study underscores the effectiveness of LLMs as judges in complex retrieval tasks, enabling more nuanced and accurate relevance evaluations across diverse applications.
5.3.2 Retrieval-Augmented Generation (RAG) Recent developments in Retrieval-Augmented Generation (RAG) have explored the capacity of LLMs to self-evaluate and self-improve without the need for annotated datasets or parameter adjustments (Chen et al., 2024d).
Li and Qiu (2023) introduce the Memory-of-Thought (MoT) framework, a two-stage self-reflective model that enhances an LLM's reasoning abilities autonomously.
In the first stage, the model generates high-confidence reasoning on an unlabeled dataset, storing this as memory.
During the testing phase, the model recalls this memory by judging the relevance of each of them with the current question and choose the most relevant ones as demonstration.
In a similar vein, Tang et al. (2024a) propose Self-Retrieval, an innovative architecture that consolidates information retrieval (IR) capabilities within a single LLM, utilizing natural language indexing to internalize the corpus.
This approach transforms retrieval into a document generation and self-assessment process, achieving a fully endto-end IR workflow within one model.
Furthermore, Asai et al. (2024) present SELF-RAG (SelfReflective Retrieval-Augmented Generation), a model that enhances LLM response quality and factuality through a retrieval and self-reflection cycle.
By using "reflection tokens" to guide adaptive responses, SELF-RAG allows the model to dynamically judge and adjust its responses based on task-specific requirements.
In the domain of question answering, LLMs are increasingly employed as evaluative agents to assess answer relevance, quality, and utility in real-time.
Rackauckas et al. (2024) introduce an LLM-based evaluation framework that generates synthetic queries from actual user interactions and domain-specific documents.
Within this framework, LLMs function as judges, evaluating retrieved documents and ranking RetrievalAugmented Generation (RAG) agent variants via RAGElo, an Elo-based automated competition.
This structure offers a scalable solution for quality control in QA systems.
Additionally, Zhang et al. (2024b) conduct an extensive study on LLMs' ability to assess relevance versus utility in open-domain QA.
Their findings demonstrate that LLMs can effectively distinguish between the two and are highly adaptable when presented with counterfactual passages.
This utility assessment capability allows LLMs to provide more nuanced and contextually relevant responses during evaluations.
RAG systems tailored to specific domains reveal the potential for LLMs to navigate complex queries by integrating specialized knowledge structures.
In domain-specific retrieval, Wang et al. (2024b) present BIORAG, an advanced RAG framework that enhances vector retrieval with hierarchical knowledge structures.
BIORAG adopts a selfaware evaluated retriever to continuously judge the adequacy and relevance of the information it has collected, thus improving the accuracy of the retrieved document.
For biomedical research, Li et al. (2024c) introduce DALK (Dynamic CoAugmentation of LLMs and Knowledge Graphs), a novel system that combines an LLM with a continuously evolving Alzheimer's Disease (AD) knowledge graph derived from the scientific literature.
Using the novel self-aware knowledge retrieval method, DALK employs the judging capability of LLMs to do noise filtering, enhancing the LLM's inferential performance in AD-related queries.
Similarly, Jeong et al. (2024) propose SelfBioRAG, a framework that adapts RAG principles to biomedical applications.
Self-BioRAG employs the LLM to select the best evidence and generate the answer based on the selected evidence and encoded knowledge.
5.4 Reasoning Unlocking the reasoning capability of LLMs offers a way to relieve the limitations of scaling laws, which alone may not fully reveal the models' potential.
Reasoning is a critical aspect of LLMs because it directly affects their ability to solve complex problems, make decisions, and provide accurate, context-aware responses.
Recently, many studies about LLMs' reasoning capability have focused on how to leverage LLM-as-a-judge to select reasoning paths (Section 5.4.1) and utilize external tools (Section 5.4.2).
5.4.1 Reasoning Path Selection Wei et al. (2022b) introduces the concept of Chain-of-Thought (CoT) prompting to encourage models to generate step-by-step reasoning processes.
While other more complex cognition structures (Yao et al., 2023a; Hao et al., 2023) have been proposed to boost LLMs' reasoning capabilities, one crucial challenge is how to select a reasonable and reliable reasoning path or trajectory for LLMs to follow.
To address this problem, LLM-asa-judge has been adopted by many works.
Some works focus on sample-level selection in the reasoning process.
(Kawabata and Sugawara, 2024) introduce REPS (Rationale Enhancement through Pairwise Selection), judging and selecting valid rationales via pairwise self-evaluation with LLMs and training a verifier based on these data.
Another reasoning path selection problem for LLMs is diversity, Lahoti et al. (2023) find that LLMs grasp the concept of diversity and can identify aspects where responses lack diversity.
By selecting and aggregating multiple critiques, LLMs can achieve similar gains, compared to multiple iterations of critique and revision.
In multi-agent collaboration frameworks, Liang et al. (2023) propose multi-agent debating (MAD), a novel paradigm to facilitate debating and discussion among multiple agents.
They leverage the judge LLM to select the most reasonable response as the final output at the end of the debating process.
Similarly, Li et al. (2024b) propose new roles in layer-based multiagent collaboration, employing the judge LLM to select the high-quality and reasonable response and thus enhancing the whole system's token utilizing efficiency by a considerable margin.
Additionally, there are also many works that focus on step-level reasoning path selection, leveraging the judge LLM as the process reward model (PRM) to evaluate state scores.
Creswell et al. (2023) decompose the reasoning process into Selection and Inference.
In the selection step, they leverage the LLM itself to judge and evaluate each potential reasoning trace, selecting the appropriate one for the following inference step.
Xie et al. (2024b) propose the Kwai-STaR framework, which transforms LLMs into state-transition reasoners to judge and select the best reasoning state for themselves in mathematical reasoning.
Lightman et al. (2023) train a LLM as the PRM to conduct inference-time supervision and perform best-of-N sampling strategy during reasoning stage.
Following them, Setlur et al. (2024) further propose process advantage verifiers (PAVs), generating rewards based on the changes in the likelihood of producing correct responses in the future.
Other works simulate advanced cognitive structures as the reasoning process.
Hao et al. (2023) employ LLMs as a world model to simulate the state of the environment and perform Monte Carlo Tree Search (MCTS) to improve performance on tasks requiring deliberate path selection.
Furthermore, Besta et al. (2024) takes the outputs generated by LLMs as an arbitrary graph.
An LLM thought is modeled as a vertex, while an edge is a dependency between thoughts.
This framework enables a systematic judge of coherence and logical reasoning for each reasoning state.
Yao et al. (2023a) propose Treeof-Thoughts (ToT), where each thought serves as an intermediate step toward problem-solving.
It decomposes reasoning into steps, self-evaluates and judges progress at each state, and uses search algorithms with LMs to judge thought paths through lookahead and backtracking.
5.4.2 Reasoning with External Tools Yao et al. (2023b) first propose to use LLMs in an interleaved manner to generate reasoning traces and task-specific actions.
Reasoning traces help the model to judge and update action plans, while actions enable it to interact with external sources.
Further, Auto-GPT was introduced by (Yang et al., 2023) to deliver more accurate information with LLM-as-a-judge for tool-using.
By equipping with a range of external complex tools, LLMs become more versatile and capable, improving planning performance by judging and reasoning which tools to use.
Sha et al. (2023) explore LLMs' potential in reasoning and judging, employing them as decisionmaking components for complex autonomous driving scenarios that require human commonsense understanding.
Zhou et al. (2024b) utilize a selfdiscovery process where LLMs perform judgment based on the given query and select the most feasible reasoning structure for the following inference stage.
While LLMs have demonstrated impressive judging capability across a variety of tools.
However, selecting which model or API to use often involves a trade-off between performance and cost.
More powerful models, though effective, are also more expensive, while less capable models are more costefficient.
To address this dilemma, the authors in Ong et al. (2024) propose a routing model that can dynamically choose between a stronger or weaker LLM during the judging process, aiming to balance the cost and response quality.
Similarly for efficiency consideration, DiffAgent was introduced in Zhao et al. (2024b) to act as an agent designed to judge and select different text-to-image APIs with user-specific prompts.
DiffAgent's judgment aligns more closely with human preferences, outperforming traditional API selection methods.
6 Benchmark: Judging LLM-as-a-judge The evaluation of large language models (LLMs) as judges necessitates robust and purpose-driven benchmarks to capture the multifaceted nature of this task.
We categorize existing benchmarks into the following dimensions: general performance, bias quantification, domain-specific performance, multimodal evaluation, multilingual capabilities, evaluation instruction following, vulnerability assessment, and challenging task performance.
This taxonomy exhibits the diverse goals of LLM-asa-judge evaluation frameworks, providing a structured lens through which to analyze their design, scope, and impact.
Table 2 demonstrates the collection of benchmarks for LLM-as-a-judge.
6.1 General Performance Benchmarks focusing on general performance aim to evaluate the overall competence of LLMs across a variety of tasks.
These benchmarks typically measure agreement with human judgments, accuracy, and correlation.
Notable examples include: • MT-Bench and Chatbot Arena (Zheng et al., 2023), which assess conversational settings using metrics such as consistency, bias, and error.
These benchmarks further explore specific biases, including position bias, verbosity, and selfenhancement tendencies.
• JUDGE-BENCH (Tan et al., 2024a), DHP (Wang et al., 2024g), and SOS-BENCH (Penfever et al.,
2024), which operate at larger scales, utilizing metrics like Cohen’s kappa, Discernment Score,
and normalized accuracy to benchmark general LLM performance.
• LLM-judge-eval (Wei et al., 2024a), which evaluates tasks such as summarization and alignment
with additional metrics like flipping noise and length bias.
6.2 Bias Quantification
Mitigating bias in LLM judgments is critical to ensuring fairness and reliability. Typical benchmarks
include EVALBIAS-BENCH (Wang et al., 2024c) and CALM (Ye et al., 2024a), focus explicitly on
quantifying biases, including those emerging from alignment and robustness under adversarial conditions.
Besides, Shi et al. (2024) evaluate metrics such as position bias and percent agreement in question-answering tasks.
6.3 Challenging Task Performance Benchmarks designed for difficult tasks push the boundaries of LLM evaluation.
For example, Arena-Hard Auto (Li et al., 2024h) and JudgeBench (Tan et al., 2024a) select harder questions based on LLMs' performance for conversational QA and various reasoning tasks, respectively.
CALM (Ye et al., 2024a) explores alignment and challenging scenarios, using metrics such as separability, agreement, and hacked accuracy to evaluate performance in manually identified hard data sets.
6.4 Domain-Specific Performance Domain-specific benchmarks provide task-focused evaluations to assess LLMs' effectiveness in specialized contexts.
Concretely, Raju et al. (2024) measure separability and agreement across tasks, leveraging metrics such as Brier scores for insights in specific domains such as coding, medical, finance, law and mathematics.
CodeJudge-Eval (Zhao et al., 2024a) specifically evaluates LLMs for judging code generation with execution-focused metrics such as accuracy and F1 score.
6.5 Other Evaluation Dimensions Beyond general performance and bias quantification, several benchmarks address additional evaluation dimensions essential for assessing specific aspects of using LLMs as judges:
• Multimodal: MLLM-as-a-judge (Chen et al., 2024a) extends evaluation frameworks to tasks involving multiple data modalities, focusing on agreement with human judgments, grading analysis, and hallucination detection.
• Multilingual: Benchmarks such as MM-EVAL (Son et al., 2024b) and KUDGE (Son et al., 2024a) evaluate multilingual and non-English performance, measuring metrics like accuracy and correlation, particularly in challenging scenarios.
• Instruction Following: Murugadoss et al. (2024) examine the extent to which LLMs adhere to specific evaluation instructions, utilizing correlation metrics to quantify performance.
7 Challenges & Future Works In this section, we outline the current challenges for LLM-as-a-judge and promising direction for future works to explore, including bias & vulnerability, dynamic & complex judgment, self-judging and human-LLM co-judgment.
7.1 Bias & Vulnerability The use of LLMs-as-a-judge inherently frames evaluation as a generation task, introducing significant challenges related to bias and vulnerability.
These biases often stem from the models' training data, which frequently embeds societal stereotypes tied to demographic identities such as race, gender, religion, culture, and ideology (Sheng et al., 2021).
Such biases can significantly compromise fairness and reliability when LLMs are deployed for diverse judging tasks.
In addition to these general biases, specific evaluative biases emerge when LLMs act as judges.
Order Bias is a prominent issue where the sequence of candidates influences preferences (Zheng et al., 2023; Wang et al., 2023c; Koo et al., 2023a; LLMS).
This bias can distort evaluation outcomes, particularly in pairwise comparisons, and is more pronounced when the quality gap between competing responses is small (LLMS; Wang et al., 2023c).
Egocentric Bias arises when LLMs favor outputs generated by the same model, compromising objectivity (Liu et al., 2023c; Koo et al., 2023a; Wataoka et al., 2024).
This issue is particularly pronounced when evaluation metrics are designed using the same model, leading to inflated scores for outputs from the originating model (Liu et al., 2023c).
Length Bias, another prevalent challenge, skews evaluations by disproportionately favoring longer or shorter responses regardless of quality (Zheng et al., 2023; Koo et al., 2023a).
Additional biases, such as Misinformation Oversight Bias, Authority Bias, and Beauty Bias, further complicate LLM evaluations.
For instance, misinformation oversight bias reflects a tendency to overlook factual errors, authority bias favors statements from perceived authoritative sources, and beauty bias prioritizes visually appealing content over substantive quality (Chen et al., 2024b,d; Stephan et al., 2024).
Similarly, Verbosity Bias shows a preference for lengthier explanations, often equating verbosity with quality, which can mislead the judgment process (Yuan et al., 2024a).
Furthermore, Sentiment Bias skews evaluations based on emotional tone, favoring responses with positive phrasing (Ye et al., 2024a).
LLM judges are also highly susceptible to adversarial manipulations.
Techniques like JudgeDeceiver highlight the risks posed by optimizationbased prompt injection attacks, where carefully crafted adversarial sequences can manipulate LLM judgments to favor specific responses (Shi et al., 2024).
Similarly, universal adversarial phrases can drastically inflate scores in absolute scoring paradigms, revealing vulnerabilities in zero-shot assessment setups (Liusie et al., 2023; Raina et al., 2024; Doddapaneni et al., 2024).
These manipulations raise concerns about the reliability of LLM judges in high-stakes scenarios such as leaderboards and academic assessments (Shi et al., 2024; Raina et al., 2024).
To address these biases and vulnerabilities, frameworks like CALM (Ye et al., 2024a) and BWRS (Gao et al., 2024b) offer systematic approaches for bias quantification and mitigation.
Techniques such as Multiple Evidence Calibration (MEC), Balanced Position Calibration (BPC), and Human-in-the-Loop Calibration (HITLC) have proven effective in aligning model judgments with human evaluations while reducing positional and other biases (Wang et al., 2023c).
Additionally, cognitive bias benchmarks like COBBLER have identified six key biases, including salience bias and bandwagon effect, that need systematic mitigation in LLM evaluations (Koo et al., 2023b).
Future Direction. One promising direction for future research is to integrate Retrieval-Augmented Generation (RAG) frameworks into LLM evaluation processes (Chen et al., 2024d).
By combining generative and retrieval capabilities, these frameworks can reduce biases such as self-preference and factuality issues by grounding evaluations in external, verifiable data sources.
Another promising avenue is the use of bias-aware datasets, like OFFSETBIAS, to systematically address inherent biases in LLM-as-a-judge systems (Park et al., 2024).
Incorporating such datasets into training pipelines allows LLMs to better distinguish superficial qualities from substantive correctness, thereby enhancing fairness and reliability.
Exploring finetuned LLMs as scalable judges, exemplified by the JudgeLM framework, represents another intriguing direction (Zhu et al., 2023).
Techniques like swap augmentation and reference support embedded in this framework can systematically mitigate biases, improve evaluation consistency, and extend the applicability of LLM-based judgments to open-ended tasks.
Additionally, advancing zeroshot comparative assessment frameworks offers significant promise (Liusie et al., 2023).
These frameworks can refine pairwise comparison techniques and implement debiasing strategies, improving fairness and reliability across diverse evaluation domains without the need for extensive prompt engineering or fine-tuning.
Finally, methods like JudgeDeceiver-resistant calibration and adversarial phrase detection strategies need further exploration to secure LLM-as-a-judge frameworks from attacks (Shi et al., 2024; Raina et al., 2024; Zhou et al., 2024a; Li et al., 2024d).
7.2 Dynamic & Complex Judgment Earlier works about LLM-as-a-judge usually adopt static and straightforward approaches that directly prompt the judge LLM to perform the assessment (Zheng et al., 2023).
Recently, more dynamic and complex judgment pipelines have been proposed to address various limitations, improving the robustness and effectiveness of LLM-as-ajudge.
One approach in direction is to follow the concept of "LLM-as-a-examiner", where the system dynamically and interactively generates both questions and judgments based on the candidate LLMs' performance (Yu et al., 2024; Bai et al., 2023a).
Other works focus on making judgments based on two or more candidate LLMs battle and debating results (Moniri et al., 2024; Zhao et al., 2024c).
These dynamic judgment methods largely improve the judge LLM's understanding of each candidate and potentially prevent the data contamination problem (Deng et al., 2024; Golchin and Surdeanu) in LLMs' evaluation.
Additionally, building complex and complicated judgment pipelines or agents is another popular research area (Li et al., 2023; Chan et al., 2023; Zhuge et al., 2024).
These methods typically involve multi-agent collaboration, along with well-designed planning and memory systems, enabling the judge LLMs to handle more complex and diverse judgment scenarios.
Future Direction. One promising direction for future research is to equip LLMs with human-like judgment capabilities (Yuan et al., 2024a; Liang et al., 2024b).
These designs can borrow insights from human behavior when making a judgment, such as anchoring and comparing, hindsight and reflection, and meta-judgment.
Another intriguing avenue would be to develop an adaptive difficulty assessment system using LLMs (Hu, 2024).
This system would adjust the difficulty of questions based on the current performance of the candidates.
Such an adaptive and dynamic system could address a significant limitation in LLM evaluation, as static benchmarks often fail to accurately assess LLMs with varying capabilities.
7.3 Self-Judging LLM-based evaluators, such as GPT-4, are widely used for assessing outputs but face significant challenges, particularly Egocentric Bias, where models favor their own responses over those from external systems (Liu et al., 2023b; Bai et al., 2023a; Zheng et al., 2023).
This self-preference undermines impartiality and creates a "chicken or the egg" dilemma: robust evaluators are essential for developing powerful LLMs, yet advancing LLMs depends on unbiased evaluators.
Other issues include Self-Enhancement Bias, where models overrate their own outputs (Li et al., 2023), and Reward Hacking, where over-optimization of specific signals leads to less generalizable evaluations (Wu et al., 2024).
Additionally, reliance on Static Reward Models limits adaptability, while biases like Positional and Verbosity Biases distort judgments by favoring response order or length over quality (Yuan et al., 2024c; Wang et al., 2024f).
The high cost and limited scalability of human annotations further complicate the creation of dynamic and reliable evaluation systems (Srivastava et al., 2022; Liang et al., 2022).
Future Direction. One promising direction for future research is the development of collaborative evaluation frameworks like Peer Rank and Discussion (PRD) (Li et al., 2023).
These frameworks leverage multiple LLMs to collectively evaluate outputs, using weighted pairwise judgments and multi-turn dialogues to reduce self-enhancement bias and align evaluations closer to human standards.
Another intriguing avenue is the adoption of Self-Taught Evaluator frameworks, which generate synthetic preference pairs and reasoning traces to iteratively refine model evaluation capabilities (Wang et al., 2024f).
This approach eliminates dependency on costly human annotations while ensuring that evaluation criteria adapt to evolving tasks and models.
The integration of Self-Rewarding Language Models (SRLM) offers another promising path (Yuan et al., 2024c).
By employing iterative mechanisms like Direct Preference Optimization (DPO), these models continuously improve their instruction-following and reward-modeling capabilities, mitigating issues like reward hacking and overfitting.
Building on SRLM, the use of Meta-Rewarding mechanisms introduces a meta-judge role to evaluate and refine judgment quality (Wu et al., 2024).
This iterative process addresses biases such as verbosity and positional bias, enhancing both alignment and the ability to evaluate complex tasks.
Finally, leveraging synthetic data creation for generating contrasting responses offers a scalable solution for training evaluators (Wang et al., 2024f).
By iteratively refining evaluations on synthetic preference pairs, models can progressively improve their robustness and adaptability.
Combining these approaches with diverse benchmarks (Srivastava et al., 2022; Liang et al., 2022), multifaceted evaluation criteria (Sellam et al., 2020), and human feedback (Xu et al., 2023; Ouyang et al., 2022) can ensure evaluations that are fair, reliable, and aligned with human expectations across various domains.
7.4 Human-LLMs Co-judgement As mentioned earlier, the biases and vulnerabilities in LLM-as-a-judge can be addressed through human involvement in the judgment process for further intervention and proofreading.
However, only a few studies have focused on this approach.
Wang et al. (2023c) introduce human-in-the-loop calibration, which employs a balanced position diversity entropy to measure the difficulty of each example and seeks human assistance when necessary.
In the context of relevance judgment, Faggioli et al. (2023) propose a human-machine collaboration spectrum, which categorizes different relevance judgment strategies based on the extent to which humans rely on machines.
Future Direction. As data selection (Xie et al., 2023; Albalak et al., 2024) becomes an increasingly popular research area for improving the efficiency of LLMs' training and inference, it also holds the potential for enhancing LLMs' evaluation.
LLM-as-a-judge can draw insights from data selection to enable judge LLMs to serve as a critical sample selector, choosing a small subset of samples for human annotators to evaluate based on specific criteria (e.g., representativeness or difficulty).
Additionally, the development of human-LLM co-judgment can benefit from mature human-in-the-loop solutions in other fields, such as data annotation (Tan et al., 2024b) and active learning (Margatina et al., 2023).