https://arxiv.org/pdf/2411.04282
Here's the text with each sentence on a new line:
Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps.
While prompt-based methods like Chain-of-Thought (CoT) can improve LLM reasoning at inference time, optimizing reasoning capabilities during training remains challenging.
We introduce LaTent Reasoning Optimization (LaTRO), a principled framework that formulates reasoning as sampling from a latent distribution and optimizes it via variational approaches.
LaTRO enables LLMs to concurrently improve both their reasoning process and ability to evaluate reasoning quality, without requiring external feedback or reward models.
We validate LaTRO through experiments on GSM8K and ARC-Challenge datasets using multiple model architectures.
On GSM8K, LaTRO improves zero-shot accuracy by an average of 12.5% over base models and 9.6% over supervised fine-tuning across Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B.
Our findings suggest that pre-trained LLMs possess latent reasoning capabilities that can be unlocked and enhanced through our proposed optimization approach in a self-improvement manner.
The code of LaTRO is available at https://github.com/SalesforceAIResearch/LaTRO.
[Section 1: INTRODUCTION]
The development of large language models (LLMs) with enhanced reasoning capabilities has emerged as a crucial area of research.
Despite their impressive advances, the inherent next-token prediction mechanism of LLMs makes it challenging for these models to solve complex problems requiring multiple reasoning steps (Wang et al., 2022; Huang et al., 2023).
For instance, LLMs often struggle to directly provide accurate solutions to mathematical problems or even simple puzzles like counting specific letters in a word.
Consequently, researchers have explored various prompting strategies that guide LLMs to generate reasoning trajectories or rationales—sequences of tokens that build a step-by-step progression toward an answer.
Techniques such as Chain-of-Thought (CoT) (Wei et al., 2022), Tree-of-Thought (ToT) (Yao et al., 2024), and Program-of-Thought (PoT) (Chen et al., 2023) prompting methods exemplify this approach.
Recent progress has also focused on inference-time techniques to enhance the reasoning abilities of LLMs (Wu et al., 2024; Brown et al., 2024), as observed in the OpenAI o1 model (OpenAI, 2024).
These methods have demonstrated remarkable performance in diverse reasoning tasks, including mathematics (Cobbe et al., 2021b; Trinh et al., 2024; Luo et al., 2024), coding (Jimenez et al., 2023; Guo et al., 2024; Zhang et al., 2024), and scientific problem-solving (Rein et al., 2023).
Notable inference-time methods, such as CoT with Self-Consistency (CoT-SC) (Wang et al., 2023) and CoT-Decoding (Wang & Zhou, 2024), extend the CoT approach by generating multiple reasoning paths and selecting the most consistent one.
Additionally, techniques like ReAct (Yao et al., 2023a) and Reflexion (Shinn et al., 2023) integrate reasoning into LLM agent loops, further enhancing their problem-solving capabilities.
Despite the promising results at inference time, improving the reasoning abilities of LLMs during their training phase remains a challenging problem.
Several obstacles impede progress in this area.
Firstly, there is a scarcity of high-quality reasoning data for complex problems, limiting the applicability of traditional supervised fine-tuning (SFT) approaches (Zelikman et al., 2022).
Moreover, when such data is available, SFT on deterministic reasoning paths may result in a lack of diversity in problem-solving strategies, potentially causing over-confidence issues and performance degradation (Cobbe et al., 2021b), especially in domains needing multiple valid approaches, such as mathematical proofs and coding.
Alternatively, improving reasoning through reinforcement learning from human feedback (RLHF) presents its own challenges (Havrilla et al., 2024; Luo et al., 2024).
Developing a reward model that accurately evaluates the quality and validity of reasoning paths is a formidable task, susceptible to distribution shifts and biased evaluations.
Self-improvement approaches like STaR (Self-Taught Reasoner) (Zelikman et al., 2022) and QuietSTaR (Zelikman et al., 2024) have shown promise in enhancing language models' reasoning capabilities without external feedback.
However, STaR relies on task-specific few-shot examples to bootstrap its reasoning process, which can limit its generalizability across diverse tasks.
While Quiet-STaR attempts to overcome this by inferring implicit rationales across arbitrary text, it does not directly optimize the reasoning process itself.
Through these findings, we observe that pretrained LLMs already possess innate reasoning capabilities but just have not been fully activated or utilized, inspiring us to propose our approach.
Our proposed method, LaTent Reasoning Optimization (LaTRO), addresses the limitations of previous approaches by formulating reasoning as sampling from a latent distribution and optimizing it through a principled variational framework.
As illustrated in Fig. 1, LaTRO enables language models to concurrently improve both their reasoning process and ability to evaluate reasoning quality, without requiring task-specific few-shot examples or external reward models.
Key contributions of LaTRO include:
1. A theoretical formulation connecting LLM reasoning optimization to latent variable models;
2. A self-rewarding mechanism leveraging the model's own probability estimates;
3. Significant performance gains across multiple model architectures and reasoning tasks, demonstrating LaTRO's effectiveness in unlocking latent reasoning capabilities of language models.
Our findings suggest that pre-trained LLMs are not only capable reasoners but also possess the potential to act as explicit reward models for evaluating reasoning paths.
We term this approach of utilizing explicit reward functions induced by LLMs themselves as "self-rewarding."
Empirically, LaTRO outperforms both baseline models and supervised fine-tuning approaches on reasoning tasks like GSM8K, while also demonstrating the capacity to compress reasoning processes and shift computational burdens from inference to training time.