LLaVA-o1: Step-by-Step Visual Reasoning VLM Explained

LLaVA-o1, later named LLaVA-CoT, is a vision-language model trained to work through visual questions in four stages: summarization, visual interpretation, logical reasoning, and conclusion. Instead of answering immediately, it separates what it sees from how it reasons about it. The research also explores using extra computation at inference time to compare or revisit reasoning steps before giving an answer.

This article originally examined the November 2024 release. Its model comparisons and stage-level beam search discussion reflect that version. The July 2025 paper revision describes stage-wise retracing search (SWIRES); the original analysis is preserved below.

VLMs face unique challenges compared to Large Language Models (LLMs), which are more extensively developed and reliable. While LLMs often excel in reasoning tasks, VLMs tend to struggle with systematic reasoning, frequently bypassing structured thought processes and arriving at final answers prematurely – leading to errors. TTo address these gaps, researchers from leading Chinese universities, Lehigh University, and prominent research labs collaborated to develop LLaVA-o1, a VLM designed to reason step-by-step and maintain a well-organized thought process. The model also employs a specific inference-time scaling technique to make its answers more reliable.

Social networks and media flashed titles: “LLaVA-o1 to challenge OpenAI's o1 model.” Is that so? And what sets LLaVA-o1 apart? Let’s find out!

In today’s episode, we will cover:

Vision-Language Models (VLMs) are gaining popularity as they can understand the world from both sides – words and images. But they often struggle with challenging problems, like answering complex questions about images. The problem is that most VLMs just give quick answers without working through the problem in a structured way. And it’s not what we all need. And even Chain-of-Thought (CoT) reasoning isn’t a perfect solution, because VLMs still make mistakes, jump to conclusions, or produce nonsense, and their reasoning process remains poorly organized at its core.

Compared to VLM, LLMs are more reliable for tasks that need logical thinking. For example, OpenAI's o1 is great at reasoning, which is widely credited to inference-time scaling. OpenAI hasn’t disclosed the details, but this is believed to include: breaking down complex problems step-by-step, using multiple attempts (answers) to solve a problem and iterative reasoning.

So how can we enhance VLMs reasoning? Can we use these o1 tips to achieve better performance of VLMs?

A group of researchers from Peking University, Tsinghua University, Peng Cheng Laboratory, Peking University Shenzhen Graduate School, Alibaba DAMO Academy, and Lehigh University, motivated by the idea of making VLMs more efficient in complex reasoning, introduced LLaVA-o1. Despite the name, it is built on Meta’s Llama-3.2-11B-Vision-Instruct, not on LLaVA.

It’s a smart VLM that can reason step-by-step in a clear, structured way. Instead of jumping to answers, it divides reasoning into four stages and uses stage-level beam search, a specialized inference-time scaling method, to create multiple answers for each stage. This brings LLaVA-o1 closer to the kind of reasoning OpenAI’s o1 is known for.

Let's take a closer look at this model in more detail.

To make reasoning more structured and systematic, LLaVA-o1 breaks the process into four stages:

Each stage is clearly marked with special tags like <SUMMARY>...</SUMMARY>, so the model stays focused and avoids going off track. The stages happen automatically, and the user doesn’t need to guide the whole process. The first three stages are hidden from the user, while the conclusion is what the user sees. Whether you need a short answer or a detailed explanation, LLaVA-o1 will provide it.

This stage of processing organizes the reasoning of problems, but it alone isn’t enough to elevate a VLM to a new level of reasoning with high accuracy. This is where another feature of LLaVA-o1 comes into play.

Inference-time scaling is about making the model smarter when it’s actually solving tasks in real-time. LLaVA-o1 uses a new method called stage-level beam search to improve its reasoning process. At every reasoning stage, it generates and compares multiple possible answers and keeps the best one before moving forward to the next stage.

If we break this process down, it will look like this:

The stage-level beam search approach means each step of the final answer is built on the candidate the model judged strongest at that stage.

Let's examine LLaVA-o1's performance results and the factors they depend on.

When researchers trained the model directly on the original question-answer pairs from existing datasets, the results were weak: overall gains were modest, and on MMVet it even fell below the base model. Since existing datasets don’t explain reasoning well, researchers created a special LLaVA-o1-100k dataset to teach LLaVA-o1 to reason logically. This dataset includes about 100,000 question-answer pairs generated by GPT-4o, showcasing step-by-step reasoning examples.

Even though LLaVA-o1 was trained on only 100k examples, it showed clear improvements. Even without inference-time search, it scored 6.9% higher on average than the base model (with stage-level beam search, the paper reports 8.9%). This means it’s much better at general question answering tasks and avoiding hallucinations. The model improved the most on tasks that need reasoning, such as:

The structured tags, such as <SUMMARY> and <REASONING>, are essential because they help the model organize its thoughts. When the model was trained on the same data with these tags removed, performance dropped significantly.

When researchers compared inference approaches on the MMVet benchmark under comparable compute, they found that LLaVA-o1 with stage-level beam search outperformed other inference scaling methods:

And finally, LLaVA-o1 was tested against both open-source and closed-source models, including some much larger ones, on reasoning-focused versions of the benchmarks. The experiments showed that:

The advantages of the LLaVA-o1 performance have been mentioned earlier and in this section, so let’s summarize them in one place to make it easier to digest.

How does LLaVA-o1 stand out?

Here are the advantages of the LLaVA-o1 model compared to other VLMs:

But what about the disadvantages of LLaVA-o1?

Several potential limitations can become a problem for efficient implementation of LLaVA-o1. They include:

LLaVA-o1’s ability to think through problems in a structured way makes it more accurate and reliable than older models. It sets a new benchmark for models that combine language and vision, especially for tasks requiring deep reasoning. It also proposes an alternative approach to step-by-step reasoning (thinking through stages) and an efficient, scalable stage-level beam search method for refining the best answers. The future will show how these approaches can be expanded and what else can be added to VLMs to enhance their complex multimodal reasoning.

As for the question about rivaling OpenAI o1, here is the answer:

The paper doesn’t compare the two directly. While OpenAI’s o1 dominates textual reasoning, LLaVA-o1 excels in visual reasoning, introducing structured, step-by-step processes and stage-level beam search.

Though LLaVA-o1 outperforms some larger models in reasoning-heavy tasks, its smaller dataset and reliance on structured tags limit generalizability. It’s not a direct rival to OpenAI’s o1 but a significant step forward for Vision-Language Models, highlighting their complementary roles in advancing AI reasoning.

LLaVA-o1 is a research vision-language model that structures visual reasoning into stages and uses inference-time search to improve answers on multimodal reasoning tasks.

No. The names refer to different projects. LLaVA-o1 is an open research model built on Meta’s Llama-3.2-Vision (not on LLaVA, despite the name). OpenAI o1 is a separate proprietary reasoning-model line.

It generates a summary, interprets visual evidence, performs logical reasoning, and produces a conclusion. The original work explored stage-level beam search; the July 2025 LLaVA-CoT revision uses stage-wise retracing search (SWIRES).

The associated research and released artifacts provide an open research path, but users should verify the exact repository, checkpoint license, base-model terms, and allowed commercial use.

It can hallucinate visual details, produce unfaithful rationales, depend on benchmark-specific prompting, consume more compute during search, and lag newer multimodal models on some tasks.

LLaVA-o1 is a research vision-language model that structures visual reasoning into stages and uses inference-time search to improve answers on multimodal reasoning tasks.

No. The names refer to different projects. LLaVA-o1 is an open research model in the LLaVA family; OpenAI o1 is a separate proprietary reasoning-model line.

It generates a summary, interprets visual evidence, performs logical reasoning, and produces a conclusion. The original work explored stage-level beam search; the July 2025 LLaVA-CoT revision uses stage-wise retracing search (SWIRES).

The associated research and released artifacts provide an open research path, but users should verify the exact repository, checkpoint license, base-model terms, and allowed commercial use.

It can hallucinate visual details, produce unfaithful rationales, depend on benchmark-specific prompting, consume more compute during search, and lag newer multimodal models on some tasks.