Token 1.5: From Chain-of-Thoughts to Skeleton-of-Thoughts, and everything in between

Chain-of-Thought prompting, or CoT prompting, is a technique that encourages a language model to work through intermediate reasoning steps before producing its final answer. Unlike a standard prompt that asks directly for an output, a CoT prompt asks the model to decompose a problem and approach it step by step.

The groundbreaking paper “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, published by Google researchers in 2022, helped establish one of the most influential ideas in modern artificial intelligence: giving a model more room to reason can improve its performance on tasks involving arithmetic, logic, planning, and multi-step decision-making.

But it didn’t stop there. The original paper inspired a large family of methods, including Self-Consistency, Automatic Chain-of-Thought, Program-of-Thoughts, Tree of Thoughts, Graph of Thoughts, Algorithm of Thoughts, and Skeleton-of-Thought.

In chronological order, we will unfold the Chain-of-Thought lineage and explain:

No more confusion around CoT. Use this updated dictionary as a reference for understanding how these methods work, when to apply them, and how each differs from a standard prompt.

Chain-of-Thought prompting is a prompting method in which a language model produces a sequence of intermediate steps before reaching its final answer. The basic idea is simple: instead of asking only for the answer, the prompt demonstrates or requests a reasoning process that connects the input to the output.

A standard prompt might ask:

A Chain-of-Thought prompt also demonstrates the calculation:

This extra structure can help the model maintain intermediate information and avoid skipping necessary operations. It is most useful for tasks involving several dependent steps, rather than simple retrieval, classification, or rewriting tasks that can be answered directly.

A standard prompt provides an instruction or question and asks the model to return the result. It is efficient for straightforward tasks such as summarization, extraction, translation, sentiment classification, or answering a factual question.

A Chain-of-Thought prompt introduces intermediate reasoning between the question and the final answer. Apply it when solving a problem requires decomposition, calculation, comparison, planning, or tracking several constraints. It may improve accuracy, but it also increases token use, latency, and inference cost.

Chain-of-Thought should also not be confused with retrieval-augmented generation. RAG supplies external information to a model, while CoT structures how the model reasons over the information it has received.

A system can use both. Retrieval-augmented generation retrieves relevant documents, while a reasoning method evaluates, compares, or combines the evidence.

The term zero-shot prompting derives from the concept of zero-shot learning.

Zero-shot learning is a model’s ability to complete a task without having received task-specific training examples.

In prompting, this means the prompt does not contain demonstrations of the task. The model must rely on the knowledge it acquired during training and the information contained in the instruction.

Task: Named Entity Recognition

Prompt:

Response: Steve Jobs

Use zero-shot prompting for familiar and clearly defined tasks where examples would add little value. Compared with Chain-of-Thought, a standard zero-shot prompt does not explicitly ask the model to produce intermediate reasoning.

Few-shot prompting includes a small number of input-output examples before presenting a new task. These demonstrations help the model infer the required format, categories, or decision boundary through in-context learning.

In-context learning is a method in which demonstrations of a task are provided to a model as part of the prompt rather than through additional model training.

Task: Sentiment Analysis

Prompt:

Response: Positive

Use few-shot prompting when the task is unusual, the output must follow a specific pattern, or a zero-shot prompt produces inconsistent results. Unlike Chain-of-Thought prompting, ordinary few-shot examples show the expected answers but do not necessarily show how those answers were derived.

For tasks demanding sequential reasoning, merely providing a few examples may not be enough. The original Chain-of-Thought method extends few-shot prompting by including demonstrations with both answers and intermediate reasoning steps.

Instead of teaching only the input-output relationship, the examples demonstrate a process the model can imitate.

The authors of the original CoT paper found that this approach improved performance on arithmetic, commonsense, and symbolic reasoning tasks. Their strongest results came from PaLM, a model with 540 billion parameters.

That historical result is sometimes simplified into the claim that Chain-of-Thought works only with models containing approximately 100 billion parameters. That is no longer a reliable general rule.

Smaller models can acquire step-by-step reasoning behavior through fine-tuning, distillation, specialized datasets, and task-specific training. For example, the authors of Multimodal Chain-of-Thought Reasoning reported state-of-the-art ScienceQA results using a model with fewer than one billion parameters.

CoT still does not automatically improve every task. It tends to be most useful when:

For easy questions, a direct answer from a standard prompt can be faster and equally accurate.

While various prompting techniques exist, this review focuses on those intimately connected with chain-of-thought prompting. To ensure coherence, we'll proceed in chronological order to trace the evolution of ideas.

In March 2022, researchers introduced Self-Consistency, a decoding strategy designed to improve the reliability of Chain-of-Thought reasoning.

Rather than generating one definitive reasoning path, Self-Consistency samples several different paths and selects the answer that appears most consistently across them. The intuition is that a complex problem may have multiple valid solution paths, while an incorrect answer is less likely to recur across independently generated reasoning chains.

Apply Self-Consistency when a difficult problem can be solved through several approaches and a single generated chain may contain a random error. Compared with a standard CoT prompt, it can improve reliability, but it requires multiple generations and therefore increases inference cost and latency.

The process consists of four main steps:

Self-Consistency changes the response-generation stage rather than the underlying prompt. It replaces greedy decoding, which selects one path, with sampling and aggregation.

A few months after the original CoT paper, researchers from the University of Tokyo and Google published “Large Language Models Are Zero-Shot Reasoners.”

The paper showed that adding a simple phrase such as “Let’s think step by step” could improve model performance on several reasoning benchmarks without providing manually written demonstrations.

Apply Zero-Shot CoT when you need a quick reasoning prompt but do not have high-quality examples available. Compared with an ordinary zero-shot prompt, it adds an explicit reasoning instruction. Compared with few-shot CoT, it requires less prompt engineering but usually gives you less control over the reasoning format.

Importantly, Zero-Shot CoT does not replace the entire Chain-of-Thought framework. It is one variant that removes the need for worked examples.

The authors of “Automatic Chain of Thought Prompting in Large Language Models” proposed a way to automate the construction of CoT demonstrations.

The original few-shot CoT approach required humans to write examples containing questions, reasoning chains, and answers. Auto-CoT reduces this manual effort by clustering representative questions, selecting examples from different clusters, and using Zero-Shot CoT to generate candidate reasoning chains.

Apply Auto-CoT when you have a dataset of representative tasks and want to construct reusable few-shot prompts at scale. Compared with a standard CoT prompt, it automates demonstration creation, although the generated examples still need validation because incorrect reasoning can affect subsequent outputs.

The original method works in three stages:

Diversity is important because examples drawn from different clusters are less likely to repeat the same reasoning mistake.

The authors released the official Auto-CoT implementation on GitHub.

Program-of-Thoughts Prompting separates semantic reasoning from numerical computation.

Instead of asking the language model to perform every calculation in natural language, the model expresses the computational part of the solution as a program. An external interpreter, such as Python, executes that program and returns the result.

Apply Program-of-Thoughts to tasks involving precise arithmetic, equations, financial analysis, or repeated numerical operations. Unlike standard CoT, which asks the model to perform both reasoning and calculation in text, PoT delegates exact computation to a tool.

This can make results easier to verify and reduce arithmetic errors, but generated code should be executed in a controlled environment. Inputs, code, and outputs must also be validated before the result is used in production.

The official Program-of-Thoughts repository includes the code and datasets used in the paper.

Multimodal Chain-of-Thought extends step-by-step reasoning beyond text by incorporating visual information.

The original framework operates in two stages:

Apply Multimodal-CoT to tasks involving charts, diagrams, scientific illustrations, medical images, video, or spatial relationships that cannot be captured through text alone. Compared with a standard text prompt, the model must ground its reasoning in evidence from more than one modality.

The original model used fewer than one billion parameters and achieved state-of-the-art performance on the ScienceQA benchmark. The researchers released the official Multimodal-CoT code on GitHub.

The field has since expanded into image, video, audio, speech, 3D information, healthcare, robotics, and autonomous-driving applications, as documented in the 2025 Multimodal Chain-of-Thought Reasoning survey. A related approach, Whiteboard-of-Thought prompting, extends visual reasoning further by having multimodal models draw intermediate steps as images before producing an answer.

However, multimodal CoT should be applied selectively. A 2026 study of multimodal reasoning models found that CoT was more useful for mathematical, scientific, and multi-image reasoning than for basic visual perception tasks such as grounding and object counting.

Tree of Thoughts, or ToT, treats problem-solving as a search across branching possibilities.

Instead of committing to one linear sequence, the model generates multiple candidate intermediate states, evaluates them, and continues exploring the most promising branches. The framework can also look ahead or backtrack when an earlier decision leads to a weak solution.

Apply ToT when a problem requires planning, exploration, comparison among possible strategies, or recovery from an incorrect early decision. Compared with standard CoT, it can explore alternative paths rather than following one uninterrupted chain, but it requires more orchestration and substantially more computation.

A Tree of Thoughts system typically defines four components:

The task is divided into meaningful intermediate states or steps.

The model proposes one or more possible next steps from the current state.

Candidate states are scored, ranked, or compared through voting.

A procedure such as breadth-first search or depth-first search determines which states to explore next.

The authors tested ToT on the Game of 24, creative writing, and mini crosswords. The official Tree of Thoughts repository includes the prompts and implementations used in the experiments.

Graph of Thoughts, or GoT, represents intermediate reasoning units as vertices in a graph and their dependencies as edges.

Unlike a tree, a graph can merge branches, reuse previous results, combine information generated along separate paths, and introduce feedback loops. This allows the reasoning structure to reflect tasks that are not naturally linear or hierarchical.

Apply GoT to elaborate tasks where reasoning components interact nonlinearly, such as document synthesis, iterative refinement, sorting, ranking, or workflows that combine several partial results. Compared with a standard CoT prompt, GoT requires more state management, parsing, scoring, and coordination outside the model.

The original framework includes interacting modules such as:

These modules enable the system to generate, transform, evaluate, combine, and revise intermediate thoughts.

The authors released the official Graph of Thoughts implementation.

Algorithm of Thoughts, or AoT, uses algorithm-inspired demonstrations to guide a model through a structured search process.

Rather than repeatedly stopping and restarting the model while an external framework explores a tree, AoT places examples of an algorithmic reasoning pathway directly into the context. The model can then imitate the procedure in one or a small number of queries.

Apply AoT when a problem has a clear search structure and can benefit from procedures resembling breadth-first or depth-first search. Compared with standard CoT, AoT imposes a more deliberate algorithmic pattern. Compared with Tree of Thoughts, it aims to reduce the number of separate model calls and the associated computational overhead.

AoT should not be interpreted as proof that the language model executes an algorithm exactly. It provides an algorithm-like template that guides exploration, and the resulting answer still requires validation.

The project’s official website and implementation resources provide examples and additional material.

Skeleton-of-Thought, or SoT, was developed primarily to reduce the latency of long language-model responses.

Instead of producing an answer sequentially from beginning to end, the method first asks the model to generate a compact skeleton or outline. The individual points are then expanded in parallel and assembled into the final response.

Apply SoT to long-form answers that can be divided into relatively independent sections, such as reports, explainers, research summaries, and structured documentation. Compared with a standard prompt or linear CoT response, SoT is designed mainly to improve generation speed rather than explore competing reasoning paths.

The method has two stages:

SoT is less suitable when later sections depend heavily on reasoning or conclusions produced in earlier ones.

The authors released the official Skeleton-of-Thought implementation.

Chain-of-Thought began as a prompting technique, but the underlying idea has moved into model training and inference.

Modern reasoning models may be trained or reinforced to spend additional computation on difficult tasks, revisit earlier steps, test alternatives, use external tools, and produce an answer only after an extended reasoning process.

This means users do not always need to add “Let’s think step by step.” A model may already perform an automatic chain of thought or another form of internal reasoning as part of its normal inference procedure. Requesting a longer explanation can sometimes add unnecessary tokens without improving the answer.

The practical distinction is now:

These approaches frequently overlap.

Not necessarily.

A generated rationale can be useful for checking calculations, identifying assumptions, or making an answer easier to audit. But it should not automatically be treated as a faithful record of the model’s internal computation.

The 2025 paper “Reasoning Models Don’t Always Say What They Think” found that reasoning models do not consistently disclose factors that influence their answers. Other research has also shown that a model can produce a plausible explanation that omits relevant influences or rationalizes an answer after it has effectively been selected.

This is particularly important in healthcare, finance, safety, and other high-stakes applications where a fluent explanation can create false confidence.

For production systems, verification should rely on external evidence and checks, including:

Visible reasoning is evidence to inspect, not proof that the answer is correct.

Use Chain-of-Thought or one of its variants for tasks involving:

A standard prompt is usually preferable for simple extraction, rewriting, classification, translation, direct factual retrieval, or tasks where additional explanation does not improve the result.

The trade-off is straightforward: reasoning methods can improve performance on difficult problems, but they usually consume more tokens, cost more, and take longer to run.

Choose standard Chain-of-Thought for a relatively straightforward multi-step problem.

Choose Zero-Shot CoT when you need step-by-step reasoning but have no demonstrations.

Choose few-shot CoT when the reasoning format or domain requires carefully designed examples.

Choose Auto-CoT when you need many demonstrations and want to reduce manual effort.

Choose Self-Consistency when one reasoning path may be unreliable and the value of greater accuracy justifies multiple generations.

Choose Program-of-Thoughts when exact calculation matters more than natural-language computation.

Choose Multimodal-CoT when the problem depends on images, charts, video, or other non-textual information.

Choose Tree of Thoughts when the model must explore alternatives and backtrack.

Choose Graph of Thoughts when intermediate results need to be merged, reused, or revised through feedback loops.

Choose Algorithm of Thoughts when a known search procedure can be demonstrated efficiently.

Choose Skeleton-of-Thought when the primary goal is to generate a long, structured answer with lower latency.

Chain-of-Thought-inspired methods have been applied across translation, planning, knowledge grounding, question answering, finance, science, and multimodal reasoning.

Selected papers include:

Chain-of-Thought prompting is a technique that encourages a language model to produce intermediate reasoning steps before giving its final answer. It is most useful for tasks requiring arithmetic, logic, planning, or several dependent decisions.

A standard prompt asks directly for an output, while a Chain-of-Thought prompt requests or demonstrates a step-by-step reasoning process. CoT can improve difficult reasoning tasks but generally uses more tokens and takes longer to run.

The phrase is associated with Zero-Shot Chain-of-Thought. It signals that the model should decompose the problem before answering, although modern reasoning models may not need this exact wording.

Automatic Chain-of-Thought, or Auto-CoT, creates reasoning demonstrations automatically. It reduces the manual effort required to construct few-shot CoT prompts by selecting representative questions and generating candidate reasoning chains.

Yes. The original 2022 research found the strongest benefits in very large models, but smaller models can acquire reasoning capabilities through fine-tuning, distillation, specialized datasets, or multimodal training. Performance still depends on the model, task, and quality of the reasoning examples.

No. Chain-of-Thought organizes the model’s reasoning, while retrieval-augmented generation supplies external information. They can be combined when a model needs both relevant evidence and a structured method for evaluating it.

No. It is most effective for genuinely multi-step tasks. For simple questions, weak models, misleading demonstrations, or poorly specified problems, CoT may add cost without improving the answer and can even produce convincing but incorrect reasoning.

Chain-of-Thought can make an answer easier to inspect, but a written rationale is not guaranteed to reflect the model’s actual internal process. Important outputs should be checked using sources, tools, calculations, validators, or human review.

Chain-of-Thought follows one mostly linear reasoning path. Tree of Thoughts generates and evaluates multiple branches, allowing the system to explore alternatives, reject weak paths, and backtrack.

Use standard or Zero-Shot CoT for ordinary multi-step reasoning, Self-Consistency for greater reliability, Program-of-Thoughts for calculations, Tree or Graph of Thoughts for search and complex orchestration, and Skeleton-of-Thought for parallel generation of long structured answers.

Chain-of-Thought prompting changed how researchers and engineers approach language-model reasoning. What began as a method for adding worked examples to a prompt has grown into a broader family of techniques for sampling, searching, calculating, verifying, and reasoning across multiple modalities.

But CoT is no longer synonymous with simply typing “Let’s think step by step.” Modern systems combine trained reasoning, test-time compute, external tools, retrieval-augmented generation, verifiers, and structured search.

The right method depends on the task: use additional reasoning where it improves the result, and keep the standard prompt when the problem does not need an intellectual parade.

Thank you for reading. You can leave your feedback in the comment section.