AI 101: The State of Reinforcement Learning in 2025

Reinforcement Learning (RL) – a topic that’s constantly present in 2025. Ever since DeepSeek-R1 kicked off the year with a major reasoning breakthrough, researchers and developers have increasingly turned to RL-based training as a way to strengthen reasoning and agentic behavior. We saw hundreds of research on RL everywhere: in LLMs, VLMs coding domain, agentic systems, robotics, and beyond.

Unlike supervised learning (depends on labeled datasets), or unsupervised learning (searches for patterns without guidance), RL is built on interaction. The system learns by doing, by nudging the environment and watching how it reacts. Rewards, penalties, surprises, and mistakes shape the policy more than any static dataset ever could. In other words, it pushes models closer to real-world behavior.

But is reinforcement learning really as strong as it looks right now? This year also surfaced a set of concerns about the strategy.

Today we’re zooming out to see what RL achieved in 2025, where it ran into walls, and what momentum it brings into 2026. Let’s begin the full year-in-review.

In today’s episode, we will cover:

Reinforcement learning is a type of machine learning where a model learns through trial-and-error interactions with an environment. Traditionally, RL centered on building explicit agents that take actions and receive rewards. The landscape shifted this year: RL techniques moved well beyond classical agent settings, and showed up across model training, optimization, reasoning, and even data curation and as Richard Sutton and David Silver argue in their Era of Experience framework, this shift is structural: the next phase of AI will be driven not by human data, but by agent experience.

At its core, the setup is simple. The agent takes an action, observes the outcome, and receives a reward or penalty based on how good that action was. Over time it tries to learn a policy that picks actions leading to high long-term reward. RL problems are usually framed as a Markov Decision Process (MDP) with the following elements:

When RL works with LLMs:

RL algorithms aim to find the optimal policy, meaning the best possible strategy for earning the most reward over time. RL lets models learn from types of feedback that are not differentiable, such as human opinions or preference comparisons. It is especially useful for two things:

2025 taught us a lot about both parts.

Let’s start with alignment. Models can hallucinate, produce biased or unsafe content, and stumble on complex instructions. Alignment matters because it nudges AI toward being more helpful and more consistent, whether it’s a chatbot holding a coherent conversation or a code model finally passing its unit tests.

Reinforcement Learning from Human Feedback (RLHF) became the default alignment strategy for LLMs in 2025. It typically follows three steps:

However, there are some difficulties with human feedback: it is expensive and slow to collect. That's why, in our era of AI-powered optimization, what has also become popular is Reinforcement Learning from AI Feedback (RLAIF).

RLAIF follows the same steps as RLHF, but instead of collecting human preference data, it replaces or supplements human evaluations with AI evaluators that judge model outputs. These evaluators may be:

The reward model is then trained to match the AI feedback, and different optimization algorithms are used to fine-tune the LLM just like in RLHF.

Benefits of RLAIF:

But there are some limitations too:

One of the interesting examples of RLAIF is Curriculum-RLAIF that helps the reward model generalize better and produce stronger alignment during RL. It trains reward models the same way you’d teach a student: start with simple, clear-cut examples and gradually introduce harder, noisier ones. Easy pairs are generated using guided prompts, hard pairs – using random sampling, and medium-difficulty “bridge” pairs in between. These pairs are sorted from easy to hard and fed to the reward model in that order.

Beyond alignment, RL has also been used to boost reasoning abilities of models, leading to the emergency of the entire class of Reasoning Models, with even early examples like DeepSeek-R1 and Kimi-1.5.

The beginning of 2025 shocked everyone with the release of DeepSeek-R1 on January 20. It was the first model to deliver advanced reasoning on par with OpenAI’s o1 (at that time) and make it available to the public. For the first time, we could actually watch the reasoning unfold, since the open model produced long, detailed chains of thought that showed how it arrived at its answers. The backbone for this was the Group Relative Policy Optimization (GRPO) algorithm, which sits inside a broader training approach that strengthens the model’s reasoning ability called Reinforcement Learning with Verifiable Rewards (RLVR).

RLVR is a part of the current trend of moving from subjective alignment, like humans labeling outputs, to objective alignment, using rewards that can be proven correct or automatically checked, like math or code.

Here is how RLVR works:

Researchers from LeapLab (Tsinghua University) and Shanghai Jiao Tong University decided to test whether RLVR is actually as impressive as many claim.

In their paper called “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”, they tested RLVR-trained models across different model families, RL methods, and benchmarks for math, coding, and visual reasoning and evaluated them by checking pass@k at very large k values – when the model is allowed many chances to sample a correct solution.

Their findings challenge the common assumptions:

When we allow multiple samples, the base models actually outperform their RLVR-trained versions. This suggests that RLVR models lean on reasoning paths the base model already knew how to produce. They don’t discover fundamentally new strategies. RLVR mostly shifts the sampling distribution so those paths appear more often.

Coverage and perplexity analyses confirm this: the reasoning steps produced by RLVR models already exist somewhere in the sampling distribution of the base model. All of this leads to one conclusion: RLVR model’s abilities come from and are limited by the base model itself.

But later, researchers from Microsoft Research Asia and Peking University published a “Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs” paper which argues with these results.

Oh we love a good paper fight.

What did they do? By repeating pass@k experiments for math and code tasks, the researchers show that RLVR solves problems the base model cannot, even with many tries.

They also noted though that pass@k can be misleading on math tasks, because it scores only the correct answers, not correct reasoning. Sometimes the base model gives wrong reasoning but still lands on the right final answer by chance. So they proposed a new metric, CoT-Pass@K, which checks if both the final answer and the reasoning steps (CoT) are correct. Using this stricter metric, they observe a clear improvement after RLVR – reasoning itself gets better.

Their main insight on why RLVR works is this: pretrained LLMs already know a lot about what good reasoning looks like, but RLVR increases the probability of those correct reasoning paths because they align with the reward for correct, verifiable answers. Even if the reward only checks the final answer, the gradient still pushes the model toward generating valid reasoning steps.

They also found that:

So the verdict from this study is that RLVR does expand the reasoning abilities of models. It’s best understood through one key idea: RLVR works when it pushes the model toward verifiable reasoning paths, not just correct answers.

This paper argues that using an LLM as a reasoning judge is a promising direction and that we need better benchmarks to evaluate how reliable these LLM judges are.

As you see, it all comes down to the evaluation issue again.

For convenience, let’s summarize how to distinguish the core RL approaches:

Since we’ve already touched on the criticism of RL, let’s go to →

To understand this, we need to revisit Karpathy’s explanation of how RL works.

For example, suppose a model needs to solve a math problem. In RL, it will first try many approaches and variations in parallel. These hundreds of attempts can be quite complex. Once the model produces an answer, it compares it with the correct one and receives a reward if it matches. The model then infers that the entire trace of decisions (or possibly several traces) led to the correct outcome. But the thing is — and this connects to a broader question about when models should stop reasoning altogether:

“Every single token gets up-weighted – do more of this. People will say that your estimator has high variance, but it’s just noisy,” Karpathy says. He notes that RL does two main things that human would never do:

So the question remains open: How far can the classic reward-model + policy-gradient framework go? One direction that addresses interpretability directly is Natural Language Reinforcement Learning (NLRL) — an approach that redefines RL's core concepts like policy, value function, and Bellman equation entirely in natural language.

It was definitely the year when dozens of policy optimizations algorithms could appear in just one week. Policy optimization refers to RL methods that directly adjust the parameters of a policy to increase expected return (the sum of rewards over time). These methods use gradient-based updates to push the policy toward outputs that yield higher rewards while maintaining stability (e.g., via clipping or trust regions). Here are a few policy optimization approaches that stand out the most:

DeepSeek-V3.2 applies GRPO with stability fixes – an unbiased KL estimator, off-policy masking, and consistent MoE routing. It unifies reasoning, agentic behavior, and alignment into one RL stage. After distillation from specialists, the model is refined with large-scale RL using rule-based and generative rewards, enabling reliable scaling and combining many skills in a single model.

If models improve their performance when scaled, why not implement this idea in RL? Researchers from Princeton University and Warsaw University of Technology demonstrate that scaling RL networks much deeper up to 1024 layers can greatly boost the performance of self-supervised, goal-conditioned reinforcement learning. Using Contrastive RL (CRL) with residual blocks, LayerNorm, and Swish activations, they train agents in sparse-reward Brax and MJX environments and find large gains:

CRL learns by pulling together embeddings of real (state, action, goal) triples and pushing apart mismatched ones. Deeper models develop richer representations and more capable behaviors once they cross certain “critical depth” thresholds. Greater depth also allows larger batch sizes and better generalization, although it offers limited benefit in offline RL where exploration is missing.

Microsoft Research proposed a way to scale language models by reframing next-token prediction as a small RL reasoning task. The model is trained with RL and receives a verifiable reward each time it predicts the correct next token from the text corpus. This approach allows RL to scale across massive web-text datasets without human labels or hand-crafted reward functions.

RPT rewards the model for explaining why a token should follow, which encourages deeper next-token reasoning and builds a stronger foundation for later RL stages. A newer approach, PretrainZero, uses reinforcement active pre-training: the model selects which spans to mask and then learns to predict them using two RL-trained policies. This turns raw text into a self-supervised RL signal and bypasses the “verification data wall” created by limited ground-truth answers. By gradually increasing difficulty with harder masks, PretrainZero improves general reasoning and math skills even before post-training RL begins.

The idea behind it is to train LMs to balance multiple goals at once, like being helpful but also concise or both harmless and creative, ensuring that improving one goal doesn’t ruin another. RLHF can’t do this because it optimizes a single reward. One of the interesting approaches here is PAreto Multi-Objective Alignment (PAMA) which reframes alignment as a convex optimization problem using a PPO variant (Noon PPO) that only applies positive-advantage updates. PAMA converges to a Pareto-stationary point, improving multiple objectives at once – for example, boosting sentiment and length rewards by 30–50% and increasing harmlessness by +0.5 to +1.0 over baselines.

RL shows up in every domain now. Since it began as a framework for agents, agentic RL became the natural direction of progress in 2025.

The power moments are seen in:

RL works not only for individual agents but can be also applied to multi-agent systems (MAS) to train agents jointly with the main optimization methods like PPO, GRPO and others.

Interest in robotics surged this year, and robots continue to learn through a mix of reinforcement learning and imitation learning (behavior cloning). RL trains a robot to learn a control policy that maximizes cumulative reward within an MDP, using algorithms such as TRPO (trust-region updates), PPO (clipped policy optimization), and SAC (an off-policy method that maximizes reward plus entropy). Training typically begins in simulation, with domain randomization to shrink the reality gap, while real-world learning improves through off-policy data reuse and human demonstrations.

Robots rely on tight feedback loops to manage noise and complex dynamics, and human feedback often mirrors RLHF. Human-in-the-loop systems like HIL-SERL have reached success rates near 99 percent on challenging manipulation tasks within one to two hours on low-cost robots.

That covered where and how RL is applied today. But across all these systems there are still crucial challenges the community is trying to solve.

The state of RL in 2025 is one of optimistic progress mixed with a clear view of what still needs work. This year’s advances – LLM alignment via RL, RLVR in reasoning models, autonomous code-generation improvements, and more capable robotic planning – created real momentum. RL helped produce agents that can reflect and reason in ways that felt out of reach only a few years ago. Moving into 2026, the trend line points toward reinforcement learning becoming a more reliable, integrative layer inside AI systems rather than an isolated technique.

At the same time, questions about whether RL is the right long-term strategy for broad AI development remain on the table. The field wants better optimization across every step of a model’s solution path so we avoid amplifying weak reasoning and burning unnecessary compute. Verification is becoming the central pillar: stronger verifiers, more interpretable reasoning traces, and clearer signals that can steer models without destabilizing them. Much of the 2026 work is likely to focus on this intersection of optimization, verification, and stability.

And here’s the bigger perspective. RL has deep roots going back to the 1950s. It could stay central, or it could end up as one piece in a larger family of algorithms. The community is increasingly open to that possibility. The real promise of 2026 isn’t a bet on “better RL,” but an environment where RL mixes with new architectures, new objective functions, and new ways of shaping model behavior. It’s a year for trying ideas that don’t fit neatly into today’s playbook.

In 2026, we get to reopen the frontier and experiment again.

Policy Optimization Techniques

Resources from Turing Post