11 New Interesting Policy Optimization Techniques (2026)

Policy optimization is one of the most exciting topics for the AI community right now. Why? Mostly because of reinforcement learning (RL) popularity. Policy optimization is the way to apply RL – you directly train the policy (the model’s behavior) using rewards. The most widely-used policy optimization methods are PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), and their extensions, but there is always much to learn because a lot of new approaches appear almost every week.

In 2026, the center of gravity is shifting from single-turn RLVR (Reinforcement Learning from Verifiable Rewards) and GRPO-style optimization toward agentic training: multi-turn environments, tool calls, reward decomposition, DAPO-style variants, and policy methods that can handle longer decision chains.

So let’s discover 11 quiet recent policy optimization methods:

Agentic Turn-based Policy Optimization via Tree Search (AT²PO)

This technique from Tencent is a way to train AI agents on multi-step tasks by organizing decisions into turns, exploring options with tree search, and assigning rewards per turn so learning becomes more stable and effective. → Agentic Turn-based Policy Optimization via Tree Search

Turn-PPO (Turn Proximal Policy Optimization)

A PPO-based training method by Amazon for multi-turn AI agents. It treats each interaction turn as a decision step, improving stability and long-horizon reasoning compared to token-level or GRPO-based approaches. → Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs

Instruction-Policy co-evolution (INSPO)

Optimizes both the agent’s policy and the instructions it uses, evolving instructions dynamically with policy learning for better multi-turn reasoning. → Agentic Policy Optimization via Instruction-Policy Co-Evolution

Group reward-Decoupled Normalization Policy Optimization (GDPO)

by NVIDIA keeps multiple rewards separate during learning, preventing them from collapsing, and helping models learn user preferences more accurately and stably. → GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

Puzzle Curriculum GRPO (PC-GRPO)

A way to train vision-language models using self-supervised visual puzzles instead of labels, with a difficulty-based curriculum that gives more stable rewards and helps models reason more consistently. → PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs

Momentum-Anchored Group Relative Policy Optimization (M-GRPO)

Stabilizes self-supervised RL by anchoring the policy update with a momentum model and filtering low-entropy trajectories to avoid premature collapse. → M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization

Tool-Augmented Policy Optimization (TAPO)

Integrates reasoning and adaptive tool usage in one RL framework, enabling models to interleave reasoning tokens with tool calls during optimization. → Tool-Augmented Policy Optimization: Synergizing Reasoning and Adaptive Tool Use with Reinforcement Learning

Progressive Reward Shaping (PRS) + Value-based Sampling Policy Optimization (VSPO)

Combines dense curriculum rewards, that guide models from correct tool usage to high-quality answers, and value-based sampling which stabilizes training by focusing updates on informative samples and avoiding flat or collapsed rewards. → Enhancing Agentic RL with Progressive Reward Shaping and Value-based Sampling Policy Optimization

Bottom-up Policy Optimization (BuPO)

Applies reinforcement learning directly to lower Transformer layers, using their intermediate token distributions to shape early exploration and reasoning, rather than only optimizing final outputs. → Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

VA-π (Variational Policy Alignment for Pixel-Aware Autoregressive Generation)

It is a post-training method that aligns autoregressive image generators with pixel quality. It treats generation as a policy and uses image reconstruction quality as a direct reinforcement learning reward. → VA-π

Distributional Value Modeling-based Policy Optimization (DVPO)

This one uses distributional token-level value estimates and risk-aware regularization to improve robustness and generalization under noisy supervision. → DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training

What is the difference between PPO and GRPO in LLM training?

PPO, or Proximal Policy Optimization, usually uses a learned value model or critic to estimate advantages and stabilize policy updates. GRPO, or Group Relative Policy Optimization, removes the critic and compares multiple sampled outputs within a group, using relative rewards as the training signal. PPO can be more robust in complex multi-turn environments, while GRPO is simpler and popular for RLVR-style reasoning tasks.

What are the newest policy optimization methods for AI agents?

The newest policy optimization methods for AI agents focus on multi-turn credit assignment, tool use, reward shaping, and more stable GRPO variants. Examples include Turn-PPO for turn-level advantage estimation, AT²PO for tree-search-based turn optimization, INSPO for instruction-policy co-evolution, TAPO for tool-augmented RL, GDPO for multi-reward optimization, and M-GRPO for stabilizing self-supervised RL.

How does GDPO differ from standard GRPO?

GDPO differs from standard GRPO by decoupling reward normalization across multiple reward types. In standard GRPO, different reward combinations can collapse into the same advantage values, which weakens the learning signal. GDPO keeps rewards separate during normalization, so the model can better learn trade-offs between correctness, format, length, tool use, and other preferences.

What is Turn-PPO and why does it improve multi-turn RL?

Turn-PPO is a PPO-based method that estimates advantages at the level of interaction turns instead of individual tokens. This makes it better suited for agentic tasks where each turn is a meaningful decision and earlier actions affect later outcomes. It improves multi-turn RL by giving the model more stable credit assignment across long-horizon environments.

Which policy optimization method is best for agentic LLMs?

There is no single best method for all agentic LLMs. For multi-turn interaction, Turn-PPO and AT²PO are strong candidates. For tool-use agents, TAPO or PRS + VSPO are more relevant. For multi-reward alignment, GDPO is useful. For self-supervised stability, M-GRPO is a better fit. The best choice depends on whether the agent needs long-horizon planning, tool use, multiple rewards, or safer training dynamics.

Why are policy optimization methods changing in 2026?

Policy optimization is changing because LLMs are no longer trained only for single-turn answers. New AI agents need to browse, call tools, write code, revise plans, and act across multiple steps. That makes classic token-level optimization insufficient, pushing the field toward turn-level RL, agentic credit assignment, reward shaping, tool-augmented optimization, and more stable GRPO-style variants.