AI 101: Reinforcement Learning: The Ultimate Guide to Past, Present, and Future

In 1988, Richard Sutton introduced Temporal-Difference (TD) learning – a method for computers or agents to learn to make predictions about the future by using past experience even when the system they are in isn’t fully known. Unlike traditional prediction methods that compare predictions to actual outcomes (for example, supervised learning), TD learning uses intermediate hints. It doesn’t wait for the final outcome, it compares successive predictions and updates incrementally, learning a little bit every step. An easy example: Monday’s weather guess for Saturday can already be adjusted on Tuesday, using Tuesday’s improved guess.

You can see TD as minimizing prediction error through gradient steps. The key difference is the target used:

TD learning approach appeared to be a more accurate one, because it avoids being misled by chance. Imagine a board game, where a new position that leads into a usually bad state might still end in a lucky win. Taking this “win“ result into account, supervised learning would wrongly label it as good, while TD learning links it only to the bad state and so learns correctly.

So TD learning has notable advantages:

In 1992 the concept of temporal-difference (TD) learning was used in an important RL milestone – TD-Gammon, a neural network created by IBM researcher Gerald Tesauro and trained to play backgammon. It played about 300,000 games against itself, where every game generated experience: board states, moves, outcomes. With this “knowledge”, TD-Gammon steadily improved its judgment of moves and reached near-professional strength.

Also, many years later, in 2015, Richard Sutton and Brian Tanner extended TD learning idea into TD networks, connecting multiple predictions – each prediction can depend on others in the future. These systems allow time-based predictions (e.g. what will happen 5 steps later) and can build predictive state representations when the environment isn’t fully observable.

Straight after the TD learning, in 1989, Christopher Watkins and Peter Dayan invented the Q-learning model-free RL algorithm. As Sutton’s TD learning it also learned just from experience and didn’t need a full model or map of the world.

In Q-learning, the agent keeps track of the “quality”, updating Q-value, of taking certain actions in certain states. Over many episodes, as the agent repeatedly explores all state-action pairs, the Q-values gradually converge to the optimal action-values. Once converged, the agent can act, choosing the action with the highest Q-value in each state.

Later, in 1994, SARSA algorithm, proposed by G.A. Rummery and M. Niranjan, came out. It is an on-policy TD control method that learns action-values (Q), using the action taken under the current policy. It uses the full sequence of experience: (State, Action, Reward, Next State, Next Action), which gives the name S-A-R-S-A. The process goes like this:

SARSA keeps learning while also improving the policy bit by bit. Over time, if every state–action pair is explored enough, the algorithm converges to the optimal policy.

So SARSA and Q-learning are the illustrative examples of on and off-policy RL algorithms:

Apart from the TD learning branch another important, model-free path was formalizing “learn from experience” concept – and it is Monte Carlo methods.

While TD methods update predictions step by step using other predictions (bootstrapping), Monte Carlo (MC) methods learn directly from the actual return – the sum of rewards, collected when an episode (the sequence of steps until the agent reaches the goal) finishes.

Typical MC method follows the general policy iteration (GPI) framework:

MC algorithms’ strong point is that they only need samples and learn purely from real outcomes, averaging real interaction with the environment.

MC naturally led to the next step in the RL story, but this time training the policy directly. That’s where REINFORCE comes in.

Everyone is now urgently exploring REINFORCE algorithm (1992, Ronald Williams). It is because of today’s most popular Group Relative Policy Optimization (GRPO) that seems very close to it. (That’s why at Turing Post we explore history and earlier methods – they’re the key to today’s machine learning.)

So what is this REINFORCE?

It is a Monte Carlo policy-gradient method that also runs episodes, uses full-episode returns as the learning signal, and nudges policy parameters to make rewarding action choices more likely. And more precisely – REINFORCE is a family of RL algorithms for neural networks with randomness built in. They adjust the network’s weights after each trial (episode step) based on:

The update rule is: Weight change = learning rate × (reward – baseline) × eligibility.

This formula explains the name REINFORCE: REward INcrement = Nonnegative factor × (reward – baseline) × Characteristic Eligibility.

REINFORCE is an important basic method because:

Monte Carlo and TD learning methods focus either on learning value functions of states (how good it is to be there) or state-action pairs (how good it is to take that move in that spot), or direct policy updates. But there is another way to perform RL that includes using critic models.

Researchers in the late 1970-1980s (notably Barto, Sutton, and Anderson) developed actor–critic architectures, which combined the two parts:

The two work together: the critic provides feedback (a temporal-difference error) that tells the actor how to adjust its policy parameters – strengthen or weaken the choice of action. Over time, this back-and-forth helps the agent improve its behavior.

This approach overcomes the high variance of policy gradients and the limits of value-based methods, which don’t directly optimize policies.

Actor-critic methods work well when there are too many possible actions to check one by one. Plus it doesn’t always pick the same move, but assigns probabilities to actions.

Actor-critic algorithms became the foundation for many modern methods, such as Advantage Actor-Critic (A2C), Asynchronous A3C, Deep Deterministic Policy Gradient (DDPG), and also one of the most popular methods now – Proximal Policy Optimization (PPO).

The RL history of the 20th century laid the foundation, and the 21st century brings not only breakthroughs but also reveals limitations: what works and what doesn’t in the race for efficiency. Now we move to this new 21st century stage.

In the 2000s, RL research continued steadily, though without the fanfare of later decades. A major focus was how to scale RL to larger, more realistic problems with function approximation, stabilizing policy optimization, and exploring hierarchy. Some of the examples are Natural Policy Gradient (2001), Least-Squares Policy Iteration, LSPI (2001), R-MAX (2002), Neural Fitted Q Iteration (2005).

By the late 2000s, advances in GPUs and large datasets transformed supervised learning with deep neural networks, laying the foundation for the new stage of RL – deep reinforcement learning (fascinating story of ImageNet+GPU+AlexNet is covered here).

This phase began with the founding of DeepMind in 2010 by Demis Hassabis, Shane Legg, and Mustafa Suleyman. Their breakthrough came with the Deep Q-Network (DQN), first presented in 2013 and made famous by the 2015 Nature paper. DQN combined Q-learning with convolutional neural networks to approximate the Q-value function and play Atari games directly from raw pixels. Here is how it works:

DQN was the first deep RL system to reach human-level play across Atari games, surpassing earlier methods and even expert humans. It proved that deep networks could power modern RL.

Also, around 2015, policy gradient and actor–critics methods started to be updated with the new efficient ideas. Different approaches like Trust Region Policy Optimization (TRPO), Deep Deterministic Policy Gradient (DDPG), Asynchronous Advantage Actor–Critic (A3C) appeared and in 2017 the outstanding Proximal Policy Optimization (PPO), introduced by OpenAI, came out.

Why is PPO so popular?

It’s main feature is stability. It updates policies in small, controlled steps by clipping probability ratios, preventing large and risky changes. PPO combines policy improvement, value estimation, and an entropy bonus for balanced exploration, and uses Generalized Advantage Estimation (GAE) to smooth reward signals. This approach is simple, robust, and effective across diverse tasks, and that’s why it became one of the most widely used algorithms in current history of RL.

Meanwhile, DeepMind was creating a bunch of development, showing the huge potential of RL in different application fields.

Strict RL breakthroughs

RL-inspired and adjacent systems

These breakthroughs caught a lot of attention during a decade and shows the progression of RL capabilities over years. But what reinforcement learning in Lange Language Models?

The dominance of LLMs opened remarkable generative capabilities, but still pointed out at a challenge: Can they predict not just text, but text that’s useful, safe, and aligned with intent? That is where RL entered center stage. Ideas developed for games, control, and simulated environments were applied to human preference alignment.

The concept of Reinforcement Learning from Human Feedback (RLHF) became a defining moment for aligning LLMs with human values with the following strategy:

This pipeline was popularized by OpenAI in the training of InstructGPT (2022) and became the industry’s go-to method. We have a detailed overview of RLHF here.

The fact that “RL is usually PPO” hints at the next wave of developments. PPO relies on a separate critic network, which adds memory and compute overhead during training. This spurred researchers to explore lighter alternatives – critic-free or baseline-efficient methods like DPO, GRPO, and RLOO – to make RLHF training more efficient and scalable.

One of the most popular now is DeepSeek’s Group Relative Policy Optimization (GRPO) that returns back to policy-gradient–style thinking.

GRPO can be thought of as a modern REINFORCE-style algorithm, tailored for LLMs. Instead of updating based on a single trajectory and reward, GRPO compares groups of model outputs against each other. The group-relative advantage provides a variance-reduced learning signal, closer in spirit to the “reward minus baseline” update rule of REINFORCE. This approach:

In one of our previous episodes we covered GRPO workflow with every details, so you can explore it here.

However, last weekend GRPO caused a huge debate in the community, expressing doubts that it actually works properly. We also explored why this happened in our last newsletter.

Both PPO and GRPO are a part of a huge family of policy optimization techniques that is expanding almost every week with the newcomers. We gathered some of them in our post on Hugging Face:

Other developing areas on RL now are:

So what’s for the future?

From Thorndike’s puzzle-box cats and Bellman’s dynamic programming to PPO, GRPO, and beyond, reinforcement learning has always returned to the same principle: intelligence emerges through trial, feedback, and adaptation. But where to move further?

Andrej Karpathy, a leading voice in deep learning, ex-OpenAI founding member and now building Eureka Labs, remains optimistic about agent-based learning but is cautious about current RL methods. He points out that reward functions – the basis for RLHF and policy gradients – are weak and hackable, 'super sus,' as he put it. They don’t generalize to open-ended reasoning and conversation. He suggests that RLHF can feel more like preference matching than true problem-solving.

Still, Karpathy notes that reinforcement fine-tuning has made models more practical than supervised training alone, and he expects its role to grow, though he questions how far the classic reward-model + policy-gradient framework can go.

Looking ahead, he believes breakthroughs will require new paradigms: interactive environments and experimental ideas like system-prompt learning, where models adapt by rewriting their own context instead of just updating weights.

Richard Sutton’s critique of LLMs in recent conversation with Dwarkesh Patel also highlights a fundamental divide between supervised prediction and RL. LLMs optimize next-word prediction on static text, but lack explicit states, actions, and rewards, and so can’t adapt through consequences. Sutton contrasts this with RL agents, which learn by interacting with environments. His analogy “reading every cookbook vs. actually cooking” illustrates the real gap between imitation and grounded understanding. He warns that LLMs as “offline learners” may plateau on human-written data, while RL agents, as “online learners,” can continue to adapt and generalize.

This raises a central research question: will intelligence emerge from scaling static predictors, or from agents that learn through interaction? What about Causal AI and Symbolic AI? And what might the next stage of RL look like as we move further into continual trial-and-error learning?

An absolute golden time for researchers to finally combine it all.

Resources from Turing Post