Today, training is very, very RL-focused. The range of where reinforcement learning is used is enormous: alignment training optimizing models to match human preferences, AI evaluators, or verifiable outcomes; multi-step reasoning capability training; agent training, teaching models to plan actions, use tools, and operate in multi-step environments (coding agents, GUI agents); and even pre-training and system-level learning.

The motivation for using RL is largely practical, inspired by the success of many reasoning models. It allows us to optimize objectives as standard training losses and simply train models to maximize the defined reward signal.

But researchers are increasingly questioning whether reinforcement learning should remain the dominant post-training stage. Why?

So where exactly are the parts of a model that should be modified? How can we make post-training more efficient? The search for better training approaches is not over. RL’s limitations motivate exploring alternative post-training strategies in optimization and parameterization spaces, including parameter-efficient methods like LoRA (Low-Rank Adaptation) and gradient-free optimization approaches such as evolutionary methods.

Today we’ll discuss what can make post-training cheaper, more stable and dynamic, and more modular than RL-based pipelines. There is a lot to learn about the new LoRAs – Doc-to-LoRA, Text-to-LoRA, LoRA-Squeeze, Kron-LoRA and MoA (Mixture of Adapters) – plus a very interesting direction Evolution Strategies, and how they can fit together. Let’s unpack what is beyond RL.

In today’s episode:

Before we move on to exploring new ideas for fine-tuning, we need to clarify what training pipelines we use today, and where we can place new alternatives.

When we talk about LLM training, we mean a lifecycle divided into two broad phases: pre-training and post-training. And the distinction is not about when something happens chronologically, but about what objective is being optimized and what capabilities are being added to the model.

In general, a modern LLM pipeline can be seen as a following combination: pre-training → supervised fine-tuning (SFT) → alignment training → post-training optimization.

In SFT, the model is trained on prompt–response examples to learn how to follow instructions and generate structured answers. This is about how to behave as an assistant.

Alignment training stage attempts to align model outputs with human preferences or task-specific objectives. The most common approach has been RLHF (Reinforcement Learning from Human Feedback), where humans rank model outputs, a reward model learns those preferences, scores model’s outputs with a reward function, and then RL updates the model to maximize reward using gradient-based optimization.

More and more systems now replace human feedback with automated evaluation signals, such as math solutions, code execution, database queries, programmatic tests and others, leading to RLVR (Reinforcement Learning with Verifiable Rewards).

On top of this, AI models often include additional improvements: retrieval augmentation (RAG), reasoning training to produce multi-step reasoning traces, tool use and agent behavior training, long-context optimization, safety and calibration training and more.

And today, we are witnessing Low-Rank Adaptation (LoRA) methods evolve into something new. Latest approaches like Doc-to-LoRA, Text-to-LoRA, LoRA-Sqeeze and others suggest that post-training is no longer just about running RL on top of a base model. It is becoming a modular stack, where some behaviors are optimized through rewards, while others are injected instantly through generated adapters. Fine-tuning starts to look less like one monolithic training stage and more like a dynamic system for composing capabilities.

Basically, LoRA changes how model parameters are updated during post-training. Early LoRA approaches treated adapters purely as fine-tuning mechanisms. They were trained on datasets to specialize a model for a task.

In standard LoRA fine-tuning, the base model is frozen and only small low-rank matrices are trained: W' = W + BA. Here, W is the original weight matrix and BA is a low-rank update whose rank is usually between 4 and 64. So instead of modifying billions of parameters, the training procedure modifies a few million.

In practice, LoRA is often used for domain specialization, instruction tuning, reasoning tasks and alignment experiments. So the typical workflow pipelines would be: Pretraining → Instruction fine-tuning → Alignment / capability training → Parameter-efficient fine-tuning (LoRA).

Now let’s break down how the latest LoRA methods work and what new angles of post-training they reveal.

Recent works from Sakana AI on Doc-to-LoRA and Text-to-LoRA significantly expand the idea of generated adapters. They work similarly but in different directions: