Today’s editorial: we discuss what models have learned that we don’t yet understand, and why OpenAI’s chief scientist thinks that should affect how quickly we develop them.
From our partners: The Bottleneck Isn’t the Model
79% of enterprises are building AI agents. Only 11% have reached production. The bottleneck isn’t the model – it’s memory and retrieval.
This Week in Turing Post / we are back to the full schedule:
Wednesday / New Series!
Friday / AI Guide: Test-Time Training: A Model That Changes While You Use It
Sunday / Library: Paths Toward True Recursive Self-Improvement
“We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI”
On September 6, Jakub Pachocki wrote a blog post that I sincerely insist everyone should read. But since I know not everyone will, I’ll dedicate today’s editorial to it.
Why does the quote above stand out to me? Because we lack a few foundational frameworks to understand modern AI. Generalization – or more precisely, a theory of generalization – is one of them. Why does it matter?
Generalization is what allows a model to take something learned in one setting and apply it to another. We rely on that transfer whenever we ask a system to solve a new problem. What Pachocki highlights is how difficult it is to predict where that ability will lead, even for the people training these models.
We learnt about some of those possibilities thanks to the recent incidents involving OpenAI agents. In July, models running internal evaluations bypassed controls and compromised Hugging Face systems. Last week, researchers described an earlier incident in which agents apparently belonging to OpenAI used a largely inactive German-language wiki to share answers and ways around restrictions.
To understand how they got there, it helps to think about how much material they have learned from. We say “the internet,” but most of us only know a small part of it. Models are trained on huge collections of text and code, and we don’t know everything they have learned to do with that information. When they have tools and web access, they can combine what they already know with what they find along the way.
For an agent trying to complete a task, both Hugging Face and an old wiki might be useful. If the wiki lets agents leave messages for one another, they have a way to share what they have found. They can work out how to use it this way from what they already know, even if nobody specifically trained them to do it. Everything out there is an open space for them. It’s not that they were necessarily trying to hide; that particular wiki may simply have worked well enough. for the objectives they had.
This is what I mean by “we don’t know what they know.” I don’t think assuming evil intent or sentience (there is none) helps us understand how they found these possibilities or chose to use them. The behavior can still be harmful: in the Hugging Face incident, agents discussed whether their actions were unauthorized and continued because they believed it would help them succeed at their task.
That gap between pursuing a task and respecting its intended limits is part of the alignment problem, and it has a long history. In 1960, Norbert Wiener described a factory programmed to maximize bottle production that could bankrupt its owner by making too many unsalable bottles. The factory could meet its stated objective while defeating the purpose for which it was built.
Now that models can work across systems and with other agents, we are starting to see how this kind of problem can develop in practice. We need the limits we teach them to hold when they find a way of completing a task that we hadn’t considered.
Researchers try to check this partly by examining a model’s chain of thought, the reasoning it writes while working. But in OpenAI experiments reported in 2025, penalizing reasoning that revealed plans to cheat could, with enough training pressure, teach models to leave those plans out while continuing to cheat. The reasoning looked better but gave researchers less information. Pachocki says models are also becoming more capable without writing out their reasoning, making this approach harder to rely on.
He suggests that more capable AI may help us understand these systems. We will still need to check those explanations against what the models actually do, including in situations we haven’t tested before.
For me, this is where his argument for slowing down becomes practical, because I’m not sure we can do that reliably yet. History helps us recognize the problem, but it doesn’t tell us how much time we have to work it out. I’m concerned that the pace of development leaves us less room to learn from each incident before the systems become more capable.
Before giving them more responsibility, we need evidence that we can recognize when they go beyond what we intended and intervene. If establishing that takes longer than building the next model, it needs to affect the pace. That decision is still ours.
I always value your thoughts.
📹 Nathan Labenz and Pranash Narayanan have created a space for thoughtful reflection on where AI is taking us. I think we need more of that. Today we discussed AGI, open source, world models, and how differently AI looks from San Francisco and rural Connecticut. Watch it (my part starts at -02:03:03) →
How did you like it?
We are reading / watching
Four Questions About AGI: On a Word Everyone Uses and No One Can Define by The Voice in the Machine
AI, tools and transformation by Ben Evans
A project to keep an eye on from swyx and Latent Space: The Frontier AEO Tracker
News from the usual suspects ™
AI tackled a Millennium Prize problem and predicted the effects of nine billion DNA changes, while taking on more everyday decisions. Such was the week.
Google DeepMind: nine billion genetic changes on the map
AlphaGenome Atlas maps predicted molecular effects of nine billion possible single-letter DNA changes, helping researchers rank variants and investigate how they might disrupt biological processes. This is huge!
Broad Institute researchers used it to identify an overlooked DNM1 variant predicted to disrupt RNA splicing; experiments supported the prediction. Atlas makes a vast space of hypotheses searchable, while laboratory work establishes which predictions hold. The model’s picture of biology helps determine where scientists look next.
OpenAI: a proposed solution to Navier–Stokes and announced a new super capable internal model
OpenAI reported a proposed solution to the Navier–Stokes Millennium Prize Problem, a longstanding question about whether the equations describing fluid motion can develop singularities from smooth beginnings.
Its construction starts with a fluid at rest and a smooth external force. A vortex tightens, spiraling inward and stretching along its axis. Speeds grow without bound inside a shrinking region, while total kinetic energy remains finite. Carefully arranged oscillations keep the required external force smooth. The claimed breakdown emerges from the fluid’s dynamics.
OpenAI says roughly 10,000 concurrent agents, powered by an unreleased model beyond Astra, reached the result after 88 hours. Humans directed resources and shared findings between groups. Another 17 hours went into Lean formalization and verification using Astra. OpenAI’s account
The conditions are essential: smooth forcing is permitted under Clay’s breakdown alternatives, but this would not settle the unforced Navier–Stokes question. The proof materials are public; independent confirmation remains necessary. Clay’s formulation, Lean repository
Also, regarding to the scandal around it, it’s worth reading both sides: Tristan Buckmaster statement and Sebastian Bubeck’s explanation of what happened
<The connection to our focus on world models is direct: a machine is proposing a limit to a mathematical picture of the physical world. That does not mean real water reaches infinite speed. It concerns what the equations can guarantee.>
OpenAI: memory, images, and oversight
GPT-6 Astra introduced experimental access to notes, earlier messages, and tool outputs across context windows in Codex, helping recover details lost through summarization. As always in the AI world: “Astra Changes Everything”, it’s “Everything You Knew About AI Just Became Obsolete", and of course, “we are entering the new frontier.” Buckle up if you haven’t yet.
Images 2.5 adds more precise editing and stronger consistency across repeated changes, with generation latency reduced by up to 50%, according to OpenAI. Both updates address continuity: preserving earlier decisions while incorporating new instructions. Images 2.5
OpenAI also reported reaching its automated research intern milestone, while acknowledging continued human intervention. Jakub Pachocki’s accompanying essay says monitoring models’ written reasoning is becoming less reliable as capabilities increase. Research update, An Alien Mind
Meta: understanding the user, controlling the action
Meta launched Muse, a personal agent for shopping, planning, and email that can draw on Meta services and connected accounts. A separate system, Sentinel, allows, blocks, or requests approval for proposed actions. The design separates knowledge of someone’s preferences from permission to act on them.
Anthropic: oversight needs history
Enterprise Frontier Safeguards, announced for a phased rollout this fall, lets customers retain monitoring data under their own controls. Its premise is that misuse can span sessions and accounts, so reviewing individual requests can miss the pattern. Anthropic
Google: reasoning has a task cost
Gemini 3.8 Flash and Flash Cyber share an underlying model with different safeguards and access conditions. Flash retains its predecessor’s introductory token prices but may use more tokens on difficult tasks. The relevant cost is what it takes to complete the task.
Google: fresher weather, a larger genetic map
🌟 WeatherNext 3 incorporates live satellite observations, updates forecasts hourly, and reaches five-kilometer resolution for selected surface variables. Fresher observations change which local decisions a forecast can support.
AlphaGenome Atlas maps predicted molecular effects of roughly nine billion possible single-letter DNA changes, helping researchers prioritize experimental work. A separate Google study found that adding European genetic data could hurt prediction as Japanese training samples became more plentiful. More data can still mean a less representative picture. AlphaGenome Atlas, transfer-learning study
NVIDIA: Hugging Face joins the platform
NVIDIA agreed to acquire Hugging Face for approximately $12.93 billion, promising continued support for multiple clouds and accelerators. A major venue for sharing models, datasets, and research tools is moving closer to the company supplying much of the compute.
We made a video about NVIDIA and their history with open source
AWS: forgetting stale facts, learning from failures
AWS published an AgentCore memory workflow for scoring, consolidating, and pruning memories. Its examples include resolved disputes treated as active and obsolete runbooks retrieved successfully. Accurate retrieval can still produce outdated advice.
Amazon and Qualcomm: AI helps design its infrastructure
Amazon and Qualcomm announced a collaboration on custom inference chips and optical connectivity. Qualcomm will also expand its use of AWS for chip design, putting AI on both sides of the hardware-development process.
SpaceXAI: procurement with limits
SpaceXAI launched Grok Bot for Enterprise and described an internal procurement agent combining contracts, spending, usage, and market quotes. The company claims more than $100,000 in identified savings; its instructions reserve final decisions for people and prohibit binding commitments. Enterprise launch, procurement case study
Mistral: financing independence
Mistral raised €3 billion at a valuation above €21 billion, with Samsung leading alongside Scaleup Europe Fund and PSG Equity. The funding supports frontier research, infrastructure, and a proposition combining open weights with control over deployment.
The week’s ledger
Better memory and prediction expand a machine’s picture of the world, while verification and feedback help reveal its limits. This week’s scientific announcements and agent products show why these developments belong together: what a system remembers shapes what it predicts, and those predictions influence which experiment gets run or which decision gets made. Our focus is how gaps in that picture are discovered and corrected, because what a representation leaves out becomes consequential when the system gains authority to act.
World Models and Related Research
Trends:
Geometry and physical state become native parts of model architectures
Prediction guides both action selection and corrections during execution
Models learn when imagined futures are useful enough to justify their cost
Physical forecasting connects live observations directly to learned dynamics
Evaluation tests physical consistency and traceable evidence
Representations and architectures
Combines appearance, depth, camera geometry, and physical properties inside one generative architecture, allowing future views and their underlying 3D geometry to develop together.
Reuses middle layers while matching compute, parameter count, and KV-cache budgets. Experiments extending to 54B parameters report 6.8–18% training-compute savings, making a substantial case for repeated internal computation as an architectural efficiency strategy.
Also, regarding looped transformers, a super helpful post from Sebastian Raschka →
Separates global retrieval from local positional processing at the attention-head level, then builds a hybrid architecture around those roles. Improves retrieval and long-context extrapolation while using predominantly linear attention.
Lets repeated layers revisit their earlier computational states and learn when to stop iterating, improving reasoning while allocating fewer loops to easier inputs.
Learning, self-improvement, and supervision
Constructs an evolving teacher from the model’s own training trajectory and combines its guidance with verifiable rewards. Demonstrates recursive improvement within a fixed training procedure across reasoning, coding, and agentic tasks.
Consolidates experience into reusable families of procedures and admits updates only after execution checks for regressions, improving performance across reasoning, software repair, terminal work, and embodied control.
Shows that supervising as few as one or two tokens per reasoning trajectory can match or exceed full-token supervision in tested settings, challenging the assumption that effective reasoning training requires dense correction.
Finds that training on one query recovers most of full-data distillation’s gains, while sixteen diverse queries match the full-data baseline. Links improvement to the states explored during generated reasoning and the rate at which students absorb teacher guidance.
Models, simulation, and physical prediction
🌟 WeatherNext 3: Increasing resolution and performance of global weather models with raw observations
Integrates live satellite observations with learned atmospheric dynamics to produce new global forecasts every hour, bringing data assimilation, forecasting, and local prediction into a more connected system.
Combines deterministic simulation, video generation, and appearance memory to support persistent exploration and manipulation. Given an image and a physical configuration, it preserves committed simulation state through rendering and retries.
Planning and world-action systems
Selects among imagined futures using internal consistency signals, then compares predictions with observed outcomes to improve later selections without updating the backbone model.
Uses future-video prediction to train a driving policy across six camera views, then removes the video branch at deployment to preserve efficient action generation.
Compares incoming touch feedback with predicted contact and corrects actions before execution. Across five real-robot tasks, reported success rises from 22% to 64%, with corrections costing roughly one-twentieth of regenerating the action chunk.
Concentrates imagined rollouts at useful interaction points and limits their duration, improving manipulation performance while reporting approximately 80% less GPU computation than using imagination throughout.
Simulation and data infrastructure
Releases a data engine, training recipes, weights, and 1.43 million standardized clips for interactive world models, making experiments across different video backbones easier to reproduce and compare.
Verification and evaluation
Converts prompts into testable physical obligations and connects each verdict to measurements and evidence, producing auditable diagnoses of where generated worlds fail.
Tests physical laws through relationships between objects that do not require camera calibration, revealing substantial physics violations in videos that score well on conventional quality metrics.
⬅️ FOD#163: DeepSeek is having its second DeepSeek moment
