Most AI today runs on general-purpose hardware. To be more concrete: GPUs. For years, it felt borderline unrealistic to crack the ecosystem NVIDIA built around them. Then something interesting happened, and it happened fast.

While this year is still young, we’ve got some really exciting development in the hardware news. And it’s focused on inference. This January, at CES, NVIDIA introduced its next platform, Vera Rubin, as a rack-scale AI supercomputer made out of six chips, tuned for inference-heavy, communication-heavy workloads. Last week, Taalas raised $169 million and is demoing a product that takes the most extreme inference bet in the market: instead of running a model on hardware, it turns a specific model into hardware (model-as-hardware concept). And only yesterday, MatX raised a $500 million Series B for MatX One, an LLM-first accelerator that is ambitious on paper and still a dark horse in practice because it is not shipping yet (but has a giants backing it).

All together it demonstrates that the competition now happens on the operational realities of inference: latency, cost per token, power, and the physics of moving parameters and KV cache around. GPUs remain the baseline. The crack is in the assumption that one baseline will serve every inference regime forever.

In today’s episode, we will do three things:

If you want the takeaway upfront: MatX is a bold programmable challenger with a long runway. Taalas is the most different idea in the market, and it deserves a closer look because it forces you to ask a hard question: how stable does a model need to be before it makes sense to etch it into silicon?

Training is the glamorous part of the AI lifecycle. It is also the part that looks the most like classic High-Performance Computing (HPC): huge matrix multiplies, enormous clusters, performance measured in throughput at scale.

Inference is where the invoice shows up: once a model ships, you run it millions of times in the messy, user-driven world that benchmarks barely approximate. Prompts get weird, context windows bloat, and the system stack keeps accreting retrieval, tool calls, and multi-agent coordination, each layer adding latency, cost, and fresh new ways to fail. Even a “single response” can turn into a chain of generation steps and intermediate reasoning, which changes what you are paying for. For the fundamentals of LLM inference optimization — TensorRT, quantization, and hardware choices — see our guide to optimizing LLM inference.

This shift matters because the bottlenecks shift too. Token generation is sequential at the user level. You can parallelize many things, but you still produce outputs token by token. The model’s “knowledge” lives in its weights, and inference constantly streams weights and reads and writes KV cache. When the workload is constrained by data movement, a design that reduces data movement can beat a design that does more compute, even if the compute is more flexible.

NVIDIA’s own Rubin messaging makes this explicit: inference is now communication-heavy and system-bound, so the platform story centers on rack-scale interconnect, predictable latency, high utilization, and infrastructure for context and data movement.

(If you want to refresh you knowledge about chip types, here is our full guide on AI hardware, where we explained how the main types work)

A GPU is still a simple idea at heart: lots of parallel compute cores, fed by high-bandwidth memory, executing a program compiled from software. Inside a modern data-center GPU, you will find billions of transistors etched onto one or more silicon dies, arranged into many lightweight processing cores. Those cores sit next to a hierarchy of on-chip SRAM – registers, shared memory, and large L2 cache – which acts as a high-speed staging area and tries to catch reuse before the GPU has to reach out to external memory.

The cores are also paired with stacked High-Bandwidth Memory (HBM), usually sitting on the same package. HBM itself is built from vertically stacked DRAM layers connected through a silicon interposer so data can move quickly between memory and compute.

When you run a language model on a GPU, the model’s weights live in that memory. The GPU keeps fetching them as it generates tokens. Intermediate results shuttle between SRAM and HBM. Attention reads and writes its KV cache. It is a powerful, flexible pipeline, and it is also why inference can become expensive: moving the model’s own “stuff” is often the limiting factor.

First, they fit the math of deep learning. Matrix and tensor operations parallelize well, and GPUs are built to split a big job into thousands of smaller tasks and compute them simultaneously.

Second, they are programmable. A model is software in the GPU paradigm. You can run many models on the same hardware, swap checkpoints, tune kernels, change frameworks, and keep your infrastructure useful while the model landscape mutates.

Third, they come with an ecosystem. NVIDIA’s software stack (CUDA) has been a compounding advantage for over a decade. Even when a competing chip has an interesting architecture, it still has to earn its place in real deployment pipelines.

Vera Rubin is NVIDIA treating the rack as the unit of performance and efficiency. NVIDIA describes the Rubin platform as “six new chips” that together form an AI supercomputer, and positions the flagship Vera Rubin NVL72 rack as a rack-scale accelerator within a larger AI factory. Production ramps in the second half of 2026, which is also why “Blackwell vs challengers” already reads like last season’s debate.

Here are the Rubin facts that matter for an inference-centric conversation:

This doesn’t make Blackwell “bad.” It makes Vera Rubin a clearer description of where NVIDIA believes the next bottleneck is: system-level memory and interconnect, sustained utilization, and the practical conversion of megawatts into usable tokens. It also clarifies what challengers have to do: they do not need to beat Rubin everywhere. They need to dominate a specific inference regime and make the tradeoffs acceptable.

Taalas is one of the rare cases where a hardware company forces you to re-evaluate what “deployment” even means. Their bet is simple to state and hard to execute →

inference is important enough that it deserves fully specialized chips, one optimized for a specific model.

Taalas proposes a conceptual shift: instead of accelerating models on hardware, hard-code the model itself into hardware. Their term for this is Hardcore Models, abbreviated HC. The name is a bold choice, and you can almost hear the person in the room who suggested “Model-as-Silicon” getting politely ignored. Though, we think that Model-as-Hardware is a better term.

What makes Taalas different is not that it uses an Application-Specific Integrated Circuit (ASIC). Many companies use ASICs. Taalas takes ASIC logic and applies it at the level of a specific neural network instance.

Taalas assembles a nearly complete chip with roughly 100 layers, then performs the final customization on two metal layers to hardwire a particular model, with TSMC taking about two months to complete fabrication of a chip customized for a specific model. That is incredibly fast.

And this matters because it turns “one model per chip” from a philosophical statement into an operational loop. Full custom silicon cycles are long. If you can change two layers on a largely prefabricated base and do it on a two-month cadence, you have something closer to a hardware compiler pipeline, with a new strategic question attached: what is the minimum stability threshold where it becomes rational to tape out a model?

Taalas’ first released system, HC1, is optimized for Llama 3.1 8B. From what we know, the entire 8B model fits on a single 815 mm² die (TSMC N6), with weights stored on-chip in a mask-ROM recall fabric, and SRAM used for KV cache and for updates such as LoRA. The announced speed is 16,000-17,000 per second per user. If you try it yourself on chatjimmy.ai, you can see how this model on a chip builds a game for 0,06 seconds. It’s faster then you can think.

For comparison, the latest fast Codex-Spark from OpenAI is 1200 tokens per second.

Taalas frames the core principle as transforming a model into custom silicon, with Hardcore Models that support fine-tuning and are faster and lower power than software-based implementations. But how do they actually do it?

The Taalas story is a memory story.

In the standard GPU workflow, the model is software, weights live in HBM, and inference repeatedly fetches weights and moves intermediate values through a stack of abstractions: kernels, scheduling, memory hierarchies, interconnect. That is flexible and powerful. It is also overhead.

Taalas collapses the abstraction layers. If the model’s weights and structure are physically realized on-chip, you bypass a large share of memory traffic and scheduling complexity. There is no dynamic model loading. There is no need to keep HBM fed with weights from outside. There is no need to schedule kernels for a model that never changes.

The main shift in this architecture is that the chip stops running the model and becomes the model itself.

Taalas’ method borrows from structured ASIC approaches, where changing only a couple of masks can alter the circuit, allowing Taalas to hardwire a model without a full redesign each time. It’s not new in semiconductor history, but no one has ever done it in AI.

If your bottleneck is memory bandwidth and data movement, this kind of architecture can produce an outlier outcome.

The tradeoffs here are not “engineering inconveniences.” They are the core of the bet.

Taalas makes the most sense when you can say “yes” to all of the following:

If your reality is constant model iteration and a zoo of models, GPUs remain the simpler answer. Taalas can still be useful for a subset of the stack, but it will not replace a general-purpose fleet. That is why this is interesting: inference hardware is splitting by workload regime, and no single architecture is going to look optimal across all of them.

Here is the analytical overview of Taalas hardware. Don’t forget to watch it to get the full picture →

If Taalas is extreme specialization, MatX is closer to the classic dream: build a programmable chip that outperforms GPUs for LLM workloads without forcing you into one-model-per-silicon commitments.

MatX just raised a $500M Series B and says the funding will “wrap up development and quickly scale manufacturing,” with tapeout in under a year. TechCrunch reports MatX’s goal is to make its processors 10× better at training LLMs and delivering results than NVIDIA’s GPUs.

MatX’s positioning for MatX One is explicit about the pain points it wants to fix:

So why call it a dark horse? Because what exists today is the promise, the funding from big names like Jane Street and Karpathy, and the architectural intent, while the deployable reality is still ahead. MatX’s own statements frame the product as not yet shipping, and public reporting points to 2027 as the start of shipping and deployment. A lot can happen between “tapeout in under a year” and “a system you can run 24/7 with a real compiler, real kernels, reliability, and support.”

MatX still deserves attention because it represents the programmable challenger category with serious capital behind it. Its promise is familiar: give teams a chip that feels like a better GPU for LLMs. Its challenge is also familiar: silicon is hard, and ecosystems are harder.

Taalas and MatX are the headline contrasts: extreme specialization versus ambitious programmability. The rest of the market is more diverse than a list of company names suggests. A better map is to group by what each architecture is optimizing.

Google’s Ironwood is hyperscaler logic in its purest form: build custom silicon, deploy it at enormous scale, and offer it through the cloud.

Google introduced Ironwood as its seventh-generation TPU and described it as the first TPU designed specifically for inference, aimed at “thinking” and inference-heavy AI models at scale. Google’s TPU documentation describes TPU7x as the first release in the Ironwood family, designed for large-scale training and inference, including dense and MoE models and decode-heavy inference, scaling up to 9,216 chips per pod.

Ironwood matters in this story because it shows where the biggest operators are placing their bets: inference gets dedicated silicon, and scaling happens as a system design problem, not a single-chip spec fight. In October, 2025 Google’s TPUs had a big moment: Anthropic signed a deal for up to 1 million TPUs – it is over a gigawatt of compute – making it one of the biggest AI infrastructure agreements to date.

AWS takes a different hyperscaler approach: build inference silicon as a cost-performance lever inside a broader cloud platform. Inferentia is not trying to replace GPUs everywhere. It is trying to offer a cheaper, efficient inference path for a large share of production workloads.

AWS’ launch materials for Inf2 instances powered by Inferentia2 claim up to 50% better performance per watt than comparable EC2 instances. AWS’ machine learning blog describes Inferentia2 as delivering up to 4× higher throughput, up to 10× lower latency, and up to 50% better performance per watt than comparable inference-optimized instances.

And you can’t pass by AWS because it is a default deployment target for a huge share of the industry. Even if Inferentia captures only part of that market, it changes what “reasonable” inference cost looks like inside the cloud.

For in-depth explanation of these ASICs and other types of hardware the exist today, check out this guide → AI 101: Intelligence Processing Unit (IPU) and other alternatives to GPU/TPU/CPU

Groq built its identity around low-latency inference through a deterministic execution model. Groq’s Language Processing Unit (LPU), designed specifically for LLM inference, falls exactly into the category of ASICs.

The notable update is that NVIDIA signed a non-exclusive licensing agreement with Groq for Groq’s inference technology, and Groq’s founder Jonathan Ross and president Sunny Madra joined NVIDIA as part of that deal.

From a competitive standpoint, this is revealing. NVIDIA is still selling the most advanced GPU platforms in the world, and it still chose to license inference technology and hire leadership from a specialized inference startup. That makes it hard to treat inference specialization as a side project.

d-Matrix represents a design philosophy that keeps gaining momentum: if inference is bandwidth-bound, put compute in or next to memory.

In its Corsair technical whitepaper, d-Matrix describes Corsair as an integrated compute-and-memory PCIe card with 3200 mm² of silicon, 2 GB on-chip “Performance Memory” delivering 150 TB/s memory bandwidth, and up to 256 GB off-chip capacity memory, with dense compute figures reported in MXINT formats.

Even if you treat every marketing number with suspicion, the architectural intent is clear: reduce off-chip memory movement and make token generation cheaper and faster by keeping the hot path on-chip. This direction is also a practical response to HBM realities: it’s expensive, power hungry, and constrained by supply.

Cerebras remains the canonical radical form-factor story. Its WSE-3 press release lists 4 trillion transistors, 900,000 AI cores, 125 petaflops of peak AI performance, and 44 GB on-chip SRAM on a wafer-scale chip built on TSMC 5nm.

Cerebras is more often discussed in the training context, but it is relevant to inference for the same reason it is relevant everywhere: it attacks coordination overhead and memory locality by collapsing a massive amount of compute and SRAM onto a single device.

It is tempting to write the dramatic version where the GPU monopoly ends in a single season. The reality is messier and more interesting: the GPU era is fragmenting by workload regime, and inference is the first place where that fragmentation becomes economically meaningful.

Vera Rubin is NVIDIA’s answer: treat inference as a system problem and optimize rack-scale utilization, interconnect, and context handling as first-class design constraints. MatX is a challenger bet that tries to beat GPUs on their own turf: programmable acceleration for LLM workloads, with architectural choices aimed at utilization, latency, and long context. Taalas is the most different bet: if inference for a model becomes stable enough, you can treat the model as hardware and remove a large share of overhead, at the price of flexibility.

These bets can all be true at the same time because they are optimizing different contracts between the model and the machine.

The GPU era is still here, but it is becoming less monolithic.

NVIDIA is moving up the abstraction ladder: Vera Rubin frames the rack as the accelerator and treats context and interconnect as part of the product. Hyperscalers are doing the same with their own silicon and their own system-level architectures. Startups are exploring the space in between, from MatX’s programmable LLM-first ambitions to Taalas’ extreme specialization.

If you want to watch where this goes next, don’t stare at peak FLOPs. Watch cost per token, latency under long context, and the shape of real production stacks. That is where the next generation of inference hardware winners will be decided, one painfully expensive infrastructure bill at a time.

Don’t forget to watch this video about Taalas to understand this new wave in the hardware more thoroughly.

Resources from Turing Post