Most AI today runs on general-purpose hardware. To be more concrete: GPUs. For years, it felt borderline unrealistic to crack the ecosystem NVIDIA built around them. Then something interesting happened, and it happened fast.

While this year is still young, we’ve got some really exciting development in the hardware news. And it’s focused on inference. This January, at CES, NVIDIA introduced its next platform, Vera Rubin, as a rack-scale AI supercomputer made out of six chips, tuned for inference-heavy, communication-heavy workloads. Last week, Taalas raised $169 million and is demoing a product that takes the most extreme inference bet in the market: instead of running a model on hardware, it turns a specific model into hardware (model-as-hardware concept). And only yesterday, MatX raised a $500 million Series B for MatX One, an LLM-first accelerator that is ambitious on paper and still a dark horse in practice because it is not shipping yet (but has a giants backing it).

All together it demonstrates that the competition now happens on the operational realities of inference: latency, cost per token, power, and the physics of moving parameters and KV cache around. GPUs remain the baseline. The crack is in the assumption that one baseline will serve every inference regime forever.

In today’s episode, we will do three things:

If you want the takeaway upfront: MatX is a bold programmable challenger with a long runway. Taalas is the most different idea in the market, and it deserves a closer look because it forces you to ask a hard question: how stable does a model need to be before it makes sense to etch it into silicon?

Training is the glamorous part of the AI lifecycle. It is also the part that looks the most like classic High-Performance Computing (HPC): huge matrix multiplies, enormous clusters, performance measured in throughput at scale.

Inference is where the invoice shows up: once a model ships, you run it millions of times in the messy, user-driven world that benchmarks barely approximate. Prompts get weird, context windows bloat, and the system stack keeps accreting retrieval, tool calls, and multi-agent coordination, each layer adding latency, cost, and fresh new ways to fail. Even a “single response” can turn into a chain of generation steps and intermediate reasoning, which changes what you are paying for.

This shift matters because the bottlenecks shift too. Token generation is sequential at the user level. You can parallelize many things, but you still produce outputs token by token. The model’s “knowledge” lives in its weights, and inference constantly streams weights and reads and writes KV cache. When the workload is constrained by data movement, a design that reduces data movement can beat a design that does more compute, even if the compute is more flexible.

NVIDIA’s own Rubin messaging makes this explicit: inference is now communication-heavy and system-bound, so the platform story centers on rack-scale interconnect, predictable latency, high utilization, and infrastructure for context and data movement.

(If you want to refresh you knowledge about chip types, here is our full guide on AI hardware, where we explained how the main types work)

A GPU is still a simple idea at heart: lots of parallel compute cores, fed by high-bandwidth memory, executing a program compiled from software. Inside a modern data-center GPU, you will find billions of transistors etched onto one or more silicon dies, arranged into many lightweight processing cores. Those cores sit next to a hierarchy of on-chip SRAM – registers, shared memory, and large L2 cache – which acts as a high-speed staging area and tries to catch reuse before the GPU has to reach out to external memory.

The cores are also paired with stacked High-Bandwidth Memory (HBM), usually sitting on the same package. HBM itself is built from vertically stacked DRAM layers connected through a silicon interposer so data can move quickly between memory and compute.

When you run a language model on a GPU, the model’s weights live in that memory. The GPU keeps fetching them as it generates tokens. Intermediate results shuttle between SRAM and HBM. Attention reads and writes its KV cache. It is a powerful, flexible pipeline, and it is also why inference can become expensive: moving the model’s own “stuff” is often the limiting factor.

First, they fit the math of deep learning. Matrix and tensor operations parallelize well, and GPUs are built to split a big job into thousands of smaller tasks and compute them simultaneously.

Second, they are programmable. A model is software in the GPU paradigm. You can run many models on the same hardware, swap checkpoints, tune kernels, change frameworks, and keep your infrastructure useful while the model landscape mutates.

Third, they come with an ecosystem. NVIDIA’s software stack (CUDA) has been a compounding advantage for over a decade. Even when a competing chip has an interesting architecture, it still has to earn its place in real deployment pipelines.

Vera Rubin is NVIDIA treating the rack as the unit of performance and efficiency. NVIDIA describes the Rubin platform as “six new chips” that together form an AI supercomputer, and positions the flagship Vera Rubin NVL72 rack as a rack-scale accelerator within a larger AI factory. Production ramps in the second half of 2026, which is also why “Blackwell vs challengers” already reads like last season’s debate.

Here are the Rubin facts that matter for an inference-centric conversation:

This doesn’t make Blackwell “bad.” It makes Vera Rubin a clearer description of where NVIDIA believes the next bottleneck is: system-level memory and interconnect, sustained utilization, and the practical conversion of megawatts into usable tokens. It also clarifies what challengers have to do: they do not need to beat Rubin everywhere. They need to dominate a specific inference regime and make the tradeoffs acceptable.

Taalas is one of the rare cases where a hardware company forces you to re-evaluate what “deployment” even means. Their bet is simple to state and hard to execute →