Transcript 0:00 Etched is faster. That line caught my attention when I was scrolling their website. Ha, that's bold. Faster how? At raising money, definitely. Is this the fastest billion dollars ever raised by a chip startup? 0:13 Quite possibly. On July 23rd, Etched raised $300 million at a valuation of 10.3 billion. 26 day later, less than a month, it raised another 700 million at a valuation of $21 billion. 0:29 And the second round was led by Jane Street, a major global quant trading firm and liquidity provider. These guys do not [censored] around. 0:37 Jane Street runs tens of thousands of GPUs, writes optimized CUDA, builds compilers and custom hardware, and operates machine learning systems on microsecond trading horizons. 0:47 A beautiful vision and publicity means little there unless it survives contact with latency, and Jane Street said it tested Etched its chip. 0:56 Etched shipped its first rack to Jane Street's data center, then Jane Street led the round. So yes, Etched is faster at raising money. Whether it is faster at inference is this $21 billion question. 1:09 [upbeat music] Attention Span. Welcome to Attention Span by Turing Post. I'm Ksenia. 1:18 Let's understand what Etched built, why Jane Street is such an important signal, and exactly where this still tiny company can sufficiently hurt giant Nvidia. First, what is inference? Training is how a model learns. 1:32 Inference is everything it does afterward, every prompt answered, image generated, every trade evaluated, and action taken by an agent. Training happens when a model is created. 1:43 Inference happens every time it is used, making it the recurring cost of AI. It is also important to understand that one response actually contains two hardware jobs. Prefill is when the model reads the prompt. 1:55 Input tokens can be processed largely in parallel, so prefill wants enormous arithmetic throughput. 2:02 Decode generates the answer one token at a time and repeatedly pulls weights and the growing key-value cache from memory, so it wants fast memory access and very little waiting between chips. 2:14 Usually, batching more requests raises total throughput, but makes each user wait longer. 2:20 Prioritizing one request improves responsiveness, but can leave expensive hardware less efficiently used, which is inappropriate for Jane Street that tries to make everything super efficient. 2:32 So Etched claims it can improve both more total tokens and lower response latency within the same power budget. That is their big promise. How are they going to do that? 2:43 Etched was founded by Gavin Uberti, Robert Waten, and Chris Xu, three Harvard dropouts who became Thiel Fellows, all of them at this moment, at this very moment, are 24 years old. 2:55 Jensen Huang was 30 when he started Nvidia. What? So when they started Etched in 2023, basically just passing their 20 years, that same year, company nearly ran out of money. 3:08 Investors would not return their calls, and the founders, in their early 20s, had never shipped a semiconductor and wanted to challenge Nvidia with a chip that deliberately did less. In 2024, that chip had a name, Soho. 3:23 Soho was presented as a transformer-only ASIC. Etched wanted to remove much of a GPU's flexibility and devote more silicon to the few operations repeated inside transformers. 3:35 The company projected that an eight-chip Soho server could run Llama 70 billion at more than 500,000 tokens per second. That meant total server throughput, not one user speed, and it was never independently demonstrated. 3:49 The bargain was clear. If transformers conquered AI, if they're so important for AGI, Etched could win by deleting everything they did not need. If you remember, Talos provides a useful comparison here. 4:03 It goes further by encoding a particular model, including its weights, into silicon. Etched kept weights in memory, allowing one chip to load different transformers. 4:13 It sat between a flexible GPU and Talos model-specific hardware. Then the story completely changed. By 2026, the Soho name and the 500,000 token claim had disappeared. 4:27 Etched began calling its product a frontier inference cluster. It said the current hardware was architecture agnostic and already running DeepSeek, Qwen, Llama, and Mamba. 4:39 DeepSeek and Qwen did not prove anything because many of their models still use transformer components, but Mamba is the interesting moment here. Mamba is a state-space model. 4:51 Instead of standard attention, it carries forward a compressed internal state and updates that state as it processes each token. That creates an obvious question. 5:02 In 2024, Etched said Soho could not run state-space models. How can its current hardware run Mamba? A 2023 Etched patent provides a little clue. 5:13 It does not describe one complete transformer graph permanently frozen into silicon. 5:18 It describes a large configurable matrix multiplication engine, separate hardware for attention, fused software kernels, and a small processor that configures how the chip executes the model. 5:30 The patent even says the attention unit can be bypassed so that other neural networks can run, though potentially less efficiently. 5:37 So hard coding the transformer appears to have been an aggressive simplification, probably in the narrative of AGI based on transformers. 5:47 Now, Etched removed much of the general purpose machinery found in a GPU and added hardware specifically optimized for attention, but the chip still retained configurable matrix hardware because matrix multiplication is the heavy computational work inside many kinds of neural networks. 6:05 Mamba 2 offers one plausible path. Its state-space duality formulation expresses much of the computation as structured matrix multiplication, which Etched's matrix fabric can handle. 6:18 The distinctive state updates still need to run somewhere, but there is no obvious contradiction. The question is efficiency. 6:25 A model can run while leaving specialized circuits idle or sending awkward operations through slower paths. 6:31 Etched has published no Mamba throughput yet, no latency, no utilization, or power numbers, and has not disclosed how much the production chip co-Changed from SoPo. 6:41 We're very much looking forward for them to explain what has changed in their previous architecture. But the chip is only part of the machine. 6:51 The big change happened when they said that the current product is a liquid-cooled rack. The company went full on. It designs its chips, boards, power delivery, interconnect, cooling, and software together. 7:06 TSMC fabricates the N4P silicon. Edge's Thailand factory handles system production and testing. Large models span many processors, so data movement, memory, cooling, and power can limit performance as much as the chip. 7:21 Edge has disclosed two main ideas how they're going to improve prefill and decode. The first is low-voltage inference, aimed mainly at high throughput prefill. 7:33 Math blocks are grids of arithmetic circuits that perform matrix multiplications. NVIDIA calls comparable machinery tensor cores. Edge says its math blocks operate at less than half the voltage used by most AI chips. 7:48 Because dynamic power rises roughly with the square of voltage, this can keep more arithmetic active within the same power and cooling limits, but low-voltage transistors become slower and less reliable. 8:00 So Edge says it's redesigned the circuits, power delivery, packaging, cooling, tiling, and scheduling around that constraint. 8:09 Its splittable math arrays appear able to divide into smaller, independently scheduled sections. That could keep more hardware busy in sparse mixture-of-experts models, where work is distributed unevenly. 8:23 Edge says its system can keep more than 80% of its theoretical computing capacity in use, even when running enormous trillion-parameter mixture-of-experts models without overheating and slowing down. 8:35 That would be unusually high, but Edge has not yet disclosed which model it tested, the numerical precision used, the rack's power consumption, or results from a comparable NVIDIA system, so the claim cannot yet be independently evaluated. 8:50 The second idea is cluster-scale memory, aimed mainly at low-latency decode. SRAM, or static random-access memory, is a small amount of extremely fast memory built directly into the processor. 9:04 HBM, or high-bandwidth memory, consists of stacked memory chips placed beside the processor. It stores far more data than SRAM, but it takes longer to access it. 9:14 Frontier models are too large for one processor, so their weights and working data are divided across many chips. 9:20 When one processor needs data stored in the memory attached to another, that data must travel across the connection between them. Edge calls the resulting delay mem-to-mem latency. 9:32 This happens frequently in mixture-of-experts models. Different experts, which are specialized parts of the model, may be placed on different chips. 9:40 A token sent to an expert on another chip must wait for data to travel there and back, leaving the processors idle in the meantime. Inefficiency. 9:49 Edge's proprietary interconnect, the network connecting its chips, is designed to reduce that delay. Its goal is to make the SRAM and HBM distributed throughout the rack behave more like one large, fast memory pool. 10:03 GPUs also combine SRAM and HBM. Edge's claimed advantage is faster access to memory attached to other processors. 10:11 Together, low-voltage inference and cluster-scale memory attack both halves of serving a model, keep more math working during prefill, then spend less time waiting for data during decode. 10:23 Okay, why would customers believe it and would order it already? Because many AI buyers are constrained tremendously by power and supply. 10:32 Their practical question is how many users and agents they can serve at an acceptable speed from the megawatts they can obtain. 10:40 If Edge delivers even part of its claimed improvement, customers could add capacity without waiting years for another grid connection. 10:47 Edge also says its first system uses different process and HBM supply from NVIDIA Rubin. 10:53 In a market short of advanced wafers, memory, packaging, cooling equipment, and completed data centers, Edge capacity may be additive rather than replacing an NVIDIA order, but it can eat NVIDIA's market. 11:06 Why Jane Street matters so much here. Jane Street is almost the perfect early customer for Edge. It's like an ideal customer that everyone else will follow. 11:16 Its engineers already rewrite software, build custom compilers, and experiment with specialized hardware whenever conventional systems are too slow, and in trading, even a small reduction in latency can be worth great deal of money, millions and billions. 11:33 Jane Street says it tested the chip and was pleased with the early results. It also says Edge delivered the precision needed for its most demanding workloads. 11:43 It means huge money for them, and that is a very strong endorsement. But Jane Street is certainly not abandoning NVIDIA at the same moment. 11:51 In April, it committed about 6 billion to CoreWeave cloud services, including access to NVIDIA Vera Rubin, and invested another 1 billion in CoreWeave itself. 12:01 So Edge is an extremely interesting case of the new inference cluster, but it's not, at least yet, a serious rival for NVIDIA. Jane Street's own behavior shows that it wants several kinds of compute. 12:14 NVIDIA remains useful for training and workloads that require flexibility. Edge may handle specialized inference, where speed and predictable latency justify moving to a new platform. 12:25 By investing, Jane Street also gains a closer relationship with the supplier, some influence over its roadmap, and substantial upside if Edge succeeds. 12:34 But the strongest validation of Edge's promises Comes from NVIDIA's own roadmap. Vera Rubin now combines flexible HBM-rich Rubin GPUs with SRAM-heavy Groq 3 LPX systems for latency-sensitive inference. 12:50 NVIDIA's separating parts of the serving workload across specialized processors, and claims up to 35 times more inference throughput per megawatt for trillion parameter models against a chosen baseline. 13:03 So the battlefield that Etched joined is very real. 13:07 In this competition between Etched's inference rack against NVIDIA's heterogeneous AI factory, Etched does not need to replace NVIDIA to matter, and matter is the word that I'm using here because it does matter. 13:21 It's super interesting how Etched can influence NVIDIA behavior. A credible alternative can give buyers leverage, pressure inference pricing, and move selected workloads away from NVIDIA's highest margin systems. 13:37 NVIDIA's latest non-GAAP gross margin was 75%. Pricing pressure can matter before market share moves very far. Could NVIDIA buy Etched? 13:48 Well, at 21 billion of valuation, it would be an expensive way to acquire a thesis NVIDIA already shares. 13:55 I assume they will try to do something, but I don't see the acquisition right away, especially since the founders are only 24 years old and hungry for fun. Here's the connection I want you to keep. 14:06 The GPU was once the middle ground. Because CPUs were extremely flexible but inefficient at the parallel mathematics required by deep learning. 14:15 Fixed function chips were faster, but too rigid for a field changing every few months. GPUs offered enough specialization to accelerate neural networks, and enough programmability to follow the research. 14:28 That middle ground became one of the most valuable businesses in history. Now the ground may be moving again. For inference, GPUs increasingly look like the more general option. 14:39 Talos sits at the other extreme, turning one model into silicon. 14:43 Etched is trying to occupy the space between them, specialized enough to remove expensive overhead, but flexible enough to run transformers, mixture of experts, and architectures such as Mamba. 14:55 Perhaps the winner is not the most flexible chip, or the most specialized one. Perhaps it is the company that finds exactly how much flexibility the next stage of AI still needs. That's my question to you today. 15:08 The GPU once found that point. Has Etched found the next one? Thank you for watching. As always, I'd love to see your comments, and please share, share, share. Have a wonderful weekend, and I will see you next week. 15:20 [outro music]