Transcript 0:00 When OpenAI talks about democratizing intelligence, these are some of the people who have to figure out how to make that happen. Welcome to the Inference Show. 0:08 My name is Ksenia, and today I'm talking with Philippe Tillet and Matthew Ferrari, two people whose work becomes part of your experience every time you send a request to ChatGPT. 0:20 Phil is the creator of Triton, a programming language that lets people write efficient GPU code without having to become CUDA specialists. 0:28 He joined OpenAI as an intern in two thousand nineteen before there was an API or a product, and he spent years working on training efficiency. So he has watched this whole operation change from the inside. 0:42 Now, the models are surprising him with their own optimizations and even changing what he needs from the tools he helped build. 0:50 He also has a philosophical side with an interest in Bertrand Russell and philosophy of mind. Matt is a phenomenally bright engineer and very much a practitioner. 0:59 He works across the inference system, looking at how requests are distributed, how hardware is used, and how models can help investigate problems. 1:07 One thing he explains particularly well is why making one piece faster might barely improve the experience for the person waiting for an answer. 1:16 You have to understand how everything works together and decide where your effort will make a difference. And that's what interested me about their collaboration with AI. 1:24 Coding is great, but when did they realize a model could help them think through the engineering too? We recorded this interview on Astra's launch day, with our examples coming from Sol. 1:35 But the question applies to every new model. Once you've built something more capable, how do you make that capability available to more people? Phil and Matt take us into the work behind that. Enjoy. Hello, guys. 1:49 Thank you so much for joining me for this interview. It's a very fascinating area of research. 1:55 I wanted to address the blog that you posted in July, and there you wrote that efficiency has been central to distributing the intelligence to a wider audience. Can you elaborate what does it mean for you? 2:06 Efficiency and democratiz- democratizing intelligence is a big part of our mission here at OpenAI, and obviously, like we have a limited finite number of compute that we have to distribute between everyone that we want to serve. 2:18 And every time we make our inference stack faster and more efficient, this allows us to just distribute our products more broadly to free users and to pay users and to everyone, and this is why we're pushing so hard on this. 2:32 Phil, you joined OpenAI in two thousand nineteen. You were an intern, and then you were an author of Triton at that time. 2:39 I know it's a wide question, but what has changed for you as a, a efficiency and inference specialist thinking about two thousand nineteen and now? Yeah, a lot. 2:48 So when I joined in two thousand nineteen, inference was not a thing. Like, OpenAI didn't even have an API or a product. 2:56 [laughs] I joined to do pre-training work, so I spent my first, uh, four or five years actually working on pre-training efficiency all the way from GPT three point five to, uh, to our latest models. 3:07 And over the past few years, I've seen like inference become more and more and more important- Mm-hmm... as, you know, the-- our, our products have hit the market and, uh, the industry has grown. 3:16 And so it's been- You say it so casually. [laughs] Yeah. It has been... Yeah, maybe desensitized a little bit. [laughs] Yeah, and I think just the scale has become li- like mind-blowing. Mm-hmm. 3:27 Uh, when I joined, we had one cluster of ten thousand V100s that we were using for pre-training, and now like it would be a drop in the bucket today. Yeah, that's been qui-quite the journey. Mm-hmm. 3:39 Matt, what does excite you the most since you joined when it was already all happening? Yes. What's exciting-- What excites you the most about inference? Yeah. So I think that it's, it's changed so dramatically. Mm. 3:51 It's so crazy to say that only in, in one year's time to see it transform, uh, so dramatically. 3:56 When I joined, a lot of the work that we had mentioned in the blog post that you spoke of, it looked so dramatically different than it does today. 4:04 There was a lot of tuning kernels by hand, investigating inefficiencies, maybe like one or two layers deep into the stack. 4:12 But now, with all of the tools that are available, Codex, GPT five point six Sol, enables you to do so much more and dive so much deeper into- Mm-hmm... the performance. 4:21 And so the work has dramatically changed and also, like the impact that, uh, you're able to have just, just as one person working on efficiency end to end. And, and so I think it's been, it's been really exciting. 4:32 When was this moment when you realized that the models can help you not only code, but actually engineer stuff? Yeah, that's a great question. 4:38 So for me, I think this, it's-- this is a question that we ask like other people internally as well. [laughs] It's like a casual lunch, uh, conversation. Mm-hmm. Like when did it like really transform your day-to-day? 4:48 For me, it was, uh, GPT five point four was when I really had the moment that, wow, this thing is really incredible. 4:57 I remember I was working on a project from December to January of December 2025 through January 2026, and I was sitting there with, with 5.4 once it was launched. 5:08 With just a little bit of iteration, I was able to basically duplicate- Mm-hmm... 5:11 all of the work that I had spent doing two months manually in just a couple of days, heavily assisted with, uh, the GPT models, and that, for me, it was like, "Wow," like, this is such an accelerant to the day-to-day work that I'm doing. 5:23 Mm-hmm. Uh, and, and really how when I saw and thought that this, you know, this will be transformational for, for our efficiency and for, for the industry at large. What about you, Phil? 5:32 Yeah, I think about the same moment. I think about the time we released Codex and really started ramping up. Mm-hmm. 5:38 I think that's when we started also, like, using it internally and seeing, like, increasing benefits from using it. 5.4, I, I would agree that it was the inflection point, and then just kept increasing with 5.5, 5.6. 5:51 Now our, um, you know, all our upcoming models, yeah, and e- every new generation of model, we, we sort of see the, the benefits, like exponentially increasing- Mm-hmm... 5:59 uh, which is, like, ch- changing the nature of our work, like even quicker than we can realize. 6:04 Can I ask you a question about just, like, practical thing, like when a user send request this newest models, uh, what happens from the inference side? Also, how, how it changed from model to model? 6:15 Yeah, so I think a very interesting question about that is, like, how also, like, it impacts efficiency- Mm-hmm... and end-to-end. 6:22 So when a user is using Codex, uh, the first thing that will happen is what's called, like, the harness or the piece of code that runs on the endpoint device, whether that's, like, in ChatGPT or on the user's laptop, uh, will send a request to the OpenAI API, uh, or the, the Codex API, and that request has what's called, like, tokens. 6:44 Mm-hmm. And/or text, and then they get turned into tokens, and that's really where the efficiency starts. 6:49 And, like, one thing that I've also found super fascinating to, like, really dive deep into firsthand is that if you don't have efficiency at the higher layers of the stack, it's impossible to get efficiency, no matter how hard you work at the lower layers of the stack- Mm-hmm... 7:03 because it, it's what dictates the workload. 7:05 And, and even if we spend an infinite amount of effort trying to make the most foundational layers of our stack efficient, it doesn't really amount to all that much if the higher layers aren't taking advantage of it and also sending the request in an efficient manner. 7:20 As mentioned, the requests get turned in from, like, text into tokens. The tokens get sent to what we call the inference stack, where it goes through a series of, like, global load balancing- Mm-hmm... 7:30 and regional load balancing and through, like, gateways, and eventually land on what's called, like, the inference engine, which is code that runs on the GPU and calls our, our kernels, which are, like, the foundational device code- Mm-hmm... 7:42 that is doing tons and tons and tons and tons of mathematical operations to predict the next token that, uh, completes the user's- Mm-hmm... uh, set of input tokens. Mm-hmm. 7:51 Uh, the, the inference engine is doing a ton of work, basically making all the decisions about how to orchestrate the work- Mm-hmm... how to batch different requests together, how to schedule them. 8:00 There's, like, just a whole lot of things that can go wrong at this layer. Uh, and maybe o- one interesting thing is m- a lot of this code is actually running on CPUs- Mm-hmm... 8:09 very traditional hardware, and if you don't use CPUs efficiently, you can also hit, like, very important performance bottlenecks. So I think there's generally, like, a lot of discussions about using GPU as well. Mm-hmm. 8:21 But, uh, CPUs we've seen can also become a bottleneck. Right. Um, I think they're also, like, the whole storage piece. It's like, okay, like, how do you avoid doing, like, duplicate work- Mm-hmm... 8:30 if, like, two users send the exact same request or, like, very similar requests that have the sa- the same prefix. Mm-hmm. How do we make sure that, that you don't do redundant work, like, unnecessarily? 8:40 So we have, like, an entire, like, caching infrastructure around this that, that helps us also, like, reduce not just the efficiency of the work that we do, but the amount of work that we do. That's super interesting. 8:50 How many areas of optimization of efficiency? You said CPU, GPU, then it's on the, uh, in the cache level, like, there's so much more, right? Yeah. I think that there's two ways to, to think about it. 9:03 There's the, the hardware that we touch and that we have to optimize- Mm-hmm... and there's the, there's, like, all the accelerators, so GPUs of course, but increasingly our custom accelerator as well. 9:13 There's all the, the CPUs, there's the, the networking. Obviously, it's very important to use our network efficiently, all the storage component. Mm-hmm. 9:21 All, all of this we need to make sure we use well, and if you take, like, maybe a more, like, architectural view of our system- Mm-hmm... then, yeah, you need to optimize the harness, as Ferrari, uh, mentioned. 9:31 You need to optimize the, the API layer, the inference engine, the, the device code that, like, the kernels that run on, on the accelerators. The load balancing. Mm. The load balancing, the, the scheduling in the engine. 9:42 There's just so many different components. So you mentioned GPUs, CPUs. There is new chip, um, that OpenAI has. 9:49 So the compute become heterogeneous, and you need to-- the, all these things need to talk, uh, between each other. Yeah. And you wrote Triton for humans who doesn't kn- know how to, like, use CUDA, basically. Yeah. 10:02 But from the blog post, I understand that- Triton becomes the language for the models. Yeah. 10:09 So th-there's, like, an interesting story here, which is, uh, you're right that Tri-Triton is something I made at first for myself- Mm-hmm... 10:16 uh, because I was hired to write kernels, and I could write them faster with Triton. 10:20 And then eventually, as our models got more and more capable, what we discovered is they were, like, getting increasingly limited by Triton. 10:26 So Triton is a relatively high level language, which means that the compiler still does a lot of-- a lot for you, makes a lot of decisions. Mm-hmm. 10:33 But as your AIs get better, you don't really want a compiler to make decisions that an AI could make. 10:38 So what we've done is we've developed, like, um, kind of an offshoot of Triton that we call Gluon, which is a lower level language, and now this is mostly what our models are using, um, both for our custom accelerator, also for GPUs. 10:50 Yeah, and I think having this level of control has really allowed the model to, to be, like, increasingly accelerant, an increasing accelerant to, to our work and our efficiency. 11:00 In that same blog post, you said that kernel improvements reduce the end-to-end cost of serving GPT 5.6 by twenty percent. What does it mean exactly? 11:09 Does the-- Does it-- For the user, does it mean that the price goes down, the latency is-- like, the speed is higher? What's happening? 11:17 Yeah, so, so there's a variety of things that, like, we can do with this twenty percent. One of the things that we can do is we can just, like, uh, increase the number of people we can deliver- Mm... 11:27 the product to, uh, without, you know, changing anything else about our product. 11:30 Another thing we can do is we can keep the total amount of traffic the same, but we can maybe make the product a little bit more responsive by decreasing latency. Yeah. Can you think of any- Yeah. 11:40 So I think ano-another big one is, like, reliability and having capacity to be able to support at scale, uh, and where the customers can rely on being able to, to use in their day-to-day workflows. 11:49 And also, as you mentioned, price. I think, uh, in GPT 5.6 in the API, there was a twenty percent, if, if, if I remember correctly, price reduction recently. Mm-hmm. 11:58 That's, that's related to- As we make our stack more efficient, and we're able to pass on the wins- Mm... to our customers, whether that be in the f- in the shape of costs- Mm... 12:08 or efficiency in terms of speed or reliability, and so, so related, generally, yes. That's very interesting. How was the work structured with the model working with you? Where the human was in this loop? 12:22 [laughs] I'm trying to understand, you know, the dynamics that happening between models and humans. Yeah. So I think it, it depends on the layer of the stack. One example that we talked about in the... 12:32 or two examples that we talked about in the GPT 5.5 original launch blog were cases that we were able to use the model, again, itself to be able to increase performance- Mm-hmm... 12:44 and that influencing, like, the costs that, that, uh, the model is available for as well. 12:48 And so in those cases, it was Codex and the model looking at large amounts of production data and production-- or production traffic patterns. Mm-hmm. 12:56 That's like the shape of requests, the number of requests, the general makeup of them, and figuring out how we could best load balance those requests- Mm-hmm... across our individual accelerators. 13:06 And this was work that previously we had our own heuristics for, but, you know, they're not perfect. 13:11 Uh, and, and they were tuned by humans, as opposed to giving, uh, the model access to thousands and thousands and thousands of data points to be able to, to tune heuristics upon. Mm-hmm. 13:21 And in that case, we're able to dramatically increase efficiency as well by reducing imbalances, uh, across the accelerators, reducing imbalances across the clusters, reducing imbalances at the network layer, and being able to provide a, a smoother experience. 13:34 So in those types of examples, it's humans working with the model to be able to, um, analyze very, very large amounts of data- Mm-hmm... and then improve the heuristics. 13:44 But there are other examples as well that maybe Phil can talk about. 13:46 Yeah, I think that that's, uh, generally correct is we still have humans, uh, supervise, uh, a lot of the optimizations that we do because ultimately, like, someone has to be responsible- Right... 13:57 if something goes wrong. So there's always, like, a human in the loop at some level. And, and what we've seen is as the models get better, the humans can give, like, you know, higher and higher level directions maybe. 14:08 Uh, but ultimately, there's still a human responsible for, like, fixing issues if we roll out something and we figure out there's a bug or something like that. Mm-hmm. 14:17 That's why we, we've been careful about, like, not, like, completely YOLOing everything and still, like, keeping some humans in the loop. Do you do YOLOing sometimes, like, when it's safe? I think, yeah. 14:29 So when we have, like, extremely high confidence in the tests that pass, we can lean a little bit more on the models. 14:35 So there, there are some components that are, like, very testable, like kernels or, like, maybe example is, okay, it's like mathematical function. Mm-hmm. You have an input, uh, and you know the output. 14:44 The problem is, like, fairly static in the sense that you know the shapes of the input are, are, like, not super variable. So th-they are, like, we can lean on, on our models more. 14:52 But the moment you start to touch into the runtime, or you have, like, many, many edge cases, what we've seen is if you're not careful and you lean on the models too much with incorrect directions- Mm-hmm... 15:01 then you can have bad surprises. I think another example, uh, of a case that I and others on the team have leaned into, like, more autonomous usage- Mm-hmm... 15:11 is, uh, in letting the model kind of run free is for debugging issues. Mm-hmm. 15:16 At the scale that we operate, there's Crazy, crazy issues that happen in production due to hardware, software that are outside of the inference stack. 15:25 Like bugs that we've seen that I struggle to imagine a human getting to the root cause in, in any sort of reasonable amount of time. Right. 15:32 But we're able to work with the model, let it kind of run, debug all of the symptoms, come back with an answer in, in many, many, many, many hours- Mm. 15:40 -of it working independently, uh, to be able to try to root cause issues. Uh, and then with those findings, then we're able to go take them back and work on fixes more so human in the loop. Right. 15:49 Um, so that's, that's another, I think, great example of place that we're able to let the model be a little bit more, you know, free- Mm. 15:56 -in the sense that, uh, it's not actually, um, like writing code that could, could impact production, more just debugging existing. Mm-hmm. Yeah. I think that, that's a great point. 16:04 Um, I think some example here that have stuck with me is we're always kind of early adopters of- Mm. -you know, hardware. 16:10 So we rolled out like GBs extremely early, and there's always like these, uh, firmware issues early on. 16:16 We can let Codex run loose, and it will get to the root cause of, okay, like it's like a problem, like the link scroll driver or something like that. 16:23 And then we can like escalate through the right channels to get it fixed. Yeah, I think this is absolutely a, a use case where like agents can autonomously, uh- Yeah. -go wild. This is continuous loop. 16:33 Constant, constant, continuous loop. Yeah. That reminds me... So, so the model, n-not specifically Solo, a model observes the state of a system, chooses when to intervene, and evaluates what's changed. 16:46 And it reminds me the observe, predict, act loop that I talk often about when regarding word models. 16:54 So my question to you is, do you think a model that you use needs some sort of an internal representation, the internal representation of how the system behaves and how to predict stuff? 17:04 Or are you just like, I'm stretching it too much, and the language-based reasoning is enough? I personally have been extremely surprised with the-- what the models are able to accomplish. 17:14 I'm sure Phil can talk about this as well, but there's many things that I've seen it do that it g- as we mentioned, like back with the GPT 5.4, kind of that i-inflection point where, I mean, now you look at some of the things, some of the just fascinating and creative solutions that it has to problems, uh- Mm. 17:29 -where it really kind of makes you think it's like, wow, that's extremely impressive that it was able to come up with that. Can you give an example? Yeah. Can you think of something off the top of your head? 17:37 I mean, some of the solutions that it has in, in kernels, some of-- again, going back to the heuristics things that we had talked about where it's able to determine how to best balance requests across accelerators. 17:48 Another one that comes to mind is, again, like the way that it's able to debug issues by analyzing all the different logs from all different devices, like hardware devices, and figure out what caused like different types of potential corruption, et cetera. 18:01 It's, it's, yeah, remarkable. Mm. Yeah. I'd say the way I think about it, there's maybe like three components. 18:07 You have the reasoning component, you have the factual knowledge component, and then you have everything that's within the context of the model. 18:13 And what we've seen is the, yeah, our models are just getting like extremely good with just these three things. 18:20 Uh, sometimes the limiting factor is just like putting the right things in the context, because otherwise it can take your instructions very literally, and it- Mm-hmm. 18:26 -will not always do exactly what you mean, or it won't have like the right amount of, uh, knowledge to lean upon to do its task. 18:33 Uh, yeah, on the kernels front, we, we, we see like very impressive things all the time where we have a strong history of kernel expertise at OpenAI, but I think now we're at the point where our models are just like surprising us, uh- Mm-hmm. 18:45 -all the time there. The models definitely need like an internal understanding of, you know, how the world work and, and how like computers work- Mm. -and software works. 18:54 They're also able to reason extremely well, and that's why we have this extremely long chain of thoughts, uh, as they try to think through the problems. 19:00 And then coding, one of the bottlenecks is verification and evaluation. Same for you? Yeah, I think very much. 19:07 Usually what, what happens is every major real-world system has edge cases that it's very hard to test for, uh, until you know about them. So unknowns unknowns in general. 19:18 We spend a lot of time and effort trying to formalize the problems as best as we can, so we can have some guarantee that- Mm. -the model is doing what we want. The higher you go up the stack, the harder it is. 19:29 Like at the lower level, like kernels, again, I think is reasonably simple to verify. Mm-hmm. 19:33 But then, okay, how do you verify that the load balancer doesn't have like edge cases, your queuing system doesn't have like edge cases? It's much harder. 19:41 So from being a mathematician, you become a manager of a huge, huge scale operation. Like you need to think on so many different levels. I just wonder how you think about, again, like the human role in this process. 19:55 One thing that I've found to be very helpful is to try to-- The models are, again, as Phil has mentioned, like very good, but they don't necessarily know like where to best prioritize- Mm. 20:03 -their time across the entire stack at large. 20:06 And so there's lots of times where like you might identify certain bottlenecks, and then you're able to kind of point the model at it, uh, and then help solve that problem, as opposed to if you just say... 20:17 Like, if you, if you point it at some issue that's not actually a problem and say like, "Try to optimize this," it'll happily go optimize it. Mm-hmm. Even though the end-to-end impact might not be very substantial. 20:26 And I think that goes to like another point of something that I've found so impactful while working on all of the optimizations, uh, across the entire stack to, is that it, it really does require like end-to-end focus. 20:38 If you just focus on one specific layer, the end-to-end impact isn't, uh, as large as it could be- Mm-hmm. -if you look at everything holistically. In the same blog post, you were talking about speculative decoding. 20:49 That's one of the things that model, uh, suggested, or how did you come up with- Oh. -that type of optimization? 20:56 Yeah, so speculative decoding, uh, so Matthew knows more about this than me, so I'll let him chat a bit more, but i-it's a technique that predates us. Mm-hmm. It's like a very well-known like research area. 21:06 You know, as our models get better, they can help us with, you know, some of this research. They can also help us with the implementation. For like 5.6 specifically, I think Matthew knows more. Yeah. 21:15 So we heavily take advantage, uh-Of speculative decoding. The m- the-- we see it as a great way to provide a faster experience for our users to predict ahead, uh, tokens and then allow the model to, to respond faster. 21:31 Mm-hmm. And get more efficiency on the, the accelerators as well. In trying to improve this for, uh, our users for five point six, we heavily used the model to improve the speculation model as well. 21:45 Everything from the architecture that was being used to the system side to, as we even mentioned in the post, like, the actual training loop end to end. 21:54 And this is something that previously might have taken a human many, many, many, many, many, many hours- Mm-hmm... uh, to do. 22:00 To do the research, to do the experimentation, and then to do the actual monitoring of the run itself. 22:06 And now being able to rely on and leverage the models to be able to do that, it's, it's a huge accelerant, and we're able-- it makes the search space and what you're able to try- Mm-hmm... 22:15 so, so, so much significantly broader. 22:17 And another thing that I really appreciated about it is you're able to be a lot more creative because it almost in a way kind of trivializes the, the implementation of it, and you're allowed to come up with these crazy ideas that previously would've been bottlenecked by your ability to implement the system. 22:31 Right. 22:31 Some of the, uh, potential ideas that we've, uh, thought about in the past, it's like the systems of it would just be too complex, and it's not even worth potentially trying to ideate how good of a solution it might be- Mm-hmm... 22:42 just because even implementing it would be way too complex. Um, but now with the models being able to assist in the implementation, it really broadens the search space of what we're able to try. 22:52 Yeah, that's a great example. Um, I guess my question was, was also about, uh, like as you said, the research and the idea predates, but also there were so many different ideas, so many different research. 23:02 Do the models help to like also ideate what to use, or it's a human who throws the ideas if before it was impossible to try, but now you throw it in the model, and it just does it? Yeah. 23:12 I mean, I think O- OpenAI has, uh, many, many years of development, uh, and experimentation around, uh, models. 23:20 And so we have, like, lots of past work that has been done in trying to see how you can make a model more intelligent. 23:26 And if we, like my team, were to try to experiment with every one of those past ideas that had been tried to improve the accelerator model, uh, it would be a, a lot of work that we wouldn't be able to manually do. 23:39 But if we're able to tell the model, "Hey, here's like many, many years' worth of, uh, model research that has been done- Mm-hmm... 23:48 please try to take advantage of some of this and see what might work best," it allows us to try many, many more things and, and then the model to try many, many more things and sort of automate a lot of the experimentation- Mm-hmm... 24:01 to see what a more optimal design might look like. How many different teams work on inference and efficiency? How many- I, I'd say one of the things- Yeah... 24:09 that I really enjoy about OpenAI is there's not, like, extremely strong- Mm-hmm... uh, organizational boundaries. 24:14 So it's sort of hard to classify teams because everybody just kind of works so collaboratively, uh, at least on the inference side. I think it's a bit hard. 24:21 I think roughly you can think about all the layers we describe, you know, like harness, API, service, engine, kernels, compilers. Mm-hmm. 24:31 I think maybe that's one way to, to think about the way the stack is structured, and maybe like one, one component is owned by like one team that may be part of product or inference. 24:41 Uh, I'm asking because it's interesting. If you have different specialization, you see different angles. 24:47 And like, for example, for two of you do have disagreement about what bo- bottleneck to tackle next or what might-- what optimization might bring more results? 24:55 Uh, I would say like our approach to this is we should have a lot of clarity around, uh, sources of inefficiency precisely. 25:02 So there's like source of truth and objective data we don't argue about, and this is also another thing we spend a lot of effort and time on, is to just to understand like where we're losing performance, that we can focus, focus as an org on addressing these bottlenecks. 25:17 One other thing that Codex is doing is it's giving everyone a lot of fluidity. 25:20 Anyone with sort of strong doctor engineering fundamentals can touch much more easily different parts of the stack in ways that wasn't possible before. 25:28 So that also gives us like more flex to, to allocate resources to the biggest problems at a certain time. Mm-hmm. 25:34 I wanted to ask, uh, with the current situation, the current models, they're so big and so sophisticated and so, uh, powerful, and recently a lot of people complain about like models, like model is down. 25:46 And it's not about any specific lab. It's like all, all over the place. When people say the model is down, it might be slower, it might be less capable, whatever. 25:54 How do you see it from the inference, from the efficiency side? What happens when the model is down? I think Matthew knows a bit more than me about that. 26:01 I think fr- from my perspective, like model can be down because the cloud provider is down, for example. That's the simplest cause. Or-- yeah, [chuckles] simplest cause. A model can be down because maybe our API is down. 26:12 At the layer of like the way it intersects with our optimization, something that can happen is we're, we're maybe like optimizing things a bit too aggressively. 26:20 Our queuing system is not like super well-balanced, and maybe like so some queues can overflow or things like that. Uh, there can be many causes. Yeah. I'd say one thing, you mentioned less capable. 26:29 I'd say the one thing we strive and guarantee-- try to guarantee is that, uh, that wouldn't happen. We always try to provide like a consistent experience for, for our customers. 26:39 And so then if you're using GPT 5.6, you're, you're using GPT 5.6 and wouldn't get a different model experience, uh, because of some incident. 26:47 But there are, as Phil had mentioned, like at, at every single layer of the stack, whether the cloud provider or, you know, way, way, way further upstream at the API layer, lots of things that can go wrong. 26:58 I think that there's also something, um, interesting, which is sometimes the, the model is not really down, but it's a bit different. Yeah. But it's not necessarily like less intelligent or something. 27:07 Maybe we'll get like slightly better at some tasks, slightly worse at some others, and I think this can also bring noise. So M- Matthew mentioned, you know, we have like five point six. This is five point six. 27:17 We have a lot of- Correctness test to make sure that this is the same experience for the users. Sometimes, like, things have slipped, uh, because, like, maybe we'll change some accumulation order, right? 27:28 Between floating points and, and there will be like a little bit of, uh, numerical differences. But now we, we can actually catch these things, and we can ensure that the behavior of the model doesn't change- Mm. 27:40 -when we roll out optimizations. As an active user, I still find it difficult to understand between all the adva- different levels of intelligence of the model. How will they chose from the inference perspective? 27:52 Like light version is lighter on the system, right, I suppose. In terms of, like, different reasoning efforts or different models themselves? Yeah. Yeah. 27:59 So the reasoning efforts indicate to the model how much time it should spend in what Phil previously referred to as its, uh, chain of thought. Mm-hmm. 28:07 Uh, the-- where the model is doing reasoning to itself to think through different steps of the process. 28:11 And so to the inference stack, those just look as additional, uh, tokens that are being generated by the model, and that process is sort of opaque to us, it... 28:20 no different than the request just took a little bit longer and generated, uh, additional tokens. Well, we can still, like, monitor the chain of thought, so it's not, like, completely- Oh, I- -opaque. Yeah. Yes. 28:31 O-opaque to us on the inference side as, like- Yeah... uh, not an optimization that we're doing. But yes, Phil, Phil is absolutely correct. Yeah. 28:37 But the, the choice to give different levels is to make it easier for the system? Yeah. 28:43 I think one thing that, uh, my mental model of this is essentially AI is like a big, uh, three-way trade-off space where you have, like, intelligence, you have latency, and you have cost. 28:53 And what we found is it's not possible to have, like, one solution that is, like, best in class on all axes. 28:59 So we have this sort of Pareto frontier we have to think through, and different ways to explore this Pareto frontier is with different reasoning efforts or different, like, model classes. 29:09 And the goal here is really just to meet developers where they are. Mm-hmm. 29:13 There are some developers who are, like, more latency sensitive, some others who are, like, more cost sensitive, and so that's mostly why we provide all these options, you know, at the risk of, uh, maybe, like, overwhelming some people with options that do provide a solution for everyone. 29:26 Mm-hmm. Thank you. That was very interesting. I have two last questions. One is about the next bottlenecks and, like, specifically what excites you to tackle. Yeah, there's so many things. 29:36 I think I usually try to stay grounded in, you know, the OpenAI mission, which is democratizing intelligence to as many people so every-everyone can, can reap the benefit of it, and I usually try to think through this lens when I look at problems. 29:53 And, you know, making the model cheaper, I think, is probably, like- Mm... very important, uh, aspect of this. Taking safety extremely seriously i-is, is something that we really take to heart. 30:04 You know, back to optimizations where we have to be extremely careful and increasingly careful that we don't, like, do any efficiency improvement- Mm... that decreases the safety profile of the models. 30:14 A-and another aspect I'm always, like, interested in as a performance engineer is just, uh, trying to use the hardware more efficiently, get more intelligence out of every watt that we deploy, uh, so that we can use our data centers as, as efficiently as possible. 30:29 Do you see a lot of po-potential there? Yeah, I think there's still, like, room to make things better, both in terms of hardware efficiency. We had-- we released, like, some data about Jalapeno. 30:38 Our custom chip is very exciting, I think. Very exciting, like, performance efficiency characteristics. Mm. But even at the pure software layers, there's, like, so much more that we can do still. 30:48 Likewise, uh, I think Jalapeno is something that's extremely exciting for the future and being able to integrate more with that. Can you elaborate a little bit more in terms of relating to other chips? 30:59 Uh, I think I, I will say that w-we love Nvidia, and I think we always try to get the most of our-- of their hardware. 31:06 Ultimately, our goal is to democratize intelligence as much as possible, so we're always looking for the chips that- Mm-hmm... let us do that. 31:13 So we're doing a lot of work trying to have very fair comparisons between different chips and then just go with whatever solution is best aligned with our mission. Mm-hmm. Uh, usually that's the way I look at it. 31:25 Right now, there's very exciting data about Jalapeno. This is, like, preliminary data. Obviously, there's, like, caveats and asterisks. So we're still in the process of evaluating our chip. 31:34 But yeah, the, the way I think about it is, as I said, just staying grounded, grounded in the mission and looking at the chips- Mm... uh, through this lens. Thank you so much. And my last question is always about a book. 31:45 What was the book that influenced you a lot, like influenced how you think about things now? Might be from your childhood or something very recent. The book that influenced me the most. 31:55 I don't know if it's, like, a single book. I mean, mayb-maybe like an author. I used to really like Bertrand Russell- Mm-hmm... 32:02 uh, way of phrasing things, and I remember I read a introduction to philosophy book that he wrote. 32:08 I think definitely that's something that stuck with me and also a bunch of, like, philosophy of mind articles that I read during my graduate school. Mm. 32:17 I think for me it's-- it was less, uh, a book and some professors that I had in school where I remember we were doing programming, implementing different algorithms, uh, on CPU, and then going through all the different efficiency levers that we could do to make the algorithms even faster and faster- Mm... 32:35 and faster by exploiting different, uh, programming models on the GPU. 32:40 And I remember just being so mind-blown that, like, wow, I had spent so hard, uh, trying to optimize my CPU algorithm that was-- ended up being a thousand times slower- [chuckles]... 32:49 than what I could trivially do on GPU. And so from that point forward, it was really something that, uh, I was fascinated about and, you know, put me down the road that I am today. 32:57 And you're still optimizing and accelerating. Yes, yes. [chuckles] Still, still, still working with accelerators and, and trying to, to make the algorithms faster. [chuckles] Thank you so much. It was a pleasure. 33:08 Thank you. Thank you. [outro music]