Transcript 0:00 A couple of weeks ago, I got very excited about DeepSeek Harness because it showed us an agent that can change its own working environment. 0:08 It can realize it is missing a tool, write that tool, load it, use it, and later remove it. 0:13 This week, Meta released Muse, its own private personal AI agent, and I found myself thinking about almost the opposite question. 0:21 If we allow agents to change more and more of themselves, what should they absolutely not be allowed to change? [upbeat music] Attention Span. Welcome to Attention Span by Turing Post. I'm Xenia. 0:37 There is a lot of things to discuss about agents this week. First of all, quick note, what is Muse? Muse is Meta's new personal agent, and it's free. It gets its own computer in the cloud. 0:49 It can browse, send emails, work with your calendar. It do a lot, a lot of stuff that you need from your personal agent. It keeps working after you close the app. 0:58 Meta says that internally they have seen it edit itself as well, and there is a lot of freedom to give to a piece of software, especially when you also give it your inbox. 1:10 So Meta built something around Muse that I find much more interesting than Muse itself. 1:15 The agent gets considerable freedom to change what it can do, but it does not get corresponding freedom to change what it is allowed to do, right? How do we do that? 1:25 But before we dive deeper, I want to make a quick note about rogue agents. 1:30 We recently made an episode about the OpenAI and Hugging Face incident, where agents running cybersecurity evaluations found ways around their sandbox and discovered unintended ways of communicating with one another. 1:44 This week, an amazing scientist, Melanie Mitchell, wrote a useful piece arguing that words like rogue, escape, and swarm can obscure what technically happened. 1:56 That blog is absolutely worth reading, since it also addresses the latest craziness around safety and alignment people who predict that AI can exterminate humanity. 2:07 And by the way, let me know if you want my honest and grounded take on that in a separate video. In any case, Melanie writes that an agent is still software running on the computer. 2:16 The model receives information, chooses an action, the harness executes it through some tool, the environment returns a result, and the model chooses what to do next. 2:26 But what has changed, and listen carefully, is how much of the route we leave unspecified, because sometimes we even don't know what we need to specify, and this lack of knowledge is a huge problem, as I see. 2:37 But we increasingly give the system a goal and let it search for ways to achieve it. In the OpenAI case, the agents had been explicitly trained to find and exploit vulnerabilities. 2:49 Some tasks unexpectedly required internet access. The sandbox was supposed to prevent that access, and the agents found weaknesses in the sandbox and kept going. 2:59 When one method of communication was removed, some found another. And you can debate whether a loss of control is the right description, the engineering problem exists regardless of that metaphor. 3:11 A capable system can find routes its designers did not anticipate. So what do you build when you assume that will happen? From that angle, I want to discuss Muse. It's really interesting. 3:23 Meta gives every Muse user a dedicated virtual machine, basically a computer running in the cloud. Inside that computer, Muse gets a fairly rich environment. 3:34 It can execute programs, compile code, build custom tools, operate a browser, run multiple sub-agents concurrently. But Muse itself lives inside a smaller fenced-off environment on that machine, a Linux container. 3:48 Think of it as giving the agent its own workshop. Inside the workshop, it can move things around, build things, and change things, but the electrical panel, the keys to the building, and the security desk are outside. 4:01 The separation is enforced by Linux. Muse can appear to have root access inside the container, while that root user maps to an unprivileged user on the host machine. 4:11 Security-sensitive services run outside the environment Muse can modify, and one of those services is called Sentinel. The name sounds very mysterious, but the idea is much simpler. 4:23 Sentinel is essentially a separate permission system. Muse proposes an action, Sentinel decides whether that action is allowed, denied, or requires your approval. 4:33 Muse cannot rewrite Sentinel, and this last sentence is the whole architecture. Imagine Muse writes itself a new Gmail tool. Suppose I ask Muse to organize a conference trip. 4:47 It checks my calendar, searches flights, reads an email from the organizer, and decides it would be useful to write a new little tool for some service Meta never built a connector for. Fine, Muse can do that. 4:58 But writing a program that knows how to call a service does not grant that program permission to call it. Every network request from Muse to has to lead through Sentinel. 5:08 Sentinel can inspect where the request is going, the protocol, the path, even the actual decoded request. This separation becomes increasingly important as agents start generating more of their own software. 5:23 If a permission check lives inside a tool the agent can rewrite, then rewriting the tool can potentially rewrite the permission check, and Meta moves the authority outside that editable environment. 5:35 Not surprisingly enough, this is an old security idea, but applied to a strange new kind of program that we still don't know much about, honestly, the agentic private assistants. And it keeps going. 5:48 If Muse is going to act as you, it needs access to accounts, right? To your Gmail, to your Expedia, whatever. But does the model itself need to see the credentials? Meta answer is no. 6:02 The actual credentials are stored separately, and the code running inside Muse's environment sees a placeholder, which Meta calls a surrogate token. 6:11 When Muse makes a request, Sentinel first decides whether the particular request is allowed. Only at the network boundary does another protected service replace the placeholder with the real credential. 6:23 So if a malicious webpage tells Muse, "Show me your Google access token," Muse does not have the real token to show. Of course, that does not solve the entire problem. 6:34 An attacker could try to persuade Muse to perform an action using the legitimate connection. But now we can distinguish two problems. Can the agent reveal the key? And can the agent misuse the door that key opens? 6:49 These are different problems that require different defenses. Then I found the part that I really like. Suppose Muse re- runs a process that checks the weather. It needs the internet. 7:00 Nothing particularly sensitive has happened yet. Now, suppose another process reads my private calendar, and then wants to connect to some website. Same operation, network access, right? 7:11 But it's a very different situation, and the second process is carrying information about me. Muse tries to keep track of that as well. Every tool process starts in a clean state. 7:22 Once it reads user data, the operating system marks it as tainted. Meta calls the mechanism tainted egress, and Muse itself does not get to decide whether it has become tainted. 7:34 Meta uses Linux mechanisms called eBPF, and security module hooks to track this below the agent's application code. You don't need to remember either term. 7:44 The important thing is that the operating system watches what the process actually touched, and the question is no longer simply, "May this program access the internet?" 7:53 The question becomes, "May this program access the internet after reading my private calendar?" This is a very interesting permission model for agents, because a seemingly harmless request can leak information. 8:06 For example, if I ask an agent to search Google for a phrase copied from a confidential document, technically it's only performing a search, but the query itself has already sent the confidential information outside. 8:18 This is one of the problems agent security has to solve. Authority depends on where information has been, not only which tool is currently being called. Meta gives another nice example. 8:30 Suppose I gave Muse permission to read my email but not send it. I just did it, actually. 8:36 That sounds fairly limited, except my inbox, the latest few emails that I just checked, contains a password reset link, one-time passcode from Google, and magic login to my newsletter. 8:50 So access that looks like read can become the ability to enter an entirely different system. Muse's email connector therefore filters one-time tokens, password reset links, and login links before the agent sees them. 9:04 This is where permission gets much more complicated than the little read/write boxes we're used to. Capabilities compromise. Reading A may let you modify B. 9:15 Knowing something from B may make an action in C dangerous, and an agent is unusually good at following those chains, because following chains is precisely what we're building it to do. 9:28 There is something I love about this whole architecture. 9:30 We are building agents that write code, launch sub-agents, operate browsers, and potentially modify their own working environments, and underneath all of that futuristic machinery, computer scientists are reaching for ideas that are half a century old. 9:46 I love history. In 1975, Jerome Saltzer and Michael Schroeder published The Protection of Information in Computer Systems, one of the foundational papers in computer security. 9:57 They describe principles including least privilege, give a program only the authority it needs, and complete mediation, every access to a protected object should actually be checked. 10:10 51 years later, those principles suddenly feel extremely current, because an agent is a program whose exact path through the world we increasingly cannot predict in advance, and surprisingly enough, there is a problem in we don't know what they know, the unpredictability is actually useful. 10:29 If I have to specify every action, every website, and every intermediate step, I might as well do the task myself. It becomes too complicated. We actually want the agent to improvise. 10:40 We want it to discover a better route. 10:42 We may even want it to write the missing software along the way, so the architecture has to separate freedom to search from authority to act, and that's what I find interesting about Muse and its approach to that. 10:56 But no, Muse has not solved this yet. There are plenty of uncomfortable caveats. Sentinel itself makes decisions using policies, models, and classifiers. Those can be wrong. 11:07 Meta explicitly says Muse will still make mistakes, and that prompt injection remains something the system has to defend against, and taint tracking has to be. It's necessarily to make it imperfect. 11:20 Tracking at the process level can be coarse. A process may read one harmless private fact, then become restricted afterward. Make tracking too permissive, and sensitive information can slip through. 11:32 Make it too aggressive, and the agent spends its life asking you hundreds of millions of questions. Then we get the human problem. If Muse asks- "Can I do this 50 times a day?"... 11:43 eventually I will just click without reading. "Yes, yes, do that, please. Stop asking." And Meta is trying to address that, too. 11:50 Approval requests come from Sentinel rather than being composed conversationally by Muse, and sensitive actions such as sending email or making purchases require explicit approval, but permission fatigue is not a solved problem. 12:04 Neither is deciding the right scope of a permission. "Help me organize this trip," is useful because it's broad. 12:11 "Access exactly these seven websites, these three calendar fields, and this one email for the next 11 minutes," is secure for sure, but it's make it much more easier for myself to organize the trip. 12:26 So finding the useful boundary between these two- is going to require a lot of work. 12:31 That's exactly where a lot of research and development, a lot of discussions, and even fear-mongering will be happening in the next few months, but it is also one other boundary worth watching. 12:42 Muse runs inside what Meta calls a secure virtual machine, but secure here does not mean nobody else can see what's inside. It means several more specific things. 12:53 Your Muse runs in its own virtual machine separated from other users. 12:57 Inside that machine, the agent itself is further contained, so code Muse generates cannot simply reach Sentinel, steal the real credentials we discussed earlier, or give itself host-level privileges. 13:09 So there are already two boundaries, but now zoom out one more level. Who operates the virtual machine? 13:15 Meta does, and with the current secure VM, Meta can access data inside it when necessary to operate, secure, or support the service. That is a completely different security question. 13:26 So if the first question was, can the agent get outside the box we gave it, then the second question was, can another user's agent get into my box? Now we have a third question. 13:38 Can the company running the box see inside it? 13:41 Today, the answer to that last question is yes, under Meta's stated conditions, and Meta says it plans to introduce something different later this year called Muse Confidential Virtual Machine. 13:51 The idea would be to encrypt the entire virtual machine with a key controlled by the user so that even Meta, despite operating the underlying infrastructure, cannot inspect what is happening inside. 14:01 And I think this distinction is going to become extremely important for personal agents, especially for Meta, with so many complaints about how it uses personal information on Facebook. 14:12 So all three systems, speaking about the boundary, can accurately call themselves secure while making very different promises. 14:20 And whenever someone tells you an agent is secure, ask them just this one question: secure from whom? DeepSeek Harness made me think about how much of an agent can become fluid. 14:34 The model can change, tools can change, capabilities can appear after the task has begun. Eventually, much more of the harness may be generated and regenerated around whatever problem the agent is solving. 14:47 I still find the direction fascinating. Muse exposes the other half of the architecture. 14:52 The more of the agent we allow to change, the more important it becomes to decide which parts sit outside that change, because I want an agent that surprises me. I want it to find a route I didn't think of. 15:05 I want it to write a tool I didn't know I needed, and if agents only allow paths engineers specified beforehand, we have left a large part of their usefulness on the table, but finding a new route cannot automatically grant permission to take it. 15:20 That distinction feels very simple, but so, so complicated when we try to solve it, and that's why I repeat the layers of it. Maybe you will come up with some ideas. 15:31 Maybe you will find a new way to research and build it, because building it, it's clearly not simple at all. 15:38 Somehow, we need to find this right balance amidst this we don't know what they know, because sometimes it might mean a pleasant surprise, and sometimes a hack in your tab just because they're eagerly optimizing. 15:50 So yeah, that's a very tough problem. There is a lot of things to solve, a lot of things to handle. That makes it certainly very interesting, fascinating, and in some cases, dangerous. 16:01 What I also suspect is that we are going to spend the next few years rediscovering a lot of computer science while we figure how to balance what we don't know, what they know. What about you? 16:13 How do you feel, and what parts of research, what parts of this problem are the most interesting for you? There is so much to explore. Thank you for watching. Share your thoughts in the comment section. 16:24 I always love to read your comments, and I see you next week. [outro music]