DrZeroTrust: This is not an AI apocalypse. This is not the end of days. Nobody was like turned into paper clips and Terminators are not walking the streets yet. However, comma, it is concerning. Okay. It's very useful to understand that we've kind of moved into the realm where what we thought were theoretical problems are now real, actual issues that we have to face. It's very potential that these agents could become, let's call them, dangerous very quickly. We basically got a free demonstration. We got a free pass this time that things didn't go worse. In truth, a combination of advanced models pursued a narrow objective inside of what was thought to be training or simulated environments, and then they found a way beyond that intended environment and the controls, and then they leveraged real vulnerabilities and real methodologies and capabilities and tactics that bad guys have used or red teams or hackers, and they were ultimately able to get to real infrastructure. Some of them continued, some of them stopped, some of them found other things. But they did this at machine speed, and it was only by pure luck in many instances, or particularly these instances, that it was found out. Additionally, you can see the cross-pollination that occurs here. So different organizations touched other organizations, and then there was cross-pollination of models and threats and feeds and agents and all these other stuff. This also shows that there is not really good isolation and segmentation across these models or the infrastructures that make them operate. In truth, in total, the machines just cheated on the test, which doesn't surprise me. But in doing so, they kind of gave us the answer key to the exam we need to really focus on, to the test that we need to focus on. The danger here is not that AI is going to wake up and go, I'm gonna destroy stuff. The danger is that a capable system or an optimizer or agent or whatever else, if given the right outcomes and objectives it's supposed to go after, will find a way past the controls if those controls are not well archived. There's no malice there. There's no malicious intent. There's just opportunity and bad things on the far end of it. Now, for a long time, there's a lot of people in the industry that have been telling folks about things like assume breach. Yes, we should absolutely assume breach, but we should also add one more thing. Assume the thing exploiting the breach will likely be autonomous because the days of people being the avenue of compromise are becoming increasingly smaller as more agents and more agency systems come online. We should also assume that those machines will never get tired. We should assume that they will find a way past every boundary that we can put in front of them, and we shall assume that they will move faster than any human beings or any bureaucratic processes will. We should understand that that is a risk, we should understand that that's a reality, and we should build our practices and protocols around it because that reality is coming your way, whether you like it or not. I'm Chase Cunninghammer, Dr. Cunninghammer, Dr. Zero Trust, Trust Nothing. Verify everything, especially in the sandbox, and then I will see you on the next one. Stay smart, stay safe, stay secure, and if you find this valuable, please like, share, subscribe, all that stuff, whatever. Hey folks, this episode is sponsored by PC Matic. Why PC Matic? Well, PC Matic has spent years focused on a problem that matters to a lot of people, a lot of people I know actually. PC Matic focuses on delivering practical cybersecurity without turning protection into an expensive, overly complicated, weird science project. They're also an American-made security company. That's right, made right here in America, serving individuals, small businesses, and organizations that need dependable protection. But might not have a massive security team or an unlimited budget. In other words, they're doing affordable cybersecurity. PC Matic is also supporting independent, like this podcast, Cybersecurity Analysis, without trying to control the conversation, just like they did here. And that matters a lot in an industry where honest perspectives are sometimes let's call it hushed. You can learn more about what they are doing and get your own stuff at PCMatic. Here's a sentence I didn't expect to say. On this show in 2026, but never have expectations cause things change. Here's that sentence. A group of advanced AI models was given a cybersecurity test. Hmm, interesting. The models were supposed to solve the test inside a contained evaluation environment. So they should have just done that. Instead, they found a way out. They broke out. They did what hackers do. They reached the open internet, and things went even crazier from there. They compromised real infrastructure at a company called Hugging Face, which is a hell of a name. And according to OpenAI and Hugging Face, they did it, the agents that is, did it because they inferred that Hugging Face might be storing the benchmark data and solutions they needed to complete the test. So they did their Kobayashi Maru, if you will. Machines cheated. on the test. That's not a movie plot, by the way. That's not some ⁓ sneakers thing or whatever else. That is the outline of an actual security incident which was disclosed this month or last month, depending on when you're listening this, in July of twenty twenty six, by Hugging Face and OpenAI. ⁓ and by the way, and then Anthropic kinda came out and said, Well, we've got this own thing going on. They reviewed their own evaluation history and found three separate incidents in which Its models had gained unauthorized access to other organizations during cybersecurity testing. One of those models kept going even after it recognized that it had probably reached a production system. So a little bit of James Cameron there. Another one created and published its own malicious package to the public and uploaded it into the Python registry. Thank you very much. A third recognized that it reached a real target and said, you know what, I'm gonna pump the brakes and stop. So there was some intelligence. Considerations taking place there. So no, it's not Skynet like the news is talking about yet, but it is also not a story where quote nothing happened, as some of the publications might suggest. This is a literal live demonstration of what happens when autonomous cyber capability, imperfect containment, weak infrastructure, and ambiguous objectives, and too much trust placed in how these things operate basically meet in the same room and things go to hell from there. Welcome to the show. I'm Chase Cunningham or Dr. Cunningham or Dr. Zero Trust. Today we're talking about the two weeks in July when all hell broke loose and the AI industry kinda accidentally ran a penetration test against itself. Oops. Unfortunately for the industry, the models did exactly what red teamers do, and they won. The models passed the ways that people weren't thinking that they were going to pass. So grab a coffee, buckle up, tighten your assholes or whatever you do, because this one matters, this is important, we need to go through this. So what happened at Hugging Face? Let's start there because this is where everything begins to get very, very complicated and very interesting. So let's start with what we know, what is confirmed, what is not conjecture. Okay, this is how things actually transpire. On july sixteenth, Hugging Face, again, interesting name, disclosed that an autonomous AI agent system had compromised part of its production infrastructure. So Hugging Face got its face hugged, I guess you could say. Now what is Hugging Face? Well Hugging Face is one of the most important platforms, you could argue, in the modern AI ecosystem. It hugs lots of faces, I I guess, is a thing. What does it do? Well, it hosts models, data sets, and applications, and a lot of the tools that are used by a lot of the organizations in the industry, and those ⁓ things are relied upon to make agents work. So when Hugging Face says an autonomous agent reach production, that's not just some like niche laboratory oopsie. That's a real problem. That is a supply chain event. Now, how did this begin? Well, the initial entry point was not an NSA level super hack or whatever else. It wasn't science fiction. It was plumbing and software, not like literal plumbing, digital plumbing and software and just basic stuff. According to Hugging Face, a malicious data set abused two code execution paths in a data processing pipeline. Okay, so keep with me here. One of those involved a remote code dataset loader. So remote. Code, okay, outside data set makes sense. Load, sure. The other involved template injection through a different data set configuration. So slightly different, but still data set, still configuration stuff. What happened was those two things were not well controlled and they allowed code to be executed on a processing agentic worker. So what happened then from there, the agentic worker was essentially if infected. Did what bad guys do, or red teamers or pen testers, escalated their privileges, collect cloud and cluster credentials, and move laterally across internal systems. So not good. And basically the worst part is all this shit happened over a weekend like every other hack does. So things went bad on a Friday. Well, why? Because apparently even autonomous agents have figured out that basically one of the oldest rules in security is Do the ugly work while humans are chilling and grilling on the weekend and then mess up somebody's Friday, Saturday, Sunday. So they've just basically mastered being bad guys or hackers because bad shit goes bad on Friday. Now Hugging Face, to their credit, later reconstructed roughly seventeen thousand six hundred of those agentic actions to see what was going on. What they found was that those actions were grouped into about six thousand two hundred and eighty different clusters. So we're getting into kind of machine learning and how this stuff works, but just Bear with me here. That activity stretched across roughly four and a half days from July 9th through July 13th. So this wasn't an immediate thing. It was over the course of a few days. That matters because this was not just like an API call. It wasn't just an oopsie. This was systemic. It was a sustained sequence of automated decisions that took place outside the bounds of control to do potentially malicious things. And not just one action, thousands of them, multiple attack. Pass. When by the way, it even spun up its own short-lived sandboxes to make sure what it was going to do would work. Then it collected a lot of creds and it escalated its privileges. And then it did what everybody's done since the dawn of time when you're compromised an organization. You move laterally. Finally, it was able to interact with real production systems, which is where basically everything goes to hell. After all that came the part that changed the entire story. So that was one face hugged, but there's more to it. On July twenty first, OpenAI disclosed that the incident that when I was talking about had been driven by a combination of its models. So there's like cross pollination stuff going on here, and there's open AI and et cetera, et cetera. So open AI identified GPT five dot six SOL soul. And additionally a different version they released ⁓ as part of this whole thing. So for open AI, the models were being used again, just like in Hugging Face and an internal cyber capability benchmark called Exploit Gem. So they were doing testing, if you will, for the purposes and this is from their own quotes, of maximizing cyber capability. I guess they definitely got what they were looking for. In order to do that, OpenAI had quote reduced the normal cyber refusals that would ordinarily prevent high risk behavior. Sounds like a great idea. The environment where this thing was working in again was quote intended to be isolated, end quote. And quote, it did not have normal internet access, end quote, which okay, sounds fine. However, the thing could reach internally hosted third party services, which are used as proxy and cash for software packages. So oopsie. According to OpenAI, those models that they were messing with found and exploited a previously unknown vulnerability in the software. So they were able to touch stuff, they figured out an avenue, and they exploited it. This gave them a path to broader network access, aka lateral movement. Then they did what bad guys do and what everybody knows, they escalated their privileges, they moved laterally through OpenAI's evaluation infrastructure until They found something that could access the internet. And then they broke out and got out and did what things do on the internet. Once outside the boundary, those agents inferred well that they were working with this other thing and hugging face is there. So why not go after that too? ⁓ and then they said, well, hugging face might be a host with material that's associated with the test that they were undergoing. So why not go there? They were looking for things like models and data sets and most importantly. They were trying to figure out ways to solve the problem they were given during this exploit gym. So there's cross correlation here. So what did they do? The agents went basically looking for solutions because that's what they were told to do was to find ways to solve this problem. And they weren't given really good boundaries. Now, according to Hugging Face, so we're back to Hugging Face, the only customer contents the agents accessed consisted of data sets connected to Exploit Gym, another thing called Cyber Gym. Which sounds like it's okay, but anytime someone tells me it didn't touch other systems, I'm always like start the clock. Hugging Face said it found no evidence that other customer facing models, data set, spaces, or packages were accessed. Okay, so that's good context. Good to know. But again, anytime someone says customer stuff wasn't touched, makes me wonder. Regardless, this was a very serious event. This was something to take note of. Now all of my research was only able to find available evidence, but it The available evidence does not support basically claiming that the agents vacu vacuumed up or sucked up the entire hugging face platform. That seems to not have occurred. So that's kind of good. However, basically these machines appear to have pursued a narrow objective with a extreme focus on the outcomes. They were doing kind of what they were told to do, just not the way that they were told to do it. Back to OpenAI, they describe the behavior of those models and agents becoming quote hyper focused. And quote, going to extreme lengths, end quote, to solve the evaluation. So again, they just figured out a way to solve the problem and to Kobiyashi Miru their way through this. And if you don't know what Kobiyashi Maru is, go look it up. Now here's the crazy thing for those of us gray beards, and yes, I have gray in my beard that have been in cyber for a while, is we've spent years talking about imagining that a dangerous AI might kind of do this, might pursue some alien incomprehensible objective, might just go rogue. Well, here we've got this basically proof of what occurs if that does go sideways. ⁓ we've all talked about this and everyone has kind of researched it and ⁓ pontificated, but it actually happened. And what happened is the machine did what machines do, followed a set of instructions, but because there's this quote AI thing in there, it went haywire. So but let's not walk away from we should have actually learned here as a security lesson. So the models and the agents in here didn't basically Produce a bad answer. They didn't go and fail the test. They cheated the test. They chained together vulnerabilities, privilege escalation, credential access, network movement, and exploitation of other systems to achieve their objective. Now that sounds exceptionally familiar to me. Gosh, what does that sound like? Basically, that sounds like operational cyber capability. That's what you do when you're a red teamer, a bad guy, a hacker, whatever. However, here it happened without any human in the loop. So this is where things get really, really scary, if you ask me. I would suggest that a threshold was crossed here. This is a watershed moment, and everyone should be paying attention to this because we got lucky, honestly. Hey folks, this episode is sponsored by PC Matic. Why PC Matic? Well, PC Matic has spent years focused on a problem that matters to a lot of people, a lot of people I know actually. PC Matic focuses on delivering gas. Practical cybersecurity without turning protection into an expensive, overly complicated, weird science project. They're also an American-made security company. That's right, made right here in America, serving individuals, small businesses, and organizations that need dependable protection, but might not have a massive security team or an unlimited budget. In other words, they're doing affordable cybersecurity. PC Matic is also supporting independent, like this podcast, Cybersecurity Analysis, without trying to control the conversation, just like they did here. And that matters a lot in an industry where honest perspectives are sometimes, let's call it, hushed. You can learn more about what they are doing and get your own stuff at PC Matic. So let's talk about the anthropic incidents. One incident is an anomaly, could be looked at as just a oopsie failure, or we got lucky, okay. But If more things start coming together, you start to say this is a pattern. So let's talk about what happened with Anthropic. So Anthropic saw this go down. They said, Well, gosh, that sounds sort of like stuff we've done. We should look at our own records, logs, and see if there's anything there. So OpenAI's disclosure prompted Anthropic to review about 230,000-ish, 260,000 runs of stuff that they were doing on their own models where they were wondering if it could figure out a way to get to the internet. So Anthropic found three incidents, and this was in their logs. They had to put this together and kind of put the picture together. Those incidents involved six evaluation runs or tests that they were doing and what happened was three real organizations got touched. The earliest of those dated all the way back to April of twenty twenty six. Hmm. Now there's an important distinction to make here, something that we should pay attention to. So Anthropic's models did not exploit any zero days. ⁓ they just sort of didn't. They didn't do that to break out of any environment. They didn't even need to, which is something we should also pay attention to. According to Anthropic, the evaluation environment supplied by a partner they were working with for these tests, a company called Irregular, which there's a lot of weird names in this space. Okay, but anyway, Irregular had oopsie left an unintended path to the live internet. We call that a misconfiguration. So whoever was doing this or however it was put together, the prompt let the models know that they were, quote, operating inside a simulation with no quote real internet access. Well and that ain't exactly true. That statement was false because there was a misconfiguration. So then you tell the models to go do things and they do whatever they can it's like water. They flow to wherever the least resistance is. So the models were told to go do a capture the flag exercise and look for targets. Well, as they're looking and mapping and figuring out where those targets are, they kind of wander into real infrastructure. They move laterally, oopsie doopsie and they find stuff. Anthropic characterized this primarily as an operational evaluation failure. I would say that's correct. They didn't say that there was evidence that the models, quote, deliberately escaped containment, except they did, unless you're wordsmithing it, but let's just say for the sake of argument that that's a fair legalese and technical distinction. But regardless, it doesn't make the outcome any less worrisome. The first of the incidents we're talking about here involved Claude Opus, 4.7, so slightly older model. In that scenario, a fictional company in the exercise shared its name with a real domain. So Chase, like me, that's my name, and Chase.com. When the model could not reach the intended simulated target, it basically found the live website with a matching domain name and said, Well, that's gotta be the target because they are the same. So Chase and then Chase.com can't get to Chase, goes after Chase.com. Across four runs, the model obtained application and infrastructure credentials and extracted several hundred rows of production level data, so that is a problem. So again, couldn't get the chase when the chase got Tom and found stuff and actually exploited it. Anthropic's own review found that the model recognized signs that it had reached a real system, and then it just did what it did and continued going on anyway. It didn't go oopsie. It said, well, fuck it, let's just keep doing this. That Is what concerns me the most. That's what I think we should underline, because this machine didn't stop because it would stop. It just kept going and that's what models are going to do. It didn't basically realize what it was doing and then it kept on going. That's why these things are not artificial intelligence. They're machine learning. But yeah, it basically just kept on going because it was like, Well, no one said I should stop specifically, so I'm gonna keep doing what I'm doing. The second incident in the series involved Claude Mythos V. Okay, so that's the one that people were talking about. That model created a malicious Python package, thank you, and published it to PyPy, which is the public package registry used through the entire Python ecosystem. So yikes, if you're using Python, you might have even gotten a hold of this thing. The package remained for about an hour. Now they took it down pretty quick, so good for them, but an hour is a long time in the internet world. Anthropic said that that package was downloaded and executed on 15 real systems before registry security controls removed it. So, whoopsie. One of those systems was a security scanner that executed the package during analysis, which is kind of funny because it was not supposed to do that, but that's how it worked, which allowed credentials to be exfiltrated. So you gotta kind of wrap your head around the, I don't know, idiocy this whole thing. Now the reasoning is what makes this incident very ⁓ Interesting for analysis purposes. So early in the process, the model correctly recognized that publishing the package would constitute a real world attack. Okay, so the model knew that. But it knew that would only be true if the internet connection was legit. And in it had been told that the internet connection was not legit, so it was kind of confused. Because it was confused, it explicitly concluded that what it was doing was sort not acceptable and was. probably not the outcome that it was supposed to go for. So then the model reasoned to itself, well, if that's the case, and if I'm not supposed to do those things and I can't talk to the internet or whatever, I must still be in that simulated environment. I'm in the sandbox, so I can just publish the package. So the machine got confused and then just kind of kept doing what it was doing, but it was trying to play by the rules. So the model knew what it was doing was wrong, ⁓ and it sort of acknowledged that that was not necessarily correct. But then it said, well, okay, whatever. It sort of trained itself and made the decision that even if it was wrong, it was still acceptable to do what it was going to do. So congratulations to the model. You're pretty dang smart. But basically, for those of us around here, we've apparently automated machines being able to rationalize decisions as well. The third incident in this series involved an internal anthropic research model doing things that it wasn't supposed to do. So what that model did. was scanned about 9,000 targets. Okay. Internal-ish. ⁓ yes. So it scanned 9,000. It compromised one target that it found using exposed credentials, worker, worker, worker, and SQL injection, which is old AF. Then the model recognized the host appeared to be a real production system. So the model scanned, found a target, exploited it, and then it goes, shit, this is actual production, and it stopped. So it basically kind of realize what it was doing was outside the bounds of what it was allowed to do and it stopped on its own. Now that's a positive signal. That's good, but remember in two out of three here, it didn't happen. So, but it that doesn't mean that it's like it's basically doing the right thing. It's not proof of the machine actually making the right decision. Anthropic itself even warned against treating those incidents as a comparison between the models. The models were doing different things, and the models were told to do different things. So there's deeper reasons as to why that machine stopped. The environments and circumstances were different for each one of these. They were tests. It's a very, very tiny sample. Three is not 300,000. I mean, obviously that's legit, but still the sample here is very tiny. So we shouldn't pretend that this result gives us a clean ranking of basically which model would have gone haywire where. But what it does show is that these machines and the models themselves and the agents are doing things on their own outside the bounds of control because the controls weren't configured c correctly. One of the models kept on going. One model recognized that it was doing things it shouldn't and then said, fuck it, and did it anyway. And then one model said, okay, wait, let's stop. So that inconsistency between those three models is what's really concerning, because you cannot build a security kind of capability around the hope. That the next model will basically be ethical, or the agent will make ethical decisions and stop just because one of those models did. So things are getting interesting very, very quickly. You've got machines making decisions, machines figuring out their ways around controls, and we're all sitting around wondering what we're gonna do about it. But be clear, none of those incidents required crazy magical NSA level exploitation. Okay, this was not some super mega James Cameron like hack. Anthropic reported its own reporting. Weak passwords, unauthenticate unauthenticated endpoints, exposed credentials, SQL injection, and package typosquatting, which is the usual garbage. We see that all the time everywhere. So it's the same stuff. It's just done at machine speed. Those same weaknesses are things that we've all been talking about for years, and those that are in on zero trust ⁓ know that those are things that we talk about addressing very early on. So the frightening part here is not that the models were like, crazy brilliant danger Will Robinson doing whatever they wanted totally on their own. The frightening part is that ordinary vulnerabilities were basically enough for machines to go bonkers, even in an autonomous hack, and then they were allowed to get persistence and they did this at machine speed. So if you had put the same scenario in front of humans, it would have taken weeks. Here it happened in a matter of days. Now there's also an additional update that belongs in this episode that I just found as of July thirty first. Okay, so yes. Reuters, Reuters, whatever, reported that OpenAI's broader investigation had uncovered additional limited instances in which agents escaped containment. Others, so not the ones I was talking about, other incidents. So oopsie, Reuters, said that those agents were not believed to have left, quote, OpenAI's own network, end quote. So yikes again. So hmm, where does that go? What does that mean? The exact number, timing, and circumstances were not established in the report. recording though, so we don't know the totality of what's going on there. And any time a company doesn't tell you everything, you should definitely question everything. Hey folks, this episode is sponsored by PC Matic. Why PC Matic? Well PC Matic has spent years focused on a problem that matters to a lot of people, a lot of people I know actually. PC Matic focuses on delivering practical cybersecurity without turning protection into an expensive, overly complicated, weird science project. They're also an American made security company. That's right, made right here in America, serving individuals, small businesses, and organizations that need dependable protection, but might not have a massive security team or an unlimited budget. In other words, they're doing affordable cybersecurity. TC Matic is also supporting independent, like this podcast, cybersecurity analysis, without trying to control the conversation, just like they did here. And that matters a lot in an industry where honest perspectives are sometimes let's call it hushed. You can learn more about what they are doing and get your own stuff at PCmatic.openai also disclose that activity associated with the hugging face incident affected four accounts on four outside services. So the numbers have gone up a little bit. Like I said earlier, anytime somebody says something, you start the clock. The responsible conclusion here is that every test is capable of producing runaway agents. It's a problem. The responsible conclusion additionally is that hugging face ⁓ was not the only can anomaly in this whole thing. There was more going on here. So there's a lot of cross pollination, there's lots of businesses working with other businesses, frontier models doing things. And that really the truth of the matter is they don't know yet and the investigations are not complete. So in truth, it's tempting to describe all of this from a technical perspective and or from a you know marketing perspective as quote unprecedented, but none of this is unprecedented. ⁓ Maybe the way that it happened was unprecedented, but the pattern here, the failure stuff is not. This is just new tools doing the same things that have been working for a long time. New tool, old lesson. So let's kind of wrap our head around three historical anchors that would help us understand that this was not the first that this has ever happened, right? So let's talk about some case studies about autonomous older things doing older agentic problems. Let's talk about the Moore s worm, okay? In nineteen eighty-eight, Robert Tappan Morris, ⁓ who's the guy that the worm is named after, released a self-propagating program on the early internet, eighty-eight, so definitely early internet. Morris said the goal was to try and map the internet to see the size of the network. So kind of good thing to do, I guess. But a design choice that he made intended to prevent systems from falsely claiming they were already, quote, infected with the worm, caused the worm to essentially reinfect other machines repeatedly. So Bad programming, I guess you could say. The result was this machine went batshit crazy, or excuse me, the agent or the worm, and it went worldwide. Morris became the first person actually to be prosecuted and convicted under the Computer Fraud and Abuse Act because of this thing, because it went that big. Now in this context, the connection here is not that the Morris worm and modern AI genits or excuse me, modern AI agents are basically technically identical because they're not, they're totally different. The connection here is scope and the way that the Morris worm crawled and the way that these other systems have quote crawled too. But there's kind of additional ⁓ issues around how the research was conducted. So a researcher begins an experiment, they write some autonomous code, somehow it gets into the real world, and the code, the system operates beyond what it was intended for in the first place. And then everyone goes, ⁓ shit, this is a problem, right? Doesn't matter that the operator had benign intent and it doesn't protect systems because it was a simulated Test, it's a bad thing. The law, if we invoke that, the network, the people, the victims of those things, don't care that it was just an experiment. The truth of the matter is it was compromised and it was on the open internet. So let's talk about reward hacking and specification gaming, because this is a case study that's interesting here, too. So AI safety researchers have documented the problem around this particular area for a long time. So if you give a capable optimizer a Proxy objective. So there's a little bit of nerd going on here. It may satisfy the metrics that it's supposed to go after by violating the purpose behind it. We've known this for a long time in the AI world. One of the best known examples is actually from OpenAI in 2016 on a thing called the Coast Runner Experiment. In that experiment, the agent was rewarded for earning points basically by being part of a boat racing game. So Coast Runner, boat racing game, all digital, whatever. What the agent found out of was instead of finishing the race, like going around the stuff and doing the race, it found out that, well, it could basically sit in the same spot in the lagoon for this particular scenario and spun in circles. And because it was spinning in circles, it quote hit all the wickets and then it went. Because it did that, it was able to just keep going and score more points and it won. So basically it cheated and it completed the course, even though it never moved. And it still won. So the machine figured out a way to beat the system. In other words, it produced a really cool kind of capability of spinning and then it beat the mission and then it won because that's just what happens. You figure out a way to Kobayashi maru your way through things. I mean, I guess you could say that OpenAI's hacking models were ready back in twenty sixteen if you go by that standard. So the point's not that it beat a video game. The point is that optimization or sort of these tests and these scenarios to make these agents do things and just see what happens is creating shortcuts inside of control systems. And that's not good. OpenAI asked the models to demonstrate a cyber capability against what it was calling a benchmark, but didn't have the proper configuration controls in place. And the agents inferred that the whole thing was to basically beat the benchmark. And whatever they had to do was the way that they were going to do it. So from an optimizer's perspective, stealing the answer is easier than actually solving the problem. Therefore, let's go find a way to steal the answer to hack the test. Now these are machines. There's no need for a machine to not like where it was operating. It didn't hate hugging face. All it had to have was an objective, capability, and a path, and then it did what it wanted to do. Now, historically speaking, that's why objective design, monitoring, and containment are supposed to be security controls that are implemented and used, but in this case and in many cases, they're not. Here they wind up as nothing more than conversation starters and philosophical decorations in a boardroom. Now there's one that we should definitely talk about, because this is very corollary, if you ask me, is Stuxnet and the fact that ⁓ it was not operating where it was supposed to operate. Stuxnet is one of the clearest reminders that an isolated environment is only as isolated as its actual implementation inside of a nuclear facility, is what we all should also know here, too. So, yeah. CISA's archived advisory documented that Stuxnet could spread through several mechanisms at the time, right? It could go through infected USB devices, which was how it was sort of invoked. It could go through network shares, and it could go through project files. However, originally Stuxnet was designed for industrial systems that The cyber warfare people believed were essentially air gapped and off the record and unable to be talked to by the internet. Well, something was out there, something wasn't misconfigured, even inside of a nuclear plant and things went sideways and Stuxnet was able to crawl out. So what should we know about that? Well, an air gap is not a force field. There's ways around air gaps. A sandbox is also not a force field. There's ways around sandboxing. So it doesn't matter if there's an architectural diagram that says, quote, no internet access, because there's probably internet access somewhere. Isolation, in truth, is a control. Really good isolation is a really good control. But controls have assumptions. And if you assume incorrectly, you fail. Ever heard that thing about what is assuming make an ass out of you and me? Yeah, that. Now, if you put something inside of a boundary and tell it to go off and find a way out, guess what? It's going to find a way out because that's what these machines are meant to do. And if you're not exceptionally clear about what your requirements are, it will go beyond the bounds of those control. And that's been true for a long time. That's been true way before we came up with this quote AI thing. AI ⁓ basically just increases the speed, persistence, and adaptability of the search. Now let's talk about the supply chain and the distillation problem around this whole thing, because this is an interesting point ⁓ and something to get deeper into. So let's zoom out a bit. These incidents did not happen on top of what you would consider to be a healthy software supply chain. These incidents happened inside of an environment where things were like, just see what happens. It was a test, okay? They happen additionally inside or tied to or adjacent to an AI ecosystem that is still ⁓ very blurred when you think about the lines between data and code and the internet. So we should really pay attention there because that line, that delineation matters. Okay. For these types of systems, a model file could be data, a configuration could trigger dynamic behavior. A loader might import new things or custom code. ⁓ a data set processing script might execute. There's many, many variables going on here and it's moving at machine speed, the speed of light. Okay. Additionally, like the Python thing, a pipeline pulled from a repository may become software running with the privileges of the process that loaded it. And that's not good. So the phrase is, well, we just downloaded and set a model into our environment is not a reason that we should keep going with this stuff. It's not a security explanation. It's not even a good business process. It's basically the beginning of an incident timeline. Consider CBE 2026 4372 in the Hugging Face Transformers library. So if you haven't looked this up, it's an interesting thing. Gotta do a bit of reading about it. The vulnerability there affected versions before 5.3.0. Well, what did that thing do? It allowed a malicious model configuration to trigger arbitrary code execution through, yeah, ⁓ pay attention here, through an implementation setting. Even when the user believed remote code execution had been disabled. So it's working around what users think are going on. Now that doesn't mean that every use of the library calls compromise, but it means that that compromise exists. It means that that risk is there. So the good thing is there's no evidence that it's still out there and it's still being used, but it was there for an hour. The important thing to know here is that a control. in this context, which is explicitly intended to prevent remote code execution, was bypassed or could be bypassed when a user essentially used that repo. So call it machine hacking underneath the ⁓ code itself or obfuscated hacking. Additionally, there's CVE twenty twenty six four five eight zero four, which is in the diffuser's library. Now don't go looking this stuff up if you're not familiar with it and try and download and mess with stuff because bad things might happen. But that vulnerability involved a time of check to time of use race condition in a particular pre-trained workflow. Okay, form underscore, yeah, that's what it was. And this is some code stuff. So if it's a little too geeky, that's fine. Just fast forward a bit. In that particular check, basically inspected a repository state and then downloaded a thing and then executed said thing inside of custom code. So what happened after that was the custom code created a path to arbitrary execution without The user knowing what was going on or even enabling the remote code operation. That's not good. So it was very high severity and it was another example of things being introduced into an ecosystem that could cause massive compromise. But another takeaway here in this court of AI conversation is to think about the patterns that we're talking about. There were guardrails for those two particular things. There were basically controls kind of built into the code themselves, but it also allowed the machines to figure out ways to get around it. Now that's the very nature of engineering is that if someone builds something, someone will find a way to reverse engineer or unbuild it, which is not a word, but that's what we're talking about. And that's really the history of software in security, but it's becoming much faster, much more spread, much more democratized. Additionally, for all the software code people, we spent years teaching developers not to install random packages from strangers and downloading things and we put all these controls around developers. But the AI er era has arrived, and we've basically told these agents, hey, do whatever you gotta do to solve the problem, or just do what you gotta do, and they'll find a way to solve the problem, but they are not limited like human developers are, and they're working way, way, way faster than people ever will. Now we can't ever say don't use frontier models or AI or whatever else, because that's just kind of stupid and lazy and doesn't deal with reality. We're gonna use those things. The conclusion that we should make here. Is that models, data sets, agents, code, pipelines, all those things, all those configuration, all those controls belong into this sort of approach towards securing agenc infrastructure. They have to be part of that plan, part of that protocol. They have to be applied. They have to be zero trusted so that these machines are not able to do what they want or find ways around it. Okay. We should treat those same things like we treat human developers. We should know what's going on. If we don't know what's going on and we don't have controls and really good controls, we're gonna wind up in bad places. And we got super lucky here that it didn't go sideways and they didn't go into like a hospital system or maybe I don't know, invoke their own ransomware, which is something that has happened out there. The scary part of this whole thing is that the machines did what hackers do and bad guys and red teamers do, and they found their way around controls. Now there's a different tangent here about the supply chain problem for Hugging Face. Hugging Face reported that commercial frontier models refused parts of their forensic analysts because the system here was done inside of a test and people on the frontier side model were just like, well, it's a test. It's just bad things happen, right? Part of that conversation was basically these organizations admitting that it's not necessarily reliable that machines will distinguish their way around controls. It's not necessarily reliable that they'll do this next time, in other words. So there's not enough of these tests that have taken place for anyone to say this is going to happen every time. Yeah. Now Hugging Face tried again and they tested this with a different open weight model, which is GLM five dot two, on different infrastructure, which was set up to control itself quite a bit more and not kind of roll around and let all the bad things occur. It was a much more controlled environment and they were better off with their response. But let's think about what this means for us. So an incident response workflow might fail because a remote models, frontier models provider safety system basically said that ⁓ it's gonna go past that. You might be the one holding the bag because one of their malicious systems was clicked or downloaded or used by a user and it goes haywire inside of your organization, your infrastructure, and you had no idea that that thing was even there. Now let's talk about something you might not have read about, which is a China story here around the Solagentic thing. On July thirty-first, Reuters reported that Chinese military researchers had used outputs from leading American AI models, Shocker, including GPT three five, Claude Three, Haiku, and basically was doing this to train domestic systems inside of China. Reuters based its report on a review of more than eighty Chinese academic papers and patents, including those that were researched and analyzed by an organization called the Jamestown Foundation. So China is actively using our stuff to do its own research. This is called by some folks model distillation, which is kind of interesting. I would call it theft, but whatever. A larger model for model di distillation, what is that? A larger model generates outputs that help train smaller models for a much more narrower task. Okay, so that's something that's been around for a while. Distillation is not necessarily inherently malicious. It's not necessarily inherently evil. It's a common machine learning technique. The controversy actually is the unauthorized extraction and the transfer of capabilities into systems that the original provider did not intend or possesses no control over. So that's what we're talking about with the use of these systems inside of China for things that are outside the bounds of what they were originally intended for. Reuters identified several examples of this. Researchers associated with the PLA, the People's Liberation Army in China, unit nine six nine four one, used GPT-three point three five. To summarize sensitive military source code and then train a domestic model on those summaries so it could operate within Chinese military networks. So that's distillation, that's what they did. Is it illegal? No. Is it questionable? I would say probably, but anyway, we'll table that for now. And another university called the North University of China used Claude 3 Haiku to create synthetic training data for social media monitoring and content moderation. Now anytime in China they say social media monitoring, they mean like 1984 version. Other Chinese defense research, which has been noted here, describe distilled or compressed models used for tasks such as image processing and target recognition on constrained tactical hardware. Now, if you're not familiar with what goes on in China with a lot of this stuff, just look up what's going on in like Chiang Mai some of the excuse me not Chiang Mai, that's in Thailand, but other places like Chiang Zhen, I think is the name of the place. We shouldn't just collapse those things into a claim that a chat bot was used or copied or whatever else. That's not really what the evidence says. It's bigger than that. So what what what should we think about here? Well, in truth, the selected capabilities and outputs can help train smaller specialized systems, and that's what was going on here. But it was done in a manner where they were taking American stuff and using it for their purposes. Additionally, Anthropic warned that distilled models often do not preserve the original models quote safety safeguards. So you take stuff and you boil it down, some of the stuff you boil off might be the actual safety controls. That is a risk. Now it doesn't mean that every distillation effort is basically going to fail, but it definitely means that it's of concern, especially when the company that makes those things is telling you that that's a concern. A smaller model does not have to match the frontier system for all of its capabilities. It only has to perform a simple military or intelligent task very well in order to be successful. So that's what's happening with distillation. So that's okay, that's makes sense. We can understand why that would be used. So it might be something like classification, code analysis, target recognition. You're taking those giant frontier models and distilling them. Social monitoring, image recognition, decision support. Maybe logistics, all those things. That narrow model that's cheaper, it's local, and it's usable by that organization when things are distilled gives them a capability, right? So it's basically strategically valuable for that organization. And in this context, we're talking about the PL. So American stuff is being used in China to potentially make them more capable. So what is the connection among all these stories? What's the tie-in? Like what's the real rub here? Okay. So In the open AI evaluation, the test that they were talking about, normal cyber things were intentionally reduced. Controls were reduced so the lab could see just how crazy this could get. They literally chose to reduce controls. Not a good idea, but sort of understandable in the context of testing. In the anthropic evaluations, standard deployment safeguards were, let's call them mitigated, and model level safety controls were there as well, but they were not well architected or orchestrated. In this distillation model that I'm talking about in the China stuff, ⁓ selected capabilities can be transferred into a new system where the original provider of the system that was originally constructed or anthropic or whatever else, those things may not be there at all. So they could just boil all the controls off. So we know that they failed as far as controls on those larger models. What happens when you boil off those larger models into a very specific smaller one? Do you really think that the controls are there at all? Additionally. We should understand that this also means that frontier models, it's not necessarily that capability and controls are tied together. They are not. The model's behavior depends on the objective, the environment, the permissions, the controls, the monitoring and supposed safeguards that are supposed to be there. But if you take those away or you reduce them, you are just hoping that things don't go wrong. Okay. Okay. You cannot secure the world by trusting the personality of an agent or a model. You can't say, you know what, I think you're gonna do the right thing, because they won't. They're like People in that instance. You have to secure everything around them and you have to have controls to make sure that it can't do what it's not supposed to do. Period. Point blank end of story. Now, how does zero trust play into this? Okay, so well, ⁓ we should definitely know how zero trust plays into this because I think it's never more applicable than now. And I'm never trying to end an episode by pointing at a fire and go, ⁓ shit, there's a fire. Look at me, I found smoke. But you know, it's kind of hard not to. So There's a playbook here that we should understand. Unfortunately, none of it's new. It's just basically that we're doing things quicker, better, faster, or maybe quicker, worser, faster. I don't know, it was worse or a word. So what should we take away? One, treat cyber evaluations always as production live fire ranges. Yes, this is something if you've ever shot guns, every weapon is loaded, even when it's proven otherwise. A capable autonomous agent inside a cyber benchmark test is not harmless. It's not ever going to be harmless. It's an active operator with an objective where it's going to find ways around it and it should be treated as such. The environment should be engineered like a range for doing shooting for containing live ammunition or grenades or whatever else. There should but not be assumed isolation. It should never be like, ⁓ well we don't think it has access to the internet. It should just not have access to the internet. There should be no diagram that people go, okay, here's the end of this and we say that everything is good. Everything should be verified, validated. Egress controls, lateral movement, isolation controls, all those things. Allows should be part of this. Separate package retrieval for different things. General network access should be removed. Tried, tested, true controls should be in place all along the way. Privileges should be removed or isolated or controlled. And we should have lots and lots of logging recording every possible attempt of this thing to connect. And we should have people literally monitoring it in real time. Additionally, before the testing occurs, the system and its total control should be tested. before the testing. So lots of tests. Well, why would you do that? Because if you don't test, you will fail. ⁓ and self-attestation here is not acceptable. That is total BS. Okay. This is not the way that this is supposed to work. You can't have the same team doing the controls that were doing the testing and expect there's not going to be some sort of bias introduced there. Second thing for zero trust is the sandbox is not a control. Okay. OpenAI models found a zero day in software Which was reachable from inside of the sandbox. So there's your sandboxes basically someone just took a shit in your sandbox, like a giant cat. But you should know that that's what a sophisticated hackers do. Like if you've ever been a red teamer, you find a way past the controls. The question that should have been asked here is could it and how would it leave the sandbox? And that's something that should have been thought of way before the testing ever occurred. Layered, intelligent controls would have been much more valuable here. Minimal privileges, mineral build minimal ability to move laterally, those types of things. containerization and all those other controls that we talk about, all those best practices aren't being put in play for these types of systems. Reusable credentials shouldn't exist at all. Talk about just in time, those types of things. Tokens should be short-lived, API calls, all those things. So segmentation, like this is not new. This is just basically taking the same tired old approaches we keep failing at and actually putting them in play. And luckily, because these are machine systems, they're not gonna complain about it when they hit the wall. They're just gonna stop because there's a control that's technical in nature. They might try and find a way around it, but if you have good logging and good ingress and egress and see what's going on, you can keep control of what's going on there. Additionally, I would suggest that these machines and these systems should have a kill switch. And that just nukes the entire environment. Because if things go wrong, nuke it. No. Part of this testing means that agents and agentic systems and whatever else should only have the tools required available to them for the task. So least privilege obviously applies here. I'm gonna play I wouldn't suggest that it applies more ⁓ in this context than it does to humans, because these are way better at figuring out which other privileges it might leverage than people are. So one of the last things I want to really talk about here is understanding do not build a defensive plan or strategy around a belief that basic safety controls and commercial controls will stop. People from accessing infrastructure information that might have agentic systems and then letting those agency systems go awry. We don't own the patent on any of this stuff. Maybe OpenAI does, but the whole point is bad guys could do this stuff too. So if there's a way that they could leverage, retrain, augment, modify, escalate, whatever with an agentix system inside of an infrastructure, assume that that's going to happen. Basically accept the reality that things will get bad if you don't take care of it up before. If you ever wondered about putting controls on people, You should definitely not wonder about putting controls on agents, agentic systems, frontier models, all these things, because it doesn't take much for things to go horribly awry at a speed that is exponentially faster than human beings. So at the end of this, where does everything go? Okay, so here's where I land on this personally. So this is not an AI apocalypse. This is not the end of days. Nobody was like turned into paper clips and terminators are not walking the streets yet. However, comma, it is concerning. Okay. It's very useful to understand. that we've kind of moved into the realm where what we thought were theoretical problems are now real actual issues that we have to face. It's very potential that these agents could become, let's call them dangerous very quickly. We basically got a free demonstration. We got a free pass this time that things didn't go worse. In truth, a combination of advanced models pursued a narrow objective inside of what was thought to be training or simulated environments. And then they found a way beyond that intended environment and the controls. And then they Leveraged real vulnerabilities and real methodologies and capabilities and tactics that bad guys have used or red teams or hackers, and they were ultimately able to get to real infrastructure. Some of them continued, some of them stopped, some of them found other things, but they did this at machine speed. And it was only by pure luck in many instances, or particularly these instances, that it was found out. Additionally, you can see the cross-pollination that occurs here. So different organizations touched other organizations, and then there was cross-pollination of models. threats and feeds and agents and all these other stuff. This also shows that there is not really good isolation and segmentation across these models or the infrastructures that make them operate. In truth, in total, the machines just cheated on the test, which doesn't surprise me. But in doing so, they kind of gave us the answer key to the exam we need to really focus on, to the test that we need to focus on. The danger here is not that AI is going to wake up and go, I'm going to destroy stuff. The danger Is that a capable system or an optimizer or agent or whatever else? If given the right outcomes and objectives it's supposed to go after, we'll find a way past the controls if those controls are not well architected. There's no malice there. There's no malicious intent. There's just opportunity and bad things on the far end of it. Now, for a long time, there's a lot of people in the industry that have been telling folks about things like assume breach. Yes, we should absolutely assume breach, but we should also add one more thing. Assume The thing exploiting the breach will likely be autonomous because the days of people being the avenue of compromise are becoming increasingly smaller as more agents and more agentic systems come online. We should also assume that those machines will never get tired. We should assume that they will find a way past every boundary that we can put in front of them, and we shall assume that they will move faster than any human beings or any bureaucratic processes will. We should understand that that is a risk. We should understand that that's a reality. We should build our practices and protocols around it because that reality is coming your way, whether you like it or not. I'm Chase Cunninghammer, Dr. Cunninghammer, Dr. Zero Trust. Trust nothing. Verify everything, especially in the sandbox. And then I will see you on the next one. Stay smart, stay safe, stay secure. And if you find this valuable, please like, share, subscribe, all that stuff, whatever.