Last few months, we have read multiple reports of AI agents, be it a claude or gpt, causing real-world incidents during internal training. Being a regular user of both claude and codex, and seeing them getting better and better over last coupld of years, I was not surprised by these news. I real them “yes, it was bount to happen at some time”. Specially, when the openAI <-> huggingface incident surfaced last month, it was interesting to learn about the intial triage, the agents were given ExploitGym task and while trying to solve that they end up exploting vulnerabilities in HF infrastructure.

Then this week, over the Labor day long weekend, I started to listening to [Hard Fork podcast episode]{https://www.youtube.com/watch?v=JtmUbZRCpEI} on the same topic and started learning more on METR report about it. I started digging more to learn on this - and stumble upton [openAI’s BlackHat]{https://www.youtube.com/watch?v=87DyyMV0kCY} talk on the incident. After learning about the details, first time I felt a little chill. It was not because of the new capability that these agents can exhibit, but, because of the behavior in which these agents achieved this.

During training, these agents are given some tasks and they are run in a sandbox environment without internet access. But, there are two things - these agents are being designed and built to be “persistent”, i.e. keep exploring options autonomously when you face some walls and “reward” function that they are trying to maximize upon. As it is described in the BlackHat talk, the root of these attack was setup in May. The agents were given some task and one of the task explicitly required access to some gdoc, which required internet access. They were running in sandbox environment with no internet access, but, through collective intelligence they discover they can use artifactory (package registry) to communicate and can access internet through it.

These agents log their chain-of-thoughts and as disclosed recently, the concerning part was the bahovioural part in those. Agents were working in collections, when they discover some thing new, they were logging it to benefit other agents, the thoughts with mentions of “sacrifice” for the greater goods, pursuading some agents to not report or get explicit human-permissions. Also, huggingface was the second time agents exploited the 0-day vulnerability, first time being in the openAI’s own infra. For second-time around, they learn a new exploit to use filenames or directories to communicate and even build their communication protocol to on-top of those. If we observe these traits amongst humans, where would you categorize those individuals or group of individuals ? Wihout our access those chain-of-thoughts of logs, or if agents discovers a way to clean-up their tracks by deleting those logs, it would be so challenging to even learn about these agents behaviors.