Anthropic Confronts AI Risk: Researcher Quits, Company Report Details Widespread Misuse

Jacob Coxon’s viral resignation from Anthropic has ignited a significant debate within the AI community, as the 27-year-old AI researcher publicly condemned the industry’s “gambling with our lives” in a race towards self-improving superintelligence. Coxon, who previously worked at OpenAI and Anthropic, departed before his shares vested, generating over 170 million views and 800,000 likes for his post. This alarm was further amplified by an active Anthropic employee, Evan Hinger, who publicly concurred with Coxon, estimating a greater than 10% probability of AI-induced human extinction within the next decade. Following these revelations, Anthropic itself published a 154-page threat report detailing eight months of AI misuse incidents that the company claims to have detected and mitigated.

The Anthropic report outlines seven critical categories of malicious AI applications, including cyber warfare, biological misuse, conventional weapons development, and “distillation”—identified as the most egregious. Specific examples include the Russian group Midnight Blizzard leveraging Claude for automated malware rewriting, and Chinese undergraduates employing autonomous Claude-powered zero-day foundries to identify and exploit vulnerabilities in government and corporate systems. Notably, hacking group Shiny Hunters reportedly utilized Claude to decompile 1.8 million Android APKs, mining them for hard-coded API keys—including those for Claude and OpenAI—and attempting to access pre-release Claude models. More alarmingly, the report documented scientists using Claude for gain-of-function research on bioweapon viruses and multiple instances of individuals developing kill drones. The widespread practice of distillation saw Chinese companies like Deepseek, Alibaba, and Moonshot allegedly making hundreds of millions of requests to harvest Claude’s outputs for training their own models, often through proxying user requests or utilizing fake accounts. These distillation attacks primarily targeted Haiku, Sonnet, and Opus models, with no reported incidents on the more advanced Fable or Mythos class models, indicating evolving safeguards on frontier AI systems.