In Japan, when Anthropic announced Claude Mythos in April, there was an enormous urgency in the Japanese government and the news was everywhere on the media. Finance Minister Satsuki Katayama called it “clear and present danger”. Prime Minister Sanae Takaichi commented that “finding vulnerabilities is a race against time” when she directed her team to “come up with concrete measures and implement them”.
Three months have passed since then and nobody seems to talk about Claude Mythos on Japanese media anymore. I searched for related news to update myself. On August 27th a couple of major newspapers reported that Japan’s three megabanks finally gained access to Mythos and started evaluating how to use it. That is how the newspaper article is written, if I translate directly. TV and other media didn’t actively report on it, as far as I am aware.
I also searched on English media. There were a lot more updates available, including the info about Project Glasswing. The ones that caught my eye are those that started questioning the practicality of Mythos. While still admitting how amazing and terrifying Mythos is at finding vulnerabilities, some sources point out that Mythos might not be so effective on assessing the severity of the vulnerabilities. For example, software supply chain security company Echo mentioned in its Mythos Readiness report that many of the issues rated as “Critical” by Mythos turned out to be not so critical. It says that, of the 27 vulnerabilities Anthropic has publicly disclosed, only one of the eight findings Mythos originally rated "Critical" held up under independent review.
This indicates that a human-in-the-loop would be necessary when triaging vulnerabilities. Meanwhile, the number of vulnerabilities to triage could be as many as 200 a day, according to other reports regarding Project Glasswing. If many of them are unnecessary to patch from the human’s point of view, despite how critical they look to the AI’s point of view, triaging and deciding which to patch would be an extremely important task. At the same time, the OpenAI’s Hugging Face incident indicated that a cyber attack could be executed solely by AI agents at machine speed. Triaging hundreds of vulnerabilities at human speed wouldn’t sound practical at all.
The obvious answer might seem to be training AI models further on assessment and triaging, but that is easier said than done. While finding vulnerabilities is a very logical task, the assessment and triaging are not that logical. It might require taking into account many variables of “human-side” conditions including the target users’ needs, user experience, IT resource staffing, budget, etc. Training AI models to be able to judge flexibly considering many subtle human conditions would take long. It should be doable and AI models could eventually learn, but it is just not as simple as learning to analyze codes to find vulnerabilities.
This reminds me of the clinical applications of AI. While AI excels at analyzing medical images and lab test results to find health issues, communicating those findings to a patient is far more complex. Finding as many issues as possible is not always beneficial for the patients, or for the public medical systems either. It would only be beneficial if treating the findings is possible, doable, and makes sense for the patient’s quality of life. This is a very difficult and sensitive issue so I would rather not go on here. I am just suggesting the analogy of finding as many issues as possible while leaving aftercare aside.
Whether and how much to let AI models judge on subtle human factors must be sorted out before letting AI models keep getting better and better at what they are already good at. Even now, they keep improving rapidly on what they are already good at.
Nevertheless, the ability of Mythos to find vulnerabilities may not be so scary. Long before Mythos was announced, the number of cyber security incidents was already skyrocketing. The well-known quote “It is not a matter of if but when” describing your possibility of getting a cyber attack was not a joke. Every cyber security specialist took it very seriously. In that sense, Claude Mythos or any other advanced cyber security AI models did not increase your risk of suffering a cyber attack. The possibility was already 100%. It is just a matter of time. So, if you have time to worry about the AI advancement of finding vulnerabilities, you should take a backup. I don’t doubt you have already been practicing that. The point is, now is the time to reinforce your backup and recovery practices. Minimizing RPO, keeping extra backup off-site, air-gapped, immutable, etc. so that your backup will be intact when attacked. Prepare the incident playbook and drill it regularly. That is what we have to focus on when we have time to worry about Mythos’s capabilities.