How should organizations evaluate which AI models and providers to use for offensive security tasks?
Answered at 9:00 in The Agents Ran All Weekend. They Came Back With Zero Days. ft. Omar Santos, Distinguished Engineer, AI Security @ Cisco · Security & GRC Decoded
How should organizations evaluate which AI models and providers to use for offensive security tasks?
Organizations should define what is 'good enough' for specific tasks rather than always using frontier models like Mythos. For example, static analysis does not need Mythos—cheaper models like Opus or GPT-4o work better. Companies should create an inventory of approved models and use agents for budget optimization, selecting models based on task completion, definition of done, and measuring efficiency to avoid unnecessary token burn.
Full answer and detailsRead the whole answer, key facts and every other question from this episode
In short
Organizations should define what is 'good enough' for specific tasks rather than always using frontier models like Mythos.
Omar Santos, Distinguished Engineer, AI Security at Cisco · The Agents Ran All Weekend. They Came Back With Zero Days. ft. Omar Santos, Distinguished Engineer, AI Security @ Cisco · Security & GRC Decoded · answer at 9:00
Key takeaways
- Organizations should define what is 'good enough' for specific tasks rather than always using frontier models like Mythos.
Quote to remember
“The agents ran all weekend. They came back with zero days: the exploit, the mitigation, even compensating controls, the fix, the unit tests, the regression tests, and documentation.”
— Omar Santos
The full answer · about 2 min
Organizations should define what is 'good enough' for specific tasks rather than always using frontier models like Mythos. For example, static analysis does not need Mythos—cheaper models like Opus or GPT-4o work better. Companies should create an inventory of approved models and use agents for budget optimization, selecting models based on task completion, definition of done, and measuring efficiency to avoid unnecessary token burn.
Key facts
- 400,000 repositories: scale at which Cisco is attempting to find vulnerabilities and simulate attacks
- 75% of unnecessary components: proportion eliminated from lightweight base images like Chainguard or Socket
- Milliseconds to seconds: time frame in which agent drift can occur in non-deterministic agentic environments
People, companies and topics mentioned
More answers from this episode
- 2:34Can AI agents autonomously find zero-day vulnerabilities in live systems without human intervention?
- 4:05What is model alignment and agent drift in agentic AI security systems?
- 16:52What are the three modalities for evaluating whether AI models effectively find zero-day vulnerabilities?
- 22:40What future capability beyond vulnerability finding is the security industry exploring with agentic AI at scale?
- 24:02Can organizations reduce vulnerability whack-a-mole cycles by using memory-safe languages and trusted software repositories?