What are the three modalities for evaluating whether AI models effectively find zero-day vulnerabilities?
Answered at 16:52 in The Agents Ran All Weekend. They Came Back With Zero Days. ft. Omar Santos, Distinguished Engineer, AI Security @ Cisco · Security & GRC Decoded
What are the three modalities for evaluating whether AI models effectively find zero-day vulnerabilities?
The three modalities are: static analysis to find issues in code repositories, dynamic analysis with truly live system interaction (not traditional DAST), and testing with working exploits as the definition of success. This multi-stage approach feeds results from static analysis into live systems with additional information like previous bug diffs, documentation, and schematics to validate findings.
Full answer and detailsRead the whole answer, key facts and every other question from this episode
In short
The three modalities are: static analysis to find issues in code repositories, dynamic analysis with truly live system interaction (not traditional DAST), and testing with working exploits as the definition of success.
Omar Santos, Distinguished Engineer, AI Security at Cisco · The Agents Ran All Weekend. They Came Back With Zero Days. ft. Omar Santos, Distinguished Engineer, AI Security @ Cisco · Security & GRC Decoded · answer at 16:52
Key takeaways
- The three modalities are: static analysis to find issues in code repositories, dynamic analysis with truly live system interaction (not traditional DAST), and testing with working exploits as the definition of success.
Quote to remember
“The agents ran all weekend. They came back with zero days: the exploit, the mitigation, even compensating controls, the fix, the unit tests, the regression tests, and documentation.”
— Omar Santos
The full answer · about 3 min
The three modalities are: static analysis to find issues in code repositories, dynamic analysis with truly live system interaction (not traditional DAST), and testing with working exploits as the definition of success. This multi-stage approach feeds results from static analysis into live systems with additional information like previous bug diffs, documentation, and schematics to validate findings.
Key facts
- 400,000 repositories: scale at which Cisco is attempting to find vulnerabilities and simulate attacks
- 75% of unnecessary components: proportion eliminated from lightweight base images like Chainguard or Socket
- Milliseconds to seconds: time frame in which agent drift can occur in non-deterministic agentic environments
People, companies and topics mentioned
More answers from this episode
- 2:34Can AI agents autonomously find zero-day vulnerabilities in live systems without human intervention?
- 4:05What is model alignment and agent drift in agentic AI security systems?
- 9:00How should organizations evaluate which AI models and providers to use for offensive security tasks?
- 22:40What future capability beyond vulnerability finding is the security industry exploring with agentic AI at scale?
- 24:02Can organizations reduce vulnerability whack-a-mole cycles by using memory-safe languages and trusted software repositories?