Listenly

What are the three modalities for evaluating whether AI models effectively find zero-day vulnerabilities?

Answered at 16:52 in The Agents Ran All Weekend. They Came Back With Zero Days. ft. Omar Santos, Distinguished Engineer, AI Security @ Cisco · Security & GRC Decoded

— questions searchable right now

What are the three modalities for evaluating whether AI models effectively find zero-day vulnerabilities?

The three modalities are: static analysis to find issues in code repositories, dynamic analysis with truly live system interaction (not traditional DAST), and testing with working exploits as the definition of success. This multi-stage approach feeds results from static analysis into live systems with additional information like previous bug diffs, documentation, and schematics to validate findings.

Omar Santos on Security & GRC DecodedAll answers from this podcast →
Full answer and detailsRead the whole answer, key facts and every other question from this episode

In short

The three modalities are: static analysis to find issues in code repositories, dynamic analysis with truly live system interaction (not traditional DAST), and testing with working exploits as the definition of success.

Omar Santos, Distinguished Engineer, AI Security at Cisco · The Agents Ran All Weekend. They Came Back With Zero Days. ft. Omar Santos, Distinguished Engineer, AI Security @ Cisco · Security & GRC Decoded · answer at 16:52

Key takeaways

  • The three modalities are: static analysis to find issues in code repositories, dynamic analysis with truly live system interaction (not traditional DAST), and testing with working exploits as the definition of success.

Quote to remember

“The agents ran all weekend. They came back with zero days: the exploit, the mitigation, even compensating controls, the fix, the unit tests, the regression tests, and documentation.”

— Omar Santos

The full answer · about 3 min

The three modalities are: static analysis to find issues in code repositories, dynamic analysis with truly live system interaction (not traditional DAST), and testing with working exploits as the definition of success. This multi-stage approach feeds results from static analysis into live systems with additional information like previous bug diffs, documentation, and schematics to validate findings.

Key facts

  • 400,000 repositories: scale at which Cisco is attempting to find vulnerabilities and simulate attacks
  • 75% of unnecessary components: proportion eliminated from lightweight base images like Chainguard or Socket
  • Milliseconds to seconds: time frame in which agent drift can occur in non-deterministic agentic environments

People, companies and topics mentioned

CiscoAnthropicOpenAIGoogleModel Context Protocol (MCP)ChainguardSocketOpenSSL

More answers from this episode

 ·  All answers from Security & GRC Decoded →
—
—
0:00
0:00