Listenly

How should organizations evaluate which AI models and providers to use for offensive security tasks?

Answered at 9:00 in The Agents Ran All Weekend. They Came Back With Zero Days. ft. Omar Santos, Distinguished Engineer, AI Security @ Cisco · Security & GRC Decoded

— questions searchable right now

How should organizations evaluate which AI models and providers to use for offensive security tasks?

Organizations should define what is 'good enough' for specific tasks rather than always using frontier models like Mythos. For example, static analysis does not need Mythos—cheaper models like Opus or GPT-4o work better. Companies should create an inventory of approved models and use agents for budget optimization, selecting models based on task completion, definition of done, and measuring efficiency to avoid unnecessary token burn.

Omar Santos on Security & GRC DecodedAll answers from this podcast →
Full answer and detailsRead the whole answer, key facts and every other question from this episode

In short

Organizations should define what is 'good enough' for specific tasks rather than always using frontier models like Mythos.

Omar Santos, Distinguished Engineer, AI Security at Cisco · The Agents Ran All Weekend. They Came Back With Zero Days. ft. Omar Santos, Distinguished Engineer, AI Security @ Cisco · Security & GRC Decoded · answer at 9:00

Key takeaways

  • Organizations should define what is 'good enough' for specific tasks rather than always using frontier models like Mythos.

Quote to remember

“The agents ran all weekend. They came back with zero days: the exploit, the mitigation, even compensating controls, the fix, the unit tests, the regression tests, and documentation.”

— Omar Santos

The full answer · about 2 min

Organizations should define what is 'good enough' for specific tasks rather than always using frontier models like Mythos. For example, static analysis does not need Mythos—cheaper models like Opus or GPT-4o work better. Companies should create an inventory of approved models and use agents for budget optimization, selecting models based on task completion, definition of done, and measuring efficiency to avoid unnecessary token burn.

Key facts

  • 400,000 repositories: scale at which Cisco is attempting to find vulnerabilities and simulate attacks
  • 75% of unnecessary components: proportion eliminated from lightweight base images like Chainguard or Socket
  • Milliseconds to seconds: time frame in which agent drift can occur in non-deterministic agentic environments

People, companies and topics mentioned

CiscoAnthropicOpenAIGoogleModel Context Protocol (MCP)ChainguardSocketOpenSSL

More answers from this episode

 ·  All answers from Security & GRC Decoded →
—
—
0:00
0:00