Why is token cost the right metric to optimize for in AI inference rather than other measures?
Answered at 1:38 in Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper · Invest Like The Best
Why is token cost the right metric to optimize for in AI inference rather than other measures?
Token cost is the current North Star for measuring AI inference work because it's the industry standard today —but it's not the final unit of measurement. As the guest notes, tokens will eventually give way to outcome-based pricing, where agents manage their own token budgets dynamically based on task requirements rather than companies imposing fixed monthly token allocations.
The practical reality is straightforward: tokens are what everyone measures right now. They're the common language across all AI infrastructure providers, making it easy to benchmark, optimize, and improve delivery costs. This focus creates a clear North Star for engineers and infrastructure companies alike.
What makes tokens temporary as a metric is the shift already underway toward agentic reasoning at scale. Once agents become the primary unit of work—autonomous systems that run for hours, days, or weeks and decide how much computation they need—the entire cost model inverts. Instead of a company predicting an engineer's token budget for the month, an individual agent will self-allocate tokens based on the difficulty and nature of the task it's solving.
As discussed in this episode of Invest Like The Best, this outcome-based future means companies will eventually stop thinking about token efficiency as the primary constraint. Instead, they'll optimize for whether an agent solved the problem—and how much it cost to do so. The token ceases to be the unit of work; the solution becomes it.
"My job is to make the tokens as cheap as humanly possible, and I will achieve that through every layer in the stack available to me."
Guest Speaker — Founder and engineer focused on AI infrastructure optimization, formerly at NVIDIA. Expertise spans semiconductor design, cost optimization across inference systems, and long-running agentic workloads. Currently building infrastructure to serve tokens at scale while architecting the transition toward outcome-based pricing models.
Full answer and detailsRead the whole answer, key facts and every other question from this episode
In short
Token cost is the current North Star for measuring AI inference work because it's the industry standard today —but it's not the final unit of measurement.
Guest · Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper · Invest Like The Best · answer at 1:38
Key takeaways
- As the guest notes, tokens will eventually give way to outcome-based pricing, where agents manage their own token budgets dynamically based on task requirements rather than companies imposing fixed monthly token allocations.
- The practical reality is straightforward: tokens are what everyone measures right now.
- What makes tokens temporary as a metric is the shift already underway toward agentic reasoning at scale.
- As discussed in this episode of Invest Like The Best, this outcome-based future means companies will eventually stop thinking about token efficiency as the primary constraint.
- Guest Speaker — Founder and engineer focused on AI infrastructure optimization, formerly at NVIDIA.
Quote to remember
“My job is to make the tokens as cheap as humanly possible, and I will achieve that through every layer in the stack available to me.”
The full answer · about 1 min
Token cost is the current North Star for measuring AI inference work because it's the industry standard today —but it's not the final unit of measurement. As the guest notes, tokens will eventually give way to outcome-based pricing, where agents manage their own token budgets dynamically based on task requirements rather than companies imposing fixed monthly token allocations.
The practical reality is straightforward: tokens are what everyone measures right now. They're the common language across all AI infrastructure providers, making it easy to benchmark, optimize, and improve delivery costs. This focus creates a clear North Star for engineers and infrastructure companies alike.
What makes tokens temporary as a metric is the shift already underway toward agentic reasoning at scale. Once agents become the primary unit of work—autonomous systems that run for hours, days, or weeks and decide how much computation they need—the entire cost model inverts. Instead of a company predicting an engineer's token budget for the month, an individual agent will self-allocate tokens based on the difficulty and nature of the task it's solving.
As discussed in this episode of Invest Like The Best, this outcome-based future means companies will eventually stop thinking about token efficiency as the primary constraint. Instead, they'll optimize for whether an agent solved the problem—and how much it cost to do so. The token ceases to be the unit of work; the solution becomes it.
"My job is to make the tokens as cheap as humanly possible, and I will achieve that through every layer in the stack available to me."
Guest Speaker — Founder and engineer focused on AI infrastructure optimization, formerly at NVIDIA. Expertise spans semiconductor design, cost optimization across inference systems, and long-running agentic workloads. Currently building infrastructure to serve tokens at scale while architecting the transition toward outcome-based pricing models.
More answers from this episode
- 0:41What is the core business model and technical offering of a token factory API service?
- 2:45What market opportunity exists in serving open source language models at scale compared to proprietary closed models?
- 3:42Why should AI inference for agents prioritize long-running background tasks over low-latency interactive responses?
- 5:13What evidence supports the shift toward longer-running agents as a viable product direction?
- 6:00What is the predicted shift in token consumption patterns between real-time and background workloads over the next several years?
- 6:38How can long-running agents improve deep research and information synthesis compared to interactive queries?
- 7:33What role do long-running AI agents play in autonomous cybersecurity vulnerability detection?
- 8:33How does diversity in AI model capabilities improve the detection of software vulnerabilities compared to relying on a single frontier model?
- 9:20What new product categories could abundant, cheap long-running AI agents unlock for individual users?
- 10:27What types of problems become economically solvable with dramatically cheaper token costs for long-horizon AI tasks?
- 12:59What is the three-layer technical stack approach to dramatically reducing AI inference costs?
- 14:01What is the role of TensorCores in GPU-accelerated AI inference and how were they developed?