Listenly

Why is there a fundamental tradeoff between GPU throughput optimization and latency optimization for AI inference?

Answered at 16:26 in Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper · Invest Like The Best

— questions searchable right now

Why is there a fundamental tradeoff between GPU throughput optimization and latency optimization for AI inference?

GPUs are throughput machines happiest when given lots of work and left to chew through it at peak compute utilization, but recent AI usage pushed toward interactive chatbot tools caring about spitting answers out quickly to users. There is a classic computer science tradeoff between getting a small amount of data through the system quickly (narrow and fast) versus running wide and slow, which manifests in GPUs through batching where larger batches fill compute units but add latency per individual user request.

Unknown on Invest Like The BestAll answers from this podcast →
Full answer and detailsRead the whole answer, key facts and every other question from this episode

In short

GPUs are throughput machines happiest when given lots of work and left to chew through it at peak compute utilization, but recent AI usage pushed toward interactive chatbot tools caring about spitting answers out quickly to users.

Guest · Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper · Invest Like The Best · answer at 16:26

Key takeaways

  • GPUs are throughput machines happiest when given lots of work and left to chew through it at peak compute utilization, but recent AI usage pushed toward interactive chatbot tools caring about spitting answers out quickly to users.

The full answer · about 1 min

GPUs are throughput machines happiest when given lots of work and left to chew through it at peak compute utilization, but recent AI usage pushed toward interactive chatbot tools caring about spitting answers out quickly to users. There is a classic computer science tradeoff between getting a small amount of data through the system quickly (narrow and fast) versus running wide and slow, which manifests in GPUs through batching where larger batches fill compute units but add latency per individual user request.

More answers from this episode

 ·  All answers from Invest Like The Best →
—
—
0:00
0:00