Listenly

How does GPU batching create a fundamental latency-throughput tradeoff for AI inference services?

Answered at 17:45 in Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper · Invest Like The Best

— questions searchable right now

How does GPU batching create a fundamental latency-throughput tradeoff for AI inference services?

Batching groups multiple users' work together to exploit GPU parallelism, but you do net more work when running a large batch, meaning each individual token or user request spends longer on the GPU being carried with other people's traffic. This is analogous to a bus versus private transit—a bus serves many people and must work for everyone, taking a slower path with stops, while private transit takes the direct path.

Unknown on Invest Like The BestAll answers from this podcast →
Full answer and detailsRead the whole answer, key facts and every other question from this episode

In short

Batching groups multiple users' work together to exploit GPU parallelism, but you do net more work when running a large batch, meaning each individual token or user request spends longer on the GPU being carried with other people's traffic.

Guest · Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper · Invest Like The Best · answer at 17:45

Key takeaways

  • Batching groups multiple users' work together to exploit GPU parallelism, but you do net more work when running a large batch, meaning each individual token or user request spends longer on the GPU being carried with other people's traffic.

The full answer · about 1 min

Batching groups multiple users' work together to exploit GPU parallelism, but you do net more work when running a large batch, meaning each individual token or user request spends longer on the GPU being carried with other people's traffic. This is analogous to a bus versus private transit—a bus serves many people and must work for everyone, taking a slower path with stops, while private transit takes the direct path.

More answers from this episode

 ·  All answers from Invest Like The Best →
—
—
0:00
0:00