Oh my, where did all that inference go?

Analysts anticipate Nvidia’s data center revenue to exceed $100B this fiscal year. According to management, about 40% of their data center revenue is currently allocated to inference purposes (vs. 60% for training). This suggests that around $40B of Nvidia’s chips will be dedicated to inference workloads this year.

Yet, we estimate the GenAI application revenue to be only in the range of $5-10B this year, concentrated among a few key players such as OpenAI and Anthropic.

This raises an interesting question/dilemma: why is there such a discrepancy between the expenditure on inference compute and the relatively small revenue generated from GenAI today? In an economically rational world, one would expect revenue to be much higher than inference spend (as an example, cloud hosting cost usually account for only 10% of SaaS revenue)

One possible explanation is that most of the inference is being spent on non-GenAI applications. However, this seems unlikely, as management has repeatedly indicated that GenAI applications have been the primary driver of Nvidia’s recent growth.

Another possible explanation is that companies are heavily subsidizing users. For instance, Github Copilot was rumored to be LOSING $20 per month per user. While this “loss leader” strategy may be feasible for some products in the short run, it is unlikely to be sustainable or widespread.

A more plausible explanation is that the vast majority of inference compute is being spent by existing incumbents for internal initiatives. Examples include Meta’s new “Ask Meta AI” feature that’s integrated into all their apps, or Notion’s rollout of GenAI features (https://lnkd.in/g6xFMQS9), or Klarna’s AI customer support agent (https://lnkd.in/gdKMYn5v). This “internal spend” has driven the majority of inference spend to date.

My rough estimate is that incumbent spending vs. startup spending is currently at least a 5:1 ratio but could be up to 10:1. The reality is that incumbents are typically best positioned to capture the lower hanging fruits at the beginning of any technological revolution. Likewise, it takes time for startups to achieve product-market fit and scale revenue.

However, as an early-stage AI investor, I’m optimistic that startups will increasingly capture a larger share of the pie. Though it will take time, the transformative potential of AI will inevitably give rise to innovative startups that redefine entire industries. Just as Apple revolutionized mobile technology and AWS reshaped the cloud landscape, new AI companies will emerge to challenge the status quo and push the boundaries of what’s possible. These players will establish themselves as the new category-defining players of the 21st century.