Inference cost
The price of running a model in production — usually measured per token or per request — shaped by model size, hardware, batching, caching, and API margins.
Learn
When to use it
Use inference cost metrics when evaluating the financial viability of deploying a model in production. Understanding inference costs helps balance model performance with budget constraints, enabling decisions like choosing between larger models with higher accuracy or smaller, more cost-effective models.
Quick example
In ChatGPT, each user query incurs an inference cost based on the model's size and the number of tokens processed. OpenAI uses inference cost metrics to optimize resource allocation and pricing tiers, ensuring that the service remains both affordable for users and profitable for the company. Here, inference cost is a key factor in determining how to scale the service efficiently.
Ecosystem
Inference cost is closely linked to model deployment and optimization strategies, influencing choices around hardware and batching.
┌─ model size ─┐
request → inference cost → API margins
└─ hardware ───┘
Misconceptions
| Misconception | Rebuttal |
|---|---|
| Inference cost is fixed | It varies with model size and usage patterns |
| Only model size affects cost | Hardware, batching, and caching also play roles |
Trade-offs
- Cost savings — may require using less powerful models
- Performance — higher accuracy can lead to increased costs
- Scalability — lower costs enable broader deployment