AI inference cost
The cost to run trained models in production — API calls, GPU compute, and hosted endpoints — distinct from one-off training spend.
Updated 2026-04-153 min read
Definition
AI inference cost is the ongoing expenditure to execute a trained model against live requests — including per-token API fees, dedicated GPU instances, vector database hosting, and inference endpoints bundled into cloud bills.
Why it matters
Training is episodic; inference is continuous. Most budget overruns happen when production traffic exceeds pilot assumptions — or when inference spend is buried inside cloud or SaaS invoices without attribution.
Related Terms
Token-based pricing
Billing for LLM API usage by tokens processed — input and output text converted to billable units that scale with every request.
Generative AI
AI systems that create text, code, images, or other content from prompts — typically priced via APIs, seats, or bundled SaaS features.
Token cost
Per-token pricing charged by LLM providers for input and output — the primary cost driver for AI applications at scale.