What is inference cost and how is it calculated?
Inference cost is what a provider bills to run a model for each answer, per million input and output tokens. Here is the formula with worked examples.
Published on
Inference cost is the price of running a trained AI model to produce an answer. Providers bill it per token, the unit of text a model reads and writes, at one price per million input tokens and a higher one per million output tokens, so an answer costs its input tokens plus its output tokens at those two rates.
The formula
One answer costs:
input tokens × input price ÷ 1,000,000 + output tokens × output price ÷ 1,000,000
The input holds everything the model reads for that answer: the instructions, the conversation so far, the tool definitions and every tool result. A site that searches its own pages before answering sends those pages as input. The two retrieval rows below carry 2 to 7 times the input of the admissions chatbot.
Worked examples at November 2026 prices
The table prices three measured token sizes on two models. A university admissions chatbot averaged 1,748 input and 60 output tokens per message. A plain retrieval pipeline averaged 3,865 input and 426 output tokens per query, and an iterative one 12,814 and 1,207.
| Answer | gpt-5-mini ($0.25 / $2) | Claude Sonnet 5.5 ($2 / $10) | 100,000 answers on Sonnet 5.5 |
|---|---|---|---|
| Admissions chatbot message | $0.00056 | $0.0041 | $410 |
| Plain retrieval answer | $0.0018 | $0.012 | $1,199 |
| Iterative retrieval answer | $0.0056 | $0.038 | $3,770 |
The plain retrieval answer on Sonnet 5.5 works out as 3,865 × $2 ÷ 1,000,000 = $0.00773, plus 426 × $10 ÷ 1,000,000 = $0.00426, for $0.01199.
Two Google price lists change on announced dates. gemini-3.8-flash bills $0.75 and $3.75 until 31 December 2026, then $1.50 and $7.50 from 1 January 2027. The same answer then costs twice as much. Gemini 4 Carbon and Argon move from $2 and $10 to $4 and $20 on 1 December 2026.
How AI Connect moves the bill
A site built with AI Connect sends its conversations through AI Wallet, which answers with the visitor's own plan, credits or API key. The provider bills that account, and the site's inference cost for those answers is zero. The site still runs its MCP server.
The site receives the token counts the provider reported for each answer. The wallet caps each site at 2,000,000 tokens a day by default and records each answered call in a usage ledger the visitor reads per day, per site and per account. The security model gives the limits, and the study of the cost of one AI answer shows the arithmetic for more models.
Related terms
- Bring your own AI: the model that moves inference to the user's account.
- AI wallet: the extension that pays from the visitor's plan.
- Paying for your own inference: a site's own key compared with visitors' AI.
Sources
- OpenAI, API pricing, read 12 November 2026
- Anthropic, Pricing, read 12 November 2026
- Google, Gemini API pricing, read 12 November 2026
- arXiv, An Empirical Study of Multi-Agent RAG for Real-World University Admissions Counseling, 15 July 2025
- arXiv, Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering, 4 September 2024
- TensorPass, Security model, 12 November 2026
Questions
- What is a token in AI pricing?
- A token is the unit of text a model reads and writes, and providers price both in US dollars per million tokens. Output tokens cost more than input tokens on every price list cited here, 5 times more for Claude Sonnet 5.5 at $2 and $10.
- How much does one chatbot answer cost a website?
- A retrieval answer of 3,865 input and 426 output tokens costs about $0.012 on Claude Sonnet 5.5 and $0.0018 on gpt-5-mini at November 2026 prices. At 100,000 answers that comes to $1,199 and $182.
- Do model prices change over time?
- Providers publish dated changes. Google bills gemini-3.8-flash $0.75 per million input tokens and $3.75 per million output tokens until 31 December 2026, then $1.50 and $7.50 from 1 January 2027.
- Who pays the inference cost on a site built with AI Connect?
- The visitor's own account pays it, through the plan or the credits AI Wallet answers with. The site pays for its MCP server and receives the token counts the provider reported.