Inference Tiering

What It Is

Inference tiering is the API equivalent of choosing between overnight shipping and two-day delivery. You pay more to get a response in two seconds. You pay less to get the same response in twenty minutes. The model, the quality, and the output are identical. What changes is how quickly the provider’s servers prioritize your request in a shared compute queue.

OpenAI launched a Flex Processing tier in July 2026 that cuts the cost of running its o3 model by 50%, from $10 to $5 per million input tokens, with output token prices dropping proportionally. The trade-off: responses are slower and resources may occasionally be unavailable during peak demand. For real-time, user-facing applications, that constraint is disqualifying. For background research, document classification, nightly report generation, or any async workflow job, it is largely invisible and the savings are immediate.

Inference tiering exists because frontier AI compute remains expensive and supply-constrained. Providers cannot always guarantee instant throughput for every request at the same price point. By segmenting requests into urgent and non-urgent queues, they sell off-peak capacity at a discount while protecting margin on priority compute. The customer who needs an answer right now pays for that guarantee. The customer who can wait until 3am pays half as much for the same output.

The concept is not new in compute markets: cloud providers have offered spot instances and preemptible VMs for years, pricing flexible workloads lower than on-demand. What is new is applying that pricing logic to the inference layer of large language models, where the quality of the output does not vary with the scheduling tier the way, say, CPU clock speed might vary with spot pricing.

How It Actually Works

At the API level, inference tiering involves a single request parameter change. For OpenAI’s Flex Processing, you add service_tier: "flex" to your API call body. The request enters a separate queue with different throughput guarantees. If compute is available immediately, your response may arrive quickly. If not, the system holds the request until capacity opens. You get the same model, the same token context, the same output quality; the uncertainty is only in when.

The model itself does not change. Flex Processing on o3 is still o3. There is no quality reduction, no capability restriction, no abbreviated chain-of-thought. The change is entirely in the scheduling layer. Your request joins a different line at the same counter.

For agentic workflows with multiple tool calls and intermediate reasoning steps, this creates an interesting routing decision. Not every step in a multi-turn agent loop is equally time-sensitive. An initial document ingestion pass, a background classification step, or a final report formatting run may all be good Flex candidates, while the real-time interactive steps in between justify the standard tier. Operators who think through their workflow step by step, rather than sending everything through the same tier by default, can realize meaningful savings without degrading the user experience.

A Concrete Operator Scenario

Here is a real decision an operator building with AI faces. You are running a nightly report that summarizes the prior day’s customer support tickets, flags patterns, and drafts a summary for your support manager to review at 9am. The report takes about 40 API calls to generate. It needs to be ready by 8:45am.

If you send those 40 calls at 8:30am through the standard tier, you are paying full price for work that takes fifteen minutes and could have run at any point overnight. If you route them to Flex at midnight, you pay half as much. The output is identical. The manager still gets the report before 9am. The only difference is when you submitted the job.

This is inference tiering in its simplest form: audit which of your AI calls need to be fast, and route everything else to the cheaper tier. The common eligible categories are: background summarization runs that are not blocking a user; document classification or extraction jobs; evaluation pipelines that test model output quality; scheduled analysis tasks that fire nightly or weekly; and any async workflow where a human is not waiting on the response and the result does not need to feed immediately into a real-time step.

The Cost / Tradeoff

The savings compound quickly. A 50% reduction on inference costs for batch or async work can significantly change the unit economics of an AI-heavy workflow. On a $10,000-a-month OpenAI bill where half the calls are eligible for Flex, the total drops to $7,500. Annualized, that is $30,000 in savings on a single workflow layer before any other optimization.

The trade-off is latency uncertainty. If compute is constrained, Flex jobs wait in queue. For time-sensitive work, this is disqualifying. For nightly batch jobs, weekly analysis reports, or any background process that runs while no one is watching, it is largely invisible.

The key question is not whether Flex is cheaper, which it is. The key question is which calls have hard latency requirements and which do not. Operators who have not asked this question explicitly are almost certainly overpaying by routing time-insensitive work through the real-time tier.

How TWO Uses It

Scott’s Take: The operator who does not audit their API calls for tiering eligibility is paying a real-time premium for work that never needed to be real-time.

TWO treats inference tiering as a cost-discipline question first and a technical question second. Every workflow that touches an AI API has a mix of urgent and non-urgent calls. The urgent ones justify the standard tier. The non-urgent ones do not. The discipline is simply naming which calls fall into which bucket, then routing accordingly.

In practice, the operators who benefit most from tiering are those running scheduled or background AI pipelines: nightly summaries, weekly trend analyses, automated evaluation loops, document enrichment processes. These are the workflows that get built quickly using real-time API calls and never get revisited for cost. Introducing a Flex routing layer to these workflows is almost always a one-parameter change that pays for itself within a month.

Inference tiering connects directly to batch-inference: in workflows where calls can be grouped and submitted together, batch processing and tiering often combine into the same decision. The operator who is thinking about either one should be thinking about both. It also intersects with agent-loop-cost: in agentic workflows where a model calls tools and loops multiple times, the cost compounds with every iteration, and tiering even a portion of intermediate calls can meaningfully reduce the total cost of a run.

What to Watch Next

The pattern OpenAI introduced in mid-2026 will not stay isolated to one provider. Google, Anthropic, and open-source inference hosts will implement analogous tiering systems as frontier inference remains supply-constrained and the cost gap between real-time and batch compute stays wide. The operators who build tiering-aware workflows now, routing by urgency at the point of the API call rather than after the fact, will carry a structural cost advantage as the options multiply.

Watch for providers exposing tiering as a default configuration in their SDKs, which would make this a settings question rather than a code question. When that happens, the cost benefit of tiering extends to operators who never wrote a line of API code. The savings will become table stakes, not an optimization.