Executive Summary
AI Gateway has launched Service Tiers, a new feature enabling developers to optimize API requests for specific use cases. Users can now choose between 'priority,' 'default,' and 'flex' tiers to balance latency, throughput, and cost. Initially supporting OpenAI and Gemini models, this feature provides granular control for interactive applications and cost-effective background jobs.
Key Takeaways
* Three Service Tiers: Offers `priority` for high-throughput, low-latency workloads; `default` for standard performance; and `flex` for lower-cost, latency-tolerant background jobs.
* Dynamic Pricing: The `priority` tier costs approximately 1.8-2x the default rate, while the `flex` tier costs about half the default rate. Billing is automatically applied based on the tier that successfully serves the request.
* Initial Model Support: The feature is available at launch for OpenAI and Gemini models and works across all AI Gateway API formats.
* Smart Fallback: If a requested tier (e.g., `priority`) is unavailable, the request automatically falls back to the `default` tier and is billed at the standard rate, ensuring uptime.
* Simple Implementation: Tiers can be specified in the API call via the `serviceTier` option under `providerOptions`.
Strategic Importance
This feature makes AI Gateway a more flexible and economically viable platform, allowing customers to precisely manage costs and performance for different AI workloads.