Executive Summary
AI Gateway has launched a new `sort` feature, giving developers explicit control over how they route requests to different AI model providers. This update allows users to programmatically rank providers based on three specific criteria: cost, time to first token (TTFT), or tokens per second (TPS). The feature enables optimization for specific use cases, such as prioritizing the cheapest provider for batch jobs or the fastest for interactive applications, with rankings calculated dynamically at request time.
Key Takeaways
* New Sorting Capability: Users can now add a `sort` parameter to their API calls to rank the available providers for a given model.
* Three Optimization Criteria:
* `cost`: Sorts by the lowest input price per million tokens.
* `ttft`: Sorts by the lowest median time to first token, for latency-sensitive tasks.
* `tps`: Sorts by the highest median throughput, for long-output generation.
* Dynamic, Real-time Ranking: The sorting logic is computed at request time, automatically accounting for provider price changes or shifts in performance without requiring code updates.
* Composable with Other Controls: The `sort` feature can be combined with other routing options like Zero Data Retention (ZDR), allowing for multi-factor routing decisions (e.g., finding the lowest latency provider that is also ZDR-compliant).
* Transparent Routing: API responses include detailed metadata showing the sort criteria used, the ranked order of providers attempted, their performance metrics, and any providers that were deprioritized due to poor health.
Strategic Importance
This feature strengthens AI Gateway's position as an intelligent routing layer by offering granular, performance-aware controls that help customers optimize both cost and user experience in production.