Executive Summary
AI Gateway has introduced a new beta feature enabling developers to set specific timeouts for individual AI model providers. This allows for faster, more resilient applications by automatically aborting a request to a non-responsive provider and failing over to the next one in a predefined sequence. This functionality is currently available only for users with "Bring Your Own Key" (BYOK) credentials.
Key Takeaways
* Per-Provider Timeouts: Developers can now configure custom timeouts in milliseconds for each AI provider (e.g., OpenAI, Anthropic, Bedrock).
* Automatic Failover: If a provider exceeds its configured timeout, AI Gateway aborts the request and automatically retries with the next provider in the specified `order`.
* Beta Availability: The feature is currently available in beta.
* BYOK Credential Only: This functionality is exclusively for users utilizing their own API keys (BYOK).
* Cost Caveat: Users may still be charged by providers for timed-out requests, as some do not support stream cancellation.
Strategic Importance
This update enhances the reliability of applications using AI Gateway by giving developers granular control to mitigate latency and outages from specific upstream providers.