Executive Summary
Vercel has successfully deployed an internal AI-powered workflow to automate the code review process for one of its largest applications. The system uses an LLM-based classifier to distinguish between low-risk changes (e.g., UI, docs) and high-risk changes (e.g., authentication, payments). This allows an AI agent to safely auto-merge 58% of all pull requests without human intervention, significantly reducing development bottlenecks. As a result, the average time-to-merge has dropped by 62%, while the speed and focus of human reviews on critical code have improved.
Key Takeaways
* Primary Function: An LLM (Gemini) classifies pull requests (PRs) as either "HIGH" or "LOW" risk based on an analysis of the code diff, title, and description.
* Automated Workflow: Low-risk PRs are automatically approved and merged by the agent, while high-risk PRs are flagged and require mandatory human review.
* Performance Metrics:
* 58% of all PRs are now handled automatically by the agent.
* Average merge time decreased by 62%, from 29 hours to 10.9 hours.
* Individual developer throughput increased by 46%.
* Safety & Reliability:
* Zero auto-merged PRs were reverted during the experiment.
* The system is tuned to favor "false HIGHs" over "false LOWs" to err on the side of caution.
* Deployment rollbacks decreased, and the only incident-causing rollback was from a human-approved PR that the AI had correctly flagged as high-risk.
* Improved Human Review: With fewer PRs to review, human engineers focused on more critical changes, with the time-to-first-review on high-risk PRs improving by 2.7x.
Strategic Importance
This initiative serves as a powerful case study for applying AI agents to solve real-world bottlenecks in the software development lifecycle, increasing engineering velocity without compromising safety. For Vercel, it validates its own AI infrastructure and demonstrates a tangible method for other engineering organizations to scale development practices.