Executive Summary
Braintrust published findings from an experiment testing the "bash is all you need" hypothesis for AI agents querying semi-structured data. They compared agents using SQL, bash, and basic filesystem tools, finding that a pure SQL agent was initially far superior in accuracy, cost, and speed. After debugging both the tools and the evaluation itself, they discovered a hybrid agent—with access to both SQL and bash—was the most reliable, using bash to verify the results of its SQL queries.
Key Takeaways
* Initial Test: In the first round, a SQL-based agent achieved 100% accuracy, significantly outperforming a bash agent (52.7%) and a basic filesystem agent (63.0%). The SQL agent was also 6.5x cheaper and 9x faster than the bash agent.
* Debugging is Crucial: The initial poor performance of the bash agent was traced to performance bottlenecks in the tool and errors within the evaluation dataset itself, which were subsequently corrected.
* Hybrid Approach Wins: The most effective strategy was a hybrid agent given access to both a SQLite database and bash commands. This agent consistently achieved 100% accuracy.
* Self-Verification Emerges: The hybrid agent developed a strategy of running a SQL query and then using bash commands (`grep`, etc.) to cross-reference the raw data files, verifying its own answers and catching errors a pure SQL approach might miss.
* Reliability vs. Cost: While the hybrid approach proved most reliable, it came at a higher cost, using roughly double the tokens of the pure SQL agent due to the extra reasoning and verification steps.
* Open Source Eval: The evaluation harness and experiment results are open source, allowing others to run their own benchmarks.
Strategic Importance
These findings challenge the single-tool "bash is all you need" theory, demonstrating that reliable AI agents benefit from flexible, multi-tool architectures and self-verification. This emphasizes the critical need for robust evaluation frameworks to move beyond simple accuracy metrics and uncover optimal agent design patterns.