AutomationBench 1.0.6 (with fallbacks)

Tool use

AutomationBench (from Zapier) measures how reliably a model can build and run real-world automations end to end — wiring up triggers, transforming data and chaining actions across third-party apps with minimal human help. Scores are low across the board, leaving plenty of headroom. This row reports release 1.0.6 of Zapier’s private evaluation set with the vendor’s production safeguards and fallback models enabled, which is how these models behave in real deployments; OpenAI’s models have no fallback mechanism, so for them the setting makes no difference.