What tau-bench measures, and where it misleads

An agent that solves a task once is a demo. One that solves it every time is a product. tau-bench was built to measure that gap.

The benchmark puts an agent in a realistic customer-service setting. It talks to a simulated user, calls tools against a database, and has to follow the domain's rules, like a refund policy. The paper's finding is that function-calling agents are inconsistent. In 2024, even gpt-4o succeeded on under 50% of tasks, and on the retail tasks its pass^8 was under 25%. It could pass a task on one run and fail the same task on the next.

That's why it reports a reliability number, pass^k, as well as a plain success rate. Pass^k asks whether the agent succeeds on all k independent attempts at the same task. It falls quickly as k grows, because every extra attempt is another chance to slip.

What it tells you: whether an agent can hold to a policy through a conversation that wanders, with a user who changes their mind.

What it doesn't: how the agent behaves in your domain, with your tools and your rules. The retail and airline scenarios are a proxy.

Borrow the idea. Run each of your test cases five times, and report how many pass all five.

How many of your agent's wins would survive that?