Your test set for an agent is a set of realistic tasks with known good outcomes and a rough sense of what a sensible path looks like. Twenty to forty is usually enough to start, because each one takes much longer to review than a single-turn answer.

Category	Share	What It Tests
Straightforward	40%	The common case, and the efficient path
Needs several steps	20%	Whether it keeps the goal in mind over a long run
A tool fails mid-task	15%	Recovery, and whether it notices at all
Not enough information to finish	15%	Whether it gives up cleanly or invents
Tempting wrong action available	10%	Whether it stays inside its permissions

The Last Category Is the One to Build Carefully

Construct a few tasks where an unauthorised or damaging action would plausibly look like the helpful thing to do, a case where sending the customer an update immediately seems obviously right, but policy says it needs approval. This is the only way to find out whether your constraints hold under pressure rather than only in easy cases.

Agentic AI and MCP for Product Managers

Module 1What 'Agentic' Actually Means

Module 2How an Agent Actually Works

Module 3MCP: What It Is and Why You Should Care

Module 4Designing the Tool Layer

Module 5Permissions and the Human in the Loop

Module 6Evaluating Agents

Module 7Cost, Latency and Unpredictability

Module 8Designing What the User Sees

Module 9Risk and Blast Radius

Module 10When an Agent Is the Wrong Answer

Module 11Final Project: Design One Real Agent End to End

Module 12Appendix: The Prompt Library

A Test Set of Tasks, Not Questions