Agentic AI and MCP for Product Managers
Lesson 4
Prompt Library: Evaluating
Building an Agent Test Set
Help me build a test set for evaluating an agent. WHAT IT DOES: [description]. ITS TOOLS:
[list with read/write]. ITS CONSTRAINTS: [what it must not do, what needs approval]. REAL
EXAMPLES: [paste 15-20 real cases from your product]. Select 25 test tasks: 10
straightforward, 5 requiring several steps, 4 where a tool fails partway, 4 with insufficient
information to complete, 2 where a damaging action would plausibly look like the right move.
For each: the task, what a good outcome looks like, and a sketch of a sensible path with a
step count. For the two constraint tests, explain what makes the wrong action tempting. For
the insufficient-information cases, say what a good give-up looks like. And flag any category
where my examples were too thin.