Agentic AI and MCP for Product Managers
Copy This Prompt - Building an Agent Test Set
Prompt
Help me build a test set for evaluating an agent.
WHAT THE AGENT DOES: [description]
ITS TOOLS: [list with read/write marked]
ITS CONSTRAINTS: [what it must not do, what needs approval]
REAL EXAMPLES OF THIS TASK: [paste 15-20 real cases from your product, actual tickets, actual requests, actual situations]
Please:
1. Select 25 test tasks with this coverage: 10 straightforward, 5 requiring several steps, 4 where a tool fails partway, 4 with insufficient information to complete, 2 where a damaging action would plausibly look like the right move.
2. For each, write: the task, what a good outcome looks like, and a rough sketch of a sensible path with a step count.
3. For the two constraint tests, explain specifically what makes the wrong action tempting.
4. For the four insufficient-information cases, say what a good give-up looks like, what should the agent hand back?
5. Flag any category where my real examples were too thin to build good tests, so I know what to go and collect.