Exercise - Build and Run an Agent Eval | Agentic AI and MCP for Product Managers

Agentic AI and MCP for Product Managers

What 'Agentic' Actually Means

The Autonomy Ladder

Why Most Things Called Agents Aren't

What This Course Covers

Tools Are the Interesting Part

What the Agent Remembers

Stopping Conditions

Copy This Prompt - Mapping Your Agent's Loop

Exercise - Trace Your Own Agent

The Problem MCP Solves

Servers and Clients

The Three Things a Server Offers

What MCP Gives You and What It Doesn't

Copy This Prompt: Working Out What You'd Connect To

Exercise: Map Your Connections

Why Tool Design Is Product Design

Names and Descriptions

What a Tool Returns When It Fails

Copy This Prompt - Designing Your Tool Inventory

Exercise - Write Your Tool Inventory

Three Categories, Not Five

Where the Approval Goes

Approval Fatigue

Designing the Approval Moment

Copy This Prompt - Designing Your Permission Model

Exercise - Build Your Permission Model

Why the Final Answer Isn't Enough

Trajectory Evaluation, Plainly

A Test Set of Tasks, Not Questions

The Four Things to Measure

Copy This Prompt - Building an Agent Test Set

Exercise - Build and Run an Agent Eval

Why Cost Per Task Has a Long Tail

Where the Seconds Go

Loops, Retries and Runaways

Budgets and What Happens at the Limit

Copy This Prompt - Modelling Agent Cost

Exercise - Model Your Costs and Set Your Caps

The Waiting Problem

Showing Work Without Showing Everything

Interruption, Correction and Undo

Building Trust One Delegation at a Time

Copy This Prompt - Designing the Agent Experience

Exercise - Design the Experience

The Blast Radius Question

Prompt Injection When the Agent Can Act

Compounding Errors

Rehearse the Incident

Exercise - Work Out Your Blast Radius

Five Situations Where a Workflow Wins

Agent, Workflow, or Just a Button

What Agents Structurally Struggle With

Reducing Autonomy Instead of Adding Capability

Copy This Prompt - Checking Whether You Need an Agent

Exercise - Check Your Own Choice

Lesson FP.1: What You're Producing

Prompt Library: Understanding the Task

Prompt Library: Designing Tools

Prompt Library: Permissions

Prompt Library: Evaluating

Prompt Library: Cost and Caps

Prompt Library: Experience and Risk

Prompt Library: Checking Yourself

Words Explained: Glossary

Lesson 6

Exercise - Build and Run an Agent Eval

Collect 15-20 real examples of the task from your actual product, real tickets, real requests, with the messiness intact.
Run the test set prompt. Fill in the 'good outcome' column yourself rather than accepting what it drafts.
Ask engineering for the full trace of each run, every tool call and every result. You can't evaluate an agent without traces.
Score 10 runs by hand on all four measures. Note every wasted step and every near-miss.
Count constraint breaches separately. Any number above zero is a shipping blocker, regardless of how good the success rate looks.