Exercise - Build and Run an Agent Eval
Back to Course
Agentic AI and MCP for Product Managers
Module 1
What 'Agentic' Actually Means
What 'Agentic' Actually Means
The Autonomy Ladder
Why Most Things Called Agents Aren't
What This Course Covers
Module 2
How an Agent Actually Works
The Loop
Tools Are the Interesting Part
What the Agent Remembers
Stopping Conditions
Copy This Prompt - Mapping Your Agent's Loop
Exercise - Trace Your Own Agent
Module 3
MCP: What It Is and Why You Should Care
The Problem MCP Solves
Servers and Clients
The Three Things a Server Offers
What MCP Gives You and What It Doesn't
Copy This Prompt: Working Out What You'd Connect To
Exercise: Map Your Connections
Module 4
Designing the Tool Layer
Why Tool Design Is Product Design
Names and Descriptions
Granularity
What a Tool Returns When It Fails
Copy This Prompt - Designing Your Tool Inventory
Exercise - Write Your Tool Inventory
Module 5
Permissions and the Human in the Loop
Three Categories, Not Five
Where the Approval Goes
Approval Fatigue
Designing the Approval Moment
Copy This Prompt - Designing Your Permission Model
Exercise - Build Your Permission Model
Module 6
Evaluating Agents
Why the Final Answer Isn't Enough
Trajectory Evaluation, Plainly
A Test Set of Tasks, Not Questions
The Four Things to Measure
Copy This Prompt - Building an Agent Test Set
Exercise - Build and Run an Agent Eval
Module 7
Cost, Latency and Unpredictability
Why Cost Per Task Has a Long Tail
Where the Seconds Go
Loops, Retries and Runaways
Budgets and What Happens at the Limit
Copy This Prompt - Modelling Agent Cost
Exercise - Model Your Costs and Set Your Caps
Module 8
Designing What the User Sees
The Waiting Problem
Showing Work Without Showing Everything
Interruption, Correction and Undo
Building Trust One Delegation at a Time
Copy This Prompt - Designing the Agent Experience
Exercise - Design the Experience
Module 9
Risk and Blast Radius
The Blast Radius Question
Prompt Injection When the Agent Can Act
Compounding Errors
Rehearse the Incident
Exercise - Work Out Your Blast Radius
Module 10
When an Agent Is the Wrong Answer
Five Situations Where a Workflow Wins
Agent, Workflow, or Just a Button
What Agents Structurally Struggle With
Reducing Autonomy Instead of Adding Capability
Copy This Prompt - Checking Whether You Need an Agent
Exercise - Check Your Own Choice
Module 11
Final Project: Design One Real Agent End to End
Lesson FP.1: What You're Producing
Module 12
Appendix: The Prompt Library
Prompt Library: Understanding the Task
Prompt Library: Designing Tools
Prompt Library: Permissions
Prompt Library: Evaluating
Prompt Library: Cost and Caps
Prompt Library: Experience and Risk
Prompt Library: Checking Yourself
Words Explained: Glossary
Lesson 6
Exercise - Build and Run an Agent Eval
Mark as Complete