The Four Things to Measure | Agentic AI and MCP for Product Managers

Agentic AI and MCP for Product Managers

What 'Agentic' Actually Means

The Autonomy Ladder

Why Most Things Called Agents Aren't

What This Course Covers

Tools Are the Interesting Part

What the Agent Remembers

Stopping Conditions

Copy This Prompt - Mapping Your Agent's Loop

Exercise - Trace Your Own Agent

The Problem MCP Solves

Servers and Clients

The Three Things a Server Offers

What MCP Gives You and What It Doesn't

Copy This Prompt: Working Out What You'd Connect To

Exercise: Map Your Connections

Why Tool Design Is Product Design

Names and Descriptions

What a Tool Returns When It Fails

Copy This Prompt - Designing Your Tool Inventory

Exercise - Write Your Tool Inventory

Three Categories, Not Five

Where the Approval Goes

Approval Fatigue

Designing the Approval Moment

Copy This Prompt - Designing Your Permission Model

Exercise - Build Your Permission Model

Why the Final Answer Isn't Enough

Trajectory Evaluation, Plainly

A Test Set of Tasks, Not Questions

The Four Things to Measure

Copy This Prompt - Building an Agent Test Set

Exercise - Build and Run an Agent Eval

Why Cost Per Task Has a Long Tail

Where the Seconds Go

Loops, Retries and Runaways

Budgets and What Happens at the Limit

Copy This Prompt - Modelling Agent Cost

Exercise - Model Your Costs and Set Your Caps

The Waiting Problem

Showing Work Without Showing Everything

Interruption, Correction and Undo

Building Trust One Delegation at a Time

Copy This Prompt - Designing the Agent Experience

Exercise - Design the Experience

The Blast Radius Question

Prompt Injection When the Agent Can Act

Compounding Errors

Rehearse the Incident

Exercise - Work Out Your Blast Radius

Five Situations Where a Workflow Wins

Agent, Workflow, or Just a Button

What Agents Structurally Struggle With

Reducing Autonomy Instead of Adding Capability

Copy This Prompt - Checking Whether You Need an Agent

Exercise - Check Your Own Choice

Lesson FP.1: What You're Producing

Prompt Library: Understanding the Task

Prompt Library: Designing Tools

Prompt Library: Permissions

Prompt Library: Evaluating

Prompt Library: Cost and Caps

Prompt Library: Experience and Risk

Prompt Library: Checking Yourself

Words Explained: Glossary

Lesson 4

The Four Things to Measure

Measure	The Question	Why It Matters
Task success	Did it achieve the goal?	The headline, and the least informative on its own
Path quality	Were the steps sensible and necessary?	Predicts cost, latency and future failures
Constraint adherence	Did it stay inside its permissions, every time?	This one is pass/fail. Any breach is a blocker.
Give-up quality	When it couldn't finish, did it stop well?	Determines what the user experiences on hard cases

Constraint adherence is the one to treat differently from the others. It isn't a percentage you're trying to improve, it's a gate. An agent that respects its permissions 97% of the time is not 97% safe; it's an agent that will breach three times in every hundred tasks, forever, at whatever volume you run it.