Trajectory Evaluation, Plainly | Agentic AI and MCP for Product Managers

Agentic AI and MCP for Product Managers

What 'Agentic' Actually Means

The Autonomy Ladder

Why Most Things Called Agents Aren't

What This Course Covers

Tools Are the Interesting Part

What the Agent Remembers

Stopping Conditions

Copy This Prompt - Mapping Your Agent's Loop

Exercise - Trace Your Own Agent

The Problem MCP Solves

Servers and Clients

The Three Things a Server Offers

What MCP Gives You and What It Doesn't

Copy This Prompt: Working Out What You'd Connect To

Exercise: Map Your Connections

Why Tool Design Is Product Design

Names and Descriptions

What a Tool Returns When It Fails

Copy This Prompt - Designing Your Tool Inventory

Exercise - Write Your Tool Inventory

Three Categories, Not Five

Where the Approval Goes

Approval Fatigue

Designing the Approval Moment

Copy This Prompt - Designing Your Permission Model

Exercise - Build Your Permission Model

Why the Final Answer Isn't Enough

Trajectory Evaluation, Plainly

A Test Set of Tasks, Not Questions

The Four Things to Measure

Copy This Prompt - Building an Agent Test Set

Exercise - Build and Run an Agent Eval

Why Cost Per Task Has a Long Tail

Where the Seconds Go

Loops, Retries and Runaways

Budgets and What Happens at the Limit

Copy This Prompt - Modelling Agent Cost

Exercise - Model Your Costs and Set Your Caps

The Waiting Problem

Showing Work Without Showing Everything

Interruption, Correction and Undo

Building Trust One Delegation at a Time

Copy This Prompt - Designing the Agent Experience

Exercise - Design the Experience

The Blast Radius Question

Prompt Injection When the Agent Can Act

Compounding Errors

Rehearse the Incident

Exercise - Work Out Your Blast Radius

Five Situations Where a Workflow Wins

Agent, Workflow, or Just a Button

What Agents Structurally Struggle With

Reducing Autonomy Instead of Adding Capability

Copy This Prompt - Checking Whether You Need an Agent

Exercise - Check Your Own Choice

Lesson FP.1: What You're Producing

Prompt Library: Understanding the Task

Prompt Library: Designing Tools

Prompt Library: Permissions

Prompt Library: Evaluating

Prompt Library: Cost and Caps

Prompt Library: Experience and Risk

Prompt Library: Checking Yourself

Words Explained: Glossary

Lesson 2

Trajectory Evaluation, Plainly

Trajectory evaluation means scoring the sequence of steps, not just the destination. It sounds technical and it's mostly just reading traces and asking sensible questions.

A Trace, Scored

TASK: "Why is shipment 44182 late? Draft an update."

1. get_shipment_status(44182) -> held at customs OK

2. get_customs_records(44182) -> missing form C88 OK

3. search_docs("form C88") -> policy article OK

4. get_shipment_status(44182) -> (same as step 1) WASTE

5. get_customer_contact(44182) -> Karin, Bexley OK

6. draft_update(...) -> drafted OK

SCORING

Did it reach the right answer? Yes

Were all steps necessary? No - step 4

Did it use the right tools? Yes

Did it take any unauthorised action? No

Steps taken vs a good path (5)? 6

Verdict: pass, with one wasted call. Not a problem

once. A problem if it happens in 40% of runs.