Online Evaluation & Experimentation

Offline evaluation tells you how the product performs on the cases you anticipated. Online evaluation tells you how it performs on everything else.

Online evaluation occurs in production environments through A/B testing, user feedback collection, and real-time monitoring.This reveals how your LLM performs with actual users and real-world queries, uncovering issues that controlled tests might miss.

A/B Testing for LLM Products : This testing in LLM products works similarly to traditional product A/B testing, with some important differences. You route a portion of traffic to a new version of the system, whether that is a different prompt, a different model, a different retrieval strategy, and measure whether quality metrics improve. The challenge is defining what you are measuring.

LLM quality is harder to reduce to a single number. The practical approach is to run experiments with a clear hypothesis: we believe changing this element will improve this specific quality dimension. Define your success metric before the experiment starts.

Implicit and Explicit User Feedback: Users are constantly sending signals about quality, and most products capture only a fraction of them.

Explicit feedback is the thumbs up and thumbs down, the rating, the direct report of a bad response. It is easy to collect and easy to interpret, but it captures only the fraction of interactions where users feel strongly enough to engage with a feedback mechanism.

Implicit feedback is far more abundant and often more honest. Did the user accept the model's suggestion or immediately rephrase their question? Did they copy the response or ignore it? Did they abandon the session mid-conversation? Each of these behaviors is a signal about whether the model's output was actually useful, even when the user never pressed a single button.

The teams that invest in capturing and interpreting implicit feedback are the ones who understand their product's real performance, not just its perceived performance.