Data for AI Products
Case Study: Choosing an Architecture Based on What Data Was Actually Available
A composite case, built from patterns that repeat across B2B SaaS and fintech teams.
The setup. A mid-size B2B payments company. Around 40,000 merchants. The support team handles roughly 9,000 tickets a month, and 62% of them are variations of four questions: why was a payout delayed, why was a transaction declined, how do I change my settlement account, and what does this fee mean. The CEO has asked for an "AI support agent" and has, in the way of CEOs, already mentioned it to the board.
The engineering lead's first instinct: fine-tune a model on three years of support transcripts. It sounds right. It is wrong, and the reason it is wrong lives entirely in the data audit.
The data audit, which took four days and saved four months
| Asset | Volume | What the audit found | Usable? |
|---|---|---|---|
| Support transcripts | ~310,000 | Only 34% marked resolved. Of those, a sample of 200 showed agents frequently pasted a macro then handled the real issue over a phone call that was never logged. | Partly. Good for tone, unreliable for answers. |
| Help centre articles | 412 | Written over five years. 19% referenced a pricing structure retired in 2024. No owner. No review date. | Not yet. Content problem. |
| Internal ops runbooks | ~60 | Accurate, actively maintained by the ops team, because their own jobs depend on them. | Yes. The best asset in the building. |
| Transaction and payout logs | Millions | Contain the actual reason codes for every decline and delay, per merchant. | Yes, but they are live data, not training data. They must be queried at request time. |
| Escalation notes from senior agents | ~2,800 | Genuinely excellent explanations of hard cases, written by the five best people in the company. | Yes. This is the fine-tuning gold. |
What the audit changed. Notice what happened. The asset the team assumed would be the training set, 310,000 transcripts, turned out to be mostly noise with a resolution label that did not mean resolution. Meanwhile two assets nobody had mentioned in the kick-off, the ops runbooks and the escalation notes, turned out to be the real foundation. And the single most important data source, the transaction logs, is not training data at all. It is a live lookup, which means the architecture needs tool access, not a bigger model.
The resulting architecture.
6. Retrieval over the ops runbooks plus a rewritten subset of the help centre, for policy and process answers. The team cut 412 articles down to 140, assigned an owner and a review date to each, and treated that curation as the actual project. It took six weeks and it was not glamorous.
7. Live tool call into the transaction API for anything merchant-specific. The model is never allowed to guess a decline reason. It either has the reason code or it says it will fetch it. This is a product rule, enforced in the system design, not a hope expressed in a prompt.
8. A light fine-tune on 900 examples, built from the 2,800 escalation notes, curated down by the PM and two senior agents. Purpose: tone, structure, and knowing when to hand off to a human. Not facts. Never facts.
9. A golden evaluation set of 250 cases, written by the PM, covering the four common questions plus twenty deliberately nasty edge cases, including one where the correct answer is "I cannot tell you that, here is a human".
Fig 3.3 - Three techniques, each doing the one job its data supports. The fine-tune is the smallest part of the system, and that is correct.
The three lessons to carry out of this.
10. The data audit is the project's first real milestone. Four days of reading actual rows changed the architecture, the budget, and the timeline. No amount of model expertise would have surfaced the fact that "resolved" did not mean resolved.
11. The biggest dataset was the least useful one. 310,000 transcripts lost to 2,800 escalation notes, because the escalation notes were written by people who were being careful.
12. The hardest work was content operations, not machine learning. Rewriting 140 help articles and assigning them owners was the unglamorous six weeks that made everything else possible. Somebody has to be willing to own that. In practice, that somebody is the PM.
YOUR MOVE THIS WEEK
Run the audit table above on your own feature. Five rows, four columns. Do not let anyone propose an architecture until it is filled in. If your team resists on the grounds that it will slow things down, remind them that the alternative is discovering this in month three.