5 posts tagged with "Evaluation"

On a Friday afternoon, someone on the ops team builds a dashboard in an hour. They describe it to a coding agent, click around, ask for a couple of changes, and share the link. Everyone loves it. On Monday a billing service renames one field, the dashboard quietly shows zero revenue for three regions, and nobody notices until Wednesday, because the only test the dashboard ever had was a person looking at it on Friday.

Nobody did anything wrong. The dashboard was built the right way for a prototype, and then it stopped being a prototype without anyone deciding that it had.

"AI hallucinates" is a true sentence that has been repeated into harmlessness. It sounds like a quirk — the chatbot got a trivia question wrong, everyone screenshots it, nothing happens.

Now move the same behavior inside a company. A finance agent invents a figure and it lands in a board deck. A support agent invents a refund policy and tells a customer, which in several jurisdictions means you now have that policy. A coding agent invents an API, writes code around it, and the failure surfaces in production at 2 a.m. Same underlying behavior in all four cases. Wildly different consequences, and the difference has nothing to do with the model.

An agent can give a completely correct answer based on completely wrong company knowledge. The reasoning is sound, the citation is real, the document exists — and the number in it has been wrong since March.

This is the failure enterprises underestimate, because it doesn't look like a failure. There's no error, no refusal, no hedge. Retrieval worked. Generation worked. The system did exactly what it was built to do, on top of a corpus nobody was accountable for.

We argued that you can't ship what you can't measure. Here's the uncomfortable half of that sentence: your measurement is a model too, and it is also wrong sometimes.

Every team that has tried to ship an AI feature has hit the same wall. The demo works. A stakeholder asks, "is it good?" and the honest answer is a shrug. You feel like it's good. You changed a prompt and it feels a little better, or a little worse, and you cannot say which, because "feels" is the only instrument you have.