4 posts tagged with "Evaluation"

"AI hallucinates" is a true sentence that has been repeated until it stopped meaning anything. It sounds like a quirk. The chatbot got a trivia question wrong, everyone screenshots it, nothing happens.

Now move the same behavior inside a company.

A finance agent invents a figure and it lands in a board deck. A support agent invents a refund policy and tells a customer, which in several jurisdictions means you now have that policy. A coding agent invents an API, writes code around it, and you find out in production at 2 a.m.

Same underlying behavior in all four cases. Wildly different consequences. And the difference has nothing to do with the model.

An agent can give a perfectly correct answer built on completely wrong company knowledge. The reasoning holds up. The citation is real. The document exists. And the number in it has been wrong since March.

This is the failure enterprises underestimate, because it does not look like a failure. No error. No refusal. No hedge. Retrieval worked, generation worked, and the system did exactly what you built it to do, on top of a corpus nobody owns.

We argued that you can't ship what you can't measure. Here's the uncomfortable half of that sentence: your measurement is a model too, and it is also wrong sometimes.

Every team that has tried to ship an AI feature has hit the same wall. The demo works. A stakeholder asks, "is it good?" and the honest answer is a shrug. You feel like it's good. You changed a prompt and it feels a little better, or a little worse, and you cannot say which, because "feels" is the only instrument you have.