2 posts tagged with "Evaluation"

We argued that you can't ship what you can't measure. Here's the uncomfortable half of that sentence: your measurement is a model too, and it is also wrong sometimes.

Every team that has tried to ship an AI feature has hit the same wall. The demo works. A stakeholder asks, "is it good?" and the honest answer is a shrug. You feel like it's good. You changed a prompt and it feels a little better, or a little worse, and you cannot say which, because "feels" is the only instrument you have.