10 posts tagged with "AI in Production"

A chatbot that says something wrong is embarrassing. An agent that sends something wrong is a fact about the world. The moment an agent can send, post, create, or delete, it stops being a text generator and becomes a principal in your system — with credentials and reach, taking actions other people receive.

The instinct at that point is to write a better system prompt. Tell the model to be careful. Tell it to confirm before sending. This doesn't work, and it doesn't fail in an exotic way — it fails because a system prompt is a suggestion addressed to the thing you're trying to constrain.

Most advice about token efficiency is prompt golf. Trim the system prompt. Cut the pleasantries. Replace the paragraph with a bullet. Shave two hundred tokens off the instructions and feel briefly virtuous about it.

It's the wrong game. Two hundred tokens is a rounding error next to a single tool result you didn't need to look at, dragged forward through every turn of a thirty-turn run. The wins that matter are structural: they come from deciding what never enters the context window in the first place. Not what gets said more tersely — what never gets said at all.

This is a practical guide to making that decision well. Four moves, each one a place where a real agent either touches something or doesn't, and then a section on what to do about it when you sit down to build your own.

Ask an engineer how to stop an agent that won't stop, and you'll get the same answer every time: add a limit. It's the right instinct and the wrong plan. There is no single limit that catches every shape of runaway, because runaways don't have one shape — and each shape slips past a different guard for a different structural reason.

"Run this agent every morning at 9" sounds like a one-line feature. There's a cron expression, there's a job table, there's a thing that reads the table. Ship it Friday.

Cron is the easy 5%. The other 95% is not firing twice, not losing a fire, not holding a database transaction open across a task that runs for minutes, and deciding whose 9am you actually mean. A cron expression is a trigger. A scheduler is everything you build around it.

"What can your agent do?" is two questions wearing one coat. The first is about the action space: once the agent is awake and thinking, what can it reach, call, read, and change? The second is about the trigger space: what makes it wake up in the first place? Teams conflate them constantly, and the conflation is expensive, because the two want opposite answers.

From the outside, creating an agent looks like filling in a form. You describe what you want, you look at a preview, you click a button. Thirty seconds, maybe less. The interface is doing everything it can to feel like nothing much is happening.

Every team that builds seriously with AI walks the same road, usually without noticing. It starts with a chat window and a good feeling, and — if the work survives contact with production — it ends somewhere far more disciplined. This is a field note on that road: the three stages we see teams pass through, what each one is genuinely good at, and the specific failure that forces the jump to the next.

"Give the agent memory" is one of those phrases that sounds like a feature and is actually an architecture. Everyone agrees agents should remember things across sessions. Almost no one agrees on what that means — and the default implementation, a vector database that everything gets dumped into, produces agents that are confidently wrong about their own past.

The most expensive misconception in applied AI is that performance lives in the model. Pick the best model, the thinking goes, and the rest is glue. So teams spend their attention on which model to call and treat everything around it — the tools, the loop, the context they assemble, the way they check the output — as plumbing.

When you build software with an AI agent day to day, the thing you run out of first isn't intelligence — it's context. The window is finite, attention inside it isn't free, and a session that starts sharp gets duller as it fills with the exhaust of the work: verbose command output, whole files read to find one function, stale documentation, the residue of three tasks ago.