The boring parts of shipping an AI feature

Getting a language model to do something impressive takes an afternoon. Getting it to do the same thing on a Tuesday for a user you have never met takes considerably longer, and almost none of that time is spent on prompts.

Here is where it actually goes.

The model is the easy part

A demo has one person, one input, and a developer watching. Production has a thousand people, inputs nobody predicted, and no one watching at 3am.

The first thing that breaks is not quality. It is shape. Your code expects JSON with four keys. Most of the time it gets that. Then someone pastes in a document with a stray brace, the model returns prose apologising for the confusion, your parser throws, and a user sees a spinner forever.

So the first real work is refusing to trust the output. Validate every response against a schema, retry once with a corrective message, and define what happens when the retry also fails. Not a crash. A specific, honest failure state that a human can read and act on.

Cost is a product decision

In a demo, cost is invisible. In production it is a line item that grows with usage, which means the feature gets more expensive exactly when it starts working.

We size this before we build. How many calls per user action, what the token count looks like at the ninetieth percentile, what that comes to per month at ten times current volume. Sometimes the answer is that the feature does not make sense at that price, so we cache aggressively, or use a smaller model for the first pass, or do it without a model at all.

That last option comes up more often than people expect. A good share of what gets scoped as an AI feature is a search problem or a rules problem wearing a costume.

Latency changes the interface

A response that takes eight seconds is not a slower version of a response that takes one second. It is a different product. It needs streaming, or a progress state, or it needs to move off the critical path entirely and become a background job that notifies the user when it finishes.

Deciding this after the feature works is how you end up rewriting the front end. We decide it during the plan phase, before anyone writes a component.

Guardrails that hold

Every AI feature touching user input needs an answer for what happens when someone tries to push it somewhere it should not go. In a business tool this is less about headlines and more about liability. A claims assistant that confidently states a policy covers something it does not is a real problem with a real cost.

The guardrails that work are boring. Scope the model to a narrow job. Give it retrieved context instead of asking it to remember. Check the output against known data before it reaches the screen. Log everything, so that when a user says it told them something wrong, you can find out what it actually said instead of guessing.

The question we ask first

When a client asks for AI in their product, our first question is what happens when it is wrong.

If the answer is "a person reviews it anyway", it is a good fit and we can move quickly. If the answer is "it goes straight to a customer" or "it triggers a payment", the scope gets narrower and the review layer gets built before the clever part.

The features that survive are the ones where being wrong occasionally is tolerable and being right usually is valuable. That is a smaller set than the pitch decks suggest. It is not an empty one.