← Back to home
AI Engineering

AI Engineering Begins Where the Demo Ends

A compelling prototype is a starting point, not evidence. Reliable AI products need evaluation, security, observability, cost controls, and clear failure behaviour.

I have yet to see an AI demo fail in a steering committee.

The pattern is familiar. Someone builds an internal knowledge assistant over a weekend. It is connected to a few hundred policy documents and a capable model. In the meeting, it answers ten questions fluently, cites its sources, and even handles a follow-up with a little humour.

The room is impressed. Funding is approved. A pilot is launched with a few hundred users.

Three weeks later, the first serious complaint arrives. An employee asked about the leave policy and received a confident, well-written, and completely outdated answer. The assistant had retrieved a document from two years ago that nobody had removed from the index.

Nothing about the demo was dishonest. But it was curated. The presenter knew which questions worked, which documents were in the index, and which phrasing the model handled well. Real users know none of that.

A demo shows that something is possible. It says very little about whether it is reliable.

To me, that gap between "it worked in the demo" and "it works on Monday morning for four thousand people" is where AI engineering actually begins.

Two teams, the same problem

Consider two teams building the same capability: an assistant that helps service-desk agents resolve tickets faster.

The first team iterates on prompts. Every time a user reports a bad answer, they adjust the instructions, try a few examples, and ship the change when it looks better. Reviews are lively but subjective. One person thinks the latest version is better; another has found a case where it is worse. When the model provider releases a new version, the behaviour shifts in ways nobody can quite explain.

The second team does something less exciting first. They pull three hundred real tickets from the last six months, covering the common cases, the awkward ones, and a few that should be refused. With a couple of senior agents, they agree on what a good answer looks like for each. Then they automate the comparison, so every change to the prompt, the model, the retrieval logic, or the documents is scored against that set before it reaches users.

For the first month, the second team looks slower.

By the third month, they are the only ones who can answer a simple question with confidence: is the system getting better or worse?

That is why, before anything else, the first thing I ask any AI team is to show me their evaluation set. Not five hand-picked examples. A representative set, with a definition of "good" that someone other than the builder agreed to.

Teams that have one ship with confidence. Teams that don't end up debating impressions in every review meeting.

Treat the model as a dependency, not a colleague

Conventional software dependencies behave consistently. A given input to a library produces the same output today and tomorrow.

Language models do not work that way. Their behaviour is probabilistic. It changes across versions, sometimes subtly. They can fail confidently, in ways that look exactly like success.

I believe that changes how we should design around them.

Every model call deserves an explicit boundary. What are we asking the model to do? How do we validate what comes back? What does the product do when the answer is uncertain, malformed, or unavailable because the provider is having a bad day?

A simple example: an assistant that drafts customer emails should never send them directly. It drafts; a person or a rules engine approves. The difference in design effort is small. The difference in risk is enormous.

Keep deterministic things deterministic

One of the most important design decisions in an AI system is deciding what the model should not do.

I've seen a proposal for an agent that would handle customer refund requests end to end: read the complaint, check the order, decide on the refund, and issue it. It worked well in testing. It also meant that the refund policy, including the financial limits, effectively lived inside a prompt.

"The model usually gets it right" is not a control.

The better design used the model for what it does well: understanding the complaint, summarizing the history, and drafting a recommended action. The decision itself, including thresholds, eligibility rules, and approval limits, stayed in ordinary code that could be tested, audited, and explained.

My rule of thumb is straightforward. Use language models where interpretation, synthesis, and flexible interaction create value. Keep authorization, financial calculations, safety limits, and compliance rules in deterministic systems where predictable behaviour is essential.

Most "agents" should start as workflows

Agentic systems are genuinely powerful, and I don't want to dismiss them. But I've noticed that many proposed agents are, on closer inspection, workflows with a model in the middle.

If you can draw the steps on a whiteboard (retrieve the customer record, classify the request, draft a response, route for approval), then write those steps as code, and use the model only for the parts that genuinely need judgment.

That approach is easier to test, easier to bound in cost, and much easier to explain when something goes wrong. Autonomy can be introduced gradually, in the specific places where evaluation shows the model makes better decisions than the fixed path.

Earn the autonomy. Don't assume it.

The operating model is the product

Production AI needs the same discipline as any other critical system, and a few additional ones.

Cost. A pilot with two hundred users can look inexpensive. The same design at twenty thousand users, with long conversations and large retrieved contexts, can cost an order of magnitude more than anyone budgeted. I've seen projects stall not because the technology failed, but because nobody modelled the cost per successful task before scaling.

Security. Retrieval-based systems read documents, and documents can contain instructions. A malicious or simply careless piece of text in an indexed file can influence the model's behaviour. Access control must apply to what the assistant can retrieve, not just to who can open the chat window.

Observability. Latency and token usage are easy to measure. The more meaningful signals are task success rate, how often a human has to correct the output, groundedness where it matters, and how the system behaves when it doesn't know. A product that says "I'm not sure, here's who can help" will earn more trust than one that is confidently wrong three percent of the time.

Ownership. Someone needs to own the evaluation set, the index, the prompts, and the incident process. When an AI system gives a bad answer, "the model did it" is not an acceptable root cause.

My perspective: evaluation is the specification

I have come to believe that for AI systems, the evaluation set is the specification.

In traditional software, we write requirements and test against them. In AI systems, the requirements are often too nuanced to write down precisely, but they can be demonstrated through examples: this question should get this kind of answer, this request should be refused, this ambiguity should trigger a clarifying question.

Teams that invest in that early make better decisions about everything else, including which model to use, whether a change is an improvement, and when the system is ready for more users.

There is a fair counterpoint. In early exploration, speed matters more than rigour, and building a full evaluation harness for an idea that may be abandoned next week is wasteful. I agree. The problem isn't skipping evaluation while exploring. The problem is shipping the exploration and calling it a product.

My view: the transition from demo to product is not primarily a prompt-engineering exercise. It is an engineering and operating-model challenge, and the teams that treat it that way are the ones whose systems are still trusted a year later.


A demo earns attention. Evaluation, guardrails, and operational discipline earn trust.

END OF ARTICLEExplore more thinking ↗
DISCUSSION

Reactions & comments

Sign in with GitHub to react or join the conversation.

Discussion is not configured yet.

Add your giscus repoId and categoryId to site.config.ts.