AI systems that stay in production

The pilot that impressed everyone

You have probably watched one die. The pilot that lit up the room in the demo – the proposal drafter, the inbox that triaged itself – and then, within a quarter, quietly stopped mattering. The intern who built it left for a job. The prompt drifted. Nobody trusted the output enough to stop checking it by hand, so the checking became the job again, and then the checking became the whole job. No one announced a failure. There was no failure to announce. The team simply went back to the old way, and after a while nobody remembered to cancel the subscription.

When I audit a firm’s past AI attempts, this is the pattern I find most often. The model was almost never the problem. The engineering around the model was. Keeping AI in production for a small business is not a research problem; it is an ownership problem, and ownership can be designed.

Six ways a system dies

In my experience, small-firm AI projects die in predictable ways.

• No named owner. The person who built it leaves – the intern, the agency, the enthusiastic associate – and nobody inherits it. Software without an owner decays on a schedule.
• No monitoring. Failures are silent. The system stops working on a Tuesday and the firm finds out three weeks later, usually from a client.
• No evaluation suite. Someone tweaks a prompt, or the underlying model changes, and quality quietly regresses. Nothing catches it, because nothing was measuring it.
• No fallback and no kill switch. The first bad output reaches a client and trust dies permanently. There was no graceful way to pause the system, so the firm abandoned it instead.
• No training. The team never learned to operate it, so they route around it, and the old manual process reasserts itself within weeks.
• Built as a demo. It lived in a side tool nobody opens, not inside the workflow where the work actually happens. Integration was always going to come later, and later never came.

Notice that none of these six is about the model’s intelligence.

A system that fails silently loses trust exactly once. After that, no one ever stops checking by hand.”

The shipping standard

The counter-discipline is not exotic. A production AI deployment deserves the same engineering that any serious production software gets – the discipline small firms are routinely sold without. Every system I ship carries all of it, every time, before I call it done:

• Production deployment on your stack. Versioned and reproducible, inside the tools your firm already runs – not a demo on my laptop.
• Observability. Every call traced, every error logged. When something fails, it fails loudly, in front of a person whose job is to see it.
• An evaluation suite. A fixed set of test cases run before any change goes live, so a prompt edit or a model update cannot silently regress quality.
• A kill switch you control. One toggle, no tickets, no waiting on me. If you want the system off at eleven at night, it is off at one minute past.
• A one-page runbook. What it does, who owns it, what to do if. One page, because a runbook nobody reads is just another silent failure.
• Team training. The people who will operate it, trained before I step back. This is the difference between a system your firm uses and one it routes around.
• A thirty-day stability commitment. After handover, I stay on the hook. If it breaks in the first thirty days, that is my problem, not yours.

None of this argues for building everything from scratch. Sometimes the right production system is bought, not built – I have set out how I make that call in build vs. buy for small firms. But bought systems die the same six deaths when nobody owns them. The standard applies either way.

The test

There is one test that matters, and it is blunt: the system keeps running when the builder leaves the chair. Everything on the list above exists to pass it. The named owner, the runbook, and the training answer who keeps it running. The monitoring and the evaluation suite answer how you know it still works. The kill switch and the fallback answer what happens when it doesn’t. Remove any one of them and you are back where you started: a pilot that impressed everyone, dead within a quarter.

Anyone can ship a demo. The discipline is what still runs a year later.

What this looks like over ninety days

In a Jovadan engagement, this standard is not an aspiration; it is the definition of shipped. Over roughly ninety days I put about three systems into production, one a month, and each one carries the full list before it counts. That accountability – for AI in production rather than AI in a demo – is most of what separates the role from a pilot vendor, as I laid out in what a Fractional AI Officer does.

It is also what makes the outcomes durable. The results a Day-90 review is built around – roughly twenty hours a week reclaimed, one full-time hire’s capacity unlocked, a 15–25% lift in margin – only mean something if the systems are still running in month six, with the founder’s hand on the switch and the team operating them without me. The first documented engagement, Carol Coelho Náutica, was run to exactly this standard, and it is written up so you can check the claim rather than take it.