NEWMake Viral Reels turns any link into a ready-to-post reel.Try it free →
AI for Enterprise · Playbook

Why AI pilots never make it into production

The pilot went well. The demo worked. The room nodded. Someone senior said "this is great, let's roll it out."

Six months later it is still a pilot. Nobody cancelled it. Nobody shipped it either. It sits in a status report under "in progress."

If that sounds familiar, the cause is usually not the model, and it is usually not your team. The pilot was built to pass one test: impress a room. Production is a different test: run every day, on real inputs, with real people depending on it. A pilot built for the first test rarely passes the second.

A demo and a daily system are built differently

Look at what a demo needs. It needs a handful of inputs that someone picked by hand. It needs one person who knows the tool well and drives it. It needs to work for twenty minutes.

Now look at what a daily system needs. It has to take whatever arrives, including the messy, incomplete, and odd cases. It has to work for people who were never in the project. It has to connect to the systems where the work happens. Someone has to notice when it is wrong, and someone has to fix it.

None of that second list shows up in a demo. So if the goal of the pilot is a good demo, none of it gets built. That is the gap your pilot falls into.

Where pilots get stuck

The same five problems come up again and again.

  • The data was a sample. The pilot ran on a clean export that someone prepared. The live data sits in another system, behind an access request nobody filed, in a shape nobody checked.
  • Nobody owns it after the demo. The innovation team built it. The business unit is supposed to use it. IT is supposed to run it. Each one assumes one of the others will pick it up.
  • "Good" was never defined. The pilot was judged by how the output looked in a meeting. There is no agreed pass mark, so nobody can say it is ready, and nobody can say it is not.
  • Security and legal saw it last. They were invited in after the build. Their questions are fair, and each answer means rework.
  • It lives outside the work. The pilot is a separate screen. Using it means copying things in and pasting things out. People try it for a week, then go back to the old way.

Every one of these is cheap to settle before you build. Every one is expensive to settle after.

Six things that have to be true before you start

Treat this as an entry checklist. If you cannot tick a line, that line is your first piece of work. Do it before anyone opens a prompt window.

1. A named person owns the outcome

Not a committee, and not the team that builds it. You want the person who runs the process today and will still run it in a year. They decide what "good" means. They take the call when it breaks.

What to do: write the name down. If nobody will put their name on it, stop here. You have found out, for free, that this pilot would not have shipped.

2. The work happens often and has a number today

Pick work that happens every day, in volume, and that you can already measure: minutes per case, error rate, backlog size. A task that happens twice a quarter will never give you enough cases to trust the system.

What to do: measure the current process for two weeks before you build. Without a baseline, your results are opinions.

3. You have access to the real data

The pilot should run on the same records, from the same systems, that production will use. That includes the ugly ones: missing fields, scanned pages, duplicates, three date formats.

What to do: file the access request in week one. Pull a random sample from live data, not a selected one. If access will take three months, it is far better to know that now.

4. You wrote down the pass mark

Agree, in plain numbers, what the system must do to go live. Agree what it must never do. Agree what result would make you stop.

What to do: collect real cases with the correct answer attached, and have the owner sign off on the threshold. Here is an illustrative version, not a benchmark: "Correct on 90 of 100 test cases, no wrong payment amounts at all, and anything the system is unsure about goes to a person." Your numbers will be different. What matters is that they exist before the first result comes in, so nobody can move the line afterward.

5. Security, legal, and IT are in the first meeting

Ask them one question early: "What would stop this from going live?" Their answers become your requirements. That is much cheaper than hearing the same answers as objections at the end.

What to do: get the list in writing. Build to it. Go back to the same people with the same list when you ask for approval.

6. Someone has agreed to run it

Running it means monitoring, a budget line, a person who handles failures, and a plan for when the model or the vendor changes. If the pilot team disbands on demo day, the system is an orphan on day one.

What to do: name the team that will run it and get their agreement before the build. Put the monthly running cost in the plan, not only the build cost.

Build the pilot as the first version of production

Once the six lines are ticked, change how you build. The pilot is not a model of the real thing. It is the real thing, kept small.

  • Keep the scope small and the conditions real. One team, one document type, one region. But live data, real users, and the real system of record.
  • Put it where the work already is. If your people work in a ticketing tool, the output should appear in the ticket. Do not ask them to open another tab.
  • Build the dull parts first. Logging, a review queue for uncertain cases, an off switch, a way to correct a wrong answer. These are what your risk team will ask about, and what your users will lean on.
  • Test with people who were not in the project. If only the builders can get good results from it, it is not ready.

A concrete example. Say your finance team wants help matching supplier invoices to purchase orders. The demo version takes ten clean PDFs and shows perfect matches on a big screen. The version that can ship takes last month's real invoices, including the scanned and handwritten ones. It writes its result into the finance system. It sends anything it is unsure about to a clerk. It records what the clerk changed. It looks less impressive in a meeting. It is the one that will still be running next year.

Change what the demo is for

You will still have a demo day. Change what you show. Skip the best five outputs. Show two weeks of real use next to your baseline: how many cases it handled, how many it got wrong, how many went to a person, what it cost, and what your users said.

Then ask the room for a decision, not applause. Ship it, fix it, or stop it. All three are good outcomes. The bad outcome is the fourth one, where the pilot drifts on with no decision at all.

Before your next pilot

Go through the six lines: an owner, frequent work with a baseline, real data, a written pass mark, the blockers in the room, and a team to run it. If you can tick all six, your pilot has a path to production before you write a line of code. If you cannot, you now know where to spend the first month.

The question to ask at kickoff is not "can we make this work in a demo?" It is "who will be running this a year from now, and what do they need from us today?"

If you want more writing like this, the Hyperautomation AI Report is a free newsletter, three times a week, with practical notes on building and using AI. You can subscribe on the newsletter page.

Talk to me about AI in your company

If you're working through an AI rollout, tell me what you're trying to do. I read every message and reply by email.

Or connect with me on LinkedIn.