When we benchmarked an off-the-shelf AI tool on a real business question for one of our clients, it returned 370 records. The correct answer was 21,314. The tool had picked the wrong table, silently dropped a filter and delivered its answer with full confidence.

That gap explains why so many AI pilots stop at the demo stage. Most mid-sized companies we talk to in the US, the UK and Germany already have a pilot: a chatbot over internal documents, an agent that drafts replies, a proof of concept built in two weeks. Far fewer have it running in production a year later.

In our experience the model is rarely the problem. Claude and comparable models handle most business processes well today. Pilots stall on the data the model needs, the permissions it must respect, the people who must trust it and the team that must keep it running. Below are the patterns we have seen in projects that made it through, with the numbers behind them.

Why do AI pilots stall?

The same five problems come up again and again in the companies we work with.

  • No measurable goal. A pilot which started as "let’s see what AI can do" has no finish line. If nobody can say which number it should move, nobody can justify the budget for the next phase.
  • Data that isn't ready. The demo runs on a clean sample. Production runs on inconsistent PDFs, legacy tables and business rules that exist only in people's heads.
  • Governance that arrives last. In Europe, GDPR, client contracts and internal security reviews apply from day one. When they show up after the demo, the pilot often has to be rebuilt.
  • No owner after the demo. An innovation team or an external agency builds the pilot, then hands it to nobody.
  • Trust that breaks on one wrong answer. People rarely go back to an assistant that once gave them a wrong number with confidence.

What separates pilots that reach production?

The projects that reached production had four things in common.

1. The assistant looks things up before it answers

The 370-record answer came from our work with Barton Malow, a large US construction company. Their teams wanted answers to business questions that used to wait in a BI analyst's queue. Off-the-shelf assistants answered anyway, even when they could not reach the right tables or apply rules such as a fiscal year that does not follow the calendar.

We changed the order of work. An orchestrator routes each task to the model best suited for it. The assistant first searches the certified data catalog for the right tables and join rules, and only then writes and runs the query. Arithmetic goes to a deterministic calculator. On real business questions, this approach gave better answers than the native NL2SQL tool in 62% of cases, and it returned the correct 21,314 records in the benchmark above.

2. The data layer decides who sees what

A common shortcut in pilots is to connect the assistant to the data warehouse through one shared account with broad access, then tell the model in its instructions what it must not reveal, for example "don't show salaries". That works in a demo. In production it is fragile: a user can rephrase a question until the model complies, or the model can simply make a mistake. One leaked salary table is enough to end a project, and security teams know it. This is why many assistants never get past a small, trusted pilot group.

Barton Malow took a different route. Every user signs in with their own identity, and the assistant queries the data warehouse with that person's credentials. Access follows each person's data domain, so a project manager, for example, sees project data and a finance analyst sees finance data. The model never receives data the user is not allowed to see, so no prompt can reveal it. With access enforced at the data layer, the company could roll the assistant out to the whole organization instead of keeping it with a pilot group.

3. Uncertain cases go to a person

For Finansi.bg, a Bulgarian financial data platform, Claude extracts structured financial data from more than 2 million company reports. Validation rules catch uncertain results and send them for manual review. Fewer than 8,000 documents needed a human, and the estimated processing cost fell from about €2 million to €18,000.

Human review stayed in the process on purpose. The validation rules decide which 0.4% of documents a person should check.

4. AI sits inside the tools people already use

Many pilots launch as a separate chat app in a new browser tab. In our experience, usage is high in the first weeks and then drops. People go back to the tools where their work already happens, because switching windows and copying context into a chat box takes longer than doing the task the old way. An assistant nobody opens produces no value, however accurate it is.

For Onno Plus, a German review-management business, we built AI into the custom CRM their team uses every day. Lawyers draft review-removal complaints with an AI tool inside the CRM, which combines Anthropic's models with the company's historical cases. There is nothing to copy and no separate app to open. Barton Malow's assistant follows the same principle: people reach it through the web, inside Microsoft Teams or through an API.

What does it realistically take?

Start with one process and one team. In our projects, the first production release usually takes 8 to 12 weeks with a team of 2 to 3 people, plus a process owner and someone from IT on the client side. Four things matter most:

  • One process, one owner, one target metric. Pick the single number the project should move, such as hours saved, response time, error rate or cost per document, and measure it before you start. One clear target is easier to prove than five.
  • A test set of real questions. Fifty to a hundred real cases with known correct answers, collected before launch and rerun after every change.
  • Security and IT involved from week one. Identity, data access, logging and the location of data processing all need answers before the build starts.
  • A budget for running the system. Models change, data changes and usage grows. Someone has to monitor quality and cost every month after launch.

What should you ask a partner before you sign?

These questions help you choose a partner and avoid paying for another demo:

  1. Can you show us an AI system you built that people use every day, and can we speak with that client?
  2. How will you check whether our data is ready before we commit the full budget? A good partner starts with a short review of the data and the process, before writing code.
  3. What will the first production release include, and which number will it move?
  4. Where will our data be processed, and who will be able to see it?
  5. What will the system cost to run per month at our expected usage?

How can you tell if your partner is building for production?

Once the work has started, four questions show whether the project is heading for production or for another demo:

  1. What does the system do when it does not have the data to answer? A good system says so and points the user to the right source instead of guessing.
  2. Where are user permissions enforced: in the prompt or in the data layer?
  3. How do you measure answer quality, and how did the results change after the last update?
  4. Who will run the system after go-live, and who fixes it when an answer is wrong?

A partner who answers these clearly has probably taken pilots to production before. One who talks mainly about model benchmarks probably has not.

The model is the easy part

The companies that get value from AI agents treat them like any other production system. They pick one process, measure it, protect the data and assign someone to run it. Model choice matters, but it rarely decides whether a project survives its first year. For European mid-sized companies, where budgets are tighter and GDPR applies from the first line of code, that discipline turns a two-week demo into a system people use every day.

Share