When the pilot meets your real data

Most AI pilots are shown on a handful of well-chosen examples: a clear email, a tidy PDF, a question with an obvious answer. They work, everyone is impressed, and the project is approved. Then the system meets the inbox.

Real requests are messier than demos. They arrive as replies to old threads, with the important detail in the third attachment, in a scanned PDF, or in a sentence that contradicts the drawing. Some are in languages nobody tested. A few aren't requests at all.

What usually breaks

• Attachments: scanned drawings, spreadsheets with merged cells, and files called scan0042.pdf.
• Threads: the question in today's email only makes sense with the one from three weeks ago.
• Exceptions: the urgent order, the regular client who skips the form, the request that's really a complaint.
• Ownership: nobody was named to look after the system, so when it misreads something, nobody notices.

Test it on your own past work

Before you commit to any AI system, ours included, ask for it to be run on a sample of your own past requests, say the last fifty, chosen by you and not by the vendor. Because they're past requests, you already know the right answers. Compare what the system produced with what your team actually did, request by request.

Count three things: how often it was right, how often it was wrong in a way someone would have caught, and how often it was wrong in a way that could have reached a client. The last number matters most, and it's the one demos never show.

Decide with the numbers

If the results are good, you also have a baseline to measure the system against once it's live. If they aren't, you've found out for the price of a test instead of a project. Either way, the decision is made on your data, not on a demo.

This is how the audit looks at every opportunity it ranks: what it's worth on your real work, and what it would take to get there.