Most AI pilots do not fail on the model. They fail on everything around the model: data nobody trusts, a process nobody wrote down, an owner who moved teams, a risk review that started after launch. We have seen this enough times to say it plainly. The pilot was dead before the first prompt was written.
This is the audit we run with a client before we agree to build anything. Fourteen questions in four groups. Answer them honestly and you will know whether you are ready to automate, or whether you have a data project or a process project wearing an AI costume.
Data
1. Can you name the system of record for every field the automation will read?
If the answer is “the ERP, mostly, but sales keeps a sheet”, you do not have a system of record. An LLM will happily reconcile two conflicting truths into one confident wrong answer. Decide which source wins before you build.
2. Is the same thing called the same thing everywhere?
Customer ID in the CRM, account code in the ERP, party name on the invoice. Every mismatch is a join the model has to guess at. On one invoice-OCR project, 30% of early review flags were pure naming drift, not extraction errors.
3. Do you know the error rate of the data today?
You cannot measure improvement from a baseline you never took. Pull 200 recent records and check them by hand. It takes an afternoon and it tells you whether the model or the data will be the bottleneck.
4. Can the automation reach the data through an API or a database, not a screen?
Screen-scraping and RPA over a UI is a maintenance debt with a launch date. If the only path in is a login page, the first item on the plan is an integration, not a model.
Process
5. Can someone write the process on one page?
Not a flowchart with forty boxes. One page. If the person who runs it cannot describe it, the model is going to learn the process from examples that quietly encode every bad habit in the team.
6. Does the process have a defined "done"?
“Order is posted to the ERP with a matching PO and no open exceptions” is done. “Order is handled” is not. Automations that lack an exit condition loop, retry, or silently drop work.
7. What happens on the exception path today?
The happy path is maybe 80% of volume and 20% of the effort. Ask who handles the odd cases, how they find out, and what they do. That is the part you are automating whether you planned to or not.
8. Is the volume high enough to matter?
A process that runs 15 times a month does not justify a pipeline with monitoring, retries and a review queue. Pick the thing that eats hours every day. For one logistics client that was order entry: 4 hours a day, every day.
People
9. Who owns the output after go-live?
Not the vendor. Not “IT”. A named person whose job gets easier when the automation works and harder when it does not. If nobody wants that role, nobody will notice when accuracy drifts.
10. Will the people whose work changes help design it?
The clerk who keys orders knows every supplier that sends PDFs sideways. Build without them and you will rediscover all of it in week two, at production cost.
11. Who reviews what the model got wrong, and how often?
Human-in-the-loop is not a checkbox, it is a rota. Somebody has to open the review queue daily, clear it, and feed corrections back. Budget the hours or the queue becomes a graveyard.
Risk
12. What does one wrong action cost?
A mis-tagged support ticket costs minutes. A wrong quantity posted to a purchase order costs a truck. The answer sets your confidence threshold and decides whether the model gets to write or only recommend.
13. Where does customer or employee data go when it hits the model?
Which provider, which region, retained for how long, under which contract. If you cannot answer in one sentence, legal will stop the project later at a much worse time. Encrypt tokens and credentials at rest (we use AES-256-GCM) and log every model call.
14. Can you switch it off in under five minutes?
A feature flag, a kill switch, a fallback to the manual path. Every automation we ship has one. The day you need it is not the day to be reading deployment docs.
The model is the cheapest part of the system to replace. The data, the process and the owner are not.
— Jaimin Shah, Founder & Principal Engineer, CodeCrafters
Scoring yourself
Add up your fourteen scores. Maximum is 28.
- 0 to 12: not ready. You have a data or process project. Do that first. It is cheaper and it makes the AI project real later.
- 13 to 21: pilot with guardrails. Pick one high-volume process, keep the model in recommend-only mode, put a human on every write for the first month.
- 22 to 28: automate. Build the pipeline, set confidence thresholds from your baseline error rate, and let the model act on the easy 80% from day one.
One more rule. Any single question scored 0 in the Risk group vetoes the total. A 26 with no kill switch is a 26 you cannot ship.
If you want a second pair of eyes on your scores, we run this audit as a fixed-scope, half-day session. You walk out with a scored sheet and a one-page plan, whether or not you build with us.
Written by
Jaimin Shah
Founder & Principal Engineer, CodeCrafters
CodeCrafters exists because most enterprise software is fragile, and the reason is almost never the software. It is the depth of thinking behind it. We were built out of years spent inside a major ERP vendor watching rollouts that technically shipped and never actually landed.