Approach & Methodology
Our AI Consultancy Approach: Problem First, Architecture Second
Most AI pilots never reach production, and the reason is rarely the model. Here is the architecture and engagement method we use to avoid that.
Quarza Team · · 8 min read
The most consistent finding in enterprise AI over the past two years is not that the technology underdelivers. It is that most organisations never get far enough to find out.
The numbers vary by source and methodology, and they should be read with that caveat, but they point the same direction. RAND's analysis of enterprise AI initiatives put the failure-to-deliver-value rate above eighty percent, roughly double the rate for comparable non-AI IT projects. MIT's review of publicly disclosed generative AI deployments found the overwhelming majority produced no measurable profit and loss impact. Gartner has projected that a large share of AI projects lacking production-ready data foundations will be abandoned. Surveys of enterprise pilots consistently find that a small minority reach production scale.
Read across those sources and a pattern emerges that is worth stating plainly: the models generally work. The projects generally do not. And the causes cited most often are not algorithmic. They are data readiness, integration complexity, absent success metrics and no defined owner.
That diagnosis shapes how we work.
Key Takeaways
- We start from a problem and how it is currently solved, not from a technology looking for an application.
- Not every problem needs AI. Sometimes the correct answer is a rules engine and a better workflow.
- The same five-layer architecture underlies our logistics, healthcare and transport work.
- Confidence is a first-class output. Every system needs a defined path for what happens when it is not sure.
- Integration into existing systems of record is where projects fail, so it is scoped first rather than last.
Principle One: The Problem Comes First
Every engagement starts with the same sequence of questions, and none of them are about models.
- What is the problem, stated specifically enough to be measured?
- How is it solved today, including the workarounds nobody documented?
- What does a failure currently cost, and who absorbs that cost?
- Who would use the output, inside what workflow, on what device?
- What would have to be true for this to be considered working?
The last question is the one most often skipped. A project without an agreed success metric cannot be finished, only abandoned, which is precisely the pattern the failure research describes.
Understanding the current manual process matters more than it sounds. The people doing a task by hand have usually developed heuristics that encode real domain knowledge. A warehouse supervisor who can tell at a glance which courier a label belongs to is applying a set of visual rules worth extracting. Automation that ignores that knowledge tends to rediscover it slowly and expensively.
Principle Two: Not Every Problem Needs AI
The question we ask is not "where can we apply AI." It is "what is the right technology for this problem."
Sometimes that is computer vision. Sometimes OCR. Sometimes location tracking. Sometimes a deterministic rules engine and a redesigned workflow, with no model anywhere in the system.
This matters commercially as well as technically. A model introduces probabilistic behaviour, a data pipeline, a monitoring obligation and a retraining cycle. If a problem can be solved deterministically, adding a model adds cost and fragility for no benefit. The discipline is knowing which is which.
A practical heuristic: use rules where the logic is known, stable and expressible. Use models where the input is perceptual, variable or too diverse to enumerate. Most real systems need both, with the model handling perception and the rules handling decisions.
The Reference Architecture
Across very different domains, we keep arriving at the same five layers. This is not a coincidence. It reflects the structure of problems where software has to understand something happening in the physical world.
- Input
- AI
- Intelligence
- Business logic
- Integration
Layer 1: Input
Camera, scanner, PDF, mobile photo, video stream, GPS feed, sensor telemetry, API.
The work here is unglamorous and disproportionately valuable: normalising, deskewing, denoising, buffering when connectivity drops, and handling the input nobody anticipated. Systems that assume clean input fail immediately, because production input is never clean.
Layer 2: AI
OCR, object detection, classification, vision-language models, embeddings, speech.
This is the layer that gets discussed most and determines project outcomes least. Model selection matters, but it is a smaller decision than the surrounding architecture.
Layer 3: Intelligence
Extraction, validation, matching, anomaly detection, confidence scoring.
This is where raw model output becomes trustworthy. A value read from an image is a hypothesis. Cross-checking it against an independent source, a format rule, a reference range or a master record is what turns it into data. The most valuable thing this layer produces is a well-calibrated signal about what the system does not know.
Layer 4: Business Logic
Thresholds, routing rules, approval paths, alerting, escalation, workflow state.
This layer encodes the organisation's risk tolerance. It decides what gets automated, what gets reviewed, who is told, and how quickly. It is the layer where consultancy work is genuinely consultative, because these are policy decisions rather than technical ones.
Layer 5: Integration
ERP, WMS, order management, hospital information systems, school platforms, mobile apps, APIs.
The integration layer is where enterprise AI projects most often die. The failure research is consistent on this: the gap between a pilot and production is largely the compounding complexity of connecting to systems of record that the pilot deliberately avoided.
Our response is to scope integration in the first conversation rather than the last. If a working prototype cannot get its output into the system that runs the business, it has not reduced anyone's workload.
Confidence as a Design Output
If there is one idea that carries across every project we take on, it is this.
Traditional software is deterministic. Given an input, it returns an answer or an error. Systems built on perception are different. They return an answer and a degree of belief, and pretending otherwise is how automation causes damage.
So every system we build produces confidence explicitly, at field or event level rather than aggregate, and every system has a defined behaviour for each confidence band.
Extraction plus validation plus confidence, rather than extraction equals truth.
Three exits, in every domain:
| Lane | Condition | Behaviour |
|---|---|---|
| Automatic | All required outputs above threshold, validations passed, no exception rule triggered | System acts without human involvement |
| Assisted review | One or more outputs below threshold, or a soft rule triggered | Human sees the flagged item and the reason, confirms or corrects |
| Mandatory manual | Unreadable input, unknown class, or a hard rule requiring sign-off | Human decides regardless of model confidence |
The third lane matters because some decisions should never be automated, and encoding that as a deterministic rule is safer than trusting a threshold to hold.
Thresholds themselves are an economic decision, not a technical one. They depend on what a wrong output costs downstream and how much review capacity actually exists. A threshold set to hit an automation target rather than a risk target produces impressive metrics and eventual incidents.
How We Run an Engagement
- Problem
- Prototype
- Real-world test
- Failure
- Improvement
- Product
Phase 1: Understand. Sit with the current process. Collect real examples, including the bad ones. Define the success metric and who judges it. Identify the integration target.
Phase 2: Prototype narrowly. Prove one workflow is technically possible on real data. Not a platform. One decision, end to end, on the messiest inputs available.
Phase 3: Shadow mode. The system runs and produces outputs. Nothing acts on them. Outputs are compared against current practice, which produces the only accuracy figure that means anything: performance on your data, in your conditions.
Phase 4: Controlled automation. Enable automatic processing for the highest-confidence, lowest-risk slice only. Watch the leaked error rate ahead of every other metric. Expand from evidence.
Phase 5: Widen and monitor. Add document types, routes, sites or categories one at a time, calibrating each separately. Put monitoring in place for drift, because these systems degrade quietly.
Real users surface problems that no controlled test produces. Those problems become the requirements for the next iteration. This is why we prefer to start small: a narrow thing in production teaches more than a broad thing in a demo.
What We Measure
Vague success criteria are the most reliable predictor of a stalled project. The metrics we agree upfront usually include:
- Field-level or event-level accuracy on a calibration set drawn from the client's own data
- Straight-through rate, meaning the share requiring no human involvement
- Escalation rate, checked against actual review capacity rather than a target
- Leaked error rate, meaning wrong outputs that were automatically accepted
- Time saved in the specific workflow being replaced
- Exception resolution time
Leaked error rate is the one we watch hardest. Every other metric can be improved by loosening thresholds. That one gets worse when you do, which is what makes it honest.
The Same Architecture, Different Domains
This is not a theoretical framework. It is a description of what our projects have converged on.
| Logistics documents | Hospital documents | School transport | |
|---|---|---|---|
| Input | Labels, manifests, invoices | Reports, prescriptions, forms | GPS, sensors, cameras |
| AI | OCR, logo and layout detection | OCR, vision-language models | Detection and pattern models |
| Intelligence | AWB and barcode cross-check | Range and dictionary validation | Deviation and anomaly detection |
| Business logic | Auto-process or flag | Auto-record or clinician verify | Alert, escalate or log |
| Integration | OMS, WMS, ERP | HIS, ABDM-aligned records | School dashboard, parent app |
The domains share almost no subject matter. They share nearly all of their structure. That is what allows a small team to work credibly across them, and it is why we invest in the architecture rather than in one vertical product.
What We Do Not Do
Stating the boundary is part of doing the work honestly.
- We do not promise perfect accuracy. Any vendor who does has not measured on your data.
- We do not replace working systems of record. We feed them.
- We do not automate decisions whose failure cost the client has not explicitly accepted.
- We do not treat regulatory constraints as a final-phase task. In healthcare and children's data, they are design inputs.
- We do not build models where rules would work.
Frequently Asked Questions
Why do most AI projects fail to reach production?
The research points consistently at non-technical causes: data that was not ready, integration into systems of record that was underestimated, no agreed success metric, and no single owner. Models are rarely the bottleneck. Estimates of the failure rate vary widely by definition and source, but every major study points the same direction.
Do you always recommend AI?
No. Where logic is known, stable and expressible, deterministic rules are cheaper, more predictable and easier to maintain. Models earn their place when the input is perceptual or too variable to enumerate. Most good systems combine both.
How long does a first project take?
The modelling is rarely the constraint. Collecting and labelling representative data from the client's own environment usually is, and shadow-mode measurement takes as long as it takes to accumulate a fair sample. A narrow first workflow is measured in weeks to a few months depending on data availability, which is why we scope narrowly.
What does human-in-the-loop actually mean in practice?
It means a defined review lane, not an informal fallback. Low-confidence or rule-flagged outputs route to a person with the specific reason highlighted, that person confirms or corrects, and the correction becomes training data. The review path and the improvement path are the same path.
How do you set confidence thresholds?
By starting conservatively, measuring escalation and leaked error rates on the client's own documents or events, and tuning against the cost of a wrong output and the client's real review capacity. Anyone quoting a threshold before seeing your data is guessing.
Can you work with our existing systems?
That is the intended design. The output is delivered through an API into whatever already runs the business. Keeping the integration surface small makes rollout reversible, which is what makes it possible to start.
Start With the Problem
If you have a process that is currently manual, error-prone or invisible, the useful first conversation is about that process rather than about technology. Get in touch and we will tell you honestly whether it needs AI at all.
