/ AI reliability
Proving AI is right, before it ships and while it runs.
AI is fast, fluent and confident, and it is wrong more often than it looks. It shows up in two places: writing your code, and running inside your product. This page is the loop we put around each, and the rules that keep a model from grading its own work.
AI that builds your software.
AI coding tools have made writing code cheap. They have not made knowing whether it is right any cheaper. This is the loop between a prompt and a merge.
Bad code used to look bad. AI code looks great.
It is clean, commented and compiles. Nothing about it tells you which lines are wrong. Three patterns, from teams that had already adopted AI coding tools before we arrived.
Output doubled. So did incidents.
More code, the same reviewers, the same tests. The bottleneck did not move. It got buried under volume that looks finished.
90 percent coverage. Nothing caught.
The agent wrote the tests for its own code. They assert what the code does, not what it should do. Every one of them passes.
"Tests pass." Two of them were edited to pass.
Left alone with a failing suite, an agent will change the test before it changes the code. Nobody noticed until production did.
Six steps between a prompt and a merge.
Generation is one step of six. The other five are where reliability comes from, and they are the five most teams skip.
- 01
Specify
A human writes down what correct means for this change: three to five statements, each one checkable.
- 02
Test first
Those statements become tests before any code is generated. The spec is the test. The agent builds to it.
- 03
Generate
The agent writes the implementation. It can read the tests. It cannot change them.
- 04
Prove
Mutation testing breaks the code on purpose and checks the tests notice. Contract checks catch any API shape the agent quietly changed.
- 05
Gate
Nothing merges without the evidence: spec coverage, mutation score on the touched modules, the critical flows green end to end.
- 06
Watch
One number in production: defects found by customers per release. If AI is helping, it goes down. If it is not, nothing else matters.
Five things that stop a merge.
Each of these is a CI check, not a review comment. An agent cannot negotiate with a failing build.
- BLOCK 01
Test files changed in the same PR as the code
Tests move in their own PR, reviewed by a person. An agent that needs a test changed has to ask.
- BLOCK 02
Mutation score below threshold on touched modules
Coverage is not the metric. The share of deliberate bugs the tests catch is.
- BLOCK 03
A spec statement with no test pointing at it
Every statement of what correct means is traced to at least one test, the same way requirements are traced in our quality system.
- BLOCK 04
A public API shape changed without a contract update
Agents rename fields and reorder responses "for clarity". The consumer finds out in production unless the contract catches it first.
- BLOCK 05
A critical flow not exercised end to end
Signup, checkout, the report your biggest customer opens on Monday. Real browser, real API, every build.
Use it for the reading. Never for the verdict.
The same agents that run our quality system run here. The difference is what they are allowed to touch.
Edge cases from the spec
Give it the requirement, not the code, and ask what inputs would break it. This is where it earns its keep.
Reviewing the diff against house rules
Weak assertions, happy-path-only tests, fragile locators, fixed sleeps. It reads every PR and never gets tired.
Classifying failures with evidence
Product bug, test bug, environment, flake. Each verdict points at a file, a line or a metric.
Writing tests from the implementation
It will describe what the code does and call it a test. Useless by construction.
Deciding what correct means
That is the one job that cannot be delegated. Someone accountable writes it down.
Approving its own work
Every output leaves sign-off pending for a named role. The agent cannot close the loop.
Four rules the system enforces on us.
“Writing code is no longer the hard part. Knowing whether it is right is.”
“The spec is the test.”
“Coverage is a lie you tell yourself. Mutation score is not.”
“Build the gates no agent can talk its way past.”
The files arrive as the engagements run.
Our quality engineering reference shows real files from a running system. This page is younger. The method is in use; the artefacts below will be published here, with the real configuration behind them, as each one is proven on client code rather than described.
- Adapted from the quality reference
Spec-to-test tracing
The YAML format a spec statement takes and the annotation a test carries to claim it.
- Shared with the quality reference
Agent review rules
The checks the PR review agent applies and the schema its findings have to fit.
- Next
Locked test policy
The CI rule and the review template that keep test files out of implementation PRs.
- Next
Mutation score as a gate
Per-module thresholds, the runner configuration, and what a failing report looks like.
- Next
Escape rate dashboard
Defects found by customers per release, the only production number this method reports on.
AI that runs in your product.
We build agents and AI workflows that run inside real products. Once a model is live, the question changes from whether its code is right to whether its decisions are safe. A model in production is an untrusted component, and we build around it that way.
The demo works. Production is where it fails quietly.
Confident and wrong.
A model does not know when it is guessing. A made-up name, date or figure reads exactly like a real one, and it reads well.
It follows instructions nobody gave it.
Everything it reads is input: a customer email, a scraped page, a document. Any of it can carry an instruction, and there is no filter that reliably catches them all.
Nothing crashes.
A wrong answer raises no error and trips no alert. It looks like a good day in the logs until a customer forwards it to you.
Six steps around a model that acts.
The model is one component. These six steps are the system around it, and they are what decides whether it is safe to put in front of a customer.
- 01
Scope
Give the model the least freedom that does the job. Known steps become a fixed workflow. An agent only where the path genuinely cannot be predicted.
- 02
Ground
The model works from data we hand it, never from recollection. "Not found" is always an allowed answer.
- 03
Bound
Permissions live in code. The model requests, the code decides. Anything that sends, pays or deletes waits for a person.
- 04
Test
A fixed set of real cases, including attempts to trick it, runs on every prompt or model change. There is a score and a bar.
- 05
Trace
Every model call, every version and every human edit is recorded, so any output can be explained after the fact.
- 06
Watch
What it costs, what it blocked, what people corrected, release over release.
An outbound sales agent, step by step.
An example of an agent built on this loop. A model researches nothing and sends nothing on its own: it drafts, classifies replies and suggests the next move, and every one of those outputs passes through rules in code before anything happens.
- 01 · SCOPE
A fixed workflow, not an agent. The model writes three of six blocks in each email. The other three are inserted word for word by code, so it cannot drift on who the sender is or what is on offer.
- 02 · GROUND
Each draft is written from one researched lead record, and must cite exactly one concrete thing from the evidence on it. No evidence, nothing to say.
- 03 · BOUND
Only one named person can approve a send. By default the system cannot send at all: it holds read-only access to the mailbox and a person sends by hand.
- 04 · TEST
Offline tests assert the rules on every change: fixed blocks in place, cost only in the closing block, length limits, no lead in a working state without a due date.
- 05 · TRACE
Every draft version, every instruction the reviewer gave and every classified reply is kept against the lead. That history is the eval set.
- 06 · WATCH
A daily brief of what needs a person, what moved and what is stalling. The blocks the reviewer rewrites most are the next prompt change.
Five rules that do not depend on the model behaving.
A prompt can ask a model to be careful. These hold even when it is not, because the model never gets the chance to break them.
- RULE 01
Nothing sends without a named person
The approval comes from one identified user, checked in code. The model cannot approve its own output, and a message in the thread from anyone else is ignored.
- RULE 02
Least access, granted at the source
In manual mode the system asks the mail provider for read-only access. It is not told not to send. It is unable to.
- RULE 03
Hard limits and an off switch
A daily send cap, a maximum number of follow-ups per lead, and a kill command that halts every send at once.
- RULE 04
Answers from a fixed list
When the model classifies a reply, its label must be one of a known set. Anything else is forced to "other" before the code acts on it.
- RULE 05
The model labels. The code decides what happens.
An unsubscribe or a bounce closes the lead with no further contact. Anything that needs judgement goes to a person. A reply from an unexpected sender is flagged, not trusted.
Four rules for a model that acts.
“Treat the model as untrusted.”
“The prompt asks. The code enforces.”
“No evals, no autonomy.”
“If it was not recorded, it cannot be explained.”
What is in place, and what is next.
Guardrails and a full record come first. Evals are built from that record, which is why an agent like this waits for a person before it sends until a score earns it more freedom.
- In place
Runtime guardrails
Named approver, read-only by default, daily cap, touch limit, kill switch, fixed label set, automatic close on unsubscribe and bounce.
- In place
Version and instruction history
Every draft, every human edit and every classified reply, kept per lead.
- In place
Offline rule tests
Template assembly and state rules checked on every change, with no model calls.
- In progress
Reply classifier eval
Hand-labelled replies scored against the classifier, with the bar it has to clear before it acts unattended.
- Next
Injection test set
Inbound messages written to steer the model, run against every prompt change.
- Next
Cost and latency per run
Tokens, spend and time per step, so a change that doubles the bill shows up before the invoice does.
We make testing disappear into development.
We find where quality is not built in. We wire the loop into your daily routine so it cannot be skipped. We stay until your developers own it. Then we leave. The goal of every engagement is to not be needed anymore.