/ AI reliability

Proving AI is right, before it ships and while it runs.

AI is fast, fluent and confident, and it is wrong more often than it looks. It shows up in two places: writing your code, and running inside your product. This page is the loop we put around each, and the rules that keep a model from grading its own work.

Part one

AI that builds your software.

AI coding tools have made writing code cheap. They have not made knowing whether it is right any cheaper. This is the loop between a prompt and a merge.

/ What we see

Bad code used to look bad. AI code looks great.

It is clean, commented and compiles. Nothing about it tells you which lines are wrong. Three patterns, from teams that had already adopted AI coding tools before we arrived.

01

Output doubled. So did incidents.

More code, the same reviewers, the same tests. The bottleneck did not move. It got buried under volume that looks finished.

02

90 percent coverage. Nothing caught.

The agent wrote the tests for its own code. They assert what the code does, not what it should do. Every one of them passes.

03

"Tests pass." Two of them were edited to pass.

Left alone with a failing suite, an agent will change the test before it changes the code. Nobody noticed until production did.

/ The loop

Six steps between a prompt and a merge.

Generation is one step of six. The other five are where reliability comes from, and they are the five most teams skip.

  1. 01

    Specify

    A human writes down what correct means for this change: three to five statements, each one checkable.

  2. 02

    Test first

    Those statements become tests before any code is generated. The spec is the test. The agent builds to it.

  3. 03

    Generate

    The agent writes the implementation. It can read the tests. It cannot change them.

  4. 04

    Prove

    Mutation testing breaks the code on purpose and checks the tests notice. Contract checks catch any API shape the agent quietly changed.

  5. 05

    Gate

    Nothing merges without the evidence: spec coverage, mutation score on the touched modules, the critical flows green end to end.

  6. 06

    Watch

    One number in production: defects found by customers per release. If AI is helping, it goes down. If it is not, nothing else matters.

/ Gates

Five things that stop a merge.

Each of these is a CI check, not a review comment. An agent cannot negotiate with a failing build.

  1. BLOCK 01

    Test files changed in the same PR as the code

    Tests move in their own PR, reviewed by a person. An agent that needs a test changed has to ask.

  2. BLOCK 02

    Mutation score below threshold on touched modules

    Coverage is not the metric. The share of deliberate bugs the tests catch is.

  3. BLOCK 03

    A spec statement with no test pointing at it

    Every statement of what correct means is traced to at least one test, the same way requirements are traced in our quality system.

  4. BLOCK 04

    A public API shape changed without a contract update

    Agents rename fields and reorder responses "for clarity". The consumer finds out in production unless the contract catches it first.

  5. BLOCK 05

    A critical flow not exercised end to end

    Signup, checkout, the report your biggest customer opens on Monday. Real browser, real API, every build.

/ Where AI belongs

Use it for the reading. Never for the verdict.

The same agents that run our quality system run here. The difference is what they are allowed to touch.

YES

Edge cases from the spec

Give it the requirement, not the code, and ask what inputs would break it. This is where it earns its keep.

YES

Reviewing the diff against house rules

Weak assertions, happy-path-only tests, fragile locators, fixed sleeps. It reads every PR and never gets tired.

YES

Classifying failures with evidence

Product bug, test bug, environment, flake. Each verdict points at a file, a line or a metric.

NO

Writing tests from the implementation

It will describe what the code does and call it a test. Useless by construction.

NO

Deciding what correct means

That is the one job that cannot be delegated. Someone accountable writes it down.

NO

Approving its own work

Every output leaves sign-off pending for a named role. The agent cannot close the loop.

/ The position

Four rules the system enforces on us.

“Writing code is no longer the hard part. Knowing whether it is right is.”
Every gate on this page exists because an agent can produce fluent, plausible, wrong code faster than a reviewer can read it.
“The spec is the test.”
If what correct means is not written as something executable before generation starts, the agent decides, and it decides in its own favour.
“Coverage is a lie you tell yourself. Mutation score is not.”
Ninety percent of lines executed with thirty percent of deliberate bugs caught is the normal shape of AI-written test suites.
“Build the gates no agent can talk its way past.”
A rule that lives in a prompt is a suggestion. A rule that lives in CI is a gate.
/ Built in the open

The files arrive as the engagements run.

Our quality engineering reference shows real files from a running system. This page is younger. The method is in use; the artefacts below will be published here, with the real configuration behind them, as each one is proven on client code rather than described.

  • Spec-to-test tracing

    The YAML format a spec statement takes and the annotation a test carries to claim it.

    Adapted from the quality reference
  • Agent review rules

    The checks the PR review agent applies and the schema its findings have to fit.

    Shared with the quality reference
  • Locked test policy

    The CI rule and the review template that keep test files out of implementation PRs.

    Next
  • Mutation score as a gate

    Per-module thresholds, the runner configuration, and what a failing report looks like.

    Next
  • Escape rate dashboard

    Defects found by customers per release, the only production number this method reports on.

    Next
Part two

AI that runs in your product.

We build agents and AI workflows that run inside real products. Once a model is live, the question changes from whether its code is right to whether its decisions are safe. A model in production is an untrusted component, and we build around it that way.

/ What goes wrong

The demo works. Production is where it fails quietly.

01

Confident and wrong.

A model does not know when it is guessing. A made-up name, date or figure reads exactly like a real one, and it reads well.

02

It follows instructions nobody gave it.

Everything it reads is input: a customer email, a scraped page, a document. Any of it can carry an instruction, and there is no filter that reliably catches them all.

03

Nothing crashes.

A wrong answer raises no error and trips no alert. It looks like a good day in the logs until a customer forwards it to you.

/ The loop

Six steps around a model that acts.

The model is one component. These six steps are the system around it, and they are what decides whether it is safe to put in front of a customer.

  1. 01

    Scope

    Give the model the least freedom that does the job. Known steps become a fixed workflow. An agent only where the path genuinely cannot be predicted.

  2. 02

    Ground

    The model works from data we hand it, never from recollection. "Not found" is always an allowed answer.

  3. 03

    Bound

    Permissions live in code. The model requests, the code decides. Anything that sends, pays or deletes waits for a person.

  4. 04

    Test

    A fixed set of real cases, including attempts to trick it, runs on every prompt or model change. There is a score and a bar.

  5. 05

    Trace

    Every model call, every version and every human edit is recorded, so any output can be explained after the fact.

  6. 06

    Watch

    What it costs, what it blocked, what people corrected, release over release.

/ Example

An outbound sales agent, step by step.

An example of an agent built on this loop. A model researches nothing and sends nothing on its own: it drafts, classifies replies and suggests the next move, and every one of those outputs passes through rules in code before anything happens.

  1. 01 · SCOPE

    A fixed workflow, not an agent. The model writes three of six blocks in each email. The other three are inserted word for word by code, so it cannot drift on who the sender is or what is on offer.

  2. 02 · GROUND

    Each draft is written from one researched lead record, and must cite exactly one concrete thing from the evidence on it. No evidence, nothing to say.

  3. 03 · BOUND

    Only one named person can approve a send. By default the system cannot send at all: it holds read-only access to the mailbox and a person sends by hand.

  4. 04 · TEST

    Offline tests assert the rules on every change: fixed blocks in place, cost only in the closing block, length limits, no lead in a working state without a due date.

  5. 05 · TRACE

    Every draft version, every instruction the reviewer gave and every classified reply is kept against the lead. That history is the eval set.

  6. 06 · WATCH

    A daily brief of what needs a person, what moved and what is stalling. The blocks the reviewer rewrites most are the next prompt change.

/ Guardrails

Five rules that do not depend on the model behaving.

A prompt can ask a model to be careful. These hold even when it is not, because the model never gets the chance to break them.

  1. RULE 01

    Nothing sends without a named person

    The approval comes from one identified user, checked in code. The model cannot approve its own output, and a message in the thread from anyone else is ignored.

  2. RULE 02

    Least access, granted at the source

    In manual mode the system asks the mail provider for read-only access. It is not told not to send. It is unable to.

  3. RULE 03

    Hard limits and an off switch

    A daily send cap, a maximum number of follow-ups per lead, and a kill command that halts every send at once.

  4. RULE 04

    Answers from a fixed list

    When the model classifies a reply, its label must be one of a known set. Anything else is forced to "other" before the code acts on it.

  5. RULE 05

    The model labels. The code decides what happens.

    An unsubscribe or a bounce closes the lead with no further contact. Anything that needs judgement goes to a person. A reply from an unexpected sender is flagged, not trusted.

/ The position

Four rules for a model that acts.

“Treat the model as untrusted.”
Not because it is bad, because it cannot tell you when it is wrong, and anything it reads can try to steer it.
“The prompt asks. The code enforces.”
The same rule as the merge gates above, applied at runtime. Anything that must never happen is made impossible in code.
“No evals, no autonomy.”
A model earns the right to act alone with a score, not a good feeling. Until then a person sends.
“If it was not recorded, it cannot be explained.”
The same input can give a different output tomorrow. The record is the only way to answer why.
/ Built in the open

What is in place, and what is next.

Guardrails and a full record come first. Evals are built from that record, which is why an agent like this waits for a person before it sends until a score earns it more freedom.

  • Runtime guardrails

    Named approver, read-only by default, daily cap, touch limit, kill switch, fixed label set, automatic close on unsubscribe and bounce.

    In place
  • Version and instruction history

    Every draft, every human edit and every classified reply, kept per lead.

    In place
  • Offline rule tests

    Template assembly and state rules checked on every change, with no model calls.

    In place
  • Reply classifier eval

    Hand-labelled replies scored against the classifier, with the bar it has to clear before it acts unattended.

    In progress
  • Injection test set

    Inbound messages written to steer the model, run against every prompt change.

    Next
  • Cost and latency per run

    Tokens, spend and time per step, so a change that doubles the bill shows up before the invoice does.

    Next
/ How we work

We make testing disappear into development.

We find where quality is not built in. We wire the loop into your daily routine so it cannot be skipped. We stay until your developers own it. Then we leave. The goal of every engagement is to not be needed anymore.

See the quality system this builds on