00 / Where AI sits
We build the version with no model in it first.
We put language models into systems that already run a business — invoices, tickets, schedules, a database somebody else wrote. This page is the whole discipline that comes with one: what we ask before agreeing to build it, what it gets measured against, what is logged, what it is allowed to cost, and the point at which we say the answer is not a model.
01
What do we ask before agreeing to build one?
THREE QUESTIONS
What does a right answer look like, and who in your building can tell one from a wrong one? What does a wrong answer cost when it reaches a customer? And what does the same job cost today, done by a person or by a query? If the first question has no owner, there is nothing to score against, and the honest answer on that call is that this is not an AI project yet.
What happens without it
The build starts against a definition of correct that nobody ever agreed to, and the argument about whether it is good enough arrives after the invoice instead of before the work.
02
What is in the evaluation set?
BEFORE THE FIRST PROMPT
Rows out of your own system, each one with the answer you would accept written beside it, signed off by whoever has to live with a wrong one. Nobody writes a prompt until that file exists, and it sits in the repository next to what it tests — which is the only reason a Tuesday model swap is survivable rather than an emergency: you re-run the file and read a score off it. The awkward rows go in first, the ambiguous ticket and the message that asks two things at once, because a set made only of the cases it already handles is a screenshot.
What happens without it
Every later question — is this better, did the upgrade break it, is the cheaper model good enough — gets settled by opinion, and the opinion that wins belongs to whoever is most confident that week.
03
Why build the version with no model in it?
SCORED ON THE SAME SET
Because it often wins. Before a model goes near production we build the dull one — an index, a join, a table of rules somebody in your office could read out loud — and put it through the same set. If the dull one clears the bar, that is what ships and the invoice is smaller. If the model clears it by more, you know the size of the gap, measured on your own rows, and what you are paying for is a difference rather than a demo.
What happens without it
A model with nothing to beat produces numbers that read as results. A score only becomes information once you know what the dull one scored on the same rows.
04
What gets logged?
INPUT · OUTPUT · LATENCY · COST
Every input, every output, the latency and the cost, on every call. A language model has a failure mode nobody can fully enumerate, so the only way to answer “why did it tell my customer that” a week after it happened is to have kept the record. Logging cost per call is also what turns a monthly bill into a list of the specific requests that produced it.
What happens without it
The first serious complaint becomes a reconstruction from memory. You end up comparing your recollection of the prompt against a bill that arrives a month later and explains nothing.
05
What is the ceiling on spend?
A NUMBER, AND AN ALERT UNDER IT
A monthly number, agreed before launch, with an alert that fires under the ceiling rather than at it. Model pricing is per token, so the cost of a feature is set by how much people use it, and usage is the variable nobody forecasts correctly. Month six always costs more than the demo implied, and the ceiling is what turns that from a surprise into a number somebody already signed.
What happens without it
The bill is the alert. It arrives after the month it describes, which is the exact moment nothing about that month can be changed any more.
06
What happens when the API is down?
THE PATH IS WRITTEN FIRST
There is a defined path, decided before launch, because the API will be down. Which path depends on the job: queue the work and come back to it, fall through to the no-model version that already exists and has already been scored, or fail visibly and tell the person what to do instead. The outage becomes a slower afternoon rather than a product that stopped working.
What happens without it
The outage surfaces as a spinner, a stack trace or silence, and the person who has to explain it to a customer finds out about the dependency on the day it fails.
07
What still goes past a person?
WRITES, AND ANYTHING THAT LEAVES
Anything that writes to your database, and anything that leaves your building. A draft a person approves is a different risk from a message that sends itself. The review is not permanent — it comes off when the logged record and the evaluation set together show it can, which is a decision made in front of a number rather than at the point somebody gets tired of clicking.
What happens without it
The first anyone hears about a wrong answer is the customer replying to it. There is no draft stage to catch it in and no record of who decided it was fine.
08 / Saying no
When is a model the wrong answer?
Most of the time, on the work I am shown. Retrieval and generation are narrow tools and dear ones: they earn their keep on a small class of problems and lose it on everything else. Much of what arrives here with AI written on it is something a query already does, wrapped in a component that fails less predictably and bills by the token — a search box an index already answers, a summary of a document the reader was going to open anyway, a classifier for a form with six options on it.
The tell is which direction the idea travelled. When it starts with a job somebody does by hand every Tuesday and ends at a model, it is usually worth measuring. When it starts at the model and goes looking for a job, what gets built is a query with a language model in front of it, and the bill and the failure mode both stay with you afterwards.
So I turn some of this work down, and that is not scepticism about the technology. The three model vendors we use are filed on the homepage among the databases and the runtimes, at the same size, because a vendor is a dependency. What they do not get is the benefit of the doubt. If the set says the version with no model in it wins, it wins, and you are told that before the build rather than after it.
That is an easy sentence to put on a website. It is a harder one to say on a call to somebody who has already told their board they are doing AI this year, and that call is the only place it is worth anything.
Amel Spahić
Founder · info@velvict.com