Everything before “let’s build it”.
Everything long after “it’s live”.
The model alone rarely solves a real problem. Most of the work is what surrounds it: retrieval, evaluation, the human in the loop, and what it costs to keep running.
I build systems that answer from a company’s own documents, and systems that carry out multi-step work on their own. RAG and agentic systems, in the trade. I take both as far as production. But much of the work happens upstream of that certainty: whether the thing should exist, what would prove it works, which parts will have to be thrown away. I’ve been shipping LLM systems since 2023 (first inside a SaaS product, now on my own) and there is no team behind this page: you get me, and the work I sign.
Where I come in
Three stages, and you can enter at any of them. Nothing here assumes the one before it. A scoping note that ends in “don’t build this” is a finished job, and a system already in production is a normal place to start.
Before anyone commits
What is feasible, at what cost, and what should not be built at all.
-
Scoping note
Feasibility before commitment
The moment nobody yet knows whether the thing is possible. Sometimes the answer is to build nothing, which costs less now than it does eighteen months in.
-
Acceptance criteria
An intention, turned into something you can measure
Every project starts as a sentence, not a specification. “We’d like AI on our documents” becomes a criterion you can measure, stress-tested against the incentive it creates before anyone measures against it.
-
Prototype · disposable
Prototypes that settle something
A prototype whose job is to answer one question, written down before anything is built. Not a demo that impresses: a device that makes a decision possible, and gets thrown away once it’s made.
Into production
Deployed, and then kept running, not handed over at the demo.
-
Running system
Building it, and putting it in production
Document retrieval and agentic systems, from the first version that works to the one that survives real use. Mostly Python, on infrastructure I operate myself.
-
Evaluation harness
Evaluation before anyone trusts it
A test set built from the questions people actually ask, scored the same way every time, so a change can be shown to help. Without one, every release is an opinion.
Dependable, not impressive
The half that decides whether anyone is still using it in a year.
-
Ongoing operation
Keeping it running, over time
The unglamorous half, and the rarest: keeping real-world data usable as it keeps arriving, operating the infrastructure, and being the one who is called when it stops. I stay with the systems I deliver, in daily use.
-
Cost and quality envelope
Costs and quality that stay predictable
What it costs to run each month, and what that figure does when usage doubles. Answer quality is tracked against the same test set as on day one, so drift shows up before someone has to report it.
Reading the near horizon
Not predicting — arbitrating. What is stable enough to build on now, what will move in the next twelve to eighteen months, and which parts of the architecture to keep interchangeable so that movement costs a swap rather than a rewrite. It ends in a decision you can act on, never in a trend.
How I think
Every answer here comes out of a note I have kept alongside the work for years, across the systems I have scoped, run or deployed, added to one failure at a time. Call it a list of things I would rather not learn twice. Open a question, then click a §-number to read the note behind it.
Because a general assistant is optimised to produce an answer, not to prove that one exists in your documents.§1
What you need instead is a system that is allowed to return nothing, and that says so where you can see it rather than in a log. That behaviour is written and tested, not hoped for.§2
And every citation is checked against the source before you see it, not taken on the model’s word. Without that check, a citation is just a plausible-looking number.§7
By treating the brief as the risk, not the code.
When something is visibly wrong and the technical analysis answers everything is normal, the conclusion to favour is not that I am misreading the screen — it’s that I misunderstood what we were trying to obtain.§9 A diagnosis validates execution, never the objective.
So work gets ranked by what breaks without it rather than by how easy it is to build,§13 and when a line of investigation stops producing new information I go back for fresh data instead of adding complexity to avoid abandoning a hypothesis I’ve grown attached to.§10
Take the common case, a system answering over your own documents. Ideally it says nothing rather than guessing, and says so where you can see it: an unsupported question comes back as an explicit not found in this corpus, not as a section quietly dropped into a log.§2
When it does answer and the answer is wrong, the citation takes you to the exact paragraph, so you can see where the reasoning left the source.§7 That traceability is the difference between a bug and a liability.
The third part holds whatever the system is: on the day it breaks badly, what matters is what leaks. Secrets are typed so they are masked everywhere, error traces included.§8
Because we agree on what would count as failure before anyone starts measuring.
An acceptance criterion I was once given was a minimum count of logged errors — and errors are what a system produces when it malfunctions, so a bad run scored better than a good one. It was caught before the pilot began.§12
Then the instrument ships before the promise: figures for a proof of concept are labelled expected, and where there are stages to time, the timings come back in every response.§3
I’d rather tell you I don’t yet know the gain and show you how it will be measured, than quote a percentage borrowed from someone else’s benchmark.
A role, rather than a second person. Nobody reviews their own work well while still holding the context that produced it — and that is as true of me as it is of anything I have automated.
An agent in one of my pipelines once shipped two failures it was entirely capable of catching, then found them in detail the moment it was asked to look again from a clean slate.§4 The capability was never missing; the context was. So review here is a separate pass that has not seen the writing: a different day, or a check that never had an opinion in the first place.
And nothing ships on a diagnosis alone: every fix has to answer a failure someone reproduced. A diagnosis read from an exit code alone once named the wrong culprit; the fix went in, and guarded nothing.§11
Yes, because it runs through the same execution path.
There is no separate demo build. A pilot runs the code you would be buying — not a copy tuned to answer demonstration questions well.§6
What a pilot is for is the rest. Three incidents, on different systems, came down to defaults nobody had chosen to inherit: a scanner defaulting to silence, a credential defaulting to the more expensive one.§14 The pilot exists to surface those while they are still cheap.
You keep the decisions, the rules, and the reasons behind them — handed over, not left behind.
Two hours of debugging once produced thirty lines of configuration and seven written rules. The configuration fixed the failure in front of me. The rules are what stopped the same family coming back.§5
What stays written is what no check can hold: what was decided, and what was deliberately left out — documented as a decision with a condition for revisiting it, rather than discovered later as a gap.§13
Method note — working principles
Fourteen so far. It has been shorter, and it will be longer.
§1–§6 — on the wall
-
The most expensive moment I have had on a production system was not an error message. It was a fluent, well-structured, entirely fabricated diagnosis, delivered in exactly the register of the correct ones.
Automated systems produce their wrong answers in the same voice as their right ones, which is why fluency cannot be counted as evidence either way.
-
A system that cannot say there is nothing here will eventually invent something. On a document pipeline that means an empty result is treated as a valid result: the section is skipped, the event is logged at INFO, the run continues. It is written and it is tested.
-
Ship the instrument before the promise. Performance figures for a proof of concept are labelled expected, and per-stage timings are returned in every API response.
I would rather tell you I don’t yet know the accuracy gain and show you how it will be measured, than quote a percentage borrowed from a public benchmark.
-
An agent in a pipeline I run shipped code containing two failures it was entirely capable of finding — and did find, in detail, the moment someone else pointed at them. The capability was never in question; the context was.
Authors read their own work for confirmation, reviewers read it for holes, and what separates the two is not which agent it is but a review seat that never saw the writing.
-
About two hours of debugging produced some thirty lines of configuration — and seven written rules that have stopped the same family of failure from recurring since. The debugging was the cost; the rules were the only thing that made it an investment.
The window for writing a lesson down is minutes wide, and it closes.
-
No intermediate service was written for the API; it reuses the same orchestrator as the command line, and a first version that duplicated the logic was thrown away. There is no showroom build tuned to answer demonstration questions well. What you test during a pilot is what you buy.
§7–§14 — what makes them true
-
Chunk identifiers are generated in Python, never by the model, in a deterministic format. Query filters are typed and validated rather than left as a free-form dictionary.
The model proposes meaning; the application keeps the reference honest — a citation lands on a paragraph that exists, because that link was never entrusted to text generation.
-
Secrets are typed so they are masked everywhere, error traces included, and exceptions coming from third-party libraries are rewritten before they surface.
Where the documents are not public, the question that matters is not whether it is encrypted but what is visible on the day the system falls over. (Scope: this covers secret handling, not access control or corpus isolation.)
-
When something is visibly wrong on screen and the technical analysis answers everything is normal, the conclusion to favour is not that I am misreading the screen — it is that I misunderstood what we were trying to obtain.
The code is faithfully executing a false intention. A diagnosis validates execution, never the objective.
-
The instinct after a failed attempt is to repeat it, more elaborately. Once the attempts stop producing new information, that instinct is a trap: the complexity being added mostly serves to avoid abandoning a hypothesis one has grown attached to.
Perseverance and stubbornness feel identical from the inside; what separates them is whether the last attempt taught you anything, not whether the next one might.
-
An automated agent diagnosed a failure from its exit code alone, named a plausible culprit, and wrote a fix into a configuration file.
The culprit did not exist; the real cause was three lines further down a log nobody had opened. The fix stayed in the repository, protecting against nothing.
Every fix I ship points at something someone actually saw.
-
An acceptance criterion I was once given was a minimum count of logged errors. Those errors are produced by the system malfunctioning, so a bad run scored better than a good one. It was caught before the pilot began.
Success criteria are stress-tested against the incentive they create before anyone starts measuring against them.
-
A drift named and corrected mid-project: features were being ordered by how easy they were to build rather than by what breaks without them.
Effort belongs in the plan; it has no business in the ranking. What gets left out is documented as a decision with a condition for revisiting it, rather than discovered later as a gap.
-
Three separate incidents, on different systems, reduce to the same shape: a scanner defaulting to silence, a credential defaulting to the more expensive one, a container defaulting to a single usable command.
None of them was a bug. Each was a documented default that nobody had chosen to inherit.
Bring me the one that didn’t make it to production.
Or the one that has been six weeks from done,
for a year.
What happens next is a conversation.