Everything before “let’s build it”.
Everything long after “it’s live”.
The model alone rarely solves a real problem. Most of the work is what surrounds it: retrieval, evaluation, the human in the loop, and what it costs to keep running.
I build systems that answer from a company’s own documents, and systems that carry out multi-step work on their own. RAG and agentic systems, in the trade. I take both as far as production. But much of the work happens upstream of that certainty: whether the thing should exist, what would prove it works, which parts will have to be thrown away. I’ve been shipping backend and LLM systems since 2021 (first as an employee, now on my own) and there is no team behind this page: you get me, and the work I sign.
Where I come in
Three stages, and you can enter at any of them. Nothing here assumes the one before it. A scoping note that ends in “don’t build this” is a finished job, and a system already in production is a normal place to start.
Before anyone commits
What is feasible, at what cost, and what should not be built at all.
-
Scoping note
Feasibility before commitment
Three questions first: who does things differently once this exists, what breaks today and at what cost, what nobody would give up. Sometimes the answers end it, which costs less now than eighteen months in.
-
Acceptance criteria
An intention, turned into something you can measure
Every project starts as a sentence, not a specification. “We’d like AI on our documents” becomes a criterion you can measure, stress-tested against the incentive it creates before anyone measures against it.
-
Prototype · disposable
Prototypes that settle something
A prototype whose job is to answer one question, written down before anything is built. Not a demo that impresses: a device that makes a decision possible, and gets thrown away once it’s made.
Into production
Deployed, and then kept running, not handed over at the demo.
-
Running system
Building it, and putting it in production
Document retrieval and agentic systems, from the first version that works to the one that survives real use. Mostly Python, on infrastructure you own and I operate.
-
Evaluation harness
Evaluation before anyone trusts it
A test set built from the questions people actually ask, scored the same way every time, so a change can be shown to help. Without one, every release is an opinion.
Still in use a year on
The half of the work that decides it.
-
Ongoing operation
Keeping it running, over time
The unglamorous half, and the rarest: keeping real-world data usable as it keeps arriving, operating the infrastructure, and knowing why it stops before anyone has to ask. I stay reachable for the systems I deliver.
-
Cost and quality envelope
Costs and quality that stay predictable
What it costs to run each month, visible before it lands on an invoice, and what that figure does when usage doubles. Answer quality is tracked against the same test set as on day one, so drift shows up before someone has to report it.
Reading the near horizon
Not predicting — arbitrating. What is stable enough to build on now, what will move in the next twelve to eighteen months, and which parts of the architecture to keep interchangeable so that movement costs a swap rather than a rewrite. It ends in a decision you can act on, never in a trend.
How I think
Every answer here comes out of a note I have kept alongside the work for years, across the systems I have scoped, run or deployed, added to one failure at a time. Call it a list of things I would rather not learn twice. Open a question, then click a §-number to read the note behind it.
Because a general assistant is optimised to produce an answer, not to prove that one exists in your documents.§1
What you need instead is a system that is allowed to return nothing, and that says so where you can see it rather than in a log. That behaviour is written and tested, not hoped for.§2
And every citation is checked against the source before you see it, not taken on the model’s word. Without that check, a citation is just a plausible-looking number.§7
By asking what the answer would change, before asking whether it can be built.
A scoping note starts from the decision the system is supposed to make possible. If nobody can name it, the note ends there, and that is a finished job.
I was once asked to scope an A/B test on an opt-in feature. Opt-in meant too few users would ever reach it for the two variants to be told apart. The note said so, and the feature was dropped before a line was written.
When the answer is yes, the note becomes a plan we could only have written together: what would prove it works before anyone commits,§3 what it costs to run, and which parts will move before they are paid off.
Take the common case, a system answering over your own documents. Ideally it says nothing rather than guessing, and says so where you can see it: an unsupported question comes back as an explicit not found in this corpus, in the answer rather than in a log.§2
When it does answer and the answer is wrong, the citation takes you to the exact paragraph, so you can see where the reasoning left the source.§7 That traceability is the difference between a bug and a liability.
Whatever the system is, there is a third case: the day it breaks badly. What matters then is what leaks. Secrets are typed so they are masked everywhere, error traces included.§8
We agree on what would count as failure before anyone starts measuring.§12
An acceptance criterion I was once given was a minimum count of logged errors. Errors are what a system produces when it malfunctions, so a bad run would have scored better than a good one. It was caught before the pilot began, and the criterion that replaced it is the one the pilot was judged against.
From there, the measurement exists before the result does.§3 Figures on a proof of concept are labelled expected until a measurement replaces them, and I will tell you I don’t yet know the gain rather than quote a percentage from someone else’s benchmark.
A review seat, which does not have to be a second person. Nobody reviews their own work well while still holding the context that produced it, and that is as true of me as of anything I have automated.§4
An agent in one of my pipelines once shipped two failures it was entirely capable of catching. Asked to review the same code from a clean context, it found both, in detail. The capability was never missing; the context was. So review here is a separate pass that has not seen the writing: a different day, or a check that never had an opinion in the first place.
And nothing ships on a diagnosis alone. Every fix answers a failure someone has reproduced.§11
Two different things get shown before a build, and they are named apart. A prototype answers one question and is thrown away. A pilot is the first version of the system itself, run on a smaller scope: same code path, smaller corpus, fewer users. There is no build maintained for demonstration purposes.§6 What changes between pilot and production is scope, hardening and data, and that is written down before the pilot starts.
What a pilot is for is the rest. Three incidents, on different systems, came down to defaults nobody had chosen to inherit: a scanner defaulting to silence, a credential defaulting to the more expensive one, a container defaulting to a single usable command. The pilot exists to surface those while they are still cheap.§14
What stays with you is written down: the decisions, the rules, and the reasons behind them.
Two hours of debugging once produced thirty lines of configuration and seven written rules. The configuration fixed the failure in front of me. The rules are what stopped the same family coming back.§5
What stays written is what no check can hold: what was decided, and what was deliberately left out, recorded as a decision with a condition for revisiting it.§5 That is what you receive when the mission ends, with the documentation that produced it.
If it is stuck when I arrive, I start from what it was meant to do, and only then from what it does.
A project six weeks from done for a year may be executing, faithfully, an intention nobody has reread. So the brief comes first, in the words of whoever wrote it.§9
Then the attempts. When the same hypothesis has been retried without producing anything new, the useful next step is data, and it may point elsewhere.§10
Then the fixes already in place. An agent once read a failure from its exit code, named a plausible culprit and wrote a fix into a configuration file. The culprit did not exist; the real cause was three lines further down a log nobody had opened. A fix like that is worth a second look.§11
Method note — working principles
Fourteen so far. It has been shorter, and it will be longer.
§1–§6 — on the wall
-
The most expensive moment I have had on a production system was not an error message. It was a fluent, well-structured, entirely fabricated diagnosis, delivered in exactly the register of the correct ones.
Automated systems produce their wrong answers in the same voice as their right ones, which is why fluency cannot be counted as evidence either way.
-
A system that cannot say there is nothing here will eventually invent something.
So an empty result is a valid result, and it is handled at two levels. In the pipeline, the section is skipped, the event is logged at INFO, and the run continues instead of failing. In the answer, the absence is stated where the content would have been, so the reader sees it without opening anything.
Both behaviours are written, and both are tested.
-
The instrument ships before the promise.
On a proof of concept, every performance figure carries the label expected until a measurement replaces it.
On my platform, per-stage timings are returned in every API response, so a slow answer can be attributed to a stage rather than to the model.
The test set used on day one is the one every later version is scored against. A gain is a comparison, never a claim.
-
Authors read their own work for confirmation; reviewers read it for holes. What separates the two is the context they hold, and that holds for an agent as it holds for me.
In my pipelines the review seat is a pass that never saw the writing: a fresh session given the diff and the acceptance criteria, and none of the reasoning that produced them. A process still holding the author’s intent cannot fill that seat.
-
The window for writing a lesson down is minutes wide, and it closes.
So a debugging session ends with a rule, or with the written note that none was found. Rules live next to the code, in a file read at the start of every session, and the debugging that produced them is the only thing that paid for them.
Decisions to leave something out are written the same way, with the condition under which they should be revisited.
-
A build kept for demonstrations answers demonstration questions well.
On my own retrieval platform, the API and the command line call the same orchestrator, the component that runs a question through retrieval and generation; a first version that duplicated that logic for the API was thrown away. What a pilot exercises is the path that stays.
§7–§14 — what makes them true
-
Chunk identifiers are generated in Python, never by the model, in a deterministic format. Query filters are typed and validated rather than left as a free-form dictionary.
The model proposes meaning; the application keeps the reference honest — a citation lands on a paragraph that exists, because that link was never entrusted to text generation.
-
Secrets are typed, so they are masked wherever they appear, error traces included.
Exceptions raised by third-party libraries are rewritten before they surface, because a vendor’s error message is where a key usually ends up in a log.
This note covers secret handling. Access control and corpus isolation are separate questions, handled separately.
-
A diagnosis can tell you the code did what it was asked. It cannot tell you the ask was right.
So when the screen and the analysis disagree, the first thing reread is the brief: what was asked, and whether the system was built to that or to my reading of it. The code is checked second.
-
Perseverance and stubbornness feel identical from the inside. What separates them is whether the last attempt taught you anything.
The threshold I use: once an attempt on the same hypothesis has taught nothing new, the next step is data, and it may end in a different hypothesis. The complexity that would have gone into another attempt is the cost of that hypothesis, and it is paid before the hypothesis has earned it.
-
A diagnosis that has not read the failure is a guess in the shape of an explanation.
So every fix is tied to evidence before it is merged: the reproduced failure, the log line, the failing test. When the failure cannot be reproduced, that is recorded as the finding, and the fix waits.
-
A success criterion is a target, and a target changes what gets optimised.
So before a criterion is adopted, it is tested against the behaviour it rewards. The question is simple: could a worse system score better on it? If it could, the criterion is rejected.
The test is written next to the criterion, so the next person can rerun it.
-
A drift named and corrected mid-project: features were being ordered by how easy they were to build rather than by what breaks without them.
Effort belongs in the plan. It has no place in the ranking.
-
A default is a decision the software makes when nobody makes one: how long to wait, how many times to retry, which model version to load, what to do with an empty result. Each one is documented somewhere, and each one is inherited by a system that never chose it.
The incidents I have traced back to defaults were never bugs. The software did exactly what its documentation said; nobody had read that page. So the defaults a system inherits are among the first things I read, while changing one is still cheap.
Bring me the one that didn’t make it to production.
Or the one that has been six weeks from done,
for a year.
What happens next is a conversation.