Kape Tools

LLM Ops: Keeping It Working

Building it is day one. This is every day after.

LLM Ops is the loop you run after your AI system is live: record what happened, check if it was good, work out why it broke, fix it, and ship the fix safely.

Why this exists

An AI system is not finished when it works once. The questions people ask change. Documents go stale. Someone edits a prompt. A tool starts failing quietly. Nothing announces itself.

LLM Ops is the routine that catches all of that. It is unglamorous and it is the difference between a demo and something people can rely on.

The loop

A cycle: trace one run, evaluate whether it was good, diagnose why it broke, pass a gate, release the fix, and feed the improved settings back into the system
Five steps, then back to the start. The gate is the step people skip.Open full size
  1. Trace. Record every run. One record per run, so nothing is invisible later.
  2. Evaluate. Score the answers. Was it actually good, or did it just sound good?
  3. Diagnose. When a score drops, find out where and why. Was it the search, a tool, or the instructions?
  4. Gate. Try the fix against your saved test set. If the score does not improve, it does not ship. This is the step people skip.
  5. Release. Ship the change: a new prompt version, a setting, a tool fix. Write down what you changed so you can undo it.

Then it starts again, because the fix you just shipped is now the thing being watched.

What actually gets changed

The three questions this answers

QuestionThe step that answers it
What happened?Trace
Was it any good?Eval
Did my fix actually help?Gate

The freelancer version

You do not need software for this. If you run a prompt for client work, the whole loop fits in a document:

One markdown file holding the last runs, five test cases, every complaint added as a case, the before and after check, and the old prompt with a date
Five sections in one file. That is the whole loop, at the size most people need.Open full size
  1. Keep the last few real inputs and outputs. That is your trace.
  2. Keep five real examples with what a good answer looks like. That is your eval.
  3. When a client complains, add their case to the five.
  4. Before you change the prompt, run the five. After you change it, run them again.
  5. Keep the old prompt in the same file with a date, so you can go back.

That is real LLM Ops. The tools people pay for do the same five things at a scale a document cannot handle.

The mistake that costs most

Editing a live prompt directly, with no copy of the old one and no test. It feels fast. Then something is worse a week later, nobody remembers what changed, and there is nothing to roll back to. Keep the old version. Always.

More words

See all 11 explanations