Measuring AI Output: Evals Instead of Gut Feeling

The prototype runs, the demo impresses – and then comes the question that derails most AI projects: how do we actually know the thing answers correctly? Without a measurable answer, you end up negotiating about taste instead of quality.

1. The Question That Follows the Prototype

There is a moment in every AI project that repeats itself with total reliability. The prototype runs. The demo in the meeting works. Someone asks three questions, the system answers fluently and plausibly, heads nod around the table. And then someone from the business side says the one sentence that changes the mood: "And how do we know it is always right?"

That question is entirely legitimate, and it marks the actual start of the project. Because a demo proves feasibility, not reliability. It shows that the system produced a good answer to three selected questions. About the four-hundredth question from a caseworker on a Tuesday afternoon it says nothing at all.

The distinction is fundamental, and it has a technical cause. Conventional software is deterministic: same input, same output, every time. A language model is not. It produces probabilities, not certainties. The same question can yield two phrasings, one of which is correct while the other is subtly off. You can work with that – but not with methods designed for deterministic systems.

Whoever leaves that gap open pays for it later, usually in one of three ways:

The good news: the problem is solved, the tools exist, and they are not particularly exotic. They are called evals.

2. What an Eval Is – and How It Differs From a Unit Test

An Eval is a reproducible test run that scores the output of an AI system against a fixed set of inputs and vetted reference answers. You feed the system a hundred or a thousand real queries, compare the answers against what would have been correct, and end up with a number.

That number is the decisive point. An eval does not say "correct" or "wrong". It says: sufficient in 94 out of 100 cases, incomplete in 4, factually wrong in 2. That is workable – you can compare it against last week's value, set a threshold, and discuss improvements without discussing impressions.

From this difference follows the most important organisational consequence: an eval needs a data foundation created by someone with domain expertise. That foundation is called a Golden Dataset – a curated, versioned set of real queries and domain-approved reference answers.

This is where the real work sits in practice. Writing the eval code is not the expensive part – those are modest scripts. What is expensive is collecting fifty to two hundred real cases and having them reviewed by someone who commands the subject matter. That effort should not be hidden but stated openly as a project line item. It is the investment that makes everything downstream measurable in the first place.

3. RAG Evaluation: Measuring Retrieval and Generation Separately

Most production AI systems in an enterprise context are not plain language models but RAG systems: they first retrieve matching documents from a knowledge base and then let the model compose the answer from those documents. How such a system is built is covered in detail in our guide to local enterprise RAG with pgvector.

For quality assurance, the decisive fact is that a RAG system has two independent failure modes. And scoring only the final answer cannot tell them apart.

A RAG system can answer wrongly despite flawless generation – because retrieval missed. And it can find the right source and still produce nonsense from it. Measure only the end and you will fix the wrong half.

Evaluation therefore needs two separate measurement points:

This separation has a very practical payoff: it tells you where to intervene. Low recall is a data problem – the Chunking strategy does not fit, the documents are poorly structured, or a Reranking step to sort the preselection is missing. Tweaking the prompt helps not at all in that case. Conversely, a poor answer on a perfectly retrieved document is a prompt or model problem, and rebuilding the knowledge base would be wasted time.

The typical metrics at a glance:

4. Hallucination Control: Groundedness, Guardrails, Human-in-the-Loop

The term Hallucination is mostly used too loosely. Quality assurance needs a sharper distinction, because the different failure types call for different countermeasures.

The practically most important dividing line runs between factually wrong and not grounded. An answer can be entirely correct and still constitute a problem – namely when the model supplied the information from its training knowledge instead of drawing it from the provided company documents. In case of doubt, general internet knowledge then sits inside an answer that looks as though it came from the company's own quality manual.

That is precisely what Groundedness measures: for each individual statement in the answer, it checks whether that statement is evidenced in the supplied context. The underlying procedure is mechanical – the answer is decomposed into individual claims, each claim is held against the context, and the results are aggregated.

Groundedness cannot sensibly be assessed with string comparison – language is far too variable for that. The established route is LLM-as-a-Judge: a second model receives context, answer, and a precise list of criteria, and grades.

The method scales beautifully and has one weakness that has to be said out loud: the judge is itself a language model. It can share the systematic biases of the system under test, it tends towards leniency with verbose answers, and it measurably favours text whose style resembles its own output. An uncalibrated judge produces numbers that look trustworthy and are not.

Beyond measurement, production operation needs two further layers. Guardrails act at runtime: they inspect the answer before the user sees it and block or escalate on insufficient groundedness, missing source references, or topics outside the permitted scope. And Human-in-the-Loop remains mandatory wherever a wrong answer causes real harm – legal advice, medical statements, or binding price commitments. That verification is the actual bottleneck of autonomous systems holds for knowledge agents just as it does for the coding agents in software development.

5. Regression Tests for Prompts: Golden Dataset and CI

Prompts are source code. They govern behaviour, they have versions, they break on change. In practice, however, they are frequently treated like sticky notes: embedded somewhere in the code, without history, edited on a colleague's say-so in chat.

This is dangerous because prompt changes have non-local effects. You add a sentence to correct one specific behaviour – and thereby alter the answers to every other query as well. In conventional code, a change has a bounded blast radius that you can read. In a prompt, that boundary does not exist.

The same applies to model upgrades. Moving to a newer model version is not a pure improvement but a behavioural change: some cases get better, some get worse, and without measurement you do not know which. That is exactly what the Regression Test exists for.

Setting it up is unspectacular and takes a few days:

1. Extract Prompts From the Code

   Every prompt gets its own versioned file with a clear identifier. That makes each change visible in the diff and lets you trace which version was in production when something broke. It is the precondition for everything that follows – without it, a behavioural break cannot be attributed.

2. Create and Version the Golden Dataset

   The test set lives as a structured file in the repository next to the code, not in a spreadsheet on a network drive. Each entry holds the query, the expected answer, the expected source document, and which criteria apply to that case. Edge cases where a refusal is the right answer belong in it explicitly.

3. Automate the Eval Run as a Script

   One command plays the entire test set against the system, computes the metrics from chapter 3, and writes the result as a machine-readable report. Keeping the run reproducible matters: pinned model version, fixed parameters, logged configuration. Otherwise you will later compare numbers that are not comparable.

4. Define Thresholds and Anchor Them in CI

   The eval run executes automatically on every change to prompt, model, or knowledge base. If a score falls below the agreed threshold, the change is not merged. Quality then stops being a matter of discipline and becomes a technical condition – the same shift unit tests brought to conventional software twenty years ago.

5. Feed Production Failures Back Into the Test Set

   Every legitimate complaint from operation becomes a new case in the golden dataset, with the correct answer beside it. The yardstick grows with the system, and the same failure cannot slip through twice. This feedback channel is the difference between a test set that ages and one that improves.

A note on cost, because it is routinely missing from proposals: an eval run over two hundred cases with LLM-as-a-Judge incurs real API charges, on every execution. At today's prices that is no reason to abstain, but it belongs in the calculation – especially when the run is triggered on every commit. A common split is a small, fast set on every change plus the full run once daily or before each release. How such recurring items fit into an honest business case is covered in our article on the measurable ROI of AI projects.

6. From Eval to Acceptance Criterion in a Fixed-Price Proposal

Now to the commercial part, which is what turns all of the above into an advantage. A fixed price only works if both sides know in advance when the work is done. With conventional software that is uncontroversial: the function exists or it does not. With an AI system whose quality is gradual, that point is missing – unless you create it.

That is exactly what evals deliver. They convert a matter of taste into a number, and numbers can be written into a contract. "The answers should be good" becomes a testable Acceptance Criterion.

It is not the eval score that protects the fixed price, but the fact that both parties defined it together before work began. A test set created at acceptance time is a weapon – one that stands at the beginning is a contract.

In practice this means the golden dataset becomes the first project phase, not the last. Client and supplier sit down together, collect real queries, formulate the expected answers, and agree on the thresholds. That takes a day to a week depending on the domain – and it is the single most valuable meeting in the entire project. Because it regularly emerges that the business unit itself disagrees on what the correct answer would be. Having that insight before building is worth a great deal.

Four points belong in a defensible acceptance clause:

The last point deserves emphasis, because it is the most common source of conflict. The quality of a RAG system depends directly on the quality of the documents supplied to it. If those are contradictory, outdated, or unstructured, the score drops – without the supplier having done anything wrong. A clean clause therefore separates the areas of responsibility and makes the data foundation a duty of cooperation on the client's side.

Conclusion

The question that follows the prototype – how do we know it is right? – is not a vote of no confidence but the transition from experiment to product. Answer it with another demo and you have merely postponed the problem. Answer it with a score on a jointly defined test set and you have put the project on a foundation that survives changes, model upgrades, and years of operation.

The effort involved is modest and sits almost entirely in the domain groundwork: collect real cases, fix the correct answers, agree the thresholds. The eval code itself is an afternoon. What that groundwork changes, though, is the nature of the project. An open-ended engagement becomes a deliverable with a defined acceptance – and that is precisely what makes a fixed price fair for both sides.

An AI system without evals is not a finished product. It is a demo running in production.

Our Regional Expertise

We are your digital partner – regionally anchored and successfully scaling across borders.

Start a project request in your region →

Automation BerlinAutomation HamburgAutomation MünchenAutomation KölnAutomation Frankfurt am MainAutomation StuttgartAutomation DüsseldorfAutomation LeipzigAutomation DortmundAutomation EssenAutomation BremenAutomation HannoverAutomation NürnbergAutomation WienAutomation GrazAutomation LinzAutomation SalzburgAutomation ZürichAutomation GenfAutomation BaselAutomation Bern

Extended Specialized Glossary