Evals Everyone Can Read

Project Overview

The Challenge

Governance in a regulated organisation is a chain of decisions, a deep log of processes, and each link belongs to someone with a different question. Remember, this was early 2025, governance standards for AI were evolving rapidly, and there were no strong playbooks for preparedness.

The development of early vibe coding tooling and code-assistants were increasing the creation of internal applications by 50%, so a governance process that could scale and also deliver innovation and rolled out usage was also integral.

• Model ownerWhat do I have to do before my use case can launch?
• Oversight teamHow risky is this use case?
• AI councilWhat has this use case been through before it reached me?
• AI engineerWhich metric failed, and why?

Each of them was working from a different tool, system or platform to organise growing AI use cases. None of the roles could follow a use case from idea to production, and the process couldn't stretch to match the rate at which new AI use cases were arriving.

The brief was to show that one platform could represent the existing process faithfully, govern every kind of AI the organisation used (built in house, bought, or built on top of a vendor's model), and give every stakeholder the same view of it.

Eight Roles, One Lifecycle

We started with workshops alongside the client's risk, controls and model monitoring teams. Instead of a feature list, we agreed what success would look like, written as opportunity statements: if we provide this, we'll know we've arrived when that happens. I wrote the user stories each sprint was measured against from those sessions.

From the workshops I built eight personas across the two workstreams. Each demo in the playback was told through them, so a risk director and an AI engineer followed the same story.

“I want a step-by-step process for risk and controls inside the platform, so I know my use case meets compliance requirements before it's deployed.”

User story for the Model Owner, written from the discovery workshops

Designing the Loop ∞

The journey became four stages. Each is owned by a named role and ends in something the next person can pick up.

1. 01

   Intake that sets the risk tier

   MModel OwnerOOversight Team

   The organisation's pre-screen questionnaire became a structured flow inside the platform. A model owner completes it when registering a use case, and the answers set the risk tier using the organisation's own grading framework. Putting it first meant every use case arrived with a tier attached, instead of being graded later from a separate document.

2. 02

   Approval that follows the tier

   OOversight TeamAAI Council

   Different tiers need different sign-off and have different levels of risk. Higher-risk use cases route to the AI council, and the workflow can't progress until a data protection impact assessment from an external system is attached. Comments and decisions stay on the use case, so the council sees its full history in one place.

3. 03

   Evaluation matched to the use case

   SAI EngineerMQA Analyst

   A summariser, a retrieval assistant and a code generator fail in different ways, so each type got its own monitor with metrics picked for it, instead of one generic quality score. When the business asked for a new type of use case, an engineer could add custom metrics to an existing monitor.

4. 04

   From breach to issue

   MAI Ops LeadMModel Owner

   The monitoring view answers one question before any other: which metrics are in breach? From a breach, an engineer or ops lead raises an issue with a priority and a type, and it routes back to the model owner who is accountable for the fix.
   
   Here's a look at some of the initial metrics from our RAG use case.

   All metrics by categoryRetrieval assistant (RAG)

   Breach 40000070 Warning 20100030 Passing 1232425133 Not run 1000000170 QualityFairnessDriftPerfor­manceExplain­abilityModel healthRetrievalSecurity

   Category and severity come first, so people see what is wrong before they dig into why.

Making Evals Legible

Most people in this loop would never open an evaluation notebook. For every metric I asked two things: what question does this answer, and who is asking it? Then I worked across the engineers, QA and risk teams until each answer made sense to all of them.

</> Code generation: three questions, three graders

We catalogued eight candidate metrics for generated code, from parse checks to prompt repetition. For the demo we narrowed them to three, and I wrote each as the question a reviewer would actually ask.

• Is it valid code for the language?Parsable
• Is it maintainable, or too verbose and complex?Cognitive complexity
• How similar is it to a known-good answer?Similarity to reference (CodeBERT)

Then we ran them against pairs of good and bad code, so the audience could see what each grader catches and what it misses. The simplified Python examples below illustrate validity, complexity and output differences. The verdicts are illustrative, not measured on the actual CodeBERT scores that we used.

def add_numbers(a, b):
    return a + b

def add_numbers(a, b):
    return a - b

def sort_list(lst):
    return sorted(lst)

def sort_list(nums):
  for i in range(len(nums)):
    for j in range(i+1, len(nums)):
      if nums[i] > nums[j]:
        nums[i], nums[j] = (
            nums[j], nums[i]
        )
  return nums

No single grader caught every failure. The useful unit is a panel of graders, each with a failure it is known to catch, and a view that shows which one fired.

Agents: a second model as the judge (LLM-as-a-judge)

For retrieval assistants and agents, we used a second model which acted as the judge. It scores the relationships between the question, what the system retrieved and the answer it gave, with similarity to a ground-truth answer added where one exists. I paired each judge with a worked example, so people saw a failure before they saw a score.

Why It Had to Be Automated

The organisation planned to add new AI use cases every month, and every one produces output that someone has to review. If people review everything, the headcount needed grows with each launch. This was 2025 and only the beginning of scaling valuable AI use cases internally.

Automated monitors change that job and graders did the first pass, and reviewers spend their time on high-risk cases instead of reading every output. That became the core of the value case we played back. Where we wanted to catch serious cases with the staff already in place, miss fewer high-risk ones, and launch each new use case sooner.

What I'd Do Differently

This was early 2025 and there wasn't any existing established patterns for evals or AI Governance. I've since spent my time building agents and the evaluations around them, and four things stand out.

1. Start from the rubric, not the metric

   We began with a catalogue of metrics and worked backwards to the questions they answered. In future, I'd run it the other way and get a reviewer writes what good looks like in plain English, and that becomes the grader. Every new metric in our flow still went through an engineer. The people who know what good looks like should be able to write the check themselves.

2. Close the loop

   Our journey ended when an issue was raised. A fix should ship with the grader that proves it worked, and that grader should re-run whenever the model underneath changes. Without that, a model upgrade can quietly undo last month's fix.

3. Put the transcript behind every red warning

   The breach view told people where something failed and it didn't show them what the model said. The finding is usually in the transcript, including the failures no metric was designed to catch.

4. Evaluate the decision as well as the answer

   In the agent example, the failure began when the agent chose web search for a question it could never answer that way. That choice is shaped by a few lines of tool description and toolcalls had only just started to come into the flow.

Impact

What I Took From This

With little wider governance process or framework in place, this project highlighted how important strong evals, a rigid process and metrics were for use cases. If you get that wrong at an organisation of this scale, then you're slowing down your own innovation and internal AI development lifecycle.

The same holds for the prompts behind any AI product. A system prompt, a tool description or a formatting instruction is only as trustworthy as the eval that checks it, and only as improvable as that eval is readable. The evals that matter most are the ones the accountable people can read, interpret, validate and understand.

This project sits between Cora+, where I learned that aggregate scores hide failures for specific groups of users, and Flint, where I built validation into the product itself.

watsonx.governance · LLM-as-judge · RAG evaluation · Code evaluation · Agent evaluation · Risk tiering · Feb 2025