An evaluation harness for Claude in regulated environments

In a regulated enterprise, the question that decides whether an AI system ever reaches production is not "is it good?" It is "how would you know if it got worse?" Teams that cannot answer that question convincingly end up in a slow loop of manual review, where every prompt change requires a committee, and the system ossifies within a quarter of launch.

An evaluation harness is the answer, and it is not a research artifact. It is a piece of production infrastructure: a versioned dataset, a set of graders, a runner in CI, and a release gate. It is also, in our experience, the single deliverable that most changes the tone of a model risk conversation — because it converts a debate about model behavior into a table of numbers with a trend line.

Start with the golden dataset, and build it from your own history rather than from public benchmarks. For a claims summarization workload, that means real claim files with the summary an experienced adjuster actually wrote. For a KYC review assistant, real cases with the analyst's disposition and reasoning. You need fewer examples than people expect — 150 to 300 well-chosen cases beat 5,000 scraped ones — but they have to span the distribution honestly.

Stratify deliberately. We build every dataset from four buckets: routine cases that represent the bulk of volume, hard cases where experienced humans disagreed, edge cases that broke the system before, and adversarial cases constructed to probe a specific failure. The last two buckets are small and disproportionately valuable, because they are where regressions show up first. Every production incident becomes a new adversarial case; that rule alone compounds the suite's value over a year.

Freeze and version the dataset like code. It lives in the repository — or in object storage with a content hash referenced from the repository — with a changelog. When a score moves, you need to know whether the model changed or the ruler did. Teams that edit their eval set casually lose the ability to compare across quarters, which is precisely the comparison a regulator asks for.

Then pick graders that match the question. Three kinds cover almost everything. Deterministic checks handle anything with a right answer: extracted fields, valid dates, schema conformance, required citations present, prohibited content absent. These are cheap, exact, and should carry as much of the load as possible. Push work into this category wherever you can.

Rubric grading with a model handles the qualitative dimensions: is the summary faithful to the source, is the tone appropriate for a customer, does the recommendation follow the policy. The rubric must be written down, specific, and scored on a small integer scale with anchored descriptions for each point. "Rate the quality from 1 to 10" produces noise. "Score 0 if any claim in the summary is unsupported by the source; 1 if all claims are supported but material facts are omitted; 2 if all claims are supported and the material facts are present" produces a number you can act on.

const FAITHFULNESS_RUBRIC = `
You are grading a claim summary against its source documents.
Score strictly. Output JSON only.

0 — Contains at least one statement not supported by the source.
1 — All statements supported, but omits a material fact
    (loss date, coverage decision, reserve change, injury).
2 — All statements supported and all material facts present.

Return: {"score": 0|1|2, "unsupported": string[], "missing": string[]}
`;

export async function gradeFaithfulness(source: string, summary: string) {
  const res = await claude.messages.create({
    model: "claude-opus-4-6",         // grade with a stronger model than you ship
    temperature: 0,
    max_tokens: 512,
    system: FAITHFULNESS_RUBRIC,
    messages: [
      { role: "user", content: `<source>${source}</source>\n<summary>${summary}</summary>` },
    ],
  });
  return JSON.parse(extractJson(res));
}

Two practical rules for model-graded evaluation. Grade with a stronger and different model than the one you are shipping, at temperature zero, and validate the grader itself against human labels before you trust it — we require the grader to agree with two human reviewers on at least 90 percent of a 100-case calibration set, and we re-run that calibration whenever the grader model version changes. A grader nobody has calibrated is an opinion with a decimal point.

The third grader is human, and it should be small and targeted. Sample twenty cases per release, weighted toward the ones where the rubric grader was least confident or where the automated and deterministic signals disagreed. This is the sample that keeps the automated graders honest, and it is the evidence a review committee actually reads.

Wire the whole thing into CI and gate on it. Any change to a prompt, a model version, a retrieval configuration, a tool schema, or a chunking strategy triggers a full run. The gate is not a single score — it is a set of thresholds, and at least one of them should be a floor on the worst stratum rather than an average. A system whose mean improves while its adversarial bucket collapses has gotten more dangerous, not better.

gates:
  faithfulness_mean:        { min: 1.85 }
  faithfulness_adversarial: { min: 1.60 }   # worst stratum floor
  citation_present_rate:    { min: 0.99 }
  pii_leak_rate:            { max: 0.0 }    # zero tolerance, deterministic
  p95_latency_ms:           { max: 6000 }
  cost_per_case_usd:        { max: 0.14 }

Report cost and latency alongside quality in the same table. In regulated environments the trade-off conversation is unavoidable, and having it with numbers in front of everyone is much better than having it after the invoice arrives. We have shipped a slightly weaker model into production more than once because the harness showed the quality difference was inside the noise band and the cost difference was not.

Offline evaluation tells you about the cases you thought of. Production drift monitoring tells you about the ones you did not. The bridge between them is a sampled online evaluation: take a small percentage of live traffic, run the same rubric graders asynchronously, and chart the scores against the offline baseline. When the live score diverges from the offline score, either your input distribution has moved or something upstream has changed. Both are worth a page.

Watch the input distribution directly too. Track document length, language mix, retrieval hit rate, and the rate at which the model declines or escalates. In practice, the declination and escalation rates are the most sensitive early indicator we have found — they move before quality scores do, because the model notices unfamiliar inputs before the graders do.

For the model risk committee, the artifact that works is short and standard: what the system does and does not decide, the dataset and how it was constructed, the graders and their calibration against humans, the current scores with a twelve-month trend, the release gates and who can override them, the incident log with the eval case each incident produced, and the rollback procedure. Six pages, refreshed quarterly. We have never had that package rejected; we have frequently seen a fifty-page model card rejected.

One structural point worth insisting on. The team that builds the system should not be the only team that can run the harness. Put the runner behind a command any reviewer can execute, keep the results in a place risk and compliance can read without asking, and let them add cases. The moment the second line of defense can independently verify a claim about the system, the approval conversation stops being adversarial.

Effort, honestly: building the first harness for a workload takes two to three weeks alongside the build, and roughly half of that is arguing productively about what "correct" means. That argument is the valuable part. Every regulated AI system we have shipped that stayed shippable had one of these; every one that stalled did not.