Evaluating LLMs the way you evaluate engineers

Benchmarks are the phone screen

A benchmark tells you a model can produce a plausible answer under ideal framing. That is the phone screen. It does not tell you whether the model degrades gracefully, admits uncertainty, or holds a constraint across eight turns.

Structure the judgement

Building evaluation pipelines for frontier engagements, the single highest-leverage move was forcing every judgement through a JSON schema:

{ "verdict": "pass" | "fail",
  "criteria": [{ "id": string, "met": boolean, "evidence": string }],
  "confidence": number }

A free-text rating is unfalsifiable. A schema with an evidence field per criterion forces the grader — human or model — to point at the token that justified the score. It also makes disagreement measurable.

Latency is a quality metric

We benchmarked accuracy, latency and robustness on the same axis, because a model that is right in nine seconds is wrong for an interactive product. Treat p95 as a correctness constraint and the architecture conversation changes shape.