LLM Evals and Grading: Building Production-Grade Evaluation Infrastructure

**What it is:** A curated set of representative inputs + expected-correct outputs for each production workload. The 'ground truth' against which you measure model + prompt changes. Typically 50-500 examples per workload depending on complexity.

**How to build:** Start by sampling 100 real production queries. For each, generate the expected-correct output (manually or via your best current process). Label edge cases, common mistakes, and tricky scenarios explicitly. Maintain in version control like code — golden datasets evolve as workloads evolve.

**Common mistake:** Treating golden datasets as static. Production query distribution shifts; your golden set should be refreshed quarterly to reflect current real-world inputs. Stale golden datasets produce evaluation that doesn't predict production quality.

**Reference:** Stanford's HELM framework at crfm.stanford.edu/helm is the academic gold standard for systematic evaluation methodology; for production-tactical guidance, LangChain's evaluation guide is a starting point.

**What it is:** Multi-dimensional scoring of LLM outputs against explicit criteria. Each dimension scores 1-5; total score is a weighted sum. Captures the texture of 'good' that automated metrics can't.

**Dimensions to include:** Vary by workload. Common: factual accuracy, format adherence, audience-appropriateness, completeness, specificity. For creative outputs: coherence, voice, novelty.

**Who grades:** Three options. (1) Human raters — gold standard but expensive ($5-15 per output rated). (2) LLM-as-judge — cheap and fast but introduces evaluator biases. (3) Hybrid — humans calibrate the LLM-judge on a sample; LLM-judge scales the rest.

**LLM-as-judge research:** Per Zheng et al. 2023 'Judging LLM-as-a-Judge with MT-Bench' (arXiv:2306.05685), GPT-4-class judges agree with human ratings at ~85% rate — comparable to human-human agreement on subjective tasks. Strong frontier-model judges are now the cost-effective default for high-volume eval.

**What it is:** Programmatic checks that don't require LLM-as-judge or human review. Schema validation, exact-match scoring, classification accuracy, BLEU/ROUGE scores for similarity-based tasks.

**Where they apply:** Structured extraction (does the output match the schema? are required fields populated correctly?), classification (precision/recall/F1), code generation (does it compile? do tests pass?), translation/paraphrasing (similarity scores against reference).

**Why they matter:** Free to compute at scale. Use them as the first-line filter — outputs that fail automated metrics never need expensive human/LLM evaluation; outputs that pass move to subjective rubric grading.

**Reference:** Eleuther AI's lm-evaluation-harness at github.com/EleutherAI is the most-used open-source framework for automated LLM benchmark evaluation.

**What it is:** Running two versions of a prompt or two model configurations against the same evaluation set, with statistical comparison of outputs. The mechanism for confidently deploying changes.

**How it works:** For each example in the eval set, generate output from version A and version B (with temperature 0 if you want determinism, otherwise N samples to average). Score both via your rubric or automated metrics. Use a paired statistical test (Wilcoxon signed-rank for ordinal scores, McNemar's for binary correctness) to determine whether the difference is significant.

**Sample size matters:** Per general A/B testing statistics, detecting a 5% quality difference with 95% confidence requires roughly 200-400 examples per variant for typical effect sizes. Smaller eval sets only detect larger differences.

**Reference implementation:** Promptfoo at promptfoo.dev is the most popular open-source LLM A/B testing framework as of 2026. Helicone at helicone.ai and Phoenix at phoenix.arize.com provide production observability + eval infrastructure.

**What it is:** Continuous evaluation of a sample of production outputs to catch quality drift. Eval infrastructure isn't only for pre-deployment testing; it's for production monitoring.

**How it works:** Sample 1-5% of production outputs daily. Run them through the same eval rubric you use for pre-deployment. Track quality scores over time. Alert on degradation.

**What causes drift:** Model provider updates (silent improvements or regressions), input distribution shift (your users' query patterns change), prompt-injection attacks, accumulated edge cases.

**Reference frameworks:** Phoenix open-source at phoenix.arize.com, Helicone at helicone.ai, LangSmith from LangChain — all provide production-grade eval + monitoring infrastructure with different feature trade-offs.

Shipping LLM workloads without systematic eval: vibes-based testing, regressions discovered in customer complaints, no confident model upgrades, prompt iteration based on gut feel.
Production eval stack (golden datasets + rubric + automated metrics + A/B + monitoring): regressions caught pre-deployment, model upgrades confident, prompt changes data-driven, quality drift detected at 1-2% impact instead of 30% impact at the customer-complaint stage.