Evaluation explained

The demo vs production gap

Every LLM product passes a hallway test: three beautiful answers, everyone applauds. Production bots fail on the fourth ticket—monthly refund edge cases, jailbreaks, missing KB topics, or wrong RAG chunks. Without evals, you learn from angry users, not CI.

Traditional software: same input → same bytes. LLMs sample from a distribution—paraphrase is normal. Evals use scorers—rules, embeddings, judges—that tolerate wording while catching policy violations.

The vibe-check trap

The vibe check is the most common failure mode: after every prompt edit, someone opens a chat UI, asks five questions, squints, declares “looks good.” That workflow is non-reproducible, non-comparable, biased toward recent edits, and invisible in CI.

Escape with a golden dataset: 30–200 real questions as JSONL in git with must_contain, forbid_phrases, refusal flags, citation requirements. Run on every PR. When a case fails, fix the pipeline or update gold deliberately.

flowchart LR
  subgraph trap["Vibe-check trap"]
    E1[Edit prompt] --> M1[Manual chat test]
    M1 --> S1[Ship on feeling]
    S1 --> E1
  end
  subgraph eval["Eval discipline"]
    E2[Edit prompt] --> R2[Run gold JSONL]
    R2 --> C2{Pass rate OK?}
    C2 -->|yes| S2[Merge]
    C2 -->|no| F2[Fix or update gold]
    F2 --> E2
  end

Why eval is harder than building the feature

• Ground truth is fuzzy — many valid answers; scorers encode policy.
• Multi-component failures — bad answers may be retrieval, prompt, model, or tools.
• Eval drift — gold sets stale when policy changes; judges need calibration.
• Cost vs coverage — live CI calls are expensive; mocks miss real failures.
• Agent trajectories — multi-step paths need trajectory assertions, not answer-only checks.

import json
from dataclasses import dataclass

@dataclass
class EvalCase:
    id: str
    question: str
    must_contain: list[str]
    forbid_phrases: list[str]
    expect_refusal: bool = False
    expect_citation: bool = False
    tags: list[str] | None = None

def load_cases(path: str) -> list[EvalCase]:
    cases = []
    with open(path) as f:
        for line in f:
            row = json.loads(line)
            cases.append(EvalCase(**{k: row[k] for k in row if k != "prompt_version"}))
    return cases

public record EvalCase(
    String id,
    String question,
    List mustContain,
    List forbidPhrases,
    boolean expectRefusal,
    boolean expectCitation,
    List tags
) {
  public static EvalCase fromJson(JsonNode n) {
    return new EvalCase(
        n.get("id").asText(),
        n.get("question").asText(),
        jsonStrings(n.get("must_contain")),
        jsonStrings(n.get("forbid_phrases")),
        n.path("expect_refusal").asBoolean(false),
        n.path("expect_citation").asBoolean(false),
        jsonStrings(n.path("tags"))
    );
  }
}

Offline regression vs online monitoring

Offline eval runs against frozen gold before deploy—your safety net in CI. Online eval samples live traffic after deploy—catches distribution shift, new user phrasing, and doc updates not yet re-indexed. You need both; offline alone misses production-only failures, online alone ships broken prompts to the first thousand users.

Track 3 introduced prompt golden inputs; Track 2 introduced recall@k. Track 5 unifies them into a product-level eval strategy—PR smoke, nightly live runs, and human review of production samples. See Reliable output formatting for schema-level unit tests that complement answer-level evals.

Three layers

flowchart TB
  H[Human eval sampling]
  I[Integration pipeline evals]
  U[Unit scorers and schema tests]
  H --> I --> U

Unit layer — scorers and fixtures

Unit tests never call the LLM. They assert scorers behave: given answer text and gold expectations, return pass/fail. Also test JSONL loaders, report writers, and exit-code logic—same as testing pytest helpers.

Integration layer — mock vs live runners

Mock runners return deterministic fake answers—free CI on every PR, catches scorer and wiring regressions.Live runners call real RAG/agents—nightly or pre-release—catch model and retrieval drift mocks hide.

Human layer — when automation stops

Judges and rules miss empathy, subtle policy, and novel failure modes. Sample production traces into a review queue; label pass/fail with reasons; promote failures to gold JSONL. Human eval is not “skip automation”—it feeds the automation.

Smoke vs full suites

Agent evals add a fourth suite type: agent_trajectory.jsonl with expected tool names and order—see Agents explained for why final-answer-only checks miss wrong tool paths.

def score_must_contain(answer: str, phrases: list[str]) -> tuple[bool, str]:
    lower = answer.lower()
    missing = [p for p in phrases if p.lower() not in lower]
    if missing:
        return False, f"missing phrases: {missing}"
    return True, "ok"

def score_forbid(answer: str, forbidden: list[str]) -> tuple[bool, str]:
    hits = [p for p in forbidden if p.lower() in answer.lower()]
    if hits:
        return False, f"forbidden phrases: {hits}"
    return True, "ok"

public final class RuleScorers {
  public static ScoreResult mustContain(String answer, List phrases) {
    String lower = answer.toLowerCase();
    List missing = phrases.stream()
        .filter(p -> !lower.contains(p.toLowerCase()))
        .toList();
    return missing.isEmpty()
        ? ScoreResult.pass("ok")
        : ScoreResult.fail("missing: " + missing);
  }
}

Core metrics for LLM apps

From vibes to numbers — workflow

1. Establish baseline pass rate on main with pinned config.
2. Change one variable (prompt, chunk size, model)—not three at once.
3. Re-run suite; compute delta and per-tag breakdown.
4. Inspect failed case IDs—not only the aggregate score.
5. Accept, revert, or update gold with documented rationale.

Plot pass rate over time in Grafana or your observability stack. A slow drift across prompt edits is invisible in vibes but obvious on a chart.

Leading vs lagging indicators

• Leading — offline pass rate delta, parse success rate, retrieval recall@5 on gold queries (predicts user pain before deploy).
• Lagging — support escalation rate, thumbs-down on bot replies, average handle time (confirms eval gaps after deploy).

When leading and lagging diverge—eval pass rate high but escalations rising—your gold set no longer represents real traffic. Refresh gold from production samples (Guide 5) before tuning prompts again.

from dataclasses import dataclass, asdict
import json

@dataclass
class EvalReport:
    suite: str
    pass_rate: float
    passed: int
    failed: int
    baseline_pass_rate: float
    delta: float
    failed_ids: list[str]
    model: str
    prompt_version: str
    git_sha: str

def write_report(path: str, report: EvalReport) -> None:
    with open(path, "w") as f:
        json.dump(asdict(report), f, indent=2)

def should_fail_ci(report: EvalReport, max_regression: float = 0.02) -> bool:
    return report.delta < -max_regression

public record EvalReport(
    String suite,
    double passRate,
    int passed,
    int failed,
    double baselinePassRate,
    double delta,
    List failedIds,
    String model,
    String promptVersion,
    String gitSha
) {
  public boolean shouldFailCi(double maxRegression) {
    return delta < -maxRegression;
  }
}

flowchart LR
  B[Baseline run on main]
  C[Change prompt RAG model]
  R[Re-run gold suite]
  D{Delta within threshold?}
  S[Ship merge deploy]
  F[Fix or revert]
  B --> C --> R --> D
  D -->|yes| S
  D -->|no| F --> C

Step 1 — Baseline

Run full suite on main with production config. Store reports/baseline.json: pass rate, failed IDs, latency, token cost. This is your contract for “currently acceptable.”

Step 2 — Change

One change per experiment: new system prompt version, chunk size 512→768, swap gpt-4o-mini for Claude, add a tool. Log the hypothesis in the PR description.

Step 3 — Compare

Re-run the same suite. Compute delta. Open failed cases side-by-side: old answer vs new. Distinguish improvements (new pass), regressions (new fail), and gold drift (policy changed—update JSONL).

Step 4 — Ship

Merge when delta ≥ threshold (e.g. no worse than −1% overall, no safety tag below 100%). Tag prompt registry version; deploy with feature flag; monitor online metrics (Guide 5).

Promote production failures into gold

When support escalates a bad bot answer, capture question + expected behavior + actual failure mode. Add JSONL row with same id pattern. That closes the loop—eval suites grow from real pain.

Example PR workflow

1. Engineer bumps support_system_v3 → v4 in prompt registry.
2. CI runs support_smoke.jsonl with mock runner (free) — wiring OK.
3. Optional: CI runs 8-case live smoke if EVAL_LIVE=1 on protected branches.
4. Nightly job runs full suite; posts pass rate to Slack if delta < −1% vs baseline.
5. On merge, deploy behind flag; online sampler watches thumbs-down for 24h before full rollout.

RAG pipeline changes follow the same loop but add retrieval metrics first—if recall@5 drops, fix chunking or hybrid search before rewriting the generation prompt. See RAG explained eval basics and Guide 3 in this track for RAGAS-style decomposition.

import sys
from eval.lib.load_cases import load_cases
from eval.scorers import score_case
from runners.live_support import answer as pipeline

def run_suite(gold_path: str, baseline_rate: float) -> int:
    cases = load_cases(gold_path)
    passed, failed_ids = 0, []
    for case in cases:
        ans = pipeline(case.question)
        ok, _ = score_case(ans, case)
        if ok:
            passed += 1
        else:
            failed_ids.append(case.id)
    rate = passed / len(cases)
    delta = rate - baseline_rate
    print(f"pass_rate={rate:.3f} delta={delta:+.3f} failed={failed_ids}")
    return 1 if delta < -0.02 else 0

if __name__ == "__main__":
    sys.exit(run_suite("eval/gold/support_smoke.jsonl", baseline_rate=0.94))

@Service
public class EvalSuiteRunner {
  public int runAndExitCode(Path goldPath, double baselineRate) {
    List cases = caseLoader.load(goldPath);
    List failed = new ArrayList<>();
    int passed = 0;
    for (EvalCase c : cases) {
      String ans = pipeline.answer(c.question());
      if (scorer.score(ans, c).passed()) passed++;
      else failed.add(c.id());
    }
    double rate = (double) passed / cases.size();
    double delta = rate - baselineRate;
    log.info("pass_rate={} delta={} failed={}", rate, delta, failed);
    return delta < -0.02 ? 1 : 0;
  }
}

Reference-based evals

Fast, deterministic, CI-friendly. Encode legal/compliance language literally: “partial months,” “30-day window.” Combine with forbid_phrases for hallucinated guarantees. Track 3 formatting guides pair with schema validators for JSON outputs.

LLM-as-judge

When rules cannot enumerate good answers, a separate model scores against a rubric: “Answer cites policy, refuses harmful requests, does not invent refund amounts.” Calibrate judges against human labels; use chain-of-thought rubrics with structured JSON output. Deep dive in LLM-as-judge.

Human eval

Sample 50–200 cases for blind A/B comparison between prompt versions. Human scores become labels to tune judges and detect judge drift. Never skip human review for high-stakes launches—automate the long tail only.

Task-based evals

For agents: assert tool sequence contains lookup_order before search_policy; mock CRM returns expected state; ticket marked resolved in sandbox. Final answer quality is insufficient when wrong tools ran.

Mapping eval types to Tracks 1–4

Composite pipelines stack scorers: retrieval recall (unit/integration on index), generation faithfulness (judge), policy phrases (rules), safety (red-team reference + judge). Failures in composite stacks need failure attribution—log which stage failed so you do not “fix” prompts when retrieval broke.

flowchart TD
  Q[What must you prove?]
  Q -->|Exact policy text| R[Reference rules]
  Q -->|Paraphrase OK| E[Embedding similarity]
  Q -->|Subjective quality| J[LLM-as-judge]
  Q -->|High stakes launch| H[Human rubric]
  Q -->|Agent completed action| T[Task-based sandbox]

def score_case(answer: str, case: EvalCase, use_judge: bool = False) -> tuple[bool, str]:
    ok, msg = score_must_contain(answer, case.must_contain)
    if not ok:
        return ok, msg
    ok, msg = score_forbid(answer, case.forbid_phrases)
    if not ok:
        return ok, msg
    if case.expect_refusal and not looks_like_refusal(answer):
        return False, "expected refusal"
    if use_judge and case.tags and "subjective" in case.tags:
        return llm_judge(answer, case.question, rubric=JUDGE_RUBRIC)
    return True, "ok"

public ScoreResult scoreCase(String answer, EvalCase c, boolean useJudge) {
  ScoreResult r = RuleScorers.mustContain(answer, c.mustContain());
  if (!r.passed()) return r;
  r = RuleScorers.forbid(answer, c.forbidPhrases());
  if (!r.passed()) return r;
  if (c.expectRefusal() && !RefusalDetector.isRefusal(answer))
    return ScoreResult.fail("expected refusal");
  if (useJudge && c.tags() != null && c.tags().contains("subjective"))
    return llmJudge.score(answer, c.question(), JUDGE_RUBRIC);
  return ScoreResult.pass("ok");
}

Architecture sketch

flowchart LR
  GIT[Gold JSONL in git]
  CI[CI smoke runner]
  NIGHT[Nightly live runner]
  RPT[Report artifact]
  OBS[Online trace sampling]
  GOLD[Promote to gold]
  GIT --> CI --> RPT
  GIT --> NIGHT --> RPT
  OBS --> GOLD --> GIT

Eval readiness checklist

• Gold JSONL in git with unique id per case; schema validated in unit tests
• Smoke suite (≤15 cases) runs on every PR—mock or live with budget cap
• Full suite nightly with pinned model and prompt_version
• CI fails when pass rate drops more than agreed threshold vs baseline
• Report artifact uploaded: pass rate, delta, failed IDs, latency, token cost
• Per-tag breakdown for billing, safety, RAG, agent trajectories
• Rule scorers for policy phrases; judges scheduled—not on every PR by default
• Human review queue for 1–5% production samples
• Process to add production failures as new gold rows within SLA
• Prompt/registry version linked to eval score at deploy time
• Feature flag rollback when online metrics diverge from offline eval
• RAG evals separate retrieval recall from generation faithfulness (Guide 3)

Minimum viable eval platform (week one)

You do not need LangSmith or Braintrust on day one. Minimum stack:

• eval/gold/*.jsonl — 20 cases from real tickets
• eval/scorers.py — rule scorers + unit tests
• runners/mock_*.py — deterministic pipeline stub
• eval/run_suite.py — exit code 0/1
• .github/workflows/eval-smoke.yml — runs on PR
• reports/latest.json — CI artifact

Upgrade to live runners, judges, and observability exporters as coverage and traffic grow—Guide 6 walks CI wiring end-to-end.

Logging every eval run

from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel

app = FastAPI()
ADMIN_TOKEN = "rotate-me"

class EvalTrigger(BaseModel):
    suite: str = "support_smoke"
    runner: str = "mock"

@app.post("/internal/eval/run")
def trigger_eval(body: EvalTrigger, x_admin_token: str = Header(...)):
    if x_admin_token != ADMIN_TOKEN:
        raise HTTPException(403)
    report = run_suite(body.suite, runner=body.runner)
    return {"run_id": report.run_id, "pass_rate": report.pass_rate}

@RestController
@RequestMapping("/internal/eval")
public class EvalController {
  @PostMapping("/run")
  public EvalRunResponse trigger(@RequestBody EvalTrigger body,
      @RequestHeader("X-Admin-Token") String token) {
    if (!adminToken.equals(token)) throw new ResponseStatusException(HttpStatus.FORBIDDEN);
    EvalReport report = suiteRunner.run(body.suite(), body.runner());
    return new EvalRunResponse(report.runId(), report.passRate());
  }
}