Evals & Benchmarking

Do Not Start With a Framework Decision

The most common way an eval effort dies is planning it. You do not need a labeled dataset, a metrics dashboard, or a platform decision. You need one failure you can reproduce.

Find a case where your feature did the wrong thing. Write down the input and what should have happened. That is test case number one, and everything in this pack is built by repeating it.

The Example Used Here

In this study pack, we'll show how you can get started with libraries like Promptfoo — a config-file eval runner that works against any provider and runs from the CLI. Other tools differ in syntax, but the shape is the same everywhere: a file listing your prompts, your models, and your test cases, plus a command that runs them and tells you what broke.

npx promptfoo@latest init --example getting-started

That writes a promptfooconfig.yaml you can edit immediately.

Ten Minutes to First Signal

Replace the generated config with your one real failure:

description: Support ticket categorization

prompts:
  - 'Categorize this support ticket into exactly one of: Hardware, Software, Billing. Reply with only the category.\n\nTicket: {{ticket}}'

providers:
  - anthropic:messages:claude-sonnet-5

tests:
  - vars:
      ticket: 'My monitor will not turn on'
    assert:
      - type: equals
        value: Hardware

Then run it and look at the results:

npx promptfoo@latest eval
npx promptfoo@latest view

view opens a browser table of every prompt and model combination side by side. One passing test is not an eval suite, but it is a harness that now exists — and adding case two costs nothing.

Related Concepts

External Resources

Four Keys Do the Work

Everything in an eval config answers one of four questions: what am I asking, who am I asking, what am I asking about, and what counts as right.

description: Support ticket categorization   # label for the run

prompts:                                      # what you are asking
  - file://prompts/categorize.txt

providers:                                    # who you are asking
  - anthropic:messages:claude-sonnet-5
  - anthropic:messages:claude-opus-5

defaultTest:                                  # applies to every case
  assert:
    - type: latency
      threshold: 3000

tests: file://tests/cases.csv                 # the cases, and what counts as right

List two providers and every case runs against both, which is how you compare models on your task instead of on a leaderboard.

Where Things Live

Once you are past a handful of cases, split them out. Nothing here is magic — file:// just points at a path relative to the config.

my-project/
  promptfooconfig.yaml     # the run definition
  prompts/
    categorize.txt         # the prompt under test, version controlled
  tests/
    cases.csv              # your growing set of real failures
    edge-cases.csv         # deliberately nasty inputs

Keeping the prompt in its own file matters more than it looks. It means a prompt change shows up as a diff in code review, and the eval that guards it sits in the same commit.

defaultTest Is the One to Know

defaultTest sets properties every case inherits. Put your universal rules there — a latency ceiling, a "never mentions a competitor" check, the model you grade with — instead of repeating them on 300 cases.

defaultTest:
  options:
    provider: anthropic:messages:claude-opus-5   # grader, not the model under test
  assert:
    - type: is-json

That options.provider line is doing real work. It sets which model grades the model-graded assertions, and it should not be the model you are testing.

Related Concepts

External Resources

Code First, Always

An assertion is just a grader with a name. The deterministic ones run locally, cost nothing, and never drift — so exhaust them before reaching for anything smarter.

is-json takes a schema inline, which turns "did it return the right shape" into a one-liner:

assert:
  - type: is-json
    value:
      type: object
      required: [category, confidence]
      properties:
        category: { type: string }
        confidence: { type: number }

When no built-in fits, javascript gets you output and context:

assert:
  - type: javascript
    value: output.split(' ').length < 50

Model Graders for Judgment

Tone, faithfulness to a source, and "is this actually helpful" need a judge. llm-rubric scores the output against criteria you write, from 0.0 to 1.0:

assert:
  - type: llm-rubric
    value: |
      Every factual claim appears in the provided source document.
      No claim is added, softened, or exaggerated.
    threshold: 0.8
    provider: anthropic:messages:claude-opus-5

Strong judges agree with human preferences over 80% of the time — roughly the rate two humans agree with each other. That makes them useful and not authoritative.

Three Biases to Design Around

Write rubrics as checkable statements, not adjectives. "Every claim appears in the source" produces stable scores; "high quality" does not.

Related Concepts

External Resources

A Spreadsheet Beats a Schema

Once you have more than a few cases, move them out of the config. A CSV is the right format because non-engineers can add rows.

ticket,__expected,__description
"My monitor will not turn on","equals: Hardware","basic hardware"
"I was charged twice","equals: Billing","basic billing"
"the thing is broken???","llm-rubric: picks a category and does not ask a clarifying question","vague input"

Column headers become variables. __expected is the assertion — bare text means equals, or name a type explicitly. Use __expected1, __expected2 for several. Point the config at it:

tests: file://tests/cases.csv

Every Bug Becomes a Row

This is the loop that makes the suite worth having. A user reports a bad answer, you reproduce it, you add the row. That failure can now never silently return.

The corollary: do not write cases from imagination. A set invented at your desk tests what you expected users to do. Production logs test what they actually did.

Volume Over Polish

Anthropic's eval guidance is blunt about the trade: "More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals."

Fifty hand-curated cases feel rigorous and cannot detect a three-point regression. Published examples run to 200 articles for summarization, 1,000 tweets for sentiment, 100 multi-turn conversations for a chat assistant. A few hundred automatically-graded rows is the target.

Add the Nasty Ones on Purpose

A set drawn only from typical traffic scores well and catches nothing. Keep a second file for inputs you know are hard:

• Irrelevant or empty input
• Input far longer than you designed for
• Hostile or off-topic user turns
• Cases ambiguous enough that two reviewers would disagree

Keep that last group even though it lowers your score. It is the only way to see whether the system asks, guesses, or invents when there is no right answer.

Related Concepts

External Resources

The Answer Text Is Not the Result

An agent's real output is what it did — files written, rows updated, tickets closed. Grading the summary it writes about its work measures its prose, not its behavior.

Two levels of grading, and you want both.

Level One: Did It Call the Right Tools?

Assertions that read the trace rather than the final message:

Sequence is the one people skip and then regret. "Looked up the account before issuing the refund" is an ordering property, and an agent that gets it backwards passes every other check.

tests:
  - vars:
      request: 'refund my last order'
    assert:
      - type: trajectory:tool-sequence
        value: [lookup_order, check_refund_policy, issue_refund]
      - type: is-valid-function-call

Level Two: Did the World End Up Right?

The strongest agent grader compares end state against a goal state. This is exactly what τ-bench does: it simulates a user conversation, then compares the database at the end against an annotated target. What the agent claimed is never consulted.

For your own agent, that means a fixture — a seeded test database, a scratch repo, a fake queue. Run the task, diff the end state, throw away the prose. In promptfoo that is a javascript assertion that inspects your fixture and returns a result:

module.exports = async (output, context) => {
  const order = await db.getOrder(context.vars.orderId);
  return {
    pass: order.status === 'refunded' && order.refundAmount === 4200,
    score: order.status === 'refunded' ? 1 : 0,
    reason: `order ended as ${order.status}`,
  };
};

Do Not Give Partial Credit

Multi-step tasks tempt you into scoring "4 of 6 steps correct." That number climbs while the task still fails every time. Score the task binary, and keep step detail for debugging only.

Related Concepts

External Resources

One Run Is a Coin Flip

Sampling is stochastic. An agent that passes your suite once may pass it a third of the time, and the single-run score is exactly the number that breaks after launch.

The fix is one flag:

npx promptfoo@latest eval --repeat 5

Every case runs five times. What you are looking for is not the average — it is any case that passed some runs and failed others. Those are your real risks, and a single run hides them completely.

pass^k vs pass@k

τ-bench formalized the distinction, and it is the most useful idea here:

The gap is wide in practice. Leading function-calling agents solved under half of τ-bench's tasks on a single attempt, and GPT-4o's pass^8 in the retail domain landed below 25% — succeeding in all eight tries less than a quarter of the time.

Pick your metric by supervision. A human reviewing every output means pass@k is fine. Nothing watching means pass^k is the only honest number.

Make It Affordable

Repeating everything is expensive, so tier it:

Remove the Variance You Own First

Some inconsistency is yours, not the model's. Pin an explicit model version rather than a moving alias, set temperature deliberately, and use --no-cache when you actually want fresh samples — cached responses will happily report perfect consistency that does not exist.

Related Concepts

External Resources

Run It on Every Prompt Change

An eval you run when you remember is documentation. The version that changes behavior runs whenever someone edits a prompt, a tool definition, or a model version.

- uses: actions/setup-node@v4
  with:
    node-version: '24'
- name: Run eval
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json
- name: Quality gate
  run: |
    FAILURES=$(jq '.results.stats.failures' results.json)
    if [ "$FAILURES" -gt 0 ]; then exit 1; fi

The gate is a separate step on purpose. Parsing results.json yourself means you decide what "too many failures" is, rather than accepting a binary all-pass rule that nobody can satisfy.

This Is Where the Threshold Lives

Every eval needs a number written down before the run, or a result of 78% starts a debate instead of a decision. CI is the honest place to put it, because it is the only place the number has consequences.

Set one per axis rather than a single accuracy figure:

Gate on Movement, Not Perfection

A hard "95% or the build fails" rule breaks on the first genuinely ambiguous case and gets bypassed within a month. Compare against the last run on main and fail on a drop instead.

Report which cases changed, not just the total. A steady 88% concealing twelve newly-broken cases and twelve newly-fixed ones is precisely the regression this is supposed to catch.

Keep It Fast and Cheap

• Cache between runs by setting PROMPTFOO_CACHE_PATH and restoring it in the workflow.
• Use -j to control concurrency when you hit provider rate limits.
• Re-run only what broke while iterating locally with --filter-failing.
• Keep the per-PR set to a few minutes. A gate that adds fifteen minutes gets disabled.

Related Concepts

External Resources

Your Set Goes Stale by Default

The suite is a snapshot of failures you knew about when you wrote it. Traffic shifts, hosted models change under you, prompts get edited. Offline scores can hold flat while real quality slides.

Three production signals catch what the suite cannot:

• Task completion rate. Did the user get what they came for, or rephrase and try again? Implicit, always available, hard to game.
• Explicit feedback. Thumbs and reports are sparse and skew negative — a great source of new cases, a poor headline metric.
• Escalation and retry rate. How often a human took over, or the same request returned within minutes.

Ship Changes as Comparisons

A prompt change that looks better offline still goes out as an A/B test against the current version. Your eval set is small and self-selected; production traffic is neither.

The Loop That Compounds

Every production failure becomes a row in the CSV:

• Step 1 — a user reports a bad answer.
• Step 2 — you reproduce it and write down the correct behavior.
• Step 3 — it becomes a row in tests/cases.csv, in the same PR as the fix.
• Step 4 — CI guards it from then on.

Teams that do this end up with a suite that predicts production. Teams that skip step 3 end up with a suite that passes while users complain.

Re-Run on a Schedule

Even with no code changes, run the full set weekly. A hosted model can shift beneath a stable version alias, and a scheduled run is how you find out from your own dashboard rather than from a customer.

Related Concepts

External Resources

Shortlisting, Not Deciding

Public benchmarks measure a model against problems someone else chose. SWE-bench, for instance, is 2,294 real GitHub issues across 12 Python repositories, where the model writes a patch and the repository's own tests decide if it passed. Genuinely hard, genuinely useful — and not your product.

Use them to narrow a field of models to two or three candidates. Then run your own suite from the previous sections against those candidates, which is the comparison that actually predicts your outcome.

Two effects mean the published numbers read high.

Contamination

If the test set was in the training data, the score measures recall, not capability. A survey of 31 models on mathematical reasoning found "substantial instances of training even test set misuse," detected via perplexity and n-gram comparisons.

You cannot audit a closed model's training data. Assume any benchmark published well before a model's release is partly contaminated, and weight recent or private evaluations higher.

Optimizing for the Leaderboard

A fixed benchmark becomes a target. An audit of Chatbot Arena found that access to arena data produced up to 112% relative gains on the arena's own distribution — improvement on the test, not the task. It also found sampling was lopsided: Google and OpenAI received roughly 19.2% and 20.4% of arena data, while 83 open-weight models shared 29.7%.

Benchmarks Get Corrected Too

SWE-bench Verified exists because engineers reviewed the original set and found problems that were underspecified or graded by tests no correct patch could pass. It is 500 human-validated instances rather than 2,294 scraped ones.

That lesson applies directly to your own CSV. When a case fails, read the case before you blame the model — a meaningful share of the time the expected answer is the thing that is wrong.

Related Concepts

External Resources

Start From What You Are Checking

Two Rules That Settle the Rest

Grade the outcome, not the explanation. A model describing its own work is generating text, not reporting a measurement. Prefer any grader that inspects the artifact — the patch, the row, the parsed object.

Match the metric to the supervision. If a human checks every output, measure whether it can succeed. If nothing is watching, measure whether it succeeds every time.

The Smallest Suite Worth Having

If you have nothing today, this is a real afternoon's work and it beats any plan:

• Step 1 — run npx promptfoo@latest init.
• Step 2 — pull 30 real failures out of your logs into tests/cases.csv.
• Step 3 — grade with equals and is-json where you can, llm-rubric where you cannot.
• Step 4 — add the GitHub Actions job from Wire It Into CI.
• Step 5 — add a row every time something breaks in production.

That is a worse suite than the one described here, and enormously better than none.

Related Concepts

External Resources