Model Evals: How to Know If You Can Use a Cheaper Model

An eval, in the AI FinOps context, is a structured comparison: run a representative sample of real production inputs through your current model and a cheaper candidate, score both against a defined quality bar, and switch only if the candidate passes. It’s the mechanism that turns “use a cheaper model” from advice into a decision you can defend.

Why “Use a Cheaper Model” Needs a Mechanism

Model right-sizing is the highest-leverage cost lever in AI FinOps — the per-token difference between Haiku and Opus is 5x, and closer to 10x against the most expensive frontier models. The What Is AI FinOps? post covers where right-sizing sits among the other levers.

But unlike compute right-sizing, there’s no utilization metric to point at. An EC2 instance at 4% CPU is objectively oversized. A model is only oversized relative to a quality bar — and most teams never define one. The result is paralysis in both directions: teams either stay on the expensive model indefinitely because nobody can prove the cheap one is safe, or they switch on vibes and discover the quality regression through customer complaints.

An eval replaces both failure modes with a measurement.

The Eval Loop

The loop has four steps, and none of them requires specialized infrastructure:

1. Collect representative inputs. Pull real production requests — not synthetic examples. The distribution matters: include the routine cases, the edge cases, and the malformed inputs your system actually receives. If you’ve tagged your Bedrock calls by feature, filtering the right sample out of your logs is straightforward.
2. Define the quality bar. A pass/fail criterion you can apply to every output, decided before you look at any results. The next section covers what this looks like per task type.
3. Run both models. Same prompts, same inputs, both models. Capture the outputs side by side.
4. Compare and decide. If the candidate meets the bar, switch and start saving on every future call. If it doesn’t, you’ve confirmed the expensive model is earning its cost.

Defining the Quality Bar by Task Type

What “good enough” means depends on what the task produces:

Classification — the bar is an accuracy threshold against labeled examples. If your current model routes support tickets at 96% accuracy and your business tolerates 94%, that’s the bar. Exact-match scoring makes this the cheapest eval to run and the easiest to automate.

Structured extraction — the bar is a pass rate on field-level checks: does the output parse, are required fields present, do values match the source document. Schema validation plus spot checks on a labeled subset covers most cases.

Summarization and generation — there’s no exact match to score against, so the bar is a rubric: faithfulness to the source, coverage of key points, absence of fabricated claims. This is where LLM-as-judge earns its place.

Simple Q&A — usually reducible to one of the above: either there’s a reference answer (score like classification) or there isn’t (score with a rubric).

LLM-as-Judge for Generation Tasks

For outputs that can’t be string-matched, use a frontier model to grade the candidate’s outputs against your rubric. The judge sees the input, the output, and the rubric — and returns a score or a pass/fail with a justification.

Two practices keep judge scores honest:

• Calibrate the judge first. Have it grade a small set of outputs you’ve already graded yourself. If the judge disagrees with you more than occasionally, fix the rubric before trusting it at scale.
• Judge outputs blind. Don’t tell the judge which model produced which output. Models exhibit measurable bias toward outputs they recognize as their own family’s style.

The economics work because judging is a one-time cost while the savings recur: you pay frontier-model rates to grade a few hundred outputs once, then collect the cheaper model’s rate on every production call after the switch.

How Many Inputs You Need

For classification, 50–200 inputs is usually sufficient to detect a meaningful accuracy gap — if two models differ by less than what that sample can resolve, the difference probably doesn’t matter to your business either. Structured extraction sits in the same range, provided the sample covers your document variety.

Generation tasks need more, typically several hundred, because rubric scores are noisier than exact matches and quality failures concentrate in the tail: the unusual request, the ambiguous source document. Weight your sample toward the inputs you’d least like to get wrong.

Tooling

You can DIY a basic eval. It is 50 lines of Python: loop over a CSV of inputs, call Bedrock with each model, compare outputs to labels, print two accuracy numbers. Keep the script and the input sample in version control so you can re-run the comparison when a prompt changes or a new model version ships. For deterministic scoring — classification, extraction, anything with an exact match — this works perfectly fine.

Dedicated tooling earns its place when the eval outgrows that: model-graded scoring is easy to get subtly wrong (a miscalibrated DIY judge passes models that should fail), and investigating a failed run means reading dozens of input/output pairs side by side. The tools below are essentially that same script with the judge implementation done correctly and a review surface attached:

• Promptfoo — open-source, configured in YAML, runs against Bedrock directly. Deterministic assertions (equality, regex, JSON validity) plus model-graded rubrics via llm-rubric.
• LangSmith — datasets and evaluators integrated with tracing. The right fit if you’re already tracing your application through LangSmith, because production traces become eval datasets with little friction.
• Braintrust — a hosted eval platform with a scorer library and side-by-side output diffing. Strongest UI for the human-review portion of the workflow.

A minimal Promptfoo comparison of two Bedrock models looks like this:

prompts:
  - "Classify this support ticket into one of: billing, technical, account. Reply with the category only.\n\nTicket: {{ticket}}"

providers:
  - id: bedrock:us.anthropic.claude-sonnet-4-20250514-v1:0
  - id: bedrock:us.anthropic.claude-haiku-4-5-20251001-v1:0

tests:
  - vars:
      ticket: "I was charged twice for my June invoice"
    assert:
      - type: equals
        value: billing

In practice, you’d replace the single hardcoded test case with your exported production sample. Running promptfoo eval then sends every input to both models and produces a side-by-side table of results — pass rates per model, ready to compare against the quality bar.

When Evals Aren’t Enough

Some tasks resist measurement. Complex multi-step reasoning, nuanced judgment calls, and open-ended advisory outputs degrade in ways a rubric struggles to capture — the cheaper model’s output looks complete but misses the consideration that mattered. For these, an eval is a starting signal, but can’t be the final verdict.

You may have to handle it operationally: roll the cheaper model out to a fraction of traffic, watch downstream signals like user retries, escalation rates, correction edits. If quality degradation would be expensive and hard to detect, the frontier model’s premium is insurance — and a defensible line item.

Conclusion

“Eval” is short for evaluation, and in this context it’s a glorified term for a comparison. You are simply comparing two models by passing the same inputs through both and scoring the outputs. The mechanics — sampling requests, running both models, tallying results — are the easy part. The effort lies in defining the bar, because quality is mostly qualitative to begin with — “the summary covers what matters” is a judgment, not a number. An eval is the exercise of turning that judgment into something countable: a threshold, a pass rate, a rubric score. This translation is also where the errors creep in — measuring what’s easy to count rather than what you care about, or reading false precision into a judge’s score. The safeguard is to read a sample of failures yourself and confirm the numbers fail for the reasons you think. After a proper eval, the right-sizing decision makes itself.