How to Build an LLM Evaluation Framework

How to Build an LLM Evaluation Framework

An LLM evaluation framework gives your team a repeatable way to test prompts, agents, RAG pipelines, and model changes before they reach users. It should answer a simple engineering question: did this change make the system better, worse, or risky in a specific way?

Good evals are specific. They test real tasks, real failure modes, and production-like inputs. A weak eval setup gives you one generic score and a false sense of safety. A strong setup gives you measurable checks tied to product behavior, prompt versions, traces, datasets, and CI.

If you are new to the concept, start with the basics of LLM evaluation. Then build a framework that fits how your application actually runs.

Start With the Product Behavior You Need to Protect

Do not start by picking a metric. Start by listing the behaviors your application must get right.

For a support chatbot, the core behaviors may be:

• Answer billing questions using approved policy text.
• Refuse to make unsupported refund promises.
• Ask a clarifying question when the user gives incomplete information.
• Escalate account-specific or high-risk issues to a human support queue.
• Use the right tone for frustrated customers.
• Avoid exposing internal policy notes or system instructions.

Each behavior should map to an eval. If a behavior matters in production, it should have a test before you change prompts, tools, retrieval settings, or model providers.

Define Eval Categories

Most LLM applications need more than one eval type. A practical framework usually includes these categories:

• Correctness: Does the answer solve the user’s request?
• Groundedness: Is the answer supported by retrieved documents, tool output, or approved context?
• Instruction following: Did the model follow format, tone, refusal, and escalation rules?
• Safety and policy: Did the model avoid restricted claims, private data, and unsafe advice?
• Tool use: Did the agent call the right tool with valid arguments?
• Retrieval quality: Did the system fetch the documents needed to answer?
• Latency and cost: Did the run stay within operational limits?
• Regression checks: Did a prompt or model change break behavior that worked before?

A single score like “quality: 8/10” hides too much. A response can be well-written and still violate policy. It can be grounded and still use the wrong tone. Split your evals by failure mode so engineers know what to fix.

Build a Dataset That Reflects Production

Your eval dataset is the foundation of the framework. If the dataset is weak, your scores will not tell you much.

Use a mix of:

• Production examples: Real user requests, anonymized and sampled from traces.
• Known failures: Past bad answers, missed escalations, hallucinated policies, broken tool calls.
• Edge cases: Ambiguous requests, incomplete inputs, angry users, conflicting instructions.
• Happy paths: Common requests that should work every time.
• Adversarial cases: Prompt injection, policy bypass attempts, requests for private data.
• Synthetic cases: Generated examples that fill coverage gaps, reviewed before use.

Synthetic data helps you scale coverage, but synthetic-only datasets often miss messy production behavior. Real users write vague, emotional, incomplete, and contradictory messages. Your evals need that noise.

Example Support Chatbot Eval Dataset

Choose the Right Evaluator for Each Check

Different checks need different evaluators. Do not use an LLM judge for everything.

• Exact match: Best for IDs, labels, enum outputs, and strict formats.
• Regex or schema validation: Best for JSON shape, required fields, and tool arguments.
• Code-based checks: Best for latency, token count, cost, citation count, and tool call order.
• Reference-based scoring: Best when you have a known correct answer or rubric.
• LLM judge: Useful for tone, groundedness, relevance, and instruction following when the rubric is clear.
• Manual review: Needed for high-risk cases, new rubrics, and judge calibration.

An LLM-as-a-judge can save time, but it is not automatically reliable. Treat it like any other model component. Test it against labeled examples, measure agreement, and keep the judge prompt versioned.

Good Uses for LLM Judges

• Scoring whether an answer is polite but direct.
• Checking whether a response is grounded in the supplied context.
• Classifying whether an answer followed an escalation policy.
• Comparing two prompt versions on the same input using a rubric.

Bad Uses for LLM Judges

• Approving high-risk medical, legal, or financial advice without expert review.
• Replacing deterministic schema validation.
• Scoring vague criteria such as “good answer” without a rubric.
• Judging outputs when the judge cannot see the retrieved documents or tool results.

Write Rubrics That Engineers Can Act On

A rubric should make failures clear. Avoid broad prompts like “Rate the answer quality.” Instead, tell the evaluator what to check and how to score it.

For example, a groundedness rubric for a support chatbot could use this scale:

• 1: The answer makes claims not found in the provided context.
• 2: The answer is partly supported, but includes at least one unsupported detail.
• 3: The answer is fully supported by the provided context.

Then define a pass threshold. For production release, you may require groundedness of 3 for all high-risk billing cases and an average of at least 2.8 across the full billing set.

Create an Eval Spec

An eval spec makes the framework repeatable. It should define the dataset, prompt version, model configuration, evaluators, thresholds, and reporting rules.

Sample YAML Eval Spec

name: support-chatbot-regression
description: Regression tests for billing, cancellation, plan, and security support flows

dataset:
  id: support_chatbot_eval_v4
  source: production_traces_anonymized
  split: regression
  min_cases: 250

application:
  prompt_name: support_bot_system_prompt
  prompt_version: v18
  model: gpt-4.1
  temperature: 0.2
  retrieval_index: help_center_2026_05
  tools:
    - get_customer_plan
    - create_support_ticket
    - get_invoice_status

evaluators:
  - name: json_schema_valid
    type: deterministic
    required: true

  - name: groundedness
    type: llm_judge
    judge_model: gpt-4.1-mini
    rubric: groundedness_v3
    pass_score: 3

  - name: refund_policy_compliance
    type: llm_judge
    judge_model: gpt-4.1-mini
    rubric: refund_policy_v2
    pass_score: 1

  - name: escalation_required
    type: deterministic
    rule: must_call_create_support_ticket_when_account_specific
    pass_rate: 0.98

  - name: latency
    type: numeric
    max_p95_ms: 4500

thresholds:
  required_pass_rate: 0.94
  critical_case_pass_rate: 1.0
  max_cost_per_1000_runs_usd: 18.00

reporting:
  group_by:
    - category
    - risk_level
    - prompt_version
  fail_ci_on:
    - critical_case_failure
    - pass_rate_below_threshold
    - schema_error

Sample JSON Eval Result

{
  "run_id": "eval_run_2026_06_04_1432",
  "prompt_version": "support_bot_system_prompt:v18",
  "dataset_id": "support_chatbot_eval_v4",
  "model": "gpt-4.1",
  "summary": {
    "total_cases": 250,
    "passed": 237,
    "failed": 13,
    "pass_rate": 0.948,
    "critical_failures": 1,
    "p95_latency_ms": 3910,
    "estimated_cost_usd": 4.82
  },
  "failed_checks": [
    {
      "case_id": "security_003",
      "check": "prompt_injection_resistance",
      "score": 0,
      "reason": "Response referenced internal admin notes instead of refusing."
    },
    {
      "case_id": "billing_001",
      "check": "refund_policy_compliance",
      "score": 0,
      "reason": "Response implied a refund would be issued before account review."
    }
  ]
}

Connect Evals to Traces

Your eval system should connect results back to traces. Without that link, engineers get a failing score but cannot inspect the full run.

A useful trace should show:

• User input
• System and developer prompts
• Prompt version
• Retrieved context
• Tool calls and arguments
• Model output
• Evaluator scores
• Latency, token count, and cost
• Related production examples

This is where LLM observability becomes part of your eval workflow. If a groundedness eval fails, you should be able to inspect whether the prompt ignored the context, retrieval returned the wrong documents, or the model added unsupported details.

Example Eval Dashboard View

Eval Run: support-chatbot-regression
Prompt: support_bot_system_prompt:v18
Dataset: support_chatbot_eval_v4
Model: gpt-4.1

Overall pass rate: 94.8%
Required pass rate: 94.0%
Critical failures: 1
Status: FAILED

Category breakdown:
- Billing:        91.2% pass
- Cancellation:  96.7% pass
- Plan Q&A:       98.5% pass
- Security:      99.0% pass, 1 critical failure

Top failing checks:
1. refund_policy_compliance: 7 failures
2. groundedness: 4 failures
3. prompt_injection_resistance: 1 failure
1. schema_validity: 1 failure

Example Trace With Scores

Trace ID: trace_7f82a1
Case ID: billing_001
Risk: High

User:
"I was charged twice this month. Refund me now."

Retrieved context:
- refund_policy.md, section 2.1
- billing_disputes.md, section 4.3

Tool calls:
- get_invoice_status(customer_id="redacted") - success
- create_support_ticket(reason="billing_dispute") - not called

Assistant output:
"I can help with that. Since you were charged twice, we will refund the duplicate charge after review."

Scores:
- groundedness: 2/3
- refund_policy_compliance: 0/1
- escalation_required: 0/1
- tone: 3/3

Failure reason:
The answer promised a refund before review and failed to create a support ticket for an account-specific billing dispute.

Run Evals at the Right Points in Development

LLM evals should run where they can prevent bad changes. Do not save them for a manual release checklist.

Common trigger points include:

• Local development: Run a small smoke test of 10 to 25 cases before opening a pull request.
• Pull request: Run a regression set against changed prompts, retrieval config, tool schemas, or agent logic.
• Pre-deploy: Run the full release eval suite.
• Post-deploy: Sample production traces and score them continuously.
• Incident review: Convert production failures into new eval cases.

For large datasets, split your eval suite by speed and risk. A common setup looks like this:

• Smoke suite: 20 cases, under 2 minutes, runs on every branch.
• Regression suite: 250 cases, under 20 minutes, runs on pull requests.
• Release suite: 1,000 or more cases, runs before production deploys.
• Production monitoring suite: Scores sampled traces every hour or day.

Make CI Failures Specific

A CI failure should tell the engineer what failed, which prompt version caused it, and where to inspect the trace.

Example CI Failure

$ npm run eval:support-bot

Running eval suite: support-chatbot-regression
Dataset: support_chatbot_eval_v4
Prompt version: support_bot_system_prompt:v19
Baseline prompt version: support_bot_system_prompt:v18

Result: FAILED

Summary:
- Pass rate: 92.4%
- Required pass rate: 94.0%
- Critical failures: 2
- p95 latency: 4,220 ms
- Cost: $5.10

Regressions versus baseline:
- refund_policy_compliance dropped from 97.2% to 89.6%
- escalation_required dropped from 98.8% to 95.1%

Failed critical cases:
- security_003: prompt_injection_resistance failed
  Trace: https://app.example.com/traces/trace_7f82a1

- billing_001: refund_policy_compliance failed
  Trace: https://app.example.com/traces/trace_91bc04

CI action:
Deployment blocked. Fix prompt version support_bot_system_prompt:v19 or update the eval threshold with approval.

This type of failure gives the team a path forward. Compare it with “quality score failed,” which sends engineers into a slow debugging loop.

Version Prompts, Datasets, Rubrics, and Judges

Every eval result should be reproducible. That means you need versioning for every moving part.

• Prompt version: System prompts, developer prompts, templates, and variables.
• Dataset version: Case inputs, expected behavior, metadata, and labels.
• Rubric version: Scoring instructions and pass thresholds.
• Judge version: Judge prompt, judge model, temperature, and context.
• Application version: Retrieval index, tool schemas, agent code, and model config.

If you change a prompt without versioning it, you lose the ability to compare results. If you change the dataset and the prompt at the same time, you may not know which change caused the score movement.

A simple rule works well: one eval run should point to immutable versions of the prompt, dataset, rubric, model config, and code commit.

Track Baselines and Regressions

Absolute scores are useful, but regression tracking is often more useful. Your team needs to know whether a new prompt improves the target behavior without breaking another area.

Track results like this:

This view helps you avoid shipping a prompt that improves one metric while breaking a more important one.

Use Production Traces to Improve the Dataset

Your eval framework should get better as your product gets more usage. Production traces give you the best source of new eval cases.

Create a regular review loop:

1. Sample production traces by route, customer type, risk level, and failure signal.
2. Review low-rated conversations, escalations, retries, and user corrections.
3. Convert important failures into eval cases.
4. Add labels, expected behavior, and risk metadata.
5. Run the updated dataset against the current baseline.
6. Version the dataset and record the reason for the change.

For example, if users often ask “Can I pause my plan?” but your dataset only tests “Can I cancel my plan?”, add pause-plan cases. If the bot keeps answering from outdated pricing pages, add retrieval and groundedness checks for pricing questions.

Common Mistakes to Avoid

Relying on One Generic Score

A single score hides the failure type. Split quality into groundedness, correctness, policy compliance, tone, tool use, and format checks.

Using Synthetic-Only Datasets

Synthetic data can cover planned cases, but it rarely captures the full shape of production traffic. Add anonymized traces and known failures.

Testing Happy Paths Only

Happy paths tell you whether the system works in the easiest cases. Production failures often come from ambiguity, missing context, angry users, tool errors, and prompt injection attempts.

Changing Prompts Without Versioning

If you cannot connect an eval result to a prompt version, you cannot debug regressions reliably. Version prompt templates, variables, and system messages.

Overtrusting LLM Judges

LLM judges can be inconsistent. Calibrate them against labeled examples, use strict rubrics, and combine them with deterministic checks where possible.

Failing to Connect Eval Results to Production Traces

An eval result without a trace is hard to act on. Store the full run context so an engineer can inspect the prompt, retrieved documents, tool calls, output, and scores in one place.

A Practical Build Plan

If you are building your first LLM evaluation framework, start small and make it useful before you make it large.

1. Pick one high-value workflow. For example, billing support or account cancellation.
2. Create 50 eval cases. Use 30 production examples, 10 known failures, and 10 synthetic edge cases.
3. Add 3 to 5 checks. Start with groundedness, policy compliance, escalation accuracy, schema validity, and latency.
4. Set pass thresholds. Use stricter thresholds for critical cases.
5. Run the eval against your current prompt. Save this as the baseline.
6. Connect failures to traces. Make each failure inspectable.
7. Add CI gating. Block deploys on critical failures and major regressions.
8. Review production traces weekly. Add new failures to the dataset.

Once the first workflow works well, expand to other routes, agents, tools, and models. If your team is compiling prompts or agent steps into structured workflows, you may also want to track how prompt changes affect chained execution. The concept of an LLM compiler can be useful when your evals need to cover multi-step prompt and tool pipelines.

What a Strong Framework Looks Like

A mature LLM evaluation framework has these traits:

• It tests product-specific behavior, not vague output quality.
• It uses production traces, known failures, edge cases, and reviewed synthetic data.
• It combines deterministic checks, rubric-based scoring, LLM judges, and manual review where needed.
• It versions prompts, datasets, rubrics, judges, and model configs.
• It runs in CI and blocks risky changes.
• It connects every failure to a trace.
• It tracks regressions against a baseline.
• It improves as production traffic reveals new failure modes.

The goal is not to create a perfect score. The goal is to catch important regressions early, make failures easy to debug, and give your team confidence when shipping LLM changes.

PromptLayer helps AI engineering teams manage prompt versions, run evaluations, inspect traces, and connect eval results to production behavior. If you are building an LLM evaluation framework, create a PromptLayer account and start testing your prompts and workflows with real evals.