Eval-Driven Prompt Development: Define Success Before You Write the Prompt

Your team ships a new prompt. Two hours later, someone says the output looks off. You read it. It seems fine. Your colleague is not so sure. There is no objective answer because nobody wrote down what “good” actually means. Eval-driven development flips the process. You write the test before you finalize the prompt. You change the prompt until the tests pass. The feedback loop is not a feeling. It is a number.

The Problem with Prompt Engineering as Usual

Most prompt work looks like this: write a prompt, call the API, read the output, decide if it is good enough, tweak, call again. Repeat until the output looks right. This loop has no memory and no standards.

When a second person reviews the output, they apply a different internal bar. What feels helpful to you might feel too terse to someone else. There is no regression detection, so when you change the prompt to fix one problem, you have no way to know whether you broke something else. And the process leaves no record of why a prompt was chosen over alternatives.

These are not edge cases. Every team I have seen ship LLM features hits these problems within the first month. The outputs feel right until they suddenly do not, and by then the bad outputs have already reached users.

What Eval-Driven Development Is

Eval-driven development (EDD) borrowed its core idea from test-driven development. You define the target before you do the work. The three steps: collect a golden dataset, define three to five scoring metrics, then write and change the prompt until every metric passes on every example.

The difference from TDD is important. Unit tests are fast and free. You run thousands of them per CI pipeline without thinking about cost. EDD evals are not free. Every eval run calls an LLM to judge the output, and those calls cost money and take time. That means you keep the golden dataset small, around 50 to 100 examples, and you pick metrics that actually correlate with your product goals rather than measuring everything you can think of.

Writing Your First Golden Dataset

Goldens are test cases. Each one has an input and an expected output. The expected output can be a concrete string, a structured format, or simply a label like “should refuse” for inputs you want the model to reject.

Start with real examples from your actual use case. If you are building a support chatbot, pull 20 real user messages from your logs. If you are building a code summarizer, pull 20 actual function signatures and docstrings your users work with. Do not generate synthetic examples for your first dataset. Real inputs surface the distribution your application actually handles.

Here is what a golden dataset looks like for a summarize-and-tag pipeline:

import json

goldens = [
    {
        "input": "The Federal Reserve raised interest rates by 25 basis points.",
        "expected": "The Fed raised rates by 0.25%.",
        "tags": ["finance", "news"]
    },
    {
        "input": "Man lands on moon, 1969.",
        "expected": "Apollo 11 landed on the Moon in 1969.",
        "tags": ["history", "science"]
    },
    {
        "input": "",
        "expected": None,
        "tags": ["edge-case"]
    },
    {
        "input": "Tell me everything about everything immediately.",
        "expected": "REFUSE",
        "tags": ["adversarial"]
    }
]

with open("goldens.json", "w") as f:
    json.dump(goldens, f, indent=2)

Notice the two edge cases. Empty input tests whether your pipeline handles missing data gracefully. The adversarial input tests whether the model refuses appropriately when someone tries to bait it into an over-promise. Include these on purpose. Happy-path examples alone will not catch the failure modes that actually hurt you.

Defining Metrics That Actually Matter

A metric is a function that takes an output and returns a score. There are three types worth knowing about.

Exact match is for structured outputs. If the expected output is a JSON object, you validate the schema and check field values directly. Python’s jsonschema library handles this cleanly:

from jsonschema import validate, ValidationError

def validate_json_output(output: str, expected_schema: dict) -> float:
    try:
        data = json.loads(output)
        validate(instance=data, schema=expected_schema)
        return 1.0
    except (json.JSONDecodeError, ValidationError):
        return 0.0

Fuzzy match using embedding similarity works for free-text responses where you cannot specify an exact expected string. You compute cosine similarity between the embedding of the output and the embedding of the expected answer. Scores above a threshold pass. This catches outputs that are semantically close even when the wording differs:

from numpy import dot
from numpy.linalg import norm

def cosine_sim(a: list, b: list) -> float:
    return dot(a, b) / (norm(a) * norm(b))

def embedding_score(output: str, reference: str, embed_fn) -> float:
    v1 = embed_fn(output)
    v2 = embed_fn(reference)
    return cosine_sim(v1, v2)

LLM-as-judge is the third type. A second LLM evaluates the output against a written criterion. This is necessary for qualities that are not easily formalized, like whether a response sounds helpful or whether it stays on topic. The judge needs a rubric, just like a human reviewer:

JUDGE_PROMPT = """You are evaluating a chatbot response for helpfulness.

User question: {input}
Actual response: {output}

Score the response on helpfulness from 1 to 5. A score of 5 means the
response directly and concisely answers the user's question. A score of 1
means the response is off-topic, wrong, or unhelpful.

Respond with only the integer score, nothing else."""

def score_helpfulness(input_text: str, output: str, model: str = "gpt-4o") -> float:
    prompt = JUDGE_PROMPT.format(input=input_text, output=output)
    response = openai.ChatCompletion.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    raw = response.choices[0].message.content.strip()
    return int(raw) / 5.0  # normalize to 0.0-1.0

The judge prompt is the make-or-break piece. A vague criterion like “score the response for quality” produces inconsistent results. The rubric needs specificity. “Does the response answer the user’s question without adding information the user did not ask for?” is better. “Is the response between 20 and 60 words and does it contain the word ‘rate’ if the question mentions interest rates?” is better still.

The Iterate Loop

Once you have goldens and metrics, the loop is simple. Run the evals. Read the failures. Change the prompt. Run again.

from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase

correctness_metric = GEval(
    name="Correctness",
    criteria="The response states a mathematically correct result.",
    evaluation_steps=[
        "Check whether the response contains the expected answer.",
        "Penalize if the response contains a factual error.",
        "Return a score between 0 and 1."
    ],
    model="gpt-4o",
    threshold=0.8
)

def run_evals(prompt: str, goldens: list) -> dict:
    results = {"passed": [], "failed": []}
    for golden in goldens:
        test_case = LLMTestCase(
            input=golden["input"],
            expected_output=golden["expected"],
            actual_output=call_llm(prompt, golden["input"])
        )
        try:
            assert_test(test_case, [correctness_metric])
            results["passed"].append(golden["input"][:50])
        except AssertionError:
            results["failed"].append(golden["input"][:50])
    return results

def call_llm(prompt: str, user_input: str) -> str:
    response = openai.ChatCompletion.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": user_input}
        ]
    )
    return response.choices[0].message.content

A single failing golden often exposes an entire class of problems. The prompt might handle short inputs but fail on long ones. It might refuse appropriately on obviously adversarial queries but be too eager on borderline ones. Reading the failures tells you where the prompt actually breaks, which is far more useful than reading a summary score.

Unit Evals and Regression Evals

Split your evals into two levels. Unit evals test individual pieces in isolation. If your pipeline has a retrieval step, test whether it returns relevant chunks before you call the generation model. If your pipeline classifies intent before routing, test classification accuracy separately. Unit evals run fast and tell you exactly which component failed.

Regression evals test the full pipeline end to end. You run every golden through the complete flow and score the final output. These are the tests that catch prompt changes breaking things that used to work. The Red Hat AI team ran 20 conversation traces through their eval pipeline every night and used CI to gate prompt changes. One failing golden blocked the merge.

# Regression eval runs the full pipeline
def regression_eval(prompt: str, goldens: list) -> dict:
    """Run full pipeline on all goldens. Returns pass rate."""
    passed = 0
    for golden in goldens:
        output = call_llm(prompt, golden["input"])
        test_case = LLMTestCase(
            input=golden["input"],
            expected_output=golden["expected"],
            actual_output=output
        )
        try:
            assert_test(test_case, [correctness_metric])
            passed += 1
        except AssertionError:
            pass
    return {"pass_rate": passed / len(goldens), "total": len(goldens)}

Run regression evals before every deployment. Run unit evals during development when you are changing a specific component. This distinction matters because full pipeline evals are expensive and slow. Unit evals are cheap and fast.

Putting It in CI

Prompt updates belong in pull requests, same as code. CI runs the eval suite before the merge. If any golden drops below threshold, the build fails. This is what makes prompt changes safe to ship.

Promptfoo was built for exactly this. You describe your test cases in YAML, point it at your prompts, and it runs the full suite:

prompts:
  - id: summarize_v1
    prompt: "Summarize the following text in one sentence: {{input}}"

tests:
  - id: finance_summary
    vars:
      input: "The Federal Reserve raised interest rates by 25 basis points."
    assert:
      - assertion: "contains"
        value: "0.25%"
      - metric: llm-rubric
        prompt: "Does the summary correctly convey the rate change?"
        score: 4

Running `promptfoo eval` produces a table showing exactly which tests pass and fail, by prompt version. You compare v1 against v2 before deciding what to ship. This is not optional once you have more than one person touching prompts. Without it, you have no record of why a prompt changed and no way to catch regressions before they reach production.

---

Ready to automate your development workflow? InterServer VPS plans include root access and scriptable environments for running eval pipelines, CI runners, and LLM serving on the same machine. Prices start at $5 per month.

Related InterServer guide: If you want AI agents to connect directly to InterServer hosting tools, read InterServer MCP Server: Connect AI Agents to Your Hosting Account.