Custom Harnesses

Use createHarness() when the first-party runtime adapters do not fit. The app can be a workflow, service function, CLI wrapper, RAG pipeline, or custom agent. This page uses the same Paris example as the runtime harnesses so the only new idea is how to normalize your own app result.

Install

Custom Harness

pnpm add -D vitest-evals

App Shape

Start with the production entrypoint. It should do real app work and return the small value tests and judges care about.

src/questionFlow.ts

export async function answerQuestion(input: string): Promise<string> {
  const response = await appAssistant.respond(input);

  return response.text.trim();
}

Configure Harness

createHarness() adapts that production entrypoint into normalized eval data. Return output for assertions and judges, and always return at least one ordered transcript event for the run. Strict camelCase messages are also accepted at this boundary and are normalized into events, including assistant toolCalls, separate role: "tool" results keyed by toolCallId, and AI SDK-style tool-call/tool-result content parts. If another provider uses content blocks, item streams, or snake_case wire fields, adapt those into events or the camelCase message transport directly. Attach usage and artifacts when reports need them. It automatically attaches a fallback run span unless the harness returns its own traces.

evals/qaHarness.ts

import { createHarness } from "vitest-evals";
import { answerQuestion } from "../src/questionFlow";

export const qaHarness = createHarness<string, string>({
  name: "qa-app",
  run: async ({ input, setArtifact }) => {
    const output = await answerQuestion(input);

    setArtifact("question", { input });

    return {
      output,
      events: [
        { type: "message", role: "user", content: input },
        { type: "message", role: "assistant", content: output },
      ],
      usage: {
        provider: "openai",
        model: "gpt-4o-mini",
      },
    };
  },
});

Return a full HarnessRun only when you need complete control over the session, usage, timings, artifacts, traces, and errors. Full runs must already use canonical session.events; createHarness() is the preferred custom harness entrypoint when you want lightweight normalization.

Writing Evals

Use the custom harness like any first-party harness. The judge scores the normalized output from the single harness run.

evals/capital.eval.ts

import { expect } from "vitest";
import { createJudge, describeEval } from "vitest-evals";
import { qaHarness } from "./qaHarness";

const CapitalJudge = createJudge<string, string>(
  "CapitalJudge",
  async ({ output }) => {
    const passed = output.toLowerCase().includes("paris");

    return {
      score: passed ? 1 : 0,
      metadata: {
        rationale: passed
          ? "The answer names Paris."
          : `Expected Paris, got: ${output}`,
      },
    };
  },
);

describeEval("capital questions", { harness: qaHarness }, (it) => {
  it("knows the capital of France", async ({ run }) => {
    const result = await run("What is the capital of France?");

    expect(result.output).toContain("Paris");
    await expect(result).toSatisfyJudge(CapitalJudge);
  });
});