OpenAI Agents Harness

Use the OpenAI Agents harness when your app already owns an Agent and runs it with a Runner. This page keeps the same Paris example as the other harnesses: configure the app once, wrap it with the harness, then score the harness result with a tiny judge.

Install

OpenAI Agents

pnpm add -D vitest-evals @vitest-evals/harness-openai-agents @openai/agents

App Shape

The app should expose the same Agent production code runs. Keep instructions, tools, handoffs, and structured output policy here.

src/questionAgent.ts

import { Agent } from "@openai/agents";

export function createQuestionAgent() {
  return new Agent({
    name: "qa-agent",
    instructions:
      "Answer geography questions directly. Keep answers short.",
  });
}

Configure Harness

The harness supplies the agent, runner defaults, turn limits, and output parser for every eval in the suite.

evals/qaHarness.ts

import { Runner } from "@openai/agents";
import { openaiAgentsHarness } from "@vitest-evals/harness-openai-agents";
import { createQuestionAgent } from "../src/questionAgent";

export const qaHarness = openaiAgentsHarness({
  agent: () => createQuestionAgent(),
  runner: () =>
    new Runner({
      tracingDisabled: true,
    }),
  runOptions: {
    maxTurns: 3,
  },
  output: ({ result }) => String(result.finalOutput ?? "").trim(),
});

Use output when finalOutput needs validation or conversion before Vitest assertions and judges read result.output. The harness also attaches native run/model spans to result.traces when provider data is available. Tool-call transcript data comes from OpenAI Agents generated items: newItems first, then output only when it contains recognizable SDK model-output items. SDK history is used as a complete-history source when no generated items are available. Runtime tool wrappers enrich matching run items and provide transcript events for custom entrypoints that return no provider items.

Writing Evals

Run the same question a user would ask the agent. CapitalJudge scores the normalized output from that run; it does not run the agent a second time.

evals/capital.eval.ts

import { expect } from "vitest";
import { createJudge, describeEval } from "vitest-evals";
import { qaHarness } from "./qaHarness";

const CapitalJudge = createJudge<string, string>(
  "CapitalJudge",
  async ({ output }) => {
    const passed = output.toLowerCase().includes("paris");

    return {
      score: passed ? 1 : 0,
      metadata: {
        rationale: passed
          ? "The answer names Paris."
          : `Expected Paris, got: ${output}`,
      },
    };
  },
);

describeEval("capital questions", { harness: qaHarness }, (it) => {
  it("knows the capital of France", async ({ run }) => {
    const result = await run("What is the capital of France?");

    expect(result.output).toContain("Paris");
    await expect(result).toSatisfyJudge(CapitalJudge);
  });
});