Pi Harness

Use the Pi harness when your app exposes a Pi agent, toolset, or runtime-compatible run(input, runtime) entrypoint. The same Paris example applies here: the app answers the question, the harness normalizes the run, and the judge scores whether the output names Paris.

Install

Pi

pnpm add -D vitest-evals @vitest-evals/harness-pi-ai

App Shape

The agent factory is the app boundary. It can close over prompts and services, then expose the runtime-compatible run(input, runtime) shape the harness expects.

src/questionAgent.ts

type PiRuntime = {
  events?: {
    assistant(content: string): void;
  };
};

export function createQuestionAgent({
  instructions,
}: {
  instructions: string;
}) {
  return {
    async run(input: string, runtime: PiRuntime) {
      const output = await answerQuestion(input, { instructions });

      runtime.events?.assistant(output);

      return {
        output,
      };
    },
  };
}

Configure Harness

The harness creates the agent for each run and can derive app setup from the input or close over shared services.

evals/qaHarness.ts

import { piAiHarness } from "@vitest-evals/harness-pi-ai";
import { createQuestionAgent } from "../src/questionAgent";

export const qaHarness = piAiHarness({
  agent: ({ input }) =>
    createQuestionAgent({
      instructions: buildInstructions(input),
    }),
});

Return { output } from the Pi entrypoint when tests and judges should assert on a parsed domain value. The harness attaches native run/model spans to result.traces when runtime usage data is available. The Pi runtime event stream and wrapped tool calls are the transcript source for message and tool-call assertions.

Writing Evals

Write the eval against the harness, not the Pi runtime directly. The judge reads the normalized output, so the scoring code stays independent of Pi internals.

evals/capital.eval.ts

import { expect } from "vitest";
import { createJudge, describeEval } from "vitest-evals";
import { qaHarness } from "./qaHarness";

const CapitalJudge = createJudge<string, string>(
  "CapitalJudge",
  async ({ output }) => {
    const passed = output.toLowerCase().includes("paris");

    return {
      score: passed ? 1 : 0,
      metadata: {
        rationale: passed
          ? "The answer names Paris."
          : `Expected Paris, got: ${output}`,
      },
    };
  },
);

describeEval("capital questions", { harness: qaHarness }, (it) => {
  it("knows the capital of France", async ({ run }) => {
    const result = await run("What is the capital of France?");

    expect(result.output).toContain("Paris");
    await expect(result).toSatisfyJudge(CapitalJudge);
  });
});