Harnesses

A harness is the boundary between production behavior and eval infrastructure. It should be thin: call the same app entrypoint you use in production, then return the output, ordered transcript events, optional traces, and usage that tests and judges need. Tool-call helpers derive their results from the transcript.

Use the first-party adapter that matches the runtime your app already uses. Use createHarness() when the app is not built on a supported SDK or when you need full control over normalized run data.

Available Harnesses

AI SDKUse when your app calls generateText, streamText, or an AI SDK wrapper.OpenAI AgentsUse when your app owns an Agent and runs it with a Runner.PiUse when your app exposes a Pi agent, toolset, or runtime-compatible entrypoint.Custom HarnessesUse for workflows, service functions, CLIs, RAG pipelines, and custom agents.

Normalized Results

Every harness returns a JSON-serializable result. Judges and reports read the same shape regardless of runtime:

Field	Purpose
output	The domain value your tests usually assert on.
session.events	Ordered transcript events, including messages, tool-call requests, and tool results.
toolCalls(result)	Deterministic tool-call projection for tool judges and replay checks.
traces	Operation spans for runs, model calls, tools, guardrails, handoffs, or custom work.
usage	Stable usage units such as provider, model, tokens, tools, and retries.

First-party harnesses attach trace spans automatically from native runtime data they observe. createHarness() attaches a fallback run span when a custom harness does not return traces itself. Span attributes are JSON-safe and include typed OpenTelemetry GenAI semantic keys where they apply.