createJudge

Call Signature

createJudge<TOptions>(name, assess): Judge<TOptions>

Creates a named judge object from an assessment function.

Type Parameters

TOptions

TOptions extends JudgeContext<any, any, any, any>

Parameters

name

string

Stable judge name shown in assertion messages and reports.

assess

JudgeAssessFn<TOptions>

Function that scores one normalized judge context.

Returns

Judge<TOptions>

Example

import { createJudge, type JudgeContext } from "vitest-evals";

type RefundOutput = { status: "approved" | "denied" };
type RefundMetadata = { expected: { status: RefundOutput["status"] } };

export const RefundStatusJudge = createJudge(
  "RefundStatusJudge",
  async ({ output, metadata }: JudgeContext<string, RefundOutput, RefundMetadata>) => ({
    score: output.status === metadata.expected.status ? 1 : 0,
    metadata: {
      rationale: `Expected ${metadata.expected.status}, got ${output.status}`,
    },
  }),
);

For LLM-backed judges, prefer the object form with ctx.runJudge(...) so provider-specific model configuration stays in the judge harness.

Call Signature

createJudge<TOptions>(config): Judge<TOptions>

Creates a named judge object from an assessment function.

Type Parameters

TOptions

TOptions extends JudgeContext<any, any, any, any>

Parameters

config

CreateJudgeConfig<TOptions>

Returns

Judge<TOptions>

Example

import { createJudge, type JudgeContext } from "vitest-evals";

type RefundOutput = { status: "approved" | "denied" };
type RefundMetadata = { expected: { status: RefundOutput["status"] } };

export const RefundStatusJudge = createJudge(
  "RefundStatusJudge",
  async ({ output, metadata }: JudgeContext<string, RefundOutput, RefundMetadata>) => ({
    score: output.status === metadata.expected.status ? 1 : 0,
    metadata: {
      rationale: `Expected ${metadata.expected.status}, got ${output.status}`,
    },
  }),
);

For LLM-backed judges, prefer the object form with ctx.runJudge(...) so provider-specific model configuration stays in the judge harness.

Call Signature

createJudge<TOptions, TInput, TOutput>(name, assessor, assess): Judge<TOptions>

Type Parameters

TOptions

TOptions extends JudgeContext<any, any, any, any>

TInput

TInput

TOutput

TOutput

Parameters

name

string

assessor

JudgeAssessor<TInput, TOutput>

assess

JudgeAssessWithAssessorFn<TOptions, TInput, TOutput>

Returns

Judge<TOptions>