createJudge

Call Signature

createJudge<TOptions>(name, assess): Judge<TOptions>

Creates a named judge object from an assessment function.

Type Parameters

TOptions

TOptions extends JudgeContext<any, any, any>

Parameters

name

string

Stable judge name shown in assertion messages and reports.

assess

JudgeAssessFn<TOptions>

Function that scores one normalized judge context.

Returns

Judge<TOptions>

Example

import { createJudge } from "vitest-evals";

type RefundOutput = { status: "approved" | "denied" };

export const RefundStatusJudge = createJudge<
  string,
  RefundOutput,
  { expectedStatus: RefundOutput["status"] }
>(
  "RefundStatusJudge",
  async ({ output, expectedStatus }) => ({
    score: output.status === expectedStatus ? 1 : 0,
    metadata: {
      rationale: `Expected ${expectedStatus}, got ${output.status}`,
    },
  }),
);

For LLM-backed judges, prefer the object form with ctx.runJudge(...) so provider-specific model configuration stays in the judge harness.

Call Signature

createJudge<TOptions>(config): Judge<TOptions>

Creates a named judge object from an assessment function.

Type Parameters

TOptions

TOptions extends JudgeContext<any, any, any>

Parameters

config

CreateJudgeConfig<TOptions>

Returns

Judge<TOptions>

Example

import { createJudge } from "vitest-evals";

type RefundOutput = { status: "approved" | "denied" };

export const RefundStatusJudge = createJudge<
  string,
  RefundOutput,
  { expectedStatus: RefundOutput["status"] }
>(
  "RefundStatusJudge",
  async ({ output, expectedStatus }) => ({
    score: output.status === expectedStatus ? 1 : 0,
    metadata: {
      rationale: `Expected ${expectedStatus}, got ${output.status}`,
    },
  }),
);

For LLM-backed judges, prefer the object form with ctx.runJudge(...) so provider-specific model configuration stays in the judge harness.

Call Signature

createJudge<TInput, TOutput, TOptions, THarness>(name, assess): Judge<CreateJudgeContext<TInput, TOutput, TOptions, THarness>>

Creates a named judge object from an assessment function.

Type Parameters

TInput

TInput

TOutput

TOutput extends JsonValue | undefined

TOptions

TOptions extends object = Record<never, never>

THarness

THarness extends Harness<TInput, TOutput> | undefined = Harness<TInput, TOutput> | undefined

Parameters

name

string

Stable judge name shown in assertion messages and reports.

assess

JudgeAssessFn<CreateJudgeContext<TInput, TOutput, TOptions, THarness>>

Function that scores one normalized judge context.

Returns

Judge<CreateJudgeContext<TInput, TOutput, TOptions, THarness>>

Example

import { createJudge } from "vitest-evals";

type RefundOutput = { status: "approved" | "denied" };

export const RefundStatusJudge = createJudge<
  string,
  RefundOutput,
  { expectedStatus: RefundOutput["status"] }
>(
  "RefundStatusJudge",
  async ({ output, expectedStatus }) => ({
    score: output.status === expectedStatus ? 1 : 0,
    metadata: {
      rationale: `Expected ${expectedStatus}, got ${output.status}`,
    },
  }),
);

For LLM-backed judges, prefer the object form with ctx.runJudge(...) so provider-specific model configuration stays in the judge harness.

Call Signature

createJudge<TInput, TOutput, TOptions, THarness>(config): Judge<CreateJudgeContext<TInput, TOutput, TOptions, THarness>>

Creates a named judge object from an assessment function.

Type Parameters

TInput

TInput

TOutput

TOutput extends JsonValue | undefined

TOptions

TOptions extends object = Record<never, never>

THarness

THarness extends Harness<TInput, TOutput> | undefined = Harness<TInput, TOutput> | undefined

Parameters

config

CreateJudgeConfig<CreateJudgeContext<TInput, TOutput, TOptions, THarness>>

Returns

Judge<CreateJudgeContext<TInput, TOutput, TOptions, THarness>>

Example

import { createJudge } from "vitest-evals";

type RefundOutput = { status: "approved" | "denied" };

export const RefundStatusJudge = createJudge<
  string,
  RefundOutput,
  { expectedStatus: RefundOutput["status"] }
>(
  "RefundStatusJudge",
  async ({ output, expectedStatus }) => ({
    score: output.status === expectedStatus ? 1 : 0,
    metadata: {
      rationale: `Expected ${expectedStatus}, got ${output.status}`,
    },
  }),
);

For LLM-backed judges, prefer the object form with ctx.runJudge(...) so provider-specific model configuration stays in the judge harness.

Call Signature

createJudge<TOptions, TInput, TOutput>(name, assessor, assess): Judge<TOptions>

Type Parameters

TOptions

TOptions extends JudgeContext<any, any, any>

TInput

TInput

TOutput

TOutput

Parameters

name

string

assessor

JudgeAssessor<TInput, TOutput>

assess

JudgeAssessWithAssessorFn<TOptions, TInput, TOutput>

Returns

Judge<TOptions>