DescribeEvalOptions

Suite-level configuration for a harness-backed eval block.

Example

const options: DescribeEvalOptions<
  string,
  { status: "approved" | "denied" },
  { expected: { status: "approved" | "denied" } }
> = {
  harness: refundHarness,
  judges: [ToolCallJudge(), StructuredOutputJudge()],
  judgeThreshold: 1,
};

Type Parameters

TInput

TInput = unknown

TOutput

TOutput extends JsonValue | undefined = JsonValue | undefined

TMetadata

TMetadata extends HarnessMetadata = HarnessMetadata

THarness

THarness extends Harness<TInput, TOutput, TMetadata> = Harness<TInput, TOutput, TMetadata>

Properties

harness

harness: THarness

Harness used for every explicit run(...) call in the suite.

---

judgeHarness?

optional judgeHarness?: JudgeHarness

Optional judge-side harness used only by judges that call ctx.runJudge(...).

---

judges?

optional judges?: Judge<JudgeContext<TInput, TOutput, TMetadata, THarness>>[]

Automatic judges applied after each successful run(...).

---

judgeThreshold?

optional judgeThreshold?: number | null

Passing threshold for automatic suite-level judges. null disables fail-on-score.

---

skipIf?

optional skipIf?: () => boolean

Skips the entire eval suite when the predicate returns true.

Returns

boolean