Evaluation & Metrics

Define metrics and test cases, then score any version — manually, in a batch, or automatically on every save.

Test cases

A test case is a named set of variable values plus (optionally) an expected output. You can add them by hand or import them in bulk, and export them again later.

Metric types

There are seven built-in metric types. Each scores a run as pass/fail (1.0 or 0.0):

MetricPasses when…
keyword_matchthe output contains the configured keyword(s).
json_validthe output parses as valid JSON.
max_lengththe output is at or under a length limit.
regex_matchthe output matches a regular expression.
exact_matchthe output equals the expected value exactly.
contains_expectedthe output contains the test case's expected output.
llm_judgea separate LLM call judges the output as acceptable.

Batch evaluation

Run all metrics against all test cases for a chosen version in one go. Results are stored per (test case × version × metric), and a summary { score, total, passed }is rolled up so you can see at a glance how a version performs.

Auto-eval on save & regression warnings

When saving a version, tick “Run regression tests after save” and pick an eval provider and model. PromptVersions runs the batch eval in the background and writes the summary back to the version. The History tab polls until it's ready and shows a regression warning if the new version's score dropped compared to the previous one.

Guardrails
Evaluation has safety limits: regex patterns are capped in length and time, outputs are capped in size, and keyword lists are bounded — so a misconfigured metric can't hang an eval.
PreviousRunning & ComparingNextPrompt Optimizer