A/B Testing

Split live traffic between two versions to compare them on real usage — with live stats on runs, latency, error rate, and cost.

How it works

Start an A/B test by choosing two versions (variant A and variant B) and a traffic split for B. When an inference call resolves to the prod/live version that matches variant A, the test randomly routes a share of that traffic to variant B instead.

• One active test per prompt (and per chain) at a time — enforced in app logic and by a database constraint.
• Traffic split — set the probability that a request is routed to variant B.
• Live stats — runs, error rate, latency, and cost for each variant, computed from logged runs within the test's time window.

Lifecycle