Skip to content
FixedAgent-ready

Pricing experiments now fail loudly instead of quietly serving one price

Experiments that could never run are rejected when you create them, and the traffic split you configure is now applied.

A pricing experiment could previously run to completion while showing every visitor the same price, and still look healthy on the results page. That happens no longer.

  • Creating an experiment now fails with a clear message unless it has exactly two variants named control and experiment. Those are the only two arms the assignment engine can assign, so any other setup recorded its results under those two names while serving prices from arms you had named something else.
  • If an experiment somehow reaches the pricing endpoint with an arm that cannot be served, the request now returns an error instead of silently falling back to the first variant. No view or revenue signal is recorded for a price that was never shown.
  • The per-variant weight you set is now honoured. Previously every experiment ran on a fixed even split regardless of the weights you configured. Set 70 and 30 and traffic now splits 70/30; leave both unset to keep the even split.

If you are running an experiment whose variants are not named control and experiment, its results to date are not valid and should be discarded. The price a visitor saw did not reliably match the arm their result was recorded under, and depending on the order the variants were stored, one arm's price may have reached nobody at all while the results page still reported figures for it. Where neither arm carried a supported name, every visitor saw the same price. Those experiments now return an error instead of a price, so recreate them with the supported names to collect real data.

Separately, if an experiment set a weight on only one of its two arms, it has been running on an even split rather than the split you configured, because the weight was never applied. It keeps serving, and you can correct it by setting a weight on both arms or on neither. Testing more than two prices at once is not supported yet.