2 minute readNewAgent-ready
Forge only keeps a new pricing model that beats luck
A new pricing model now has to beat the current one by more than luck alone could, and the Forge card shows that bar next to each candidate's gain.
Forge writes new pricing models as code, tests each on a pricing test bench against the current model, and keeps the ones that win. Try enough candidates and one will look better by chance, so a win now has to clear a luck bar.
- The bar is the gain a candidate with no real edge could reach from test noise alone. Noise is measured as how each market's gain moves from one test run to the next.
- It rises with every candidate tried against the same current model, counted across nightly runs too, so trying many cannot buy a pass. The chance that any no-edge candidate passes stays at 5% however many are tried.
- Each candidate is now tested on 4 runs per market instead of 2.
- The Forge card on the Learned pricing page draws a bar per candidate, with a caption such as "Gain 2.31 points; luck alone reaches 1.03 points after 2 tries. Clear of luck."
- In a real run of 36 candidates, the real win (2.31 points, mean regret down from 6.20% to 4.84% on held-back tests) still passed. A 0.32 point gain that the old fixed 0.25 point margin would have let through was held back as within luck, against a 0.79 point bar.
- In a simulation of candidates with no edge, the old fixed margin let something through in over 70% of runs that tested 6 candidates. The luck bar holds that under 10%.
