Compare AI models

Two models, one window, one asset class, one horizon. Accuracy is shown with a 95% confidence interval, because a rate without one cannot be compared to another rate.

Rule-Based Analysis

rule-based-analysis
12.2%
95% CI 11.5% – 12.9%
  • Sample size8410
  • Correct1023
  • Mean stated confidence 32.0%
  • Confidence gap 19.9 pts
  • Symbols covered 1020 / 1032
  • Coverage of the field 98.8%
  • Horizons used1
Calibration — stated confidence against measured accuracy
Confidence n Accuracy
0–50% 8058 11.4%
50–70% 190 38.9%
70–85% 42 14.3%
85–100% 120 22.5%

Mistral Small Latest

mistral-small-latest
36.4%
95% CI 33.6% – 39.3%
  • Sample size1071
  • Correct390
  • Mean stated confidence 74.5%
  • Confidence gap 38.1 pts
  • Symbols covered 138 / 1032
  • Coverage of the field 13.4%
  • Horizons used2
Calibration — stated confidence against measured accuracy
Confidence n Accuracy
0–50% 20 55.0%
50–70% 380 45.3%
70–85% 247 38.1%
85–100% 424 26.7%

Over the last 30 days the two confidence intervals do not overlap, so this sample does separate the models.

Accuracy counts a prediction correct when the direction it stated matches the direction the price moved past a 1% threshold over the stated horizon. Coverage is the share of symbols scored in this window that the model expressed an opinion on — a high rate over four symbols is not the same claim as the same rate over ninety. Full method and disclosures