Compare AI models

Two models, one window, one asset class, one horizon. Accuracy is shown with a 95% confidence interval, because a rate without one cannot be compared to another rate.

Rule-Based Analysis

rule-based-analysis
33.5%
95% CI 32.9% – 34.0%
  • Sample size29268
  • Correct9795
  • Mean stated confidence 35.1%
  • Confidence gap 1.7 pts
  • Symbols covered 307 / 336
  • Coverage of the field 91.4%
  • Horizons used1
Calibration — stated confidence against measured accuracy
Confidence n Accuracy
0–50% 26341 33.0%
50–70% 1742 39.7%
70–85% 591 31.5%
85–100% 594 39.1%

FinBERT

huggingface-ProsusAI/finbert
55.6%
95% CI 53.6% – 57.6%
  • Sample size2407
  • Correct1338
  • Mean stated confidence 94.0%
  • Confidence gap 38.4 pts
  • Symbols covered 29 / 336
  • Coverage of the field 8.6%
  • Horizons used1
Calibration — stated confidence against measured accuracy
Confidence n Accuracy
0–50% 0 —
50–70% 0 —
70–85% 0 —
85–100% 2407 55.6%

Over the last 30 days the two confidence intervals do not overlap, so this sample does separate the models.

Accuracy counts a prediction correct when the direction it stated matches the direction the price moved past a 1% threshold over the stated horizon. Coverage is the share of symbols scored in this window that the model expressed an opinion on — a high rate over four symbols is not the same claim as the same rate over ninety. Full method and disclosures