Compare AI models

Two models, one window, one asset class, one horizon. Accuracy is shown with a 95% confidence interval, because a rate without one cannot be compared to another rate.

Rule-Based Analysis

rule-based-analysis
33.6%
95% CI 33.0% – 34.1%
  • Sample size31175
  • Correct10460
  • Mean stated confidence 36.3%
  • Confidence gap 2.7 pts
  • Symbols covered 515 / 537
  • Coverage of the field 95.9%
  • Horizons used2
Calibration — stated confidence against measured accuracy
Confidence n Accuracy
0–50% 26541 32.6%
50–70% 3196 42.0%
70–85% 853 28.5%
85–100% 585 38.1%

FinBERT

huggingface-prosusai/finbert
56.1%
95% CI 54.1% – 58.1%
  • Sample size2384
  • Correct1337
  • Mean stated confidence 94.0%
  • Confidence gap 37.9 pts
  • Symbols covered 29 / 537
  • Coverage of the field 5.4%
  • Horizons used1
Calibration — stated confidence against measured accuracy
Confidence n Accuracy
0–50% 0 —
50–70% 0 —
70–85% 0 —
85–100% 2384 56.1%

Over the last 30 days the two confidence intervals do not overlap, so this sample does separate the models.

Accuracy counts a prediction correct when the direction it stated matches the direction the price moved past a 1% threshold over the stated horizon. Coverage is the share of symbols scored in this window that the model expressed an opinion on — a high rate over four symbols is not the same claim as the same rate over ninety. Full method and disclosures