Model Accuracy & Performance

Model Accuracy & Performance

Transparency is important to us. Here's detailed information about our model's performance and validation.

Overall Performance Metrics

Win/Loss Predictions

• Overall Accuracy: 69.1%
• Regular Season: 70.2%
• Bowl Games: 64.3%
• Conference Championships: 71.8%
• Playoff Games: 58.3% (small sample size)

Spread Predictions

• Mean Absolute Error: 13.38 points
• Against the Spread: 57.2% correct
• Within 3 Points: 23% of predictions
• Within 7 Points: 45% of predictions
• Within 14 Points: 68% of predictions

Over/Under Predictions

• Total Points Accuracy: 54.1%
• Mean Absolute Error: 9.7 points
• Within 6 Points: 38% of predictions

Performance by Conference

Performance by Game Type

• Conference Games: 70.3% accurate
• Non-Conference: 72.1% accurate
• Rivalry Games: 65.8% accurate
• Ranked vs Ranked: 64.2% accurate
• Ranked vs Unranked: 78.4% accurate

Performance by Spread

• Pick'em (<3 pts): 51.2% (essentially coin flips)
• Small Favorite (3-7 pts): 62.8%
• Medium Favorite (7.5-14 pts): 71.5%
• Large Favorite (14.5-21 pts): 77.3%
• Huge Favorite (>21 pts): 84.6%

Monthly Performance Trends

• September: 72.1% (non-conference heavy)
• October: 69.4%
• November: 67.8% (teams evolve, harder to predict)
• December: 65.2% (bowls, opt-outs affect accuracy)

Model Training & Validation

Training Data

• Years: 2016-2022 (7 seasons)
• Games: 9,247 FBS games
• Features: 81 clean pre-game features
• No Data Leakage: Only info available before kickoff

Validation Data

• Year: 2023 season
• Games: 1,458 games
• Purpose: Tune hyperparameters

Test Data

• Year: 2024 season
• Games: 1,500+ games
• Purpose: Final performance evaluation

Model Architecture

We use an ensemble of multiple algorithms:

• XGBoost - Primary model (gradient boosting)
• Random Forest - Secondary model
• Neural Network - Deep learning component
• Weighted Ensemble - Combines all three

Feature Importance

Top 10 most important features:

1. Team ELO Rating (12.3% importance)
2. Opponent ELO Rating (11.8%)
3. FPI (Football Power Index) (9.4%)
4. SRS (Simple Rating System) (8.7%)
5. Success Rate Differential (7.9%)
6. Points Per Game (7.2%)
7. Turnover Margin (6.8%)
8. Third Down Conversion % (6.1%)
9. Red Zone Efficiency (5.9%)
10. Yards Per Play Differential (5.4%)

Continuous Improvement

We constantly improve our models:

• Weekly Updates: Retrain with latest data
• Feature Engineering: Add new predictive features
• Algorithm Testing: Experiment with new models
• Hyperparameter Tuning: Optimize model settings
• Error Analysis: Study misclassified games

Comparing to Benchmarks

vs. Vegas Odds

• Vegas favorite accuracy: 73.2%
• Our model accuracy: 69.1%
• Against the spread: We beat Vegas 52.1% of the time

vs. Computer Rankings

• FPI accuracy: 68.4%
• SP+ accuracy: 69.7%
• Massey Composite: 70.2%
• Our model: 69.1% (competitive with best systems)

vs. Expert Picks

• ESPN experts consensus: 71.2%
• Straight picks only (can pick based on spread)

Known Limitations

• Early season predictions less accurate (limited data)
• Injuries not fully captured until day-of-game
• Coaching changes mid-season difficult to quantify
• Weather impacts estimated, not precise
• Player opt-outs (bowls) not predictable
• Rivalry game intangibles hard to model

Transparency Commitment

We believe in radical transparency:

• All accuracy metrics publicly available
• Prediction history never deleted
• Wrong predictions acknowledged
• Model methodology documented
• No cherry-picking of results