After 6 months of work, I’m finally more or less done with version 3. Currently it’s testing at 64% accuracy, .65 logloss on the latest 10% of UFC fights that the model has never seen.
Updates
- Async scrapers of fight data
- Thorough unit tests to prevent hidden bugs
- Better feature engineering leading to 81 high quality, highly predictive features
- Complex custom Elo scoring
- Ensemble of top 6 models
- MMA-AI.net is now an ensemble of the best 3 XGBoost models, and the best 3 CatBoost models
Custom Elo
The Elo scoring system was overhauled. It now has 17 custom scoring weights that affect the Elo score modification with everything from which round the fight ended in to the age bracket of the fighter. For instance, a KO of any time adds 2.4 score. A 1st round KO adds another 1.8 score. These scores are added up and put into a formula which affects the final increase or decrease of each fighter’s Elo. Additionally, the age bracket of the fighter is accounted for. Older fighters gain less Elo per win, and prior to each fight prediction every fighter’s Elo is adjusted for their age and days since their last competition, so a 45 year old fighter who hasn’t fought in 5 years will take a big hit to their Elo prior to their stats being digested by the models.
These scoring weights were fine tuned using hyper parameter optimization, they’re not arbitrarily assigned by my opinion. One interesting thing that came out of this is that submission victories were significantly lower utility in creating an accurate Elo score. A KO of any kind scores 2.4. A submission of any kind scores only 1.62. The Elo score feature alone is about 61% accurate in predicting which fighter will win.
Training
The data is split into train/validation/test. 80% of all fights from 2014 onward are used for training. The next latest 10% of fights is used for validation of the training and the final most recent 10% of fights is used for testing. This final 10% of fights were never seen by the model and only used to check how the model performs against unseen, recent fights. How the model performs on this test set is how we’re identifying the model’s accuracy and logloss.
Hyperparameter optimization is performed with a 5 fold time series split. The data is broken up into 5 consecutive groups ordered oldest fights to latest fights. The model is trained using the first block of fights then validated against the next block of fights in a rolling manner. More recent fights are given more weight in the algorithm so the earliest fights get a weight of 1 and the latest fights get a weight of 1.5 or 2 making them 50-100% more important to get correct for the model. This helps the model identify hyperparameters that are better at predicting the most recent fights which is what we’re trying to accomplish with a profitable betting model.
The model is trained to optimize only logloss. Logloss is the confidence the model has in it’s prediction. It is a significantly better metric than accuracy when trying to be profitable in betting. This is because you can bet the favorite 100% of the time and be 66% accurate, but you’ll lose money. However, you could be only 50% accurate betting heavy underdogs yet if they’re smart bets then you’ll be significantly profitable. So the model is designed to identify mismatches in the win percentage the odds dictate versus the real life win percentage of each fighter. You can use a strategy like Kelley Criterion and only bet on fighters that the model predicts as a winner that also have >5% difference between the betting odds win percentage and the model’s predicted win percentage. Example: Fighter1’s odds say he’ll win 50% of the time. His odds are -100. The model predicts he’ll win 55% of the time. If this is the case then we throw our max bankroll percentage that we’re comfortable with on this fight as it’s a profitable opportunity. I usually limit my max bankroll percentage to about 5-10% of money available.
Limitations
This model does not make predictions on any fights where one of the contestants has never fought in the UFC. Additionally, it does not predict women’s fights as they tend to be much more unpredictable and their fight stats poison the well when looking for generalized patterns to apply to the more common men’s fights.