Did I announce MMA-AI v2.0 months ago? Yes. Did I announce that too early? Sure did.
Originally, my intention was to release MMA-AI 2.0 with a model that was trained on profitability rather than accuracy. I discussed this idea with Chris from wolftickets.ai and he got a stellar implementation working almost immediately which is available to his subscribers (it’s only $5 a month, I recommend it). I am not as versed in machine learning as he is and it took me weeks to even get a beta model out when I originally announced version 2.0. However, the models I was producing though were not of the quality I expected and I haven’t updated the site or predictions since. This is because I went back to the drawing board for the last several months to work on the basics of machine learning: model evaluation, feature selection, reducing overfitting, and hyperparameter optimization.
After hundreds of hours of work and study, I took a different direction and am seeing very high quality results. Instead of doing a model which trains itself on profitability of each prediction, I turned to tuning my model for logloss + accuracy. Logloss is essentially the model’s confidence in it’s prediction. Good logloss models are good at setting accurate real world win % chances for fighters. The idea behind this is that a logloss model which can provide realistic win % chances for fighters can be used with the Kelley Criterion to determine how much to bet on a fighter based on the difference between the AI predicted odds and the real world odds. The combination of quality predictions + quality betting strategy has sent my test profits through the roof compared to my older models.
I completely revamped how I was doing feature selection and hyperparameter optimization. For a long time I thought more features = more accuracy. Chris alerted me to the fact that this is not the case; in general more features = more likelihood of a model that is overfit to the training data and bad at making predictions on unseen data. Overfitting is when you train the model to be amazing at predicting the fights that it’s trained to predict, and bad at predicting future fights. So I used my domain expertise in MMA, as a fan and fighter for a decade and a half, to pare down the features to much smaller selection that I felt would directly impact whether a fighter is winning or losing a fight. After that, I went on a deep study course in hyperparameter optimization and model evaluation to make sure I never overfit my models again. The culmination of this is my latest model.
The latest model when combined with the Kelley Criterion (KC) betting strategy is showing hundreds of percent profit gain when tested against the last 365 days of fights that the model has never seen before. The profit numbers are frankly absurd which makes me concerned I’m making a mistake somewhere but I’ve implemented stronger unit testing to fight bugs and I’ve poured over every test prediction it’s made to confirm no data is leaking and it’s calculating the numbers correctly. Either way, the basic machine learning tests I’m doing are coming back much much better than any of my previous models and I’m given significant confidence that my future model pipeline will be as devoid of overfitting on the training data as possible. See the homepage for new model details.