Right around Memorial Day of this year, I became focused on trying to create a game prediction model for MLB games. I had attempted this in the past with limited success, but I felt that now was a good time to try again. There is an abundance of publicly available data to use, and building prediction models is something I love to do (see March Madness). Within a week, I had created a model that I decided to track for the rest of the season. And, that is what I did. I input the teams, starting pitchers, and odds for all 1,557 games from May 31st to the last regular season games on September 27th. Each day, my data was refreshed so that all games were run through the model accurately. Before I get into the results, I want to explain what the model is, and what it is trying to accomplish.
The Model
So, let’s begin by examining the inputs of the model. For a given game, I enter the two teams (away and home), and each team’s starting pitchers. The model pulls the starting pitcher’s average length of outing (some teams use relievers to start games, and thus it is better to use innings per appearance rather than innings per start). This also gives the bullpen its expected innings for this game (9 innings – Starter Average Innings / Game). I use a special blend of pitching statistics to give the starter and the bullpen their expected number of runs allowed per 9 innings. This blend is then adjusted for the amount of innings expected for the starter and the bullpen. This is the basis for the “Pitcher Runs” for that game. For “Batter Runs,” I use teams’ wRC+ and scale it to league average runs. But to make it more in-depth, I use a team’s wRC+ vs Righties (if the opposing starting pitcher is right-handed), and wRC+ vs Lefties (if the opposing starting pitcher is left-handed). For the bullpen portion, I simply use the teams’ total wRC+. This gives an expected number of “Batter Runs” per team for that specific game. Also involved in “Batter Runs,” is Fielding Run Value, to measure the defensive effects of the opposing team. To find the expected final score of the game, I use a weighted average formula with Pitching Runs and Batter Runs. And finally, I use Park Factor to scale the final result to the park that it is being played in (Playing at Coors Field leads to a lot more runs than at Globe Life Field).
Without giving away too much, that is the model. It gives a predicted final score for any game between any two pitchers. As a game prediction model, I wanted to use odds to help track its success. I input each team’s moneyline odds, as well as the game’s total. This gave me more data to track the success of my model. i was focused on tracking the following statistics/categories: moneyline win percentage, 10% expected value win percentage, predicted underdog win percentage, and total (o/u) win percentage. These are the main four categories, but the winning percentage does not tell the full story. I wanted to track profit/loss based on a standard unit. This is much more telling in determining the success of a model. Now let’s get into the results:
Results
Predicting the Winner of Each Game
I am going to start with the most fundamental thing this model was built to predict: the winning team. Yes, over/unders are inherent to the model as well, but the only important thing in real baseball is winning the game. Did my model do a good job of accurately predicting the winning team? I would say “yes.” Here are its results compared to two less-complex methods of predicting the winning team:
As shown above, my model was better than the oddsmakers at predicting the winning team in a given game. The Karn Model won 57.48% of its games, whereas picking the favored team only won 56.82% of the team. And although that might seem small, look the the next column to see how big of a difference that really is. The Karn Model was over 63 units better than simply betting one unit on the team favored to win. This is not totally unexpected though. Because you are betting a standard one unit on every game, the team that is favored to win will always return less than 1 unit back. This means that the breakeven win percentage is much higher because there are no underdogs to return over 1 unit per win. The Home Team Only method performed the worst of the three models by far. Not only did it have the lowest win percentage, it also lost the most units.
Some may say that my model was a failure because it lost units over the course of the season (May 31-End of Regular Season), but I would disagree for two main reasons. First, putting one unit on all 1,557 games is not realistic. Secondly, it discounts the fact that the model can favor some games more than others. This is the basis for the 10% Expected Value category.
10% Expected Value
As I mentioned above, I needed a way to sort which games that model favored the most. To accomplish this, I used an adjusted Pythagorean Win Expectancy formula. Because this formula is meant to project a winning percentage across a whole season’s worth a runs and runs allowed, and my model only predicts the score of a single game, I had to increase the exponents to 3.5. This felt like a good balance of creating separation in win probability between the two teams. I did some testing in May and June, and exponents of 3.5 performed the best. This gave me my win probability per game. When compared to the implied win probability from the sportsbook odds, I could then use the expected value formula to see which games were more favorable than others. From there, I isolated only the expected value percentage of the team my model projected to win. There is no point in positive expected value in a team that was not projected to win the game. I then tracked all games in which the projected winning teams’ expected value was above 10%.
This is really where the model shined. Out of the 1,557 games, there were 350 in which the expected value (from here on out, “of the winning team” is implied) was above 10%. That’s roughly 22.5% of all games. Of the 350 games, the model correctly predicted the winner in 203 of them, for a winning percentage of 58.00%. While that isn’t significantly above the winning percentage of predicting every game, the margins were far better in this section. If a bettor placed a 1 unit bet on all 350 games, they would have finished the season with a profit of 32.15 units. Not only is that a very strong return, but it is also much more realistic for actual bettors.
You might be asking, “how is it possible that only around half of a percentage point can make such a big difference in the profit/loss of each model?” The answer: “Return.” Let’s look at presumptive AL Cy Young Award Winner: Cam Schlittler and the New York Yankees. Schlittler pitched in 20 games while I was tracking games. The Yankees won 11 of those games. But even though the Yankees won more than they lost in which he was the starting pitcher, they had a profit/loss of -2.51 units. This is because the Yankees were favored in all 20 games, with average odds of -177. Meanwhile, Boston Red Sox won 13 of Sonny Gray’s 19 games on the season, and his games netted a profit of 3.69 units. The reason is because Gray was only favored by more than -177 (Schlittler’s average) three times all season. Another example is Cade Cavalli from the Washington Nationals. He was even less of a favorite than Sonny Gray: only favored -177 or more once in 20 games. The Nationals won 13 of Cavalli’s 20 starts, and netted bettors a profit 4.38 units with 1 unit bets on all 20 games. The Nationals were the underdog in half of his starts. This shows that there is profitability in being able to predict underdog teams accurately.
Predicted Underdogs
That brings us to the Predicted Underdogs section of my model. To measure this, I only counted games in which the team with “plus” odds (+100 or higher) was predicted to win the game by my model. This does not count the games in which the underdog team had odds lower than their opponent but still “minus” odds. An example of a game that would not be counted is if Team A was favored to team Team B with odds -112 for Team A and -108 for Team B. So in total, of the 1,557 games, only in 192 of them was a “plus” underdog predicted to win. That’s a percentage of 12.33%, so we can say roughly 1 out every 8 games, a plus underdog was predicted to win. In those 192 games, the predicted plus underdog won 93 games, a win percentage of 48.44%. Now on the surface, this does not look great: winning less than 1 out of every 2 games. However, because every team predicted to win in this section has plus odds, every win is worth more than 1 unit. This means that the breakeven point is much lower than with all the games included. In fact, if a bettor had placed 1 unit on all 192 predicted plus underdogs to win, he would have profited 6.3 units. I see this is an absolute success. Being able to predict underdog victories is difficult, and being able to profit off of them is even more difficult. This is definitely a plus for my model.
Totals (Over/Under)
Now here is where the model struggled. Predicting totals can be difficult because of sample sizes and averages. The model uses season-long averages to predict the results in one game. This can be a flawed concept because anything can happen in one game. However, as the season progresses that we should get closer and closer to the true averages. If a game was played 10,000 times, we should see the average of the results get close to my model’s prediction. When predicting the winner of the game, it does not matter how many runs score, or how much a team wins by, it is only about the team that wins the game. With totals, every run matters. There are a wider outcome of results, which makes it tougher to predict. Two other reasons that totals can be tough to predict in baseball: weather and park factor. Sometimes the wind will be blowing in, usually leading to fewer runs, or it will be colder, also usually leading to fewer runs. The weather has a big impact on the total. Additionally, the location of a game matters a great deal as well. The altitude at Coors Field leads to more runs than a park at sea level and on the water like Oracle Park in San Francisco. I used BallparkPal.com for the Park Factors of every game throughout this process. It combines weather and park effects to give each game a park factor. The park factor predicts how many more or fewer runs will be scored in that park than on average. This was perfect to help tailor my season-long averages to each individual game. Unfortunately, the results were not as good as I had hoped for. Of the 1,557 games I tracked, I correctly predicted the total in only 801 of them, a winning percentage of 51.45%. While still above 50%, it fell far below the breakeven line for a loss of 19.55 units. This definitely was not the result that I had hoped for, however my model was still able to predict the correct result more than half the time, so it was not completely wrong. There is certainly room for improvement.
Analysis
Overall, the model performed quite well in its first partial season. I will be “running it back” next year with the same framework. However, there are some improvements that need to be made in the offseason. The first thing I will focus on is automating my data refreshing. Throughout the season, I would have to manually refresh my data each day to ensure that the model was handling the most up-to-date information to make predictions. If I can automate this, I can save myself around 5-10 minutes of copy and pasting. Secondly, regarding the model itself, I want to try to predict bullpen usage. This is going to be a tall task, but it’s not as accurate to use a team’s entire season long bullpen averages for a game in which only 3 or 4 of them may pitch. If I can find a way to predict usage, or even just use the averages of the “available” arms, I feel this will lead to an increase in accuracy. Finally, I want to find a way to introduce variance into the data for pitchers. The reason being that pitchers almost never throw exactly their season averages in a game. The same goes for hitters, but it should be much easier to track in pitchers. The more consistent the pitcher is around his season averages, the more confident the model is in predicting the result. This will require me to gather game-by-game data for the pitchers, but it could be well worth it if it increases prediction accuracy. I’m sure I will think of more upgrades to make over the offseason, but as of now, those are the three that are at the front of my mind.
Conclusion
Overall, I am very pleased with my model and its results. As I mentioned above, there are certainly things that can be improved, but I feel that it was a very good first attempt. I will bring the MLB Karn Model back next year, new and improved, and I will try to give updates along the way! Coming out over the course of the next couple of weeks will be my 2026 MLB Season Predictions Review, where I will go over all of my predictions that I made at the beginning of the season. Some are good, some are great, and some are down-right awful. Stay tuned for that article soon.
Leave a Reply