Who Had The Best Projections in 2013? – Part 1: Hitters

The list of baseball projection systems seems to grow each year as more people think they can predict the future. It’s important to know which of those projections are the best because you can win or lose your league depending on what projection system you use in fantasy baseball. This website gives you cheatsheets and you are given you the choose which projection system you want to use and that becomes a heavy decision. To help understand which projections are best, I’ll look at the 2013 hitter projections and see which projections would have helped you most for your fantasy leagues last year.

The Method

If you’re using one projection system for your fantasy baseball draft analysis, you’re actually not totally focused on whether a prediction of 40 HR, for instance, ends up being correct. Because, what if that entire projection system had predicted fifty guys to have 40+ HR? In that case, you may not have paid as much attention the guy that was accurately predicted from that group. What you want to be correct is how far above or below average they are projecting a player to be in each stat.

So, with that being said, I standardize all of the projections for each statistic so that I look at the predicted z-score in that stat for each player (z-score being how many standard deviations above/below the mean that projection was).

At that point, there are a variety of statistical methods to use to analyze whether the projections were accurate in projecting a player to be above or below average. I chose to use Mean Absolute Error (MAE) for this analysis. Compared to Root Mean Squared Error (RMSE), it doesn’t penalize large mistakes quite as much which I find to be a good thing because you have the option to bench a player in fantasy baseball that is performing vastly different than expected. Mean Absolute Error, in this case, is the difference between the projected z-score and the actual z-score from the actual 2013 results averaged out among the whole pool of players we are comparing.

I only included players in this analysis that were showing up in drafts last pre-season and were shared among all of these analyzed projection systems. Though some players did well that weren’t drafted, this is about who helped you the most on draft day. I also removed players who ended up not playing last season or played in an extremely limited capacity. This left 259 hitters in my pool of payers from last season.

The 2013 Rankings

For a full rundown of the competitors, see my post that introduces the baseball projections.

After performing the analysis as stated above, here are the rankings for how well each projection system performed:

*WERTH is the value of adding the five roto z-score values together for a player to get their total projected rotisserie value.

Those results tell part of the story but sometimes the gap between first and second place is larger than we think. Here’s a visual representation of the rankings that shows precisely how above or below average each system was in comparison to one another.

Trends or Anomalies

I’m not surprised that my optimized combination of the projections performed the best in a majority of categories. I am actually somewhat surprised it didn’t perform even better so I’ll need to do some more tweaking to the weights now that I have more data to analyze. This sort of projection combination is a bit of a cop-out as it minimizes outliers and brings everyone closer to a safe middle.

My biggest surprise is the weak performance by Steamer in the HR category. Otherwise, Steamer nearly swept the rest of the race. The Fangraphs Fans seemed to have done a phenomenal job at predicting HR totals so I went back and looked at past years and this seems to be a trend as the Fans had the best HR projections last season (and a 2nd place finish in 2011).CAIRO also had a very good performance across the board with the exception of the AVG and SB categories. Somehow the marriage of the Fangraphs Fans and Steamer projection did not go as well as expected for the hitting projections. We’ll have to see if that trend continues for the pitching projections when we look at that next.

The Awards

Well, it’s clear that Steamer deserves the gold medal for the third straight year here. If you want projections for your fantasy baseball hitters that will be most closely aligned with actual results, Steamer is the clear choice for you. The only concerning part of their projections in 2013 was the very poor performance in projecting HR’s. Aside from that, they “steamed” the competition (puns are fun?).

CAIRO deserves the silver medal for a strong but inconsistent performance. Despite being a bit all over the map, they still did a great job at predicting the overall WERTH value for each player and that’s certainly a good thing.

In addition, the non-scientific method of just letting the Fans do the picking has shown some value, especially when it comes to projecting HR totals. It’s a bit hit-or-miss beyond that but we have to start to acknowledge the power of the fan HR projection.