Ranking The Accuracy Of Last Year’s Projections [2020 Edition]

Fantasy baseball owners put a lot of blind faith in the projection system they use in their draft preparation. It is only natural that they’d want the most accurate projection system possible. In order to help figure out which projections have been most accurate for fantasy baseball purposes, I’ve been running an annual analysis of the baseball projections since 2012. It’s been an educational process to better learn what works and what doesn’t work, and I consistently share my results with you each year to help educate you as well. Let’s kick off a new decade by doing it again.

I recently saw a projection analysis done by the staff at FantasyPros. I certainly have respect for what their site sets out to accomplish, but I did feel this analysis wasn’t done in the most optimal way. Their analysis directly compared the 2019 projections to the actual 2019 stats. My friends, that is not the way to analyze the puzzle that is baseball projections and I’ll explain why as we go.

The Projection Systems

In my life outside of this website, I make documentary films (quite different from spreadsheets and statistical analysis, I know!). If I were to make a documentary about baseball projections, I’d start by talking to Tom Tango. He started the trend of statistical projections with his Marcels projections where he would weight stats from the three previous seasons and adjust for regression. It was a simple system, but every system likely owes some credit to that initial work as they’ve done their own adaptions of it.

Looking back at the 2019 projections, there are now ten competitors that I feel are worth taking a look at. Here’s the list of them, noting how they fared last time I did this analysis and their methodology:

• Steamer (ranked 3rd in hitters & pitchers for 2018 projections) – Steamer uses five years of player data and treats each stat differently for regression purposes so each component within their projections basically uses a different projection methodology. In the many years I’ve done this test, they’ve typically finished at the top.
• Steamer 600 (ranked 11th in hitters for 2018 projections, 8th in pitchers) – This is a modified version of the Steamer projection where it eliminates playing time projection and assumes every projected hitter will have the same PA (600) and every pitcher will have the same IP. It’s not really met for this type of testing as the playing time will obviously be way off, but it’s fun to include for reference.
• ZiPS (ranked 6th in hitters for 2018 projections, 7th in pitchers) – Developed by Dan Szymborski, this system takes the past four years of stats for each player, weights the more recent years heavily but then takes those results to look for comparable historical players to determine the aging regression trend to apply.
• Clay Davenport (ranked 9th in hitters for 2018 projections, 6th in pitchers) – This method also looks for comparable players like ZiPS but then applies playing time weights and redistributes the stats afterwards based on team projections and to mimic last year’s league totals.
• Fangraphs Depth Chart (ranked 5th in hitters for 2018 projections, 4th in pitchers) – This is a combination of projections. First, it combines ZiPS and Steamer projections, but then playing time is adjusted by the humans on Fangraphs staff as opposed to letting the projections blindly project playing time.
• Mr. Cheatsheet’s Special Blend (ranked 1st in hitters & pitchers for 2018 projections) – I started doing these in 2013. Like the Fangraphs Depth Chart, these combine other projections. However, each individual stat uses a different weighting system for that combination of projections. Based on my historical research, I may know that one system is much better with HR’s so I give them additional weight over other projections.
• ATC (ranked 4th in hitters for 2018 projections, 5th in pitchers) – On the surface, the idea behind ATC seems fairly identical to my Special Blend approach. They take different projection sources and past stats, then weight them differently for each category based on their research. I guess it means that there’s validity to such a process! This is my first time including them in this analysis and I’ll be curious to see how their weights match up versus mine.
• THE BAT (ranked 8th in hitters for 2018 projections) – Developed by Derek Carty and, like Steamer and ZiPS, this system uses regression and aging curves but it has unique twists like factoring in umpires, weather and platoon splits. This is likely due to the fact that it was originally designed for DFS play. This is my first year including it, so we’ll see how it stacks up to the other regression-focused models.
• Pod (ranked 2nd in hitters & pitchers for 2018 projections) – Mike Podhorzer has been doing projections since 2001. He incorporates xStats into his analysis but also touts his manual approach of accounting for things like injuries, mechanical changes and more. I can’t determine the exact methodology but it seems to be a delicate balance of science and human touch. This is also the first time I’ve included them in this test.
• Razzball (not analyzed last year) – From my understanding, they take the Steamer projections and do playing time adjustments to them to make their own slightly modified version. I’d be happy to hear more insight into their process if that is incorrect though.

The Method

Two of the better statistical approaches to analyzing projections are Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). They’re both statistics that are designed to measure the magnitude of errors in a sample of predictions. I calculate both in my analysis, but I show MAE here because I feel it is a better tool for this test as it doesn’t penalize large errors as heavily as RMSE. I am running this test for fantasy baseball purposes, and if there were a grossly large error in a projection then you would likely just bench that player in your league so I feel that MAE is a bit more realistic in a fantasy baseball universe.

There are many ways that you can judge the accuracy of projections. Earlier I mentioned how the folks at FantasyPros have a different system than mine. They specifically look at how each projection compared to actual results. I don’t see that as the way to go about this type of analysis.

It may seem like common sense to compare projections to actual results but there are reasons we don’t do that.

Each projection system operates within its own baseline as far as what would be an average player. The baseline they set should not be held against any projection system for fantasy purposes. If Steamer thinks the average fantasy hitter in their universe will hit 15 HRs, a player they project for 25 HRs would be way above average. During the actual season, we may find the average fantasy hitter actually hits 22 HRs, so a hitter hitting 25 HRs would only be a bit above average. An accurate projection in Steamer’s universe for that hitter wouldn’t have been 25 HRs; it would have been 17 HRs (just slightly above their average baseline). If we’re drafting based on projections, it doesn’t quite matter if they nailed that 25 HRs or not, it matters if they nailed how far above or below average that player was in that projection universe.

To account for this reality, I calculate how far above/below average a player’s projection was in a system for each fantasy stat compared to how far above/below average they actually finished in the season.

The math goes like this. I calculate averages and standard deviations for each stat in each projection universe and convert each projected statistic to z-scores. I then use the Mean Absolute Error to compare those projected z-scores to the actual z-scores from the season. This is done with the typical 5×5 roto stats for both pitching and hitting while taking into account playing time adjustments for rate-based stats like AVG, ERA, and WHIP (a player who hit for a .300 AVG over 200 PA is not as impactful as a player who hit for a .300 AVG over 650 PA so I make adjustments accordingly).

This analysis is specifically for fantasy baseball draft purposes so I start with the list of players that were being drafted in fantasy drafts last preseason. I accumulate all of the projections for them and their actual stats from last year. I eliminate players that didn’t have a projection across the board, or those that didn’t play enough in the actual season to be analyzed. That left me with 264 hitters and 170 pitchers to analyze this year.

The Hitter Results

When analyzing the fantasy hitter projections from last year, I look at the main 5×5 roto stats of HR, AVG, R, RBI, and SB. I’ve been doing this test since 2012 and Steamer has typically finished at the top among non-weighted projection systems. My own Special Blend system (which aggregates the other projections) has often offered an improvement over Steamer. This year, we have three aggregated projection systems and seven true projection systems to analyze.

My own Special Blend projections were able to top all of the weighted projection systems (and all projections) when it came to the overall success. This year, the ZiPS projections were able to take the top spot when it came to actual projection systems, and that was partially aided by their huge success in the HR category (which overcame their poor performance in AVG and SB). This is the best performance I’ve seen from the ZiPS system in a long time.

Steamer is usually a top dog in this category and still did great when it came to the rate-based stat of batting average. The fact that they didn’t perform great in the other categories was likely just due to a slightly off-year in terms of playing time projections. I should note that an “off-year” for Steamer is still the second-best true projection system overall. Given their history of outperforming everyone, I don’t doubt a comeback is in order.

The Pitcher Results

Just like with the hitters, I’m looking at the main roto 5×5 categories when it comes to pitching too but I do leave out Saves. The managers on the baseball field control this stat more than projection systems do, and I don’t feel it is right to fault anybody for not projecting that correctly.

I’ve stated before that Steamer is usually the vanguard of projections and that typically rings true with hitting, but pitching is where the system really stands out. It is almost always at the top of the pack. We have some new contenders this year with THE BAT and the Razzball version of Steamer projections, so perhaps they could give Steamer a run for its money?

Nope. Steamer displayed no mercy this past year with pitcher projections. They almost broke the needle with how far ahead of the pack they were overall. They dominated nearly every category across the board. I can’t understate how impressive their work was last year with pitching. The people behind the weighted systems try to mathematically come up with specific weightings combining the best projection systems out there, and yet we can’t even beat a projection system such as Steamer when it comes to pitching. Needless to say, it’s a good projection system.

That being said, this was the first time THE BAT appeared in the pitcher rankings here and they were quite impressive too. The Pod projections also beat out any weighted projection and that should be applauded. All in all, there is work to do for the weighted systems to catch up to the actual projection systems themselves when it comes to pitching.

Conclusions

When it comes to the weighted systems, my Special Blend projections had a good year overall. I was happy to see my aggregated rankings finish in 1st for hitting, even if they finished in 5th for pitching. It’s pretty amazing to me how well Steamer did with the pitcher projections though. For a free service, they really do a tremendous job every single year. If you’ve read these posts in past years then you know that Steamer almost always finishes at the top.

I was impressed with the year that ZiPS had for hitting. In addition, THE BAT seems like a promising system when it comes to pitching as well and I’ll be curious to see if that trend continues in the years ahead. It’s an exciting time for baseball projections as there are lots of good options out there now. I offer many kudos to the hard work of all the people behind these projections. They all do a great job and it shows here.

With this year’s analysis is done, I’ll be releasing my Special Blend projections tomorrow with slight updates based on the work you see here. Until then, keep working hard on prepping for your drafts!