Who Had The Best Fantasy Baseball Hitter Projections in 2014?

Given that a big part of using these fantasy baseball cheatsheets is making sure you have good projections within them, I’ve made a habit of doing an analysis of the projections each season. In the past three years, Steamer has swept the field by being named my best projection system for hitters every year. I still found that combining projection systems (or, like with my Special Blend projections, combining while using appropriate weights for each stat) is often even more accurate though. When looking back at 2014, will anybody be able to catch up to Steamer this time around and will the combined projections still work best?

Last year, I profiled the main projection systems (broken into the categories of Age Regression systems, Comparable Player systems and the Human Element systems). I’ve opted to include RotoChamp, Guru and CBS Sportsline as new sources for the analysis this year though. I can’t find information on RotoChamp’s methodology but Guru is an age regression model. On the other hand, CBS Sportsline represents your standard “expert” projections that are more human-based. I haven’t analyzed a system like CBS in the past to compare to the more scientific-minded models so I am interested to see how it does.

In this particular analysis, the goal isn’t to find which system most accurately predicted the entire baseball world nor is it to see who predicted which teams would do best. No, the goal here is simply to find out which one of these systems did the best job at projecting our fantasy baseball hitters last year.

Before we get started, I want to note that the Baseball Projection Project is a great resource if you ever want to gather up this old information and do your own analysis too. All it takes is a bit of drive, passion and tons of free time.

The Projection Sources

In addition to the sources that I already listed above, I’ll be analyzing some of our usual competitors (which you can read about in my previous projection introduction post):

• Steamer
• ZiPS
• CAIRO
• Oliver
• Marcel
• Clay Davenport
• MORPS
• Fangraphs
• Mr. Cheatsheet’s Special Blend – For those unfamiliar, this is where I combine the projections into a master projection but I apply different weighting of the combinations for each individual stat to try to come up with the best possible combination of available projections.

The Method

If you’re using a single projection system for your fantasy baseball draft then you’re not exactly worried about whether a projection comes close to the actual end-of-year total. Sounds crazy, right? What I mean is that you actually want to the system to be accurate in telling you how far above or below average each player is within those projections. One system might award home runs more generously than another but all you really want to know is who has the best projected stats in that universe because that helps you determine who you should draft on draft day.

Keeping that fact in mind, I standardize all of the projections for each statistic so that I look at the predicted z-score in that stat for each player (z-score being how many standard deviations above/below the mean that projection was). At that point, I choose to use Mean Absolute Error (MAE) to compare the results. It basically averages out the difference between the projected z-score and the z-score from the actual 2014 stats among our pool of comparable players.

This isn’t about who predicted the waiver wire wonders. This is about draft day. So I only included players in this analysis that were showing up in drafts last pre-season and were shared among all of these analyzed projection systems. I also removed players who ended up not playing last season or played in an extremely limited capacity. This left 228 hitters in my pool of players from last season.

The Initial Results

My analysis focused on the five main rotisserie categories to see how well the systems did in projecting the actual results there. All of these results were dependent on playing time in my analysis, including batting average (which I had weighted by number of plate appearances for rotisserie purposes). After crunching the numbers, I expected to see Steamer rise to the top again but was surprised to see some different results for 2014.

	HR	AVG	R	RBI	SB	Avg Rank
Special Blend	1	1	3	2	3	2
CBS	4	6	1	1	2	2.8
RotoChamp	6	12	2	3	1	4.8
Fangraphs	2	4	8	4	7	5
Steamer	3	2	7	5	9	5.2
ZiPS	5	3	6	9	6	5.8
CAIRO	7	9	5	6	4	6.2
MORPS	10	7	4	7	5	6.6
Clay	8	10	10	8	12	9.6
Oliver	11	8	12	11	8	10
Marcel	9	11	9	10	11	10
Guru	12	5	11	12	10	10

My Special Blend of projections finished first, which did not shock me, but the CBS Sportsline projections actually finished in second place and even the Fangraphs Fans finished above Steamer. Perhaps it shouldn’t be a shock that two systems that rely on humans to project playing time finished at the top in this type of analysis. As stated above, I’m not sure of RotoChamp’s methods but they might also use human intervention for determining playing time instead of using a scientific projection.

To see just how high or low each system ranked in those categories, check out the chart below.

Notably, the projections that relied most heavily on age regression (Marcel, Guru, Oliver) were the ones that performed the worst in this type of analysis.

Seeing that the “human systems” did so well in this analysis, it begs the question of whether they would do as well when playing time was not a factor in their projections so let’s analyze that.

The Results Per AB

I couldn’t break down the results to look at each stat per plate appearance (which in my mind would have been the best method) because I didn’t have projected PA’s for all of my data. However, I did have projected AB’s for all of my data so I used that instead to see how well the projections did when looking at HR’s per AB, R’s per AB and such. These results looked a bit more familiar but, still, with some strange twists.

	HR	AVG	R	RBI	SB	Avg Rank
Special Blend	2	1	1	2	1	1.4
ZiPS	3	3	2	5	4	3.4
Steamer	1	2	10	1	6	4
Fangraphs	4	6	12	3	5	6
CBS	7	5	8	8	2	6
Oliver	11	7	4	4	7	6.6
CAIRO	8	10	5	6	8	7.4
Marcel	6	9	3	7	12	7.4
Clay	5	8	9	11	11	8.8
RotoChamp	9	11	11	10	3	8.8
Guru	12	4	7	12	9	8.8
MORPS	10	12	6	9	10	9.4

The first thing to note is that ZiPS and Steamer benefited the most from taking away playing time projections as a factor. In fact, both systems performed better than any other method when simply projecting raw production per at bat.

The human systems still fared fairly well with Fangraphs and CBS both being close to the top. RotoChamp, however, fell drastically in this version of the projections so it seems that their playing time projection was a big part of their success from the first analysis.

The age regression systems of Marcel, Oliver and CAIRO all saw a boost here without playing time dragging them down as well.

When looking at the chart below breaking down just how far well each system performed in each category, we see that Steamer and ZiPS were pretty far above the rest of the pack (with Steamer actually finishing second despite what you see above). On top of that, we also see that the Special Blend of projections fared the best of all.

Conclusions

The first conclusion to draw is that playing time projections have a huge influence on the accuracy of projections. The scientific models struggle at projecting playing time because it really isn’t something that predictable. Humans have a better idea of who is projected to start or just get playing time. This is likely the reason that Steamer has tried out a few variations of their methodology over the years like taking playing time projections from the Fans at Fangraphs (I think they stopped doing this though) or just doing a flat projection of 600 PA’s for everyone so that playing time isn’t a factor.

With playing time removed, Steamer and ZiPS do a great job at projecting how well the players will do for you. And, in either case of using playing time or not, the CBS Sportsline expert projections actually did a damn fine job.

However, as in the past, the power of combining projections still wins out. Even when playing time is a factor, a combination of projections still is the best method for your fantasy success.

Winners

I struggle to determine first place, second place or third place prizes this year because of the variety of results that we saw. There just isn’t a clear-cut winner (outside of combining projections) so I’ll award my top three, in no special order, instead. Without further ado, the 2014 best projections were:

CBS Sportsline
Steamer
ZiPS