Analyzing the Accuracy of the 2012 Projections | Part 1: Hitters

Unlike other fantasy sports, baseball has the luxury of having lots of really advanced projection systems to look at prior to your drafts. If you’re a person who relies on one set of projections then you put yourself at risk because you’re only as good as the projection system that you choose. Projections represent the lens in which you view the baseball universe so picking the correct set of lenses is paramount for fantasy baseball success. So, let’s look back at the 2012 projections and see how successful they were in projecting batting statistics for our roto leagues.

For the 2012 draft cheatsheets, I offered up seven different projection choices and here’s a rundown on who they are and what they do:

• Marcel: The most basic forecasting system around – it takes three years of player data, weights the most recent years heaviest and regresses the players towards a mean (age factor included)
• Steamer: For hitters, Steamer takes five years of player data and regresses certain stats more heavily than others using an aging factor as well. Each component within their projections uses a different projection system.
• ZiPS: Does a little of the weighted regression like Marcel but for four years and does a bit of the comparable player regression based on aging trends
• Cairo: It’s like Marcel but with more bells and whistles (stat-specific regression and position-specific regression for instance)
• Fangraphs Fans: These are user-entered projections for a player that are entered into the Fangraphs site. They average out all of those user-entered projections for a look at how the public views a player as opposed to a computer algorithm. A minimum of 8 “fans” must have submitted a projection for a player in order for them to be eligible.
• RotoChamp: Also takes three years of player data and weights the most recent years heavier but has more tricks such as looking at batted ball type (line drive percentage, fly ball percentage, etc) and lineup projections and more.
• Combined (MSZC): Averaging the four main free projections for a player into one projection

In order to analyze who did the best job in helping fantasy baseball drafters select hitters, I took a few steps here. The first step was to merge all of these projections, the actual 2012 stats and the ADP data from the preseason. I only included players who were in all seven projections, had at least 100 plate appearances and had an 2012 ADP under 400 so that we were only looking at players who were relevant on fantasy baseball draft days last preseason. This left a player pool of 236 batters to analyze.

There are a few statistics which could help us compare actual results to projected results. The two statistics that I used in this analysis were the correlation coefficients of the projections related to actual results and the root mean square error (RMSE). The correlation coefficient shows how closely related the the projection is with the actual result and is a measure of the linear relationship between the two variables. On the other hand, the RMSE value analyzes the misses and the extent to which the projections varied from the actual results. Good RMSE results indicate a better agreement between the projected and actual results.

When doing my analysis, I standardized all of the stats within their own universe because it ends up being  irrelevant if the actual fantasy league average was 15 HR’s per player yet a projection system had the league average set higher or lower. On draft day, we’re comparing players within their own projection systems and if the system projects Miguel Cabrera to be 3 standard deviations above the mean in HR’s then it doesn’t matter if that equates to 35 or 45 HR’s.

After running the tests for correlation and RMSE, I could have ranked the results and displayed them here as rankings in each category. However, that wouldn’t recognize the times when 1st, 2nd and 3rd place for a category were a virtual tie and when last place was far, far behind the others. To account for that, I converted the rankings to standardized z-scores to show how far above or below average each projection was for each stat.

So, now that you know the method, let’s see those results for how each projection system ranked above or below average in each roto 5×5 category, the overall total WERTH ranking as well as an average of those six z-scores.

Correlation rankings (results converted to z-scores)
	HR	AVG	R	RBI	SB	WER	Av
Comb	0.8	1.4	1.3	1.5	0.5	1.2	1.1
Steamer	1.0	1.1	1.2	1.4	0.4	1.3	1.1
Fans	1.0	-0.5	0.2	-0.5	0.6	0.1	0.2
Zips	0.2	0.3	-0.8	-0.6	1.0	0.0	0.0
RC	-0.9	-1.3	0.0	-0.1	0.3	-0.4	-0.4
CAIRO	-0.4	-0.7	-1.0	-0.6	-1.3	-1.2	-0.9
Marcel	-1.6	-0.3	-1.1	-1.0	-1.6	-1.1	-1.1
RMSE rankings (results converted to z-scores)
	HR	AVG	R	RBI	SB	WER	Av
Comb	0.8	1.4	1.3	1.5	0.5	1.2	1.1
Steamer	1.0	1.1	1.2	1.4	0.4	1.3	1.1
Fans	1.0	-0.5	0.2	-0.5	0.6	0.1	0.1
Zips	0.2	0.3	-0.8	-0.6	1.0	0.0	0.0
RC	-0.9	-1.3	0.0	-0.2	0.3	-0.4	-0.4
CAIRO	-0.4	-0.7	-1.0	-0.6	-1.3	-1.2	-0.9
Marcel	-1.5	-0.3	-1.1	-1.0	-1.6	-1.1	-1.1

When looking at all 236 draftable hitters from last preseason, the Combined and Steamer projections were #1 and #2 by quite a bit in regards to both correlation and RMSE. That’s the same top two as 2011 but a reversal as Steamer was #1 with a 1.0 average z-score for RMSE ranking and the Combined projections were #2 with a 0.9 average z-score for the same. The other positions shifted around in comparison to last year but it is enlightening to see that the champions stayed the same. Seeing that the success of Steamer was not a just a one-year wonder last year is valuable as it shows we’re looking at a very strong and consistent projection system.

Looking at the entire draft pool tells us one story for fantasy baseball but not all leagues are concerned with all 200+ draftable hitters. So I also wanted to specifically look at those who were among the top portion of draft boards since those are often who we are most concerned about on draft day. When looking at those within the top 200 ADP last year, we had 117 hitters to look at.

Correlation
	HR	AVG	R	RBI	SB	WER	Av
Comb	0.2	1.1	1.8	1.4	0.2	1.2	1.0
Steamer	0.6	0.5	0.4	0.5	0.8	1.3	0.7
Fans	1.5	-0.6	-0.2	-1.2	1.0	0.1	0.1
RC	0.5	-0.5	-1.4	-0.1	0.8	-0.4	-0.2
Zips	-0.8	-1.8	-0.3	0.8	-0.1	0.0	-0.3
CAIRO	-0.7	0.6	0.2	-0.3	-1.7	-1.2	-0.5
Marcel	-1.4	0.6	-0.5	-1.2	-0.9	-1.1	-0.8
RMSE
	HR	AVG	R	RBI	SB	WER	Av
Steamer	0.7	0.7	1.0	1.2	0.6	1.3	0.9
Comb	0.0	1.0	0.5	0.7	0.1	1.2	0.6
RC	0.8	0.6	1.0	0.7	0.7	-0.4	0.6
Fans	1.4	-1.2	0.2	-0.6	0.9	0.1	0.1
Zips	-0.6	-1.5	0.0	0.4	0.5	0.0	-0.2
Marcel	-1.2	0.6	-1.0	-1.0	-0.9	-1.1	-0.8
CAIRO	-1.1	-0.1	-1.7	-1.4	-1.8	-1.2	-1.2

With this dataset trimmed down to those 117, we see a bit more variety in the results. the Combined projections and Steamer are still tops in correlation and RMSE. RotoChamp makes a push to contend in RMSE but is still lagging a little bit.

We can trim that pool even further by only looking at those with an ADP in the top 100. By doing this, we start to get into dangerous territory as the sample size gets very small with only 66 batters to analyze.

Correlation
	HR	AVG	R	RBI	SB	WER	Av
Comb	0.8	1.1	1.8	1.4	0.2	1.2	1.1
Steamer	0.4	0.5	0.4	0.5	0.8	1.3	0.7
Fans	0.5	-0.6	-0.2	-1.2	1.0	0.1	-0.1
CAIRO	0.9	0.6	0.2	-0.3	-1.7	-1.2	-0.2
Zips	-1.0	-1.8	-0.3	0.8	-0.1	0.0	-0.4
Marcel	0.1	0.6	-0.5	-1.2	-0.9	-1.1	-0.5
RC	-1.8	-0.5	-1.4	-0.1	0.8	-0.4	-0.6
RMSE
	HR	AVG	R	RBI	SB	WER	Av
Steamer	0.8	0.7	1.0	1.2	0.6	1.3	0.9
Comb	0.8	1.0	0.5	0.7	0.1	1.2	0.7
RC	-2.1	0.6	1.0	0.7	0.7	-0.4	0.1
Fans	0.4	-1.2	0.2	-0.6	0.9	0.1	-0.1
Zips	-0.2	-1.5	0.0	0.4	0.5	0.0	-0.1
Marcel	-0.1	0.6	-1.0	-1.0	-0.9	-1.1	-0.6
CAIRO	0.4	-0.1	-1.7	-1.4	-1.8	-1.2	-1.0

In these results, we see some more shifting in the rankings all over the place but Steamer and the Combined projections are still the top two by a large margin in both areas. The success of these two systems is starting to embarrass the other contenders at this point so let’s just call the fight.

Conclusion

The takeaway that I have from these results is that some projections are better than others in certain ways but Steamer and the Combined projections are the two that consistently are at the top in each scenario. This is not a one-year occurrence either as they were also consistently at the top if you look at last year’s results as well. So, when it comes to hitters, I feel very safe in saying that the Steamer projections seem to be your best bet for projections in fantasy baseball and, well, they’re free! I highly recommend utilizing them when loading up your cheatsheets here but pay special attention to the Combined projections as well for a nice second opinion when analyzing a player.

Update on 2/27/13

It was suggested by a few of the friendly readers from around these parts that I should add in a few other projection systems to this analysis. Previously, I had not heard of MORPS but was happy to bring another contender into the ring here. Also, I had used PECOTA in the analysis last year so it was only fair to bring it back. I ran the analysis using the same process as above and the results weren’t wildly different as the Combined projections and Steamer projections continued their dominance. However, MORPS and PECOTA showed themselves to be strong contenders for fantasy usage.

Looking at all 236 draftable hitters again, PECOTA was better than average in correlation and RMSE while MORPS was lagging towards the bottom.

Correlation
	HR	AVG	R	RBI	SB	WER	Av
Comb	0.8	1.5	1.5	1.6	0.7	1.4	1.3
Steamer	1.0	1.2	1.4	1.5	0.6	1.4	1.2
Fans	1.1	-0.3	0.2	-0.6	0.8	0.1	0.2
PECOTA	0.6	0.4	0.0	0.6	-1.2	0.4	0.1
Zips	0.2	0.4	-0.9	-0.7	1.2	0.0	0.0
RC	-1.0	-1.1	0.0	-0.2	0.5	-0.5	-0.4
MORPS	-0.5	-1.5	0.2	-0.3	-0.2	-0.1	-0.4
CAIRO	-0.5	-0.5	-1.1	-0.8	-1.1	-1.4	-0.9
Marcel	-1.7	-0.1	-1.3	-1.2	-1.4	-1.3	-1.2
RMSE
	HR	AVG	R	RBI	SB	WER	Av
Comb	0.8	1.6	1.5	1.6	0.7	1.4	1.3
Steamer	1.1	1.2	1.4	1.5	0.6	1.5	1.2
Fans	1.1	-0.3	0.2	-0.6	0.8	0.1	0.2
PECOTA	0.6	0.4	0.0	0.6	-1.2	0.3	0.1
Zips	0.2	0.4	-0.9	-0.7	1.2	0.0	0.0
RC	-1.0	-1.1	0.0	-0.2	0.5	-0.5	-0.4
MORPS	-0.5	-1.5	0.2	-0.3	-0.2	-0.1	-0.4
CAIRO	-0.5	-0.5	-1.1	-0.8	-1.1	-1.4	-0.9
Marcel	-1.7	-0.2	-1.3	-1.1	-1.4	-1.3	-1.2

When trimming the pool down and only looking at the top 200 drafted hitters from last year, MORPS and PECOTA both fare even better. PECOTA doesn’t necessarily top Steamer but definitely creeps up as a strong silver medalist.

Correlation
	HR	AVG	R	RBI	SB	WER	Av
Comb	0.3	1.2	1.7	1.5	0.0	0.9	0.9
Steamer	0.6	1.0	-0.1	0.8	0.7	0.8	0.6
PECOTA	0.4	-0.1	1.4	0.1	-0.1	1.8	0.6
Fans	1.8	-0.4	-0.4	-1.0	0.8	-0.4	0.1
RC	0.7	-0.3	-1.6	0.1	0.7	-1.5	-0.3
CAIRO	-0.7	0.7	0.0	0.0	-1.9	-0.4	-0.4
MORPS	-0.8	-1.3	0.0	-1.5	1.2	0.0	-0.4
Zips	-0.8	-1.5	-0.5	0.9	-0.3	-0.9	-0.5
Marcel	-1.5	0.7	-0.6	-0.9	-1.1	-0.3	-0.6
RMSE
	HR	AVG	R	RBI	SB	WER	Av
Steamer	1.1	1.3	1.3	1.7	0.5	1.5	1.2
PECOTA	0.6	-0.8	1.3	1.1	-0.4	1.2	0.5
RC	0.7	0.7	0.3	0.2	0.6	-0.1	0.4
MORPS	-0.2	-1.3	0.6	0.1	1.2	0.7	0.2
Comb	-0.2	1.1	0.0	0.2	0.0	-0.1	0.2
Fans	1.3	-0.8	-0.2	-0.8	0.8	-0.2	0.0
Zips	-0.7	-1.1	-0.4	0.0	0.4	-0.4	-0.4
Marcel	-1.4	0.7	-1.1	-1.1	-1.0	-0.8	-0.8
CAIRO	-1.3	0.1	-1.7	-1.4	-2.0	-1.7	-1.3

Then, when we go down to just the top 100, we see MORPS and PECOTA continue to grow stronger. They both pass Steamer in regards to correlation power but Steamer still does the best job of aligning with the actual results.

Correlation
	HR	AVG	R	RBI	SB	WER	Av
Comb	0.8	0.4	0.8	1.2	0.1	0.6	0.6
MORPS	-0.3	0.5	0.9	-0.1	1.2	1.0	0.5
PECOTA	0.9	-0.9	0.0	0.7	-0.1	0.9	0.2
Marcel	0.0	1.1	0.4	0.0	-1.4	0.3	0.1
Steamer	0.5	0.0	-1.0	0.0	0.8	-0.2	0.0
Zips	-1.2	-1.7	1.0	1.1	0.4	0.2	0.0
Fans	0.5	0.7	-1.1	-1.3	0.9	-0.7	-0.2
CAIRO	0.8	-1.0	0.6	0.2	-1.8	0.1	-0.2
RC	-2.0	0.9	-1.7	-1.8	0.0	-2.3	-1.1
RMSE
	HR	AVG	R	RBI	SB	WER	Av
Steamer	1.1	1.3	1.3	1.7	0.5	1.5	1.2
PECOTA	0.6	-0.8	1.3	1.1	-0.4	1.2	0.5
RC	0.7	0.7	0.3	0.2	0.6	-0.1	0.4
MORPS	-0.2	-1.3	0.6	0.1	1.2	0.7	0.2
Comb	-0.2	1.1	0.0	0.2	0.0	-0.1	0.2
Fans	1.3	-0.8	-0.2	-0.8	0.8	-0.2	0.0
Zips	-0.7	-1.1	-0.4	0.0	0.4	-0.4	-0.4
Marcel	-1.4	0.7	-1.1	-1.1	-1.0	-0.8	-0.8
CAIRO	-1.3	0.1	-1.7	-1.4	-2.0	-1.7	-1.3

All in all, MORPS shows itself as being a solid new contender when it comes to the top players in baseball (though a bit weaker in the later portions of the draft) and PECOTA shows why it may be worth plopping some money down for. However, when all is said and done, the refrain remains the same as Steamer continues to top the field.