Renowned Baseball Prospectus researcher Russell Carleton wrote an interesting article earlier this week on Baseball Prospectus that utilized the Baseball Info Solutions shifts data made available by the great folks at FanGraphs and questioned whether defensive shifts are actually an effective technique. However, there are a couple of small flaws in the technique that lead to misleading results.

Carleton came up with a very good approach to apply hitter batting averages on balls in play (BABIP) in non-shifted situations to their number of shifted balls in play. This allowed him to compare hitters’ actual totals of hits to their expected totals of hits, finding that the actual hits exceeded the expected hits and concluding that the shift has been detrimental in net.

However, Carleton’s choice of technique creates a few problems. His technique considered single season BABIP in non-shift situations with a minimum of 100 such plate appearances in that individual season. Unfortunately, this eliminates a large number of the most shifted batters in baseball. In 2012, for example, the most shifted batters in baseball were Carlos Pena, Adam Dunn, and David Ortiz. None of them reached Carleton’s threshold of 100 balls in play in non-shift situations, meaning Carleton is leaving out the three players facing the shift the most. In 2015, when shifts were much more prevalent, this approach throws out 7 of the top 11 shifted batters.

More importantly, this approach relies on single season BABIPs based on as few as 100 plate appearances. Contemporary analysis has taught us that while hitter BABIP is more stable than pitcher BABIP, even full season samples should be regressed heavily towards the league BABIP or hitter’s historical BABIP.

For comparison, we can tweak Carleton’s approach ever so slightly, which will lead us to reach the exact opposite conclusion. Instead of treating each season individually, we can group the full 2012-15 sample together and apply the same 100 no-shift plate appearance minimum over 2012-15. This will allow for more stable non-shift sample sizes to help calculate the Expected Hits and allow more players into the sample.

As you can see, this small change of approach leads us to a much stronger conclusion in the opposite direction, demonstrating that shifts have in fact been working on a large scale.

Neither of these is a perfect approach, of course. You can focus on groundballs and short line drives specifically, for example. We’ve tried this and many other approaches (some of which were previously published Stat of the Week articles here, here, here, here, and here and in the four volumes of The Fielding Bible), and have unanimously reached the conclusion that the shift works. In fact, the results of this tweaked version of Carleton’s study match up quite well with our previous research. In this case, a slight flaw in Carleton’s technique seems to be skewing the results and conclusions, but that should encourage, not detract, from the larger discussion. Carleton is a fantastic analyst, and we’re thrilled to see him dig into a topic so near and dear to our hearts.