The Strikeout Push/Pull Effect

              Hey, guys.  I am aware that I owe you one article from the Rookie series, and I’ll get to that next, and then we’ll get back to the Bill James Life in Prison Baseball Game, try to re-launch that, but first of all I got carried away with a serious research question, and I’ve spent a week or so on that issue.   This is an issue that I have discussed before and researched before, but this time I did it much better than I have before, studied it more systematically and in more depth, and came away with a better understanding of what is happening. 

              It has been my theory for about ten years that the historical upward drift of strikeouts, the ever-increasing number of strikeouts in the game, is caused by an imbalance of forces in the selection criteria.  Basically, the problem is that

(a)    Pitchers who get strikeouts are better pitchers than those who do not, but

(b)    Batters who strike out are NOT significantly worse, on balance, than batters who don’t strike out. 

Not only are strikeout-prone hitters historically not worse, they are, over a period of time, slightly better.  Hitters who strike out, like Babe Ruth, Mickey Mantle, Mike Schmidt, Reggie Jackson and Aaron Judge, are still enormously valuable despite the strikeouts.

Because of (a) and (b) above, teams are constantly looking for strikeout pitchers—but are NOT looking to avoid strikeout hitters.   They’re not looking to avoid guys like Mike Schmidt and Aaron Judge; they’re looking to find them.   So you have upward push on the strikeout column from pitcher selection, but no downward push from batter selection.   The result of this is that strikeouts go up over time. 

In other words, what should (you would think) be a relationship of push and push back is, in practice, a relationship of push and pull.   What we will call PUSH is the pitcher’s contribution—the fact that teams are always looking for pitchers who can strike people out.  But what should be pushBACK—teams looking for hitters who don’t strike out—isn’t there.  Very often, it’s a push/pull relationship in which both forces are operating in the same direction. 

This has been my theory for about ten years, but now I need to stress that this is simply a theory.  It is not a proven fact.  It could be true; it could be partially true but not absolutely true.  It could be false; it could be that there is some other explanation that works better.   I am trying to keep an open mind as I study the issue. 

This issue was brought to the front of my mind as I was writing an article for the 2019 Bill James Handbook about trendlines in baseball.  I wrote the following:

At some point, this has to end, right?  At some point, strikeouts have to become SO high that batter effectiveness and strikeouts have to part company, or at least this is what I believe.

In 1978 the ten teams which struck out the most often (as hitters) had a collective won-lost record of 849-769, 80 games over .500.  The ten teams which struck out LEAST often had a collective record of 762-852, 90 games under .500.  The high-strikeout teams were better teams—indicating that strikeouts were not a negative for their teams, but merely the fellow travelers of home runs.

By 2008 the gap perhaps was narrowing; the advantage of the strikeout teams was diminishing.  In 2008 the ten teams which struck out the most were collectively 27 games over .500 (822-795), but the ten teams that struck out the least were 7 games over (814-807).  The gap may (or may not) have become less than it was.

But in 2018 there was a different picture.  In 2018 the ten teams that struck out the most often had a collective record of 765-857, or 92 games under .500.  The ten teams which struck out LEAST often had a collective record of 861-758, or 103 games over .500.  The pattern is actually stronger if you stick to five teams.  The five teams which struck out most—the White Sox, San Diego, Philadelphia, Texas and San Francisco—all had losing records, while the five teams which struck out least—Cleveland, Houston, Seattle, Pittsburgh and Boston—all had winning records, and three of the five won their divisions.

What I am saying is that in modern baseball, it may no longer be true that strikeouts are not a meaningful drain on a team’s offensive energy.  It may be that we have reached the point, or at least are near the point, when teams WILL begin to avoid hitters who strike out. 

After reading that, I took note of this passage in Rob Neyer’s new book, Power Ball (page 128):

From the mid-1960s through the mid-90s, the strikeout rate essentially held steady—aside from an odd downturn in the late 70s—at around 15 percent of plate appearances, give or take a percentage point or so. But for the next twenty years, the rate edged toward 20 percent. Finally, three years ago fully one-fifth of all plate appearances ended in strikeouts. Last year it was 21 percent; this year, nearly 22 percent. 

And there’s simply no reason to think this trend is going to reverse itself anytime soon.

So I was hoping to be able to tell Neyer, "Actually, Neyer, there IS a reason to think this trend is going to reverse itself soon."  My studies didn’t exactly work out that way (he said, tipping his pitches), but that was my thinking.

My observation in the Handbook is valid and relevant, I think, but there are four obvious problems with it:

First, the observation spot-checks data from just a few seasons (1978, 2008), and compares that to 2018, leaving the appearance of a trend where no trend may exist in fact. 

Second, the data that I cited is team-level data, but the effect, if it is really there, would operate on the level of individual players, not teams.

Third, I used the ten teams on the extremes to represent the "slant" of the data, rather than doing a correlation study that would include all of the teams in the study and evaluate the degree to which each one provided evidence about the issue.  In other words, you can get a different answer, perhaps, if you use eight teams or six teams or twelve teams.  Just using X number of teams is a loosey-goosey approach to the problem.

And fourth, the data that I sited addresses only the "batter" side of the issue, while ignoring the "pitcher" side of the issue, which may be more important.

In other words, what I referenced in the Handbook is not a real study of the issue; it is just picking and choosing data that seems to support my point—exactly what traditional sportswriters do when they say that Lefty O’Doul ought to be in the Hall of Fame because he had a career batting average of .349.   THIS is a real study of the issue, what we’re doing today.   I looked at both sides of the issue, looked at much larger sections of the data, looked at it on the player level rather than the team level, and did actual correlation studies although, being the obstreperous SOB that I am, I then aggregated the correlation data in ways that would probably cause a legitimate data scientist to sneer viciously and shout at me to leave the stage.  My skills only go so far. 

By doing this, I did in fact gain a much better understanding of the issue I was studying.  I picked up a major new insight into the problem.  This was satisfying, because very often you put 8 days into a study like this and you don’t learn a damn thing.  This time I actually gained understanding of the issue. 

First we have to define the data set.  Perhaps I should put in a sub-heading there:

Outlining the Study, Part I—Batters

This is boring math, and also, if you don’t get math, you may not understand what I am talking about.   If you don’t get math, you may want to skip ahead to "Conclusions, Part I—Batters". 

First we have to define the data set.   I studied all players (hitters) who had 300 plate appearances in a season in the last 100 years, 1919 to 2018.   For each player, I used two data points:  his strikeout rate and his runs created per 27 outs.   For strikeout rate, I used strikeouts divided by plate appearances. 

For each season, I determined the correlation of runs created/27 outs with the strikeout rate, using a Pearson Product/Moment correlation.  If the two were positively correlated, that means that, in that season, the good hitters (as a group) struck out more often than the less-productive hitters.  This was true, for example, in every season from 1960 to 1984.  If the two were negatively correlated, as they were, for example, this season, that means that the good hitters struck out less often than the weaker hitters. 

In 1970, picking a season at random for illustration.  In 1970 there were 212 players who had 300 or more plate appearances.  With 24 teams that is 8.83 players per team with 300 plate appearances, a normal number.  The average strikeout rate of those 212 players was 130 strikeouts per 1000 plate appearances, while the average runs created per 27 outs was 4.985. 

The three best hitters of 1970, runs created per 27 outs, were Carl Yastrzemski (10.42), Rico Carty (10.24) and the late Willie McCovey (10.12).   All three of those players had BELOW average strikeout rates, so all three of them made NEGATIVE correlation contributions.   Yastrzemski, for example, struck out only 66 times in 698 plate appearances, or .095.   We multiply the extent to which he was better than the average hitter in the group (10.42 minus 4.985) by the extent to which he struck out less than the average hitter (.095 minus .130), and you get negative .193.   His contribution to the correlation is negative .193. 

That’s important—it is one of the largest contribution figures of the 1970 season—but it is less than the correlation contribution of Bobby Bonds.   Bonds was also an above-average hitter, contributing 7.45 runs to his team for each 27 outs, but he struck out 189 times, which was a major league record at that time and a record that stood for many years after that.  He struck out in 25.4% of his plate appearances—actually not that far above the 2018 average, but way above average in 1970.  Bonds’ contribution to the correlation in 1970 was (7.45 – 4.985) X (.254-.130), or +.304.  

Of the 212 players from 1970, 122 had positive contributions to the correlation (meaning that they were either good hitters who struck out a lot or weak hitters who struck out little), while 90 (like Yastrzemski, Carty and McCovey) had negative correlation contributions.  Overall, the sum total of the 212 players was +2.94.  

If the two categories had been perfectly aligned—that is, if every hitter who struck out a lot was also a good hitter, and if the runs created/27 outs were in exactly the same order as the strikeout rates, and if the proportions between them were exactly the same relative to the standard deviations of each. . ..if the two were perfectly aligned, then the sum total would have been +18.63.   It was +2.94, so we divide 2.94 by 18.63, and we get +.158.   The correlation between strikeouts and offensive productivity, for 1970 hitters, was +.158. 

This is a high figure by historical standards.  For the 100 seasons that I studied, this was the fourth-highest.   The correlation was positive in 53 seasons, negative in 47 seasons, and this was the fourth-highest positive correlation. 

Conclusions—Part I, Batters

There were 53 seasons in which strikeouts were positively correlated with offensive productivity, and 47 seasons in which the correlation was negative (you just said that.)   

The correlation was positive in every season from 1926 to 1942 except two (1936 and 1937).  This is what we could refer to as the Babe Ruth/Jimmie Foxx/Hank Greenberg era, when there were several or many big sluggers who struck out a lot by the standards of the era. 

The correlation was negative for every season between 1943 and 1959, except 1956.  This is what we could refer to as the Joe DiMaggio/Ted Williams/Stan Musial/Yogi Berra era, when most of the best hitters in the game did NOT strike out much. 

From 1960 to 1984, the correlation was positive in every season without exception.  This is what we could refer to as the Mickey Mantle/Harmon Killebrew/Dick Allen/Reggie Jackson/Mike Schmidt era, when the game was dominated by star hitters who often led the league in strikeouts. 

From 1985 to the present, the correlation has generally been negative.   It has been negative 25 times in that 34-year period, and positive 9 times.   For the last 34 years, the majority of good hitters have NOT been hitters who struck out a lot. 

While the correlation has generally been negative for a third of a century now, and while I definitely should have picked this up many years ago, the negative correlation is stronger now than at any other point over the last 100 years.   The 2018 figure was negative .141, the second strongest negative relationship of any season in the study.   The only stronger negative connection was in 2014, when it was -.161.   This chart gives the correlation of strikeouts to offensive productivity in every season of the last 100 years. 

And here is the same chart for the benefit of Dave Studeman and any others of you who are color-blind:

The total for the last five seasons (2014 to 2018) is negative .513, which is the lowest total in the history of baseball or at least the last 100 seasons.   In other words, we ARE (in a certain sense) back to the Ted Williams/Stan Musial/Joe DiMaggio/Yogi Berra era, when the best hitters generally do NOT have high strikeout totals.

For those who would like to see an end to the endless escalation of strikeouts, that’s an encouraging development.   If we stopped the study at this point, we could conclude that the endless increase is strikeouts may in fact be near an end, because there is now significant pushback from the hitters.   If we stopped now, we could say "see this, Rob Neyer, there actually IS reason to believe that this trend will reverse itself." 

Tempted as I was to do that, I am unfortunately too intellectually honest to let myself do it.   It has never been the hitters who were basically causing the upward trend in strikeouts.  It has always been the pitchers (pitcher selection) who were basically causing the upward push in strikeouts.   The role of hitters (hitter selection) was simply that it was not effectively pushing BACK. 

So now hitter selection is pushing back a little bit, but is it enough? 

By my theory, by the strikeout push/pull theory, the upward trend in strikeouts should end when the negative correlation of hitter strikeouts to productivity is equal to the positive correlation of pitcher strikeouts to pitcher effectiveness.   But have we reached that point, or are we near that point?   To answer that question, we have to study the pitchers. 

Outlining the Study, Part II—Pitchers

The process for pitchers was essentially the same as the process for hitters.   I included in the study all pitchers who

(a)    Pitched 100 or more innings in the season, or

(b)    Had 50 or more "points", where points are Games Started + Game Appearances, a combination very commonly used in pitcher contracts. 

Then, after defining the group, I did essentially the same things I had done with the hitters.  The two things that I correlated for pitchers were:

(a)     ERA, and

(b)    The percentage of batters struck out.   Strikeouts, divided by batters facing pitcher. 

For illustration, in 1919 there were 90 major league pitchers who met the workload standards to be included in the study.   Those 90 pitchers had an average strikeout rate of .083 (8.3%) and an average ERA of 2.99.  (The major league ERA was 3.07, because those pitchers not included in the study were somewhat less effective than those who were included in the study.)   Hod Eller of Cincinnati had the highest strikeout rate in baseball, 13.9%, and also had a good ERA (2.40), so we mark that as a positive.  He had a positive correlation contribution of +.033. 

The sum for all major league pitchers was +.372 out of a theoretically possible 1.508, for a 1919 correlation of +.247.

The Major New Insight that I Have into this Problem

That I gained from doing this study

The increase in strikeout rates is feeding on itself.

I feel stupid for not having realized this before now, but as you no doubt can recognize, there are many processes in nature that feed on themselves, growing stronger because they have grown stronger.   A hurricane is one such effect, or a tornado.  A straight wind may reach a certain maximum speed, 50 MPH or 60 MPH or whatever, with GUSTS of wind that may be higher.   What happens with a tornado or a hurricane is that the wind forms a circle and blows into itself, making a sustained wind that is much more powerful than a gust of wind.   It is generating its own momentum.  A snowball rolling downhill is the clichéd example of this, or an avalanche. 

Strikeouts are increasing because strikeouts are increasing.   Stupid of me not to see it before now.  

Look, Hod Eller in 1919 pitched 248 innings and struck out 137 batters.   This was the highest strikeout rate in baseball, five strikeouts per nine innings.   A typical pitcher, pitching the same number of innings in 1919, would have struck out 80 to 85 batters. 

The correlation of strikeout rate to pitcher effectiveness was positive in 1919 (+.247), but weak by historic standards.   When you think about it, this obviously has to be true.  When there aren’t that many strikeouts in the game, strikeouts are just not that important, and therefore the extent to which the best pitchers are strikeout pitchers is limited.   Other things become more important.   Not walking batters is more important.  Getting ground balls is more important.  Holding the baserunners is more important.   Getting strikeouts is not that important because it’s just a few plays a game. 

Because strikeouts are not that important, the ability to get a strikeout is not a central or not the central selection mechanism for pitchers. 

As strikeouts become more common, they become more important.  

As strikeouts become more important, the extent to which the best pitchers are strikeout pitchers increases.

As the extent to which the best pitchers are strikeout pitchers increases, the extent to which future pitchers are selected by their ability to get strikeouts increases.

As the extent to which pitchers are selected by their ability to get strikeouts increases, strikeouts become more common.

As strikeouts become more common, they become more important. 

              As strikeouts become more important, dot dot dot. 

              For ten years or more, I have explained the relentless increase in strikeouts as a function of the asymmetry in the relationship of pitcher strikeouts/pitcher effectiveness to the relationship of batter strikeouts/batter effectiveness.   But I now realize that there is a second dynamic involved here, which is probably as important or more important than the other one.  Like a hurricane, the cycle of strikeouts increasing is feeding on itself.

Conclusions—Part II, Pitchers

We are nowhere near the point at which the downward pressure on strikeouts (from the batters) is equal to the upward pressure (from the pitchers), and, in fact, we may not be much closer to that point now than we were years ago. 

The realization in the point above is an outgrowth of the realization that, over time, the correlation of strikeouts to effectiveness among pitchers has increased dramatically, as the number of strikeouts has increased.  Once you realize that, it’s an "Oh, of course."   I had just never before realized that, never before realized that the extent to which the best pitchers were strikeout pitchers had increased over time.

Unlike the batter correlation, which is sometimes positive and sometimes negative, the correlation between pitcher strikeouts and pitcher success has always been positive.  There has never been a year when it wasn’t positive.   

The correlation has always been positive, but it has become more strongly positive over the years.   In the first 25 years of the century we are studying, the correlation average was +.286.  In the second 25 years, it averaged +.309.   In the third 25 years, it averaged +.388.   In the fourth 25 years, it averaged +.494.   In the last five seasons it has averaged +.513.    As strikeouts have risen, so too has the value of being a strikeout pitcher.

Conclusions—Part III, Combining the Pitching and Hitting Studies

It could be true, then, that:

a)       There is now meaningful pushback against rising strikeout rates, caused by the fact that the best hitters now are players who don’t strike out as much, but

b)      This effect is still very small compared to the "push" effect of strikeout pitchers being better than non-strikeout pitchers, and

c)       The growth in strikeout rates is still pushing on itself, more than negating the increase in pushback in recent years.

So it remains possible and perhaps likely that strikeout rates will continue to go up.

Sorry.

That wasn’t the result I wanted, either.   This chart combines the pitcher’s correlation (PUSH) and the batter’s correlation (PULL) into one number. 

I’ll go over that one quickly, because the one-year chart doesn’t show trends clearly.  It’s not a one-year effect.   The theory here is that pitcher selection is driven by the types of pitchers who are successful.   That’s not a one-year process; it’s a process that happens over time.  These charts below combine the one-year effects (above) into five-year totals. 

The "1.05" number above combines the Push and Pull Effects (that is, the pitcher’s correlation and the hitter’s correlation) from 1921 to 1925 into one number, which is 1.05.   That means that the average of those ten effects is just +.105—a very weak effect.  Strikeouts are being pushed upward, but at a relatively slow pace. 

Now I will take the "from" and "to" out of the charts, and just refer to "1921 to 1925" as "1925".  That means it is the five-year average, measured from 1925.  From 1925 to 1934, this number went steadily upward:

By 1934 the upward pressure on strikeouts had nearly doubled.   From 1934 until 1957, however, this number went DOWN steadily:

For the sake of clarity, there was always upward pressure on strikeouts resulting from the asymmetrical pressures of pitcher selection (Push) and batter selection (Pull).   However, that pressure, during this period of time, was dropping steadily, and was much lower in 1957 than it had been in 1934.   The 1957 measurement is the all-time low point of this chart.   After 1957, the upward pressures on strikeouts accelerated at a remarkable rate:

Let us ask what was happening, in that era, which would have caused this quite remarkable jump in the upward measurable pressures on the strikeout rate?

1)      Sandy Koufax emerged as clearly and obviously the best pitcher in baseball.  In 1957 Koufax was in the majors and had a very high strikeout rate, but was an ineffective pitcher.  

2)      Koufax’ emergence (and Drysdale’s dominance) was followed by the emergence of many other power pitchers—Bob Gibson, Bob Veale, Sam McDowell and others.  In the 1950s the Cy Young Award winners were Don Newcombe and Vern Law, who "dominated" by never walking anybody, a 36-year-old Warren Spahn and a 39-year-old Early Wynn.  (Law actually won the Cy Young in 1960.)  In the 1960s they were power pitchers. 

3)       Many teams built up their mounds, beyond regulation height, to give their pitchers an advantage.

4)      The strike zone was re-defined (1963).

5)       There was a very dramatic shift in the thinking of managers about the irregular use of pitchers.  In the 1950s managers commonly used pitchers in what would later be regarded as highly irregular patterns.   Between 1959 and 1965 this changed, to an enormous extent, largely unrecognized by baseball historians although it can be clearly documented.  

The upward pressure on strikeouts can be seen in the chart beginning in 1958, which is actually the years 1954 to 1958.  Thus, it can be argued that the push/pull effect predicts the pitching-dominated 1960s, that the signs of this happening emerged several years before it did happen.   That can be argued, and it may be true, or it may be just picking and choosing data from the chart that is convenient to the argument.   I don’t actually know which one of those it is.

In any case the push/pull effect after 1966 was flat for ten years, but remained at a very high level:

After 1976 the Push/Pull Pressure grew more intense once again, reaching a new peak of 2.66 in 1981:

After 1981 the pressure for more strikeouts relaxed significantly for six years:

But beginning in 1987, it then went up for another 23 years, reaching an all-time high of 2.79 in 2010:

Since 2010 the upward pressure on strikeouts does appear to have relaxed somewhat:

However, while the upward pressures on the strikeout total are not now what they were in 2010, they remain relatively high by historic standards.   

By any possible interpretation of the data, we are nowhere near the point at which the downward pressures on strikeouts resulting from the fact that the best hitters don’t strike out much are equal to the upward pressures resulting from (a) the fact that the best pitchers get strikeouts, and (b) the hurricane effect, that as strikeouts become more common, that pushes managers to find strikeout pitchers.   In other words, there is no evidence in the data that the 120-year trend toward more strikeouts in the game is near its end. 

That doesn’t mean that it isn’t near its end, either.   The exact relationship between future strikeouts and the upward pressure on strikeouts resulting from asymmetrical forces and is unclear, if in fact such a relationship exists.  It’s a long-term, multi-year observation of the difference which creates a long-term increase in strikeout rates (if that is in fact what is happening.)  You can’t look at the data and see an obvious relationship between how strong the forces are and how rapidly the strikeout rate increases.   There may be such a relationship in there somewhere, but you can’t easily spot it.   The rules of science require that if we can’t demonstrate a relationship between the measured forces and the "resulting" increase in strikeouts, we have to assume that there is no such relationship.  

I do believe, honestly, that we could be near to a turning point.   The forces we are trying to measure here operate through the collective mind of baseball people.  Two or three years ago, when most of baseball was focused on "launch angle" and apparently ready to accept unlimited strikeouts to buy launch angle, the Red Sox and the Astros went in the other direction.   They went the non-strikeout route.   In 2018, the two best teams in baseball, probably, were the Red Sox and the Astros.   I do believe that this may turn out to be a significant development.  Baseball imitates its winners, of course, understanding that you have a new winner every year or most years.  What won last year may not win next, and there may be no lasting effect to the 2018 success of the low-strikeout teams. 

But then again, there could be.