Suppose that you took all seasons by relief pitchers, ever, and sorted them into ten levels of contribution.   Aroldis Chapman in 2013, for example, had 38 saves, a 2.54 ERA, and struck out 112 batters in 63.2 innings.   That’s a "10" season—not that he was the best reliever in the majors, but that he was in the top 10%.

Neal Cotts of Texas had only 1 save and 11 holds—unimpressive numbers—but won 8 games, lost 3, and had a 65-18 strikeout to walk ratio and a 1.11 ERA.   That’s a  "9".

Dane de la Rosa of the Angels had only 2 saves but 20 holds, a 6-1 won-lost record, 65-28 strikeout to walk ratio and a 2.86 ERA.   That’s an "8".

Kevin Gregg of the Cubs had 33 saves, but the rest of his numbers were just fair—3.48 ERA, 2-6 won-lost record, 56-32 strikeout to walk ratio.   That’s a "7"; we’re not ignoring the Saves, but it’s not a super season, either.

Will Harris of the Diamondbacks had no saves and only four holds, but he was 4-1, 2.91 ERA, 53-15 strikeout to walk ratio.   That’s a "6".

Kelvin Herrera of the Royals struck out 74 batters in 58.1 innings (74-21) and had 20 holds, but his ERA was just 3.86, 5-7 won-lost record.   That’s a "5".

Nate Jones of the White Sox pitched 70 times, 89 strikeouts in 78 innings (89-26), and had 16 holds, but his ERA was over four (4.15).   That’s a "4".

Brandon League of the Dodgers had 14 saves and 6-4 won-lost record, but struck out only 28 hitters (28-15) and had an ugly 5.30 ERA.    That’s a "3".

His teammate Carlos Marmol, inventor of Marmolaide, had a better ERA, 4.41, but he had only 2 saves and 2 wins (2-4), and he walked 40 men in 49 innings (59-40).    That’s a "2".

Marmol’s first-half-of-the-season teammate, James Russell, had a yet-better ERA, 3.59, but had no saves, a 1-6 won-lost record, unimpressive strikeouts and walks (37-18), and gave up an OPS higher than 1.000 to right-handed hitters.   That’s a "1", a bottom-ten-percent season for a reliever.

And, I don’t know if you noticed, but I gave you those pitchers not only in order of performance, but also in alphabetical order—Chapman (10), Cotts (9), de la Rosa (8), Gregg (7), Harris (6), Herrera (5), Jones (4), League (3), Marmol (2) and Russell (1).

                I took all pitchers in baseball history who had 40 or more game appearances, and no more than 4 starts. . .basically, all relievers ever who got in something close to a full season’s work.  There were 5,207 such pitchers.   I sorted them into 10 levels—520 in the top group, the "10s", 520 in the bottom two groups, 521 in the each of the other 7 groups.

                The first pitcher who qualified, chronologically, was Lou North in 1921—40 games, no starts, 4-4 record, 3.56 ERA, 7 saves credited in the modern encyclopedias, although of course there was no such concept at the time.  That’s a "6" season, not a bad season, not really much of an impact.   This is a chronological scan of the 5,207 seasons:

                One can see from this how impossible it would be to address the question, "When does the modern bullpen really start?", because each decade is different from the one before, and the increases are steady and proportional, rather than having a sudden increase at any point in time which would allow us to draw a dividing line.

                The first reliever to have a "10" season was Garland Braxton in 1927.   Braxton, who was called "Gob" (presumably he had been in the Navy. . .sailors were called "Gobs") was a rail-thin, Gomer-Pyle-looking like guy who kept his hat pushed way back on his head, the bill standing straight up in the air.   In 1927 he made only 2 starts but pitched 156 innings with a 2.94 ERA, 10-9 record, 13 saves and a 96-33 strikeout to walk.    The strikeout to walk ratio is sensational for that era; in the American League in 1927 there were 600 more walks than strikeouts.   This is a chronological spectrum of the "10" seasons:

                Some of you will remember that Wilcy Moore won 19 games, primarily as a reliever, for the 1927 Yankees, but that season doesn’t qualify because Moore made 12 starts.   I would like to have 10% of the best seasons by relievers be designated "10s" in each decade, but of course the only way you can make that happen is to overpower reality and force it to happen.    In the 1970s 17% of reliever seasons were "10s"; now, it is down to 7%.    Teams in 1970 typically had three-man and four-man bullpens; now, most teams have 7-man and 8-man bullpens.   When you have three pitchers doing the work of eight, you have more Big Seasons; there’s really no way of getting around it except to ignore it and pretend that pitching 70 innings now is the same as pitching 140 innings of relief in 1973.

                Some of you will insist on knowing how I sorted the seasons into levels, so I suppose I will have to explain that.   I started with the pitcher’s Season Score, which I have explained before several times.  To this, I added:

                1)  One point for each run that the pitcher saved compared to the league average in that season (since Season Scores are not otherwise era-adjusted), and

                2)  One-half of one point for each strikeout above (or below) the league strikeout average (since strikeout levels have changed so much over time), and

                3)  100 points to each season (so that I wouldn’t have to deal with negative numbers.)

                An average "10" reliever pitched in 66 games, 93 innings, with a 7-4 won-lost record, 81-30 strikeout to walk ratio, 2.25 ERA and 29 saves.    This chart summarizes the performance averages for each level of success:

                By doing this, I was able to create a simple "scan line" for every reliever.   These would be the top ten relievers of all time, giving them 10 points for a "10" season, etc.:

                The "EYOB" or "Effective Year of Birth" is given so that you can relate the peaks and valleys of each pitcher’s career to calendar years.   Mariano was born in 1969 but in late 1969, so he is treated as 30 years old in 2000, so his Effective Year of Birth is 1970.   2000 is the year he was a "10".   It’s a joke.

                I can foresee one consequence of publishing this right away:  those people who are advocating for John Franco to be in the Hall of Fame will now start saying that "Bill James says that he was the 4^th greatest reliever of all time."   I’m not really saying that; that is merely the conclusion you would reach by the use of his approach.   Maybe a "10" season is worth 20 times as much as a "1" season, rather than 10 times as much; I don’t know.

                Anyway, this is the "second ten", which has to be eleven because there is a tie:

                And this would be the next ten:

                Mike Timlin would be the #1 reliever of all time who never had a "10" season.    There has been at least one "10" reliever every season since 1949 except for 1994, when the strike and a couple of injuries kept the top shelf cabinet bare for a year.

                I note that relievers are acquiring a true history; more and more of the relievers now are from 30, 40 years ago.   A few years ago, when you did a list of the greatest relievers, it was dominated by active and recently retired pitchers.   Not so much now; Roy Face, Ron Perranoski, Hoyt Wilhelm and Don McMahon were a half-century ago, and many of the pitchers listed above were active or finished by 1984, 30 years ago—Garber, Sutter, Dave Smith, Tekulve, Gossage, Lee Smith, Reardon, Sparky Lyle, Tug McGraw, Rollie Fingers.   The position is beginning to develop historical depth and historical standards. 

                OK, turning now to the serious research.  I created these summary lines in order to improve my ability to generalize about relievers sustaining success levels.    Of pitchers who are "10s" in this study, the average next-season performance level is 6.58—6.29 before 1990, 6.93 since 1990.

                Prior to 1990, 53% of "10s" in this study were at the 8, 9 or 10 level the next season, and 44% were at the 8-9-10 level two years later.   Since 1990, those percentages are 63% and 51%.  It may be that something has changed since 1990 because:

                a)  Up until 1990, relievers who had outstanding seasons were sometimes moved to the bullpen,

                b)  Relievers in the 1970s and 1980s were worked much harder than they are today, leading to more injuries, or

                c)  Early relievers were often second-line talents with trick pitches, older guys and broken-down starters and failed starters.   Now front-line relievers are more likely to feature some serious fastballs. 

                From my standpoint, it doesn’t much matter WHY these numbers have changed; what really matters is the fact that they have, which means that I will have to be cautious in using data from before 1990 in studying anything else relevant to today.

                Restating, "10" relievers since 1990 have had an average next-season score of 6.93, 63% of them were still highly effective relievers one year later, and 51% were still effective relievers two years later, "highly effective"  being defined as "8-9-10" on this scale.    For the "9" relievers, since 1990, the average next-season score is 5.46, 40% are still in the group one year later, and 30% are still in the group two years later.

                For the "8" relievers since 1990, these numbers are 4.82, 26% and 19%.     Charting:

                Generalizing, the chance that a mid-range reliever will become a highly effective reliever one or two years later is around 10%.    An ineffective reliever becoming a highly effective reliever, it’s a real long shot, although obviously is does happen.

                My next little study here deals only with pitchers in the top three levels of effectiveness, 8-9-10, and only since 1990, and obviously not including the 2012-2013 pitchers, since there is no "lookahead" data on them.    There are 686 relief pitchers in the data, all of whom had very good seasons. 

                Those 686 pitchers contain some who had amazing control—Dennis Eckersley, for example--and some who did not have amazing control, such as Mitch Williams and John Rocker.   483 of the pitchers had better-than-league walk rates, adjusting for Intentional Walks, but 203 did not.   Those who had better-than-league control were slightly more effective than those who had below-average control, but only slightly more.

                In subsequent seasons, would those who had good control be more likely to continue to be effective than those who issued more walks?   Remember John Axford, had a couple of good years as a closer but would walk people?   Remember David Aardsma, Don (Fullpack) Stanhouse, Ryne Duren?   It seems like that is kind of a transient thing, that sometimes a reliever is able to be highly effective for a year or two with a scattershot approach, but then it gets away from him.

                No.   I would have guessed the answer would be "yes", but it isn’t.   In the base year, the "good control—high effectiveness" group had an average grouping of 9.03, and an average next-season grouping of 5.70.   42% of them remained in the "highly effective" groups the next season, and 34% did so two seasons later.

                In the "poor control—high effectiveness" group, the base year group average was 8.77.   But the next year, their average grouping was 5.71—the same as the control pitchers—and 42% of them remained in the "highly effective" groups, the same percentage as for the good control pitchers.    The percentage remaining effective two years later did drop slightly, to 31%, but that’s really nothing.

                If you focus on the extremes—the 50 pitchers with the BEST control and the 50 pitchers with the WORST control—then it does appear that there is some "persistence advantage" to the pitchers with good control (46%/36% versus 32%/24%).    But small groups become less reliable.

                Suppose that we sort the highly effective relievers by strikeout rates.   To begin with, 88% of the highly effective relievers (603 out of 686) had above-average strikeout rates, and those who had the highest strikeout rates were more effective than those whose strikeout rates were just moderately high.   They had a few more saves and a little bit better ERAs.

                But suppose we divide those pitchers into quadrants, by their strikeout rates (171-172-172-171).    Pay dirt:

                The low strikeout pitchers who are highly effective are much less able to maintain their effectiveness, in future years, than the high strikeout pitchers.  (The "Below Avg. Strikeouts" group there is not really below the LEAGUE average.   They are merely below the midpoint of this group, although 100% of those pitchers are still above the league average.   Even about one-half of Quadrant 4, the Lowest Strikeout Group, is still above the league strikeout average.)  

                Anyway, that’s really useful to know:   Control rates (for effective relievers) are not a variable predicting the pitcher’s ability to remain effective, but strikeout rates ARE.     I would assume that batting average on balls in play is?

                Not really, no.

                There is some survival advantage for the pitchers who have high batting averages on balls in play, but it is much smaller than I would have expected.   I don’t know how to interpret that.    Just trying to make sure everybody is keeping up. . .. .when a pitcher has a low batting average on balls in play (BABIP), we assume that he has been lucky, and has had the at’em ball working.   We thus assume that he is likely to be LESS lucky the next season, that he will be no more lucky or less lucky than anyone else, thus that he may lose effectiveness.    This is true in this group, but not profoundly true, not true enough that there would be any profit in focusing on it.

                The highest BABIP in this group was .365, against Roberto Hernandez with the 1995 White Sox.   Roberto Hernandez could throw 97, 98 MPH, pitch after pitch, when he was 37 years old, closing games for the Royals, and was still in the mid-90s when he was 40.   At one point we were thinking of making a trade offer for him, so we sent Bill Lajoie to scout him for several days.   Bill returned with the best ten-word scouting report I ever heard:  "I don’t know how they hit him," he said, "but they do."

                Carlos Marmol one year had an in-play average of .169.   Well, I think that’s about all I’ve got here that is worth saying.   I did this research, hoping to construct a way to look BACK at relievers, asking "Where did this good reliever come from?   Was he a minor league starter before he became an effective reliever, a major league starter, a minor league reliever, an ineffective major league reliever, a college pitcher?   What was he, before he became the guy we are trying to find?"

                I don’t know that I’ve found any answers to that one; perhaps I can continue to work on it.   When we publish this article I’ll post the file which reduces each reliever’s career to a single line of 1s through 10s, just in case that file (spreadsheet) is of any use to any of you; its name is Relievers Scaled 1 to 10.xls   Thanks for reading.

Bill James