I am reporting today the results of my study, which I described yesterday, in which I asked "which categories of a pitcher’s record are the best indicators of his overall value?" I’m going to start at the bottom.
1) Home Runs Allowed (55.4%). Among the twelve pitching categories that I studied, Home Runs Allowed are the poorest predictor of overall value.
I figured Home Runs Allowed per 27 innings, even though the innings were essentially a constant. The pitcher allowing fewer home runs was the better pitcher 55% of the time.
Two pitchers in the data, Denny Driscoll in 1882 and Bill Steen in 1914, allowed no home runs at all in their 200 innings. Both pitchers are scored at 2.7 WAR, which is above average, but barely above average, the average being 2.48. On the other end of the scale is Bronson Arroyo, 2011, who gave up 46 bombs in 199 innings.
2) ERA (57.4%). Among the twelve elements studied here, ERA was the poorest predictor of true value, other than the rate of Home Runs Allowed. Obviously this is a surprising conclusion, and I will discuss the implications of this below, when we are talking about Winning Percentages.
3) Runs Allowed Per 9 Innings (58.9%). Runs Allowed Per 9 innings did slightly better, as a predictor of true value, than Earned Run Average.
4) WHIP (60.01%). Walks + Hits per Inning Pitched was 60% accurate as a predictor of actual value in this study—better than ERA and Runs Allowed, but not much better.
5) Strikeout Rate (60.5%).
6) Won-Lost Records (60.9%). OK, the most interesting conclusion from these studies is the fact that won-lost record and its brother, winning percentage, perform better as a predictor of true value than ERA and its brothers, runs allowed per 9 innings and WHIP, so let’s deal with that here.
First of all, we should make sure everyone understands that the "ERA" that performs poorly here is ERA not adjusted for the league ERA, and that this comparison is of different pitchers over a long period of time. We are comparing here, for example, Johnny Lush in 1907 with Jamie Moyer in 2005. Lush in 1907 had an ERA of 2.64, but the National League ERA was 2.46. Moyer in 2005 had an ERA of 4.28, but the American League ERA in 2005 was 4.76. Lush had a "better" ERA only if one fails to adjust the ERA for context, and it is not surprising that Moyer’s won-lost record (13-7) was better than Lush’s (10-15). ERA is of course better as a predictor of true value than the won-lost record if one is comparing two pitchers in the same league, and ERA may be better than the won-lost record if one restricts the time frame to a limited period of years, so that there would be less variation in the league standard.
I started to think about this article in January, when I was on Brian Kenny’s show, and Brian asked me whether I would join in the effort to get rid of the "Win" stat. I said "No", and the reason I said "No" is that I’m not generally in favor of getting rid of information. I’m not in the business of eliminating data.
Critics of the won-lost record are focused on the flaws in won-lost records, the biases and inconsistencies—but all statistics have biases and flaws of the same nature. The won-lost record is misleading because some teams score many more runs than other teams, which makes it much easier for their pitchers to win—but it is also much easier for a pitcher to post a low Earned Run Average in some leagues than it is in others, and in some parks than it is in others.
I started thinking, after that, about how I could compare the validity of a pitcher’s won-lost record to the validity of his ERA. I am very surprised that won-lost records outperformed ERA in this study, and I certainly did not design the study so as to favor that result. I assumed that ERA would be a better predictor of actual value than the won-lost record. It isn’t.
It isn’t over time; it might be over a shorter period of time. The critic of the won-lost record could make the following argument: that we mentally remove the biases of the ERA when we use it, whereas people who cite won-lost records speak about them as if the pitcher actually "won" or "lost" the game. All by himself.
But is that true? This is an argument about historical stats, about the fair comparison of Jack Morris to Jim Palmer to Burleigh Grimes. I might argue that fans in general have a stronger grasp of the biases inherent in won-lost records from the teams on which people pitched than they do of the ERA bias from leagues. Quick now, which league has a higher ERA: The National League in 1933, or in 1969?
The National League ERA in 1933 was 3.33; in 1969 it was 3.60, and in 1970 it was 4.05. Maybe you got that one right, because you guessed that I was throwing you a curve ball, or maybe you got it because you are an expert in the history of league ERAs, but I think most people would assume that the National League ERA was higher—and probably much higher—in 1933 than it was in 1969. We organize data into batches, in our minds; we simplify to make storage room. Comparing a 1930s season to a 1960s season, most people are going to assume that the 1960s ERA should be lower, but in fact the National League ERA was higher in 1961 than in 1931, 1932, 1933, 1935, 1936, 1937, 1938 or 1939. The National League ERA in 1966 was higher than in 1933, and at the same level as most of the 1930s. We don’t really have specific enough information in our heads to counter-act that bias.
And that is just one of two major biases in the ERA stat (and similar and related stats, such as WHIP.) After you adjust out the leagues, then you’ve got to worry about the parks.
The won-lost record has a tremendous advantage over almost any other stat, in that the winning percentage is always .500 in every decade, in every year, in every month—and in every park. If you remember that the Orioles of the 1970s were a consistently outstanding team—which I think that most people do—then you can adjust for the ONLY major bias in Jim Palmer’s won-lost record.
Both won-lost record and ERA have numerous other flaws. About the won-lost record:
1) It is biased by the team on which the pitcher pitched,
2) It is heavily subject to random fluctuations based on small data samples, and
3) The accounting system is often irrational, and will sometimes credit a pitcher with a "win" only because he has allowed the tying run to score.
But about the ERA:
1) It is subject to the bias of league norms,
2) It is subject to park effects,
3) It is subject to weather effects (league ERAs are always lower in September than in July, so a pitcher who pitches more in the late season has an ERA advantage),
4) It is subject to the randomness of balls in play being hit at fielders or not being hit at fielders (an effect which also reaches won-lost records),
5) It is subject to the vagaries of scoring decisions. In 1900 32% of all runs were scored as un-earned, whereas in modern baseball it is less than 10%.
6) The accounting system for ERA is also irrational on numerous points, and
7) The clustering of run events causes ERA also to be more subject to random fluctuations than many people assume that it is. The fact that a pitcher pitches 200 innings in a season causes people to assume that the random effects even out, when in fact the more relevant number is not the innings pitched but the runs scored, which is normally more like 80 than 200. The process of normalization from larger numbers is slowed down by the fact that runs are normally scored in clusters. In fact, it is quite common for a pitcher to have a 2.85 ERA one year, 4.10 the next, when in reality he has pitched as well one year as the other.
The critic of won-lost records could say, "OK, but we have better summary methods now. We have Fangraphs WAR and other summary methods which fill the place of won-lost records. We don’t need them any more."
But that is no more an argument to get rid of won-lost records than it is an argument to get rid of ERA, or WHIP, or any other less accurate stat. Since won-lost records are more accurate at predicting a pitcher’s true value than ERA, why not get rid of ERA?
We don’t get rid of ERA because we’re not in the business of getting rid of information. It’s not constructive. We need to know more information, not less. But if you can only know a pitcher’s ERA, or his won-lost record, you’re better off knowing the won-lost record.
6) Walks Per Nine Innings (61.2%). I will switch now from ranking the categories from the bottom to ranking them from the top. . .walks per inning are the 7th-worst category, which makes them the sixth-best. Walks per 9 innings outperformed strikeouts, but only by a thin margin.
5) Winning Percentage (61.5%). Winning Percentage is the 5th most accurate indicator of a pitcher’s true value, given the assumptions of this study. I do not know why the winning percentage performed better, as an indicator of true value, than the won-lost record, but I suspect that it had to do with ties.
There are a lot of ties in winning percentage, even given a constant number of innings; 10 and 10 is the same as 9-9, 11-11, 12-12 or 13-13; 8-10 is the same as 12-15, 12-9 the same as 16-12. The won-lost record breaks those ties, considering 12-12 to be better than 10-10. This is consistent with the assumption of WAR; if the replacement level is .300, then 12-12 is +4.8, whereas 10-10 is +4.0. But given that all of these pitchers pitched the same number of innings, the won-lost record probably breaks those ties in a random direction, being "right" as often as it is "wrong", and thus probably enters a .500 component into what is otherwise a .615 category, which tends to drag the measured success of the category downward.
4) Season Score (62.6%) The Season Score has three components—the innings/ERA component, the won/lost/saves component, and the strikeout/walk component. The process of figuring a season score starts by figuring a "CLI" or "Crude Leverage Index", based on the pitcher’s saves. The purpose of the CLI is to put relievers and starters on an equal footing. Since all of these pitchers are starters, the CLI has very limited relevance to this study, and I’ll just skip the explanation of that.
I multiply the pitchers outs recorded (which is three times his innings pitched) by .435, subtract runs allowed, and subtract earned runs allowed. This is, in essence, a way of saying that a pitcher with a 5.50 ERA has no value. .. parallel to the "replacement level" concept used in WAR. (.435 times 27) outs is 11.75 (runs+earned runs) per 27 outs. If a pitcher has a 5.50 ERA he might also allow another 0.75 un-earned runs per game, in general, so that’s 11.75 runs per 27 outs (5.50 + 6.25). A pitcher’s score is based on his runs allowed below that level, and both earned and un-earned runs count against him, but earned runs count twice as heavily against the pitcher as un-earned runs.
In addition to that I give the pitcher eight points for a win, one point for a save, take away five points for each loss. Then I take two times strikeouts minus three times walks, divide that total by 15, and add that. The highest Season Score in the study, by far, was 306, by Pedro Martinez in 2002, and the worst Season Score in the study was by John Harkins of the 1887 Dodgers. Harkins finished 10-14—a decent won-lost record--but struck out 36 batters, walked 77, and had a 6.02 ERA.
Season Scores were 63% accurate as a predictor of true value in this study. I was disappointed that the Season Score did not do better as a predictor of true value, and I had expected it to do better. I used to have a different way of figuring Season Scores, which relied more heavily on Wins and Losses. I changed the method in 2009 to de-emphasize Won-Lost records. I didn’t check this out, because I’m not going backward, but it seems obvious that the older version of Season Scores, with more reliance on won-lost records, would have performed better in this study than the new version does. John Harkins, 1887, is a good example of why this is true; he was 11-14 with a 6.02 ERA—but Fangraphs WAR puts his value (1.7) at a point entirely consistent with his won-lost record, but not at all consistent with his ERA. On the other end of that would be Barney Pelty, 1909; Pelty was 11-11 with a 2.39 ERA. Fangraphs gives him 1.8 WAR—like Harkins, consistent with his won-lost record, not at all consistent with his ERA.
But this, of course, makes again the point I made before: that the won-lost category holds value because won-lost records automatically adjust to rising and scoring levels of runs scored, whereas all of the other categories studied here go up and down with changes in time and place.
3) Relative ERA. Relative ERA was 63.6% accurate as a predictor of value, making it the third-best predictor of value among the twelve stat categories studied. This Relative ERA was NOT park-adjusted; an ERA of 3.00 in a league with an ERA of 4.00 was 0.75, regardless of whether the park factor was 109 or 90. Presumably, Relative ERA would have done even better, had I been able to study the park-adjusted relative ERAs.
1 and 2) Strikeouts Minus Walks (65.2%) and Strikeout to Walk Ratio (65.4%) essentially tied as the best predictors of True Value, among the twelve categories studied.
Of course, it is possible that Strikeouts and Walks perform best as a predictor of Wins Above Replacement because Fangraphs is relying too heavily on strikeouts and walks in their calculation of the pitcher’s value. To this point I have assumed that WAR is a perfect and unassailable measure of a pitcher’s true value. This, of course, is not necessarily true.
Going into the study, I assumed that it would not make very much difference what measure of true value I used, that one measure of overall value would be as closely tied to the elements of a pitcher’s record as another, and I had actually written a couple of sentences explaining to you why it was unlikely to matter what we used as the bottom line. I see now, at the conclusion of the study, that this assumption could be entirely wrong, and that very probably the version of WAR that we use does matter.
I would have liked to come out of this study entirely convinced that the Fangraphs WAR system does work almost perfectly, and convinced that the confidence placed in F-WAR by this study was not an issue. I can’t say that I’m there; I’m not criticizing F-WAR, I am not saying that it is wrong in any particular case or that I have a better method, but I can’t tell you that I have unbounded confidence in it, either. I will have to tell you, honestly, that I was surprised by how many cases there are in which F-WAR gives seemingly irrational answers in comparing two pitchers.
For instance?
Spud Chandler in 1942 was 16-5 with a 2.37 ERA. Harry Howell in 1902 was 9-15 with a 4.12 ERA. We’d tend to assume that Chandler might possibly be the better pitcher, no?
Fangraphs likes Howell. The league ERAs are about the same—3.66 for Chandler, 3.57 for Howell. Chandler’s ERA was 1.29 under the league average; Howell was .55 over. Chandler’s strikeout to walk ratio was 74 to 74; Howell’s was 33 to 48. Why exactly should we consider Howell to be a better pitcher? Not saying it is wrong, you understand; I just don’t quite get it.
Ah, but Harry Howell in 1902, in addition to his work on the mound, also played 26 games at second base, 15 at third base, 18 in the outfield, 11 at shortstop and one at first base. Maybe his value comes from his work in the field?
No, not really. . .at least if I understand the Fangraphs WAR charts. Howell is +2.7 as a pitcher, +0.2 for his work as a fielder and hitter. Chandler, on the other hand, hit a nice .211 with 8 walks, so he is +0.4 as a hitter/fielder. The non-pitching stuff is actually helping Chandler get closer to Howell, rather than pushing Howell ahead of Chandler. If I understand the charts correctly. ..and if I don’t understand the charts correctly, then they’re not of much use to me.
Maybe there is a good reason why Howell is better than Chandler, but I don’t know what it is, and I’m a little skeptical.
Gil Heredia in 1999 was 13-8 but with a 4.82 ERA; Chuck Finley in 1989 was 16-9 with a 2.57 ERA. American League ERA in 1999 was 4.87; in 1989 it was 3.89. The Park Effect for Oakland in 1999 (Heredia) was .92; for California in 1999 it was .93. Finley was 26 runs better than league, park adjusted; Heredia was three runs worse than league. DH rule; hitting is basically no factor here. Finley gave up 13 homers; Heredia gave up 22.
Fangraphs WAR says that Heredia was better than Finley (+4.0 to +3.6). Well, OK, but I’m a little bit skeptical. Garret Stephenson in 2000 was 16-9 with a 4.49 ERA against a league norm of 4.64, strikeout to walk ratio of 123 to 63. Joe Coleman in 1975 was 10-18 with a 5.55 ERA against 3.79 league ERA, strikeout to walk ratio of 125 to 85. Fangraphs says that Coleman (+1.3) was better than Stephenson (+1.2). Say what?
Johnny Broaca in 1935 was 15-7, 3.58 ERA, 78 strikeouts, 79 walks. Danny MacFayden in 1932 was 8-15, 4.39 ERA, 62 strikeouts and 70 walks. The league ERAs are basically the same, and neither pitcher was a good hitter. Even the park is the same for half of the year; Broaca was with the Yankees, and MacFayden was with the Yankees the second half of the year. Fangraphs says that MacFayden was better than Broaca. Are you sure?
Hank Wyse, 1946 Cubs, and Sad Sam Jones, 1932 White Sox. Wyse was 14-12; Jones was 10-15. Wyse’ ERA was 2.69; Jones was 4.48. Wyse’ strikeout to walk ratio was 52-52; Jones was 64 to 75. Wyse hit .243; Jones hit .193. You might assume that Wyse would be a better pitcher?
Nope.
Mule Watson in 1922 was 8-14, 4.70 ERA; Sheldon Jones in 1948 was 16-8, 3.36 ERA. The league ERAs are almost the same, 15 points difference, strikeout to walk ratios are about the same. Fangraphs says that Watson was better.
Mace Brown in 1939 was 9-13, 3.38 ERA, 71-52 K-W, 8 home runs allowed. Oral Hildebrand in 1937 was 8-17, 5.15 ERA, 75-87 K-W, 18 home runs allowed. Fangraphs says that Hildebrand was better than Brown.
Art Ditmar in 1960 was 15-9, 3.06 ERA; Rodrigo Lopez in 2010 was 7-16 with a 5.00 ERA. The league ERAs are about the same, 15 points difference again. Each pitcher walked 56 batters. Ditmar struck out 65 batters but gave up 25 homers; Lopez struck out 116 but gave up 37 homers. Again, 15-9, 3.06 ERA vs. 7-16, 5.00 ERA. ..I think I would go with the guy who is 15-9 with a good ERA. Fangraphs goes the other way.
Here’s a few more comparisons in which Fangraphs seems to offer a counter-intuitive answer to the question, "Which of these two pitchers had a better season?" (These WAR numbers include the batting adjustment):
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Hank
|
Johnson
|
1928
|
31
|
14
|
9
|
.609
|
199.0
|
110
|
104
|
16
|
4.30
|
4.04
|
0.6
|
Johnny
|
Lindell
|
1953
|
32
|
6
|
17
|
.261
|
199.0
|
118
|
139
|
17
|
4.66
|
4.28
|
2.8
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Doug
|
Rau
|
1978
|
30
|
15
|
9
|
.625
|
199.0
|
95
|
68
|
17
|
3.26
|
3.58
|
1.5
|
Gene
|
Conley
|
1961
|
33
|
11
|
14
|
.440
|
200.0
|
113
|
65
|
33
|
4.91
|
4.02
|
2.1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Roger
|
Craig
|
1956
|
35
|
12
|
11
|
.522
|
199.0
|
109
|
87
|
25
|
3.71
|
3.77
|
1.0
|
Joe
|
Genewich
|
1924
|
34
|
10
|
19
|
.345
|
200.0
|
43
|
65
|
4
|
5.22
|
3.86
|
1.2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Gio
|
Gonzalez
|
2010
|
33
|
15
|
9
|
.625
|
200.2
|
171
|
92
|
15
|
3.23
|
4.14
|
2.9
|
Esteban
|
Loaiza
|
2000
|
34
|
10
|
13
|
.435
|
199.1
|
137
|
57
|
29
|
4.56
|
4.92
|
3.3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Bill
|
Sherdel
|
1925
|
32
|
15
|
6
|
.714
|
200.0
|
53
|
42
|
8
|
3.10
|
4.27
|
3.2
|
Frank
|
Foreman
|
1901
|
25
|
12
|
7
|
.632
|
199.1
|
42
|
60
|
3
|
3.88
|
3.66
|
3.5
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Fritz
|
Ostermueller
|
1934
|
33
|
10
|
13
|
.435
|
199.0
|
75
|
99
|
7
|
3.48
|
4.50
|
2.6
|
Doc
|
Crandall
|
1911
|
41
|
15
|
5
|
.750
|
199.0
|
94
|
51
|
10
|
2.62
|
3.39
|
1.1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Kevin
|
Gross
|
1987
|
34
|
9
|
16
|
.360
|
200.2
|
110
|
87
|
26
|
4.35
|
4.09
|
1.3
|
Doyle
|
Alexander
|
1976
|
30
|
13
|
9
|
.591
|
201.0
|
58
|
63
|
12
|
3.36
|
3.52
|
1.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Kris
|
Benson
|
2004
|
31
|
12
|
12
|
.500
|
200.1
|
134
|
61
|
15
|
4.31
|
4.31
|
3.6
|
Shawn
|
Estes
|
1997
|
32
|
19
|
5
|
.792
|
201.0
|
181
|
100
|
12
|
3.18
|
4.21
|
3.4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Tommy
|
Bridges
|
1932
|
34
|
14
|
12
|
.538
|
201.0
|
108
|
119
|
14
|
3.36
|
4.48
|
2.1
|
Johnny
|
Lindell
|
1953
|
32
|
6
|
17
|
.261
|
199.0
|
118
|
139
|
17
|
4.66
|
4.28
|
2.8
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Hank
|
Johnson
|
1928
|
31
|
14
|
9
|
.609
|
199.0
|
110
|
104
|
16
|
4.30
|
4.04
|
0.6
|
Ray
|
Benge
|
1929
|
38
|
11
|
15
|
.423
|
199.0
|
78
|
77
|
24
|
6.29
|
4.71
|
1.6
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Marcelino
|
Lopez
|
1966
|
37
|
7
|
14
|
.333
|
199.0
|
132
|
68
|
20
|
3.93
|
3.44
|
1.5
|
Art
|
Ditmar
|
1960
|
34
|
15
|
9
|
.625
|
200.0
|
65
|
56
|
25
|
3.06
|
3.87
|
0.7
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Pat
|
Dobson
|
1973
|
34
|
12
|
15
|
.444
|
200.0
|
93
|
53
|
23
|
4.41
|
3.67
|
1.7
|
Doc
|
Crandall
|
1911
|
41
|
15
|
5
|
.750
|
199.0
|
94
|
51
|
10
|
2.62
|
3.39
|
1.1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Ray
|
Collins
|
1912
|
27
|
13
|
8
|
.619
|
199.1
|
82
|
42
|
4
|
2.53
|
3.34
|
3.2
|
Jim
|
Kaat
|
1961
|
36
|
9
|
17
|
.346
|
200.2
|
122
|
82
|
12
|
3.90
|
4.02
|
4.1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First
|
Last
|
Year
|
G
|
W
|
L
|
WPct
|
IP
|
SO
|
BB
|
HR
|
ERA
|
Lg ERA
|
Total
|
Al
|
Demaree
|
1913
|
31
|
13
|
4
|
.765
|
200.0
|
76
|
38
|
4
|
2.21
|
3.20
|
2.6
|
Andy
|
Ashby
|
1997
|
30
|
9
|
11
|
.450
|
200.2
|
144
|
49
|
17
|
4.13
|
4.21
|
3.0
|
Again, and very sincerely, I am not saying that Fangraphs is wrong in any of these comparisons; it is quite possible that, if I knew more about the calculations, I would agree with them. I know very well that many times what seems like an obvious conclusion from the statistics will not stand up to closer scrutiny, and the researcher will wind up arguing that what appears to be true is not actually true. I’ve seen that myself a thousand times.
But I also know very well, from years of doing it, that the process of weighing and measuring every stat so as to determine overall value is a treacherous and difficult task, and that there are thousands of ways you can get the wrong answer. I’ve done that myself a great many times, as well. I am less convinced than I would like to be that these evaluations are accurate.