Trying to explain what it is that we really do.

(I have been asked to speak to a Risk Management Seminar being organized by someone who has some connection with the Red Sox.  I thought about how I would explain Sabermetrics to a group of intelligent people who didn’t really get what we doing, and this is what I came up with.   The following is the text for a talk that will be delivered in mid-June, and, since I had to write it, I decided to share it with you.)

            Good afternoon, and thank you for inviting me to speak to you for a few minutes here.    I have been asked to say a few words about sabermetrics, about what I do for the Red Sox, and about how this might relate to risk management, if it does.

            Sabermetrics is descended from traditional sportswriting.   Sportswriting consists of two types of things—reporting, and analysis.  Sabermetrics came from that part of sportswriting which consists of analysis, argument, evaluation, opinion and bullshit.   I can tell you very precisely when and how we parted ways with traditional analysis.

            Sportswriters discuss a range of questions which are much the same from generation to generation.   Who is the Most Valuable Player?   Who should go into the Hall of Fame?   Who will win the pennant?   What factors are important in winning the pennant?   If Boston won the pennant, why did they win it?   If Kansas City finished last, why did they finish last?   How has baseball changed over the last few years?   Who is the best third baseman in baseball today?   Who is better, Mike Lowell or Eric Chavez?

            The questions that we deal with in our work are the same as the questions that are discussed by sports columnists and by radio talk show hosts every day.   To the best of my knowledge, there is no difference whatsoever in the underlying issues that we discuss.   The difference between us is very simple.   Sportswriters always or almost always begin their analysis with a position on the issue.   We always begin our analysis with the question itself. 

            If you find a sportswriter debating who should be the National League’s Most Valuable Player this season, his article will probably begin by asserting a position on the issue, and then will argue for that position.   If you find 100 articles by sportswriters debating issues of this type, in all likelihood all 100 articles will do this.  

            What we do is simply to begin by asking “Who is the National League’s Most Valuable Player this season?” rather than to begin by stating that “Albert Pujols is the National League’s Most Valuable Player this season, and let me tell you why.”  That’s all.   That is the entire difference between sabermetrics and traditional sportswriting.  It isn’t the use of statistics.   It isn’t the use of formulas.   It is merely the habit of beginning with a question, rather than beginning with an answer.

            From this very small difference, profound changes arise.   A person who begins by asserting a position on an issue naturally focuses on what he KNOWS.   “What facts do I know, he asks himself, which will help me to explain why Mike Lowell is better than Eric Chavez?”

            The person who begins with the question itself. ...who is better, Mike Lowell or Eric Chavez. . ..the person who begins with the question itself naturally focuses not on what he DOES know, but on what he does NOT know.   I misstated that just a little bit.   The person who begins with the question itself naturally focuses first on what he NEEDS to know.    As soon as he begins to think through what he needs to know, however, it will become apparent that he does not know many of the things that he needs to know to really answer the question.

            Every large general question in baseball can be broken down into smaller and more specific questions, which can be broken down into yet smaller and yet more specific questions, which can be broken down into yet smaller and yet more specific questions.   Eventually you reach the level at which the questions have small and definite answers, even though you may not know what those answers are.  The question of who is better, Mike Lowell or Eric Chavez, for example, leads to the question “What are the elements of a third baseman’s job?”   Well, there is hitting, fielding, baserunning, and off-the-field contributions if you want to count those.  

            The question of who is a better fielder breaks down into 20 other questions.   Who is quicker?  Who is more reliable?  Who makes more mistakes?  Who has a better arm?  Who is better at anticipating the play, and positioning himself correctly?   Who is better at going to his left, or to his right?  Who is better at applying the tag if he needs to make a tag on a runner?

            At that level, the questions that we ask ourselves are actually very much like the questions that scouts ask.   Scouts try to answer them by their expert judgment.   We are not experts, and so we try to find ways to answer them that do not rely on our judgment.  In that way, sabermetrics forms a kind of transition between journalism, from which it descended, and scouting.   In any case, the question of who is better at applying the tag if he needs to make a tag play leads to the question “How many times a year does a third baseman have to make a tag play?”    That is a question that has an actual answer.   I don’t know what the answer is, but there is an answer to that question.   In a few years, somebody will figure out a way to get that answer. 

            The question of who is better, Mike Lowell or Eric Chavez, contains within it probably several hundred smaller questions.   The person who begins by asking that question naturally realizes that he does not know the answers to most of those questions.   He then proceeds inevitably to ask, “How can I find the best answer to that question?” 

            When a person begins with the question itself he inevitably winds up confronting his own ignorance, and trying to find ways to fill in the gaps in his knowledge.   The person who begins with a position on the issue never sees his own ignorance, and, in fact, deliberately avoids seeing his own ignorance.  The person who begins with a position on the issue and argues for that position naturally tries to hide his ignorance of the other internal issues, since the things that he doesn’t know are a weakness in his argument.   The person who begins with the question itself, on the other hand, inevitably winds up reveling in his own ignorance, celebrating his ignorance, and sharing it freely with the world at large.  

            But the person who begins with a position on the issue, by this process, becomes a borrower from the Bank of Knowledge.   He borrows from the things that he knows, and uses them to construct an argument.

            The person who begins with the issue itself, on the other hand, eventually becomes a contributor to the Bank of Knowledge.   Forced to confront his own ignorance, he is forced to find ways to figure out the information that he is missing—ways to count things that haven’t been counted, or ways to estimate the parameters of things that are unknown.    Through this process, he winds up knowing things that were not known before.  

            This is essentially what we do:  We try to construct knowledge to fill in some of the spaces in our massive ignorance.   We are not people who know things.   We are people who are honest enough to admit that we don’t understand things, and frankly, we don’t believe that you understand them, either.  

            Because we have been involved in the effort to figure out the things that we don’t understand for a long time now, we have developed an inventory of a few hundred standard methods that can be used to analyze a new question.   What I do for the Red Sox is to try to draw from that inventory tools that can be used to study whatever it is that the Red Sox need to study.   Suppose that we are considering trading for a new pitcher, for example.   Our General Manager might ask me “What do you think is the chance that this pitcher will retain value over the life of his contract?”. .. the “risk management” that I am supposedly discussing here.   There are several ways to approach that problem.   One is to look for similar pitchers.   If you can identify 25 other very similar pitchers over the last twenty years, you can study their health records as they moved forward.   You can do a characteristics analysis.   There are observable characteristics of durable pitchers, such as a high strikeout rate and a record of moderate but consistent workloads, and you can compare this pitcher to the characteristics of healthy pitchers.

            When our GM or one of his associates asks me a question, I almost never give him an immediate answer—nor does he expect me to.   If he wanted an immediate answer, he would ask an expert, or he would ask a scout.   What I always say, when asked anything, is “I’ll study the issue.”

            I shouldn’t say “never”.   Occasionally I’ll slip up and give an immediate answer.   A year ago, for example, Theo asked me “How confident are you that Dustin Pedroia will hit what your projections show that he will hit?”   Without thinking about it, I said “Absolutely 100%.”  I knew three things that caused that to slip out.   First, I knew that our hitting projections for minor leaguers have a very high degree of reliability—much higher than most people assume that they do.   Second, I knew that our projections for Dustin Pedroia were actually very conservative, that he was actually a better hitter than we were saying that he was.  Third, I knew that, even though Pedroia had hit .191 as a late-season callup in 2006, he had made contact with more than 90% of his swings.  If you swing as hard as he does and you make contact with every swing, how can you possibly fail to hit?

            That’s risk management, of course. . . .knowing how reliable the projections are.   But that’s also very much the exception to the rule, actually—the exception to two rules.   The first rule is:  We are not experts.   We are not people who “know” things about baseball players.   We are people who know how to study things about baseball players.   When Theo asks me a question about a player or about the game in general, he is not asking me for what I know.   He is asking me to study the issue. 

            And the other rule is, we are never certain.  I said that I was 100% certain that Pedroia would hit in the majors, but in reality, we are never 100% certain of anything.   The talk show hosts are 100% certain.   The sports columnists are 100% certain.   We are just doing the best we can.   Our methods are always flawed, and our answers are usually tenative and muddled.  

            But the difference between knowledge and BS is that knowledge moves forward, whereas BS moves in circles.   When I develop a new method to analyze a baseball question, someone will point out the flaws in that method, and then someone else will suggest an entirely different method to approach the problem, and then somebody else will point out the flaws in that entirely different method, and then somebody else will figure out a way to combine the best features of both methods into one method that is better than either one.  We wind up with methods that get better over time. 

            And also, we wind up with lots of numbers, which is why people perceive us as statisticians.  Going back to the number of tag plays made at third base.   It’s a significant piece of information that doesn’t exist.    It’s very difficult to take a throw from right field—a long throw—and apply the tag to a fast-moving target which is taking evasive action.   It’s not an easy thing to do, and if the third baseman makes an extra play in that situation, that has the same value as hitting a triple.   Eventually, somebody will find some way to measure this, and the product of this will be stated as a number, which, since we are dealing with baseball, will be called a statistic.  

            We are no more statisticians than we are historians, or scouts, or accountants, or computer programmers.   I suspect that everything we do is much the same as what many of you do.  We look to the past, and we try to organize the things we have seen so that they make some sense.   We ask ourselves “how many of those were there?” and “how many of those others were there?” and “How many of them ended well?” and “How many of them ended badly?”, just as I would imagine most of you do.  

            Often people wonder if we will run out of things to study, and what I always say is that there will never be a shortage of ignorance.   I have realized recently that some people take this wrong, and that when I say that there will never be a shortage of ignorance people think that I am referring to some other group of people, some “ignorant” group of people, when in reality what I am talking about is myself.   I will never understand baseball; I will never understand 1% of what I need to understand.   My view of our work is that we are attacking a mountain range of ignorance with a spoon and a used toothbrush.   The things that we do not know are inexhaustible.  

            But our time here is limited, and I appreciate your patience.