Bill James on What Makes Good Stats

More wisdom from the bard

Bill James on What Makes Good Stats

Here’s the latest snippet from the old Bill James Baseball Abstract - this time from the 1978 edition:

Now, I really really really wish that I had a copy of that 4 1/2 page essay. This is why I wish somebody would just scan those old handmade Abstracts and stick them online in PDF format for us to study. They’re fading as it is: let’s digitize them while we still can and put them up where the community can actually use them. But I digress.

James is spot on here — and he identifies precisely what is wrong with a lot of modern baseball statistics and sabermetric thought.

First of all — if you’re going to do something, tell us why you’re doing it! Now, I should note that James himself is guilty of breaking this rule all the time, which is something that will become apparent as we continue our deep dive into his writings.

The second point stands as well, though I will note that we have discovered over the years that our assumed understanding of how baseball works was not always all that accurate. For example, it turns out that power hitting is a lot more effective in run production than hitting for singles, even if your singles hitter can also steal 100 bases or more per season.

The third point is also important, though I worry that James got off on the wrong track here. Baseball is not a game of wins as much as it is a game of runs. The truth, though, is that it’s more a game of outs than anything else. Outs come at a premium, and a game cannot be won until the last man is out (well, barring being shortened because of rain or darkness, that is). When we look at whether an action is efficient or not, we need to see what we get in return for an out, or for a possible out. That leads us naturally to wins and losses.

However, I agree in principle with James’ third point.

Then comes the 4th point, which is the big kicker.

I don’t care if you like messing around with potential statistics or not. Knock yourself out. Heck, I read blogs like the MLB Data Warehouse all the time:

However, if you’re going to use one of these stats in a discussion, you need to understand what it is and how it works. The biggest test of whether you understand something is whether you can explain it.

And I’m talking about explaining it in basic terms, too. I’m not talking about a long mathematical explanation. I’m talking about explaining the logic behind the data and the approach that you’ve taken.

I think that WAR in general fits that requirement, though specific aspects of WAR don’t. For example, the way that WAR deals with fielding for the 1999 season and before is mysterious, strange, and does not seem to match up with any ordinary fielding statistics that I’m aware of. Similarly, the considerable modifications that WAR creates for players based on the positions that they play is something that is difficult to understand, is not intuitive, and is really not well explained anywhere.

If you can’t figure out the why behind a metric, you really shouldn’t use it. It’s extremely likely that there’s a bias hidden behind the scenes, or that the person who stuck it together made it obtuse to hide their own bias, or that a bunch of stat geeks got together with wild ideas and refused to allow people to challenge their assumptions.

To put it simply: if a 12 year old can’t understand what you did and why, you need to think it through again.

I’d also argue that the same applies to baseball sim mechanics — but we’ll talk more about that in the future.