Snippets of a conversation among two economists this morning:
A: Is a sequence of eight heads conditioned on three tails a random walk?
B: Almost certainly not. Why?
A: The Red Sox.
B: Yeah, wow. That was some performance.
A: Did they have a hot hand?
B: I think so. And apparently the Cardinals did too. They hardly showed up last night.
The conventional wisdom in the academic literature is that psychologists and behavioral economists have debunked the "myth of the hot hand," i.e. that players or teams are subject to streaky performance. Performance is just a random walk, they say. I don't buy it. The Red Sox were on a roll.
Note: The odds against LLL-WWWW-WWWW in a sequence of coin flips are 2047 to 1. Yes, Sox fans, deliverance from the curse of 1918 was a near miraculous event.
Update: Eric McErlain points out that the Red Sox' post-season was even streakier than the sequence listed above.
Update II: David Pinto writes in the comments section: "Hold it. Any 11 game sequence of 3 losses and 8 wins would have the same probability. So if the sequence was WLWLWWWWLWW, the probability would still be 2047 to 1, and no one would have thought it was very miraculous."
I anticipated this comment when I cut a line mentioning "1 of 2048 sequences, all equally likely" from the version I posted, since I didn't want to get into the following digression. Now is the right time I suppose. Thanks for prompting it.
The short answer is that David's sequence is much less streaky than the Sox's sequence. Of the 2048 combinations of an 11 game sequence, some are streakier than LLL-WWWW-WWWW, but not many. And fewer still would have resulted in a Red Sox victory in the World Series.
There is a statistical test for whether a sequence is streaky - a "runs test." If I get a chance I'll execute it and post it in Update III. If someone has it handy it the result may make it here before the weekend. I believe that the Sox' streak is highly unlikely under a coin flip scenario, and is thus evidence against the random walk model.
Update III: I've had a chance to check my trusty stats book from days of yore (Walpole and Myers, 1985). The probability of having only 2 runs in a sequence of eleven trials, i.e. one run of Ls and one run of Ws, where either outcome is equally likely, is .012. That "p-value" is sufficiently low to reject the null hypothesis at the 95% confidence level, the standard used by David in his critique of my original post. Skeptics may wish to employ a higher standard of proof, but it's good enough for me. Indeed, Walpole and Myers employ a similar example from a twelve trial sequence, and state that "a sample containing only two runs is most unlikely to occur from a random selection process."
I continue to believe the Red Sox post-season performance was streaky.