In a comment to a recent item on my own blog, Jabber asked whether I thought an analyst who beat the market 4 or 5 times running was a good analyst. As any good economist would answer, “It all depends.”
Let’s use sports as an example first. I haven’t looked for awhile, but many sports publications used to run ads “Call us toll free and get one free pick for this week’s line-up of NFL games.”
My guess is that they gave out one side of the bet to half the callers and the other side of the bet to the other half of the callers. Then when people called for a free pick the next week, even (especially!) if they had called before, they were given a free pick again. And again, half the callers were given one side of the bet, while the other half were given the other side of the bet.
After 4 replications of this purely random strategy, roughly 1 out of every 16 of the original callers would have received a correct pick every single week (and presumably the others would have tried a different service). There is a very good chance that these people would be glad to order the full service of predictions from this service, believing that the service operators were brilliant when, in fact, they were just lucky. And, I expect, once they ordered the service they probably found that they were doing little better than 50-50 in their bets. (note: I am aware of some complex models that claim to have done slightly better than the spread over time, so don’t get too worked up over this point).
In other words, just because someone predicts something correctly four times in a row, don’t necessarily conclude they’re good. They might just be lucky.
Now apply this same analysis to financial advisors. What if you learned that 1 out of every 32 financial analysts beat the market for the past five years running? That is exactly what you would expect if they were all just tossing coins. In other words, with these results it would be difficult to reject the hypothesis that the financial advisors were no better than coin tosses would be.
