Can you beat the market? If so, how? If not, why not? I wrote this primarily for a group blog about rationality, so it’s a bit wonkier than usual. Life is short and posts are long; unless you’re an investing nerd you might give this one a miss.

There are whispers that the Efficient-Market Hypothesis (EMH) is dead. Smart people say it may have been the real victim of the coronavirus. These people, or their friends, were able to get ahead of the recent crash. They sold stocks before the market reacted, or shorted them, or bought ‘put’ options, and made handsome profits.

The EMH states that “asset prices reflect all available information”. The direct implication is that if you don’t have any non-available information, you shouldn’t expect to be able to beat the market, except by chance.

But the people who beat the market didn’t have any special information! They were reading the same news and reports as everyone else. They made a profit by acting on public information that was right there for anyone to see. And so, the EMH is dead, or dying, or at the very least, has a very nasty cough.

I say this is wishful thinking, and rumours of the death of efficient markets have been greatly exaggerated.

First: how the heck did the market get the coronavirus so wrong?

The Great Coronavirus Trade

Lots of people initially underreacted to COVID-19. We are only human. But the stockmarket is not only human—it’s meant to be better than this.

Here’s Scott Alexander, in A Failure, But Not of Prediction:

The stock market is a giant coordinated attempt to predict the economy, and it reached an all-time high on February 12, suggesting that analysts expected the economy to do great over the following few months. On February 20th it fell in a way that suggested a mild inconvenience to the economy, but it didn’t really start plummeting until mid-March – the same time the media finally got a clue. These aren’t empty suits on cable TV with no skin in the game. These are the best predictive institutions we have, and they got it wrong.

But… this isn’t how things went down. The market started reacting in the last week of February, with news headlines directly linking the decline to the ‘coronavirus’. By the time we get to mid-March, we’re not far off the bottom.

You can confirm this for yourself in three seconds. People pointed out the mistake in the comments, but to no avail.

(It’s important to note here that Scott is a brilliant guy, and my favourite blogger of all time. I am not claiming he was deliberately misleading anyone, or trying to single him out in particular.)

There is a general sense in which COVID-19 seems to be a magnet for revisionist history and wishful thinking. In other comments under the same post, the notion that our ‘tribe’ got it right in advance comes under scrutiny—in fact, it looks like many of the names thrown around seem to have jumped on the bandwagon after it was obvious, and certainly long after the market was moving.

The facts are that the market reacted faster than almost all of us. But not before a few prescient people placed their bets!

So now the question becomes: why didn’t the market react earlier than February 20, like those smart people did?

The null hypothesis is that the market reacted exactly appropriately on the basis of the information available. After all, there were other potential pandemics in the recent past that were successfully contained or eradicated.

On February 20, there were only 4 known cases in Italy. We were a long ways from the bloodbath that was coming. Maybe it was correct to move cautiously until further information came in. And as that information came in—confirming community spread, and triggering a tipping point over the next few days—the market responded in exactly the manner we would expect.

If the null hypothesis is true, then those early trades were not quite as prescient as they look. We might be making the mistake of ‘resulting’, and confusing the reality we ended up in with all the others which were possible at the time, in which those traders lost their shirts. We will never know for sure, because these events are one-offs. Hopefully we never get a chance to run this experiment again!

That didn’t stop one trader who bought out-of-the-money puts in mid-February from claiming that “at least for me this puts a final nail in the coffin of EMH.”

This is a polite way of saying that you might be a brilliant investing wizard with the power to beat the market. Honestly, after making such a beautiful trade—and my gosh it really was beautiful—whom amongst us could resist that temptation? Certainly not me. And anyway, it might even be true!

In making sense of this claim, the first thing we have to establish: what does it even mean to be able to beat the market?

Can Uncle George Beat the Market?

Uncle George really likes his new iPhone. Man, these things are nifty! The dancing poop emoji is hilarious. On the strength of this insight, George dials his broker and loads up on AAPL stock.

Over the next year, AAPL stock goes up 15 per cent, while the broader S&P 500 only goes up 10 per cent. George becomes insufferable at family dinners as he holds forth on his stock-picking powers. Guess the market isn’t so ‘efficient’ after all, huh. Suck it, Eugene Fama!

So: did Uncle George beat the market?

In the narrowest possible sense… yes.

In the sense in which we aim to string words together so that they mean things: no, of course not. By this definition, every single trade leads to one of the two parties ‘beating the market’. Millions of people beat the market while I wrote this sentence. I can flip a coin between Pepsi and Coke right now, and have a 50 per cent chance of becoming a market-beating genius.

The Uncle George example makes it glaringly obvious that a successful trade does not somehow ‘break’ efficient markets. And yet, this is the same naive criticism constantly leveled against the EMH: if the market moves in literally any direction, that must mean it was wrong before! My cousin who sold/bought before it went up/down beat the market!

Same goes for the Great Coronavirus Trade. The fact that some people got out of the market early is no surprise whatsoever. Investors constantly think the market is going to crash, for any number of plausible reasons. This is the default state of affairs: we have successfully predicted 73 of the last five market crashes, etc.

These predictions are almost always wrong, and almost all the people who make them would have been much better off taking the boring ‘buy and hold forever’ strategy:

But even a stopped clock is right twice a day. And of course, we’re much more likely to hear about the occasional brilliant successes than the near-constant dull failures.

So the dumbest critique of the EMH boils down to ‘it is possible to make a good trade’. This is just a property of trading. It tells us exactly nothing about the market’s efficiency.

But some people really do beat the market—and not in the trivial sense. I’ll suggest a definition later on which strips out the effect of randomness.

Before we get there—doesn’t the concession that people can non-trivially beat the market already drive a stake through the EMH’s heart?

This is the second great misunderstanding: there is no conflict between the EMH and beating the market. That’s how the market gets efficient! You find an information asymmetry that isn’t priced in yet, and in exploiting it, you move the market a little further towards efficiency.

Let’s call this information asymmetry an ‘edge’.

If the EMH is true—or even just true-ish—that doesn’t mean the market can’t be beat. It means:

You shouldn’t expect to beat the market without a unique edge, except by chance

Now, this usually gets simplified down to ‘you can’t beat the market’. And most of the time, this simplification is good enough: you might get lucky and win in the Uncle George sense, but over an investing lifetime, you’ll almost certainly revert to the mean (which isn’t matching the market return—it’s underperforming it).

But if you can find some kind of edge, you really can win! So, what might a genuine edge look like?

Anomalies Exist!

The Uncle Georges of the world don’t have an edge. All of their thoughts have already been thunk by someone else (probably by millions of someone elses). Instead, their fortunes are entirely at the mercy of the myriad other forces that drive stock prices: consumer demand, workplace harassment scandals, money printers going brrr, the exact virulence of a novel coronavirus, the price of cheese in Spain last Friday afternoon, etc.

All of this stuff—billions of inputs processed by the greatest collective intelligence ever built—is a black box unto us mere mortals. It’s impossible to assign perfect causal explanations to stock prices, which means we can pick whichever story suits us best.¹

All Uncle George can see is that he placed his bet, and AAPL went up. It was the poop emoji for sure!

And so, Uncle George spends his days dishing out hot stock tips on online forums, oblivious to the fact that his success was meaningless.

A century ago, investors started noticing they could consistently pick up bargains by running very simple formulas over stock prices. The most famous is the ‘value investing’ approach developed by Ben Graham, and used by Warren Buffett and Charlie Munger. There was a genuine, big old inefficiency in the markets, and these guys had a great time exploiting it.

I think this might be the image most people have in their head when they think of ‘beating the market’—diligently studying The Intelligent Investor and learning about PE ratios or whatever.

But this is like trying to use a stone-age axe against a fighter jet. The Ben Graham information asymmetry has long since disappeared because…markets are efficient(ish)! Once the formula was widely known, it stopped working. Investors developed more sophisticated versions, more formulas, more pricing models. Once those got out, they stopped working too. Now there’s a great debate as to whether even the most complicated descendants of value might be totally dead. In which case, the anomaly has officially gone for good.

Either way, this is not how Buffett gets his edge, and it hasn’t been for decades. Here’s his partner Charlie Munger:

The trouble with what I call the classic Ben Graham concept is that gradually the world wised up and those real obvious bargains disappeared. You could run your Geiger counter over the rubble and it wouldn’t click.

Buffett’s most brilliant achievement is weaving this folksy narrative that he is a cute old grandpa who beats the market by backing the best companies. Let me tell you how market-beating investors really make their money.

Modern Edges are Completely Bonkers

1. The Warren Buffett Halo Effect

In recent decades, Buffett has made a killing through juicy private deals which are completely out of reach of the average investor. Like, six billion dollar deals with three billion in preference rights and a guaranteed dividend. Like, lobbying the government to bail out the banks, then carving off a huge piece of the action. Like, being able to play around with Berkshire Hathaway’s $115 billion insurance float. Much of his fortune is built on taxpayer largesse.

Warren Buffett’s brand is so powerful that at this point, his success is a self-fulfilling prophecy: when Berkshire invests in a stock, everyone else piles in after him and drive the price up. Buffett even lends out his ‘halo’ to companies that need it—most famously during the GFC—so long as they give him a generous discount to the market price, of course.²

And yet, and yet… Berkshire Hathaway has underperformed for the last decade. Buffett would have been better off if he’d taken his own advice and put it all in index funds.

2. Hedge funds with armies of drones

There you are, sitting in your home office going through Walmart’s quarterly report and calculating PE ratios or whatever. Meanwhile, the professionals are using an army of drones to monitor the movement of shopping carts in Walmart parking lots in real time.

See also: sending foot soldiers out to every branch of a bakery chain at the close of business each day, because the numbered dockets start out at zero, and thus contain live sales data unavailable to the market.

And so, when renowned hedge fund manager Michael Steinhardt was asked the most important thing average investors could learn from him, here’s what he suggested:

“I’m their competition.”

And yet, and yet…almost all hedge funds underperform. Not all of them are trying to beat the market, but it gives a sense of the difficulty.

3. High-frequency traders move mountains

If multiple people have access to the same information, speed in bringing it to market also matters. So, we have high-frequency traders.

One firm spent $300 million laying a direct cable between Chicago to New Jersey. They cut straight through mountains and crossed rivers. The cable stretched 1331 kilometres. And they did this to shave four milliseconds off their transmission time.

And yet, and yet…microwaves came along and rendered the whole project obsolete. Trying to get an edge is expensive.

4. Being willing and able to commit felonies

Insider trading is a thing. See also: criminals who hack or otherwise steal sensitive private information.

And yet, and yet…even when criminals have advance access to earnings reports, they still don’t do all that well, which is evidence for the very strongest form of the EMH (the one that no-one believes can possibly be true).³

So…what was your edge again?

If you’re mumbling something about having ‘good intuition’, or ‘subscribing to the Wall Street Journal‘ then you should consider the strong possibility that you might be Uncle George.

If your answer involves ‘fundamental analysis’ or ‘Fibonacci retracements’, you’re still in Uncle George territory. The only difference is that doing something complicated makes it easier to internally justify the delusion that you know a secret no-one else does. But it is still (probably!) a delusion.

The EMH Gets Stronger With Every Attack

So we know for sure that market-beating edges exist—I’ve written them down for everyone to see!

I can only dream of possessing a halo effect so strong that everyone piles into a stock right after I announce I have graced it with my favour. I don’t have an army of drones at my command, or the ability to bore through mountains to shave milliseconds off my trading times, or a weekly round of golf with the CEO of a Fortune 500 company.

The market is never perfectly efficient. But relative to me, it might as well be.

Critics have pointed out plenty of cases in which the EMH doesn’t jive with reality—and they are absolutely right. So this is where it gets really weird.

The EMH is the only theory that grows stronger with every attack against it.

Edges are constantly at risk of being gobbled up by an efficient-ish market. The ones I’ve mentioned are somewhat defensible: they’re based on personal relationships, capital investment, proprietary technology, etc. But they disappear too.

Most edges can’t even be spoken out loud without disappearing. If stocks systematically rise on the third Thursday of each month but only under a waxing moon, and then someone writes about it in public, you can kiss that anomaly goodbye. The EMH sucks it into its gigantic heaving maw, and it’s gone forever.

In other words: every time someone picks a hole in the theory and points out an inefficiency, they make the predictions generated by the EMH more robust! It’s like some freaky shoggoth thing that Just. Won’t. Die.

Which gets us to the totally justified criticism of the theory: the only reason the EMH can pull this stunt is because it’s bullshit science.

It’s unfalsifiable! It responds to criticism by saying, ‘OK, good point, but now that I’ve factored that in, you should believe in my theory even more.’

And…we really should?

The only way to think about the EMH without going insane is to note that it generates a useful heuristic. It’s not a stable law, like we might find in hard sciences. It’s not perfectly accurate. At any given point in time, there are always competing models that do a better job of describing reality. But all those other models can stop working at any moment, with no warning! By the time you find out their predictive power is gone, it’s too late, and you probably lost a bunch of money! By contrast, the EMH is a reliable model—reliably vague and hand-wavy, yes, but also reliably useful.

We know there are inefficiencies in the market. In the fullness of time, they will be absorbed into the gelatinous alien-god’s hivemind. But before that happens, maybe we can make money off of them.

So now we come to the final test. How do you tell if you’ve really found a market-beating edge, or you’re fooling yourself like Uncle George?

If You’re so Smart, Why Aren’t You Rich?

Everyone knows a secret about themselves, or the people they know well, or can arbitrage some opportunity in a niche that few people are paying attention to. These illiquid private ‘markets’ are much more fertile hunting grounds for asymmetries, and something I encourage everyone to think about.

But the public security markets are a giant agglomeration of everyone’s predictions, which constantly hoovers up every new fragment of information, and recalibrates itself in real time. Your challenge is to try and reliably predict why this giant meta-prediction is wrong, and in which direction.

If you think you can reliably beat, say, an index fund that passively tracks the S&P 500, this is a much stronger claim than it first appears. For one thing, you’re claiming to be better than Warren Buffett, who has failed to pull this off in the last 10 years, despite his huge advantages. But that’s nothing. What you are really saying is that you have the power to beat the greatest collective intelligence humanity has ever created.

This is an extraordinary claim, and the thing about extraordinary claims is that they require extraordinary evidence.

Uncle George’s AAPL trade ain’t going to cut it. Here is the extraordinary evidence that I would personally want to see before agreeing that an investor can beat the market:

1. Big heaps of money

This is the one area of life where there really is no dodging that most venerable of sick burns: if you’re so smart, why aren’t you rich?

So the first piece of evidence I would accept is the fact that someone is very, very rich. Sitting atop a big old pile of cash. And of course, they’d probably also open a hedge fund so they can take other people’s money and turn it into millions more.

2. Track record of outperformance

Maybe a genuine market-beater doesn’t have enough starting capital to make big piles of money on a relatively slim edge, and for some reason is unable to come up with any scheme to beg or borrow more? Or the anomaly is real, but disappears before they can get filthy rich?

In these scenarios, I would also accept a complete record of out-of-sample investment returns over time—no backtests! no selecting the best trades!—as compared against the appropriate risk-adjusted benchmark.

These evidential standards work both in the case that the EMH is ‘dead’, i.e. you can reliably beat the market using public domain info which ought to already be priced in, and in the case that it’s not, i.e. you really do need novel information to get an edge.

As for evidence that the EMH really is dead…hmm. It’s not a proper theory to begin with. But I guess it would be ‘dead’ when the predictions it generates stop being accurate or useful? Which would look something like: ‘finding out that plenty of people meet the standards above, despite having never been in possession of a scrap of information that wasn’t already available to the market’.

In doing so, we’d have to be very careful to make sure we aren’t just looking at the Uncle Georges who unwittingly drew the winning lottery ticket. After all, we should expect plenty of investors to beat the market for a long time—even years on end—entirely by chance.

The Great Coin-Tossing Experiment

Say we held a national coin-flipping contest. After 15 rounds, one in every ~32,800 people would have managed to call every single toss correctly, perfectly predicting a sequence like this:

H T H H T T H H H H T H T T T 

Pretty impressive, huh!

Well, only in a world where we don’t know about probability. In that world, we might mistake blind randomness for skill. The lucky few winners would be hailed as the heroes of their hometowns, do interviews with breathless breakfast TV hosts, and explain that it’s all about the precise flick of the wrist. Aspiring flippers would queue up to buy the inevitable best-selling book, Flip Me Off, and pay exorbitant sums for one-on-one coaching sessions with the master tossers.

Depressingly, this is exactly what happens in the world of investing. What does it mean to achieve the kind of success which only happens by chance with a 0.0003 probability? In the United States alone, it means you end up with 10,000 lucky dopes who are indistinguishable from brilliant investors.

And fund managers don’t need to do anywhere near that well to attract a market-beating aura. They’re incentivised to swing for the fences, increasing the odds they beat the market in some highly visible fashion over some shorter period—say, a lucky season or two. They inevitably regress to the mean, sometimes crashing and burning in spectacular fashion, but it doesn’t matter so long as they manage to hose naive investors in the meantime.

We can never entirely rule out the effect of randomness—there will always be some tiny chance that Warren Buffett is really just the world’s greatest coin-flipper—but we have to draw the line somewhere.

Once the odds of a fluke get pretty slim—someone is super duper rich, and they’ve made a ton of consistently good trades over time—I’d happily congratulate them on their market-beating prowess, give them all my money to invest, listen eagerly to their advice, etc.

The corollary: if you are confident you can beat the market, and yet you notice that you are not sitting atop an enormous pile of wealth right now, you might consider the possibility that you are fooling yourself.

The Four Types of Investors

There are very obvious and well-known reasons why everyone loves to think they can beat the market: overconfidence, confirmation bias, ‘resulting’, selective memory, survivorship bias, etc.

These forces are so powerful that many people—myself included—blithely ignore the vast piles of evidence that suggest beating the market is incredibly difficult, and go ahead and try anyway. All of us think we are special, and (almost) all of us are wrong.

So we can roughly place investors into one of the following four quadrants:

Deluded losers (‘Suckers’)

“Apple stock really is undervalued, but the market hasn’t recognised it yet. I just got unlucky—it was because of [elaborate rationalization]. Also, even if I was wrong this time, I’m usually right. Next time!

“What’s that? Do I track my portfolio returns over time, and compare against the relevant risk-adjusted benchmark to see whether I’m actually outperforming? Well, there’s no need. I usually do pretty well for myself, and I’m expecting to improve—in fact, I just picked up this classic book called The Intelligent Investor…”

Deluded winners (‘Dumb Luck’)

“I knew Apple stock was undervalued! And I remember that other time I made a really good trade, too. Guess I’m pretty good at this game!

“…What’s that? I might have just got lucky? Hah, no. I even did the Fibonacci retracements and everything.”

Realistic losers (‘Clear-Eyed Fools’)

“I keep a meticulous record of my portfolio returns, which forces me to acknowledge the fact that even though I occasionally do well, I am underperforming my benchmarks on a risk-adjusted basis. I am under no delusions about my prospects of finding an edge, and I know I really ought to take Warren Buffett’s advice and put all my money in index funds.

“But I enjoy playing the markets! The same way a night in Vegas is fun, despite having negative expected value on strict financials. So I’m gonna keep gambling with a small part of my portfolio, just for shits and giggles. In the event that I ‘win’, I will try really hard to resist the incredible internal pressure to start thinking of myself as a brilliant investing guru.”

Realistic winners (‘Chosen One’)

“I keep a meticulous record of my portfolio returns, which have outperformed the appropriate risk-adjusted benchmarks to such a degree that I am confident I have found a genuine informational asymmetry. I will of course never tell anyone about it, or it will become useless.

“And I can never be entirely sure: it’s also possible that I just got lucky. But at the very least, I am sitting atop great piles of money, which is pretty nice.”

***

The vast majority of people who actively trade their account are ‘Suckers’. Some smaller number fall into the ‘Dumb Luck’ quadrant (Uncle George would stay there if he never places another trade, but he almost certainly won’t be able to help himself.)

The right-hand quadrants are much more sparsely populated. I guess there are a few ‘Clear-Eyed Losers’ floating around, and a tiny handful of ‘Chosen Ones’.

This distribution is unlikely to be hugely controversial. The question is, which one are you?

Trying to Beat the Market is Like Crack for Smart People

There is a tendency for smart people to wander into areas they know very little about, and think they can do better than the actual experts who have years or decades of domain-specific knowledge, on the basis of being very smart, or having read some blog posts online about ‘being more rational’.

This would be OK if it was just a bit cringe. I love armchair pontificating as much as the next guy! The consequences are usually limited to mildly annoying the people who actually know what they’re talking about, and much eye-rolling when you triumphantly reinvent the wheel.

There is some upside too: reinventing the wheel is fun, because you get to, like, invent wheels. And very occasionally, it might even be true! No doubt smart outsiders are occasionally able to breeze into a new field and exploit some obvious inefficiencies.

But…oh boy. It’s really not true of this particular domain. And it’s not harmless either.

The central prediction generated by the EMH is that you should not expect to be able to beat the market (in the non-trivial sense) unless you have unique information or some similar edge.

This prediction is tested every day. We have great piles of evidence which suggest that it is correct: the vast majority of active investors do really badly.

Crucially, it’s not only regular schmucks who underperform. So do paid professionals, and active managers, and hedge funds, and all sorts of brilliant people who have made this their life’s work.

These days I would put myself in the ‘Clear-Eyed Fool’ quadrant, but only by a fingernail. It’s a constant battle even to stay there. I still do clever things that contradict my own boring advice, and annoyingly, am rewarded for my hubris just often enough to start entertaining the thought that I’m a brilliant investing genius after all. Then I force myself to calculate the IRR on my publicly-traded investments, and compare it against appropriate benchmarks, and manage to get a fingernail-hold back on boring old reality.

To the extent that I have succeeded as an investor, and I am doing quite nicely thank you, it has only come through forcing myself to acknowledge the main prediction that emerges from the very-much-alive-and-kicking EMH.

This has one huge and underappreciated benefit: I occasionally divert some of my attention elsewhere, to domains where I actually do have an edge—and then I win.

Notes:

1. This was literally my job as a market reporter—calling brokers and economists to wrap a plausible narrative around totally inexplicable events, and generate sage nodding of heads.
2. Matt Levine has written some great columns about this.
3. The weak form of efficient markets: historic data is already priced in. The semi-strong form: all public information is priced in. The strong form: all private information is priced in, too.

Not Sure What the Future Holds? Get Your Copy of Optionality Now.