The ‘Bias’ Bias: No, Your Brain is Not Made of Swiss Cheese

a cross-section of the brain of homo irrationibilis

When I grew out of my annoying teenage atheist phase I needed a new way to feel superior to other people. In a stroke of luck, this was around the peak ascendancy of behavioural economics, a field devoted to pointing out how everyone was going around being WRONG!!! all the time.

From 2012-2018ish, I was obsessed with this stuff. And I wasn’t alone: Daniel Kahneman’s Thinking Fast and Slow was a phenomenon, selling millions of copies. I discovered an online community of ‘rationalists’ devoted to practicing the art of thinking clearly, and whose founding blog was literally called Overcoming Bias. I lapped up everything they wrote.

Fast forward to a few months ago, when Kahneman died. It got me thinking about just how little impact any of that stuff has had on my life and my thinking, despite all the hype at the time.

I don’t think it’s controversial to say that the field of behavioural economics has failed to live up to its promise. The stronger claim I want to make is that its very foundations are crumbling, right up to and including the great central pillar of prospect theory for which Kahneman received his Nobel prize.

A quick refresher: we are capable of slow and conscious deliberation—what Kahneman calls System 2—but we very often use fast cognitive shortcuts instead. Usually, these shortcuts work well (in which case we call them ‘heuristics’), but in certain situations they misfire (in which case we call them ‘biases’).

I think that most apparent biases are, in fact, perfectly serviceable heuristics, and that people are far more rational than we have been led to believe.

I realise this is a bold claim. But I’m really only setting it up in contrast with the Wikipedia page for cognitive biases, which turns up some 200+ types of mistakes we’re making, implying our brains are absolutely riddled with exploitable errors and generally have all the consistency of swiss cheese.

The proliferation of new biases is partly explained by a quirk in our behaviour called The ‘Bias’ Bias: it feels good to believe we have special insights that make us smarter than other people.

The other explanation is that it is very easy to make up biases, like I did just now: they are bad explanations because they are easy to vary. You can find a bias to explain any behaviour you want, including in opposite and contradictory directions. If you ever find yourself losing an online argument or lacking inspiration for a dunk on the outgroup, you have 200+ smart-sounding Science Effects right there at your fingertips.

Of course, most biases come from a literature that attempts to actually ground their existence empirically. So let’s start there.

Broadly speaking we could say there are four kinds of biases:

1. Biases that are totally bogus, i.e. the result of shoddy or fraudulent science
2. Biases that are actually heuristics, i.e they describe perfectly rational behaviour
3. Biases that do not track the truth but it would be a mistake to try and ‘correct’
4. Biases that are a source of error and we should totally try to correct

1. Biases that are totally bogus

The book that got me hooked on this stuff all those years ago was Predictably Irrational, written by a famous Duke professor named Dan Ariely. In 2021, independent researchers discovered that Ariely had faked the data on a study about…dishonesty, of all things. Incredible stuff.

For some reason Ariely still has his job. You can see why I’m kind of jaded about all of this.

The field of social psychology is showing its whole ass, but outright fraud is just a tiny little pimple on the bum. The replication crisis revealed that the majority of studies—even those published in top journals—could not be reproduced, usually for mundane methodological reasons.

Thinking, Fast and Slow contains gems like ‘The Marvels of Priming’, in which Kahneman claims that if you prime people’s subconscious with a bunch of words related to elderly people, they walk more slowly down the hallway afterwards. Here’s the money quote:

Disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true.

Whoops! Looks like someone fell victim to overconfidence bias, cos the major conclusions of these studies were…not true.¹

Besides shoddy research methodology, the other great scourge of the field is publication bias (hehe), in which studies with interesting results are vastly more likely to be published than those that support the null hypothesis.

This is pretty much what happened with the whole ‘nudging’ obsession: making people eat less by giving them small plates, changing retirement plans from opt-in to opt-out, and so on. There were elements of outright fraud there too—Brian Wansink’s p-hacking is breathtakingly audacious stuff, he wasn’t even hiding it—but it’s mostly just more mundane selection effects. Once you adjust for publication bias, the effect size of nudging either mostly or entirely disappears.

2. Biases that are not biases, i.e. they describe perfectly rational behaviour

The central pillar of Kahneman’s work, and the foundation of his prospect theory, is the concept of loss aversion: that losses loom larger than gains.

Here’s the classic diagram with the kink in the utility curve:

Loss aversion crumbles when you account for ergodicity: in plain English, the path dependence of wins and losses. It’s perfectly rational to be ultra-cautious in a non-ergodic system—like, say, life—where the ensemble average is different to the time average.²

A simple example to pump intuition: you own a stock which returns either +2 per cent or -2 per cent every day. How much money will you make?

The arithmetic mean is 0 per cent, so it seems like you’d just crab along sideways. But the geometric mean—the growth rate of your returns over time, which is the only thing we actually care about—is negative:³

Over a long enough timeframe, you’ll go bust with 100 per cent probability. And if we crank up the volatility—for e.g. the stock returns plus or minus 10 per cent each day—you’ll go bust much, much faster:

Kahneman himself gives the example of a coin-tossing game in which you win $110 if the coin comes up heads, and lose $100 if it comes up tails. He is surprised to find that most people don’t want to play the game, even though the expected return to playing is a gain of $5.

But to the extent this is true, most people are right to not participate. Again: the simple average return is positive, but the geometric mean is negative! If Kahneman had run this past literally any trader or serious gambler, they would have pointed out his mistake.

What if we constructed a game with a slightly positive long-run growth rate, and found that people still refused to play? Would that be enough to rescue loss aversion?

I don’t think so.

Knightian Uncertainty Rules Everything Around Me

The economist Frank Knight drew a distinction between risk, which can be quantified and measured, and uncertainty, which cannot.

If you have perfect information, as in the coin-flipping game, then you’re operating in the realm of risk. In this case, loss aversion really is a source of error: instead of going with your gut, you could have just run the numbers and got the actual, correct answer about the geometric return.

The only problem is that literally nothing looks like this in real life.

The closest analogue I can think of is quant trading. You build a model that tells you the returns and volatility of a given trading strategy. Then you use the Kelly criterion to calculate the optimal proportion of your bankroll you should bet on each trade in order to maximise the growth rate of your account.

All nice and clean and based on hard numbers, except…any serious trader knows you never bet full Kelly!

Why not? Because even the best models trail vast clouds of uncertainty: volatility is based on past events, which will not repeat (perhaps the mother of all black swans will come along and wreck you). Your edge is similarly ephemeral: you have no idea if it will disappear overnight, you won’t be able to tell when it does, and there’s always the possibility that it never existed in the first place (e.g. it was an artefact caused by overfitting the model).

And this is the best-case example. Everything else in life is much, much messier!

The big lesson that Nassim Taleb is always banging on about is that outside of sterile lab conditions, risks are not quantifiable. Reality is high-dimensional, and full of unknown unknowns: it’s uncertainty all the way down, baby.

Of course, we still have to make decisions, and trying to quantify them is often more useful than not. But since the path dependence of losses hurts us more than gains help us, and we never have perfect information, a rational person should always bake in a safety buffer. Perhaps we could call it some kind of… ‘aversion’ to ‘losses’.

We could no doubt come up with contrived examples where loss aversion hurts us. But presenting this as if it were some generalised mistake is not only wrong, but dangerous: try to correct for it and you’ll end up with a much worse outcome (maybe even a ruinous one).

A lot of biases turn out to be perfectly rational when you’re uncertain or you have limited time to run the numbers—which is to say, the normal conditions of life. Let’s quickly run through a few more examples.

Sunk Costs and Status Quo Bias

Gwern asks Are Sunk Cost Fallacies? and concludes: not really. It’s surprisingly hard to find actual concrete examples, and there are all sorts of useful reasons we might stick with things longer than we ‘should’.

Again, this has to do with uncertainty: if we had perfect information about the costs and benefits of sticking with a decision, then yes, sunk costs would be irrational.

But it also has to do with computational resources. Consider this tweet from Eliezer Yudkowsky, the grand chief poobah of the rationalists:

This is silly in part because there is massive uncertainty in your assessment of how much “better” someone is, which means the specific quantified thresholds are meaningless.

But the main problem here, and the source of most of the dunks that followed, is that this is no way to live, man. Imagine that every time you meet someone new, you have to fire up Excel and run the numbers on whether or not to leave your wife. You’ll also need to run the numbers on whether you should abandon your startup every time you see a job listing, whether or not you should sell each of your possessions, and whether or not you should change your sports team after every game.

This is stupid not because you should never reassess these things—in some circumstances you absolutely should—but because it would drive you insane and occupy every waking moment of your life. It almost always makes sense to stick with the status quo!

The Availability Heuristic

This is the one about how we tend to rely on things that come to mind quickly, either because we encountered them recently or they’re more memorable (e.g. we think shark attacks are much more common than they really are).

I’ll just quote Jason Collins here, who is great on all this kind of stuff, and from whom I’m cribbing throughout this post:

The unbiased way to make a decision under uncertainty is to sum the utility of all possible outcomes weighted by their probability. This, however, is typically computationally intractable where there are many possible outcomes. In that case, a more tractable approach is to instead sample a limited number of possible outcomes, which comes with the cost of possibly not including rare but extreme outcomes. Falk Lieder, Ming Hsu, and Tom Griffiths showed that the ‘rational’ solution to this computational constraint is to over-sample extreme outcomes. That is, you should apply something like the availability heuristic by calling those more extreme (easily accessible) outcomes to mind. The result is a biased estimate, but one that is optimal given the finite computational resources at hand.

3. Biases that do not track the truth but it would be a mistake to try and ‘correct’

Type 3 biases are defined by conflating ‘what is true’ with ‘what is effective’. Evolution doesn’t give a shit about the truth. Evolution cares about winning. That means there are all kinds of scenarios in which it makes sense to strategically delude ourselves, and in which we might actually make our lives worse by trying to ‘fix’ our wiring.

Overconfidence Bias

Kahneman clearly suffered from the dreadful scourge of overconfidence bias. It may not have always done wonders for his truth-seeking powers, but it did give him the kind of moxie required to carve out an entire new field of research, write a best-selling book, win the Nobel prize, etc.

I have no idea as to whether he was self-aware about this stuff, but here’s the relevant snippet from his Conversations With Tyler interview:

Overconfidence has many virtues. In the first place, it’s nice, it’s pleasant to be overconfident, especially if you’re an optimist. […] To exaggerate the odds of success is a very useful thing for people. It will make them more appealing to others, they will get more resources, and they will take risks.

Kahneman thinks overconfidence is not necessarily a net benefit for individuals—it causes them to take risks that usually won’t pay off—but it is good for society as a whole.

I agree that entrepreneurs are taking one for the team, but am a little more optimistic about the individual benefits of positive thinking. Check out ‘Is the Law of Attraction Real?’ for my take on self-fulfilling prophecy, and how it dovetails nicely with the predictive processing model of how the brain works.

One of the reasons I parted ways with the rationalists is that they are optimising narrowly for truth-seeking, which inevitably trades off against things like ‘happiness’ and ‘effectiveness in the world’. To the extent that your values include literally anything other than being right, it is perfectly rational to be irrational. Or as Taleb would say: suckers try to win arguments, nonsuckers try to win.

The Planning Fallacy

This is the one about how we always underestimate how long it’ll take to complete a project. I have experienced this myself so many times that I don’t doubt it’s true. But if we eliminated the planning fallacy overnight, I’m pretty sure we would achieve way less stuff, and especially the really big ambitious stuff that changes the world.

Here’s Paul Graham in How to Do Great Work:

Work has a sort of activation energy, both per day and per project. And since this threshold is fake in the sense that it’s higher than the energy required to keep going, it’s ok to tell yourself a lie of corresponding magnitude to get over it.

So you just have to fool yourself long enough to get over that initial resistance. Once you’re invested, sunk costs take over and force you to keep going.

4. Biases that are a source of error and we should totally try to correct

…

???

I did zero research for this post. I just used whatever examples I happened to find in my notes. But if you poke and prod at any of the other hundreds of biases categorised as Type 4, I am confident that most will also fall apart under scrutiny (i.e. they are actually Type 1, 2, or 3).

The aforementioned Jason Collins has a great Works in Progress piece analysing what went wrong here. The problem is that instead of having a sound theoretical framework to guide our decisions, we have a grab-bag of tricks that can be used to tell whatever story we want:

Suppose you are studying a person deciding on their retirement savings plans. You want to help them make a better decision (assuming you can define it). So which biases could lead them to err? Will they be loss averse? Present biased? Regret averse? Ambiguity averse? Overconfident? Will they neglect the base rate? Are they hungry?

From a predictive point of view, you have a range of countervailing biases that you need to disentangle. From a diagnostic point of view, you have an explanation no matter what decision they make. And if you can explain everything, you explain nothing.⁴

**

Collins wants a new unified theory to replace the scrappy patchwork collection of biases, and is against merely tweaking the original rational-actor model.

I don’t fully understand his reasoning here. To me, the original model seems like it still works OK, so long as it avoids the following mistakes:

• conflating bias with truth-seeking
• assuming perfect information
• assuming unlimited computation
• measuring one-off instances instead of the global pattern of behaviour

Which you could formulate as something like:

Humans make decisions that maximise the expected geometric growth rate of utility, under constraints of uncertainty and finite computational resources.

Call it the ‘rational-ish’ model. I really don’t understand why we should expect to find a new, highly elegant theory that does any better than this. OK, it’s hard to quantify things like utility or uncertainty, but those problems will remain no matter which framework we use.

Crucially, the rational-ish model gives us a strong theoretical grounding against which to judge apparent biases: it tells us that evolution really did get things broadly right, and if you think otherwise, you better be sure you’ve accounted for all the factors above.

It also tells us where we’re most likely to find actual Type 4 biases: by looking for strong mismatches between modernity and the ancestral environment in which our cognitive shortcuts developed.

So I guess what I really want is to shift the burden of proof. An experiment that finds some apparent bias is nowhere near enough to proclaim the discovery of an actual gaping hole in our thinking. If we don’t have a good explanation for why it should exist—ideally one that we came up with before we went out looking for it—then we are very likely making one or more of the Type 1-3 mistakes.

How to change your behaviour in light of the above

My guess is that most people engage with the biases and heuristics literature only so far as it provides handy bludgeons to malign the motives of their enemies. In which case: carry on!

But if you are still saying things like ‘ah, I need to adjust for blah blah bias’, or genuinely trying to make predictions about human behaviour by invoking this stuff, then it might be time for a rethink.

At best, it’s a gigantic waste of time that distracts you from the process that will actually lead to better outcomes: namely, going out into the world and doing stuff. If you want to improve your System 2’s explicit models, you have to gather information and reduce uncertainty. If you want to sharpen up your intuitive System 1’s cognitive shortcuts, you have to get plenty of time on the tools: experts tend to use less information than amateurs do.

Both point to less time sitting around memorising lists of biases and fallacies, and more time taking action.

A few heuristics to wrap up: As a starting point, assume that your biases are useful. Do not update based on sterile lab experiments. Remember that it is often instrumentally useful to fool yourself. If you have a lot of uncertainty, fast and frugal heuristics will serve you much better than complicated models. And if someone tries to tell you your brain is hopelessly riddled with cognitive defects, send them this post.

Notes

**