Do not be too timid and squeamish about your actions. All life is an experiment. The more experiments you make, the better. What if they are a little coarse and you may get your coat soiled or torn? What if you do fail, and get fairly rolled in the dirt once or twice? Up again; you shall never be so afraid of a tumble.
I started this blog with a vague premise that there was something valuable about trying weird things, and taking a few steps off the beaten path. I didn’t have any actual justification for this at the time, so I borrowed an impressive-sounding quote from some famous sage or other: ‘All life is an experiment.’
This became the Deep Dish strapline, and its unifying theme. I’ve never made it explicit until now because I didn’t have the models to even understand what I was doing, let alone the ability to communicate whatever this thing was to anyone else (“umm… an old dead proto-hippie said it was cool!”)
Emerson’s non-conformity schtick was enough to get me started. But now I have a much better model for why this is important.
I explained that part in the Embarrassing Problem of Premature Exploitation: powerful structural forces push us away from self-experimentation, at the exact same time as our dynamic world increasingly rewards those inclined towards tinkering.
But I also have a much better model for how to make experiments, which is the topic of this post.
Over the last five years, I’ve done something like 100 lifestyle experiments great and small.
Generally these have ended well; occasionally not so well. When I say I have a ‘much better’ model of how to run experiments, that’s a relative term. I have at least managed to avoid doing really dumb shit, although this was initially as much by accident as by design.
Anyway. Here’s my rough guide to self-experimentation:
1. Define ‘Experiments’ Broadly
Moving to a country where you don’t speak the language is certainly an experiment. But so is changing your brand of toothpaste.1
Or trying on a different identity, learning a new productivity system, testing out products and tools, taking up a sport or hobby, inserting yourself into a different social group, ‘reinventing’ some aspect of your persona, trying a business venture or artistic endeavor, interning or moonlighting in a different career, etc.
Before we start winnowing down the field, it helps to think as broadly as possible about which areas of life can be experimented upon.
2. Start with a Strong (or Plausible) Hypothesis
In 2017, I stopped washing my hair. I’d heard that shampoo and conditioner were ‘solutions’ manufactured by consumer capitalism to solve a problem of its own making: if you left your hair alone to do its thing, it was perfectly capable of self-regulating oiliness, etc.
This was entirely consistent with everything I knew about consumer capitalism, so I triumphantly ditched the ‘poo. Another victory for frugality and simple living!
After six months or so, a friend tactfully let me know my hair… smelled kind of funky. When I washed it for the first time, it was a goddamn revelation. Like one of those ads where the model joyfully tosses her hair around in slow motion. Whoops. Guess consumer capitalism got that one right!
Now, you’re never going to hear about failed experiments like this because of the file-drawer effect: they’re too boring, or too embarrassing, or fail in such a way that I learned nothing.
Fortunately, this is usually not the case!
I have a pretty good hit rate because I’m mostly replicating experiments run by other people, or trying something that seems plausible from first principles.
My hit rate would be a whole lot worse if I came up with a hypothesis at random: what would happen if I started washing my hair with… chipotle mayonnaise?
Of course, testing other people’s ideas isn’t guaranteed to increase the hit rate. There’s plenty of VICE-style clickbait along the lines of “I tried Dwayne the Rock Johnson’s workout for a week!”, or “I tested every one of Gwyneth Paltrow’s yoni eggs at the same time!”
This is a problem of discernment, i.e. being so open-minded that your brain falls out. You could quite easily waste an entire lifetime trying junk experiments inspired by the vast field of health and wellness-related quackery.
There is no way to get around this, except by first learning the meta-skill of how not to be a credulous sucker, how to think statistically, how to evaluate which ‘experts’ are trustworthy, etc.2
3. Bet on Asymmetric Returns
Optimising for a ‘success vs failure’ hit rate is too simplistic. What if you run an experiment which has a 99 per cent chance of paying off, but a 1 per cent chance of ruining your life? What if it’s almost certainly going to flop, but has a tiny chance of bringing you great fortune?
This is why I’ve moved to the optionality model, which tells us to:
- run a million miles from anything with a risk of ruin,
- not waste too much time or effort on anything with a bounded upside, and
- look for asymmetric returns: a capped downside cost, and the possibility of a large/unbounded upside
A good example that fits this approach was starting a blog. The downside was limited, with some reliably modest benefits, along with exposure to occasional strokes of life-changing serendipity.
A bad example was attempting to hike across the Himalayas in flip-flops. There was essentially zero upside, except the amusement of myself and my friend, and, uh… writing some clickbait for VICE.3
4. Make it Reversible
A big part of capping the downside risk is not getting locked into anything.
This usually looks like making a series of small bets to gather data, rather than jumping right in the deep end. Sometimes this is very obvious: if you’re thinking of permanently moving to San Francisco, of course you’re gonna make several trips to scope it out before you uproot your entire life.
But judging by the mausoleums to aborted lifestyle experiments known as ‘garages’, it’s not always that obvious. In the Barbell Strategy for Buying Stuff, I warned against buying anything for life if it’s hobby-related or whimsical:
No matter how strong your burst of initial enthusiasm, try not to rush in. Get the entry-level version, preferably secondhand, and see if the passion persists. Once you get skilled enough to actually benefit from having triple-forged titanium alloy frames or whatever, go ahead and splash out.
To run with the blogging example: I initially spent $20 on a domain name, the cheapest possible hosting, and a default WordPress theme. If I abandoned the project in the first few months—which is what happens to most blogs—I wouldn’t have wasted much time or money.
As the results have trickled in, I’ve invested a lot more resources into design, functionality, etc. And I’m still experimenting: if certain things work out, I’ll take the next leap. If they don’t, I won’t.
There are some experiments which aren’t strictly irreversible, but I would still avoid, to the extent that they create strong path dependencies.
For example, I encourage trying on mostly-harmless identities to see how they fit, like ‘rock climber’ or ‘minimalist’, but I would be way more cautious with ‘Bernie Bro’ or ‘libertarian’. Ideology-based movements tend to lock you into path dependencies, and in the worst case, rot your brain until it falls out your nose.
In a similar fashion, I’d avoid experimenting with substances that have a risk of ruin (overdose, psychosis), but also with anything which creates path dependencies (addiction).
5. Go Big or Go Home
I’m not actually sure if this is a recommendation or a disrecommendation. But I have noticed a pattern where I go hard on some experiment, reap the rewards in a major and obvious fashion, then swing back to a more nuanced position. Some examples:
Minimalism
Thesis: I started out by getting rid of everything I owned and living out of a 22L backpack. This was a useful and liberating constraint.
Antithesis: More recently, I noticed I was getting a lot of pleasure out of owning high-quality stuff, and that minimalism could easily degenerate into a dumb signaling game.
Synthesis: Now I have a checked bag, and own something like 300 items: several times more than my ultralight backpacking days, but still ~1000x less than the contents of the average American home.
Frugality
Thesis: I started out by pruning my personal expenses and saving half my pay. This was far and away the best experiment I’ve ever done, and gave me the slack needed to do everything else.
Antithesis: More recently, I loosened the purse-strings a little, have become less puritanical about how I (and others) spend money, and am feeling good about that.
Synthesis: My expenses are up ~50 per cent from my most ascetic phase, but are still extremely low in the grand scheme of things. And I’m looking for more ways to spend money, not less.
Reading
Thesis: I started out by setting a goal to read 100 books a year, and implemented a bunch of strategies for reading more.
Antithesis: In the course of achieving the goal, I noticed I was confusing the measure with the target, and kind of sucking the fun out of it.
Synthesis: Now I read more intentionally: a higher volume than before I tried the experiment, but with no pressure to hit any particular number, and no risk of Goodharting myself.
Wouldn’t it be better to try and shoot for that more nuanced position right away? Yeah, no kidding, but I’m not sure that makes sense. If I knew the best outcome in advance, I wouldn’t need to run the experiment!
The benefit of testing the strongest possible version is that it gives you a much more pronounced result, with an exaggerated sense of both the benefits and the downsides.4 So long as it fits the other guidelines (reversible, capped downside) you don’t have much to lose.
6. Do Less in Stable Domains
Life is volatile and only getting more so—hence the importance of constant tinkering—but some domains are relatively stable.
This includes ‘the laws of physics’, human physiology, etc. In these areas, doing endless experiments is pretty quickly going to run you up against the law of diminishing returns. After all, the whole point of exploring more is to get you in a better position to exploit.
You might keep a little window open for exploration, in case your preferences or circumstances change. Other than that, it’s fine—optimal, even!—to keep repeating the same old thing until you’re a fully calcified fossil or the sun implodes; whichever comes first.
For example: I started out vaguely ‘exercising’, got into powerlifting, moved to calisthenics, and ended up with a hybrid of the two. Now I’m done. These practices have been around for millennia, and are Lindy-approved. Unless someone magically invents a new activity tomorrow that moves mass through space in a way that is even more cheap, fun, and effective than calisthenics, then I’m not interested. I’ll keep making tweaks here and there—this or that specific exercise, or program—but I’m not going to suddenly pivot to camelback polo steeplechasing or whatever.
I hope I’m not too far away from being done with the same optimizing process in diet, sleep hygiene, and routines, which, again, are relatively stable. The aim is to get to good enough and leave it there, for my own sanity as much as anything else.
By contrast, I hope I never stop tinkering with my career, skill acquisition, learning, curiosity, etc—personal preferences aren’t stable over time, and neither are the activities the world rewards us for.
7. Experiments Don’t Have to be Scientifically Rigorous
I had a crushing realisation the other day. Even though I’ve read a zillion pop-science books and a fair bit of original research, I’m still a skittish dilettante, bordering on scientifically illiterate.
This is something I aim to fix, but the cool thing is that it hasn’t hampered my efforts at all!
So long as you follow point 3 (asymmetric returns) and point 5 (go big or go home), there’s no need to set up a double-blinded trial or whatever, because the results are gonna be about as subtle as an elephant juggling a neon sign flashing “I’M SUBTLE”.
If you want to know whether weightlifting works, you just try to pick up something heavier than you picked up last month, or look in the mirror, and then you have your result. Ta-da!
If you want to know whether frugality works, you ask yourself: ‘am I having fun, and saving lots of money?’ and then you answer ‘yes’ or ‘not really’, and then you have your result. Ta-da!
The most promising self-experiments, as defined in this post, have nothing to do with being a human guinea pig on the bleeding edge of scientific research.
You’re not testing whether something works in a universal sense—you’re testing whether it works for you. n=1 is fine. In fact, it’s ideal! The questions that matter: can I make this part of my life? How should I best implement it?
Sometimes the results are invisible, and the only thing you’re testing is personal preference.
For example: I have no way of knowing for sure whether long-term fasting promotes autophagy, short of carving out a cross-section of my thigh to count the autophages. This is something I am trying solely on the basis that smart people think it has a reasonable chance of extending my life. But I have gathered a ton of valuable personal data on, for example:
- which of the many versions of fasting suit me best
- the optimal timing of fasts and how they jive with other activities (work, fitness, sleep, etc)
- which foods I should eat to prepare for a fast, or to comfortably break a long fast
- what to expect psychologically; what to do and not do
Some of which—get this—might even be of use to others!
Again, the saving grace here is not gonna be whether I preregistered the experiment, or obeyed the conventions of open science or whatever. It’s gonna be the degree to which I followed point 2, i.e. how good an idea it was to begin with.
8. But Sometimes… They Really Do!
OK, sometimes you have to do the hard work.
I wanted to try micro-dosing LSD. All the cool kids were doing it, and reporting enhanced creativity and feelings of wellbeing. If I had tried it, I would have no doubt reported the same…and almost certainly ended up fooling myself.
Fortunately, a mysterious online researcher named gwern saved me from wasting my time. Having noticed that no-one had rigorously tested the efficacy in ~50 years, he ran the experiment himself—but randomised and blinded, with distilled water as a control/placebo. Here are his findings:
No beneficial effects reached statistical-significance and there were worrisome negative trends.
That this cuts against hundreds of positive anecdotes isn’t really all that surprising. Experimenting with heroic doses of LSD generates extremely obvious ‘results’: if you don’t know whether it’s working, the coruscating trapezoid rabbit creature will be sure to tell you.
But taking microdoses and then trying to judge subjective things like ‘equanimity’ and ‘creativity’ is impossible, almost by definition: the dose is below the threshold of perception! If there was ever a time to get suckered by placebo or expectancy effects, this is it.
In this case, one rigorous gwern post is worth 100 vague thinkpieces on Medium. Without a proper experiment design, it’s garbage-in, garbage-out.
But this usually doesn’t matter. The low-hanging fruit come from following points 2, 3, and 5. You could reap almost all the rewards of self-experimentation with a level of literacy that never goes beyond liking ‘I Fucking Love Science!!!’ on Facebook.
So why go any further?
9. Experiments Are Not About Outcomes
When you’re trying to measure things that don’t have obvious results, like taking tiny, sub-threshold doses of LSD, you’ve already crossed the line into fiddling around on the margin.
For a baseline healthy person with no deficiencies, it’s not as if you start tinkering in this fashion and your whole life suddenly changes (and if it did, once again, you wouldn’t need to measure anything!). We’re talking about marginally better sleep, or whatever. The upside is bounded.
The case in favour: Even modest upside is worth having. If I can accumulate a bunch of 0.5 per cent improvements here and there, that really starts to add up.
The case against: Maybe it’s a distraction from the things that actually matter. In the strength training world, this is known as fuckarounditis: if you are not proficient in the heavy compound lifts, you have no business doing split stance bosu-ball cable rows to isolate the third head of your non-dominant coracobrachialis.
Through this lens, something like Quantified Self looks a lot more like a hobby than a high-impact way to improve health or performance. A fun and cool and interesting hobby! But a hobby.
And that’s exactly why I’m planning on getting into it. After a certain point, self-experiments are not about the outcomes. They’re just a handy excuse to learn more about the world.
My compulsion for tracking stuff means I already have a baseline of raw data going back years: sleep, productivity, resting heart rate, measurements, weight, mood, etc. Now that gives me a fun, real-world application for learning actually useful stuff.
A relevant passage from gwern’s 2019 review:
I think [running personal self-experiments] is an underappreciated way of learning statistics, as it renders concrete all sorts of abstruse issues: blocking vs randomization, blinding, power analysis, normality, missingness, time-series & auto-correlation, carryover, informative priors & meta-analysis, subjective Bayesian decision theory—all of these arise naturally in considering, planning, carrying out, and analyzing a self-experiment.
I only understand about half of those words, which is a little bit intimidating, but also exciting. If anyone has advice on how to go about getting started with statistics for someone who hasn’t done it since high school, and presumably Python or R or something, let me know.
Again, I don’t expect this to generate especially exciting results. It’s much more like fine-tuning, rather than hunting open-ended upside with the potential to be transformative.
But you never know! Maybe there are some juicy insights that I could only stumble across through careful number-crunching. In which case, I look forward to sharing them with you in the name of open “science”—if they’re not too embarrassing, and if I actually manage to learn some maths.
Those are some pretty big ‘ifs’. In the meantime—and seeing as my biggest wins often involve copying smart weirdos—you should all tell me about your best experiments, so I can steal those too.
Notes:
- The active ingredient in most toothpaste is sodium fluoride. But there’s another version—stannous fluoride—which does a better job of preventing tooth decay and gum disease. The only reason we’re not all using it is a historical quirk: it used to taste funky and cause some staining, but the new formulations have resolved those problems. If you want to find out whether it works, this experiment will cost you approximately seven dollars to run. (Hat-tip to Rob Wiblin.)
- I might put together a reading list along these lines at some point, with a bunch of pop science books I’ve found useful (Thinking in Bets, Risk Savvy, Algorithms to Live By) and some of the better material from the rationalist community.
- I took precautions to manage the downside—there was never any serious risk of frostbite—but it probably did make me slightly more likely to slip on scree or ice and fall down a cliff or something.
- This is not true for anything that follows a U-shaped dose response curve, where doing ‘too much’ might give you a false negative (which is another reason why I’m not sure whether this is good advice.)
Not Sure What the Future Holds? Get Your Copy of Optionality Now.
Hi Richard,
I’m an engineer who turned to statistics after falling in love with it (don’t ask) and I might be able to help you with the “how to get into statistics part”.
I suggest you to read Discovering Statistics with SPSS (and sex and drug and rock ‘n roll) by Andy Field, a brilliantly written book that is more than capable of getting you into it, oftentimes making you laugh in the process – which, as you can probably guess, comes as a surprise. You can find it online for free in the usual ways.
As for R, you could start by downloading the RCommander package, which creates a Graphical User Interface rather than forcing you to learn coding (basically you point and click from drop-down menus as you do in Excel or as you woud do in SPSS).
Let me (us) know how that goes!
Hi Luca, thanks very much for the suggestions.
I am intrigued both by your forbidden love, and by a statistics text that is not only funny, but generates reviews like this: “The book is a constant bombardment on anyone with some morality…It is a shame that people would help pay royalties to this author’s crass writing.” Sounds right up my alley!
Do you have an opinion on R vs Python? I’m weakly leaning towards the latter because it seems more broadly useful in the event I want to use it for other stuff. But the point-and-click package does sound appealing.
R vs Python is a forever young debate in the stat/programming community, with (as usual) good arguments for both options.
They are both extremely powerful and I highly doubt that you will ever find yourself wishing that they could more to manage your “other stuff”. And, should it come to that, you will have quit this blog and opened one on data analysis/programming instead anyway, because your life will have become so data-centric that you won’t have time for anything else.
Back to the point.
At the end, it all comes down to what you want to do with it: do you need to become a programmer or a data scientists? Then Python – as the mail you shared suggests – may be the way to go (even though I’m a statistician working in the industry and I only use R).
If you just want to learn how to do some analyses on your own data for little projects, possibly concluding them with some nice plotting, then I think any point-and-click solution would work just fine; should you then fall in love (or get sick – I guess it depends from the perspective) like I did, you can always up your game further down the road.
I suggested R Commander because that’s the one I know, but, again, for starting they’re all the same.
Cool, thanks Luca—It’s all a little bewildering to an outsider so this is really helpful context.
is the question I want to answer before I go any further. Still have other projects that take me through until the end of the year, so will sleep on it until then.
On the subject of washing, you might be interested to read Ruth Goodman (How to be a Victorian, Penguin 2013), who experiments with doing things as the Victorians actually did them. One technique for keeping clean involved dry rubbing the body, which apparently works pretty well. She writes: “The longest I have been without washing with water is four months – and nobody noticed.”
For hair washing, she notes (without reporting personal experiments): “At the start of the nineteenth century, washing your hair was a daring task, and one that went against conventional health advice – you were supposed to brush and comb is thoroughly twice a day to dislodge any dirt. Brushing would also spread the natural oils and ‘fluids’ across the length of the hair, improving its condition.”
Maybe you should have tried brushing your hair more thoroughly!
That’s fascinating, thanks. I know quite a few people who are sold on ‘no poo’, and their hair looks fine, so maybe I was doing it wrong!
One from the mailbag:
The Manga Guide to Statistics is a fun, easy read. It won’t get you into Bayesian probability theory, but it’s a good place to start if you haven’t seen it since high school.
Thanks for the recommendation. I can’t believe I didn’t know there was an entire genre of ‘EduManga’, this is amazing.
I am also a self experimenting junkie so I found myself agreeing with most of what you wrote but I would point out applying more scientific methods to lifestyle issues often isn’t all that helpful (my background is engineering by the way…I fall into problem solving mode at the drop of a hat). It’s like trying to use a two handed sword as a letter opener…it can work but generally it is just overkill. You don’t need a pile of data sliced twenty different ways to make a decision of ‘did this work for me’ which is often the point of self experiments.
But I have noticed over time as my life hits a more optimized point that my experiments are getting less generally life improving and more I want to try out something that may not have much general upside overall but some specific upside for me. Case in point my latest experiment was getting into all grain brewing of beer (after doing simple wort kits for a while) where I spent more than I needed on equipment because I wanted a simple setup and was willing to pay for ease of use (but I have yet to make a beer I didn’t enjoy). So yes, I can make a bloody cheap pint but I won’t really get my costs recovered for VERY long time but the point was really about more flexibility on brewing options (kits only came in so many options, now I can literally make just about any ale).
So in time you will hit a vanishing point where a lot of the big gains are gone but you can still try new things and make adjustments as you go. So just keep that in mind as you go forward. Best of luck on the journey.
Hey Tim, good to hear from someone who has been doing this for a long time.
That’s a great way of putting it, I’m definitely stealing this line.
Hey Richard Love the post! Big fan, also did you see that Roam Research got covered by Thomas Frank? He’s got like two million subscribers on youtube. He’s a big study hacks guy, he’s been doing video on Cal Newport before it was cool. You like invest in Roam right?
Anyways I’ve told a bunch of people about it and am a daily user myself.
Hey Jack, thanks bud. Yeah I did see that, although I didn’t recognise the name – will have to take a look. So glad you’re hooked on Roam! If you’re on twitter, come join the #roamcult. There’s a slack group too, lots of amazing people doing all sorts of cool stuff with it.