Does It Really Work? What a Single Self-Experiment Shows

A single self-experiment shows you what happened on that one occasion, and that is a good start. Whether the result came from the thing you tested, from chance or from your own expectation, one try alone cannot tell apart. For that it needs a comparison, a few repeats and, where possible, blinding, and all of this can be set up at home. Research puts clear numbers on how much these building blocks matter. A 2001 analysis of 130 clinical trials found that small trials show larger effects than big ones, and a self-experiment with n = 1 is the smallest trial there is. When 100 psychology experiments were repeated, 97 percent of the original studies had been statistically significant, and 36 percent of the repeats were. To judge whether something really works, look at the control condition, at independent repeats, at an established measuring method and at the significance test.

What does a self-experiment with a single measurement show?

A self-experiment is a single observation on a single subject, usually without a comparison group and without repeats. Statisticians call this situation n = 1.

The result may well be right. It just has a mathematical problem: a sample of size 1 has no spread that could be estimated. Every measurement fluctuates, even the same measurement on the same subject on two different days. How large that fluctuation is only becomes visible with a second measurement. Only then can a real difference be separated from random ups and downs.

Selection plays a part as well. When you test yourself, you alone decide when to measure, what counts as a result and when to stop. These decisions are often made after a glance at the data, with no intent at all. The building blocks of a proper study exist to guard against exactly that.

The step from 1 to 2 or 3 is therefore a big one. A single observation becomes a hypothesis, a good question for the next measurement. An anecdote turns into data once a comparison and repeats are added, and repeating is the simplest step in that direction.

Why does a control condition decide the result?

A control condition is the second arm of an experiment, treated the same way, in which only the thing being tested is missing. An effect is calculated against this point of comparison.

Asbjørn Hróbjartsson and Peter Gøtzsche measured how much this matters in 2001, across 130 clinical trials. In these trials, participants were randomly given either a placebo or no treatment at all. The analysis covered 32 trials with binary outcomes in 3,795 patients and 82 trials with continuous outcomes in 4,730 people.

“As compared with no treatment, placebo had no significant effect on binary outcomes (pooled relative risk of an unwanted outcome with placebo, 0.95; 95 percent confidence interval, 0.88 to 1.02), regardless of whether these outcomes were subjective or objective.”

>

Source: Hróbjartsson A, Gøtzsche PC, Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment, New England Journal of Medicine 344(21), 2001, pages 1594–1602, DOI 10.1056/NEJM200105243442106

The continuous outcomes are where it gets interesting. There the placebo did show a benefit, with a standardized mean difference of −0.28. The larger the trials were, the smaller this benefit became. Small trials systematically show more effect than large ones, and a self-experiment is the smallest trial possible.

For your own observation, the practical lesson is simple: first ask what you are comparing against, and then ask how big the difference is.

What does expectation do in an experiment?

The nocebo effect is a worsening of how a person feels that comes from their expectation, independent of the stimulus they suspect. Its counterpart, the placebo effect, works in the other direction.

For electromagnetic fields (EMF), the data on this are unusually clear. In 2010, a team led by James Rubin at King's College London brought together all available blinded provocation studies. In these studies, people who describe themselves as sensitive to electromagnetic fields were exposed to real or sham fields.

“Including studies reported in our earlier review, 46 blind or double-blind provocation studies in all, involving 1175 IEI-EMF volunteers, have tested whether exposure to electromagnetic fields is responsible for triggering symptoms in IEI-EMF. No robust evidence could be found to support this theory.”

>

Source: Rubin GJ, Nieto-Hernandez R, Wessely S, Idiopathic environmental intolerance attributed to electromagnetic fields: An updated systematic review of provocation studies, Bioelectromagnetics 31(1), 2010, pages 1–11, DOI 10.1002/bem.20536

The same review supports a role for the nocebo effect in triggering acute symptoms. The people in these studies really did feel their symptoms. Whether a field was switched on at that moment, however, they could not tell any better than by guessing.

For an experiment of your own, that is a useful insight. As soon as you know when the thing you are testing is present, your expectation comes into play, in either direction.

How do you make your own experiment meaningful?

Want to try it yourself? That is a good way to find out, and a few simple steps get far more out of your experiment. A good self-experiment is a small trial built from the same parts researchers work with.

• Decide in advance what you will watch for. Write down what you want to observe, such as how you slept or how alert you feel in the afternoon, and how long the experiment will run. That way you do not decide it while looking at the results.
• Use comparison days. Alternate days with and days without, so that weather, stress or a long workday hit both sides equally.
• Repeat. A single evening is a one-off event. Over several weeks you see how much your well-being varies anyway.
• Take expectation out of the picture. Where you can arrange it, someone else decides which day gets what, and you only find out at the end.
• Write it down. A short note in the morning, right next to the alarm clock on your nightstand, is more accurate than your memory a few weeks later.

A vague impression becomes an observation you can actually work with.

How can you tell a study is solid?

A solid study is one that protects its result against chance, expectation and bias. Research has fixed building blocks for that, and you can spot each of them while reading.

Building block	What it protects against	How to recognize it
Defined model	an unclear subject	the cell line, animal species or group of people is named
Control group	mistaking the starting point for an effect	a second arm without the thing being tested
Blinding	expectations of participants and investigators	no one knows during the measurement who received what
Independent repeats	lucky hits	the number of experimental runs is stated, for example n = 3
Significance test	overrating small differences	a p-value or confidence interval appears with the result

The 2015 replication study by the Open Science Collaboration shows how much repetition matters. A network of 270 researchers repeated 100 published experiments from three psychology journals, with larger samples and, where possible, with the original materials.

“Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results.”

>

Source: Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349(6251), 2015, aac4716, DOI 10.1126/science.aac4716

On average, the repeated effects were half the size of the original ones. John Ioannidis had derived this situation ten years earlier. He concluded that for most study designs, a research claim is more likely to be false than true (DOI 10.1371/journal.pmed.0020124). Research lives on repetition, and a finding gains weight when it shows up again in further experiments.

Which claims can you check for yourself?

A verifiable claim is a statement that names its measured quantity, its measuring conditions and its source along with it. Then someone else can recalculate it, without any help from the sender.

A good example is in your pocket. For cell phones, Germany's Federal Office for Radiation Protection (BfS) cites a maximum specific absorption rate (SAR) of 2 watts per kilogram. It is averaged over six minutes of exposure and over 10 grams of tissue, about a third of an ounce (Federal Office for Radiation Protection, SAR values of cell phones). The manufacturer stores the measured value and test distance for your own device in the phone itself. On an iPhone you find them in Settings, under the legal and regulatory information. The figure comes from the manufacturer, and you can still work with it well, because the quantity, the condition and the test standard are all out in the open.

These questions help you place any claim about an effect, including one from your own experiment:

1. What was measured, and how often?
2. What was it compared with?
3. How large is the spread, and is there a p-value or confidence interval next to it?

If one question stays open, the claim is a hypothesis, and that makes it a good starting point for the next measurement. Our article on structured water walks through what that looks like for a popular term.

Why Qi Blanco writes about testing claims

Qi Blanco is a brand of jewelry and everyday objects. Our subjects are water, radiation and the body, and around all three, people ask us the same thing: how would I know? We follow the research on these subjects closely and keep track of what has been measured and what is still open. The same three questions about measurement, comparison and spread are the ones we put to every study we cite.

The original papers we cite on our own subjects are collected in full on our studies page. For the physics behind water and order, read Does Water Have Memory? What Physics Has Actually Measured.

Frequently asked questions

Does a successful self-experiment prove nothing at all?

It shows you that the result occurred on that one occasion. Whether an effect, chance or expectation lies behind it, it cannot separate on its own, because it lacks a comparison and repeats. As a starting point it is perfectly useful, and with comparison days and a few repeats it becomes a small study.

Why is a before-and-after comparison not enough?

Between before and after, more changes than the thing being tested: sleep, weather, how you feel that day and what you expect from the result. A before-and-after comparison measures the difference between two points in time. A comparison of two conditions says more, for example alternating days with and days without.

What does p ≤ 0.01 mean in a study?

The p-value states how likely a difference at least this large would be if there were in truth no difference. At p ≤ 0.01, that probability is at most 1 percent. How large the difference is shows in the measured values themselves, not in the p-value.

What does a study on cells tell you about everyday life?

A cell study, also called an in vitro study, is an experiment on cells in culture, outside a living body, under precisely defined conditions. It shows whether an effect occurs at all under controlled conditions, and that is where research on a question begins. How something works for you in everyday life is best found out with an experiment of your own, done well.

How can I tell whether a claim about an effect holds up?

Check whether a comparison condition is named, how often it was measured, and whether a spread is given next to the average. When these details are there, you can judge the claim well for yourself.

Key takeaways on self-experiments

• A self-experiment with n = 1 shows a one-off event and provides a hypothesis for the next measurement.
• Comparison days, repeats and blinding make even an experiment at home meaningful.
• Only the control condition separates an effect from chance in the numbers.
• 46 blinded provocation studies with 1175 participants show how strongly expectation shapes how people feel (the nocebo effect).
• Of 100 repeated experiments, 36 percent stayed statistically significant, compared with 97 percent of the original studies.
• What was measured, what was it compared with, and how large is the spread: these questions let you place any claim about an effect.

Sources and further reading

• Hróbjartsson A, Gøtzsche PC, Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment, New England Journal of Medicine 344(21), 2001, 1594–1602. https://doi.org/10.1056/NEJM200105243442106 · Record: https://pubmed.ncbi.nlm.nih.gov/11372012/
• Rubin GJ, Nieto-Hernandez R, Wessely S, Idiopathic environmental intolerance attributed to electromagnetic fields: An updated systematic review of provocation studies, Bioelectromagnetics 31(1), 2010, 1–11. https://doi.org/10.1002/bem.20536 · Record: https://pubmed.ncbi.nlm.nih.gov/19681059/
• Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349(6251), 2015, aac4716. https://doi.org/10.1126/science.aac4716 · Record: https://pubmed.ncbi.nlm.nih.gov/26315443/
• Ioannidis JPA, Why Most Published Research Findings Are False, PLoS Medicine 2(8), 2005, e124. https://doi.org/10.1371/journal.pmed.0020124
• Federal Office for Radiation Protection (Bundesamt für Strahlenschutz, BfS), SAR values of cell phones (German). https://www.bfs.de/DE/themen/emf/mobilfunk/vorsorge/sar-handy/sar-handy_node.html