When research fails to replicate: the "brain drain" smartphone study

What we know

In 2017, a US research team (Ward and colleagues) published a paper called “Brain Drain”. Participants left their phone in one of three places: on the desk, in a pocket or bag, or in another room. Then they took tests of memory and reasoning. The finding: the closer their own phone was, the lower their scores on working memory and fluid intelligence tests. The gap was reported to be larger for people who depended more on their phones.

It was a simple, memorable story: your phone affects you even when you are not using it, just by being there. News articles and books repeated it widely.

In 2022, however, Ruiz Pardo and Minda in Canada published a repeat of the experiment. Their replication was preregistered: the methods and analysis plan were made public before any data were collected. Following Ward’s second experiment, they used six conditions that combined phone location (desk, pocket or bag, outside the room) with power (on or off), and they measured working memory and the ability to hold back a response.

The result: phone location made no difference to scores on either test. The authors concluded that the mere presence of one’s own smartphone may not be enough to affect cognitive performance.

How to read it

It is not unusual for a result from one study to disappear when other researchers repeat the same method.

In 2015, the journal Science published the results of a large project in which researchers around the world repeated 100 psychology studies. 97% of the original studies had statistically significant results. In the repeats, only 36% did. On average, the effects in the repeats were about half the size of the originals.

That does not mean research cannot be trusted. There are several reasons this happens:

• Chance. Small experiments sometimes show a difference by luck.
• Publication bias. Studies that find a difference are more likely to be published, while null results get filed away.
• Flexible analysis. If researchers can choose their analysis after seeing the data, it is easy to pick the one that shows a difference. Preregistration is designed to prevent this.

So a preregistered replication, or a meta-analysis that pools many studies, deserves more weight than a single study. In the smartphone case, we gave more weight to the preregistered replication, and we no longer use the original study on its own.

A failed replication also does not prove that phones have no effect on focus at all. What we can say is narrower: the strong claim that “just having it nearby dulls your mind” is not currently supported.

How UPSHIFT chooses evidence

The app and these articles follow a few rules:

1. Name the type of study. Only randomized experiments and natural experiments get wording like “doing X increased Y”. Observational data get “people who do X tend to Y”.
2. Name the country and data. We do not present a US study as if it had been shown in Japan or Korea. When there is no Japanese study, we say so.
3. Use only numbers found in the original source. We also check that each DOI is registered.
4. Do not use studies that failed to replicate. The “brain drain” study in this article is one example.
5. Treat results as averages. Research numbers describe group averages, not a forecast of your own future.

A small step for today

There are other things current research can say about phones. One experiment found that reading on a glowing screen before bed delayed sleep. Another found that people felt less stressed in a week when they limited how often they checked email.

• Log “put my phone down an hour before bed”. Keep your distance from your phone for the reason that has evidence behind it.
• When you see “studies show”, check three things. How many people, in which country? Was it an experiment, or just a comparison? Has anyone repeated it?

Sources

• [B5] Ward, Duke, Gneezy & Bos (2017) Brain Drain: The Mere Presence of One’s Own Smartphone Reduces Available Cognitive Capacity. Journal of the Association for Consumer Research.
• [B6] Ruiz Pardo & Minda (2022) Reexamining the “brain drain” effect: A replication of Ward et al. (2017). Acta Psychologica.
• [X-osc-2015] Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science.
• [B4] Chang et al. (2015) Evening use of light-emitting eReaders negatively affects sleep, circadian timing, and next-morning alertness. PNAS.
• [B7] Kushlev & Dunn (2015) Checking email less frequently reduces stress. Computers in Human Behavior.