As a double Ph.D. in neuroscience and cognitive science with an active research program, I am in a position to give some useful advice on how to evaluate the ‘evidence’ behind different brain training apps and interventions.
Scientific ‘Evidence’ – What Is It?
When brain training providers claim their products are ‘backed by science’ this sounds credible, but ‘scientific evidence’ comes in many guises, including:
- ‘In house’ research
- Conference talks, theses, papers ‘under review’ and other non-published material
- Single peer-reviewed journal articles that have not been replicated
- Meta-studies of multiple peer-reviewed journal articles with replications
It turns out that only the last of these – independent meta review studies – is a truly reliable guide to the effectiveness of a brain training intervention. Let us look at each of these in turn.
In house research
This is research on interventions conducted by the very company selling or promoting the product. There are two problems with this kind of research. 1. It is not independent, but there is a conflict of interest. It comes with a built-in bias to find support for the product. 2. There is no external check for the truth or validity of the claims made. It is not peer-reviewed by other scientists. This leaves more room for data ‘manipulation’ – a well-known problem in science.
Conference talks, Theses, etc
We have the same kind of problem with conference talks, Masters/Ph.D theses, etc. They are not peer-reviewed to check for scientific rigor and validity, even if they are independent. As with in-house research you often find this kind of ‘evidence’ with popular brain training providers.[space]
Single, Peer Reviewed Journal Articles
Published, peer-reviewed articles are a better source of evidence. Peer-review means that independent experts have evaluated the paper and consider it worthy of publication in a particular scientific journal. However, single papers showing an effect are not conclusive in themselves. Why? Because many ‘effects’ shown in peer-reviewed scientific journals have been shown to vanish when other labs try to repeat the experiment. This has been dubbed the ‘crisis of replication’ in psychology. As this Nature commentary notes:
“Psychologists are going through a period of intense self-reflection regarding the reliability of research in their field, fuelled by recently uncovered cases of fraud, failed attempts to replicate classic results, and calls from prominent psychologists to replicate key results in disputed fields.”
Multiple Peer Reviewed Journal Articles & Meta-Reviews
So what kind of evidence can we trust? First we should look for replications – that is, repeats of the study in different labs, published in different journal articles. Second, we should trust systematic reviews and meta-studies rather than individual studies – reviews of multiple studies looking at the same effect. Popular brain training vendors generally do not provide this kind of evidence.
In addition to being firmly based in well-established scientific evidence, your brain training apps should be assessed on a personal level – through self-quantification.
Self-quantification means that using carefully designed technologies and protocols you track your own cognitive capacities with independent scientific measures. Brain training vendors should provide independent, scientifically valid measures of your cognition for before and after training.
These measures should be scientifically valid, and developed in the field of cognitive neuroscience. They should enable you to objectively track your cognitive gains across a full spectrum of cognitive abilities relating to memory, attention, and learning – as well as IQ.