Large-scale automated synthesis of human functional neuroimaging data

Tal Yarkoni, Russell A Poldrack, Thomas E Nichols, David C Van Essen, Tor D Wager
2011 Nature Methods  
The rapid growth of the literature on neuroimaging in humans has led to major advances in our understanding of human brain function but has also made it increasingly difficult to aggregate and synthesize neuroimaging findings. Here we describe and validate an automated brain-mapping framework that uses text-mining, meta-analysis and machine-learning techniques to generate a large database of mappings between neural and cognitive states. We show that our approach can be used to automatically
more » ... uct large-scale, high-quality neuroimaging meta-analyses, address long-standing inferential problems in the neuroimaging literature and support accurate 'decoding' of broad cognitive states from brain activity in both entire studies and individual human subjects. Collectively, our results have validated a powerful and generative framework for synthesizing human neuroimaging data on an unprecedented scale. The development of noninvasive neuroimaging techniques such as functional magnetic resonance imaging (fMRI) has spurred rapid growth of literature on human brain imaging in recent years. In 2010 alone, more than 1,000 fMRI articles had been published 1 . This proliferation has led to substantial advances in our understanding of the human brain and cognitive function; however, it has also introduced important challenges. In place of too little data, researchers are now besieged with too much. Because individual neuroimaging studies are often underpowered and have relatively high false positive rates 2-4 , multiple studies are required to achieve consensus regarding even broad relationships between brain and cognitive function. It is therefore necessary to develop new techniques for the large-scale aggregation and synthesis of human neuroimaging data 4-6 . Here we describe and validate a new framework for brain mapping, NeuroSynth, that takes an instrumental step toward automated large-scale synthesis of the neuroimaging literature. NeuroSynth combines text-mining, meta-analysis and machinelearning techniques to generate probabilistic mappings between cognitive and neural states that can be used for a broad range of neuroimaging applications. Whereas previous approaches have relied heavily on researchers' manual efforts (for example, refs. 7,8), which limits the scope and efficiency of resulting analyses 1 , our framework is fully automated and allows rapid and scalable synthesis of the neuroimaging literature. We show that this framework can be used to generate large-scale meta-analyses for hundreds of broad psychological concepts; support quantitative inferences about the consistency and specificity with which different cognitive processes elicit regional changes in brain activity; and decode and classify broad cognitive states in new data solely on the basis of observed brain activity. RESULTS Overview Our methodological approach includes several steps (Fig. 1a) . First, we used text-mining techniques to identify neuroimaging studies that used specific terms of interest (for example, 'pain' , 'emotion' , 'working memory' and so on) at a high frequency (>1 in 1,000 words) in the article text. Second, we automatically extracted activation coordinates from all tables reported in these studies. This approach produced a large database of term-to-coordinate mappings; here we report results based on 100,953 activation foci drawn from 3,489 neuroimaging studies published in 17 journals (Online Methods). Third, we conducted automated meta-analyses of hundreds of psychological concepts, producing an extensive set of whole-brain images that quantified relationships between brain activity and cognition (Fig. 1b) . Finally, we used a machine-learning technique (naive Bayes classification) to estimate the likelihood that new activation maps were associated with specific psychological terms, which allowed relatively open-ended decoding of psychological constructs from patterns of brain activity (Fig. 1c) . Automated coordinate extraction Our approach differs from previous work in its heavy reliance on automatically extracted information, raising several potential concerns about data quality. For example, the software might incorrectly classify noncoordinate information in a table as an activation focus (a false positive); different articles report foci in different stereotactic spaces, resulting in potential discrepancies between anatomical locations represented by the same set of coordinates; and the software did not discriminate activations from deactivations. ARTICLES To assess the effect of these issues on data quality, we conducted supporting analyses (Supplementary Note). First, we compared automatically extracted coordinates with a reference set of manually entered foci in the Surface Management System Database (SumsDB) 7,9 , and found high rates of sensitivity (84%) and specificity (97%). Second, we quantified the proportion of activation increases versus decreases reported in the neuroimaging literature. We found that decreases constituted a small proportion of results and had minimal effect on our results. Third, we developed a preliminary algorithm (based on ref. 10) to automatically detect and correct for between-study differences in stereotactic space (Supplementary Fig. 1) . Although automated extraction missed a minority of valid coordinates, and work remains to be done to increase the specificity of the extracted information, most coordinates were extracted accurately and several factors of a priori concern had relatively small influences on the results. Large-scale automated meta-analysis We used the database of automatically extracted activation coordinates to conduct a comprehensive set of automated meta-analyses for several hundred terms of interest. For each term, we identified all studies that used the term at high frequency anywhere in the article text 11 and submitted all associated activation foci to a metaanalysis. This approach generated whole-brain maps that showed the strength of association between each term and every location in the brain, enabling us to make multiple kinds of quantitative inference (for example, if the term 'language' had been used in a study, how likely was the study to report activation in Broca's area? If activation had been observed in the amygdala, what was the probability that the study frequently used the term 'fear'?). To validate this automated approach, which rests on the assumption that simple word counts are a reasonable proxy for the substantive content of articles, we conducted several supporting analyses (Supplementary Note). First, we found that NeuroSynth accurately recaptured conventional boundaries between distinct anatomical regions by comparing lexically defined regions of interest to anatomically defined regions of interest (Supplementary Fig. 2) . Second, we used NeuroSynth to replicate previous findings of visual category-specific activation X Y Z | a b c Figure 1 | Schematic overview of NeuroSynth framework and applications. (a) Outline of the NeuroSynth approach. The full text of a large corpus of articles is retrieved and terms of scientific interest are stored in a database. Articles are retrieved from the database on the basis of a user-entered search string (for example, 'pain') and peak coordinates from the associated articles are extracted from tables. A meta-analysis of the peak coordinates is automatically performed, producing a whole-brain map of the posterior probability of the term given activation at each voxel (P(pain|activation)). (b) Outlines of forward and reverse inference in brain imaging. Given a known psychological manipulation, one can quantify the corresponding changes in brain activity and generate a forward inference, but given an observed pattern of activity, drawing a reverse inference about associated cognitive states is more difficult because multiple cognitive states could have similar neural signatures. (c) Given meta-analytic posterior probability maps for multiple terms (for example, working memory, emotion and pain), one can classify a new activation map by identifying the class with the highest probability, P, given the new data (in this example, pain). Figure 2 | Comparison of previous meta-analysis results with forward and reverse inference maps produced automatically using the NeuroSynth framework. (a) Meta-analytic maps produced manually in previous studies 14-16 . (b) Automatically generated forward inference maps showing the probability of activation given the presence of the term (P(act.|term)). (c) Automatically generated reverse inference maps showing the probability of the term given observed activation (P(term|act.)). Metaanalyses were carried out for working memory (top), emotion (middle) and physical pain (bottom) and mapped to the PALS-B12 atlas 30 . Regions in b were consistently associated with the term and regions in c were selectively associated with the term. To account for base differences in term frequencies, reverse inference maps assumed uniform priors (equal 50% probabilities of 'term' and 'no term'). Activation in orange or red regions implies a high probability that a term is present, and activation in blue regions implies a high probability that a term is not present. Values for all images are shown only for regions that survived a test of association between term and activation, with a whole-brain correction for multiple comparisons (false discovery rate was 0.05). DLPFC, dorsolateral prefrontal cortex; DACC, dorsal anterior cingulate cortex; AI, anterior insula. Forward inference Reverse inference DLPFC a b c Working memory DACC Emotion Pain 0 0.4 0.1 0.9 P(act.|term) P(term|act.) Al ARTICLES in regions such as the fusiform face area 12 and visual word form area 13 (Supplementary Fig. 3) . Third, we found that more conservative meta-analyses in which the lexical search space had been restricted to article titles yielded similar, but less sensitive, metaanalysis results (Supplementary Fig. 4) . Finally, we compared our results with those produced by previous manual approaches. Comparison of automated metaanalyses of three broad psychological terms ('working memory' , 'emotion' and 'pain') with previously published meta-or mega-analytic maps 14-16 revealed marked qualitative (Fig. 2) and quantitative convergence (Supplementary Fig. 5 ) between approaches. To directly test the convergence of automated and manual approaches when applied to similar data, we manually validated 265 automatically extracted pain studies and performed a standard multilevel kernel density analysis 15 to compare experimental pain stimulation with baseline (66 valid studies). There was a notable overlap between automated and manual results (correlation across voxels, 0.84; Supplementary Fig. 6) . These results showed that, at least for broad domains, an automated meta-analysis approach generated results that were comparable in sensitivity and scope to those produced with more effort in previous studies. Quantitative reverse inference The relatively comprehensive nature of the NeuroSynth database enabled us to address a long-standing inferential problem in the neuroimaging literature, namely how to quantitatively identify cognitive states from patterns of observed brain activity. This problem of 'reverse inference' 17 arises because most neuroimaging studies are designed to identify neural changes that result from known psychological manipulations and not to determine what cognitive state(s) a given pattern of activity implies 17 (Fig. 1b) . For instance, fear consistently activates the human amygdala, but this does not imply that people in whom the amygdala is activated must be experiencing fear because other affective and nonaffective states have also been reported to activate the amygdala 4,18 . True reverse inference requires knowledge of which brain regions and networks are selectively, and not just consistently, associated with particular cognitive states 15,17 . Because the NeuroSynth database contains a broad set of termto-activation mappings, our framework is well suited for drawing quantitative inferences about mind-brain relationships in both the forward and reverse directions. We could quantify both the probability that there would be activation in specific brain regions given the presence of a particular term (P(activation|term) or 'forward inference'), and the probability that a term would occur in an article given the presence of activation in a particular brain region (P(term|activation) or reverse inference). Comparison of these two analyses allowed us to assess the validity of many common inferences about the relationship between neural and cognitive states. For illustration, we focused on the sample domains of working memory, emotion and pain, which are of substantial basic and clinical interest and have been extensively studied using fMRI (for additional examples, see Supplementary Fig. 7) . These domains are excellent candidates for quantitative reverse inference, as they are thought to have confusable neural correlates, with common activation of regions such as the dorsal anterior cingulate cortex (DACC) 19 and anterior insula. Our results showed differences between the forward and reverse inference maps in all three domains (Fig. 2) . For working memory, the forward inference map revealed the most consistent associations in the dorsolateral prefrontal cortex, anterior insula and dorsal medial frontal cortex, replicating previous findings 15,20 . However, the reverse inference map instead implicated the anterior prefrontal cortex and posterior parietal cortex as the regions that were most selectively activated by working memory tasks. We observed a similar pattern for pain and emotion. In both domains, frontal regions that have been broadly implicated in 0.3
doi:10.1038/nmeth.1635 pmid:21706013 pmcid:PMC3146590 fatcat:bq3tzv4hjbbhdn62aldujs52oa