Denizenslab

Recent studies have shown that contextual semantic embeddings from language models can accurately predict human brain activity during language processing. However, most studies use contextual embeddings with the same context length and model layer for all voxels, potentially overlooking meaningful variations across the brain. In this study, we investigate whether optimizing contextual embeddings for individual voxels improves their ability to predict brain activity during reading. We optimize embeddings for each voxel by selecting the best-predicting context length, model layer, or both. We perform this optimization with two different types of stimuli (isolated sentences and narratives), and quantify the performance gains of optimized embeddings over standard fixed embeddings. Our results show that voxel-specific optimization substantially improves the prediction accuracy of contextual semantic embeddings. These findings demonstrate that voxel-specific contextual tuning provides a more accurate and nuanced account of how the contextual semantic information is represented across the cortex.

One of the major goals of cognitive neuroscience is understanding how the brain represents information about its own internal states and about the external world. This goal can be addressed by creating encoding models that reveal the information represented explicitly in measured brain activity. Here we describe the Voxelwise Encoding Model (VEM) framework for creating encoding models with functional magnetic resonance imaging (fMRI) data. The VEM framework provides several key advantages over traditional neuroimaging approaches. First, the VEM framework is applicable to most experimental designs, from classic factorial designs to complex naturalistic experiments such as movie watching or video games. This flexibility enables researchers to study brain function across multiple domains with the same analytical approach. Second, hypotheses about functional representations are defined and tested quantitatively by extracting feature spaces from experimental stimuli or tasks. These feature spaces quantify specific types of information potentially represented in brain activity and can range from simple human-derived labels to complex features generated by deep neural networks. If a feature space can be used to linearly predict brain activity, the brain is considered to explicitly represent features within that feature space. This prediction approach provides a systematic way to test hypotheses about functional representations. Third, the VEM framework implements robust data science methods to improve model estimation and minimize false positive results. Encoding models are estimated on a training set and validated on an independent test set. Testing in an independent dataset provides direct evidence that experimental findings generalize beyond the dataset used for model estimation. Finally, voxelwise encoding models can be created in each participant’s native brain space without unnecessary information loss due to spatial averaging or template resampling required by conventional group analyses. This enables the VEM approach to reveal fine-grained functional organization in individual participants that might otherwise be ignored. In this article, we provide a comprehensive guide to the Voxelwise Encoding Model framework through all phases of research, from experimental design and data collection to the estimation and interpretation of voxelwise encoding models.

Language comprehension involves integrating low-level sensory inputs into a hierarchy of increasingly high-level features. Prior work studied brain representations of different levels of the language hierarchy, but has not determined whether these brain representations are shared between written and spoken language. To address this issue, we analyze fMRI BOLD data that were recorded while participants read and listened to the same narratives in each modality. Levels of the language hierarchy are operationalized as timescales, where each timescale refers to a set of spectral components of a language stimulus. Voxelwise encoding models are used to determine where different timescales are represented across the cerebral cortex, for each modality separately. These models reveal that between the two modalities timescale representations are organized similarly across the cortical surface. Our results suggest that, after low-level sensory processing, language integration proceeds similarly regardless of stimulus modality.