Publications | Denizenslab

2025

Optimizing Language Model Embeddings to Voxel Activity Improves Brain Activity Predictions

Anuja Negi, Christine Tseng, Anwar O. Nunez-Elizalde, and 2 more authors

Sep 2025

ISSN: 2692-8205 Pages: 2025.09.18.676935 Section: New Results

Abs DOI

Recent studies have shown that contextual semantic embeddings from language models can accurately predict human brain activity during language processing. However, most studies use contextual embeddings with the same context length and model layer for all voxels, potentially overlooking meaningful variations across the brain. In this study, we investigate whether optimizing contextual embeddings for individual voxels improves their ability to predict brain activity during reading. We optimize embeddings for each voxel by selecting the best-predicting context length, model layer, or both. We perform this optimization with two different types of stimuli (isolated sentences and narratives), and quantify the performance gains of optimized embeddings over standard fixed embeddings. Our results show that voxel-specific optimization substantially improves the prediction accuracy of contextual semantic embeddings. These findings demonstrate that voxel-specific contextual tuning provides a more accurate and nuanced account of how the contextual semantic information is represented across the cortex.
Encoding models in functional magnetic resonance imaging: the Voxelwise Encoding Model framework

M. Oleggio Castello, Fatma Deniz, Tom Tour, and 1 more author

Sep 2025

Abs

One of the major goals of cognitive neuroscience is understanding how the brain represents information about its own internal states and about the external world. This goal can be addressed by creating encoding models that reveal the information represented explicitly in measured brain activity. Here we describe the Voxelwise Encoding Model (VEM) framework for creating encoding models with functional magnetic resonance imaging (fMRI) data. The VEM framework provides several key advantages over traditional neuroimaging approaches. First, the VEM framework is applicable to most experimental designs, from classic factorial designs to complex naturalistic experiments such as movie watching or video games. This flexibility enables researchers to study brain function across multiple domains with the same analytical approach. Second, hypotheses about functional representations are defined and tested quantitatively by extracting feature spaces from experimental stimuli or tasks. These feature spaces quantify specific types of information potentially represented in brain activity and can range from simple human-derived labels to complex features generated by deep neural networks. If a feature space can be used to linearly predict brain activity, the brain is considered to explicitly represent features within that feature space. This prediction approach provides a systematic way to test hypotheses about functional representations. Third, the VEM framework implements robust data science methods to improve model estimation and minimize false positive results. Encoding models are estimated on a training set and validated on an independent test set. Testing in an independent dataset provides direct evidence that experimental findings generalize beyond the dataset used for model estimation. Finally, voxelwise encoding models can be created in each participant’s native brain space without unnecessary information loss due to spatial averaging or template resampling required by conventional group analyses. This enables the VEM approach to reveal fine-grained functional organization in individual participants that might otherwise be ignored. In this article, we provide a comprehensive guide to the Voxelwise Encoding Model framework through all phases of research, from experimental design and data collection to the estimation and interpretation of voxelwise encoding models.
Brain-Informed Fine-Tuning for Improved Multilingual Understanding in Language Models

Anuja Negi, Subba Reddy Oota, Anwar Nunez-Elizalde, and 2 more authors

Jul 2025

ISSN: 2692-8205 Pages: 2025.07.07.662360 Section: New Results

Abs DOI

Recent studies have demonstrated that fine-tuning language models with brain data can improve their semantic understanding, although these findings have so far been limited to English. Interestingly, similar to the shared multilingual embedding space of pretrained multilingual language models, human studies provide strong evidence for a shared semantic system in bilingual individuals. Here, we investigate whether fine-tuning language models with bilingual brain data changes model representations in a way that improves them across multiple languages. To test this, we fine-tune monolingual and multilingual language models using brain activity recorded while bilingual participants read stories in English and Chinese. We then evaluate how well these representations generalize to the bilingual participants’ first language, their second language, and several other languages that the participant is not fluent in. We assess the fine-tuned language models on brain encoding performance and downstream NLP tasks. Our results show that bilingual brain-informed fine-tuned language models outperform their vanilla (pretrained) counterparts in both brain encoding performance and most downstream NLP tasks across multiple languages. These findings suggest that brain-informed fine-tuning improves multilingual understanding in language models, offering a bridge between cognitive neuroscience and NLP research.

2024

The cortical representation of language timescales is shared between reading and listening

Catherine Chen, Tom Tour, Jack L. Gallant, and 2 more authors

Communications Biology, Mar 2024

Publisher: Nature Publishing Group

Abs DOI

Language comprehension involves integrating low-level sensory inputs into a hierarchy of increasingly high-level features. Prior work studied brain representations of different levels of the language hierarchy, but has not determined whether these brain representations are shared between written and spoken language. To address this issue, we analyze fMRI BOLD data that were recorded while participants read and listened to the same narratives in each modality. Levels of the language hierarchy are operationalized as timescales, where each timescale refers to a set of spectral components of a language stimulus. Voxelwise encoding models are used to determine where different timescales are represented across the cerebral cortex, for each modality separately. These models reveal that between the two modalities timescale representations are organized similarly across the cortical surface. Our results suggest that, after low-level sensory processing, language integration proceeds similarly regardless of stimulus modality.
On managing large collections of scientific workflows

Nourhan Elfaramawy, Fatma Deniz, Lars Grunske, and 5 more authors

Mar 2024

Publication title: Modellierung 2024 satellite events

DOI
Speech language models lack important brain-relevant semantics

Subba Reddy Oota, Emin Çelik, Fatma Deniz, and 1 more author

In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024

Abs DOI

Despite known differences between reading and listening in the brain, recent work has shown that text-based language models predict both text-evoked and speech-evoked brain activity to an impressive degree. This poses the question of what types of information language models truly predict in the brain. We investigate this question via a direct approach, in which we systematically remove specific low-level stimulus features (textual, speech, and visual) from language model representations to assess their impact on alignment with fMRI brain recordings during reading and listening. Comparing these findings with speech-based language models reveals starkly different effects of low-level features on brain alignment. While text-based models show reduced alignment in early sensory regions post-removal, they retain significant predictive power in late language regions. In contrast, speech-based models maintain strong alignment in early auditory regions even after feature removal but lose all predictive power in late language regions. These results suggest that speech-based models provide insights into additional information processed by early auditory regions, but caution is needed when using them to model processing in late language regions. We make our code publicly available. [https://github.com/subbareddy248/speech-llm-brain]
Bilingual language processing relies on shared semantic representations that are modulated by each language

Catherine Chen, Xue L. Gong, Christine Tseng, and 3 more authors

Jun 2024

Pages: 2024.06.24.600505 Section: New Results

Abs DOI

Billions of people throughout the world are bilingual and can understand semantic concepts in multiple languages. However, there is little agreement about how the brains of bilinguals represent semantic information from different languages. Some theories suggest that bilingual speakers’ brains contain separate representations for semantic information from different languages, while others suggest that different languages evoke the same semantic representations in the brain. To determine how the brains of bilinguals represent semantic information from different languages, we used functional magnetic resonance imaging (fMRI) to record brain responses while participants who are fluent in both English and Chinese read several hours of natural narratives in each language. We then used this data to specifically and comprehensively compare semantic representations between the two languages. We show that while semantic representations are largely shared between languages, these representations undergo fine-grained shifts between languages. These shifts systematically alter how different concept categories are represented in each language. Our results suggest that for bilinguals, semantic brain representations are shared across languages but modulated by each language. These results reconcile competing theories of bilingual language processing.

2023

Phonemic segmentation of narrative speech in human cerebral cortex

Xue L. Gong, Alexander G. Huth, Fatma Deniz, and 3 more authors

Nature Communications, Jul 2023

Publisher: Nature Publishing Group

Abs DOI

Speech processing requires extracting meaning from acoustic patterns using a set of intermediate representations based on a dynamic segmentation of the speech stream. Using whole brain mapping obtained in fMRI, we investigate the locus of cortical phonemic processing not only for single phonemes but also for short combinations made of diphones and triphones. We find that phonemic processing areas are much larger than previously described: they include not only the classical areas in the dorsal superior temporal gyrus but also a larger region in the lateral temporal cortex where diphone features are best represented. These identified phonemic regions overlap with the lexical retrieval region, but we show that short word retrieval is not sufficient to explain the observed responses to diphones. Behavioral studies have shown that phonemic processing and lexical retrieval are intertwined. Here, we also have identified candidate regions within the speech cortical network where this joint processing occurs.
Semantic Representations during Language Comprehension Are Affected by Context

Fatma Deniz, Christine Tseng, Leila Wehbe, and 2 more authors

Journal of Neuroscience, Apr 2023

Publisher: Society for Neuroscience Section: Research Articles

Abs DOI

The meaning of words in natural language depends crucially on context. However, most neuroimaging studies of word meaning use isolated words and isolated sentences with little context. Because the brain may process natural language differently from how it processes simplified stimuli, there is a pressing need to determine whether prior results on word meaning generalize to natural language. fMRI was used to record human brain activity while four subjects (two female) read words in four conditions that vary in context: narratives, isolated sentences, blocks of semantically similar words, and isolated words. We then compared the signal-to-noise ratio (SNR) of evoked brain responses, and we used a voxelwise encoding modeling approach to compare the representation of semantic information across the four conditions. We find four consistent effects of varying context. First, stimuli with more context evoke brain responses with higher SNR across bilateral visual, temporal, parietal, and prefrontal cortices compared with stimuli with little context. Second, increasing context increases the representation of semantic information across bilateral temporal, parietal, and prefrontal cortices at the group level. In individual subjects, only natural language stimuli consistently evoke widespread representation of semantic information. Third, context affects voxel semantic tuning. Finally, models estimated using stimuli with little context do not generalize well to natural language. These results show that context has large effects on the quality of neuroimaging data and on the representation of meaning in the brain. Thus, neuroimaging studies that use stimuli with little context may not generalize well to the natural regime. SIGNIFICANCE STATEMENT Context is an important part of understanding the meaning of natural language, but most neuroimaging studies of meaning use isolated words and isolated sentences with little context. Here, we examined whether the results of neuroimaging studies that use out-of-context stimuli generalize to natural language. We find that increasing context improves the quality of neuro-imaging data and changes where and how semantic information is represented in the brain. These results suggest that findings from studies using out-of-context stimuli may not generalize to natural language used in daily life.

2022

Attention weights accurately predict language representations in the brain

Mathis Lamarre, Catherine Chen, and Fatma Deniz

In Findings of the Association for Computational Linguistics: EMNLP 2022, Dec 2022

Abs DOI

In Transformer-based language models (LMs) the attention mechanism converts token embeddings into contextual embeddings that incorporate information from neighboring words. The resulting contextual hidden state embeddings have enabled highly accurate models of brain responses, suggesting that the attention mechanism constructs contextual embeddings that carry information reflected in language-related brain representations. However, it is unclear whether the attention weights that are used to integrate information across words are themselves related to language representations in the brain. To address this question we analyzed functional magnetic resonance imaging (fMRI) recordings of participants reading English language narratives. We provided the narrative text as input to two LMs (BERT and GPT-2) and extracted their corresponding attention weights. We then used encoding models to determine how well attention weights can predict recorded brain responses. We find that attention weights accurately predict brain responses in much of the frontal and temporal cortices. Our results suggest that the attention mechanism itself carries information that is reflected in brain representations. Moreover, these results indicate cortical areas in which context integration may occur.

2021

Visual and linguistic semantic representations are aligned at the border of human visual cortex

Sara F. Popham, Alexander G. Huth, Natalia Y. Bilenko, and 4 more authors

Nature Neuroscience, Nov 2021

Publisher: Nature Publishing Group

Abs DOI

Semantic information in the human brain is organized into multiple networks, but the fine-grain relationships between them are poorly understood. In this study, we compared semantic maps obtained from two functional magnetic resonance imaging experiments in the same participants: one that used silent movies as stimuli and another that used narrative stories. Movies evoked activity from a network of modality-specific, semantically selective areas in visual cortex. Stories evoked activity from another network of semantically selective areas immediately anterior to visual cortex. Remarkably, the pattern of semantic selectivity in these two distinct networks corresponded along the boundary of visual cortex: for visual categories represented posterior to the boundary, the same categories were represented linguistically on the anterior side. These results suggest that these two networks are smoothly joined to form one contiguous map.
Non-Complementarity of Information in Word-Embedding and Brain Representations in Distinguishing between Concrete and Abstract Words

Kalyan Ramakrishnan and Fatma Deniz

In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, Jun 2021

Abs DOI

Word concreteness and imageability have proven crucial in understanding how humans process and represent language in the brain. While word-embeddings do not explicitly incorporate the concreteness of words into their computations, they have been shown to accurately predict human judgments of concreteness and imageability. Inspired by the recent interest in using neural activity patterns to analyze distributed meaning representations, we first show that brain responses acquired while human subjects passively comprehend natural stories can significantly distinguish the concreteness levels of the words encountered. We then examine for the same task whether the additional perceptual information in the brain representations can complement the contextual information in the word-embeddings. However, the results of our predictive models and residual analyses indicate the contrary. We find that the relevant information in the brain representations is a subset of the relevant information in the contextualized word-embeddings, providing new insight into the existing state of natural language processing models.

2019

The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality

Fatma Deniz, Anwar O. Nunez-Elizalde, Alexander G. Huth, and 1 more author

Journal of Neuroscience, Sep 2019

Publisher: Society for Neuroscience Section: Research Articles

Abs DOI

An integral part of human language is the capacity to extract meaning from spoken and written words, but the precise relationship between brain representations of information perceived by listening versus reading is unclear. Prior neuroimaging studies have shown that semantic information in spoken language is represented in multiple regions in the human cerebral cortex, while amodal semantic information appears to be represented in a few broad brain regions. However, previous studies were too insensitive to determine whether semantic representations were shared at a fine level of detail rather than merely at a coarse scale. We used fMRI to record brain activity in two separate experiments while participants listened to or read several hours of the same narrative stories, and then created voxelwise encoding models to characterize semantic selectivity in each voxel and in each individual participant. We find that semantic tuning during listening and reading are highly correlated in most semantically selective regions of cortex, and models estimated using one modality accurately predict voxel responses in the other modality. These results suggest that the representation of language semantics is independent of the sensory modality through which the semantic information is received. SIGNIFICANCE STATEMENT Humans can comprehend the meaning of words from both spoken and written language. It is therefore important to understand the relationship between the brain representations of spoken or written text. Here, we show that although the representation of semantic information in the human brain is quite complex, the semantic representations evoked by listening versus reading are almost identical. These results suggest that the representation of language semantics is independent of the sensory modality through which the semantic information is received.

2018

The unified maximum a posteriori (MAP) framework for neuronal system identification

Michael C.-K. Wu, Fatma Deniz, Ryan J. Prenger, and 1 more author

Nov 2018

arXiv:1811.01043 [q-bio]

Abs DOI

The functional relationship between an input and a sensory neuron’s response can be described by the neuron’s stimulus-response mapping function. A general approach for characterizing the stimulus-response mapping function is called system identification. Many different names have been used for the stimulus-response mapping function: kernel or transfer function, transducer, spatiotemporal receptive field. Many algorithms have been developed to estimate a neuron’s mapping function from an ensemble of stimulus-response pairs. These include the spike-triggered average, normalized reverse correlation, linearized reverse correlation, ridge regression, local spectral reverse correlation, spike-triggered covariance, artificial neural networks, maximally informative dimensions, kernel regression, boosting, and models based on leaky integrate-and-fire neurons. Because many of these system identification algorithms were developed in other disciplines, they seem very different superficially and bear little relationship with each other. Each algorithm makes different assumptions about the neuron and how the data is generated. Without a unified framework it is difficult to select the most suitable algorithm for estimating the neuron’s mapping function. In this review, we present a unified framework for describing these algorithms called maximum a posteriori estimation (MAP). In the MAP framework, the implicit assumptions built into any system identification algorithm are made explicit in three MAP constituents: model class, noise distributions, and priors. Understanding the interplay between these three MAP constituents will simplify the task of selecting the most appropriate algorithms for a given data set. The MAP framework can also facilitate the development of novel system identification algorithms by incorporating biophysically plausible assumptions and mechanisms into the MAP constituents.

2017

The Practice of Reproducible Research Case Studies and Lessons from the Data-Intensive Sciences

Nov 2017

Abs

Scholarship is a powerful tool for changing how people think, plan, and govern. By giving voice to bright minds and bold ideas, we seek to foster understanding and drive progressive change.
pyMooney: Generating a Database of Two-Tone, Mooney Images

F. Deniz

In The Practice of Reproducible Research: Case Studies in Data Science, Nov 2017
Introducing the Case Studies.

D. Turek and F. Deniz

In The Practice of Reproducible Research: Case Studies in Data Science, Nov 2017
Towards Implicit Visual Memory-Based Authentication

Nov 2017

Abs

Author(s): Claude Castelluccia, Markus Dürmuth, Maximilian Golla, Fatma Deniz Download: Paper (PDF) Date: 27 Feb 2017 Document Type: Reports Additional Documents: Slides Video Associated Event: NDSS Symposium 2017 Abstract: Selecting and remembering secure passwords puts a high cognitive burden on the user, which has adverse effects on usability and security. Authentication schemes based on implicit memory can relieve … Continued
Portable Learning Environments for Hands-On Computational Instruction: Using Container- and Cloud-Based Technology to Teach Data Science

Chris Holdgraf, Aaron Culich, Ariel Rokem, and 3 more authors

In Practice and Experience in Advanced Research Computing 2017: Sustainability, Success and Impact, Jul 2017

Abs DOI

There is an increasing interest in learning outside of the traditional classroom setting. This is especially true for topics covering computational tools and data science, as both are challenging to incorporate in the standard curriculum. These atypical learning environments offer new opportunities for teaching, particularly when it comes to combining conceptual knowledge with hands-on experience/expertise with methods and skills. Advances in cloud computing and containerized environments provide an attractive opportunity to improve the efficiency and ease with which students can learn. This manuscript details recent advances towards using commonly-available cloud computing services and advanced cyberinfrastructure support for improving the learning experience in bootcamp-style events. We cover the benefits (and challenges) of using a server hosted remotely instead of relying on student laptops, discuss the technology that was used in order to make this possible, and give suggestions for how others could implement and improve upon this model for pedagogy and reproducibility.

2016

Generation and the subjective feeling of “aha!” are independently related to learning from insight

Jasmin M. Kizilirmak, Joana Silva, Fatma Imamoglu, and 1 more author

Psychological Research, Jul 2016

Abs DOI

It has been proposed that sudden insight into the solutions of problems can enhance long-term memory for those solutions. However, the nature of insight has been operationalized differently across studies. Here, we examined two main aspects of insight problem-solving—the generation of a solution and the subjective “aha!” experience—and experimentally evaluated their respective relationships to long-term memory formation (encoding). Our results suggest that generation (generated solution vs. presented solution) and the “aha!” experience (“aha!” vs. no “aha!”) are independently related to learning from insight, as well as to the emotional response towards understanding the solution during encoding. Moreover, we analyzed the relationship between generation and the “aha!” experience and two different kinds of later memory tests, direct (intentional) and indirect (incidental). Here, we found that the generation effect was larger for indirect testing, reflecting more automatic retrieval processes, while the relationship with the occurrence of an “aha!” experience was somewhat larger for direct testing. Our results suggest that both the generation of a solution and the subjective experience of “aha!” indicate processes that benefit long-term memory formation, though differently. This beneficial effect is possibly due to the intrinsic reward associated with sudden comprehension and the detection of schema-consistency, i.e., that novel information can be easily integrated into existing knowledge.

2014

Learning from Neuroscience to Improve Internet Security

Jul 2014
Activity in high-level brain regions reflects visibility of low-level stimuli

F. Imamoglu, J. Heinzle, A. Imfeld, and 1 more author

NeuroImage, Nov 2014

Abs DOI

Stimulus visibility is associated with neural signals in multiple brain regions, ranging from visual cortex to prefrontal regions. Here we used functional magnetic resonance imaging (fMRI) to investigate to which extent the perceived visibility of a "low-level" grating stimulus is reflected in the brain activity in high-level brain regions. Oriented grating stimuli were presented under varying visibility conditions created by backward masking. Visibility was manipulated using four different stimulus onset asynchronies (SOAs), which created a continuum from invisible to highly visible target stimuli. Brain activity in early visual areas, high-level visual brain regions (fusiform gyrus), as well as parietal and prefrontal brain regions was significantly correlated with subjects’ psychometric visibility functions. In addition, increased stimulus visibility was reflected in the functional coupling between low and high-level visual areas. Specifically, neuroimaging signals in the middle occipital gyrus were significantly more correlated with signals in the inferior temporal gyrus when subjects successfully perceived the target stimulus than when they did not. These results provide evidence that not only low-level visual but also high-level brain regions reflect visibility of low-level grating stimuli and that changes in functional connectivity reflect perceived stimulus visibility.

2012

Changes in functional connectivity support conscious object recognition

Fatma Imamoglu, Thorsten Kahnt, Christof Koch, and 1 more author

NeuroImage, Dec 2012

Abs DOI

What are the brain mechanisms that mediate conscious object recognition? To investigate this question, it is essential to distinguish between brain processes that cause conscious recognition of a stimulus from other correlates of its sensory processing. Previous fMRI studies have identified large-scale brain activity ranging from striate to high-level sensory and prefrontal regions associated with conscious visual perception or recognition. However, the possible role of changes in connectivity during conscious perception between these regions has only rarely been studied. Here, we used fMRI and connectivity analyses, together with 120 custom-generated, two-tone, Mooney images to directly assess whether conscious recognition of an object is accompanied by a dynamical change in the functional coupling between extrastriate cortex and prefrontal areas. We compared recognizing an object versus not recognizing it in 19 naïve subjects using two different response modalities. We find that connectivity between the extrastriate cortex and the dorsolateral prefrontal cortex (DLPFC) increases when objects are consciously recognized. This interaction was independent of the response modality used to report conscious recognition. Furthermore, computing the difference in Granger causality between recognized and not recognized conditions reveals stronger feedforward connectivity than feedback connectivity when subjects recognized the objects. We suggest that frontal and visual brain regions are part of a functional network that supports conscious object recognition by changes in functional connectivity.