| Brain Analysis |
| ❏ Estimation of Brain Activity evoked Linguistic Stimuli utilizing the General-Purpose Language Model: BrainBERT |
| Overview |
| Currently, many researchers have used language models to achieve excellent results in various fields, such as understanding the semantics of text and extracting multimedia information like videos. Furthermore, many investigations have also been conducted to capture the generative correspondence between text and the brain. In this way, we constructed a model based on the correspondence between brain activity data and semantic representation by BERT, called BrainBERT. The BrainBERT was used to build an encoding model between text and brain activity states, and to estimate the brain activity. In brief, we have achieved the two primary achievements of this research. 1) We verified the superiority of the BrainBERT model for brain signal extraction compared to the other 20 popular language models. 2) Using visualization tools such as PyCortex, we visualized the correlation of brain activity data according to the regions of interest in the brain. |
| Slides |
| Ying Luo |
| 羅 桜，小林一郎「汎用言語モデルBrainBERTを用いた言語刺激下の脳内状態推定」人工知能学会全国大会（第36回），国立京都国際会館，京都，2022年6月．(in Japanese) |
| ❏ Towards Understanding the Hierarchical Processing of Visual and Linguistic Information in the Human Brain |
| Overview |
| In recent years, many studies have been conducted to elucidate the information processing mechanisms of the human brain using deep learning models. For example, studies using convolutional neural networks (CNNs) confirmed the hierarchical processing of visual information in the brain. Studies using language models also confirmed that it is possible to model the representation of semantic information in the brain. However, these two modalities are often studied separately, and their similarities and differences have not been adequately discussed. With this background, we have been investigating the similarities and differences between the two modalities by using deep learning models that handle visual and language information to predict brain activity and analyzing the predictions in terms of the modalities and hierarchy of the deep learning models. |
| Slides |
