| Temporal Commonsense Reasoning |
| ❏ ALICE Model |
| Overview |
| State-of-the-art natural language processing (NLP) models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely high complexity of pre-trained models, aggressive fine-tuning often causes the fine-tuned model to overfit the training data of downstream tasks and fail to perform well on unseen data, and also on domain shift and adversarial scenarios. In this research, we aim to leverage these issues and explore how to improve model generalization and robustness of pre-trained language models (e.g., BERT) on downstream NLP tasks by adopting adversarial training. |
| Slides |
| ❏ An Approach to Building a General-Purpose Language Model for Understanding Temporal Commonsense |
| Overview |
| The ability to capture commonsense temporal relations for time-related events expressed in sentences is a very important task in natural language understanding. On the other hand, pre-trained language models such as BERT, which have achieved significant results in a wide range of natural language processing tasks in recent years, are still said to perform poorly in temporal inference. In this study, we focus on developing language models for temporal commonsense inference for several pre-trained language models. The models are created through multi-step fine-tuning using multiple corpora and masked language modeling to predict masked temporal indicators that are important for temporal commonsense inference, and the results show that the models significantly improve accuracy over standard fine-tuning in the temporal commonsense inference task. |
| Slides |
| Mayuko Kimura |
| 木村 麻友子，Kanashiro Pereira Lis，浅原 正幸，Cheng Fei，越智 綾子，小林 一郎「時間的常識理解へ向けた言語モデル構築への取り組み」人工知能学会全国大会（第36回），国立京都国際会館，京都，2022年6月．(in Japanese) |
| ❏ A Study on Building a Japanese General-Purpose Language Model for Temporal Commonsense Understanding |
| Overview |
| In order to understand events expressed in natural language, it is important to understand time. However, since they are often omitted from descriptions, it is necessary to have common sense knowledge about various temporal aspects of events. Therefore, we aim to build a Japanese general-purpose language model that can identify time-related common sense using the English dataset on temporal common sense, MC-TACO, which is translated into Japanese. Through experiments with fine-tuning by Masked Language Modeling at various mask settings and mask proportions and by modifying pre-trained models, we were able to construct a language model with improved accuracy for tasks in temporal common sense. We have confirmed that high accuracy can be obtained in Japanese with the same settings as in English during Masked Language Modeling. However, the accuracy in the Japanese experiment was lower than that in the English experiment, and it was also found that the task in Japanese temporal common sense is difficult. Now, we are working toward using the language model we constructed for research on the analysis of the processing of temporal information in the human brain. |
| Slides |
