Transformer-based Korean Pretrained Language Models: A Survey on Three Years of Progress [article]

Kichang Yang
2021 arXiv   pre-print
With the advent of Transformer, which was used in translation models in 2017, attention-based architectures began to attract attention. Furthermore, after the emergence of BERT, which strengthened the NLU-specific encoder part, which is a part of the Transformer, and the GPT architecture, which strengthened the NLG-specific decoder part, various methodologies, data, and models for learning the Pretrained Language Model began to appear. Furthermore, in the past three years, various Pretrained
more » ... guage Models specialized for Korean have appeared. In this paper, we intend to numerically and qualitatively compare and analyze various Korean PLMs released to the public.
arXiv:2112.03014v1 fatcat:65ilb7fuijcmzb6coxaeqdsnp4