Filters








49 Hits in 3.2 sec

TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue [article]

Chien-Sheng Wu, Steven Hoi, Richard Socher, Caiming Xiong
2020 arXiv   pre-print
Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task-oriented dialogue applications, including intention recognition, dialogue state tracking  ...  We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue.  ...  We collect and combine nine human-human and multi-turn task-oriented dialogue corpora to train a task-oriented dialogue BERT (TOD-BERT).  ... 
arXiv:2004.06871v3 fatcat:plfdyo7j3nfdhiqdrztyzbou3m

TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue

Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, Caiming Xiong
2020 Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)   unpublished
We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue.  ...  Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream taskoriented dialogue applications, including intention recognition, dialogue state tracking  ...  We collect and combine nine human-human and multi-turn task-oriented dialogue corpora to train a task-oriented dialogue BERT (TOD-BERT).  ... 
doi:10.18653/v1/2020.emnlp-main.66 fatcat:lekm4zgpzbeo7ln24su3xwemia

Probing Task-Oriented Dialogue Representation from Language Models [article]

Chien-Sheng Wu, Caiming Xiong
2020 arXiv   pre-print
This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks.  ...  the dialogue research community, 3) find insights of pre-training factors for dialogue application that may be the key to success.  ...  for task-oriented dialogue. • Pre-trained language models intrinsically contain more information about intents and dialogue acts but less for slots. • ConveRT (Henderson et al., 2019) and TOD-BERT-jnt  ... 
arXiv:2010.13912v1 fatcat:aib5nkug4bbevjbswmbwqpy3ym

Learning Dialogue Representations from Consecutive Utterances [article]

Zhihan Zhou, Dejiao Zhang, Wei Xiao, Nicholas Dingwall, Xiaofei Ma, Andrew O. Arnold, Bing Xiang
2022 arXiv   pre-print
Learning high-quality dialogue representations is essential for solving a variety of dialogue-oriented tasks, especially considering that dialogue systems often suffer from data scarcity.  ...  In this paper, we introduce Dialogue Sentence Embedding (DSE), a self-supervised contrastive learning method that learns effective dialogue representations suitable for a wide range of dialogue tasks.  ...  However, they did not release code or pre-trained models for comparison. Wu et al. (2020a) combines nine dialogue datasets to obtain a large and high-quality task-oriented dialogue corpus.  ... 
arXiv:2205.13568v2 fatcat:6yjq53blw5gsbhlqrathtdzmty

Integrating Pre-trained Model into Rule-based Dialogue Management [article]

Jun Quan, Meng Yang, Qiang Gan, Deyi Xiong, Yiming Liu, Yuchen Dong, Fangxin Ouyang, Jun Tian, Ruiling Deng, Yongzhi Li, Yang Yang, Daxin Jiang
2021 arXiv   pre-print
Rule-based dialogue management is still the most popular solution for industrial task-oriented dialogue systems for their interpretablility.  ...  Furthermore, we integrate pre-trained models and empower the DM with few-shot capability. The experimental results demonstrate the effectiveness and strong few-shot capability of our method.  ...  We would also like to thank the anonymous reviewers for their insightful comments.  ... 
arXiv:2102.08553v1 fatcat:lvkvovokhvdczakhgqyxpj2nzy

Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems [article]

Andrea Madotto, Zihan Liu, Zhaojiang Lin, Pascale Fung
2020 arXiv   pre-print
Task-oriented dialogue systems use four connected modules, namely, Natural Language Understanding (NLU), a Dialogue State Tracking (DST), Dialogue Policy (DP) and Natural Language Generation (NLG).  ...  The most common and effective technique to solve this problem is transfer learning, where large language models, either pre-trained on text or task-specific data, are fine-tuned on the few samples.  ...  Peng et al. (2020a) proposed a pre-trained language model (LM) for end-toend pipe-lined task-oriented dialogue systems.  ... 
arXiv:2008.06239v2 fatcat:ejzu3qeq55bijl3alqntkqyaoa

Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems [article]

Fei Mi, Wanhao Zhou, Fengyu Cai, Lingjing Kong, Minlie Huang, Boi Faltings
2021 arXiv   pre-print
Recently, large-scale pre-trained language models, have shown promising results for few-shot learning in ToD.  ...  Empirical results demonstrate that the proposed self-training approach consistently improves state-of-the-art pre-trained models (BERT, ToD-BERT) when only a small number of labeled data are available.  ...  Du et al. (2020) also recently showed the benefit of ST over pre-training for general natural language understanding.  ... 
arXiv:2108.12589v1 fatcat:otctbzsg5jgptb7sjhuzj3tcaa

Effectiveness of Pre-training for Few-shot Intent Classification [article]

Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao-Ming Wu, Albert Y.S. Lam
2021 arXiv   pre-print
This paper investigates the effectiveness of pre-training for few-shot intent classification.  ...  Specifically, fine-tuning BERT with roughly 1,000 labeled data yields a pre-trained model -- IntentBERT, which can easily surpass the performance of existing pre-trained models for few-shot intent classification  ...  Acknowledgments We would like to thank the anonymous reviewers for their helpful comments. This research was supported by the grants of HK ITF UIM/377 and DaSAIL project P0030935.  ... 
arXiv:2109.05782v1 fatcat:ea65zk6f6jdbtm55i3q67g6y6q

Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation

Wanwei He, Yinpei Dai, Min Yang, Jian Sun, Fei Huang, Luo Si, Yongbin Li
2022 Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval  
Recently, pre-training methods have shown remarkable success in task-oriented dialog (TOD) systems.  ...  We design a dedicated pre-training objective for each component. Concretely, we pre-train the dialog encoding module with span mask language modeling to learn contextualized dialog information.  ...  For example, although TOD-BERT significantly improves the performance of a wide range of dialog understanding tasks [99] , it is difficult to apply TOD-BERT for dialog generation due to its bidirectional  ... 
doi:10.1145/3477495.3532069 fatcat:mns2pgtjynhkzh46zgqfbb2vee

DialogZoo: Large-Scale Dialog-Oriented Task Learning [article]

Zhi Chen, Jijia Bao, Lu Chen, Yuncong Liu, Da Ma, Bei Chen, Mengyue Wu, Su Zhu, Jian-Guang Lou, Kai Yu
2022 arXiv   pre-print
In addition to this dataset, we further propose two dialogue-oriented self-supervised tasks, and finally use the mixture of supervised and self-supervised datasets to train our foundation model.  ...  We evaluate our model on various downstream dialogue tasks.  ...  corpus, and TOD-BERT , and ConvBERT (Mehri et al., 2020) , which use task-oriented dialogues as the training corpus.  ... 
arXiv:2205.12662v1 fatcat:bjjvinmorzbljk7274juubbevm

Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System [article]

Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, Yi Zhang
2022 arXiv   pre-print
Pre-trained language models have been recently shown to benefit task-oriented dialogue (TOD) systems.  ...  In this study, we present PPTOD, a unified plug-and-play model for task-oriented dialogue.  ...  : In the dialogue multi-task pre-training stage, we pre-train our model with four TOD-related tasks, including natural language understanding (NLU), dialogue state tracking (DST), dialogue policy learning  ... 
arXiv:2109.14739v2 fatcat:dsy25xhxlbbk3f2qccoluwtpiy

MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems [article]

Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Pascale Fung
2020 arXiv   pre-print
In this paper, we propose Minimalist Transfer Learning (MinTL) to simplify the system design process of task-oriented dialogue systems and alleviate the over-dependency on annotated data.  ...  MinTL is a simple yet effective transfer learning framework, which allows us to plug-and-play pre-trained seq2seq models, and jointly learn dialogue state tracking and dialogue response generation.  ...  However, adapting pre-trained language models to task-oriented dialogue systems is not trivial.  ... 
arXiv:2009.12005v2 fatcat:f442ggfkgbe3navnwefapq5khy

AraConv: Developing an Arabic Task-Oriented Dialogue System Using Multi-Lingual Transformer Model mT5

Ahlam Fuad, Maha Al-Yahya
2022 Applied Sciences  
Task-oriented dialogue systems (DS) are designed to help users perform daily activities using natural language.  ...  Task-oriented DS for English language have demonstrated promising performance outcomes; however, developing such systems to support Arabic remains a challenge.  ...  [34] used deep learning approaches to build a natural language understanding module for Arabic task-oriented DS for home automation.  ... 
doi:10.3390/app12041881 fatcat:o77vdjfthbevxl6re2m6cydch4

Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization [article]

Haode Zhang, Haowen Liang, Yuwei Zhang, Liming Zhan, Xiao-Ming Wu, Xiaolei Lu, Albert Y.S. Lam
2022 arXiv   pre-print
It is challenging to train a good intent classifier for a task-oriented dialogue system with only a few annotations.  ...  Recent studies have shown that fine-tuning pre-trained language models with a small amount of labeled utterances from public benchmarks in a supervised manner is extremely helpful.  ...  Acknowledgments We would like to thank the anonymous reviewers for their valuable comments. This research was supported by the grants of HK ITF UIM/377 and PolyU DaSAIL project P0030935 funded by RGC.  ... 
arXiv:2205.07208v2 fatcat:ugzofe2vejhqncmkbkobqrwgty

Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding [article]

Shiyang Li, Semih Yavuz, Wenhu Chen, Xifeng Yan
2021 arXiv   pre-print
Task-adaptive pre-training (TAPT) and Self-training (ST) have emerged as the major semi-supervised approaches to improve natural language understanding (NLU) tasks with massive amount of unlabeled data  ...  language inference, named entity recognition and dialogue slot classification.  ...  Acknowledgement The authors would like to thank the anonymous reviewers for their thoughtful comments.  ... 
arXiv:2109.06466v1 fatcat:op47sc3fmzhqbkqswwhjmqqm4a
« Previous Showing results 1 — 15 out of 49 results