2,028 Hits in 6.1 sec

Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision

Mieradilijiang Maimaiti, Yang Liu, Yuanhang Zheng, Gang Chen, Kaiyu Huang, Ji Zhang, Huanbo Luan, Maosong Sun
2021 Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing   unpublished
Limited efforts have been made by previous studies to deal with such problems. In this work, we propose a self-supervised CWS approach with a straightforward and effective architecture.  ...  Recent state-of-the-art (SOTA) effective neural network methods and fine-tuning methods based on pre-trained models (PTM) have been used in Chinese word segmentation (CWS), and they achieve great results  ...  We thank all anonymous reviewers for their valuable comments and suggestions on this work.  ... 
doi:10.18653/v1/2021.emnlp-main.158 fatcat:mzqhyi3uxrgyjm5keamixlj27e

Discriminative Self-training for Punctuation Prediction [article]

Qian Chen, Wen Wang, Mengzhe Chen, Qinglin Zhang
2021 arXiv   pre-print
In this paper, we propose a Discriminative Self-Training approach with weighted loss and discriminative label smoothing to exploit unlabeled speech transcripts.  ...  Experimental results on the English IWSLT2011 benchmark test set and an internal Chinese spoken language dataset demonstrate that the proposed approach achieves significant improvement on punctuation prediction  ...  We also compare the proposed approach with previous methods on a large internal Chinese dataset [1] . We use Jieba 2 for Chinese word segmentation.  ... 
arXiv:2104.10339v2 fatcat:gfnf5xxkzzej7akmebwulho7bm

How Context Affects Language Models' Factual Predictions [article]

Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, Sebastian Riedel
2020 arXiv   pre-print
We report that augmenting pre-trained language models in this way dramatically improves performance and that the resulting system, despite being unsupervised, is competitive with a supervised machine reading  ...  Furthermore, processing query and context with different segment tokens allows BERT to utilize its Next Sentence Prediction pre-trained classifier to determine whether the context is relevant or not, substantially  ...  One possible reason for this phenomenon resides in the Next Sentence Prediction (NSP) classifier of BERT, learned with self-supervision during pretraining by training the model to distinguish contiguous  ... 
arXiv:2005.04611v1 fatcat:uylkyzvscve5dgcofihq7g3mfu

SpanBERT: Improving Pre-training by Representing and Predicting Spans

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, Omer Levy
2020 Transactions of the Association for Computational Linguistics  
Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without  ...  We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text.  ...  We thank the anonymous reviewers, the action editor, and our colleagues at Facebook AI Research and the University of Washington for their insightful feedback that helped improve the paper.  ... 
doi:10.1162/tacl_a_00300 fatcat:2i52pub37jcghhxp43gjw5cvve

SiBert: Enhanced Chinese Pre-trained Language Model with Sentence Insertion

Jiahao Chen, Chenjie Cao, Xiuyan Jiang
2020 International Conference on Language Resources and Evaluation  
Moreover, a word segmentation method called SentencePiece is utilized to further enhance Chinese Bert performance for tasks with long texts.  ...  Hence a new pre-training task called Sentence Insertion (SI) is proposed in this paper for Chinese query-passage pairs NLP tasks including answer span prediction, retrieval question answering and sentence  ...  In this table, WWM means the Whole Word Masking for Chinese. DA indicates the data augmentation for samples with unpaired queries and passages.  ... 
dblp:conf/lrec/ChenCJ20 fatcat:b2ub26e3zfbe5anwbsyc7gyili

Polyphone Disambiguition in Mandarin Chinese with Semi-Supervised Learning [article]

Yi Shi and Congyi Wang and Yu Chen and Bin Wang
2021 arXiv   pre-print
Although the problem has been well explored with both knowledge-based and learning-based approaches, it remains challenging due to the lack of publicly available labeled datasets and the irregular nature  ...  In this paper, we propose a novel semi-supervised learning (SSL) framework for Mandarin Chinese polyphone disambiguation that can potentially leverage unlimited unlabeled text data.  ...  It is a common practice in the industry to leverage an existing text segmenter and a Chinese word-topinyin dictionary.  ... 
arXiv:2102.00621v2 fatcat:ahnwiycy4baizgasvvx5uuqbea

From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video [article]

Junwei Liang
2021 arXiv   pre-print
This design hinders prediction performance in video data from diverse domains and unseen scenarios.  ...  However, human trajectory prediction still remains a challenging task, as scene semantics and human intent are difficult to model.  ...  In the following methods (Word Hard Matching and Latent Topic with Word Embedding), we rst extract bag-of-words features from di erent modalities for each video and then match them using speci c matching  ... 
arXiv:2011.10670v3 fatcat:mlom5zqk6jdvjndcsfwimpj7xu

Self-Supervised Speech Representation Learning: A Review [article]

Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, Shinji Watanabe
2022 arXiv   pre-print
Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active  ...  This review presents approaches for self-supervised speech representation learning and their connection to other research areas.  ...  Networks are usually pre-trained with SSL techniques, augmented with prediction heads, and fine-tuned (or trained) with labeled data in downstream tasks for benchmarking.  ... 
arXiv:2205.10643v2 fatcat:6pveqmlbh5ebrhv2wuvb5hcp7q

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking [article]

Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
2022 arXiv   pre-print
Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked.  ...  Self-supervised pre-training techniques have achieved remarkable progress in Document AI.  ...  The WPA objective is to predict whether the corresponding image patches of a text word are masked.  ... 
arXiv:2204.08387v3 fatcat:3tgkjyllzzhkdfasf25ytfrrfi

Stroke-Based Autoencoders: Self-Supervised Learners for Efficient Zero-Shot Chinese Character Recognition [article]

Zongze Chen and Wenxia Yang and Xin Li
2022 arXiv   pre-print
In this paper, we develop a stroke-based autoencoder(SAE), to model the sophisticated morphology of Chinese characters with the self-supervised method.  ...  One is fine-tuned on existing stroke-based method for zero-shot recognition of handwritten Chinese characters, and the other is applied to enrich the Chinese word embeddings from their morphological features  ...  Acknowledgement This work is partially supported by the National Natural Science Foundation of China (61573012), China Scholarship Council (201906955038), and National innovation and entrepreneurship training  ... 
arXiv:2207.08191v1 fatcat:xpngz2xy7vchjlsvxr4xz33kge

A Feasible Framework for Arbitrary-Shaped Scene Text Recognition [article]

Jinjin Zhang, Wei Wang, Di Huang, Qingjie Liu, Yunhong Wang
2019 arXiv   pre-print
Our method wins the championship on Scene Text Spotting Task (Latin Only, Latin and Chinese) of ICDAR2019 Robust Reading Challenge on ArbitraryShaped Text Competition.  ...  In this paper, we propose a feasible framework for multi-lingual arbitrary-shaped STR, including instance segmentation based text detection and language model based attention mechanism for text recognition  ...  Then the mask branch predicts the final segmented text region for latter text recognition.  ... 
arXiv:1912.04561v2 fatcat:5rtipn2hsjefjifv57ouizawoi

Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models [article]

Yuxuan Lai, Yijia Liu, Yansong Feng, Songfang Huang, Dongyan Zhao
2021 arXiv   pre-print
We further propose a masked segment prediction task to push the model to learn from rich but redundant information inherent in lattices, while avoiding learning unexpected tricks.  ...  In this work, we propose a novel pre-training paradigm for Chinese -- Lattice-BERT, which explicitly incorporates word representations along with characters, thus can model a sentence in a multi-granularity  ...  We also adopt the whole word masking trick. LBERT is our proposed Lattice-BERT model, with word lattices as inputs, equipping with lattice position attentions and masked segment prediction.  ... 
arXiv:2104.07204v2 fatcat:2xxwbavxqbc6bbodlmdhfirn5a

Instance Segmentation Network with Selfdistillation for Scene Text Detection

Peng Yang, Guowei Yang, Xun Gong, Pingping Wu, Xu Han, Jiasong Wu, Caisen Chen
2020 IEEE Access  
Segmentation based methods have become the mainstream for detecting scene text with arbitrary orientations and shapes.  ...  In this paper, we propose an instance segmentation network (ISNet), which simultaneously generates prototype masks and per-instance mask coefficients.  ...  OPTIMIZATION Multiple loss functions, including L mask for instance segmentation, L sd for self-distillation, and L norm , can be back-propagated to optimize the text detection model in a supervised manner  ... 
doi:10.1109/access.2020.2978225 fatcat:tylrcfwzcvfijn47dgthjnvtui

A Novel Chinese Dialect TTS Frontend with Non-Autoregressive Neural Machine Translation [article]

Wudi Bao, Junhui Zhang, Junjie Pan, Xiang Yin, Zejun Ma
2022 arXiv   pre-print
For Mandarin text inputs, Chinese dialect TTS can only generate partly-meaningful speech with relatively poor prosody and naturalness.  ...  To lower the bar of use and make it more practical in commercial, we propose a novel Chinese dialect TTS frontend with a translation module.  ...  CWS aims to address that Chinese words have no formal natural segmentation symbols.  ... 
arXiv:2206.04922v2 fatcat:u46vg7c24rglrd653g4uiv47va

Sequence Model with Self-Adaptive Sliding Window for Efficient Spoken Document Segmentation [article]

Qinglin Zhang, Qian Chen, Yali Li, Jiaqing Liu, Wen Wang
2021 arXiv   pre-print
We propose a sequence model with self-adaptive sliding window for accurate and efficient paragraph segmentation.  ...  Automatically predicting paragraph segmentation for spoken documents may both improve readability and downstream NLP performance such as summarization and machine reading comprehension.  ...  StructBERT [15] augments the BERT pre-training objectives with a word structural objective and a sentence structural objective.  ... 
arXiv:2107.09278v2 fatcat:3t7rya67ofgnzey3ntwfrmuole
« Previous Showing results 1 — 15 out of 2,028 results