907 Hits in 7.4 sec

Natural Language Generation through Character-based RNNs with Finite-state Prior Knowledge

Raghav Goyal, Marc Dymetman, Éric Gaussier
2016 International Conference on Computational Linguistics  
In order to avoid generating non-words and inventing information not present in the input, we propose a method for incorporating prior knowledge into the RNN in the form of a weighted finite-state automaton  ...  than a rule-based system, it is able to improve certain aspects of the utterances, in particular their naturalness.  ...  In order to improve this, we propose to constrain the generation of characters through a certain weighted finite-state automaton that incorporates prior knowledge: (i) about well-formed strings of characters  ... 
dblp:conf/coling/GoyalDG16 fatcat:fj7xxs4uprew3dbxhvfuxrykxm

Symbolic Priors for RNN-based Semantic Parsing

Chunyang Xiao, Marc Dymetman, Claire Gardent
2017 Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence  
entities present in the input utterance are also present in the logical form is modeled by weighted finite-state automata.  ...  While in principle they can be trained directly on pairs (natural language utterances, logical forms), their performance is limited by the amount of available data.  ...  In the context of Natural Language Generation (NLG), Goyal et al. [2016] describe an RNN model that generates sentences character-by-character, conditional on a semantic input.  ... 
doi:10.24963/ijcai.2017/585 dblp:conf/ijcai/XiaoDG17 fatcat:ldjznvlrpze6bapemn2coasspe

Log-Linear RNNs: Towards Recurrent Neural Networks with Flexible Prior Knowledge [article]

Marc Dymetman, Chunyang Xiao
2016 arXiv   pre-print
We conduct experiments in the domain of language modelling of French, that exploit morphological prior knowledge and show an important decrease in perplexity relative to a baseline RNN.  ...  rich prior knowledge, while the NN aspect, according to the "representation learning" paradigm, allows the model to discover novel combination of characteristics.  ...  In this approach, the background b, formulated as a weighted finite-state automaton over characters, is used both for encouraging the system to generate character strings that correspond to possible dictionary  ... 
arXiv:1607.02467v2 fatcat:m6qomk243rbpncbetg57zuk4sq

Comparison of Hidden Markov Model and Recurrent Neural Network in Automatic Speech Recognition

Akshay Madhav Deshmukh
2020 European Journal of Engineering Research and Science  
RNN-CTC).  ...  survey on state of the art researches on two major models, namely Deep Neural Network - Hidden Markov Model (DNN-HMM) and Recurrent Neural Networks trained with Connectionist Temporal Classification (  ...  Miao et al. on the other hand used a decoding approach which is generic and based on Weighted finite-state transducers (WFST) [25] .  ... 
doi:10.24018/ejers.2020.5.8.2077 fatcat:yxf3da6jyzcoxcjftcsytvytbq

Recovering Missing Characters in Old Hawaiian Writing

Brendan Shillingford, Oiwi Parker Jones
2018 Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing  
One approach is implemented, endto-end, using finite state transducers (FSTs).  ...  The other is a hybrid deep learning approach, which approximately composes an FST with a recurrent neural network language model.  ...  The beam element holds the current FST and RNN states, and the path taken through the FST so far.  ... 
doi:10.18653/v1/d18-1533 dblp:conf/emnlp/ShillingfordJ18 fatcat:yw6r2j2sffcsdgxujjwh2vytem

Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM [article]

Takaaki Hori, Shinji Watanabe, Yu Zhang, William Chan
2017 arXiv   pre-print
We learn to listen and write characters with a joint Connectionist Temporal Classification (CTC) and attention-based encoder-decoder network.  ...  During the beam search process, we combine the CTC predictions, the attention-based decoder predictions and a separately trained LSTM language model.  ...  Finally, these modules are integrated into a Weighted Finite-State Transducer (WFST) for efficient decoding.  ... 
arXiv:1706.02737v1 fatcat:cdswlmukebhnvepo6phfezmnae

Representing Formal Languages: A Comparison Between Finite Automata and Recurrent Neural Networks [article]

Joshua J. Michalenko, Ameesh Shah, Abhinav Verma, Richard G. Baraniuk, Swarat Chaudhuri, Ankit B. Patel
2019 arXiv   pre-print
finite automaton (MDFA) for the language.  ...  Specifically, we train a RNN on positive and negative examples from a regular language, and ask if there is a simple decoding function that maps states of this RNN to states of the minimal deterministic  ...  We aim to explain this internal algorithm of RNNs through comparison to fundamental concepts in formal languages, namely, finite automata and regular languages.  ... 
arXiv:1902.10297v1 fatcat:cx5vo7rplje7loet3asg3ek5qm

Tabula Nearly Rasa: Probing the Linguistic Knowledge of Character-level Neural Language Models Trained on Unsegmented Text

Michael Hahn, Marco Baroni
2019 Transactions of the Association for Computational Linguistics  
We present a multi-lingual study of the linguistic knowledge encoded in RNNs trained as character-level language models, on input data with word boundaries removed.  ...  Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks.  ...  RNNs are very general devices for sequence processing, hardly assuming any prior linguistic knowledge.  ... 
doi:10.1162/tacl_a_00283 fatcat:lw3p6pttabfylp7t7u53xx2smi

Recognizing Long Grammatical Sequences Using Recurrent Networks Augmented With An External Differentiable Stack [article]

Ankur Mali, Alexander Ororbia, Daniel Kifer, Clyde Lee Giles
2020 arXiv   pre-print
In this paper, we improve the memory-augmented RNN with important architectural and state updating mechanisms that ensure that the model learns to properly balance the use of its latent states with external  ...  For example, RNNs struggle in recognizing complex context free languages (CFLs), never reaching 100% accuracy on training.  ...  While some of the languages from the Chomsky hierarchy [8] can be modelled based on counting, counting does not capture the actual structural property underlying natural languages -formal languages are  ... 
arXiv:2004.07623v2 fatcat:xh5usvh2hvczzkia2g6tdppn5e

The Neural State Pushdown Automata [article]

Ankur Mali, Alexander Ororbia, C. Lee Giles
2019 arXiv   pre-print
We observe that many RNNs with and without memory, but no prior knowledge, fail to converge and generalize poorly on CFGs.  ...  Our results show that, for Dyck(2) languages, prior rule-based knowledge is critical for optimization convergence and for ensuring generalization to longer sequences at test time.  ...  Experimentally, later we will see that this regularizer improves generalization even when prior knowledge is not integrated into the RNN.  ... 
arXiv:1909.05233v2 fatcat:imaxvlic4fepppsleoxw3xmez4

Memorize or generalize? Searching for a compositional RNN in a haystack [article]

Adam Liška, Germán Kruszewski, Marco Baroni
2018 arXiv   pre-print
In this paper, we explore the compositional generalization capabilities of recurrent neural networks (RNNs).  ...  this domain when trained with standard gradient descent and provided with additional supervision.  ...  through finite means.  ... 
arXiv:1802.06467v2 fatcat:xvazpuk3yfgxtcs7gvjd3drnpq

Improvisational Computational Storytelling in Open Worlds [chapter]

Lara J. Martin, Brent Harrison, Mark O. Riedl
2016 Lecture Notes in Computer Science  
Human improvisation occurs in an open-world that can be in any state and characters can perform any behaviors expressible through natural language.  ...  The goal is to develop an intelligent agent that can sensibly co-create a story with one or more humans through natural language.  ...  Third, character actions are conveyed through language and gesture.  ... 
doi:10.1007/978-3-319-48279-8_7 fatcat:6wvedzvzmzguvee6ra62svp5d4

Chatterbot implementation using Transfer Learning and LSTM Encoder-Decoder Architecture

Kolla Bhanu Prakash
2020 International Journal of Emerging Trends in Engineering Research  
Chatterbot is an existing research area whose main goal is to appear as human as possible and most of the current models(which use RNN and related sequential learning models) are unable to achieve this  ...  This model also shows promise as a generative based model which prior generative approaches were not able to show.  ...  Natural language processing is one important area in AI research.  ... 
doi:10.30534/ijeter/2020/35852020 fatcat:obt76fz54banfoxeh5vh2wmv4u

A New Approach to Knowledge-Based Design of Recurrent Neural Networks

E. Kolman, M. Margaliot
2008 IEEE Transactions on Neural Networks  
This yields a modular and systematic approach for knowledge-based design of RNNs.  ...  Inferring the FARB yields an input-output mapping that is mathematically equivalent to that of an RNN. We use this equivalence to develop two new knowledge-based design methods for RNNs.  ...  The most common KBD technique for RNNs is based on representing the prior knowledge in the form of a deterministic finite-state automaton (DFA) [1] , [43] , [44] .  ... 
doi:10.1109/tnn.2008.2000393 pmid:18701369 fatcat:ehvzzv6p25bmdbqxtvexo2ilue

On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages [article]

Satwik Bhattamishra, Kabir Ahuja, Navin Goyal
2020 arXiv   pre-print
Since natural language datasets have nested dependencies of bounded depth, this may help explain why they perform well in modeling hierarchical dependencies in natural language data despite prior works  ...  Hence, we evaluate our models on samples generated from Dyck languages with bounded depth and find that they are indeed able to generalize to much higher lengths.  ...  To our knowledge, prior works have not empirically analyzed Transformers on formal languages, particularly context-free languages.  ... 
arXiv:2011.03965v1 fatcat:rrh2kr7ifbhxpcktizgcb6biwy
« Previous Showing results 1 — 15 out of 907 results