Filters








2,754 Hits in 5.6 sec

Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models [article]

Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, Dhruv Batra
2018 arXiv   pre-print
Equally ubiquitous is the usage of beam search (BS) as an approximate inference algorithm to decode output sequences from these models.  ...  We observe that our method consistently outperforms BS and previously proposed techniques for diverse decoding from neural sequence models.  ...  CONCLUSION Beam search is the most commonly used approximate inference algorithm to decode sequences from RNNs; however, it suffers from a lack of diversity.  ... 
arXiv:1610.02424v2 fatcat:jskj532wabelpfq32cmpbmlsdi

Diverse Beam Search for Improved Description of Complex Scenes

Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, Dhruv Batra
2018 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
Beam Search, the standard work-horse for decoding sequences from these models, is an approximate inference algorithm that decodes the top-B sequences in a greedy left-to-right fashion.  ...  Recently, neural sequence models such as RNNs and LSTMs have been employed to produce visually-grounded language.  ...  Conclusion In this work, we propose Diverse Beam Search that modifies classical Beam Search decoding with a diversityaugmented sequence decoding objective.  ... 
doi:10.1609/aaai.v32i1.12340 fatcat:cv7sog6g6baqxm3abizcemjlja

Diverse Beam Search for Increased Novelty in Abstractive Summarization [article]

André Cibils, Claudiu Musat, Andreea Hossman, Michael Baeriswyl
2018 arXiv   pre-print
beam search.  ...  Secondly, we present a novel method, that relies on a diversity factor in computing the neural network loss, to improve the diversity of the summaries generated by any neural abstractive model implementing  ...  It is a variation of the classic beam search designed for neural sequence models which addresses the lack of diversity of the original algorithm [Gimpel et al., 2013] .  ... 
arXiv:1802.01457v1 fatcat:37wmwf5acjb33ba2mr3f566kfm

Determinantal Beam Search [article]

Clara Meister, Martina Forster, Ryan Cotterell
2021 arXiv   pre-print
Beam search is a go-to strategy for decoding neural sequence models.  ...  In a case study, we use the string subsequence kernel to explicitly encourage n-gram coverage in text generated from a sequence model.  ...  Introduction The decoding of neural sequence models is a fundamental component of many tasks in NLP.  ... 
arXiv:2106.07400v3 fatcat:qugbtczsn5hgzg2to5gwhd3sme

Searching for Search Errors in Neural Morphological Inflection [article]

Martina Forster, Clara Meister, Ryan Cotterell
2021 arXiv   pre-print
Neural sequence-to-sequence models are currently the predominant choice for language generation tasks.  ...  However, in the case of morphological inflection, we find that the empty string is almost never the most probable solution under the model. Further, greedy search often finds the global optimum.  ...  a deterministic task, greedy search is optimal under a Bayes optimal model, 3 most text generation tasks benefit from using beam search.  ... 
arXiv:2102.08424v1 fatcat:txdjpecrxjdcxavwfabir3ge3a

BSDAR: Beam Search Decoding with Attention Reward in Neural Keyphrase Generation [article]

Iftitahu Ni'mah, Vlado Menkovski, Mykola Pechenizkiy
2019 arXiv   pre-print
This study mainly investigates two decoding problems in neural keyphrase generation: sequence length bias and beam diversity.  ...  Results show that our proposed solution can overcome the algorithm bias to shorter and nearly identical sequences, resulting in a significant improvement of the decoding performance on generating keyphrases  ...  Figure 1 :Figure 2 : 12 Beam Search Decoding Issues: sequence length bias and beam diversity Beam Search Decoding Issues: generating Present (C) vs.  ... 
arXiv:1909.09485v1 fatcat:bdxonqnhkre53jhloeefaq33aq

Best-First Beam Search [article]

Clara Meister, Tim Vieira, Ryan Cotterell
2021 arXiv   pre-print
The default algorithm for this job is beam search -- a pruned version of breadth-first search.  ...  Quite surprisingly, beam search often returns better results than exact inference due to beneficial search bias for NLP tasks.  ...  We provide results on several sequence-to-sequence transduction tasks that show the speed-ups our algorithm provides over standard beam search for decoding neural models.  ... 
arXiv:2007.03909v4 fatcat:uyi4nb4wlvfx7be7qvl6fljywu

Best-First Beam Search

Clara Meister, Tim Vieira, Ryan Cotterell
2020 Transactions of the Association for Computational Linguistics  
The default algorithm for this job is beam search—a pruned version of breadth-first search.  ...  Quite surprisingly, beam search often returns better results than exact inference due to beneficial search bias for NLP tasks.  ...  We provide results on several sequence-to-sequence transduction tasks that show the speed-ups that our algorithm provides over standard beam search for decoding neural models.  ... 
doi:10.1162/tacl_a_00346 fatcat:zt7mtst34fgkrghbyamyzbll74

Beam Search with Bidirectional Strategies for Neural Response Generation [article]

Pierre Colombo and Chouchang Yang and Giovanna Varni and Chloé Clavel
2021 arXiv   pre-print
Sequence-to-sequence neural networks have been widely used in language-based applications as they have flexible capabilities to learn various language models.  ...  Instead of developing various decoder strategies based on a "regular sentence order" neural network (a trained model by outputting sentences from left-to-right order), we leveraged "reverse" order as additional  ...  During decoding, the reverse model estimates right-to-left dependencies while the regular model estimates left-to-right dependencies. 1 Two different settings 1 From graph topology viewpoints, the decoder  ... 
arXiv:2110.03389v1 fatcat:krwnfqg4xbhaji4uw3vocxet44

Conditional Poisson Stochastic Beam Search [article]

Clara Meister, Afra Amini, Tim Viera, Ryan Cotterell
2021 arXiv   pre-print
Beam search is the default decoding strategy for many sequence generation tasks in NLP.  ...  Furthermore, we show how samples generated under the CPSBS design can be used to build consistent estimators and sample diverse sets from sequence models.  ...  Beam Search In this section, we overview the necessary background on neural sequence models and beam search in order to motivate our algorithm in §3. Neural Sequence Models.  ... 
arXiv:2109.11034v2 fatcat:ccou4owlv5c7vldt6jtlin2kga

Multi-Turn Beam Search for Neural Dialogue Modeling [article]

Ilia Kulikov, Jason Lee, Kyunghyun Cho
2019 arXiv   pre-print
In neural dialogue modeling, a neural network is trained to predict the next utterance, and at inference time, an approximate decoding algorithm is used to generate next utterances given previous ones.  ...  We propose a novel approach for conversation-level inference by explicitly modeling the dialogue partner and running beam search across multiple conversation turns.  ...  Vijayakumar et al. (2018) propose an alternative to beam search that decodes a list of diverse outputs by optimizing for a diversity-augmented objective.  ... 
arXiv:1906.00141v2 fatcat:5r7sktsbhbfhhiuakv2grdyfra

Leveraging Sentence Similarity in Natural Language Generation: Improving Beam Search using Range Voting [article]

Sebastian Borgeaud, Guy Emerson
2020 arXiv   pre-print
The proposed method can be applied when generating from any probabilistic language model, including n-gram models and neural network models.  ...  We evaluate different similarity measures on an image captioning task and a machine translation task, and show that our method generates longer and more diverse sentences, providing a solution to the common  ...  Acknowledgements We would like to thank Kris Cao for discussions about distributions over sequences, which prompted the initial idea for this project. We would like to thank Dr.  ... 
arXiv:1908.06288v2 fatcat:bcgmr6oorbfsrirf7pvnwfxpmi

If beam search is the answer, what was the question? [article]

Clara Meister, Tim Vieira, Ryan Cotterell
2021 arXiv   pre-print
We frame beam search as the exact solution to a different decoding objective in order to gain insights into why high probability under a model alone may not indicate adequacy.  ...  generation models.  ...  Conclusion We analyze beam search as a decoding strategy for text generation models by framing it as the solution to an exact decoding problem.  ... 
arXiv:2010.02650v2 fatcat:kg3lqiwlcbalbf7uxrhzizpejm

Improving Beam Search by Removing Monotonic Constraint for Neural Machine Translation

Raphael Shu, Hideki Nakayama
2018 Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)  
To achieve high translation performance, neural machine translation models usually rely on the beam search algorithm for decoding sentences.  ...  The beam search finds good candidate translations by considering multiple hypotheses of translations simultaneously.  ...  In practice, NMT models use the beam search algorithm to generate output sequences in a limited time budget (Graves, 2012; .  ... 
doi:10.18653/v1/p18-2054 dblp:conf/acl/ShuN18 fatcat:lx5syoicvnadbeaj3i62v3byjq

Empirical Analysis of Beam Search Performance Degradation in Neural Sequence Models

Eldan Cohen, J. Christopher Beck
2019 International Conference on Machine Learning  
Beam search is the most popular inference algorithm for decoding neural sequence models.  ...  We perform an empirical study of the behavior of beam search across three sequence synthesis tasks.  ...  The most commonly used inference algorithm for decoding neural sequence models is beam search, a search algorithm that generates the sequence tokens one-by-one while keeping a fixed number of active candidates  ... 
dblp:conf/icml/CohenB19 fatcat:4odozvxwevbqnk3exdu5vwfsjy
« Previous Showing results 1 — 15 out of 2,754 results