143,277 Hits in 5.2 sec

Simple Local Attentions Remain Competitive for Long-Context Tasks [article]

Wenhan Xiong, Barlas Oğuz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Wen-tau Yih, Yashar Mehdad
2022 arXiv   pre-print
For each attention variant, we pretrain large-size models using the same long-doc corpus and then finetune these models for real-world long-context tasks.  ...  Many NLP tasks require processing long contexts beyond the length limit of pretrained models.  ...  We did not observe significant performance gaps between different attention variants and simple local attentions remain strong compared to the best Nyström attention. has Table 2 2 , after we scale up  ... 
arXiv:2112.07210v2 fatcat:hyng2ppaezdrpafd5mv2mak5yi

Sparse Sinkhorn Attention [article]

Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, Da-Cheng Juan
2020 arXiv   pre-print
Given sorted sequences, we are then able to compute quasi-global attention with only local windows, improving the memory efficiency of the attention module.  ...  method is competitive with vanilla attention and consistently outperforms recently proposed efficient Transformer models such as Sparse Transformers.  ...  Our proposed Sinkhorn attention remains competitive to the dense fully-connected attention while outperforming local attention and Sparse Transformers.  ... 
arXiv:2002.11296v1 fatcat:rhsnvybbyjhrbgmnbofqszu6xi

Cognitive Vision Needs Attention to Link Sensing with Recognition [chapter]

John K. Tsotsos
2006 Lecture Notes in Computer Science  
The presentation begins with a brief discussion on a definition for attention followed by an enumeration of the different ways in which attention should play a role in computer vision and cognitive vision  ...  Cognitive computer vision is concerned with integration and control of vision systems using explicit but not necessarily symbolic models of context, situation and goaldirected behaviour" (Vernon 2003)  ...  These must remain topics for future research.  ... 
doi:10.1007/11414353_3 fatcat:s2nstgmsengg7idsvtcasil3vi

Psychophysical Measurement of Attentional Modulation in Low-Level Vision Using the Lateral-Interactions Paradigm [chapter]

Elliot Freeman, Jon Driver, Dov Sagi
2002 Visual Attention Mechanisms  
Thanks to Ariella Popple for helpful comments.  ...  on evidence for long-range horizontal connections in early visual cortex.  ...  For example, anatomical studies of V1 have revealed long-range horizontal connections between cells, often extending for long distances outside each cell's classical receptive field along an axis defined  ... 
doi:10.1007/978-1-4615-0111-4_3 fatcat:3bbjze2z35adfj2rhx4dmxcixu

Recent Advances in Vision Transformer: A Survey and Outlook of Recent Work [article]

Khawar Islam
2022 arXiv   pre-print
Vision Transformers (ViTs) are becoming more popular and dominating technique for various vision tasks, compare to Convolutional Neural Networks (CNNs).  ...  In this paper, we begin by introducing the fundamental concepts and background of the self-attention mechanism.  ...  [15] 3D Object Detection Point clouds backbone and local and global context [16] Medical Image Segmentation Convolution for features and long-range association [17] Object Goal Navigation Object instances  ... 
arXiv:2203.01536v2 fatcat:m26h7ll4xzeylmdoezftbel52m

SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition [article]

Jing Pan, Tao Lei, Kwangyoun Kim, Kyu Han, Shinji Watanabe
2021 arXiv   pre-print
long-range dependencies.  ...  The Transformer architecture has been well adopted as a dominant architecture in most sequence transduction tasks including automatic speech recognition (ASR), since its attention mechanism excels in capturing  ...  fine-grain local context [8, 9] .  ... 
arXiv:2110.05571v1 fatcat:bsqrxudj5zftpgdxgsmdsclvkq

Attention links sensing to recognition

Albert L. Rothenstein, John K. Tsotsos
2008 Image and Vision Computing  
it may form an appropriate starting point for the difficult task of integrating attention into cognitive vision systems. q 2006 Published by Elsevier B.V.  ...  This paper presents arguments that explicit strategies for visual attentional selection are important for cognitive vision systems, and shows that a number of proposals currently exist for exactly how  ...  The authors thank the two anonymous reviewers for their comments and suggestions.  ... 
doi:10.1016/j.imavis.2005.08.011 fatcat:al6fcjnwqza43ol4azju7iobai

Investigating implicit statistical learning mechanisms through contextual cueing

Annabelle Goujon, André Didierjean, Simon Thorpe
2015 Trends in Cognitive Sciences  
This review was supported by grants from l'Agence Nationale de la Recherche (ANR) for the ELMA project and from the European Research Council (ERC) for the M4 project.  ...  Acknowledgments We thank Dr Thomas Geyer and five other anonymous reviewers, as well as Dr Rebecca Schwarzlose, Editor of Trends in Cognitive Sciences, Dr Eric Ruthruff, and Dr Nicholas Turk-Browne for  ...  How does the local context facilitate the search task?  ... 
doi:10.1016/j.tics.2015.07.009 pmid:26255970 fatcat:a5pnsgoppzddlgrz57fpqajrzq

How visual salience wins the battle for awareness

Steven Yantis
2005 Nature Neuroscience  
Attention can also be drawn involuntarily by salient objects-for example, by the sudden onset of a bright stimulus.  ...  A study now shows how this involuntary type of attention may mediate competition between representations in human visual cortex.  ...  simple, local visual properties such as edge orientation and contrast (Fig. 1) .  ... 
doi:10.1038/nn0805-975 pmid:16047021 fatcat:njt3bh3vibhkhbdym2ix5ydftu

Attention demands of language production in adults who stutter

Nathan D. Maxfield, Wendy L. Olsen, Daniel Kleinman, Stefan A. Frisch, Victor S. Ferreira, Jennifer J. Lister
2016 Clinical Neurophysiology  
For AWS, availability of attentional resources for secondary task processing was reduced as competition in word retrieval was resolved.  ...  P3 amplitude was also investigated separately in a simple tone oddball task. Results-P3 morphology was similar between groups in the simple task.  ...  We are grateful to Kalie Morris, Caitlin Kellar, Fanne L'Abbe and Alissa Belmont for their efforts in collecting and processing the data presented here. We thank Dr. Emanuel Maxfield et al. Page 21  ... 
doi:10.1016/j.clinph.2016.01.016 pmid:26971476 pmcid:PMC4792332 fatcat:uv272xtk6bg5xosogrox5bpode

The Effects of Information Format and Shopping Task on Consumers' Online Shopping Behavior: A Cognitive Fit Perspective

2004 Journal of Management Information Systems  
However, the suitability of the prevalent information formats in supporting various online shopping tasks is not known.  ...  Using the cognitive fit theory as the theoretical framework, we developed a research model to investigate the fit between information format and shopping task, and exam-NOTES  ...  Acknowledgments: The authors thank the three anonymous reviewers for their constructive comments on early versions of the paper.  ... 
doi:10.1080/07421222.2004.11045812 fatcat:pscn2xlstvbdzkk5fjaimmsihq

The Infant Orienting With Attention Task: Assessing the Neural Basis of Spatial Attention in Infancy

Shannon Ross-Sheehy, Sebastian Schneegans, John P. Spencer
2015 Infancy  
Results indicate rapid development of spatial attention and visual response competition between 5 and 10 months.  ...  After only a few hours of visual experience, neonates demonstrate visual preferences for simple highcontrast objects and shapes (Johnson, 1992) and begin to recognize their mother's face (Bushnell, 2001  ...  Fox at the University of Maryland, for his invaluable role in the conception of this project.  ... 
doi:10.1111/infa.12087 pmid:26273232 pmcid:PMC4530987 fatcat:jdmhbtsh7vhvzgwnkpx3oyqfay

Withdrawing attention at little or no cost: Detection and discrimination tasks

Jochen Braun, Bela Julesz
1998 Perception & Psychophysics  
Weused a concurrent-task paradigm to investigate the attentional cost of simple visual tasks.  ...  We hypothesize that the information in question is selected by a competition for saliency at the level of early vision.  ...  Competition for Saliency Competition between stimuli is not a new idea and has long been postulated to account for "visual saliency."  ... 
doi:10.3758/bf03211915 pmid:9503909 fatcat:sbdplmzuxveexmuuf64umvodqi

Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation [article]

Alessandro Raganato, Yves Scherrer, Jörg Tiedemann
2020 arXiv   pre-print
However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns.  ...  In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed -- non-learnable -- attentive patterns that are solely based on position and do not require any external  ...  The authors gratefully acknowledge the support of the CSC -IT Center for Science, Finland, for computational resources. Finally, We would also like to acknowledge NVIDIA and their GPU grant.  ... 
arXiv:2002.10260v3 fatcat:lajp4hcpbfdz7pjkhp6r2q67tu

Selection from perceptual and conceptual representations

2004 Cognitive, Affective, & Behavioral Neuroscience  
., visual selective attention) and selecting relevant information from conceptual representations (e.g., selecting a specific attribute about an object from long-term memory).  ...  Selection mechanisms have been described in several domains spanning different stages of processing, from visual attention to response execution.  ...  For example, consider the simple task of retrieving the color of an apple when the object is not immediately available in front of you.  ... 
doi:10.3758/cabn.4.4.466 pmid:15849891 fatcat:2dkxrkx5a5b35fag7e463yy644
« Previous Showing results 1 — 15 out of 143,277 results