A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
Filters
Simple Local Attentions Remain Competitive for Long-Context Tasks
[article]
2022
arXiv
pre-print
Preserved Fulltext
Other Versions
For each attention variant, we pretrain large-size models using the same long-doc corpus and then finetune these models for real-world long-context tasks. ...
Many NLP tasks require processing long contexts beyond the length limit of pretrained models. ...
We did not observe significant performance gaps between different attention variants and simple local attentions remain strong compared to the best Nyström attention. has
Table 2 2 , after we scale up ...
arXiv:2112.07210v2
fatcat:hyng2ppaezdrpafd5mv2mak5yi
Sparse Sinkhorn Attention
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Given sorted sequences, we are then able to compute quasi-global attention with only local windows, improving the memory efficiency of the attention module. ...
method is competitive with vanilla attention and consistently outperforms recently proposed efficient Transformer models such as Sparse Transformers. ...
Our proposed Sinkhorn attention remains competitive to the dense fully-connected attention while outperforming local attention and Sparse Transformers. ...
arXiv:2002.11296v1
fatcat:rhsnvybbyjhrbgmnbofqszu6xi
Cognitive Vision Needs Attention to Link Sensing with Recognition
[chapter]
2006
Lecture Notes in Computer Science
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
The presentation begins with a brief discussion on a definition for attention followed by an enumeration of the different ways in which attention should play a role in computer vision and cognitive vision ...
Cognitive computer vision is concerned with integration and control of vision systems using explicit but not necessarily symbolic models of context, situation and goaldirected behaviour" (Vernon 2003) ...
These must remain topics for future research. ...
doi:10.1007/11414353_3
fatcat:s2nstgmsengg7idsvtcasil3vi
Psychophysical Measurement of Attentional Modulation in Low-Level Vision Using the Lateral-Interactions Paradigm
[chapter]
2002
Visual Attention Mechanisms
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
Thanks to Ariella Popple for helpful comments. ...
on evidence for long-range horizontal connections in early visual cortex. ...
For example, anatomical studies of V1 have revealed long-range horizontal connections between cells, often extending for long distances outside each cell's classical receptive field along an axis defined ...
doi:10.1007/978-1-4615-0111-4_3
fatcat:3bbjze2z35adfj2rhx4dmxcixu
Recent Advances in Vision Transformer: A Survey and Outlook of Recent Work
[article]
2022
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
Other Versions
Vision Transformers (ViTs) are becoming more popular and dominating technique for various vision tasks, compare to Convolutional Neural Networks (CNNs). ...
In this paper, we begin by introducing the fundamental concepts and background of the self-attention mechanism. ...
[15] 3D Object Detection Point clouds backbone and local and global context [16] Medical Image Segmentation Convolution for features and long-range association [17] Object Goal Navigation Object instances ...
arXiv:2203.01536v2
fatcat:m26h7ll4xzeylmdoezftbel52m
SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition
[article]
2021
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL.
The file type is application/pdf.
long-range dependencies. ...
The Transformer architecture has been well adopted as a dominant architecture in most sequence transduction tasks including automatic speech recognition (ASR), since its attention mechanism excels in capturing ...
fine-grain local context [8, 9] . ...
arXiv:2110.05571v1
fatcat:bsqrxudj5zftpgdxgsmdsclvkq
Attention links sensing to recognition
2008
Image and Vision Computing
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
it may form an appropriate starting point for the difficult task of integrating attention into cognitive vision systems. q 2006 Published by Elsevier B.V. ...
This paper presents arguments that explicit strategies for visual attentional selection are important for cognitive vision systems, and shows that a number of proposals currently exist for exactly how ...
The authors thank the two anonymous reviewers for their comments and suggestions. ...
doi:10.1016/j.imavis.2005.08.011
fatcat:al6fcjnwqza43ol4azju7iobai
Investigating implicit statistical learning mechanisms through contextual cueing
2015
Trends in Cognitive Sciences
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
This review was supported by grants from l'Agence Nationale de la Recherche (ANR) for the ELMA project and from the European Research Council (ERC) for the M4 project. ...
Acknowledgments We thank Dr Thomas Geyer and five other anonymous reviewers, as well as Dr Rebecca Schwarzlose, Editor of Trends in Cognitive Sciences, Dr Eric Ruthruff, and Dr Nicholas Turk-Browne for ...
How does the local context facilitate the search task? ...
doi:10.1016/j.tics.2015.07.009
pmid:26255970
fatcat:a5pnsgoppzddlgrz57fpqajrzq
How visual salience wins the battle for awareness
2005
Nature Neuroscience
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2007; you can also visit the original URL.
The file type is application/pdf.
Attention can also be drawn involuntarily by salient objects-for example, by the sudden onset of a bright stimulus. ...
A study now shows how this involuntary type of attention may mediate competition between representations in human visual cortex. ...
simple, local visual properties such as edge orientation and contrast (Fig. 1) . ...
doi:10.1038/nn0805-975
pmid:16047021
fatcat:njt3bh3vibhkhbdym2ix5ydftu
Attention demands of language production in adults who stutter
2016
Clinical Neurophysiology
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2019; you can also visit the original URL.
The file type is application/pdf.
For AWS, availability of attentional resources for secondary task processing was reduced as competition in word retrieval was resolved. ...
P3 amplitude was also investigated separately in a simple tone oddball task. Results-P3 morphology was similar between groups in the simple task. ...
We are grateful to Kalie Morris, Caitlin Kellar, Fanne L'Abbe and Alissa Belmont for their efforts in collecting and processing the data presented here. We thank Dr. Emanuel Maxfield et al. Page 21 ...
doi:10.1016/j.clinph.2016.01.016
pmid:26971476
pmcid:PMC4792332
fatcat:uv272xtk6bg5xosogrox5bpode
The Effects of Information Format and Shopping Task on Consumers' Online Shopping Behavior: A Cognitive Fit Perspective
2004
Journal of Management Information Systems
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
However, the suitability of the prevalent information formats in supporting various online shopping tasks is not known. ...
Using the cognitive fit theory as the theoretical framework, we developed a research model to investigate the fit between information format and shopping task, and exam-NOTES ...
Acknowledgments: The authors thank the three anonymous reviewers for their constructive comments on early versions of the paper. ...
doi:10.1080/07421222.2004.11045812
fatcat:pscn2xlstvbdzkk5fjaimmsihq
The Infant Orienting With Attention Task: Assessing the Neural Basis of Spatial Attention in Infancy
2015
Infancy
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Results indicate rapid development of spatial attention and visual response competition between 5 and 10 months. ...
After only a few hours of visual experience, neonates demonstrate visual preferences for simple highcontrast objects and shapes (Johnson, 1992) and begin to recognize their mother's face (Bushnell, 2001 ...
Fox at the University of Maryland, for his invaluable role in the conception of this project. ...
doi:10.1111/infa.12087
pmid:26273232
pmcid:PMC4530987
fatcat:jdmhbtsh7vhvzgwnkpx3oyqfay
Withdrawing attention at little or no cost: Detection and discrimination tasks
1998
Perception & Psychophysics
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
Weused a concurrent-task paradigm to investigate the attentional cost of simple visual tasks. ...
We hypothesize that the information in question is selected by a competition for saliency at the level of early vision. ...
Competition for Saliency Competition between stimuli is not a new idea and has long been postulated to account for "visual saliency." ...
doi:10.3758/bf03211915
pmid:9503909
fatcat:sbdplmzuxveexmuuf64umvodqi
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Other Versions
However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns. ...
In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed -- non-learnable -- attentive patterns that are solely based on position and do not require any external ...
The authors gratefully acknowledge the support of the CSC -IT Center for Science, Finland, for computational resources. Finally, We would also like to acknowledge NVIDIA and their GPU grant. ...
arXiv:2002.10260v3
fatcat:lajp4hcpbfdz7pjkhp6r2q67tu
Selection from perceptual and conceptual representations
2004
Cognitive, Affective, & Behavioral Neuroscience
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
., visual selective attention) and selecting relevant information from conceptual representations (e.g., selecting a specific attribute about an object from long-term memory). ...
Selection mechanisms have been described in several domains spanning different stages of processing, from visual attention to response execution. ...
For example, consider the simple task of retrieving the color of an apple when the object is not immediately available in front of you. ...
doi:10.3758/cabn.4.4.466
pmid:15849891
fatcat:2dkxrkx5a5b35fag7e463yy644
« Previous
Showing results 1 — 15 out of 143,277 results