Beyond Context: A New Perspective for Word Embeddings

Yichu Zhou, Vivek Srikumar
<span title="">2019</span> <i title="Association for Computational Linguistics"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/62lejef5rzghbafcrd6wj6pc7u" style="color: black;">Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*</a> </i> &nbsp;
Most word embeddings today are trained by optimizing a language modeling goal of scoring words in their context, modeled as a multiclass classification problem. Despite the successes of this assumption, it is incomplete: in addition to its context, orthographical or morphological aspects of words can offer clues about their meaning. In this paper, we define a new modeling framework for training word embeddings that captures this intuition. Our framework is based on the well-studied problem of
more &raquo; ... lti-label classification and, consequently, exposes several design choices for featurizing words and contexts, loss functions for training and score normalization. Indeed, standard models such as CBOW and FAST-TEXT are specific choices along each of these axes. We show via experiments that by combining feature engineering with embedding learning, our method can outperform CBOW using only 10% of the training data in both the standard word embedding evaluations and also text classification experiments.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.18653/v1/s19-1003">doi:10.18653/v1/s19-1003</a> <a target="_blank" rel="external noopener" href="https://dblp.org/rec/conf/starsem/ZhouS19.html">dblp:conf/starsem/ZhouS19</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/udz2aedmdnd2nnjh5rtgatrfr4">fatcat:udz2aedmdnd2nnjh5rtgatrfr4</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200507234337/https://www.aclweb.org/anthology/S19-1003.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/da/c7/dac731a9bf707b35ed351950740654184db83938.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.18653/v1/s19-1003"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> Publisher / doi.org </button> </a>