WordRank: Learning Word Embeddings via Robust Ranking

Shihao Ji, Hyokun Yun, Pinar Yanardag, Shin Matsushima, S. V. N. Vishwanathan
2016 Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing  
Embedding words in a vector space has gained a lot of attention in recent years. While stateof-the-art methods provide efficient computation of word similarities via a low-dimensional matrix embedding, their motivation is often left unclear. In this paper, we argue that word embedding can be naturally viewed as a ranking problem due to the ranking nature of the evaluation metrics. Then, based on this insight, we propose a novel framework Wor-dRank that efficiently estimates word representations
more » ... ord representations via robust ranking, in which the attention mechanism and robustness to noise are readily achieved via the DCG-like ranking losses. The performance of WordRank is measured in word similarity and word analogy benchmarks, and the results are compared to the state-of-the-art word embedding techniques. Our algorithm is very competitive to the state-of-the-arts on large corpora, while outperforms them by a significant margin when the training set is limited (i.e., sparse and noisy). With 17 million tokens, WordRank performs almost as well as existing methods using 7.2 billion tokens on a popular word similarity benchmark. Our multi-node distributed implementation of WordRank is publicly available for general usage.
doi:10.18653/v1/d16-1063 dblp:conf/emnlp/JiYYMV16 fatcat:b6ankxk7x5f3le53xyd3m2rc5i