Filters








110 Hits in 2.8 sec

Riemannian Optimization for Skip-Gram Negative Sampling

Alexander Fonarev, Oleksii Grinchuk, Gleb Gusev, Pavel Serdyukov, Ivan Oseledets
2017 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)  
Skip-Gram Negative Sampling (SGNS) word embedding model, well known by its implementation in "word2vec" software, is usually optimized by stochastic gradient descent.  ...  However, the optimization of SGNS objective can be viewed as a problem of searching for a good matrix with the low-rank constraint.  ...  ., 2013 ) is a discriminative neural network that optimizes Skip-Gram Negative Sampling (SGNS) objective (see Equation 3 ).  ... 
doi:10.18653/v1/p17-1185 dblp:conf/acl/FonarevGGSO17 fatcat:vfohomk7ajdtnge2ig6z4l3yqa

Riemannian Optimization for Skip-Gram Negative Sampling [article]

Alexander Fonarev, Oleksii Hrinchuk, Gleb Gusev, Pavel Serdyukov, and Ivan Oseledets
2017 arXiv   pre-print
Skip-Gram Negative Sampling (SGNS) word embedding model, well known by its implementation in "word2vec" software, is usually optimized by stochastic gradient descent.  ...  However, the optimization of SGNS objective can be viewed as a problem of searching for a good matrix with the low-rank constraint.  ...  ., 2013 ) is a discriminative neural network that optimizes Skip-Gram Negative Sampling (SGNS) objective (see Equation 3 ).  ... 
arXiv:1704.08059v1 fatcat:l3gr73n6nbbjvoz7fwxt262q4q

Skip-gram word embeddings in hyperbolic space [article]

Matthias Leimeister, Benjamin J. Wilson
2019 arXiv   pre-print
An objective function based on the hyperbolic distance is derived and included in the skip-gram negative-sampling architecture of word2vec.  ...  Inspired by these results and scale-free structure in the word co-occurrence graph, we present an algorithm for learning word embeddings in hyperbolic space from free text.  ...  Code The implementation of the hyperbolic skip-gram training and experiments is available online. 3  ... 
arXiv:1809.01498v2 fatcat:mjdeovouhfdkhpqauilv7lmybm

Learning Mixed-Curvature Representations in Product Spaces

Albert Gu, Frederic Sala, Beliz Gunel, Christopher Ré
2019 International Conference on Learning Representations  
Empirically, we jointly learn the curvature and the embedding in the product space via Riemannian optimization.  ...  We discuss how to define and compute intrinsic quantities such as means-a challenging notion for product manifolds-and provably learnable optimization functions.  ...  Aside from choice of model, the training setup including hyperparameters (window size, negative samples, etc.) is identical to LW for all models.  ... 
dblp:conf/iclr/GuSGR19 fatcat:g7wtnbgknbexjameeru25cyc44

Natural Alpha Embeddings [article]

Riccardo Volpi, Luigi Malagò
2019 arXiv   pre-print
The aim of item embedding is to learn a low dimensional space for the representations, able to capture with its geometry relevant features or relationships for the data at hand.  ...  This can be achieved for example by exploiting adjacencies among items in large sets of unlabelled data.  ...  [32] , who proposed the use of Riemannian methods to optimize the Skip-Gram Negative Sampling objective function over the manifold of required low-rank matrices, and Nickel and Kiela [33] who introduced  ... 
arXiv:1912.02280v2 fatcat:lnql7yj64re4nhq2tj45zuj3uy

Hyperbolic Deep Learning for Chinese Natural Language Understanding [article]

Marko Valentin Micic, Hugo Chu
2018 arXiv   pre-print
In this paper we first train a large scale hyperboloid skip-gram model on a Chinese corpus, then apply the character embeddings to a downstream hyperbolic Transformer model derived from the principles  ...  of gyrovector space for Poincare disk model.  ...  ., w n ), where w i ∈ V, skip-gram learns a vector representation in Euclidean space for each word by using it to predict surrounding words.  ... 
arXiv:1812.10408v1 fatcat:kkc2eexfyzd33o34iaogxjgd4a

Unsupervised Word Embedding Learning by Incorporating Local and Global Contexts

Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Jiawei Han
2020 Frontiers in Big Data  
It is also shown in Levy and Goldberg (2014 ) that word2vec's Skip-Gram model with negative sampling is equivalent to factorizing a shifted PMI matrix.  ...  For fair comparison, we set the hyperparameters as below for all methods: word embedding dimension 11 p = 100, local context window size h = 5, number of negative samples k = 5, number of training iterations  ...  Government was authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.  ... 
doi:10.3389/fdata.2020.00009 pmid:33693384 pmcid:PMC7931948 fatcat:fyf5w7rdzffe3cwhmydrqtulni

Embedding Text in Hyperbolic Spaces

Bhuwan Dhingra, Christopher Shallue, Mohammad Norouzi, Andrew Dai, George Dahl
2018 Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12)  
However, the implicit continuous hierarchy in the learned hyperbolic space makes interrogating the model's learned hierarchies more difficult than for models that learn explicit edges between items.  ...  The learned hyperbolic embeddings show improvements over Euclidean embeddings in some -but not all -downstream tasks, suggesting that hierarchical organization is more useful for some tasks than others  ...  We quantitatively evaluate hyperbolic embeddings on two tasks against the baseline Skip-Gram with Negative Sampling (SGNS) embeddings (Mikolov et al., 2013a) 4 .  ... 
doi:10.18653/v1/w18-1708 dblp:conf/textgraphs/DhingraSNDD18 fatcat:xmfgkg7jg5fhpnk2qv2zi6wk5q

Embedding Text in Hyperbolic Spaces [article]

Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, George E. Dahl
2018 arXiv   pre-print
However, the implicit continuous hierarchy in the learned hyperbolic space makes interrogating the model's learned hierarchies more difficult than for models that learn explicit edges between items.  ...  The learned hyperbolic embeddings show improvements over Euclidean embeddings in some -- but not all -- downstream tasks, suggesting that hierarchical organization is more useful for some tasks than others  ...  We quantitatively evaluate hyperbolic embeddings on two tasks against the baseline Skip-Gram with Negative Sampling (SGNS) embeddings (Mikolov et al., 2013a) 4 .  ... 
arXiv:1806.04313v1 fatcat:nvbydehtqncsbktgi674fqerca

Ricci Curvature-Based Semi-Supervised Learning on an Attributed Network

Wei Wu, Guangmin Hu, Fucai Yu
2021 Entropy  
Ricci curvature is related to the optimal transport distance, which can well reflect the geometric structure of the underlying space of the network.  ...  Since graph-structured data are inherently non-Euclidean, we seek to use a non-Euclidean mathematical tool, namely, Riemannian geometry, to analyze graphs (networks).  ...  Planetoid [25] : It trains the samples to predict both the category labels of the samples and the contexts in the graph. It learns node embedding by optimizing for class label loss and context loss.  ... 
doi:10.3390/e23030292 pmid:33673440 pmcid:PMC7997130 fatcat:us4fzdj2fbep3cq4xezssp564y

An Unsupervised Learning Method for Attributed Network Based on Non-Euclidean Geometry

Wei Wu, Guangmin Hu, Fucai Yu
2021 Symmetry  
autoencoder module in which each layer is provided with a network information aggregation layer based on the Ricci curvature and an embedding layer based on hyperbolic geometry; (2) the second module is a skip-gram  ...  For u, v ∈ T p H n , let g p (u, v) =< u, v > M ; then, H n is a manifold equipped with a Riemannian metric g p .  ...  The other is the selected partial attribute data related to the skip-gram module, denoted as x index , which will be explained in detail in the section of the introduction to the skip-gram module.  ... 
doi:10.3390/sym13050905 fatcat:5e5336nc6zhibfmsn57jlfxqs4

Hyperbolic Heterogeneous Information Network Embedding

Xiao Wang, Yiding Zhang, Chuan Shi
2019 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
Specifically, to capture the structure and semantic relations between nodes, we employ the meta-path guided random walk to sample the sequences for each node.  ...  We further derive the effective optimization strategy to update the hyperbolic embeddings iteratively.  ...  For LINE, metapath2vec, PoincaréEmb and HHNE, we set the number of negative samples as 10.  ... 
doi:10.1609/aaai.v33i01.33015337 fatcat:uxkwqtfsj5evnmv3xjsro2dbn4

Low-dimensional statistical manifold embedding of directed graphs [article]

Thorben Funke, Tian Guo, Alen Lancic, Nino Antulov-Fantulin
2020 arXiv   pre-print
Like APP, DeepWalk uses the skip-gram model, trains the representation with random walks, and evaluates by cosine similarity between two node representations.  ...  BASELINES APP is the asymmetric proximity preserving graph embedding method [59] based on the skip-gram model, which is used by many other methods like Node2Vec and DeepWalk.  ...  A.9 IMPORTANCE SAMPLING As there exists no closed form of KL divergence for the generalized exponential power distributions, we propose an efficient importance sampling Monte Carlo estimation, which is  ... 
arXiv:1905.10227v3 fatcat:zaxy7tsejvgsjnvplae47kydyq

Changing the Geometry of Representations: α-Embeddings for NLP Tasks

Riccardo Volpi, Uddhipan Thakur, Luigi Malagò
2021 Entropy  
These distributions form a Riemannian statistical manifold, where word embeddings can be interpreted as vectors in the tangent space of a specific reference measure on the manifold.  ...  Their computation is based on the maximization of the likelihood of a conditional probability distribution for each word of the dictionary.  ...  In this paper, we focus on Skip-Gram (SG), a well-known log-linear model for the conditional probability of the context of a given central word.  ... 
doi:10.3390/e23030287 pmid:33652911 pmcid:PMC7996742 fatcat:4uqcmr76ojcrvk47cmzl3eke6i

Representing Hierarchical Structure by Using Cone Embedding [article]

Daisuke Takehara, Kei Kobayashi
2022 arXiv   pre-print
RELATED WORK RANDOM WALK MODELS DeepWalk [Perozzi et al., 2014] samples series by random walks on graphs and applies a method for embedding series data, such as the skip-gram model (word2vec Mikolov  ...  NON-EUCLIDEAN MODELS (HYPERBOLIC EMBEDDING MODELS) Poincaré embedding [Nickel and Kiela, 2017] is a method to embed the adjacency matrix of a graph in a skip-gram model, large-scale information network  ...  Note that this becomes negative when θ 1 + θ 2 + θ 3 > 2π.  ... 
arXiv:2102.08014v2 fatcat:6ee2s7hhcfcqbaqi7qm22wcg5u
« Previous Showing results 1 — 15 out of 110 results