A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/2002.09812v1.pdf">the original URL</a>. The file type is <code>application/pdf</code>.
Sketching Transformed Matrices with Applications to Natural Language Processing
[article]
<span title="2020-02-23">2020</span>
<i >
arXiv
</i>
<span class="release-stage" >pre-print</span>
Preserved Fulltext
Suppose we are given a large matrix A=(a_i,j) that cannot be stored in memory but is in a disk or is presented in a data stream. However, we need to compute a matrix decomposition of the entry-wisely transformed matrix, f(A):=(f(a_i,j)) for some function f. Is it possible to do it in a space efficient way? Many machine learning applications indeed need to deal with such large transformed matrices, for example word embedding method in NLP needs to work with the pointwise mutual information (PMI)
<span class="external-identifiers">
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2002.09812v1">arXiv:2002.09812v1</a>
<a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/gcwy7iysnbhafhu4mcxydmk3hm">fatcat:gcwy7iysnbhafhu4mcxydmk3hm</a>
</span>
more »
... matrix, while the entrywise transformation makes it difficult to apply known linear algebraic tools. Existing approaches for this problem either need to store the whole matrix and perform the entry-wise transformation afterwards, which is space consuming or infeasible, or need to redesign the learning method, which is application specific and requires substantial remodeling. In this paper, we first propose a space-efficient sketching algorithm for computing the product of a given small matrix with the transformed matrix. It works for a general family of transformations with provable small error bounds and thus can be used as a primitive in downstream learning tasks. We then apply this primitive to a concrete application: low-rank approximation. We show that our approach obtains small error and is efficient in both space and time. We complement our theoretical results with experiments on synthetic and real data.
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200322050550/https://arxiv.org/pdf/2002.09812v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext">
<button class="ui simple right pointing dropdown compact black labeled icon button serp-button">
<i class="icon ia-icon"></i>
Web Archive
[PDF]
</button>
</a>
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2002.09812v1" title="arxiv.org access">
<button class="ui compact blue labeled icon button serp-button">
<i class="file alternate outline icon"></i>
arxiv.org
</button>
</a>