Filters








1,734 Hits in 5.4 sec

Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization [article]

Blake Woodworth, Jialei Wang, Adam Smith, Brendan McMahan, Nathan Srebro
2019 arXiv   pre-print
We suggest a general oracle-based framework that captures different parallel stochastic optimization settings described by a dependency graph, and derive generic lower bounds in terms of this graph.  ...  We highlight gaps between lower and upper bounds on the oracle complexity, and cases where the "natural" algorithms are not known to be optimal.  ...  Acknowledgements We would like to thank Ohad Shamir for helpful discussions.  ... 
arXiv:1805.10222v3 fatcat:islrouwldjdarbl64qmrkt2wli

Optimal Complexity in Decentralized Training [article]

Yucheng Lu, Christopher De Sa
2022 arXiv   pre-print
In this paper, we provide a tight lower bound on the iteration complexity for such methods in a stochastic non-convex setting.  ...  We prove by construction this lower bound is tight and achievable.  ...  Feder Cooper, Jerry Chee, Zheng Li, Ran Xin, Jiaqi Zhang and anonymous reviewers from ICML 2021 for providing valuable feedbacks on earlier versions of this paper.  ... 
arXiv:2006.08085v4 fatcat:zsbovarrqbgubgnwe3jbqgzaxa

The Minimax Complexity of Distributed Optimization [article]

Blake Woodworth
2021 arXiv   pre-print
First, I present the "graph oracle model", an extension of the classic oracle complexity framework that can be applied to study distributed optimization algorithms.  ...  In this thesis, I study the minimax oracle complexity of distributed stochastic optimization.  ...  A Generic Graph Oracle Lower Bound In this section, we prove a lower bound in the graph oracle model for any distributed, stochastic first-order optimization algorithm which depends on the associated graph  ... 
arXiv:2109.00534v1 fatcat:ibkwtyfd3bawzftakx7ebpwod4

Decentralized and Parallel Primal and Dual Accelerated Methods for Stochastic Convex Programming Problems [article]

Darina Dvinskikh, Alexander Gasnikov
2021 arXiv   pre-print
We introduce primal and dual stochastic gradient oracle methods for decentralized convex optimization problems.  ...  Both for primal and dual oracles, the proposed methods are optimal in terms of the number of communication steps.  ...  Note, that by using restarts with STM(L,0,x 0 ) one can eliminate the gap from ln(LR 2 /ε) to ln(µR 2 /ε) between lower bounds and the bounds for STM(L,µ,x 0 ) without restarts [27] .  ... 
arXiv:1904.09015v17 fatcat:7j5ueplfsbcshfv75kd7nxndne

The Complexity of Making the Gradient Small in Stochastic Convex Optimization [article]

Dylan J. Foster, Ayush Sekhari, Ohad Shamir, Nathan Srebro, Karthik Sridharan, Blake Woodworth
2019 arXiv   pre-print
We give nearly matching upper and lower bounds on the oracle complexity of finding ϵ-stationary points (∇ F(x) ≤ϵ) in stochastic convex optimization.  ...  We jointly analyze the oracle complexity in both the local stochastic oracle model and the global oracle (or, statistical learning) model.  ...  Acknowledgements We would like to thank Srinadh Bhojanapalli and Robert D. Kleinberg for helpful discussions.  ... 
arXiv:1902.04686v2 fatcat:ddl3n2zfcfardg2yjethtyozni

The Adaptive Complexity of Maximizing a Gross Substitutes Valuation

Ron Kupfer, Sharon Qian, Eric Balkanski, Yaron Singer
2020 Neural Information Processing Systems  
Both the upper and lower bounds are under the assumption that queries are only on feasible sets (i.e., of size at most k).  ...  Both lower bounds extend to the class of OXS functions.  ...  This near-optimal algorithm provides an exponential improvement in parallel runtime compared to previous algorithms for maximizing gross substitutes. We also provide two lower bounds.  ... 
dblp:conf/nips/KupferQBS20 fatcat:lpy5ta2nqzgibonsdabjcc3dxa

The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication [article]

Blake Woodworth, Brian Bullins, Ohad Shamir, Nathan Srebro
2021 arXiv   pre-print
We present a novel lower bound with a matching upper bound that establishes an optimal algorithm.  ...  of communication to optimize the objective, and during each round of communication, each machine may sequentially compute K stochastic gradient estimates.  ...  This work is also partially supported by NSF-CCF/BSF award 1718970/2016741, and NS is also supported by a Google Faculty Research Award.  ... 
arXiv:2102.01583v2 fatcat:yypfhqxtzjdvxn5kua3qtofbqu

Approximation Algorithms for Reliable Stochastic Combinatorial Optimization [chapter]

Evdokia Nikolova
2010 Lecture Notes in Computer Science  
A natural and important objective that incorporates risk in this stochastic setting is to look for a feasible solution whose stochastic cost has a small tail or a small convex combination of mean and standard  ...  Our models can be equivalently reformulated as nonconvex programs for which no efficient algorithms are known. In this paper, we make progress on these hard problems.  ...  Let s min and s max be a lower and upper bound respectively for the variance of the optimal solution.  ... 
doi:10.1007/978-3-642-15369-3_26 fatcat:odnjeoieenad7eahwd45abb3ru

Recent theoretical advances in decentralized distributed convex optimization [article]

Eduard Gorbunov, Alexander Rogozin, Aleksandr Beznosikov, Darina Dvinskikh, Alexander Gasnikov
2021 arXiv   pre-print
The lower bounds on communications rounds and oracle calls have appeared, as well as methods that reach both of these bounds.  ...  In this paper, we focus on how these results can be explained based on optimal algorithms for the non-distributed setup.  ...  Uribe and P. Dvurechensky for fruitful discussions.  ... 
arXiv:2011.13259v3 fatcat:dwqhc32ht5gqxc342sycoignsu

Scheduling (Dagstuhl Seminar 18101)

Magnús M. Halldórson, Nicole Megow, Clifford Stein, Michael Wagner
2018 Dagstuhl Reports  
The primary objective of the seminar was to expose each community to the important problems and techniques from the other community, and to facilitate dialog and collaboration between researchers.  ...  The seminar brought together algorithmically oriented researchers from two communities with interests in resource management: (i) the scheduling community and (ii) the networking and distributed computing  ...  Stochastic input models are a promising way to bypass lower bounds of worst-case analysis.  ... 
doi:10.4230/dagrep.8.3.1 dblp:journals/dagstuhl-reports/HalldorssonMS18 fatcat:xlfjfzwt3fdpdk3ryxr2ldra5m

Learning a Loopy Model For Semantic Segmentation Exactly [article]

Andreas Christian Mueller, Sven Behnke
2013 arXiv   pre-print
We show that our proposed method yields exact solutions with an optimality guarantees in a computer vision application, for little additional computational cost.  ...  As exact inference in loopy graphs is NP-hard in general, learning these models without approximations is usually deemed infeasible.  ...  Learning with any under-generating approach, we can use 1. to maintain a lower bound on the objective.  ... 
arXiv:1309.4061v1 fatcat:olaovxp4ofcvbazvkikdjyrooi

BUNDLE METHODS IN THE XXIst CENTURY: A BIRD'S-EYE VIEW

Welington de Oliveira, Claudia Sagastizábal
2014 Pesquisa Operacional  
Bundle methods are often the algorithms of choice for nonsmooth convex optimization, especially if accuracy in the solution and reliability are a concern.  ...  We adopt an approach that is by no means exhaustive, but covers different proximal and level bundle methods dealing with inexact oracles, for both unconstrained and constrained problems.  ...  ACKNOWLEDGMENTS We are grateful to Robert Mifflin for his careful reading and useful remarks.  ... 
doi:10.1590/0101-7438.2014.034.03.0647 fatcat:sixs6rtmjbeadhi2khfw75jjje

Compositional system-level design exploration with planning of high-level synthesis

Hung-Yi Liu, M. Petracca, L. P. Carloni
2012 2012 Design, Automation & Test in Europe Conference & Exhibition (DATE)  
The growing complexity of System-on-Chip (SoC) design calls for an increased usage of transaction-level modeling (TLM), high-level synthesis tools, and reuse of pre-designed components.  ...  The two algorithms are computationally efficient and enable an effective parallelization of the synthesis runs.  ...  Sloan Foundation fellowship, and the Gigascale Systems Research Center, one of six research centers funded under the Focus Center Research Program (FCRP), a Semiconductor Research Corporation entity.  ... 
doi:10.1109/date.2012.6176550 dblp:conf/date/LiuPC12 fatcat:pj5am7yjpbg6nnvmj5bfp7u3je

Adaptive Stochastic Dual Coordinate Ascent for Conditional Random Fields [article]

Rémi Le Priol, Alexandre Piché, Simon Lacoste-Julien
2018 arXiv   pre-print
In this paper, we adapt SDCA to train CRFs, and we enhance it with an adaptive non-uniform sampling strategy based on block duality gaps.  ...  SDCA enjoys a linear convergence rate and a strong empirical performance for binary classification problems. However, it has never been used to train CRFs.  ...  Acknowledgments We are thankful to Thomas Schweizer for his numerous software engineering advices. We thank Gauthier Gidel and Akram Erraqabi who started this project.  ... 
arXiv:1712.08577v2 fatcat:ocxs6k2jlzewzbvatauo4lwcey

Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression [article]

Xinmeng Huang, Yiming Chen, Wotao Yin, Kun Yuan
2022 arXiv   pre-print
To close the gap between the lower bound and the existing upper bounds, we further propose an algorithm, NEOLITHIC, which almost reaches our lower bound (up to logarithm factors) under mild conditions.  ...  In this paper, we consider distributed stochastic algorithms for minimizing smooth and non-convex objective functions under communication compression.  ...  Recently, [41] show a lower bound in the decentralized setting (for the linear graph) by assigning disjoint components of the model to remote nodes in the graph topology.  ... 
arXiv:2206.03665v1 fatcat:gdyqnri3tje2rb42u34kljexme
« Previous Showing results 1 — 15 out of 1,734 results