Representation Ensembling for Synergistic Lifelong Learning with Quasilinear Complexity [article]

Joshua T. Vogelstein, Jayanta Dey, Hayden S. Helm, Will LeVine, Ronak D. Mehta, Tyler M. Tomita, Haoyin Xu, Ali Geisa, Qingyang Wang, Gido M. van de Ven, Chenyu Gao, Weiwei Yang (+4 others)
2024 arXiv   pre-print
In lifelong learning, data are used to improve performance not only on the current task, but also on previously encountered, and as yet unencountered tasks. While typical transfer learning algorithms can improve performance on future tasks, their performance on prior tasks degrades upon learning new tasks (called forgetting). Many recent approaches for continual or lifelong learning have attempted to maintain performance on old tasks given new tasks. But striving to avoid forgetting sets the
more » ... l unnecessarily low. The goal of lifelong learning should be to use data to improve performance on both future tasks (forward transfer) and past tasks (backward transfer). Our key insight is that we can ensemble representations that were learned independently on disparate tasks to enable both forward and backward transfer, with algorithms that run in quasilinear time. Our algorithms demonstrate both forward and backward transfer in a variety of simulated and benchmark data scenarios, including tabular, vision (CIFAR-100, 5-dataset, Split Mini-Imagenet, and Food1k), and audition (spoken digit), including adversarial tasks, in contrast to various reference algorithms, which typically failed to transfer either forward or backward, or both.
arXiv:2004.12908v18 fatcat:nxkqtdhb5zdonh2uaxmpcphhoy