3,027 Hits in 3.3 sec

Mathematical Reasoning via Self-supervised Skip-tree Training [article]

Markus N. Rabe and Dennis Lee and Kshitij Bansal and Christian Szegedy
2020 arXiv   pre-print
We examine whether self-supervised language modeling applied to mathematical formulas enables logical reasoning.  ...  We find that models trained on the skip-tree task show surprisingly strong mathematical reasoning abilities, and outperform models trained on standard skip-sequence tasks.  ...  Self-supervised training techniques for formal mathematics have received much less attention.  ... 
arXiv:2006.04757v3 fatcat:yrmqpmijjzh6rcnkm3b77k3mwy

Towards the Automatic Mathematician [chapter]

Markus N. Rabe, Christian Szegedy
2021 Lecture Notes in Computer Science  
AbstractOver the recent years deep learning has found successful applications in mathematical reasoning.  ...  This extended abstract summarizes recent developments of machine learning in mathematical reasoning and the vision of the N2Formal group at Google Research to create an automatic mathematician.  ...  (skip-tree training) [41] .  ... 
doi:10.1007/978-3-030-79876-5_2 fatcat:w6hmshg5gzbabdfkyj5om3yeme

Proof Artifact Co-training for Theorem Proving with Language Models [article]

Jesse Michael Han, Jason Rute, Yuhuai Wu, Edward W. Ayers, Stanislas Polu
2022 arXiv   pre-print
We propose PACT (Proof Artifact Co-Training), a general methodology for extracting abundant self-supervised data from kernel-level proof terms for co-training alongside the usual tactic prediction objective  ...  We apply this methodology to Lean, an interactive proof assistant which hosts some of the most sophisticated formalized mathematics to date.  ...  Unlike skip-tree training , which focuses solely on predicting masked subterms of theorem statements, PACT derives its self-supervised training data from far more complex proofs.  ... 
arXiv:2102.06203v2 fatcat:ly2rlwm2erhjhakw42d2nht2gy

Survey of Neural Text Representation Models

Karlo Babić, Sanda Martinčić-Ipšić, Ana Meštrović
2020 Information  
Furthermore, we categorize these models by representation level, input level, model type, and model supervision.  ...  Tree-LSTM performs better than an RecNN with basic RNNs because of the same reasons LSTM outperforms RNN.  ...  Tree-LSTM [75] introduced a generalization of LSTM to tree-structured network topologies. The model is trained on a supervised sentiment task and it requires parse trees upon input.  ... 
doi:10.3390/info11110511 fatcat:veamykmme5cm5jhsllyc4xl7ma

Ensemble deep learning: A review [article]

M.A. Ganaie and Minghui Hu and A.K. Malik and M. Tanveer and P.N. Suganthan
2022 arXiv   pre-print
like bagging, boosting and stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous /heterogeneous ensemble, decision fusion strategies, unsupervised, semi-supervised  ...  Stochastic depth is an improvement on ResNet [13] wherein residual blocks are randomly dropped during training and bypassing these transformation blocks connections via skip connections.  ...  Let h j be the hypothesis generated on the training data D evaluated on test data (x, t), mathematically, h j (t|x) = P[y|x, h j , D].  ... 
arXiv:2104.02395v2 fatcat:lq73jqso5vadvnqfnnmw4zul4q

FoldingZero: Protein Folding from Scratch in Hydrophobic-Polar Model [article]

Yanjun Li, Hengtong Kang, Ketian Ye, Shuyu Yin, Xiaolin Li
2018 arXiv   pre-print
It is trained solely by a reinforcement learning algorithm, which improves HPNet and R-UCT iteratively through iterative policy optimization.  ...  In this paper, we propose a novel protein folding framework FoldingZero, self-folding a de novo protein 2D HP structure from scratch based on deep reinforcement learning.  ...  In FoldingZero, HPNet is trained in a supervised manner to match the R-UCT search results closely.  ... 
arXiv:1812.00967v1 fatcat:vjkzsbpjr5fczp4x4y3cod4onu

A Survey of Knowledge Enhanced Pre-trained Models [article]

Jian Yang, Gang Xiao, Yulong Shen, Wei Jiang, Xinyu Hu, Ying Zhang, Jinghui Peng
2022 arXiv   pre-print
Pre-trained models learn contextualized word representations on large-scale text corpus through a self-supervised learning method, which has achieved promising performance after fine-tuning.  ...  Pre-trained models with knowledge injection, which we call knowledge enhanced pre-trained models (KEPTMs), possess deep understanding and logical reasoning and introduce interpretability to some extent  ...  Transformers via a syntax-aware self-attention mechanism.  ... 
arXiv:2110.00269v2 fatcat:miinw6thcbasbjnvodp5535w5e

A Deep Learning Approach for a Source Code Detection Model Using Self-Attention

Yao Meng, Long Liu
2020 Complexity  
Finally, the representation model encodes the sequence of statement vectors via a bidirectional LSTM network, which is a classical deep learning framework, with a self-attention layer and outputs a vector  ...  The representation model firstly transforms the source code into an abstract syntactic tree and splits it into a sequence of statement trees; then, it encodes each of the statement trees with a deep-first  ...  It gives a reasonable prediction of code clone by comparing the code similarity in a supervised learning way.  ... 
doi:10.1155/2020/5027198 doaj:a8882994dfc040c8be3d2f74a2cc9ca6 fatcat:3khymjwlzngwbnvv3h6zj7ym5y

Contrastive Self-supervised Neural Architecture Search [article]

Nam Nguyen, J. Morris Chang
2021 arXiv   pre-print
First, using only a small amount of unlabeled train data under contrastive self-supervised learning allow us to search on a more extensive search space, discovering better neural architectures without  ...  Finally, we tackle the inherent discrete search space of the NAS problem by sequential model-based optimization via the tree-parzen estimator (SMBO-TPE), enabling us to reduce the computational expense  ...  Hence, it is not reasonable to compare supervised NAS and self-supervised NAS based on the test accuracy.  ... 
arXiv:2102.10557v3 fatcat:3qtpuha3njauhjpl72p7j4gyom

Diving Deep into Deep Learning:History, Evolution, Types and Applications

While machine learning is busy in supervised and unsupervised methods, deep learning continues its motivation for replicating the human nervous system by incorporating advanced types of Neural Networks  ...  Skip-gram is a model for training word embedding algorithms.  ...  When it is trained without supervision, it can learn to probabilistically restrict its inputs and acts as feature detectors and with supervision to perform classification [38] .  ... 
doi:10.35940/ijitee.a4865.019320 fatcat:orn2asvoxfaxvlc5iv7kec4nm4

Self-Supervised Damage-Avoiding Manipulation Strategy Optimization via Mental Simulation [article]

Tobias Fromm
2017 arXiv   pre-print
This learned manipulation strategy is continuously refined in a self-supervised optimization cycle dur- ing load-free times of the system.  ...  Such a simulation-in-the-loop setup is commonly known as mental simulation and allows for efficient, fully automatic generation of training data as opposed to classical supervised learning approaches.  ...  classifier and optimize them in a self-supervised manner.  ... 
arXiv:1712.07452v1 fatcat:n366fugsmbctxhqm2q6vc6musa

Semi-supervised emotion lexicon expansion with label propagation and specialized word embeddings [article]

Mario Giulianelli
2017 arXiv   pre-print
In a similar attempt, we learn task-specific word embeddings via a supervised task.  ...  As a consequence, the accuracy of a self-training classifier decrements at each iteration. A different method is transductive inference (Vapnik and Vapnik, 1998) .  ... 
arXiv:1708.03910v1 fatcat:uw3yfzjjfbfxldqmpoybfifzz4

Solos: A Dataset for Audio-Visual Music Analysis [article]

Juan F. Montesinos, Olga Slizovskaia, Gloria Haro
2020 arXiv   pre-print
, cross-modal correspondences, cross-modal generation and, in general, any audio-visual self-supervised task.  ...  URMP was intented to be used for source separation, thus, we evaluate the performance on the URMP dataset of two different source-separation models trained on Solos.  ...  Besides, it is possible to carry out self-supervised tasks in which one modality supervises the other one. This entails another research field, the cross-modal correspondence (CMC).  ... 
arXiv:2006.07931v2 fatcat:3fwx6dxifvczlp445ejr72fohu

Machine Learning on Graphs: A Model and Comprehensive Taxonomy [article]

Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, Kevin Murphy
2022 arXiv   pre-print
Specifically, we propose a Graph Encoder Decoder Model (GRAPHEDM), which generalizes popular algorithms for semi-supervised learning on graphs (e.g.  ...  The second, graph regularized neural networks, leverages graphs to augment neural network losses with a regularization objective for semi-supervised learning.  ...  The attention parameters are trained through backpropagation, and the GAT self-attention mechanism is: g k (H ) = LeakyReLU(H B b 0 ⊕ b 1 BH ) where ⊕ indicates summation of row and column vectors with  ... 
arXiv:2005.03675v3 fatcat:6eoicgprdvfbze732nsmpaumqe

Synergy of physics-based reasoning and machine learning in biomedical applications: towards unlimited deep learning with limited data

Valeriy Gavrishchaka, Olga Senyukova, Mark Koepke
2019 Advances in Physics: X  
Beyond obvious use in initial-factor selection, existing simplified models are effectively employed for generation of realistic synthetic data for later DNN pre-training.  ...  We outline our hybrid framework that leverages existing domain-expert models/knowledge, boosting-like model combination, DNN-based DL and other machine learning algorithms for drastic reduction of training-data  ...  After that, DNN could be further fine-tuned via supervised training using available labeled data.  ... 
doi:10.1080/23746149.2019.1582361 fatcat:wkmef4jmgreurnseofsaqa5dva
« Previous Showing results 1 — 15 out of 3,027 results