Contrastive Code Representation Learning.

Recent work learns contextual representations of source code by reconstructing tokens from their context. ... We propose ContraCode: a contrastive pre-training task that learns code functionality, not form. ... The augmentations discussed in Section 3.1 enable adapting recent contrastive learning objectives for images to code representation learning. ...

arXiv:2007.04973v3 fatcat:bpqzjwtoebhh3in5q7qkk4sggq

Open Access Multiple Versions

In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. ... While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four ... Predictive Coding, the proposed representation learning approach. ...

arXiv:1807.03748v2 fatcat:l7bjdp4x7bdg3pprz42rzi3zqq

Multiple Versions

In recent years self-supervised learning has emerged as a promising candidate for unsupervised representation learning. ... In this work, we conduct an extensive analysis of the applicability of self-supervised learning in remote sensing image classification. ... Self-supervised pre-training As the first step in our experiments, we use Contrastive Multiview Coding -CMC [36] for self-supervised pretraining. ...

arXiv:2104.07070v2 fatcat:irqhsya535galjojxuy3dn57ci

Multiple Versions

In this paper, we propose HELoC, a hierarchical contrastive learning model for source code representation. ... syntax trees (ASTs) play a crucial role in source code representation. ... In this paper, we present HELoC (short for Hierarchical Con-strastivE Learning of Code), a hierarchical contrastive learning model for source code representation. ...

arXiv:2203.14285v1 fatcat:ulclwu5e3ffxtivec76hblhosu

To foster disentanglement, we propose adversarial contrastive predictive coding. This new disentanglement method does neither need parallel data nor any supervision. ... We further demonstrate an increased robustness of the content representation against a train-test mismatch compared to spectral features, when used for phone recognition. ... CONTRASTIVE PREDICTIVE CODING SUPPORTED FACTORIZED VARIATIONAL AUTOENCODER To learn disentangled representations of style and content, we propose a fully convolutional FVAE which is illustrated in Fig ...

arXiv:2005.12963v2 fatcat:qznumov53bau7ptmntlh3g74le

Multiple Versions

A convolutional neural network learns frame-level representation from the raw waveform via noise-contrastive estimation (NCE). ... Recent attempts employ self-supervised learning, such as contrastive predictive coding (CPC), where the next frame is predicted given past context. ... Recent attempts employ self-supervised learning, such as contrastive predictive coding (CPC), where the next frame is predicted given past context. ...

arXiv:2110.02345v2 fatcat:ecmpikwm5bggjj72alu57thhea

Multiple Versions

learning set-up. ... Using this set of labeled data, we show that a popular natural language modeling processing pipeline, namely vector representation of words, a.k.a word embeddings, followed by Long Short Term Memory (LSTM ... Given this set-up, our approach is to learn useful representations via contrastive learning using the triplet loss [21, 22] . ...

arXiv:2011.13384v2 fatcat:ybywlsghfrcpzhlgkxj7aajfaq

Multiple Versions

Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound on the mutual information (MI) between different views of an instance ... [Code is available at: ] ... APPROACH We present cross-modal active contrastive coding (CM-ACC) to learn audio-visual representations from unlabeled videos. Fig. 1 highlights the main idea of our approach. ...

arXiv:2009.09805v2 fatcat:oybaqjvq6fefbbshaclckrqc6a

Multiple Versions

Code representation learning, which aims to encode the semantics of source code into distributed vectors, plays an important role in recent deep-learning-based models for code intelligence. ... To further explore the properties of programming languages, this paper proposes SynCoBERT, a syntax-guided multi-modal contrastive pre-training approach for better code representations. ... We propose a multi-modal contrastive learning (MCL) objective to obtain more comprehensive representations by learning from three modalities (code, comment, and AST) through contrastive learning. ...

arXiv:2108.04556v3 fatcat:vhp2fpfkpnh7lpfomyn5yqwyba

Multiple Versions

To address these limitations, we are proposing Corder, a self-supervised contrastive learning framework that trains code representation models on unlabeled data. ... The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through a contrastive learning paradigm. ... In this case, it should map˜and˜into two code vectors and , respectively. • A contrastive loss function is defined for the contrastive learning task. ...

arXiv:2009.02731v5 fatcat:sdyhezkr4rhmbpmgoyssrtiu5i

Multiple Versions

A simple framework ference on Learning Representations. for contrastive learning of visual representations. ... Contrastive Code Representation Learning Paras Jain∗ and Ajay Jain∗ and Tianjun Zhang and Pieter Abbeel and Joseph E. ...

doi:10.18653/v1/2021.emnlp-main.482 fatcat:ojvnpzix5fc4rffg6tqxg3epza

Knowledge representation learning (KRL) has been used in plenty of knowledge-driven tasks. ... In this paper, we propose Contrastive Completion Coding (C 3 ), a novel KRL framework that is composed of two functional components: 1. ... Contrastive Loss Contrastive losses measure the distance, or similarity, between representations in the latent space, which is one of the key differences between contrastive learning methods and other ...

doi:10.18653/v1/2021.findings-emnlp.263 fatcat:nmb4pp7vqvcbtpn225yjxvmm5a

via multi-level graph contrastive learning framework. ... Specifically, KnowAugNet first leverages the graph contrastive learning using graph attention network as the encoder to capture the implicit relations between homogeneous medical codes from the medical ... incorporate the unsupervised graph contrastive learning method based on DGI [30] to learn the medical codes representations by maximizing the mutual information of graph representation for the first ...

arXiv:2204.11736v2 fatcat:vvncqo2tejgwrbpo2cxpstqdt4

Open Access Multiple Versions

In this paper, we propose the CodeRetriever model, which combines the unimodal and bimodal contrastive learning to train function-level code semantic representations, specifically for the code search task ... For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build text-code pairs. ... It shows that the unimodal contrastive learning helps to learn a unified representation space of code with different programming languages. ...

arXiv:2201.10866v1 fatcat:zywntl7z3zhuleci7ic4ymegru

We seek to improve the performance for both frequent and rare ICD codes by using a contrastive graph-based EHR coding framework, CoGraph, which re-casts EHR coding as a few-shot learning task. ... To mitigate this risk, CoGraph devises two graph contrastive learning schemes, GSCL and GECL, that exploit the HEWE graph structures so as to encode transferable features. ... Graph contrastive learning extends the paradigm to representation learning on graphs. Peng et al. ...

arXiv:2106.15467v1 fatcat:3tnzxc6qgzduli3fcmepxwi3ti

Open Access

Contrastive Code Representation Learning [article]

Preserved Fulltext

Other Versions

Representation Learning with Contrastive Predictive Coding [article]

Preserved Fulltext

Other Versions

Self-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Coding [article]

Preserved Fulltext

Other Versions

HELoC: Hierarchical Contrastive Learning of Source Code Representation [article]

Preserved Fulltext

Contrastive Predictive Coding Supported Factorized Variational Autoencoder for Unsupervised Learning of Disentangled Speech Representations [article]

Preserved Fulltext

Other Versions

Unsupervised Speech Segmentation and Variable Rate Representation Learning using Segmental Contrastive Predictive Coding [article]

Preserved Fulltext

Other Versions

Automatic coding of students' writing via Contrastive Representation Learning in the Wasserstein space [article]

Preserved Fulltext

Other Versions

Active Contrastive Learning of Audio-Visual Video Representations [article]

Preserved Fulltext

SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation [article]

Preserved Fulltext

Other Versions

Self-Supervised Learning for Code Retrieval and Summarization through Semantic-Preserving Program Transformations [article]

Preserved Fulltext

Other Versions

Contrastive Code Representation Learning

Preserved Fulltext

Knowledge Representation Learning with Contrastive Completion Coding

Preserved Fulltext

KnowAugNet: Multi-Source Medical Knowledge Augmented Medication Prediction Network with Multi-Level Graph Contrastive Learning [article]

Preserved Fulltext

Other Versions

CodeRetriever: Unimodal and Bimodal Contrastive Learning [article]

Preserved Fulltext

Few-Shot Electronic Health Record Coding through Graph Contrastive Learning [article]

Preserved Fulltext