Deep Modality Invariant Adversarial Network for Shared Representation Learning.

We call our method the Deep Modality Invariant Adversarial Network (DeMIAN). We demonstrate the effectiveness of our method in experiments. ... In this work, we propose a novel method to learn the mapping to the common space wherein different modalities have the same information for shared representation learning. ... Conclusion In this paper, we proposed a novel method to learn modality-invariant representations for shared representation learning, called the Deep Modal Invariant Adversarial Network (DeMIAN). ...

doi:10.1109/iccvw.2017.311 dblp:conf/iccvw/HaradaSMU17 fatcat:ys3e7grypjctxlczcu3j24oode

In particular, we propose a novel algorithm for modality-invariant representation learning, named Deep Modality Invariant Adversarial Network (DeMIAN), which utilizes the idea of Domain Adaptation (DA) ... Using the modality-invariant representations learned by DeMIAN, we achieved better classification accuracy than with the state-of-the-art methods, especially for some benchmark datasets of zero-shot learning ... This work was funded by ImPACT Program of Council for Science, Technology and Innovation (Cabinet Office, Government of Japan) and supported by CREST, JST. ...

arXiv:1612.07976v2 fatcat:dyr3t25yfvf3vm2px2asfylv64

Multiple Versions

Recently, deep neural networks have exhibited as a powerful architecture to well capture the nonlinear distribution of high-dimensional multimedia data, so naturally does for multi-modal data. ... Throughout this survey, we further indicate that the critical components for this field go to collaboration, adversarial competition and fusion over multi-modal spaces. ... Deep Modality Invariant Adversarial Network (DeMIAN) [38] aimed to learn modality-invariant representations from paired modalities, while enabling to train a classifier on source modality samples, which ...

arXiv:2006.08159v1 fatcat:g4467zmutndglmy35n3eyfwxku

INDEX TERMS Multimodal representation learning, multimodal deep learning, deep multimodal fusion, multimodal translation, multimodal adversarial learning. ... This paper highlights on the key issues of newly developed technologies, such as encoder-decoder model, generative adversarial networks, and attention mechanism in a multimodal representation learning ... Two methods used for improving modality-invariant property via adversarial learning. ...

doi:10.1109/access.2019.2916887 fatcat:ms4wcgl5rncsbiywz27uss4ysq

DOAJ

In addition, we propose a view-adversarial training method to enhance learning of view-invariant features. ... By learning to extrapolate cross-view motions, the representation can capture view-invariant motion dynamics which is discriminative for the action. ... The second category uses deep networks to jointly learn feature representation and classifier. ...

arXiv:1809.01844v1 fatcat:u4w3kzv6ofaplkawtapmkscisi

These two losses enhance domain-invariant feature learning and modality independent noise removing. ... This framework integrates cross-spectral face hallucination and discriminative feature learning into an end-to-end adversarial network. ... In , Restricted Boltzmann Machines (RBMs) is used to learn a shared representation between different modalities. ...

arXiv:1709.03675v1 fatcat:gnjwnkszrfgnlikmla6vgqnqhm

Adversarial learning methods are a promising approach to training robust deep networks, and can generate complex samples across diverse domains. ... We first outline a novel generalized framework for adversarial adaptation, which subsumes recent state-of-the-art approaches as special cases, and we use this generalized view to better relate the prior ... Conclusion We have proposed a unified framework for unsupervised domain adaptation techniques based on adversarial learning objectives. ...

arXiv:1702.05464v1 fatcat:zzepyxgtyzbanhxrrehknzeexa

These two losses enhance domain-invariant feature learning and modality independent noise removing. ... This framework integrates cross-spectral face hallucination and discriminative feature learning into an end-to-end adversarial network. ... In (Yi, Lei, and Li 2015) , Restricted Boltzmann Machines (RBMs) is used to learn a shared representation between different modalities. ...

doi:10.1609/aaai.v32i1.12291 fatcat:l26znnxzfbds7lpqqlcvmsqkqq

In this paper, we present a novel Adversarial Cross-Modal Retrieval (ACMR) method, which seeks an effective common subspace based on adversarial learning. ... The first process, a feature projector, tries to generate a modality-invariant representation in the common subspace and to confuse the other process, modality classifier, which tries to discriminate between ... With ACMR, we combine for the first time the concepts of supervised representation learning for cross-modal retrieval and adversarial learning. ...

doi:10.1145/3123266.3123326 dblp:conf/mm/WangYXHS17 fatcat:7pbuzhkqsbbt7bmhyj45qazyea

To address these challenges posited by the heterogeneity gap and the semantic gap, we propose integrating Shannon information theory and adversarial learning. ... For this purpose, a modality classifier (as a discriminator) is built to distinguish the text and image modalities according to their different statistical properties. ... We would like to thank Theodoros Georgiou and Nan Pu for several discussions and thank NVIDIA for the donation of GPU cards. ...

doi:10.1016/j.patcog.2021.107983 fatcat:jzx5h5yfzvhvphpzdycowd5c24

Adversarial learning methods are a promising approach to training robust deep networks, and can generate complex samples across diverse domains. ... We then propose a previously unexplored instance of our general framework which combines discriminative modeling, untied weight sharing, and a GAN loss, which we call Adversarial Discriminative Domain ... Introduction Deep convolutional networks, when trained on large-scale datasets, can learn representations which are generically usefull across a variety of tasks and visual domains [1, 2] . ...

doi:10.1109/cvpr.2017.316 dblp:conf/cvpr/TzengHSD17 fatcat:lpnyganiercatbsofc5z326vhe

Deep learning has empowered analysis for single-cell sequencing data in many ways and has generated deep understanding about a range of complex cellular systems. ... of high dimensional data that come from different sources and represent cellular systems with different features, there is an equivalent rise and challenge of integrating single-cell sequence across modalities ... Unsupervised representation learning Deep metric learning has shown effective representation learning without supervision. ...

arXiv:2104.12320v1 fatcat:sb6cokdqpbhm5cyu3nxcgo5f6a

There has been a rapid development of deep learning (DL) models for medical imaging. However, DL requires a large labeled dataset for training the models. ... Using adversarial techniques, unsupervised DA has achieved good performance, especially for segmentation tasks. ... to learn a domain-invariant feature representation. ...

doi:10.1055/s-0040-1702009 pmid:32823306 fatcat:gtlhoh6m3fh4hcumfzdlpdohr4

Open Access

The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. ... In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. ... Similar to the third category, we also learn common modality-invariant subspaces. However, we do not use adversarial discriminators to learn shared mappings. ...

arXiv:2005.03545v3 fatcat:nyomobnpojcefpyllea3scjdpq

Open Access Multiple Versions

In this paper, we highlight key points of recent cross-modal retrieval approaches based on deep-learning, especially in the image-text retrieval context, and classify them into four categories according ... Cross-modal retrieval aims to retrieve relevant information across different modalities. ... After that, Sarafianos et al. propose Text-Image Modality Adversarial Matching (TIMAM [34] ), which adopts an Adversarial Representation Learning (ARL) framework to learn modality-invariant representations ...

doi:10.1109/mipr49039.2020.00042 dblp:conf/mipr/ChenZBK20 fatcat:fps5wiw4ezf7teko3vegaxq4tq

Deep Modality Invariant Adversarial Network for Shared Representation Learning

Preserved Fulltext

DeMIAN: Deep Modality Invariant Adversarial Network [article]

Preserved Fulltext

Other Versions

Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion [article]

Preserved Fulltext

Deep multimodal representation learning: a survey

Preserved Fulltext

Unsupervised Learning of View-invariant Action Representations [article]

Preserved Fulltext

Adversarial Discriminative Heterogeneous Face Recognition [article]

Preserved Fulltext

Adversarial Discriminative Domain Adaptation [article]

Preserved Fulltext

Adversarial Discriminative Heterogeneous Face Recognition

Preserved Fulltext

Adversarial Cross-Modal Retrieval

Preserved Fulltext

Integrating Information Theory and Adversarial Learning for Cross-modal Retrieval

Preserved Fulltext

Adversarial Discriminative Domain Adaptation

Preserved Fulltext

Integration of Unpaired Single-cell Chromatin Accessibility and Gene Expression Data via Adversarial Learning [article]

Preserved Fulltext

Advancing Medical Imaging Informatics by Deep Learning-Based Domain Adaptation

Preserved Fulltext

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis [article]

Preserved Fulltext

Other Versions

Review of Recent Deep Learning Based Methods for Image-Text Retrieval

Preserved Fulltext