Domain Adaptation with Randomized Multilinear Adversarial Networks

[leo-p]

https://arxiv.org/pdf/1705.10667.pdf

Adversarial learning has been successfully embedded into deep networks to learn transferable features for domain adaptation, which reduce distribution discrepancy between the source and target domains and improve generalization performance. Prior domain adversarial adaptation methods could not align complex multimode distributions since the discriminative structures and inter-layer interactions across multiple domain-specific layers have not been exploited for distribution alignment. In this paper, we present randomized multilinear adversarial networks (RMAN), which exploit multiple feature layers and the classifier layer based on a randomized multilinear adversary to enable both deep and discriminative adversarial adaptation. The learning can be performed by stochastic gradient descent with the gradients computed by back-propagation in linear-time. Experiments demonstrate that our models exceed the state-of-the-art results on standard domain adaptation datasets.

[leo-p]

Summary:

• Dataset: Office and ImageCLEF-DA1.
• Objective: Perform domain-adaptation by adapting several layers using a randomized representation and not just the final layer thus performing alignment of the joint distribution and not just the marginals.

Inner-workings:

Basically an improvement on RevGrad where instead of using the last embedding layer for the discriminator, a bunch of them is used.
To avoid dimension explosion when using the tensor product of all layers they instead use a randomized multi-linear representation:

Where:

• d is the dimension of the embedding (they use 1024)
• R is random matrix for which each element as a null average and variance of 1 (Bernoulli, Gaussian and Uniform are tried)
• z^l is the l-th layer
• ⊙ represents the Hadamard product
  In practice they don't use all layers but just the 3-4 last layers for ResNet and AlexNet.

Architecture:

They use the usual losses for domain adaptation with: - F minimizing the cross-entropy loss for classification and trying to reduce the gap between the distributions (indicated by D). - D maximizing the gap between the distributions.

Results:

Improvement on state-of-the-art results for most tasks in the dataset, very easy to implement with any pre-trained network out of the box.