MISA

Fusion starts with a representation problem

Language, facial motion and acoustics describe the same utterance, but their feature distributions are not naturally comparable. A common-space model can align them by discarding their differences. A separate-stream model can preserve those differences but leave the fusion layer to bridge the entire modality gap.

MISA keeps both views. For every utterance, each modality is projected once into a modality-invariant space and once into a modality-specific space. The shared vectors are trained to capture cross-modal regularities, such as the speaker’s common communicative intent. The private vectors retain cues that need not occur elsewhere, such as a vocal pattern or a speaker-specific facial expression.

Six vectors, four losses and one attention block

Text is encoded with BERT; visual and acoustic sequences use stacked bidirectional LSTMs. A shared feed-forward encoder maps all three utterance vectors into the invariant space. Three separate encoders map them into language-, vision- and audio-specific spaces. This produces six vectors.

Four objectives give the split a testable meaning:

1. Task loss trains sentiment regression or classification, or humour classification.
2. Similarity loss uses Central Moment Discrepancy to align the distributions of the three invariant vectors, matching higher-order moments without an adversarial discriminator.
3. Difference loss makes each invariant vector approximately orthogonal to its private counterpart and makes the three private spaces approximately orthogonal to one another.
4. Reconstruction loss decodes the sum of a modality’s shared and private vectors back to its original representation, preventing the private encoder from satisfying orthogonality with an empty or arbitrary code.

Multi-head self-attention operates over the six-vector sequence. The attended vectors are concatenated and passed to the prediction head. MISA therefore changes what fusion receives; it does not depend on an elaborate tensor fusion operator.

Three datasets test sentiment and humour

CMU-MOSI contains 2,198 subjective utterance-video segments from 93 videos and 89 speakers, with sentiment labels from −3 to +3. CMU-MOSEI expands this to 23,453 segments from 5,000 videos, 1,000 speakers and 250 topics. UR-FUNNY changes the target from sentiment to humour, testing whether the representation split transfers to another affective task.

The paired binary scores follow the paper’s two conventions: negative/non-negative and negative/positive with neutral examples excluded. On UR-FUNNY, BERT-based MISA reached 70.61% accuracy, compared with 68.57% for the BERT-based TFN baseline reported in the same experiment.

The ablations test the split, not only the final score

Full MISA achieved 0.783 MAE and 0.761 correlation on MOSI. Its plain three-encoder base, without the shared/private factorisation, reached 0.810 and 0.750. On MOSEI the corresponding figures were 0.555/0.756 for MISA and 0.568/0.752 for the base. Using only invariant representations, only private representations, or learning both but fusing only one set all underperformed the complete design. Removing the similarity, difference or reconstruction loss also reduced performance, with the model more sensitive to the first two.

The strongest ablation is modality removal. Without language, MOSI performance fell from 0.783 MAE and 0.761 correlation to 1.450 and 0.041; MOSEI fell from 0.555/0.756 to 0.801/0.090. Removing vision or audio caused much smaller changes. The authors attribute part of this dominance to manually transcribed text and pretrained BERT, compared with noisier hand-engineered nonverbal features and randomly initialised recurrent encoders. They explicitly treat it as a property of these datasets and encoders, not a universal ranking of modalities.

The t-SNE plots are a qualitative check, not proof of disentanglement. MISA is evaluated on word-aligned affect datasets, and “invariant” means distributionally aligned under CMD rather than semantically identical. The method also inherits the annotation, feature-quality and language biases of MOSI, MOSEI and UR-FUNNY. Its result is narrower and more useful: on these tasks, forcing all information into one shared space was less effective than preserving shared and modality-specific representations together.