Multimodal-InfoMax

A correct label does not show what fusion discarded

A sentiment model can get the label right by following the transcript while learning very little from the face or voice. If that shortcut works during training, the prediction loss gives it little reason to preserve the other signals. Multimodal-InfoMax (MMIM) adds objectives that reward retaining information from each modality. Because mutual information is difficult to compute directly, the method trains with tractable lower bounds at two places in the network.

The experiments use unaligned CMU-MOSI and CMU-MOSEI sequences. BERT encodes text; separate LSTMs encode COVAREP acoustic features and visual features aligned with P2FA. A feed-forward fusion network produces a 128-dimensional vector and a sentiment regressor. The task loss is mean absolute error.

The first level connects text to vision and audio

At the input level, MMIM optimises Barber–Agakov lower bounds for two pairs: text–vision and text–audio. Text is used to predict the mean and variance of a Gaussian distribution over the lower-dimensional nonverbal representation. The log likelihood supplies one part of the bound.

The remaining entropy term is estimated with a two-component Gaussian mixture, one component for negative examples and one for non-negative examples. Estimating the covariance from a small minibatch is unstable, so MMIM keeps recent visual and acoustic embeddings in a first-in, first-out queue. The current batch and cached batches are used together for the Gaussian-mixture estimate; after the update, the oldest batch leaves the queue.

The paper chooses text as the predictor because its 768-dimensional BERT representation maps more readily to the visual and acoustic vectors, each below 50 dimensions, and because text was already the dominant signal in these benchmarks. It does not optimise a direct vision–audio lower bound in the final configuration.

The second level asks the fused vector to reconstruct each modality

At the fusion level, three contrastive predictive coding losses connect the fused representation to text, vision and audio. A small network predicts each modality vector from the fused vector. The matching vector is the positive example; other examples of the same modality in the current minibatch are negatives. Normalising both vectors prevents the score from increasing only by stretching their norms.

Training alternates between fitting the conditional predictors used by the input-level bound and updating the encoders, fusion network and task head. The final objective is

MAE + α · fusion-level CPC + β · input-level BA.

The two levels have different jobs. The input objective encourages text-correlated signal to survive in the visual and acoustic encoders. The fusion objective makes the final vector predictive of each modality instead of supervising it only through the sentiment label.

Five repeated runs on MOSI and MOSEI

The paper reports the mean of five runs under fixed hyperparameters. It evaluates regression with MAE and correlation, seven-class accuracy, and binary accuracy/F1 under both neutral-included and neutral-excluded conventions.

The MOSEI ablations identify where the gain comes from. Removing both input-level mutual-information terms changed MAE from 0.526 to 0.541, correlation from 0.772 to 0.752 and seven-class accuracy from 54.24 to 53.57. Removing all three fusion-level CPC terms produced 0.543, 0.759 and 53.49. Replacing the polarity-conditioned mixture with one Gaussian produced 0.533, 0.768 and 53.4. Removing the FIFO history data made covariance estimation fail with NaN values under the reported implementation.

What the extra objectives can preserve

One paper example shows where the model still fails: the phrase “I’m sorry” pulls the prediction in the wrong direction when neither the face nor the acoustics provides a corrective cue. Maximising dependence cannot create evidence that a modality does not contain.

The ablations show that these extra training objectives help on MOSI and MOSEI, but their construction is tied to the task. The entropy estimate divides examples into two polarity-conditioned Gaussian components, assuming equal priors and approximately disjoint distributions; the FIFO queue stabilises it with embeddings from nearby training steps. Those choices explain both how the method works and why adapting it to another problem requires more than reusing the loss. A task without sentiment polarity, or with missing modalities or long video, would need its own evaluation of what the estimates preserve.