Multimodal-InfoMax

A correct label does not show what fusion discarded

A sentiment model can minimise its prediction loss while its fused vector largely follows the transcript and neglects the face or voice. Multimodal-InfoMax (MMIM) adds auxiliary objectives that make information retention part of training. It does not compute mutual information exactly. It optimises tractable lower bounds at two places in the network and tests whether they improve the downstream task.

The experiments use unaligned CMU-MOSI and CMU-MOSEI sequences. BERT encodes text; separate LSTMs encode COVAREP acoustic features and visual features aligned with P2FA. A feed-forward fusion network produces a 128-dimensional vector and a sentiment regressor. The task loss is mean absolute error.

The first level connects text to vision and audio

At the input level, MMIM optimises Barber–Agakov lower bounds for two pairs: text–vision and text–audio. Text is used to predict the mean and variance of a Gaussian distribution over the lower-dimensional nonverbal representation. The log likelihood supplies one part of the bound.

The remaining entropy term is estimated with a two-component Gaussian mixture, one component for negative examples and one for non-negative examples. Estimating the covariance from a small minibatch is unstable, so MMIM keeps recent visual and acoustic embeddings in a first-in, first-out queue. The current batch and cached batches are used together for the Gaussian-mixture estimate; after the update, the oldest batch leaves the queue.

The paper chooses text as the predictor because its 768-dimensional BERT representation maps more readily to the visual and acoustic vectors, each below 50 dimensions, and because text was already the dominant signal in these benchmarks. It does not optimise a direct vision–audio lower bound in the final configuration.

The second level asks the fused vector to reconstruct each modality

At the fusion level, three contrastive predictive coding losses connect the fused representation to text, vision and audio. A small network predicts each modality vector from the fused vector. The matching vector is the positive example; other examples of the same modality in the current minibatch are negatives. Normalising both vectors prevents the score from increasing only by stretching their norms.

Training alternates between fitting the conditional predictors used by the input-level bound and updating the encoders, fusion network and task head. The final objective is

MAE + α · fusion-level CPC + β · input-level BA.

The two levels have different jobs. The input objective encourages text-correlated signal to survive in the visual and acoustic encoders. The fusion objective makes the final vector predictive of each modality instead of supervising it only through the sentiment label.

Five repeated runs on MOSI and MOSEI

The paper reports the mean of five runs under fixed hyperparameters. It evaluates regression with MAE and correlation, seven-class accuracy, and binary accuracy/F1 under both neutral-included and neutral-excluded conventions.

The MOSEI ablations identify where the gain comes from. Removing both input-level mutual-information terms changed MAE from 0.526 to 0.541, correlation from 0.772 to 0.752 and seven-class accuracy from 54.24 to 53.57. Removing all three fusion-level CPC terms produced 0.543, 0.759 and 53.49. Replacing the polarity-conditioned mixture with one Gaussian produced 0.533, 0.768 and 53.4. Removing the FIFO history data made covariance estimation fail with NaN values under the reported implementation.

The estimator is also the limitation

One paper example shows where the model still fails: the phrase “I’m sorry” pulls the prediction in the wrong direction when neither the face nor the acoustics provides a corrective cue. Maximising dependence cannot create evidence that a modality does not contain.

The mutual-information values are lower-bound estimates, not direct measurements of all information in the inputs. The Gaussian-mixture entropy calculation assumes two polarity-conditioned components with equal priors and approximately disjoint distributions. The FIFO queue mixes embeddings produced at nearby optimisation steps to stabilise that estimate. These are practical modelling choices whose success is shown on sentiment prediction, not guarantees of semantic preservation. The evidence is confined to MOSI and MOSEI; it does not establish the same estimator for images, long video, missing modalities or tasks without a polarity structure.