Tensor Fusion Network

The same words can change with a face or a voice

“This movie is sick” is ambiguous as text. A smile can make it positive; a frown can make it negative; a loud voice can intensify the judgement when the other cues agree. Concatenating the three feature vectors gives a network access to every modality, but it does not explicitly show which dimensions interact.

Tensor Fusion Network (TFN) makes those products visible. It was developed for utterance-level sentiment in opinion videos, where language, facial behaviour and acoustics can reinforce or contradict one another.

Three modality encoders produce one vector each

The language encoder reads 300-dimensional GloVe vectors with an LSTM. A following feed-forward layer produces a 128-dimensional spoken-language vector. The visual pipeline samples the face at 30 Hz and extracts emotion indicators, 20 facial action units, head pose and 68 landmarks with FACET and OpenFace. Mean-pooled visual features pass through three 32-unit ReLU layers. The acoustic pipeline samples at 100 Hz and uses COVAREP features including MFCCs, pitch, voiced/unvoiced measures and glottal-source parameters; another three-layer network produces a 32-dimensional vector.

These encoders handle dynamics within a modality. TFN then appends the scalar 1 to each vector and computes

[language; 1] ⊗ [vision; 1] ⊗ [audio; 1].

With 128 language dimensions and 32 dimensions for both vision and audio, the result is a 129 × 33 × 33 tensor: 140,481 coordinates. The appended constants are what preserve lower-order terms. One region contains language alone, one vision alone and one audio alone; three more contain the pairwise products; the final region contains the three-way products. The outer product has no learned parameters. Two 128-unit layers after it learn how those coordinates predict sentiment.

Early fusion is contained inside this construction: its concatenated unimodal terms are three of the tensor’s seven regions. The scientific question is whether the additional product regions carry useful evidence.

The experiment keeps speakers out of both train and test

The evaluation uses CMU-MOSI: 2,199 opinion segments from 93 YouTube videos and 89 distinct speakers. Five annotators rate each segment on a seven-point scale from −3 to +3; agreement is Krippendorff’s α = 0.77. The paper evaluates binary sentiment, five-class sentiment and continuous regression. All systems use the same five-fold cross-validation, and no speaker appears in both the training and test split.

The controlled variants help locate the gain. A learned network over concatenated features reached 75.2 binary accuracy, 76.2 F1, 39.0 five-class accuracy, 0.96 MAE and 0.63 correlation. Keeping unimodal and bimodal tensor regions but removing the trimodal region reached 75.3, 76.2, 39.7, 0.919 and 0.66. Full TFN reached 77.1, 77.9, 42.0, 0.87 and 0.70. Both explicit cross-modal products and the three-way region contributed under this protocol.

The qualitative cases show when nonverbal evidence moves the answer

The paper compares predictions from TFN with early fusion and single-modality models. In one segment, a frown makes a weakly negative statement easier to classify. In another, the transcript contains an opaque reference to “a B,” while facial and acoustic cues make the positive judgement clearer. A strongly negative face can pull an otherwise positive sentence towards neutral. These examples support the interaction hypothesis without implying that voice or face always dominates language.

The boundary of the result is equally concrete. TFN was tested on one small, English, movie-review dataset using hand-engineered visual and acoustic features. The full tensor grows multiplicatively with modality dimension; replacing the 32-dimensional streams with modern high-dimensional encoders would make the representation impractical without factorisation or low-rank approximation. TFN established that explicit high-order interactions can outperform concatenation in this setting. It did not establish that a full outer product is the right fusion mechanism for every modality or task.