Emotion lives in the conversation

The problem: the same words can carry different emotions

“Fine” can be agreement, resignation or anger. In MELD, 42% of utterances contain fewer than five words, so the current sentence often does not contain enough evidence by itself. The speaker’s earlier turns, another person’s reply, vocal delivery and facial expression can all change the label.

MELD keeps the exchange intact

MELD contains 13,708 utterances from 1,433 dialogues in the television series Friends: 9,989 utterances for training, 1,109 for development and 2,610 for testing. Dialogues include as many as nine speakers. Every utterance has a transcript, audio and video, plus one of seven emotion labels and one of three sentiment labels. Three graduate annotators watched the clips, rather than reading only the transcript, and the released label follows majority vote.

The distribution is sharply imbalanced. Across the three splits, 6,436 utterances are neutral, while only 358 are fear and 361 are disgust. The source is also scripted television, so the benchmark does not represent every culture, setting or speaking style.

DialogueRNN keeps one state per participant

DialogueRNN separates three jobs that a single sequence model would mix together. A global GRU summarizes the conversation; a party GRU updates only the state of the person speaking; and an emotion GRU turns that speaker state into the representation used for classification. Attention over earlier global states lets the current turn use relevant utterances from any participant. The bidirectional variant also reads future turns, which is useful for offline annotation but unavailable in a live system.

The original DialogueRNN experiments used speaker-disjoint train/test partitions of IEMOCAP and AVEC. On IEMOCAP text, DialogueRNN reached 59.89 weighted F1, compared with 56.13 for the Conversational Memory Network baseline. Adding bidirectional context and emotion-level attention raised weighted F1 to 62.75; the text-audio-video version scored 62.9, versus 58.5 for the multimodal CMN baseline.

What the MELD baseline measured

The MELD experiments encoded text with GloVe vectors and a one-dimensional CNN, extracted acoustic features with openSMILE, and concatenated the two representations for fusion. On the seven-class test set, DialogueRNN reached 57.03 weighted F1 with text, 41.79 with audio and 60.25 with text plus audio. The last result is 1.00 point above the multimodal bidirectional-LSTM baseline and 5.23 points above the context-free text CNN. Vision was not used because identifying and localizing the active speaker in multi-person video remained unresolved.

The paper also inspected where the model placed attention. In 882 of 1,381 correct test predictions, the highest-weighted earlier utterance came from another speaker. This is a useful diagnostic, not proof that the attended turn caused the decision. Emotion changes were a clearer failure point: recall was 66% for detecting a within-speaker shift, but only 36.7% when the model also had to name the new emotion correctly.

What remains unresolved

Rare labels such as fear, disgust and sadness were frequently confused, simple feature concatenation left much of the audio and video signal unused, and even human labels compress an ambiguous interpretation into one class. MELD and DialogueRNN therefore establish a controlled test of speaker-aware context; they do not establish that emotion can be read unambiguously from a face, voice or sentence.