INFDok - Dokumentenvolltextserver des Fachbereichs Informatik

Kurzfassung auf Englisch:

This thesis examines the effectiveness of Cross-modal Random Network Prediction (CRNP) fusion for audio-visual continuous emotion recognition. The method was previously shown to achieve state-of-the-art results in multimodal medical image segmentation. We chose to adapt the CRNP to the task of multi-dimensional emotion regression. Reviewing recent literature on emotion recognition and limitations of naive multi-modal fusion methods, we decided on 3 strong baselines for comparison. After defining the baselines and the proposed method, we evaluated the emotion recognition performance of the CRNP against the baselines on the AffWild2 dataset. Additionally we conducted experiments with added Gaussian noise, to analyze the capabilities of the models to function with unreliable data. Our analysis has shown that the CRNP fusion can be successfully used for continuous emotion recognition, however it was outperformed by attentionbased baselines in nearly all cases. We also found that the fusion mechanism did not provide sufficient benefit over the baselines for noisy data. Experiments analyzing the behavior of the CRNP were however able to show that the uncertainty estimation was correct. We conclude that, even though it performed worse than the baselines, the CRNP’s results are promising for audio-visual emotion recognition. We identified multiple venues for future research that have potential to increase the CRNP’s performance and might help close the gap between it and the examined baselines.