Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session B: Wednesday, August 13, 1:00 – 4:00 pm, de Brug & E‑Hall

Toward Real-World Emotion Decoding: A Transformer-Based Approach Using Movie fMRI

Bo-Gyeom Kim¹, Jiook Cha¹, Patrick Styll²; ¹Seoul National University, ²Technische Universität Wien

Presenter: Bo-Gyeom Kim

Real-world emotion recognition arises through continuous interactions among multiple sensory cues—dynamics often missed by standard laboratory paradigms. To investigate these dynamics, we applied a Transformer-based deep-learning model (SwiFT combined with Perceiver IO) to functional MRI data from 512 youths (ages 5–21) watching a 10-minute movie. By modeling neural signals as continuous time-series, we tracked short-term (~40s, 50 TRs) changes in seven emotions (e.g., positive, fear). Longer sequence windows and explicit hemodynamic modeling (double-gamma HRF) improved decoding accuracy, highlighting the importance of extended temporal context and precise BOLD-delay modeling. The prominent contribution of the visual cortex suggests reliance on low-level visual features within rich audiovisual stimuli. These findings demonstrate that flexible sequence-to-sequence methods effectively capture the temporal dynamics of emotion recognition under realistic conditions, deepening our understanding of real-world emotional processing.

Topic Area: Methods & Computational Tools

Extended Abstract: Full Text PDF