Sensory Perception & Interaction Research Group

University of British Columbia

Personalized Emotion Modelling from Affective Touch with Multimodal Support

Start Year: 

We investigate how multipass emotion labelling protocols generate ecologically valid dynamic emotion transition labelling of affective touch behaviour (and other modalities). Multipass emotion labels affords emotion metaphors that do not assume static emotion states consisting of scalar values. We demonstrate the construction of emotion labels that distinguish between emotions by transition (best-fit line) -- eg ``anxious getting more stressed'' vs ``anxious but resolving towards relaxed''. Using keypress force (FSR) and brain activity (EEG) data collected from participants playing a horror video game and/or skin conductance, heart rate, respiratory rate, and touch behaviour from a custom fabric sensor during emotional storytelling for elicitation, we trained subject-dependent hierarchical models of contextualized individual experience to compare emotion classification across modality, sessions (in longitudinal collection), emotions, individuals, we report benchmark F1-scores=[0.44, 0.82] (chance empirically determined at F1=0.22, sd=0.01).  

Our FEEL (FSR EEG Emotion-Labelled) dataset is available for download here.