Abstract

In recent times, the incorporation of EI into AI systems has increased user engagement, improved content moderation, and provided mental health support. This research proposes a security-based UEIM that uses multiple modalities to perform emotion recognition and behavioural trajectory analysis on social media platforms. Unlike conventional approaches that depend on only one source of input, UEIM leverages the capabilities of texts, images, and videos using a cross-modal attention fusion network known as CMAFNet, which helps secure the emotional data. To achieve accurate emotion trajectory analysis, Bi-LSTM networks have been used in the proposed approach. Graph attention networks (GATv2) have also been utilized to predict the spread of emotions among users. The UEIM can guarantee the privacy and security of the users’ data even when there is noise or missing data present, which is a frequent problem in real-life scenarios related to social media applications. From experimental results, the performance of the model is excellent since it obtained 88.2% F1-score in emotion recognition, 91.3% top-1 accuracy in emotion sequence prediction, and 85.2% micro-F1 for emotion propagation modelling. The MSE for emotion valence/arousal regression is 0.018. These results show the great accuracy of the model in emotion prediction and behaviour analysis, and this can be used in health monitoring and content moderation, among others.