International Research Journal of Engineering and Technology (IRJET)
e-ISSN: 2395-0056
Volume: 12 Issue: 11 | Nov 2025
p-ISSN: 2395-0072
www.irjet.net
Multimodal Artificial Intelligence Detection of Hidden Emotional Signal For Early Mental Health Support Using Vision Transformer Prof. S. M. Malode1 MAHI NAKHATE2, KALYANI FATING3, SAKSHI MILMILE4, AADITYA NARWADKAR5 ¹Assistant Professor, AI & DS Department K. D. K. COLLEGE OF ENGINEERING, Nagpur
2345B.Tech Final Year, AI & DS Department K. D. K. COLLEGE OF ENGINEERING, Nagpur
-----------------------------------------------------------------------***----------------------------------------------------------------------ABSTRACT affective behaviour. As such, emotion recognition is inherently multimodal, and relying solely on one modality—such as facial images or speech—restricts system reliability.
Emotion recognition is a cornerstone capability in modern affective computing and human–AI interaction systems. As artificial intelligence increasingly enters domains such as mental-health support, cognitive monitoring, conversational systems, autonomous driving, and adaptive learning, the ability to decode human emotional cues in real time becomes critical. Traditional unimodal systems based solely on facial expressions or speech tend to suffer from reduced robustness when exposed to real-world noise, occlusions, and behavioural variability. The emergence of Transformer-based models—particularly Vision Transformers (ViT) for visual understanding and selfattentive encoders for speech—has reshaped the landscape of emotion recognition by offering superior global context modelling and scalability. This review critically analyses this system, a real-time bimodal (vision + audio) emotion recognition system that integrates ViTbased facial emotion recognition, LSTM-based vocal emotion analysis, and a transformer-based multimodal fusion network enabled by agentic AI memory through LangChain. The system aims to create longitudinal emotional awareness rather than isolated predictions. Through a detailed literature review, architectural analysis, comparative assessment, and identification of research gaps, this paper presents a consolidated reference for researchers building efficient real-time multimodal emotional intelligence frameworks. Diagrams, architectural illustrations, and IEEE-style references are provided to support further exploration.
In recent years, two technological shifts have redefined the field:
Multimodal fusion techniques now integrate visual, auditory, and contextual signals within a unified representation space.
ViT for vision LSTM for speech Transformer-based fusion Real-time synchronous processing Memory-enabled agentic reasoning
Together, these components form a holistic emotional intelligence system capable of real-time inference, emotional continuity tracking, and adaptive support. 1.1 Emotion Recognition and Its Applications Emotion recognition is relevant across numerous domains: (a) Mental Health Monitoring AI-driven systems can support emotional wellbeing by tracking variations in affect, detecting indicators of stress or depressive states, and offering grounding exercises or supportive interventions. These tools do not replace clinical professionals but provide continuous monitoring outside clinical settings.
Emotion recognition enables machines to perceive and respond to human emotions, bridging cognitive gaps between humans and AI systems. Unlike traditional AI models that interpret explicit commands or textual content, emotion-aware systems must decode subtle cues expressed through facial muscle contractions, tone of voice, micro expressions, and temporal shifts in
Impact Factor value: 8.315
2.
1. INTRODUCTION
|
Transformer-based architectures have surpassed CNNs and RNNs by allowing attention-driven global receptive fields.
Multimodal Artificial Intelligence Detection of Hidden Emotional Signal For Early Mental Health Support Using Vision Transformer is situated at this intersection, combining:
Keywords: Multimodal Emotion Recognition, Vision Transformer (ViT), Audio-Visual Fusion, Speech Emotion Recognition, Transformer Models, Mental Health AI.
© 2025, IRJET
1.
|
ISO 9001:2008 Certified Journal
|
Page 941