International Research Journal of Engineering and Technology (IRJET)
e-ISSN: 2395-0056
Volume: 12 Issue: 10 | Oct 2025
p-ISSN: 2395-0072
www.irjet.net
Interpretable Visual Reasoning through Human Feedback and Symbolic Explanation Laraib Ahmad Siddiqui1, Mohd Shahzad2 1Program Control Services Analyst, Accenture, India 2AWS and DevOps Consultant, Deloitte, India
---------------------------------------------------------------------***--------------------------------------------------------------------1.2 Problem Statement Abstract - Modern vision–language models achieve nearhuman performance on image captioning and visual question answering, yet they often remain opaque, producing correct answers for the wrong reasons. This work proposes a human-feedback-driven interpretability framework that trains models not only to predict outcomes but also to explain their reasoning in natural language anchored to visual evidence. We combine visual-language transformers with a symbolic concept layer that grounds explanations in identifiable scene elements, objects, actions, and attributes, and use human preference data to iteratively refine these explanations. Our framework introduces two key innovations: A rationale-generation head that learns from paired human-expert rationales, and A feedbackguided explanation scorer that rewards faithfulness and penalizes hallucinated justifications. Empirical studies on VQA-X and e-SNLI-VE show substantial gains in explanation fidelity (+19%) and user comprehension (+27%) over baseline caption-based methods. The results suggest a scalable path toward trustworthy and transparent visual reasoning aligned with human cognitive expectations.
Most explainability efforts in vision are post-hoc: heatmaps or saliency overlays generated after inference. These methods may highlight relevant pixels but fail to reveal the conceptual chain of reasoning. Conversely, textual rationales generated by large multimodal models (e.g., GPT-4V, LLaVA) are often linguistically fluent but semantically unfaithful; they “sound right” yet lack factual grounding. We therefore pose a central question: Can we train vision–language models to explain decisions the way humans do, through structured, symbolic, and context-aware reasoning reinforced by human feedback?
1.3 Proposed Approach We introduce a hybrid architecture that fuses transformer-based perception with symbolic explanation modules, optimized through human feedback loops:
Key Words: Human Feedback, Symbolic Reasoning, Interpretable AI, Visual–Language Models, Explanation Faithfulness, Neuro-Symbolic Learning, Vision Transformer, Explainable Artificial Intelligence (XAI)
1. 2. 3.
1. INTRODUCTION 1.1 Motivation
Unlike prior models that optimize only predictive accuracy, our system explicitly learns to balance correctness and interpretability.
As vision–language systems become decision-support tools in fields such as healthcare, robotics, and surveillance, understanding their reasoning is crucial. A model that outputs the correct answer but for an incorrect internal rationale can mislead users, degrade trust, and cause ethical harm.
1.4 Contributions 1.
Traditional accuracy-driven training overlooks the underlying reasons behind predictions, focusing instead on the outcome. Recent legislative and industrial frameworks, such as the EU AI Act and NIST AI Risk Management Guidelines, explicitly call for explainability and human oversight. Thus, interpretable visual reasoning is no longer optional; it is an operational and compliance requirement.
© 2025, IRJET
|
Impact Factor value: 8.315
A visual-semantic parser extracts object and relation graphs from the image. A rationale generator converts these symbolic elements into natural-language explanations. A human-feedback scorer evaluates explanations for faithfulness, completeness, and readability, updating the generator via preference-based fine-tuning.
2. 3.
|
A unified framework combining human feedback and symbolic reasoning for interpretable visual explanations. A feedback-guided Explanation Scorer that quantifies alignment between visual evidence and generated rationales. An open-benchmark evaluation demonstrating improved explanation fidelity and user trust metrics.
ISO 9001:2008 Certified Journal
|
Page 574