International Research Journal of Engineering and Technology (IRJET)
e-ISSN: 2395-0056
Volume: 12 Issue: 06 | Jun 2024
p-ISSN: 2395-0072
www.irjet.net
An Image Caption Generator based on Convolutional Neural Networks Ayush Tolani1, Prathamesh Landge2, Sagar Naphade3, Vijayendra S. Gaikwad4 1 Dept. of Computer Engineering, PICT, Maharashtra, India 2 Dept. of Computer Engineering, PICT, Maharashtra, India 3 Dept. of Computer Engineering, PICT, Maharashtra, India 4 Dept. of Computer Engineering, PICT, Maharashtra, India
---------------------------------------------------------------------***---------------------------------------------------------------------
of the visual content and the capability to express that understanding fluently in human language.
Abstract - The integration of computer vision and natural
language processing has resulted in significant advancements in enabling machines to interpret and communicate visual content. One of the most compelling outcomes of this integration is the development of captioning systems that automatically generate descriptive sentences for a given image. This project focuses on generating captions using deep learning-based architectures to effectively accomplish this task. Specifically, the VGG16 model is used to extract features from images with high levels of semantic and spatial detail. These extracted features are passed to a Bidirectional Long Short-Term Memory (BiLSTM) network, which captures the sequential nature of language and enhances contextual understanding. The bidirectional structure enables the model to consider both past and future contexts in the caption, resulting in more accurate and fluent descriptions. The system is trained and evaluated on standard benchmark datasets, and its performance is assessed using metrics such as BLEU, METEOR, CIDEr, and more. The results highlight the potential of combining visual perception with natural language to build intelligent systems capable of a comprehensive understanding of scenes.
The complexity of this task lies in the dual demand for visual comprehension and linguistic fluency. To produce meaningful captions, a system must first analyze and interpret the image's content and then construct a coherent, grammatically sound sentence that reflects its semantic essence. This demands an effective fusion of computer vision and NLP, ensuring that the generated descriptions are not only syntactically correct but also contextually and semantically accurate. One major challenge is ensuring that captions resemble the natural, context-aware language used by humans. This requires a nuanced understanding
2. RELATED WORK The task of automatic image captioning has garnered significant attention in recent years due to its potential to bridge computer vision and natural language processing. Early approaches primarily relied on template-based methods, which used predefined sentence structures combined with object detection results. While these methods were interpretable and simple, they lacked flexibility and struggled with generalization across diverse image contexts.
Key Words: Neural Image Caption Generator, CNN, BiLSTM, Attention Mechanism, Feature Extraction, VGG16, Deep Learning, Multimodal Understanding
1.INTRODUCTION
Subsequent advancements introduced retrieval-based approaches, where the system retrieved the most similar image from a dataset and reused its human-written caption. Although effective in some cases, these methods were limited by the diversity and coverage of the caption database, and often failed to generate novel captions.
Humans naturally possess the remarkable ability to observe and describe intricate details in their environment. With just a single glance, they can provide rich, meaningful descriptions of visual scenes. This cognitive skill is fundamental to human communication. For decades, researchers in artificial intelligence have endeavored to replicate this human-like ability in machines.
With the emergence of deep learning, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), end-to-end learning models became the standard. The seminal work by Vinyals et al. proposed the "Show and Tell" model, which utilized a CNN to encode the image and an LSTM to decode the features into a caption. This approach demonstrated the feasibility of training a unified neural model for the task.
While substantial progress has been made in computer vision tasks—such as object detection, attribute classification, action recognition, image classification, and scene understanding—the challenge of generating humanlike sentences to describe images remains relatively new and complex. Image captioning, the task of automatically generating descriptive natural language for a given image, bridges the gap between computer vision and natural language processing (NLP). It requires a deep understanding
© 2025, IRJET
|
Impact Factor value: 8.315
Further improvements were introduced with attention mechanisms, as seen in the "Show, Attend and Tell" model by
|
ISO 9001:2008 Certified Journal
|
Page 299