International Research Journal of Engineering and Technology (IRJET)
e-ISSN: 2395-0056
Volume: 12 Issue: 05 | May 2025
p-ISSN: 2395-0072
www.irjet.net
Scalable Image Captioning with Transformer-Based Joint Learning of Visual and Language Models ELAMARAN R1, HARISH V2, KARTHICK BALA S3, KARTHICK P4, Asst Prof. MUTHUMARI5 1Bachelor of Engineering, Computer Science and Engineering, Dhanalakshmi College of Engineering,
Tamil Nadu, India.
2Bachelor of Engineering, Computer Science and Engineering, Dhanalakshmi College of Engineering,
Tamil Nadu, India.
3Bachelor of Engineering, Computer Science and Engineering, Dhanalakshmi College of Engineering,
Tamil Nadu, India.
4Bachelor of Engineering, Computer Science and Engineering, Dhanalakshmi College of Engineering,
Tamil Nadu, India
5Assistant Professor, Computer Science and Engineering, Dhanalakshmi College of Engineering,
Tamil Nadu, India. ---------------------------------------------------------------------***--------------------------------------------------------------------(GPT-2), T5 Text-to-Text Transfer Transformer, BART Abstract - Image captioning is one of the most important Decoder Network.
areas in the intersection of computer vision and natural language generation, aiming to generate grammatically and semantically relevant textual descriptions of visual contents. In this paper, we provide a set of implementations for a transformer-based captioning framework using existing pre-trained modules in the Hugging Face Transformers library. Following this philosophy, we adopt a dual-architecture configuration to utilize visual encoders, e.g. the Vision Transformer (ViT) and ResNet as semantic feature extractors, and subsequently leverage state-of-theart decoders such as GPT2, T5, or BART as language generators within the VisionEncoderDecoderModel framework. The visual encoder extracts high-level features from input sequences and the transformer-style language model decodes these embeddings into semantic, wellformed captions. This enables it to learn strong visuallinguistic alignments by training on large datasets of images and their descriptions, like MS COCO. Fluency and accuracy are then improved by tokenization strategies, beam search, and sampling techniques used to generate captions. Results are measured using the standard benchmarking metrics: BLEU, ROUGE and METEOR, whereas the proposed approach is shown to perform competitively with existing state-of-the-art approaches. Hugging Face has a very modular architecture, which immensely reduces the overhead in developing models, facilitating quick experimentation and deployment. Our results suggest that recent advances in combining vision and language transformers can create a scalable and efficient captioning methodology, suitable for many different assistive and retrieval-oriented applications.
1.INTRODUCTION The marriage of computer vision and natural language processing has triggered a wave of remarkable advances in the interpretation and verbalization of visual information. Image captioning is one such crossdisciplinary challenges that serves as an important area that enables to convert visualization of scenes to semantically accurate natural language representation. Such applications range from accessibility solutions for blind people, to automatic content generation, to improved human-computer interaction using multimodal interfaces. The mainstream image captioning methods were either template generation methods or rule-driven models with a tendency to overfit unseen data. Then, deep learning came along with encoder-decoder structures, with CNNs (convolutional neural networks) for encoding and RNNs (recurrent neural networks) for sequence generation. However, these approaches faced challenges with modeling long-range dependencies and being efficiently parallelized during training (you can read all about it in this paper). Overcoming this limitation, however, recent transformer-based architectures have been developed that utilize self-attention methods to allow for contextual representation across sequences, and while still being more efficient when compared to RNNs. Such unified schema is effectively offered by the VisionEncoderDecoderModel paradigm, especially through the implementation offered by HuggingFace Transformers library to connect powerful visual encoders with powerful language decoders. The ViT and ResNet
Key Words: Transformer-Based Architectures, VisionLanguage Integration, Image Captioning Framework, Vision Encoder Decoder Model, Hugging Face Transformers, Semantic Feature Extraction, Beam Search Decoding, Vision Transformer (ViT), Generative Pre-trained Transformer
© 2025, IRJET
|
Impact Factor value: 8.315
|
ISO 9001:2008 Certified Journal
|
Page 507