International Research Journal of Engineering and Technology (IRJET) Volume: 11 Issue: 05 | May 2024
www.irjet.net
e-ISSN: 2395-0056 p-ISSN: 2395-0072
THE PIVOTAL ROLE OF DATA ENGINEERING IN ADVANCING LARGE LANGUAGE MODELS (LLMS) Vishnu Vardhan Amdiyala Binghamton University, USA ----------------------------------------------------------------------------***--------------------------------------------------------------------ABSTRACT: Large Language Models (LLMs) have revolutionized the field of natural language processing (NLP) by demonstrating remarkable capabilities in generating human-like text and understanding language context. However, data engineering's crucial role in ensuring the availability of high-quality training data and effective processing pipelines is crucial to the success of LLMs. This article explores the vital contributions of data engineering to the development and deployment of LLMs, focusing on key aspects such as data collection, scalable infrastructure, feature engineering, model training, and deployment [1]. Keywords: Data Engineering, Large Language Models (LLMs), Scalable Infrastructure, Feature Engineering, Model Training and Optimization
© 2024, IRJET
|
Impact Factor value: 8.226
|
ISO 9001:2008 Certified Journal
|
Page 1936