International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 Volume: 11 Issue: 05 | May 2024
www.irjet.net
p-ISSN: 2395-0072
Storage Solutions for AI/ML Workloads: An Evaluation of Performance, Scalability and Efficiency Ramprasad Chinthekindi1, Shyam Burkule2, Senthilbharanidhar Boganavijayakumar3 ---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - The proliferation of artificial intelligence (AI)
volumes and a high demand for processing power, the efficiency of storage infrastructure has a dramatic impact on the flexibility and effectiveness of AI/ML workflows [2] [3]. This study thoroughly examines several storage architectures, including both conventional systems and newer technologies such as cloud-based platforms, to evaluate their capabilities, limitations, and appropriateness for a wide range of AI/ML applications. The diagram below illustrates three common options to consider when choosing the initial storage solution for your artificial intelligence and machine learning workload.
and machine learning (ML) applications has greatly intensified the requirements for storage systems. These systems must now facilitate high-speed and quick access to vast datasets. When evaluating storage options for AI/ML applications, the most important factors to consider are performance, scalability, and efficiency. Optimal performance requires achieving a high rate of input/output operations and minimizing the time delay, which is essential for efficiently managing massive quantities of data and meeting real-time processing requirements. The primary objective of this study is to investigate storage solutions for AI/ML workloads, specifically evaluating their performance, scalability, and efficiency. The study utilizes a comprehensive literature review methodology. This study examined a total of 20 papers published between the years 2018 and 2024. Data is gathered from several web databases. This paper provides an in-depth examination of modern storage solutions specifically built for AI/ML workloads. It covers distributed file systems, object storage, and specialized block storage systems that are optimized to enhance the performance of AI/ML models. The study also examines the incorporation of sophisticated storage capabilities, such as automated data tiering, in-storage processing, and hardware accelerations, which play a crucial role in improving data access speeds and processing efficiency. This study's findings not only emphasize the present condition of storage technologies in aiding advanced AI/ML environments, but also propose future avenues for innovation in storage solutions to more effectively address the changing requirements of the AI/ML community. This study offers a fundamental reference for enhancing the storage infrastructure required for the future generation of intelligent applications. Key Words: AI; ML; Scalability; Efficiency; Performance; Storage systems; Throughput; Latency; AI/ML Environments
Figure 1: An overview of Google Cloud AI/ML storage services [21]
1.INTRODUCTION
1.1 Scope of this study:
In the ever-evolving field of AI and ML, the pursuit of efficient storage solutions is crucial for harnessing the complete capabilities of data-driven advancements [1]. This assessment aims to thoroughly analyse the performance, scalability, and efficiency of storage solutions designed specifically for AI/ML workloads. In a context where there is a significant increase in data
© 2024, IRJET
|
Impact Factor value: 8.226
This study provides a complete assessment of the performance, scalability, and efficiency of different storage options in the context of AI/ML workloads. It includes conventional storage systems, cloud-based platforms, and emerging technologies to offer a comprehensive
|
ISO 9001:2008 Certified Journal
|
Page 2251