Big Data idea implementation in organizations: potential, roadblocks

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 12 | Dec 2022 www.irjet.net p-ISSN: 2395-0072

Big Data idea implementation in organizations: potential, roadblocks

Nishchitha S1, Anusha N M2, Priya D S3

Nishchitha S, Sapthagiri College of Engineering, Bangalore, Karnataka, India Anusha N M, Sapthagiri College of Engineering, Bangalore, Karnataka, India Priya D S, JSS Academy of Technical Education, Bangalore, Karnataka, India ***

Abstract - The examination of the Big Data phenomena is the focus of this essay. There are seven sections to it. In the first, it is explored how data and information are playing a larger and faster-growing role in the new socioeconomic reality. The concept of "Big Data" is then described,alongwith the primary factors driving data expansion. The most important options related to big data are givenandaddressed in the section that follows. The description of the tools, approaches, and the most beneficial data in the context of Big Data projects is the focus of the next section. The success aspects of Big Data efforts are examined in the section that follows, which is then followed by an examination of the most significant issues and difficulties related to big data. The paper's most important results and recommendations are presented in the final section.

Key Words: Big Data; Road block; Algorithms; Technology; Tools;

1. INTRODUCTION

Increasingamountsofdataareenteringmodernbusinesses asaresultofthevolumeofdatabeinggeneratedbythose organizations’stakeholdersandotherentitiesoperatingin theirbusinessenvironments,inadditiontotheorganizations themselves,whichisexpandingquickly.Thus,phraseslike"a data-centricworld"arebecomingmoreandmorecommon inthiscontext

Theprocesseslistedaboveareimportantcomponentsofthe global socioeconomic shifts occurring now, where the extraordinarilydynamicgrowthofincreasinglypotentand widespread information technology plays a vital role. The modern economyhasundergonetremendous change,and theestablishmentofa"interconnectedeconomy"hasbeen significantly accelerated by developments in this field. In terms of resources, this new type of economy is a knowledge-basedoneinwhichintellectualcapitalranksas the most significant form of capital. Under these circumstances,thecapacityofanorganizationtogatherthe appropriate data and information and to efficiently transformitintousefulknowledgebecomesanincreasingly crucialissue.

Duetotheaccelerationofadvancementsmaderecently,the fieldof information technologyhasstarted tomoveintoa newera.Processingpoweranddatastoragearealmostfree today, and networks and cloud-based services offer users

ubiquitous access to a wide range of services. These procedures lead to the creation of Big Data sets that have rapidlyincreasedinsize.Everydayin2012,2.5exabytesof data were produced, tripling from 2011. around every 40 months. In general, the previous two years have seen the creationof90%oftheworld'sdata

Asaresult,companiesnowhaveaccesstoanunprecedented amountofdataandinformationforanalysis[4].Thiscreates a wide range of brand-new operational opportunities for firmsaswellascountlessnewdifficulties.

The phrase "Big Data" has evolved in this context and is beingusedmorefrequentlyinthebusinessworld.

2. THE MOST SIGNIFICANT DATA GROWTH SOURCES AND BIG DATA

Becausetheterm"BigData"isnotalwaysunderstoodand applied,othermethodsofanalysishavebeendeveloped.The LeadershipCouncil forInformationAdvantageclaimsthat thisphraseisnotprecise."(...)it'sadescriptionoftheneverendingassemblageofvarioustypesofdata,themajorityof which is unstructured. It speaks of data sets that are exponentially expanding and too big, unprocessed, or unstructured to be analysed with relational database approaches.Contrarily,BigDataisdefinedbyNewVantage Partners as "a term used to describe data sets that are so large, so complex, or that require such rapid processing (sometimescalledtheVolume/Variety/Velocityproblem), that they become challenging or impossible to work with usingstandarddatabasemanagementoranalyticaltools."It iscrucialtoemphasisetostressthatBigDatareferstoboth the data associated with this consumption as well as the storageandconsumptionoforiginalmaterial.

Ingeneral,anumberofimportantphenomenahaveledtoa considerableincreaseindatageneration

Thegrowthoftraditionaltransactionaldatabases,thefirst trend, is mostly related to the fact that businesses are gatheringdatamoreoftenandwithgreatergranularity.

Thisisbecauseofanumberoffactors,includingrisingclient demands, escalating competition, and turbulent business environments.Organizationsmustrespondtothechanges occurringasquicklyandflexibleaspossible,andthenmake necessaryadjustments.Tobeabletoaccomplishthis,they

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 12 | Dec 2022 www.irjet.net p-ISSN: 2395-0072

are compelled to conduct ever-increasingly in-depth analysesofmarkets,rivalry,andcustomerbehaviour.

The second trend, an increase in multimedia material, is related to the quick uptake of multimedia throughout a rangeofmoderneconomicsectors,includingthehealthcare industry,whereover95%ofclinicaldataisnowdigitalvideo format has been generated. In general, multimedia data already makes up more than half of Internet backbone traffic, and by the end of 2013, it is anticipated that this sharewillreach70%.

The advent of "The Internet of Things," a phenomenon wherephysicalitemsorgadgetsconnectwithoneanother withoutanyhumaninvolvement,isthenexttrendthathas contributed to an increase in the amount of data being generated. They connect to one another wirelessly or throughwiredconnections,frequentlyutilisingIPprotocols. Theygatherandtransmitenormousvolumesofdatabecause theyarefittedwithseveralsensorsoractuators.Thevolume of data produced by the "Internet of Things" will increase dramaticallyby2015,asthenumberofdeployedconnected nodesworldwideisanticipatedtoincreaseatarateofover 30%eachyear.

Thenextveryimportantsourceoftheriseindataissocial media.JustFacebookusersproduceatonneofdata.In2011, Facebook's 600 million active users spent more than 9.3 billionhourspermonthonthesite,producinganaverageof 90piecesofcontent(pictures,notes,blogentries,links,or news articles) One billion people were using Facebook a yearlater.Ifonlytextsaretakenintoaccount,accordingto researchdoneatthestartof2012,userssendandreceivean average of nine messages every month. In the instance of YouTube,24hoursofvideoareuploadedperminute,while 98000 tweets are sent during the same period of time on Twitter. Smart phones are also becoming more and more significantinsocialnetworks.Socialnetworkusageisrising on both PCs and smartphones, but smartphone usage is substantiallyhigher.Iffrequentusersaretakenintoaccount, the percentage for PCs is 11% each year while the percentage for cellphones is 28% per year. Due to this, mobiledatatraffichasrapidlyincreasedanddoubledduring thethirdquartersof2011and2012.By2018,mobiledata trafficisexpectedtomultiplytwelvetimes

3. OPPORTUNITIES AND BENEFITS RELATED TO THE USE OF BIG DATA

TheevolutionoftheBigDataphenomenonandthetoolsand proceduresthatgoalongwithitcannotbedivorcedfromthe broaderorganisationalchangesthathavebeengoingonin recentyears.Infact,itisbecomingmoreandmoreprevalent infirmsthatareinterestedintheanalyticssectorand has greatly increased the possibilities offered by business intelligence(BI)technologies.Businessintelligencesystems arehighlysuitedforgatheringandanalysingstructureddata

duetotheir roleinoffering firmsa varietyof possibilities and chances in the field of analytics. However, there are severalanalysiskindsthatBIcannothandle.Thesemostly concern instances where data sets expanding in diversity, granularity,real-time,anditeration.Whenattemptingtouse conventional methodologies based on relational database models, these forms of unstructured, large volume, and rapidlychangingdatapresentchallenges.Itisnowclearthat a newclassoftechnologyandanalytical techniquesare in increasingdemand.

ThefindingsofresearchundertakenbytheMcKinseyGlobal Institutehave verified that usingBig Data hasa variety of benefits depending on the sector of the economy. These findings demonstrate how Big Data has the power to transform industries as diverse as healthcare, public administration,retail,manufacturing,andpersonallocation data. There are seven primary groupings of benefits associatedwithBigDataprojects,accordingtothefindings ofapollperformedinthesummerof2012byNewVantage Partners among C-level executives and department heads frommanyofAmerica'sleadingfirms.Themostsignificant oftheseadvantagesarebetter,fact-baseddecisionmaking (22%) and improved customer experiences (22%), along with the general message that the expectation is to take quicker, smarter decisions. Increased sales (15%), new productdevelopments(11%),decreasedrisk(11%),more efficient operations (10%), and higher-quality goods and services(10%)areamongtheothergroupsofadvantages.

BigDataplatformsenableorganisationstoreceiveanswers to critical queries instantly rather than over the course of months. Big Data's main benefit is to shorten the time it takestogetananswer,whichspeedsupdecision-makingat both the operational and tactical levels. The ability for ongoingbusinessexperimentationtoinformdecisionsand test new goods, business models, and customer-focused innovationsisacrucialnewfeaturerelatedtotheBigData phenomenoninthecontextofdecision-making.Sometimes, astrategylikethisevenenablesreal-timedecision-making. Therearenumerousinstancesofbusinessesadoptingthisin reallife.Forinstance,CapitalOne'smultidisciplinaryteams domorethan65,000testsannually.Theyplayaroundwith combiningdifferentmarketsegmentsandfreshgoods.

Basedononlinedatastreams,theonlinegrocerFreshDirect changespricesanddiscountseverydayorevenmoreoften. Tesco is another illustration. Through a loyalty card programme, this corporation collects transaction data on millions of its clients, which it then utilises to assess potentialnewbusinessventures.Forinstance,itexamines how to tell customers about pricing, promotion, and shelf allocation decisions as well as how to design the most successful promotions for particular consumer segments OnesuchillustrationisWalmart.TheBigDataplatform(The OnlineMarketingPlatform)wasdevelopedbythiscompany, and it is used, among other things, to conduct several

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 12 | Dec 2022 www.irjet.net p-ISSN: 2395-0072

concurrentteststoevaluatenewdatamodels.Additionally, InternetpowerhouseslikeAmazon,eBay,andGooglehave beenleveragingtodrivetheirperformancethroughtesting.

There are five main ways that big data adds value for organisations,accordingtotheMcKinseyGlobalInstitute:

• enabling experimentation to identify needs, reveal variability,andenhanceperformance;

• fosteringtransparencybyintegratingdataandmakingit morereadilyaccessibletoallpertinentparties;

•Segmentingpopulationstotailorinterventions,Segmenting populationstotailorinterventions,

• Using automated algorithms to supplement or replace humandecision-making.

•Developingnovelbusinessmodels,goods,andservices

Overall, the research findings from the Economist IntelligenceUnit,whichpolled607executivesfromaround the world in February 2012, support the importance of businesses utilising big data. According to the CEOs who participatedinthepoll,bigdataprojectshaveraisedtheir firms'performanceoverthepastthreeyearsbyabout26%. They anticipate that over the following three years, these initiatives will boost performance by an average of 41% Additionally, it is important to note that businesses with decision-making processes based on data and business analytics have 5-6% higher output and productivity, accordingtothefindingsofresearchbyBrynjolfssonetal. Businessanalytics-baseddecision-makingalsoaffectsother performance metrics such as market value, equity return, andassetutilization.

Big Data systems have been utilised for both decision automation and human decision support, similar to BI initiatives.Basedonthedegreeofriskassociatedwiththe decision,BigDataisutilised,onaverage,fordecisionsupport 58%ofthetimeandfordecisionautomationaround29%of the time, according to the findings of the aforementioned studydonebytheEconomistIntelligenceUnit

4.MAXIMUM USEFUL DATA, TOOLS, AND TECHNIQUES IN THE CONTEXT OF BIG DATA INITIATIVES

The efficient implementation of big data initiatives necessitatesadoptingtheproperorganizationalsteps,such asensuringthatbusinesseshaveaccesstoalltheresources requiredtoenableanalysisofthealwaysexpandingdatasets theyhaveaccessto.Oneofthemostimportantissuesinthis area is the application of appropriate methods and technologies. In reality, businesses combine, manipulate, analyse, and visualise big data using a wide range of techniques and technology. They come from a variety of

value:

disciplines,includingeconomics,computerscience,statistics, and applied mathematics. Some of them have been specificallycreatedforthis,whileothershavebeenmodified forit.ExamplesofmethodsusedforBigDataanalysisare: time series analysis, sentiment analysis, spatial analysis, simulation, data mining, data fusion and integration, A/B testing,andmachinelearning.BigTable,Cassandra,Google FileSystem,Hadoop,Hbase,MapReduce,streamprocessing, and visualisation (tag cloud, clustergram, history flow, spatial information flow) are a few examples of the technologiesusedtogather,modify,manage,andanalysebig data.

There are more and more brand-new analytical toolkits availablefortheexaminationofBigData.Examplesofthese remediesinclude:

• NM Incite, Social Mention, SocMetrics, Traackr, Tweepi (sentimentanalysistoolsforestimatingthebuzzarounda product or service, influencer intelligence tools for identifying key influencers and targeting for marketing or insights)

•Attensity,Autonomy(livetestingtoolsforgettingdirect userfeedbackonnewproducts)

•Alterian,TweetReach(networkintelligencetoolsforrealtimeanalysisofthereactionsandresponsestochangesof industryplayers),

In addition, adequately trained personnel are a crucial componentofBigDatainitiatives.Inthiscontext,aspecial category of worker known as a data scientist who has received the necessary training to deal with Big Data is mentioned.Inactuality,itmeansthattheyshouldbeadeptat miningvastamountsofunstructureddataforthesolutions to an organization's most pressing problems. These individualsoughttocombinetheskillsofananalyst,adata hacker,acommunicator,andatrustedadvisor.Theyshould have strong analytical capabilities as well as strong, innovative IT skills, and they should be familiar with the company'sinternalproductsandoperations.Themajorityof businesses use platforms to bridge the knowledge gap because data scientists often require years to acquire indepthdomainexpertise

AppropriatedataisafundamentalresourceneededforBig Data initiatives, in addition to appropriate methodology, tools,andpeople.Aswasalreadynoted,modernenterprises are currently receiving a large amount of data from numerous sources, but not all Big Data sets are equally valuable.Themostcrucialsourceofdataisunquestionably informationaboutbusinessactivity,suchassales,purchases, costs,andsoforth.Thesecondmajorsourceofdataisoffice documents, closely followed by social media. In several industries, like healthcare, pharmaceuticals, and biotechnology, social media data sets are more significant than office documentation. POS data, website clickstream

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 12 | Dec 2022 www.irjet.net p-ISSN: 2395-0072

data,RFID/logisticsdata,GISdata,telecommunicationsdata, andtelemetrydataaresomeoftheadditionalcrucialtypes ofdatasets.

5. REASONS BIG DATA INITIATIVES SUCCESS

Numerous success elements can be identified through an examinationofBigDataeffortsthathavealreadybeenput intoaction,eachwithitsownsetofsuggestions.

Fivecrucial ruleshave been highlighted byMarchand and Peppard as essential to a Big Data project's success. They consistof:

1.MakingtheBigDatainitiative'sfocusonpeople.

2. Putting a focus on information use as a means of maximisingthevalueofinformationtechnology.

3. Including cognitive and behavioural scientists on IT projectteams.

4.Concentratingoneducation

5. Focusing on using technology rather than resolving businessissues.

Barton and Court, on the other hand, came to the opinion thatcompleteexploitationofdataandanalyticsneedsthree competences based on their experiences working with businessesindata-richindustries:

1. Picking the appropriate data. In this environment, it's crucial to upgrade IT architecture and infrastructure for simpledatamergingaswellassourceinternalandexternal datacreatively.

2.Stressingtheuseofinformationasakeytomaximisingthe potential of information technology. In this setting, it's crucial to concentrate on the performance factors that matter most and to create models that strike a balance betweencomplexityandusability.

3. Including cognitive and behavioural scientists on IT projectteams.Inthiscontext,upgradingbusinessprocesses anddevelopingcapacitiestoenabletoolusearetwocrucial factors.Thefirstisbuildingstraightforward,intelligibletools forworkersonthefrontlines.

OrganizationsthatutiliseBigDatabasetheiroperationson threekeyconcerns,accordingtoBarthetal.are:

1.Focusingondataflowratherthanstockprices

2. Relying on product and process developers and data scientistsratherthandataanalysts.

3.Bringing operational and production processes intothe centreofthebusinessandseparatinganalyticsfromtheIT function.

6. THE BIG DATA EFERENCES' MOST SIGNIFICANT OBSTACLES AND CHALLENGES

BigDatainitiatives,likeotherIT-relatedendeavours,arenot withoutissuesanddifficulties.TheresearchbytheEconomic IntelligenceUnitreferencedpreviouslyidentifiessomeofthe challengestoeffectivelyusingbigdatafordecision-making. Thebiggesthurdlewas"organizationalsilos"(55,7%),which are caused by the fact that data related to specific organizationalfunctions(suchassales,distribution,etc.)are collectedin"functionsilos"ratherthanbeingpooledforthe benefitoftheentirebusiness.Thesecondproblem,whichis alsosignificant(50,6%),isthedearthofdatascientistswith thenecessaryqualifications.Thethirdfactor(43,7%)ishow longittakesbusinessestoanalyselargedatasets.Thethird factor (43,7%) is how long it takes businesses to analyse large data sets. Organizations anticipate being able to examine data in real time and take action on it, as was alreadysaid.Thechallengesassociatedwithanalysingeverincreasingamountsofunstructureddatamakeupthefourth barrier (41,7%). The fifth major barrier is, finally, senior management's failure to view big data in a sufficiently strategicway(34,9%).

Five management issues, according to McAfee and Brynjolfsson,areholdingbackfirmsfromutilisingbigdata toitsfullpotential.Theyare:companyculture,leadership, people management, technology, and decision-making. Havingmoreorbetterdatadoesnot ensuresuccesswhen evaluatingleadership.Thefirm'smanagementstillneedto have a clear understanding of the market, set attainable goals,andseehowtheorganizationwilldevelop.Numerous organizationaldecisionsarealteredbybigdata.Theneedto provide the organization with the necessary individuals (such as data scientists) who are capable of working with largeamountsofdataisrelatedtotalentmanagement.The nextissueishowtoguaranteethatthedatascientistshave the right equipment to handle Big Data. Technology is a crucialcomponentofbigdataendeavours,eventhoughitis not sufficient on its own to be successful. The second obstacle is related to the issue of making sure there exist mechanismsinplacetoensurethattheinformationandthe appropriatedecision-makersarepresentinthesameplace. Makingensuringthatthosewhocomprehendtheissuescan make the appropriate use of information and collaborate with those who possess the essential problem-solving abilitiesiscrucial.Thelastdifficultyislinkedtoalterations inorganizationalculture.Makingdecisionsthatareasdatadrivenaspossibleratherthanrelyingsolelyonintuitionand gutfeelingiscrucialinthissituation.Itisimportanttonote thatotherresearch,suchasthatpertainingtosectoralBig Data projects, has also discussed the significance of such culturaltransformation.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 12 | Dec 2022 www.irjet.net p-ISSN: 2395-0072

It's also important to consider the numerous difficulties relatedtodataandlegalrights.Theyhavetodowiththings likecopyright,databaserights,privacy,trademarks,contract law,andcompetitionlaw.Anothersignificantissuerelating tolegalmattersalsoexists.Ithastodowiththeopennessof data collection procedures. The use of Big Data to boost decisionautomationisafurthersignificantrisk.Onefurther significantriskishighlightedinthecontextofbigdata.Ithas todowiththepossibilitythatBigDataisn'tgivingacertain circumstance the full picture. This is due to a number of factors, including biases in the data gathering process, exclusions or gaps in the data signals, or the ongoing requirementforcontextinconclusions.

Atthesametime,pre-BigDatadangersanddifficultiesare stillevolving,suchastheissueofprotectinginformationand data that has been gathered. These problems primarily concern how to safeguard information that must be kept private by businesses (such as different categories of consumer data) and information that is competitively sensitive.Asaresult,theissuesrelatedtobroadlydefined security of enterprises' IT infrastructure and protection against various assaults become even more crucial than before.Securingithasbecomeevenmorecrucialasaresult ofenterprises'growingrelianceontheefficientandstable operationoftheirITinfrastructureasaresultoftheBigData phenomena.

7. CONCLUSIONS

The processes pertaining to decision-making at different organizational levels are evolving as a result of the fast expanding amount of data that companies have at their disposalandtheopportunitiesconnectedwithitspractical usage.BigDatathushastheabilitytosignificantlyimprove how businesses operate generally and gives them a competitiveedge.Now,businessesareattemptingtomake evengreateruseofthepossibilitiesandopportunitiesthat areappearing.

Itisnotsufficienttosimplycollectandownthenecessary datasetsifprojectsaimedattheactualexploitationofBig Datasetsaretobesuccessfulatprovidinganorganization withacompetitiveedgeandbeofvalue.Inactuality,thisis just where every Big Data endeavor begins. Additional crucialcomponentsareappropriateanalyticalmodels,tools, qualifiedpersonnel,andorganizationalskills.Whenanyone oftheseessentialelementsisabsent,theresultcouldbethat instead of the anticipated benefits, there is merely disappointment.BigDataprojectsaremerelythelatestina longhistoryofmanagerialfads,leadingtodisillusionment and the notion. In general, the speed at which Big Data solutionscanbeusedinasecureandpracticalwayremains questionable, despite the fact that they offer enormous potentialforbothcommercialenterprisesandgovernments.

REFERENCES

[1] Kemp Little LLP, “Big Data – Legal Rights and Obligations”, http://www.kemplittle.com /Publications/WhitePapers/Big%20Data %20%20Legal%20Rights%20and%20Obligations%202013.pdf, January2013.

[2] D. Tapscot, A. Williams, Radical Openness. New York: TEDBooks(KindleEdition),2013.

[3] National Intelligence Council, “Global Trends 2030”, http://globaltrends2030.files.wordpress.com /2012/11/global-trends-20 30-november2012.pdf, December2012.

[4] E. Brynjolfsson, A. McAfee, (2012), “Big data: The managementrevolution”,HarvardBusinessReview,pp.6068,October2012.

[5] A. Lampitt, “Hadoop: Analysis at massive scale”, in InfoWorld, http://resources.idgenterprise.com/original /AST-0084522_IW_ Big Data_ rerun_1_all_sm.pdf, pp. 812, Winter2013.

[6] LCIA, “Big Data: Big Opportunities to Create Business Value”,http://poland.emc.com/microsites/cio/articles/bigdata-big-opportunities / LCIA-BigData-OpportunitiesValue.pdf,2011.

[7]McKinseyGlobalInstitute,“Bigdata:Thenextfrontierfor innovation, competition, and productivity”, http://www.mckinsey.com /mgi/publications/big_data/pdfs/MGI_ big_data_full_report.pdf,May2011.

[8] NewVantage Partners, “Big Data Executive Survey: Themes & Trends”, http://newvantage.com /wpcontent/uploads/2012/12/NVP-Big-Data-Survey-ThemesTrends.pdf,2012.

[9]J.Gantz,D.Reinsel,“ExtractingValuefromChaos”