Risk and Controls for Artificial Intelligence and Machine Learning Systems. May 27, 2024

Risksandcontrolsfor artificialintelligenceand machinelearningsystems

Report

Version1.0

May27,2024

D-16-432

Projectleads:LiinaKamm(CyberneticaAS)

HendrikPillmann(RIA)

Authors:DanBogdanov

PaulaEtti

LiinaKamm

AndreOstrak

TanelPern

FedorStomakhin

MariaToomsalu

Sandhra-MirellaValdma

AntoVeldre

CyberneticaAS,Mäealuse2/1,12618Tallinn,Estonia.

E-mail: info@cyber.ee ,Website: https://www.cyber.ee,Phone: +3726397991

Co-fundedbytheEuropeanUnion.Viewsandopinionsexpressedarehoweverthoseofthe author(s)onlyanddonotnecessarilyreflectthoseoftheEuropeanUnionortheEuropeanCybersecurityCompetenceCentre.NeithertheEuropeanUnionnortheEuropeanCybersecurity CompetenceCentrecanbeheldresponsibleforthem.

©EstonianInformationSystemAuthority,2024

6.1 Informationsecuritycontrols

8.1 DescribeyourAIsystem

8.1.1 Howtogoevenfurther?

8.2 Findadeploymentmodelsuitingyoursystem

8.3 Identifyapplicablelegalnorms

8.3.1

8.3.2 DM2:systemusinganexternally-trainedAImodel

8.3.4 Howtogoevenfurther?

8.4 Evaluatethreatstousers,society,andenvironment.

8.4.1 DM1:systemusingAIasaservice

8.4.3

8.4.4

8.5.4 Howtogoevenfurther?

8.6 AIsysteminasingleslide

1Introduction

1.1Purpose

TheEstoniansocietyhasadopteddigitalservicesforimprovingworkefficiency.Ourdigital stateisrenownedforitslowadministrativeoverhead.Transactionsbetweenstateagenciestake placeovertheX-Roaddataexchangelayer.Boththepublicandtheprivatesectorhaveadopted digitalidentitysolutions.ForEstonia,adigitalsocietyisanobjectofconstantdevelopment.

Rapidadvancesincomputingpowerhavetakenthedevelopmentofartificialintelligencetechnologytoaqualitativelynewlevel.Artificialintelligencesystemscapableofgeneratingtext, images,sounds,music,andvideobasedonanaturallanguagedescriptionhavemadethetechnologyaccessibletoawidepopulation,leadingtoanincreasingbeliefthatinformationtechnologywillenablethedevelopmentofanewgenerationofsystemscapableofperformingsuch tasksbetterthanhumans.

ArtificialintelligencesystemsarebeingdevelopedinEstoniaandtherestoftheworldbyboth publicandprivatesectorinstitutions.Thepurposeofthisreportistosupporttheimplementationofthistechnologybyprovidingguidanceinensuringcybersecurity,fulfillingoflegalrequirements,andsocietalsafety.

Thereportiswrittenforabroadaudience.Itwillbemostusefulforsmallandmedium-size organisationsandprivateindividualswhomaynothavelegal,informationsecurity,orartificial intelligenceexpertsontheirstaff.Theseuserswillbeabletoutilisethequick-referenceguide attheendofthereportforAIsystemriskassessmentandchoiceofmeasures.Ourgoalisfor everyonetouseAIlawfully,safely,andwithoutharmingthesocietyandenvironment.

Morematureorganisationsemployingqualitymanagementsystemsandmorelabour-intensive riskmanagementprocesseswillbeprovidedwithguidanceontheapplicationofartificialintelligence.Theywillbegivenrecommendationsonwhichstandardsandreportstofollowtoensure anadequatelevelofmaturity.

1.2Definitionsandabbreviations

AGI Artificialgeneralintelligence. AI Artificialintelligence. AIsystem Artificialintelligencesystem. AIHLEG

EUHigh-LevelExpertGrouponAI.

API Applicationprogramminginterface.

ASI

Artificialsuperintelligence. BERT

BidirectionalEncoderRepresentationfromTransformers.

CaaS

Computeasaservice.

CNN

Convolutionalneuralnetwork.Amodelarchitectureusedinimagerecognition.

CPU Centralprocessingunit.

CUDA

ComputeUnifiedDeviceArchitecture,atoolkitdevelopedbytheNvidiaCorporationforacceleratedgeneral-purposecomputing.

DPO

Directpreferenceoptimisation.Fine-tuningmethod.

FLOP

Floating-pointoperation.Computationalresourcesrequiredformodeltrainingismeasured infloating-pointoperations.

GAN

Generativeadversarialnetwork.Modelarchitectureusedinimagesynthesis.

GPT

Generativepretrainedtransformer.AImodelarchitecture.

GPU Graphicsprocessingunit.

IaaS Infrastructureasaservice.

AItechnology

IPO

LLM

Thestudyanddevelopmentofartificialintelligence.

Identitypreferenceoptimisation.Fine-tuningmethod.

Largelanguagemodel.Artificialintelligencemodelusedfornaturallanguageprocessing, distinguishedbythelargenumberofparametersinvolved.

LSTM

Longshort-termmemory.Modelarchitecturewidelyusedinlanguagemodelsbeforethe adoptionoftransformers.

ML Machinelearning.

MoE

MixtureofExperts.Modelarchitecture.

NPU

Neuralprocessingunit.Artificialintelligenceacceleratormainlyusedinphones.

OWASP

OpenWorldwideApplicationSecurityProject.Webcommunityaggregatingandproducing webapplicationandsoftwaresecurityresources.

PaaS Platformasaservice.

RAG

Retrieval-augmentedgeneration.Methodusedforthedeploymentofartificialintelligence applicationswherethelanguagemodelsinheritsadditionalcontextfromadatabaseoranotherexternalsourcebasedonauserpromptforimprovingresponsequality.

RLHF

Reinforcementlearningwithhumanfeedback.Fine-tuningtechnologyutilisingreinforcementlearning.

RNN

Recurrentneuralnetwork.Modelarchitecturewidelyusedinlanguagemodelsbeforethe adoptionoftransformersandLSTM.

SaaS

Softwareasaservice.

SFT

Supervisedfine-tuning.AImodeltrainingmethodthat,unlikepre-training,issupervisedand isusedforthefurtherguidanceofthemodel’swork.

TPU

Tensorprocessingunit.AIacceleratordevelopedbyGoogle.Corporation

VAE

Variationalautoencoder.Modelarchitectureusedinimagesynthesis.

XAI

ExplainableAI.Collectionofmethodsfortheexplanation,interpretation,andvalidationof theworkofAImodelsandtheresultsofthiswork

1.3Structureofthereport

WebeginourreportwithanoverviewofthehistoryofAIandmainAItechnologies(Section 2 ). Wewillthenmoveontotheirapplications,presentingexamplesofareasoflifeinwhichadditionalvalueishopedtobegainedfromAI.Thefielditselfhasbeendevelopingrapidlyduring thewritingofthisreport;hence,wewillalsoincludeanoverviewofcurrenttrends.

Countriesallacrosstheworldhavebeguntolegallyregulateartificialintelligence.Section 3 providesareviewofthecurrentstateofthislegislation.Section 4 focusesonthearchitecture ofAIsystemsandpresentsthreegeneralmodelsforthedeploymentofAIapplications.These threedeploymentmodelsformagoodbasisfororganisationsfortheapplicationoftheirrisk assessmentmethodologies.

Alongsidelegalconsiderations,applicationsofAItechnologymustalsotakeintoaccountcybersecurityandsocietalsafetyrequirements.Guidelinesforrelevantriskassessmentmeasures arepresentedinSection 5 .Theexistenceofrisks,meanwhile,alsonecessitatestheapplication ofmitigatingmeasures.ThesearereviewedinSection 6 .

Section 7 summarisestherecommendationsforthepromotionoftheapplicationofAIsystems inEstoniadevelopedinthecourseofthisstudy.

Thelastpartofthereportisthemostpracticalandismainlytargetedatthoselookingforquick solutionsforanalysingtherisksofAIsystems.Thispartpresentsspecificandeasy-to-follow guidanceforidentifyinganddealingwiththemainrisksinthecreationordevelopmentofanAI system.RelevantguidelineswithsupportingfigurescanbefoundinSection 8 .

2OverviewandusecasesofAI applications

2.1Historyofartificialintelligencetechnology

Artificialintelligence(AI)isunderstoodhereinasanysystemcapableofperformingtasksseeminglyemployinghuman-levelintelligence.Figure 1 presentsanoverviewofimportantmilestones inthehistoryofAI.AIasafieldgrewoutofcybernetics,thegoalofwhichwasthestudyof feedbacksystems,includingbiological,technological,andsocialsystems.Althoughtheidea andstructureofartificialneuronswasalreadyproposedinthe1940s,thehistoryofartificial intelligenceistracedtoasummerseminarheldatDartmouthin1956wherethetermwasfirst proposed.

Theparticipantsoftheseminarreachedtheconclusionthatmachinescanbemadetoperform alltaskstiedtohumanintelligence.Indeed,theyconsideredcomputerstobecapableofindependentlearning,languageuse,andcreativity.Eventhoughnobigbreakthroughsweremade duringthetwo-monthseminar,overthenext20years,itsparticipantsfiguredamongthemain promotersofAItechnology.TheAIsystemsdevelopedinthisperiodwerecapableofsolving mathematicalproblems,playingcheckers,andtranslatingtextsfromonelanguagetoanother.

Figure1.HistoryofAIdevelopment

1958sawthebirthofthehigh-levelLispprogramminglanguagethatbecamethemainlanguage ofAIsoftwareforthenextthreedecades.Theseeminglymajoradvancesandsolutionsdevelopedinthisperiodfellrathershortinreality,though.Translationprogramsemployedliteral translationandremainedthusunabletorelatethemeaningofphrases.Programsforproving mathematicaltheoremsorplayingcheckerswereonlycapableofreviewingalimitednumberof statesandfailedtosolvemorecomplexproblems.

Problem-solvingwasdemonstratedinsmallplay-environmentscalledmicroworlds.Perhaps themostfamousofthemicroworldswerevirtualblocksworldsthattheusercouldmanipulate

usingEnglish-languagecommands,e.g.viatheSHRDLUlanguageparser.Eventhoughgenetic algorithmsandthebasicprinciplesofartificialneuralnetworkswerealreadyproposedinthe late1960s,littleprogresswasmadewiththesealgorithmsduetotheirlowlevelofoptimisation andinsufficientcomputationalpower.

ThehopesraisedbytheemergenceofthefirstAIsystemsledmanyresearcherstomake promisesthatcouldnotbefulfilled.ThisledtodisappointmentamongthebackersofAIresearchandadeclineinAIresearchanddevelopmentinthe1970s.BoththeUKandtheUS significantlycutAIfundingforuniversities,andtheUSDefenseAdvancedResearchProjects Agency(DARPA)stoppedfundingAIprojectsaltogether.Thiserafrom1974to1980iscalled thefirstAIwinter.

Inspiteofthefundingcuts,thedevelopmentofAIstillcontinued,butinsteadofsolvinglarge andcomplexproblemsthefocusnowturnedtosystemsconcentratingknowledgeprovidedby expertsindifferentfieldsandusingthisforthesolutionofnarrowerproblems.Suchso-called expertsystemswereusedine.g.medicineandanalyticalchemistry.Expertsystemswerealso successfullystudiedbyEstonianresearchers(includingEnnTõuguandLeoVõhandu).

Thesuccessofexpertsystemsledtorenewedpublicinterestinartificialintelligenceintheearly 1980s.Oneofthefirstcommercialrules-basedsystemswasR1,asystemthatassistedclients inconfiguringcomputersinaccordancewiththeirrequirements.In1981,theso-calledFifth GenerationComputerSystemsprojectwasannouncedinJapan.Theprojectinvolvedadecadelongplanforthedevelopmentofintelligentcomputers.Thisalsocreatedrenewedinterestin artificialintelligenceintheUSandtheUK.

ThenewAIboompeakedinthesecondhalfofthe1980s.LargeAmericancorporationscreated workinggroupsfocusingonAIsystems.Thefocusonceagainturnedtoartificialneuralnetworks andtheirtrainingusingbackpropagationalgorithms.Mathematicalandstatisticaloptimisation methods,aswellasspecialisedlanguagesandsoftwarewereincreasinglyemployedforthe developmentofAIalgorithms.Thebest-knownAI-specificlanguagesweretheallpartsofthe Lispfamilyofprogramminglanguages.Specialcomputers–Lispmachines–weredevelopedto runprogramswrittenintheselanguagesmoreefficiently.

Inspiteofthelargeadvancesmade,1987markedthebeginningofasecondAIwinter.The maintenanceandupdatingofspecialisedartificialintelligenceswascomplicated;theywere alsounabletoindependentlyhandlepreviouslyunfamiliarinputs,leadingtothemquicklybecomingobsolete.IBMandAppleproducedeverhigher-performancegeneral-purposedesktop computers.Special-purposemachines(includingLispmachines)losttheirusefulness.Thefifth generationcomputerprojectfailedtoyieldthehoped-forresults.Thus,1991shouldhaveseen thecompletionofartificialintelligencecapableofholdingeverydayconversationswiththeuser; itwouldtakedecadesbeforethisgoalwasfinallyreached.Disappointedinthelimitedcapabilitiesofexpertsystems,DARPAagaindrasticallyreducedfundingforAIsystemsresearch.

SubsequentdevelopmentofAItechnologywasincreasinglyfoundedonexactmathematical methodsdevelopedinthepast.Thefocusonceagainmovedtorigorouslogicandsolutions weresoughtfromcontroltheory,asubfieldofcybernetics.Atthesametime,researchersalso begantoutiliseprobabilitymodelsandfuzzylogicenablingthemtodescriberelationshipsand conditionalprobabilitiesoffeaturesand,unlikepurelogic,expresslackofknowledgeanduncertaintyinforecasts.

The1990ssawtheriseofdataminingandmachinelearningalgorithms.Systemswerenolonger describedonlybyprogrammersandexperts:thecomputersbecamecapableofindependent learningthroughtheanalysisoflargedatasets.AItechnologyandprobabilitymethodsweretied

togetherbyBayesiannetworksallowingtheconditionalprobabilitieslinkingdifferentvariables tobeexpressedintheformofdirectedgraphs.AnewparadigmemergedinAIthatsawartificial intelligencesasagentsreceivingsignalsfromtheenvironmentandattemptingtooptimisetheir behaviourfortheachievementofcertaingoals.ThegreatestachievementofAItechnologyin the1990scouldbeconsideredtobethevictorybythechess-playingsystemDeepBlueover thereigningchessworldchampionGarryKasparovonMay11th,1997.Bythispoint,AIsystems alsobegantobeutilisedineverydayservices,especiallyweb-basedsolutions.Naturallanguage processingwasthusemployedbytheGooglePageRanksearchalgorithm,alsocreatedin1997. Thealgorithmrankedthepagesdisplayedafteruserqueries;thisisconsideredoneofthecritical piecesoffunctionalitysettingGoogleapartfromotherexistingsearchengines.

Naturallanguageprocessingwasalsoemployedinspeechsynthesismodels,suchasDECtalk, usedashisspeechsynthesiserbyStephenHawking,aswellastheslightlymorecomplexBell LabsTTS(Text-to-Speechsystem),capableofsynthesisingspeechinseveraldifferentlanguages.Fornearly20years,startingfromtheearly1990s,machinetranslationasafieldwas dominatedbystatisticalmodelsdevelopedatIBM.Meanwhile,hiddenMarkovmodelsbecame predominantinspeechrecognition.Themainapproachtofacerecognitioninthe1990sconsistedintheuseofeigenfacealgorithmsemployinglinearalgebraicmethodsfortheanalysisof facialfeatures.

Inspiteoftheadvancesmadebyartificialintelligencesystems,thetermAIwasstillfrowned uponattheendofthe1990s.Researchersavoidedtheterm,preferringtospeakofstatistical methods,machinelearning,andcontroltheoryinstead.TheendofthesecondAIwinteris notclearlydefined,butitisgenerallyagreedtohaveendedby2005whentheStanford-built self-drivingcarStanleycoveredthe212kmDARPAGrandChallengetrailintheNevadadesert inlessthansevenhours.Thiswasamajorstepforward,consideringthatduringtheprevious year’sten-houreventnoneofthecompetingvehicleswereabletocovermorethan12km.Two yearslater,DARPArepeatedthecompetitioninacitysetting.Thewinnerofthischallengewas theCarnegieMellonUniversityBossrobotwhichcovered96kminlessthansixhoursinthese conditions.

In2011,IBMdemonstratedtheirquestion-answeringsystemWatsonontheUSTV-showJeopardy!(AlsopopularinEstoniaunderthenameKuldvillak).Intwoconsecutiveshows,Watson competedagainsttwohumanplayers(oneofwhomwasKenJennings,regardedasoneofthe bestJeopardy!playersinhistory)winningbothgamesbyagoodmargin.Watson’ssuccess wasfoundedonideasderivedfromavarietyoflanguagemodelsandlargecomputingpower, enablingthesystemtobetrainedonlargedatasets.Erroranalysiswascontinuouslycarried outthroughoutthetraining,andtheprogramwasconstantlyimproved.Nevertheless,Watson’s performancewasnotcompletelyflawless.Forinstance,duringtheFinalJeopardy!roundofthe firstshow,Watsongavetheanswer’Toronto’toaquestionaboutUScities.

Oneofthegreatestbreakthroughsoftheartificialintelligenceeracamein2012whenthe AlexNetconvolutionalneuralnetworkwontheImageNetLargeScaleVisualRecognitionChallenge(ILSVRC)byalargemargin.AlexNetwasnotthefirstconvolutionalneuralnetwork;the architecturewasfirstproposedbyYannLeCunbackin1989.Thebreakthroughwascatalyzed bytrainingalgorithmsoptimisedforspecialisedgraphicsprocessingunitsenablingthetrainingoflargeranddeeperneuralnetworksthaneverbefore.TheImageNetdatabasecontained 15millionimagesfrommorethan22000categories.InthefollowingImageNetcontests,all winningideaswerebasedonconvolutionalneuralnetworksandAlexNet’sresultwasimproved multipletimes.Today,theImageNetchallengeisconsideredtohavebeensolved.

AftertheAlexNetbreakthrough,neuralnetworkshavebeensubjecttoactivedevelopment.

Alongsideconvolutionalneuralnetworks,significantattentionwasalsogarneredbylargelanguagemodels,recurrentneuralnetworks,longshort-termmemorymodels.This,inturn,ledto therapiddevelopmentofspeechrecognitionandsynthesisandtranslationmodels.Artificial intelligencewaswidelyadoptedinmedicine,industry,andfinance.Recurrentnetworksbegan toseeuseintimeseriesanalysis,robotics,andgames.Notably,theAlphaGosystemreceived greatattentionafterdefeatingaprofessionalhumanplayeratGoin2015.

Asatthetimeofthisreport,themainpublicattentionisdirectedtogenerativeAImodelscapableofcommunicatinginhumanlanguage,answeringquestions,seeminglylogicalreasoning, generatingimagesandmusic,andassistingprogrammersinwritingcode.Whiletheconcept ofgenerativemachinelearningmodelsishardlynew,themainachievementsrelatedtodeep generativeneuralnetworksdatetothepreviousdecade.Generativeadversarialmodelsand variationalautoencoderswereintroducedin2014,bothofwhichareimportanttoolsforimagesynthesis.Generativeadversarialmodelsallowedsynthesisinghigh-resolutionimagesof humanfacesforthefirsttime.

In2015,itwasdemonstratedthatthemethodsofstatisticalphysicscanbeusedfortraining generativediffusionmodels.Perhapsthebiggeststepforward,however,cameintheformof attentionmechanismtransformers,thebasicarchitectureofwhichwasproposedbyGooglein 2017.Transformersareatthecoreofanumberofwell-knowngenerativelanguagemodels, suchasGPTandBERT,aswellastheGitHubCopilotcodecompletiontool.

Transformersenabletheconstructionofparallelisablemodelswithlongcontextwindowsthat canbetrainedunsupervisedonlargedatasets.Unsupervisedmodelscanalsoberetrained forspecifictasksthroughtransferlearning.Thisisavitalfeature,foratime-andresourceconsuminguniversalmodelonlyhastobetrainedonceinsuchcase.Thismodelcanthenlater beeasilyadaptedtoaspecificproblemusingamuchsmallerdatasetandfarfewerresources.

Imagesynthesis,ormorespecifically,text-to-imagemodelsalsousetransformers,buttheir architectureisgenerallymorecomplex.DALL-E3andStableDiffusionuseanautoencoderfor encodingimages;theencodeddataareusedfortrainingdiffusionmodels,inturnmadeupof convolutionalneuralnetworks.

2.2Artificialintelligencealgorithmsandtaxonomies

Theterm’artificialintelligence’isverybroadandencompassesmethodswithlargedifferences incomplexity,explanatorypoweranddepth,aswellasareasofuseandtrainingalgorithms.On ahigherlevel,artificialintelligencealgorithmsaredividedintorule-basedsystems,traditional machinelearningalgorithms,andneuralnetworks.

2.2.1Rule-basedsystems

Rule-basedsystemsarethesimplestartificialintelligencesystems.Ingeneral,thesesystems consistofrulescreatedbyhumanexpertsthatthecomputercanthenfollowtosolveproblems seeminglyrequiringhumanintellect.Forexample,rule-basedsystemsaregoodatsolvingcertaintypesoflogicalthinkingexercisesandpuzzles(e.g.so-calledEinstein’spuzzlesandzebra puzzles).

2.2.2Machinelearning

Machinelearningmeansthatthecomputerlearnstosolveataskbasedonexistingdata(which couldincludemachine-readablerepresentationsofsensors,previousevents,etc.).Machine learningutilisesmathematicaloptimisationmethodswhichtheprogramusesforfindingamaximallyaccuratesolutiontotheinitialproblem.Thisallowsthesystemtosolvetaskswherethe solutionalgorithmisdifficultforahumantodescribeusingpreciseinstructions.

Machinelearningmethodscanbecategorisedinvariousways.Forexample,fromtheperspectiveofapplicationsandtrainingdata,machinelearningcanbedividedintosupervisedand unsupervisedmachinelearningandreinforcementlearning.

2.2.2.1Supervisedandunsupervisedmachinelearning,reinforcementlearning

In supervisedmachinelearning ,thegoalofthetrainingalgorithmistocreateamodelcapableof predictingvaluesorvectors,alsoknownaslabels,basedontheinputreceived.Inunsupervised learning,themodelbeingtrainedispresentedwithtrainingdatawhichincludesbothinputs andthecorrespondinglabels.Themodelcancontinuouslycompareitspredictionswithcorrect labelsandusethecomparisonresultsforimprovingitspredictioncapacity.Supervisedmachine learningisusedinalmostallfieldswheremachinelearningisutilised,suchasmedicalresearch, image,text,andvoicerecognitionorprocessing,andthetrainingofsearchenginesandspam filters.

Supervisedmachinelearningtasksaredividedintoclassificationandregressiontasks.Thegoal ofclassificationmodelsistopredictwhichofthetwoormoreclassesagivenrecordbelongs to.Regressionmodelstrytoprovideamaximallyaccuratepredictionofthenumericalvalue correspondingtotherecord.

In unsupervisedmachinelearning ,labelscorrespondingtotherecordseitherdonotexistor themodelcannotseethem.Thegoalofthealgorithminsuchcasesistoidentifyrelationships orstructurewithinthedatawithouttheaidoftraininglabels.Unsupervisedalgorithmspermitthedimensionalreductionofthebasedata(principalcomponentanalysis)orgroupingof similarrecords(clustering).Unsupervisedmachinelearningmethodsareusede.g.ingenetics fortheidentificationofsub-populations,aswellasfortraininggenerativemodels,suchasautoencoders.Unsupervisedmethodsareoftenalsousedpriortotheemploymentofsupervised machinelearning.

Anotherclassofmethodsalongsidesupervisedandunsupervisedmachinelearningalgorithms isreinforcementlearning.Inthecaseofreinforcementlearning,noteverysingleinputwillbe pairedtoanoutput.Thealgorithmwillinsteadlearntoselectactionsbasedontheenvironment sothattherewardfortheseactionsismaximised.Forexample,reinforcementlearningcanbe usedforspeechprocessingorteachingthecomputertoplaygames.Reinforcementlearning wasthususedfore.g.trainingAlphaGo.

Transferlearning isamachinelearningtechniquewhereininformationacquiredfortheperformanceofonetaskisalsousedforperformingothertasks.Forexample,trainedgeneral-purpose languagemodelscanbeusedfortheperformanceofdifferentlinguistictaskswithoutanyadditionalfine-tuningofthemodel(seeSection 2.2.4.1).

2.2.2.2Machinelearningalgorithms

Linearregression (Figure 2 )isoneofthesimplestsupervisedmachinelearningmodels.Asa statisticalmodel,ithasactuallybeenusedforcenturies.Themodelisusedfortheprediction ofarealnumberoutputvaluefrominputdata.Asperthename,linearregressionisusedfor modellingalinearrelationshipbetweenaninputandanoutput.Thetrainedmodelisthuseasily explainable,asiteasytosurmisefromthemodelitselfhowachangeintheinputvaluewill influencetheprediction.

Logisticregression (Figure 2 )isverysimilarinnaturetolinearregression;inspiteofitsname, however,itismainlyusedforclassificationanalysis.Inthecaseofbinarylogisticregression, thepredictionalgorithmfirstemploysalinearfunction,theoutputofwhichcanbeinterpreted asthelogarithmoftheprobabilityofalabel.Theoutputisthenpassedtoasigmoidfunction thattransformstheoutputvaluetoaprobabilityintherange [0, 1].Logisticregressioncanalso easilybeadaptedtosituationswheretherearemorethantwooutputclasses.

Supportvectormachines aresupervisedmachinelearningmethodsinitiallydevelopedforclassificationtasks.Thesimplestsupportvectormachineisalinearclassifiertaskedwithfindinghyper-levelsdemarcatingrecordsofdifferentclasses.Linearclassifierspresumethatdata classesarelinearlyseparable,whichis,however,generallynotthecase.Thishasledtothe developmentofanumberofadaptationsovertimewhichenablesupportvectormachinestobe trainedfornon-linearclassification,regressionanalysis,exceptionfinding,anddimensionality reduction.

Supportvectormachinesareusedinimageandtextclassification,butalsoine.g.biology.The mainweaknessofsupportvectormachinesistheirdifficultexplainabilityandhighercomputationalcomplexityintraining.

Decisiontrees (Figure 3 )aresupervisedhierarchicaldatastructure-basedmodelsutilisedfor regressionandclassificationanalysisasaseriesofrecursivedecisions.Thetreeconsistsof testnodesandendnodesorleaves.Inthetestnodes,theinputissubjectedtotestswhich areusedforchoosingthenextbranches.Leavesreturntheoutputcorrespondingtotheinput basedonthetestsperformed.

Decision-makingcanbeenvisionedasaseriesofyes/noquestionswereeachnewquestion dependsonapreviousoneandthefinalpredictedvaluedependsoneachsingleanswer.Decisiontreesareeasilyexplainableandintuitivelyunderstandablemodelswhichhasmadethem historicallyextremelypopular.

ThenaiveBayesmethod isaclassificationalgorithmutilisingtheBayestheoremforthepre-

dictionofthemostprobablelabelsbasedonaninput.Thismethodpresumesthattheinput featuresusedfortrainingthemodelareindependentofeachother.Nevertheless,thenaive Bayesmethodhasbeenhistoricallypopularduetoitssufficientpower,aswellassimpleexplainabilityandtrainability.Unlikemanyothermachineslearningalgorithms,thesolutionofthe naiveBayesmethoddoesnothavetobefoundiniterativesteps,astheformulaforassessing thehighestprobabilitycanbepresentedinanexplicitform.

The k-nearestneighbouralgorithm (Figure 4 )isasupervisedalgorithmthatcanbeusedfor solvingbothregressionandclassificationtasks.Asperthemethod’sname,predictionsare madebasedon k nearestneighbourswhere k isapositiveinteger.Inthecaseofclassification tasks,thealgorithmdetermineswhichclasshasthehighestrepresentationamongthe k nearest neighbours.Inthecaseofregression,thepredictedvalueistheaverageofthevaluesof k nearestneighbours.Thepredictionscanbemodifiedbyassigningweightstotheneighbours basedontheirdistancefromtheoriginalrecord.Distancesbetweendifferentpointscanbe measuredusingdifferentmetricsbasedontheinitialproblem.

Thenearest-neighbourmethodispopular,asthereisnoneedforpre-training:predictionsare madebasedonthetrainingdata.Themodelisalsoeasilyexplainable.Themaindrawbackof themodelisseeninthefactthatthemethodisalocalone,i.e.predictionsarebasedonafew individualrecordswhiletherestofthetrainingdatasetisignored.

Principalcomponentanalysis isanunsupervisedalgorithmthatallowstranslatingdatatoa moreeasilyexplainablecoordinatesystemusinglineartransformations.Principalcomponent analysisisoftenutilisedthedimensionalreductionofthedataset.Thisisespeciallyusefulin situationswheremanyfeaturesfoundinthedatasetarestronglycorrelatedtoeachother.First principalcomponentsarevectorsthatmaximallyrepresentthevarianceofthedatauponmapping.Mappingthedataontothefirstprincipalcomponentsalsoenablestheclusteringofthe datatobestudiedvisually.

The k-meansmethod or k-meansclusteringmethodisanunsupervisedmachinelearningalgorithmthatdividesthedatarecordsinto k differentclusterswhere k isapositiveinteger.The k-meansmethodshouldnotbeconfusedforthe k-nearestneighbourmethodwhichisasupervisedmethodology.Whereas,inthecaseofthe k-nearestneighbourmethod,predictionscan bemadebyonlylookingatthenearestpointstotherecord.The k-meansmethodlooksforan optimumclusteringforallpointswhichmakestrainingmuchmoredifficultandtheinterpretation oftheoutputrequiresidentifyingalltherecordsthatwereclusteredtogether.Clusterscanbe usedforidentifyingrelationshipswithinthedataset.Clusteringyieldsthecentreofeachcluster whichcanbeusedine.g.,signalprocessingasarepresentativeclusterpoint.Themethodcan alsobeusedforautomaticfeaturelearningwhichallowsinputdatatobetranslatedtoaform suitableforothermachinelearningmethods.

HiddenMarkovmodels (Figure 5 )arestatisticalalgorithmsmodellingMarkovprocesses,i.e., seriesofpossibleeventswheretheprobabilityofeachfollowingeventonlydependsonthe stateoftheprocessafterthepreviousevent.Markovprocessstatesarenotobservableina hiddenMarkovmodel.Theonlythingsthatareobservablearetheeventsdirectlyinfluencedby thehiddenstates/events.Thegoalistousetheobservableeventstostudythehiddenstates andevent.

Figure5.ExampleofahiddenMarkovmodelofactivitiesfordifferentweatherconditions

Ensemblemethods (Figure 6 )aretechniquescombiningdifferentmachineslearningmodels. Combinedmodelsareoftenbetterandmorestablethanindividualmodelsbythemselves.Var-

iousmethodsexistforthecombinationofmodels:bootstrapaggregatingorbagging,stacking, boosting.Thebest-knownensemblemethods,suchasdecisionforestsandgradient-boosted treescombinedifferentdecisiontrees.Diffusionmodelshavealsobeenusedforthegeneration ofneuralnetworkparameters[1].Ensemblelearningisalsoknownasmeta-learning.

2.2.3Artificialneuralnetworks

Artificialneuralnetworksaremachinelearningmodelsthatattempttoimitatetheoperationof thehumanbrain.Neuralnetworksconsistoflayersofnodes,thebehaviourofwhichshouldbe similartotheneuronsfoundinthebrain.Eventhoughthefirstneuralnetworkswerebuiltas earlyasinthe1950s,theyonlysawrealsuccessaboutadecadeagowiththecreationofthe firstconvolutionalneuralnetworkscapableofachievingbetterresultsinimageprocessingand facerecognitionthananyotherexistingalgorithm.

Increasesincomputingpowerandreductionsinrelatedcostshavecreatedtheconditionsfor traininglargeandcomplexneuralnetworkswhichhasledtoakindofraceinbothresearch andimplementationofsuchsystems.Today,modelsbasedonneuralnetworksarecapable ofsolvingtasksthatwereconsideredimpossibleamerefewyearsago.Neuralnetworksare generallydifficulttoexplainandthetrainedmodelsareseenasblackboxes.Asaresult,more easilyexplainablemachinelearningmodelsofsimilarpredictivecapacityareoftenpreferredto neuralnetworks.Thestudyoftheexplainabilityofneuralnetworksisanactivefieldofresearch.

2.2.3.1Neuralnetworkarchitectures

Fullyconnectedneuralnetworks (Figure 7 )areoneofthefirstneuralnetworkarchitectures everdeveloped.Afullyconnectednetworkismadeupofaseriesoffullyconnectedlayers whichinturnconsistoflinearnodes,theoutputsofwhicharesubjectedtonon-linearactivation functions.

Convolutionalneuralnetworks areneuralnetworkscomprisingoneorseveralhiddenconvolutionallayers.Whereasfullyconnectedlayerscompriselinearnodesorweightscorresponding toeachinputvalue,aconvolutionallayerismadeupofsmallkernels/filtersmakingthelayers

Figure7.Artificialneuralnetworksconsistofdifferentlayersandnodes.

smallerandenablingthemtobeusedforthecreationofdeeper(i.e.involvingmorelayers) neuralnetworks.

Thebest-knownapplicationofconvolutionalneuralnetworksisartificialvision.Inthecaseof facerecognition,aconvolutionalnetworkcanidentifydifferentfeatureslayerbylayer,beginning withlinesandangles,followedbyeyesandthemouth,andendingwiththecompletehuman face.Convolutionalnetworkshadarealbreakthroughin2012.withAlexNetwhichbeatother contestantsintheImageNetLargeScaleVisualRecognitionChallengebyahugemargin.From thenon,convolutionalnetworkshavebeenthemaintoolofartificialvision.Convolutionalnetworksarealsosuccessfullyusedintextprocessingand,toalesserextent,otherspecialtytasks.

Bothfullyconnectedandconvolutionalneuralnetworksareexamplesoffeed-forwardnetworks wheretheoutputofahiddenlayeristheinputforthenextlayer,i.e.informationonlyflowsina singledirectionthroughthenetworklayers.Incaseinformationcanalsoflowinacyclicmanner withintheneuralnetwork,i.e.alayer’soutputisfedbackintothenetworkandcaninfluence laterinputstothesamelayer,thiskindofneuralnetworkiscalledarecurrentneuralnetworks. Recurrentneuralnetworksaremainlyusedfortheanalysisofdataseries,astheycankeep trackoftheprecedinginputswithinthesameserieswhenprocessingatraininginput.Recurrent neuralnetworksarewidelyusedin,e.g.,languagemodels,textgeneration,speechrecognition, artificialvision,videolabelling.

Thetrainingofrecurrentneuralnetworkscanberenderedunstablebythe’explosion’or’vanishing’ofgradientsduringbackpropagation.Tomitigatethisproblem,longshort-termmemory (LSTM)neuralnetworkshavebeenadoptedasasubsetofrecurrentnetworks.Atthecoreof LSTMarecellswithinput,output,andforgetgatescontrollingtheflowofinformationthrough thecellinordertopreventgradientexplosionorvanishingduringbackpropagation.

Transformers aredeeplearningmodelsusingattentionmechanismsfortheanalysisofsequentialdata.Transformerscameintothelimelightin2017whenitwasshownthat,whenapplied tonaturallanguageprocessing,theyarecapableofidentifyingthecontextcorrespondingto atokenbasedontheprecedingsequencewithouttheiterativeanalysisofthissequence.An inputofacertainlengthisanalyzedasawholeandanattentionmechanismisusedtoidentify thesignalsmostrelevanttoeachtokenintheprecedingsequenceoftokens.Thisenablesthe modelstobetrainedinparallel,thusreducingcomputingcostscomparedtoe.g.LSTMs. UnlikeLSTMs,inthecaseoflonginputs,transformerslackthecapacitytokeeptrackofthe

entireprecedingseriesandcanonlytrackacertainsegmentoftheserieswhichcanprove problematicinanalyzinglongtexts.Transformersgenerallyconsistsofanencoderandadecoder,thefirstofwhichanalyzestheinputantthesecondgeneratestheoutputstep-by-step. Thecoderanddecodercanbeusedbothsimultaneouslyandseparately.Forinstance,GPTis apurelydecoder-basedandBERTapurelyencoder-basedmodels;thereare,however,models suchasT5thatemploybothanencoderandadecoder.

Transformersareusedinthetrainingofbothsupervisedandunsupervised,aswellashybrid models.Largelanguagemodels,suchasBERTandGPT,arefirsttrainedunsupervisedona largesetoftexts.Themodelwillthenbetrainedonasmaller,labelleddatasetforaspecific task.Transformer-basedmodelshaveachievedalmostcompletedominanceamonglanguage modelintherecentyears.Inotherfields,however,nosimilarsuccesshasbeenobserved.For instance,inartificialvision,convolutionalneuralnetworksarestillpreferredtotransformers, evenifattentionmechanismsarealreadyemployedinthese.

An autoencoder isanunsupervisedneuralnetworkcomprisingbothanencoderandadecoder. Inputisreceivedbytheencoderandtransformedtoanotherform,whilethedecoderattempts toreconstructtheoriginalinputfromthetransformedinput.Thetrainedencodercanthenbe usedfordimensionalreductionoftheinputdataandthedecoder,forthegenerationofnew data.Inmostcases,thegenerativecapacityofanautoencoderislimited,astheproximityof thedecoder’sinputsdoesnotguaranteethesimilarityofoutputs.

Variationalautoencoders (VAE)havebeenproposedforuseindatageneration,e.g.image synthesis.VAEsdifferfromordinaryautoencodersinthattheencodermapsaninputtoadistribution,ratherthanasinglepoint,e.g.,byoutputtinganormaldistributionmeanvalueand covariancematrix,whereasthedecoderwillbegivenarandomvectorfromthisdistributionas aninputwhichitwillthentryanduseforreconstructingtheencoder’soriginalinput.Unlike autoencoders,trainedVAEdecoderswillusuallygeneratesimilaroutputsforproximateinputs.

A generativeadversarialnetwork (GAN)isagenerativemodelwheretwoneuralnetworks–a generativeandadiscriminativeone–contestwitheachotherfortrainingthemodel.Bothneural networksaretrainedsimultaneously.Thegenerativemodelreceivesaninputfromasimpledistributionandattemptstousethistogenerateanoutputfromacomplexdescriptivedistribution, whereasthediscriminativemodelattemptstodistinguishtheoutputsofthegenerativemodel fromrealdata,thedistributionofwhichthegenerativemodelisattemptingtoimitate.GANscan beusedin,e.g.,imagesynthesiswherethegenerativemodelisgeneratingimagesofhumans whilethediscriminativemodelisattemptingtodistinguishtherealimagesfromthegenerated ones.Generativeadversarialnetworksarealsousedinspeechandtextsynthesis.

Diffusionmodels aregenerativemodelsbasedonMarkovprocesses.Diffusionmodelsare somewhatsimilartoautoencoders,inthattheycompriseaforwardprocesswherenoiseis addedtorealdatastep-by-step,andareverseprocessattemptingtorecreatetheoriginalinput throughthegradualremovalofthenoise.Ingeneral,noiseusedfortrainingdiffusionmodels isgeneratedusinganormaldistribution;aftertheadditionofasufficientamountofnoisethe originalinputwilldisappearcompletelyandtheoutputwillonlyconsistofrandomnoise.

Ifalotofnoiseisaddedtotheinputatonceitwillbeextremelydifficulttopredicttheoriginal input,butitturnsoutthatwhennoiseisaddedinsufficientlysmallincrements,themostrecent additionofnoisecanbepredictedandremovedusing,e.g.,aneuralnetworkfortheprediction.Thetrainedmodelcanbesequentiallyappliedtoacompletelyrandominputandusedto generateanoutputsimilartorealdata.

Theefficiencyofsuchtrainingstemsfromtheknowledgethatifnoiseisgeneratedfromanormal

distributionandaddedsequentially,thenalloftheaddednoisealsooriginatesfromanormal distribution.Thesumofnoisefromseveralincrementscanthusbesimultaneouslyaddedto theoriginalinputduringtraining,andtheneuralnetworkcanbeaskedtopredictonlythesmall amountofnoiseaddedinthelateststep.

Diffusionmodelsoriginatefromstatisticalphysics.In2015,itwasdemonstratedthattheycan alsobeusedforimagesynthesis.Subsequentstudyofthesemodelshasgivenrisetothe realisationthatdiffusionmodelsaremorepowerfulandstableyetlessresource-intensivethan, forexample,generativeadversarialnetworksthatwerepreviouslythebestimage-generating models.Today,diffusionmodelsandtransformersarethemaincomponentsoftext-to-image models,suchasDALL-E3andStableDiffusion.

2.2.4Largelanguagemodels

Largelanguagemodels(LLMs)aregenerallytransformer-basedtextsynthesismodels,distinguishedbythelargenumberofparametersandamountoftrainingdataused.Non-transformer basedlanguagemodelsalsoexist.Variousarchitectures,suchasRetNet[2 ],RWKV[3 ],and Mamba[4 ]havebeendevelopedthatcanalsobeusedforthecreationoflanguagemodels,offeringsolutionsfortheweaksidesoftransformerarchitecture.Alargepartofrecentinnovation inmachinelearningandartificialintelligencehasbeenrelatedtothedevelopmentofLLMsand theadoptionofproducts(suchasChatGPT)builtonLLMs.

Accordingtoonehypothesis,shouldartificialgeneralintelligence(AGI)provepossibleatall,it canonlybedevelopedonthebasisofmultimodallargelanguagemodels[5 ].DemisHassabis fromtheAIdeveloperDeepMindhasopinedthat’multimodalfoundationalmodelsaregoingto bekeycomponentofAGI’1 .ConceptionsanddefinitionsofAGIvary,however,andsomeclaim thenecessaryleveloftechnologyhasalreadybeenreached[6 ].

2.2.4.1Training

Aswithallotherartificialintelligencemodels,themodelarchitectureneedstobetrainedafter beingestablished.ThetrainingofLLMsusuallyinvolvesseveralsteps,noneofwhichare,however,strictlyrequired.ThetrainingprocessofLLMsandthechoicesmadeintheprocessare closelytiedtothedeploymentmodelsoftheAIapplicationsfoundedupontheLLM.

Pre-training isthefirst,unsupervisedstageoftrainingwherethemodelisfedtextsequences containingmaskedelementsandisinstructedtopredicttheseelements.Theselectionofthe maskedelementsisautomatic.Pre-trainingisthemostcompute-intensiveprocessinvolving hugeamounts( ∼trilliontokens)ofunlabelled,low-qualitydata,usuallyacquiredthroughweb crawling.Pre-trainingyieldsapre-trainedmodelthatcangenerateacontinuationtoaninput basedonwhatithaslearnedfromthetrainingdata.Thiscontinuationmaynotnecessarily beuseful:whenthepre-trainedmodelisaskedaquestionitcangenerateananswertothe question,oritmaygenerateacontinuationorfollow-upquestions.

Supervisedfine-tuning (SFT)isthesecondstageoftrainingmeanttotunethemodelfora specificpurpose.Forinstance,inthecaseofchatbots,itisspecificallypreferredthatthesystem generateanswers,nototherkindsofoutputs.Trainingdatausedforfine-tuningareoften, althoughnotalways,assembledandlabelledbyhumans.Theirqualityishigherandquantity

1 TheGuardian:’GooglesaysnewAImodelGeminioutperformsChatGPTinmosttests’. https: //www.theguardian.com/technology/2023/dec/06/google-new-ai-model-gemini-bard-upgrade Visited December11th,2023

muchlower( ∼tensofthousandssamplepairs)comparedtopre-trainingdata.

Reinforcementlearningwithhumanfeedback (RLHF)isthethird,reinforcement-basedphase oftrainingwherethemodelistunedtohumanpreferences.Arewardmodeliscreatedforthis purpose,whichisthenappliedtothefine-tunedmodelfortheevaluationofitsoutputs.The rewardmodelistrainedusingadatasetcreatedwithhumanassistancewhereeachqueryis mappedto1ormore (good answer,bad answer) pairswherethegoalistomaximiseforeach pairthedifferencebetweentherewardmodel’sevaluationofthegoodandbadanswers.After therewardmodelhaslearnedtodistinguishthedesirableanswersfromtheundesirableones, itwillbeemployedtoadditionallyfine-tunethemodelthathasalreadyundergoneSFTduring reinforcementlearning.

Directpreferenceoptimisation and identitypreferenceoptimisation (DPO,IPO)arealternative approachestofine-tuningwhere,similarlytoRLHF,adatasetofhumanpreferencesisused forpreferencelearning.ThetwoapproachesaredistinguishedbythefactthatunlikeRLHF, DPOandIPOdonotrequiretheemploymentofarewardmodelbecausetheLLMitselfcan fulfiltheroleoftherewardmodel[7 8 ],usingthedifferencebetweentheevaluationsofgood andbadanswersasthelossfunction.Whereasamodelthathasonlybeenpre-trainedcangive irrelevantordangerousanswers,SFTandRLHF/DPO/IPOaspartsofthetrainingprocessenable usinghumansupervisiontotrainthemodeltomakeitmoresecureandmorecompatiblewith userandbusinessrequirements.

2.2.4.2Inferenceandcontextlearning

Aprompt isauserinputtokenusedbyagenerativeimageorlanguagemodelforthegeneration ofanoutput.Thisprocessiscalled inference .Apromptisusuallymadeupofanatural-language text.ThepromptsusedbyLLM-basedchatbotsarecombinedwithapre-promptcontaining additionalinformationonthecontextoftheconversation,theuser,andthelanguagemodel. Amongotherthings,thisiscrucialforensuringthatthechatbot’soutputisbasedonitsrole asachatbotrespondingtoquestions,ratherthangeneratingacontinuationtotheuserinput. Apre-promptcanalsobeusedforprovidinginformationabouttheoutsideworld,suchasthe date,time,username,contentsofadocumentortextfile,andotherfeaturesoftheuserorthe environment.

Modelsareunabletodistinguishapromptfromapre-prompt,afactexploitedbynumerous promptinjectiontechniques.Asthepre-promptiseasyfortheusertoacquirethroughawellcraftedprompt,itshouldnotcontaininformationthattheusershouldnothaveaccessto.Inthe caseoftransformerarchitecture,thepromptalongwiththepre-promptmustfitintothemodel’s contextwindow whichismeasuredintokensandcontainsthe(pre-)informationnecessaryfor generatinganoutput.Another,morecomplexformofthisapproachisretrieval-augmentedgeneration(RAG)whereinthelanguagemodelscreatesadatabasequerybasedontheuserprompt andAPIinformationfoundinthepre-prompt,andusestheresultsofthisqueryforgenerating aresponse.Thisalsosolvesoftheproblemoftheuser-provideddatabeingtoolargetoinsert intothecontextwindowusingaprompt.Modelarchitectureswithunlimitedpromptlengthalso exist,e.g.Mamba[4 ]andRWKV[3 ].

Whereassimplerlanguagemodelsrequireretrainingorfine-tuningforeachnewtask,thelanguageknowledgeandgeneralisationcapacityofLLMsmeanthat,inmanycases,articulating thetaskandaddingafewexamplestothepromptisallittakes[9 ].Giventhatinformation relatedtothetaskisfedintothemodel’scontextwindow,thisapproachiscalled in-context learning .

In-contextlearningisdividedintonumeroussub-methods:few-shotlearningwheretheprompt issupplementedseveralexamplesalongsidetheinstructions,one-shotlearningwhereasingle exampleisprovided,andzero-shotlearningwherethequeryismadewithoutprovidingany examples.Themoreparametersthelanguagemodelcontains,thefewerexampleshavetobe normallyaddedtothepromptforthesuccessfulcompletionofthetask.

2.3Applicationsofartificialintelligence

Imagesynthesis meanstheautomaticgenerationofanimagewithpredeterminedfeatures,e.g. basedonaverbaldescription(oranotherimageandaverbaldescription).Imagesynthesissubfieldsinclude,inanorderofincreasinggranularity,inpainting,outpainting,styletransfer,deep learning-basednoiseremoval,videosynthesis,andrefinement.Thesedays,imagesynthesis generallyemploysgenerativeadversarialnetworks[10 ]and,increasingly,diffusionmodels[11, 12 ].

Thepurposeof artificialvision istheartificialextractionofinformationfromimages.Thiscomprisesclasssegmentationandinstancesegmentation,labelling,andobjectrecognition.Artificialvisiongenerallyutilisesconvolutionalneuralnetwork(CNN)andtransformer-baseddeep learningmodels[13 14 ].Commonusecasesincludethemonitoringofcattleandagricultural equipment,monitoringofroadconditionsandthesurroundingsbyself-drivingcarsordelivery bots,facerecognition,andaugmentedreality.

Thepurposeof speechsynthesis istogeneratehuman-understandablespeechfromagiven text.Primitivespeechsynthesismodelsoperatedbysequentiallylinkingpre-recordedphonemes orwords,buttoday,transformer-basedneuralnetworksaregenerallyusedforthistask[15 , 16 ]. Speechsynthesisisusedinchatbots,automatedmessagedelivery,screenreaders,computer gamelocalisation,anddubbing.Subfieldsofspeechsynthesisincludespeechstyletransfer, i.e.,imitationofthetoneandpatternsofsamplespeech.

Incontrasttospeechsynthesis,thepurposeof speechrecognition istheextractionofinformationfromhumanspeech.Speechrecognitionincludesspeechtranscription,inthecaseoff whichtextualinformationisextractedfromthespeech.Whereaspastspeechrecognitionmodelsemployedstatisticalmethods,today’ssystemsaremainlybuiltuponneuralnetworksbased onCNNsandtransformers[17 ].Speechrecognitionisusedinsmarthomesandhands-free devicesforvoiceinstructionsanddictation.

Naturallanguageprocessing isabroadfieldcomprisingthegenerationandclassification,as wellastheinterpretationoftexts.Textgenerationgenerallymeanspredictingthenexttoken, withprevioustokensprovidingthecontextfortheprediction.Textclassificationandinterpretationareusedinsemanticsearchwherecandidatephrasesfoundinadocumentortextexcerpt arecomparednotbasedonkeywordmatchingbutsemanticproximity.Deeplearningnetworks comprisingrecurrentneuralnetworks(RNN)andlongshort-termmemory(LSTM)werepreviouslyusedinspeechsynthesis.Amajorbreakthroughinthefieldcamewiththeemergenceof largelanguagemodels(LLMs),thearchitectureofwhichisgenerallytransformer-based[18 , 19 , 20 ].LLMsareusedin,e.g.,copywriting,chatbots,neuralmachinetranslation,emotionanalysis, andcodegeneration.

Generaldataprocessingandanalysis. Machinelearningmethodsarealsousedindataanalysisinotherapplications.Theseincludevariousclassification,clusteranalysis,anddiscreteor continuousfeaturepredictiontasks,suchaspredictingstockpricemovements,processingof brainsignalscollectedbyabrain-computerinterface,orclusteranalysisbasedonclients’consumptionhabits.Dependingonthenatureofthetask,bothdeeplearningneuralnetworksand

statisticalmachinelearningmethodscanbeused.

2.4Areasofuseofartificialintelligence

Thetechnologiesdiscussedabovehavefounduseinmanywalksoflife:e-governance,the privatesector,educationandresearch,healthcare,andunspecifiedpersonaluses.Wewillnext takealookatsomeofthesefieldsandapplications.

E-stateande-governance. TheAIstrategiespublishedbytheEstonianMinistryofEconomic AffairsandCommunicationsforeseewidespreadadoptionofAIinthepublicsector.Thenatural languageprocessing-basedvirtualassistantBürokrattenablescommunicationwithpublicsectorservicesviaachatwindow.TheEstonianParliament’sdigitalstenographerHansusesspeech recognitiontotranscribespeechesmadeintheplenaryhall.Severalministrieshaveusedthe Textatextanalysistoolkitforauditingtheirdocumentation.Ilme,aserviceprovidedtheNational ArchivesofEstonia,allowsusingartificialvisiontofindpeoplesimilartouser-uploadedimages inhistoricalphotos.

Education. Artificialintelligencehasnumeroususesineducation,e.g.considertheEducation technologycompasspublishedbytheEstonianEducationandYouthBoard(HARNO) 2 .Theeducationalnon-profitKhanAcademyusesachatbotbasedonGPT-4forthepersonalisationof studies.TheDuolingolanguagelearningapplicationcontainsasimilarGPT-4-basedinteractivechatbotsolution;naturallanguageprocessingmethodsarealsoemployedbytheLingvist languagelearningapplication.

Research. Artificialintelligenceandmachinelearninghavebeenusedbyresearchersforboth discoveringnewknowledgeandfindingandsystematisingexistinginformation 3 .TheSemanticSearchsearchportalusesnaturallanguageprocessingandartificialvisionforsummarising, indexing,andsearchingscientificpublications,whereastheAlphaFoldAIsystemdevelopedby Alphabethasmadeitpossibletopredicttheshapesofproteinswithpreviouslyunknownstructures.Modelsbasedonmachinelearningandartificialintelligencehavebeenadoptedinparticle physicsfordataanalysisandsimulationdesign,andinbiomedicine,forthedevelopmentofnew pharmaceuticals.

Healthcare. Artificialintelligencehasbeensuccessfullyemployedinpersonalmedicine,clinical research,aswellasdrugdevelopment4 .Machinelearning-basedbigdataanalysismethods allowusingthepatient’sgenedataforprovidingbettertreatment.Artificialvisionishelpfulinthe interpretationofmedicalimagesanddiagnosingthepatient.Naturallanguageprocessingand textanalysismethodsenablefindingandorganisingpatientdata.Machinelearningmethodsare usedindrugdevelopment,e.g.inmolecularsimulations,predictionoftherapeuticproperties, aswellasthegenerationofmolecularstructuresandsynthesispaths.

Privatesector. Machinelearning-basedaudioprocessing,noiseremoval,andaudioandvideo streampackingtechniques(Skype)areutilisedintelecommunications.Artificialvisionisused ine.g.robotics(Milrem,Cleveron),agriculture,identityverification(Veriff).Chatbotsbasedon naturallanguageprocessingareincreasinglycommonincustomersupport.

2 EducationandYouthBoard.Educationtechnologycompass. https://kompass.harno.ee/tehisintellekt VisitedAugust10th,2023

3 OECD,ArtificialIntelligenceinScience. https://www.oecd.org/publications/ artificial-intelligence-in-science-a8d820bd-en.htm VisitedAugust10th,2023

4 NationalInstituteforHealthDevelopment.Artificialintelligenceasthefoundationforpersonalmedicineinoncology. https://www.tai.ee/et/personaalmeditsiini-uudiskirjad/ tehisintellekt-kui-personaalmeditsiini-alus-onkoloogias VisitedAugust11th,2023

Personaluse. AI-basedpersonalassistants,suchasGoogleAssistant,AmazonAlexa,andSiri werecommonevenbeforetheemergenceofLLMsanddiffusion-basedimagesynthesismodels.TheproliferationandincreasedaccessibilityoffLLMsanddiffusion-basedimagesynthesis modelshasledtoanevolutionaryleapinthisarea,includingthewidespreadadoptionofthe AIaaS(artificialintelligenceasaservice)businessmodel.Modelsdevelopedforpresonaluse andpluginsandapplicationsbuiltuponthesecananalyzecode(GitHubCopilot),readdocumentsorwebpagesandextractnecessaryinformation(BingChat),generatetextsfrombirthdayinvitationstomarketingmaterials(ChatGPT).

Imagesynthesismodelscanbeusedbyindividualsforcreatingillustrationsinthedesiredstyle, generateinteriordesignideas,increasetheresolutionofimagesorphotos(StableDiffusion, Midjourney),andevenidentifycertainspeciesofmushroomsinthewoods.

2.5Explainabilityinmachinelearning

Theemergenceofdeeplearningmethodsandincreasesinthecomplexityofmachinelearning modelshavegivenrisetoquestionsregardingtheexplainabilityofthemodels.Explainabilityof amodelmeanstheabilitytoprovideahuman-understandableexplanationoftherelationship betweenthemodel’soutputandinput.EUdataprotectionregulationsconsiderthetransparency oftheusedartificialintelligencetechnologyvitalforsituationswhereautomateddecisionsare madeusingmachinelearningmodels[21].Thiscanbeachievedthroughtheexplainabilityof themodel.

ExplainableAI(XAI)hasbeenproposedasasolutionfacilitatingmovementtowardsmoretransparentartificialintelligenceandthusavoidinglimitationsontheadoptionofAIincriticalareas [22 ].Asatthetimeofthisreport,thereisasofyetnoglobalconsensusregardingthedesirable thresholdofalgorithmicexplainability[23 ].

ExplainabilityiscloselytiedtotheissuesoftransparencyandtrustworthinessofAIsystems. Thesystematicdefinitionofexplainabilityrequirementsisthusavitalstepinthedevelopment oftransparentandtrustworthyartificialintelligencesystems[24 ].TheOECDhasfound[25 ] that,inordertoensuretransparencyandexplainability,AIactorsshouldprovidemeaningful information,appropriatetothecontext,andconsistentwiththestateofart:

• tofosterageneralunderstandingofAIsystems;

• tomakestakeholdersawareoftheirinteractionswithAIsystems;

• toenablethoseaffectedbyanAIsystemtounderstandtheoutcomeand

• toenablethoseadverselyaffectedbyanAIsystemtochallengeitsoutcomebasedonplain andeasy-to-understandinformationonthefactors,andthelogicthatservedasthebasis fortheresult.

Real-worldinterpretationsoftheexplainabilityrequirementshavealsobeenstudied[24 ].These studieshaveledtothefindingthattheexplainabilityofAIis, interalia ,fosteredbytheestablishmentofsystematicdefinitionsandtheformalisationandquantificationofexplanationsand performanceindicators[22 ].Fourcomponentsofexplainabilityhavebeenproposed[24 ]:

• addressees–towhomtoexplain?

• aspects–whattoexplain?

• context–inwhatkindofsituationtoexplain?

• explainers–whoexplains?

Anexplainablemodelismoretrustworthy,itiseasiertodevelop,test,andaudit;itisalsoeasier todetectanybiasesandexplainabnormalbehaviour.Explainabilityisvitalinmedicinewhere, e.g.,animagemodeldesignedtodetecttumoursismoretrustworthy,ifthepredictionisaccompaniedbyanexplanationofwhichfeaturesoftheimage(contrast,shape)ledtothedetection ofthepresence(orabsence)ofatumour.Likewise,someonereceivinganegativeresponseto theirloanapplicationfromabankmightbeinterestedtolearnwhattheyneedtodoforthebank togivethemaloan(aso-calledcounterfactualexplanation).Aprofanityfilterhighlightingthe inputwordscontributingthemosttotheclassificationofamessageasobscenewillbeeasier todevelopandtestthanonethatdoesnot.

Explainabilityisnotalwaysrequired.Wheretherisksarelowandtheproblemitselfhasalready beenstudiedindepth,itcanprovesuperfluous.Asarule,thereisalsoatrade-offbetween thecapabilitiesandexplainabilityofthemodel[26 ].Whereas,inthecaseofalinearregression model,therelationshipbetweentheoutputandtheinputcanbegleamedfrommerelylooking attheregressioncoefficients,morecomplexandmorepowerfulmodels,suchasdeepneural networks,areakindofa’blackbox’[27 ]forhumanswherethemodel’spredictionordecision principlesarenolongeridentifiableonthebasisofthemodel’sstructureandparameters.

Explainabilitycanbedividedintointrinsicandpost-hocexplainability.Inthecaseofintrinsic explainability(alsoknownastransparency),themodel’scomplexityislimitedinordertoprevent itfrombecomingablackboxandtomaintaintheexplainabilityofitsparametersovertheentire modelfromthestart.Modelswithasimplestructure,suchasdecisiontreesandsimpleregressionmodels,areconsideredself-explainable.Wherethetaskathandcallsfortheemployment ofamorecomplexmodel,post-hocmethodsareusedforincreasingitstransparency.

Post-hocmethodsaregenerallymodel-agnostic–theydonotdependonthearchitectureofthe model,nordotheypresumethepossessionofanoverviewofitsinternalcomponents.Post-hoc explanationstreatallmodels,includingthosethatareself-explainableduetotheirsimplicity,as blackboxes.So-calledlocalpost-hocexplainabilitymethodsdemonstratehowmuchandin whichdirectionsmallindividualchangesininputfeatureswillshiftthemodel’soutput,orwhat arethesmallestnecessarychangesininputfeaturesrequiredforthemodeltopredictanother class.Globalpost-hocexplainabilitymethodsallowunderstandingtheintermediarylayersof analready-trainedmodel:thus,OpenAIhascreatedMicroscope 5 ,acollectionofvisualisations, thancanbeusedtoacquireanoverviewoftheintermediarylayersofdifferentimagemodels, theneuronscontainedtherein,andtheirproperties.Italsoallowsstudyingwhichpictureswithin theinputdatasetactivatetheneuroninquestionthemost.

2.6Globaltrends

2.6.1Fasterandlarger

Increasingmodelsizes. Justascomputingpower,thesizeofneuralnetworkshasalsoundergoneanexponentialgrowth.In1989,YannLeCun’steamusedaconvolutionalneuralnetwork toidentifynumbersinimages.Thenetworkconsistedoftwoconvolutionalandonefullyconnectedlayer,foratotaloffewerthantenthousandtrainableparameters.TheAlexNetmodel introducedin2012comprisedoffiveconvolutionalandthreefullyconnectedlayer,withasmany as61millionparameters.

Withthespreadoftransformerarchitecture,thenumberoftrainableparameterskeptincreasing (Figure 8 ):TheBERT-baseandGPT-1languagemodels(2018)alreadycontained ∼110million, 5 OpenAIMicroscope https://microscope.openai.com VisitedDecember10th,2023

Figure8.Growthinthenumberofmodelparametershasbeenexponential.

GPT-2(2019)–1,5billion,andGPT-3(2020)–175billiontrainableparameters.Thenumber ofparametersusedinGPT-4hasnotbeenpublicisedbutithasbeenspeculatedthatitisa so-called’mixtureofexperts’(MoE)modelwith ∼1.76trillionparameters.Theincreaseinthe numberofparametersalsomeansincreaseddemandsforcomputingpowerandmemoryrequiredforbothtrainingamodelandtheapplicationofatrainedmodel(inference).Efficient trainingofamodelalsorequiresever-largeramountsoftrainingdata.

Withtheincreaseinthenumberofparameters,languagemodelshavebeguntoexhibitemergentabilities,generallyunderstoodascapabilitiesthatexistinmodelswithlargernumbersof parametersbutlackinginsmallerones[28 ].Forinstance,largerlanguagemodelsarecapableof summarisingandtranslatingtexts,generatingcode,findingpatternsinatextandunderstandinghumour,whilesmallermodelsarelimitedtoansweringtosimplerquestionsorgenerating seeminglygrammaticallycorrecttext.Somehavealsoattemptedtoexplainsuchostensibly emergentabilitieswiththebettermemorisationcapacityandimprovedsteerabilityviaprompts characteristictolargermodels[29 ].Untiltheadoptionofmodelweightquantisationandmodel pruning,suchfeatureswerethoughttoappearinalanguagemodelfrom ∼7billionparameters, eventhoughcertainemergentpropertieshadbeenobservedinthe1.5billionparameterGPT-2. Today,however,ithasbecomeclearthatsmallerorcompressedmodelsmayalsopossesssuch abilitiestoacertainextent.

Alanguagemodelwithahighernumberofparametersrequiresalargertrainingdatasetforthe efficientutilisationoftheseparameters.LargerEnglish-languagetrainingsetscomprisetrillions oftokens,whereasthesizeofEstoniandatasetsdoesnotexceedseveralbillions.Thismeans thatalanguagemodeltrainedbasedontheEstonianlanguagewillgenerallybesmallerand lesscapable.TheshareofEstonianinmulti-languagedatasetsisverysmall,meaningthata modeltrainedontheEstonianlanguagemaynotalwaysbecapableofmasteringthelanguage. Onestrategyformitigatingthisdisparityisfine-tuningmodelstrainedonanEnglish-language datasetusingEstonian-languagedata.

Increasinghardwarerequirements. ComputerGPUsusetheSIMD(singleinstruction,multiple data)architecturewhichallowsthesameoperationtobeperformedsimultaneouslyonseveral piecesofdata.Thisallowssignificantlyspeedinguprenderingworkflowsandothergraphicsrelatedtaskswhereacertainoperationneedstoberepeatedoneachbufferelement.This featuredidnotgounnoticedfortheresearchersofdeeplearningneuralnetworkswhoproposed

theideain2009thatmatrixoperationsfrequentlyusedinneuralnetworkscouldbespedup usinggraphicsprocessingunits[30 ].

Foreachnewtokenbeinggenerated,transformer-basedlargelanguagemodelsmustaccessall theweightsandattentionvectors (q,k,v) employedbythemodelandmovethemfromRAMto GPUregisters.Asufficientlylargenumberofsufficientlylargeweightmatriceswillresultinincreasedloadingtimes.Thismakesmemorycapacityandthroughputcomparableinimportance toplainFLOPS 6 .

Unlikefine-tuning,in-contextlearningdoesnotrequirethecomputationallyexpensiveupdatingofmodelweightsalongsideinference(prediction).Thein-contextlearningfunctionalityof certainLLMscanalsobeimplementedonahigh-performancepersonalcomputer 7 .Quantisation[31]–reductionintheaccuracyandmemoryrequirementsofmodelparameters–isused tofacilitatefittingthemodelweightsinthePC’sGPUmemory.E.g.16-bitfloatingpointnumbersareusedinplaceof32-bitones;themostpowerfulquantisationmethodsre-encodethe parameterssothatasingleparameterwillonlyrequireabitmorethan2bitsofmemory[32 ]. Onthedownside,themodel’sabilitiesmaysufferfromquantisation.

Theexpansionoffieldsemployingparallelprocessing(machinelearning,simulations,scientific modelling,cryptocurrencymining)hasincreaseddemandforbothhardwareandfirmwaresuitableforthetask.NvidiahasthusdevelopedtheCUDAplatformcomprisingbothhardwarecomponentsandasoftwareframeworkfortheutilisationofGPUsinparallelprocessingtasks.Apple haddevelopedtheOpenCLparallelprocessingstandardthat,unlikeCUDA,wasnotbasedon aspecifictypeofhardwarebuttoday,they,too,haveswitchedtotheirownhardware-specific framework,calledMetal.

ClassicserverarchitecturesarenolongeradequateforofferingAIasacloudservice.Extremely largevolumesofdataalsomeanthatspecialiseddatacentresorcloudservicesareusedfordata storageandprocessing.Whenscalingaservice,cloudinfrastructureandspecialisedhardware arerecommendedforbothinferenceandtraining.Meanwhile,specialisedhardwarenolonger meansonlyGPUs–italsocoverssolutionsevenmorespecifictoneuralnetworks,suchasthe tensorprocessingunit(TPU)developedbyGoogle,ortheneuralprocessingunit(NPU)usedin smartphonesandInternet-of-Things(IoT)devices.

2.6.2Fromgeneral-purposetospecial-purpose

Fromfoundationmodelstoapplications. FoundationmodelsareoftenmentionedinthecontextofLLMs.Thesearegeneral-purposemodelsthatcanbeusedforperformingmanydifferenttasks.Chatbotsareoneofthemostbasicapplicationsoffoundationmodels,asthey onlyrequirecommandofnaturallanguageandgeneralknowledgethatcanbederivedfrom modelweightsanddonotrequireaseparatedatabaseinterface.Non-deterministicmodeloutputisalsoacceptableinchatbots.Indomain-specificapplications,thegeneralisationability andknowledgeofthefoundationmodelmaynotalwaysbeadequateforthetask.Specialised solutionsandmodelshavethereforebeendevelopedalongsideandbasedonlargefoundation models.Theseareespeciallygoodatprocessingmedicalandlegaltext,summarisinglarge

6 FLOPS(floatingpointoperationspersecond)isameasureofcomputerperformance.

7 llama.cppisanopen-sourceapplicationthatfacilitatesrunninginferenceonLLaMA,LLaMA2,andotherlanguage modelsusingquantisation.

documents 8 ,programminglanguagesandpatterns 9 ,imagerecognition 10 ,andcanevaluatethe likelihoodofanimageortextbeingcreatedbyagenerativemodel11 .

SimplersolutionshavealsoemergedthatconnecttoanexistingAImodelusingitsAPI,e.g., forinteractingwithandsummarisingdocumentsintheformofPDFfiles.Thebusinessrisk involvedinsuch’thin’solutionsisthattheprovidersofAPIsandmodelscaneasilyimplement suchfunctionalityintheirownproducts,justlikeOpenAIhasdonewiththeanalysisofPDFfiles inChatGPT 12 .

Fromthesynthesisofasingletypeofcontenttothecreationofheterogenouscontent. When amodelinteractswithdifferentinputoroutputmodalitiesitcanbeclassedasmultimodal.In otherwords,evenasimpleimageclassifiercouldbeconsideredmultimodalinthatitreceivesan imagesasaninputandoutputsatextlabel.Inreality,thetermismainlyusedformodelswhere inputswithdifferentmodalitiesaremappedtothesameembedding,suchasOpenAICLIP 13 andGPT-4V 14 .Multimodaltext-to-videomodelsalsoexistthatgenerateanimagesequence correspondingtotheprompt,eitherrelyingonreferenceimages[33 ]orwithout[34 35 ].

Whereasmultimodalinputshavebeensimpletoprocessthusfar,generatinganoutputcomprisingdifferentmodalitiesismoredifficult.Themostcommon(andeasiest)solutionsofaristhe combinationoftheoutputsandinputsofmultiplemodels.Thus,ChatGPTcomprisesanimage generationfunctionalitywheretextualinstructionsgeneratedusingtheGPT-4languagemodel basedonauserpromptarefedtotheDALL-E3imagesynthesismodelwhichwillthenreturn thegeneratedimagestotheusers.TheInvideoAIservice 15 (alongsideseveralothersimilar services)composesvideosbasedoninputtext:itgeneratesascriptbasedonauserprompt andsearchesthedatabaseforclipswhicharethenassembledintoavideo,afterwhichitalso generatesasoundtrack.

OneoptionforcombiningAIservicesisanAIagent(insomecasesagenerativeagent)capable ofinterfacingwithdifferentservices,e.g.makingInternetqueriesforperformingthetaskit hasbeengiven.AIagentsarecharacterisedbyacontinuousfeedbackcyclebetweenmaking queries(interfacingwiththeoutsideenvironment)andupdatingtheirinternalstate.Forthis reason,itisvitalforAIagentstobecapableofplanningtheirnextstepswhilealsokeeping trackoftheresultsoftheprevioussteps,theirinternalstate,andthebroadercontentsand purposeofthetask[36 ].Aself-drivingcarcanbeconsideredanAIagent.

Thesedays,AIagentsgenerallymeansolutionsbasedonlargelanguagemodelsthatfacilitate automatisingmulti-stepactionsrequiringthedivisionoftasksintosubtasks,additionalplanning, andconstantfeedbackbasedonnaturallanguageinstructions.Someofthecurrentlypopular (asofwritingthisreport)frameworksforcreatingandmanagingAIagentsincludeAutoGPT, BabyAGI,andAiAgent.App.

8 Claude2: https://www.anthropic.com/index/claude-2

9 GitHubCopilotX: https://github.com/features/preview/copilot-x

10 Gpt-4Vision: https://openai.com/research/gpt-4v-system-card

11 StableSignature: https://ai.meta.com/blog/stable-signature-watermarking-generative-ai/

12 ChatGPTPlusmemberscanuploadandanalyzefilesinthelatestbeta. https://www.theverge.com/2023/ 10/29/23937497/chatgpt-plus-new-beta-all-tools-update-pdf-data-analysis VisitedFebruary25th, 2024

13 CLIP:Connectingtextandimages. https://openai.com/research/clip

14 GPT-4V(ision)systemcard. https://openai.com/research/gpt-4v-system-card

15 InvideoAI. https://invideo.io/

2.6.3Fromclosedtoopen

Modelsforprovidingaccesstoclosedmodels. ThelargerAImodelsgot,themoreexpensive theirtraining,management,anddeploymentbecame.Themorepowerfultheygot,thegreater therisksofexploitingtheirgenerativecapabilitiesbecame.OpenAIwasfoundedin2015asa non-profitwiththegoalofresearchingartificialintelligenceandamainfocusondeeplearning neuralnetworks 16 .Intheearlydays,theorganisationputastressonopennessandcreating valueforthewholesociety.

On8April2019,afewmonthsaftertheannouncementandunveilingoftheGPT-2language model,thedecisionwasmadetosplitthecompanyintoa’limitedprofit’company(OpenAILP) andanon-profit(theexistingOpenAINonprofit),withtheboardofthelatterremainingthegoverningbodyofthetwonewpartnerorganisations 17 .Thisstepwaspurportedlytakenbecause ofthehighmaintenanceexpensesofmodernAIsystems:trainingthesesystemsiscomputeintensive,maintenanceofthebigdatainfrastructureusedforthetrainingiscostly,andanNGO’s opportunitiesforraisingcapitalarefarexceededbythoseofcompanies.Thiswasfollowedby apartnershipwithMicrosoftwhoinvestedonebillionUSdollarsintothecompany,andanother 10billiondollarsin2023.

GPT-2wasOpenAI’slastcompletelyopenlanguagemodels.In2020,OpenAIreleasedGPT-3, buttheparametersofthetrainedmodelwerenotmadeaccessibletothepublic–accessto themodelwaslimitedtotheOpenAIAPI18 andGPT-3itselflicensedtoMicrosoft19 underthe cooperationagreementsignedearlier.ThedecisiontocreateanAPIwasmotivatedbysecurity requirements,aswellasfinancialconsiderations.AsthemaintaineroftheAPI,OpenAIretains therighttorestrictaccesstothemodeltoexploiters;theAPIwasalsothefirstcommercial productofOpenAILPthathelpedfundfurtherresearchandmaintaintheexpensiveserverinfrastructure.

Emergenceofpublicmodels. In2023,Metaannounceditsownseriesoflanguagemodels, LLaMA 20 ,surprisingtheworldbymakingthemodelscompletelypubliclyaccessible,evenfor commercialuse.ThelicenceoftheLLaMA2modelseriesreleasedafewmonthslaterexcluded companieswithmorethan700millionannualusersinordertoprotectMetafromitsbiggest competitors.Thesameyearalsosawthereleaseofthesourcecodeandparametersofstability.ai’sgenerativeimagemodel,StableDiffusion 21 .Theemergenceofmodelsfarsurpassing GPT-2intheircapabilities,suchasLLaMA2,hasunleashedanavalancheofsmallerbut,in someways,morepowerfulAImodelsfine-tunedforspecificareasofuse.Theperformance ofthesemodelsisonlymarginallyinferiortofoundationmodelswithamuchhighernumberof parameters.Mistral-7B 22 andSSD-1B 23 aregreatexamplesofsuchmodels.

Hobbyists,smallenterprises,andresearchinstitutionscanhardlyaffordtheinformationinfras-

16 OpenAI. https://openai.com/blog/introducing-openai VisitedOctober20th,2023

17 OpenAILP. https://openai.com/blog/openai-lp VisitedOctober23rd,2023

18 OpenAIAPI. https://openai.com/blog/openai-api VisitedOctober23rd,2023

19 OpenAIlicensesGPT-3technologytoMicrosoft. https://openai.com/blog/ openai-licenses-gpt-3-technology-to-microsoft VisitedOctober23rd,2023

20 IntroducingLLaMA:Afoundational,65-billion-parameterlargelanguagemodel. https://ai.meta.com/blog/ large-language-model-llama-meta-ai/ VisitedOctober24th,2023

21 StableDiffusionPublicRelease. https://stability.ai/blog/stable-diffusion-public-release VisitedOctober24th,2023

22 MistralAI. https://mistral.ai/ VisitedOctober24th,2023

23 AnnouncingSSD-1B:ALeapinEfficientT2IGeneration. https://blog.segmind.com/ introducing-segmind-ssd-1b/ VisitedOctober24th,2023 Risksandcontrolsforartificialintelligenceandmachinelearningsystems

tructureortrainingbudgetsofthelikesofOpenAI,Google,orMeta,whichhascausedashiftin focusfromthenumberofparameterstotheirefficientuse,thequalityoftrainingdata,andalternativemodelarchitectures.AsdemonstratedbyGoogle’sleaked’WeHaveNoMoat’24 memo, theirsuccesshasbeenacauseforconcernforlargecorporations.Theemergenceofmoreefficientandcheaperfine-tuningmethods,suchasLoRA[37 ],hasallowedhobbyiststokeepup withlargetechnologycompaniesinspiteofthegapininvestmentcapacity.

Motivatedby,ononehand,thetechnologyindustry’sdesiretouseAIonportabledevicesand, ontheotherhand,thelimitedresourcesofsmallenterprisesandtheopensourcecommunity,a numberof’smalllanguagemodels’(SLMs)withfewerparametershavenowemerged,suchas Microsoft’sPhi-1.5[38 ]andPhi-2,Google’sGeminiNano 25 andGemma[39 ],aswellasMistral 7B[40 ]andtheQwen1.5familyofSLMs[41]whichareonlyslightlyinferiorinperformanceto muchlargermodels.

2.6.3.1Developmentsindeploymentmodels

AnAImodelinitselfisnotsufficientforperformingbusinesstasks.Themodelmusthaveaccess toinputdataandmustbecapableofproducingproperlyformatted,high-qualityoutputdata. DeploymentmodelsrefertothestructureofAIapps,interfacesbetweentheAImodelandother componentsoftheapp,andtheflowsofdatabetweenthesecomponents(includingusers’ personaldata).

Thefirst,moreprimitiveAImodels(e.g.linearregression,perceptrons,rules-basedexpertsystems)werenotcompute-intensive,makingtheinformationinfrastructureforrunningthemodel lesscriticalthandatastorageinfrastructure.AIapplicationdeploymentmodelsonlybecame relevantwiththewidespreadadoptionofAIinthe2010s,accompaniedbygrowingdatasets, proliferationofneuralnetworks,andtheresultingneedtoacceleratetrainingandinferenceusingGPUsthatwerenotalwaysreadilyphysicallyaccessibletothetrainersorusersofAImodels. Alongsidedatastorageandnetworking,cloudinfrastructureprovidersbegantoofferhardware andcloudcomputingenvironmentsforAImodels(e.g.GoogleColab,AmazonSageMaker),but theuserswerestillresponsibleforthedevelopment,training,anduseoftheirmodels.

Thegeneral-purposenatureofsubsequentlargetextandimagesynthesismodelsmeantthat forcertaintasks,themodelnolongerneededtobetrainedfromthegroundup.Thisgaveriseto AIaaSorAIasaservice,allowingcompaniesandindividualstouselargeAImodelsevenwithout investmentsintohardware,training,andotherinformationinfrastructure.

TheemergenceofChatGPTandAIAPIshastriggeredadelugeofthin’APIwrapperaps’using thegeneralisationabilityofChatGPToranotherAItextsynthesissolutionforsolvingdomainspecifictasks.Someoftheseapplicationsprovidelittlebesidesaconvenientuserexperience andacarefullycraftedpre-prompt;meanwhile,thereproducibilityofsuchsolutionscreatessignificantbusinessrisksforthecreatorsofwrapperapps.ThisriskmaterialisedattheOpenAIDev DaywhereOpenAIintroduceda’customGPT’serviceallowinguserstobuildspecial-purpose chatbotswithoutwritingasinglelineofcode 26

ThebusinessnicheofAIserviceprovidersisnotgenerallyfoundedoninnovativemodelarchitecture,astheseareusuallypublic,buttheinformationinfrastructurebuiltaroundthemodel,

24 Google:”WeHaveNoMoat,AndNeitherDoesOpenAI”. https://www.semianalysis.com/p/ google-we-have-no-moat-and-neither VisitedOctober26th,2023

25 GoogleBlog:IntroducingGemini https://blog.google/technology/ai/google-gemini-ai/ VisitedDecember14th,2023

26 IntroducingGPTs. https://openai.com/blog/introducing-gpts VisitedNovember20th,2023

theuserexperienceprovidedbythesolution,andthequantityandqualityofdomain-specific trainingdata.TheX(formerTwitter)AIserviceGrokhasreal-timeaccesstothedatabaseof userpostsandMicrosoft’sCopilotXcodingassistantwouldnotbenearasefficientwithoutthe constantlyupdatedGitHubrepository.ChatGPT,meanwhile,allowstheusertogivefeedback toallchatbot’sanswerswhichhasenabledOpenAItocollectlargeamountsofvaluabledataon users’interactionswiththechatbottofacilitatethefurtherimprovementofthequalityoftheir languagemodels.

Trainingdataqualitymanagementisvitalasitallowssignificantlyreducingtheamountofdata requiredforthetrainingofanequivalentmodel[42 ],butalsobecausetheproportionofsynthetic contentontheInternethasrisensharplyasoflateand,accordingtoexperts,mightreach90 percentby2026[43 ].

2.6.4Fromunregulatedtoregulated

2.6.4.1AIethics

Theethicsofcomputerscienceisamultifaceted,comprisingbothmoralandethicalconsiderationsrelatedtothedevelopment,deployment,anduseofcomputingtechnologies,suchasAI. Itisvitaltoensurethatthesetechnologiesaredevelopedandusedinwaysthatmirrorhuman valuesandpromotesocialwellness[44 ].Ethicalprinciplesaredynamic,meaningthattheycan changeintime,adaptingtodevelopmentsinscienceandthesociety[45 ].

TheemploymentofAItechnologiesisontherise–by2027,themarketcapitalisationofthe fieldisexpectedtoreach407billiondollars[46 ].Estoniancompaniesarealsoincreasingly usingAItechnologies–asatQ1of2023,themarkethasseena2%increasecomparedto2021. AccordingtoStatisticsEstonia,AItechnologiesaremostfrequentlyusedinEstoniabyfinance andinsurance,informationandcommunication,andenergysectorenterprises[47 ].

Eventhoughartificialintelligencetechnologiesdemonstrateenormouspotential,theuseofAI alsogivesrisetonumerousquestionsandfears.Forexample,asurveycarriedoutinEnglandin2023showedthatpeoplearethemostworriedaboutself-drivingcarsandautonomous weapons.TheyalsofearthatifAIisusedforprofessionaldecision-making,theartificialintelligencemayproveunabletoaccountforindividualreal-worldcircumstancesanddecision-making maysufferfromalackoftransparencyandresponsibility[48 ].

In2018–2021,ascandalbrokeoutinTheNetherlandswhenitwasfoundthatthenationaltax officehadusedaflawedriskanalysisalgorithmindecision-making,resultinginthousandsof childsupportreceiversbeingbaselesslyaccusedoffraud[49 ].Thisledtotensofthousandsof families,oftenfromlowerincomebracketsorethnicminorities,fallingintopoverty.Someofthe victimsperformedsuicideandoverathousandchildrenwhereplacedintofosterfamilies[50 ].

Professionaldecisionsofthiskindmayalsoincludecourtrulings.Thisraisesthequestion whetherarulingmadebyanartificialintelligenceisequivalentinqualitytoonemadebyahuman judge,whetherthesysteminquestionhasbeentrainedonhigh-qualitydata,andwhethercare hasbeentakentoruleoutdiscriminationonanygrounds,suchasgender,race,orincome.ResearchershavepointedoutthatAImodelsbasedoninformationderivedfromolderinputdata aremorelikelytofollowmoreconservativepracticesandmaynotbecapableofadaptingto significantpoliticalchangesovertime[51].IthasalsobefoundthattheuseofAIformaking courtrulingsmayproveathreattotheintegrityofdatawhich,duetotheirverynature,would requirethehighestlevelofsecurity[52 ].

IthasbeenfoundthatLLMsmaytendtoreinforceincorrectlegalassumptionsandbeliefswhich inturngivesrisetosignificantconcernsoverthereliabilityoftheresultsinalegalcontext[53 , 54 ].ThetransparencyandaccuracyoftheAImodelalsobecomecriticalinthecontextof trials[55 ].

Ethicalissuesemerginginthedevelopment,deployment,anduseofAIarethesubjectofAI ethicswhichisconsideredoneofthesubdomainsofappliedethics.ThegoalofAIethicsis todeterminehowanartificialintelligencesystemcanincreaseordecreasehumanwell-being throughchangesinqualityoflifeorautonomyandindependence.DifferentAIethicsframeworks aregenerallybuiltaroundfundamentalrights[45 ].

OnApril8th,2019,theEUHigh-LevelExpertGrouponAI(hereinafterAIHLEG)presentedits ethicsguidelinesfortrustworthyAI[45 , 56 ]withthegoalofprovidingguidanceforpromoting andsupportingethicalandrobustartificialintelligence.Lessattentionispaidtothelegalaspectsofthesystem.ThedocumentpresentsapreliminaryframeworkfortrustworthyAIwhile alsodiscussingissuesrelatedtotheimplementationandevaluationofAIsystems[45 ].

2.6.4.2AIregulationintheEU

InApril2021,theEuropeanCommissionproposedthefirstlegalframeworkregulatingAI[57 ]. Theproposalwasbuiltaroundarisk-basedapproach,assertingthatartificialintelligencesystemsshouldbeanalyzedandclassifiedbasedonthethreattheyposetousers[58 ].NegotiationsovertheAIActendedonDecember8th,2023.Inearly2024,theAIActisexpectedtobe publishedintheOfficialJournaloftheEuropeanUnion. Neithershouldoneoverlooktheexistinglegalframework.Morespecifically,theGeneralData ProtectionRegulation(GDPR)of2016[59 ]stressestheimportanceoftheprotectionofnaturalpersonsintheautomatedprocessingofpersonaldata 27 .Inadditiontothetheabove,the development,implementation,anduseofartificialintelligencemustalsoaccountforotherrequirements,suchasintellectualpropertyrights.Formoredetailsonthelegalaspectsofartificial intelligence,seeSection 3 ofthereport.

27 GDPRregulatestheautomatedprocessingofpersonaldata,includingprofiling,andconfersonthedatasubject therighttoopposeindividualdecisionsbasedonsuchprocessing(seeGDPRarticles2,21,and22,andrecitals15 and71).

3Legalaspects

3.1Internationallegalinitiatives

3.1.1Regulation

ExperiencefromrecentyearsindicatesthatAIregulationisrapidlydevelopingallovertheworld. TheexamplespresentedbelowpertaintojustsomeofthestatesregulatingAIsystems.

OnOctober30th,2023,thePresidentoftheUnitedStatesJoeBidenissuedanexecutiveorder toensurethattheUSmaintainsaleadingpositionintheworldinAIsystems.TheExecutive OrderestablishesnewstandardsforAIsafetyandsecurity,protectsAmericans’privacy,advancesequityandcivilrights,standsupforconsumersandworkers,promotesinnovationand competition,advancesAmericanleadershiparoundtheworld,andmore[60 ].

TheUKParliamenthaspublishedabilltoregulatetheuseofAItechnologiesintheworkplaceand makeprovisionaboutworkers’andtradeunionrightsinrelationtotheuseofartificialintelligence technologies.ThefirstreadingofthebilltookplaceonMay17th,2023[61, 62 ].InSeptember 2023,theUKgovernmentpublishedawhitepaperonapro-innovationapproachtoAIregulation. Thisframeworkisunderpinnedbyfiveprinciples[63 ]:

1. safety,securityandrobustness;

2. transparencyandexplainability;

3. fairness;

4. accountabilityandgovernance;

5. contestabilityandredress.

DiscussionsovertheregulationofartificialintelligencearealsounderwayinAustralia[64 ].In 2022,theAustraliangovernmentpublishedaconsultationontherulesforartificialintelligence andautomateddecision-making.TheconsultationwasdrivenbytheAustraliangovernment’s digitaleconomystrategylayingoutanambitiousvisionAustraliabecomingoneofthe10best digitaleconomiesandsocietiesby2030[65 66 ].AccordingtothenewdraftlawofsearchenginespresentedonSeptember8th,2023,theAustraliangovernmentrequiresInternetsearch serviceproviderstoreviewandregularlyupdatetheirartificialintelligencetoolsinordertoensurethatclass1Amaterials(e.g.,materialsrelatedtothesexualabuseofchildren,supportof terrorism,andextremeviolence)arenotreturnedinsearchresults.Thedraftactalsomandates thatusersmustbeabletoidentifywhetheranimageaccessiblethroughasearchengineisa deepfake[67 , 68 , 69 ].

InSeptember2023,CanadapublishedavoluntarycodeofconductontheresponsibledevelopmentandmanagementofgenerativeAIsystems[70 ].WorkisalsoonthewayontheArtificial IntelligenceandDataAct(AIDA)thatwouldsetthefoundationfortheresponsibledesign,developmentanddeploymentofAIsystemsthatimpactthelivesofCanadians[70 ].Theactwould ensurethatAIsystemsdeployedinCanadaaresafeandnon-discriminatoryandwouldhold businessesaccountableforhowtheydevelopandusethesetechnologies.Inadditiontothe above,onOctober12th,2023,theCanadiangovernmentannouncedapublicconsultationon theeffectsofgenerativeartificialintelligenceoncopyright[71].

Alongsidetheabove-listedstates,legalinitiativesrelatedtoAIsystemshavealsobeenundertakeninIsrael,Japan,China,Chile,Mexico,Peru,Singapore,andotherplaces[72 ].EUlegal

actsonartificialintelligencesystemsarecoveredinSection 3.3 ofthereport.

3.1.2Standards

TurningourattentionnexttoapproachestoAIfoundininternationalsoftlaw,variousnon-binding recommendationsandguidelineshavebeenpublishedtopromotethedevelopmentandadoptionofethical,responsible,andtrustworthyAI.Thesearegenerallyfoundedonprincipleslike privacy,explainability,impartiality,security,andbeinghuman-centered.

OneofsuchstandardsisISO/IEC22989establishingterminologyforAIanddescribingconcepts inthefieldofAI[73 ].CommonterminologyensuresbetterunderstandingofAIsystemsand isvitaltocooperation,regulation,adoptionofresponsibleAIsystems,andinformationsharing[74 ].TheISO/IEC23053standarddescribesartificialintelligencesystemsusingmachine learning[75 ].Thestandarddescribesthecomponentsofamachinelearningsystemandtheir functionsintheAIecosystem[74 ].

Next,theISO/IEC5259standardestablishesaframeworkforensuringdataqualityinanalytics andmachinelearning[76 77 ].ISO/IEC4213describestherequirementsforevaluatingclassificationperformanceinmachinelearning[78 ].Variousotherstandardsandframeworksalso exist,suchastheBSIvalidationframeworkBS30440:2023fortheuseofartificialintelligence withinhealthcare[79 ],theIEEEethicaldesignstandard[80 ],GoogleAIprinciples[81]andresponsibleAIpractices[82 ]andtheMicrosoftresponsibleAIstandard[83 ].

Adherencetostandardswillcontributetothesafety,quality,andreliabilityofproductsorservices;theycanalsohelpenhanceandimprovethecompany’ssystemsandprocesses.Standardsapplicabletothedifferentlifecyclesofartificialintelligencesystemsarecoveredinthe ENISAgoodcybersecuritypracticesforAIsystems[84 ].

3.2EUtrustworthyAIinitiative

OnApril8th,2019,theEUhigh-levelexpertgrouponartificialintelligence(AIHLEG)presented itsethicsguidelinesfortrustworthyAI[85 ]coveringanoverallframeworkforandimplementation andevaluationoftrustworthyartificialintelligence[86 ].Accordingtotheethicsguidelines,the lifecycleofatrustworthyAIsystemshouldbe[86 ]:

1. lawful–respectingallapplicablelawsandregulations;

2. ethical–respectingethicalprinciplesandvalues;and

3. robust–bothfromatechnicalperspectivewhiletakingintoaccountitssocialenvironment.

SectionIoftheguidelinessetsoutthethreemainethicalprinciplesfoundedonfundamental rights.First,thedevelopmentofAIsystemsmustrespecthumanautonomy,ensurethefairnessandexplainabilityofthesystem,andpreventharm.Thesecondprinciplerequirespaying particularattentiontosituationsinvolvingmorevulnerablegroups(suchaschildren,persons withdisabilities)andsituationswhicharecharacterisedbyasymmetriesofpowerorinformation.Finally,attentionisdrawntotherisksposedbyAIsystemsandtheadoptionofmeasures tomitigatetheserisks[86 ].

SectionIIoftheethicsguidelinespresentsanoverviewofhowtocreateatrustworthyAIsystem, andproposessevencriteriaorsuchasystem.

1. Aboveall,itisrecommendedtoensurethatthedevelopment,deploymentanduseofAIsystemsmeetsthesevenkeyrequirementsfortrustworthyAI: ’(1)humanagencyandoversight,

(2)technicalrobustnessandsafety,(3)privacyanddatagovernance,(4)transparency,(5) diversity,non-discriminationandfairness,(6)environmentalandsocietalwell-beingand(7) accountability.’ [86 ].

2. Usingbothtechnicalandnon-technicalmethodstoensuretheimplementationofthose requirementsisrecommended.

3. Researchandinnovationshouldbefosteredtoincreasetheamountofknowledgeavailable aboutAIsystems–amongotherthings,orthetrainingofnewAIethicsexperts.

4. ClearinformationshouldbeprovidedonthecapabilitiesandlimitsoftheAIsystemtoenable settingrealisticexpectations.

5. Systemsshouldbedevelopedtobeexplainabletofacilitatetheirauditabilitywhichmay proveparticularlyvitalincriticalsituations.

6. StakeholdersshouldbeinvolvedthroughouttheAIsystem’slifecycle,andpeopleshouldbe trainedtoincreasetheirawarenessoftrustworthyAI.

7. Ithastobetakenintoaccountthattensionsmightarisebetweenthedifferentprinciples andrequirementsfortrustworthyAI.Itisrecommendedtocontinuouslydocumentallconsiderations,trade-offs,anddecisions[86 ].

SectionIIIoftheethicsguidelinesprovidesanassessmentlistforoperationalisingtrustworthy AI,tobeadaptedbasedonthepurposeoftheAIsystem.Complianceshouldbeassessed, stakeholdersinvolved,andresultscontinuouslyimprovedthroughouttheentirelifecycleofan AIsystem[86 ].ThetrustworthinessofanAIsystemdependsonallofitsfeatures;unfortunately, theexhaustiveunderstandingofcompromisesbetweenthesefeaturesstillremainsanimportant unsolvedproblem[87 ].

Thefinalsectionoftheethicsguidelineselaboratesuponsomeoftheissuesaddressedinthe document,offeringexamplesofbeneficialopportunitiesthatshouldbepursued,anddiscussing criticalconcernsraisedbyAIsystemsthatshouldbecarefullyconsidered[86 ].TheEUhighlevelexpertgrouphasalsopublishedpolicyandinvestmentrecommendationsfortrustworthy artificialintelligenceexplaininghowtrustworthyAIshouldbedeveloped,deployed,promoted, andexpandedinEuropewhilemaximisingitsbenefitsandminimisingandpreventingpossible risks[88 89 ].OnJuly17th,2020,theAIHLEGadditionallypublishedtheirassessmentlistfor trustworthyAI(ALTAI)[90 ].TheALTAIisatoolthatfacilitatesevaluatingtheextenttowhich anAIsystemmeetstherequirementsfortrustworthyAI.Theseguidelinesarealsoavailablein aweb-basedtoolversion[91].

Theyalsopublishedadocumentonsectoralconsiderationsregardingpolicyandinvestment recommendations,analyzingthepotentialapplicationofrecommendationspreviouslypublished bytheAIHLEGinthreespecificsectors:(1)thepublicsector,(2)healthcare,(3)manufacturing andInternetofThings(IoT)[92 ].

Onthe19thofFebruaryin2020,theEuropeanCommissionpublishedareportonthesafetyand liabilityimplicationsofartificialintelligence,theInternetofThingsandrobotics[93 ].Allproducts andservicesmustoperatesafely,reliablyandconsistently,andanydamagemustberemedied –thesearethegoalsoflegalframeworksforsafetyandliability.AccordingtotheCommission, aclearsafetyandliabilityframeworkisparticularlyimportantwhennewtechnologiesemerge, bothwithaviewtoensureconsumerprotectionandlegalcertaintyforbusinesses[93 ].

Onthesameday,theECalsopublishedawhitepaperonartificialintelligence[94 ]discussing aspectsrelatedtothemostimportantoutputsofdataeconomy–artificialintelligence,acollectionoftechnologiesthatcombinedata,algorithmsandcomputingpower.Thewhitepaper

notesthattheuseofdigitaltechnologiesisbasedontrustanddiscusseshowactionneedsto besteppedupatmultiplelevelsinordertosupporttheuptakeofAI[94 ].

3.3EUproposalforanArtificialIntelligenceAct

AnumberoflegalproposalsrelatedtoAIhavebeenproposedintheEUwiththegoalofensuring thatartificialintelligencesystemsusedintheEUaresafe,transparent,ethical,impartial,and human-controllable[95 ].

InApril2021,theEuropeanCommissionpresentedaproposalforaregulationlayingdownharmonisedrulesonartificialintelligence(ArtificialIntelligenceAct)[57 ].Accordingtotheexplanatorymemorandum,theactwouldsetdownharmonisedrequirementsfollowingaproportionate risk-basedapproachtothedevelopment,placingonthemarket,anduseofAIsystemsinthe EU[57 ].OnDecember8th,2023,apoliticalagreementwasreachedonthefinaltextofthe act[96 97 ],followedbytechnicaldiscussionsonfinalisingthetext.Particularattentionwas paidtothequestionofathresholdforhigh-impactgeneral-purposeAI(GPAI)models,which wasdecidedtobeestablishedbasedonthecumulativeamountofcomputingpowerusedfor thetraining(10^25).HarmonisedstandardsfortheregulationofGPAImodelswillbedeveloped inthefuture[98 ].

OnJanuary26th,2024,theBelgianPresidencyoftheCounciloftheEUofficiallysharedthe finalcompromisetextoftheAIActwithmemberstates’representatives[99 ].OnFebruary2nd, 2024,theAIActwasadoptedbytheCommitteeofPermanentRepresentatives(COREPER).The compromisewasbasedonamulti-levelapproachcomprisinghorizontaltransparencyrulesfor allmodelsandadditionalrequirementsforAIsystemsposingapotentialsystemicrisk[98 ].

TheAIActproposal[57 ]servesfourmainobjectives.

1. ThefirstgoalistoensurethatAIsystemsplacedontheEUmarketandusedaresafeand meetexistinglawsandEUvalues.

2. Next,itshouldensurelegalcertaintytofacilitateinvestmentandinnovationinAI.

3. Third,itshouldenhancegovernanceandeffectiveenforcementofexistinglawonfundamentalrightsandsafetyrequirementsapplicabletoAIsystems.

4. Finally,itshouldfacilitatethedevelopmentofasinglemarketforlawful,safeandtrustworthy AIapplications.

Accordingtotheproposal,artificialintelligencesystemswouldbedividedintofourriskcategoriesinordertoestablishrequirementsconsistentwiththerisksinvolved(seeTable 3 ).In thecourseofthenegotiations,thetextoftheAIActwasamendedwithprovisionsconcerning non-systemicandsystemicrisksrelatedtogeneral-purposeAIsystems[99 ].

InthefinalcompromisetextoftheAIAct[99 ],anAIsystemisdefinedasamachine-based systemdesignedtooperatewithvaryinglevelsofautonomyandthatmayexhibitadaptiveness afterdeploymentandthat,forexplicitorimplicitobjectives,infers,fromtheinputitreceives, howtogenerateoutputssuchaspredictions,content,recommendations,ordecisionsthatcan influencephysicalorvirtualenvironments(SeeArticle3(1))1 .

1 Hereinafter,requirementsforAIsystemsarediscussedintheformtheyarefoundinthefinalcompromisetextof theAIAct,insofarastheofficialadoptedversionoftheregulationwasyettobepublishedintheOfficialJournalof theEUatthetimeofpreparingthisreport.Itmustbekeptinmindthatthespecificarticles,points,orrecitalsofthe compromisetextcitedheremaydifferfromthetextoftheAIActpublishedintheOfficialJournal,asthenumbering inthecompromisetexthasnotbeencorrected.–AccessibleontheInternet: https://data.consilium.europa. eu/doc/document/ST-5662-2024-INIT/en/pdf LastvisitedFebruary24th,2024

Thecitedcompromisetext[99 ]statesthatthepurposeoftheregulationistopromotetheuptakeofhuman-centeredandtrustworthyartificialintelligencewhilepromotinginnovationand ensuringahighlevelofprotectionofhealth,safety,fundamentalrights,democracy,ruleoflaw, andtheenvironmentagainstharmfuleffectsofartificialintelligencesystems.Theregulation setsoutharmonisedrequirementsforplacingonthemarket,puttingintouse,anduseofAIsystemsintheEU.Itprohibitscertainusesofartificialintelligence,laysdownspecificrequirements forhigh-riskAIsystems,andtheobligationsoftheoperatorsofsuchsystems.Italsosetsout harmonizedtransparencystandardsforcertainAIsystems,andrequirementsfortheplacingon themarketofgeneral-purposeAImodels.Theregulationalsolaysoutrulesformarketsurveillanceandmonitoringandmeasuresforsupportinginnovation,withamainfocusonsmalland mediumenterprises,includingstarts-ups.

3.3.1PersonsfallingwithinthescopeoftheAIAct

ThefollowingpersonsfallwithinthescopeoftheAIAct:

1. providersplacingAIsystemsonthemarketintheEUorusingthemintheirservicesorplacingonthemarketageneral-purposeAImodel,irrespectiveofwhethertheyareestablished orlocatedwithintheEUorinathirdcountry;

2. deployersofAIsystemsoperatingorestablishedwithintheEU;

3. providersanddeployersofAIsystemsoperatingorlocatedinathirdcountry,totheextent thattheoutputoftheirAIsystemisusedwithintheEU;

4. importersordistributorsofAIsystems;

5. productmanufacturerswhoareplacingonthemarketorputtingintouseAIsystemsalong withtheirproductundertheirnameortrademark;

6. authorisedrepresentativesofprovidersestablishedoutsidetheEUand

7. affectedpersonslocatedwithintheEU.

Article3oftheAIActsetsoutanumberofnewterms,includingthedefinitionsofdeepfakesand AIliteracy,aswellastraining,validation,testing,andinputdata.AIliteracyiseventhesubject ofaseparatearticle(Article4b)thatobligatestheprovidersanddeployersofAIsystemstotake measuresto,e.g.,ensureasufficientlevelofAIliteracyoftheirstaffandotherpersonsdealing withtheoperationanduseofAIsystems.

Below,wehavepresentedsomeofthemoreimportantrequirementsforAIstakeholdersfound inthefinalcompromisetextoftheAIAct[99 ].

3.3.2ExclusionsfromthescopeoftheAIAct

TheregulationdoesnotapplytodeployerswhoarenaturalpersonsusingAIsystemsinthe courseofapurelypersonalnon-professionalactivity.Italsodoesnotapplyto,e.g.,AIsystems usedsolelyformilitary,defenceornationalsecuritypurposes.Excludedfromthescopeof theAIActarealsoAIsystemsandmodels,includingtheoutputsofsuchmodels,specifically developedandputintoserviceforthesolepurposeofscientificresearchanddevelopment.It alsodoesnotapplytoscientificresearch,testinganddevelopmentactivityonAIsystemsor modelspriortobeingplacedonthemarketorputintoservice,withoutprejudicetothetesting ofAIsystemsinreal-lifeconditions.Finally,thescopeoftheregulationdoesnotincludeAI systemsreleasedunderfreeandopensourcelicences,withoutprejudicetosystemsplacedon themarketorputintoservicease.g.,high-riskAIsystems.

3.3.3Prohibitedartificialintelligencepracticesanduses

TheregulationprohibitsanumberofAIpractices(seeArticle5fordetails).TheseincludeprohibitionsonusesofAIsystemsthatpurposefullymanipulatewithapersonwiththeobjectiveto distorttheirbehaviourandappreciablyimpairtheperson’sabilitytomakeaninformeddecision. TheregulationalsoprohibitsAIsystemsexploitinganyofthevulnerabilitiesofapersonora specificgroupofpersonsduetotheirage,disabilityoraspecificsocialoreconomicsituation. Anotherprohibitionisrelatedtotheuseofbiometriccategorisationsystemsthatcategorisenaturalpersonsbasedontheirbiometricdatatodeduceorinfertheirrace,politicalopinions,trade unionmembership,religiousorphilosophicalbeliefs,sexlifeorsexualorientation.AIsystems arealsonotallowedtobeusedfortheclassificationofnaturalpersonsbasedontheirsocial behaviourorpersonalitycharacteristicsalongwithasocialscoreleadingtothedetrimentalor unfavourabletreatmentoftheperson.

3.3.4Criteriaforhigh-riskAIsystems

CriteriafortheclassificationofAIsystemsashigh-riskarelaidoutinArticle6oftheregulation proposal.Forexample,anAIsystemisalwaysconsideredahigh-risksystemifitisintended fortheprofilingofnaturalpersons.AproviderwhoconsidersthatanAIsystemreferredto inAnnexIIIisnothigh-riskmustdocumentitsassessmentbeforethatsystemisplacedon themarketorputintoservice.Suchproviderissubjecttotheregistrationobligationsetoutin Article51(1a)anduponrequestofnationalcompetentauthorities,theprovidermustprovidethe documentationoftheassessment.Nolaterthan18monthsaftertheentryintoforceoftheAI Act,theEuropeanCommissionmustprovideguidelinesspecifyingthepracticalimplementation ofArticle6completedbyacomprehensivelistofpracticalexamplesofhighriskandnon-high riskusecasesonAIsystems.

Article9setsoutrequirementsforriskmanagementsystemsforhigh-riskAIsystems.According topoint2inthearticle,theriskmanagementsystemisunderstoodasacontinuousiterative processplannedandrunthroughouttheentirelifecycleofahigh-riskAIsystem,requiringregular systematicreviewandupdating.Itcomprisesthefollowingsteps:

a) identificationandanalysisoftheknownandthereasonablyforeseeablerisksthatthehighriskAIsystemcanposetothehealth,safetyorfundamentalrightswhenthehigh-riskAI systemisusedinaccordancewithitsintendedpurpose;

b) estimationandevaluationoftherisksthatmayemergewhenthehigh-riskAIsystemisused inaccordancewithitsintendedpurposeandunderconditionsofreasonablyforeseeable misuse;

c) evaluationofotherpossiblyarisingrisksbasedontheanalysisofdatagatheredfromthe post-marketmonitoringsystem(seeArticle61)and

d) adoptionofappropriateriskmanagementmeasures.

High-riskAIsystemsmustmeettherequirementssetoutintheAIAct(seeChapter2),taking intoconsiderationthepurposeofsuchsystems,aswellasthelevelofAIandrelatedtechnologies.Morespecifically,theriskmanagementmeasuresmustbesuchthatrelevantresidual riskassociatedwitheachhazardaswellastheoverallresidualriskisjudgedtobeacceptable (Article9(4)).

High-riskAIsystemsmustalsobetestedforthepurposesofidentifyingthemostappropriateriskmanagementmeasures(Article9(5)).Testingproceduresmayincludetestinginreal

worldconditions(Article9(6);seealsoArticle54a).Considerationmustalsobegivenpotential impactstopersonsundertheageof18andothervulnerablegroupsofpeople(Article9(8)).

High-riskAIsystemswhichmakeuseoftechniquesinvolvingthetrainingofmodelswithdata mustbedevelopedonthebasisoftraining,validationandtestingdatasetsthatmeetthequality criteriasetoutintheAIAct(Article10(1)).Training,validationandtestingdatasetsmustalsobe subjecttoappropriatedatagovernanceandmanagementpracticesappropriatefortheintended purposeoftheAIsystem,e.g.,todetect,preventandmitigatepossiblebiases(Article10(2)(fa)).

Training,validationandtestingdatasetsmustberelevant,sufficientlyrepresentative,and,to thebestextentpossible,freeoferrorsandcompleteinviewoftheintendedpurpose,aswell aspossessingtheappropriatestatisticalproperties(Article10(3)).

Theprocessingofspecialcategoriesofpersonaldataforthepurposesofensuringbiasdetectionandcorrectioninhigh-riskAIsystemissubjecttostrictregulation.ItmustmeetallEUdata protectionregulationsandforsuchprocessingtooccur,criteriasetoutinpoints(a)–(f)ofArticle 10(5)mustbefulfilled.First,itmustbeexplainedwhythebiasdetectionandcorrectioncannot beeffectivelyfulfilledbyprocessingotherdata,includingsyntheticoranonymiseddata.

Specialcategoriesofdatamustbeprocessedusingstate-of-the-artsecurityandprivacy-preserving measures,includingpseudonymisation,orprivacyenhancingtechnologies.Measuresmustbe takentoensurethesecurityofthedata,includingincludingstrictcontrolsanddocumentation oftheaccesstoavoidmisuseandensureonlyauthorisedpersonshaveaccesstothosepersonaldatawithappropriateconfidentialityobligations.Suchdataarenottobetransmitted, transferredorotherwiseaccessedbyotherparties.Thedatamustbedeletedoncethebias hasbeencorrectedorthepersonaldatahasreachedtheendofitsretentionperiod,whatever comesfirst.

Thetechnicaldocumentationofahigh-riskAIsystemmustbedrawnupbeforethesystemis placedonthemarketorputintoserviceandhastobekeptup-todate.Thedocumentationmust contain,ataminimum,theelementssetoutinAnnexIV(Article11(1)).High-riskAIsystemsmust technicallyallowfortheautomaticrecordingofevents(logs)overthedurationofthelifetimeof thesystem(Article12(1)).

High-riskAIsystemsmustbedesignedanddevelopedinsuchawaytoensurethattheiroperationissufficientlytransparenttoenabledeployerstointerpretthesystem’soutputanduse itappropriately(Article13(1)).High-riskAIsystemsmustbeaccompaniedbyinstructionsfor useinanappropriatedigitalformatorotherwisethatincludeconcise,complete,correctand clearinformationthatisrelevant,accessibleandcomprehensibletousers(Article13(2)).Said instructionsmustcorrespondtotheminimalrequirementssetoutinArticle13(3)oftheregulation.

High-riskAIsystemsmustbeequippedwithmeanstoensurethattheycanbeeffectivelyoverseenbyhumansduringtheperiodinwhichtheAIsystemisinuse(seehumanoversightrequirementsandprinciplessetoutinArticle14).Forexample,humansneedtobeabletointervenein theoperationofahigh-riskAIsystemorinterrupttheoperationofthesystemthrougha’stop’ buttonorasimilarprocedure(Article14(4)(e)).

High-riskAIsystemsmustbedesignedanddevelopedinsuchawaythattheyachieveanappropriatelevelofaccuracy,robustness,andcybersecurity,andperformconsistentlyinthose respectsthroughouttheirlifecycle(Article15(1)).Suchsystemsneedtoberesilientasregards toattemptsbyunauthorisedthirdpartiestoaltertheiruse,outputsorperformance(Article 15(4)).

Article21oftheregulationmandatesthatprovidersofhigh-riskAIsystemswhichhavereason toconsiderthatahigh-riskAIsystemwhichtheyhaveplacedonthemarketorputintoservice isnotinconformitywiththeAIActmustimmediatelytakethenecessarycorrectiveactions,e.g., tobringthatsystemintoconformityortodisableit.Theprovidermustalsoinformdistributors and,ifapplicable,deployers,authorisedrepresentatives,andimportersofthesystem.

3.3.5RequirementsforparticipantsintheAIvaluechain

TheAIActalsosetsoutavarietyofrequirementsforotherAIsystemstakeholders,suchasdeployers,authorisedrepresentativesofnon-EUproviders,importers,andmarketers.Itisthereforeimportanttoassessanyspecificperson’sroleintheAIvaluechaininaccordancewiththe AIActtoidentifythespecificrequirementstheyneedtofollow.

TheAIActisanewadditiontotheEUlaw;thenewnormsandthoseimplementingthesenorms thusneedsometimetoadapttothenewsituation.Thiswillhopefullybefacilitatedbythe EuropeanAIOffice–thecentreofAIexpertiseacrosstheEU.TheAIOfficeplaysacentralrole intheimplementationoftheAIAct,supportingthedevelopmentanduseoftrustworthyAIand internationalcooperation[100 ].

3.4AILiabilityDirectiveproposal

InordertomitigateAI-relatedrisks,theAIActproposalwasfollowedbyaproposalforadirective onAIliabilityinSeptember2022[101],theaimofwhichistoensurethatpersonsharmedby AIsystemshavereasonablemeansavailableforprotectingtheirrights.Thedirectivewould harmonisenationalnormsfornon-contractualliability.Itisalsomeanttoincreaselegalcertainty forbusinessesdevelopingorusingartificialintelligence.

Oneofthemeasuresforeseenbythedirectiveistoexpediatecourtproceedingsforvictimswho havebeenharmedbyanAIsystem.Thevictimswillbeabletoclaimcompensationbothindividuallyorcollectively,asappropriate.Ifaviolationhastakenplaceandapotentialcausallink existstoanAIsystem,arebuttablepresumptionofcausalitywillbeapplied.Morespecifically, apresumptionofcausalitycanonlybeappliedwhenitcanbeconsideredlikelythatthegiven faulthasinfluencedtherelevantAIsystemoutputorlackthereof,whichcanbeassessedonthe basisoftheoverallcircumstancesofthecase.Atthesametime,theclaimantstillhastoprove thattheAIsystem(i.e.itsoutputorfailuretoproduceone)gaverisetothedamage[101].

Theproposeddirectivewillalsoprovidebetteropportunitiesforensuringlegalprotection.For instance,acourtmayorderthedisclosureofrelevantevidencetoavictimtodeterminethe causeofthedamageandidentifywhichpersonisliableforcompensatingthedamage.

3.5Productsafety

Regulation(EU)2023/988oftheEuropeanParliamentandoftheCouncilofMay10th,2023 ongeneralproductsafety[102 ]laysdownessentialrulesonthesafetyofconsumerproducts placedormadeavailableonthemarket(Regulation(EU)2023/988,Article1(2)).Recital5of theregulationnotesthat ’[d]angerousproductscanhaveverynegativeconsequencesforconsumersandcitizens.Allconsumers,includingthemostvulnerable,suchaschildren,olderpersonsorpersonswithdisabilities,havetherighttosafeproducts.Consumersshouldhaveat theirdisposalsufficientmeanstoenforcethatrightandMemberStatesshouldhaveadequate instrumentsandmeasuresattheirdisposaltoenforcethisRegulation’.

OnSeptember28th,2022,theEuropeanCommissionpublishedaproposalforenactingadirectiveonliabilityfordefectiveproducts[103 ].Theobjectiveofthisdirectiveistolaydownthe rulesgoverningtheliabilityofeconomicoperatorsfordamagecausedbydefectiveproducts andtheconditionsunderwhichnaturalpersonshavearighttocompensation.Thedirective alsoforeseessolidaryliability.Accordingtothedirective,economicoperatorsareliablefor defectiveproductsfor10yearsfollowingplacingtheproductonthemarket.

Theexplanatorymemorandumfortheproposeddirectiveexplainsthatoneofitsobjectivesis alsotoensureliabilityfordefectsinartificialintelligencesystemswhichhavecausedphysical harm,propertydamage,ordataloss.InsuchsituationstheuserwillhavetherighttoseekcompensationfromtheprovideroftheAIsystemoranymanufacturerintegratinganAIsysteminto anotherproduct.Thescopeoftheproposalalsoincludessoftwareproviders,businessesthat makesubstantialmodificationstoproducts,authorisedrepresentatives,andfulfilmentservice providers,givinginjuredpersonsabetterchanceofbeingcompensatedfordamagesuffered 2 .

3.6Intellectualproperty

Thepurposeofintellectualpropertylawistoprotectthecreationsofthemind.Generativeartificialintelligencehaschangedthesociety’sunderstandingofcreativityandpropertyrights, raisingquestionsregardinghumaninputandintellectualproperty[104 ].Atthetimeofpreparingthisreport,theinteractionsbetweenintellectualpropertyrightsandartificialintelligence havebecomeoneofthemainareasofdevelopmentofintellectualpropertylaw,mainlythanks todevelopmentsrelatedtoAI,initialrelevantcaselaw,andpoliticalinitiativesundertakenby internationalorganisationandlegislators[105 ].

Inrecentyears,legalscholarshaveincreasinglyturnedtheirattentiontoissuesrelatedtoartificialintelligenceandintellectualproperty.Thiscanbedividedintotwomaincategories.

1. Legalprotectionforautomatedcreation–e.g.arethereanycircumstancesunderwhichAIgeneratedworkscouldbesubjecttocopyrightorinventionstheyhavecreatedbepatented?

2. Intellectualpropertyviolations–e.g.howtoefficientlyprotecttheholdersofintellectual propertyrightsfromthedevelopersofartificialintelligencesystemswhouseworksprotectedunderintellectualpropertylawfortrainingtheirAIsystemswithouttherightsholder’s knowledgeand/orconsent?

GenerativeAIcapableofwritingcohesivetexts,creatingartorarchitecturaldesignshasgiven risetoallmannersofquestionsregardingthenatureofintellectualpropertyandhasbecomea causeforlegaldisputes.ExamplesexistofbothcasesofauthorstakinglegalactionagainstAI developerswhohaveusedprohibiteddataorworksfordevelopingtheirAIsystems(e.g.used copyrightedtexts,images,etc.withoutpermission)[106 107 ],aswellascasesofintellectual propertyrightsbeingclaimedforAI-generatedworks[108 ].

CurrentintellectualpropertylawgenerallygivesnoconsiderationstocreatorslikeAIsystems. Theregimeinplacetodaywascreatedtopromotehumancreationandinnovation.Fromtheperspectiveoftheintellectualpropertysystem,AI’sautonomyraisesfundamentalquestionsabout allformsofintellectualpropertyrights[109 ].Meanwhile,stronginteractionsandcorrelationcan beobservedbetweenAIsystemsandintellectualpropertylaw[110 ].Inmostcases,thefollowingtwoprinciplesareconsideredcritical:theoriginalityofthework,thedichotomyofideaand expression,andrenderingtheaboveinahuman-perceptibleform[111].

Thus,inaccordancewithSection4(2)oftheEstonianCopyrightAct,worksmean ’anyoriginal

2 Seeexplanatorymemorandumfortheproposeddirective,Section1.2andChapter2.

resultsintheliterary,artisticorscientificdomainwhichareexpressedinanobjectiveformand canbeperceivedandreproducedinthisformeitherdirectlyorbymeansoftechnicaldevices. Aworkisoriginalifitistheauthor’sownintellectualcreation.’ Oneoftheproposedsolutionsto theseissuesistheadoptionofahybridownershipmodel(AiLE)[111].Otherhave,meanwhile, foundthattheadditionofnewlayerstotheexistingintellectualpropertyrightssystemisnota goodsolutionforbalancingthesocialimpactoftechnologicalprogress[112 ],andthatthecreationsofAIarenotprotectable[113 ].TheEuropeanParliamentfindsitimportanttodistinguish betweenAI-assistedhumancreationsandcreationsautonomouslygeneratedbyAI[114 ].

TimewilltellwhatthefuturewillbringforintellectualpropertyrightsasAIsystemscontinueto develop.Itisclear,however,thatthereisanabundanceofdifferentopinionsregardingintellectualpropertyrightsandtherearecurrentlynosimplesolutionsonoffer.Itcannotevenbe ruledoutthatnowisnottherighttimetomakesuchdecisions,thatdevelopmentsrelatedto AIsystemsrequirecarefulconsiderationandacertainlevelofmaturityfromthesocietybefore anychangesaremadetofunctionallegalsystems.

3.7Legalrequirementsforcybersecurity

Justlikewithotherinformationsystems,thesecurityofartificialintelligencesystemsstarts fromensuringconfidentiality,availability,andintegrity.Dependingontheirroles,contexts,and operationalcapability,AIstakeholdersshouldapplysystematicriskmanagementineverystage oftheAIsystem’slifecycleinordertohandleriskstoprivacy,digitalsecurityandsafety,andto preventalgorithmicbias[25 ].

InaccordancewithOECDrecommendations,AIsystemsshouldremainsecure,reliable,andsafe throughouttheirentirelifecycle.Thisappliestobothroutineandplanneduseaswellasabuse andunfavourableconditions.EnsuringthemonitorabilityoftheAIsystemiscriticalforensuring theabove.Itappliesequallytothedataordatasetsused,variousprocessesanddecisions,and allowsperformingcontext-specificanalysesoftheoperationofanAIsystem,e.g.itsoutputs orreactionstoqueries[25 ].

ENISAliststhefollowingtypesofthreatstoICTinfrastructures[84 ]:

• adversarialthreats–theseareresultsofmaliciousintentions(e.g.denialofserviceattacks, non-authorisedaccess,masqueradingofidentity);

• accidentalthreats–thesearecausedaccidentally,e.g.throughhumanerror,orthrough legitimatecomponents.Usually,theyoccurduringtheconfigurationoroperationofdevices orinformationsystems,ortheexecutionofprocesses;

• environmentalthreats–theseincludenaturaldisasters(floods,earthquakes),human-caused disasters(fire,explosions),andfailuresofsupportinginfrastructures(poweroutage,communicationloss);

• vulnerabilities–existingweaknessesofAIsystemsthatmightbeexploitedbyanadversary. VariouslegalactshavebeenenactedinEuropetoreacttosuchthreats.TheSecondCybersecurityDirective(NIS2)[21]andtheCybersecurityAct[115 ]areconsideredtobethetwomost importantlegalactsoncybersecurityinEurope.AnotherkeylegalactistheGeneralDataProtectionRegulation(GDPR)[59 ].Saidlegalactsstresstheimportanceofsupplychainsecurity, privacy,andprotectionofpersonaldata,allofwhicharealsocentralthelifecycleofartificial intelligencesystems[84 ].

TheNIS2DirectiveenteredintoforceonJanuary16th,2023andalsocoversartificialintelligence systems.Morespecifically,thedirectiveseekstopromotetheuseofAIfor,e.g.,thediscovery

andpreventionofcyberattacks,andtheplanningofrelevantresources 3 .Essentialandimportantentitiesarerecommendedtoadoptbasiccyberhygienepracticesand,whereappropriate, integrateartificialintelligenceormachine-learningtechnologiestoenhancesecurity 4 .NIS2also requiresAIusetocomplywithEUdataprotectionlaw,includingincludingthedataprotection principlesofdataaccuracy,dataminimisation,fairnessandtransparency,anddatasecurity, suchasstate-of-the-artcryptography.TherequirementsofintegratedanddefaultdataprotectionlaiddownintheGDPRmustalsobefollowed[21].AnoverviewofNIS2canbefoundon thewebsiteofCentreforCybersecurityBelgium[116 ].

AproposalforaregulationoftheEuropeanParliamentandoftheCouncilononhorizontalcybersecurityrequirementsforproductswithdigitalelements(alsoknownastheCyberResilience ActortheCRA)introducesaEuropeancybersecuritycertificationframeworkforproductsand services[117 ].Thenecessityforsuchregulationisexplainedwiththelowlevelofcybersecurity ofproducts,servicesandaninsufficientunderstandingandaccesstoinformationbyuserson thesecurityoftheseproductsandservices.Article8oftheCRAlaysdownrequirementsfor high-riskAIsystems.

CybersecurityalsooccupiesacentralplaceintheAIActproposal[118 ].Forinstance,itplaysan importantroleinensuringtheresilienceofartificialintelligencesystemstoattemptstochange theiruse,behaviourorperformance,orputtheirsecurityfeaturesatriskbymaliciousthird partiesseekingtoexploitthesystem’svulnerabilities.Adversariesmaythustarget,e.g.,training data(datapoisoning),thetrainedmodels(adversarialattacksorre-identificationattacks),or exploitthevulnerabilitiesoftheAIsystem’sdigitalassetsortheunderlyingITinfrastructure. Adequateandefficientmeasuresaccountingforthecurrentleveloftechnologymustbetaken toensurerisk-appropriatecybersecurity.

3.8Dataprotectionandprivacy

Dataandprivacyenhancementconsiderationsmustbetakenintoaccountwhereverpersonal dataareprocessedinanystageoftheAIsystem’slifecycle(e.g.,intrainingorapplication).The mainlegalactregulatingtheprocessingofpersonaldataintheEUistheGeneralDataProtectionRegulation[59 ].OnJuly4th,2023,theEuropeanCommissionpublishedaproposalfora regulationlayingdownadditionalproceduralrulesrelatingtotheenforcementoftheGDPR[119 ]. Specialruleshavealsobeenestablishedforlawenforcementauthorities[120 ]andEUinstitutions[121].

Nationaldataprotectionandprivacynormsalsohavetobetakenintoconsideration,andin somecases,sectoralrequirementsmayalsoapply.Accordingly,foreachspecificsectorand activity,itisvitaltoconsiderthespecialnormsoftherelevantfieldalongsiderequirementsset outintheGDPR.Conditionsagreeduponbydifferentparties(e.g.,contracts,dataprotection agreements,termsofservice)mustalsobetakenintoaccount.

Thedeploymentofartificialintelligencedemandssolutionsforcomplexlegalproblems.Privacy anddataprotectionareafewofthemosturgentissues,especiallyinthelightofGDPRrules. TheGDPRintroduceshighstandardsfordataprotectionwhich,inturn,haveagreatimpacton AIsystemsdependentonlargeamountsofdata[122 ].ToensureanAIsystem’scompliancewith dataprotectionrequirements,itmusttakeintoaccountthepersonaldataprocessingprinciples laidoutinGDPRArticle5(1).Thecontrollerisresponsibleformustbeabletodemonstrate compliancewiththeseprinciples(GDPRArticle5(2)).Personaldatamustbe:

3 SeeNIS2,recital51.

4 SeeNIS2,recital89.

(a) processedlawfully,fairlyandinatransparentmannerinrelationtothedatasubject(‘lawfulness,fairnessandtransparency’);

(b) collectedforspecified,explicitandlegitimatepurposesandnotfurtherprocessed inamannerthatisincompatiblewiththosepurposes;furtherprocessingforarchivingpurposesinthepublicinterest,scientificorhistoricalresearchpurposesor statisticalpurposesshall,inaccordancewithArticle89(1),notbeconsideredto beincompatiblewiththeinitialpurposes(‘purposelimitation’);

(c) adequate,relevantandlimitedtowhatisnecessaryinrelationtothepurposesfor whichtheyareprocessed(‘dataminimisation’)

(d) accurateand,wherenecessary,keptuptodate;everyreasonablestepmustbe takentoensurethatpersonaldatathatareinaccurate,havingregardtothepurposesforwhichtheyareprocessed,areerasedorrectifiedwithoutdelay(‘accuracy’);

(e) keptinaformwhichpermitsidentificationofdatasubjectsfornolongerthanis necessaryforthepurposesforwhichthepersonaldataareprocessed;personal datamaybestoredforlongerperiodsinsofarasthepersonaldatawillbeprocessedsolelyforarchivingpurposesinthepublicinterest,scientificorhistorical researchpurposesorstatisticalpurposesinaccordancewithArticle89(1)subjecttoimplementationoftheappropriatetechnicalandorganisationalmeasures requiredbythisRegulationinordertosafeguardtherightsandfreedomsofthe datasubject(‘storagelimitation’)

(f) processedinamannerthatensuresappropriatesecurityofthepersonaldata, includingprotectionagainstunauthorisedorunlawfulprocessingandagainstaccidentalloss,destructionordamage,usingappropriatetechnicalororganisational measures(‘integrityandconfidentiality’)

ConsideringthesizeofdatasetsusedfordevelopingandtestingAIsystems,itmayprovedifficulttoensurethecomplianceofAIsystemswithcertaindataprotectionrules(e.g.,dataminimisation,purposeandstoragelimitations).Therapiddevelopmentofgenerativeartificialintelligenceandlargelanguagemodelshasposedthequestionofadaptingexistingdataprotection rulesinthisnewcontext.

Differentdataprotectionauthoritieshavepublishedguidancedocumentsonfollowingdataprotectionprinciplesandrulesinthedevelopment,deployment,anduseofAIsystems.Someof theseauthoritiesincludetheFrenchNationalCommissiononInformaticsandLiberty(CNIL)[123 ] andtheUKInformationCommissioner’sOffice(ICO)[124 ].Inearly2024,theICOalsolaunched aseriesofconsultationsongenerativeAIwiththeobjectiveofdetermininghowdataprotectionrulesshouldbeappliedinthedevelopmentanduseofAItechnology[125 ].Theconsultationsstudyvariousaspectsrelatedtodataprotection,e.g.,traininggenerativeAImodelson web-scrapeddata,accuracyofgenerativeAIoutputs,implementationofthepurposelimitation principle,guaranteeingdatasubjects’rights[126 ].Theconsultationswillbeusedtopublish relevantrecommendations.

PrivacyanddataprotectionneedtobeensuredthroughouttheentirelifecycleofanAIsystem[45 ].Privacyanddataprotectionareespeciallyimportantduetothefactthatbehavioural datamaypermitAIsystemstoinfernotjustaperson’spreferencesbutalsootherpersonaland relativelyprivateinformation,e.g.,sexualorientation,age,gender,religiousbeliefsorpolitical views.ItisthereforevitalforAIsystemstoensurethatprivacyanddataprotectionrequirementsaremetnotonlyinthecaseoftheinitialdataprovidedbythesystem’suserbutalso thedataproducedinthecourseofusingthesystem(outputs,reactionstorecommendations,

etc.).Anykindofunlawfulandunfairdiscriminationonthebasisofdatamustbeoutruled[45 ]. TherehavebeencaseswhereAIsystemshaveleakedsensitiveinformation,e.g.,conversation histories[87 ].

TheEUAIHLEGhasfoundthatprivacyissuesarecloselytiedtotheprincipleofpreventionof harm.Relevantdatamanagementmeasuresmustbeappliedtoensureprivacy,whichincludes managingthequalityandintegrityofthedatabeingused,andaccessprotocols[45 ].

TheAIActproposalincludesanassessmentoftheneedarisingincertainsituationstoconductevaluationsoftheimpactofAIsystemsonfundamentalrightsandtocarryoutadata protectionimpactassessment[118 ].Theproposalfindsthattheconductionofsuchimpactassessmentsmustbeplannedasapartofanoverarchingprocessinordertoreduceredundancy andunnecessaryadministrativeburden.ThefutureAIOfficewouldbetaskedwithdeveloping aquestionnairethatthedeployersofAIsystemscouldusetomeettherelevantcriteria[118 ].In anycase,thedevelopmentanduseofAIsystemsmustcomplywithexistingprivacyanddata protectionrules.

SinceAIsystemsarefoundedondata,thequalityofthisdataiscritical.Dataqualityisalso importantforthecreationofthestructureofAIsystemsandensuringtheiroperability.Training, validation,andtestdatamustberelevant,sufficientlyrepresentative,maximallyerror-freeand completefromthepointofviewofthepurposeoftheAIsystem.Therequirementfordatasets tobemaximallycompleteanderror-freeshouldnotimpacttheuseofprivacy-preservingtechnologiesinthecontextofthedevelopmentandtestingofAIsystems[118 ].

Itmustalsobetakenintoaccountthatthecompilationofdatasetsmustbebasedonthelawful useofdataincompliancewithdataprotectionregulations[127 ].Theprocessingofpersonal dataisonlylawfulifatleastoneoftheconditionsofGDPRArticle6(1)(pointsa–f)ismet.There havebeencaseswherecompetentauthoritieshavedemandedthedeletionofmodelsbased onunlawfullycollecteddata[128 ].Inordertopreventanyformofdiscrimination,thedatasets shouldalsopossesstherelevantstatisticalpropertiesandaccountforfeaturescharacteristic tothespecificsituationorgroupofpersons.

InordertocomplywithGDPRrequirements,anartificialintelligencesystemmustbedeveloped, trained,andputintoservicewithaclearlydefinedpurpose.TheFrenchNationalCommission onInformaticsandLiberty(CNIL)recommendsthepurposeoftheAItobedeterminedinthe earlyplanningstagesoftheprojec.Thepurposeofthesystemmustbelawful,clear,andunderstandable,andusablefordeterminingwhichkindsofdataneedtobeprocessedforthis specificpurpose,aswellashowlongtheywillhavetoberetainedinordertoachievetheenvisagedobjective[127 ].

Eventhoughtheprincipleoflimitedpurposerequiresusingpersonaldataonlyforachievinga specificpredeterminedgoal,thismayprovecomplicatedinthecaseofanAIsystem.TheCNIL hasfoundthatatthealgorithmtrainingstageitisnotalwayspossibletodefineallthepossible futureusesoftheartificialintelligence;nevertheless,thetypeandmainpotentialfunctionsof thesystemshouldstillbedefinedasclearlyaspossible[129 ].

DiscussionsrevolvingaroundtheextraterritorialenforcementoftheGDPRgivereasontobelieve thatthejurisdictionalmodelimplementedinsaidregulationwhichhasalsobeenintroducedinto theEUAIActmaynotbeapplicableinpractice[130 , 131, 132 , 133 ].AccordingtoArticle3(2), points(a)and(b)oftheGDPR,theregulationalsoappliestotheprocessingofpersonaldata ofdatasubjectswhoareintheEUbyacontrollerorprocessornotestablishedintheEUifthe processingisrelatedtotheofferingofgoodsorservicestosuchdatasubjectsintheEUorthe monitoringoftheirbehaviourasfarastheirbehaviourtakesplacewithintheEU.

InthecourseoftheimplementationoftheGDPR,therehavebeennumerousdisputesover specificallytheprocessingofpersonaldatabycontrollersorprocessorswhofallwithinthe scopeofArticle3(2)oftheGDPRbutwhorefusetocooperatewithEuropeandataprotection authoritiesordonotrecognisetheEU’sjurisdiction(see,e.g.,theClearviewAIcase)[134 , 132 ]. TheAIActproposalalsousesanapproachsimilartotheGDPRwherebusinessesfromnonEUstatesareincludedwithinthescopeoftheregulation(seeArticle2(1)(c))[99 ].Inpractice, competentauthoritiesmaybefacingproblemssimilartothosethathaveariseninconnection totheextraterritorialenforcementoftheGDPR.

Thetransferofpersonaldatatonon-EUstatesandinternationalorganisationsisregulatedby ChapterVoftheGDPR.Thetransferofdataisgenerallypermittedonlyifsuitablelegalgrounds existforsuchtransfer(GDPR,Articles6and9)andrelevantandefficientprotectionmeasures aretaken[135 ].Article45oftheGDPRgivestheEuropeanCommissiontherighttodetermine whetheranon-EUstateorinternationalorganisationprovidesanadequatelevelofdataprotection[136 , 137 ].Forexample,inJuly2023,theCommissionadoptedanadequacydecisionfor theEU-USDataPrivacyFramework[138 ]5 .TheexistenceofarelevantdecisionbytheCommissionremovestheneedforaspecificauthorisationforthetransferofdata(GDPR,Article45(1)). EEAstates(Norway,Iceland,Liechtenstein)areconsideredtobestateswithanadequatelevel ofdataprotection.

Additionalsafeguardsmustbeimplementedwhentransferringdatatostateslackinganadequatelevelofdataprotection(see,e.g.[139 ]),oroneofthederogationslaiddownintheGDPR mustbeapplicable(GDPR,articles46–49)[140 ].TheEuropeanDataProtectionBoard(EDPB) hasfoundthatincertainsituationsremoteaccessfromanon-EUstate(e.g.,supportservices, troubleshooting),aswellasstorageinacloudsituatedoutsidetheEEAmaybeconsideredto beatransferinthemeaningoftheGDPR[141].Itisthereforestrictlyadvisabletoplanoutthe AIinfrastructurebeforeenteringintoanyagreementswithserviceprovidersinordertoavoid laterlegaldisputesorsanctions.

3.9Importanceofthelegalframework

PersonscentraltothelifecycleofanAIsystemneedtobeup-to-dateonthelegalandregulatory requirementsshapingthelegalframeworktheyoperatein.Thisdeterminestherequirements thattheAIsystemaswellasthepersonoperatingthesystemmustmeet.Variousaspects ofadministeringandmanagingprocessesrelatedtotheAIsystem,suchasthedevelopment, testing,andmonitoringofthesystemarealsotiedtotheabove.

Aholisticapproachtoinformationtechnology,security,andlegalissuesisincreasinglyimportant fororganisations.Thisalsomeansclosecooperationbetweenpeoplefulfillingtherelevantroles fromthestageofdesigninganAIsystemtotheendofitslifecycle.This,inturn,facilitates expandinglegalspecialists’knowledgeoftechnologyandviceversa,thuscontributingtoan increaseoforganisationalknowledge.

Thegreatertheawarenessoftherequirementsrelatedtothelegalframework–evenatthe stageofdesigninganAIsystem–andthemoresaidrequirementsareactuallyadheredto,the smallertheprobabilityoftheoccurrenceofundesirablescenarios.Meanwhile,itmustbetaken intoaccountthatAIlawisstillfarfrommatureandthelegalenvironmentcanbeexpectedto continuetochange.

5 EarliersimilaragreementsanddecisionsbetweentheEUandtheUShaverepeatedlybeendeclaredvoid.We recommendthereadersofthisreporttomonitorthecurrentlegalsituationbeforetransferringEUcitizens’datato theUS.

4AIapplicationdeploymentmodels

4.1Introduction

DevelopersofAIapplicationscanchoosefromavarietyofarchetypeswhendeployingtheir apps.ThebiggestdifferenceisinthewaytheAImodelisusedbytheapp.SomeAImodels areaccessibleforfreewhileotherscanonlybeaccessedviapaidapplicationprogramming interfaces(APIs).Inthefollowing,wewillpayextraattentiontoapplicationsusingcloud-based services,asthetransferofdatabetweendifferentdataprocessorsbringsadditionalrisksto privacy.Cloudprocessing(ortheuseofsecond-partydatacentresingeneral)isalsovery commonintoday’sITsystems.

ThetechnologicallysimplestAIsystemisanapplicationimplementingaspecificbusinesslogic onthebasisofanexistingAIAPI.Oneexampleofsuchasolutioncouldbeachatbotusingthe OpenAIGPTAPIwherethemainvalueproposalistheuserexperienceandpromptsprovidedby theapp.Thinapplicationsofthistypemaybelimitedbythecontextlearningcapabilityofthe modelbehindtheAPI.

Morecomplexandmoreexpensivesolutionsuseanexistingmodel’sAPIcallswhilemanagingthe user’sstatusandservicingtheirdatawhichmaybedomain-specific.Solutionslikethisrequire databaseintegration,usermanagementandalsoinputandoutputvalidation.Thedeployerof theappmaythususe,e.g.,someRetrieval-AugmentedGeneration(RAG)solutionwherethe model’sgenericknowledgeisaugmentedwithinformationfoundintheapp’sowndatabase. SolutionsofthistypearediscussedinSection 4.4.2 .

SomesolutionsinvolvetheserviceproviderdeployinganAImodelthemselves.Thispresumes thattheserviceprovidereithertrainstheirmodelthemselves,fine-tunesanexistingmodelor adoptsanexternalmodelwhileindependentlyrunninginference(i.e.,computingtheAI’soutputs ontheirowninfrastructure).Thisrequiresinvestmentsintoinformationinfrastructurewhich growwiththesizeofthemodelanduserbase,butmayatthesametimereducerisksrelatedto APIavailability,dataconfidentialityandprivacy,asthenumberofdataprocessorsisreduced.In situations,whereservingalargeuserbaseisnotthegoal,quantificationandotheroptimisation methodsallowrunninginferenceonmanyfreelyaccessiblemodelsevenonapowerfulpersonal computer.SolutionsofthistypearediscussedinSections 4.4.3 and 4.4.4 .

Alldeploymentmodelscoveredheresharesomesimilarcharacteristics.Forexample,aservice providermayuseIaaS(infrastructureasaservice),CaaS(computingasaservice),andPaaS (platformasaservice)servicesforbusinesslogic,model,anddatamanagement.InthecontextoftheGeneralDataProtectionRegulation(GDPR),theseserviceprovidersareconsidered processorsofuserdata.Incaseuserdataareusednotonlyforserviceprovisionbutalsofor improvingthequalityofthemodelorothersidetasks,alegalbasismustbeestablished(e.g., theusermayhavetogivetheirinformedconsent)forsuchuses.Thiscomesintoplayinthe contextofinterfacingtheservicewithotherservicesanddata.

4.2Methodology

Inthedevelopmentofthedeploymentmodelsdiscussedherewetookintoaccounttheconsiderationsandneedsofpotentialserviceproviders,aswellastheireverydaypractices.We especiallyfocusedonstatutoryrequirementsandthemovementofuserdatabetweendifferent processors.Theoverviewofdeploymentmodelspresentedbelowisnotexhaustive,asthere

arecountlesswaysforconnectingservices,APIs,anddatasources.Itshould,however,provide asufficientpictureofthecriticalpointsofmorecommonapproachesthatarerelatedtousers’ andserviceprovider(s)’rolesandresponsibilitiesinthecontextofthestructureofthedeploymentmodelanddataflow.Simplermodelsalsofacilitateprovidingfasteradviceforcarryingout riskanalysis.

Arrowsinthefiguresrepresentdataflows,indicatingthemovementofdatabetweendifferent componentsofthedeploymentmodel.Representingdataflowsisvitalbecausethemovement ofdataacrossbetweenareasofresponsibilitycomeswithrisks(e.g.,toprivacy)whichmust beaccountedfor.Privacyandresponsibilityareunderstoodhereinthesensetheyareused intheGDPR.Tofacilitatebetterunderstandingoftheboundariesofresponsibility,aswellas othercharacteristicsofthedeploymentmodeltiedtothestructureofthespecificAIsupply chain,wehavepresentedbothservicesandcriticaldataelementsoftheAIsystem(training data,model,input,output)ascomponentsofthedeploymentmodel.OurfocushereisonAIbasedcloudservices,asduetotheirperformancerequirements,AIsystemsoftenneedtouse specialisedhardwareacceleratorsincloudservicesforacceleratingcomputations.Itmustbe keptinmind,however,thatAIsystemsnotdeployedviathecloudaresomewhatlessexposed toconfidentialityrisks;systemsofthiskindwillbediscussedseparately.IaaS,CaaSandPaaS componentsarenotspecificallyrepresentedinthedeploymentfigurediagrams,astheycan easilybeusedwithdifferentelementsofthedeploymentmodel.Wewill,however,discussthe consequencesoftheiruse.

WehaveusedperformanceanalysistoprovideamoredetailedpictureofAIapplicationdeploymentmodels.ModelsarepresentedusingBusinessProcessModellingNotation(BPMN).This hasallowedustospecifythedataobjectsprocessedbythemodel,aswellastheprocessing parties.

4.3LegalrolesofAIsystemstakeholders

FromtheperspectiveofboththeGDPRandtheAIAct,itiscrucialtoassesstheapplicabilityof theregulations.TheapplicabilityofGDPRrulesmustbeconsideredifanAIsystemprocesses personaldataanywhereinitslifecycle.TheapplicabilityofAIActrulesmustbeconsideredif thepersonisanAIdeveloperorifitusesanAIsystemorAPIdevelopedbysomeoneelseintheir services.AnAIsysteminthesenseoftheAIActisamachine-basedsystemdesignedtooperate withvaryinglevelsofautonomy,thatmayexhibitadaptivenessafterdeploymentandthat,for explicitorimplicitobjectives,infers,fromtheinputitreceives,howtogenerateoutputssuch aspredictions,content,recommendations,ordecisionsthatcaninfluencephysicalorvirtual environments[99 ].

IfanAIsystemorapersonoperatingthesystemisfoundtofallwithinthescopeoftheregulation(s),thespecificrequirementsarisingfromtheregulation(s)mustbeidentified.Fromthe perspectiveoftheGDPRitisimportantto,e.g.,determinewhethertheorganisationqualifies asacontrolleroraprocessorofpersonaldata;inthecaseoftheAIAct,however,whetherthe organisationqualifiesasaproviderordeployeroftheAIsystem.Bothregulationsalsodefine severalotherroles,whicharealsoadvisabletoreview.Theroleslistedabovearethemostcritical,though–especiallythoseofthecontroller(GDPR)andtheprovider(AIAct),asbothare subjecttostrictcompliancerules.Insomecases,asinglepersonmayalsosimultaneouslyact inseveraldifferentrolesdependingonprocesses,relationshipsbetweentheparties,oragreements.Identificationofrolesiscrucialbecauseofthedependenceofresponsibilityonroles. AccordingtotheGDPR,acontrolleristhenaturalorlegalperson,publicauthority,agencyor

otherbodywhich,aloneorjointlywithothers,determinesthepurposesandmeansoftheprocessingofpersonaldata(GDPR,Article4(7)).Aprocessorisanaturalorlegalperson,publicauthority,agencyorotherbodywhichprocessespersonaldataonbehalfofthecontroller(GDPR, Article4(8)).

Aproviderisanaturalorlegalperson,publicauthority,agencyorotherbodythatdevelopsanAI systemorageneral-purposeAImodelorthathasanAIsystemorageneral-purposeAImodel developedandplacesitonthemarketorputstheAIsystemintoserviceunderitsownnameor trademark,whetherforpaymentorfreeofcharge[99 ].Adeployerisanaturalorlegalperson, publicauthority,agencyorotherbodyusinganAIsystemunderitsauthorityexceptwherethe AIsystemisusedinthecourseofapersonalnon-professionalactivity[99 ].

Inordertoidentifywhichrequirementsapplyinthespecificcase,itisalsonecessarytodeterminetheobjectiveofthedataprocessingandAIuse,thetypesofdataprocessingprocesses operatinginthesystem,thetypesofdatabeingtransferredandthepartiesofthesetransfers, andtheAIsystemorcomponent(includingtherisklevelofthesystem)beingused.

4.4Deploymentmodels

4.4.1Overviewofmodels

WehaveidentifiedthreedistinctdeploymentmodelsforAIapplicationsdifferentiatedbythe transferofdatabetweenparties,thedeployingparty,andtheoriginoftheAImodel.Therelationshipsbetweenthesemodels,aswellasillustrativeapplications,arepresentedinFigure 9

Figure9.Deploymentmodelsfromtheperspectiveofthedeployer’stasksinrelationtotheAImodel

ThemodelshavebeenlistedintheorderoftheextenttowhichtheAIapplicationservice providercanrelyonexistingAIservicesandproducts.Themorespecificandcomplexthe commercialpurposeandthestrictertherequirementsfortheprocessingofdata,thebiggerthe proportionofnecessaryservicesthatusuallyneedtobedevelopedin-house.Thisheuristic isjustanapproximation,however.Fromtheperspectiveofdataflowstructure,thelastofthe listeddeploymentmodelsincludesbothsimpleandcomplexsolutions. Theupperpartofthefigurerepresentsthescopeofthedeployer’stasksindifferentdeployment models.Inallcases,thedeployerprocessessomekindofdata.Beginningwithcloudservices importinganexternalmodel,thedeployerdeploysthemodelitselfalongsidetheirbusinesslogic, fine-tuningthemodelifnecessary.Inthecaseofalocally-trainedmodel,nothirdpartyisany longerresponsibleforthecreationandtrainingofthemodel;both(aswellasthemanagement oftrainingdata)arecompletelyinthehandsofthedeployer.

4.4.2DM1:ServiceusinganAIAPI

OnecommonchoiceofarchitectureforAI-basedservicesisusingathird-partyAIAPIinyour businesslogic.Ifnecessary,theserviceprovidercanalsoprocessorstoreuserdata,which thedeployercanalsoimplementusingcloudservices.Theinitialdatausedfortrainingthe third-partymodelcaninturncomefromexternalsources.Alternatively,thethird-partyAIcloud serviceorAPIcantrainitsmodelsonuserdatareceivedfromtheserviceprovider.Allsuch cloudservicescanrelyonsomeIaaS(infrastructureasaservice)solution.

Themodeldescribedabovehasbeenusedin,e.g.,machinevisionapplications.Itgainedin popularityafterthepublishingoftheOpenAIAPIwhichfacilitatedsimpleinterfacingofyour servicewithpowerfullanguageandimagemodels.TheAImodelisexternaltotheapplication (i.e.,outsidetheserviceprovider’scontrol).Thetrainingdataforthemodelarealsoexternal inorigin.Userdataflowstotheservice,fromtheservicetotheAIAPIprovider,thenbackto theservice,andfinallybacktotheuser.Iftheserviceisinterfacedwiththird-partyservices anddatathentheuserdatamayalsobetransferredthere.Userdatacanmeanwhilebestored byboththeserviceproviderandtheAIAPIprovider(e.g.,storinginputsandoutputsincache, butalsointhetrainingdatabase).Inonespecialcaseofthisdeploymentmodel,theAIAPI provideralsoprovidestheoptionoffine-tuningthemodelontheserviceprovider’sdatabutthe APIproviderstilldeploysthefine-tunedmodel.Thisapproachpartiallyoverlapswiththenext deploymentmodel(seeSection 4.4.3 ).

DM1:ServiceusinganAIAPI

Overview: Serviceinterfaceswithanexternal APItoprocessuserdatausingtheAIAPI provider’smodel.BoththeserviceandtheAPI providercanalsosharedatawiththirdpartiesfor additionalprocessing.Theinitialdatausedfor trainingthemodelmaycomefromthird-party sources.

Examples: copy.ai,StreamlitandGradioAIdemo applications,servicesusingtheOpenAIAPI

Originofmodel: External

Originoftrainingdata: External

Inputdataarestored: Optional

Inputdatatransfers: Tocloudservice,then(if necessary)tootherservicesandtheAPI,backto theservice,thenbacktotheuser,potentiallyusing differentinfrastructures.

Figure:

Risksandconsiderations:

1. ProcessingofpersonaldatabytheserviceproviderorAPIprovider.

2. InformationontheexplainabilityofthemodelusedbytheAPIprovidermaybeincomplete.

3. Userqueriesandmodeloutputsarevalidatedbytheserviceprovider.

4. Non-serviceproviderrelatedfailuresintheworkoftheAIAPIarearisktoavailability.

5. Lowestcapitalinvestmentsandtechnologicallyleastcomplexofalldeploymentmodels.

Figure10.Copy.aiasanexampleofaserviceusingAIviaanAPI

Copy.aiisanexampleofaDM1-typeservice 1 .Copy.aiusestheOpenAIAPItoassisttheuserinwriting marketingandadvertisingtexts.Theuserprovidestheservicewithadescriptionofthetextrequired anditscharacteristics(e.g.,writingstyle),theserviceprocessesthedescriptions,andpresentsthemto theAIAPIintheformofaquery.TheusercanchoosewhichAPItheywishtouse(AnthropicorOpenAI). Afterreceivingaresponsetothequerytheservicefurtherprocessestheresponseandreturnstheresult totheuser.Thecopy.aideploymentmodelispresentedinFigure 10

Figure 11 describesthedataflowsinaDM1-typedeploymentmodel.InthecaseofaDM1model,the userusesaservicewhichinturnusesanAIAPItogenerateanoutput.TheAPIproviderisdividedinto twodivisionswithdifferenttasks–modeldevelopmentandservicedeployment.Theobjectiveofmodel developmentistodesignthemodelarchitecture,trainandtestthemodeland,ifnecessary,generate fine-tuningdatasetsandfine-tunethemodel.Modeldevelopmentisalsoresponsibleformonitoringthe model.

Theservicedeploymentprocessbeginswiththeuserwhousestheirinputdatatogeneratedatatobe senttotheservice.TheserviceusesthesetogenerateaquerywhichistransmittedtotheAPI.When theservicesendsaquerytotheAPIincludesthequerydatainitsinputandtheinputinthemodel.The modelwillbeusedtogenerateanoutputandrespondtothequery.Dependingonthetermsofservice, interactionhistorymaybestoredandusedforbothmodelmonitoringandthegenerationoffine-tuning datasets.

OncetheAPIhassenttheservicearesponsetothequery(i.e.,anoutput),theservicewill,inturn, generateauseroutputandtransmitthistotheuser.TheusercanusetheoutputreceivedfromtheAI serviceforfulfillingtheirpersonalobjectives.

4.4.3DM2:ServiceimplementinganexternalAImodel

InterfacingwithanexternalAIAPIorwebservicemakesthedeployerdependentontheaccessibilityof theserviceused.Thedeployermayalsoneedtofine-tunethemodelwhichisnotofferedbyallAIAPI providers.Inordertosolvetheseproblemsthedeployercanadoptapre-trainedmodelfromamodel provider(orafreelyprovidedAImodel)andintegratethisdirectlyintotheirapplication.Incasethe deployerisfine-tuningthemodel–thisiscalledtransferlearning–theywillfaceanadditionalneedfor managingtrainingdataandmonitoringthemodel’ssecurityandqualityindicators.Thismodelisalso applicabletothespecialcaseswherethemodelproviderprovidesafederatedlearningservicewith centralisedcomponents.

1 Copy.ai. https://www.copy.ai/ LastvisitedMay25th,2024.

DM2:ServiceusinganexternalAImodel

Overview: Serviceprovideruses(and fine-tunes,ifnecessary)anexternallyimported model.Theinitialmodelcomesfromanexternal source;thecreatorofthemodeltrainsand transfersthemodelfortheircustomers, includingtheserviceprovider.Theservice providerdeploysthemodel,usingitonin-house andclientdata.Theymayusein-housedatafor fine-tuningthemodel.Cloudservicescan interfacetootherdataandservices,e.g.,vector databasesinthecaseofRAGsolutions.

Examples: Serviceimportingamodelfrom,e.g., theHuggingfacerepository,AndroidGboard(asan exampleoffederatedlearning)

Originofmodel: External

Originoftrainingdata: External,in-house,user data

Inputdataarestored: Optional

Inputdatatransfers: Tocloudservice,optionallyto otherservicesandbacktotheuser,optionally usingthird-partyinfrastructure.Inthecaseof federatedlearning,weightupdatesarealso transferredtothemodel’strainer.

Figure:

Risksandconsiderations:

1. Infine-tuning,monitorsecurityandqualityindicators,aswellasqualityofin-housedataandchanges intheirdistribution.

2. Informationmustbecollectedonthesecurityandexplainabilityofasecondparty-trainedmodel.

3. Insomecases,weightupdatesmaybeconsideredpersonaldatainfederatedlearning.

Figure12.TranslationservicedeploymentmodelasanexampleofaserviceusinganexternalAImodel

Figure 12 representsanexampleofatranslationservicewherethetranslationservicedeployedbytheAI serviceproviderusesamodelpre-trainedbyamodelprovider(e.g.,TartuNLP).Theusersendsaquery totheservice(e.g.,throughtheapplication’swebinterface),thequeryissenttotheservicewherethe dataareprocessed(translated).Thetranslatedoutputisreturnedtotheuser.Dataarenottransferred fromtheserviceprovidertothemodelprovider.Theserviceprovidercanfurtherfine-tunethemodel basedonuserdata.

Figure 13 describesthedataflowsinaDM2-typedeploymentmodel.Theprocessinvolvesthreeparties: user,AIservice,andmodelprovider.Themodelproviderdevelopsthemodelarchitecture,trainsand/or fine-tunesandteststhemodel,andprovidesthemodeltoAI-basedserviceproviders.

TheAIservicesisdividedintotwodivisions:furtherdevelopmentofthemodelanddeploymentofthe service.Tofurtherdevelopthemodel,theAIserviceintegratesthemodelprovidedtothemintotheir ownservice,generatesafine-tuningdatasetifnecessary,andfine-tunesthemodel.Theserviceprovider thencontinuestomonitortheoperationofthemodel.Nodataistransmittedbacktothemodelprovider fromtheAIservice.

WhentheusercreatesandtransmitsdatatotheAIservice,theAIservicedeploymentbranchaddsthe datatotheinput,thentothemodeltogenerateanoutputwhichitwillthentransmittotheuser.TheAI servicethenstorestheinteractionhistorywhichwillbeusedformonitoringthemodelandcanalsobe usedforgeneratingfine-tuningdatasets.TheusercanusetheoutputreceivedfromtheAIservicefor fulfillingtheirpersonalobjectives.

Figure 14 depictsaspecialcaseoftheseconddeploymentmodel.Thedifferencebetweenthetwo figuresliesinadatabasequeryaddedtotheservicedeploymentstage,theresultofwhichisaddedto theuserinput.ThismethodiscalledRAG(RetrievalAugmentedGeneration);itcanalsobeusedwith deploymentmodelsDM1andDM3.

4.4.4DM3:AIserviceusinganin-housemodel

ThethirddeploymentmodelcoverssolutionswheretheAImodelistrainedanddeployedin-houseby theserviceprovider.Theseincludebothsimplesolutions,suchasdecisiontreesandregression-based solutions,wherethesimplicityofthemodelmakesitimpracticaltoimportfromanexternalsource,as wellassolutionsdevelopedbylargeAIproducers.TrainersoflargeAImodelsgenerallyonlyofferservicesbasedonmodelstheyhavedevelopedandtheypossesssufficientresourcesfortheirautonomous deployment.

DM3:AIserviceusinganin-housemodel

Overview: Thetrainerofthemodelcollectsdata, trains,deploys(and,optionally,implements)the model.Usingin-housemodelsisanimportant usecase.Thiscanfacilitatesituationswhere neitherthetrainingdatanorthemodelitselfor userdataaretransferredtothirdparties.

Examples: Neuraltranslation,ChatGPTandOpenAI API,Grok,DALL-E,Midjourney

Originofmodel: Internal

Originoftrainingdata: User,serviceprovider,third parties

Inputdataarestored: Optional

Inputdatatransfers: Tocloudservice,optionallyto otherservices,thenbacktotheuser,optionallyvia aninfrastructureserviceprovider.

Figure:

Risksandconsiderations:

1. Thetrainerofthemodelisexpectedtoprovideinformationontheexplainabilityandqualityofthe model.

2. Thetrainermusthavealawfulbasisforprocessingthetrainingdata.

3. Incaseoflargemodelsandlargetrainingdatasets,solutionsofthistypearethemostexpensiveto build.

xamplesofthisdeploymentmodelincludeallorganisationsusingAIforbuildinginternalservices,aswell ase.g.OpenAI.Theusersendsqueriestotheservice;theservice,inturn,returnstheoutputfromthe selectedmodel.OpenAI’sdeploymentmethodsforthemodelstheytraindependonthetargetgroup: somearedeployedintheformofAPIs,others–intheformofweights.OpenAIcollectsandpurchases trainingdataitself.Atthesametime,notallthedetailsoftheoriginofthedataarepublic.OpenAIAPI modelsarenottrained(asofNovember2023)onqueriesreceivedoverAPI;theyare,however,trained onChatGPTqueries,exceptforChatGPTEnterprise 2 .

ChatGPTitselfisalsoanexampleofthisdeploymentmodel,asitusesin-housemodels(developedby OpenAI).OnenoteworthythingaboutChatGPTisthefactthat,iftheuseremploysplugins,thesecan makequeriestothirdpartiesforadditionalprocessingoracquisitionofdata.Itisimportanttokeepin mindthatthemodeldoesnotcommunicatedirectlywiththepluginsortheservicestheyinterfaceto: thisdataexchangetakesplaceasapartoftheservice’sbusinesslogic.Asarule,thismeansthatif themodeldecidestouseapluginitwillusetheinformationcontainedinthepre-promptandtheuser’s requesttocomposeaquerytotheserviceinterfacedviatheplugin.Aresponsebasedonthequery composedbythemodelisreturnedtothemodelwhereitisformattedintoaresponseutilisablebythe user.TheChatGPTdeploymentmodelisshowninFigure 15

Figure15.ChatGPTdeploymentmodel

Anotherversionofanapplicationcompatiblewiththisdeploymentmodelisacomputationallyinexpensive rules-basedorothersimplemachinelearningalgorithm(e.g.,linearregression,decisiontree,ornaive Bayesianclassifier)easilytrainedonin-housedatasets.Thisdeploymentmodelissuitablefor,e.g.,bank creditriskmodels:thebanktrainsthemodelin-houseonitsown(clients’)dataandimplementsthemodel in-house.Thebankalsousessupplementarydata:creditdefaultdata,financialindicators,andinternal bankdata.ThedeploymentmodelforaserviceofthistypeisshowninFigure 16

Figure 17 describesathirdtypeofdeploymentmodels.Thismodelinvolvestwoparties:AIserviceand user.Inthisdeploymentmodel,theAIservice,thedeployer,andthemodelproviderareallthesame party.

TheAIserviceisdividedintotwo:modeldevelopmentandservicedeployment.Modeldevelopment involvesthesamestepsasthedeploymentmodelsdiscussedabove:modelarchitecturedevelopment, trainingandtestingthemodel,andoptionallyfine-tuningandmonitoringthemodel.Afterreceivingdata fromtheuser,theAIservicedevelopmentdivisionaddsthedatatotheinputandtothemodel,composes anoutput,andsendstheoutputtotheuser.Interactionhistoryisstoredandcanbeusedformonitoring themodelandassemblingfine-tuningdatasets.

2 EnterpriseprivacyatOpenAI. https://openai.com/enterprise-privacy VisitedDecember1st,2023

Figure16.Deploymentmodelforacreditinstitution’sretailcreditriskevaluationmodel

Figure17.DataflowsindeploymentmodelDM3

5RisksofAIapplications

5.1Riskmanagementmethodology

ThemainstandardsonriskassessmentaretheISO31000riskmanagementstandard[142 ]andthe NISTSP800-37riskmanagementframework(RMF)[143 ].Thecharacteristicsofinformationsecurity risksarecoveredbyISO/IEC27005[144 ],thoseofcybersecuritybytheNISTcybersecurityframework (CSF)[145 ].AI-specificriskmanagementguidelinesarecoveredbyISO/IEC23984[146 ];thisstandard describeshowtoadaptanISO31000compliantriskmanagementprocesstoanorganisationusing, developing,orimplementingartificialintelligencesystems.IftheorganisationhasanISO/IEC27001 certificateforitsinformationsecuritymanagementsystem(ISMS),thenourrecommendationistoadd AIsystemstotheexistingriskmanagementprocess.

ThesimplifiedmethodologydescribedhereiscompliantwithISO31000andISO/IEC27005workflows, butcanalsobeadaptedtotheNISTRMFandCSFframeworks,ifneeded.Iftheorganisationwishesto employamorecomplexriskmanagementstrategyanytimeinthefuture,itwillbeeasytointegrateexistingAIsystemriskmanagementintothegeneralframework.TheEstonianinformationsecuritystandard (E-ITS)[147 ]isalignedwiththeISO/IEC27001series,meaningthatthoseimplementingE-ITScanalso optforarisk-basedapproach.Thus,themethodologydescribedhereisalsoadaptablebyorganisations implementingE-ITS.

Theriskmanagementprocesscomprisesthreesteps:contextestablishment,riskassessment,andrisk treatment.ThescopeoftheriskmanagementmethodologypresentedinourreportcoversITsystems thatincludeanAIcomponent.

5.1.1AI-specificconsiderationsincontextestablishment

Contextestablishmentinvolvesidentifyinganddocumentingstakeholdersandassetsrelatedtotheprocess.Theorganisationdefinesitsriskreadiness,riskappetiteandriskowners,andidentifiestheinternal, national,andstatutoryrequirementsforstakeholders.Theorganisationdeterminestheconditionsforrisk acceptanceandselectsanappropriateriskmanagementmethodology.

Contextestablishmentforartificialintelligencesystemsrequiresidentifyinganddocumentingallstakeholders.Thisincludestakingintoaccountanypartiesthatmaynotseemtobedirectlyconnectedto serviceprovision(e.g.,personsappearingintrainingdata,ownersofworks,aswellasthird-partyinfrastructureandserviceproviders).Regardlessofwhethertheorganisationcreatesanin-houseAIsystem andusesthissystemwithintheorganisationortheAIsystemisusedasaservice,theanalysismust include:

• datasubjectsordataownerswhosedatahavebeenusedintrainingthemachinelearningmodel;

• thepartythattrainedthemodel;

• theserviceprovider;

• theserviceuser.

Theorganisationmustidentifythestakeholdersandaccountfortheirrightsandinterestinriskassessmentandrisktreatment.Newstakeholdersmayalsonecessitatetheneedtotakeintoaccountnew foundationdocumentsorregulations.Itisimportanttodeterminewhetherthesenewstakeholdersare apartof,orexternalto,theorganisation.Theorganisationmustmapeveryone’sstatutoryrightsand obligationsandwhooperateswhichpartofthesystem.

Theorganisationmustidentifytheoriginofdifferenttypesofdata(models,training,input,andoutput data)andsoftwarecomponents,aswellasthedataflowbetweenthedifferentcomponents.Stakeholder andcomponentmappingisnecessaryforunderstandingthecontextoftheAIsystem.Somerisksmay alsoarisefromtheuseofcertaintypesofdataorsystems.Forvisualisingthemapping,toolsnormally usedforsystemsmodelling(UML,BPMN)canbeused.Specifictoolsexist(e.g.,PE-BPMN[148 ])for

describingthemovementandvisibilityofdataobjectsfromtheperspectivesofdifferentstakeholders. Stakeholders’accesstodatacanbedocumentedusingvisibilitytables.Table 1 isanexampleofavisibility tabledescribingtheaccessofdifferentstakeholderstodifferenttypesofdatainanAIsystem.Inthis example,therearethreestakeholders:theenduser,theserviceprovider(AIclientapplication),andthe AIAPIprovider(whotrainsandsharesthemodel).Allstakeholdersseeenduserinputdataandmodel outputs.TheserviceproviderandAIAPIproviderhaveaccesstotheserviceprovider’sbusinessdata. Themodel,inthiscase,isvisibleonlytotheAIAPIprovider.

Table1.Simplifiedexampleofavisibilitytable

5.1.2AIsystemriskassessment

Riskisoftenexpressedasacombinationofthelikelihoodoftheoccurrenceofathreateventandits potentialdamage.Riskassessmentinvolvestheidentification,analysis,andevaluationofrisks.Risk identification,inturn,involvessearchingforrisks,determiningtherelevanceofrisks,anddescription ofrelevantrisks.Ariskownerisassignedtoeachidentifiedrisk.Riskanalysiscoversdeterminingthe reasonsandsourcesofrisksandevaluatingthepotentialdataandthelikelihoodoftheoccurrenceof therisk.Inthecourseofriskevaluation,theriskleveldeterminedasaresultoftheanalysisiscompared tothecriteriaforacceptablerisksdefinedinthecourseofcontextestablishmentinordertoevaluate whethertherisklevelistolerableandacceptable.

AIriskassessmentisbasedontheestablishedcontext.ForeachcomponentoftheAIsystem,risksare evaluatedinthecontextofthestakeholders.Findingtheserelationshipsiseasybasedonthevisibility tablecreatedduringcontextestablishment.Foreachidentifiedstakeholderandcomponentpair,we analyseandevaluatethreetypesofrisks:risksrelatedtocybersecurity,regulations,andAI-specific threats.CybersecurityrisksareusuallyconnectedtotheadequacyoftheAIsystem’sprocessesorthe confidentiality,integrity,andavailabilityoftheAIsystem’scomponents(software,data,services).Risks relatedtoregulationsareconnectedtothelegalobligationsforstakeholdersoperatingAIsystems(AIspecificregulations)orsystemcomponents(e.g.,regulationsonpersonaldata,copyrighteddata,critical infrastructure).AIrisksareconnectedtothecharacteristicsofAIalgorithms,aswellastheimpactof AIsystemsonthesocietyandethicalaspects.AIsystemriskassessmentiscoveredinmoredetailin Section 5.2

Table 2 providesanexampleofdefiningrisksviasecurityvulnerabilitiesandthreats.Foreachthreat, theorganisationmustevaluatethelikelihoodofthethreatmaterialisingandthepotentialdamage.The likelihoodofanddamagecausedbyasimilareventcanbedifferentfordifferentorganisations.Insome cases,itwillbebeneficialtocomparetherisksofdifferentsolutionsinordertochoosethemostsuitable solutionfortheorganisation.Forinstance,eventhoughacloudserviceprovidermayofferbettersecurity measuresthanasmallorganisationcouldimplementitself,dependenceonacloudservicemaybean availabilityrisk,shouldtheconnectiontothecloudproviderbelost.

5.1.3AIsystemrisktreatment

Differentsolutionsexistforrisktreatment:riskavoidance,riskmitigation,risktransfer,orriskretention. Thesolutiontobeusedwillbechosenbasedonriskanalysisresults.

Table2.Examplesofsecurityvulnerabilitiesandthreats

Data Risktype Security vulnerability Threat

Output AI-specificrisk Biasedordefective model

Trainingdata Regulatoryrisk Lackoflegalbasis forprocessing personaldata

Model Informationsecurity risk Defectiveidentity management

Enduserreceivesanoutput guidingthemtoharm themselvesorothers

Finefordataprotection regulationviolation

AIAPIproviderlosesaccessto theirinfrastructureandis unabletoprovideinference service

Theorganisationisunlikelytobeabletomitigateallrisks.Risksmaybeorderedaccordingtotheir importance.Suitableinformationsecuritymeasures,AI-specificmeasures,orlegalmeasuresarechosen tofacilitatealigningpotentialriskswiththeorganisation’sriskappetite.

Riskscanbeavoidedthroughtheeliminationofthesourceoftherisk,discardingfunctionalities,orreorganisationofthebusinessprocess.Risksaremitigatedthroughtheadoptionofsecuritymeasures. Theemploymentofadditionalsecuritymeasurestomitigateariskisnotalwayspossibleorrational. RiskmitigatingmeasuresaredescribedinSection 6 .Risktransfermeanssharingtheriskwithanother organisation,orthecompensationofdamagearisingfromtherisk,e.g.,byusinginsurance.

Iftherisklevelremainingafterrisktreatmentcorrespondstotheorganisation’sriskappetite,therestof theriskscanbeaccepted.Thismeansthattheriskinquestionwillnolongerbeworkedonandtheriskis retained.Periodicsurveyandrevisionofrisksisrequiredtokeepriskmanagementuptodate,asthreat occurrencesofimpactsevolve.Anotherimportantelementoftheprocessisriskcommunication,the objectiveofwhichistokeeptheemployeesinformedoftheprocessandresultsofinformationsecurity riskmanagement.

5.2Riskassessment

5.2.1Informationsecurityrisks

Digitalrisksarethemostlikelyandhavethebiggestimpact[149 ].Themainthreathereiscybercrime[149 150 ].GenerativeAItechnologiescan,however,supportmoreefficienthandlingofdigital risks[149 ]whendevelopedandimplementedforthispurpose.Researchanddevelopmentrelatedto thecreationofautomatedorsemi-automatedcybersecuritymeasuresisalsorecommendedbytheNIS2 directive[21].

Informationsecurityrisksareidentifiedandanalysedonthebasisofthreats,probabilityofthreatevents, andpotentialdamage.TheEstonianE-ITSinformationsecuritystandarddescribesabaselinesecurity process,oneelementofwhichisthebaselinesecuritycatalogue.Thecatalogueconsistsofprocess modulesandsystemmodules.These,inturn,containalistofthreatsandadescriptionofmeasures.Usingthebaselinesecurityprocesswillsimplifyriskidentificationandisalsocompliant(whenimplemented atahighlevel)withtheISO/IEC27000seriesofstandards.

ThefollowingE-ITSbaselinesecuritymodules[151]arerelevanttotheimplementationanduseofAI systems:processmodulesORP(organisationandpersonnel),CON(conceptsandmethodologies),OPS (operations),DER(detectionandreaction),andsystemmodulesSYS(ITsystems)andAPP(applications). ThemoduleslistedaboveonlyincludethosewhichrequiretakingseparatemeasuresrelatedtotheimplementationoruseofAIsystems.Thelistdoesnotincludemodulesnecessaryforsettinguptherest

oftheorganisation’sinfrastructureorsecuritymanagement.Iftheorganisationdoesnotassessortreat risksinsurroundingsystemsthenevenstronglevelsofprotectionfortheAIsystemwillbemeaningless. TheadditionofAIsystemstotheorganisation’sworkflowwillprobablygiverisetothefollowingprocess threats.Notethatthelistofthreatsisnotlimitedtothoselistedinthestandard.

• ORP1.NoclearrulesfortheuseofAIsystemsexist;theAIsystemisincompatiblewithothertools.

• ORP2.TheemployeesareinsufficientlyfamiliarwithAIsystems;theyarecarelessaboutusingdata inAIsystems;theyareinsufficientlyqualified.

• ORP3.TheemployeeshavenotreceivedsufficienttrainingonthreatsandattacksrelatedtoAI systems.

• ORP5.UseoftheAIsystemisinviolationofthelaworcontractualobligations;unauthorisedpublicationofinformationintheAIsystem;internalinformationisaccidentallyrevealedtoanexternalAI system.

• CON2.InputstoAIsystemsareprovidedinneglectofdataprotectionrequirements;dataprocessing proceduresareinadequateanddonotaccountfortheworkingprinciplesofAIsystems;noresources areallocatedtotheprotectionofpersonaldatainAIsystems;theprivacyofdatasubjectsisnot ensuredfordataprocessedbyAIsystems;theconfidentialityofdataintheAIsystemisnotensured, asdatacanfallinthehandsofunauthorisedpersonsorareaccessibleinthetrainedmodel;the reputationofthedataprocessorisdamaged.

• CON3.ProblemsrelatedtobackingupAIsystemdata(boththeinputsandmodel,aswellas,in somecases,theoutputs).

• CON6.InadequatedeletionanddestructionofAIsystemdata.

• CON8.UnsuitabledevelopmentmethodsusedforAIsystemdevelopment;insufficientqualitymanagement;inadequatedocumentation;insufficientdevelopmentenvironmentsecurity;AIsystemdesignerrors;inadequateAIsystemtestingandacceptanceprocedures;usingproductionenvironment datafortestingtheAIsystem.

• CON10.IncaseoftheAIsystemusedasawebapp:displayingsensitivebackgroundinformation foundintheAIsysteminthewebapp;useofautomatedattacksforattackingtheAIsystemweb app.

• OPS2.2.Allthreatsrelatedtotheuseofcloudservicesapply:inadequateAIcloudserviceuse strategy;dependenceonAIcloudserviceprovider;insufficientrequirementmanagementinusing AIcloudservices;violationofstatutoryrequirements;deficienciesinagreementsignedwiththeAI cloudserviceprovider;insufficientintegrationofAIcloudserviceswithin-houseITsystems;insufficientregulationoftheendofAIcloudserviceuse;deficienciesinemergencyreadinessplan;AI cloudprovidersystemfailure.

• OPS2.3.Allthreatsrelatedtooutsourcingapply:inadequateAIsystemoutsourcingstrategy;insufficientcontroloverbusinesscriticalprocesses;dependenceonAIserviceprovider;insufficientlevel ofinformationsecurityattheAIserviceprovider;insufficientcontrolovertheprovidedAIservice; deficienciesinagreementsregulatingtheAIservice;inadequateaccessrightsmanagement;lackof controloverAIserviceprovider’ssubcontracting;lackofkeyperformanceindicators(KPI);inadequatestipulationsregardingtheendofAIsystemoutsourcing;inadequateemergencymanagement inoutsourcedAIservice.

• OPS3.2.Allserviceproviderinformationsecuritythreatsapply:inadequateinformationsecurity managementbytheAIserviceprovider;inadequateemergencymanagementbytheAIserviceprovider; inadequateserviceagreementswithAIservicereceivers;vulnerabilitiesininterfacingwithAIservice provider’sITsystems;dependenceofAIservicereceiveronserviceprovider;inadequatemanagementofaccessrights;lackofmulti-tenancycapacityattheAIserviceprovider;AIserviceprovider’s dependenceonsubcontractors;inadequateprocedureforendingAIserviceagreement;AIsystem providerITsystemfailure;socialengineering.

• DER2.1.InadequatehandlingofsecurityincidentsrelatedtoAIsystems;destructionofevidencein securityincidenthandling.

• DER3.1.InadequateorunplannedimplementationofsecuritymeasuresinAIsystems;verifier’sinadequatequalification;inadequateauditplanningandcoordination;non-coordinateduseofpersonal data;intentionalhidingofsecurityissues.

ThesystemmoduleSYSdescribesthreatstoITsystems,includingservers(SYS1.1,1.2,1.3,1.9),virtualisationsystems(SYS1.5),containers(SYS1.6),storagesolutions(SYS1.8),clientcomputers,(SYS 2.1,2.2,2.3,2.4),laptopcomputers(SYS3.1),smartphonesandtablets(SYS3.2),printers(SYS4.1), embeddedsystems(SYS4.3),IoTdevices(SYS4.4),andexternalstoragedevices(SYS4.5).TheSYS modulealsodescribesthreatsrelatedtotheuseoftheEstonianX-Roadsecurityserver(SYS.EE1)and eIDcomponents(SYS.EE2).DependingontheAIsystemorservicebeingcreatedorused,therelevant threatscanbefoundintherelevantmodules.

ThesystemmoduleAPPdescribesthreatstoapplications:mobileapplications(APP1.4),webapplications(APP3.1),databasesystems(APP4.3),Kubernetesclusters(APP4.4),softwareingeneral(APP 6),andcustomsoftwaredevelopment(APP7).APP.EE1additionallydescribesthreatstotheEstonian X-Roaddataservices.

TheAIsystemdeveloperorimplementercanusecontextestablishmenttodeterminewhichofthese threatsarerelevanttothem.Identificationofthreatsenablesthedescription,analysis,andevaluationof risks.

5.2.2Legalrisks

NotablelegalrisksrelatedtoAIsystemsincludenon-compliancewithstatutoryrequirementswhichmay leadto:

1. damageclaims;

2. legaldisputes;

3. sanctionsfromcompetentsupervisoryauthorities,includingnoticestoensurecompliance,impositionofpenaltypayments,suspensionorcessationofoperations.

Thelistedrisksmayleadtoadditionaltimespentbyemployeesonworkingonthedamageclaimsor legaldisputes,costsrelatedtoexternallegalservices,financiallossfromcompliancewithdamageclaim orcourtrulingorcompensationoflegalexpenses,lossofincomefromsuspensionofoperations,or reputationaldamage.Thelattermaymaterialiseintheformoflossofclientsandreducedincomeor,in theworstcase,lossoftrustandcessationofoperations.

Finesrelatedtoprocessingpersonaldatacanreachupto20millioneurosor,inthecaseofenterprises, upto4%ofglobalturnoverfrompreviousfinancialyear,whicheverisgreater.AccordingtotheAIAct proposal,certainviolationswouldbeliabletofinesupto35millioneurosor,inthecaseofenterprises,up to7%ofglobalturnoverfrompreviousfinancialyear,whicheverisgreater.Thesubmissionofinaccurate, incomplete,ormisleadingdatawouldbeliabletoafineofeitherupto7.5millioneurosor,inthecaseof enterprises,upto1%ofglobalturnoverfrompreviousfinancialyear,whicheverisgreater.

AccordingtotheAIActproposal,theEuropeanCommissionmayimposefinesongenerativeAIsystem serviceprovidersfornon-complianceofupto35millioneurosor,inthecaseofenterprises,upto3%of globalturnoverfrompreviousfinancialyear,whicheverisgreater.TheAIActproposalalsoforeseesthe rightforcompetentauthoritiestoremoveanAIsystemfromthemarket.

ItisalsocrucialtoensurethattheAIstakeholdershavewrittenagreementsinplacelistingtherights, obligations,andresponsibilitiesoftheparties.Dataprocessingagreementsbetweenthepartiesalso playanimportantroleintheprocessingofpersonaldata.Non-compliancewithanagreementcanalso resultinpenaltyanddamageclaims,aswellaslegaldisputes.

Inthepastfewyearstherehavebeennumerouscourtcasesinvolvingdisputesoverinputs(texts,photos, etc.)usedfortrainingAIsystems(see,e.g.,[152 , 153 154 ]).Thesehavepredominantlyconcerned copyrightviolations.Atthesametime,therehavealsobeendisputesoverresponsibilitiesrelatedtoAI systems.Thus,inMoffattvsAirCanada[155 ],thecourtfoundthatanenterpriseisresponsibleforall informationfoundontheirwebsite,regardlessofwhethertheinformationcomesfromastaticpageor

achatbot.CourtcasestestthelegalboundariesofAIandwillhopefullybringclaritytothisareainthe verynearfuture,helpingtocreatemoreuniformpracticesfortheinterpretationoflegalnorms.

5.2.3AIrisks

DevelopmentsinAI,especiallyinlargelanguageandimagesynthesismodels,havestimulateddiscussionsofrisksofthesetechnologies.Therisksthemselvescanbeconnectedtoboththeharmfulor unintendedoutputsof(universally)powerfulmodels,aswellasthespreadandincreasedadoptionof thesemodelsandthesocietalconsequencesoftheiradoption.

Themostpowerfulimageandlanguagemodelsareexpensivetotrain;widelyusedsmalleropen-source modelsare,however,notfarbehindintheircapabilitiesandcanbeexpectedtogrowevenmorepowerful inthenearfuture.TheadoptionofAImodelsforautomateddecision-makingincriticalareas,suchas medicineorwarfare,hasgivenrisetoadditionalrisksandnumerousethicalconcerns.

Risksrelatedtoartificialsuperintelligencecapableofindependentactionandhumanabilitytocontroland guideitsactionscallforseparateconsideration.Thefurtherdevelopmentofartificialintelligencemay giverisetonew,previouslyunknownrisks,aswellascompoundingexistingones,meaningthattheir mitigationhastobeacontinuous,iterativeprocess.

5.2.3.1ClassificationofrisksbasedontheproposedAIAct

TheAIActisbasedonarisk-basedapproach,distinguishingbetweenfourlevelsofrisk:unacceptable, high,limited,andminimal(seeTable 3 ).RequirementsforAIsystemsarebasedontherisklevel.For generalpurposeAI(GPAI),theregulationalsodefinestwootherriskclasses:non-systemicandsystemic risks.

Table3.AIActrisklevelsforAIsystems

No. Risk Description

1 Unacceptable risk ProhibitedAIsystems

2 Highrisk Regulatedhigh-riskAI systems

3 Limitedrisk Compliancerequirements

ExamplesofAIsystems

AIsystemscausingsignificantriskstohuman healthandsafetyorfundamentalrights(manipulative,exploitativeAIsystems),e.g.,socialscoring systems

E.g.biometricidentificationsystems,emotion identificationsystems,securitycomponentsof criticalinfrastructuresystems,recruitmentsystems,polygraphs,interpretationoflawincourts

AIsystemhasnosignificantimpactonnature oroutcomeofdecisions.AIsystemisdesigned forperforminglimitedproceduraltasks,structured datacreation,groupingincomingdocumentsby subjectordetectionofduplicatesamonglarge numbersofapplications

4 Minimalrisk Noobligations

5.2.3.2Algorithmicrisks

AIsystemsthatcanbeusedwithoutlimitations, e.g.,spamfilters,AIbasedvideooraudioenhancementsystems

ThefollowingsectionfocusesonrisksrelatedtospecificAIsystemsandtheimmediateconsequences oftheiruse.Insomecases,thematerialisationoftheserisksisconnectedtoattacksagainstAIsystems,

whicharediscussedinSection 5.3

Limitedgeneralisationability. Theutilisationofautomatedartificialintelligencesystemsinhighlycritical fields(e.g.,medicine,warfare,orself-drivingvehicles)comeswiththeriskofthemodelnotreturning aviableoutputforaninputdeviatingtoofarfromthetrainingdata.Largelanguagemodelshavebeen observedtoproduce’hallucinations’wherethemodelreturnsasuperficiallyconvincingbutfactuallyunfoundedresult[156 ].ThisriskiscompoundedbyAIsystems’lackoftransparencywhichmayleadtothe dangerofblindlytrustingaharmfulormisleadingoutput.

ExcessivedependenceonAIandlossofhumansupervision. Theincreasingadoptionofartificialintelligence,includingincriticalsystems,threatenstoleavehumansinthepassengerseat.Themorecomplex AImodelsandsystemsbecome,themoredifficulttheyareforahumantograspwhichmaydecrease humanabilitytomonitorthesesystems.Thereductionofhumansupervision,inturn,reducesourability tointerfereintheoperationofAIsystemsandpreventundesirableoutcomes.Atthesametime,the benefitsprovidedbythesesystemsmaybelargeenoughthatthepriceoflossofsupervisionandcontrollabilitywillbedeemedacceptable.Giventhatcomplexsystemstendtobemorefragilethansimple ones,excessivedependencewithoutunderstandingcanbeabigrisk.

Biasedanddangerousresponses. Evensystemstunedtobesaferusingreinforcementlearningcanbe madetogeneratediscriminatory,abusive,orotherwisepotentiallyharmfulcontentusingpromptinjection techniques[157 ].Inadditiontotheriskofpromptinjection,themodel’ssecuritymechanismscanbe disabledquitecheaplyoreveninadvertentlyviafine-tuning[158 ]whilesomemodels(especiallyopensourceones)donotevencontainanymeaningfulprotectionsofthiskind.Sincethemodelsareprimarily trainedonhuman-madedatasetscontainingbiasescharacteristictohumans,themodelstrainedonthese datasetsarealsoinherentlybiased.Atthesametime,correctionsforalgorithmicdiscriminationrequire careintheselectionoftargetindicators,forthelattermayalsobebiased.Excessiveuseofcorrective measurescanhaveanoutsizenegativeimpactonthecapabilitiesofthemodelorapplication,whichis exactlywhathappenedto,e.g.,theGoogleGeminiAIimagesynthesistool 1

5.2.3.3Societalrisks.

AItechnologyisinrapiddevelopment.ThewidespreadadoptionofAIpromisestobringenormouseconomicandsocialbenefits,yetitalsothreatenstoleadtoanupheavalatleastassweepingastheone causedbythewidespreadadoptionoftheInternet.The(human)societalrisksofAIareconnectedto boththeexpansionofhumanagencymediatedbyAIandtheunpredictabilityoftheaccompanyingsocial changesasthepossibilityoftheemergenceofartificialsuperintelligence(ASI).

Autonomousartificialsuperintelligence. Artificialsuperintelligence,familiartomanyfromsciencefiction,hasrecentlyfounditselfinthelimelightofdiscussionsovertheexistentialrisksofAI.Increasesin AImodelsizeandcomputingpower,aswellastheappearanceofemergentproperties,giverisetocertainexpectationsforevenmorepowerfulandmulti-modalmodelsorapplicationspossessingasuperior generalisationability.Shouldsuchamodelpossessesasufficientlevelofautonomy,accesstocritical (e.g.,financial)systems,andtheabilitytoremainundetectedoravoidpotentialcountermeasures,the riskwouldbeevengreater[5 ].

AsufficientlypowerfulandautonomousAIagentcan(whetherwithhumanassistanceorwithoutit) becomeathreatjustbygainingaccesstotheInternetandtheabilitytomakeGETqueries,usingsecurity holes,suchasLog4Shell[159 ].Additionalriskfactorsincludetheagent’sabilityforself-enhancement andsituationalawareness.Researchershavenotreachedaconsensusoverthepotentialtimelineofthe emergenceofsuchabilitiesbutthisinitselfdoesnotruleouteitherthepossibilityofrelevantrisksor evenapotentialexistentialrisktothehumanity.

UncontrolledspreadofAImodels. ThefreedistributionandwidespreadadoptionofAIfoundationmodels,whichbynowseemsavoidable,willmagnifyAI-relatedrisks.Themoreusersanddeveloperscan accessthemodel,thehigherthenumberofpotentialexploitersandthegreaterthescopeofrequired regulation[160 ].Thisriskisevenbiggerinthecaseofthespreadofpre-trainedmodelswhichhavenot

1 Google’s‘Woke’ImageGeneratorShowstheLimitationsofAI https://www.wired.com/story/ google-gemini-woke-ai-image-generation/ VisitedFebruary23rd,2024

beenfine-tunedtoprovidesafeanswers.

Biologicalandchemicalweapons. Inthecontextofthespreadofpowerfulfoundationmodels,researchershavehighlightedtheriskofterroristgroupsgainingaccesstoatoolthatcanhelpthemacquire chemicalorbiologicalweaponsmoreeasily[161].AutonomousAIagentscapableofperformingthenecessaryresearchautonomouslydeservespecialattentionhere[162 ].Someresearchershavepointedout that,inthecontextoftheevaluationofAI-specificrisksofthespreadofbiologicalandchemicalweapons, AIshouldbeequatedtoaccesstotheInternetwhichthemaliciouspartiescertainlyhave,andthereal questionis,which 2 bottlenecks 3 intheirmanufacturingprocessareeliminatedbyAI[163 ].

Theavailabilityofinformationisgenerallynotoneofsuchbottlenecks–inanycase,LLMscanbeconsideredacompressed(withlosses)versionofinformationalreadyfoundontheInternet–,unlike,e.g.,navigatingthemountainsofinformationorthemanufacturingprocess.TheabilityofmodernAItorespond toquestionsbasedonitsextensivegeneralknowledgeandnavigateinandsummarisetextualdatacan acceleratethisprocess.Eventhoughthescepticshavepointedoutthatthecapacitytoproducechemicalorbiologicalweaponsiscurrentlyratherrare,theexpectedimprovementsintheperformanceofAI modelsandapplicationsisthreateningtomagnifysuchrisks.

AIininformationwarfare. High-qualitytext,image,speech,andvideosynthesismodelsenablecarrying outextensiveautomateddisinformationcampaignswhich,inturn,isportendingdistrustofwebcontent ingeneral.Thisisaproblemtostatesandauthorities[164 ]whichwillnowhavetoseekforwaystoaffirm theauthenticityoftheirmessages.AIprovidesallpartiesininformationwarswithpowerfulweapons; meanwhile,defensivemeasureshavenotdevelopingatasimilarrate.

Artificialintelligenceandfraud. Thespreadofgenerativeartificialintelligencehasalsoprovidednew toolstoscammers[165 ].Imageandtextsynthesismodelsallowgeneratingcrediblefakeidentities, includingpassportsandotheridentitydocuments.Speechsynthesisenablestheimitationofanother person’svoicewhichfacilitatesidentitytheft.Languagemodelshaveautomatedthecreationofever morebelievable,customisedphishinge-mails.Videoscreatedusingdeepfaketechnologiescancause significantharmtothepersonstheydepict.

5.2.3.4Ethicaldilemmas.

TheadoptionofAIgivesrisetonumerousethicalissues.CanAIbethefinalarbiterinmattersoflifeand death?IftheeconomictransformationcausedbyAIistoosudden,shoulditbesloweddown?CananAI systemormodelbeconsideredtheauthorofawork?WhoisresponsibletoproblemswiththeAIsystem orthedamageithascaused?

Lossofjobs. Largelanguageandimagesynthesismodelsthreatentoreplacehumansinnumerousfields. Thegeneralabilitiesofmodernlanguagemodelsarenoworsethanhumans’intasksdemandingcommunicatingwiththeclientinnaturallanguagefollowingpredeterminedalgorithmicrulesorthecomposition andsummarisingofmarketingandotherspecialtytextsbasedonexistingsourcesofinformation.Speech synthesisisendangeringcallcentres,imagesynthesis–artdirectorsandartists,textsynthesis–marketingcopywritersandtechnicalsupportspecialists.ThemorepowerfulAIsolutionsbecome,thegreater theirimpactonthelabourmarket;widespreadlossofjobswillcomewitheconomicandsocialrisks.This processcanbeconsideredpartofabroadertrendofautomation,hithertomainlyconnectedtotheevolutionofroboticswheretheethicaldilemmaconcernsthetrade-offsbetweenproductivityandsecurity oftheemploymentrelationship.

Ethicaldilemmasinautonomoussystems. Thesedays,AIisusedinsystemsmakingautonomousdecisionsthatcanhaveasignificantimpactonhumanautonomy.Theadoptionofsuchsystemsrequires considerationoftheethicalandmoralaspectsofdecisionsmadebytheAI.Ifafast-movingself-driving carfindsitselfabouttorunoverababyandagrandmother,theAIsystemisforcedtomakeamoralcallon whoitshouldputatrisk:thebaby,thegrandmother,orthedriver?Theproblemofdecidingoverhuman

2 Anthropic:FrontierThreatsRedTeamingforAISafety: https://www.anthropic.com/index/ frontier-threats-red-teaming-for-ai-safety VisitedNovember9th,2023

3 PropagandaorScience:OpenSourceAIandBioterrorismRisk: PropagandaorScience: OpenSourceAIandBioterrorismRisk VisitedNovember9th,2023

livesisencounteredinallsystemswherehumanshavenowaytocontrolandpromptlyinterfereinthe decision-makingprocess.Theethicalriskisespeciallygreatinthecaseoffullyautonomousweapons systems,suchasturretsordroneswarms,whichhavetomakefriend-or-foedecisionsinafractionofa second[166 ].

Addictivechatbots. Modernimageandtextsynthesisenablesthecreationoftrulyengrossingchatbots andcompanions.Dependingonthebusinessmodel,theprovidersofsuchservicescouldhaveafinancial incentivetomaketheserviceasaddictiveaspossiblebycustomisingtheAIcompaniontourgetheuserto spendmoretimeinitscompany.Thiscanbecompoundedbylanguagemodels’tendencytosycophancy acquiredthroughRLHF(reinforcementlearningfromhumanfeedback)[167 ].Constantpositivefeedback providedbyaddictivechatbotscreatesanechochambereffectandisespeciallyharmfultomentallyand sociallyvulnerablepeople.

AIinthelegalsystem. AItechnologiesareincreasinglyeitherdirectlyorindirectlyrelevanttotheadministrationofjustice.AIapplicationscansimplifytheworkofjudgesandlawyersbyprocessinglarge amountsofdata.Theadoptionofsuchtechnologiesrequiresconsideringthetransparencyofthedecisionsandrecommendationsprovided,aswellasriskstopersonalprivacy(e.g.,inthecaseofautomated surveillanceorinformationgathering).

Artificialintelligenceandintellectualproperty. Today’sgenerativeAIiscapableofsynthesiSingtext, music,images,video,andothercontent.Thesecapabilitiesarearealchallengetoartists–notjustby threateningtoreplacethembutalsofromtheperspectiveofintellectualproperty.Ifanimagesynthesis modeliscapableofsynthesisingimagesinthestyleofaspecificartist,doesthisqualifyasacopyright violation?Ifnot,thenhowsimilartotheartist’sworksdoesthesynthesisedimagehavetobetoqualify asone?And,lastbutnotleast,cangenerativeAIbeconsideredtheauthorofanythingatall?Fromthe artist’spointofview,theseareallunansweredquestions.Afurthercriticalissuepertainsto,e.g.,image banksandwebcrawlerscollectingtrainingdataforthemodel.Howtoprovethatamodelhasbeen trainedoncopyrightedorotherwiselicense-protecteddata?

Artificialintelligenceandprivacy. TheevolutionofAImagnifiesprivacyrisksinseveralways.ThecapabilityofidentifyingconnectionsbetweenpiecesofinformationfoundontheInternetcanhelpdeanonymise userswishingtoremainanonymous.AForbesreporterwasthusabletoidentifythepersonbehindthe X(formerTwitter)userBeffJezos,usingAItocompareaudiorecordingsofBeffJezosandtalksgiven bytheformerquantumcomputingengineerGuillaumeVerdon[168 ]andconcludingthattheyare,with averyhighlikelihood,thesameperson.Othermethodsarealsoavailable–theanalysisofsocialmediausagetimes,relationstootheraccounts,andlanguageusecanallbeemployedtoinfertheperson behindanaccount.

Anotherrisktoprivacyisconnectedtotrainingdataleaks.Languagemodelsareknowntohaveatendencytoreproducetheirtrainingdatasetwordforword,andcertainpromptingtechniquescanbeexploitedtofurtheraggravatethistendency[169 ].Trainingdatasetscancontainsensitiveorcopyrighted information.

MissingoutonthebenefitsofAIduetooverregulation. ThedebateoverthedangersofAIandthescope oftherelatedregulationproposalsmaymeanthatsomeofthebenefitsoftheadoptionofAImayfailto materialiseduetotheimplementationofsomeoftheproposals.Insteadofjustfocusingonthepossibility ofthreats,thedebatesshould,therefore,begroundedincomprehensiveanalysisofsuchrisks.

5.3Attacksagainstartificialintelligencesystems

AIsystemsmakedecisionsbasedondata.Ingeneral,thedecision-makingtakesplacewithouthuman surveillancewhilebeingpotentiallyamatteroflifeanddeath(e.g.inmedicineorself-drivingcars);the datausedmayalsobesensitiveinnature.AdversariescouldexploitthecharacteristicsofAIsystemsfor influencingtheirbehaviourorextractingsensitiveinformation.Thismeansthat,inadditiontoeveryday ITsystemsecuritymeasures,onealsoneedstoconsiderAIsystem-specificmeasures.Tothisend, wewillnextreviewattacksspecificallycharacteristictoAIsystems.Ourreviewofattacksisbasedon

theGermanFederalOfficeforInformationSecurity’s’AIsecurityconcernsinanutshell’4 andtheOWASP Foundation’s’OWASPTop10forLLMApplications’5 reports.WewillnotfocushereonattacksagainstAI systemsalreadycoveredinthesectionsonalgorithmicandethicalrisksofAI.

5.3.1Evasionattacks

EvasionattacksareattackswheretheadversaryattemptstomaketheAImodelreturnanoutputnot intendedbythesystem’sdeployer,oftenusingaseeminglyinnocentinputcontainingahiddenattack. Theirobjectiveinthismaybeeitherobtainingaspecificoutputorsimplyreducingoutputquality(fora specificallychoseninput).

Adversarialexamples areinputsconcealinganevasionattack.Forexample,iftheadversaryhasaccess totheentireimagesynthesismodel,theycantakeanynormalinputasabasisandnudgethisinput alongthegradienttowardsthesoughtoutputclass,asseeninFigure 18 .Tiniestnudgessuchasthis willimpactthemodel’soutputwhileoftenremainingcompletelyinvisibletotheeye[170 171].

Figure18.Distortionofanimageusingcarefullychosennoisemakesthemodelpredictthewrong outputclass[170 ].

Promptinjection isaformofattackagainstlargelanguagemodelsandAIapplicationsbuiltuponsuch modelsusingthecharacteristicsofthepromptandthecontextwindowtoobtainanoutputnotintended bythemodel’sdeployer[172 ].Asthelanguagemodelisunabletodifferentiatethedeployer-createdprepromptinthecontextwindowfromauserprompt,theusercanexploitpromptinjectiontomakethemodel ignoreinstructionspresentedinthepre-promptorrevealtheseinstructionstotheuser.Instructions containedinthepromptinjectioncanruncodeorquerywebpagesviainsecurelyinterfacedplugins[173 ]. Apromptinjectionmaymeanwhilenotoriginatefromamalicioususerbutsomeoneloadingtheprompt toawebresourcethatcanbequeriedbyanInternet-connectedLLMapplication[174 ].Anattacklikethis canbeclassifiedasanindirectpromptinjection.Themodelwillthusendupwithanewsetofinstructions; Figure 19 showsaschematicdepictionofthistypeofattack.Promptinjectionattacksaresimilartocode injectioncommoninwebapplicationswhereinsecureinputhandlingcanresultintheapplicationrunning codefoundintheinput.

Insecureoutputhandling meanslackofcontroloverthequeriesandcommandcomposedbythemodel itself.ThiscanleadtoanadversaryusingpromptinjectiontogainaccesstotheAIapplication’sback-end systems,shouldthemodelbeinterfacedtoany.Forexample,theuserpromptcouldcontaininstructions toruncode,usingan exec or eval call.Alternatively,apluginorthird-partyserviceinterfacedtothe modelcouldreturnaninsecureoutputtothemodelwhichwill,inturn,returnthisoutputtotheuser.The

4 https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/Practical_Al-Security_Guide_ 2023.html LastvisitedDecember8th,2023

5 OWASPTop10forLargeLanguageModelApplications. https://owasp.org/ www-project-top-10-for-large-language-model-applications/)LastvisitedFebruary26th,2024

Figure19.Indirectpromptinjection(adaptedfrom[174 ]).

outputcancontain,e.g.,codewritteninaprogramminglanguage(suchasJavaScript)thatwillthenbe runontheuser’swebbrowser.

Foundationmodelvulnerabilitytransfer isariskinherenttotransferlearning[171]duetothedatasets usedforfine-tuningamodelbeingmuchsmallercomparedtonormaltraining.Anadversarycanconsequentlyuseanopen-sourcemodel’sknownvulnerabilitiestodevelopmaliciousinputsagainstanother modelfine-tunedonthismodelwithoutdirectaccesstothefine-tunedmodel.

5.3.2Dataextractionattacks

Dataextractionattacksincludeattackswheretheadversarytriestoextractinformationtheyshouldnot haveaccesstofromthemodelanditsoutputs.Theadversarycouldthusbeabletomakeinferences aboutaperson’sinclusioninthetrainingdataset,obtainsensitivedetailsaboutthem,stealthemodel,or reconstructthetrainingdataset.

Modeltheft isaformofattackwheretheobjectiveistoreconstructa’shadowmodel’trainedonthe adversary’sinputsandtheattackedmodel’soutputs[175 ].Apowerfulandaccurateshadowmodelcan openthewayforotherattacks,suchasevasionattacks. Membershipinferenceattack isaformofattack wheretheadversarytriestodeterminewhetheracertainrecordwasincludedinthetrainingdataset[176 , 177 ].Giventhatmereinformationaboutarecord’sinclusioninthetrainingdatasetcouldbesensitivein nature(e.g.,inthecaseofmodelstrainedonmedicaldata),suchattacksposeasignificantprivacyrisk. Anattackofthistyperequiresaccesstothemodel’soutputandcanadditionallyexploitinformationabout thestatisticalrelationshipsrepresentedinthetrainingdatasettodeterminetheprobabilityofaspecific outputwithandwithoutaspecificrecordinthetrainingdataset.

Attributeinferenceattack isaformofattackwheretheadversarytriestoinferadditionalsensitiveattributesofarecordthattheyknowtobeincludedinthemodel’strainingdataset.Itworksinasimilar fashiontomembershipinference–basedonknowledgeaboutstatisticalrelationshipsbetweenknown sensitiveattributesinthetrainingdataset,theadversaryusesmodeloutputstoassesstheprobability

oftheconcurrenceoftheseattributes.

Modelinversion ortrainingdatasetreconstructionisaformofattackwheretheadversary’sobjectiveis toinfertheproperties(inputsortheirelements)ofthemodel’soutputclasses[178 ].Theadversaryhas accesstothemodel,whichtheyuse(e.g.,bytrainingagenerativemodelagainstthismodel[179 ])to reconstructthetrainingdatasetrecordscorrespondingtothetargetclasses,whichcanpotentiallyreveal sensitiveinformation.

5.3.3Poisoningandbackdoorattacks

Datapoisoning meansinfluencingthetrainingdatasetwiththegoalofeitherinfluencingthemodel’s performanceinacertaindirectionorsimplyreducingitsperformance.Theobjectiveofdatapoisoning istochangeoutputclassesintrainingdatasetrecordswiththegoalofcausingmaximumdamage[180 , 181, 171, 182 , 183 ].Amodeltrainedonpoisoneddataeitherhaspoorperformanceingeneralorisunable tohandlecertainspecificinputcategories.

Abackdoorattack isaspecialcaseofdatapoisoningwherethetrainingdatasetispoisonedwithaset ofexampleswheretheclasstokenwillbeincorrectonlyinthecaseoftheexistenceofacertaintrigger intheexample[184 , 185 , 183 ].Thiswillresulteitherinreducedmodelperformanceorthemodelwill onlypredictthewrongclassiftheexampleprovidedtothemodelcontainsthechosentrigger.Amodel poisonedinthisfashionwilloperateproperlyinothersituations,makingitmoredifficulttodetectthe attackcomparedtonormaldatapoisoning.Abackdooredmodelwillbevulnerabletoevasionattacks. Figure 20 depictsanexampleofabackdoorattack.

Figure20.Backdoorattackwhereamodeltrainedonpoisoneddatawillincorrectlyclassifyastopsign inthecaseoftheexistenceofacertainpatternintheinput[185 ].

5.3.4Denialofservice

Adenialofserviceattackisatypeofattackwheretheoperationofacomputersystemisparalysed byqueriesthatareeitheroverwhelmingintheirnumberorinitiatecompute-intensiveprocedures.Large languagemodelsareautoregressive,meaningthattheentiretyoftheoutputpreviouslyassembledbythe modelwillbetakenintoaccounttoconstructthenextoutputtoken.Responsetime(andcomputational intensity)isthereforecorrelatedtooutputlength[186 ].Thispropertycanbeexploitedbyanadversary byqueryingthemodelwithinputsforcingittoreturnlongoutputsequences[187 ].Amodel’soperation canalsobeparalysedbysubmittinginputsbarelyfittingthecontextwindow,thusincreasingthemodel’s memoryusage.

6Controls

6.1Informationsecuritycontrols

Justlikeinthedescriptionsofthreatsunderlyinginformationsecurityrisks,ourdiscussionofcontrols isbasedontheE-ITSbaselinesecuritycatalogue[151].Allcontrolsaresystematicallydescribedinthe catalogue,easilyaccessible,anddownloadableinXLSorPDFformat.Ourdiscussionhereis,therefore, limitedtolistingtherelevantcontrols.

6.1.1Processcontrols

Informationsecurityorganisation(ORP1)controls:

• TasksandobligationsrelatedtoAIsystemsaredefined,communicatedtoallemployees,andreviewedonaregularbasis(ORP.1.M1).

• TheAIsystemorsignificantcomponentsofthesystemareincludedinthelistoftoolsandequipment, theircompatibilityandsecurityistakenintoaccountinprocurement(ORP.1.M8).

• SecureuseguidelinesareestablishedforAIsystems,keptuptodate,andpresentedtotheemployees(ORP.1.M16).

Personnelcontrols(ORP2):

• Employeesreceiveregularinstructionandtrainingrelatedtotheirareaofwork,employeesaremotivatedtoconstantlydeveloptheirskills,theeducation,qualifications,andskillsrequiredfromnew employeesareclearlydescribed,accuracyofqualificationsrequiredforspecificpositionsarereviewedonaregularbasis(ORP.2.M15).

• Personsparticipatinginpersonnelselectionverifythecandidate’strustworthiness(ORP.2.M7).

Informationsecurityawarenessraisingandinstruction(ORP3)controls:

• ManagementreceivesregularupdatesonrisksconnectedtoAIsystems,potentialresultinglosses andimpactonbusinessprocesses,themanagementisawareofstatutoryrequirementsforAIsystems,leadingemployeessetanexampleintheresponsibleuseofartificialintelligencesystems (ORP.3.M1).

• EmployeesareinstructedinthesafeuseofAIsystems(ORP.3.M3).

• AnawarenessandtrainingplanontherisksandlegalaspectsofAIsystemsiscreated(ORP.3.M4).

• AnawarenessandtrainingprogramontherisksandlegalaspectsofAIsystemsisdesignedandimplemented,allemployeesreceivetrainingrelevanttotheirtasksandareasofresponsibility(ORP.3.M6).

• Trainingresultsaremeasuredandassessed(ORP.3.M8).

• Peopleandorganisationsatriskareprovidedspecialtrainingonconfidentiality,integrity,andavailability(ORP.3.M9).

Compliancemanagement(ORP5)controls:

• Legalframeworkisdefined,aprocessisdevelopedfordeterminingalllegalacts,agreements,and otherrequirementsimpactingsecuritymanagement,thelegalframeworkistakenintoaccountin designingthebusinessprocesses,applications,andarchitectureofAIsystemsandintheprocurementofAIsystemsortheirelements.SpecialregulatoryrequirementsforAIsystemsarecarefully consideredespeciallyinthefollowingareas:personaldata,businesssecretandintellectualproperty protection(ORP.5.M1).

• Thelegalframeworkistakenintoaccountalreadyintheplanninganddesignstages(ORP.5.M2).

• Compliancemanagementisplannedandimplemented(ORP.5.M4).

• Compliancemanagementisreviewedonaregularbasis(ORP.5.M8).

Personaldataprotection(CON2)controls:

• Organisationhasanalysedthelocations,types,andprotectionrequirementsofpersonaldataprocessedbytheAIsystem(CON.2.M1).

• ProcessingofpersonaldataintheAIsystemismappedovertheentirelifecycleofthedata(CON.2.M3)

• DesignoradditionofAIsystemstotheprocessensuresthatpersonaldataareprocessedinalegal andtargetedmannerandtheprincipleofdataminimisationisfollowed(CON.2.M6).

• Datasubjects’rightsareprotected(CON.2.M8).

• IntheprocessingofpersonaldatabytheAIsystem,theorganisationminimisestheuseofdata directlyorindirectlytraceabletoaperson;wherepossible,dataarepseudonymisedoranonymised (CON.2.M9).

• AIsystem-specificdataprotectionimpactassessmentsarecarriedout(CON.2.M13).

• Theprivacy-by-designandprivacy-by-defaultprinciplesarefollowedinthedesignandadditionof AIsystemstoprocesses,e.g.,employingprivacyenhancingtechnologies(CON.2.M22).

• CookiesandmonitoringtoolsusedinAIwebapplicationsareincompliancewiththeGDPRandother relevantlegalacts(CON.2.M24).

Databackupconcept(CON3)controls:

• DatabackuprulesincludethedataoftheAIsystem(CON.3.M2).

• DatabackupplansaccountforthespecificsofAIsystems(whetherthebackupincludestraining data,model,inputs,outputs)(CON.3.M4).

• AdatabackupconceptisdrawnupforAIsystems(CON.3.M6)

Dataerasureanddestruction(CON6)controls.

• DataerasureanddestructionproceduresaccountforthespecificsoftheAIsystem(CON.6.M1).

• ProceduresforthesecureerasureofdataaccountforthespecificsoftheAIsystems(CON.6.M12).

Softwaredevelopment(CON8)controls:

• AsuitablesoftwaredevelopmentmethodologyandaprocessmodelcorrespondingtothemethodologyarechosenforthedevelopmentoftheAIsystemandtheyarefollowed.Thesoftwaredevelopmentprocessmodelincludesinformationsecurityrequirements.Informationsecurityrequirements aretakenintoaccountinthedevelopmentprocess(CON.8.M2).

• PrinciplesofsecuresystemdesignaretakenintoaccountinthedevelopmentoftheAIsystem,they aredocumented,andcompliancewiththemismonitored(CON.8.M5).

• SoftwarelibrariesoriginatingfromtrustworthysourcesareusedinthedevelopmentoftheAIsystem (CON.8.M6).

• AIsystemsaretestedinthecourseofdevelopment,andcodereviewsarecarriedout.Testing takesplaceindevelopmentandtestingenvironmentsisolatedfromtheoperationalenvironment (CON.8.M7).

• Security-criticalpatchesandupdatesaredevelopedandinstalledpromptly(CON.8.M8).

• Suitableversionmanagementtoolsareusedtoensurethesafetyofthesourcecodeandcodechange managementoftheAIsystem(CON.8.M10).

• Externalsoftwarecomponentsandlibraries,thatarenotguaranteedtobecompletelysecure,pass securitytestingbeforeadoption(CON.8.M20).

• DetailedandcomprehensivedocumentationexistsfortheAIsystem(CON.8.M12).

• RiskassessmentiscarriedoutinthefirststageofthedevelopmentoftheAIsystem(CON.8.M21).

• ArchitectureoftheAIsystemisselectedbasedonrequirementsandriskassessmentresults(CON.8.M22). Webapplicationdevelopment(CON10)controls:

• SecureauthenticationisensuredintheAIwebapplication(CON.10.M1).

• Users’accessrightsarelimitedtotheirneeds(CON.10.M2).

• AIwebapplicationonlyoutputsintendedandpermitteddataandcontenttotheusers(CON.10.M4).

• AIwebapplicationisprotectedfromunauthorisedautomatedaccess(CON.10.M6).

• Protectionofconfidentialdataisensured(CON.10.M7).

• InputdatasubmittedtotheAIwebapplicationaretreatedaspotentiallyharmfuldata;theyarefiltered andvalidatedbeforefurtherprocessing(CON.10.M8).

• Disclosureofsensitivebackgroundinformationinoutputsanderrormessagesislimited(CON.10.M10).

• AIwebapplicationisdevelopedonthebasisofasecuresoftwarearchitecture;allcomponentsand dependenciesaredocumented(CON.10.M11).

• ResolutionoffailuresencounteredintheoperationoftheAIwebapplicationmaintainstheintegrity ofthewebapplication;allerrormessagesarelogged(CON.10.M13).

• Denialofserviceiscounteractedtoensureavailability(CON.10.M17).

• Sensitivedataareprotectedusingcryptographicmechanismstoensuretheirconfidentialityand integrity(CON.10.M18).

Cloudserviceusage(OPS2.2)controls.

• Acloudservicestrategyisestablished,coveringtheobjectives,benefits,andrisksofcloudservices, aswellastherelevantlegal,organisational,financial,andtechnicalframeworks.Feasibility,costbenefit,andsecurityanalysesarecarriedout.Astep-by-stepserviceadoptionplanisdrawnup (OPS.2.2.M1).

• Thisstrategyisusedfordrawingupacloudservicesecuritypolicy.Nationalspecificsandstatutory requirementsaretakenintoaccountforinternationalserviceproviders(OPS.2.2.M2).

• AIsystemsusingacloudserviceareincludedinthelistofcloudservices(OPS.2.2.M3).

• Responsibilitiesrelatedtotheuseofthecloudserviceandthetasksoftheservicepartiesaredefined anddocumented(OPS.2.2.M4).

• Cloudservicesecuritypolicyisusedasthebasisforacloudservicesecurityprogrammefocusingon cloud-specificrisks(e.g.,dependenceoncloudserviceprovider,multi-tenancy,fixeddataformats, accesstodata).Thecloudservicesecurityprogrammeiscompliantwiththeagreementssigned withthecloudserviceproviderandnetworkprovider,aswellasthetermsofservice(OPS.2.2.M7).

• Cloudserviceproviderischosenbasedonarequirementsspecification(OPS.2.2.M8).

• Acloudserviceagreementconformingtotheclient’srequirementsissigned(OPS.2.2.M9).

• Migrationtothecloudserviceiscarriedoutsecurely(OPS.2.2.M10).

• Anemergencyreadinessprogrammeisdevelopedforcloudservices(OPS.2.2.M11).

• Correspondenceofthecloudservicetotheconditionsandsecurityrequirementssetoutintheserviceagreement,aswellascompliancewiththecloudservicesecurityprogramme,ismonitoredon aregularbasis(OPS.2.2.M12).

• Cloudserviceprovidercertifiesthecomplianceofinformationsecuritywithstatutoryrequirements and/orinternationallyacceptedcriteria(OPS.2.2.M13).

• Cloudserviceagreementsareterminatedonanordinarybasis(OPS.2.2.M14)

• Specificcriteriaareestablishedforswitchingcloudserviceprovidersortransitiontoaninternalservicewhichincludeportabilityrequirementsandservicemigrationtestingobligations(OPS.2.2.M15).

• Detaileddatabackuprequirementsarepresentedtothecloudserviceprovider(OPS.2.2.M16).

• Necessityofdataencryptionandencryptionmechanismsareagreedon(OPS.2.2.M17).

Outsourcing(OPS2.3)controls:

• Securityrequirementsareestablishedforalloutsourcedservices,definedwithconsiderationtothe typesofdatabeingprocessedandthenecessarylevelofsecurityfordataexchangeprocedures andinterfaces.Dependenciesbetweenbusinessprocesses,aswellastheinputsandoutputsofthe processesarealsotakenintoaccount(OPS.2.3.M1).

• Feasibilityofoutsourcingtheserviceisdecidedonthebasisofresultingrisks.Continuedcorrespondenceoftheservicetothepermittedriskprofileisverifiedonaregularbasis(OPS.2.3.M2).

• Arequirementsprofileincludingsecurityrequirementsisdrawnupforthechoiceofserviceprovider (OPS.2.3.M3).

• Aserviceagreementcorrespondingtotheclient’srequirementsissigned(OPS.2.3.M4).

• Serviceprovidermustensuresecureisolationofclientdatawhenofferingsimilarservicestodifferent clients(OPS.2.3.M5).

• Outsourcedservicesecurityprinciplesaredocumentedandfollowed(OPS.2.3.M6).

• Outsideserviceagreementsareterminatedaspercontract(OPS.2.3.M7).

• OutsourcingstrategyincludesconditionsforAIsystemsandservices(OPS.2.3.M8).

• ProcurementpolicyisupdatedwithinformationofAIsystemsandservicesbasedontheoutsourcing strategy(OPS.2.3.M9).

• AIsystemsandservicesareincludedintheoutsourcedservicesregistry(OPS.2.3.M11).

• Theserviceagreementdefineswhichobjectsandnetworkservicestheserviceprovidercanaccessontheclient’snetwork.Keyperformanceindicators(KPIs)oftheservicearedocumentedasa partoftheserviceagreement.Serviceagreementincludesdifferentconditionsforterminatingthe outsourcedserviceagreementandrelevantproceduresforreturningtheclient’sdataandproperty. Serviceagreementincludesguidanceontheobligationsandactionsofthepartiesinanemergency situation(OPS.2.3.M14).

• Alternativeserviceproviderswithasuitablecompanyprofileandadequatelevelofinformationsecurityaremapped.Actionplanforservicemigrationisdrawnup(OPS.2.3.M19).

• Anemergencyreadinessplanisdevelopedfortheoutsourcedservice(OPS.2.3.M20).

• SensitivedataexchangedbetweentheserviceproviderandtheclientintheAIsystemaredelivered inanencryptedform(OPS.2.3.M23).

Serviceproviderinformationsecurity(OPS3.2)controls:

• AIserviceproviderhasaccountedforservicereceivers’informationsecurityrequirementsinthe designoftheirservices.Theserviceconformstoregulatory(includingdataprotection)requirements (OPS.3.2.M1).

• AIserviceproviderhasdevelopedstandardtermsandconditionsforserviceagreements(OPS.3.2.M2).

• AIserviceprovideraccountsforsecurityrequirementsintheuseofsubcontractors(OPS.3.2.M3).

• AIserviceproviderensuressufficientlysecureisolationofdifferentclients’dataandoperationalenvironmentsintheirsystems(OPS.3.2.M4).

• AIserviceproviderhasdrawnupasecurityconceptcoveringallservicesprovidedtoclients(OPS.3.2.M5).

• Serviceagreementincludesconditionsforbothordinaryandextraordinaryterminationoftheagreement(OPS.3.2.M6).

• AIserviceproviderusingtheservicesofsubcontractorsdrawsupalistofalternativesubcontractors (OPS.3.2.M7).

• AIserviceproviderhasdocumentedtheprinciplesforthecreation,testing,anddeploymentofservices(OPS.3.2.M8).

• Compliancewithsecuritycontrolsstipulatedinserviceagreementsandcontinuedviabilityofthe securitycontrolsisverifiedonaregularand/orcase-by-casebasis(OPS.3.2.M9).

• Aserviceemergencyreadinessplaniscreated(OPS.3.2.M11).

• TherisksoftheAIserviceprovider’sprocessesandITsystemshavebeenanalysed(OPS.3.2.M12).

• AIserviceproviderensuresthetransparencyofthesupplychain(OPS.3.2.M16).

• AccessoftheAIserviceprovider’sandclient’semployeestorooms,systemsandnetworks,aswell asaccesstoAIsystemdataandsoftware,isregulatedusingappropriateorganisationalandtechnical controls(OPS.3.2.M17).

• Subcontractor’semployeesareinstructedinperformingtheirtasksandinformedofcurrentinformationsecurityrequirementsanddocumentsregulatinginformationsecurity(OPS.3.2.M18).

• SecureencryptionmechanismsareagreedonforthesafetransferandstorageofdataattheAI serviceprovider(OPS.3.2.M20).

Securityincidenttreatment(DER2.1)controls:

• DefinitionofpossiblesecurityincidentsincludesthedefinitionsofsecurityincidentsrelatedtoAI systems(DER.2.1.M1).

• SecurityincidenttreatmentguidecoverssecurityincidentsrelatedtoAIsystems(DER.2.1.M2).

• SecurityincidenttreatmentmethodologycoverssecurityincidentsrelatedtoAIsystems(DER.2.1.M7).

• SecurityincidentreportingguidecoversreportingofsecurityincidentsrelatedtoAIsystems(DER.2.1.M9).

• ImpactofsecurityincidentsrelatedtoAIsystemsisassessed(DER.2.1.M10).

• EmployeesoftheITdepartmentarereadytotreatsecurityincidentsrelatedtoAIsystems(DER.2.1.M15).

• PrioritiesforthetreatmentofincidentsrelatedtoAIsystemsareestablishedbasedontheimpactof differentbusinessprocesses(DER.2.1.M19).

Auditandreview(DER3.1)controls:

• AIsystemsareaddedtothescopeofaudits(DER.3.1.M2).

• Reviewsverifytheintegrity,adequacy,andup-to-datestatusoftheimplementationoftheinformationsecuritycontrolsunderreviewinAIsystems(DER.3.1.M4).

• ListofreviewobjectsincludesAIsystemcomponents(DER.3.1.M8).

• AIsystemsareauditedbyasuitableauditorreviewgroup(DER.3.1.M9).

6.1.2Systemcontrols

SystemcontrolsareidenticalfornormalITsystemsandAIsystems,aswellasnormalapplicationsand AIapplications.ThesecontrolsaredescribedintheSYSandAPPmodulesoftheE-ITSbaselinesecurity catalogue.

6.2AI-specificriskcontrols

6.2.1ImprovementofthequalityandsafetyofAIsystems

SeveraldifferentapproachesexistforthemitigationofrisksrelatedtothequalityoftheoutputsofAIsystems.Inthecaseofanimportedmodel,thefirstcontrolistosimplyacquireabetterAImodel(assuming thatoneexists).Thisrequiresresearchintothemodelproviderandthemodel’schainofdelivery(e.g., datasetqualityindicators).Next,thequalityofthesystem’soutputsmustbecontinuouslymonitoredto determinewhetherthequalityoftheAImodelremainsstableovertimeandwhetheritcanhandlepreviouslyunseeninputs.Bothtechnologicalindicatorsanduserfeedbackcanbeusedforthispurpose.Ifa shiftinthequalityofthemodel(e.g.,inrelationtoaspecificinputclass)oranyotherincidentisdetected, modelexplainabilitymethodscanhelpinterprettheshift.

Varioussolutionsexistforpreventinghallucinationsinlanguagemodels.Fromtheperspectiveofapplicationarchitecture,aRAG(Retrieval-AugmentedGeneration)solutioncanbeusefulwherequeriesare madetoanexisting(text)datasettocomposetheoutput.InterfacingtheAImodelwithanexisting knowledgebasecanhelpreducetheoccurrenceofincorrectorunverifiableresponses.RAGsolutions,in whichtheoutputofthelanguagemodelincludesreferencestosearchengineresults,areusedforensuringthecontrollabilityandexplainabilityofanAIsystem.Theoutputofalanguagemodelcanadditionally beinfluencedbyusingpromptingtechniquestoinstructthemodeltouseonlyinformationfoundbya searchengine.Additionalfine-tuningofthemodelandtrainingdataqualitymanagementcanalsohelp preventhallucinations.

ToavoiddependenceonandlosingcontroloverAI,human-in-the-looptechnologiesshouldbepreferred. Thisisespeciallyvitalinthecaseofcriticalorhigh-riskusecases.Thefreedomofactionofanagentbasedartificialintelligencemustbelimitedtoaspecifictaskdomain,e.g.bylimitingthepermissions giventotheAIagent.Adoptionofartificialintelligenceindifferentworkflowsrequirestransparency,as wellascompliancewithrelevantregulations.

MitigationofrisksrelatedtobiasedandharmfulresponsesisaprocesscoveringtheentireAIchain ofdelivery.Thequalityanddiversityoftrainingdatamustberequired,themodelmustbefine-tuned

basedonthequalityandsecurityindicators,andtheseindicatorsmustbemeasuredandmonitoredin thedeploymentoftheAIapplicationusingthemodel,blockingunapprovedinputsoroutputs.

6.2.2ControlsfortechnologicalattacksagainstAIsystems

Inthefollowing,wewillusetheabbreviationsusedfordeploymentmodelsinSection 4 ,asnotallprotectivecontrolsarerelevanttoallmodels.Foreachcontrol,wewilllistthemodelswhichthecontrolapplies to.

Thepre-promptofthelanguagemodelshouldnotcontaininformationtheusershouldnothaveaccess to.Thedeployermustproceedfromtheassumptionthatthecontentsofthepre-promptarealways extractablebytheuser. DM1,DM2,DM3

Ifthelanguagemodelusestheuserinputtoconstructqueriestoaninterfacedservice(e.g.,RAGsystem components)thequeryshouldnothavemorerightsthantheuser.Inotherwords,ifaserviceorapplication(e.g.,database)isinterfacedtoalanguagemodel,itmustbeassumedthattheuserisalsocapable ofmanuallycomposingqueriestotheinterfacedservice.Thishelpsmitigateunauthorisedaccessand sensitivedataleakrisks. DM1,DM2,DM3

Iftheuserinputcontainssectionsofcodetoberun,therunningenvironmentshouldbeisolated.Evenif runningcodeisnotanintendedfunctionality,userinputprocessingmustaccountforthepossibilitythat theinputcontainscallsto eval exec,orsimilarcommandsorfunctionsthatstillattempttodoso.Such inputsmustbefilteredtopreventremotecodeexecution.Indirectpromptinjectioncanbemitigatedby validatingtheresponsestoAPIcallsandqueriestootherinterfacedapplications. DM1,DM2,DM3

ProxyandfirewallarchitecturesareusedinAIapplicationswheretheuserqueryfirstreachesaproxy loggingandfilteringmaliciousqueries,sanitisesandrewordsthemifneeded,andselectstheapplicable models.Thequeriesarethenpassedontothefirewallprotectingthemodelsandtheirinfrastructure. Fromthefirewall,thequeryispassedontothemodel.Themodel’sresponsepassesthroughtheproxy andthefirewallintheoppositeorder,andtheresponseisvalidatedinbothstagesbeforereturningitto theuser. DM1,DM2,DM3

Topreventtheinterpretationofthemodel’soutputbytheuser’swebbrowserasJavaScriptorMarkdown code(scriptinjection),themodel’soutputmustbeencoded. DM1,DM2,DM3

Datapoisoningandbackdoorattackspresumeaccesstotrainingorfine-tuningdatasets.Controlsagainst thesecoverthemodel’sentirelifecycleandsupplychain.Thefirstcontrolagainstsuchattacksisdataset curation.Qualitymetricsmustbeappliedwhenthetrainingdatasetisassembledviadatacrawling(automateddatacollectionontheInternet),datasourcesvalidatedandfilteredbasedontheirtrustworthiness whilepayingspecialattentiontothequalityofdataclassesrelevanttothespecificsofthemodel(e.g., legalormedicalsources). DM3

Toavoidbackdoorattacks,variousreliabilityenhancementtechniquescanbeusedwhentrainingimage models,e.g.,imagetransformation,suchasnoiseadditionandmaskingportionsoftheimage–thiscan reducetheimpactofbackdoor-openinginputs. DM3

Ifapre-trainedmodelisadoptedfromanexternalsource,themodelprovidermustbeverifiedtobe trustworthyandtransparentregardingtheirdatasupplychain,andtoprovideadequateinformationon thecapabilitiesandweaknessesofthemodel(modelmaps). DM1,DM2

Amodel’sperformancemustbecontinuouslymonitoredwhenusedinanapplication,includinginrelationtospecificinputcategoriesorclasses,toensuretheabilitytodetectsituationswherethemodel’s performanceinrelatedtoaspecificdatacategoryorclassfallsbelowacertainthreshold–thismaybe asignofdatapoisoning. DM1,DM2,DM3

Tomitigatetheriskoftransferofvulnerabilitiesintransferlearning(whichishighestinthecaseofthe adoptionofpre-trainedopensourcemodels),itisrecommendedtoperformadditionalfine-tuningofthe model,althougheventhismayproveinsufficient.Afterfine-tuning,thequalityandsecurityindicatorsof theoriginalmodelcannolongerbereliedupon[158 ]–theymustbere-applied. DM2

Languagemodelscanbemadetoquotethecontentsoftheirtrainingdatasets[169 ].Differentcon-

trolsexistforthemitigationoftheriskofleakingsensitivepersonallyidentifiabledatafoundintrainingdatasets.First,attemptscanbemadetoexcludethemfromtrainingdatasetseitherindividuallyor dataset-by-dataset.Alternatively,syntheticdatacanbeusedwhichpreservetherelationshipsfound intheoriginaldatabutdonotcontainsensitiveorpersonallyidentifiableinformation.Datacanalso bepseudonymised,e.g.,byreplacingpersonallyidentifiablepiecesofdatawithcorrespondinglabels. Pseudonymisationcanalsobeappliedontheoutputsideofthemodel,i.e.,asapartofthelogicofthe AIapplication,butsuchoutputscouldstillprovepersonallyidentifiable[188 ]andamodelofthiskindis morevulnerabletodataextractionattacks,ifithappenstobeleaked.

Themodelreturningdatathatisseeminglypersonallyidentifiableorevenoverlapswithpersonallyidentifiableinformationcannot,ineverycase,beconsideredaprivacyviolation,asitcouldbearandom coincidenceresultingfromrelationshipsfoundinthemodel.Thus,alanguagemodelmayoutputthe medicalhistoryofapatientwithacommonnameandsymptomsasaresponsetoaspecificquery.To verifythatthisisindeedacoincidence–notaleakofpersonaldata–adifferentialprivacymethodcanbe usedwheretheprobabilityofreturningthatspecificoutputiscomparedinsituationswheretherelevant recordwereorwerenotincludedinthetrainingdataset.Anotheroptionistousedifferentiallyprivate(or otherprivacyenhancingtechnology-based[189 ])trainingandfine-tuningmethods[190 ]. DM3

Tomitigatedenialofserviceattheapplicationlevel,applicationinformationsecuritypracticesshouldbe followed.TopreventdenialofserviceattacksexploitingthefeaturesoftheAImodel,itisimportantto limitinputlength,whichshouldcorrespondtothemodel’sfeatures(e.g.,inthecaseoftransformer-based languagemodels,lengthofthecontextwindow),aswellasresourceuseconnectedtoasinglequery, andthenumberofsubstepsorsubqueries. DM1,DM2,DM3

Limitingthenumberofqueriesmadebyasingleusercanhelpfightmodelinversionandmodeltheft, hamstringingtheadversariesinattemptsofaccumulatingasufficienttrainingdatasetorlogitderivation. DM1,DM2,DM3

6.3Controlsforsocietalrisks

6.3.1Controlsoperatingatthesocietallevel

AIsystemshavemadeamajorleapforwardinthepastfewyears.Eventhoughthesesystemshavethe potentialtoimproveefficiencyandcreatenewopportunities,thismaycomeatthepriceofnumerous riskstothesociety,someofwhichwillbediscussedbelow.

• Dataprotectionandprivacy LargedatasetsusedbyAIsystemscomewiththeriskofexploitation ofthesedata,includingtheviolationofprivacy.Onepossiblecontrolisraisingtheawarenessof thesocietyofAIsystemsanddataprotectionandprivacyissuesrelatedtothesesystems,e.g., bypublishingguidelinesonthecollection,processing,andstorageofdata.Anotherveryefficient methodforthemitigationofrisksistoreducetheprocessingofpersonallyidentifiabledata.This canbeachievedeitherviachangestothebusinesslogicorasystemimplementingAIusingprivacy enhancingtechnologies.

• Changesinthelabourmarket.TheevolutionofAIwillalsoleadtovariouschangesinthelabour market.Thetechnologyenablessimplifyingcertainworkprocessesandmakingthemmoreefficient, whichwillresultinarestructuringoftheworkforce.Atthesametime,certainpositionsintraditional industriesmaydisappear.Todealwiththelabourmarketchanges,noveleducationalorretraining programmescouldbeintroducedtohelppeopleadapttothenewtechnologyandlearntousethe possibilitiesofferedbyAI.

• Socialdivides .IfcertainsocialgroupslackaccesstoAItechnologyortheskillstomakeefficientuse ofthistechnology,thismayleadtotheexacerbationofthedigitaldivide.Itisthereforeimportantto thinkofhowtomakeAItechnologyaccessibletodifferentsocialgroupsfromchildrentotheelderly, e.g.,throughtheintroductionofwidelyaccessibleeducationalprogrammes.

• Discrimination .DevelopmentofunprejudicedAIsystemsisarelativelycomplexprocess.AnAI systemcharacterisedbyprejudiceandapatternofdiscriminationcould,however,increasesocial inequalityandviolatebasichumanrights.TheAIsystem’salgorithmsmustthereforebesystemati-

callyassessedand,ifnecessary,improved(or,intheworstcase,disablethem)toensurecompliance withtheprinciplesofdiversityandjustice.

• Technologicaldependenceandvulnerability .ThedependenceofthesocietyonAIsystemsison therise.Thismay,inturn,increaseitsvulnerability.Asacontrol,thetechnologicalinfrastructure needstobediversifiedandresourcesinvestedintothedevelopmentofthesafetyandresilienceof AI.

• Ecologicalfootprint.Artificialintelligencesystemsarebasedonmassivedatasetsandtheintense useofcomputationalresources.Thedevelopmentandoperationofsuchsystemsthusincreases energyuseandhencealsoourecologicalfootprint.Onewaytomitigatethiscouldbetocarryout researchintosustainableandmoreenergy-efficientAIsystems.Specificindicatorsshouldalsobe agreedontoassessAI’senvironmentalimpact.

TheimpactofAIsystemsonthesocietyismanifoldinnatureandpotentialassociatedthreatsrequirethe approachesusedfortheimplementationofsuitablecontrolstoaccountfortheimpactofAIonavariety ofaspects.ResearchanddevelopmentandpolicymakingshouldstrivetowardstheuseofAIsystems supportinggeneralsocietalwell-being,inclusion,andsustainability.

6.3.2AIsystemlevelcontrols

TheimmaturityoflegalactsandsupervisoryauthoritiesregulatingAImeansthattheeasiestwayto ensurethesafetyofapplicationsisthroughself-assessment.Duringthedevelopmentofanartificialintelligenceserviceorapplication,itisnecessarytoassessthesystem’simpactonindividualsand,through them,thesociety.Theefficiencyofthisevaluationnaturallydependsonthedeveloper’sethicalconvictionsandtechnologicalmaturity.

Statesandenterpriseshavegloballydevelopedvariousrecommendationsandguidelinesforapproaching thisissue.Termssuchasresponsible,trustworthy,andsafeAIarefrequentlyused.Wewillhighlight herethetrustworthyAIself-assessmentmodel[90 ]developedbytheEUAIHLEG,wheresevenkey requirementsaresetoutfortrustworthiness:

1. humanagencyandoversight;

2. technicalrobustnessandsafety;

3. privacyanddatagovernance;

4. transparency;

5. diversity,non-discrimination,andfairness;

6. environmentalandsocietalwell-beingand

7. accountability.

Below,wewilllistasetofguidelinesthatwerecommendtobefollowedinthedevelopment,implementation,anduseofAIsystems.

• Human-centredvalues .ThedevelopmentofanAIsystemshouldbefoundedontheprinciplesof human-centreddesign,respectingandprotectingtheindividual’sphysicalandmentalintegrityand theirsenseofidentity[45 ].

• Preventionofharm .AIsystemsmustbesafeandsecure,technologicallyrobust,andtheirmalicious useshouldbeprecluded[45 ].

• Fairness .TheAIsystemshouldbeensuredtopromoteequalopportunitiesandnotbeunfairlybiased ordiscriminatespecificindividualsorsocialgroups[45 ].

• Accountability .AccountabilitymeansthatthepartiesinvolvedinAIdevelopmentassumeresponsibilityforthesystem’sproperoperationbasedontheirroleandaccountingforboththecontextof useofthesystemandconsistencywiththestateoftheart[191].

• Explainability .ThepurposeandcapabilitiesoftheAIsystemmustbeknownandallprocessesshould bemaximallyexplainabletopersonsimpactedbythem[191].

• Inclusiveeconomicgrowth,sustainabledevelopment,andwell-being. TheuseoftrustworthyAI shouldcreatevalueforindividuals,thesociety,aswellastheentireplanet,increasecreativity,reduce inequality,andprotectthenaturalenvironment[192 ].

7Policyrecommendations

TheimplementationofthefollowingpolicyrecommendationswillsupportthegrowthoftheEstonianAI ecosystemandAIeconomy.Still,theymayhaveinternationalrelevanceinotherterritories,basedon thelocalregulations,standardsandtechnologicalmaturity.Thedevelopmentofethicalandresponsible AIrequiresafunctionalecosystemtoencourage,inspire,andsupportitsdevelopment.Wide-ranging cooperationbetweendifferentpublicandprivatesectorstakeholdersisvital.Sustainableuserequires workingonawarenessofrisksrelatedtoAIsystemsandtimelyimplementationofmitigationmeasures.

• InvestmentsinAIresearchanddevelopment.TofacilitatetheemergenceofcompetitiveAIcompaniesinEstonia,AI-relatedresearchanddevelopmentshouldbesupported.Publicinvestments shouldbeprovided,andprivateinvestmentsencouraged.TheNIS2directivealsoencouragesAIrelatedresearchanddevelopmenttoimprovethedetectionandpreventionofcyberattacks,andthe planningofresourcesforthispurpose.

• Talentreproduction .Scholarshipprogrammesandcooperationprojectswithuniversitiesshouldbe createdtoincreasethenumberoflocalexperts.This,inturn,willcreatetheprerequisitesforthe developmentofanationalcommunityofAIexperts.Talenttrainingfacilitatesdevelopinghuman capabilitieswhichisalsoimportantforadaptationtochangesinthelabourmarket.

• CreationofAIsystemsandboxes,developmentcentres,orincubators .Controlledenvironments canbecreatedforAIdeveloperstoprovideentrepreneursaccesstonecessaryresources(e.g.,funding,infrastructure,mentoring,technicalsupport)andallowingtestingofnewAIsolutions.SuchcontrolledenvironmentswouldfacilitatesafertransitionofAIsystemsfromresearchanddevelopment todeploymentandoperation.Fromtheregulators’perspective,itwillfacilitategainingknowledgeof newAItechnologiesandtakingthisknowledgeintoaccountinpolicydecisions,ifneeded.According totheAIAct,eachEUmemberstatemustcreateatleastoneregulativeAIsandbox.

• Creationofapublicdataplatformordatafoundation .AIsystemsarecharacterisedbyasignificant dependenceondata.Publicdataplatformswouldprovidebusinessesandresearchersaccessto largedatasetsthatcouldbeusedforthetrainingandtestingofAIalgorithmsindifferentspheres.For newAIdevelopers,thecreationoftrainingdatasetscanbetime-consumingandcomplicated(e.g., fromtheperspectiveofdataprotectionandintellectualpropertylaw).Whileopendataarepublished inEurope,includinginEstonia,theiruseforthetrainingofAImodelsisimpractical.Thisisduetothe factthattheyarenotagoodreflectionofthereallifesituation–thelevelof’cleanliness’ofopendata isveryhigh,whichdoesnotfacilitatediversity,andedgecasesaregenerallyremoved.Thestate couldthereforehelpcreatepublicsyntheticdatasetswhichwouldberepresentative,unprejudiced, wouldrespectprivacy,andcomplywithbothpersonaldataprotectionrequirementsandintellectual propertylaw.

• StandardsforthedescriptionofAImodels .StandardsforAImodelswouldbebeneficialforidentifyingwhatkindsofdatasetstheyweretrainedonandhowthedatawasacquired.Standardscould alsobeusefullyadoptedtolabelsynthesisedimages,text,andotherinformation.

• TechnologicaltoolkitforensuringthesecurityofAIsystems .AwarenessoftechnologicaldevelopmentsisvitalforprotectingthesecurityofAIsystems.Itisthereforerecommendedtoprotect thesesystemsbyusingefficienttools,suchasend-to-endprivacywhichpreventsoutsidersfrom accessingdataontheAIsystem(e.g.,unauthorisedreadingorsecretlychangingdata).

• CreationofafavourablepoliticalenvironmentforAI.Atransparentlegalframeworkwillencourage businessestoinvestinAIsystems.ThiscallsforthecompositionofguidelinesandsharingbestAI practices,e.g.,bysharingthegovernment’sexperiencesandlessonsfromthedevelopmentofAI applications.PolicymakingshouldalsobeusedtoencourageinnovationandcompetitioninthedevelopmentoftrustworthyAI.Holdinginnovationcompetitionsisrecommendedtoinspirethecreation ofinnovativeAIapplicationsindifferentareas.

• Promotionofinternationalcooperation .Internationalpartnershipsareimportantforsharingknowledge,experience,andresources(e.g.,throughcooperationprojects).This,inturn,willcreatethe conditionsforfastertechnologicaldevelopmentandincreaseexportopportunities.

• Preservationandpromotionoftheevolutionofthenationallanguageinadigitalera .Datasets usedfortrainingAIs,aswellasInternetcontentingeneral,aremainlyinEnglish.Inspiteofthis, AIwillcreatenewopportunitiesforcontributingtotheevolutionofotherlanguagesthroughhighqualityautomatedtranslations,automateddigitisationofandextractingstructureddatafromarchive materials,aswellasboostinginnovativeteachingmaterialsandothermethodsofthedigitalhumanities.ThecontinueddevelopmentofEstoniantextandspeechcorporaisextremelyvaluableforthe preservationoftheEstonianculture.

• RaisingsocietalawarenessofAIsystems .PublicdebateoverAIshouldbeencouragedandawarenesscampaignscarriedout.ThisisvitalforexplainingthebenefitsaswellasthechallengesofAI.It isalsoimportanttocollectfeedbackfromcitizensinordertodesignpoliciesinlinewiththedemands ofthesociety.

8Quickreferenceguidefororganizations

8.1DescribeyourAIsystem

UsetheworksheetinFigure 21 andfollowtheinstructionsbelowtofillinallfourcolumns.

ListtheendusersoftheAIsystem(sectionsA 1–A n oftheform).

1. WhoarethedirectusersoftheAIsystem?Listusersbothontheserviceprovideranduserside. IdentifythemainroleswhosedataareprocessedbytheAIsystemorwhousetheresultsofthe processing.NOTE:endusersshouldalsoincludepotentialinformationsystemsusingautomated decision-making,asthisinformationwillbeneededlateronintheimpactanalysis.

2. Listwhattheuserneedsthesystemfor.Thiswilllaterassistyouinimpactassessment.

3. ListthetypesofdataprovidedtoandreceivedfromtheAIsystembytheuser.Thesewilllaterform thebasisforariskandimpactassessment.Wherepossible,alsonotewhetherthedataisstructured, tabular,textual,image,audio,video,oracombinationofmorethanone.

DescribetheserviceusingAItechnology(sectionsB 1 andB 2 oftheform).

1. WhatisthepurposethattheAIsystem(apporservice)wascreatedtofulfil,whatisthevaluethat itgenerates?

2. Listthemodelsandtechnologiesused,toyourbestknowledge,bytheserviceproviderwhosemodel underliestheapporservice.

3. Describetheinfrastructure(in-housedatacentre,cloudservice)theserviceoperatesonandinwhich countryisthisinfrastructurelocated.

4. BasedontheinformationprovidedaboveontheusersoftheAIsystems,provideasummaryofthe datatransmittedbytheservicetotheAIcomponentandvice-versa.

ExplainwhetherrunningtheAImodelisoutsourcedordoneusingin-houseinfrastructure.

1. Ifitisoutsourcedtoaserviceprovider(e.g.,throughanAPI),completesectionC 1

a. Whoistheserviceproviderandwherearetheylocated?

b. Whatdataisthemodeltrainedon?Theobjectivehereistoverifythatthetrainingofthemodel hasbeenlegal(e.g.,nounauthoriseduseofcopyrightedinformation).

c. Whatisthecountryoforiginoftheserviceproviderandwhereistheirinfrastructurelocated?

d. Addareferencetothetermsandconditionsoftheserviceprovidedorthetermsoftheagreement youhavesigned.

2. IfthecreatedAIsystemrunsthemodelsitself(irrespectiveofwhetherithasbeentrainedin-house, licensed,orbought),completesectionC 2

a. Whohastrainedthemodelandwhatcountryisthatorganisationfrom?

b. Whatdataisthemodeltrainedon?Theobjectivehereistoverifythatthetrainingofthemodel hasbeenlegal(e.g.,nounauthoriseduseofcopyrightedinformation).

c. Whattechnologydoesthemodeluse(asfarasyouknow)?

d. Whereistheinfrastructureusedforrunningthemodellocated(isitanin-housedatacentreor cloudinfrastructure)?

Finally,writedowneverythingyouknowaboutthetrainingofthemodel,regardlessofwhetheritwas trainedexternallyorinternally.

1. IftheAImodelwasbought,licensed,orisusedviaanAPI,completesectionD 1

a. Asfarasyouareaware,whatkindofdatawasthemodeltrainedon?

b. Whatarethetermsofuseofthemodel?E.g.,whatliabilitiesareassumedandwhatguarantees providedbythemodeltrainer.

2. IftheAIsystemprovidertrainsthemodelin-house,completesectionD 2.

a. Whatkindofdataisthemodeltrainedon?Whereweretheyacquiredandonwhatconditions?

b. Whatkindoftechnologyisusedfortrainingthemodel?Listalgorithmsandtools,wherepossible.

c. Whereistheinfrastructureusedfortraininglocated?

d. Describetheknow-howtheserviceproviderpossessesfortrainingAImodels.

8.1.1Howtogoevenfurther?

TheformpresentedinFigure 21 helpswiththeinitialstructuringofyourideasandaskingrelevantquestions.Oncethisisdone,itwillbeusefultobreaktheanswersdowninmoredetail.Thiscanbedoneina separatedocument.Itprovidesagoodopportunityforintegratingtheprocessintotheorganisation’sexistingquality,managementorcybersecuritysystem.Ifthisrequiresspecificprocessestobecompleted, theformpresentedhereinfacilitatecollectinginformationrelevanttothoseprocesses.

Anotherfurtherstepwouldbetheimplementationofanartificialintelligencemanagementsystem,e.g. ISO/IEC42001.Thiscan,ifnecessary,beintegratedwithISO9001andISO/IEC27001management systems.

8.2Findadeploymentmodelsuitingyoursystem

AftertheAIsystemhasbeendescribedusingtheformabove,thenextstepistoidentifythedeployment modeltobeusedforriskassessment.Ifyouhavecompletedtheformabove,thischoicewillbeeasy andrequireansweringjusttwoquestions.ThedecisionchartforthisispresentedinFigure 22

ThepurposeofquestiononeistodeterminewhetherthecreationoftheAImodelisunderthecontrolof theAIapplication’screator.Ifyes,thenthecreationofthemodelmustbetreateddifferentlyfromother deploymentmodelsinsubsequentriskanalysis(DM3).

ThepurposeofquestiontwoistodeterminewhethertheapplicationoftheAImodelisunderthecontrol oftheAIapplication’screator.Thisfacilitatesfocusingonrisksrelatedtothechoiceandhandlingofthe modelinriskassessment(DM2).

IfthecreatoroftheAIapplicationneithertrainsnorrunsthemodelitself,theyareverylikelytousea deploymentmodelwheretheAIcomponentisboughtasaservice(DM1).

Wewillnoteherethat,inallcases,eitheranin-housedatacentreorprivateorpubliccloudcomputing systemcanbeusedasinfrastructure.Thishasnoimpactonthechoiceofdeploymentmodel,andthe locationoftheinfrastructurewillbetreatedseparatelyinriskassessment.

8.3Identifyapplicablelegalnorms

Itisimportanttorecognisethattheguidelinespresentedinthisreportdonotqualifyaslegaladviceand theycannotbetreatedastheprovisionoflegaladviceoralegalservice.Themainpurposeofthese guidelinesistohelpdeterminewhichlegalactsmustbetakenintoaccountwithoutexception.Every serviceprovidermustensurethecomplianceoftheirservicetorelevantstatutory,contractual,andother stipulations.

Figure 23 isasimplifiedflowchartforidentifyingwhichlegalnormscanapplytoanAIsystemintheEU. OurfocushereisonasituationwheretheguidelinesareusedbyanAI-basedserviceprovider.

8.3.1DM1:ServiceusinganAIAPI

DoestheAI-basedapp/serviceprocesspersonallyidentifiabledata(seesectionsA 1–A n andB 2 ofthe form)?

Ifyes,thentheservicefallswithinthescopeoftheGDPRandapplicabledataprotectionlaw.

DoestheAI-basedapp/serviceprocesscopyrightedworks(seesectionsA 1–A n andB 2 oftheform)?

Ifyes,thentheservicefallswithinthescopeoftheCopyrightActandapplicablecopyrightlaw.

DoestheAI-basedapp/serviceprocessprotecteddatafromaspecificfield(e.g.,taxsecrets,banking secrets,confidentialinformation)(seesectionsA 1–A n andB 2 oftheform)?

Ifyes,thentherequirementsoflegalactsregulatingtherelevantfieldsmustbetakenintoconsiderationinthedevelopmentoftheservice.

DoestheAI-basedapp/serviceprocesscertaindatabasedonspecificagreements?(seesections A 1–A n andB 2 oftheform)?

Ifyes,thentheclausesofsaidagreementsmustbefollowedduringservicedevelopment.

DoestheAI-basedapp/serviceorthemodel-runningserviceoperateoninfrastructurelocatedina territorywithaninadequatelevelofdataprotection(seeSection 3.8 ofthereportandsectionsB 1 andC 1 oftheform)?

Ifyes,thendataprotectionrequirementsconcerningtheprocessingofpersonallyidentifiabledata onsuchinfrastructuremustbetreatedandevaluatedseparately.

WhatistheroleoftheenterpriseororganisationintermsoftheEuropeanUnionAIAct?

EvaluatethescopeoftheAIActandidentifywhetheryouqualifyas,e.g.,aprovider,deployer,or otherpersonwitharoleintheAIsystem’slifecycle.Followtherequirementsfortherelevantroles.

WhatistheriskleveloftheAI-basedapp/serviceintermsoftheAIAct(seeTable 3 andsectionsA 1–A n andB 2 oftheform)?

Table 3 providesaninitialassessmentoftheAIsystem’spotentialrisklevelwhichshouldbevalidated againstspecificrequirementssetoutintheAIAct.UsetheAIActtodeterminetherequirements applicabletoanAIsystemwiththatspecificrisklevel.

DoestheAItechnologyemployeduseageneral-purposeAImodel(seesectionB 1 oftheform)? Additionalrequirementsapplytosystemsusinggeneral-purposeAImodelundertheAIAct.

8.3.2DM2:systemusinganexternally-trainedAImodel

AnswerallquestionsinSection 8.3.1 andthefollowingadditionalquestions.

HastheAImodelbeentrainedonpersonallyidentifiabledata,copyrightedworks,orotherdatarequiringseparateauthorisationforprocessing(seesectionD 1 oftheform)?

Ifyes,thenitmustbedeterminedwhetherthemodel,whenused,couldoutputresponsesrequiring alegalbasistobeprocessedbytheservice/app’screator.

Doesthecreatoroftheapp/serviceplantoimproveorcontinuetrainingtheAImodel?

Ifyes,thentheAIappcreatormustsecurerightstousethesedataforimprovingtheAImodel.

8.3.3DM3:systemusinganAImodeltrainedin-house

AnswerallquestionsinSections 8.3.1 and 8.3.2 andthefollowingadditionalquestions.

Arepersonallyidentifiabledata,copyrightedworks,orotherdatarequiringseparateauthorisationfor processingusedfortrainingtheAImodel(seeanswerstosectionD 2 oftheform)?

Ifyes,thenitmustbedeterminedwhetherthemodel,whenused,couldoutputresponsesrequiring alegalbasistobeprocessedbytheservice/app’screator.

IstheAImodelusedintheEUasapartofanAI-basedapp/service(seesectionsB 1 andB 2 ofthe form,butalsoconsidersituationswherethemodelcouldbeusedbysomeoneelseforprovidinga service)?

Thisquestionfocusesonaspecialcasewherethetrainedmodelisactuallyappliedbysomeoneelse. Eventhoughthisspecialcasewasnotdiscussedinthedeploymentmodelsabove,werecommend youtoconsiderthispossibility.ApplicationsofthistypealsofallwithinthescopeoftheEuropean UnionAIAct.

8.3.4Howtogoevenfurther?

Thefirststepincompliancewithdataprotectionrequirementsistoestablishthesystemstakeholdersin termsoftheGDPR,followedbymappingthedataflowsbetweenthem.Theresultofthisworkcanbea tablewherelinesrepresentallstakeholdersrelatedtotheoperationoftheAIsystemandthecolumns, thedataelementsthattheyprocess.

Markeachcellofthetableifthespecificstakeholderprocessesthespecificdataelementinthesense ofdataprotectionlaw(e.g.,collection,storage,anddeletion).Ifthesystememploysprivacyenhancing technologies,thecellcanalsoshowtheleveltowhichthespecificdataelementhasbeenmademore difficulttopersonallyidentifyforthespecificstakeholder.

Artificialintelligencelawisinrapiddevelopmentatthemoment,makingitinfeasibletoprovidequickand specificrecommendationsfortheyearstocome.ItisimportanttomonitortheevolutionofAIregulation inthetargetmarketsofthedevelopedservice.

8.4Evaluatethreatstousers,society,andenvironment

8.4.1DM1:systemusingAIasaservice

Impactanalysis1.1: Foreachenduser,seetheresponsesprovided(sectionsA 1–A n oftheform)andthe generaldescriptionofthesystem(sectionsB 1 andB 2 oftheform)andwritedownthekindofdecisions whichtheusercouldmakebasedontheresponsesreceivedfromtheAIsystem,andwhetheranyof thesedecisionsmayhaveadirectimpactofanotheruserorathirdpartyorcoulddirectthemtotake anydecisionsorsteps.

Itisimportanttofocushereontheusersofthesystemonboththeclientandserviceprovidersides. AclientofthesystemcouldgetinformationfromtheAI’soutputthattheywillusetomakeadecision impactingtheirorsomeoneelse’slife.Analysingsuchthoughtprocesseswillfacilitateawarenessofthe AIsystem’simpactonhumanbehaviourand,therefore,thesociety.

Aseparateimportantstepistoalsoconsiderhereasend-usersinformationsystemsmakingautomated decisionsusingAI,andtheirimpact.Forexample,ifaserviceorappusesAI-basedautomateddecisions forapprovingallowances,loans,orrentals,theAIsystemwillhaveadirectimpactonthelivesofthird personswhichthecreatoroftheserviceneedstobeawareof.

WritedownallactionsidentifiedthroughthisthoughtexperimentthattheAIservice’soutputcandirect anindividualto.Figure 24 providesanexampleofaworksheettousefortheanalysis.Expandthe worksheetwithnewcellsasrequired.

Impactanalysis1.2: Foreachsuchaction,evaluatewhetheritcouldhaveanegativeimpactonthe individualorthesociety.Someofsuchharmfulimpactscouldincludethefollowing.

1. UseoftheAIserviceimpactsthebasicrightsofapersonoragroupofpersons.

2. AdecisionmadebasedontheAIservice’soutputdiscriminatesagainstaspecificsocialgroupbased onsomeoftheirtraits.

3. AdecisionmadebasedontheAIservice’soutputwillleadamemberofthesocietytocauseharm tothemselves(e.g.,inaccuratehealthadvice,inaccurateeducationaladvice,inaccurateinvestment advice).

4. AdecisionmadebasedontheAIservice’soutputwillleadamemberofthesocietytocauseharmto anotherperson(e.g.,inaccuratediagnosis,inaccuratetreatmentadvice,inaccuratesuspicioninan offence,inaccurateassessmentofskillsorcapabilities).

Impactanalysis1.3: Collectallscenariosinvolvingdecisionsleadingtopotentialharmfulimpact.AnalysetheextenttowhichanAImodeloperationserviceproviderassumesresponsibilityandproposes countermeasurestothese.IdentifyscenariosinwhichtheAImodeloperationserviceprovider’scountermeasuresandresponsibilityareinsufficienttomitigatetherisk.Assesswhetherthebusinesslogicof theservicescanbechangedorscaleddown,oraddsuitablecountermeasurestothesystem(e.g.,transparency,additionofahumansupervisionmechanism,strongerdatamanagement,additionalcontrolsin businesslogic,awarenesscampaigns,trainingprograms).

Impactanalysis1.4: Evaluatethegeneralimpactofthecreatedsystemonthenaturalandlivingenvironment(withoutfocusingonspecificgroupsofindividuals).Evaluatewhetherthecreationofthe systemhasanimpactontheenvironment–whetheritimpactstheuseofenergyornaturalresources, e.g.,throughsupportingwastefulorpollutingbehaviour.Iftheimpactisharmful,changeorscaledown systemfunctionalityorimplementnecessaryharmpreventionorreductionmeasures.

8.4.2DM2:systemusinganexternally-trainedAImodel

CompleteallstepslistedinSection 8.4.1,aswellasthefollowingsteps.

Impactanalysis2.1: Familiariseyourselfwiththemodelprovider’sserviceconditions,descriptionofthe model,andsafetyinformation(seesectionsC 1 andC 2 oftheform).Identifythepotentialharmfulimpacts oftheuseofthemodel.

Ifyouseethatriskscanbereducedinatechnologicallyadequate,legallysound,andethicalmannervia additionalAImodeltraining,thenaddadditionaltrainingorfine-tuningoftheAImodeltotheplanned activities.

8.4.3DM3:systemusinganAImodeltrainedin-house

CompleteallstepslistedinSections 8.4.1 and 8.4.2 ,aswellasthefollowingsteps.

Impactanalysis3.1: EvaluatethebalanceandlackofbiasesintheAImodel’strainingdataset.Isit sufficientlyrepresentativetopreventdiscriminationintheapplicationofthemodel?Ifnot,findlegaland ethicalwaystoaddmoretrainingdatasets.

Impactanalysis3.2: Evaluatetheknow-howandtechnologicalsolutionsrequiredfortrainingtheAI model.Isthetrainingofahigh-qualitymodelpossibleandaffordablein-house?Iftherearedoubts regardingitsaffordability,youshouldconsiderusinganexternally-trainedmodelratherthantrainingone in-house.

8.4.4Howtogoevenfurther?

Guidelinesdevelopedforthispurposecanbeusedinimpactanalysis.WerecommendusingtheEU AIHLEGself-assessmentmethodology[90 ],andforLLMapplications,theOWASPFoundation’sLLMAI Cybersecurity&Governancechecklist[193 ].

ItcanbeexpectedthatEUAIregulationswillclassifysomeartificialintelligencesystemsashigh-risk systemsandestablishadditionalobligationsforrelevantserviceproviders.Followthedevelopmentsof theregulationtocomplywiththese.

8.5Performrisktreatmentandselectcontrols

8.5.1KeyrisksofAIsystems

Thissectionwillprovideinstructionsonwhatshouldbetheprimaryfocusofrisktreatment.Theseshould notbeconsideredexhaustivesecurityrecommendations.Eachorganisationisdifferentandmayrequire amorein-depthapproach.IftheorganisationprovidinganAIservicehasrisktreatmentpracticesin placethenthesepracticesshouldbefollowedandtheinstructionshereusedasaninitialguideline.

Tables 4 , 5 ,and 6 listthekeyrisksof,respectively,serviceprovision,runningAImodels,andtraining AImodels.Weassesstheirimpactashighandtheserviceproviderneedstofindwaystotreatthem. Naturally,yourriskassessmentprocesscanalsoidentifyadditionalrisksnotincludedinthistable.

AllthreetableslistthekeyrisksofAIapplicationbystages(compositionofinputintheapporservice, runningthemodel,trainingthemodel)anddeploymentmodels.

8.5.2Recommendationsforcybersecuritycontrols

Figure 25 presentsaselectionofmeasuresfromtheEstonianE-ITSinformationsecuritystandardsuitable forsecuringAIsystems.Theyarealsoclassifiedinthefigurebythecontextofthesystem.

Themajorityofthemeasuresareapplicabletotheserviceprovider’sorganisation,softwaredevelopment, andcloudserviceuseandoutsourcingpractices.Forsomeofthemeasures,wehavehighlightedtheir importancetothemachineoruserinterfacescreatedforusers.Wehavealsohighlightedthesignificance ofcertainpracticestocommunicationwithAIAPIormodelproviders.

Thecloudserviceandoutsourcingmeasuresarepresentedasoptional–iftheserviceproviderdoesnot usecloud-baseddataprocessingoroutsourceanything,theirimplementationmaynotberelevanttothe createdAIapporservice.

8.5.3RecommendationsforAIcontrols

WerecommendimplementingthecontrolsfromSection 6.2 toimprovethesafetyofAI-basedservices. ThesehelpimprovethequalityoftheAIsystemandavoidrisksarisingfromspecificAItechnologies.

8.5.4Howtogoevenfurther?

Werecommendcompletelyimplementinganystandardisedinformationsecurityorcybersecuritymanagementsystemorriskassessmentmethodology.SpecificreferencesarefoundinSection 5.1.ImplementingtheE-ITSorISO/IEC27001standardstoanappropriatelevelwillgreatlysupportthedevelopmentofthesecurityofAIsystems.Theworkputintoimplementingthisquick-referenceguidewillnot bewastedandwillsupporttheimplementationofthechosenstandardsintheorganisation.

Table4.KeyrisksofrunninganAI-basedservicebasedontheidentifieddeploymentmodel

Category DM1:ServiceusinganAIAPI

Cybersecurity

Legal

DM2:ServiceusinganexternalAImodel

DM3:AIserviceusinganin-housemodel

AvailabilityoftheAIAPIdoesnotmeet servicerequirements Commonrisks Commonrisks

Serviceproviderlackslegalbasisfor processinginputoroutputdataor submittingthedatatotheAPI

Serviceproviderlackslegalbasisfor processinginputoroutputdataor submittingthedatatotheAPI

AIsafety AIAPIoutputshaveharmfulimpact SeerisksofrunningmodelsinTable 5

Table5.KeyrisksofrunninganAImodelbasedontheidentifieddeploymentmodel

Category DM1:ServiceusinganAIAPI

DM2:ServiceusinganexternalAImodel

Serviceproviderlackslegalbasisfor processinginputoroutputdataor submittingthedatatotheAPI

SeerisksofrunningmodelsinTable 5

Cybersecurity

Serviceproviderdoesnotrunthemodel themselves

Legal

AIsafety

Serviceproviderdoesnotrunthemodel themselves

InfrastructureusedforrunningtheAI modellackssufficientperformance (availabilityrisk)

AImodelproviderdoesnotprovide improvementsandupdatesforthemodel

AImodeloritsoutputsincludedatathat theserviceproviderisnotauthorisedto process

Serviceproviderisnotauthorisedto processdatausedforimprovingthemodel

DM3:AIserviceusinganin-housemodel

InfrastructureusedforrunningAImodel lackssufficientperformance(availability risk)

Serviceproviderdoesnotrunthemodel themselves

Risksandcontrolsforartificialintelligenceandmachinelearningsystems

AImodeloutputshaveaharmfulimpact Dataandtoolsusedforimprovingthe modelreducethemodel’squality

SeerisksofmodeltraininginTable 6

SeerisksofmodeltraininginTable 6

Table6.RisksoftrainingAImodelsbasedontheidentifieddeploymentmodel

Category DM1:ServiceusinganAIAPI

Cybersecurity

Legal

AIsafety

Serviceproviderdoesnottrainthemodel themselves

Serviceproviderdoesnottrainthemodel themselves

Serviceproviderdoesnottrainthemodel themselves

DM2:ServiceusinganexternalAImodel

Serviceproviderdoesnottrainthemodel themselves

Serviceproviderdoesnottrainthemodel themselves

Serviceproviderdoesnottrainthemodel themselves

DM3:AIserviceusinganin-housemodel

AImodeltraininginfrastructurelacks sufficientperformance(availabilityrisk)

Serviceproviderlacksauthorisationfor processingdatausedfortrainingthe model

AImodeloutputshaveaharmfulimpact Dataandtoolsusedfortrainingthemodel reducethequalityofthemodel

Figure25.E-ITSmodulesrecommendedforAIsystemsandcontextsoftheirimplementation

Risksandcontrolsforartificialintelligenceandmachinelearningsystems May27,2024

8.6AIsysteminasingleslide

Theapplicationofartificialintelligencemayleadtosituationswhereanoverviewofthecreatedsystem mustbepresentedinasingleimage(e.g.,presentationslidetotheorganisation’smanagement).Thefiguresbelowpresenttemplatesfordescribingthestructureofthesystem.Eachfigurepresentsatemplate foraspecificdeploymentmodel(Figure 26 forDM1,Figure 27 forDM2,andFigure 28 forDM3).

Figure27.TemplateforpresentinganapporserviceusingdeploymentmodelDM2

Figure28.TemplateforpresentinganapporserviceusingdeploymentmodelDM3

Bibliography

[1] KaiWangetal. NeuralNetworkDiffusion .2024.arXiv: 2402.13144[cs.LG]

[2] YutaoSunetal. RetentiveNetwork:ASuccessortoTransformerforLargeLanguage Models .2023.arXiv: 2307.08621[cs.CL].

[3] BoPengetal. RWKV:ReinventingRNNsfortheTransformerEra .2023.arXiv: 2305. 13048[cs.CL].

[4] AlbertGuandTriDao. Mamba:Linear-TimeSequenceModelingwithSelectiveState Spaces .2023.arXiv: 2312.00752[cs.LG].

[5] MeredithRingelMorrisetal. LevelsofAGI:OperationalizingProgressonthePathtoAGI. 2023.arXiv: 2311.02462[cs.AI].

[6] BlaiseAgüerayArcasandPeterNorvig.“ArtificialGeneralIntelligenceIsAlreadyHere”. In: NoemaMagazine (Oct.2023).URL: https://www.noemamag.com/artificialgeneral-intelligence-is-already-here/

[7] RafaelRafailovetal. DirectPreferenceOptimization:YourLanguageModelisSecretlya RewardModel.2023.arXiv: 2305.18290[cs.LG]

[8] MohammadGheshlaghiAzaretal. AGeneralTheoreticalParadigmtoUnderstandLearningfromHumanPreferences .2023.arXiv: 2310.12036[cs.AI].

[9] TomB.Brownetal. LanguageModelsareFew-ShotLearners .2020.arXiv: 2005.14165 [cs.CL].

[10] TeroKarras,SamuliLaine,andTimoAila. AStyle-BasedGeneratorArchitectureforGenerativeAdversarialNetworks .2019.arXiv: 1812.04948[cs.NE].

[11] RobinRombachetal. High-ResolutionImageSynthesiswithLatentDiffusionModels . 2022.arXiv: 2112.10752[cs.CV].

[12] AdityaRameshetal. HierarchicalText-ConditionalImageGenerationwithCLIPLatents 2022.arXiv: 2204.06125[cs.CV]

[13] WenhuiWangetal. ImageasaForeignLanguage:BEiTPretrainingforAllVisionand Vision-LanguageTasks .2022.arXiv: 2208.10442[cs.CV]

[14] WenhaiWangetal. InternImage:ExploringLarge-ScaleVisionFoundationModelswith DeformableConvolutions .2023.arXiv: 2211.05778[cs.CV].

[15] ChengyiWangetal. NeuralCodecLanguageModelsareZero-ShotTexttoSpeechSynthesizers .arXiv.Jan.2023.URL: https://www.microsoft.com/en-us/research/ publication/neural-codec-language-models-are-zero-shot-text-tospeech-synthesizers/.

[16] MatthewLeetal. Voicebox:Text-GuidedMultilingualUniversalSpeechGenerationat Scale .2023.arXiv: 2306.15687[eess.AS]

[17] AlexeiBaevskietal. wav2vec2.0:AFrameworkforSelf-SupervisedLearningofSpeech Representations .2020.arXiv: 2006.11477[cs.CL]

[18] HugoTouvronetal. LLaMA:OpenandEfficientFoundationLanguageModels .2023. arXiv: 2302.13971[cs.CL].

[19] AakankshaChowdheryetal. PaLM:ScalingLanguageModelingwithPathways .2022. arXiv: 2204.02311[cs.CL].

[20] OpenAI. GPT-4TechnicalReport.2023.arXiv: 2303.08774[cs.CL].

[21] EuropeanUnion.“Directive(EU)2022/2555oftheEuropeanParliamentandoftheCouncilof14December2022onmeasuresforahighcommonlevelofcybersecurityacross theUnion,amendingRegulation(EU)No910/2014andDirective(EU)2018/1972,and repealingDirective(EU)2016/1148(NIS2Directive)”.In: OJEUL333 65(Dec.27,2022), pp.80–152.

[22] WaddahSaeedandChristianOmlin.“ExplainableAI(XAI):Asystematicmeta-surveyof currentchallengesandfutureopportunities”.In: Knowledge-BasedSystems 263(2023), p.110273.ISSN:0950-7051.DOI: https://doi.org/10.1016/j.knosys.2023. 110273.URL: https://www.sciencedirect.com/science/article/pii/S0950705123000230

[23] LucaNannini,AgatheBalayn,andAdamLeonSmith.“ExplainabilityinAIPolicies:ACriticalReviewofCommunications,Reports,Regulations,andStandardsintheEU,US,and UK”.In: Proceedingsofthe2023ACMConferenceonFairness,Accountability,andTransparency .FAccT’23.NewYork,NY,USA:AssociationforComputingMachinery,2023, pp.1198–1212.DOI: 10.1145/3593013.3594074.URL: https://doi.org/10.1145/ 3593013.3594074.

[24] NagadivyaBalasubramaniametal.“TransparencyandexplainabilityofAIsystems:From ethicalguidelinestorequirements”.In: InformationandSoftwareTechnology 159(2023), p.107197.ISSN:0950-5849.DOI: https://doi.org/10.1016/j.infsof.2023. 107197.URL: https://www.sciencedirect.com/science/article/pii/S0950584923000514

[25] OECD. OECDLegalInstruments.RecommendationoftheCouncilonArtificialIntelligence.OECD/LEGAL/0449.Adoptedon:22.05.2019.Amendedon:08.11.2023 .Nov.2023. URL: https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449

[26] FeiyuXuetal.“ExplainableAI:ABriefSurveyonHistory,ResearchAreas,Approaches andChallenges”.In:Sept.2019,pp.563–574.ISBN:978-3-030-32235-9.DOI: 10.1007/ 978-3-030-32236-6_51

[27] ChristophMolnar. InterpretableMachineLearning:AGuideForMakingBlackBoxModels Explainable .Paperback.Feb.2022.

[28] JasonWeietal. EmergentAbilitiesofLargeLanguageModels .2022.arXiv: 2206.07682 [cs.CL].

[29] ShengLuetal. AreEmergentAbilitiesinLargeLanguageModelsjustIn-ContextLearning? 2023.arXiv: 2309.01809[cs.CL].

[30] RajatRaina,AnandMadhavan,andAndrewNg.“Large-scaledeepunsupervisedlearning usinggraphicsprocessors”.In:vol.382.June2009,p.110.DOI: 10.1145/1553374. 1553486.

[31] AmirGholamietal. ASurveyofQuantizationMethodsforEfficientNeuralNetworkInference .2021.arXiv: 2103.13630[cs.CV].

[32] AlbertTsengetal. Quip#:QuipwithLatticeCodebooks .Dec.2023.

[33] UrielSingeretal. Make-A-Video:Text-to-VideoGenerationwithoutText-VideoData 2022.arXiv: 2209.14792[cs.CV]

[34] LevonKhachatryanetal. Text2Video-Zero:Text-to-ImageDiffusionModelsareZeroShotVideoGenerators .2023.arXiv: 2303.13439[cs.CV]

[35] AndreasBlattmannetal. StableVideoDiffusion:ScalingLatentVideoDiffusionModels toLargeDatasets .2023.arXiv: 2311.15127[cs.CV].

[36] JoonSungParketal. GenerativeAgents:InteractiveSimulacraofHumanBehavior.2023. arXiv: 2304.03442[cs.HC].

[37] EdwardJ.Huetal. LoRA:Low-RankAdaptationofLargeLanguageModels .2021.arXiv: 2106.09685[cs.CL].

[38] YuanzhiLietal.“TextbooksAreAllYouNeedII:phi-1.5technicalreport”.Sept.2023. URL: https://www.microsoft.com/en-us/research/publication/textbooksare-all-you-need-ii-phi-1-5-technical-report/.

[39] GoogleDeepMindGemmaTeam. Gemma:OpenModelsBasedonGeminiResearchand Technology .2024.URL: https://storage.googleapis.com/deepmind-media/ gemma/gemma-report.pdf.

[40] AlbertQ.Jiangetal. Mistral7B .2023.arXiv: 2310.06825[cs.CL]

[41] JinzeBaietal. QwenTechnicalReport.2023.arXiv: 2309.16609[cs.CL].

[42] BenSorscheretal. Beyondneuralscalinglaws:beatingpowerlawscalingviadatapruning .2023.arXiv: 2206.14486[cs.LG].

[43] Europol. Facingreality?Lawenforcementandthechallengeofdeepfakes.AnObservatoryReportfromtheEuropolInnovationLab. 2022.DOI: 10.2813/158794|QL-02-24129-EN-N.URL: https://www.europol.europa.eu/cms/sites/default/files/ documents/Europol_Innovation_Lab_Facing_Reality_Law_Enforcement_And_ The_Challenge_Of_Deepfakes.pdf.

[44] ParthaPratimRay.“ChatGPT:Acomprehensivereviewonbackground,applications,key challenges,bias,ethics,limitationsandfuturescope”.In: InternetofThingsandCyberPhysicalSystems 3(2023),pp.121–154.ISSN:2667-3452.DOI: https://doi.org/10. 1016/j.iotcps.2023.04.003.URL: https://www.sciencedirect.com/science/ article/pii/S266734522300024X

[45] IndependentHigh-LevelExpertGrouponAIsetupbytheEuropeanCommissioninJune 2018. EthicsguidelinesfortrustworthyAI.2019.URL: https://digital-strategy. ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai

[46] Forbes. 24TopAIStatisticsAndTrendsIn2023 .Apr.2023.URL: https://www.forbes. com/advisor/business/ai-statistics/%5C#sources_section

[47] Statistikaamet[StatisticsEstonia]. Tehisintellektitehnoloogiatekasutamineettevõtetes ontõusutrendis[UseofAItechnologiesbybusinessesisontherise].Sept.2023.URL: https://www.stat.ee/et/uudised/tehisintellekti-tehnoloogiate-kasutamineettevotetes-tousutrendis

[48] AdaLovelaceInstituteandTheAlanTuringInstitute. HowdopeoplefeelaboutAI?A nationallyrepresentativesurveyofpublicattitudestoartificialintelligenceinBritain .June 2023.URL: https://www.turing.ac.uk/sites/default/files/2023-06/how% 5C_do%5C_people%5C_feel%5C_about%5C_ai%5C_-%5C_ada%5C_turing.pdf

[49] Euractiv. EUtopcourt’srulingspellstroubleforscoringalgorithms .Dec.2023.URL: https://www.euractiv.com/section/data-privacy/news/eu-top-courtsruling-spill-trouble-for-scoring-algorithms/

[50] Politico. DutchscandalservesasawarningforEuropeoverrisksofusingalgorithms . Mar.2022.URL: https://www.politico.eu/article/dutch-scandal-servesas-a-warning-for-europe-over-risks-of-using-algorithms/

[51] MaxiScherer.“’ArtificialIntelligenceandLegalDecision-Making:TheWideOpen?’”In: JournalofInternationalArbitration 36(52019),pp.539–573.URL: https://kluwerlawonline. com/journalarticle/Journal+of+International+Arbitration/36.5/JOIA2019028.

[52] MajaBrkan.“Opinions.ArtificialIntelligenceandJudicialDecision-Making”.In: European DataProtectionLawReview 9.3(2023).DOI: 10.21552/edpl/2023/3/5.URL: https: //doi.org/10.21552/edpl/2023/3/5.

[53] MatthewDahletal. LargeLegalFictions:ProfilingLegalHallucinationsinLargeLanguage Models .2024.eprint: 2401.01301.

[54] MatthewDahletal. HallucinatingLaw:LegalMistakeswithLargeLanguageModelsare Pervasive .Jan.2024.URL: https://hai.stanford.edu/news/hallucinatinglaw-legal-mistakes-large-language-models-are-pervasive.

[55] ShaunLim.“Judicialdecision-makingandexplainableartificialintelligence.AReckoning fromFirstPrinciples”.In: SAcLj 280(2021).URL: https://law.nus.edu.sg/trail/ wp-content/uploads/sites/9/2022/03/9777_09.-Shaun-Lim-JudicialDecision-Making-and-Explainable-AI.pdf.

[56] EuropeanCommission. EthicsguidelinesfortrustworthyAI.URL: https://digitalstrategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai.

[57] “ProposalforaregulationoftheEuropeanOarliamentandoftheCouncillayingdownharmonisedrulesonartificialintelligence(ArtificialIntelligenceAct)andamendingcertain unionlegislativeacts”.In:().URL: https://eur-lex.europa.eu/legal-content/ EN/TXT/?uri=CELEX:52021PC0206.

[58] EuropeanParliament. EUAIAct:firstregulationonartificialintelligence .June2023.URL: https://www.europarl.europa.eu/topics/en/article/20230601STO93804/ eu-ai-act-first-regulation-on-artificial-intelligence.

[59] EuropeanUnion.“Regulation(EU)2016/679oftheEuropeanParliamentandoftheCouncilof27April2016ontheprotectionofnaturalpersonswithregardtotheprocessingof personaldataandonthefreemovementofsuchdata,andrepealingDirective95/46/EC (GeneralDataProtectionRegulation)”.In: OfficialJournalL119 59(May4,2016),pp.1–88.

[60] TheWhiteHouse. FactSheet:PresidentBidenIssuesExecutiveOrderonSafe,Secure, andTrustworthyArtificialIntelligence .Oct.2023.URL: https://www.whitehouse. gov/briefing-room/statements-releases/2023/10/30/fact-sheet-presidentbiden-issues-executive-order-on-safe-secure-and-trustworthy-artificialintelligence/.

[61] UKParliament. ParliamentaryBills .2023.URL: https://bills.parliament.uk/ bills/3464.

[62] UKParliament. Artificialintelligenceandemploymentlaw .URL: https://commonslibrary. parliament.uk/research-briefings/cbp-9817/.

[63] OfficialWebsiteoftheInternationalTradeAdministration. UKAIregulations2023 .2023. URL: https://www.gov.uk/government/publications/ai-regulation-a-proinnovation-approach.

[64] AustralianGovernment. SupportingresponsibleAI:discussionpaper.June2023.URL: https://consult.industry.gov.au/supporting-responsible-ai.

[65] AustralianGovernment. PositioningAustraliaasaleaderindigitaleconomyregulation. AutomatedDecisionMakingandAIRegulation.IssuesPaper.Mar.2023.URL: https:// storage.googleapis.com/converlens-au-industry/industry/p/prj211c4e81fb27d147ec9c1/ public%5C_assets/automated-decision-making-ai-regulation-issuespaper.pdf.

[66] Dentons. AustralianGovernmentrequestspublicfeedbackonregulating‘SafeandResponsibleAI’.June2023.URL: https://www.dentons.com/en/insights/alerts/ 2023/june/6/australian-government-requests-public-feedback-on-regulatingsafe.

[67] eSafetyCommissioner. TechTrendsPositionStatement.GenerativeAI.URL: https: //www.esafety.gov.au/sites/default/files/2023-08/Generative%5C%20AI% 5C%20-%5C%20Position%5C%20Statement%5C%20-%5C%20August%5C%202023%5C% 20.pdf

[68] AustraliatorequireAI-madechildabusematerialberemovedfromsearchresults .Sept. 2023.URL: https://www.reuters.com/technology/australia-require-aimade-child-abuse-material-be-removed-search-results-2023-09-08/

[69] TheGuardian. SearchenginesrequiredtostampoutAI-generatedimagesofchildabuse underAustralia’snewcode .June2023.URL: https://www.theguardian.com/ technology/2023/sep/08/search-engines-required-to-stamp-out-aigenerated-images-of-child-abuse-under-australias-new-code

[70] GovernmentofCanada. ArtificialIntelligenceandDataAct.Sept.2023.URL: https: //ised-isde.canada.ca/site/innovation-better-canada/en/artificialintelligence-and-data-act.

[71] GovernmentofCanadalaunchesconsultationontheimplicationsofgenerativeartificial intelligenceforcopyright.Oct.2023.URL: https://www.canada.ca/en/innovationscience-economic-development/news/2023/10/government-of-canadalaunches-consultation-on-the-implications-of-generative-artificialintelligence-for-copyright.html.

[72] RunwayStrategies. GlobalAIRegulationTracker.Dec.2023.URL: https://www.theguardian. com/technology/2023/sep/08/search-engines-required-to-stamp-out-aigenerated-images-of-child-abuse-under-australias-new-code.

[73] ISO. ISO/IEC22989:2022 .2022.URL: https://www.iso.org/standard/74296.html.

[74] InternationalElectrotechnicalCommission. Twonewfoundationalstandardsforartificial intelligence .July2022.URL: https://www.iec.ch/blog/two-new-foundationalstandards-artificial-intelligence.

[75] ISO. ISO/IEC23053:2022 .July2022.URL: https://www.iso.org/standard/74438. html.

[76]

[77]

[78]

ISO. ISO/IECFDIS5259-1.Artificialintelligence.Dataqualityforanalyticsandmachine learning(ML).Part1:Overview,terminology,andexamples .URL: https://www.iso. org/standard/81088.html.

ISO. ISO/IECDIS5259-2.Artificialintelligence.Dataqualityforanalyticsandmachine learning(ML).Part2:Dataqualitymeasures .URL: https://www.iso.org/standard/ 81860.html.

ISO. ISO/IECTS4213:2022.Informationtechnology.Artificialintelligence.Assessmentof machinelearningclassificationperformance .URL: https://www.iso.org/standard/ 79799.html.

[79]

ISO. BS30440:2023.Validationframeworkfortheuseofartificialintelligence(AI)within healthcare.Specification.Current.Published:31Jul2023 .July2023.URL: https:// knowledge.bsigroup.com/products/validation-framework-for-the-use-ofartificial-intelligence-ai-within-healthcare-specification.

[80] IEEE. IEEEStandardsAssociation.Ethicallyaligneddesign,Version1,Translationsand reports .URL: https://standards.ieee.org/industry-connections/ec/eadv1/.

[81] Google. GoogleAI.Responsibility:Ourprinciples .URL: https://ai.google/responsibility/ principles/.

[82] Google. GoogleAI.Responsibility:ResponsibleAIpracticess .URL: https://ai.google/ responsibility/responsible-ai-practices/.

[83] Microsoft. MicrosoftResponsibleAIStandard,v2.GeneralRequirements.Forexternal release.June2022 .July2022.URL: https://blogs.microsoft.com/wp-content/ uploads/prod/sites/5/2022/06/Microsoft-Responsible-AI-Standard-v2General-Requirements-3.pdf.

[84] OECD. MultilayerFrameworkforGoodCybersecurityPracticesforAI.June2023.URL: https://www.enisa.europa.eu/publications/multilayer-framework-forgood-cybersecurity-practices-for-ai.

[85] EuropeanCommission. EthicsguidelinesfortrustworthyAI.Apr.2019.URL: https:// digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthyai.

[86] IndependentHigh-LevelExpertGrouponAIsetupbytheEuropeanCommissioninJune 2018. EthicsguidelinesfortrustworthyAI.Apr.2019.URL: https://digital-strategy. ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai.

[87] A.Vassilevetal. AdversarialMachineLearning:ATaxonomyandTerminologyofAttacksandMitigations.(NationalInstituteofStandardsandTechnology,Gaithersburg, MD)NISTArtificialIntelligence(AI)Report,NISTTrustworthyandResponsibleAINIST AI100-2e2023. 2024.DOI: https://doi.org/10.6028/NIST.AI.100-2e2023.

[88] EuropeanCommission. High-levelexpertgrouponartificialintelligence .URL: https: //digital-strategy.ec.europa.eu/en/policies/expert-group-ai.

[89] High-levelexpertgrouponartificialintelligence. PolicyandinvestmentrecommendationsfortrustworthyArtificialIntelligence .June2019.URL: https://digital-strategy. ec.europa.eu/en/library/policy-and-investment-recommendations-trustworthyartificial-intelligence.

[90] High-levelexpertgrouponartificialintelligence. TheAssessmentListforTrustworthy ArtificialIntelligence(ALTAI)forselfassessment.July2020.URL: https://digitalstrategy.ec.europa.eu/en/library/assessment-list-trustworthy-artificialintelligence-altai-self-assessment.

[91] EuropeanAIAlliance. WelcometotheALTAIportal! URL: https://futurium.ec. europa.eu/en/european-ai-alliance/pages/welcome-altai-portal.

[92] High-levelexpertgrouponartificialintelligence. AIHLEG-SectoralConsiderationson PolicyandInvestmentRecommendationsforTrustworthyAI.July2020.URL: https: //futurium.ec.europa.eu/en/european-ai-alliance/document/ai-hlegsectoral-considerations-policy-and-investment-recommendations-trustworthyai.

[93] EuropeanCommission. ReportfromtheCommissiontotheEuropeanParliament,the CouncilandtheEuropeanEconomicandSocialCommittee.Reportonthesafetyand liabilityimplicationsofArtificialIntelligence,theInternetofThingsandrobotics.Brussels, 19.2.2020,COM(2020)64final.Feb.2020.URL: https://eur-lex.europa.eu/ legal-content/EN/TXT/?uri=CELEX%5C%3A52020DC0064.

[94] EuropeanCommission. WhitePaperOnArtificialIntelligence-AEuropeanapproachto excellenceandtrust.Brussels,19.2.2020,COM(2020)65final.Feb.2020.URL: https: //eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%5C%3A52020DC0065& qid=1664351767552.

[95] EuropeanParliament.“hapingthedigitaltransformation:EUstrategyexplained”.In:(). URL: https://www.europarl.europa.eu/topics/en/article/20210414STO02010/ shaping-the-digital-transformation-eu-strategy-explained.

[96] EuropeanCommission.“CommissionwelcomespoliticalagreementonArtificialIntelligenceAct”.In:(Dec.2023).URL: https://www.europarl.europa.eu/legislativetrain/theme-a-europe-fit-for-the-digital-age/file-regulation-onartificial-intelligence.

[97] EuropeanParliament.“LegislativeTrainSchedule.Artificialintelligenceact”.In:().URL: https://www.europarl.europa.eu/legislative-train/theme-a-europe-fitfor-the-digital-age/file-regulation-on-artificial-intelligence.

[98] LucaBertuzzi. EUcountriesgivecrucialnodtofirst-of-a-kindArtificialIntelligencelaw . Feb.2024.URL: https://www.euractiv.com/section/artificial-intelligence/ news/eu-countries-give-crucial-nod-to-first-of-a-kind-artificialintelligence-law/

[99] Analysisofthefinalcompromisetextwithaviewtoagreement.InterinstitutionalFile: 2021/0106(COD).No.Ciondoc.:8115/21.Jan.2024.URL: https://data.consilium. europa.eu/doc/document/ST-5662-2024-INIT/en/pdf

[100] EuropeanCommission. EuropeanAIOffice .2024.URL: https://digital-strategy. ec.europa.eu/en/policies/ai-office.

[101] EuropeanCommission. ProposalforaDirectiveoftheEuropeanParliamentandofthe Councilonadaptingnon-contractualcivilliabilityrulestoartificialintelligence(AILiabilityDirective).Brussels,28.9.2022.COM(2022)496final.2022/0303(COD).Sept.2022. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX: 52022PC0496&from=EN

[102] EuropeanUnion.“Regulation(EU)2023/988ongeneralproductsafety,amendingRegulation(EU)No1025/2012andDirective(EU)2020/1828,andrepealingDirective2001/95/EC andDirective87/357/EEC”.In: OJEUL135 66(May23,2023),pp.1–51.

[103] EuropeanCommission. ProposalforaDirectiveoftheEuropeanParliamentandofthe Councilonliabilityfordefectiveproducts.Brussels,28.9.2022.COM(2022)495final. 2022/0302(COD).Sept.2022.URL: https://eur-lex.europa.eu/legal-content/ EN/TXT/HTML/?uri=CELEX:52022PC0495

[104] JudeUme.“WhatAIMeansforIntellectualProperty”.In: ITNOW 65.2(May2023),pp.44–45.ISSN:1746-5702.DOI: 10.1093/combul/bwad059.eprint: https://academic. oup.com/itnow/article-pdf/65/2/44/50367250/bwad059.pdf.URL: https: //doi.org/10.1093/combul/bwad059

[105] PeterGeorgPichtandFlorentThouvenin.“AIandIP:TheorytoPolicyandBackAgain–PolicyandResearchRecommendationsattheIntersectionofArtificialIntelligenceand IntellectualProperty”.In: IIC-InternationalReviewofIntellectualPropertyandCompetitionLaw 54.6(July1,2023),pp.916–940.DOI: 10.1007/s40319-023-01344-5.URL: https://doi.org/10.1007/s40319-023-01344-5.

[106] WinstonCho. AICompaniesTakeHitasJudgeSaysArtistsHave“PublicInterest”In PursuingLawsuitsAfederaljudgerejectedargumentsfromStabilityAI,Midjourneyand DeviantArtthatthesuitisintendedtosuppressitsfreespeech .Feb.2024.URL: https: //www.hollywoodreporter.com/business/business-news/artist-lawsuitai-midjourney-art-1235821096/

[107] BobbyAllyn. ’NewYorkTimes’considerslegalactionagainstOpenAIascopyrighttensionsswirl.Aug.2023.URL: https://www.npr.org/2023/08/16/1194202562/ new-york-times-considers-legal-action-against-openai-as-copyrighttensions-swirl

[108] WillBedingfield. TheInventorBehindaRushofAICopyrightSuitsIsTryingtoShowHis BotIsSentient.StephenThaler’sseriesofhigh-profilecopyrightcaseshasmadeheadlinesworldwide.He’sdoneittodemonstratehisAIiscapableofindependentthought Sept.2023.URL: https://www.wired.com/story/the-inventor-behind-a-rushof-ai-copyright-suits-is-trying-to-show-his-bot-is-sentient/

[109] EuropeanParliament. WorldIntellectualPropertyOrganization.WIPOCONVERSATION ONINTELLECTUALPROPERTY(IP)ANDARTIFICIALINTELLIGENCE(AI).WIPO/IP/AI/3/GE/20/INF/5 Jan.2021.URL: https://www.wipo.int/edocs/mdocs/mdocs/en/wipo_ip_ai_3_ ge_20/wipo_ip_ai_3_ge_20_inf_5.pdf.

[110] S.SuganyaandE.Prema.“Humancentricintellectualpropertyrightsandlegalpersonalityofartificialintelligence”.In: InternationalJournalofIntellectualPropertyManagement 13.3-4(2023),pp.252–267.DOI: 10.1504/IJIPM.2023.134051.eprint: https: //www.inderscienceonline.com/doi/pdf/10.1504/IJIPM.2023.134051.URL: https://www.inderscienceonline.com/doi/abs/10.1504/IJIPM.2023.134051.

[111] AtifAziz.“ArtificialIntelligenceProducedOriginalWork:ANewApproachtoCopyright ProtectionandOwnership”.In: EuropeanJournalofArtificialIntelligenceandMachine Learning 2.2(Mar.2023),pp.9–16.DOI: 10.24018/ejai.2023.2.2.15.URL: https: //www.ej-ai.org/index.php/ejai/article/view/15.

[112] MauritzKop.“TAI&IntellectualProperty:TowardsanArticulatedPublicDomain”.In: UniversityofTexasSchoolofLaw,TexasIntellectualPropertyLawJournal(TIPLJ) 28.1(June 2019),pp.44–45.ISSN:1746-5702.DOI: http://dx.doi.org/10.2139/ssrn. 3409715.eprint: https://academic.oup.com/itnow/article-pdf/65/2/44/ 50367250/bwad059.pdf.

[113] WinstonCho. AI-CreatedArtIsn’tCopyrightable,JudgeSaysinRulingThatCouldGive HollywoodStudiosPause.AfederaljudgeonFridayupheldafindingfromtheU.S.CopyrightOfficethatapieceofartcreatedbyAIisnotopentoprotection .Aug.2023.URL: https://www.hollywoodreporter.com/business/business-news/ai-worksnot-copyrightable-studios-1235570316/

[114] EuropeanParliament. EuropeanParliamentresolutionof20October2020onintellectual propertyrightsforthedevelopmentofartificialintelligencetechnologies(2020/2015(INI)) Oct.2020.URL: https://www.europarl.europa.eu/doceo/document/TA-9-20200277_EN.html

[115] EuropeanUnion.“Regulation(EU)2019/881oftheEuropeanParliamentandoftheCouncilof17April2019onENISA(theEuropeanUnionAgencyforCybersecurity)andon informationandcommunicationstechnologycybersecuritycertificationandrepealing Regulation(EU)No526/2013(CybersecurityAct)”.In: OJEUL151 62(June7,2019), pp.15–69.

[116] CentreforCybersecurityBelgium. TheNIS2Directive:Whatdoesitmeanformyorganization? 2024.URL: https://ccb.belgium.be/en/nis-2-directive-what-doesit-mean-my-organization

[117] EuropeanCommission. ProposalforaRegulationoftheEuropeanParliamentandof theCouncilonhorizontalcybersecurityrequirementsforproductswithdigitalelements andamendingRegulation(EU)2019/1020.Brussels,15.9.2022.COM(2022)454final. 2022/0272(COD).Sept.2022.URL: https://eur-lex.europa.eu/legal-content/ EN/TXT/HTML/?uri=CELEX:52022PC0454

[118] EUPresidency. ProposalforaRegulationoftheEuropeanParliamentandoftheCouncil layingdownharmonisedrulesonartificialintelligence(ArtificialIntelligenceAct)and amendingcertainUnionlegislativeacts.Analysisofthefinalcompromisetextwitha viewtoagreement.Brussels,26January2024.InterinstitutionalFile:2021/0106(COD). No.Ciondoc.:8115/21.Jan.2024.URL: https://data.consilium.europa.eu/doc/ document/ST-5662-2024-INIT/en/pdf

[119] EuropeanCommission. ProposalforaRegulationoftheEuropeanParliamentandofthe CouncillayingdownadditionalproceduralrulesrelatingtotheenforcementofRegulation (EU)2016/679.Brussels,4.7.2023.COM(2023)348final.2023/0202(COD).July2023. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX: 52023PC0348.

[120] EuropeanUnion.“Directive(EU)2016/680oftheEuropeanParliamentandoftheCouncilof27April2016ontheprotectionofnaturalpersonswithregardtotheprocessing ofpersonaldatabycompetentauthoritiesforthepurposesoftheprevention,investigation,detectionorprosecutionofcriminaloffencesortheexecutionofcriminalpenalties,andonthefreemovementofsuchdata,andrepealingCouncilFrameworkDecision 2008/977/JHA”.In: OJEUL119 59(May4,2016),pp.89–131.

[121] EuropeanUnion.“Regulation(EU)2018/1725oftheEuropeanParliamentandoftheCouncilof23October2018ontheprotectionofnaturalpersonswithregardtotheprocessing ofpersonaldatabytheUnioninstitutions,bodies,officesandagenciesandonthefree movementofsuchdata,andrepealingRegulation(EC)No45/2001andDecisionNo 1247/2002/EC”.In: OJEUL295 61(Nov.21,2018),pp.39–98.

[122] N.AllahRakha. AIandtheLaw:UnravelingtheComplexitiesofRegulatoryFrameworks inEurope .Aug.2023.URL: https://irshadjournals.com/index.php/ibys/ article/view/115/102.

[123] AI:ensuringGDPRcompliance .Sept.2022.URL: https://www.cnil.fr/en/aiensuring-gdpr-compliance.

[124] InformationCommissioner’sOffice. GuidanceonAIanddataprotection .URL: https: //ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/ artificial-intelligence/guidance-on-ai-and-data-protection/.

[125]

[126]

InformationCommissioner’sOffice. InformationCommissioner’sOfficelaunchesconsultationseriesongenerativeAI.Jan.2024.URL: https://ico.org.uk/about-theico/media-centre/news-and-blogs/2024/01/information-commissioner-soffice-launches-consultation-series-on-generative-ai/.

InformationCommissioner’sOffice. ICOconsultationseriesongenerativeAIanddata protection .Jan.2024.URL: https://ico.org.uk/about-the-ico/ico-andstakeholder-consultations/ico-consultation-series-on-generative-aiand-data-protection/.

[127] CNIL. AI:ensuringGDPRcompliance .Sept.2021.URL: https://www.cnil.fr/en/aiensuring-gdpr-compliance.

[128] FederalTradeCommission. FTCTakesActionAgainstCompanyFormerlyKnownasWeight WatchersforIllegallyCollectingKids’SensitiveHealthData .Mar.2022.URL: https: //www.ftc.gov/news-events/news/press-releases/2022/03/ftc-takesaction-against-company-formerly-known-weight-watchers-illegallycollecting-kids-sensitive.

[129] CNIL. Artificialintelligence:CNILunveilsitsfirstanswersforinnovativeandprivacyfriendlyAI.Oct.2023.URL: https://www.cnil.fr/en/artificial-intelligencecnil-unveils-its-first-answers-innovative-and-privacy-friendly-ai.

[130] DanSvantesson. TheEuropeanUnionArtificialIntelligenceAct:Potentialimplicationsfor Australia .2022.DOI: https://doi.org/10.1177/1037969X211052339.

[131] AdèleAzzi.“TheChallengesFacedbytheExtraterritorialScopeoftheGeneralData ProtectionRegulation”.In: JIPITEC 9.2(2018),pp.126–137.ISSN:2190-3387.URL: http: //nbn-resolving.de/urn:nbn:de:0009-29-47231.

[132] MichalCzerniawskiandDanSvantesson.“Challengestotheextraterritorialenforcement ofdataprivacylaw-EUcasestudy”.In:Jan.2024,pp.127–153.

[133] FedericoFabbriniandEdoardoCeleste.“TheRighttoBeForgottenintheDigitalAge:The ChallengesofDataProtectionBeyondBorders”.In: GermanLawJournal 21.S1(2020), pp.55–65.DOI: 10.1017/glj.2020.14.

[134] ChrisBurt.“ClearviewdeniesjurisdictionofFrenchregulatorinresponseto€20Mfine”. In: BiometricUpdate (Oct.2022).URL: https://www.biometricupdate.com/202210/ clearview-denies-jurisdiction-of-french-regulator-in-response-toe20m-fine.

[135] IAPP. Towardarisk-basedapproach?Challengingthe’zerorisk’paradigmofEUDPAsin internationaldatatransfersandforeigngovernments’dataaccessschedule .Feb.2024. URL: https://iapp.org/news/a/towards-a-risk-based-approach-challengingthe-zero-risk-paradigm-of-eu-dpas-in-international-data-transfersand-foreign-governments-data-access/.

[136] EuropeanCommission. Adequacydecisions.HowtheEUdeterminesifanon-EUcountry hasanadequatelevelofdataprotection. URL: https://commission.europa.eu/law/ law-topic/data-protection/international-dimension-data-protection/ adequacy-decisions_en.

[137] EuropeanCommission. CommissionfindsthatEUpersonaldataflowscancontinuewith 11thirdcountriesandterritories .URL: https://ec.europa.eu/commission/presscorner/ detail/en/ip_24_161.

[138] EuropeanCommission. AdequacydecisionfortheEU-USDataPrivacyFramework .URL: https://commission.europa.eu/document/fa09cbad-dd7d-4684-ae60be03fcb0fddf_en.

[139] EDPB. Recommendations1/2022ontheApplicationforApprovalandontheelements andprinciplestobefoundinControllerBindingCorporateRules(Art.47GDPR).Adopted on20June2023. 2023.URL: https://edpb.europa.eu/system/files/202306/edpb_recommendations_20221_bcr-c_v2_en.pdf.

[140] AndmekaitseInspektsioon[DataProtectionInspectorate]. Isikuandmeteedastaminevälisriiki[Internationaltransferofpersonaldata].URL: https://www.aki.ee/isikuandmed/ andmetootlejale/isikuandmete-edastamine-valisriikikuidas-aru-saadami#edastamineameerika.

[141] EuropeanDataProtectionBoard. Guidelines05/2021ontheInterplaybetweentheapplicationofArticle3andtheprovisionsoninternationaltransfersasperChapterVofthe GDPR.Version2.0.Adopted14February2023 .Feb.2023.URL: https://www.edpb. europa.eu/system/files/2023-02/edpb_guidelines_05-2021_interplay_ between_the_application_of_art3-chapter_v_of_the_gdpr_v2_en_0.pdf.

[142] Riskmanagement—Guidelines .en.StandardISO31000:2018.InternationalOrganization forStandardization,2018.URL: https://www.iso.org/standard/65694.html

[143] RiskManagementFrameworkforInformationSystemsandOrganizations:ASystemLife CycleApproachforSecurityandPrivacy .en.StandardNISTSP800-37Rev.2.USNationalInstituteofStandardsandTechnology,2018.URL: https://csrc.nist.gov/ pubs/sp/800/37/r2/final.

[144] Informationtechnology—Informationsecurity,cybersecurityandprivacyprotection— Guidanceonmanaginginformationsecurityrisks .en.StandardISO/IEC27005:2022. InternationalOrganizationforStandardization,2022.URL: https://www.iso.org/ standard/80585.html

[145] NISTCybersecurityFramework1.1.en.StandardNISTCSFv.1.1.USNationalInstituteof StandardsandTechnology,2018.URL: https://www.nist.gov/cyberframework/ framework

[146] Informationtechnology—Artificialintelligence—Guidanceonriskmanagement.en. StandardISO/IEC23984:2023.InternationalOrganizationforStandardization,2023.URL: https://www.iso.org/standard/77304.html.

[147] RiigiInfosüsteemiAmet[InformationSystemAuthority]. Eestiinfoturbestandard(E-ITS) [EstonianInformationSecurityStandard].2023.URL: https://eits.ria.ee/.

[148] PillePullonen,RaimundasMatulevičius,andDanBogdanov.“PE-BPMN:Privacy-Enhanced BusinessProcessModelandNotation”.In: BusinessProcessManagement.SpringerInternationalPublishing,2017,pp.40–56.DOI: 10.1007/978-3-319-65000-5_3

[149] HMGovernment. SafetyandSecurityRisksofGenerativeArtificialIntelligenceto2025 . URL: https://assets.publishing.service.gov.uk/media/653932db80884d0013f71b15/ generative-ai-safety-security-risks-2025-annex-b.pdf.

[150] RichardFangetal. LLMAgentscanAutonomouslyHackWebsites .2024.arXiv: 2402. 06664[cs.CR].

[151] RiigiInfosüsteemiAmet[InformationSystemAuthority]. Eestiinfoturbestandardietalonturbekataloog[E-ITSBaselineSecurityCatalogue].2023.URL: https://eits.ria. ee/et/versioon/2023/eits-poohidokumendid/etalonturbe-kataloog.

[152] TheNewYorkTimes. TheTimesSuesOpenAIandMicrosoftOverA.I.UseofCopyrighted Work.MillionsofarticlesfromTheNewYorkTimeswereusedtotrainchatbotsthatnow competewithit,thelawsuitsaid .Dec.2023.URL: https://www.nytimes.com/2023/ 12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html.

[153] TechTarget. AIlawsuitsexplained:Who’sgettingsued?Authors,artistsandothersare filinglawsuitsagainstgenerativeAIcompaniesforusingtheirdatainbulktotrainAIsystemswithoutpermission .Jan.2024.URL: https://www.techtarget.com/whatis/ feature/AI-lawsuits-explained-Whos-getting-sued.

[154] TheFashionLaw. FromChatGPTtoGettyv.StabilityAI:ARunningListofKeyAI-Lawsuits URL: https://www.thefashionlaw.com/from-chatgpt-to-deepfake-creatingapps-a-running-list-of-key-ai-lawsuits/.

[155] CivilResolutionTribunalofBritishColumbia. Moffattv.AirCanada,2024BCCRT149 (CanLII).Feb.2024.URL: https://www.canlii.org/en/bc/bccrt/doc/2024/ 2024bccrt149/2024bccrt149.html.

[156] LeiHuangetal. ASurveyonHallucinationinLargeLanguageModels:Principles,Taxonomy,Challenges,andOpenQuestions .2023.arXiv: 2311.05232[cs.CL]

[157] RushebShahetal. ScalableandTransferableBlack-BoxJailbreaksforLanguageModels viaPersonaModulation .2023.arXiv: 2311.03348[cs.CL]

[158] XiangyuQietal. Fine-tuningAlignedLanguageModelsCompromisesSafety,EvenWhen UsersDoNotIntendTo! 2023.arXiv: 2310.03693[cs.CL].

[159] JoséLuisRicón. SetSailForFail?OnAIrisk .Nintil.Availableat https://nintil.com/ ai-safety/.Aug.2022.

[160] GovernmentOfficeofScience,UK. FutureRisksofFrontierAI.Tech.rep.Technology& ScienceInsightsandForesight,Oct.2023.

[161] JonasB.Sandbrink. Artificialintelligenceandbiologicalmisuse:Differentiatingrisksof languagemodelsandbiologicaldesigntools .2023.arXiv: 2306.13952[cs.CY]

[162] DaniilA.Boiko,RobertMacKnight,andGabeGomes. Emergentautonomousscientificresearchcapabilitiesoflargelanguagemodels .2023.arXiv: 2304.05332[physics.chemph]

[163] OpenAIResearchTeam. BuildinganEarlyWarningSystemforLLM-AidedBiologicalThreat Creation .OpenAI.URL: https://openai.com/research/building-an-earlywarning-system-for-llm-aided-biological-threat-creation (visitedon 02/08/2024).

[164] JosephR.BidenJr. ExecutiveOrderontheSafe,Secure,andTrustworthyDevelopment andUseofArtificialIntelligence .ExecutiveOrder.14110.2023.

[165] T.C.King,N.Aggarwal,M.Taddeo,etal.“ArtificialIntelligenceCrime:AnInterdisciplinary AnalysisofForeseeableThreatsandSolutions”.In: SciEngEthics 26(2020),pp.89–120. DOI: 10.1007/s11948-018-00081-0.

[166] BrigittaDresp-Langley.“Theweaponizationofartificialintelligence:Whatthepublicneeds tobeawareof”.In: FrontiersinArtificialIntelligence 6(2023),p.1154184.DOI: 10.3389/ frai.2023.1154184.

[167] MrinankSharmaetal. TowardsUnderstandingSycophancyinLanguageModels .2023. arXiv: 2310.13548[cs.CL].

[168] EmilyBaker-WhiteandForbesStaff.“WhoIs@BasedBeffJezos,TheLeaderOfTheTech Elite’s‘E/Acc’Movement?”In: Forbes (Dec.2023).Külastatud04.12.2023.URL: https:// www.forbes.com/sites/emilybaker-white/2023/12/01/who-is-basedbeffjezosthe-leader-of-effective-accelerationism-eacc/.

[169] MiladNasretal. ScalableExtractionofTrainingDatafrom(Production)LanguageModels .2023.arXiv: 2311.17035[cs.LG].

[170] IanJ.Goodfellow,JonathonShlens,andChristianSzegedy. ExplainingandHarnessing AdversarialExamples .2015.arXiv: 1412.6572[stat.ML]

[171] JingLinetal. MLAttackModels:AdversarialAttacksandDataPoisoningAttacks .2021. arXiv: 2112.02797[cs.LG].

[172] JonasGeipingetal. CoercingLLMstodoandreveal(almost)anything .2024.arXiv: 2402.14020[cs.LG].

[173] JosephLucas.“MitigatingStoredPromptInjectionAttacksAgainstLLMApplications”.In: NVIDIATechnicalBlog (Aug.2023).URL: https://developer.nvidia.com/blog/ mitigating-stored-prompt-injection-attacks-against-llm-applications/ [174] KaiGreshakeetal. Notwhatyou’vesignedupfor:CompromisingReal-WorldLLM-Integrated ApplicationswithIndirectPromptInjection .2023.arXiv: 2302.12173[cs.CR]

[175] DarynaOliynyk,RudolfMayer,andAndreasRauber.“IKnowWhatYouTrainedLastSummer:ASurveyonStealingMachineLearningModelsandDefences”.In: ACMComputing Surveys 55.14s(July2023),pp.1–41.ISSN:1557-7341.DOI: 10.1145/3595292.URL: http://dx.doi.org/10.1145/3595292.

[176] RezaShokrietal. MembershipInferenceAttacksagainstMachineLearningModels .2017. arXiv: 1610.05820[cs.CR].

[177] BorisvanBreugeletal. MembershipInferenceAttacksagainstSyntheticDatathrough OverfittingDetection .2023.arXiv: 2302.12580[cs.LG]

[178] Ngoc-BaoNguyenetal. Re-thinkingModelInversionAttacksAgainstDeepNeuralNetworks .2023.arXiv: 2304.01669[cs.LG].

[179] Kuan-ChiehWangetal. VariationalModelInversionAttacks .2022.arXiv: 2201.10787 [cs.LG].

[180] LiamFowletal. AdversarialExamplesMakeStrongPoisons .2021.arXiv: 2106.10807 [cs.LG].

[181] BattistaBiggio,BlaineNelson,andPavelLaskov. PoisoningAttacksagainstSupportVectorMachines .2013.arXiv: 1206.6389[cs.LG]

[182] ShawnShanetal. Prompt-SpecificPoisoningAttacksonText-to-ImageGenerativeModels .2023.arXiv: 2310.13828[cs.CR].

[183] MicahGoldblumetal. DatasetSecurityforMachineLearning:DataPoisoning,Backdoor Attacks,andDefenses .2021.arXiv: 2012.10544[cs.LG].

[184] XinyunChenetal. TargetedBackdoorAttacksonDeepLearningSystemsUsingData Poisoning .2017.arXiv: 1712.05526[cs.CR].

[185] RuixiangTangetal. AnEmbarrassinglySimpleApproachforTrojanAttackinDeepNeural Networks .2020.arXiv: 2006.08131[cs.CR]

[186] MeghaAgarwaletal. LLMInferencePerformanceEngineering:BestPractices https: //www.databricks.com/blog/llm-inference-performance-engineeringbest-practices.Accessedon11Dec2023.2023.

[187] JiaqiRuanetal. ApplyingLargeLanguageModelstoPowerSystems:PotentialSecurity Threats .2023.arXiv: 2311.13361[cs.AI].

[188] RobinStaabetal. BeyondMemorization:ViolatingPrivacyViaInferencewithLargeLanguageModels .2023.arXiv: 2310.07298[cs.AI]

[189] CyberneticaAS. Privaatsuskaitsetehnoloogiatekontseptsioon[Privacyenhancingtechnologyconcept.Tech.rep.Majandus-jaKommunikatsiooniministeerium[MinistryofEconomicAffairsandCommunications],2023.URL: https://www.kratid.ee/analuusidja-uuringud#pet.

[190] RouzbehBehniaetal.“EW-Tune:AFrameworkforPrivatelyFine-TuningLargeLanguage ModelswithDifferentialPrivacy”.In: 2022IEEEInternationalConferenceonDataMining Workshops(ICDMW).IEEE,Nov.2022.DOI: 10.1109/icdmw58026.2022.00078.URL: http://dx.doi.org/10.1109/ICDMW58026.2022.00078.

[191] OECD. Accountability(Principle1.5).URL: https://oecd.ai/en/dashboards/aiprinciples/P9.

[192] OECD. Inclusivegrowth,sustainabledevelopmentandwell-being(Principle1.1).URL: https://oecd.ai/en/dashboards/ai-principles/P5.

[193] OWASPFoundation. LLMAICybersecurity&GovernanceChecklist.2024.URL: https: //owasp.org/www-project-top-10-for-large-language-model-applications/ llm-top-10-governance-doc/LLM_AI_Security_and_Governance_Checklistv1.pdf.