SAEED GHAHRAMANI
Western New England University
Springfield, Massachusetts, USA
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2016 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Version Date: 20151005
International Standard Book Number-13: 978-1-4987-5502-3 (eBook - PDF)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com (http:// www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com
and the CRC Press Web site at http://www.crcpress.com
ToLili,Adam,andAndrew
This page intentionally left blank
1.1Introduction1
1.2SampleSpaceandEvents3
1.3AxiomsofProbability10
1.4BasicTheorems17
1.5ContinuityofProbabilityFunction25
1.6Probabilities0and127
1.7RandomSelectionofPointsfromIntervals28 ReviewProblems33
2CombinatorialMethods36
2.1Introduction36
2.2CountingPrinciple36
NumberofSubsetsofaSet 40 TreeDiagrams 40
2.3Permutations44
2.4Combinations50
2.5Stirling’sFormula66 ReviewProblems68
3ConditionalProbabilityandIndependence71
3.1ConditionalProbability71 ReductionofSampleSpace 75
3.2LawofMultiplication80
3.3LawofTotalProbability83
3.4Bayes’Formula94
3.5Independence102
3.6ApplicationsofProbabilitytoGenetics119 Hardy-WeinbergLaw 123 Sex-LinkedGenes 125 ReviewProblems128
4 DistributionFunctionsand DiscreteRandomVariables
4.1RandomVariables131
4.2DistributionFunctions135
4.3DiscreteRandomVariables144
4.4ExpectationsofDiscreteRandomVariables150
4.5VariancesandMomentsofDiscreteRandomVariables165 Moments 170
4.6StandardizedRandomVariables173
ReviewProblems174
5SpecialDiscreteDistributions177
5.1BernoulliandBinomialRandomVariables177
ExpectationsandVariancesofBinomialRandomVariables 183
5.2PoissonRandomVariable189
PoissonasanApproximationtoBinomial 189 PoissonProcess 194
5.3OtherDiscreteRandomVariables202
GeometricRandomVariable 202
NegativeBinomialRandomVariable 205
HypergeometricRandomVariable 207
ReviewProblems214
6ContinuousRandomVariables218
6.1ProbabilityDensityFunctions218
6.2DensityFunctionofaFunctionofaRandomVariable227
6.3ExpectationsandVariances232
ExpectationsofContinuousRandomVariables 232
VariancesofContinuousRandomVariables 238
ReviewProblems244
7SpecialContinuousDistributions246
7.1UniformRandomVariable246
7.2NormalRandomVariable252
CorrectionforContinuity 255
7.3ExponentialRandomVariables268
7.4GammaDistribution274
7.5BetaDistribution280
7.6SurvivalAnalysisandHazardFunction286 ReviewProblems290
8BivariateDistributions292
8.1JointDistributionofTwoRandomVariables292
JointProbabilityMassFunctions 292
JointProbabilityDensityFunctions 296
8.2IndependentRandomVariables310
IndependenceofDiscreteRandomVariables 310 IndependenceofContinuousRandomVariables 313
8.3ConditionalDistributions322
ConditionalDistributions:DiscreteCase 322
ConditionalDistributions:ContinuousCase 327
8.4TransformationsofTwoRandomVariables334 ReviewProblems342
9MultivariateDistributions346
9.1JointDistributionof n> 2 RandomVariables346
JointProbabilityMassFunctions 346
JointProbabilityDensityFunctions 354 RandomSample 358
9.2OrderStatistics363
9.3MultinomialDistributions369 ReviewProblems373
10MoreExpectationsandVariances375
10.1ExpectedValuesofSumsofRandomVariables375 PatternAppearance 382
10.2Covariance388
10.3Correlation402
10.4ConditioningonRandomVariables407
10.5BivariateNormalDistribution421 ReviewProblems425
11.1Moment-GeneratingFunctions428
11.2SumsofIndependentRandomVariables438
11.3MarkovandChebyshevInequalities446
Chebyshev’sInequalityandSampleMean 450
11.4LawsofLargeNumbers455
ProportionversusDifferenceinCoinTossing 463
11.5CentralLimitTheorem466
ReviewProblems475
12 StochasticProcesses
12.1Introduction478
12.2MoreonPoissonProcesses479
WhatIsaQueuingSystem? 490
PASTA:PoissonArrivalsSeeTimeAverage 491
12.3MarkovChains494
ClassificationsofStatesofMarkovChains 503
AbsorptionProbability 513 Period 517
Steady-StateProbabilities 518
12.4Continuous-TimeMarkovChains530
Steady-StateProbabilities 536
BirthandDeathProcesses 539
12.5BrownianMotion548
FirstPassageTimeDistribution 555
TheMaximumofaBrownianMotion 556
TheZerosofBrownianMotion 556
BrownianMotionwithDrift 559
GeometricBrownianMotion 560
ReviewProblems563 13Simulation568
13.1Introduction568
13.2SimulationofCombinatorialProblems572
13.3SimulationofConditionalProbabilities575
13.4SimulationofRandomVariables578
13.5MonteCarloMethod586
This page intentionally left blank
Preface
Thisone-ortwo-termbasicprobabilitytextiswrittenformajorsinmathematics,physicalsciences,engineering,statistics,actuarialscience,businessand finance,operations research,andcomputerscience.Itcanalsobeusedbystudentswhohavecompletedabasiccalculuscourse.Ouraimistopresentprobabilityinanaturalway:throughinteresting andinstructiveexamplesandexercisesthatmotivatethetheory,definitions,theorems,and methodology.Examplesandexerciseshavebeencarefullydesignedtoarousecuriosityand henceencouragethestudentstodelveintothetheorywithenthusiasm.
Thisisanewprintingofthethirdeditionof FundamentalsofProbabilitywithStochasticProcesses previouslypublishedbyPearson.Allthedetectedtypographicalandother errorshavebeencorrectedinthis Chapman&Hall/CRC edition.
Authorsareusuallyfacedwithtwoopposingimpulses.Oneisatendencytoputtoo muchintothebook,because everything isimportantand everything hastobesaidthe author’sway!Ontheotherhand,authorsmustalsokeepinmindacleardefinitionofthe focus,thelevel,andtheaudienceforthebook,therebychoosingcarefullywhatshould be“in”andwhat“out.”Hopefully,thisbook isanacceptableresolutionofthetension generatedbytheseopposingforces.
Instructorsshouldenjoytheversatilityofthistext.Theycanchoosetheirfavorite problemsandexercisesfromacollectionof1558and,ifnecessary,omitsomesections and/ortheoremstoteachatanappropriatelevel.
Exercisesformostsectionsaredividedintotwocategories:AandB.Thoseincategory Aareroutine,andthoseincategoryBarechallenging.However,notallexercisesincategoryBareuniformlychallenging.Someofthoseexercisesareincludedbecausestudents findthemsomewhatdifficult.
Ihavetriedtomaintainanapproachthatismathematicallyrigorousand,atthesame time,closelymatchesthehistoricaldevelopmentofprobability.Wheneverappropriate,I includehistoricalremarks,andalsoincludediscussionsofanumberofprobabilityproblemspublishedinrecentyearsinjournalssuchas MathematicsMagazine and American MathematicalMonthly. Theseareinterestingandinstructiveproblemsthatdeservediscussioninclassrooms.
Chapter13concernscomputersimulation.Thatchapterisdividedintoseveralsections, presentingalgorithmsthatareusedto findapproximatesolutionstocomplicatedprobabilisticproblems.Thesesectionscanbediscussedindependentlywhenrelevantmaterialsfrom earlierchaptersarebeingtaught,ortheycanbediscussedconcurrently,towardtheendof thesemester.AlthoughIbelievethattheemphasisshouldremainonconcepts,methodology,andthemathematicsofthesubject,Ialsothinkthatstudentsshouldbeaskedtoread thematerialonsimulationandperhapsdosomeprojects.Computersimulationisanexcellentmeanstoacquireinsightintothenatureofaproblem,itsfunctions,itsmagnitude,and thecharacteristicsofthesolution.
OtherContinuingFeatures
• Thehistoricalrootsandapplicationsofmanyofthetheoremsanddefinitionsare presentedindetail,accompaniedbysuitableexamplesorcounterexamples.
• Asmuchaspossible,examplesandexercisesforeachsectiondonotrefertoexercisesinotherchaptersorsections—astylethatoftenfrustratesstudentsandinstructors.
• Wheneveranewconceptisintroduced,itsrelationshiptoprecedingconceptsand theoremsisexplained.
• Althoughtheusualanalyticproofsaregiven,simpleprobabilisticargumentsarepresentedtopromotedeeperunderstandingofthesubject.
• Thebookbeginswithdiscussionsonprobabilityanditsdefinition,ratherthanwith combinatorics.Ibelievethatcombinatoricsshouldbetaughtafterstudentshave learnedthepreliminaryconceptsofprobability.Theadvantageofthisapproach isthattheneedformethodsofcountingwilloccurnaturallytostudents,andthe connectionbetweenthetwoareasbecomesclearfromthebeginning.Moreover, combinatoricsbecomesmoreinterestingandenjoyable.
• Studentsbeginningtheirstudyofprobabilityhaveatendencytothinkthatsample spacesalwayshavea finitenumberofsamplepoints.Tominimizethisproclivity,the conceptof randomselectionofapointfromaninterval isintroducedinChapter1 andappliedwhereappropriatethroughoutthebook.Moreover,sincethebasisof simulatingindeterministicproblemsisselectionofrandompointsfrom (0, 1),in ordertounderstandsimulations,studentsneedtobethoroughlyfamiliarwiththat concept.
• Often,whenwethinkofacollectionofevents,wehaveatendencytothinkabout themineithertemporalorlogicalsequence.So,if,forexample,asequenceofevents A1 , A2 , , An occurintimeorinsomelogicalorder,wecanusuallyimmediately writedowntheprobabilities
withoutmuchcomputation.However,wemaybeinterestedinprobabilitiesofthe intersectionofevents,orprobabilitiesofeventsunconditionalontherest,orprobabilitiesofearlierevents,givenlaterevents.Thesethreequestionsmotivatedtheneed forthelawofmultiplication,thelawoftotalprobability,andBayes’theorem.Ihave giventhelawofmultiplicationasectionofitsownsothat eachofthesefundamental usesofconditionalprobabilitywouldhaveitsfullshareofattentionandcoverage.
• Theconceptsofexpectationandvarianceareintroducedearly,becauseimportant conceptsshouldbedefinedandusedassoonaspossible.Onebenefitofthispractice isthat,whenrandomvariablessuchasPoissonandnormalarestudied,theassociated parameterswillbeunderstoodimmediatelyratherthanremainingambiguousuntil expectationandvarianceareintroduced.Therefore,fromthebeginning,students willdevelopanaturalfeelingaboutsuchparameters.
• SpecialattentionispaidtothePoissondistribution;itismadeclearthatthisdistributionisfrequentlyapplicable,fortworeasons: first,becauseitapproximatesthe
binomialdistributionand,second,itisthemathematicalmodelforanenormous classofphenomena.ThecomprehensivepresentationofthePoissonprocessandits applicationscanbeunderstoodbyjunior-andsenior-levelstudents.
• Studentsoftenhavedifficultiesunderstandingfunctionsorquantitiessuchasthe densityfunctionofacontinuousrandomvariableandtheformulaformathematical expectation.Forexample,theymaywonderwhy xf (x) dx istheappropriatedefinitionfor E (X ) andwhycorrectionforcontinuityisnecessary.Ihaveexplainedthe reasonbehindsuchdefinitions,theorems,andconcepts,andhavedemonstratedwhy theyarethenaturalextensionsofdiscretecases.
• The firstsixchaptersincludemanyexamplesandexercisesconcerningselectionof randompointsfromintervals.Consequently,inChapter7,whendiscussinguniform randomvariables,Ihavebeenabletocalculatethedistributionand(bydifferentiation)thedensityfunctionof X ,arandompointfromaninterval (a,b ).Inthisway theconceptofauniformrandomvariableandthedefinitionofitsdensityfunction arereadilymotivated.
• InChapters7and8theusefulnessofuniformdensitiesisshownbyusingmany examples.Inparticular,applicationsofuniformdensityin geometricprobability theory areemphasized.
• Normaldensity,arguablythemostimportantdensityfunction,isreadilymotivated byDeMoivre’stheorem.InSection7.2,Iintroducethestandardnormaldensity, theelementaryversionofthecentrallimittheorem,andthenormaldensityjustas theyweredevelopedhistorically.Experienceshowsthistobeagoodpedagogical approach.Whenteachingthisapproach,thenormaldensitybecomesnaturaland doesnotlooklikeastrangefunctionappearingoutoftheblue.
• Exponentialrandomvariablesnaturallyoccuras timesbetweenconsecutiveeventsof Poissonprocesses.Thetimeofoccurrenceofthe ntheventofaPoissonprocesshas agammadistribution.ForthesereasonsIhavemotivatedexponentialandgamma distributionsbyPoissonprocesses.Inthiswaywecanobtainmanyexamplesof exponentialandgammarandomvariablesfromtheabundantexamplesofPoisson processesalreadyknown.Anotheradvantageisthatithelpsusvisualizememoryless randomvariablesbylookingattheintereventtimesofPoissonprocesses.
• Jointdistributionsandconditioningareoftentroubleareasforstudents.Adetailed explanationandmanyapplicationsconcerningtheseconceptsandtechniquesmake thesematerialssomewhateasierforstudentstounderstand.
• Theconceptsofcovarianceandcorrelationaremotivatedthoroughly.
• AsubsectiononpatternappearanceispresentedinSection10.1.Eventhoughthe methoddiscussedinthissubsectionisintuitiveandprobabilistic,itshouldhelpthe studentsunderstandsuchparadoxical-lookingresultsasthefollowing.Ontheaverage,ittakesalmosttwiceasmany flipsofafaircointoobtainasequenceof five successiveheadsasitdoestoobtainatailfollowedbyfourheads.
• Theanswerstotheodd-numberedexercisesareincludedattheendofthebook.
NewToThisEdition
Since2000,whenthesecondeditionofthisbookwaspublished,Ihavereceivedmuchadditionalcorrespondenceandfeedbackfromfacultyandstudentsinthiscountryandabroad. Thecomments,discussions,recommendations,andreviewshelpedmetoimprovethebook inmanyways.Alldetectederrorswerecorrected,andthetexthasbeen fine-tunedforaccuracy.Moreexplanationsandclarifyingcommentshavebeenaddedtoalmosteverysection. Inthisedition,278newexercisesandexamples,mostlyofanappliednature,havebeen added.Moreinsightfulandbettersolutionsaregivenforanumberofproblemsandexercises.Forexample,IhavediscussedBorel’snormalnumbertheorem,andIhavepresented aversionofafamoussetwhichisnotanevent.Ifafaircoinistossedaverylargenumber oftimes,thegeneralperceptionisthatheadsoccursasoftenastails.Inanewsubsection, inSection11.4,Ihaveexplainedwhatismeantby“headsoccursasoftenastails.”
Someoftheotherfeaturesofthepresentrevisionarethefollowing:
• Anintroductorychapteronstochasticprocessesisadded.Thatchaptercoversmore in-depthmaterialonPoissonprocesses.ItalsopresentsthebasicsofMarkovchains, continuous-timeMarkovchains,andBrownianmotion.Thetopicsarecoveredin somedepth.Therefore,thecurrenteditionhasenoughmaterialforasecondcourse inprobabilityaswell.Thelevelofdifficultyofthechapteronstochasticprocesses isconsistentwiththerestofthebook.Ibelievetheexplanationsinthenewedition ofthebookmakesomechallengingmaterialmoreeasilyaccessibletoundergraduateandbeginninggraduatestudents.Weassumeonlycalculusasaprerequisite. Throughoutthechapter,asexamples,certainimportantresultsfromsuchareasas queuingtheory,randomwalks,branchingprocesses,superpositionofPoissonprocesses,andcompoundPoissonprocessesarediscussed.Ihavealsoexplainedwhat thefamoustheorem,PASTA, PoissonArrivalsSeeTimeAverage,states.Inshort,the chapteronstochasticprocessesislayingthefoundationonwhichstudents’further pureandappliedprobabilitystudiesandworkcanbuild.
• Somepractical,meaningful,nontrivial,andrelevantapplicationsofprobabilityand stochasticprocessesin finance,economics,andactuarialsciencesarepresented.
• Eversince1853,whenGregorJohannMendel(1822–1884)beganhisbreedingexperimentswiththegardenpea Pisumsativum,probabilityhasplayedanimportant roleintheunderstandingoftheprinciplesofheredity.Inthisedition,Ihaveincluded moregeneticsexamplestodemonstratetheextentofthatrole.
• Tostudytheriskorrateof“failure,”perunitoftimeof“lifetimes”thathavealready survivedacertainlengthoftime,Ihaveaddedanewsection,SurvivalAnalysisand HazardFunctions,toChapter7.
• Forrandomsumsofrandomvariables,IhavediscussedWald’sequationanditsanalogouscaseforvariance.CertainapplicationsofWald’sequationhavebeendiscussed intheexercises,aswellasinChapter12,StochasticProcesses.
• Tomaketheorderoftopicsmorenatural,thepreviouseditions’Chapter8isbroken intotwoseparatechapters,BivariateDistributionsandMultivariateDistributions. Asaresult,thesectionTransformationsofTwoRandomVariableshasbeencovered
earlieralongwiththematerialonbivariatedistributions,andtheconvolutiontheorem hasfoundabetterhomeasanexampleoftransformationmethods.Thattheoremis nowpresentedasamotivationforintroducingmoment-generatingfunctions,sinceit cannotbeextendedsoeasilytomanyrandomvariables.
SampleSyllabi
Foraone-termcourseonprobability,instructorshavebeenabletoomitmanysections withoutdifficulty.Thebookisdesignedforstudentswithdifferentlevelsofability,and avarietyofprobabilitycourses,appliedand/orpure,canbetaughtusingthisbook.A typicalone-semestercourseonprobabilitywouldcoverChapters1and2;Sections3.1–3.5;Chapters4,5,6;Sections7.1–7.4;Sections8.1–8.3;Section9.1;Sections10.1–10.3; andChapter11.
Afollow-upcourseonintroductorystochasticprocesses,oronamoreadvancedprobabilitywouldcovertheremainingmaterialinthebookwithanemphasisonSections8.4, 9.2–9.3,10.4and,especially,theentireChapter12.
Acourseon discreteprobability wouldcoverSections1.1–1.5;Chapters2,3,4,and 5;ThesubsectionsJointProbabilityMassFunctions,IndependenceofDiscreteRandom Variables,andConditionalDistributions:DiscreteCase,fromChapter8;thesubsection JointProbabilityMassFunctions,fromChapter9;Section9.3;selecteddiscretetopics fromChapters10and11;andSection12.3.
SolutionsManual
Ihavewrittenan Instructor’sSolutionsManual thatgivesdetailedsolutionstovirtuallyall ofthe1224exercisesofthebook.Thismanualisavailable,directlyfromPrenticeHall, onlyforthoseinstructorswhoteachtheircoursesfromthisbook.
Acknowledgments
Whilewritingthemanuscript,manypeoplehelpedmeeitherdirectlyorindirectly.Lili, mybelovedwife,deservesanaccoladeforherpatienceandencouragement;asdomy wonderfulchildren.
AccordingtoEcclesiastes12:12,“ofthemakingofbooks,thereisnoend.”Improvementsandadvancementtodifferentlevelsofexcellencecannotpossiblybeachievedwithoutthehelp,criticism,suggestions,andrecommendationsofothers.Ihavebeenblessed withsomanycolleagues,friends,andstudentswhohavecontributedtotheimprovement ofthistextbook.OnereasonIlikewritingbooksisthepleasureofreceivingsomanysuggestionsandsomuchhelp,support,andencouragementfromcolleaguesandstudentsall overtheworld.Myexperiencefromwritingthethreeeditionsofthisbookindicatesthat collaborationandcamaraderieinthescientificcommunityistrulyoverwhelming.
Forthethirdeditionofthisbookanditssolutionsmanual,mybrother,Dr.Soroush Ghahramani,aprofessorofarchitecturefromSinclairCollegeinOhio,usingAutoCad,
withutmostpatienceandmeticulosity,resketchedeachandeveryoneofthe figures.Asa result,theillustrationsaremore accurateandclearerthantheywereinthepreviouseditions. Iammostindebtedtomybrotherforhishardwork.
Forthethirdedition,Iwrotemanynew AMS-LATEX files.Myassistants,AnnGuyotte andAvrilCouture,withutmostpatience,keeneyes,positiveattitude,andeagernessput thesehand-written filesontothecomputer.Mycolleague,ProfessorAnnKizanis,whois knownforbeingaperfectionist,read,verycarefully,thesenew filesandmademanygood suggestions.Whilewritingabouttheapplicationofgeneticstoprobability,Ihadseveral discussionswithWesternNewEngland’sdistinguishedgeneticist,Dr.LorraineSartori.I learnedalotfromLorraine,whoalsoreadmymaterialongeneticscarefullyandmade valuablesuggestions.Dr.MichaelMeeropol,theChairofourEconomicsDepartment, readpartsofmymanuscriptson financialapplicationsandmentionedsomenewideas. Dr.DavidMazurwasteachingfrommybookevenbeforewe werecolleagues.Overthe pastfouryears,Ihaveenjoyedhearinghiscommentsandsuggestionsaboutmybook.It givesmeadistinctpleasuretothankAnnGuyotte,Avril,AnnKizanis,Lorraine,Michael, andDavefortheirhelp.
ProfessorJayDevorefromCaliforniaPolytechnicInstitute—SanLuisObispo,made excellentcommentsthatimprovedthemanuscriptsubstantiallyforthe firstedition.From BostonUniversity,ProfessorMarkE.Glickman’scarefulreviewandinsightfulsuggestions andideashelpedmeinwritingthesecondedition.Iwasveryluckytoreceivethorough reviewsofthethirdeditionfromProfessorJamesKuelbsofUniversityofWisconsin,Madison,ProfessorRobertSmitsofNewMexicoStateUniversity,andMs.EllenGundlachfrom PurdueUniversity.Thethoughtfulsuggestionsandideasofthesecolleaguesimprovedthe currenteditionofthisbookinseveralways.IammostgratefultoDrs.Devore,Glickman, Kuelbs,Smits,andMs.Gundlach.
Forthe firsttwoeditionsofthebook,mycolleaguesandfriendsatTowsonUniversityreadortaughtfromvariousrevisionsofthetextandofferedusefuladvice.Inparticular,IamgratefultoProfessorsMostafaAminzadeh,RaoufBoules,JeromeCohen, JamesP.Coughlin,GeoffreyGoodson,SharonJones,OhoeKim,BillRose,MarthaSiegel, HoushangSohrab,EricTissue,andmylatedearfriendSayeedKayvan.Iwanttothank mycolleaguesProfessorsCoughlinandSohrab,especially,fortheirkindnessandthegenerositywithwhichtheyspenttheirtimecarefullyreadingtheentiretexteverytimeitwas revised.
Iamalsogratefultothefollowingprofessorsfortheirvaluablesuggestionsandconstructivecriticisms:ToddArbogast,TheUniversityofTexasatAustin;RobertB.Cooper, FloridaAtlanticUniversity;RichardDeVault,NorthwesternStateUniversityofLouisiana; BobDillon,AuroraUniversity;DanFitzgerald,KansasNewmanUniversity;SergeyFomin, MassachusettsInstituteofTechnology;D.H.Frank,IndianaUniversityofPennsylvania; JamesFrykman,KentStateUniversity;M.LawrenceGlasser,ClarksonUniversity;Moe Habib,GeorgeMasonUniversity;PaulT.Holmes,ClemsonUniversity;EdwardKao,UniversityofHouston;JoeKearney,DavenportCollege;EricD.Kolaczyk,BostonUniversity; PhilippeLoustaunau,GeorgeMasonUniversity;JohnMorrison,UniversityofDelaware; ElizabethPapousek,FiskUniversity;RichardJ.Rossi,CaliforniaPolytechnicInstitute— SanLuisObispo;JamesR.Schott,UniversityofCentralFlorida;SiavashShahshahani, SharifUniversityofTechnology,Tehran,Iran;YangShangjun,AnhuiUniversity,Hefei,
China;KyleSiegrist,UniversityofAlabama—Huntsville;LorenSpice,myformeradvisee, aprodigywhobecameaPh.D.studentatage16andafacultymemberattheUniversity ofMichiganatage21;OlafStackelberg,KentStateUniversity;andDonD.Warren,Texas LegislativeCouncil.
SpecialthanksareduetoCRC’svisionaryeditor,DavidGrubbs,forhisencouragement andassistanceinseeingthisneweffortthrough.
Last,butnotleast,IwanttoexpressmygratitudeforallthetechnicalhelpIreceived, for17years,frommygoodfriendandcolleagueProfessorHowardKaplonofTowson University,andalltechnicalhelpIregularlyreceivefromBillLandry,myfriendandcolleagueatWesternNewEnglandUniversity.IamalsogratefultoProfessorNakhl´eAsmar, fromtheUniversityofMissouri,whogenerouslysharedwithmehisexperiencesinthe professionaltypesettingofhisownbeautifulbook.
SaeedGhahramani sghahram@wne.edu
This page intentionally left blank
A xioms of Probability
1.1INTRODUCTION
Insearchofnaturallawsthatgovernaphenomenon,scienceoftenfaces“events”thatmay ormaynotoccur.Theeventof disintegrationofagivenatomofradium isonesuchexample because,inanygiventimeinterval,suchanatommayormaynotdisintegrate.Theevent of findingnodefectduringinspectionofamicrowaveoven isanotherexample,sincean inspectormayormaynot finddefectsinthemicrowaveoven.Theeventthatan orbital satelliteinspaceisatacertainposition isathirdexample.Inanyexperiment,anevent thatmayormaynotoccuriscalled random.Iftheoccurrenceofaneventisinevitable,it iscalled certain,andifitcanneveroccur,itiscalled impossible.Forexample,theevent thatanobjecttravelsfasterthanlightisimpossible,andtheeventthatinathunderstorm flashesoflightning precedeanythunderechoesiscertain.
Knowingthataneventisrandomdeterminesonlythattheexistingconditionsunder whichtheexperimentisbeingperformeddonotguaranteeitsoccurrence.Therefore,the knowledgeobtainedfromrandomnessitselfishardlydecisive.Itishighlydesirableto determinequantitativelytheexactvalue,oranestimate,ofthechanceoftheoccurrence ofarandomevent.Thetheoryofprobabilityhasemergedfromattemptstodealwiththis problem.Inmanydifferent fieldsofscienceandtechnology,ithasbeenobservedthat, underalongseriesofexperiments,theproportionofthetimethataneventoccursmay appeartoapproachaconstant.Itistheseconstantsthatprobabilitytheory(andstatistics) aimsatpredictinganddescribingasquantitativemeasuresofthechanceofoccurrence ofevents.Forexample,ifafaircoinistossedrepeatedly,theproportionoftheheads approaches 1/2.Henceprobabilitytheorypostulatesthatthenumber 1/2 beassignedto theeventof gettingheadsinatossofafaircoin.
Historically,fromthedawnofcivilization,humanshavebeeninterestedingamesof chanceandgambling.However,theadventofprobabilityasamathematicaldisciplineis relativelyrecent.AncientEgyptians,about3500 B.C.,wereusingastragali,afour-sided die-shapedbonefoundintheheelsofsomeanimals,toplayagamenowcalled houndsand jackals.Theordinarysix-sideddiewasmadeabout1600 B.C. andsincethenhasbeenused inallkindsofgames.Theordinarydeckofplayingcards,probablythemostpopulartool ingamesandgambling,ismuchmorerecentthandice.Although itisnotknownwhere andwhendiceoriginated,therearereasonstobelievethattheywereinventedinChina sometimebetweentheseventhandtenthcenturies.Clearly,throughgamblingandgames
ofchancepeoplehavegainedintuitiveideasaboutthefrequencyofoccurrenceofcertain eventsand,hence,aboutprobabilities.Butsurprisingly,studiesofthechancesofevents werenotbegununtilthe fifteenthcentury.TheItalianscholarsLucaPaccioli(1445–1514), Niccol`oTartaglia(1499–1557),GirolamoCardano(1501–1576),andespeciallyGalileo Galilei(1564–1642)wereamongthe firstprominentmathematicianswhocalculatedprobabilitiesconcerningmanydifferentgamesofchance.Theyalsotriedtoconstructamathematicalfoundationforprobability.Cardanoevenpublishedahandbookongambling,with sectionsdiscussingmethodsofcheating.Nevertheless,realprogressstartedinFrancein 1654,whenBlaisePascal(1623–1662)andPierredeFermat(1601–1665)exchangedseverallettersinwhichtheydiscussedgeneralmethodsforthecalculationofprobabilities.In 1655,theDutchscholarChristianHuygens(1629–1695)joinedthem.In1657Huygens publishedthe firstbookonprobability, DeRatiocinatesinAleaeLudo(OnCalculationsin GamesofChance).Thisbookmarkedthebirthofprobability.Scholarswhoreaditrealizedthattheyhadencounteredanimportanttheory.Discussionsofsolvedandunsolved problemsandthesenewideasgeneratedreadersinterestedinthischallengingnew field.
AftertheworkofPascal,Fermat,andHuygens,thebookwrittenbyJamesBernoulli (1654–1705)andpublishedin1713andthatbyAbrahamdeMoivre(1667–1754)in1730 weremajorbreakthroughs.Intheeighteenthcentury,studiesbyPierre-SimonLaplace (1749–1827),Sim´eonDenisPoisson(1781–1840),andKarlFriedrichGauss(1777–1855) expandedthegrowthofprobabilityanditsapplicationsveryrapidlyandinmanydifferent directions.Inthenineteenthcentury,prominentRussianmathematiciansPafnutyChebyshev(1821–1894),AndreiMarkov(1856–1922),andAleksandrLyapunov(1857–1918) advancedtheworksofLaplace, DeMoivre,andBernoulliconsiderably.Bytheearlytwentiethcentury,probabilitywasalreadyadevelopedtheory,butitsfoundationwasnot firm. Amajorgoalwastoputiton firmmathematicalgrounds.Untilthen,amongotherinterpretationsperhapsthe relativefrequencyinterpretation ofprobabilitywasthemostsatisfactory.Accordingtothisinterpretation,todefine p,theprobabilityoftheoccurrenceof anevent A ofanexperiment,westudyaseriesofsequentialorsimultaneousperformances oftheexperimentandobservethattheproportionoftimesthat A occursapproachesaconstant.Thenwecount n(A),thenumberoftimesthat A occursduring n performancesofthe experiment,andwedefine p =limn→∞ n(A)/n.Thisdefinitionismathematicallyproblematicandcannotbethebasisofarigorousprobabilitytheory.Someofthedifficulties thatthisdefinitioncreatesareasfollows:
1. Inpractice, limn→∞ n(A)/n cannotbecomputedsinceitisimpossibletorepeatan experimentinfinitelymanytimes.Moreover,ifforalarge n, n(A)/n istakenasan approximationfortheprobabilityof A,thereisnowaytoanalyzetheerror.
2. Thereisnoreasontobelievethatthelimitof n(A)/n,as n →∞,exists.Also,if theexistenceofthislimit isacceptedasanaxiom,manydilemmasarisethatcannot besolved.Forexample,thereisnoreasontobelievethat,inadifferentseriesof experimentsandforthesameevent A,thisratioapproachesthesamelimit.Hencethe uniquenessoftheprobabilityoftheevent A isnotguaranteed.
3. Bythisdefinition,probabilitiesthatarebasedonourpersonalbeliefandknowledgeare notjustifiable.Thusstatementssuchasthefollowingwouldbemeaningless.
• Theprobabilitythatthepriceofoilwillberaisedinthenextsixmonthsis60%.
• Theprobabilitythatthe50,000thdecimal figureofthenumber π is7exceeds 10%.
• TheprobabilitythatitwillsnownextChristmasis30%.
• TheprobabilitythatMozartwaspoisonedbySalieriis18%.
In1900,attheInternationalCongressofMathematiciansinParis,DavidHilbert(1862–1943)proposed23problemswhosesolutionswere,inhisopinion,crucialtotheadvancementofmathematics.Oneoftheseproblemswastheaxiomatictreatmentofthetheory ofprobability.Inhislecture,HilbertquotedWeierstrass,whohadsaid,“The finalobject, alwaystobekeptinmind,istoarriveatacorrectunderstandingofthefoundationsofthe science.”Hilbertaddedthatathoroughunderstandingofspecialtheoriesofascienceis necessaryforsuccessfultreatmentofitsfoundation.Probabilityhadreachedthatpoint andwasstudiedenoughtowarrantthecreationofa firmmathematicalfoundation.Some worktowardthisgoalhadbeendoneby ´ EmileBorel(1871–1956),SergeBernstein(1880–1968),andRichardvonMises(1883–1953),butitwasnotuntil1933thatAndreiKolmogorov(1903–1987),aprominentRussianmathematician,successfullyaxiomatizedthe theoryofprobability.InKolmogorov’swork,whichisnowuniversally accepted,threeselfevidentandindisputablepropertiesofprobability(discussedlater)aretakenas axioms, and theentiretheoryofprobabilityisdevelopedandrigorouslybasedontheseaxioms.Inparticular,theexistenceofaconstant p,asthelimitoftheproportionofthenumberoftimes thattheevent A occurswhenthenumberofexperimentsincreasesto ∞,insomesense, isshown.Subjectiveprobabilitiesbasedonourpersonalknowledge,feelings,andbeliefs mayalsobemodeledandstudiedbythisaxiomaticapproach.
Inthisbookwestudythemathematicsofprobabilitybasedontheaxiomaticapproach. Sinceinthisapproachtheconceptsof samplespace and event playacentralrole,wenow explaintheseconceptsindetail.
1.2SAMPLESPACEANDEVENTS
Iftheoutcomeofanexperimentisnotcertainbutallofitspossibleoutcomesarepredictable inadvance,thenthesetofallthesepossibleoutcomesiscalledthe samplespace ofthe experimentandisusuallydenotedby S .Therefore,thesamplespaceofanexperiment consistsofallpossibleoutcomesoftheexperiment.Theseoutcomesaresometimescalled samplepoints, orsimply points, ofthesamplespace.In thelanguage ofprobability, certainsubsetsof S arereferredtoas events.Soeventsaresetsofpointsofthesample space.Someexamplesfollow.
Example1.1 Fortheexperimentof tossingacoinonce,thesamplespace S consistsof twopoints(outcomes),“heads”(H)and“tails”(T).Thus S = {H, T}.
Example1.2 Supposethatanexperimentconsistsoftwosteps.Firstacoinis flipped. Iftheoutcomeistails,adieistossed.Iftheoutcomeisheads,thecoinis flippedagain. Thesamplespaceofthisexperimentis S = {T1, T2, T3, T4, T5, T6, HT, HH}.Forthis
experiment,theeventof headsinthe first flipofthecoin is E = {HT, HH},andtheevent of anoddoutcome whenthedieistossedis F = {T1, T3, T5}
Example1.3 Considermeasuringthelifetimeofalightbulb.Sinceanynonnegative realnumbercanbeconsideredasthelifetimeofthelightbulb(inhours),thesamplespace is S = {x : x ≥ 0}.Inthisexperiment, E = {x : x ≥ 100} istheeventthat thelightbulb lastsatleast100hours, F = {x : x ≤ 1000} istheeventthat itlastsatmost1000hours, and G = {505.5} istheeventthat itlastsexactly505.5hours.
Example1.4 Supposethatastudyisbeingdoneonallfamilieswithone,two,orthree children.Lettheoutcomesofthestudybethegendersofthechildrenindescendingorder oftheirages.Then
S = b,g,bg,gb,bb,gg,bbb,bgb,bbg,bgg,ggg,gbg,ggb,gbb .
Heretheoutcome b meansthatthechildisaboy,and g meansthatitisagirl.Theevents F = {b,bg,bb,bbb,bgb,bbg,bgg } and G = {gg,bgg,gbg,ggb} representfamilieswhere theeldestchildisaboyandfamilieswithexactlytwogirls,respectively.
Example1.5 Abuswithacapacityof34passengersstopsatastationsometimebetween11:00 A.M. and11:40 A.M. everyday.Thesamplespaceoftheexperiment,consisting ofcountingthenumberofpassengersonthebusandmeasuringthearrivaltimeofthebus, is
where i representsthenumberofpassengersand t thearrivaltimeofthebusinhoursand fractionsofhours.Thesubsetof S definedby F = (27,t):11 1 3 <t< 11 2 3 istheevent thatthebusarrivesbetween11:20 A.M. and11:40 A.M. with27passengers.
Remark1.1 Differentmanifestationsofoutcomesofanexperimentmightleadtodifferentrepresentationsforthesamplespaceofthesameexperiment.Forinstance,inExample 1.5,theoutcomethatthe busarrivesat t with i passengers isrepresentedby (i,t),where t isexpressedinhoursandfractionsofhours.Bythisrepresentation,(1.1)isthesample spaceoftheexperiment.Nowifthesameoutcomeisdenotedby (i,t),where t isthe numberofminutesafter11 A.M. thatthebusarrives,thenthesamplespacetakestheform S1 = (i,t):0 ≤ i ≤ 34, 0 ≤ t ≤ 40 .
Totheoutcomethatthe busarrivesat 11:20 A.M. with 31 passengers,in S thecorresponding pointis 31, 11 1 3 ,whilein S1 itis (31, 20).
Example1.6(Round-OffError) Supposethat eachtimeJaychargesanitemtohis creditcard,hewillroundtheamounttothenearestdollarinhisrecords.Therefore,the round-offerror,whichisthetruevaluechargedminustheamountrecorded,israndom, withthesamplespace
wherewehaveassumedthatforanyintegerdollaramount a,Jayrounds a.50 to a +1.The eventofroundingoffatmost3centsinarandomchargeisgivenby
0, 0 01, 0 02, 0 03, 0 01, 0 02, 0 03
Iftheoutcomeofanexperimentbelongstoanevent E ,wesaythattheevent E has occurred.Forexample,ifwedrawtwocardsfromanordinarydeckof52cardsandobserve thatoneisaspadeandtheotheraheart,alloftheevents {sh}, {sh,dd}, {cc,dh,sh}, {hc,sh,ss,hh}, and {cc,hh,sh,dd} haveoccurredbecause sh,theoutcomeoftheexperiment,belongstoallofthem.However,noneoftheevents {dh,sc}, {dd}, {ss,hh,cc}, and {hd,hc,dc,sc,sd} hasoccurredbecause sh doesnotbelongtoanyofthem.
Inthestudyofprobabilitytheorytherelationsbetweendifferenteventsofanexperimentplayacentralrole.Intheremainderofthissectionwestudytheserelations.Inallof thefollowingdefinitionstheeventsbelongtoa fixedsamplespace S .
Subset
Equality
Intersection
Anevent E issaidtobea subset oftheevent F if,whenever E occurs, F alsooccurs.Thismeansthatallofthesamplepointsof E arecontained in F .Henceconsidering E and F solelyastwosets, E isasubsetof F intheusualset-theoreticsense:thatis, E ⊆ F
Events E and F aresaidtobe equal iftheoccurrenceof E impliesthe occurrenceof F ,andviceversa;thatis,if E ⊆ F and F ⊆ E ,hence E = F .
Aneventiscalledthe intersection oftwoevents E and F ifitoccurs onlywhenever E and F occursimultaneously.Inthelanguageofsets thiseventisdenotedby EF or E ∩ F becauseitisthesetcontaining exactlythecommonpointsof E and F .
Union
Complement
Difference
Aneventiscalledthe union oftwoevents E and F ifitoccurswhenever atleastoneofthemoccurs.Thiseventis E ∪ F sinceallofitspointsare in E or F orboth.
Aneventiscalledthe complement oftheevent E ifitonlyoccurswhenever E doesnotoccur.Thecomplementof E isdenotedby E c .
Aneventiscalledthe difference oftwoevents E and F ifitoccurswhenever E occursbut F doesnot.Thedifferenceoftheevents E and F is denotedby E F .Itisclearthat E c = S E and E F = E ∩ F c .
Certainty
Impossibility
Aneventiscalled certain ifitsoccurrenceisinevitable.Thusthesample spaceisacertainevent.
Aneventiscalled impossible ifthereiscertaintyinitsnonoccurrence. Therefore,theemptyset ∅,whichis S c ,isanimpossibleevent.
MutuallyExclusiveness
Ifthejointoccurrenceoftwoevents E and F isimpossible, wesaythat E and F are mutuallyexclusive.Thus E and F aremutually exclusiveiftheoccurrenceof E precludestheoccurrenceof F ,andvice versa.Sincetheeventrepresentingthejointoccurrenceof E and F is
EF ,theirintersection, E and F ,aremutuallyexclusiveif EF = ∅. Asetofevents {E1 ,E2,...} iscalled mutuallyexclusive ifthejoint occurrenceofanytwoofthemisimpossible,thatis,if ∀i = j , E i E j = ∅ Thus {E 1 ,E2 ,...} ismutuallyexclusiveifandonlyifeverypairofthem ismutuallyexclusive.
Theevents n i=1 E i , n i=1 E i , ∞ i=1 E i ,and ∞ i=1 E i aredefinedinawaysimilarto E 1 ∪ E 2 and E 1 ∩ E 2 .Forexample,if {E 1 ,E2,...,En } isasetofevents,by n i=1 E i we meantheeventinwhichatleastoneoftheevents E i , 1 ≤ i ≤ n,occurs.By n i=1 E i we meananeventthatoccursonlywhenalloftheevents E i , 1 ≤ i ≤ n,occur.
SometimesVenndiagramsareusedtorepresenttherelationsamongeventsofasample space.Thesamplespace S oftheexperimentisusuallyshownasalargerectangleand, inside S ,circlesorothergeometricalobjectsaredrawntoindicatetheeventsofinterest. Figure1.1presentsVenndiagramsfor EF , E ∪ F , E c ,and (E c G) ∪ F .Theshadedregions aretheindicatedevents.
Figure1.1 Venndiagramsoftheeventsspecified.
first-servedbasis.Let
E = thereareatleast fiveplaneswaitingtoland, F = thereareatmostthreeplaneswaitingtoland, H = thereareexactlytwoplaneswaitingtoland.
Then
1. E c istheeventthatatmostfourplanesarewaitingtoland.
2. F c istheeventthatatleastfourplanesarewaitingtoland.
3. E isasubsetof F c ;thatis,if E occurs,then F c occurs.Therefore, EF c = E.
4. H isasubsetof F ;thatis,if H occurs,then F occurs.Therefore, FH = H .
5. E and F aremutuallyexclusive;thatis, EF = ∅ E and H arealsomutually exclusivesince EH = ∅.
6. FH c istheeventthatthenumberofplaneswaitingtolandiszero,one,orthree.
Unions,intersections,andcomplementationssatisfymanyusefulrelationsbetween events.Afewoftheserelationsareasfollows:
E c )c = E,E ∪ E c = S,ES = E, and EE c = ∅
Commutativelaws:
Associativelaws:
Distributivelaws:
∪ F =
Anotherusefulrelationbetween E and F ,twoarbitraryeventsofasamplespace S ,is
