Acknowledgments
Immeasurablethankstoallofthephilosophers,scientists,andfriends whothoughtthroughtheideasinthisbookwithme.Iamgratefulfor yourtime,yourpatience,andyourinsights.Severalofyoureadwithcare thefulldraftmanuscriptandofferedextensivefeedback.Ioweyoumy sincerethanksforyourdetailedcommentsandchallengingquestions, whichwereinvaluable.
TheInstituteofPhilosophy,inUniversityofLondon’sSchoolof AdvancedStudy,hostedmeforasummervisitingfellowshipduring whichIbeganworkonthisbook.ANationalEndowmentfortheHumanitiesFellowshipandcorrespondingresearchleavefromWashington UniversityinSt.Louisenabledmeto finishit.Igratefullyacknowledge thisgeneroussupport.
Preface
Senseperceptionisthemostvividformoflivedhumanconsciousness. Youmayseethe flashofacardinaltaking flight,hearthethumpingofa hammer,ordetectafaintcitrusysmell.Throughoursenses,weencountertheworld.Webelievewhatwesee,andoursensesguidewhatwedo. Thesensesarecentraltodistinctivelyhumanaestheticexperience. Reflectingonperceptualimpairmentsrevealshowmucheachofour sensesmatters.
Understandingtheinterfacebetweenthemindandtheworldhas animatedaphilosophicaltraditionstemmingfromPlato’ s Theaetetus andAristotle’ s DeAnima.Fromtheearlymoderneratothepresent, philosophicalthinkingaboutperceptionanditssignificancehasbeen shapedtoaremarkableextentbyattentiontovision.However,vision doesnotstandalone.Atmostwakingmoments,peopleperceiveusing theirothersenses.Wehear,smell,taste,andtouchoursurroundings. Nothingguaranteesthatwhatwesayaboutvisionextendsneatlytothe othersenses.
Recently,philosophershavechallengedclaimsaboutperceptionand perceptualconsciousnessfoundedonvisionalone.Forinstance,Ihave arguedthatthetemporalnatureofsoundsandthewayssoundsoccupy pitchspaceconfound “visuocentric” thinkingabouttheobjectsofperception.Attentionexclusivelytovisionblindsustothescopeandnature ofwhatweperceive.Thishasimplicationsnotjustforphilosophybut alsoforartandaesthetics,soundstudies,andmusictheory.
Otherphilosophershavelookedbeyondvisiontotouch,bodily perception,olfaction,andtasteforinsightsabouthowoursensesacquaint uswiththeworld.Thishasreorientedthephilosophyofperception, enrichinghowweunderstandspatialawareness,whatitisforanexperiencetorepresent,andwhetherbrainprocessescouldfullyexplain sensoryconsciousness.Whatgoesforvisiondoesnotalwaysgoforour othersenses.
Still,nosenseisanisland.Eachsenseoperatesagainstthebackground ofothers,andpeopletypicallyperceiveusingmultiplesenses.Indeed,the moststrikingdiscoveryinthecognitivesciencesofperceptionduring thepasttwodecadesisthatsensorysystemsinteractextensivelywith eachother.
Sensoryinteractionssometimesleadtosurprisingperceptualillusions. Somepeoplehearsoundsascolored,feeltouchfromsounds,ortaste shapes.SynesthesiahasinspiredartistsfromKandinskyandHockneyto PharrellWilliams(SeeingSounds).Somesayithelpsexplainmetaphor, creativity,andtheoriginsoflanguageitself.However,synesthesiaisrare, affectingjust5percentofthepopulation.
Perceptualsciencehasshownthatsensoryinteractionsarefarmore widespread.Seeingatalkingfacecanchangehowspeechsounds for instance,youmayseemtohear/da/ratherthan/ba/justbecauseyousee someonearticulate/ga/.Inthesound-induced flasheffect,hearingtwo clicksmakesone flashlookliketwo.Inventriloquism,youhearthe sound’slocationdifferentlybecauseyouseethedummy.Crossmodal illusionsarepervasive.Theyoccurintypicalperceivingsubjectsacrossa widerangeofdomainswithnumeroussensorypairings.Justasvisual illusionsilluminatehowvisionfunctions,crossmodalillusionsshowhow thesensesworktogether.
Crossmodalillusionsaresurprising.Onesensecanreshapewhatyou perceivewithanother.Thisconflictswithcommonsense,whichpresupposesthatoursensesareseparate.Whatremainsunclear whatthe sciencedoesnotsettle ishowsensoryinteractionsarereflectedinthe consciouslivesofperceivingsubjects,andwhytheymatter.Answering thesequestionsrequiresconfrontingphilosophicalquestionsaboutperceptionandperceptualconsciousness.
Thisbookaddressesthesequestions.Itexploresthemultisensory natureofperceptionanditstheoreticalandphilosophicalsignificance. Againstphilosophicalorthodoxy,whichtreatsperceptualconsciousness asacollectionofexperiencesassociatedwithvision,hearing,touch, taste,andsmell,thisbookcontendsth athumanperceptualconsciousnessisconstitutivelyandirreduciblymultisensory.Itdevelopsan accountofmultisensoryperceptioninwhichcoordinationandcooperationamongthesensesimprovesandaugmentshumanperceptual
capacities.Thenormalandoptimalfunctioningofeachsenserequires thesupportofmultiplesenses.
Accordingtothisaccount,thecoordinateduseofmultiplesenses enhancesandextendshumanperceptualcapacitiesinthreecritical ways.First,crossmodalperceptualillusionsrevealhiddensensoryinteractionsthatperformmultisensoryfunctions.Butsuchinteractionsare farmorewidespread.Typically,theymakeeachsensemorereliable andthusabettersourceofevidenceabouttheenvironment.Thecost ispredictableillusions.Multisensoryinteractionsserveanimportant purposebyimprovingperception’scoherence,accuracy,andreliability. Spatialhearingimproveswhenitlistenstovision,andlipreading supportsspeechcomprehensionasmuchasagoodhearingaid.Such perceptualimprovementscanreverberateasepistemicadvantages. Believingyoursensesworksbetterwhenyoursensesworktogether. Morereliableperceptionmeansmorereliablecognition.Multisensory epistemologythusreachesbeyondwhatmeetstheeyesandtheears. Whatispuzzlingisthatcrossmodalrecalibrationsandillusionstypicallygounnoticed youmaynotrealizethatwhatyouseeaffectswhat youhear.
Second,thejointuseofmultiplesensesdisclosesmoreoftheworld, givingusconsciousaccessinperceptiontonovelfeaturesandqualities. Newaspectsoftheworldareperceptibleonlythankstothecoordinated useofmultiplesenses.Umpiresinbaseballaretrainedtotellwhethera baserunnerissafeoroutbywatchingthefoottouchthebagwhile listeningforthesoundoftheballstrikingtheglove.Umpiresare sensitivetoperceptibleintermodaltemporalorder.Atthemovies, imagesonscreenappeartomakethesoundsyouhear amisaligned soundtrackisgrating.Somethingvisiblecanperceptiblybearaudible features.Or,thankstothewaysmell,taste,andsomatosensationwork together,novelqualities,suchas flavors the mintiness ofmint,the spiciness ofcapsaicin areexperiencedonlymultisensorily.
Multisensoryperceptionthereforedoesmorethanimprovethetestimonyofthesenses.Sometimesone’smultisensorycapabilitiesareevident.Thejointuseofoursensesenablesnewformsofperceptual awareness.Asaconsequence,perceptualconsciousnessisnotalways specifictoonesenseoranother.Contrarytothereceivedview,thisbook
arguesthatthephenomenalcharacterofsensoryconsciousnessitselfis irreduciblymultisensory.Thiscontravenesacentralassumptioninthe empiricistphilosophicaltradition,accordingtowhicheachexperience hasadistinctivesense-specificcharacter.Sometimes,perceptualconsciousnessitselfisineliminablymultisensory.
Third,eachsensedependsontheinfluenceofothers.Multisensory perceptionevenreshapesunisensoryperception.Perceptualcapacities associatedwithonesensedependonothersenses.Perceptuallearning canenableusauditorilytodetectfeaturesthatotherwiseareaccessible onlythroughsight.Crossmodalparasitismcaninfuseanauditoryexperiencewithcharacteristicsinheritedfromvision.So,onesensecanchange overtimethankstoanother.Thismeanstheauditoryexperienceofa congenitallyblindpersonmaydifferfromsomeonewhosees.Asurprisingconclusionfollows.Whileadeficitinonesensecanenhanceanother sense,deficitsinonesensealsocanramifyasdeficitselsewhere.Famously,spatialhearingimproveswithblindness,butbymyaccount blindnessalsoyieldshearingdeficits.Appreciatingcrossmodalplasticity makesroomforanovelaccountofsensoryenhancementusingprosthetics,suchascochlearimplants,andsubstitutiondevices.
Sensoryplasticityandcrossmodaldependencepresentadilemmafor thesense-by-senseapproach.Eitheritignoreswhatothersensescontributetosightandhearing,oritexciseseachsensefromtheothers, therebythrottlingbackitscapabilitiesandalteringitscharacter.
Theimplicationisthatnoonesense notevenvisionitself canbe understoodentirelyinisolationfromtheothers.Thisoverturnsaprevailingunisensoryapproachtosenseperceptionandperceptualconsciousness,whichassumesthateachsensecanbetheorizedinisolation orinabstractionfromtheothers.Perceivingisnotjustseeing,hearing, touching,tasting,andsmellingatthesametime.Nocompleteaccountof perceptualconsciousnessoritsrolecanbeformulatedwithoutconfrontingthemultisensorynatureandcharacterofperception.Thissetsthe stageforarevisionary,multisensoryphilosophyofperception.
Introduction
Acoretensionanimatesthisbook.Mosttimes,typicalhumansubjects perceiveusingmultiplesenses.Outwalking,Iseepeople,trees,colors, andmotion.Ihearchatter,footsteps,andrustlingleaves;smellcoffee, andpetrichorafterrain;feelabreeze,thebrushofashoulder,ormudon theground.Itastemycoffee’sacidity,itsearthiness.Weuseoursenses togethertonavigateandlearnabouttheworld.
Mosttheorizingaboutperception,however,isunisensory.Philosophy andsciencehavefocusedononesenseatatime.Researcherstakepains tosecureaquietsetting,isolateatactualstimulus,orteaseoutvisual experiencesfromasubject’sconsciouslife.Thissense-by-senseapproach respectsananalytic,dissectiveapproachtosenseperception.
Thisbookisaboutmultisensoryperception.Itisabouthowoursenses worktogether,incontrastwithhowtheyworkseparatelyandindependently.Inparticular,itisabouthowonesensecanimpactanotherand abouthowthejointuseofmultiplesensesenablesnovelformsof perceptionandexperience.Anditisaboutwhythismatters.Itdescribes themostcriticalrespectsinwhichperceptionismultisensory.Andit addressestheconsequences.
Mythesisisthatthecoordinateduseofmultiplesensesenhancesand extendshumanperceptualcapacities.Thismakespossiblenovelvarieties ofperceptualconsciousness.Theseformsofperceptionilluminateperception’snatureandpurposebutescapeunisensorytheorizing.
Perceivingisnotjustamatterofourseveralsensesworkingontheir own,inparallel.Thishasbothnegativeandpositiveconsequences.Onthe negativeside,itmakestroubleforunisensoryapproaches.Evenifone’ s focusisasinglesense,suchasvisionalone,investigatingonesenseata timeriskseitheranincompleteoradistortedunderstandingofperception andperceptualconsciousness.Sincenearlyeveryphilosophicalaccount ofperceptionuntilrecentlyhasfocusedononesenseoranother,this consequenceisfar-reaching.
Onthepositiveside,coordinationamongsensescanimproveour abilitytonegotiatetheenvironment.Eventhoughitcausespredictable crossmodalillusions,itneverthelesscanmakeperceptionmorecoherent andmorereliable.Ittherebyaffectsone’sepistemicposition.Moreover, itenrichesperceptualconsciousness,bothatatimeandovertime,when comparedwithparallelunisensoryperception.Itrevealsmoreofthe world.Accordingly,multisensoryperceptionhaspractical,epistemic, andaestheticbenefits.
Appreciatinghowoursensesworktogether howtheyimpact,influence,andcooperatewitheachother bringsintofocusamoreaccurate, morenuancedunderstandingofperceptionandperceptualconsciousness.Thisrevealsnewpuzzles,anditilluminatesoldones.Itoughtto shapeanyfuturephilosophyofperception.
Thischaptersetsthestage.Itpresentsthebook’splanofattackandits centralthemes.First,itdescribesunisensoryparadigmsforinvestigating perceptioninscienceandinphilosophy,anditpresentstheevidence. Next,itintroducesthecriticalrespectsinwhichperceptionismultisensory andexplainswhythisisaproblemforsense-by-sensetheorizing.Finally, itraisesthecentralquestionsanymultisensoryphilosophyofperception mustface,anditoutlinestheanswersandargumentsthatfollow.
1.1UnisensoryApproaches
Theorizingaboutperceptioninscienceandinphilosophyuntilrecently hasbeenoverwhelminglyunisensory.BythisImeanthatithasproceeded onesenseatatime.Ithasadoptedasense-by-senseapproachtoperceptualcapacities,processes,andconsciousness.Thismeansinvestigating perceptioninunisensorycontextsandabstractingphenomenaassociated withonesensefromwhatoccursinothersenses.Thistreatmentreflects twoimplicitassumptions.The firstisthateachsenseisexplanatorily independentfromtheothers.Thesecondisthatthesumoftheorizing aboutindividualsensesyieldsacompleteaccountofsensoryperception.
Theseclaimsrequireexplanationanddefense.Inwhatfollows,I’ll presentacasethatparadigmsoftheorizingaboutperceptioninscience andinphilosophyhavebeenunisensory.Then,I’lldescribewhatmakes thisisaproblem.
Theempiricalproblemofperceptionishowstimulationofsensory receptorsleadstoperceptionoftheworld.Perceptionscienceanswers bydescribingprocessesinwhichinformationisextractedfromsensory stimulationinawaythatenablesitsuseinrecognizing,acting,attending, remembering,thinking,andimagining.Itsaimistoaccountforhuman performance.
Thistaskhastwosides.The firstistodeterminewhichperceptual capacitiestypicalhumanbeingspossess.Whichsortsofthingsand featurescanwedetectanddifferentiate,andinwhatconditions?Psychophysicsmeasuresourcapacitytodiscernandmakeuseoffeaturesin theenvironment.
Theotheristodescribeandtocharacterizethenatureoftheprocesses thatareresponsible.Whatgroundsthecapacitytoperceive?Perceptual psychologyandneuroscienceeachaimtospecifywhichprocessesextract informationfromastimulusandmakeitavailableforuseincognition andaction.Cognitivepsychologydescribesabstractfunctionsthatcanbe realizedindifferingphysicalsystems,whileneurosciencefavorsneural mechanisms.
OnecelebratedparadigmisMarr’scomputationaltheoryofvision. AccordingtoMarr(1982),thesolutiontotheempiricalproblemof humanvisionhasthreecomponents.The firstarticulatesthecomputationalproblemvisionsolves.Whatareitsinputsandoutputs?This amountstocharacterizingvisualcapacities.Thesecondisanalgorithm. Howdothingsfunctioninsidetheblackbox?Thisisapictureofhow visionsolvesthecomputationalproblem.Thethirdisitsimplementation inthebrain.Whichhiddenprocessesoccurtomakehumanvisual capacitiesmanifest?Thisisadescriptionofneuralactivitythatrealizes vision.
Thisapproachtoperceptionisanalytical,intworespects.Oneis vertical.Acapacitytoperceiveisbrokendownintostages,analyzedas aseriesofsubcapacities,exercisedstepbysteptoperformaperceptual task.MotivatedbytheobservationthatGibson(1966)underestimates thedifficultyofdirectinformationpickup(29–30;seealsoFodorand Pylyshyn1981),Marrsaysseeingobjects firstreliesonextractinginformationaboutedgesfromilluminationgradients,followedbydiscerning
boundaries,surfaces,andthree-dimensionalshapesofsolids.Ascientific accountofperceptionrequiresdescribingeachstepintheprocessby whichtheempiricalproblemofperceptionissolved.
Theotherishorizontal.Perceptionsciencecarvesperceptionat distinctsenses,anditdissectseachsenseaccordingtoitsconstituent capacities.Marroffersatheoryofvisionalone,notperceptionoverall. Onepartofthisisanaccountofhowvisionmakesextendedobjects availableforcognition.Colorperceptionisanotherpartofvision(see,for instance,MaloneyandWandell1986).Othersensesmirrorthisapproach.
Marr’starget,Gibson(1966),achampionofhaptictouch,devotestwo chaptersdelineatingitscapacitiesandcomponents.Bregman(1990) the Marrofhearing describesthemeansbywhichauditioncarvesacoustical scenesintodistinctaudibleobjectsandfeatures.Bregmanarticulates howindividuatingsoundstreamsreliesonbothsimultaneousgrouping anddiscerninggoodcontinuityinfrequency,timbre,andtiming.
Intheserespects,Marr’sparadigmisrepresentative.And,ineach respect,perceptionscienceatlargeuntilrecentlyhasremainedpredominantlyunisensory.Typicalpracticeinexperimentationandexplanation hastreatedperceptionusingonesenseratherthanmultiplesenses.Ithas aimedtodiscoverwhichcapacitiesbelongtoaparticularsense,suchas vision,audition,ortouch.Andithasfocusedonspecifyingthedetailsof capacitiesdeployedusingonesenseatatime.Moreover,whileithas oftenconsideredinfluencesonperceptionfromcognition,perception sciencetypicallyhasnotconsideredat eachstageoftheperceptualprocess outsideinfluencesfromothersenses.Untilthestartofthetwenty-first century,ithasrarelyaddressedcoordinationwithinperceptiontakingplace betweensenses.
Considerthemethodology.Experimentalparadigmstendtoinvestigatesenseperceptioninunisensorycontexts.Inpractice,thismeans presentingandvaryingastimulustoonesensewhileholding fixedor tryingtoeliminatestimulationtoothersenses.Forexample,anaudition researchermaymeasureresponsivenesstoatone,ratherthantoatone and flashcombination,ortoatonepresentedwithabusyvisualscene. Or,anexperimentermayjustforgetabouttheblindfoldandignoreor abstractfromanycontributionsvisionmakestoaudition,assumingthey areinsignificantorirrelevant.Untilrecently thepastdecadeortwo littlesystematicattentionhasbeendevotedtoperceptualcontexts
involvingmorethanonesense,tothewaysinwhichonesenseinfluences another,andtocoordinationamongsenses.
Accordingly,inventoriesofperceptualcapacitiesandaccountsof perceptualprocessinghaveremainedsensespecific.BythisImeanthat theyconsidercapacitiesonlyinsofarastheydonotrelyonandremain uninformedbyothersenses,andtheydescribeprocesses fixedbyfactors stemmingonlyfromwithinagivensensorypathway.Intheserespects, thescienceofperceptionhasbeenchieflyunisensory.
Asaconsequence,theparadigmsinperceptionscienceareunisensory. Theyfailtoconsideroutsideinfluencesfromothersenses,ortheyignore andabstractawayfromsuchinfluences.Withafewnoteworthyexceptions,includingearlyworkonspeechperception,objectperception,and ventriloquism,theydonotaddresscollaborationbetweensensesor capacitiesthatrequiresensestocooperate.EvencontemporaryBayesian accounts firstfocusedalmostexclusivelyonvision,beforeturningto multisensoryperception(see,forexample,Mamassianetal.2002).¹
Unisensoryapproachestoperceptionsciencerevealtwoassumptions. The firstisthatthesensesareexplanatorilyindependentfromeach other.Inparticular,unisensorytheorizingpresumesthatwhatasense doesinisolationmatcheswhatitdoesinamultisensorycontext.Any differencesaffectneithertheprinciplesaccordingtowhichitfunctions noritsstockofcapacities.Eachsenseiscausallyorfunctionallydiscrete.
¹ Invisionscience,someinfluentialexamplesofunisensorytheorizingfromthepastfour decadesincludeGibson(1979);Marr(1982);Rock(1983);Treisman(1988);Milnerand Goodale(1995);Rock(1997);MackandRock(1998);Palmer(1999);Pylyshyn(1999); O’ReganandNoë(2001).Inauditionscience,noteworthyrecentparadigmsareBregman (1990);Handel(1993);Blauert(1997).
Contemporaryexceptionspriorto2000includegroundbreakingworkonspeechperception byMcGurkandMacDonald(1976)andFowler(1986),stemmingfrominsightsofLiberman etal.(1967);workonobjectperceptionandcorecognitionbySpelke(1990);andBertelson (1999)onventriloquism.Fodor’s(1983)theoreticalunderstandingofmodularinputsystemsas domainspecificbutnotsensespecificistrailblazing.SpenceandDriver(2004)marksthe watershed.
Whilenotfocusedonperceptionsciencealone,Google’sn-gramdataisrevealing.Priorto 1932,then-gramviewerregistersnooccurrenceinbooksoftheterm “multisensory.” However, followingasteady firsttrickleofuses,theshareofoccurrencesof “multisensory” spikesroughly fivefoldfrom1964to1976(duringthistime, “audition” roughlydoubleditsshare,and “vision” was flat),beforelevelingandthenroughlydoublingfromthemid1990sthrough2008(during thisperiod, “audition” increaseditsshareroughly20percent,and “vision” wasroughly flat). Nonetheless,in2008, “vision” stillhadashare271timesgreaterthan “multisensory,” and “audition” hadsixtimestheshareof “multisensory” (“vision” thuswasforty-fivetimesmore frequentthan “audition” in2008).(https://books.google.com/ngrams,accessedFebruary22,2018)
Thesecondisthatthesense-by-senseapproachisexplanatorily complete.Perceptualcapacitiesincludejustthoseassociatedwitheach oftherespectivesenses.Perceptualprocessesinvolveonlyfunctionsand mechanismsbelongingtodiscretemodalities.Takentogether,sensespecificaccountsofperceivingwitheachoftherespectivesensesexhaust thescienceofperception.Whatremainsisforcognition.Putsimply, Marr,Bregman,andtheirilksuffice.
Recentapproachesinperceptionsciencebreakfromthistradition.For instance,predictivetheoriestreatperceptionasahierarchyofdistinct statesthatprobabilisticallymodelincomingstimulationanddetecterrors inordertoguiderevisionsandupdates(Hohwy2013;Clark2016).Such accounts,likeotherBayesianapproaches,maytreatsensesseparatelyor jointly.Accordingly,theyprovideanattractiveframeworkinwhichto addressmultisensoryphenomena(seeChapter2).Researchersduring thetwenty-firstcenturyincreasinglyhavedevotedattentiontoexplaining perceptioninmultisensorycontexts.
1.1.2Philosophy
Thephilosophicalpuzzleofperceptionishowtoreconcilethepossibility ofillusionsandhallucinationswiththecharacterofperceptualconsciousness.Perceptionisahumansubject’smostintimateformof acquaintancewiththeenvironment.Fromasubject’sperspective,things appearjustsetoutinone’ssurroundings,accessiblebymeansofthe senses.But,appearancescanbemisleading,andthingsarenotalwaysas theyseem.Nothinginprinciplerulesoutthepossibilityofaradical, wholesalehallucinationthatasubjectcouldnottellapartfromatypical, veridicalperception.
Thepuzzlehasmetaphysicalandepistemicfacets.Whatistherelationshipinwhichwestandtothingsandfeaturesduringperceptual consciousness?Areweeveracquaintedwiththingsandfeaturesthatdo notdependonourownconsciousness?Howdoweknow?Canexperiencesenableandjustifybeliefsaboutthingsthataremindindependent? Answersconcernthenatureandroleofexperience.Onemainaimin philosophyofperceptionistocharacterizethenatureofperceptual consciousness.Forinstance,relationalistaccountsholdthatexperience
isaconscioussensoryrelationbetweenasubjectandaparticularobject orfeature,requiringthepresenceofeach.Inrepresentationalaccounts, experienceshavesatisfactionconditionsthatmaybemetorunmet.In someaccounts,experiencesaremarkedbyasubject’sconsciouslyinstantiatinginherentlyqualitativeproperties,orqualia.²
Anotheraiminphilosophyofperceptionistodescribetheroleof experiences.Thisincludestherelationshipsbetweenexperiencesand othersortsofmentalphenomena.Forinstance,experiencescausememories,learning,andbeliefs.Italsoisnaturaltothinkthatexperiences mustbecapableofrationallysupportingbeliefs.Inonetypeofaccount, experiencesdosojustbypresentingthingstobeaparticularway.Others sayexperienceitselfisneutral,orthatitreliesonfurtherassumptionsto supportempiricalbeliefs.
Arelateddebateabouttheroleofexperiencesconcernswhether informationfromcognitioneverdirectlyaffectsperceptualconsciousness.Thisbearsonwhetherornotperceptionisanunbiasedsourceof evidence.Ifpreconceptionsshapeexperiences,thatmayunderminethe supportanysuch “hijacked” experienceprovides(Siegel2017).
Thus,thefundamentalnatureofperceptualconsciousness,andthe roleofperception,especiallyinrelationtosensation,thought,andaction, areofparticularconcerntophilosophersinvestigatingperception.
Philosophyofperceptiondealswithtwoaspectsofperceptualconsciousness.The firstisawareness.Bythis,Imeantheconsciousgraspon anobjectorsituationthatperceptionaffords.Perceivingmakesthings availabletoasubjectinawaythatenablesthatsubjecttorecognize, conceptualize,demonstrate,imagine,remember,act,andotherwise cognizethem.Thatconsciousgraspisdiagnosedinpartbythepsychologicalroleitplays.Awareness figurescriticallyinaddressingepistemic questionsaboutperceptualconsciousness.
Theotherisphenomenality.Experiencesareconsciousepisodeswith phenomenalcharacter.Phenomenalcharactercomprises “whatitislike” forasubjecttoundergoaconsciousepisode(Nagel1974).Howthings
² WhileinthisbookIaimasfaraspossibletobeneutralamongsuchapproaches indeed, thebookisdesignedtoaddressconcernsdistinctivetoeach mypointinthissectionisthat suchaccountshavebeenshapedbyattemptstoaccommodateunisensoryratherthanmultisensoryphenomena.
areinconsciousnessfromasubject’sperspectiveis fixedbyitsphenomenal features.Experience,fromasubject’sperspective,ispartofthephilosophicalpuzzleofperception.Accuratelyspecifyingperceptualphenomenal characterisakeypartofevaluatingsolutionstothepuzzle.Phenomenality also figurescriticallyinaddressingmetaphysicalquestionsaboutperceptualconsciousness.
Onecandiscerntworespectsinwhichphilosophyofperceptionhas beenstarklyunisensoryinconfrontingeachoftheseaspectsofperceptualconsciousness.The firstisthatithasfocusedpredominantlyon vision.Despiteapronouncedrecentturntonon-visualformsofperception,suchashearing,touch,andsmell,mostphilosophicalworkaddresses visionandvisualconsciousness.Itisdifficulttooverstatetheinfluenceof thinkingaboutsightandseeinginunderstandingperceptualawareness, perceptualphenomenality,andtheircausalandrationalroles.
Focusingonvisionalonerisksovergeneralizing.Visuocentrismignores counterexamples,novelphenomena,andsalientfeaturesofawarenessand phenomenalcharacterdrawnfromothersenses.Moreover,somegood reasonssuggestthatvisionisexceptional.Atheoryofvisionthusshould notpassforageneraltheoryofperception.Thephilosophyofperception ismorethanthephilosophyofseeing.
Nevertheless,Iwanttosetasidevisuocentrism.Thereisasecondway inwhichphilosophicalworkonperceptionisunisensory.Untilrecently, ithasalmostexclusivelyreflectedasense-by-senseapproach.Bythis Imeanthatphilosopherswhoinvestigateperceptionbyandlargeaddress onesenseatatime.Unisensorymethodologytreatsvision,hearing,touch, taste,orsmellinisolationorinabstractionfromothersenses.
Aunisensoryapproachinphilosophyofperceptionmayinvolve examiningwhatasensedoesinaunisensorycontextorinresponseto varyingastimulustojustonesense.Morefrequently,itmeansattemptingtoextractfromamultisensoryepisodewhatbelongstoatarget sense.Thebulkofphilosophicalworkonmetaphysicalandepistemic dimensionsofperceptualawarenessandphenomenologymanifeststhis typeofunisensorytake.
Mostly,thesense-by-senseapproachisunstated.Thephilosophyof perceptionsubsistsonexamplesofperceivingwithonesenseoranother: seeingspeckledhens,hearingspeechsoundsinfamiliarorforeignlanguages,smellingscentsofroses,tastingVegemite,ortouchingspheres
andcubes.Evennoteworthyworksthatdonotfocusonvisionalonehave takenperceptiononesenseatatime(see,forinstance,Matthen2005). Somephilosophersaremoreexplicitinadoptingasense-by-sense approach.SusannaSiegel’sbook, TheContentsofVisualExperience, targetsvisualexperiencesthisway:
Whenyouwaterskiyouareinaconsciousstatewithmanyfacets. Wecancallyouroverallconsciousstateduringthestretchoftimethat youarewaterskiingyour overallexperience .Youroverallexperience includesseeingtheboatanditssurroundings,hearingitsmotor,feeling preoccupied,keepingyourbalance,andsoon. ...Fromthemanyfacets ofwaterskiing,wecanzoominonthevisualones....wecanconsider theoverallexperienceasacollectionofphenomenallyconsciousstates andthenignorethenon-visualones.(Siegel2010,19–21)
Siegeloffersanaccountofisolablevisualphenomenalstatesthatcanbe carvedoutandtheorizedseparatelyfromothersenses.Thisabstracts fromthemultisensorywholetotreatvisualexperiencesontheirown. Insomecases,theunisensoryapproachismarkedasanidealization. NicoOrlandiacknowledgesthis:
Therearealsogeneralquestionsconcerningtheindependenceofvisual processingfromtheprocessingofothermodalities.Theissuespertainingtothesequestionsarenotalwaysaddressedbyinferentialists. TheyarealsoneglectedinthisbookwhereIoperateundertheidealizationthatthereconstructionofvisualobjectsoccurssomewhatindependentlyoftheinfluenceofothersensorymodalities.
(Orlandi2014,28)
Theseexamplesillustrateageneraltrend.Philosophicalquestionsabout perceptionandperceptualconsciousnessareaddressedbyconsidering anddescribingformsofperceptioninvolvingonesenseatatime. Thisisaunisensoryapproachtothephilosophyofperception.Unisensorytheorizingbeginswiththetruismthatwehavemultiplesenses multiplewaysofperceivingtheworld.Itthentreatsvarietiesofperceptual consciousnessassociatedwitheachsenseseparately.Itabstractstoisolate independentsenses,oritanalyzesthewholeasaco-conscioussumof
sense-specificparts.Itconsiderstheroleofeachsenseonitsownrather thanamongthesensesworkingjointly.Thissortofunisensorytheorizing ismymaintarget.³
Theunisensoryapproachisreflectedinphilosophicalaccountsof awarenessandphenomenality.Forexample,intentionalistsgenerally treatexperiencesasrelationstocontentsinaspecificsensorymode, eachofwhichisunderstoodasakintoadistinctivepropositionalattitude.Intentionalistsholdthatanexperience’sphenomenalfeaturesarea subsetofitsintentionalfeatures.Thus,phenomenalfeaturesdependon agivensensemodality.Someintentionalistsmaintainthatcontent fixes phenomenalcharacteronlywithinasensorymodality.Intentionalists onlyrecentlyhaveconsideredtheconsequencesofformsofperception thatrelyonthejointuseofmultiplesenses(forinstance,Speaks2015; Tye2003).
Relationalistssayexperiencesinvolveasubject’sstandinginaspecific sensoryrelationtowhatisperceived(Brewer2011).Seeing,hearing, touching,smelling,andtastingaredistinctivesensoryrelations distinct waysofperceivingcorrespondingtoeachofthevarioussenses.Martin
³ Inphilosophy,someinfluentialexamplesofunisensorytheorizingfromthepastfour decadesincludeSnowdon(1981);Peacocke(1983);Perkins(1983);Harman(1990);Crane (1992);Valberg(1992);McDowell(1994);Lycan(1996);Tye(2000);Byrne(2001);O’Regan andNoë(2001);Campbell(2002);Noë(2004);Matthen(2005);Pylyshyn(2007);Hill(2009); Burge(2010);Siegel(2010);Orlandi(2014);Dickie(2015);Siegel(2017);Schellenberg(2018).
RecentexamplesofunisensorytheorizingbeyondvisionincludeO’Callaghan(2007);Nudds andO’Callaghan(2009);Lycan(2000);Batty(2011);Richardson(2013);Fulkerson(2014). Someexceptionstounisensorytheorizingsince2000includeNudds(2001);O’Callaghan (2008);deVignemont(2014),andessaysinBennettandHill(2014);Stokesetal.(2015);Deroy (2017).Fodor(1983)wasavant-garde.
Unisensorytheorizingisclearestintheanalytictradition,andittracestoearlymodern empiricists,suchasLockeandHume.Notably,Carnap’sphenomenalisticreconstructioninhis Aufbau (1928)treatssubjects’ overallexperiencesasbasicandunanalyzable,butabstractsto characterizeclassesofexperiencescorrespondingtoeachsense,includingvision(sections 61–94;see,especially,sections67–8and85–6).Goodman’scriticalresponsein TheStructure ofAppearance (1951)insteadbeginswithbasicqualitativepartsofexperiences,orsensory atoms. “Qualia,” forGoodman, “aretobesuchqualitiesassinglephenomenalcolors,sounds, degreesofwarmth,moments,andvisuallocations” (156),whichstandinaprimitiverelationof “togetherness” toformconcreteparticulars(158).WhereCarnapabstractstotreatsenses individually,Goodmanconstructseachsensefromsense-specificqualia.AccordingtoGoodman, “Qualia[suchascolorsandsounds]thatdonotbelongtothesamesenserealmnever occurtogether” (159).
Amongtwentieth-centurycontinentalphilosophers,Merleau-Ponty’ s Phenomenologyof Perception (1945)isthemostnoteworthyexceptiontounisensorytheorizing.Aristotle’ s De Anima (1984),whichaddressesthecommonsense,standsoutinancientphilosophy.
(1992)isskepticalwhetheranyunifyingaccountexistsforsightand touchbecauseeachdiffersfundamentallyinhowitpresentssubjectswith objectsandspace.
Qualiatheoristsinvariablytreateachphenomenalqualityassociated withaconsciousperceptualepisodeasdistinctivetoonespecificsenseor another.Jackson’s(1982)classicessaypointstoepiphenomenalqualia associatedwithsmellingarose,tastingalemon,hearingaloudnoise, seeingthesky,seeingripetomatoes,andbatsonar.Lewis’s(1988)reply addssmellingaskunk,feelinghot fireandcolddrafts,andtasting Vegemite.Multisensoryqualiaareunmentioned.
Eventheoristswhoendorseconsciousperceptionof “high-level” features,suchasnaturalkindandsemanticproperties,focusontheir perceptionbymeansofaspecificsensorymodality.Discussionsconcern seeingpinetrees,tigers,andfaces,hearingpersonsandthemeaningsof spokenutterances,smellingroses,andtastingvintages.Perceptionof higher-levelfeaturesusingmorethanonesenseatatimetypicallyis absentfromsuchaccounts(cf.Nudds2001).
Accordingly,perception’srationalrolemostoftenisrestrictedtothat whichstemsfromsingularsensorysources.Forinstance,thoughtsabout thingsandfeaturesanchoredinmultisensorydemonstrationsgounaddressed.And,ingeneral,epistemologyofperceptionfailstoconsider incrementalsupportstemmingfromthecoordinatedworkingsofseveral senses(cf.Matthen2014).
Paradigmsinphilosophyofperceptionthusrevealtwoassumptions thatparallelanearlierperceptionscience.The firstisthateachsenseis explanatorilyindependentfromtheothers.Inparticular,itispossibleto explainperceptualconsciousnessineachsenseonitsownterms.This assumesthatthepropertiesofperceptualconsciousnessassociatedwith onesensearenotrelevanttoexplainingthepropertiesofperceptual consciousnessassociatedwithanothersense.
Thesecondassumptionisthatthesense-by-senseapproachis explanatorilycomplete.Perceptualawarenessincludesjusttheforms associatedwitheachoftherespectivesenses,andperceptualphenomenalityreducestowhatisspecifictoonesenseoranother.Perception’ s roleinsupportingthoughtandactionamountstotheroleplayedbyeach individualsense.Takentogether,accountsofperceptualconsciousness ineachoftherespectivesenses,alongwiththeunityofconsciousness,
exhaustthephilosophyofperception.Perceptualconsciousnessjustis theco-conscioussumofitssense-specificparts.
1.2SensoryInteractions
Someofthemostdramaticdevelopmentsinthecognitivesciencesof perceptionduringthepasttwodecadesstemfromappreciatingthe extenttowhichhumansensesinteractwitheachother.First,onesense canimpactanothersense,atatimeorovertime,therebyaffecting perceptualconsciousness.Thisunderminestheassumptionofexplanatory independence.Second,multiplesensesworkincoordination,sometimes revealingfeaturesthatarenototherwiseperceptible.Thisunderminesthe assumptionofexplanatorycompleteness.
Synesthesiaprovidesaglimpse.Insynesthetes,stimulatingonesense cantriggersensationsassociatedwithanothersense.Asynesthetemight visualizecolorsinresponsetosoundsorseemtoexperiencetastesin responsetoshapes.Butsynesthesiaisquirky,anditisrare.Itishardto seewhatpurposeitserves.
However,onesensecanaffectanotherinawaythatreshapesexperienceinjustaboutanyone.Crossmodalperceptualillusionsarecasesin whichonesensechangesyourexperiencewithanothersense.For example,seeingapuppetoraneventonscreencanaffecttheapparent locationfromwhichyouhearasound.Thisisthebasisforventriloquism.Itisanauditoryillusionproducedbyvision.Seeingaspeaker’ s mouthmovementsalsocanchangewhichspeechsoundsyouseemto hear.ThisisknownastheMcGurkeffect.
Theexamplesmultiply,andvisiondoesnotalwayswin.Playinga flash alongwithtwobeepscanmakeitlookliketwo flashes.Thisisthesoundinduced flash,avisualillusioninducedbysound.Ataponyourskinwith twobeepsalsocanfeelliketwotaps.Smellingvanillamakesfoodtaste sweeter(sweetnessenhancement).Thetimingofasoundaffectswhena lightseemstoilluminate(temporalventriloquism).Intherubberhand effect,strokingavisibleartificialhandwithafeathercanmakeitfeellike yourownhand.
Crossmodaleffectsinmultisensorycontextsdemonstratethatstimulationtoonesensorysystemcanalterperceptualconsciousnessthatis
associatedwithanothersenseinawaythatleadstoperceptualillusions. Suchillusionsarewidespreadandsystematic,andtheimpactcanoccur veryearlyinperceptualprocessing(seeChapter2).
Crossmodalillusionsshowthatonesensecanalteranotherduringa shorttimespan.Onesensealsocanaffectanotheroveralongertime span.Forexample,visionhelpscalibratemiddle-distancelocational hearing,anditenablesperceiversauditorilytodiscernrelativespatial locationsbetweenaudibleitems.Asaresult,congenitallyblindperceiverstendtohavespatialhearingdeficitsanddistortionsinauditory space.Nonetheless,skilledecholocationcansharpenspatialhearing.For instance,itimprovesthecapacityauditorilytodiscernlocationsand shapesofextendedbodies.Asaconsequence,duetodifferingcrossmodalinfluencesandperceptuallearning,theauditoryexperienceofa congenitallyblindperceiverinagivenacousticalsettingtypicallydiffers fromthatofasightedperceiver.Thus,overthelongterm,onesense canaffectperceptualconsciousnessassociatedwithanothersense.The changecanbeevidentevenwhenperceivingwithjustonesense.
Theimpactofonesenseonanother,atatimeandovertime,demonstratesthatfeaturesofperceptionandconsciousnessinvolvingonesense candependonanothersense.Inparticular,onagivenoccasion,both perceptualprocessesandperceptualconsciousnessassociatedwithone sensemaydependoncurrentorpastactivityinvolvinganothersensory system.Intheserespects,perceptioninvolvingonesenseisnotwholly independentfromtheothersenses.
Sensesalsoworktogetherincoordination,underminingtheassumptionofexplanatorycompleteness.Psychophysicistsandneuroscientists recentlyhavefocusedonhowinformationfromdifferentsensorysystemsissharedandintegratedbetweensenses.Somewaysofintegrating informationaredistinctivelymultisensory.Forexample,thereexist processesinwhichtheresponsetoacertaincombinationofsignals differsfromthesumoftheresponsestothesamesignalstakenindividually(itmaybegreaterorlesser).Theseresponsesarecalled superadditive.Superadditiveprocessesperformdifferentlyincombining multiplecuesfromhowtheyperforminregisteringcuesseparately. Theirbehaviorisnon-linearinthefaceofdistinctcues.Whatisnoteworthyhereisthatsomesuperadditiveresponsesoccurexclusively duringmultisensorystimulation.Forinstance,aneuronmay fireata
rateaboveitsbaselineonlywhenitreceivesbothauditoryandvisual stimulation.Sucharesponseisselectivelysensitivetojointauditoryand visualstimulation.Thisresponsebearsinformationaboutthejoint presenceofaudibleandvisiblefeatures.Perceptionevenimplicates regionsofthebrainthatrespondselectivelytomultisensorystimulation. Forinstance,thesuperiortemporalsulcusisheteromodal.Itisthought tobecriticalinprocessinginformationaboutspeechfromdifferent senses,anditismostactivewithmultisensoryinput.
Coordinationgoesbeyondintegration.Perceptionscienceprovides evidencethattypicalhumansubjectsinmultisensorycontextscandetect featuresthatarenotperceptibleusinganysinglesenseandthatcouldnot beperceivedusingseveralsensesindependentlyandmerelyinparallel.For instance,subjectscanbedifferentiallysensitivetotheidentityofanobject orattributethatisperceivedusingdistinctsenses.Thisisnotjustamatter ofrespondingtothesameitemorfeatureusingdifferentsensesatthe sametime.Itrequiresacoordinated,distinctivelymultisensoryresponse. Empiricalevidenceinformedbytheoreticalconsiderationsalsosupports theclaimthatsubjectsareabletoperceivenovelintermodalinstances ofcertainrelationalfeatures,suchassimultaneity,rhythm,motion,or causality.Forexample,asubjectmayperceivesomethingvisible’scausing somethingaudible,orarhythmcomprisingsoundsandtouches.Iarguein laterchaptersthatthejointuseofmultiplesensesalsocanrevealnovel featuretypesandqualitiesthatotherwisearenotperceptible.
Ineachoftheseexamples,thejointuseofmultiplesensesmakes possibleadistinctiveformofperception.Thisdoesnotjustamountto usingseveralsensesatonce,inparallel.Instead,thesensesworktogether todosomethingnewthatcouldnotbedoneseparately.Eachinvolvesa novel,multisensorywayofperceiving.
Thecoordinateduseofmultiplesensesdemonstratesthatmultisensoryperceptionandconsciousnessisnotexhaustedbywhatisassociated witheachoftherespectivesenses.Bythis,Imeanthatmultisensory perceptioninvolvesmorethanwhateachsensecoulddoonitsown, independentlyfromothersenses.Multisensoryperceptualconsciousness isnotasimplefusionofthatwhichbelongstoeachofthesenses,along withthesimpleunityofconsciousness.Intheserespects,theorizing abouteachsenseonitsowndoesnotsufficeforacompleteaccountof multisensoryperception.
Sensoryinteractionsthusmaketroubleforthetwoassumptionsthat underwriteunisensoryapproachestothescienceandthephilosophyof perception.Theychallengetheassumptionofexplanatoryindependence, accordingtowhicheachsensecanbetheorizedfullyinisolationor inabstractionfromtheothers.Andtheychallengetheassumptionof explanatorycompleten ess,accordingtowhichassemblingaccountsof perceivingwitheachoftherespectivesensesyieldsacomprehensive understandingofperception.
1.3Thesis
Thisbook’sthesisisthatthecoordinateduseofmultiplesensesenhances humansenseperceptionbyimprovingitinkeyrespectsandaugmenting itinothers.Inparticular,itincreasescoherenceandreliability,andit extendsthereachofourperceptualcapacities.Sensoryinteractionsthus makepossiblenewformsofperceptualconsciousness.
Claimsaboutsenseperceptionmayconcernperceptualprocessesor perceptualconsciousness.Myapproachinthisbookistoaddresseachon itsownterms,answeringconsiderationsrelevanttoestablishingconclusionsineachdomain.
The firstparttakesonperceptualprocesses.Inparticular,itdealswith perceptualfunctionsandcapacities.Chapter2concernscrossmodal interactions.Itarguesthatcrossmodalillusionsresultfromprincipled interactionsbetweensensesthatarepartoftypicalmultisensoryperceptualfunctioning.Sucheffectsarewidespread,andusuallydonotcause illusions.Infact,theytendtominimizeconflict,andtheyenableustoget thingsrightmoreoften.Sensoryinteractionsthusimprovetheoverall coherenceandreliabilityofperception.Multisensoryfunctioningthereforeenhancesperceptualcapacitieswealreadypossess.
Chapter3concernscoordinationamongsenses.Itarguesthattypical humanperceiversaredifferentiallysensitivetothepresenceofnovel featuresthatcouldonlybeperceivedthankstothejointuseofmultiple senses.Thischapterargues,onempiricalandtheoreticalgrounds,that perceiverscandetectanddifferentiatetheidentityofanobjectperceived withdistinctsenses,intermodalinstancesofrelationalfeatures,suchas simultaneity,motion,causality,andrhythm,andnovelfeaturetypes,
includingcomplex flavors.Therefore,whencomparedwithseveralsenses workingindependentlyormerelyinparallel,thejointuseofseveral sensesextendsthereachofourperceptualcapacities.Moreoftheworld canbeperceivedthankstocollaborationbetweensenses.
Thesecondpartturnstoperceptualconsciousness.Inparticular,it dealswithperceptualawarenessandexperience.Chapter4concerns awareness.Perceptualawarenessinvolveshowthingsseemorappear tobetoaconsciouslyperceivingsubject.Thischapterarguesthatsome formsofperceptualawarenessintypicalhumansubjectsareconstitutivelyandirreduciblymultisensory.Forinstance,inconsciouslyperceivinganovelintermodalfeature,perceptualawarenesscomprisesmore thanwhatbelongstoeachofthevarioussensestakenseparately.Asa consequence,multisensoryperceptionenablesnewvarietiesofperceptualconsciousness.
Chapter5concernsexperienceandphenomenalconsciousness.Some maycontendthatmultisensoryawarenessdoesnotguaranteethatany perceptualepisodehasmultisensoryphenomenalcharacter.Thischapter arguesthatnotallperceptualexperienceismodalityspecific.Inparticular, itarguesthatthephenomenalfeaturesofaconsciousmultisensoryepisodeneednotbeexhaustedbythoseassociatedwitheachoftherespective sensesalongwiththosethataccruethankstomereco-consciousness. Thus,multisensoryconsciousnessismarkedbyphenomenalcharacter thatnomerecollectionofco-consciousunisensoryepisodescouldhave. Inthisrespect,multisensoryperceptionmakespossiblenewvarietiesof phenomenalconsciousness.
Asaresult,theassumptionofindependenceandtheassumptionof completenessfallshort.Eachfailsasaclaimaboutthesubjectmatter targetedinaddressingempiricalandphilosophicalquestionsabout perception.Whethertakenasclaimsaboutperceptualprocessesand capacities,orasclaimsaboutperceptualawarenessandphenomenal character,theclaimthateachsenseisexplanatorilyindependentfromthe othersandtheclaimthatthesense-by-senseapproachisexplanatorily completemustberevisedinthefaceofmultisensoryperceptionand consciousness.
Multisensoryperceptionraisestwopressingquestionsofpsychological taxonomy.Inthefaceofmultisensoryphenomena,whatdifferentiates oursenses,andwhatdistinguishesperceptionfromcognition?Chapter6
