Emoji and Social Media Paralanguage
University of New South Wales, Sydney
University of New South Wales, Sydney
Michele Zappavigna
Lorenzo Logi
Shaftesbury Road, Cambridge CB2 8EA, United Kingdom
One Liberty Plaza, 20th Floor, New York, NY 10006, USA
477 Williamstown Road, Port Melbourne, VIC 3207, Australia
314–321, 3rd Floor, Plot 3, Splendor Forum, Jasola District Centre, New Delhi – 110025, India
103 Penang Road, #05–06/07, Visioncrest Commercial, Singapore 238467
Cambridge University Press is part of Cambridge University Press & Assessment, a department of the University of Cambridge.
We share the University’s mission to contribute to society through the pursuit of education, learning and research at the highest international levels of excellence.
www.cambridge.org
Information on this title: www.cambridge.org/9781009179812
DOI: 10.1017/9781009179829
© Michele Zappavigna and Lorenzo Logi 2024
This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press & Assessment.
First published 2024
A catalogue record for this publication is available from the British Library.
Library of Congress Cataloging-in-Publication Data
Names: Zappavigna, Michele, author. | Logi, Lorenzo, author.
Title: Emoji and social media paralanguage / Michele Zappavigna, University of New South Wales, Sydney, Lorenzo Logi, University of New South Wales, Sydney.
Description: Cambridge, United Kingdom ; New York, NY : Cambridge University Press, 2024. | Includes bibliographical references and index.
Identifiers: LCCN 2023027009 (print) | LCCN 2023027010 (ebook) | ISBN 9781009179812 (hardback) | ISBN 9781009179805 (paperback) | ISBN 9781009179829 (ebook)
Subjects: LCSH: Emojis. | Social media. | Language and the Internet. | Communication – Technological innovations.
Classification: LCC P99.63 .Z37 2024 (print) | LCC P99.63 (ebook) | DDC 302.2–dc23/eng/20230816
LC record available at https://lccn.loc.gov/2023027009
LC ebook record available at https://lccn.loc.gov/2023027010
ISBN 978-1-009-17981-2 Hardback
ISBN 978-1-009-17980-5 Paperback
Cambridge University Press & Assessment has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.
To Yaegan Doran for his ongoing interest in and generous feedback on this book, as well as his inspiring work on modelling tenor.
4.
3.2
5.
5.3
5.4
5.5
5.6
5.7
6.5
6.6
6.7
6.8
6.9
7.
7.3 Quarantine Hotel Food Review TikTok Comment Corpus
7.4 D ialogi C a ffiliation
7.5 The Role of Emoji in rallying around Shared Bonds
7.6 The Role of Emoji Invoking Laughter
7.7 The Role of Emoji in Negotiating Gendered Bonds about Appearance
7.8 The Role of Emoji in rejecting Bonds
7.9 Conclusion
8. C ommuning a ffiliation : The Role of Emoji in Communing around Bonds
8.1 Introduction
8.2 The #Domicron Corpus
8.3 C ommuning a ffiliation : C onvoke , f ine SS e , and P romote
8.4 Emoji Supporting [Boosting] and [Buttressing] Affiliation
8.5 Emoji Supporting with C onvoking Affiliation
8.6 Death-Related Emoji P romoting Bonds Critical of the COVID
8.7 Conclusion
9. Beyond Emoji–Text Relations: Factoring in Images and Other Semiotic
9.1
9.2
9.3
9.4
10. Conclusion
Figures
1.1 Examples of Twemoji rendering of emoji from the Unicode ‘Smileys and People’ category page 6
2.1 GIFs and digital stickers used to express ideas about vaccination in social media paralanguage 20
2.2 An overview of how an emoji code point becomes a picture character on a screen 24
2.3 Examples of the skin tone modifier acting on the Twitter rendering of the Vulcan salute base emoji 27
2.4 Examples of emoji sequences and their fall-back positions 28
2.5 Different renderings of the HOT BEVERAGE emoji, U+2615, across a selection of vendors 29
2.6 Two types of emoji presentation 30
2.7 Adding an emoji to a tweet through a searchable palette 34
2.8 Instagram emoji suggestion (left) and iMessage emoji prediction (right) in the Apple iMessage application 35
2.9 An example of a text annotated using WebAnno 41
3.1 The SFL model of language
3.2 Fundamental organisation of i D eation
3.3 Examples of linguistic [entity] types
3.4 Emoji realising ideational discourse semantic choices (factoring out language) 49
3.5 The a PP rai S al system 51
3.6 Intermodal convergence in relations between written verbiage and emoji
3.7 Emoji C oor D inating with a [thing entity] 63
3.8 Emoji interacting with a [state figure] 64
3.9 A single emoji interacting with two linguistic thing entities
3.10 Two field-related emoji interacting with different linguistic [entities]
3.11 An emoji interacting with multiple linguistic resources in a text
3.12 Emoji interacting with a [thing entity] 67
3.13 An example of emoji interacting with positive and negative interpersonal prosodies
3.14 Attitudinal prosody and emoji–language interaction 69
3.15 An example of a system network 70
3.16 The system of emoji–language C onvergen C e
4.1 The system of emoji–text S yn C hroni C ity 76
4.2 The punctuate system 79
5.1 The system of emoji–text C on C urren C e 93
5.2 The system of depiction 94
5.3 Classification taxonomy for emoji and language resources in Text (5.28) 97
5.4 The system of embellish 102
5.5 The images used in the ‘this is fine’ meme 108
6.1 Interpersonal emoji–text re S onan C e 115
6.2 The harmonise system 117
6.3 The [rebound] system 122
7.1 D ialogi C affiliation strategies 140
8.1 The system of C ommuning affiliation 170
9.1 Example of an image–macro meme 187
9.2 An example of an emoji graphic created with Slack (Anonymous, 2022) 191
9.3 Detail from an Instagram story containing a digital sticker (Knight, 2022) 192
9.4 An example of an Instagram story including a ‘GIPHY’ selected from a palette using the search term ‘coffee’ (Knight, 2022) 193
9.5 Examples of GIF and emoji palettes on Instagram 194
9.6 A simplified version of the relation of the body and visual media in a tweet
9.7 An example of intermodal C onvergen C e
9.8 An example of intermodal re S onan C e
9.9 A Twitter interaction saturated with positive attitu D e (light grey) targeting i D eation (dark grey) 202
9.10 An example of a GIF saturated with emblems of positive attitude 204
9.11 Intermodal emoji–co-text-GIF relations incorporating a bonding icon 205
10.1 The system of emoji–text C onvergen C e
10.2 A ‘goodbye’ GIF included in a popular tweet about leaving Twitter
10.3 The system of emoji–text S yn C hroni C ity
10.4 The system of emoji–text C on C urren C e
10.5 The system of emoji–text re S onan C e 218
10.6 Opposing bond networks negotiated in excerpt 229
1.1 Emoji frequency list for the Sydney Emoji Corpus and the Hot Beverage Emoji Corpus page 15
1.2 Emoji frequency list for the Hotel Quarantine TikTok Comment Corpus and the #Domicron Corpus 16
2.1 Selection factors for including a proposed emoji character in Unicode 21
2.2 The Unicode ‘Drink’ subcategory 31
3.1 Metafunction, register, discourse semantics 46
3.2 Ideational elements 47
3.3 Examples of types of phoricity 54
5.1 Frequency list for collocates of HOT BEVERAGE emoji (window span = 1) 91
5.2 Most frequent 3-grams incorporating the HOT BEVERAGE emoji 92
6.1 Most frequent collocates of the HOT BEVERAGE emoji that likely have an attitudinal function (window span = 5) 112
6.2 Most frequent 3-grams with a likely interpersonal function 113
6.3 The five most frequent repeated emoji in the Hot Beverage Emoji corpus 130
7.1 Most frequent emoji in the Quarantine Hotel Food corpus
7.2 The most common 3-grams in the corpus 155
8.1 Most frequent emoji and 3-grams in the #Domicron corpus
8.2 Most frequent death-related emoji in the corpus 183
9.1 Herring and Dainas’ (2017) coding of the pragmatic functions of graphicons 188
9.2 The most frequent 3-grams in the Hot Beverage graphicon specialised corpus
10.1 Twitter resignation excerpt affiliation summary
Acknowledgements
We would also like to thank our colleagues Shooshi Dreyfus, Aurélie Mallet, and Josh Han for their contribution to the early stages of gathering and analysing data about emoji. And to all our friends and loved ones, thank you for your indulgence and opinions when the subject of emoji comes up around the dinner table.
Emojis used under a creative commons licence, © X Corp. Full details:
Copyright 2023 X Corp and other contributors
Code licensed under the MIT Licence: http://opensource.org/licenses/MIT Graphics licensed under CC-BY 4.0: https://creativecommons.org/licenses/ by/4.0/
Part of this work was funded by the Commonwealth of Australia.
Note on the Text
The following conventions are adopted in this book when explaining the analysis:
1 System networks are drawn using the conventions defined in Systemic Functional Linguistics. References to the names of systems occur in S mall C a PS and features occur in [square] brackets.
2 Bonds identified in affiliation analysis are shown in ALL CAPS and square brackets and i D eation - attitu D e couplings are also shown in square brackets.
3 Emoji are referred to using the Common Locale Data Repository (CLDR) Short Names, which are presented in ALL CAPS.
4 Realisation of features from system networks are shown using –>1
5 ‘ … ’ is used to indicate text has been abridged.
6 Anonymised information is shown in braces.
1 The more traditional notation of a downward slanting arrow was not used in order to avoid confusion with the DOWN-RIGHT ARROW emoji.
Abbreviations
API Application Programming Interface
CLDR Short Name
Common Locale Data Repository Short Name
JSON JavaScript Object Notation
SFL Systemic Functional Linguistics
ZWJ Zero Width Joiner
NSW New South Wales
1 Social Media Paralanguage and Emoji
1.1 Introduction
This book explores the kinds of meanings that can be made with emoji, small graphical icons encoded as Unicode characters. Emoji are a frequent feature of digital communication and, as we will see, can enact a wide range of functions in tandem with language and other semiotic resources. We began writing at home during lockdown in Sydney in the midst of the third wave of the COVID-19 pandemic in Australia. At 11 o’clock each morning, the New South Wales State Premier Gladys Berejiklian provided a televised update on the COVID-19 case numbers. These addresses generated large volumes of social media reaction on platforms such as Twitter. Emoji proliferated in these reactions, appearing in the ambient audience’s responses to the case numbers, and in their criticism or praise of the government’s policies. For example, journalists made use of emoji to encapsulate, categorise, and evaluate information provided during the premier’s press conference, as in Text (1.1).
Text (1.1)
1431 new cases in NSW
12 deaths (119)
Woman in her 30s died (unvaccinated)
7.2 M
Ambulance 160 in ICU ( 0)
979 in Hospital ( 62)
7 new cases in Wilcannia (88)
53 new cases in Western NSW ( 11 Burke)
@gladysB says ‘The next 2 weeks will be the worst’
Not only do the emoji in this tweet1 visually organise the message into a list, they articulate both key content (e.g. + ‘cases’) and key feelings (e.g. + ‘deaths’), together with the written verbiage in the post.
Everyday social media users also discussed their reactions to the daily updates in Twitter interactions such as Exchange 1.1.
1 Tweets are short character-constrained messages posted to the social media service Twitter.
Exchange 1.1
Text (1.2) User 1: Anyone else feel like the day doesn’t start until @gladysB tells us exactly how screwed we all are?
Text (1.3) User 2: Same, about to get my 11am presser coffee made
Within this exchange, there appears to be a relationship between ‘coffee’ and the HOT BEVERAGE emoji. However, this emoji also seems to be important in terms of how interpersonal alignments are negotiated in the exchange, hinting at solidarity among the interactants and perhaps even some ironic enthusiasm. This book aims to deal with these kinds of relations – of emoji to any co-occurring text and to the interactive context – that are so central to how emoji make meaning in social media discourse.
1.2 The Semiotic Flexibility of Emoji
In order to function in a wide variety of discursive arenas, emoji need to have high semiotic flexibility. By virtue of their technical encoding, emoji are treated by social media platforms, and the devices on which they are navigated, as characters. This means that they can integrate seamlessly with the rest of the Unicode characters in the body of a social media post. This property also contributes to their semiotic capacity to make meanings relatively seamlessly with the written verbiage in the rest of the post, hereon the ‘co-text’. For example, as a prelude to the discussion in the chapters that follow, let us consider the sorts of meaning emoji realise in the example tweets: Text (1.4), Text (1.5), and Text (1.6).
Text (1.4) Lovely chatting with {Name} from @{Username} about the insane endurance feats of @{Username} & @{Username} and the approach to supporting such giants.
Grab a cuppa & have a listen!
[tiny URL]
It comes after brill insights from {Name} on his PW record
Text (1.5) I drink so much I feel like even thinking bout it
Text (1.6) Good Morning Tweepies!.. .Monday Morning!. .Slept in!. have to say hello and Goodbye. … see you this arvo! … [GIF]
In Text (1.4), we see how the emoji is used as a visual deictic, pointing to the location of the ‘tiny URL’2 in the text. The emoji also potentially references
2 A ‘tiny URL’ is short alias linking to a longer URL (Uniform Resource Locator) used to locate resources on the internet such as a web page.
the implicit call to action of ‘clicking on’ the URL since it resembles the mouse pointer icon of some operating systems. We thus have an emoji making meaning by visually depicting a material action and by referencing how a stylised rendering of that hand gesture has been adopted in another mode. In Text (1.5), the and emoji are used to directly represent the entity (a hot beverage) and action (vomiting) they depict, thereby effectively replacing the written text for these. Noteworthy here, however, is that while the emoji depicts a category of things, when interpreted in combination with linguistic co-text a reader would supply just one example of that category, such as ‘coffee’ or ‘tea’. Lastly in Text (1.6), the , , , , and emoji act to imbue the text with various emotive inflections. Unlike in Text (1.5), however, we are unlikely to interpret the emoji as indexing the expressions or gestural paralanguage of the author or of other textual participants. Rather we recognise that in this context the emoji coordinate with the affective and convivial meanings made in the text, contributing to the inclusive solidarity enacted. In these small examples alone, we can see evidence of the semiotic suppleness of emoji.
On their own, emoji are rather under-specified and stylised representations that hint at a range of ideas. However, in combination with their co-text, they can make a range of complex meanings. Emoji can optionally involve themselves in the organisation of the text as a coherent semantic unit, can contribute to the articulation of entities and activities, and can resound with the emotional implications of the text, as we will explore in detail in Chapters 4, 5, and 6. In addition, each of the examples we have just explored offers a potential ‘bond’ to any interactants in the ambient social media audience through the way that emoji and language are used to share and position particular values. We will deal with such social affiliation in Chapters 7 and 8, considering both interactive exchanges and ambient communing. Emoji also enter into relationships with multimedia beyond the written co-text, such as digital stickers, memes, and simple animations. For example, Text (1.6) includes an animated image (a GIF), created from a snippet of a 1966 episode, ‘Atlantic Inferno’, from the British TV series Thunderbirds. The GIF depicts the electronic marionette puppet character, John Tracy, sitting up abruptly in bed. In terms of the meanings the tweet makes as a multimodal text, the GIF appears to coordinate intermodally with broadly shared ideas about coffee, mornings, and alertness that we will explore in Chapter 9.
1.3 The Semiotic Complexity of Encoding and Rendering
‘Picture Characters’
Emoji are fascinating in their capacity to involve themselves in intricately specific meanings made within localised social media subcultures, at the same time as being malleable and under-specified enough to make meanings across
a vast array of social contexts. In terms of their visual appearance, they are presented to users of social media platforms such as Twitter, Instagram, Facebook, and TikTok as small, coloured glyphs. While it might be tempting to interpret the meaning-potential of emoji in terms of their visual appearance alone, they are complex semiotic resources. As we will explore in Chapter 2, emoji are ‘picture characters’ with some of the affordances of written characters and some of the affordances of images. One way of conceiving of this is to treat emoji as an iconographetic mode:
… the element icono (from the Greek for image), generally refers to pictorial characters; it does not matter whether these are iconic or symbolic characters in isolation. The second part of the term, graphetic, is understood in terms of the Greek word graphé (or writing) and will hence refer to all kinds of written characters: typed characters in the case of digital communication. (Siever, 2019, p. 129, original formatting)
The concept of a picture icon used for interactive digital communication arose with the practice of mobile text messaging in Japan. The popular term ‘emoji’ is itself derived from the Japanese 絵文字 (‘e’ = ‘picture’; ‘moji’ = ‘written character’). However, just what constitutes a ‘picture’ or the property of pictoriality can be difficult to establish (Wilde, 2019). Studies have noted a range of analytical quandaries regarding how emoji, as picture icons, can represent apparently complex concepts via a visual shorthand that is both imprecise and expansive enough to range across the multitude of meanings needed in a wide range of communicative situations. Some studies appeal to a contrast with visual images to understand this semiotic potential: ‘the specifics of the individual representation is often incidental to the underlying meaning of the ideogram [i.e. an emoji] – this is unlike images where the particulars of a given image are often more crucial than what it is representing generally (i.e. it is a photo of your dog, not just a photo representing the semantic notion of “dog”)’ (Cappallo et al., 2018, p. 2, clarification added). By this logic, there is an inherent difference between an image of a dog and the DOG emoji.
In order to meaningfully account for how emoji make meaning as picture characters, we also need to factor in the way they operate inside ‘semiotic technologies’ (Zhao, Djonov, & van Leeuwen, 2014) such as social media in which they are deployed. We thus need to isolate each relevant semiotic mode and resource, as well as their particular affordances and functions. Part of this involves understanding exactly what emoji are as technical constructs and the implications this has for how they can be analysed. Emoji are ‘encoded’ as characters with unique code points in the Unicode Standard. This standard is widely adopted worldwide as a consistent method for encoding typed information in software, enabling cross-platform interoperability. In terms of their visual representation, emoji are ‘rendered’ by software as glyphs which gives them their distinctive appearance and ‘colorful cartoon form’ (Davis & Edberg, 2018).
This process is controlled by the software companies who own the platforms and is not standardised by Unicode. For instance, consider Text (1.7).
Text (1.7) Day 5 of covid. Feeling a little better. Still no taste or smell. Does anybody know how long that takes to come back I can’t taste my coffee
The first emoji in the sequence ending Text (1.7) is the HOT BEVERAGE, which has the unique hexadecimal code point U+2615 and is rendered as the glyph . The final emoji, LOUDLY CRYING FACE, has the code point U+1F62D and is rendered as
As Unicode characters, emoji form part of a designed and institutionalised pictographic lexicon defined by the Unicode Consortium’s bureaucratic processes. This consortium is a conglomerate of entities that controls which characters are added to the Unicode Standard. A total of 674 emoji were added to Unicode in 2010 and their numbers have increased with each new version of the standard. The consortium’s voting members include technology companies such as Adobe, Apple, Facebook, Google, Microsoft, and Netflix, institutional members like the Ministry of Endowments and Religious Affairs of Oman and the University of California at Berkeley, supporting members like Emojipedia, and a variety of associate and individual members (Unicode Consortium, 2021a). The Emoji Subcommittee, a part of Unicode’s Technical Committee, evaluates proposals for new emoji based on various inclusion criteria, which will be discussed in further detail in Chapter 2.
Emoji’s visual presentation as coloured glyphs also depends on the communicative channel used to create or read a social media post. Unlike encoding, the visual ‘rendering’ of emoji is controlled by the particular vendor (operating, software system, or platform) applying the font to the Unicode characters. For instance, if I enter the HOT BEVERAGE emoji from Text (1.7) into Twitter, the emoji will be rendered as a Twemoji, the distinctive rendering style used by Twitter, and displayed as: . Therefore, it will have a different visual appearance compared to the emoji typed in Microsoft Word, which would display as . Twemoji were created by Twitter designers working in collaboration with the company Iconfactory (Twitter, 2020). Vendors display emoji differently to showcase the unique branding and visual style they wish to project. According to its designers, the flat, gradient-free visual design of Twemoji aims to convey ‘light-hearted, fun versions of the familiar icons users around the world know and love’ in a visual style ‘that would be easily identified as uniquely Twitter’s’ (Iconfactory, 2022). Twemoji have a minimalist, ‘flat’ design, with rounded shapes, dots, and lines used to articulate facial expressions, and without shading or 3D effects. In terms of colour palette, Twemoji ‘tend to use colors that are similar to – or at least complement – the Twitter logo’ (Gray & Holmes, 2020, p. 16). Figure 1.1 shows instances of Twemoji 1.3 Encoding and Rendering ‘Picture Characters’
from the ‘Smileys and People’ category in the current version, Twemoji 14.0 v. This release of Twemoji includes 3,245 emoji which map to Unicode 14. All emoji in this book will be presented in this style as it is open-source and our main corpora were collected from Twitter.
To refer to emoji, we will use the naming convention adopted in Unicode, the Common Locale Data Repository (CLDR) Short Name, for instance, ‘HOT BEVERAGE’ for ‘ ’. The Common Locale Data Repository (CLDR) project, run by the Unicode Consortium, aims to provide locale data (e.g. relating to different languages) in an interoperable XML format so that it can be used in a variety of computer applications. For instance, emoji will have different CLDR Short Names depending on language, and these may be provisional for new emoji and change with version releases. However, CLDR Short Names should not be confused with the meaning of an emoji, which will instead be analysed using close text analysis and corpus-based methods. As visible in Text (1.1) and Text (1.7), because emoji are technically characters, they appear in-line with the rest of the written characters in the post (e.g. letters, digits, and symbols). This also means that the user can enter emoji through the keyboard, or a palette menu, without the author leaving the post’s preparation window.
Figure 1.1 Examples of Twemoji rendering of emoji from the Unicode ‘Smileys and People’ category
While emoji in their rendering as glyphs are images, their visual meaning potential is constrained. They cannot incorporate free-form components and are limited to the small size of accompanying textual characters. Their visual rendering also tends to be stylised rather than realistic, as seen in Figure 1.1. Emoji tend to have a limited degree of visual specificity, in part due to their constrained size. Paradoxically, it is this under-specification that means they are open-ended enough to enter into distinct relations with their co-text, and, in effect, make more complex meanings. We will return to these ideas about emoji encoding and rendering in Chapter 2, where we consider some of the technical dimensions touched on here in more detail, as well as reflect on their important implications for creating and processing corpora containing emoji.
1.4 Emoji as a Social Media Paralanguage
Our approach in this book is driven by observation of the close relationship between emoji and the linguistic meanings in social media posts; in other words, how intertwined emoji appear to be with the meanings made in their co-text. This perspective aligns with a shift in emoji research from attributing independent linguistic meanings to emoji towards analytical frameworks that prioritise the relationship between emoji and language. Research exploring the consistency of emoji interpretation (without the provision of contextual information to interpreters) has found that only a few emoji have completely unambiguous meanings (Częstochowska et al., 2022). As emoji have proliferated and become ubiquitous across digital communication, their apparent pragmatic meanings have become diluted (Konrad, Herring, & Choi, 2020) or have undergone semantic drift (Arviv & Tsur, 2021). Accordingly, an individual emoji may be considered a resource that is ‘graphematically ambiguous, as the specific linguistic unit it refers to is not fixed but variable and determined by the context’ (Dürscheid & Meletis, 2019, p. 174). As such, emoji are heavily dependent on their linguistic co-text, which acts as ‘a clear verbal anchorage’ (Sampietro, 2016, p. 110) for the meaning made by the multimodal text as a whole.
As our brief suggestions about the meanings made in Text (1.4), Text (1.5), and Text (1.6) at the beginning of this chapter have suggested, emoji may enact a range of semiotic relations with their co-text. For example, they can serve either a referential role (replacing words) or a modal role (modifying or complementing the surrounding text) (Siever, 2019). Some studies analogise emoji with co-speech gesture and suggest that, like beat gestures accompanying speech, emoji ‘are not taking on the function of grammar, but acting in relation to written text’ (McCulloch & Gawne, 2018). Other studies broaden the scale of context to consider cultural meanings, and argue that interpreting emoji requires a degree of ‘familiarity with the cultural conventions of various aspects of contemporary society, along with an eclectic range of knowledge from
Eastern and Western written and gestural languages, sign languages and even fictional communication systems’ (Seargeant, 2019, p. 25).
Given their strong connection to the meanings conveyed in their written co-text, we approach emoji as a form of paralanguage. Paralanguage is semiosis, such as gesture, which is dependent on language (Abercrombie, 1968). This dependency is sometimes described as ‘parasitic’ since it depends ‘on the fact that those who use them are articulate (“linguate”) beings’ (Halliday & Matthiessen, 1999/2006, p. 606) and will vary depending on the kind of expression plane involved. For instance, in the case of paralanguage where the body is used for expression, this dependency might be ‘sonovergent’ with spoken language, that is, in-sync or in-tune with the phonological patterns of co-speech, or ‘semovergent’, that is, coordinating with linguistic meanings made in the co-speech (Martin & Zappavigna, 2019). Thus, rather than attempting to catalogue emoji as a kind of visual lexicon, we focus our attention on modelling the meaning-potential that emoji realise in concert with language.
Even where emoji appear in isolation in a text, they are likely to be dependent on co-occurring language within the broader context of situation, for instance, a preceding linguistic move in an exchange, as suggested by research on the role of images as moves in social media interactions (Jovanovic & van Leeuwen, 2018). The idea that emoji serve a paralinguistic function is also supported by corpus-based studies that have observed their semantic coordination (Gawne & McCulloch, 2019) and syntagmatic alignment (McCulloch & Gawne, 2018) with language. Our approach is also compatible with experimental studies that have suggested that, while emoji have some capacity for very simple sequencing and tend to interact with the linguistic grammatical structure, they do not seem to have developed their own grammatical structural potential (Cohn, Engelen, & Schilperoord, 2019). Emoji’s visually stylised under-specification is also one of the reasons that emoji tend to coordinate with more elaborated meanings construed in their written co-text.
However, while we consider emoji as a form of paralanguage, we do not follow the approach taken in some studies of directly equating emoji with gestures (Gawne & McCulloch, 2019; McCulloch & Gawne, 2018). This is because we view emoji as a distinct semiotic mode with its own particular affordances and meaning potential. These affordances are realised via the expression plane of the ‘picture character’; a different expression plane to modes which realise their meaning via embodiment (e.g. gesture, posture, voice quality, etc.). As previously mentioned, emoji are a ‘designed’ resource with specific digital affordances, and it is crucial to isolate these affordances to understand their semiotic potential. In simple terms, rather the studying emoji as if they were images or gestures, we study them for their own distinct meaning potential, taking into account their unique design and digital functions. Studies which liken emoji to gesture appear to be motivated by
the apparent iconicity of popular emoji that depict stylised facial expressions and body gestures (e.g. CRYING FACE , ROLLING ON THE FLOOR LAUGHING , THUMBS UP3 , OK HAND , CLAPPING HANDS , etc.). However, a direct equivalence of emoji and gesture risks proscription of emoji’s meaning potential – as Albert observes, ‘the formal analogy between emoji faces in general and the corresponding facial expressions provokes the misleading inference that there must also be a functional analogy’ (2020, p. 68). While it may be tempting to suggest that emoji ‘share various properties and characteristics with other systems, they’re actually adding something quite new to the resources we use to express ourselves’ (Seargeant, 2019, pp. 35–6).
This is not to say that emoji are not agnate to other kinds of paralanguage. A dimension that gesture and emoji do share in common is their general dependency on their linguistic co-text. Employing McNeill’s (1992) diagnostic criteria for determining the degree to which semiotic modes can function independently of language, Gawne and McCulloch (2019) observe that ‘gestures and co-speech emoji are closely integrated into meaning with the accompanying speech/text’. This study suggests that emoji may be likened to gesture since they ‘do not decompose into smaller morphological units, they do not show predictable syntax, their meaning is shaped by context-specific use, and there is accepted variation in form’ (Gawne & McCulloch, 2019). According to this account, unlike language, emoji are global and synthetic, non-combinatoric, context-sensitive, and do not have standards of form.
1.5 A Social Semiotic Perspective on Emoji–Text Relations
The central goal of a social semiotic approach to communication is to understand how the different resources available to language users make meaning in the contexts in which they are used. In order to achieve this aim, not only do we need a theory of meaning and tools for analysing meaning-making, but we need a principled means for exploring how communicative modalities combine. In addition, we require ways of managing this complexity so that we achieve an elegant description of such semiotic coalescence. To systematically explore the meaning made in emoji–text relations, we will draw on social semiotics and its multimodal concern with understanding the semiotic systems that operate within and across modalities. We will approach these meanings methodically as ‘bundles of oppositions’ (Ngo et al., 2021, p. 8), adopting the relational theory of meaning that underlies work in Systemic Functional Linguistics (SFL). This kind of approach treats semiosis as a resource rather than a collection of rules
3 Emoji glosses are sourced from https://emojipedia.org/ (accessed 11 November 2020), an emoji dictionary developed by professional lexicographers.
and treats the relations between choices in meaning as key to understanding how those choices function in real-world contexts.
Our functional approach manifests as a concern with three essential functions of language, termed ‘metafunctions’ by Halliday and Hasan (1985): the ideational (how experience is represented), interpersonal (how relationships are enacted), and textual (how text is organised). For instance, the oppositions in meaning we touched on when considering Text (1.1) at the beginning of this chapter can be seen to span what an SFL perspective on language views as field (the domain of experience), tenor (the interpersonal construction of relationships and stances), and mode (the organisation of the information flow of text) (Halliday & Matthiessen, 2004). In terms of field, the emoji in Text (1.1) contribute to co-construing the kinds of topics and experiences at stake: the MICROBE , AMBULANCE , and HOSPITAL converge with verbal meanings about a health emergency. In terms of tenor, the PENSIVE FACE and BROKEN HEART resonate with details about deaths in the written verbiage to suggest negative emotions about this emergency. In addition, in terms of mode, the emoji themselves act as visual bullet points, organising the text into a list, at the same time as thematising the key information elaborated in the co-text. It is this kind of combinatorial meaning-making that we will focus on in the chapters which follow.
Inspired by work attempting to model paralanguage using Systemic Functional Semiotics recently consolidated in Ngo et al. (2021), one of the major assumptions that we make in this book is that language and other modalities coordinate inter-semiotically to make meaning. As such, we view written language and emoji as complementary semiotic resources and are interested in how they are interwoven, or more technically ‘converge’ to create meaning in social media texts. This assumption of complementarity is also in line with earlier research into how images and written language coordinate in picture books where three types of relations of convergence were described: concurrence in ideational meaning, resonance in interpersonal meaning, and synchronicity in textual meaning (Painter & Martin, 2012; Painter, Martin, & Unsworth, 2013). These types of relations were used by Ngo et al. (2021) to explore how gesture and co-speech interrelate in embodied semiosis, resulting in the social semiotic model of paralanguage which informs the analytical approach adopted in this book.
Parkwell’s (2019) metafunctional analysis of the meaning-potential of the TOILET emoji aligns with our approach and serves as a noteworthy example of previous social semiotic work specifically focused on emoji. The TOILET was used by popular musical artist Cher to discuss former US President Donald Trump on Twitter without using his name. The study draws on the perspective of multimodality (as outlined by Kress and van Leeuwen, 2001; O’Halloran, 2004) and Zappavigna’s (2018) metafunctional analysis of
1.6 Using Corpora to Understand
hashtags, to demonstrate how a single emoji can express experiential, interpersonal, and textual functions. The conclusion of the study highlights the highly contextual and flexible nature of emoji as a modality that is ‘likely to continue to shift and morph with the changing needs and contexts of social media users’ (Parkwell, 2019, p. 9). Another social semiotic study, conducted by He (2022), analysed the use of emoji in news story comments on the Chinese social media platform Weibo. This study adopted the ‘intermodal coupling’ of semiotic resources as its analytical unit, building upon the notion that meanings created through different modes can be complementary, as proposed by Painter et al. (2013). It found that emoji realise two distinct interpersonal functions: construing attitude targeting linguistic co-text and enacting social bonds with interactants around shared attitudes. These functions encompass emoji’s capacity to ‘not only directly reflect a commenter’s attitude through the depiction of facial expression and gesture, but … to guide readers to detect the buried implications in a text’ (He, 2022, p. 12). Other social semiotic studies of Weibo have also identified that emoji offer expanded pragmatic potential in relation to the co-text, serving as ‘a multimodal layer of meaning in which emojis may not only substitute, reinforce, or complement text, but also perform speech acts, highlight subjective interpretations, and convey higher degrees of informality and/or casualness’ (Yang & Liu, 2021, p. 166).
1.6 Using Corpora to Understand Emoji
The majority of corpus-based studies on emoji have been undertaken within the realm of computational science, utilising a corpus-driven methodology and incorporating machine learning techniques. These studies frequently aim to leverage emoji to enhance sentiment analysis (Kralj Novak et al., 2015; Shiha & Ayvaz, 2017) and typically view emoji as ‘emotion tokens’ for monitoring sentiment polarity (Wolny, 2016). Some studies aim to create emoji sentiment lexicons in an effort to surpass classification methods that are based on manual annotation or CLDR Short Names (Fernández-Gavilanes et al., 2018; Kimura & Katsurai, 2017; Kralj Novak et al., 2015), while others utilise the Unicode description as a means of classifying emoji (Eisner et al., 2016). A number of studies have centred on emoji sense prediction and disambiguation (Barbieri et al., 2018; Guibon, Ochs, & Bellot, 2018; Shardlow, Gerber, & Nawaz, 2022), and have monitored longitudinal changes in emoji semantics (Robertson et al., 2021).
This methodological context has proven fertile for research into how emoji have been used during the COVID-19 pandemic, primarily through the lens of quantitative studies using corpus-driven or sentiment analysis techniques to analyse emoji frequency and density in social media discourse (Das, 2021). This line of inquiry holds promise for yielding valuable insights that can
benefit domains such as public health initiatives and finance. For example, some studies have proposed new methods for understanding the gender-based disparities in the effects of COVID-19 (Al-Rawi et al., 2020) and for charting the correlation between emotional uncertainty and market volatility (Lazzini et al., 2021). Especially germane to this book is the vein of research examining the role of emoji in discourse related to remote work, including studies that examine emoji usage in videoconferencing chat (Dürscheid & Haralambous, 2021) and closed captions (Oomori et al., 2020).
Quantitative studies across various domains have shown a general interest in determining the most commonly used emoji. Unicode releases up-to-date information on emoji usage patterns, including the Unicode Emoji Subcommittee Chair’s report on the most frequently used emoji in 2021 (Daniel, 2021). Additionally, various tools such as ‘Emoji Tracker’ (Rothenberg, 2013) aim to monitor emoji uptake in real-time, offering a dynamic insight into emoji trends and usage patterns. The top ten emoji used worldwide in 2021, according to Unicode (Daniel, 2021), were the following.
1 FACE WITH TEARS OF JOY
2 RED HEART
3 ROLLING ON THE FLOOR LAUGHING
4 THUMBS UP
5 LOUDLY CRYING FACE
6 FOLDED HANDS
7 FACE BLOWING A KISS
8 SMILING FACE WITH HEARTS
9 SMILING FACE WITH HEART EYES
10 SMILING FACE WITH SMILING EYES
While this kind of frequency list cannot tell us the ‘meaning’ of these emoji, it does suggest that they are most likely involved with construing broadly positive meanings, depending on how they interact with their co-text.
Efforts within corpus linguistics to study emoji usage are relatively new, likely due to the technical challenges posed by the unique features of emoji that can complicate the use of traditional corpus analysis tools such as concordance software. Chapter 2 will delve into the specific challenges posed by emoji as special characters in corpus processing, exploring both their encoding and rendering. These issues require the analyst to pay close attention to what is actually being counted. While emoji might be roughly interpreted as a ‘lexical unit’, they are in fact often composed of Unicode character sequences. This means that corpus linguistic software will not necessarily be able to capture all emoji unless it has the capability for recognising these sequences as a single unit. Thus some kind of work-around for concatenating relevant emoji sequences will be required to meaningfully process emoji
Book (Zappavigna & Logi, 2021). Accordingly, this book relied on a custom script, along with a python library, to accurately count and inspect emoji concordance lines. This approach ensured that Unicode emoji sequences, which are not readily recognisable by standard concordance systems, were properly accounted for. Corpus-based studies of emoji in linguistics have nevertheless attempted to draw on standard corpus methods such as analysis of frequency lists, concordance lines, and n-grams. Some corpus-based studies have combined pragmatics and corpus analysis methods (Li & Yang, 2018; Pérez-Sabater, 2019; Sampietro, 2019). For example, Kehoe and Gee (2019) undertook a large-scale data-driven corpus pragmatic analysis of emoji use on Twitter, using a corpus of 40 million English and German language tweets. Replicating the results of previous research, the FACE WITH TEARS OF JOY was the most frequent emoji in this dataset, followed by LOUDLY CRYING FACE (in English) and the RED HEART (in German). The study employed collocational analysis to disambiguate different emoji uses. For instance, it distinguished multiple meanings for FOLDED HANDS , including ‘thanking, pleading, praying or giving a high five’ (Kehoe & Gee, 2019, p. 2). The study noted that collocational span, as well as the frequent repetition of emoji within tweets, posed challenges for corpus analysis of emoji patterning.
Another relevant study adopting a corpus analytical approach is a multimodal analysis of Facebook posts incorporating emoji and annotated for images (Collins, 2020). This study also found FACE WITH TEARS OF JOY to be the most frequent emoji. Echoing the challenges noted by Kehoe and Gee (2019), the study suggested that the traditional concept of a collocational span, established as useful for work on written text, was problematic for exploring emoji and for relations of images to text. These issues were somewhat ameliorated when dealing with images in the corpus by employing a large collocational span of 365 tokens (which was determined from the longest text in the corpus), while keeping a keen eye on confidence measures. The author decided, drawing on Spina’s (2019) work on emoticons, that emoji ‘should also be investigated within a narrower collocational span (at the “type” unit level), since research has shown that there are conventions for the position of emoji, which interact with the syntax and punctuation of written material’ (Collins, 2020, p. 190). In our own work we adopt a similar approach, drawing on collocational patterns and n-grams where possible to both motivate and buttress our more qualitative discourse semantic analysis.
1.7 Corpora Analysed in This Book
In this book we employ a corpus-based approach to understanding the use of emoji in social media posts, informed by a social semiotic orientation to discourse analysis. This kind of methodological position is sometimes termed
