Governing artificial intelligence in the media and communications sector

Introduction

Since 2018 several European policy efforts have aimed to incorporate ethics principles and European fundamental rights into the governance of artificial intelligence (AI) (Floridi et al., 2018; Hagendorff, 2020). The European Council (2017) announced its priority to establish a “high level of data protection, digital rights and ethical standards” for AI systems (p. 2). The ensuing Communication from the European Commission (2018) claims to address the ethical and legal questions surrounding AI, which is reiterated in the 2019-2024 Agenda for Europe (von der Leyen, 2019). The subsequent multistakeholder deliberations through the High Level Expert Group on Artificial Intelligence (AI HLEG), including the AI Ethics Guidelines and the subsequent European Commission White Paper on AI and the AI Alliance multistakeholder efforts, all aim to render broad abstract principles actionable. This includes the Assessment List for Trustworthy AI (ALTAI) developed by the AI HLEG, a checklist for AI developers and deployers to use when implementing AI ethics principles in their operations (HLEG, 2020).

Notwithstanding this “principle proliferation” (Floridi & Cowls, 2019), such initiatives generally fall short in addressing the broadly defined impact of AI technologies on the media and communication (MC) sector. Neither existing policy initiatives nor the literature fully consider how AI affects this sector, limiting their focus most notably to automated content moderation on social media or news sites, or the distribution of disinformation across social networks. This paper fills this gap by exploring the interplay between AI technologies and the MC sector. This sector encompasses a variety of contemporary forms of technologically supported content and communication. From the vantage point of critical media studies and critical social science, we offer two key contributions. Firstly, we identify key blind spots in the dominant trustworthy and risk-based approach to AI being pursued by the European Union. These blindspots include a lack of attention in AI policies to the potential impact of AI on the infrastructural and structural asymmetries in the MC sector, a tendency to ignore the cultural, social and democratic importance of diverse media to societies and a failure to attend to the ways in which AI can be used to undermine social cohesion and collective rights. Our contribution exclusively assesses the key non-regulatory policy documents up until Spring 2021, due to their primary relevance in establishing the “Proposal for a regulation on a European approach for AI” (AI Act) in April 2021 (European Commission, 2021). Secondly, we propose a multi-level approach to analysing the key policy challenges to governing AI. Such a perspective is needed for an understanding of the sectoral specificities and the socio-technical consequences of AI in the MC sector, more generally. A multi-level perspective is needed to understand the key areas in which AI is currently applied in this sector. To this end, we identify four dominant uses of AI in the MC sector: automating data capture and processing, automating content generation, automating content mediation and automating communication. The four levels are the outcome of extensive desk research and our expert committee meetings, part of a much larger European multi-stakeholder initiative established by the Atomium institute in Brussels. For each of these uses, we identify overarching opportunities and risks stemming from the use of AI in relation to the AI HLEG requirements for trustworthy AI. In doing so, we propose a novel analytical and macro-level framework to identify the key issues raised by the application of AI in the MC sector. We lay the groundwork for identifying who should be responsible for addressing these issues and how they might do so.

The article comprises five sections. After the introduction, Section 2 delineates how the MC sector and related AI implementations are conceptualised in the article. Section 3 summarises and assesses state-of-the-art European AI policy initiatives and governance, offering a critical perspective. Section 4 analyses the MC sector and through case studies shows how the 7 key requirements, as suggested by the AI HLEG, operate across four different levels. Section 5 offers recommendations to ensure that AI development and adoption is compliant with the key requirements for trustworthy AI in the MC sector, while reflecting on the adequacy of the 7 key requirements and the EU risk-based regulatory approach from the perspective of media and communications scholarship. We conclude by recommending three areas of oversight for governing AI: addressing data power asymmetry, empowerment by design for mitigating risks, and cooperative responsibility through stakeholder engagement.

Understanding AI and the media and communications sector

Defining and setting the boundaries of the MC sector is no simple matter given the fast evolution of digital technologies, corporate mergers and new market entrants. MC includes multiple forms of technologically supported content, interaction and communication. It embraces digital media, i.e. digitised legacy media, as well as born-digital platforms. The latter act as socio-technical intermediaries that collect and process data to enable and steer communication (Helberger, Pierson & Poell, 2018). Mobile applications (apps) are increasingly central to MC businesses, as much digital media communication today takes place in the context of an app. These apps are made available within a (mostly) privatised platform and communications infrastructure that is optimised to exploit data (Pierson, 2021; Nieborg & Helmond, 2019). Many forms of digital media increasingly rely on the data analysis of users and content. Data is collected, processed and evaluated in the MC sector for many purposes, including the automated personalisation of content (e.g. news recommendations) and targeted advertising. Finally, the media content and communication sector rely upon existing telecommunications and internet infrastructure. The latter includes wired and wireless mobile internet and communication service providers. These operators and their services are a crucial part of the MC sector, and constitute what Winseck (2019, p. 176; 2020, p. 1) calls the “network media economy”. From our perspective this sector includes those media industries usually included in the definition of creative and cultural industries, but also telecommunication service providers and parts of the wider computing industry, which provide infrastructure for the circulation of user data, media content and communication.

Our understanding of AI is based on the AI HLEG (2019a), which defines AI as human designed systems implemented in a digital or physical environment in the form of software-based systems or hardware devices. AI systems process and assess data based on reasoned decision-making, producing outputs and suggesting relevant actions to achieve given goals. In other words, AI systems intervene in society on the basis of the data they collect, and decisions the software is enabled to make. This process is guided by a set of symbolic rules or a numeric model, and also by AI’s ability to learn from its environment and previous outputs (HLEG, 2019a).

Given that AI systems provide information about the world, make recommendations that may impact our everyday decisions, and provide value-based judgements, citizens must be able to trust them (Ferrario et al., 2019). According to the AI HLEG, trustworthiness should represent a “prerequisite for people and society to develop, deploy and use AI” (HLEG, 2019b). This trust has been severely impacted by public revelations about the use of AI by states and companies for the purpose of public surveillance and manipulation (Kerr et al. 2020). Research has also revealed potential and actual harms, biases and discrimination (see, e.g., Hintz et al., 2019; Eubanks, 2018). These developments and findings suggest that the MC sector needs to be vigilant to ensure that AI systems and data processing are, and remain, trustworthy.

The Charter of Fundamental Rights of the European Union provides a sound basis upon which to build policies that protect citizens from being unwillingly influenced by AI and having their personal data used without their consent (European Parliament, 2000). The Charter emphasises the importance of freedom of expression, access to information and the protection of personal data, all essential aspects of democracy and the public sphere, but all are undergoing transformation in the current digitalisation and datafication phase of MC. Special consideration is also given to children, minority groups such as migrant communities and marginalised communities including LGBTQIA+ and refugees. Therefore, we interpret “trustworthy AI” as AI systems that are worthy of the European public’s trust from a fundamental rights perspective. We focus on the European context, given the EU’s ongoing commitment to establish a regulatory and societal framework for “trustworthy” and “human-centric” AI. The next section briefly examines the evolving EU regulatory regime and identifies some critical blindspots in relation to AI applications in the MC sector.

Critical blindspots in the European perspective on AI governance

An emphasis on democratic values, fundamental rights and the rule of law is integral to the EU vision for the development, deployment and governance of AI. At the same time, AI is a European research and innovation priority. When European principles and values are translated into emerging policies, which are also meant to support innovation, we see conflicting goals and varying emphasis to governing AI between EU institutions and national governments. The framing of emerging regulatory approaches is especially relevant for the MC sector, where automated decision making and content generation technologies are increasingly prevalent. Consider, for example, the use of AI across the entire news production process: automated content production (Willens, 2019), machine translation of text (ADAPT, 2021), correction and transcription services (Marr, 2018), facts and potential fake news (Cassauwers, 2019) or data-driven tools to personalise users’ news feeds and recommend other content (Warren, 2021).

The AI HLEG, appointed by the European Commission (EC) in June 2018, consisted of 52 members and included companies and industry representative groups, academics from computer science, law, philosophy and economics, and representatives of civil society groups and digital rights organisations. Notably, no academics from media and communication studies, nor people from traditional public service media or new digitally native European media companies were represented, whereas a number of telecommunications and technology companies, including Alphabet-Google, were. Neither the draft nor final report of the AI HLEG mentioned media specifically, although both mention that policy and regulation needs to pay attention to “situations with asymmetries of power or information” (HLEG, 2019b: 13). There was also no discussion of AI-driven mis- or disinformation.

The final AI HLEG report, “Ethics Guidelines for Trustworthy AI”, (HLEG, 2019b) listed seven requirements for trustworthy AI: 1) human agency and oversight, 2) technical robustness and safety, 3) privacy and data governance, 4) transparency, 5) diversity, non-discrimination, and fairness, 6) societal and environmental wellbeing, and 7) accountability. The AI HLEG final report was influential in shaping a “European” approach to governing AI in that it further substantiated the concepts of “ethical” and “trustworthy” AI (Hasselbalch, 2020), and was foundational for the Assessment List of Trustworthy Artificial Intelligence (ALTAI) (HLEG, 2020) and the February 2020 White Paper on Artificial Intelligence (EC, 2020). In the White Paper, the EC highlights the use and potential impact of AI (1) for information selection and content moderation by online intermediaries; (2) in tracing people’s daily habits; and (3) in creating information asymmetries by which citizens might be left powerless. The White Paper considers AI systems “high-risk” if “both the sector and the intended use involve significant risks” (EC, 2020), particularly if safety, consumer rights or fundamental rights are at stake. It is noteworthy that the White Paper sees no specific “high-risk” issues, nor mentions any use examples from the media, creative or cultural industries – while health, security, energy, farming, transport etc. are all mentioned. While the high-risk distinction promotes a risk-based approach to AI, incremental, long-term and societal-collective risks are not addressed. The White Paper, furthermore, focuses on AI as a tool or technology which disregards larger power imbalances and power structures that derive from the use of AI in a larger digital communication infrastructure.

Subsequently, the EC published a proposal for the regulation of AI in April 2021. This “AI Act” (AIA) adopts the risk-based regulatory approach. Certain applications of AI, including AI systems that could cause physical harm (MacCarthy & Propp, 2021), would be banned under the AIA, and high- and limited-risk AI systems (European Commission, 2021) would be regulated. High-risk AI systems would only be put on the market if they have conducted conformity assessments in advance and meet obligations such as transparency, human oversight, record-keeping, robustness, accuracy and security (European Commission, 2021). The proposed regulation focuses mainly on AI system providers and relies on companies and organisations to self-assess, and on national supervisory authorities to oversee compliance with the regulations (MacCarthy & Propp, 2021). The AIA does not include specific consumer rights, such as the right to redress.1 However, consumers must be informed when they are interacting with an AI system, or when their emotions are being detected (for the purpose of providing relevant services, products, etc.).

To summarise, the European policy documents we have analysed emphasise generating the conditions for trustworthiness in the development, deployment and use of AI, but fail (to date) to consider the long-term, incremental risks for democratic processes and fundamental rights. The AIA draft currently fails to provide sufficient guidance on accountability or redress mechanisms to address harms. In these documents there is a lack of attention paid to the MC sector, even though there is a call for sector-specific analysis of threats and risks. A healthy democracy depends on independent and diverse news and media outlets, especially in their role as the Fourth Estate (Csaky, 2021). However a growing part of our society, up to two thirds (Newman et al., 2021), consumes news, and consequently, forms their opinions from sources on social media. Developments like AI-driven recommender systems in digital media have helped spread online disinformation, and have been implicated in the radicalisation and polarisation of societies, partly due to their advertising-driven preference for arousing emotional and controversial content (Coalition to Fight Digital Deception, 2021; Hagey & Horwitz, 2021; Ribeiro et al., 2020). These are documented risks which are detrimental to democracy and social cohesion. It is not clear, however, if the AIA would classify these applications as high or medium risk.

From the MC sectoral perspective, our analysis of European policies related to AI governance thus reveals several fundamental blind spots. In general the focus on algorithms, data and information in AI policy documents prioritises transmission and access issues. This approach treats data and information as bits to be transferred rather than being concerned with the diversity, veracity or meaning of the content that is being generated, issues which have long been at the heart of media policy and regulation in many European countries. Indeed, we concur with the critique that the European perspective on AI regulation is technologically determinist and lacks the attention to redress relative market powers (Veale, 2020). For us the key blindspots are the lack of attention to: 1) the infrastructures of surveillance capitalism and datafication which prioritise data gathering and engagement (Zuboff, 2019), 2) a trend towards increasing corporate and market concentration and power asymmetries in MC sectors (often by large companies based and regulated outside Europe), 3) a tendency to ignore the important cultural, social and democratic role of the media, including the importance in the diversity of content and representation (Napoli, 2019), 4) a lack of understanding of how AI is used to categorise people in new ways, and to target and adapt content and communications towards them in opaque ways, and finally, 5) a preference from social and digitally native media and technology companies for self-governance and minimal content or communication oversight. Without due consideration to these blindspots a generic commitment to trustworthy AI principles and a reliance on corporate self-certification is not, in our opinion, sufficient. In short, most European AI policies neglect relevant issues that undermine trust in AI from an MC perspective or – at best – label them as low-risk.

Background and methodology

This paper emerged following the involvement of the authors in the AI4People programme run by the Atomium European Institute for Science, Media and Democracy (Atomium-EISMD 2018-2020). The Atomium initiative aimed to develop 7 industry-specific frameworks and policy recommendations for governing AI technology. Given the lack of EU AI MC sector-specific policy proposals as identified above, the authors of this paper welcomed the chance for intersectoral dialogue in the context of the Atomium initiative. They were invited to join a 12 member committee of the AI4People programme focused on the MC sector, appointed by Atomium-EISMD in consultation with chair Jo Pierson, including equal numbers of MC academics (6) and personnel from media and technology industry associations and multinational companies (6).2

The starting point of the Committee’s work was the “Key Requirements for Trustworthy AI”, as proposed by the European Commission (EC) High-Level Expert Group on Artificial Intelligence, henceforth AI HLEG (HLEG, 2019b). The goal was to discuss and debate how to translate the AI HLEG’s 7 key requirements for Trustworthy AI into a framework that could be implemented in the MC sector, and to identify opportunities and risks specific to the sector. This committee produced a 30-page report in December 2020, which is available online (Pierson et al., 2021). This article brings together the academics from the committee to reflect further on the AI4People Media and Communication committee discussions, (March–November 2020), and to update our analysis to take account of subsequent policies and draft regulations which emerged during the first half of 2021.

The four levels of intervention of AI in the MC sector described in the next section were identified through both desk-based research and discussion, as well as debate and analysis by the MC committee members, based on a multistakeholder approach (Raymond and DeNardis, 2015). First, we conducted extensive desk research on existing frameworks at the interface of AI, media, communication and EU policy. Next, a total of 44 case studies containing best practice and problematic practice use cases of AI in the MC sector were solicited from committee members and our wider networks based on their expertise and practical experience. Additional information, literature and explanation on the cases was provided. The case study repository was examined and overlapping thematic issues were identified. In a final step, the AI case studies were grouped into four themes, cross-combined with the 7 key requirements and used as the basis to develop our multi-level approach to evaluate trustworthy AI in the MC sector, and to assess the adequacy of current EU AI policies to govern the use of AI in the MC sector.

A multi-level analysis of trustworthy AI in media and communication

In order to evaluate the current European approach to trustworthy AI, from the MC sector perspective, we developed a multilevel framework of current AI applications. These were:

• Level 1: Automating data capture and processing
• Level 2: Automating content generation
• Level 3: Automating content mediation
• Level 4: Automating communication

The four levels broadly correspond to different stages in the typical (big) data life cycle (Jagadish et al., 2014).3 The four MC AI levels are not mutually exclusive, and in what follows we illustrate our discussion at each level with concrete examples from our case study research. This approach allows us to identify tensions between the 7 key requirements, while also considering complex issues that occur predominantly on one level (e.g. advertising in data capture and processing), but may also occur at other levels. We also identify high-risk applications in the MC sector which should be considered in EU AI policy.

Level 1: Automating data capture and processing

The first level encompasses a variety of AI technologies that systematically capture and process data in the MC sector. This typically includes data capture and processing by digital media, platforms and websites for profiling, personalisation, inferential predictive analytics, targeted advertising, etc. It also includes facial and voice recognition systems capturing emotional expressions, as well as GPS/location tracking and VR/AR.

The principles of human agency and oversight and privacy and data governance are most pertinent at this level. First, the EU legislative framework, in particular the General Data Protection Regulation (European Parliament & Council, 2016) and consumer protection standards, already protect individual rights to allow citizens to make informed and independent choices. These protections also apply to automated data capture and processing in AI systems: citizens should always be able to decide if and how they choose to use a certain service or be tracked by it. European citizens using a MC service should have the right to decide what and how much data will be collected, what it will be used for, where it originates from and how it will be shared. Despite the existence of these legal protections and ethical principles, many AI-driven applications and services rely on an advertising and a data-driven business model that remains opaque. This is particularly relevant for online behavioural advertising, where internet users’ behavioural data (website visits, clicks, mouse movements, etc.) and metadata (browser type, location, IP address, etc.) are collected and processed to create profiles that will be used to personalise ads and improve conversion rates.

Civil society groups have argued that automated advertising systems with real-time bidding (RTB) have been capturing and processing data in prohibited and unethical ways (Irish Council for Civil Liberties, 2021; Information Commissioner’s Office, 2019). RTB is the system by which advertisers bid on the possibility of instant targeted advertising to website visitors by using personal data that is collected through tracking and is shared with all bidders. Even advertisers who do not win the auction receive personal data that ascertains the visitor’s interest at the conclusion of the auction. Some advertisers allegedly participate in the auctions merely to enrich their data sets. The targeting is based on profiles of users built via extensive and persistent tracking of online and (possibly) offline activities (e.g. via cookies or pixels). Users’ past behaviour is a category of these profiles, but so are inferred preferences and affinities, often including sensitive categories protected by the GDPR. For example, Alphabet-Google and several data brokers have been accused of violating EU data protection rules by harvesting and processing peopleʼs personal data to build detailed online profiles, including information on sexual orientation, health status and religious beliefs (Scott & Manancourt, 2020), and in turn grouping people based on their assumed interests rather than their personal traits, becoming a digital construction of groups based on categories of identity (Cheney-Lipold, 2011).

In addition, given the value that can be generated by collecting and processing extensive amounts of (personal) data and using it to personalise content and advertising, special attention is needed to safeguard a level playing field between large and small operators to ensure the diversity of operators and content. According to the GDPR (Recital 71), personalisation through profiling involves the automated processing of personal data to evaluate individuals’ personal aspects, to analyse or predict their economic situation, health, preferences or behaviour. The GDPR (Art. 22) grants individuals the right not to be subjected to automated decision-making if it has a significant effect on their lives. When automated profiling and personalisation have significant effects, it will only be allowed if users consent. Although all companies and organisations in the media and advertising ecosystem who process European citizen data are subject to the GDPR, larger operators with more financial resources have made challenging regulatory judgements under the GDPR since it was introduced. Regardless, they are also better able to pay the fines imposed. Smaller media companies and community media with fewer resources are not only losing advertising revenue to a small number of multinational platform companies, they are also unable to compete on technical grounds. Ultimately, this model may result in less diverse and less local content generation. The consolidation of personal data in fewer hands might also increase, and perversely, negatively affect people’s rights and freedoms overall, but also their cultural and communication rights, and media diversity, per se (Kerr et al., 2019). An interesting attempt to counter this consolidation of power is the development of pre-commercial data pooling and communal processing initiatives to benefit several (competitive) companies at once in local markets (e.g. Ads & Data in Belgium) (van Zeeland et al., 2019).

To summarise, the use of AI to automate data capture and processing reinforces problems of operator concentration and power in the sector, and further threatens cultural and communication diversity, as well as rights. The application of existing legislation, such as the GDPR, and existing trustworthy AI principles, have thus far failed to address these issues. The speed, scale and opaqueness of new AI methods, and the complexity of data and advertising infrastructures, are undermining the ability of end users and media companies to give and receive informed consent.

Level 2: Automating content generation

The second level refers to online content produced by automated systems, either entirely or in combination with human agents. Examples of common AI uses in content generation are news reporting apps (based on user preferences),4 translation tools, and – critical from a democratic perspective – disinformation and deep fakes: images or videos in which identities and/or elements are digitally manipulated. Such AI-generated content may appear in text, image, audio or video. How much content is generated automatically and to what extent users are informed of this fact is unclear.

Journalists, and the companies and organisations they work for, play a major role in providing trustworthy news and information. Thus, human agency and oversight, transparency, accountability and societal well-being are highly relevant to automated content generation. In data journalism, for instance, AI helps to identify patterns in large datasets. AI-driven tools can suggest titles and photos, help writers find a new angle to a topic, and produce draft versions of articles. Automated systems assist the journalist in writing the story, but the journalist should remain the primary storyteller (Willens, 2019). Assistive content generation with human editorial input and human oversight allows for faster content generation. The news generation in some news genres, especially fact-based ones, is highly automated. For instance, specialised natural language processing tools can generate sports articles and financial reporting (Peiser, 2019); recent projects even involve video reporting (Chandler, 2020).

The use of AI at this level has had a significant impact. The increasing pace and efficiency of automated news production can put pressure on smaller newsrooms, which usually do not have access to large datasets and robust AI-systems (Helberger et al., 2019). Higher automation of content generation can lead to job losses, impacting the diversity of journalists, as well as the bias, accuracy and diversity of content and wider societal well-being (Srnicek, 2017; Lindén & Tuulonen, 2019). Automating content generation will also create new kinds of jobs, which may require substantial upskilling, training and education. In addition, content produced by AI systems is often not flagged as such to the user, undermining the principle of transparency. Since trust in news and information is often associated with its producers, great care needs to be taken with the use of AI tools in content production, as this can reduce the trustworthiness of content overall.

Technical robustness is also highly relevant in producing and translating texts. Machine translation risks replication biases (e.g. stereotypes, gender and racial biases) and errors from training datasets, affecting the principle of diversity, non-discrimination and fairness. Robust machine translation, in contrast, can help to preserve the EU’s linguistic and cultural plurality. For example, the ADAPT research centre (2021) in Ireland develops datasets and intelligent models that automatically translate online content for native speakers of low-resource languages, and make important content available to people in their language of choice. Projects have focused on developing resources for Irish, Serbian, Basque and non-European languages, including Hindi. Their approach employs both AI and humans, rather than fully automated systems.5

On social media platforms and in messaging services, deep fakes generated by AI-driven tools are increasingly common. These formats simulate in an increasingly realistic manner a speech or an action, usually by a public persona (such as politicians, celebrities and actors). Such false information is often generated without the individual’s knowledge, and viewers may be unaware that the video was altered. Thus the principles of human agency and societal wellbeing are at stake. Deep fakes can foster the spread of contentious and harmful content like “fake news”, disinformation and hate speech. A recent study found that 72 percent of people reading an AI-generated news story thought it was credible (Leibowicz, 2019). Generating deep fakes and producing disinformation challenges media integrity, erodes trust and has negative implications for democracy. This relates to the two main roles of media in a healthy democracy: correctly informing citizens to support meaningful political choices (“watchdog function”) and creating a diverse public forum that allows different ideas and opinions to be shared and discussed (Helberger, 2019; Balkin, 2018). Deep fakes and disinformation also impact diversity, non-discrimination and fairness. Current forms of redress to tackle these issues seem to be inadequate (Fanni et al., 2022). Deep fakes are specifically mentioned in the EC draft regulatory framework on AI, but without specifics on redress (European Commission, 2021). Most recently, the EC introduced a new redress and liability regime applicable to AI systems causing damage, which is a first step towards meaningful redress rights for individuals. The EU AI Liability Directive (European Commission, 2022) strengthens users’ rights to access to information and alleviates victims' burden of proof in the case of harm by the fault or omission of an AI provider, developer or user.

Level 3: Automating content mediation

The third level involves automated filtering systems in the distribution and moderation of online content and advertising. AI technologies in content distribution include recommender systems for entertainment and social media content, online news aggregators, and programmatic advertising (including RTB). These provide user-specific and context-conforming content. Other AI systems moderate content to detect and tackle contentious content like fake news, mis- and disinformation (European Parliament. Directorate General for Parliamentary Research Services, 2019a) and harmful content (Lacoma, 2020).

Employing automated filtering systems in online content and advertising mediation tasks entails the principles of diversity, non-discrimination, fairness, human agency and oversight. For example, years after the initial research into discrimination in online employment ads, higher salary positions are still advertised to predominantly (assumed) male users (Burke, Sonboli & Ordonez-Gauger, 2018). When AI technologies act as gatekeepers and set agendas in the online sphere, they can co-determine what people see or do not see, as well as what content users can generate online. This could affect freedom of expression, media diversity and the plurality of voices (Helberger et al., 2018). Algorithmic content distribution can constrain access to diverse information and create “filter bubbles” leading to “echo chambers”, i.e. personalised content. Online platform recommender systems tend to magnify hyperactive users’ interests and content, or certain producers, while passive users’ interests and other forms of content become invisible (Content Personalisation Network, 2020; Papakyriakopoulos et al., 2020). However caution is needed to not overestimate the social impact of these algorithmic developments (Löblich & Venema, 2021; Möller et al., 2018). Hence, political microtargeting and opinion formation could become subject to (un)intentional algorithmic manipulation and media content could become less diverse (Feezell, Wagner & Conroy, 2021).

Algorithmic filtering systems are required given the high volume and fast-paced production of online content. AI systems are crucial assistants to humans in the evaluation of harmful content such as child abuse, racism and harassment. These tools may reduce the physical and mental impact of this work on the human moderators (Ofcom, 2019) by flagging harmful content, blurring particularly harmful sections or engaging in “visual question answering”. AI can also be used to tackle malicious online behaviour directly using notifications or chatbots that make the user aware that a post contains harmful content, or the technology can create a short delay in the posting process to encourage the user to rethink the message (Statt, 2020). AI systems can also provide alternative content suggestions that are more positive, but express the original message. In both instances, the human agent – a content moderator or user – ultimately decides. However, AI systems have also been found to have many limitations when it comes to understanding the complexity and diversity of fast changing communications (Kerr et al., 2020; Gowra et al., 2020).

Transparency and accountability are two major principles for tackling algorithmic decision-making and countering potential abuse. The highly complex architecture of AI moderating tools, and their proprietary nature, makes it difficult to understand and assess their decision-making process (European Parliament. Directorate General for Parliamentary Research Services, 2019b). Inappropriate algorithmic standards, combined with a lack of resources, can result in negative content being left online, and/or appropriate content being removed. Transparency is the first step to developing systems of accountability for algorithmic decisions and judgements, making these processes legible to different stakeholders, e.g. general public, certain sectors, human agencies and/or oversight bodies. Abuse of algorithmic filtering systems could result in censorship, which would violate democratic principles. To prevent this, civil society groups have made software for algorithmic auditing methods open source to expose personalisation algorithms on social media and shopping platforms, for example Algorithms Exposed (ALEX) (see Beraldo et al., 2021 and Milan & Agosti, 2019). Their goal is to empower both advanced users and low-skill users with data extraction tools, and to enhance individual and societal knowledge of algorithmic content mediation.

Level 4: Automating communication

The fourth level in our analysis includes all forms of interaction and communicative actions and infrastructure enabled by AI, including: speech and language technologies, chatbots, smart speakers, voice assistants and automated marketing communication. All these AI systems that simulate a realistic conversation, and the encoding and decoding of conversational messages and data from users fall into this level.

The most important requirements for automated communication are human agency and oversight, diversity, non-discrimination, fairness and transparency. First, transparency would entail all AI-empowered communication channels being open, or making much of their data infrastructure auditable, including how information and output is compiled. Transparency would also enable users to better understand how their conversation data is used and evaluated. Especially in automated marketing and communication, users can fall prey to misleading messages or biased information, which could be avoided if transparency in marketing practices were mandatory.

Open curated datasets may improve the experience of AI systems users in terms of diversity, non-discrimination and fairness, and prevent the distortion of conversation and information between humans and machines. Further, considering the European linguistic diversity, automated communication systems can already discriminate against or disadvantage certain linguistic minorities. Finally, users should be able to choose whether they want to interact with a chatbot or with a human being, as reflected in the principle of human agency and oversight. This also links to the principle of accountability if a consumer-oriented chatbot, e.g. for a bank, gives imprecise or incorrect information that can cause harm or damage.6 Accountability in automated communication must therefore include redressing automated decisions by chatbots. The AIA refers to ensuring accountability mechanisms for affected persons through transparency and traceability, as well as ex-post controls (European Commission, 2021). This fits in the computer science tradition of explainable AI (XAI). XAI is defined as the practice of improving understandability, trustability, and the manageability of emerging AI systems (Meacham et al., 2019), however all too often the focus is on making these systems understandable to AI experts rather than end users.

The societal and environmental wellbeing principle is crucial to overall decisions about whether it is suitable, viable, sensible and sustainable to adapt AI enabled automated communication for certain cases. Automated communication AI tools require extensive datasets and the training of resource intensive models in order to automatically translate online content. While machine translation may increase social inclusion for speakers of low-resource languages (ADAPT, 2021), thereby making important content accessible to linguistically diverse communities, these models also risk replicating biases and errors from training datasets. The fact that employment opportunities for translators are significantly diminished by automated communication AI technologies threatens the societal wellbeing principle, which means that automated communication AI companies were required to mitigate the impact of their technologies on the traditional job sector. In addition, an increasing concern is the environmental costs of training large AI models, running statistical analysis and cooling data infrastructures. This is a hidden societal cost at all levels that deserves critical attention (Crawford, 2021).

Towards trustworthy AI in media and communication

This section spells out some key considerations for addressing the critical blindspots in current AI policy in Europe, moving towards the implementation of trustworthy AI in MC across the four levels discussed above. Three considerations are proposed: addressing data power asymmetries, empowerment by design for mitigating risks, and ensuring cooperative responsibility through diverse stakeholder engagement.

Addressing data power asymmetries

The use of AI is eroding meaningful, intentional and informed consent in the automatic data capture and processing of personal data by the MC sector; it is also deepening power asymmetries between MC organisations and their audiences/users. Do citizens and customers know when their personal data is being collected by AI-enabled systems? Users should be informed when their volunteered, observed or inferred personal data is being used to train machine learning algorithms before they decide whether to opt in. AI-driven businesses could be obliged to provide explanations on an ongoing basis, as has been suggested by Article 29 Working Party (WP29) in their guidelines on valid consent, which were recently updated by the European Data Protection Board (EDPB, 2020). Such positive data obligations enable citizens to act with agency in the face of data power (Kennedy et al., 2015). Clear and informed opt-in consent, combined with transparency obligations for algorithmic training and testing with user data, needs to be mandated for the MC sector. One approach might be to more widely embrace algorithmic registries, which detail the datasets that models were trained on and how algorithms are utilised, as Amsterdam and Helsinki have done (Moltzau, 2020).

Individual consent decisions will not prevent every harm stemming from abuses of automated personal data processing in the MC sector. While individuals may consent to the use of information about their emotions, political affiliation, health or sexual orientation, this use may have large-scale repercussions, for which individual choices cannot bear responsibility. Political microtargeting offers an example: individual users may consent to the use of data about their political preferences and emotional states on a platform, but aggregated data on attitudes and emotions linked to political preferences may be used to automatically manipulate other citizens’ voting behaviours with potentially major societal effects, as the Cambridge Analytica scandal has illustrated (The Guardian, n.d.). Prevention of such malignant applications of automated data processing cannot rest on an individual’s shoulders, but should be addressed with national or European regulation based on an interdisciplinary, multi-stakeholder engagement of fundamental rights and public values. We recommend multi-stakeholder processes of value-sensitive design to investigate how predictive analytics, sentiment analysis and emotional AI threaten the integrity and autonomy of digital media users, especially in online behavioural advertising and synthetic content production. This approach is in line with the remit of Art. 22 GDPR (“Automated individual decision-making, including profiling”).

Explainability is a complex, nuanced problem, considering the variety of European citizens and the complexity of some AI systems. Research and funding to increase AI transparency and explainability should be pursued and prioritised. However questions remain as to the adequacy of current technical solutions, as well as the fact that current XAI efforts are largely driven and funded by military organisations (e.g. DARPA) (Taddeo et al., 2021; Whittaker, 2021). Technical solutions should be supplemented with (co)regulatory efforts for establishing more transparency from digital platforms vis-à-vis independent regulators, especially on matters like internal processes for handling harmful and illegal content through algorithms and AI. Then we can better regulate platform-specific architectural amplifiers of contentious content, e.g. in recommendation engines, search engine features (such as autocomplete), features like “trending” and other mechanisms that predict what we want to see next. This approach fits with ex-ante principles-based co-regulatory approaches, when authorities attempt to force digital media companies to be more proactive in achieving state-determined public policy objectives. In that way, self-regulatory efforts can be better enforced, while avoiding purely punitive measures that penalise unlawful behaviour only after harm has been done (Vermeulen, 2019). Hence, we recommend strengthening research on process-based regulation and oversight on AI transparency and explainability, especially with regard to architectural elements for algorithmic amplification, e.g. avoiding widespread calls for violence by prohibiting algorithms from favouring and amplifying sensational news. Anticipatory data management policy should be a future priority in EU legislation. Privacy and data governance are moving targets, and new categories of personal data will be utilised, collected and created. Therefore, it is imperative to consider emerging sensitive AI-related personal identifiers, whether emotional data or even predictive AI systems, in the implementation and enforcement of GDPR, ePrivacy regulation revision and other European data-related policy initiatives (e.g. Data Governance Act, Digital Services Act, Digital Markets Act).

Empowerment by design for mitigating risks

Using AI technologies for MC activities like profiling, content and advertising personalisation threatens human agency, transparency and safety. Comprehensive solutions must be investigated and developed to address these threats. This fits in with the idea of “empowerment by design”, i.e. building infrastructures and systems to give (organised) citizens (e.g. civil society organisations and activists) the agency to safeguard and strengthen their fundamental rights and the public interest (Pierson & Milan, 2017).

The advertising industry in the MC sector is a complex and multi-sided market with a multitude of actors, many of them intermediaries, such as networks of third parties with tracking technology, intermediary data brokers and exchanges, all competing in the market of RTB and automated auctions (Binns et al., 2018). Sensitive information about individuals – ethnicity, gender, sexual orientation, religious beliefs – can be inferred and used for online behavioural advertising and affinity profiling, i.e. grouping people according to their assumed interests rather than their personal traits, and through soft biopolitics: how biopower constitutes a population, and how that population is diffusely developed (Cheney-Lipold, 2011). Several scholars and digital rights organisations have made suggestions for empowering consumers against the illegal or unethical automated capturing and processing of their personal data (BEUC, 2020, pp. 16-17).

AI is also used in emotion detection in the MC sector. All these uses risk manipulating human behaviour and applying biased models and data from one context to another. These systems could “nudge” people into taking certain behavioural actions (Mele et al., 2021); infer belief and attitude; and incentivise use or concealment of certain emotional expressions. Emotion detection could likewise exacerbate existing biases against vulnerable groups. A set of actions could help to mitigate the risks posed by emotion detection AI. First, users should have to opt-in if any of their data is being used to detect emotions. User consent should be mandatory for MC businesses, as required by EU data protection law, with few exceptions. However, consenting to data collection is not sufficient; the issue is its application at scale. Those developing emotion detection AI need to facilitate full transparency and evaluation by relevant experts such as sociologists, psychologists, anthropologists, media scholars and psychiatrists. Overall, emotion detection in the MC sector is a high-risk application, and next to being listed as such in forthcoming regulations, a range of mitigating measures will need to be developed.

Cooperative responsibility and accountability through stakeholder engagement

Many concerns that arise in this sector can only be tackled by means and resources beyond the sector. One way to secure societal and environmental well-being is to develop a shared responsibility for governance between civil society (users, human rights and consumer groups), public and private companies (platforms, technology and content producers, advertising and data service providers) and governments (education, policy and regulation). This type of “cooperative responsibility” requires digital media platforms, policy makers, users and other possible actors to develop a division of labour for managing their responsibility in terms of their role in public values (Helberger et al., 2018; van Dijck et al., 2018). The EU preliminary principle demands that the MC sector can only be “compliant” in the presence of an oversight body with a transparent system of compliance, an appeal (redress) and a complaints procedure. Any such system would also have to acknowledge and interface with legacy governance structures in the MC sector. An important first step will be to ensure that any advisory or co-regulatory body must include legacy media, community media and citizen representatives to balance the current dominance of computer, digital and telecommunications stakeholders.

The MC sector, especially online intermediaries, should be encouraged to set up an architecture that empowers users. Standardised methodologies and deliberation fora for facilitating ongoing exchange with specific user communities should be put in place. Also, media production cycles, such as designing websites, should involve multiple stakeholders, who should then be required to consider diversity, non-discrimination, fairness and human rights, as online game developers were by the ISFE-Council of Europe guidelines (Directorate General of Human Rights and Legal Affairs, 2008). We recommend incentivising and developing educational trajectories, guidelines, training, materials and tools for professional and technical staff (e.g. via online courses or curriculum changes in higher education) among the respective stakeholders to better understand and engage with the EU’s fundamental human rights and the principle of trustworthy human-centred AI.

Any system that might emerge should consider the stance some countries have taken in relation to the “traditional media” industries: the Press Councils and Press Ombudsman in Ireland that oversee both print and online only news media (Press Council of Ireland, 2020) and the communications regulation bodies like Ofcom in the UK which oversee telecoms and broadcast media (Ofcom, 2020). Any governance system might also need to work with established worker unions like the National Union of Journalists, both to train and educate journalists, and for whistleblowing and worker rights. In sum, we recommend strengthening workers’ rights and public interest values in the media as new AI systems evolve and emerge.

Final reflections

Consideration for the specificities of the MC sector and media examples are largely missing from current EU AI policy documents, as well as in the proposed AIA. This is surprising given high profile scandals involving the misuse of AI in the sector, including the Cambridge Analytica scandal. Further, media experts have been largely absent from key European AI policy initiatives since 2018, and thus we argue that critical social, cultural, democratic and power related issues related to the use of AI in the MC sector are insufficiently considered. Transnational computing, telecommunications and digital platform companies, industry trade bodies and academics have tended to dominate EU policy initiatives in this space. Indeed, our own experience of the AI4People initiative concurs with this, and the expert committee was quite ‘thin’ in terms of the diversity of stakeholders involved (Raymond & DeNardis, 2015).

Our analysis has proposed that AI systems have a substantial and multi-level impact across the European MC sector. This article identified four levels in which AI applications operate in the MC sector: automating data capture and processing, automating content generation, automating content mediation and automating communication. We analysed the core uses and risks of AI applications across these levels, identifying where trustworthy AI principles came to the fore, and we also identified the use of emotion recognition AI as high-risk.

It is clear that existing European charters and legislation have not been sufficiently applied to deal with the emergence of a complex socio-technical system of datafication that is built around the technical, economic and political power of a small number of very large commercial transnational corporations operating at both the national and European level. The shift from trustworthy AI to a risk-based but self-administered compliance based system in the AIA does not – in our opinion – auger well for the protection of fundamental rights, freedoms and public values. Neither is it sufficient to protect environmental or social sustainability. The AIA, as currently written, appears to exist in isolation from other policies addressing the digital media sector, and future work will need to address the relationship between AI and other EU digital policies, including the Digital Services Act package.

In concluding our paper, we offered three key areas where more research and regulatory efforts are needed to influence the development, deployment, use and governance of AI systems in the MC sector. These efforts need to involve a range of disciplines and be cognisant of the existing European policy context and sectoral specifics. The first related to the need to address data power asymmetries by rethinking informed consent in the context of ML powered AI, and in the deployment and use of predictive analytics by a complex ecosystem of companies. This will also require new systems of transparency and oversight for the use of automated decision making and the dominance of a small number of commercial companies in many of these technologies. The second relates to the need to develop systems that empower citizens and that operate in the public interest, from algorithmic registries to algorithmic auditing, and from skill development to the right to redress. Finally, we consider it necessary to move away from multi stakeholder initiatives towards new systems of cooperative responsibility and accountability to protect citizen and worker rights. While the goal of a principles based and value-driven AI governance framework across the EU is to be commended, too few voices and sectors currently dominate the policy agenda.