Technology, Autonomy, and Manipulation

Since 2016, when the Facebook/Cambridge Analytica scandal began to emerge, public concern has grown around the threat of “online manipulation�?. While these worries are familiar to privacy researchers, this paper aims to make them more salient to policymakers — first, by defining “online manipulation�?, thus enabling identification of manipulative practices; and second, by drawing attention to the specific harms online manipulation threatens. We argue that online manipulation is the use of information technology to covertly influence another person’s decision-making, by targeting and exploiting their decision-making vulnerabilities. Engaging in such practices can harm individuals by diminishing their economic interests, but its deeper, more insidious harm is its challenge to individual autonomy. We explore this autonomy harm, emphasising its implications for both individuals and society, and we briefly outline some strategies for combating online manipulation and strengthening autonomy in an increasingly digital world.

influencing someone by constraining their options, such that their only rational course of action is the one the coercer intends (Wood, 2014). Persuasion and coercion carry very different, indeed nearly opposite, normative connotations: persuading someone to do something is almost always acceptable, while coercing them almost always isn't. Yet persuasion and coercion are alike in that they are both forthright forms of influence. When someone is trying to persuade us or trying to coerce us we usually know it. Manipulation, by contrast, is hidden-we only learn that someone was trying to steer our decision-making after the fact, if we ever find out at all.
What makes manipulation distinctive, then, is the fact that when we learn we have been manipulated we feel played.11 Reflecting back on why we behaved the way we did, we realise that at the time of decision we didn't understand our own motivations. We were like puppets, strung along by a puppet master. Manipulation thus disrupts our capacity for self-authorship-it presumes to decide for us how and why we ought to live. As we discuss in what follows, this gives rise to a specific set of harms. For now, what is important to see is the kind of influence at issue here. Unlike persuasion and coercion, which address their targets openly, manipulation is covert. When we are coerced we are usually rightly upset about it, but the object of our indignation is the set of constraints placed upon us. When we are manipulated, by contrast, we are not constrained. Rather, we are directed, outside our conscious awareness, to act for reasons we can't recognise, and toward ends we may wish to avoid.
Given this picture, one can detect a hint of deception. On our view, deception is a special case of manipulation-one way to covertly influence someone is to plant false beliefs. If, for example, a manipulator wanted their partner to clean the house, they could lie and tell them that their mother was coming for a visit, thereby tricking them into doing what they wanted by prompting them to make a rational decision premised on false beliefs. But deception is not the only species of manipulation; there are other ways to exert hidden influence. First, manipulators need not focus on beliefs at all. Instead, they can covertly influence by subtly tempting, guilting, seducing, or otherwise playing upon desires and emotions. As long as the target of manipulation is not conscious of the manipulator's strategy while they are deploying it, it is "hidden" in the relevant sense.
Some argue that even overt temptation, guilting, and so on are manipulative (these arguments are often made by proponents of the "non-rational influence" view of manipulation, described above), though they almost always concede that such strategies are more effective when concealed. 12 We suspect that what is usually happening in such cases is a manipulator attempting to covertly tempt, guilt, etc., but failing to successfully hide their strategy. On our account, it is the attempted covertness that is central to manipulation, rather than the particular strategy, because once one learns that they are the target of another person's influence that knowledge becomes a regular part of their decision-making process. We are all constantly subject to myriad influences; the reason we do not feel constantly manipulated is that we can usually reflect on, understand, and account for those influences in the process of reaching our own decisions about how to act (Raz, 1986, p. 204). The influences become part of how we explain to ourselves why we make the decisions we do. When the influence is hidden, however, that process is undermined. Thus, while we might naturally call a person who frequently engages in overt temptation or seduction manipulative-meaning, they frequently attempt to manipulate-strictly speaking we would only say that they have succeeded in manipulating when their target is unaware of their machinations.
Second, behavioural economists have catalogued a long list of "cognitive biases"-unreliable mental shortcuts we use in everyday decision-making-which can be leveraged by would-be manipulators to influence the trajectory of our decision-making by shaping our beliefs, without the need for outright deception.13 Manipulators can frame information in a way that disposes us to a certain interpretation of the facts; they can strategically "anchor" our frame of reference when evaluating the costs or benefits of some decision; they can indicate to us that others have decided a certain way, in order to cue our intrinsic disposition to social conformity (the so-called "bandwagon effect"); and so on. Indeed, though deception and playing on people's desires and emotions have likely been the most common forms of manipulation in the past-which is to say, the most common strategies for covertly influencing people-as we explain in what follows, there is reason to believe that exploiting cognitive biases and vulnerabilities is the most alarming problem confronting us today.14 Talk of exploiting cognitive vulnerabilities inevitably gives rise to questions about nudging, thus finally, we briefly distinguish between nudging and manipulation. The idea of "nudging", as is well known, comes from the work of Richard Thaler and Cass Sunstein, and points to any intentional alteration of another person's decision-making context (their "choice architecture") made in order to influence their decision-making outcome (Thaler & Sunstein, 2008, p. 6). For Thaler and Sunstein, the fact that we suffer from so many decision-making vulnerabilities, that our decision-making processes are inalterably and unavoidably susceptible to even the subtlest cues from the contexts in which they are situated, suggests that when we design other people's choice-making environments-from the apps they use to find a restaurant to the menus they order from after they arrive-we can't help but influence their decisions. As such, on their account, we might as well use that power for good, by steering people's decisions in ways that benefit them individually and all of us collectively. For these reasons, Thaler and Sunstein recommend a variety of nudges, from setting defaults that encourage people to save for retirement to arranging options in a cafeteria in way that encourages people to eat healthier foods.15 Given our definition of manipulation as intentionally hidden influence, and our suggestion that influences are frequently hidden precisely by leveraging decision-making vulnerabilities like the cognitive biases nudge advocates reference, the question naturally arises as to whether or not nudges are manipulative. Much has been written on this topic and no consensus has been reached (see, e.g., Bovens, 2009;Hausman & Welch, 2010;Noggle, 2018a;Nys & Engelen, 2017;Reach, 2016;Selinger & Whyte, 2011;Sunstein, 2016). In part, this likely has to do with the fact that a wide and disparate variety of changes to choice architectures are described as nudges. In our view, some are manipulative and some are not-the distinction hinging on whether or not the nudge is hidden, and whether it exploits vulnerabilities or attempts to rectify them. Many of the nudges Thaler and Sunstein, and others, recommend are not hidden and work to correct cognitive bias. For example, purely informational nudges, such as nutrition labels, do not seem to us to be manipulative. They encourage individuals to slow down, reflect on, and make more informed decisions. By contrast, Thaler and Sunstein's famous cafeteria nudge-placing healthier foods at eye-level and less healthy foods below or above-seems plausibly manipulative, since it attempts to operate outside the individual's conscious awareness, and to leverage a decision-making bias. Of course, just because it's manipulative does not mean it isn't justified. To say that a strategy is manipulative is to draw attention to the fact that it carries a harm, which we discuss in detail below. It is possible, however, that the harm is justified by some greater benefit it brings with it.
Having defined manipulation as hidden or covert influence, and having distinguished manipulation from persuasion, coercion, deception, and nudging, it is possible to define "online manipulation" as the use of information technology to covertly influence another person's decision-making, by targeting and exploiting decision-making vulnerabilities. Importantly, we have adopted the term "online manipulation" from public discourse and interpret the word "online" expansively, recognising that there is no longer any hard boundary between online and offline life (if there ever was). "Online manipulation", as we understand it, designates manipulation facilitated by information technology, and could just as easily be termed "digital manipulation" or "automated manipulation". Since traditionally "offline" spaces are increasingly digitally mediated (because the people occupying them carry smartphones, the spaces themselves are embedded with internet-connected sensors, and so on), we should expect to encounter online manipulation beyond our computer screens.
Given this definition, it is not difficult to see why information technology is uniquely suited to facilitating manipulative influences. First, pervasive digital surveillance puts our decisionmaking vulnerabilities on permanent display. As privacy scholars have long pointed out, nearly everything we do today leaves a digital trace, and data collectors compile those traces into enormously detailed profiles (Solove, 2004). Such profiles comprise information about our demographics, finances, employment, purchasing behaviour, engagement with public services and institutions, and so on-in total, they often involve thousands of data points about each individual. By analysing patterns latent in this data, advertisers and others engaging in behavioural targeting are able to detect when and how to intervene in order to most effectively influence us (Kaptein & Eckles, 2010).
Moreover, digital surveillance enables detection of increasingly individual-or person-specific vulnerabilities. 16 Beyond the well-known cognitive biases discussed above (e.g., anchoring and framing effects), which condition most people's decision-making to some degree, we are also each subject to particular circumstances that can impact how we choose.17 We are each prone to specific fears, anxieties, hopes, and desires, as well as physical, material, and economic realities, which-if known-can be used to steer our decision-making. In 2016, the voter micro-targeting firm Cambridge Analytica claimed to construct advertisements appealing to particular voter "psychometric" traits (such as openness, extraversion, etc.) by combining information about social media use with personality profiles culled from online quizzes.18 And in 2017, an Australian newspaper exposed internal Facebook strategy documents detailing the company's alleged ability to detect when teenage users are feeling insecure. According to the report, "By monitoring posts, pictures, interactions and internet activity in real-time, Facebook can work out when young people feel 'stressed', 'defeated', 'overwhelmed', 'anxious', 'nervous', 'stupid', 'silly', 'useless', and a 'failure'" (Davidson, 2017). Though Facebook claims it never used that information to target advertisements at teenagers, it did not deny that it could. Extrapolating from this example it is easy to imagine others, such as banks targeting advertisements for highinterest loans at the financially desperate or pharmaceutical companies targeting advertisements for drugs at those suspected to be in health crisis.19 Second, digital platforms, such as websites and smartphone applications, are the ideal medium for leveraging these insights into our decision-making vulnerabilities. They are dynamic, interactive, intrusive, and adaptive choice architectures (Lanzing, 2018;Susser, 2019b;Yeung, 2017). Which is to say, the digital interfaces we interact with are configured in real time using the information about us described above, and they continue to learn about us as we interact with them. Unlike advertisements of old, they do not wait, passively, for viewers to drive past them on roads or browse over them in magazines; rather, they send text messages and push notifications, demanding our attention, and appear in our social media feeds at the precise moment they are most likely to tempt us. And because all of this is automated, digital platforms are able to adapt to each individual user, creating what Karen Yeung calls "highly personalised choice environment[s]"-decision-making contexts in which the vulnerabilities catalogued through pervasive digital surveillance are put to work in an effort to influence our choices (2017, p. 122).20 Third, if manipulation is hidden influence, then digital technologies are ideal vehicles for manipulation because they are already in a real sense hidden. We often think of technologies as objects we attend to and use with focus and attention. The language of technology design reflects this: we talk about "users" and "end users," "user interfaces," and "human-computer interaction". In fact, as philosophers (especially phenomenologists) and science and technology studies (STS) scholars have long shown, once we become habituated to a particular technology, the device or interface itself recedes from conscious attention, allowing us to focus on the tasks we are using it to accomplish.21 Think of a smartphone or computer: we pay little attention to the devices themselves, or even to the way familiar websites or app interfaces are arranged.
Instead, after becoming acclimated to them, we attend to the information, entertainment, or conveniences they offer (Rosenberger, 2009). Philosophers refer to this as "technological transparency"-the fact that we see, hear, or otherwise perceive through technologies-as though they were clear, transparent-onto the perceptual objects they convey to us (Ihde, 1990;Van Den Eede, 2011;Verbeek, 2005). Because this language of transparency can be confused with the concept of transparency familiar from technology policy discussions, we might more helpfully describe it as "invisibility" (Susser, 2019b). In addition to pervasive digital surveillance making our decision-making vulnerabilities easy to detect, and digital platforms making them easy to exploit, the ease with which our technologies become invisible to us-simply through frequent use and habituation-means the influences they facilitate are often hidden, and thus potentially manipulative.
Finally, although we focus primarily on the example of behavioural advertising to illustrate these dynamics, it is worth emphasising that advertisers are not the only ones engaging in manipulative practices. In the realm of user interface/experience (UI/UX) design, increasing attention is being paid to so-called "dark patterns"-design strategies that exploit users' decision-making vulnerabilities to nudge them into acting against their interests (or, at least, acting in the interests of the website or app), such as requiring automatically-renewing paid subscriptions that begin after an initial free trial period (Brignull, 2013;Gray, Kou, Battles, Hoggatt, & Toombs, 2018;Murgia, 2019;Singer, 2016). Though many of these strategies are as old as the internet and not all rise to the level of manipulation-sometimes overtly inconveniencing users, rather than hiding their intentions-their growing prevalence has led some to call for legislation banning them (Bartz, 2019).
Worries about online manipulation have also been raised in the context of gig economy services, such as Uber and Lyft (Veen, Goods, Josserand, & Kaine, 2017). While these platforms market themselves as freer, more flexible alternatives to traditional jobs, providing reliable and consistent service to customers requires maintaining some amount of control over workers.
However, without access to the traditional managerial controls of the office or factory floor, gig economy firms turn to "algorithmic management" strategies, such as notifications, customer satisfaction ratings, and other forms of soft control enabled through their apps (Rosenblat & Stark, 2016). Uber, for example, rather than requesting (or demanding) that workers put in longer hours, prompts drivers trying to exit the app with a reminder about their progress toward some earnings goal, exploiting the desire to continue making progress toward that goal; Lyft issues game-like "challenges" to drivers and stars and badges for accomplishing them (Mason, 2018;Scheiber, 2017).
In their current form, not all such practices necessarily manipulate-people are savvy, and many likely understand what they are facing. These examples are important, however, because they illustrate our present trajectory. Growing reliance on digital tools in all parts of our lives-tools that constantly record, aggregate, and analyse information about us-means we are revealing more and more about our individual and shared vulnerabilities. The digital platforms we interact with are increasingly capable of exploiting those insights to nudge and shape our choices, at home, in the workplace, and in the public sphere. And the more we become habituated to these systems, the less attention we pay to them.

THE HARM(S) OF ONLINE MANIPULATION
With this picture in hand, the question becomes: what exactly is the harm that results from influencing people in this way? Why should we be worried about technological mediation rendering us so susceptible to manipulative influence? In our view, there are several harms, but each flows from the same place-manipulation violates its target's autonomy.
The notion of autonomy points to an individual's capacity to make meaningfully independent decisions. As Joseph Raz puts it: "(t)he ruling idea behind the ideal of personal autonomy is that people should make their own lives" (Raz, 1986, p. 369). Making one's own life means freely facing both existential choices, like whom to spend one's life with or whether to have children, and pedestrian, everyday ones. And facing them freely means having the opportunity to think about and deliberate over one's options, considering them against the backdrop of one's beliefs, desires, and commitments, and ultimately deciding for reasons one recognises and endorses as one's own, absent unwelcome influence (J. P. Christman, 2009;Oshana, 2015;Veltman & Piper, 2014). Autonomy is in many ways the guiding normative principle of liberal democratic societies. It is because we think individuals can and should govern themselves that we value our capacity to collectively and democratically self-govern.
Philosophers sometimes operationalise the notion of autonomy by distinguishing between its competency and authenticity conditions (J. P. Christman, 2009, p. 155f). In the first place, being autonomous means having the cognitive, psychological, social, and emotional competencies to think through one's choices, form intentions about them, and act on the basis of those intentions. Second, it means that upon critical reflection one identifies with one's values, desires, and goals, and endorses them authentically as one's own. Of course, many have criticised such conceptions of autonomy as overly rationalistic and implausibly demanding, arguing that we rarely decide in this way. We are emotional actors and creatures of habit, they argue, socialised and enculturated into specific ways of choosing that we almost never reflect upon or endorse. But we understand autonomy broadly-our conception of deliberation includes not only beliefs and desires, but also emotions, convictions, and experiences, and critical reflection can be counterfactual (we must in principle be able to critically reflect on and endorse our motivations for acting, but we need not actually reflect on each and every move we make).
In addition to rejecting overly demanding and rationalistic conceptions of autonomy, we also reject overly atomistic ones. In our view, autonomous persons are socially, culturally, historically, and politically situated. Which is to say, we acknowledge the "intersubjective and social dimensions of selfhood and identity for individual autonomy and moral and political agency" (Mackenzie & Stoljar, 2000, p. 4). 22 Though social contexts can constrain our choices, by conditioning us to believe and behave in stereotypical ways (as, for example, in the case of gendered social expectations), it is also our social contexts that bestow value on autonomy, teaching us what it means to make independent decisions, and providing us with rich sets of options from which to choose. Moreover, it is crucial for present purposes that we emphasise our understanding of autonomy as more than an individual good-it is an essential social and political good too. Individuals express their autonomy across a variety of social contexts, from the home to the marketplace to the political sphere. Democratic institutions are meant to register and reflect the autonomous political decisions individuals make. Disrupting individual autonomy is thus more than an ethical concern; it has social and political import.
Against this picture of autonomy and its value, we can more carefully explain why online manipulation poses such a grave threat. To manipulate someone is, again, to covertly influence them, to intentionally alter their decision-making process without their conscious awareness.
Doing so undermines the target's autonomy in two ways: first, it can lead them to act toward ends they haven't chosen, and second, it can lead them to act for reasons not authentically their own.
To see the first problem, consider examples of targeted advertising in the commercial sphere.
Here, the aim of manipulators is fairly straightforward: they want people to buy things. Rather than simply put products on display, however, advertisers can construct decision-making environments-choice architectures-that subtly tempt or seduce shoppers to purchase their wares, and at the highest possible price (Calo, 2014). A variety of strategies might be deployed, from pointing out that one's friends have purchased the item to countdown clocks that pressure one to act before some offer expires, the goal being to hurry, evade, or undermine deliberation, and thus to encourage decisions that may or may not align with an individual's deeper, reflective, self-chosen ends and values.
Of course, these strategies are familiar from non-digital contexts; all commercial advertising (digital or otherwise) functions in part to induce consumers to buy things, and worries about manipulative ads emerged long before advertising moved online.23 Equally, not all advertising-perhaps not even all targeted advertising-involves manipulation. Purely informational ads displayed to audiences actively seeking out related products and services (e.g., online banner ads displaying a doctor's contact information shown to visitors to a health-related website) are unlikely to covertly influence their targets. Worries about manipulation arise in cases where advertisements are sneaky-which is to say, where their effects are achieved covertly. If, for example, the doctor was a psychiatrist, his advertisements were shown to people suspected of suffering from depression, and only at the specific times of day they were thought to be most afflicted, our account would offer grounds for condemning such tactics as manipulative.
It might also be the case that manipulation is not a binary phenomenon. We are the objects of countless influence campaigns and we understand some of them more than others; perhaps we ought to say that they are more or less manipulative in equal measure. On such a view, online targeted (or "behavioural") advertising could be understood as exacerbating manipulative dynamics common to other forms of advertising, by making the tweaks to individual choice architectures more subtle, and the seductions and temptations that result from them more difficult to resist (Yeung, 2017). Worse still, the fluidity and porousness of online environments makes it easy for marketers to conflate other distinct contexts with shopping, further blurring a person's reasoning about whether they truly want to make some purchase. For example, while chatting with friends over social media or searching for some place to eat, an ad may appear, thus requiring the target to juggle several tasks-in this case, communication and information retrieval-along with deliberation over whether or not to respond to the marketing ploy, thus diminishing the target's ability to sustain focus on any of the them. This problem is especially clearly illustrated by so-called "native advertising" (advertisements designed to look like usergenerated, non-commercial content). Such advertisements are a kind of Trojan horse, intentionally conflating commercial and non-commercial activities in an attempt to undermine our capacity for focused, careful deliberation.
In the philosophical language introduced above, these strategies challenge both autonomy's competency and authenticity conditions. By deliberately and covertly engineering our choice environments to steer our decision-making, online manipulation threatens our competency to deliberate about our options, form intentions about them, and act on the basis of those intentions. And since, as we've seen, manipulative practices often work by targeting and exploiting our decision-making vulnerabilities-concealing their effects, leaving us unaware of the influence on our decision-making process-they also challenge our capacity to reflect on and endorse our reasons for acting as authentically on our own. Online manipulation thus harms us both by inducing us to act toward ends not of our choosing and for reasons we haven't endorsed.
Importantly, undermining personal autonomy in the ways just described can lead to further harms. First, since autonomous individuals are wont to protect (or at least to try and protect) their own interests, we can reasonably expect that undermining people's autonomy will lead, in many cases, to a diminishment of those interests. Losing the ability to look out for ourselves is unlikely to leave us better off in the long run. This harm-e.g., being tricked into buying things we don't need or paying more for them than we otherwise would-is well described by those who have analysed the problem of online manipulation in the commercial sphere (Calo, 2014;Nadler & McGuigan, 2018;Zarsky, 2006;Zarsky, 2019). And it is a serious harm, which we would do well to take seriously, especially given the fact that law and policy around information and internet practices (at least in the US) assume that individuals are for the most part capable of safeguarding their interests (Solove, 2013). However, it is equally important to see that this harm to welfare is derivative of the deeper harm to autonomy. Attempting to "protect consumers" from threats to their economic or other interests, without addressing the more fundamental threat to their autonomy, is thus to treat the symptoms without addressing the cause.
To bring this into sharper relief, it is worth pointing out that even purely beneficent Having said that, we have not demonstrated that manipulation is necessarily wrong in every case-only that it always carries a harm. One can imagine cases where the harm to autonomy is outweighed by the benefit to welfare. (For example, a case where someone's life is in immediate danger, and the only way to save them is by manipulating them.) But such cases are likely few and far between. What is so worrying about online manipulation is precisely its banality-the fact that it threatens to become a regular part of the fabric of everyday experience. As Jeremy Waldron argues, if we allow that to happen, our lives will be drained of something deeply important: "What becomes of the self-respect we invest in our own willed actions, flawed and misguided though they often are, when so many of our choices are manipulated to promote what someone else sees (perhaps rightly) as our best interest?" (Waldron, 2014) That we also lack reason to believe online manipulators really do have our best interests at heart is only more reason to resist them.
Finally, beyond the harm to individuals, manipulation promises a collective harm. By threatening our autonomy it threatens democracy as well. For autonomy is writ small what democracy is writ large-the capacity to self-govern. It is only because we believe individuals can make meaningfully independent decisions that we value institutions designed to register and reflect them. As the Cambridge Analytica case-and the public outcry in response to it-demonstrates, online manipulation in the political sphere threatens to undermine these core collective values. The problem of online manipulation is, therefore, not simply an ethical problem; it is a social and political one too.

TECHNOLOGY AND AUTONOMY
If one accepts the arguments advanced thus far, an obvious response is that we need to devise law and policy capable of preventing and mitigating manipulative online practices. We agree that we do. But that response is not sufficient-the question for policymakers is not simply how to mitigate online manipulation, but how to strengthen autonomy in the digital age. In making this claim, we join our voices with a growing chorus of scholars and activists-like Frischmann, Selinger, and Zuboff-working to highlight the corrosive effects of digital technologies on autonomy. Meeting these challenges requires more than consumer protection-it requires creating the positive conditions necessary for supporting individual and collective selfdetermination.
We don't pretend to have a comprehensive solution to these deep and complex problems, but some suggestions follow from our brief discussion. It should be noted that these suggestions-like the discussion, above, that prompted them-are situated firmly in the terrain of contemporary liberal political discourse, and those convinced that online manipulation poses a significant threat (especially some European readers) may be struck by how moderate our responses are. While we are not opposed to more radical interventions, we formulate our analysis using the conceptual and normative frameworks familiar to existing policy discussions in hopes of having an impact on them.

CURTAIL DIGITAL SURVEILLANCE
Data, as Tal Zarsky writes, is the "fuel" powering online manipulation (2019, p. 186). Without the detailed profiles cataloguing our preferences, interests, habits, and so on, the ability of would-be manipulators to identify our weaknesses and vulnerabilities would be vastly

PROBLEMATISE PERSONALISATION
When asked to justify collecting so much data about us, data collectors routinely argue that the information is needed in order to personalise their services to the needs and interests of individual users. Mark Zuckerberg, for example, attempted recently to explain Facebook's business model in the pages of the Wall Street Journal: "People consistently tell us that if they're going to see ads, they want them to be relevant," he wrote. "That means we need to understand their interests" (2019).25 Personalisation seems, on the face of it, like an unalloyed good. Who wouldn't prefer a personalised experience to a generic one? Yet research into different forms of personalisation suggests that individualising-personalising-our experiences can carry with it significant risks.
These worries came to popular attention with Eli Pariser's book Filter Bubble (2011), which argued forcefully (though not without challenge) that the construction of increasingly singular, individualised experiences, means at the same time the loss of common, shared ones, and describes the detriments of that transformation to both individual and collective decisionmaking.26 In addition to personalised information environments-Pariser's focus-technological advances enable things like personalised pricing -sometimes called "dynamic pricing" or "price discrimination" (Calo, 2014) and personalised work scheduling -or "just-in-time" scheduling (De Stefano, 2015). For the reasons discussed above, many such strategies may well be manipulative. The targeting and exploiting of individual decision-making vulnerabilities enabled by digital technologies-the potential for online manipulation they create-gives us reason to question whether the benefits of personalisation really outweigh the costs. At the very least, we ought not to uncritically accept personalisation as a rationale for increased data collection, and we ought to approach with care (if not skepticism) the promise of an increasingly personalised digital environment.

PROMOTE AWARENESS AND UNDERSTANDING
If the central problem of online manipulation is its hiddenness, then any response must involve a drive toward increased awareness. The question is what form such awareness should take.
Yeung argues that the predominant vehicle for notifying individuals about information flows and data practices-the privacy notice, or what is often called "notice-and-consent"-is insufficient (2017). Indeed, merely notifying someone that they are the target of manipulation is not enough to neutralise its effects. Doing so would require understanding not only that one is the target of manipulation, but also who the manipulator is, what strategies they are deploying, and why. Given the well-known "transparency paradox", according to which we are bound to either deprive users of relevant information (in an attempt to be succinct) or overwhelm them with it (in an attempt to be thorough), there is little reason to believe standard forms of notice alone can equip users to face the challenges of online manipulation.27 Furthermore, the problem of online manipulation runs deeper than any particular manipulative practice. What worries many people is the fact that manipulative strategies, like targeted advertising, are becoming basic features of the digital world-so commonplace as to escape notice or mention.28 In the same way that machine learning and artificial intelligence tools have quickly and quietly been delegated vast decision-making authorities in a variety of contemporary contexts and institutions, and in response, scholars and activists have mounted calls to make their decision-making processes more explainable, transparent, and accountable, so too must we give people tools to understand and manage a digital environment designed to shape and influence them.29

ATTEND TO CONTEXT
Finally, it is important to recognise that moral intuitions about manipulation are indexed to social context. Which is to say, we are willing to tolerate different levels of outside influence on our decision-making in different decision-making spheres. As relatively lax commercial advertising regulations indicate, we are-at least in the US-willing to accept a fair amount of interference in the commercial sphere. By contrast, somewhat more stringent regulations around elections and campaign advertising suggest that we are less willing to accept such interference in the realm of politics.30 Responding to the threats of online manipulation therefore requires sensitivity to where-in which spheres of life-we encounter them.

CONCLUSION
The idea that technological advancements bring with them new arrangements of power is, of course, nothing new. That online manipulation threatens to subordinate the interests of individuals to those of data collectors and their clients is thus, in one respect, a familiar (if nonetheless troubling) problem. What we hope to have shown, however, is that the threat of online manipulation is deeper, more insidious, than that. Being steered or controlled, outside our conscious awareness, violates our autonomy, our capacity to understand and author our own lives. If the tools that facilitate such control are left unchecked, it will be to our individual and collective detriment. As we've seen, information technology is in many ways an ideal vehicle for these forms of control, but that does not mean that they are inevitable. Combating online manipulation requires both depriving it of personal data-the oxygen enabling it-and empowering its targets with awareness, understanding, and savvy about the forces attempting to influence them. June 2019 | Volume 8 | Issue 2 the notion of customer surveillance as a modern phenomenon. Although the internet's technological advances have taken advertising in new directions and the practice of 'datamining' to almost incomprehensible extremes, nearly all of what is transpiring reflects some of the basic methods developed by marketers beginning a hundred years ago" (Stole, 2014).
24. See https://eugdpr.org 25. Zuckerberg also cited needing user information for "security and operating our services". 26. Some empirical researchers have expressed skepticism about the alleged harms of filter bubbles, some even suggesting that they are beneficial (Dubois & Blank, 2018;Zuiderveen Borgesius et al., 2016). Their findings, however, are far from conclusive.
27. On the "transparency paradox," see Nissenbaum (2011). Though privacy notices are, in themselves, insufficient for shielding individuals from the effects of online manipulation, that does not mean that they are entirely without value. They might support individual autonomy, even if they can't guarantee it: see Susser (2019a).
28. For example, Marcella Vayena writes: "[N]ot just Cambridge Analytica, but most of the current online ecosystem, is an arm's race to the unconscious mind: notifications, microtargeted ads, autoplay plugins, are all strategies designed to induce addictive behavior, hence to manipulate" (Vayena, 2018).
29. For a helpful discussion about the calls for-and limits of-explainable artificial intelligence, see (Selbst & Barocas, 2018) 30. In a longer version of this paper, we also consider online manipulation in the context of the workplace. See Susser et al. (2018).