Algorithmic bias and the Value Sensitive Design approach

Recently, amid growing awareness that computer algorithms are not neutral tools but can cause harm by reproducing and amplifying bias, attempts to detect and prevent such biases have intensified. An approach that has received considerable attention in this regard is the Value Sensitive Design (VSD) methodology, which aims to contribute to both the critical analysis of (dis)values in existing technologies and the construction of novel technologies that account for specific desired values. This article provides a brief overview of the key features of the Value Sensitive Design approach, examines its contributions to understanding and addressing issues around bias in computer systems, outlines the current debates on algorithmic bias and fairness in machine learning, and discusses how such debates could profit from VSD-derived insights and recommendations. Relating these debates on values in design and algorithmic bias to research on cognitive biases, we conclude by stressing our collective duty to not only detect and counter biases in software systems, but to also address and remedy their societal origins. Issue 4 This article belongs to Concepts of the digital society, a special section of Internet Policy Review guest-edited by Christian Katzenbach and Thomas Christian Bächle.


Introduction
When, in 2016, investigative journalists at ProPublica published a report indicating that a software system used in US courts was racially biased, a lively debate ensued. In essence, the journalists had found that COMPAS, a decision support tool used by judges and parole officers to assess a defendant's likelihood to re-offend, was systematically overestimating the recidivism risk of black defendants while underestimating that of white defendants (see Angwin et al., 2016). Northpointe, the company that developed COMPAS, disputed the allegations, arguing that its assessment tool was fair because it predicted recidivism with roughly the same accuracy regardless of defendants' ethnicity (see Dieterich et al., 2016). The ProPublica journalists, in turn, held that an algorithmic model cannot be fair if it produces serious errors, that is, false positives (i.e., false alarms) and false negatives (i.e., missed detections), more frequently for one ethnicity than for another, triggering a debate about the very idea of programming fairness into a computer algorithm (see, e.g., Wong, 2019). To date, over 1,000 academic papers have cited the ProPublica article, 1 and its findings have been discussed in popular news outlets around the globe.
But the ProPublica case was not a one-off. Rather, it marked the beginning of a series of reports and studies that found evidence for algorithmic bias in a wide range of application areas: from hiring systems (Dastin, 2018) to credit scoring (O'Neil, 2016) to facial recognition software (Buolamwini and Gebru, 2018). Cases such as these, which highlight the potential for automated discrimination based on characteristics such as age, gender, ethnicity, or socio-economic status, have reinvigorated old debates regarding the relationship between technology and society (see, e.g., Winner, 1980), questioning the neutrality of algorithms and inviting discussions about their power to structure and shape, rather than merely reflect, society.
However, if technologies are not morally neutral and if the values and disvalues embedded have tangible consequences for both individuals and society at large, would this not imply that algorithms should be designed with care and that one should seek not only to detect and analyse problems, but to proactively engage with them through mindful design decisions? 2 Such questions, which are now be-ing discussed within the computer science community, are not new, but have a long and often neglected history within computer science itself-e.g., through research in participatory design-but also in other fields and disciplines such as computer ethics, philosophy of technology, history of science, or science and technology studies (STS). The most principled attempt to design responsibly and sensitively to human values, however, is the Value Sensitive Design (VSD) approach, which emerged out of this intellectual landscape in the mid-1990s and has been expanded and refined ever since. More recently, and as result of increased awareness that "data is not a panacea" and that algorithmic techniques can "affect the fortunes of whole classes of people in consistently unfavorable ways" (Barocas and Selbst, 2016, p. 673), interest in the VSD methodology has been growing, begging the question: what insights can the approach offer to ongoing debates about bias and fairness in algorithmic decision-making and machine learning?
This article provides a brief overview of the key features of Value Sensitive Design (Section 2), examines its contributions to understanding and addressing issues around bias in computer systems (Section 3), outlines the current debates on algorithmic bias and fairness in machine learning (Section 4), and discusses how such debates could profit from VSD-derived insights and recommendations (Section 5).
Relating these debates on values in design and algorithmic bias to research on cognitive biases, we conclude by stressing our collective duty to not only detect and counter biases in software systems, but to also address and remedy their societal origins (Section 6).

Value Sensitive Design: a brief overview
Value Sensitive Design as a theoretically grounded methodology emerged against the backdrop of the 1990s rapid computerisation and as a response to a perceived need for a design approach that would account for human values and social context throughout the design process (see Friedman and Hendry, 2019). Indeed, Friedman's (1997)

seminal edited book Human Values and the Design of Computer
Technology already provided an impressive demonstration on how to conceptualise and address issues around agency, privacy, and bias in computer systems, emphasising the need to "embrace value-sensitive design as part of the culture of computer science" (ibid.: p. 1). At its core, the VSD approach offers a concrete methodology for how to intentionally embed desired values into new technologies. It con-2. In this paper, we use the term value to refer to "[t]hose things that people find valuable that are both ideal and general" and the term disvalue to refer to "those general qualities that are considered to be bad or evil" (Brey, 2010, p. 46).
sists of three iterative phases, namely conceptual-philosophical, empirical, and technical investigations (see Friedman et al., 2006;Flanagan et al., 2008): 3 that are often implicated in system design. Regarding the latter, by not only considering direct but also indirect stakeholders, VSD aims to counter the frequent neglect of non-users in technology design, that is, of groups which may not use a technology themselves, but who are nonetheless affected by it (see Oudshoorn and Pinch, 2005;Wyatt, 2005). Given that values are often interrelated-consider, e.g., the ongoing debate about the relationship between privacy and security-and that what is important to one group of stakeholders may or may not be important to another group, conceptual investigations are also concerned with the relative importance of different values as well as potential trade-offs between conflicting values.

Conceptual-philosophical investigations
Empirical investigations make use of a wide range of quantitative and qualitative social science methods (e.g., surveys, interviews, observations, experiments) to provide a better understanding of how stakeholders actually conceive and prioritise values in specific socio-technical contexts. Cultural, historical, national, ethnic, and religious affiliations may play a role in this process and can determine how value conflicts are handled and resolved (see Flanagan et al., 2008, p. 328). Moreover, empirical investigations may reveal differences between espoused practice (what is said) and actual practice (what people do), enabling a more nuanced analysis of design decisions and their impact on usage, thereby complementing the conceptual investigations outlined above. Ultimately, it is through this empirical mode of inquiry that a more situated understanding of the socio-technical sys-3. The following paragraphs are a reworked and expanded version of section 1 in "Value-Sensitive Design as a Methodology" (Simon, 2017).
4. Examples of such "values with ethical import" include privacy, meaning "the right of an individual to determine what information about himself or herself can be communicated to others"; autonomy, meaning "people's ability to decide, plan, and act in ways that they believe will help them to achieve their goals"; or informed consent, which refers to "garnering people's agreement, encompassing criteria of disclosure and comprehension (for 'informed') and voluntariness, competence, and agreement (for 'consent')" (Friedman et al., 2006, p. 364).
tem can be derived, facilitating not only the observation of stakeholders' usage and appropriation patterns, but also whether the values envisioned in the design process are fulfilled, amended, or subverted.
Technical investigations are premised on the assumption that any given technological design provides "value suitabilities" (Friedman and Hendry, 2019, p. 34) in that it supports certain values and activities more readily than others. Following Friedman et al. (2008), investigations into these suitabilities can take one of two forms: in the first form, technical investigations focus on how existing technological properties can support or hinder specific human values. This approach bears similarities to the empirical mode, but instead of focusing on individuals, groups, or larger social systems, the emphasis is on the technology itself. In the second form, technological investigations involve the proactive design of systems to support and realise values identified in the conceptual investigation. If, for instance, privacy is a value that ought to be preserved, technical mechanisms must be implemented that further and promote privacy protections rather than diminish them.
As specific designs will prioritise certain values over others, technical investigations can reveal both existing (first form) or prospective (second form) value hierarchies, thus adding another layer of insight to the analysis.
Through these three modes of investigation, VSD aims to contribute to the critical analysis of socio-technical systems and the values that have been-intentionally or unintentionally-embedded into them. Accordingly, VSD on the one hand serves as an analytical tool to open up valuation processes within technology design and development that are usually black-boxed or neglected. On the other hand, it provides a constructive tool that enables and supports the realisation of specific desired values in the design and development of new technologies. 5

Bias in computer systems
Long before the current debate about algorithmic bias and its consequences, Friedman and Nissenbaum (1996)  develop a better understanding of bias in computer systems, not least because they considered biased systems to be "instruments of injustice" and stressed that "freedom from bias should be counted among the select set of criteria according to which the quality for systems in use in society should be judged" (ibid.: p. 345f). A good understanding of biases would allow us to identify potential harms in a system and either avoid them in the process of design or correct them if the system is already in use. To this end, Friedman and Nissenbaum provided a taxonomy of biases that remains highly relevant and useful for today's debate on algorithmic bias and discrimination (see, e.g., Dobbe et al., 2019;Cramer et al., 2018). Based on the respective origin of bias, they specified three different types of biases, namely preexisting bias, technical bias, and emergent bias.
According to Friedman and Nissenbaum (1996), preexisting bias has its roots in social institutions, practices, and attitudes and usually exists prior to the creation of the system. It can either originate from individuals who have significant input into the design of the system (individual preexisting bias) or from prejudices that exist in society or culture at large (societal preexisting bias). Importantly, such biases mostly enter a system implicitly and unconsciously rather than through conscious effort.
Technical bias, in turn, arises from technical constraints or considerations. Sources of technical bias may include limitations of computer tools (e.g., in terms of hardware, software, or peripherals), the use of algorithms that have been developed for a different context, and the unwarranted formalisation of human constructs, that is, the attempt to quantify the qualitative and discretise the continuous.
Finally, emergent bias is bias that arises in a context of use, typically some time after a design is completed, as a result of (a) new societal knowledge or changing cultural values that are not or cannot be incorporated into the system design or (b) a mismatch between the users-their expertise and values-assumed in the system design and the actual population using the system.  In the computer science community, where research on bias and discrimination in computational processes was conducted even prior to the current debate on the impacts of "Big Data" and artificial intelligence (see, e.g., Custers et al., 2013), attempts to detect and prevent such biases intensified. An example for this would be the organisation of the yearly FAT/ML 6 annual meeting from 2014 onwards, which in light of a growing recognition that techniques such as machine learning raise "novel challenges for ensuring non-discrimination, due process, and understandability in decision-making, " sought to "provid[e] researchers with a venue to explore how to characterize and address these issues with computationally rigorous  (Danks and London, 2017, p. 4691), and offer a "framework for understanding and remedying it" (Friedman and Nissenbaum, 1996, p. 330). In either case, the designation and characterisation of different types of biases is thus seen as a key element of the common analytical objective to recognise and remedy such biases in existing algorithmic systems.

Algorithmic bias and fairness in machine learning
With respect to (b), and in addition to the analytical task of identifying and mitigating bias, there is also a more constructive aspiration in the machine learning community to design fair algorithms. Kearns and Roth, for instance, describe this aspiration as the "science of socially aware algorithm design" that looks at how algorithms can "incorporate -in a quantitative, measurable, verifiable mannermany of the ethical values we care about as individuals and as a society" (2019, p. 7. From an VSD perspective, the development of a "fair" algorithmic system would entail the embedding of specific values such as fairness, accountability, or transparency into the system. 18). Alternatively, research on algorithmic fairness has been characterised as "translat[ing non-discrimination] regulations mathematically into non-discrimination constraints, and develop [ing] predictive modeling algorithms that would be able to take into account those constraints, and at the same time be as accurate as possible. " (Žliobaitė, 2017, p. 1061) In other words, algorithmic fairness research does not only aim at identifying and mitigating bias, but more proactively at building the value of fairness into algorithmic systems. Such research generally proceeds from some predefined fairness metrics or fairness constraints, and then aims to develop algorithmic systems that are optimised according to the proposed metrics or satisfy the specified constraints. This process can either take place (i) in the pre-process stage, where input data are modified to ensure that the outcomes of algorithmic calculations when applied to new data will be fair, (ii) during the inprocess stage, where algorithms are modified or replaced to generate fair(er) output, or (iii) in the post-process stage, where the output of any model is modified to be fairer. 8 Once again, there are obvious parallels between such computational approaches and VSD's goal of "influencing the design of technology early in and throughout the design process" (Friedman and Hendry, 2019, p. 4). In both cases, the adoption of a proactive orientation is indicative of a shared commitment to progress and improvement through ethical, value-based design. It is a constructive agenda that aims at contributing to responsible innovation rather than taking a purely analytical, after-the-fact approach. As Friedman and Hendry (2019, p. 2) put it: "While empirical study and critique of existing systems is essential, [VSD] is distinctive for its design stance -envisioning, designing, and implementing technology in moral and ethical ways that enhance our futures. "

Discussion
Despite the conceptual similarities outlined above and the fact that the VSD literature is often cited by the FAT (Fairness, Accountability, and Transparency in sociotechnical systems) community, the uptake and integration of some of VSD's core ideas in computer science remains inadequate in several important aspects. stressed the need for a broader view on algorithmic bias that takes into account all the categories of Friedman and Nissenbaum's (1996) taxonomy and considers "risks beyond those pre-existing in the data" (Dobbe et al., 2019, p. 2). Thus, in order to better fulfill the analytical objective of identifying and mitigating bias in algorithmic systems, it is important that the academic machine learning community does not resort to VSD in an eclectic, piecemeal manner, but rather draws on the full breadth of the proposed frameworks.
Second, it is important to remember that concepts such as fairness are by no means self-explanatory or clear-cut. Verma and Rubin (2018), for instance, point out that more than twenty different notions of fairness have been proposed in AIrelated research in the last few years, a lack of agreement that calls into question the very idea of operationalising fairness when seeking to design fair algorithms.
Although the idea of fairness and the related concept of 'equality of opportunity' have been extensively discussed in philosophical research (see, e.g., Ryan, 2006;Hooker, 2014;Arneson, 2018), Binns (2018) has argued that most fairness measures in machine learning research tend to be undertheorized from a philosophical perspective, resulting in approaches that focus "on a narrow, static set of prescribed protected classes [...] devoid of context" (ibid.: p. 9). Last but not least, Corbett-Davies and Goel (2018) have highlighted the divergence between formalised notions of fairness and people's common understanding of fairness in everyday decision contexts. What follows from these objections is that attempts to formalise and operationalise fairness in specific ways can be contested on numerous grounds.
Unfortunately, this contestability is often disregarded or downplayed in the presentation of technical solutions, 9 even though recent years have shown a trend toward more interdisciplinary approaches that are conscious of the need to broaden the analytical scope. Proper utilisation of VSD could support such efforts as the method not only requires diligent investigations of the values at stake (see, in particular, the philosophical and technical investigations in the VSD method), but also calls for the involvement of interdisciplinary research teams that include, for example, philosophers, social scientists, or legal scholars. Of course, such interdisciplinary approaches can be challenging and resource intensive, but ethical design ultimately demands more than mechanical, recipe-based treatments of FAT requirements (see Keyes et al., 2019). Striving for truly value-sensitive designs implies being sensitive to the manifold meanings of values in different societal and 9. For a detailed discussion of the concept of contestability and the importance of contestable design, see Kluttz et al., 2020. cultural contexts and requires recognising, relating, and applying different disciplinary competences.
Finally, and on a related note, there is not only a need to expand the breadth of disciplinary perspectives, but also to widen the scope of the object of investigation itself. Simply put, instead of focusing more narrowly on fairness, accountability, and transparency in machine learning, research on algorithmic bias should also account for (a) the broader socio-technical system in which technologies are situated and (b) the different logics and orders that these algorithmic technologies produce and engender. Regarding the former, Gangadharan and Niklas (2019) have warned that the techno-centric focus on embedding fairness in algorithms, which is based on the idea that technical tweaks will suffice to prevent or avoid discriminatory outcomes, runs the danger of ignoring the wider social, political, and economic conditions in which unfairness and inequality arise. Regarding the latter, Hoffmann (2019, p. 910) reminds us that work on algorithmic bias does not only demand sustained attention to system failures but also to "the kinds of worlds being builtboth explicitly and implicitly -by and through design, development, and implementation of data-intensive, algorithmically-mediated systems". What would thus be needed is greater attention to the "broader institutional, contextual, and social orders instantiated by algorithmically mediated systems and their logics of reduction and optimization" (ibid.). The FAT community has already made strides in this direction, with the ACM FAT* Conference 2020 explicitly seeking "to sustain and further improve the high quality of computer science research in this domain, while simultaneously extending the focus to law and social sciences and humanities research" (ACM FAcct Conference, 2020). Nevertheless, we believe that a more comprehensive uptake of VSD, which has been conceptualised as an interdisciplinary approach from the very start, could support this process.

Concluding remarks
This paper has offered a concise review of the methodology of Value Sensitive Design and the taxonomy of biases proposed by Friedman and Nissenbaum (1996). It has shown that both VSD and the taxonomy of biases remain highly relevant for current research on bias and fairness in socio-technical systems. Despite its usefulness, however, VSD is often taken up only partially and crucial insights-e.g., regarding the conceptual underpinnings of values, the need to consider both users and non-users of a technology, 10 or the importance of interdisciplinarity-are lost.
10. For a more detailed discussion on the need to also take non-users into account, see Wong (2019) and Wong and Simon (2020).
Consequently, it would be advisable to intensify efforts to revitalise and deepen the uptake of Value Sensitive Design in Fairness, Accountability, and Transparency (FAT) and related research. Fortunately, there is indeed a trend to expand the debates and move the discussion beyond the technical domain.
Clearly, the review of VSD and research on algorithmic bias in this paper does not fully capture the evolving debate. Moreover, it is important to note that research on biases goes well beyond the purview of VSD and computer science. Indeed, psychology and the cognitive sciences have long studied cognitive biases (Gigerenzer et al., 2012;Kahneman, 2011) and implicit biases (Holroyd et al., 2017). 11 While Friedman and Nissenbaum's notion of preexisting bias has, to some extent, accounted for implicit biases, the relationship between human cognitive biases and bias in computer systems requires further analyses. Especially in the context of au- Firstly, cognitive biases can be causally related to biased automated decision-making. Cognitive limitations and biases may for instance contribute to the formation of societal stereotypes, prejudices and unwarranted preferences, or poor decisionmaking practices (e.g., through the defective interpretation of probabilities), which are fed into ADM systems through training data, thereby hiding while at the same time reproducing and reinforcing such biases in seemingly neutral machines.
Secondly, and conversely, ADM systems can also reduce and/or eliminate cognitive biases by accounting for and possibly correcting flaws in human reasoning (see, e.g., Savulescu and Maslen, 2015;Sunstein, 2018). In this respect, if designers and researchers of ADM systems can a) identify the sources of cognitive biases and b) counter them through specific methodological choices in designing and implementing the system, such systems can be conceived as tools to both disclose cognitive biases in human decision-making and to reduce or even prevent their negative impacts through sophisticated human-machine interaction in decision-making.
Finally, unwarranted delegation of human decision-making to machines can be a cognitive bias in itself, known as automation bias (Mosier et al., 1996) or automation complacency (Parasuraman and Manzei, 2010). Automation bias is characterised by the human tendency to over-trust and over-rely on allegedly neutral 11. It should be noted that cognitive bias and implicit bias do not necessarily have the negative moral connotation as in the case of bias in VSD. machines in that they follow wrong (or questionable) 'decisions' from the machines without seeking further corroborative or contradictory information, or even discount information from other existing sources (Skitka et al., 1999). Relatedly, automation complacency describes human operators' belief in the system's reliability, thereby causing them to pay insufficient attention to monitoring the process and to verifying the outputs of the system. Thus, recognising the dangers of automation bias and automation complacency-i.e., of overreliance on automated decision-making-brings us right back to Friedman and Nissenbaum's early warnings regarding biases in seemingly accurate, neutral, and objective computer systems, and their timely request to actively expose and counter them for better design and informed public discourse on the merits and limitations of such software tools.
However, improving our tools will only bring us so far-accounting for values and countering bias also requires us to acknowledge and remedy existing inequalities and injustices in our societies and to concede that not all decision-making processes should be conducted by algorithms.