Algorithmic bias and the Value Sensitive Design approach

1. Introduction

When, in 2016, investigative journalists at ProPublica published a report indicating that a software system used in US courts was racially biased, a lively debate ensued. In essence, the journalists had found that COMPAS, a decision support tool used by judges and parole officers to assess a defendant's likelihood to re-offend, was systematically overestimating the recidivism risk of black defendants while underestimating that of white defendants (see Angwin et al., 2016). Northpointe, the company that developed COMPAS, disputed the allegations, arguing that its assessment tool was fair because it predicted recidivism with roughly the same accuracy regardless of defendants' ethnicity (see Dieterich et al., 2016). The ProPublica journalists, in turn, held that an algorithmic model cannot be fair if it produces serious errors, that is, false positives (i.e., false alarms) and false negatives (i.e., missed detections), more frequently for one ethnicity than for another, triggering a debate about the very idea of programming fairness into a computer algorithm (see, e.g., Wong, 2019). To date, over 1,000 academic papers have cited the ProPublica article, 1 and its findings have been discussed in popular news outlets around the globe.

But the ProPublica case was not a one-off. Rather, it marked the beginning of a series of reports and studies that found evidence for algorithmic bias in a wide range of application areas: from hiring systems (Dastin, 2018) to credit scoring (O'Neil, 2016) to facial recognition software (Buolamwini and Gebru, 2018). Cases such as these, which highlight the potential for automated discrimination based on characteristics such as age, gender, ethnicity, or socio-economic status, have reinvigorated old debates regarding the relationship between technology and society (see, e.g., Winner, 1980), questioning the neutrality of algorithms and inviting discussions about their power to structure and shape, rather than merely reflect, society. However, if technologies are not morally neutral and if the values and disvalues embedded have tangible consequences for both individuals and society at large, would this not imply that algorithms should be designed with care and that one should seek not only to detect and analyse problems, but to proactively engage with them through mindful design decisions? 2 Such questions, which are now being discussed within the computer science community, are not new, but have a long and often neglected history within computer science itself—e.g., through research in participatory design—but also in other fields and disciplines such as computer ethics, philosophy of technology, history of science, or science and technology studies (STS). The most principled attempt to design responsibly and sensitively to human values, however, is the Value Sensitive Design (VSD) approach, which emerged out of this intellectual landscape in the mid-1990s and has been expanded and refined ever since. More recently, and as result of increased awareness that "data is not a panacea" and that algorithmic techniques can "affect the fortunes of whole classes of people in consistently unfavorable ways" (Barocas and Selbst, 2016, p. 673), interest in the VSD methodology has been growing, begging the question: what insights can the approach offer to ongoing debates about bias and fairness in algorithmic decision-making and machine learning?

This article provides a brief overview of the key features of Value Sensitive Design (Section 2), examines its contributions to understanding and addressing issues around bias in computer systems (Section 3), outlines the current debates on algorithmic bias and fairness in machine learning (Section 4), and discusses how such debates could profit from VSD-derived insights and recommendations (Section 5). Relating these debates on values in design and algorithmic bias to research on cognitive biases, we conclude by stressing our collective duty to not only detect and counter biases in software systems, but to also address and remedy their societal origins (Section 6).

2. Value Sensitive Design: a brief overview

Value Sensitive Design as a theoretically grounded methodology emerged against the backdrop of the 1990s rapid computerisation and as a response to a perceived need for a design approach that would account for human values and social context throughout the design process (see Friedman and Hendry, 2019). Indeed, Friedman's (1997) seminal edited book Human Values and the Design of Computer Technology already provided an impressive demonstration on how to conceptualise and address issues around agency, privacy, and bias in computer systems, emphasising the need to "embrace value-sensitive design as part of the culture of computer science" (ibid.: p. 1). At its core, the VSD approach offers a concrete methodology for how to intentionally embed desired values into new technologies. It consists of three iterative phases, namely conceptual-philosophical, empirical, and technical investigations (see Friedman et al., 2006; Flanagan et al., 2008): 3

Conceptual-philosophical investigations encompass both the identification of relevant human values and the identification of relevant direct and indirect stakeholders. Regarding the former, careful working conceptualisations of specific values are meant to (a) clarify fundamental issues raised by the project at hand and (b) enable comparisons across VSD-based studies and research teams. While VSD defines human values relatively broadly as "what is important to people in their lives, with a focus on ethics and morality" (Friedman and Hendry, 2019, p. 24), Friedman et al. (2006, p. 364f) have provided a heuristic list of human values with ethical import 4 that are often implicated in system design. Regarding the latter, by not only considering direct but also indirect stakeholders, VSD aims to counter the frequent neglect of non-users in technology design, that is, of groups which may not use a technology themselves, but who are nonetheless affected by it (see Oudshoorn and Pinch, 2005; Wyatt, 2005). Given that values are often interrelated—consider, e.g., the ongoing debate about the relationship between privacy and security—and that what is important to one group of stakeholders may or may not be important to another group, conceptual investigations are also concerned with the relative importance of different values as well as potential trade-offs between conflicting values.

Empirical investigations make use of a wide range of quantitative and qualitative social science methods (e.g., surveys, interviews, observations, experiments) to provide a better understanding of how stakeholders actually conceive and prioritise values in specific socio-technical contexts. Cultural, historical, national, ethnic, and religious affiliations may play a role in this process and can determine how value conflicts are handled and resolved (see Flanagan et al., 2008, p. 328). Moreover, empirical investigations may reveal differences between espoused practice (what is said) and actual practice (what people do), enabling a more nuanced analysis of design decisions and their impact on usage, thereby complementing the conceptual investigations outlined above. Ultimately, it is through this empirical mode of inquiry that a more situated understanding of the socio-technical system can be derived, facilitating not only the observation of stakeholders' usage and appropriation patterns, but also whether the values envisioned in the design process are fulfilled, amended, or subverted.

Technical investigations are premised on the assumption that any given technological design provides "value suitabilities" (Friedman and Hendry, 2019, p. 34) in that it supports certain values and activities more readily than others. Following Friedman et al. (2008), investigations into these suitabilities can take one of two forms: in the first form, technical investigations focus on how existing technological properties can support or hinder specific human values. This approach bears similarities to the empirical mode, but instead of focusing on individuals, groups, or larger social systems, the emphasis is on the technology itself. In the second form, technological investigations involve the proactive design of systems to support and realise values identified in the conceptual investigation. If, for instance, privacy is a value that ought to be preserved, technical mechanisms must be implemented that further and promote privacy protections rather than diminish them. As specific designs will prioritise certain values over others, technical investigations can reveal both existing (first form) or prospective (second form) value hierarchies, thus adding another layer of insight to the analysis.

Through these three modes of investigation, VSD aims to contribute to the critical analysis of socio-technical systems and the values that have been—intentionally or unintentionally—embedded into them. Accordingly, VSD on the one hand serves as an analytical tool to open up valuation processes within technology design and development that are usually black-boxed or neglected. On the other hand, it provides a constructive tool that enables and supports the realisation of specific desired values in the design and development of new technologies. 5

3. Bias in computer systems

Long before the current debate about algorithmic bias and its consequences, Friedman and Nissenbaum (1996) had already pioneered an analysis of bias in computer systems, arguing that such systems are biased if they "systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others [by denying] an opportunity for a good or [assigning] an undesirable outcome to an individual or groups of individuals on grounds that are unreasonable or inappropriate" (ibid.: p. 332). For Friedman and Nissenbaum it was important to develop a better understanding of bias in computer systems, not least because they considered biased systems to be "instruments of injustice" and stressed that "freedom from bias should be counted among the select set of criteria according to which the quality for systems in use in society should be judged" (ibid.: p. 345f). A good understanding of biases would allow us to identify potential harms in a system and either avoid them in the process of design or correct them if the system is already in use. To this end, Friedman and Nissenbaum provided a taxonomy of biases that remains highly relevant and useful for today's debate on algorithmic bias and discrimination (see, e.g., Dobbe et al., 2019; Cramer et al., 2018). Based on the respective origin of bias, they specified three different types of biases, namely preexisting bias, technical bias, and emergent bias.

According to Friedman and Nissenbaum (1996), preexisting bias has its roots in social institutions, practices, and attitudes and usually exists prior to the creation of the system. It can either originate from individuals who have significant input into the design of the system (individual preexisting bias) or from prejudices that exist in society or culture at large (societal preexisting bias). Importantly, such biases mostly enter a system implicitly and unconsciously rather than through conscious effort.

Technical bias, in turn, arises from technical constraints or considerations. Sources of technical bias may include limitations of computer tools (e.g., in terms of hardware, software, or peripherals), the use of algorithms that have been developed for a different context, and the unwarranted formalisation of human constructs, that is, the attempt to quantify the qualitative and discretise the continuous.

Finally, emergent bias is bias that arises in a context of use, typically some time after a design is completed, as a result of (a) new societal knowledge or changing cultural values that are not or cannot be incorporated into the system design or (b) a mismatch between the users—their expertise and values—assumed in the system design and the actual population using the system.

In sum, Friedman and Nissenbaum's taxonomy of biases is meant to enable designers and researchers to identify and anticipate bias in computer systems by considering individual and societal worldviews, technological properties, and the contexts of use. Their analysis of biases foregrounds the value-laden nature of computer systems and stresses the possibility to mitigate or eliminate potential harms through proactively engaging with the design and development of the systems, which is one of the main objectives of the Value Sensitive Design approach. Consequently, their analysis reflects the double function of VSD as a tool for the analysis of (dis)values in existing technologies and for the construction of novel technologies that account for specific desired values. These two functions, the analytical and the constructive, are also central in recent research on bias and fairness in machine learning.

4. Algorithmic bias and fairness in machine learning

When mathematician Cathy O'Neil published her popular book Weapons of Math Destruction in 2016, the message was clear: Mathematical models, she wrote, can "encod[e] human prejudice, misunderstanding, and bias into the software systems that increasingly manag[e] our lives. [...] Their verdicts, even when wrong and harmful, [are] beyond dispute or appeal. And they ten[d] to punish the poor and the oppressed in our society, while making the rich richer" (O'Neil, 2016, p. 3). To support this claim, O'Neil works through a number of cases—from crime prediction software and personalised online advertising to college ranking systems and teacher evaluation tools to credit, insurance, and hiring algorithms—demonstrating the punitive power such systems can have on those who already suffer from social inequalities and emphasising the task to "explicitly embed better values into our algorithms, creating 'Big Data' models that follow our ethical lead" (ibid.: p. 204). O'Neil's book, along with a few other academic and non-academic texts, was at the forefront of a movement that sought to push back against the depiction of algorithms as fair and objective, showcasing their potential to "so[w] injustice, until we take steps to stop them" (ibid.: p. 203).

In the computer science community, where research on bias and discrimination in computational processes was conducted even prior to the current debate on the impacts of "Big Data" and artificial intelligence (see, e.g., Custers et al., 2013), attempts to detect and prevent such biases intensified. An example for this would be the organisation of the yearly FAT/ML6 annual meeting from 2014 onwards, which in light of a growing recognition that techniques such as machine learning raise "novel challenges for ensuring non-discrimination, due process, and understandability in decision-making," sought to "provid[e] researchers with a venue to explore how to characterize and address these issues with computationally rigorous methods" (FAT/ML, 2018). Other events such as the DAT (Data and Algorithmic Transparency) Workshop in 2016 or the FATREC Workshop on Responsible Recommendation in 2017 followed, and the FAT/ML meeting was eventually succeeded by the FAT* and later the ACM FAccT Conference, which seeks to bring together "researchers and practitioners interested in fairness, accountability, and transparency in socio-technical systems" (ACM FAccT Conference, 2021). As mentioned, this research and the VSD approach find common ground in their twofold objective to (a) identify bias and discrimination in algorithmic systems (analytical objective) and (b) to create and design fair algorithmic systems (constructive objective). 7

With respect to (a), researchers of algorithmic bias have proposed different frameworks for understanding and locating the sources of algorithmic biases, thereby delineating ways to mitigate or correct them (see, e.g., Baeza-Yates, 2018; Mehrabi et al., 2019; Olteanu et al., 2019). Barocas and Selbst (2016), for instance, provide a detailed description of the different ways that biases can be introduced into a machine learning system, including (i) through problem specification, where the definition of target variables rests on subjective choices that may systematically disadvantage certain populations over others; (ii) through the training data, where biased data sets can lead to discriminatory models and harmful results; (iii) through feature selection, where the reductive representation of real-world phenomena may result in inaccurate determinations and adverse effects; and (iv) through proxy variables, where specific data points are highly correlated with class membership, facilitating disparate treatment and potentially implicating less favorable outcomes for members of disadvantaged groups. In a similar vein, Danks and London (2017) identify different forms of algorithmic bias in autonomous systems, namely (i) training data bias, (ii) algorithmic focus bias, (iii) algorithmic processing bias, (iv) transfer context bias, and (v) interpretation bias. The parallels between such recent approaches and Friedman and Nissenbaum's earlier work become most apparent when considering their common goal to sound a "call for caution" (Barocas and Selbst, 2016, p. 732), provide "a taxonomy of different types and sources of algorithmic bias" (Danks and London, 2017, p. 4691), and offer a "framework for understanding and remedying it" (Friedman and Nissenbaum, 1996, p. 330). In either case, the designation and characterisation of different types of biases is thus seen as a key element of the common analytical objective to recognise and remedy such biases in existing algorithmic systems.

With respect to (b), and in addition to the analytical task of identifying and mitigating bias, there is also a more constructive aspiration in the machine learning community to design fair algorithms. Kearns and Roth, for instance, describe this aspiration as the "science of socially aware algorithm design" that looks at how algorithms can "incorporate – in a quantitative, measurable, verifiable manner – many of the ethical values we care about as individuals and as a society" (2019, p. 18). Alternatively, research on algorithmic fairness has been characterised as "translat[ing non-discrimination] regulations mathematically into non-discrimination constraints, and develop[ing] predictive modeling algorithms that would be able to take into account those constraints, and at the same time be as accurate as possible." (Žliobaitė, 2017, p. 1061) In other words, algorithmic fairness research does not only aim at identifying and mitigating bias, but more proactively at building the value of fairness into algorithmic systems. Such research generally proceeds from some predefined fairness metrics or fairness constraints, and then aims to develop algorithmic systems that are optimised according to the proposed metrics or satisfy the specified constraints. This process can either take place (i) in the pre-process stage, where input data are modified to ensure that the outcomes of algorithmic calculations when applied to new data will be fair, (ii) during the in-process stage, where algorithms are modified or replaced to generate fair(er) output, or (iii) in the post-process stage, where the output of any model is modified to be fairer. 8 Once again, there are obvious parallels between such computational approaches and VSD's goal of "influencing the design of technology early in and throughout the design process" (Friedman and Hendry, 2019, p. 4). In both cases, the adoption of a proactive orientation is indicative of a shared commitment to progress and improvement through ethical, value-based design. It is a constructive agenda that aims at contributing to responsible innovation rather than taking a purely analytical, after-the-fact approach. As Friedman and Hendry (2019, p. 2) put it: "While empirical study and critique of existing systems is essential, [VSD] is distinctive for its design stance – envisioning, designing, and implementing technology in moral and ethical ways that enhance our futures."

5. Discussion

Despite the conceptual similarities outlined above and the fact that the VSD literature is often cited by the FAT (Fairness, Accountability, and Transparency in socio-technical systems) community, the uptake and integration of some of VSD's core ideas in computer science remains inadequate in several important aspects.

First, concerns have been raised that the current literature on fairness in machine learning tends to focus too narrowly on how individual or societal biases can enter algorithmic systems—concentrating mostly on what Friedman and Nissenbaum refer to as "preexisting bias"—while ignoring other sources of bias such as technical bias or emergent bias. In response to this, Dobbe and co-authors (2019) have stressed the need for a broader view on algorithmic bias that takes into account all the categories of Friedman and Nissenbaum's (1996) taxonomy and considers "risks beyond those pre-existing in the data" (Dobbe et al., 2019, p. 2). Thus, in order to better fulfill the analytical objective of identifying and mitigating bias in algorithmic systems, it is important that the academic machine learning community does not resort to VSD in an eclectic, piecemeal manner, but rather draws on the full breadth of the proposed frameworks.

Second, it is important to remember that concepts such as fairness are by no means self-explanatory or clear-cut. Verma and Rubin (2018), for instance, point out that more than twenty different notions of fairness have been proposed in AI-related research in the last few years, a lack of agreement that calls into question the very idea of operationalising fairness when seeking to design fair algorithms. Although the idea of fairness and the related concept of 'equality of opportunity' have been extensively discussed in philosophical research (see, e.g., Ryan, 2006; Hooker, 2014; Arneson, 2018), Binns (2018) has argued that most fairness measures in machine learning research tend to be undertheorized from a philosophical perspective, resulting in approaches that focus "on a narrow, static set of prescribed protected classes [...] devoid of context" (ibid.: p. 9). Last but not least, Corbett-Davies and Goel (2018) have highlighted the divergence between formalised notions of fairness and people's common understanding of fairness in everyday decision contexts. What follows from these objections is that attempts to formalise and operationalise fairness in specific ways can be contested on numerous grounds.

Unfortunately, this contestability is often disregarded or downplayed in the presentation of technical solutions, 9 even though recent years have shown a trend toward more interdisciplinary approaches that are conscious of the need to broaden the analytical scope. Proper utilisation of VSD could support such efforts as the method not only requires diligent investigations of the values at stake (see, in particular, the philosophical and technical investigations in the VSD method), but also calls for the involvement of interdisciplinary research teams that include, for example, philosophers, social scientists, or legal scholars. Of course, such interdisciplinary approaches can be challenging and resource intensive, but ethical design ultimately demands more than mechanical, recipe-based treatments of FAT requirements (see Keyes et al., 2019). Striving for truly value-sensitive designs implies being sensitive to the manifold meanings of values in different societal and cultural contexts and requires recognising, relating, and applying different disciplinary competences.

Finally, and on a related note, there is not only a need to expand the breadth of disciplinary perspectives, but also to widen the scope of the object of investigation itself. Simply put, instead of focusing more narrowly on fairness, accountability, and transparency in machine learning, research on algorithmic bias should also account for (a) the broader socio-technical system in which technologies are situated and (b) the different logics and orders that these algorithmic technologies produce and engender. Regarding the former, Gangadharan and Niklas (2019) have warned that the techno-centric focus on embedding fairness in algorithms, which is based on the idea that technical tweaks will suffice to prevent or avoid discriminatory outcomes, runs the danger of ignoring the wider social, political, and economic conditions in which unfairness and inequality arise. Regarding the latter, Hoffmann (2019, p. 910) reminds us that work on algorithmic bias does not only demand sustained attention to system failures but also to "the kinds of worlds being built – both explicitly and implicitly – by and through design, development, and implementation of data-intensive, algorithmically-mediated systems". What would thus be needed is greater attention to the "broader institutional, contextual, and social orders instantiated by algorithmically mediated systems and their logics of reduction and optimization" (ibid.). The FAT community has already made strides in this direction, with the ACM FAT* Conference 2020 explicitly seeking "to sustain and further improve the high quality of computer science research in this domain, while simultaneously extending the focus to law and social sciences and humanities research" (ACM FAcct Conference, 2020). Nevertheless, we believe that a more comprehensive uptake of VSD, which has been conceptualised as an interdisciplinary approach from the very start, could support this process.

6. Concluding remarks

This paper has offered a concise review of the methodology of Value Sensitive Design and the taxonomy of biases proposed by Friedman and Nissenbaum (1996). It has shown that both VSD and the taxonomy of biases remain highly relevant for current research on bias and fairness in socio-technical systems. Despite its usefulness, however, VSD is often taken up only partially and crucial insights—e.g., regarding the conceptual underpinnings of values, the need to consider both users and non-users of a technology, 10 or the importance of interdisciplinarity—are lost. Consequently, it would be advisable to intensify efforts to revitalise and deepen the uptake of Value Sensitive Design in Fairness, Accountability, and Transparency (FAT) and related research. Fortunately, there is indeed a trend to expand the debates and move the discussion beyond the technical domain.

Clearly, the review of VSD and research on algorithmic bias in this paper does not fully capture the evolving debate. Moreover, it is important to note that research on biases goes well beyond the purview of VSD and computer science. Indeed, psychology and the cognitive sciences have long studied cognitive biases (Gigerenzer et al., 2012; Kahneman, 2011) and implicit biases (Holroyd et al., 2017). 11 While Friedman and Nissenbaum's notion of preexisting bias has, to some extent, accounted for implicit biases, the relationship between human cognitive biases and bias in computer systems requires further analyses. Especially in the context of automated decision-making (ADM), where human decisions are complemented—or even replaced—by machine decisions, human cognitive biases can have interesting ramifications for the design and use of ADM systems.

Firstly, cognitive biases can be causally related to biased automated decision-making. Cognitive limitations and biases may for instance contribute to the formation of societal stereotypes, prejudices and unwarranted preferences, or poor decision-making practices (e.g., through the defective interpretation of probabilities), which are fed into ADM systems through training data, thereby hiding while at the same time reproducing and reinforcing such biases in seemingly neutral machines.

Secondly, and conversely, ADM systems can also reduce and/or eliminate cognitive biases by accounting for and possibly correcting flaws in human reasoning (see, e.g., Savulescu and Maslen, 2015; Sunstein, 2018). In this respect, if designers and researchers of ADM systems can a) identify the sources of cognitive biases and b) counter them through specific methodological choices in designing and implementing the system, such systems can be conceived as tools to both disclose cognitive biases in human decision-making and to reduce or even prevent their negative impacts through sophisticated human-machine interaction in decision-making.

Finally, unwarranted delegation of human decision-making to machines can be a cognitive bias in itself, known as automation bias (Mosier et al., 1996) or automation complacency (Parasuraman and Manzei, 2010). Automation bias is characterised by the human tendency to over-trust and over-rely on allegedly neutral machines in that they follow wrong (or questionable) 'decisions' from the machines without seeking further corroborative or contradictory information, or even discount information from other existing sources (Skitka et al., 1999). Relatedly, automation complacency describes human operators' belief in the system's reliability, thereby causing them to pay insufficient attention to monitoring the process and to verifying the outputs of the system. Thus, recognising the dangers of automation bias and automation complacency—i.e., of overreliance on automated decision-making—brings us right back to Friedman and Nissenbaum's early warnings regarding biases in seemingly accurate, neutral, and objective computer systems, and their timely request to actively expose and counter them for better design and informed public discourse on the merits and limitations of such software tools. However, improving our tools will only bring us so far—accounting for values and countering bias also requires us to acknowledge and remedy existing inequalities and injustices in our societies and to concede that not all decision-making processes should be conducted by algorithms.