Google Scholar – Platforming the scholarly economy

Jake Goldenfein, Melbourne Law School, The University of Melbourne, Melbourne, Australia, jakeg@unimelb.edu.au
Daniel Griffin, UC Berkeley School of Information, University of California, Berkeley, Berkeley, United States

PUBLISHED ON: 29 Sep 2022 DOI: 10.14763/2022.3.1671

Abstract

Google Scholar has become an important player in the scholarly economy. Whereas typical academic publishers sell bibliometrics, analytics and ranking products, Alphabet, through Google Scholar, provides “free” tools for academic search and scholarly evaluation that have made it central to academic practice. Leveraging political imperatives for open access publishing, Google Scholar has managed to intermediate data flows between researchers, research managers and repositories, and built its system of citation counting into a unit of value that coordinates the scholarly economy. At the same time, Google Scholar’s user-friendly but opaque tools undermine certain academic norms, especially around academic autonomy and the academy’s capacity to understand how it evaluates itself.
Citation & publishing information
Received: October 26, 2021 Reviewed: January 6, 2022 Published: September 29, 2022
Licence: Creative Commons Attribution 3.0 Germany
Funding: Both Jake Goldenfein and Daniel Griffin were supported by the US National Science Foundation under U.S. NSF INSPIRE SES1537324 for part of the work on this project.
Competing interests: The author has declared that no competing interests exist that have influenced the text.
Keywords: Digital platforms, Autonomy, Google, Bibliometrics, Platform economy
Citation: Goldenfein, J. & Griffin, D. (2022). Google Scholar – Platforming the scholarly economy. Internet Policy Review, 11(3). https://doi.org/10.14763/2022.3.1671

Introduction

Bibliometrics have been a major part of academic publishing since the mid-20th century when Eugene Garfield developed citation indexing as a way to automate the organisation of scientific information (Garfield, 1955). Citation indexing paved the way for evaluative bibliometrics, whose centrality has long been controversial in the academic world (see e.g. European Commission, 2019; Nygaard & Bellanova, 2017; Crane & Glozer, 2022). Despite long-held concerns, research evaluators and managers, such as funders and university hiring committees, continue to use metrics to quickly approximate research and researcher quality, and the commercial academic publishing industry continues to develop evaluative metrics and analytics as next generation products. Rather than add to the vast literature critiquing academic metrics and the metrification of academic work, this article interrogates the consequences of academic metrics that are provided by platform businesses like Alphabet through services like Google Scholar.

Google Scholar is not simply a search interface. It has leveraged its system of citation counting into a tool that coordinates the scholarly economy. The service’s metrics drive the distribution, consumption and evaluation of both research and researchers. Like other digital platforms that use scoring to rank content and actors across markets, Google Scholar uses citation counting to control the visibility of research and researchers, and intermediate data flows between researchers, research managers and repositories. As its gatekeeping role (see e.g. Lynskey, 2017) in the academic domain expands, it infuses the scholarly context with platform dynamics in ways that trouble a set of contextual norms around academic autonomy.

In this article we describe how Google Scholar participates in refashioning the academic world in the image of the platform. Much has been said about the ways academic publishing replicates the logics of platform capitalism (see e.g. Mirowski, 2018), which includes commercial academic publishers’ turn to analytics as next generation products. However, perhaps because its relation to valorisation and profit is non-obvious, less attention has been paid to the specific dynamics that drive Google Scholar. These include a “free” service that enrolls scholars in systems of surveillance, multi-sided markets and behavioural advertising; Google Scholar’s user experience design that frustrates researchers’ ability to self-define search ‘relevance’; and the specific ways that Google Scholar exploits the academic open access movement as a vehicle for value extraction (vis-à-vis how commercial publishers have taken advantage of open access (see e.g. Mirowski, 2018; Meagher, 2021). We argue that these dynamics generate a particularly non-accountable and non-transparent evaluative environment that, through the sociological theory of Bourdieu (2004), presents a political and ethical challenge to the academy. In particular, as Google Scholar continues to be an intermediate in academic life, interjecting its opaque and trade-secret-protected systems of scholarly evaluation, it has begun to challenge the ability of the academy to understand how it evaluates itself.

After outlining the analytic method, this article proceeds by contextualising Google Scholar citation counts within the broader history of citation indexing and its coordinating function in the scholarly economy. The article then describes Google Scholar’s services, including its search, citation counting and scholar profiling tool. Finally, the article outlines Google Scholar’s platform dynamics, including its provision of free services, its user experience design and its exploitation of open access ideals and rhetorics.

Scholarly platforms and academic norms

The analysis of scholarly platforms and their relationship to metrics, evaluation and rankings has accelerated in recent years (Chen & Chan, 2021; Crane & Glozer; 2022). Our analysis supplements this literature on the performance, affordances and political economy of metrics and scholarly platforms, with specific attention given to the provision of evaluative bibliometrics by Google Scholar.

Our analysis uses the theoretical framework of ‘Handoff’ (Mulligan & Nissenbaum, 2020; Goldenfein et al., 2020) in order to parse the stakes of Google Scholar’s role in academic practice. The Handoff model affords a lens for a political and ethical analysis of sociotechnical systems that exposes the consequences when the performance of a given function shifts from one type of actor to another, where the inclination is to understand the latter actor as performing the same function as the former. Here, the function analysed is the provision of indicators that work as tools for coordinating the scholarly economy as they move between the world of commercial academic publishers and platform services like Google Scholar. The analysis requires a look at the different components of the system (be they humans, organisations, interfaces, users, etc.) in each arrangement, paying attention to how components act-on or engage each other (i.e. by force - including the direct constraints imposed by a technical system, or affordance, such as the ways user reactions are elicited by exploiting human actors’ tendencies to respond to cues in systematic ways). The lens “decomposes” the way these sets of actors and components engage each other, with the goal of showing the ethical and political significance of the shift from one configuration to another (Mulligan & Nissenbaum, 2020).

To that end, the analysis here compares the organisational, technical and ethical arrangements of bibliometric evaluation, especially with respect to their scholarly market coordination function, in the forms of Journal Impact Factor and Google Scholar. We note that this is particularly difficult in the context of a technical system like Google Scholar that, by design, resists efforts at systematic analysis. That absence of transparency, however, has become central to the normative stakes of Google Scholar’s popularity in the academic field.

Evaluative bibliometrics in the scholarly economy

Analysing Google Scholar’s reconfiguration of academic practices requires understanding, to whatever degree possible, the system that Google Scholar interrupts, as well as its political and ethical orientations. Historical work on scholarly publishing typically identifies its normative functions as community formation, research dissemination and the establishment of intellectual priority (Csiszar, 2018; Nissenbaum, 2002).1 Evaluative bibliometrics, however, are the product of their own set of historical forces. Geopolitics and competition in post WWII industrial science and technology meant a dramatic increase in funding for scientific research. As the quantity of research output increased, the need for better and faster indexing emerged, along with a demand for abstracting, reviewing and library services. Not long after the war, the UK Royal Society convened the Empire Scientific Conference, wherein it was reported that abstracting services only covered about half of published research papers (Shaw, 1948). The rest remained more or less undiscoverable. Support for scientific information services had not kept up with support for science research. Eventually the problem of insufficient digesting was understood to be an impediment that was slowing down scientific discovery (Bowles, 2000).

Concurrent developments in computation offered potential solutions to this crisis of information organisation, but required a method to code scientific information in a manner amenable to automated processing. Eugene Garfield’s answer to this problem was citation indexing. Rather than digesting research articles according to their semantic content as librarians had traditionally done, citation indexing enabled the organisation of articles according to the works they referenced. Characterising research according to its networks of references afforded a good proxy for content digesting, enabling researchers to trace the intellectual lineage of concepts while using a statistical language that could be parsed by computers and tabulating machines. Garfield commercialised this process in 1964, launching the Science Citation Index through his company, the Institute for Scientific Information (ISI) (which became part of Thomson Reuters in 1992 and Clarivate Analytics in 2018). The SCI initially included an index of 1.4 million citations from 613 journals. In 1972, Garfield published an article in Science titled ‘Citation Analysis as a Tool in Journal Evaluation: Journals can be ranked by frequency and impact of citations for science policy studies.’ It outlined his method for ranking journals listed in the SCI according to average citations per article over a period of time. The ranking unit developed was Journal Impact Factor (JIF).

Not only did JIF inform science policy by providing a shorthand for scientific authority and influence (Gingras, 2016, p. 7), in offering a proxy for research quality it became a mechanism for coordinating the scholarly economy. This is because JIF became a unit of equivalence capable of representing different forms of value pursued by different classes of actors in the academic field. For publishers, JIF became a proxy for economic capital through its determination of subscription prices. For scholars, reputational prestige was demonstrated through the JIF of a chosen publication forum, which for research managers, became a proxy for researcher quality (Fyfe et al., 2017; Cameron, 2005). As governments, foundations and other grant-giving institutions began to demand higher “accountability” from universities through the 1980s, JIF also became a way for universities to demonstrate value-for-money in research, often described as research “excellence” (Shore & Wright, 2000), as well as more broadly implementing science policy through competitive market dynamics under the imperatives of New Public Management (Strathern, 2000; Gläser & Laudel, 2007). In the context of such demands, universities became more interested in acquiring scholars that could demonstrate a capacity for measurable “impact”, for which JIF became the most convenient shorthand (Alexander, 2000; Salter & Tapper, 1994). JIF thus perpetuated academia’s reification into a status economy (Hamann & Beljean, 2017).

There are numerous components to the JIF evaluative ecosystem. These include the commercial businesses that derive indicators as commercial products from their database of journals and citations. Academic journals compete for entry into the SCI database, and for higher JIF scores by working to become attractive fora for highly cited scholars (and sometimes by manipulating citation counts). In order to pursue high prestige publications, individual researchers are sometimes (of course, not always) incentivised towards high JIF publications in their disciplines. Frequently, universities use journal quartile or quintile rankings (i.e. highest 25% or 20% of JIF ranked journals per discipline) as part of their research performance calculations for scholars. And JIF is commonly included in research quality evaluation for various research managers, stakeholders and funders.

The consequences of this arrangement have been clearly described in innumerable critiques of JIF and its impact on academia. These critiques are sometimes technical — i.e. the failure to adequately reflect citation performance (Russell & Rousseau, 2001); JIF’s failure to normalise citation counts across journal subject categories; its failure to correct for self-citations. They are sometimes epistemological — i.e. the impossibility of “measuring” academic quality; its incentivisation of “review” articles; its negative impact on knowledge production and agenda-setting, which includes channeling research towards universal rather than local domains (Bianco et al., 2016); its problematic cooptation of the notion of “impact” (Hecht et al., 1998). And the critiques are sometimes political — i.e. JIF’s inclusion of only a fraction of the world’s journals; its focus on certain publication formats over others; its privileging of English language content and associated geographies; its failure to take into account disciplinary citation conventions; its deployment as a marketing tool (Hecht et al., 1998); and its “off-label” use for the evaluation of individual researchers, programs and institutions.

The latter critique is related to JIF’s expanded reception in academic practice and its coordinating function. JIF was always intended to be a market coordination tool — but only between university libraries and journal publishers. But the affordances of JIF as an indicator of value enabled its coordinating function to be extended across academic interactions, such as funding decisions, the hiring and promotion of researchers, and even decisions about what materials were worth reading and citing, despite the fact that this type of evaluation was unintended by JIF’s inventors or owners. Notwithstanding repeated warnings that JIF should never be used to evaluate research performance for an article or of an individual (van Noorden, 2010, pp. 864-865; Glänzel & Moed, 2002), it is indisputably used this way (see, among others, Quan et al., 2017; Rushforth & de Rijcke, 2015; McKiernan et al., 2019). For this reason, even Garfield himself came to see JIF as a ‘mixed blessing’ (Garfield, 1999).

JIF’s coordination role was not inevitable, but it is very much a product of how academics and other stakeholders use it as an indicator. Most parties very clearly understand the limitations of this tool (Evidence Ltd, corp creators, 2007). Much like Google Scholar, JIF is not particularly sophisticated — it has been described as ‘a poor man’s citation analysis’ (Pendlebury & Adams, 2012). But its simplicity ironically augments its coordinating utility. Its uptake is a product of both technological affordance, i.e. the efficiency of indicators over subjective measures of value, and JIF’s explicit design as a market management tool, as well as the realities of academic practice, i.e. the reduction in labour necessary to engage in academic transactions. It is easy to use, access and comprehend by the various actors in the scholarly economy, enabling decisions without spending time to subjectively ascertain the value of researchers and their research output.

While JIF remains a key indicator in the contemporary evaluative ecology, we suggest these coordinating functions have now been supplemented by additional indicators — including Google Scholar. The Handoff model does not insist on the total substitution of one set of components or configuration with another. Rather, the goal is to identify “what may have changed in the reconfiguration of function across component actors” (Mulligan & Nissenbaum, 2020, p. 5), sometimes sequential and sometimes simultaneous, and unpack those differences to expose the political and ethical stakes of the shift. As we show below, the provision of indicators by Google Scholar augments the market logic that was facilitated and amplified by JIF (Shore & Wright, 2000; Shore, 2010; Burrows, 2012) with a range of platform dynamics (see e.g. Viljoen et al., 2021) that have their own political and ethical significance. For instance, whereas JIF worked to diffuse scholarly evaluation into a decentralised market for scholarly authority, Google Scholar’s interface simulates a market while introducing novel interfaces that opaquely manage the visibility of market participants. In that way, Google Scholar uses its gatekeeper power to distribute academic authority according to its own unchallengeable algorithmic designs.

It is important to note that JIF has been supplemented by numerous metrics systems, of which Google Scholar citation count is only one. Further, many of these bibliographic tools similarly enact platform-like features in explicitly profitable ways, such as channeling bibliometrics into institutional ranking systems (Chen & Chan, 2021). Those systems have been, and continue to be, subject to rigorous analysis. This will continue, as platformisation of universities, publishing and evaluative ecologies is the inevitable expression of informational capitalism’s dominant logic (Cohen, 2019). What we show below is how some of the features of platformisation, and especially the Google Scholar enactment of platformisation, have political and ethical consequences. And while Google Scholar has been persistently analysed for its comparative performance with other bibliometric services (Halevi et al., 2017), its political and ethical consequences are not frequently an analytical focus (for an exception, see van Dijck 2010).

The normative stakes

The primary ethical concerns exposed by our description of Google Scholar relate to concepts of academic autonomy. At one level, the evaluation of researchers and research centres has become central to the management of relationships between universities and their stakeholders — be they states or private funders. Financial independence often relies on performance outcomes. Indeed, if bibliometrics are central to evaluation, they are central to questions of institutional autonomy. But academic evaluation is also fundamentally connected to academic autonomy in the sense of self-governance (Whitley, 1984). For instance, beyond financial independence, academic evaluation implicates autonomy at the level of field formation — that is, the creation of a bounded academic identity and independence as a field of practice. Historically, academic autonomy, in this sense, refers to the ways in which professional scientists and scientific associations separated themselves from journalists and the broader public through conventions of authorship and evaluation, like anonymous refereeing. Csiszar (2018), for instance, describes how scientific identity and authority, what we might think of as precursors to recognition and capital, were minted originally through periodical authorship with its evaluative conventions. In this sense, autonomy reflected a boundary between the academic field and other social systems.

The social function of the university has transformed over recent decades, however. Universities’ shifting relationships with states and other stakeholders means that notions of academic autonomy have been recalibrated for the “knowledge economy” and informational capitalism (Olssen & Peters, 2005; Shore & Taitz, 2012; Cohen, 2019). Universities are now “economic engines”, expected to forge partnerships with industry and commercialise knowledge (Shore & Taitz, 2012) as a way to demonstrate efficiency and value (for money) to their various private and public stakeholders. This challenges the notion of a separation between society and the academic field. Autonomy has accordingly transformed, becoming a question of managing interactions with the market and other stakeholders, rather than avoiding industrial influences on knowledge production.

In this context, and somewhat ironically, autonomy has itself become a metricized accounting of national education policies in service to regional and global hegemonies of university performance (see e.g. Bennetot Pruvot & Estermann, 2017). Questions of independence from conflicted interests and the market, self-regulation or external influence on agendas, are subsumed into metricised levels of organisational autonomy, financial autonomy and staffing autonomy, through which autonomy is scored and compared across institutions and nations for the sake of university rankings.

But within this weird patchwork of contradictory meanings, autonomy continues to do some meaningful normative work — especially in the context of evaluation. Indeed, it has always been central to academic autonomy that the work of academics be evaluated by other academics. According to Bourdieu (2004), academic knowledge production can only be saved from its inherently contingent and political construction through transparency in the processes of evaluation, and through the pursuit of disinterested and peer-distributed forms of symbolic capital or credit. From here, Bourdieu makes the claim that:

The fact that [knowledge] producers tend to have as their clients only their most rigorous and vigorous competitors, the most competent and the most critical, those therefore most inclined and most able to give their critique full force, is for me the Archimedean point on which one can stand to give a scientific account of scientific reason, to rescue scientific reason from relativistic deduction and explain how science can constantly progress towards more rationality without having to appeal to some kind of founding miracle… The closure upon itself of the autonomous field constitutes the historical principle of the genesis of reason and the exercise of its normativity (p. 54).

That ideal has been meaningfully leveraged to limit whose evaluation of academic work and researchers ought to be considered meaningful and acceptable. For instance, Smith et al. (2011) describe, in the context of Australian and UK Research “Excellence” Frameworks, how “impact” requirements in research excellence assessments challenge academic autonomy because of the role of external, non-academic, industry assessors with the consequence of effectively replacing peer-review with ‘expert-review’ (p. 1370). Similar concerns have been raised with respect to the identity of individuals who determine the appointments of academic staff (Henkel, 2007, p. 92). We argue this imperative applies equally to the deployment of evaluative bibliometrics and their coordination function. We see academic autonomy, in this evaluative dimension, as one of the key values at stake in the handoff between systems of evaluative bibliometrics and their use as market indicators between commercial publishers and platform companies.

The use of metrics for evaluation, provided by any entity, raises complex issues. As noted above, there is no essential or inherent relationship between impact (defined by metrics) and quality. Metrics often smooth over geographic and disciplinary considerations, disconnecting those indicators from the real contexts of academic work (Bianco et al., 2016). To that end, the Leiden Manifesto for Research Metrics, for example, outlines a series of principles for the best practice use of metrics in evaluation (Hicks et al., 2015). Managers are enjoined to use metrics to support rather than usurp qualitative assessment; to localise evaluation against the mission of a research centre; to compare researchers against those in the same field; to ensure databases and data processing used for assessment are open and transparent; and to enable researchers to verify and rectify records. Central to these best practices are transparency and contestability in the construction of metrics.

The question then becomes whether Google Scholar metrics, which are used for scholarly evaluation and market coordination, challenge those ideals of transparency in evaluative mechanisms, tools and processes, and whether this is exemplified in the Google Scholar ecosystem when compared to the JIF system.

There is an unquestionable transparency deficit with JIF. On one hand, this pertains to criteria for inclusion in the relevant databases from which JIF is derived. Much like Google Scholar’s opaque definition of scholarliness, described below, building the SCIE and SSCI (the Science Citation Index Expanded and Social Sciences Citation Index from Clarivate Analytics) depends on opaque estimations of worthiness as determined by commercial operators. Similarly, despite the apparent simplicity of the calculation, JIFs have been challenged as non-reproducible (Anseel et al., 2004; Rossner et al., 2007). But subsequent research has refuted that argument. With access to the Web of Science Core Collection, authors have clearly demonstrated the reproducibility of JIF (Larivière & Sugimoto, 2019), and shown it to be an understandable and challengeable indicator. Critically, the databases from which this class of indicators derive are (commercially) available and interrogable — one can at least know if a forum is or is not included in the database. Ultimately, this means that JIF does not resist the academy’s capacity to understand how it is constructed, or how it operates as an evaluative metric. That is why scholars like Gingras (2016) suggest that the Web of Science and Scopus offer advantages over Google Scholar in that: ‘one can know the list of journals included at any time’, whereas Google Scholar content is not well defined and varies constantly. For Gingras, ‘this is problematic from an ethical point of view, as evaluations should be transparent when they affect people’s careers’ (Gingras, 2016, p. 64).

In addition, not only does the Google Scholar system incentivise, through technical affordance, the continued non-reflexive use of indicators by academics, it also, through technical and organisational constraints, challenges academia’s capacity to understand how this type of bibliometric evaluation occurs at all. As described below, and contrary to the Leiden Manifesto best practices, through the way Google Scholar reconfigures evaluative hierarchies and networks, and how scholars and research managers interact through its technical system, it encourages purely quantitative comparison. Such comparison is applied across an academic domain, with no contextual or disciplinary divisions, through non-discipline-normalised metrics that the service will not rectify. Through its system of evaluative bibliometrics and citation counting, it has managed to mint a new form of symbolic capital that is distributed according to its own opaque and non-challengeable algorithmic determinations. The scale of its index, the definition of scholarliness and the mechanism for extracting citations are not transparent. Those opaque and non-accountable systems of research and researcher evaluation explicitly challenge the academy’s understanding of how it evaluates itself.

In the sections that follows, we describe the Google Scholar ecosystem, and then how the political and normative consequences described above emanate from the way Google Scholar imports into scholarly evaluation platform logic like two-sided markets, intermediation and value extraction, non-transparent relevance ranking and problematic priorities in interaction design.

From JIF to Google Scholar citations

Google Scholar launched in 2004, the same year that Elsevier launched rival bibliometrics platform Scopus. Beginning in 2006, citation counts were included in search results. The scholar profiling service, “Google Scholar Citations”, was launched in 2011. Citation counting drives both search and scholar profiles in different ways, and we argue that it has become a new unit of value for coordinating the scholarly economy, albeit in a manner different from JIF. While JIF quite explicitly shifted research priorities and visibility around inclusion in the SCI (De Bellis, 2014), Google Scholar is much more granular and totalizing in its shaping of research and researcher visibility according to citation count.

Search

Google Scholar search scans its index of scholarly documents and provides results according to relevance ranking. Neither the boundaries of the index,2 nor the meaning of “relevance” are transparent. Typical academic search engines index explicit repositories of journals, often organised by discipline. The choice of forum dictates which search engines will index a publication, and which university library subscriptions will make it available. The visibility of research thus depends on hierarchies of journal quality and perceived value-for-money. Alphabet is not an academic publisher however, and Google Scholar’s index is repository and discipline agnostic — it offers results from multiple openly available and pay-walled repositories. Larger academic publishers opened their repositories to Google Scholar around 2009, both in response to the “open science” movement, and in recognition of Google Scholar’s growing centrality as a federated search tool. Direct access to publications was eventually offered through link resolvers (Acharya, 2012), meaning users could access PDF files from pay-walled journals their university library subscribed to directly from Google Scholar search without ever logging in to a publisher interface.

There are vague descriptions on the Google Scholar website of how “scholarliness” is determined by this system. Google Scholar employees have also offered some limited commentary.3 The tendency is towards inclusivity. However, the algorithm making such determinations remains an opaque trade secret. Studies have shown nonetheless, that the academic web indexed by Google Scholar is markedly larger than any other academic search engine (Martín-Martín et al., 2018; Gusenbauer & Haddaway, 2019). This means, query-parsing being equal, it returns a potentially greater range of “relevant” results and potentially a higher number of citations. However, the absence of an API, and the limitation of 1,000 results per query, means there is no way to accurately measure the size of the index.

Google Scholar also indexes a great deal of what is sometimes termed “grey literature”. This includes documents from journals, blogs, newspapers or free/unmoderated repositories like SSRN, ArXiv or Academia.edu. That reconfiguration of what materials are visible and accessible through scholarly indexes supplements legacy gatekeepers such as journal editors and peer-reviewers with novel actors capable of producing academic esteem and symbolic capital through their interaction with (i.e. participating in the publication of materials that are indexed by) the Google Scholar system. This trend may be generally correlated with evolutions in academic publishing in networked environments and the pluralisation of academic outputs. Google Scholar both leverages and perpetuates this phenomena.

Making available a much larger corpus of research necessitates new ways of relating that research to a search query. When Google Scholar launched, the majority of other academic search engines used keyword-in-context, or Boolean operations on specific words to determine search engine result rankings. They gave researchers the ability to self-sort results according to discipline, jurisdiction, date, keywords and other metadata, word frequency, syntax relationships and many other criteria. In other words, users defined their own parameters of relevance. Google Scholar however, offered researchers fewer options to customise results, limiting search only by date range, specific author or journal, but critically, not discipline, document type or syntax relationship.4 This document ranking system, Google Scholar’s “primary innovation” (van Noorden, 2014), remains opaque and trade-secret protected. However, empirical work suggests that relevance in the Google Scholar depends primarily on the number of times an article is referenced by other documents in the scholarly index (Beel & Gipp, 2009a; Beel & Gipp, 2009b; Rovira et al., 2019). That means the most important factor determining search ranking is estimated to be citation count.5

In this context, academic information retrieval is reorganised around a logic of aggregation and indexing as opposed to self-defined notions of relevance and publication prestige. The influence of citation counting moves academic evaluation away from the disciplinary “expertise” associated with editors and reviewers, and into new networked hierarchies of authority determined by distributed citation practices, though not necessarily constrained to the academy or publications evaluated by academics. Because citation counts influence search rankings, academic search itself then becomes the product of that distributed evaluative process. However, much like the algorithms that determine whether a work is scholarly or how relevant it is to a user, the algorithmic “parsing” systems used to count citations out of the scholarly index are opaque.

Profiles

The researcher profile interface “Google Scholar Citations” launched in 2011, the same year as Microsoft Academic’s metrics service (which closed in 2021).6 Scholar profiles are automatically generated in Google Scholar, but can be edited by scholars. To edit a profile, the author is required to have a personal or university-provided Google account. Academic users of Google Scholar are therefore subject to ordinary Google user terms of service (including around data collection, processing and advertising), rather than, for instance, the more protective and negotiable terms in ‘G Suite for Education’.7

Google Scholar Citations displays scholar profiles that include basic institutional and biographic information, lists publications with links and citation counts and shows a calculation of scholars’ h-index and i-10 index values. The i-10 index lists the number of documents with more than 10 citations. The h-index, developed in 2005 (Hirsch, 2005), is an indicator of “impact”, applied at the micro-level of individual researchers. It is based on the number of papers published by a scholar and the number of citations referring to those publications over a particular time frame. Unsurprisingly, the h-index presented on Google Scholar Citations uses Google Scholar’s system of citation counting for the calculation.

H-index has since become a dominant mechanism for evaluating scholars, albeit in some disciplines and institutions more than others.8 Aside from its dubious relationship to actual researcher quality (see e.g. Lehmann et al., 2005; Kolton & Hafner, 2021), much like JIF, it smoothes over disciplinary citation and collective authorship conventions, thus facilitating non-normalised researcher evaluations and comparisons (Bornmann & Daniel, 2008; Hirsch, 2005; Iglesias & Pecharromán, 2006; Kelly & Jennions, 2006; Malesios & Psarakis, 2017).

Some studies have sought to discern the distribution of h-indexes across certain disciplines, in particularly geographical contexts, using specific citation databases, at particular points in time (see e.g. Malesios & Psarakis, 2017). These indicate, for instance, that using the SCI, computer science has a lower h-index average than mathematics, which is lower than economics - with clinical medicine and chemistry at the top. However, different h-indexes across disciplines are also the product of different publication coverage for each discipline in the databases used to calculate the indicator. The more disciplinary fora included in the database, the higher the citation counts across the corpus that inform the h-index. In the sciences, Harzing (2010) has shown that Google Scholar, for instance, provides radically higher citation counts for computer science when compared to other databases, whereas many other sciences show roughly equivalent citation counts and h-indexes.

Various factors result in vastly different h-index averages across disciplines. At the same time, different disciplines have different interpretations and treatments of the indicator, and citation counting more broadly in their evaluative conventions. Similarly, different funding bodies and universities also have different treatments of citation counting systems, where anecdotal discussions with hiring committees suggest better-resourced institutions have the capacity to spend more time and engage more qualitative expertise and evaluation for the sake of hiring and promotion decisions. Nonetheless, metrics still inform decisions from the highest to the lowest level. Metrics are used in the evaluation of institutions (Wilsdon et al., 2015; Mingers et al., 2017) to dismiss or make scholars redundant (Else, 2021), as well as for hiring decisions. While we were not able to perform a comprehensive empirical study of the use of Google Scholar citations in academic review or promotion evaluations, as has been done previously for JIF (McKiernan et al., 2019), the escalating role of Google Scholar is both anecdotally and intuitively clear to researchers. Indeed, numerous academic job application interfaces now require Google Scholar research data.

A screenshot of a form with University of Melbourne branding that requests an applicant to fill out the following fields listed under Academic Profile: Link to LinkedIn profile (optional), Link to professional website (optional). Google scholar total citations (required), H-index (required), Link to your Google Scholar profile (required).
Figure 1: Google Scholar Metrics as compulsory inputs for Lecturer/Senior Lecturer (equivalent to Assistant Professor) job application in Information Systems at the University of Melbourne (recorded 17 October 2019).

And at the more quotidian level, citation count driven scholar profiles and search results inform ordinary decisions by scholars about who might be invited to speak, or whose work is worth reading, as well as the materials that students retrieve and analyse. While dismissing academic staff on the basis of Google Scholar metrics alone might be unthinkable, Google Scholar makes researcher evaluation extremely easy when compared to other bibliometric databases, thus centralising itself in the daily practice of academic work. That centrality is achieved, however, by infusing the scholarly world with platform dynamics that challenge scholarly norms like autonomy and transparency in evaluating researchers and their work.

Platform dynamics in the scholarly economy

 

“Free”

There are numerous commercial services that provide evaluative bibliometrics. One point of distinction between Google Scholar and other scholarly platforms is price. Large commercial academic publishers operate according to subscription models that increasingly bundle analytic and bibliometric products alongside content provision. That said, these business models are still highly opaque, often taking advantage of regulatory or funder-driven Open Access obligations, and coercive strategies for data and bibliometric valorisation through asymmetrical power relationships with universities. Open access fees are increasingly part of these bundle deals. Newer platforms like Academia.edu sometimes represent themselves as free, but they use predatory freemium models and charge for access to features, or like ResearchGate, openly disclose that they trade in user data.9 Google Scholar, however, does not charge users, and is less explicit about its treatment of user data, making its motivation in the scholarly space more difficult to decipher.

As far as we were able to discern, Google Scholar does not directly make revenue. Google scholar is free for users and does not display advertising. In an interview with Nature, co-creator Anurag Acharya described Google Scholar’s raison d’etre as follows: to ‘give back to the research community’ (van Noorden, 2014). Interviews reported in the New York Times in 2004, however, indicated an initial intention to include advertising (Markoff, 2004; Terdiman, 2004). To that end, we note analyses of “gift” structures in platform economics by Fourcade & Kluttz (2020), and of “free” online services by Hoofnagle & Whittington (2014). Describing how different business structures in the digital economy rely on non-monetary exchanges, those authors describe how zero-price exchanges enable platforms to intermediate and control data flows, which can then be taxed in different ways.

Google Scholar’s gift claim is thus better understood as a site of strategic contradiction. As described by Fourcade and Kluttz (2020, p. 6), it is more likely to be an ‘instrument to generate the primitive accumulation of data’, a mechanism to enroll users into a digital system under the guise of a benign process, to institutionalise platform approaches to daily academic practice and as a way to ‘mask[] structural asymmetry’ (p. 3). By intermediating transactions between researchers and content, Google Scholar’s data capture gives it the broadest view of academic research as a practice. From this vantage point, Alphabet becomes the best placed actor to provide “business intelligence” about trends in research and publishing, about topics and researchers obtaining traction, and about movement and flows of research between institutions.

Further, while there is no advertising presented on the Google Scholar service, when we asked Google Scholar whether Google Scholar user interaction and behavioural data was commercialised through data markets, behavioural advertising, or in the development of other commercial products pitched to academics, universities or publishers, they refused to answer. Data transparency requests subsequently lodged under Article 15 of the European General Data Protection Regulation clarified that Google Scholar data is governed by ordinary Google terms of service, and that data is processed according to the ordinary Google privacy policy. Alphabet is thus able to extract commercial value from Google Scholar through its “service/data-profile/advertising complex” (Lovink & Tkacz, 2015, p. 15), even though no ad inventory is explicitly sold on Google Scholar websites.10

At the same time, denying its commercial nature enables Google Scholar to avoid obligations of accountability to users. For instance, users are referred back to publishers to amend errors in article descriptions. Google currently suggests that updates in their system will “usually take 6-9 months” (Google Scholar Website, see also Internet Archive recordings from June 2010 and August 2021).11 Similarly, there are only limited ways of getting in contact with Google Scholar staff, making user inquiries extremely difficult. And there are no ways to challenge absent citations (even if citing papers are present on the Google Scholar index) or contest search rankings. This maintenance of distance may contribute to Google's resistance to critique. See, for instance, Google Scholar's continued 'trans-exclusionary and sexist design' as 'the most influential organisation in scientific publishing to not enter a partnership affirming the ability of trans authors to change their names' (Speer & Signers, 2021; see Nerzig (2021) for a list of parties to the partnership, including publishers discussed in the present research: Clarivate, Elsevier, Wiley and Scopus).

User experience

Google Scholar’s interaction design is a large part of its success. Because it is free, users are not required to go through library logins or publisher paywalls. The capacity to search “<researcher’s name> google scholar” in any search engine and immediately access publication history, citation counts and h-index makes researcher evaluation easy. Google is also able to leverage its dominance in web-searching by suggesting “relevant” Google Scholar results in ordinary Google searches. Other free academic services, like ResearchGate or Microsoft Academic, do not provide anything close to that level of usability for quick searches or researcher evaluation. With Google Scholar, accessing scholarly work and scholar profiles becomes part of the native internet browsing experience. However, some of these interaction design strategies contradict typical academic practices.

Bibliometricians, librarians and conscientious researchers initially critiqued the inability to manipulate Google Scholar search results according to users’ own criteria (Giglierano, 2008; Falagas et al., 2008; Lewandowski, 2010). However, the removal of that interactivity allows Google Scholar to more “transparently” intermediate scholarly reality (Day, 2014, p. 27). “Transparency” in interface and interaction design is, ironically, a design principle that substitutes complexity for intuitive representation and use. As Michael Black notes ‘we trust that interfaces present us with a faithful representation of the systems they provide access to or largely assume that those mechanisms they mask are hidden from us for good reasons’ (2020). Transparency is thus a way to align and structure user behaviour with an intended set of acceptable practices.

In this way, the capacity to perform targeted research is usurped by design practices with other priorities. For instance, the ability to search by subject area was once a feature of Google Scholar, but it was removed in 2012 in order to present a ‘new modern look’ (Harzing, 2016). Google Scholar also returns search results faster than its competitors — which research suggests is more effective for retaining users (Arapakis et al., 2014; Crescenzi et al., 2016; Brutlag, 2009). Users should not think that fewer options to manipulate results makes Google Scholar a less sophisticated or powerful tool. Rather, the Google Scholar interface dissimulates both the power of the tool, and the political consequences of the design choices made. Another example is the lack of a display for publication Digital Object Identifiers (DOI), and listing multiple versions of articles, each of which inflate citation counts for particular disciplines. Accepting these user experience strategies means accepting an ex-ante evaluation of scholarship premised on the invisible knowledge hierarchies built by Google, that are entirely opaque, unknowable and unchallengeable by the user.

While authorship and citation conventions in different fields elevate or decrease average citations, the decision to not normalise those counts by discipline in an a-contextual presentation also builds new disciplinary hierarchies. Studies suggest that while Google Scholar’s citations counts are generally larger than Web of Science and Scopus, Engineering disciplines have roughly double the number of citations on Google Scholar, whereas physical and life sciences are only boosted by approximately 50% (Harzing & Alakangas, 2016). These amplifications, alongside the much higher number of computer science researcher profiles on Google Scholar (Ortega & Aguillo, 2012), perpetuate a shift in intellectual topography through which computer science and computer engineering appear to be the disciplines around which all knowledge is organised.

Open access

A third Google Scholar dynamic that challenges certain academic politics and ethics is how it has leveraged “open access” publishing to make itself a new gatekeeper in the scholarly world. Heralded as a remedy to the predatory practices of commercial academic publishing, open access has become a pillar of science policy around the world (see e.g. European Commission, 2019). Supported by the Open Society Institute, the 2002 Budapest Open Access Initiative statement of principles declared:

An old tradition and a new technology have converged to make possible an unprecedented public good. The old tradition is the willingness of scientists and scholars to publish the fruits of their research in scholarly journals without payment, for the sake of inquiry and knowledge. The new technology is the internet. The public good they make possible is the world-wide electronic distribution of the peer-reviewed journal literature and completely free and unrestricted access to it by all scientists, scholars, teachers, students, and other curious minds (Budapest Open Access Initiative, 2002).

Library budget constraints and political intervention, especially in Europe, have made open access mainstream, and developments like Plan S indicate that it could become the default publishing mode (Aspesi & Brand, 2020). However, open access has not proved the utopian solution that early internet optimists had hoped for, and it has failed to decrease reliance on legacy commercial publishers (see e.g. Chen & Chan, 2021). Some suggest that open access has effectively morphed into its neoliberal antithesis (Mirowski, 2018), especially in its “hybrid” Green and Gold manifestations where researchers pay for open access, therefore improving publishers’ profit margins (Aspesi & Luong, 2014). More important to our purposes, open access has also made space for other commercial actors like Google Scholar to build value capturing platforms atop the infrastructure of commons-based open access content (see e.g. Bodó, 2019). As Mirowski (2018) reminds us: ‘Platform capitalism meets open science; romance ensues’. Indicative of its interest in those developments, the Google Scholar interface now polices open access mandates by offering a leaderboard of open access mandate compliance, and displaying open access information about articles on researcher profiles, even inducing scholars to upload articles to their Google Drive (see e.g. van Noorden, 2021).

To that end, the academic community has largely failed to translate open access publishing into its own value-added analytics or tools for coordinating the market for academic prestige. Instead of open access contributing to greater academic autonomy through standardised, transparent and accountable evaluative analytics, commercial platforms have managed to intermediate, make proprietary, and extract value from what the open access movement has made available. In making the content of open science available on its platform, Google Scholar intermediates the scholarly economy through which its own unit of scholarly capital is minted and flows. Open access’s failure to decrease reliance on commercial publishers has thus ironically created new dependencies on commercial platform providers with opaque business models.

Legacy publishers have followed similar strategies, for instance, acquiring pre-print platforms and open access repositories, developing new academic analytics products and services, and locking institutions into bundled access, publishing, analytics and multi-service portals. Wiley, for instance, acquired the open access repository and infrastructure, ‘Knowledge Unlatched’, in December 2021 (Fund, 2021). Prior to that, Elsevier purchased the free repository Social Science Research Network (SSRN), as well as Mendeley, Bepress/DigitalCommons, Plum Analytics and other institutional repositories.12 Although owning open access repositories may seem inconsistent with Elsevier’s business model, Elsevier describes these purchases as a move into research and technology data management (Elsevier, 2017). Publishers extract value from these acquisitions by channeling data flows acquired from open repositories into products and subscriptions sold to universities at high prices in the form of “research services”, “academic analytics” and “business intelligence” (see e.g. Kelty, 2016).13 Linking SSRN content and usage data to Scopus citation data, for example, offers a way to extract value from that commons-based infrastructure without charging for content provision.

Elsevier has also been able to make its tools indispensable to universities by integrating them with university ranking data infrastructures. In this way, Elsevier has become a university “ranking” business that, for instance, provides the exclusive metrics for the Times Higher Education rankings (see e.g. Chen & Chan, 2021). HeinOnline has taken similar steps, making substantial investments in citation counting to leverage its analytics directly into the US News University law school ranking system.14 Being the data and analytics channel for a primary law school ranking system enables HeinOnline to capture tremendous value from its catalogue, while simultaneously cementing its centrality as a repository for legal scholarship.

Chen & Chan (2021, p. 425) describe these moves as commercial publishing firms ‘leveraging rankings as part of their strategies to further extract rent and assets from the university beyond their traditional role as publishers and citation providers’. Google Scholar follows a different path, however. By offering a free platform that works as a federated search engine and simple evaluative tool, it is able to intermediate the totality of the academic corpus, capturing all the data generated as content flows between multiple institutional, commercial, pre-print and “grey” repositories and users. But how this control over scholarly capital is translated into economic capital is even less clear. Unlike Elsevier, whose products university executives, for better or worse, actively decide to purchase, Google Scholar has become an information gatekeeper, and indeed the owner and controller of a dominant and proprietary marketplace, without scholars or university administrators actively making that choice (see e.g. Tanczer et al., 2019).

While Google might claim it does not intend for Google Scholar to compete with Web of Science, Scopus or any other commercial bibliometric service, and is instead about providing an “open” platform for academic research (Butler, 2011), such comments need to be interpreted in light of the political, economic and normative configurations that this type of “openness” generates (see e.g. Powers & Jablonski, 2015). Google thrives on an “open” web, and by encouraging the web as the domain of all represented human thought, with authority determined by topology, it is restructuring the very meaning of authority and expertise in ways that are unexaminable to the academic world, let alone to other stakeholders. While JIF participated in building this market for authority, and has its own transparency deficits, Google Scholar amplifies these trends by distributing authority according to the opaque algorithmic rules of a proprietary market that it owns and controls. Google’s continuing centralisation of academic tools makes the academy more dependent (see e.g. Vaidhyanathan, 2011), more vulnerable and less autonomous.

Google Scholar and academic autonomy

The Google Scholar system reconfigures bibliometric evaluation and scholarly market coordination around a unit of value — Google Scholar citation counts — that it generates through its proprietary algorithmic system. Compared to previous systems performing those functions, like JIF, Google Scholar introduces new components and actors that are able to generate scholarly authority that operates beyond the boundaries of the academic field’s primary evaluative mechanism — peer review — in unexaminable and unknowable ways. The various components of the system also act on each other in novel ways. The affordances of Google Scholar’s interface alter the modes by which researchers interact with scholarly content, nudging researchers away from their own determination of relevance, in favour of accepting intellectual topographies and hierarchies of authority that are opaquely constructed and controlled by Google Scholar. The platform enrolls users through “free” services, leverages open access policies and rhetorics to intermediate data flows between researchers and repositories, and extracts behavioural data about scholars, all while offering nearly no accountability to users. Google Scholar’s usability, ironically limiting the degree of interaction and adjustment of its system, has centralised and institutionalised the system into a daily part of academic life, informing critical decisions about the quality of individual researchers, as well as what scholarly content is visible and authoritative.

Platforms bring new organisational and economic logic to the domains they intermediate. While large dominant platforms are the subject of a great deal of critique, they clearly bring useful tools. Improved information retrieval and processing across the academic field is a generally welcome development. But the analysis above suggests that the ways in which Google Scholar intermediates the scholarly field often contradicts what we understand to be academic best practices.

Google Scholar is not alone in its introduction of platform dynamics and minimal transparency in the scholarly context. Like Google Scholar, Web of Science also acts as a “platform” in the sense of rule-setting and market-making. Platforms like SSRN exercise profoundly non-accountable control over what is included in their index, with unclear standards for what is considered “scholarly” or of scholarly merit (Crawford, 2019), using non-transparent mechanisms for policing metrics. It is probably correct, as the journal PLOS submitted to the UK inquiry into metrics in research evaluation in 2014, that ‘there are no adequate sources of bibliometric data that are publicly accessible, usable, auditable and transparent’ (Wilsdon et al., 2015, p. 17).

Similarly, while Google Scholar enables the production of scholarly capital outside mechanisms of peer review, it is clear that peer-review is hardly ideal. There is a diversity of peer-review practices, some better and some worse, with positive developments occurring in the context of Open Review practices. But peer-review does not necessarily mean quality assurance. It also concentrates gatekeeping power in panels of editors and discipline experts, who dispense degrees, publications, rankings and funding, often for the sake of conservative disciplinary reproduction (see e.g. Sayer, 2014).

There are many studies on the limitations, biases and otherwise derogations from an ideal peer-review process (see e.g. Johnson & Hermanowicz, 2017), and unquestionably a great deal of important literature is created outside of such formal mechanisms (Lawrence et al., 2014). But despite its many problems, peer-review and other forms of collective editorship are at least performed by academics. From this perspective, even if peer-review is an inferior tool for evaluating quality and an instrument for exercising power, it at least separates scholarly evaluation from industrial interests. In other words, this is not a question of whether metrics or peer-review produce “better” evaluations. It is a question of whether it is possible to know (or influence) how a system of evaluation works, and to what degree it is accountable.

The economies of academic evaluation that emerged in the middle of the 20th century, while not necessarily transparent or independent, did not explicitly undermine the academy’s understanding of how it evaluated itself. In a changing political context, universities embraced the new marketised logic of evaluation. Within that suite of new practices and market dynamics, the evaluation of academic work and the hierarchies around it were still performed, more or less, by academics with their structures of gatekeeping and social reproduction intact. Where commercial publishers sought more editorial control, academics often resisted.15 Google Scholar’s tools, however, are less transparent and testable than the commercial alternatives, more ingeniously designed to achieve centralisation and less reliant on academic editorship and peer-review — the ‘least worst’ mechanism of academic evaluation (Wilsdon et al., 2015). Whereas JIF did not necessarily displace academic expertise at the centre of its evaluative economy, Google Scholar’s blurry operations perpetuate the belief in the epistemic superiority of the market. This neoliberal refrain: that the market knows best (echoed forcefully by the state and funding agencies), coupled with that market being entirely owned and controlled by Google, means the academy is less able to know itself.

Conclusion

Google Scholar does not have a monopoly on opacity. There is a dearth of transparency and accountability across the now numerous tools for evaluation and ranking. Google Scholar is not the only bibliometrics platform demonstrating these features or causing these issues. In many ways Google Scholar replicates issues already identified with JIF, especially its secrecy and lack of reproducibility (Rossner et al., 2007; Rossner et al., 2008). But as Onara O’Neill describes, old intermediaries are being replaced with new intermediaries ‘whose contributions are harder to grasp, and who are not and cannot be disciplined by the measures used to discipline the old intermediaries’ (O’Neill, 2020). Google Scholar insists on avoiding the accountability mechanisms that have at least some grasp on the explicitly commercial relationship between universities and academic publishers (see e.g. Davis, 2011; Davis, 2012; Davis, 2018).

Google’s ambitions in education are vast. Unquestioned reliance on such opaque measures in hiring, promotion and tenure committees, let alone decisions about what to read and who can speak, displace academic autonomy. If the academy cannot develop "an accountability relationship" with Google Scholar, it must find other ways to meet its own obligations (Rached, 2016). 

Acknowledgements

The authors wish to thank colleagues at the Digital Life Initiative at Cornell Tech, participants at the 2019 workshop “Money talks? – The impact of corporate funding on information law and policy research”, the 2019 Northeast Privacy Scholars Workshop and the Doctoral Reading and Theory Workshop at UC Berkeley’s School of Information. We also thank Sebastian Benthall and Eran Toch who contributed to earlier work that set up much of our thinking for this article, as well as Jef Ausloos who collaborated on related ideas. We would also like to acknowledge the reviewers and editors at Internet Policy Review for their comments and guidance that improved this research.

References

Acharya, A. (2012). Integrating discovery and access for scholarly articles: Successes and failures. Proceedings of the Charleston Library Conference. https://doi.org/10.5703/1288284315073

Acharya, A. (2015, June 10). Indexing repositories: Pitfalls and best practices [Conference presentation]. Open Repositories 2015 (OR2015), Indianapolis, Indiana, United States. https://media.dlib.indiana.edu/media_objects/9z903008w

Alexander, F. K. (2000). The changing face of accountability: Monitoring and assessing institutional performance in higher education. The Journal of Higher Education, 71(4), 411–431. https://doi.org/10.2307/2649146

Anseel, F., Duyck, W., De Baene, W., & Brysbaert, M. (2004). Journal impact factors and self-citations: Implications for psychology journals. American Psychologist, 59(1), 49–51. https://doi.org/10.1037/0003-066X.59.1.49

Arapakis, I., Bai, X., & Cambazoglu, B. B. (2014). Impact of response latency on user behavior in web search. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, 103–112. https://doi.org/10.1145/2600428.2609627

Aspesi, C., & Brand, A. (2020). In pursuit of open science, open access is not enough. Science, 368(6491), 574–577. https://doi.org/10.1126/science.aba3763

Aspesi, C., & Luong, H. (2014). Reed Elsevier: Goodbye to Berlin — The fading threat of open access (upgrade to market-perform) [Report]. Bernstein Research. https://www.richardpoynder.co.uk/Aspesi.pdf

Beel, J., & Gipp, B. (2009a). Google Scholar’s ranking algorithm: The impact of citation counts (an empirical study). Proceedings of the 3rd International Conference on Research Challenges in Information Science (RICS’09), 439–446. https://doi.org/10.1109/RCIS.2009.5089308

Beel, J., & Gipp, B. (2009b). Google Scholar’s ranking algorithm: An introductory overview. In B. Larsen & J. Leta (Eds.), Proceedings of the 12th International Conference on Scientometrics and Informetrics (ISSI 2009) (Vol. 1, pp. 230–241). BIREME/PAHO/WHO & Federal University of Rio de Janeiro. https://www.issi-society.org/publications/issi-conference-proceedings/proceedings-of-issi-2009/

Bennetot Pruvot, E., & Estermann, T. (2017). University autonomy in Europe III: The Scorecard 2017 [Report]. European University Association. https://eua.eu/resources/publications/350-university-autonomy%C2%A0in-europe-iii-%C2%A0the-scorecard-2017.html

Bianco, M., Gras, N., & Sutz, J. (2016). Academic evaluation: Universal instrument? Tool for development? Minerva, 54(4), 399–421. https://doi.org/10.1007/s11024-016-9306-9

Black, M. L. (2020). Usable and useful: On the origins of transparent design in personal computing. Science, Technology, & Human Values, 45(3), 515–537. https://doi.org/10.1177/0162243919865584

Bodó, B. (2019). Was the open knowledge commons idea a curse in disguise? – Towards sovereign institutions of knowledge. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3502119

Bornmann, L., & Daniel, H. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80. https://doi.org/10.1108/00220410810844150

Bourdieu, P. (2004). Science of science and reflexivity. Polity.

Bowles, M. D. (2000). Liquefying information: Controlling the flood in the Cold War and beyond. In M. R. Levin (Ed.), Cultures of control (pp. 225–246). Routledge.

Brutlag, J. (2009). Speed matters for Google web search [Report]. Google, Inc. https://services.google.com/fh/files/blogs/google_delayexp.pdf

Budapest Open Access Initiative. (2002). Budapest Open Access Initiative (BOAI) [Declaration]. https://www.budapestopenaccessinitiative.org/read/

Burrows, R. (2012). Living with the h-index? Metric assemblages in the contemporary academy. The Sociological Review, 60(2), 355–372. https://doi.org/10.1111/j.1467-954X.2012.02077.x

Butler, D. (2011). Computing giants launch free science metrics. Nature, 476(7358), 18. https://doi.org/10.1038/476018a

Cameron, B. D. (2005). Trends in the usage of ISI bibliometric data: Uses, abuses, and implications. Portal: Libraries and the Academy, 5(1), 105–125. https://doi.org/10.1353/pla.2005.0003

Chen, G., & Chan, L. (2021). University rankings and governance by metrics and algorithms. In E. Hazelkorn & G. Mihut, Research handbook on university rankings: Theory, methodology, influence, and impact (pp. 425–443). Edward Elgar Publishing. https://doi.org/10.4337/9781788974981.00043

Chen, G., Posada, A., & Chan, L. (2019). Vertical integration in academic publishing: Implications for knowledge inequality. In L. Chan & P. Mounier (Eds.), Connecting the knowledge commons — From projects to sustainable infrastructure. OpenEdition Press. https://doi.org/10.4000/books.oep.9068

Cohen, J. E. (2019). Between truth and power: The legal constructions of informational capitalism (1st ed.). Oxford University Press. https://doi.org/10.1093/oso/9780190246693.001.0001

Crane, A., & Glozer, S. (2022). What’s next for the quantified scholar? Impact, metrics, and (social) media. Business & Society, 61(4), 807–812. https://doi.org/10.1177/00076503211016778

Crawford, B. (2019, May 21). @SSRN and the (arbitrary) determination of ‘scholarly’ merit. The Faculty Lounge. Conversations about Law, Culture, and Academia. https://www.thefacultylounge.org/2019/05/ssrn-and-the-arbitrary-determination-of-scholarly-merit.html

Crescenzi, A., Kelly, D., & Azzopardi, L. (2016). Impacts of time constraints and system delays on user experience. Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, 141–150. https://doi.org/10.1145/2854946.2854976

Csiszar, A. (2018). The scientific journal: Authorship and the politics of knowledge in the nineteenth century. University of Chicago Press. https://doi.org/10.7208/chicago/9780226553375.001.0001

Davis, P. (2011, October 17). Gaming the impact factor puts journal in time-out. The Scholarly Kitchen. https://scholarlykitchen.sspnet.org/2011/10/17/gaming-the-impact-factor-puts-journal-in-time-out/

Davis, P. (2012, December 12). Gaming Google Scholar citations, made simple and easy. The Scholarly Kitchen. https://scholarlykitchen.sspnet.org/2012/12/12/gaming-google-scholar-citations-made-simple-and-easy/

Davis, P. (2018, June 27). Impact factor denied for 20 journals for self-citation, stacking. The Scholarly Kitchen. https://scholarlykitchen.sspnet.org/2018/06/27/impact-factor-denied-20-journals-self-citation-stacking/

Day, R. E. (2014). Indexing it all: The subject in the age of documentation, information, and data. MIT Press. https://doi.org/10.7551/mitpress/10073.001.0001

De Bellis, N. (2014). History and evolution of (biblio)metrics: Harnessing multidimensional indicators of scholarly impact. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics (pp. 47–66). The MIT Press. https://doi.org/10.7551/mitpress/9445.003.0004

de Búrca, G., & Weiler, J. (2020, February 5). Wiley and the European Law Journal. EJIL: Talk! Blog of the European Journal of International Law. https://www.ejiltalk.org/wiley-and-the-european-law-journal/

Dupuis, J. (2016, May 18). Elsevier buys SSRN: Another sideshow or the main event? ScienceBlogs. https://scienceblogs.com/confessions/2016/05/18/elsevier-buys-ssrn-another-sideshow-or-the-main-event

Else, H. (2021, March 25). Row erupts over university’s use of research metrics in job-cut decisions. Nature News. https://www.nature.com/articles/d41586-021-00793-7

Elsevier. (2017, August 2). Elsevier acquires bepress, a leading service provider used by academic institutions to showcase their research [Press release]. https://www.elsevier.com/about/press-releases/corporate/elsevier-acquires-bepress,-a-leading-service-provider-used-by-academic-institutions-to-showcase-their-research

European Commission. (2019). Future of scholarly publishing and scholarly communication: Report of the expert group to the European Commission [Report]. European Commission, Directorate-General for Research and Innovation. https://op.europa.eu/en/publication-detail/-/publication/464477b3-2559-11e9-8d04-01aa75ed71a1

Evidence Ltd, corp creators. (2007). The use of bibliometrics to measure research quality in UK higher education institutions [Research report]. Universities UK. https://dera.ioe.ac.uk//26316/

Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008). Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and weaknesses. The FASEB Journal, 22(2), 338–342. https://doi.org/10.1096/fj.07-9492LSF

Fister, B. (2016, May 17). Platforms and profits. Inside Higher Ed. https://www.insidehighered.com/blogs/library-babel-fish/platforms-and-profits

Fourcade, M., & Kluttz, D. N. (2020). A Maussian bargain: Accumulation by gift in the digital economy. Big Data & Society, 7(1). https://doi.org/10.1177/2053951719897092

Fund, S. (2021, December 2). Wiley acquires open access innovator Knowledge Unlatched. Open Research Community. https://openresearch.community/posts/wiley-acquires-open-access-innovator-knowledge-unlatched

Fyfe, A., Coate, K., Curry, S., Lawson, S., Moxham, N., & Røstvik, C. M. (2017). Untangling academic publishing: A history of the relationship between commercial interests, academic prestige and the circulation of research[Briefing paper]. Zenodo. https://doi.org/10.5281/ZENODO.546100

Garfield, E. (1955). Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159), 108–111. https://doi.org/10.1126/science.122.3159.108

Garfield, E. (1972). Citation analysis as a tool in journal evaluation: Journals can be ranked by frequency and impact of citations for science policy studies. Science, 178(4060), 471–479. https://doi.org/10.1126/science.178.4060.471

Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979–980. https://www.cmaj.ca/content/161/8/979

Giglierano, J. (2008). Attitudes of OhioLINK librarians toward Google Scholar. Journal of Library Administration, 47(1–2), 101–113. https://doi.org/10.1080/01930820802110951

Gillespie, T. (2014). The relevance of algorithms. In T. Gillespie, P. J. Boczkowski, & K. A. Foot (Eds.), Media technologies: Essays on communication, materiality, and society (pp. 167–194). The MIT Press. https://doi.org/10.7551/mitpress/9780262525374.003.0009

Gillula, J., & Cope, S. (2016, October 6). Google changes its tune when it comes to tracking students. Electronic Frontier Foundation Blog. https://www.eff.org/deeplinks/2016/10/google-changes-its-tune-when-it-comes-tracking-students

Gingras, Y. (2016). Bibliometrics and research evaluation: Uses and abuses. MIT Press.

Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193. https://doi.org/10.1023/A:1014848323806

Gläser, J., & Laudel, G. (2007). The social construction of bibliometric evaluations. In R. Whitley & J. Gläser (Eds.), The changing governance of the sciences (Vol. 26, pp. 101–123). Springer Netherlands. https://doi.org/10.1007/978-1-4020-6746-4_5

Gusenbauer, M., & Haddaway, N. R. (2019). Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217. https://doi.org/10.1002/jrsm.1378

Haddow, G., & Hammarfelt, B. (2019). Quality, impact, and quantification: Indicators and metrics use by social scientists. Journal of the Association for Information Science and Technology, 70(1), 16–26. https://doi.org/10.1002/asi.24097

Haider, J., & Sundin, O. (2019). Invisible search and online search engines: The ubiquity of search in everyday life (1st ed.). Routledge. https://doi.org/10.4324/9780429448546

Halevi, G., Moed, H., & Bar-Ilan, J. (2017). Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation — Review of the literature. Journal of Informetrics, 11(3), 823–834. https://doi.org/10.1016/j.joi.2017.06.005

Hamann, J., & Beljean, S. (2017). Academic evaluation in higher education. In J. C. Shin & P. Teixeira (Eds.), Encyclopedia of international higher education systems and institutions (pp. 1–7). Springer Netherlands. https://doi.org/10.1007/978-94-017-9553-1_295-1

Harzing, A. W. (2010, June 1). Citation analysis across disciplines: The impact of different data sources and citation metrics. Harzing.Com Research in International Management. https://harzing.com/publications/white-papers/citation-analysis-across-disciplines

Harzing, A. W. (2016, February 6). Google Scholar: Missing subject areas. Harzing.Com Research in International Management. https://harzing.com/resources/publish-or-perish/tutorial/google-scholar/missing-subject-areas

Harzing, A.-W., & Alakangas, S. (2016). Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2), 787–804. https://doi.org/10.1007/s11192-015-1798-9

Hecht, F., Hecht, B. K., & Sandberg, A. A. (1998). The journal “impact factor”: A misnamed, misleading, misused measure. Cancer Genetics and Cytogenetics, 104(2), 77–81. https://doi.org/10.1016/S0165-4608(97)00459-7

Henkel, M. (2007). Can academic autonomy survive in the knowledge society? A perspective from Britain. Higher Education Research & Development, 26(1), 87–99. https://doi.org/10.1080/07294360601166836

Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a

Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572. https://doi.org/10.1073/pnas.0507655102

Hoofnagle, C. J., & Whittington, J. (2014). Free: Accounting for the costs of the internet’s most popular price. 61 UCLA Law Review, 606(2014), 606–670. https://www.uclalawreview.org/free-accounting-for-the-costs-of-the-internets-most-popular-price-2/

Iglesias, J. E., & Pecharromán, C. (2007). Scaling the h-index for different scientific ISI fields. Scientometrics, 73(3), 303–320. https://doi.org/10.1007/s11192-007-1805-x

Johnson, D. R., & Hermanowicz, J. C. (2017). Peer review: From “sacred ideals” to “profane realities”. In M. B. Paulsen (Ed.), Higher education: Handbook of theory and research (Vol. 32, pp. 485–527). Springer International Publishing. https://doi.org/10.1007/978-3-319-48983-4_10

Kamrani, P., Dorsch, I., & Stock, W. G. (2021). Do researchers know what the h-index is? And how do they estimate its importance? Scientometrics, 126(7), 5489–5508. https://doi.org/10.1007/s11192-021-03968-1

Kelly, C., & Jennions, M. (2006). The h index and career assessment by numbers. Trends in Ecology & Evolution, 21(4), 167–170. https://doi.org/10.1016/j.tree.2006.01.005

Kelty, C. M. (2016, May 18). It’s the data, stupid: What Elsevier’s purchase of SSRN also means. Savage Minds. Notes and Queries in Anthropology. https://savageminds.org/2016/05/18/its-the-data-stupid-what-elseviers-purchase-of-ssrn-also-means/

Keusch, F., & Kreuter, F. (2021). Digital trace data. In U. Engel, A. Quan-Haase, S. X. Liu, & L. Lyberg, Handbook of computational social science, volume 1 (1st ed., pp. 100–118). Routledge. https://doi.org/10.4324/9781003024583-8

Larivière, V., & Sugimoto, C. R. (2019). The journal impact factor: A brief history, critique, and discussion of adverse effects. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 3–24). Springer International Publishing. https://doi.org/10.1007/978-3-030-02511-3_1

Lawrence, A., Houghton, J., Thomas, J., & Weldon, P. (2014). Where is the evidence? Realising the value of grey literature for public policy and practice [Discussion paper]. Swinburne Institute for Social Research. https://digitalcommons.unl.edu/scholcom/161

Lehmann, S., Jackson, A. D., & Lautrup, B. E. (2005). Measures and mismeasures of scientific quality. ArXiv: Physics and Society. https://doi.org/10.48550/ARXIV.PHYSICS/0512238

Lewandowski, D. (2010). Google Scholar as a tool for discovering journal articles in library and information science. Online Information Review, 34(2), 250–262. https://doi.org/10.1108/14684521011036972

Lindh, M., & Nolin, J. (2016). Information we collect: Surveillance and privacy in the implementation of Google Apps for Education. European Educational Research Journal, 15(6), 644–663. https://doi.org/10.1177/1474904116654917

Lovink, G., & Tkacz, N. (2015). Moneylab: Sprouting new digital-economic forms. In G. Lovink, N. Tkacz, & P. de Vries (Eds.), Moneylab reader: An intervention in digital economy (pp. 13–18). Amsterdam: Institute of Network Cultures. https://networkcultures.org/blog/publication/moneylab-reader-an-intervention-in-digital-economy/

Lynskey, O. (2017). Regulating ‘platform power’. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2921021

Malesios, C. C., & Psarakis, S. (2014). Comparison of the h-index for different fields of research using bootstrap methodology. Quality & Quantity, 48(1), 521–545. https://doi.org/10.1007/s11135-012-9785-1

Markoff, J. (2004, November 17). Google plans new service for scientists and scholars. The New York Times. https://www.nytimes.com/2004/11/18/technology/google-plans-new-service-for-scientists-and-scholars.html

Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160–1177. https://doi.org/10.1016/j.joi.2018.09.002

McKiernan, E. C., Schimanski, L. A., Muñoz Nieves, C., Matthias, L., Niles, M. T., & Alperin, J. P. (2019). Use of the journal impact factor in academic review, promotion, and tenure evaluations. ELife, 8, e47338. https://doi.org/10.7554/eLife.47338

Meagher, K. (2021). Introduction: The politics of open access — Decolonizing research or corporate capture? Development and Change, 52(2), 340–358. https://doi.org/10.1111/dech.12630

Mingers, J., O’Hanley, J. R., & Okunola, M. (2017). Using Google Scholar institutional level data to evaluate the quality of university research. Scientometrics, 113(3), 1627–1643. https://doi.org/10.1007/s11192-017-2532-6

Mirowski, P. (2018). The future(s) of open science. Social Studies of Science, 48(2), 171–203. https://doi.org/10.1177/0306312718772086

Morse, R. (2019, February 13). U.S. News considers evaluating law school scholarly impact. U.S. News. https://www.usnews.com/education/blogs/college-rankings-blog/articles/2019-02-13/us-news-considers-evaluating-law-school-scholarly-impact

Mulligan, D. K., & Nissenbaum, H. (2020). The concept of handoff as a model for ethical analysis and design. In M. D. Dubber, F. Pasquale, & S. Das (Eds.), The Oxford handbook of ethics of AI (pp. 231–251). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190067397.013.15

Nerzig, M. (2021, July 28). Scientific publishing organizations and national laboratories partner on transgender-inclusive name-change process for published papers. Berkeley Lab News Center. https://newscenter.lbl.gov/2021/07/28/transgender-inclusive-name-change-process-for-published-papers/

Nissenbaum, H. F. (2002). New research norms for a new medium. In N. Elkin-Koren & N. Netanel (Eds.), The commodification of information (pp. 433–457). The Hague: Kluwer Academic Press.

Nygaard, L. P., & Bellanova, R. (2017). 2 lost in quantification: Scholars and the politics of bibliometrics. In M. J. Curry & T. Lillis (Eds.), Global academic publishing (pp. 23–36). Multilingual Matters. https://doi.org/10.21832/9781783099245-007

Olssen, M., & Peters, M. A. (2005). Neoliberalism, higher education and the knowledge economy: From the free market to knowledge capitalism. Journal of Education Policy, 20(3), 313–345. https://doi.org/10.1080/02680930500108718

O’Neill, O. (2020). Trust and accountability in a digital age. Philosophy, 95(1), 3–17. https://doi.org/10.1017/S0031819119000457

Ortega, J. L., & Aguillo, I. F. (2012). Science is all in the eye of the beholder: Keyword maps in Google Scholar citations. Journal of the American Society for Information Science and Technology, 63(12), 2370–2377. https://doi.org/10.1002/asi.22761

Pendlebury, D. A., & Adams, J. (2012). Comments on a critique of the Thomson Reuters journal impact factor. Scientometrics, 92(2), 395–401. https://doi.org/10.1007/s11192-012-0689-6

Powers, S. M., & Jablonski, M. (2015). The real cyber war: The political economy of internet freedom. University of Illinois Press. https://doi.org/10.5406/illinois/9780252039126.001.0001

Quan, W., Chen, B., & Shu, F. (2017). Publish or impoverish: An investigation of the monetary reward system of science in China (1999-2016). Aslib Journal of Information Management, 69(5), 486–502. https://doi.org/10.1108/AJIM-01-2017-0014

Rached, D. H. (2016). The concept(s) of accountability: Form in search of substance. Leiden Journal of International Law, 29(2), 317–342. https://doi.org/10.1017/S0922156516000042

Rossner, M., Van Epps, H., & Hill, E. (2007). Show me the data. Journal of Experimental Medicine, 204(13), 3052–3053. https://doi.org/10.1084/jem.20072544

Rossner, M., Van Epps, H., & Hill, E. (2008). Irreproducible results: A response to Thomson Scientific. Journal of Experimental Medicine, 205(2), 260–261. https://doi.org/10.1084/jem.20080053

Rovira, C., Codina, L., Guerrero-Solé, F., & Lopezosa, C. (2019). Ranking by relevance and citation counts, a comparative study: Google Scholar, Microsoft Academic, WoS and Scopus. Future Internet, 11(9), 202. https://doi.org/10.3390/fi11090202

Rushforth, A., & de Rijcke, S. (2015). Accounting for impact? The journal impact factor and the making of biomedical research in the Netherlands. Minerva, 53(2), 117–139. https://doi.org/10.1007/s11024-015-9274-5

Russell, J. M., & Rousseau, R. (2001). Bibliometrics and institutional evaluation. Science and Technology Policy, 2.

Salter, B., & Tapper, T. (1994). State and higher education. Woburn Press.

Sayer, D. (2014, November 19). Time to abandon the gold standard? Peer review for the REF falls far short of internationally accepted standards. London School of Economics Impact of Social Sciences Blog. https://blogs.lse.ac.uk/impactofsocialsciences/2014/11/19/peer-review-metrics-ref-rank-hypocrisies-sayer/

Shaw, R. R. (1948). Royal Society Scientific Information Conference. The American Statistician, 2(4), 14–16, 23. https://doi.org/10.2307/2682685

Shore, C. (2010). Beyond the multiversity: Neoliberalism and the rise of the schizophrenic university. Social Anthropology, 18(1), 15–29. https://doi.org/10.1111/j.1469-8676.2009.00094.x

Shore, C., & Taitz, M. (2012). Who ‘owns’ the university? Institutional autonomy and academic freedom in an age of knowledge capitalism. Globalisation, Societies and Education, 10(2), 201–219. https://doi.org/10.1080/14767724.2012.677707

Shore, C., & Wright, S. (2000). Coercive accountability: The rise of audit culture in higher education. In M. Strathern (Ed.), Audit cultures: Anthropological studies in accountability, ethics and the academy (pp. 57–89). Routledge.

Smith, S., Ward, V., & House, A. (2011). ‘Impact’ in the proposals for the UK’s Research Excellence Framework: Shifting the boundaries of academic autonomy. Research Policy, 40(10), 1369–1379. https://doi.org/10.1016/j.respol.2011.05.026

Speer, R., & Signers. (2021). Google Scholar has failed us. Google Scholar is a trans-exclusionary site. Don’t use it. Help us demand change [Petition]. scholar.hasfailed.us. https://scholar.hasfailed.us/

Strathern, M. (2000). The tyranny of transparency. British Educational Research Journal, 26(3), 309–321. https://doi.org/10.1080/713651562

Tanczer, L. M., Deibert, R. J., Bigo, D., Franklin, M. I., Melgaço, L., Lyon, D., Kazansky, B., & Milan, S. (2019). Online surveillance, censorship, and encryption in academia. International Studies Perspectives, 21(1), 1–36. https://doi.org/10.1093/isp/ekz016

Tay, A., Martín-Martín, A., & Hug, S. E. (2021, May 27). Goodbye, Microsoft Academic – Hello, open research infrastructure? London School of Economics Impact of Social Sciences Blog. https://blogs.lse.ac.uk/impactofsocialsciences/2021/05/27/goodbye-microsoft-academic-hello-open-research-infrastructure/

Terdiman, D. (2004, November 25). A tool for scholars who like to dig deep. The New York Times. https://www.nytimes.com/2004/11/25/technology/a-tool-for-scholars-who-like-to-dig-deep.html

Traweek, S. (1992). Beamtimes and lifetimes: The world of high energy physicists. Harvard University Press.

Vaidhyanathan, S. (2011). The googlization of everything (and why we should worry). University of California Press.

van Dijck, J. (2010). Search engines and the production of academic knowledge. International Journal of Cultural Studies, 13(6), 574–592. https://doi.org/10.1177/1367877910376582

van Noorden, R. (2010). Metrics: A profusion of measures. Nature, 465(7300), 864–866. https://doi.org/10.1038/465864a

van Noorden, R. (2014, November 7). Google Scholar pioneer on search engine’s future. Nature News. https://doi.org/10.1038/nature.2014.16269

van Noorden, R. (2021, March 31). Do you obey public-access mandates? Google Scholar is watching. Nature News. https://doi.org/10.1038/d41586-021-00873-8

Veletsianos, G., Houlden, S., Hodson, J., & Gosse, C. (2018). Women scholars’ experiences with online harassment and abuse: Self-protection, resistance, acceptance, and self-blame. New Media & Society, 20(12), 4689–4708. https://doi.org/10.1177/1461444818781324

Viljoen, S., Goldenfein, J., & McGuigan, L. (2021). Design choices: Mechanism design and platform capitalism. Big Data & Society, 8(2). https://doi.org/10.1177/20539517211034312

Whitley, R. (1984). The intellectual and social organization of the sciences. Oxford University Press.

Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J., & Johnson, B. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management [Report]. https://doi.org/10.13140/RG.2.1.4929.1363

Footnotes

1. We thank Marcel Wrzesinski for also suggesting descriptive practices like registration, certification and preservation. We argue that those practices are enrolled in the above normative functions, to aid in maintaining or securing priority and dissemination. As well, the forces and affordances at play in scholarly communication practices outside publishing, such as the joking, gossip and other conversation in seminars and conference hallways, or in backrooms and on social media, also have broad consequences on the normative functions (Traweek, 1992; Veletsianos et al., 2018).

2. See Acharya (2015) for a presentation on indexing from a Google Scholar co-founder, largely about how publishers can make themselves ‘algorithmically recognizable’ (Gillespie, 2014).

3. In 2012: “We have built the largest scholarly search. At this point it includes every source that I can reasonably think of, and some sources may be borderline scholarly, but that is the nature of trying to do everything” (Acharya, 2012). In 2014: “'Scholarly' is what everybody else in the scholarly field considers scholarly. It sounds like a recursive definition but it does settle down. We crawl the whole web, and for a new blog, for example, you see what the connections are to the rest of scholarship that you already know about. If many people cite it, or if it cites many people, it is probably scholarly. There is no one magic formula: you bring evidence to bear from many features” (van Noorden, 2014). In 2015: “The scope of scholar as yet, and for the foreseeable future, is articles, text descriptions of research that is done by researchers and written up for other researchers” (Acharya, 2015).

4. Some advanced Boolean operators like wildcards and proximity searching are also reported to work on Google Scholar (see guides provided by university libraries), but they are not published on the tool’s “Advanced search” popup or “Search tips” page accessible via “Help” at the bottom of the screen.

5. Relevance could include factors similar to those in general-purpose web search (discussed but not explored by Rovira et al., 2019), such as the number and nature of backlinks (distinct from citations) or clicks (perhaps even incorporating clicks from general-purpose web search for items in Google Scholar). Scopus, for instance, does not use "citedness" or date, but reports to consider factors like the number of hits, significance of a term and the location of the query terms in relation to each other, as well as their position in the document. Rather than emphasising a notion of universal or algorithmically-personalised relevance, bibliographic databases and other academic search engines generally provide searchers with several ways to filter or sort results, as mentioned (in those cases, while defaults still matter, the individual searchers are themselves to navigate the ‘friction of relevance’ in the tension between competing individual and societal interests (Haider & Sundin, 2019)).

6. Tay et al., (2021) argue that Microsoft never intended to run a scholarly information service or business, but instead used Microsoft Academic to test various big data and machine learning technologies, including the capacity to extract information from word documents. Microsoft Academic shared many features with Google Scholar, but provided an API and was highly transparent.

7. See e.g. Gillula & Cope (2016) regarding a US Federal Trade Commission complaint about a similar issue with K-12 student privacy when students used Google services outside of the Google Apps for Education or G Suite for Education agreements; relatedly, see e.g. Lindh & Nolin (2016) for an exploration of the difficulty of wading through Google’s policy rhetoric to understand Google’s “back end” strategies around Google Apps for Education (the previous name of Google’s now G Suite for Education).

8. There is work looking at country-level differences in the use (and institutional requirements of use) of the h-index between social scientists in Sweden and Australia (Haddow & Hammarfelt, 2019) and the perceived importance and knowledge of the index across a range of disciplines in Germany (Kamrani et al., 2021). The latter shows a greater perceived importance and knowledge in the physical sciences and medicine (in comparison to social sciences, humanities and law) with younger researchers perceiving a greater importance and having greater knowledge of the h-index (and they preferred the h-index on Google Scholar rather than the Web of Science or Scopus).

9. A close look at ResearchGate’s data sharing arrangements demonstrates that it “shares”, or rather sells, data collected through use of the platform to almost 500 external participants, see Keusch and Kreuter (2021). Most of these are commercial entities and participants in the “ad-tech” ecosystem, including Google.

10. This is telegraphed in the subtle language shift on the Google Scholar help page for publisher support which states: ‘We will not share [publisher electronic holdings information] with third parties or use it for marketing purposes’ [emphasis added], whereas a constraint on marketing purposes is absent in the next section regarding ‘the usage of your electronic holdings’.

11. In June 2010 (the earliest Internet Archive capture of the page), Google Scholar said it could make such updates in “anywhere from a few days to 3-6 months”. In August 2021, Google Scholar’s ‘Inclusion Troubleshooting’ page states that “updates of papers that are already included usually take 6-9 months. Updates of papers on very large websites may take several years”.

12. Chen, Posada, & Chan (2019) documented Elsevier’s strategy of horizontal (acquiring otherjournal titles) and vertical integration, creating an end-to-end platform that provides servicesfrom journal submission and data archiving, to peer review and journal publication, to subsequentdissemination, metric tracking and predictive analytics.

13. Dupuis (2016) argues this as an effort to maintain hegemony in an open environment. Others like Fister (2016) have argued that it is not about owning the work “process”, but rather about owning the infrastructure of value capture.

14. HeinOnline has also been working on its scholar profile pages. By searching an author in the Hein dialogue box, you receive a profile page similar to Google Scholar that includes published articles, as well as statistics including the number of citations for each article, number of times an article is accessed on HeinOnline (i.e. view and download like SSRN), some time-based citation counts and a count of self-citations. HeinOnline also allows the downloading of CSV files of an entire institution’s scholar metrics, presumably for other forms of institutional evaluation. Like Google Scholar, author profiles are automatically generated if an author has an article on HeinOnline, but unlike GS, they request certain information so that a human staff member can complete the profile page. US News & World who has an influential system of law school rankings, has been systematically expanding data collection relevant to its ranking project. They analyse a law school’s scholarly impact according to systems of indicators measuring productivity and impact, including publications and citations. They have accordingly commenced a collaboration with HeinOnline to gather that data (although only from a specific group of approximately 2600 legal periodicals). They are specifically looking at mean citations per faculty member, median citations per faculty member and total number of publications. See Morse (2019).

15. See e.g. de Búrca & Weiler (2020) with respect to recent debates about the European Law Journal and the publisher’s effort to determine the editorial staff, which led to the resignation of the entire editorial committee.

Add new comment