Identifying potential emerging human rights implications in Chinese smart cities via machine-learning aided patent analysis

Introduction

Cities, and the interactions between their infrastructure and their populations, will be fundamental to the human experience in the future as greater and greater proportions of the global population become urban dwellers. By 2050 more than two thirds of humanity will live in urban areas (United Nations Department of Economic and Social Affairs, 2018).

Cities are undergoing a rapid process of digital transformation, many of them involving advanced systems of surveillance and internet control such as facial recognition. Roughly half of the 1,000 smart city initiatives underway are in China (Atha et al., 2020). Chinese cities are becoming synonymous with massive surveillance and rising ‘digital authoritarianism’ (Dragu & Lupu, 2021; Weber, 2019). China is also actively exporting mass surveillance technologies: AI-enabled ‘Safe Cities’ are being sold to over 75 countries globally (Feldstein, 2021).¹ Chinese companies supply AI surveillance technology to over 63 countries, 36 of which have signed up to China’s Belt and Road initiative (Feldstein, 2021).²

The spread of these domestic and exported systems endangers a host of human rights. Predictive policing, or ‘Intelligence-Led Policing’ (情报指导警务) as the model is known in the UK (Schwarck, 2018), is a core feature of smart cities, used by police to identify areas in a city that need additional police attention (expecting future crimes) based on a database of past crimes (Kempin Reuter, 2020). However, predictive policing exacerbates targeting of citizens based on ethnicity. It relies on algorithms that are inherently racially biassed (Kempin Reuter, 2020). The wider material and digital architectures of these systems have a considerable human rights impact too, as will be illustrated in more detail below (Galdon-Clavell, 2013; Williams, 2020; Weber & Ververis, 2021; Kempin Reuter, 2020).

Since IBM brought the concept of smart cities to China in 2008 the concept has been strongly embraced by the Chinese Communist Party, public security organs and industry (Xinhuanet, 2020). Hundreds of smart cities have emerged including in major cities such as Hangzhou and Shanghai (Hangzhou City Government, 2021; Shanghai City Government, 2020). Despite Chinese-developed smart cities having an impact on the lives of hundreds of millions of people within and beyond China, few non-Chinese language resources exist on how Chinese smart cities are built and what their impacts are (Chin & Lin, 2022; Walton & Muggah, 2021; Weber & Ververis, 2021).

A smart city is commonly defined as the deployment of technologies within an urban setting to overcome city specific challenges (Green, 2019; Halegoua, 2020a, 2020b; Lorinc, 2022; Townsend, 2014). This paper investigates the smart city infrastructure that is used by law enforcement to monitor citizens, such as cameras, dedicated cloud computing infrastructure and police command headquarters that are integrated into the urban surveillance infrastructure.

We make use of patent repositories to reach a better understanding of the public security aspects of smart cities, using data scraping and machine learning to process our initial list of 67,130 patents, which was reduced to a final list of 5,989 patents that were identified, with high probability, as directly related to smart cities, and that were filed in China. The process of scraping and machine learning is refined and analysed for quality purposes by a ‘human in the loop’ to remove false positive search results caused by colliding terms that match different search topics (Hinsley et al., 2023).

Based on a conceptualisation frequently cited in China, and in particular by Huawei, this work defines the major elements of a Chinese smart city as consisting of a “body-like structure” (China National Maritime Technology Co. Ltd., 2018; Weber & Ververis, 2021). Huawei, for instance, aims to create a digital twin of cities and conceptualises the smart city as if it were a digital body (Dong, 2018; Bayesteh et al., 2022). In this conceptualisation of a smart city the Internet of Things (IoT) sensors, such as cameras with microphones, represent the “eyes and ears'' of a city. Fibre optic cables in a smart city are akin to the neural networks of a body. What would, in a physical body, be considered as the brain is seen in the smart city context as a ‘city brain’. This concept of a ‘city brain’, much as the brain in a physical body, is arguably the most important individual component of a smart city, crucial to coordination and control of the overall organism (see next section). The city brain comprises a city’s physical police command centre that uses cloud computing infrastructure and algorithms to interpret the data gathered by cameras and other IoT devices throughout the city. This interpretation of data in the city brain allows police to take data-informed actions (iFlytek, n.d.).

The remainder of this paper is structured as follows: we first examine smart cities in the context of existing literature of Western sources and compare this with sources from the People’s Republic of China. We summarise the origins of the ‘city brain’ framework in the PRC literature and account for the increasing prevalence of the concept in research.

We continue by laying out our patent analysis methodology with sections on filing frequency changepoint analysis and topic modelling. The key finding from the changepoint analysis is the identification of a key shift in the rate of patent filing for smart cities in China that occurred in early 2015 (Figure 1). This correlates directly with a major mention of smart cities in the 13th Five Year Plan of the China Communist Party, presented in March 2015. The range of topics identified by the topic model show that a significant aspect of smart city innovation focuses on the development and expansion of sensing, monitoring, and – crucially – analytics of the data of citizens. The emergence of these topics as areas of significant commercial interest in China represent a potentially concerning view into the future development of such technologies.

Finally, we assess the implications of smart city patenting for human rights. The European Commission defines smart cities as “a place where traditional networks and services are made more efficient with the use of digital solutions for the benefit of its inhabitants and business” (European Commission, n.d.). As will be shown in the analysis of this paper, Chinese smart cities may not always benefit its inhabitants and are sometimes used to specifically repress parts of the population (Walton & Muggah, 2021).

Context: smart cities and the city brain

At its most fundamental level the smart city relies on sensors that are spread throughout the city (China National Maritime Technology Co. Ltd., 2018). While the human body has 5 senses – sight, hearing, touch, smell and taste – smart city technologies to date have primarily focused on visual sensing and analysis (Dong, 2018; Bayesteh et al.,2022). Within the smart city, the interconnected network transmission layer – consisting of physical cables, mobile networks, bluetooth and Wi-Fi infrastructure – transmit information from images of people and their physical location to a centralised storage and analysis hub: the city brain (Dong, 2018; Bayesteh et al.,2022).

The city brain is an integral part of a smart city in its conceptualisation within the PRC (Liu, 2021). The city brain uses artificial intelligence to analyse data (Alibaba, n.d.): for instance, automatically flagging alerts of traffic violations or of people that leave geo-fenced areas. Due to the centrality of the city brain to the public security dimension of the smart city, we use the following lines to this concept.

The concept of city brain has been endorsed by the highest echelons of the Chinese party state. It has been described as "...a powerful tool for reform - the "brain for governance" is a winner" (三、改革的利器——“治国理政大脑”应运而胜) (Guangming Net, 2021). Even General Secretary Xi Jinping visited the Hangzhou City Brain Operation Command Center and mentioned that cloud computing and big data make cities smarter and should be used in governance in order to transition from a digitised network, to an intelligent network and finally to a network full of wisdom. Xi Jinping's remarks at the Hangzhou City Brain Operation Command Centre (2020), echo the application (and modification) to smart cities (lit. 'wisdom cities') of the classic DIKW hierarchy (data, information, knowledge, wisdom) (Gu, 2013.)

A 2013 paper by academic Gu Jifa situates this data hierarchy in a Chinese smart city context and in particular with respect to Qian Xuesen who has theorised the relationships “between data, information, knowledge, wisdom and [his] new theory on meta-synthesis of wisdom” (Gu, 2013, p. 714). Xi Jinping’s comments need to be seen in the context of Qian Xuesen’s writing, a former rocket scientist who remains highly regarded within China. In the early 1990s Qian proposed what he claimed was a “new discipline of science”, drawing on his work since the 1950s (Tsien,1954) on control systems and “cybernetics” (控制论), “information theory” (信息论), and “systems theory” (系统论的合称), he posited a theory for understanding complex systems as ‘Open Complex Giant Systems’ and presented a methodology for complex systems engineering he termed “meta-synthetic engineering” - or Dacheng Wisdom (综合集成法) (Xuesen et al., 1993). Qian’s theory still exerts significant influence over policy and planning (King, 2022), including at the highest levels of the party-state (Lee, 2017) and particularly among engineers of AI-driven urban social management systems (Song et al., 2014).

In practical terms, the goal of the city brain is to create a digital twin of a real-life city, which means that each element of a city has a digital Doppelgänger (iFlytek, 2023). Police forces are visualised on a map, so are pedestrians or vehicles. At the core of the city brain's policing features is the “comprehensive management platform for public security supervision” (公安监管综合管理平台) which integrates data from the city, including from prisons (Newings, n.d.). This platform allows the police to command and dispatch its force to areas that have for instance flagged an incident that needs to be investigated. In Liaocheng the city brain includes integrated command and dispatch (融合指挥调度) (iFlytek, n.d.).

Many companies have worked towards developing city brain technologies. IFlytek for instance calls its product the Urban Super Brain (城市超脑) (iFlytek, 2023). While in China the idea of the internet as a brain dates back to at least 2008, the idea of the city brain is more recent (Liu, 2022). The name “city brain” was devised in 2016 Wang Jian (王坚) Chief Technology Officer of Alibaba. Alibaba is also one of the most prominent companies developing such technologies (Liu, 2021).

In 2016 the Hangzhou city brain was at first targeted towards reducing traffic congestion (Hangzhou Municipal People’s Government, 2023). Hangzhou was the Chinese city with the second worst traffic (Zhejiang Provincial People’s Government, 2021). In 2018, a comprehensive version of the city brain became operational. It did not only have traffic control capabilities. Its remit was extended to a city’s e-governance, digitising various industries, allowing citizens to find a parking space through the City Brain app (Zhejiang Provincial People’s Government, 2021; Chin & Lin, 2022).

Alibaba’s city brain product has been not only deployed in Hangzhou, Suzhou, Shanghai, Macau and Malaysia (Alibaba, n.d.). Within China, the city brain is present in hundreds of cities (Liu, 2022). It largely focuses on analysing pedestrians, vehicles and is according to the company "one of the largest public artificial intelligence systems worldwide” (Alibaba, n.d.). Examining the company’s marketing materials it becomes clear that the city brain is also about the speed of processing video material. The system can process 16 hours of footage in one minute and consists of a video access system, streaming platform and a video analysis feature. Alerts are being pushed by the system to the operators. The brain can also predict the flow of vehicles and pedestrians by drawing from historical and real-time data. In Tianying, Jieshou it takes only a second or two to find missing persons or people that are on the run. In Quzhou, Zhejiang the city brain is at the core of the Sharp Eyes project, a major and controversial surveillance project of the Chinese Communist Party, which aims to monitor 100% of public space through CCTV cameras (China Government Procurement Network, 2020).

The difference between smart cities and city brains is in practical terms often not that large. Chinese experts in this field deplore that they are often similar, but emphasise that the city brain is an evolution of the traditional smart cities (Liu, 2022). The same experts argue that often cities are connected and generate a lot of data but do not produce enough actionable insights. A smart city that contains elements that are brain-like can create more insights according to those scientists. One may argue that the city brain is just a rebranding effort of the smart cities concept but detailed conceptual interpretations of experts and associated city-brain technical standards suggest that the city brain is indeed an evolution of the traditional smart city concept that is still predominant in Western countries (Liu, 2022). The smart city concept present in Western countries is still very fragmented, where companies provide individual solutions, e.g. traffic management, smart street lighting (Thales, n.d.; Nokia, n.d.; Airbus, n.d.). In turn the city brain concept aims to connect and integrate everything into the city brain headquarters where insight is gained (where thinking occurs). In cities like Hangzhou the city brain has gotten very close to this goal (Chin & Lin, 2022).

In sum, the Chinese concept of city brain is similar to what has been captured in the broader smart city literature (Laufs et al., 2020; Kitchin et al., 2015; Mattern, 2015) . It combines sensors that are spread throughout the city, a usage for various purposes including smart city policing and a deployment that requires the development of digital twins (digital representations of physical objects) and a command centre where city data is visualised for the human operator. The Chinese city brain concept is unique however because it puts the deployment of artificial intelligence and big data integration at the core of city management (iFlytek, n.d.).

Methodology

To assess the growth and development of commercial interest in smart city technologies, and in particular to identify emerging topics and concepts, we make use of large scale analysis of patent filing data. Patents represent the leading edge of technological development by corporations seeking to dominate and drive an industry, responding to perceived or identified needs in the marketplace. As patent filing is necessarily public (TRIPS Art. 29, 1994), patent data provides a convenient and open data source by which to study emerging trends in a range of industries as well as the key concerns that are present in the text of existing patent filing behaviour.

To understand the landscape of patenting concerning smart cities, we scrape public patent data via the Google Patents search engine and apply both a time series frequency analysis to identify key changepoints in the rate of patent filing, representing interest in particular innovations; as well as a topical analysis of the contents of patents. This approach draws on previous work performing large-scale machine-assisted assessment of developments in technological innovation in the wildlife trade (Hinsley et al., 2023). Topic modelling has also recently been applied to the patent filings of Palantir Technologies, a notable actor in the surveillance industry (Iliadis & Acker, 2022), as well as more generally in attempts to classify patents automatically and to determine the growth rates of particular classes of patents (Kim & Kim, 2022; Yun & Geum, 2020; Suominen et al., 2017). Our work differs from these in that it focuses on changepoints in patenting rates, as well as the emergence of new terms and topics, but has similarities in its use of topic modelling to assess and group patents via automatic means.

We make use of the Google Patents search engine (Google, n.d.) to conduct full-text keyword-based search across both the text and metadata of patents. Google patents allows search across the full automatically-translated text of the patent, however in order to ensure accurate matching we conducted searches with both English and Chinese terms. The search terms used for the analysis were identified by the authors through literature and policy surveys of Chinese smart city and technology documents. The full list of terms used in the work presented here is given in Table 1.

From a full-text search, Google Patents returns a listing of patents that match the identified terms. We applied the open source browser automation library selenium, and specifically its R interface rselenium to obtain a full list of patents matching the identified search terms. Example initial search terms include ‘smart city’ and its alternative translation of the Chinese term ‘wisdom city’, as well as more specific search terms related to aspects of smart city development identified via the above theoretical study of smart cities.

Table 1: Full list of search terms for ‘smart cities’ patents

The full list of patents provided by the Google Patents search engine is then used to scrape patent text directly using the R rvest library. This results in a dataset of all filed patents matching the provided search terms in both Chinese and English, including the names of the filing entities, patent authors, the dates of first filing of a patent amongst potential revisions and related patents (the patent ‘priority date’), the text of the patent abstract and the full text description of the associated invention. For our initial set of search terms the initial downloaded dataset comprised 67,130 patents across 14 search terms, which after filtering, cleaning and deduplication was reduced to a final count of 5,989 patents for smart cities filed in China.

Following initial scraping, the patent data is subjected to filtering and processing to remove duplicate patents and irrelevant confounding search terms. Due to the nature of the data it is difficult a priori to ensure that search terms do not produce spurious irrelevant results, and manual filtering to validate the accuracy of the downloaded data is required. In order to improve the accuracy of the search, we engaged in an iterative process of sampling the list of downloaded patents to identify false positive search results and identifying confounding terms that resulted in unrelated patents being included in the list.

Following the process of manual inspection and filtering, we subject the patents to two key stages of analysis before further iterating on the results of these. As a result of our manual filtering and cleaning process, our initial list of 67,130 patents was reduced to a final list of 5,989 patents that were identified, with high probability, as directly related to smart cities, and which were listed as filed in China. We retained data concerning global, non-Chinese filed, patents for smart cities but focused on the specific development of Chinese smart city patents in this work.

Filing frequency changepoint analysis

We are particularly interested in the rate at which patents for a given technology are filed, representing commercial interest in the topics of a patent: an increased patenting rate represents the identification of a niche in the market for such a technology from commercial entities. Of particular interest to our analysis was whether there was any correlation between patenting rates and policy pronouncements, potentially indicating policy as a driver for commercial activity. More specifically, we are interested in moments at which a changepoint occurs in patent filing, representing a statistical shift in the underlying rate at which corporations file patents related to a concept such as smart cities.

We conduct a Bayesian changepoint analysis (Stan Development Team, 2022) on the rate of monthly patent filing for a given set of patent search results. To do so, we model the rate of monthly patents as drawn from a probability distribution – specifically, in this model, a negative binomial distribution, allowing for flexible description of count data. To determine the existence and location of a significant shift in this patenting rate we assess whether the data can more effectively be described by two separate models: one falling before some point in time, and one falling after. By identifying the pair of models that most effectively capture the behaviour of the overall time series, and the moment in time at which the series changes from one model to the other, we identify the highest probability moments at which the patenting rate shifted.

We fit the changepoint model using the Stan probabilistic programming language (Stan Development Team, 2022) through its R interface cmdstanr, making use of standard approaches to ensure model convergence according to a principled Bayesian workflow (Gelman et al., 2020). The Bayesian changepoint analysis provides not only a point estimate of the changepoint but a probability distribution across potential changepoints, allowing us to assess the uncertainty in the models. The approach taken here focuses on a single changepoint, however a more detailed analysis would allow us to segment the data set at identified changepoints and thus identify less significant points of change.

For this work, however, we focus on identifying a single main changepoint in the patent filing data to allow the inspection of major changes in patent filing behaviour alongside key policy decisions, identified via the same literature and policy review that produced the search keywords. This approach does not draw causative inferences between the observed patent filing behaviour and exogenous events but presents a useful comparison between shifts in technological filing and their surrounding social, technological and political context.

Topic modelling

The changepoint analysis serves to identify the timing of key shifts in patent behaviour but does not interrogate the content of the patents. Latent Dirichlet Allocation (LDA) topic modelling (Blei et al., 2003) is a widely-used natural language processing technique applied to identify commonly co-occurring ‘themes’ in a large corpus of data. Topic modelling treats each document in a corpus as resulting from an underlying probability distribution of words and splits the corpus into a defined number of separate ‘topic’ probability distributions. Each document can then be classified according to the given topic distribution that best describes its content.

The result of applying this approach is that a large corpus of documents can be automatically classified into sets of related documents according to the key terms that appear in them. The characteristic terms for each topic can then be identified to allow human-identified semantic understanding of the classifications. In this sense, topic modelling as an unsupervised machine learning technique is best considered an automated aid allowing large-scale human guided classification and understanding of large corpora of documents. In particular, we use structural topic modelling (STM) (Roberts et al., 2013) as implemented in the stm package for R.

For the purposes of this analysis, topic modelling allows us not only to classify related documents together but to identify key themes and terms that run through the patent filing data set and to determine whether certain terms are emerging, growing or falling in popularity.

Despite initially carrying out a topic model on the full text of the downloaded patents, this resulted in a substantial proportion of identified topics related to technical details of implementation that were unrelated to the core concept of the patents in question. As such, we chose to focus on a topic model of patent abstracts, in which the patent filers focus on the nature of the innovation itself rather than the implementation.

We fitted topic models to the entire corpus of 5,989 patent abstracts returned in our search for ‘smart city’ related keywords. As a semi-supervised learning approach, topic modelling typically requires manual selection of the number of topics into which to categorise the documents in the corpus. Following Roberts et al. (2014), we select an optimal number of topics through balancing the semantic coherence of the topic model against its exclusivity, aiming to split the documents into a minimal number of topics whilst preserving each topic’s focus on genuinely related concepts. This procedure resulted in a set of 20 topics, reproduced in full in Appendix A.

Findings

We have provided a short overview of smart cities and highlighted the concept of the city brain, since it is so central to the operation of smart cities today. In the next section, we show how patents evolved with respect to smart cities and the city brain. Our patent searches, and changepoint analyses and topic modelling, revolved around a set of related terms for both “smart cities” and “wisdom cities”, two terms that are used extensively within the Chinese-language literature for smart cities.

The key finding from the changepoint analysis was the identification of a key shift in the rate of patent filing for smart cities in China that occurred in early 2015 (Figure 1). Interestingly, this correlates directly with a major mention of smart cities in the 13th Five Year Plan of the China Communist Party, presented in March 2015:

We will build smart cities as we strengthen modern information infrastructure and promote the development of big data and the Internet of Things. We will build innovative cities by making full use of the concentrations of creative resources found in cities to develop business parks and cradles of innovation (Central Committee of the Communist Party of China, 2015, p. 96).

Whilst – as noted previously – the term ‘smart city’ was introduced into Chinese public discourse as early as 2008, the 12th Five Year Plan, for instance, does not mention the term. As such, the correlation between the 13th Five Year Plan and the beginnings of an ongoing surge in patents highlights a relationship between a sudden, rapid increase in commercial interest and the recognition of the importance of such a concept at the highest levels of governmental policy.

While the above analysis of patent filing rates provides an overall assessment of the relative interest of Chinese patent-filing entities in the overall concept of smart cities, it makes no attempt to analyse the specific content of the filed patents. To give further insight into the patents themselves, we apply a structural topic model to characterise both the overall topics and to identify key themes for further investigation.

Of the 20 topics in our overall topic model of smart city patents, Table 2 identifies 10 topics with greater or lesser relevance to surveillance, sensing, tracking and analytics of human behaviour. The first topic in Table 2, in particular, which we label ‘Deviance’ focuses on the identification and tracking of deviant behaviour, as characterised by terms such as abnormal, illegal, violation, and behavior. Other related topics include those that focus on monitoring (alarm, monitor, central, emergency, scanning, personal) and surveillance (analyzes, acquisition, mass, visual). Another important set of patents are those that concern the analysis of data and which relates, in this conceptualisation, to the “city brain” concept. This is primarily inducted from the terms and topics.

Table 2: Topic model of ‘smart city’ patent search results, with human-interpreted topic titles, restricted to surveillance-related topics

The range of topics identified by the topic model show that a significant aspect of smart city innovation focuses on the development and expansion of sensing, monitoring and — crucially — analytics of the data of citizens.

The topic model, however, is not a static assessment of patent filing, but allows us insight into the growth and development of emerging themes and topics in patent filing. Restricting the topic model only to those topics with particular relevance to surveillance, Figure 2 shows the evolution of patent filing behaviour and the emergence of new technologies.

In Figure 2 perhaps the most striking shift is the clear increase in the relative rate of patents filed under the ‘Intelligence’ topic, since the middle of 2020. This topic, which focuses particularly on the applications of artificial intelligence, is clearly emerging as an area of significant innovation in the smart cities landscape. Less dramatically, the ‘Deviance’ topic, which focuses on the detection of abnormal or illegal behaviour, appears to be the focus of significant attention. We suggest that the emergence of these topics as areas of significant commercial interest, i.e. companies investing in patenting in China, represent a potentially concerning view into the future development of such technologies.

Human rights implications of smart city patenting

Smart city technologies create risks to human rights both in the West and in China. The following section does not provide a comprehensive and in-depth review of the human rights literature on these topics. For broader engagements with the literature other articles and works can be consulted (Brown, 2019; Cammers-Goodwin, 2019; Mullick & Patnaick, 2022; DeNardis, 2022; Galič & Schuilenburg, 2021; Flak & Hofmann, 2020; Almeida et al., 2018). In the West there are concerns that ubiquitous sensors could exert a chilling effect on citizens that would avoid going to protests if they feared punishment (Williams, 2020). Similarly, racial discrimination that is exacerbated by facial recognition cameras is a major concern (Goujard, 2022).

Cardullo et al. examine the ethical concerns that come with smart cities and categorise them into e.g geosurveillance, privacy, dataveillance, nudging, profiling and social sorting (Cardullo et al., 2019). Chang goes into more depth and elaborates that the right to privacy can be undermined through “information collection, information processing and information dissemination or invasion (Chang, 2021, p. 8).

A smart city initiative that uses GPS tracking for crisis management impacts security positively (Article 3 of the Universal Declaration of Human Rights) but may affect privacy negatively (Article 12 of the Universal Declaration of Human Rights). The authors thereby show that technologies can have simultaneously a positive and negative impact on fundamental human rights (United Nations, 1948).

Examining human rights in Chinese smart cities first requires the recognition that smart city technologies are always embedded in a specific political constellation. Akbari demonstrates that in Iran for instance traffic cameras are being used to crack down on women that do not wear a hijab (Akbari, 2022). In Afghanistan, the Taliban used US-produced technology to identify people that have worked with the Western coalition in the past decades (Schoemaker, 2021). China, in turn, used its surveillance infrastructure to implement its zero COVID policy (Akbari, 2022). In short, smart cities are not apolitical. They are a product of the political realities in a certain geographic location.

Chinese smart cities in particular stand in stark contrast to Western smart cities concepts and deployments. As shown above, the city brain concept was endorsed on a national level by the highest echelons of power. Much effort is dedicated to integrating data from different cities. In the West, smart city deployments are much more a piecemeal effort and vary by the municipalities that are implementing them (Airbus, n.d., Nokia, n.d., Thales, n.d; Walton & Weber, 2023). Nokia for instance works on increasing 5G connectivity in several French cities (Nokia, n.d.). Airbus for its part focuses on smart city airborne mobility. In other words, there are many isolated smart projects within municipalities but little goes beyond that (Airbus, n.d.). Dourish (2016) has dubbed the western trend whereby disparate and uncoordinated municipal ICT projects are presented as a unified vision the ‘accidental smart city’ (Coletta et al., 2019).

In comparison to China, Europe has strong constitutional safeguards that protect fundamental rights (Geller, 2022). These rights protections also apply to smart cities, which is reflected in the European legal literature that lays out the legal context that grants protection for fundamental rights (Hacker & Neyer, 2023; Lane, 2023; Mobilio, 2023; Wernick et al., 2023).

As we have showcased above in the smart city topic model, we argue that the pervasive, ubiquitous, geographically constrained and physical nature of smart city surveillance makes it at least as dangerous as more traditional online surveillance, which can only touch on the virtual elements of a citizen’s activities. Previously citizens could turn off their phone or leave their computer at home to escape surveillance. Now such evasion is illusory (Weber & Ververis, 2021). DeNardis argues similarly – “[t]he data-collection practices structurally underlying material control structures, as well as the firmly established revenue models that rely on pervasive data collection, mean that society is approaching a decision point about whether any possibility for a private sphere remains possible” (DeNardis, 2020, p. 184).

Perhaps the most concerning element of these developing technologies is the increasingly “intelligent” nature of cyber-physical cities. We have tracked the emergence of the term “city brain”, designating the physical and logical location where data in cities is being fused and analysed. With ever increasing processing power of computers, and growth in data centres, the capacity of authorities to sense, store, process and act on surveillance data is accelerating.

In the next paragraphs we are delving deeper into the Chinese authoritarian imaginary of a smart city and impacts on human rights (Akbari, 2022). The topic model of smart cities in this paper identifies elements of the smart city infrastructure in which surveillance is a particular concern. Beyond the overtly public security applications discussed above, the topic model also showed that a significant aspect of PRC (People’s Republic of China) smart cities and research clusters around is in construction, energy and healthcare and gathering data about the health of citizens. As seen during the covid pandemic, health codes (green, yellow, red) can be crucial in controlling people’s movements (Mozur et al., 2020). Another batch of surveillance terms clusters around the movement of vehicles, traffic, streets, parking and pedestrians. Here the idea is to create digital twins of cars and the movement of people to know at all times where citizens are located. A third group of terms revolves around buildings and energy. While the patent data cannot show whether the patented technology is put into use – indeed, by some estimates as of 2014, of the 2.1m active patents at the time only 5% were licensed or commercialised (Fisher & Walker, 2014) – a wider uptake of these technologies would suggest exacerbation of human rights risks associated with smart cities. This could be done for instance, through Huawei’s eSight platform which can detect if people cluster inside buildings and determine the location of people within buildings (Weber & Ververis, 2021). The idea of extending surveillance inside buildings is captured in the term “intelligent buildings” which has been used in China for quite some time before the term smart cities appeared and designated sensors being spread throughout buildings to gather data (Walton, 2001). The energy consumption of buildings is also crucial in government surveillance. As seen in Xinjiang, a rise in consumption could correlate with suspicious behaviour and is therefore subject to constant surveillance (Human Rights Watch, 2019).

Monitoring is now spread throughout the city. In 2001, Hikvision, now the largest CCTV camera equipment manufacturer, was founded. In the decades since, CCTV cameras were installed visibly on almost every street corner. While the CCTV system is still fundamental to China’s surveillance state, since 2001 invisible sensors have been installed throughout cities (Walton, 2001; Weber & Ververis, 2021). This shift from visible to covert surveillance has raised concerns that citizens can no longer gauge when they are being monitored. With the addition of data sharing by manufacturers, such as that shared by vehicles with authorities or smart metering of electricity usage, pervasive and largely invisible surveillance is becoming increasingly ubiquitous.

The term smart cities was introduced to China by IBM in 2008 (Xinhuanet, 2020). As our analysis of patent filings demonstrates however, smart cities only began to receive high levels of commercial interests in innovation around 2015. This suggests that human rights defenders should pay close attention initially to high-level policy documents, such as the Five Year Plans, to identify in advance a top down push for certain surveillance systems, such as smart cities. Despite the lag from policy to commercial interest, the subsequent growth in patents surrounding the terms smart and wisdom cities has been worryingly rapid. While not all patents see implementation, commercialisation or adoption, the rapid pace of innovation in smart cities and the clear trend for the invention of technologies with significant negative implications for human rights presents a concerning perspective for the future of human rights in China.

Conclusion

In this work we have analysed developments in commercial innovation for smart cities in the particular context of Chinese policy and implementation.Through an analysis of patent filing, we have found a number of emerging trends in the underlying concepts that have been identified by commercial entities as profitable niches to exploit in the emerging technological landscape. The rise of patents related to, amongst others, artificial intelligence and the detection of abnormal or deviant behaviour in recent years highlights the potential implications for human rights in the future development and adoption of smart cities.

Through comparison with Chinese policy documents and pronouncements, we have also identified that the introduction of the term “smart city” at the highest of policy levels in China was followed almost immediately by a notable rise in the appearance of similar terms in commercial patent filing, despite the term having emerged globally much earlier. This suggests that patent filing is a rich data source with which to track emerging trends in technological developments, particularly in the Chinese context, when considered in combination with analysis of high level policy documents.

The approach presented here represents a new approach to deriving insights from patents — an almost entirely ignored, hugely valuable and freely-available large-scale data source concerning developments in technological innovation — and has significant potential for broader application. The methods shown here were originally developed in the context of understanding commercial interest in products derived from endangered wildlife (Hinsley et al., 2023) and their implications for the wildlife trade. There are clear applications of these approaches for general investigation of emerging technologies and commercial developments, as well as for deeper analysis of smart cities and their implications for human rights. While this article has focused on smart city patents filed in China, the method can easily be applied to compare trends in patent filing in other geographic locations; to identify networks of, and relationships between, patent filing-related entities – such as patent offices, patentees, the nationality of inventors and, indeed, corporations – to find emerging trends in related technologies and how they spread between entities and locations. The combination of large-scale patent data and emerging machine learning methods presents a diverse range of novel approaches to understanding emerging technologies across many fields.

The methods and findings presented here have, however, directly demonstrated the value of patent analysis as a horizon scanning technique to identify not only emerging technological trends directly but also to provide insight into the terms and concepts that drive them from a philosophical or ideological perspective. We propose that regular, systematic surveys of patent filing presents a valuable, open data source through which human rights defenders can maintain a view of emerging technologies and procedures that have potential to impact on the lives of citizens.

Our analysis has shown that emerging smart city technologies have potential impacts on various aspects of their inhabitants’ human rights, making it increasingly difficult to preserve privacy and anonymity. Whether someone goes to work, is in the streets or at home, the surveillance industry is increasingly innovating to create and maintain ‘digital twins’ that virtually replicate buildings, pedestrians, vehicles and infrastructure. Further, this trend in smart city technologies is not only increasing, but rapidly accelerating. With growing capacity for data gathering, storage, analysis and prediction, as well as a clearly demonstrated commercial interest in the development and application of these technologies, the trends identified here cast a concerning shadow over efforts to preserve fundamental freedoms in the urban future.

Appendix A: Supplementary data

Table A1: Full topic model of ‘smart city’ patent search results, labelled with human-interpreted topic titles