London Observatory

Several participants expressed a sense of exclusion from the AI design process.

All of the different stages of the design loop for AI/automated systems – from problem identification through prototype engineering to testing and deployment – were deemed too far removed from end users. It was mentioned that the design, manufacture and marketing pipeline can take years to complete, and there are few if any points where users can intervene. Another participant noted that engineers seem to be deploying fashionable technologies according to economic logic; they may appear powerful, but often reduce usability.

For example, many functions on new cars are accessed via touchscreens (that also require the driver to look at them); previously the functions were available via physical buttons and levers that the driver could find by feel. A respondent who had driven new AI-enhanced vehicles (and one who had worked on bringing one to market) remarked on this as a particularly backward step in haptic design.

Participants also voiced criticisms of the model of technological progress that is implemented in AI innovations in the street, going beyond more common concerns with data protection and surveillance.

A participant noted that the emphasis on protecting personal data ignored the interpersonal nature of much data (especially that generated by encounters in the street/public space), and mirrored notions of individual property ownership. Another mentioned data rights in Maori and Aboriginal cultures that take different forms prioritising community over individual, and such approaches could be more widely relevant. One participant remarked that 'technology itself is shining and everything else that it's attaching to seems to fade' – noting that we are too attracted to tuning technological solutions, while ignoring fundamental questions; for another, it was natural resource limitations and time constraints that led to a 'zero sum game' where increasing deployment of AI meant less capacity to address other issues.

The dependence of computation on electricity derived from fossil fuel sources and the environmental costs of manufacturing electronic components were also mentioned.

Some participants flagged trade offs between increasing automation and human autonomy.

One felt that automation brought about a loss of motivation or dignity; others acknowledged that while this might be the case, if the end result was fewer accidents (due to self-driving cars) or cleaner streets (due to smart bins), then it was worth it. The group discussed a facial recognition system deployed in supermarkets and other shops for AI-driven age-verification, noting that if the 'computer says no' then the (suspected underage) customer is locked out of the purchase, which the staff cannot override.

While some flagged that this has led to a reduction in violence against shop staff from frustrated customers, another highlighted the risk that such a system disempowers the staff and could thus lead to unwanted outcomes.

 Several participants voiced scepticism about the capacity of AI systems to anticipate and respond to human behaviour, especially given the nuances and cultural specificities of spoken and gestural language, differences in behaviour typical of individuals and groups, and the very different use of public space in Northern Europe versus, say, South Asia.

Participants mentioned their experiences of (human) failures in the installation and maintenance of otherwise well-designed systems. Others voiced concern about the general vulnerability of computational systems vs. electro/mechanical and/or social systems, which they felt was exacerbated by their complexity and opacity (e.g. undiscovered bugs, hacks, exploits, etc.).