In a new paper published in BMJ’s Journal of Medical Ethics, the TAS functionality nodes’ Jonathan Ives, John Downer and Helen Smith explore the increased use of AI in healthcare and medical settings, and the lack of professional guidance around it.

Although AI has great potential to help improve medical care and alleviate the burden on healthcare workers, the authors argue that as there is no precedent for when AI or AI-influenced medical workers make a mistake, regulation should be developed as a priority to outline the rights and expectations of those working closely with it.

There have recently been reports from National Health Service AI Lab & Health Education England which focus on healthcare workers’ understanding and confidence in AI clinical decision support systems, and are concerned with developing trust in, and the trustworthiness of these systems. However while they offer guidance to aid developers and purchasers of such systems, they offer little specific guidance for the clinical users who will be required to use them in patient care.

The clinicians who will have to decide whether or not to enact an AI’s recommendations are subject to the requirements of their professional regulatory bodies in a way that AIs (or AI developers) are not. This means that clinicians carry responsibility for not only their own actions, but also the effect of the AI that they use to inform their practice.

The paper argues that clinical, professional and reputational safety will be risked if this deficit of professional guidance for clinicians, and that this should be introduced urgently alongside the existing training for clinical users.

The authors end with a call to action for clinical regulators: to unite to draft guidance for users of AI in clinical decision-making that helps manage clinical, professional and reputational risks.

More information

Read the full paper:

http://dx.doi.org/10.1136/jme-2022-108831

Read a blog about the paper on BMJ

All authors are fully or part funded via the UKRI’s Trustworthy Autonomous Systems Node in Functionality under grant number EP/V026518/1.

Helen Smith is additionally supported by the Elizabeth Blackwell Institute, University of Bristol via the Wellcome Trust Institutional Strategic Support Fund.

Jonathan Ives is in part supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health and Social Care.

Matimba Swana, our TAS PhD student, joined other early career researchers in a “reverse think tank” project last year, organised by the Public Engagement team at University of Bristol and co-led by Kilter Theatre. Designed to be a creative approach to the new emphasis on responsible innovation, as well as encouraging debate about what that means, the reverse think tank challenges the notion of the “expert”. Eleven 15-18 year olds gathered to learn about current research which they then used in a collaborative storytelling activity, based on a Disney strategy they employed to think of new ideas for films.

Although the researchers weren’t present, the young people saw videos of Matimba’s SWARM study research, to get an idea of where the research might lead. She is looking at the ethics and regulations of nanoswarms, a future nanotechnology that could be developed to treat cancer, to help shape the first in-person clinical studies. The young people were excited about the possibility of the technology being used but were concerned about negative consequences such as autonomy and unknown long-term effects. Their story involved a man named Ralph, living in Bristol, in his early 50s. He’s wondering whether to get nanoswarm treatment for cancer, asks for advice from his wife, who is very distrustful of the medical establishment, and is “anti-vaxx” and his friend Bob, who believes this is Ralph’s chance to be immortal. This also highlighted the possible inequalities that could arise from nanoswarm use, and the general distrust of medical advances.

Matimba found the think tank outcomes really useful and says “Something that resonated with me was we conduct research that impacts the future of young people and yet their voices are often not included in this area of research. Engaging young people as peer reviewers of my research has been enriching and a chance to understand their thoughts and feelings about the technologies we are investigating that could impact their future.”

You can read more about it here

If you’d like to participate in our SWARM study focus groups, please go here.

In a new paper published in AI & Society: Knowledge, Culture and Communication, the TAS Functionality Node’s Helen Smith, Kerstin Eder and Jonathan Ives argue that using the term ‘autonomous’ to describe the capabilities of highly automated systems is misleading at best.

Drawing on examples from widely-know science fiction, the Cyberdyne Systems Model T-800 depicted in the Terminator and Terminator 2 films is presented as a great example of an adaptive system that demonstrates evolving functionality and decision-making. However, the authors observe that it can hardly be defined as autonomous when its overall goals and limitations are set by another agent, removing autonomy or ‘freedom of choice’ from its functionality.

The authors look at the implications of describing a system as autonomous, since in doing so we would be assigning moral agency to it. On this basis, the expectation would be that a system is a moral agent and therefore can be held responsible for bad decisions, which is simply not feasible (‘a computer cannot be fined or put in jail when a bad decision is made’ (Dignum et al, 2018, p.63)).

The authors conclude that the correct use of language to describe critical systems is vital to ensure responsibility for the systems’ decisions and actions is attributed to those designing, developing and operating these systems, rather than the systems themselves.

More information

Read the full paper: https://doi.org/10.1007/s00146-023-01662-9

This paper was led by Dr Helen Smith, a Research Associate in Engineering Ethics and Registered Nurse based in the Centre for Ethics in Medicine at the University of Bristol. Helen works as part of the TAS Node team bringing her expertise in the ethical and legal challenges of AI use in healthcare (and beyond).