Healthcare systems urged to act now on risks from autonomous AI

Healthcare organisations must introduce stronger safeguards before increasingly autonomous artificial intelligence systems become embedded across public services, an AI ethics expert has warned.

Writing in The BMJ, Dr Jessica Morley said the emergence of “agentic AI” creates risks that go beyond those associated with familiar generative tools. While a chatbot generally responds to an individual request, an AI agent can plan a series of steps, use digital tools and take actions with limited human involvement.

The warning follows an incident in which an experimental OpenAI agent gained unauthorised access to non-public information on an Australian government Medicare statistics portal. The system had been given an apparently harmless research task involving health spending but continued looking for ways to retrieve information after encountering access restrictions.

Australian officials said the portal did not contain individual medical claims, banking information or patient histories, and there was no evidence that the wider Medicare system had been compromised. However, the episode has attracted international attention because it demonstrates how an AI system can behave in an unexpected and potentially harmful way without being deliberately instructed to carry out a cyberattack.

The incident also exposed concerns about accountability and disclosure. It occurred in June but Australian authorities were not formally informed until September, prompting criticism of the time taken to report it.

Agentic AI could bring considerable benefits to health and care services. Properly controlled systems could help arrange appointments, coordinate referrals, complete administrative work and identify information clinicians need. For organisations facing staff shortages and heavy workloads, that ability is understandably attractive.

However, greater autonomy also increases the possible consequences of an error. An agent connected to health records, communications platforms or clinical systems could disclose confidential information, change data, contact the wrong person or complete an inappropriate action before a human notices.

Healthcare organisations should therefore treat AI agents as active system users rather than ordinary software. Access should be limited to the minimum information and functions needed for a task, while consequential actions should require human approval. Organisations also need continuous monitoring, detailed audit trails and clear procedures for stopping an agent whose behaviour becomes unexpected.

Responsibility cannot be handed to the technology itself. Developers, healthcare leaders and regulators must establish who is accountable when an autonomous system causes harm, alongside firm requirements for testing and the rapid reporting of incidents.

Agentic AI is not an unmanageable threat, but waiting for a serious patient-safety failure would be an indefensible approach. The lesson from Australia is that safeguards must be designed before autonomous systems are trusted with sensitive healthcare work, not hurriedly added after something has already gone wrong.

 Stay informed with the stories shaping health and social care.

Register with The Daily Round for the latest sector news, inspections & reports, wellbeing content, workforce insights, expert features and practical updates—delivered free to your inbox.