Artificial intelligence in primary care

3.1. Safety and regulation issues

Risks to patient safety present the largest obstacle to widespread use of AI in healthcare. An AI tool could provide unsafe advice because it has been poorly programmed, trained with inadequate or unrepresentative (biased) data, or is vulnerable to hacking.¹ There is also a risk an AI tool is used in an inappropriate setting or context, which can skew the device’s decision-making ability over time.³

AI technologies remain under-regulated in Australia and across the world. Researchers looking to implement AI solutions in the Australian primary care setting have questioned the lack of rigour in assessing the safety and efficacy of AI tools, which sits in contrast to that applied to new treatments and procedures.⁸

Where regulation is lagging behind technological advancement, there is a risk that commercial developers could rush a product to market without sufficient testing or evidence to safeguard the public.

There are specific challenges to consider in regulating AI products. By definition, machine learning involves an evolution in a computer’s ability to learn from experience, so a device that uses this technology might be compliant with standards on one day but not on the next.¹ AI might also be capable of identifying anonymised data.⁹ Another major issue is the ‘black box’ phenomenon. The complex decision making that takes place within a machine learning system is hard for a human to untangle and understand, and in some circumstances that process will be the proprietary product of the developer or technology company. This creates issues in assessing the reliability of the system and determining whether biases have compromised its decision making. 

Hand in hand with the issue of regulation is the issue of liability. The finding of fault might become complicated where a clinician ignores a recommendation made by AI, or follows an AI recommendation but it results in a bad outcome.⁸ There are many ethical questions of this nature that require deep consideration. For example, should a GP have the right to override a machine’s diagnosis or decision? Should the reverse apply, and if so, to what degree?¹

3.2. Exacerbating health inequality

There is warranted concern that AI could propagate and magnify existing problems with health equity. Biases can be ‘baked in’ to AI products. They can be trained with unrepresentative datasets, or their goals might be inappropriately chosen.¹ For example, one concern is in the use of historical training data in which women and ethnic minorities are unrepresented.

3.3. Workforce issues

While many have envisioned a future in which human tasks will be completely subsumed by AI, it is unlikely that the complex work of a doctor will ever be performed by machines operating independently. AI cannot replicate the value judgements and empathy of a doctor, but in time, it will complement human skills.^{5, 10} GPs will continue to provide the societal, clinical, and personal context for diagnostics and treatment decisions.¹¹ 

However, there are legitimate concerns about how AI might affect the delivery of primary care. If AI takes on a significant role in patient interaction and decision-making, this shift might impact what it means to have a career in medicine.¹ Just as AI has the potential to reduce the burden of administration, it might also decrease job satisfaction by stripping back the social elements of general practice and the stimulation of problem-solving in diagnostics.

Importantly, there is a need to ensure that AI does not create unnecessary and low-value work for GPs and general practice staff. There is potential for developers to create products that are intended to create efficiencies and improve effectiveness, but end up simply generating additional work for clinicians through task substitution, such as reviewing and editing clinical notes that have been automatically prepared by a digital scribe.¹²

Impacts for macro-level service delivery might also be on the horizon. For example, demand for GP services could increase if patients self-refer for care at the direction of an AI tool.¹ AI could redirect low-cost, low-risk patients away from general practice, and leave behind those who require more clinician time and resources. This has the potential to burden an already under-resourced sector and contribute to GP burnout.¹²

3.4. Trust and acceptance

A major impediment to the uptake of AI within primary care is likely to be a lack of trust in the technology. There is skepticism about AI’s ability to deliver care that takes into account patient preferences, needs and values. Many GPs consider the interaction between patient and doctor to be integral to medical information gathering, and believe that patients will not accept a technological substitute for face-to-face human care.¹³ 

The lack of collaboration between professionals from the fields of medicine and AI contributes to this lack of trust and is a barrier to uptake.¹⁴ Without clinicians involved in the design, implementation and regulation of AI technologies, there is a risk that the healthcare system will inherit products that not only fail to solve existing problems, but create new and unforeseen ones in their wake.

GPs and primary care researchers need to work in partnership with AI developers. GPs must have a say in guiding and overseeing the integration of AI as end users of these technologies. 

Failure to engage in this process could have far-reaching consequences. It could result in systems or products that do not benefit either clinicians or patients.^{12, 14} Value to technology company shareholders might be prioritised over patient outcomes.¹⁵

There is a need to build research teams that consist of AI specialists, primary care researchers, and frontline clinicians and to foster collegiality between the two disciplines.⁴ Researchers should look to conduct evaluation studies within primary care settings and GPs should play an active role in formulating research questions so the studies’ conclusions have real-world applications.¹⁵

The RACGP also has a role in bridging the gap between the profession and industry and government bodies to facilitate knowledge sharing in AI.

4.1. Governance

Lawmakers, regulators, and professional bodies must keep pace with the development of AI to ensure it is implemented safely, particularly in the medical setting where the stakes are high.

The RACGP is keen to work with all stakeholders on the puzzle of how best to regulate AI so that these technologies can benefit Australian patients.

We recognise the need to strike a balance between regulation and innovation to advance the interests of the patient. Full and frank consideration of the various ethical and legal issues before the technology is implemented widely might be the key to successful development of new technologies, as it was in the case of in vitro fertilisation (IVF).¹ Slow work in this arena or over-regulation could prevent the development of potentially life-saving technologies.⁹

There must be safeguards to protect patients after the product is released. Post-market surveillance will be critical, as systems that continuously learn from previously-collected data must be evaluated over time, in a dynamic way, to ensure they are still fit for purpose. 

There may be a need to establish a new Australian regulatory body to fill gaps in existing government portfolios and legislative instruments.¹⁶

4.2. Education

GPs at all stages of the career cycle will need new skills to work with AI and keep pace with rapid developments in this field.¹⁷ Some argue that a comprehensive overhaul of medical education is required, as the changes brought by AI will have far-reaching effects on the practice of medicine.¹⁸ What is certain is that medical students of the future will require an understanding of medical informatics, mathematical concepts, data science, and the ethics of AI.⁷ They will need tools for appraising new AI technologies for safety and efficacy.¹⁹ At the postgraduate level, there should also be a focus on relevant applications of AI in clinical practice.²⁰ 

To get the most out of AI technologies for their patients, doctors will also need to maintain and build on the skills that separate them from machines – the ability to demonstrate empathy, to see the relationship between a patient and their illness, and to consider the emotional states of the patient.^{19, 20} Skills in communication, decision making, leadership, and team-based work will continue to be of pivotal importance to future GPs.

The RACGP is committed to support GPs to develop the skills needed to work with AI as required.