F11 – Artificial intelligence (AI)

If the practice uses AI tools, it needs to inform patients of how these tools use their health information (for example, what information is collected, where it is stored and who has access to it).

Tools used to transcribe consultations may need to access patient health records or listen to the consultation in order to generate notes for the GP, and this could have implications for data security if the use of AI does not comply with Australian privacy legislation.

Patients have the right to withdraw consent for the use of AI tools in their care at any time. Practices need to inform patients of this right during the consent process and have a clear and accessible mechanism for patients to opt out. If a patient withdraws consent, the practice needs to:

• document the withdrawal in the patient’s health record
• stop using AI tools in the patient’s care unless clinically necessary and re-authorised
• provide alternative care pathways that do not rely on AI tools, where feasible
• communicate any limitations or implications of opting out in a transparent and respectful manner.

See PP4 – Informed consent for further advice on obtaining informed consent.

If the practice uses AI tools, it must consider how to meet its obligations under the Australian Privacy Act 1988 and the Australian Privacy Principles (APPs). Legal advice may be useful, particularly when determining relevant answers to the questions listed below.

• Does the AI vendor confirm compliance with the Privacy Act and APPs?
• Can collected data be used or sold for secondary purposes?
• How is patient data encrypted, stored, and destroyed?
• How is personal information managed?
• Can clinicians review and confirm AI outputs before use?

The AI systems the practice uses in clinical care need to operate within a framework that supports clinical supervision of, intervention in, and accountability for any AI output. This includes establishing:

• governance structures
• risk management processes
• safeguards to support clinicians when making informed decisions.

Practitioners have professional obligations related to the use and review of AI tools used for clinical care. Ahpra provides advice about meeting professional obligations and the potential challenges facing clinicians who use AI.

AI tools may be developed or operated by third-party vendors. Practices need to confirm that:

• any third-party technology used in patient care (including AI scribes, transcription tools, or decision-support systems) complies with privacy legislation
• patients are informed of the involvement of external providers (as per third party consent at PP4 Informed consent)
• contracts with vendors address data handling, consent, and accountability.

When AI tools are provided by third parties, practices remain responsible for their safe and appropriate use.

AI systems need to be monitored and evaluated not only before implementation, but afterwards to confirm that they perform as intended and that risks are actively managed over time. Given the potential for significant impacts on patients or on the practice’s operations, practices need to provide:

• a timely and accessible process for patients, consumers, and the practice team to question or challenge AI outputs
• transparent processes for responding to AI-related issues
• clinical autonomy for practitioners, including the ability to review and override AI outputs using clinical judgement
• access to information about how its AI systems work so that patients, consumers and members of the practice team can properly question its output if needed
• mechanisms for monitoring and reviewing AI performance, safety, and appropriateness, including regular evaluation against clinical standards and the practice’s needs.

Practices need to be particularly mindful of the potential disproportionate impact of AI on vulnerable populations. Vulnerable populations (for example, people with disabilities, older adults, people from culturally and linguistically diverse backgrounds, and those experiencing disadvantage) may be more at risk of harm from AI systems. This harm could arise from biases in the data the AI was trained on, a lack of transparency about how decisions are made, and barriers to understanding or contesting AI-driven decisions. Practices using AI need to assess and manage these risks.

To determine whether AI tools are appropriate for diverse patient populations, practices could:

• assess whether the AI system has been trained on data that reflects the diversity of the practice’s patient population
• evaluate whether outputs are culturally safe, inclusive, and clinically relevant for different groups
• involve consumer representatives and/or cultural advisors when selecting and evaluating AI tools
• evaluate accessibility for patients with low health literacy, language barriers, or disability
• review vendor documentation relating to bias mitigation, the extent to which their data represents different groups, and equity safeguards.

When using AI tools that process patient data, practices need to be mindful of Indigenous data sovereignty (the right of Aboriginal and Torres Strait Islander peoples, communities and organisations to maintain, control, protect, develop, and use data that relates to them)⁽²¹⁾, particularly when working with Aboriginal and Torres Strait Islander patients. The Maiam Nayri Wingara principles provide a framework for ensuring that Indigenous peoples have control over the collection, access, and use of data relating to them. These principles emphasise the need for:

• ownership of data by Indigenous communities
• control over how data is used and shared
• access to data that is relevant and meaningful
• custodianship of data in ways that respect cultural protocols⁽²²⁾.

Practices strengthen their approach to monitoring and reviewing AI by implementing assessment activities. The practice could:

• collect structured feedback from members of the practice team about their experiences with AI systems, including its effectiveness and usability
• audit outcomes from AI-generated recommendations (for example, checking for over-diagnosis or missed diagnosis when AI tools are used for interpretation or triage)
• monitor consumer feedback and complaints related to the use of AI in their care, to identify safety concerns or opportunities for improvement
• track the number and nature of technical support requests or issues raised about AI system performance
• collect data on how often AI is used in clinical decision-making or administrative processes, and how often clinician-only approaches are used
• review updates and changes to AI systems to confirm they are not introducing new risks or reducing effectiveness.

Some AI tools used in general practice (for example, diagnostic aids, clinical decision support systems, transcription tools) may be classified as medical devices under the Therapeutic Goods Act 1989.

If an AI tool meets the definition of a medical device, it must be included on the Australian Register of Therapeutic Goods (ARTG) before it can be lawfully supplied or used in Australia. Practices need to confirm that any AI tools used in clinical care:

• are appropriately registered on the ARTG
• comply with relevant Therapeutic Goods Administration (TGA) guidance
• come with documentation from the vendor explaining regulatory status and intended use.

If a practice is unsure whether an AI tool qualifies as a medical device, the practice can seek legal or regulatory advice to make sure there is no breach of TGA requirements. For more information on how AI medical devices are regulated, refer to the Department of Health Disability and Ageing’s Artificial Intelligence (AI) and medical device software.

The RACGP has developed the following resources for practices to use to guide the implementation of digital health technologies and/or AI in their practice.

• Artificial intelligence (AI) scribes: This resource describes AI scribe tools that can automate parts of the clinical documentation process for a medical practitioner, and explores the potential benefits and risks, as well as relevant considerations.
• Conversational AI: This resource aims to assist GPs weigh up the potential advantages and disadvantages of using conversational AI in their practice.
• Artificial intelligence in primary care position statement: This resource can help the practice team to understand the advantages and risks of AI in general practice.
• Privacy and managing health information in general practice: This resource aims to help the practice team understand relevant privacy laws, patient consent, and information management and security, and how to adapt these to digital health technology and AI.

The Australian Commission on Safety and Quality in Health Care (ACSQHC) Pragmatic Artificial Intelligence (AI) guidance for clinicians supports clinicians in the day-to-day use of AI tools.

Ahpra’s Meeting your professional obligations when using Artificial Intelligence in healthcare explains how health practitioners must apply existing professional obligations when using artificial intelligence in healthcare.

Department of Industry, Science and Resources provides detailed guidance for responsible AI governance in Guidance for AI Adoption: Implementation guidance and outlines the following six elements of responsible AI use:

1. Accountability
2. Impact assessment
3. Risk management
4. Transparency
5. Monitoring
6. Human oversight.

The World Health Organization (WHO) Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models addresses the use of different types of AI systems.

The federal Department of Industry, Science and Resources has published eight voluntary AI ethics principles which can assist the practice provide safe healthcare when using AI. These principles encourage businesses to use AI in ways that:

• benefit individuals, society and the environment
• respect human rights, diversity, and individuals’ autonomy
• are inclusive and minimise discrimination
• uphold rights to privacy
• operate in accordance with their intended purpose
• are transparent and easily understood
• allow a person, community, group or environment to easily contest the uses or outcomes of the AI system
• allow accountability and human oversight.

Medical defence organisations provide advice on medico-legal issues such as privacy, consent, accuracy and quality.