The Victorian Metropolitan Alliance (VMA), a Regional Training Provider (RTP) under the Australian General Practice Training (AGPT) program, delivered general practice training to the Melbourne metropolitan region from 2001 to 2015. The VMA’s Education Enhancement Program (EEP) began as a program of remediation and support for general practice registrars in 2006. Clearly defined guidelines, originally modelled on the UK’s National Association of Clinical Tutors (NACT)1 framework for managing remediation, were continually reviewed and refined. This was to ensure best practice2,3 was followed and the needs of the registrars were met.
A team of experienced medical educators managed the program under the direction of the Education Enhancement Officer (EEO). The role of the EEO is to support general practice registrars who experience difficulties, educational or otherwise, during their training. The role of the EEO and EEP has encompassed assessing and monitoring registrars’ progress, and formulating and delivering individually tailored educational interventions to underperforming registrars. In 2011, the VMA’s Diagnostic and Assessment Centre opened, making the services and expertise of the EEP available to other general practice training providers and organisations, such as the Australian Health Professionals Regulation Agency (AHPRA) and medical defence organisations (MDOs).
This paper, authored by the EEO, is a reflection on the work conducted by the VMA’s Diagnostic and Assessment Centre from December 2011 to June 2015. Firstly, it reflects on the formal assessment of clinical skills, which has been almost entirely with respect to non-VMA registrars whose future in general practice training rested on the determination of the assessment (ie ‘high-stakes’ assessments). Secondly, it reflects on the focused, educational interventions, mentoring and audits of practitioners who were either under investigation or where a determination had been made by AHPRA for alleged breaches of regulations. Referrals to the EEP were almost entirely from AHPRA or from the MDO assisting the practitioner.
The data
Formal assessment of clinical skills
Eleven formal assessments of clinical skills were conducted between January 2012 and June 2015 (Table 1). Only one referral was received from a private general practice for the assessment of a general practitioner’s (GP’s) communication skills, with the intention of informing an educational intervention. Sixty-four per cent of referrals were for overseas-trained doctors.
Table 1. Formal assessment of clinical skills between January 2012 and June 2015
|
| Assessment type | n |
Referred by
| Overseas trained | Gender |
|---|
| General Practice Education and Training* | Regional Training Provider | Other |
|---|
| Comprehensive |
7 |
2 |
5 |
– |
5 |
3 male and 4 female |
| Reassessment |
1† |
– |
1 |
– |
– |
1 female |
| Focused |
3 |
– |
2‡ |
1§ |
2 |
2 male, 1 female |
| Total |
11 |
2 (18%) |
8 (73%) |
1 (9%) |
7 (64%) |
5 male and 6 female (45% and 55%) |
*The organisation governing general practice training under Australian General Practice Training until the end of 2014
†Registrar reassessed seven months after the initial assessment and following implementation of a remediation plan
‡Review of video-recorded consultations
§Assessment of communication skills (referred by the practice where the doctor was working) |
Ten assessments were for registrars and seven of the assessments were considered ‘high stakes’. One registrar was reassessed following the completion of a remedial plan. All assessments encompassed:
- application of knowledge
- history-taking and physical examination
- the ability to synthesise information
- reasoning
- development and implementation of a management plan.
The seven comprehensive (high-stakes) assessments also addressed knowledge base, insight, receptiveness to feedback and capability to improve. Following each assessment, a comprehensive report was written, which commented on each of these areas and provided specific recommendations with respect to a remediation plan (Box 1). Of the nine registrars who were assessed, three left general practice training, three received educational interventions and are progressing in their training, and three are presently waiting for a suitable practice to be found and a remediation program to be put into place.
Box 1. Comprehensive assessment of clinical skills*
|
Prior to the assessment
- Referral received with outline of concerns
- Background information obtained (eg training to date, issues as perceived by the medical educators and supervisors, relevant reports)
- Appraisal of the registrar’s recently video-recorded consultations (if available)
- Case selection and briefing of simulated patients and medical educators assisting with the assessment†
|
The assessment proper
- Discussion with the registrar (eg their perspective of the issues)
- Completion of validated and benchmarked written papers (eg multiple choice questions and key feature problems)
- First round of role-play of clinical scenarios (6–10 cases), followed by feedback‡
- Second round of role-play of clinical scenarios (6–10 cases), followed by feedback§
- Overall feedback to the registrar and recommendations
|
Report includes
- Results of written papers
- Detailed information on the registrar’s performance in the role-plays
- Summary of the registrar’s strengths and weaknesses
- Determination as to the registrar’s:
- level of competence with respect to knowledge, clinical skills and overall (benchmarked)
- remediability
- Specific recommendations with respect to a remediation plan
|
*A typical assessment is conducted over three days or four days if appraisal of video-recorded consultations is also conducted. This does not include time for preparation and report writing.
†Consent obtained from the registrar and simulated patients to video record. All simulated patients and medical educators sign confidentiality agreements.
‡Role-plays are observed by the Education Enhancement Officer (EEO) and video-recorded to facilitate feedback. Brief feedback is provided by simulated patients and medical educators. More detailed feedback is provided by the EEO.
§The purpose of the second round is to see whether feedback given can be implemented and to determine remediability. |
Focused educational interventions, mentoring and audits
Forty-nine practitioners, 40 GPs and nine medical specialists or allied health professionals sought the services of the EEP (Tables 2, 3). Forty-nine per cent were overseas-trained doctors. The requests were overwhelmingly for focused educational interventions (n = 43; 88%) as opposed to requests for mentoring and audit of medical records. The interventions varied with respect to length (one to four hours) and number (one to six sessions). Each intervention was conducted face to face and was individually tailored to address the areas that had been deemed by AHPRA to have been breached. Most of the practitioners were male (82%). For most practitioners (65%), AHPRA had completed the investigation and made a final determination (Table 2). The working experience of the practitioners varied from those who were still in hospital training to those who were 70 years of age or older and still practising. The five principal topic areas that were addressed were:
- clinical skills – 33%
- prescribing – 24%
- documentation – 18%
- patient boundaries – 17%
- ethical and legal – 4%.
Table 2. Focused educational interventions, mentoring and audits between December 2011 and June 2015
|
| Intervention type | n | Primary referring source | Overseas trained | Gender | General practitioner |
|---|
| Australian Health Professionals Regulation Agency | Medical defence organisations | Other |
|---|
| Educational |
40 |
23 |
16 |
1* |
18 |
34 male and 6 female |
31† |
| Mentoring |
2 |
2 |
– |
– |
1 |
1 male and 1 female |
2 |
| Audit |
3 |
3 |
– |
– |
3 |
3 male |
3 |
| Education and mentoring |
1 |
1 |
– |
– |
1 |
1 female |
1 |
| Education and audit of records |
2 |
2 |
– |
– |
1 |
2 male |
2 |
| Mentoring and audit of records |
1 |
1 |
– |
– |
– |
1 female |
1 |
| Total |
49 |
32 (65%) |
16 (33%) |
1(2%) |
24 (49%) |
40 male and 9 female
(82% and 18%) |
40 (82%) |
*Self-referred
†The remaining nine comprised five specialists (three physicians, one surgeon and one psychiatrist), one specialist training program (anaesthetics), and three allied health professionals (one psychologist, one chiropractor, one osteopath) |
Within these topic areas, prescribing drugs of addiction was the topic most frequently addressed (23%), followed by appropriate documentation (18%). Equal third were informed consent, communication skills and professional boundaries (11% each). Communication skills were more frequently addressed than indicated in Table 3 because they were also addressed in other areas such as informed consent, managing drug-seeking patients and maintaining professional boundaries (Table 3).
Table 3.Topic areas for the educational interventions between December 2011 and June 2015
|
| Educational content | Number of practitioners |
|---|
|
Clinical skills
(n = 23; 33%)
|
Informed consent |
8 |
| Communication skills* |
8 |
| Patient assessment |
2 |
| Working with patients |
2 |
| Clinical decision making |
1 |
| Patient management |
1 |
| Keeping up to date |
1 |
Prescribing
(n = 17; 24%) |
Drugs of addiction, pain management† |
16 |
| Off-label |
1 |
| Appropriate documentation (n = 13; 18%) |
13 |
Boundaries
(n = 12; 17%) |
Professional |
8 |
| Sexual |
4 |
|
Ethical and legal
(n = 3; 4%)
|
Confidentiality |
1 |
| Privacy |
1 |
| Certificates |
1 |
| Other (n = 3; 4%) |
3 |
| Total |
71 |
*Focused on ‘communicating effectively with patients, carers and relatives’. With other topics, such as informed consent, the importance of communication skills is highlighted as well as how they can be used to improve outcomes.
†Areas covered include: regulations with respect to drugs of addiction, understanding and managing the behaviour of drug seeking patients, and principles and practice of chronic pain management. |
Discussion
When the services of the EEP were first made available externally, it was unclear what the level of demand would be. Anecdotal evidence indicated that a need existed for objective and independent assessment of clinical skills. Requests for assessment have been almost entirely for registrars, and mostly from other RTPs.
All of the assessments conducted on registrars were ‘high stakes’. Therefore, it was important to follow best practice4,5 and for processes to have validity and reliability.6,7 Careful preparation, skilled medical educators, well-trained simulated patients8 and standards for comparison5,6 were found to be necessary if the final determination was to be defensible. Processes have been continually reviewed to ensure ongoing rigour. Because of all these factors, every assessment has been labour and time intensive.
Each assessment must be individually tailored. Methods of assessment were chosen9,10 according to the referrer’s concerns, and to enable the registrar to fully demonstrate their abilities. Background information from the referrer assisted considerably in this respect. It was also important for the registrar to fully engage so they felt they had adequately demonstrated their abilities. Engagement11 also allowed the assessor to gain a better understanding of the registrar’s difficulties, including any cultural dimensions, clinical reasoning and degree of insight. However, an adequate determination of professionalism12 could not be made, given that the situation was contrived and the person under assessment knew that their behaviour was being scrutinised. All registrars cooperated fully with the assessment.
In contrast to the number of requests for formal assessments, there has been a big demand for educational interventions, including a small number from New South Wales and Queensland. What has been discovered is very few providers offer this service. As the availability of the EEP has become more widely known, the demand has grown significantly. Therefore, there appears to be a distinct need for more providers for educational interventions as well as for audits and mentoring.
Annual reports from AHPRA only provide basic data regarding doctors who have undergone investigation.13 These reports do not provide any information about the types of infringements and determinations made. Limited data are available regarding overseas-trained doctors.14 The data presented here cannot therefore be taken as being representative. The lack of more specific data from AHPRA makes it difficult for planning and program development.
Similarly to the formal assessments, engaging individuals who were mandated to undergo an intervention has indeed been a challenge but was very important for learning outcomes to be attained. Engagement was achieved by following the principles of effective feedback15,16 – gaining an understanding of the practitioner, listening to their perspective and providing constructive feedback – thus making their experience meaningful and valuable.
The EEP has been able to provide independent assessments of clinical skills, and focused educational and other interventions to medical and other health practitioners. As a result, the EEP has gained considerable expertise in the assessment and management of doctors experiencing difficulties. Other potential areas for the EEP include:
- Providing assistance to practitioners returning to the workplace after a prolonged period of absence (eg illness or maternity leave). This need has already been identified and successfully addressed with registrars who were returning to the VMA’s training program after a substantial period of absence. There have been similar experiences overseas.17–19
- Assessment of underperforming GPs in private practice18 and developing learning plans for them.
- Providing continuing professional development programs on communication skills and prescribing drugs of addiction. Unsurprisingly, these are the two main areas where the EEP has assisted practitioners, as evident in bulletins from AHPRA and MDOs, which repeatedly highlight that poor communication is one of the prime reasons for complaints against doctors. With drugs of addiction, many of the doctors were reported to AHPRA because of their lack of familiarity with the regulations and inability to manage drug-seeking behaviour, which has become a complex and difficult area.
- Creating programs for supporting overseas-trained doctors who face significant difficulties in the workplace.20
The EEP has assisted practitioners to improve their clinical skills and also potentially prevent them from making similar infringements in the future. The EEP will continue to operate in the new training landscape under the banner of Eastern Victoria General Practice Training. The aim is to continually evaluate and improve processes of assessment, and explore educational interventions that are innovative and suited to different levels of experience and need. Research into the particular needs of overseas trained doctors, as well as the long-term outcomes for doctors who have been through the program, would also be useful for the ongoing development of the EEP.
Author
George Zaharias MBBS, MFM, FRACGP, Education Enhancement Program Director, Eastern Victoria General Practice Training, Hawthorn, VIC, and previously Education Enhancement Officer with Victorian Metropolitan Alliance General Practice Training, VIC. zaharias@optushome.com.au
Competing interests: None.
Provenance and peer review: Not commissioned, externally peer reviewed.