Ear, nose and throat

June 2016

Professional

The Victorian Metropolitan Alliance’s Education Enhancement program: A reflection

Volume 45, No.6, June 2016 Pages 439-443

George Zaharias

Background

The Victorian Metropolitan Alliance’s (VMA’s) Education Enhancement Program (EEP) began as a program of remediation for general practice registrars in 2006. Considerable experience was gained in the area of remediation, culminating in the opening of the VMA’s Diagnostic and Assessment Centre in 2011. Since then, the EEP has made its services and expertise available externally and has been conducting comprehensive assessments of clinical skills, focused educational interventions, mentoring and audit of medical records for doctors.

Objective/s

This paper presents the data with respect to the external activities of the EEP, challenges encountered and lessons learnt.

Discussion

The EEP has provided some valuable services. The comprehensive assessments of clinical skills, being high-stakes assessments, required rigorous processes. There has been a greater than expected demand for educational interventions, particularly with respect to communication skills and prescribing, highlighting the need for such services to be more widely available.

The Victorian Metropolitan Alliance (VMA), a Regional Training Provider (RTP) under the Australian General Practice Training (AGPT) program, delivered general practice training to the Melbourne metropolitan region from 2001 to 2015. The VMA’s Education Enhancement Program (EEP) began as a program of remediation and support for general practice registrars in 2006. Clearly defined guidelines, originally modelled on the UK’s National Association of Clinical Tutors (NACT)1 framework for managing remediation, were continually reviewed and refined. This was to ensure best practice2,3 was followed and the needs of the registrars were met.

A team of experienced medical educators managed the program under the direction of the Education Enhancement Officer (EEO). The role of the EEO is to support general practice registrars who experience difficulties, educational or otherwise, during their training. The role of the EEO and EEP has encompassed assessing and monitoring registrars’ progress, and formulating and delivering individually tailored educational interventions to underperforming registrars. In 2011, the VMA’s Diagnostic and Assessment Centre opened, making the services and expertise of the EEP available to other general practice training providers and organisations, such as the Australian Health Professionals Regulation Agency (AHPRA) and medical defence organisations (MDOs).

This paper, authored by the EEO, is a reflection on the work conducted by the VMA’s Diagnostic and Assessment Centre from December 2011 to June 2015. Firstly, it reflects on the formal assessment of clinical skills, which has been almost entirely with respect to non-VMA registrars whose future in general practice training rested on the determination of the assessment (ie ‘high-stakes’ assessments). Secondly, it reflects on the focused, educational interventions, mentoring and audits of practitioners who were either under investigation or where a determination had been made by AHPRA for alleged breaches of regulations. Referrals to the EEP were almost entirely from AHPRA or from the MDO assisting the practitioner.

The data

Formal assessment of clinical skills

Eleven formal assessments of clinical skills were conducted between January 2012 and June 2015 (Table 1). Only one referral was received from a private general practice for the assessment of a general practitioner’s (GP’s) communication skills, with the intention of informing an educational intervention. Sixty-four per cent of referrals were for overseas-trained doctors.

Table 1. Formal assessment of clinical skills between January 2012 and June 2015
Assessment typen
Referred by
Overseas trainedGender
General Practice Education and Training*Regional Training ProviderOther
Comprehensive 7 2 5 5 3 male and 4 female
Reassessment 1 1 1 female
Focused 3 2 1§ 2 2 male, 1 female
Total 11 2 (18%) 8 (73%) 1 (9%) 7 (64%) 5 male and 6 female (45% and 55%)
*The organisation governing general practice training under Australian General Practice Training until the end of 2014
Registrar reassessed seven months after the initial assessment and following implementation of a remediation plan
Review of video-recorded consultations
§Assessment of communication skills (referred by the practice where the doctor was working)

Ten assessments were for registrars and seven of the assessments were considered ‘high stakes’. One registrar was reassessed following the completion of a remedial plan. All assessments encompassed:

  • application of knowledge
  • history-taking and physical examination
  • the ability to synthesise information
  • reasoning
  • development and implementation of a management plan.

The seven comprehensive (high-stakes) assessments also addressed knowledge base, insight, receptiveness to feedback and capability to improve. Following each assessment, a comprehensive report was written, which commented on each of these areas and provided specific recommendations with respect to a remediation plan (Box 1). Of the nine registrars who were assessed, three left general practice training, three received educational interventions and are progressing in their training, and three are presently waiting for a suitable practice to be found and a remediation program to be put into place.

Box 1. Comprehensive assessment of clinical skills*
Prior to the assessment
  • Referral received with outline of concerns
  • Background information obtained (eg training to date, issues as perceived by the medical educators and supervisors, relevant reports)
  • Appraisal of the registrar’s recently video-recorded consultations (if available)
  • Case selection and briefing of simulated patients and medical educators assisting with the assessment
The assessment proper
  • Discussion with the registrar (eg their perspective of the issues)
  • Completion of validated and benchmarked written papers (eg multiple choice questions and key feature problems)
  • First round of role-play of clinical scenarios (6–10 cases), followed by feedback
  • Second round of role-play of clinical scenarios (6–10 cases), followed by feedback§
  • Overall feedback to the registrar and recommendations
Report includes
  • Results of written papers
  • Detailed information on the registrar’s performance in the role-plays
  • Summary of the registrar’s strengths and weaknesses
  • Determination as to the registrar’s:
  • level of competence with respect to knowledge, clinical skills and overall (benchmarked)
  • remediability
  • Specific recommendations with respect to a remediation plan
*A typical assessment is conducted over three days or four days if appraisal of video-recorded consultations is also conducted. This does not include time for preparation and report writing.
Consent obtained from the registrar and simulated patients to video record. All simulated patients and medical educators sign confidentiality agreements.
Role-plays are observed by the Education Enhancement Officer (EEO) and video-recorded to facilitate feedback. Brief feedback is provided by simulated patients and medical educators. More detailed feedback is provided by the EEO.
§The purpose of the second round is to see whether feedback given can be implemented and to determine remediability.

Focused educational interventions, mentoring and audits

Forty-nine practitioners, 40 GPs and nine medical specialists or allied health professionals sought the services of the EEP (Tables 2, 3). Forty-nine per cent were overseas-trained doctors. The requests were overwhelmingly for focused educational interventions (n = 43; 88%) as opposed to requests for mentoring and audit of medical records. The interventions varied with respect to length (one to four hours) and number (one to six sessions). Each intervention was conducted face to face and was individually tailored to address the areas that had been deemed by AHPRA to have been breached. Most of the practitioners were male (82%). For most practitioners (65%), AHPRA had completed the investigation and made a final determination (Table 2). The working experience of the practitioners varied from those who were still in hospital training to those who were 70 years of age or older and still practising. The five principal topic areas that were addressed were:

  • clinical skills – 33%
  • prescribing – 24%
  • documentation – 18%
  • patient boundaries – 17%
  • ethical and legal – 4%.
Table 2. Focused educational interventions, mentoring and audits between December 2011 and June 2015
Intervention typenPrimary referring sourceOverseas trainedGenderGeneral practitioner
Australian Health Professionals Regulation AgencyMedical defence organisationsOther
Educational 40 23 16 1* 18 34 male and 6 female 31
Mentoring 2 2 1 1 male and 1 female 2
Audit 3 3 3 3 male 3
Education and mentoring 1 1 1 1 female 1
Education and audit of records 2 2 1 2 male 2
Mentoring and audit of records 1 1 1 female 1
Total 49 32 (65%) 16 (33%) 1(2%) 24 (49%) 40 male and 9 female
(82% and 18%)
40 (82%)
*Self-referred
The remaining nine comprised five specialists (three physicians, one surgeon and one psychiatrist), one specialist training program (anaesthetics), and three allied health professionals (one psychologist, one chiropractor, one osteopath)

Within these topic areas, prescribing drugs of addiction was the topic most frequently addressed (23%), followed by appropriate documentation (18%). Equal third were informed consent, communication skills and professional boundaries (11% each). Communication skills were more frequently addressed than indicated in Table 3 because they were also addressed in other areas such as informed consent, managing drug-seeking patients and maintaining professional boundaries (Table 3).

Table 3.Topic areas for the educational interventions between December 2011 and June 2015
Educational contentNumber of practitioners

Clinical skills
(n = 23; 33%)

Informed consent 8
Communication skills* 8
Patient assessment 2
Working with patients 2
Clinical decision making 1
Patient management 1
Keeping up to date 1
Prescribing
(n = 17; 24%)
Drugs of addiction, pain management 16
Off-label 1
Appropriate documentation (n = 13; 18%) 13
Boundaries
(n = 12; 17%)
Professional 8
Sexual 4

Ethical and legal
(n = 3; 4%)

Confidentiality 1
Privacy 1
Certificates 1
Other (n = 3; 4%) 3
Total 71
*Focused on ‘communicating effectively with patients, carers and relatives’. With other topics, such as informed consent, the importance of communication skills is highlighted as well as how they can be used to improve outcomes.
Areas covered include: regulations with respect to drugs of addiction, understanding and managing the behaviour of drug seeking patients, and principles and practice of chronic pain management.

Discussion

When the services of the EEP were first made available externally, it was unclear what the level of demand would be. Anecdotal evidence indicated that a need existed for objective and independent assessment of clinical skills. Requests for assessment have been almost entirely for registrars, and mostly from other RTPs.

All of the assessments conducted on registrars were ‘high stakes’. Therefore, it was important to follow best practice4,5 and for processes to have validity and reliability.6,7 Careful preparation, skilled medical educators, well-trained simulated patients8 and standards for comparison5,6 were found to be necessary if the final determination was to be defensible. Processes have been continually reviewed to ensure ongoing rigour. Because of all these factors, every assessment has been labour and time intensive.

Each assessment must be individually tailored. Methods of assessment were chosen9,10 according to the referrer’s concerns, and to enable the registrar to fully demonstrate their abilities. Background information from the referrer assisted considerably in this respect. It was also important for the registrar to fully engage so they felt they had adequately demonstrated their abilities. Engagement11 also allowed the assessor to gain a better understanding of the registrar’s difficulties, including any cultural dimensions, clinical reasoning and degree of insight. However, an adequate determination of professionalism12 could not be made, given that the situation was contrived and the person under assessment knew that their behaviour was being scrutinised. All registrars cooperated fully with the assessment.

In contrast to the number of requests for formal assessments, there has been a big demand for educational interventions, including a small number from New South Wales and Queensland. What has been discovered is very few providers offer this service. As the availability of the EEP has become more widely known, the demand has grown significantly. Therefore, there appears to be a distinct need for more providers for educational interventions as well as for audits and mentoring.

Annual reports from AHPRA only provide basic data regarding doctors who have undergone investigation.13 These reports do not provide any information about the types of infringements and determinations made. Limited data are available regarding overseas-trained doctors.14 The data presented here cannot therefore be taken as being representative. The lack of more specific data from AHPRA makes it difficult for planning and program development.

Similarly to the formal assessments, engaging individuals who were mandated to undergo an intervention has indeed been a challenge but was very important for learning outcomes to be attained. Engagement was achieved by following the principles of effective feedback15,16 – gaining an understanding of the practitioner, listening to their perspective and providing constructive feedback – thus making their experience meaningful and valuable.

The EEP has been able to provide independent assessments of clinical skills, and focused educational and other interventions to medical and other health practitioners. As a result, the EEP has gained considerable expertise in the assessment and management of doctors experiencing difficulties. Other potential areas for the EEP include:

  • Providing assistance to practitioners returning to the workplace after a prolonged period of absence (eg illness or maternity leave). This need has already been identified and successfully addressed with registrars who were returning to the VMA’s training program after a substantial period of absence. There have been similar experiences overseas.17–19
  • Assessment of underperforming GPs in private practice18 and developing learning plans for them.
  • Providing continuing professional development programs on communication skills and prescribing drugs of addiction. Unsurprisingly, these are the two main areas where the EEP has assisted practitioners, as evident in bulletins from AHPRA and MDOs, which repeatedly highlight that poor communication is one of the prime reasons for complaints against doctors. With drugs of addiction, many of the doctors were reported to AHPRA because of their lack of familiarity with the regulations and inability to manage drug-seeking behaviour, which has become a complex and difficult area.
  • Creating programs for supporting overseas-trained doctors who face significant difficulties in the workplace.20

The EEP has assisted practitioners to improve their clinical skills and also potentially prevent them from making similar infringements in the future. The EEP will continue to operate in the new training landscape under the banner of Eastern Victoria General Practice Training. The aim is to continually evaluate and improve processes of assessment, and explore educational interventions that are innovative and suited to different levels of experience and need. Research into the particular needs of overseas trained doctors, as well as the long-term outcomes for doctors who have been through the program, would also be useful for the ongoing development of the EEP.

Author

George Zaharias MBBS, MFM, FRACGP, Education Enhancement Program Director, Eastern Victoria General Practice Training, Hawthorn, VIC, and previously Education Enhancement Officer with Victorian Metropolitan Alliance General Practice Training, VIC. zaharias@optushome.com.au

Competing interests: None.
Provenance and peer review: Not commissioned, externally peer reviewed.

References

  1. National Association for Clinical Tutors. Managing trainees in difficulty. Practical advice for educational and clinical supervisors. Version Milton Keynes, Buckinghamshire: NACT, 2013.
  2. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA 2002;287(2):226–35.
  3. National Clinical Assessment Service. Handling performance concerns in primary care. London: NCAS, 2010.
  4. van der Vleuten CP, Schuwirth LW, Scheele F, Driessen EW, Hodges B. The assessment of professional competence: Building blocks for theory development. Best Pract Res Clin Obstet Gynaecol 2010;24(6):703–19
  5. Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011;33(3):206–14.
  6. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: Theory and application. Am J Med 2006;119(2):166.e7–16.
  7. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001;357(9260):945–49.
  8. Norcini J, Boulet J. Methodological issues in the use of standardized patients for assessment. Teac Learn Med 2003;15(4):293–97.
  9. Epstein RM. Assessment in medical education. N Engl J Med 2007;356(4):387–96.
  10. Norcini J, McKinley DW. Assessment methods in medical education. Teaching and Teach Edu 2007;23(3):239–50.
  11. Norcini J. The power of feedback. Med Educ 2010;44(1):16–17.
  12. van Mook WN, Gorter SL, O’Sullivan H, Wass V, Schuwirth LW, van der Vleuten CP. Approaches to professional behaviour assessment: Tools in the professionalism toolbox. Eur J Intern Med 2009;20:e153–57.
  13. Medical Board of Australia. Medical regulation at work In Australia, 2013/14. Regulating medical practitioners in the National Registration and Accreditation Scheme. Sydney: AHPRA, 2014. Available at www.ahpra.gov.au/Publications/Corporate-publications/Annual-reports.aspx [Accessed July 2015].
  14. Elkin K, Spittal MJ, Studdert DM. Risks of complaints and adverse disciplinary findings against international medical graduates in Victoria and Western Australia. Med J Aust 2012;197(8):448–52.
  15. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29:855–71.
  16. Vickery AW, Lake FR. Teaching on the run tips 10: Giving feedback. Med J Aust 183(5):267–68.
  17. Gorman A. For doctors who take a break from practice, coming back can be tough. Kaiser Health News. 15 June 2015. Available at http://khn.org/news/for-doctors-who-take-a-break-from-practice-coming-back-can-be-tough [Accessed July 2015].
  18. Academy of Medical Royal Colleges. Return to practice guidance. London: Academy of Medical Royal Colleges, 2012. Available at www.aomrc.org.uk/publications/reports-a-guidance?view=docman [Accessed July 2105].
  19. National Clinical Assessment Service. Back on track. Restoring doctors and dentists to safe professional practice. London: NCAS, 2006.
  20. Australian Health Practitioner Regulation Agency. Submission to the House of Representatives Standing Committee on Health and Aging. Inquiry into registration processes and support for overseas trained doctors. Sydney: AHPRA, 2011. Available at www.aph.gov.au/Parliamentary_Business/Committees/House_of_representatives_Committees?url=haa/overseasdoctors/subs/sub51.pdf [Accessed 1 December 2015].

Correspondence afp@racgp.org.au

Yes     No

Declaration of competing interests *

Yes No

Additional Author (remove)

Yes No

    

 

 

 

 

Competing Interests: 

Your comment is being submitted, please wait

 

Download citation in RIS format (EndNote, Zotero, RefMan, RefWorks)

Download citation in BIBTEX format (RefMan)

Download citation in REFER format (EndNote, Zotero, RefMan, RefWorks)

For more information see Wikipedia: Comparison of reference management software
Advertisement loading...

Advertisement

Type

Professional