RACGP
Australian Family Physician
Australian Family Physician

Advertising

Volume 45, Issue 12, December 2016

Adding random case analysis to direct observation (ARCADO) – Updating the external clinical teaching visit to improve general practice registrar assessments

Jennifer Fry Gerard Ingham Bernadette Ward
Download article
Cite this article    BIBTEX    REFER    RIS

Background
In response to the advent of competency-based training and the increase in the number of general practice registrars, the Australian general practice education community is seeking valid, reliable, time-efficient and cost-efficient tools to assess registrars. Despite the central role of the external clinical teaching visit (ECTV) in formative assessment of general practice registrars, the ECTV has been an infrequent subject of research or evaluation.
Objective
The objective of this article is to report on the development of a new approach to ECTV that adds random case analysis to direct observation of consultations – ARCADO ECTV.
Discussion
ARCADO ECTV is a flexible, acceptable and time-efficient formative assessment. The two assessment approaches in the ARCADO ECTV provide complementary insights into the registrar’s performance. At least three observed consultations are required to ensure adequate assessment of communications skills. Medical records need to be of recent consultations. There is scope for development of the ARCADO ECTV as a summative assessment.

The external clinical teaching visit (ECTV) is unique to Australia. It was developed in the mid-1980s in Western Australia to augment in-practice teaching.1 It was soon adopted by all states and territories.2 An evaluation of ECTV by The Royal Australian College of General Practitioners (RACGP) in 1994 identified a ‘strong element of luck’ in the clinical and educational content of ECTVs so a manual was developed to reduce variability.3 The trend towards a consistent approach was reversed by regionalisation of general practice training in 2001. A survey in 2005 found that many regional training providers (now called regional training organisations) were not using an ECTV manual or providing training for visiting medical educators and general practice supervisors.4

The ECTV has been an infrequent subject of research or evaluation.5–7 Most general practice registrars have five ECTVs during their training, which is a significant investment of time and money. Typically, the ECTV involves a visitor (either a medical educator or general practice supervisor from another practice) observing a registrar consulting. Other methods of assessment previously described within the ECTV include review of video-recorded consultations or discussions with the supervisor about the registrar’s progress.3

The ECTV is a formative assessment. These assessments occur during training and are low-stakes assessment for learning. Formative assessments aim to generate powerful learning experiences from feedback and identify learning needs. In contrast, summative assessments are high-stakes assessments of learning that typically occur at the end of training.

The adding random case analysis to direct observation (ARCADO) ECTV was developed from a theoretical basis and progressed through the synthesis of findings of several research projects and consultation with medical education experts.

Theoretical background

Workplace-based assessments examine what a practitioner does in the real world rather than what they demonstrate they can do in an artificial exam centre assessment setting. They are considered to be best-practice in summative and formative assessments of clinical competence.8,9 It is difficult to have multiple assessors in the workplace or to control the content of what walks through the door and this can lead to problems with reliability and validity of workplace-based assessments.10 To overcome this, conducting multiple workplace-based assessments and using multiple modes of assessment is recommended.11

With this pedagogical background in mind, the authors proposed adding random case analysis (RCA) to direct observation (DO) of the consultation in a formative assessment ECTV. In RCA, a selection of clinical records is chosen by the assessor and discussed with the learner. A framework for the use of RCA in an Australian context12 was recently developed. This framework promotes discussion between the assessor and learner based on the five domains of general practice from the RACGP’s curriculum, and further consideration of four contextual influences – doctor, patient, problem and system. RCA was found to be effective in uncovering registrars’ ‘unknown unknowns’ and patient safety concerns.13

Privacy and RCA

Before conducting the first ARCADO assessment, it was confirmed that state-based and territory-based health records legislation allowed the visiting medical educator or general practice supervisor to access patient records. For example, in Victoria, the Health Records Act 2001 accepts such access when:

the use or disclosure (is) for the purpose of … monitoring, improvement or evaluation of health services; or training provided by a health service provider to employees or persons working with the organisation …and (when) it is impracticable for the organisation to seek the individual’s consent to the use or disclosure.

It is important that the ‘collection statement’ of any practice involved in ARCADO includes the prospect of a visiting medical educator/general practice supervisor from another practice accessing patient medical records for such purposes.

Initial development

An existing ECTV manual was used as the basis for the initial version of ARCADO. This was reviewed by an expert reference group that included a senior medical educator at the Australian College of Rural and Remote Medicine (ACRRM), a university academic with expertise in teaching and assessment, and an experienced medical educator who is also an RACGP examiner. Following modifications, the ARCADO ECTV was piloted. This informed the development of an ARCADO training session for medical educators.

Trial and analysis

To explore ARCADO ECTV, a qualitative research project was undertaken involving 10 ARCADO ECTVs.14 Semi-structured interviews were conducted with the participating medical educators and registrars, and the data were analysed thematically. The medical educators and registrars believed ARCADO ECTV was flexible, time-efficient and acceptable. Medical educators appreciated the complementary insights into the registrars’ abilities provided by the two assessment approaches. The medical educators also considered that ARCADO ECTV provided a more valid or authentic assessment by overcoming some of the artificiality of an assessment based solely on observation.

The addition of RCA to direct observation in ARCADO results in fewer consultations being observed (Box 1). Some research participants were concerned that this risked inadequate assessment of communication skills. With the RCA component of the assessment, it was noted that there was less value in reviewing records that were not recent consultations as the general practice registrar could not recall the clinical reasoning involved in clinical decisions. When the medical records had little content, there was risk of embellishment by the general practice registrar of the detail of the consultation.

Review

Further review by the expert reference group resulted in final modifications to the ARCADO ECTV. Changes included requirements that:

  • at least three consultations be observed
  • the medical records be recent consultations – consultations recent enough to enable registrar recall of the detail and reasoning.

The ARCADO ECTV manual was expanded to include a guide for the registrar, visiting general practice supervisor/medical educator and the practice.

Key features of the final ARCADO

The ARCADO tool is used within a three-hour ECTV (Box 1). Three patient consultations are conducted by the registrar and observed by the visiting medical educator/general practice supervisor. The observed consultations are placed at the start of the ECTV as there is an opportunity to book extra patients later in the visit if the observed encounters are brief or if a patient fails to attend. The booking schedule allows time between consultations for discussion and feedback, and there is an opportunity to commence RCA if there is a delay in a patient attending.

Box 1. Suggested three-hour ARCADO ECTV scheduling

15 minutes

Registrar and visitor discussion (no patients booked)

30 minutes

Consultation – patient one and feedback

30 minutes

Consultation – patient two and feedback

30 minutes

Consultation – patient three and feedback

45 minutes

Review of medical records (random case analysis)

20 minutes

Complete assessment and update learning plan

10 minutes

Feedback from/with supervisor

The RCA component involves the visiting medical educator/general practice supervisor selecting recent consultation records at random, which are then reviewed by and discussed with the general practice registrar. The discussion with the registrar is based on the RACGP’s domains of general practice and uses ‘what if’ questions to explore beyond the content of the record – ‘What if the problem/doctor/person/system was different?’.12 The number of records reviewed varies depending upon the time available after consultation observation and the complexity of the records reviewed. During the trial, between two and seven records were reviewed.

The visiting medical educator/general practice supervisor concludes the ECTV with feedback to the registrar. The feedback is based on all consultations observed and records analysed. Direct observation and RCA are not reviewed separately. The ARCADO ECTV tool guides the formative assessment and encourages comments based broadly on the five domains of general practice from the RACGP’s curriculum. An overall assessment is provided to the registrar as to whether they are at the standard expected for their current stage of training. The formative assessment outcome does not determine whether the registrar can advance  through training, but provides the registrar with feedback regarding how they are progressing towards becoming a competent GP.

The visiting medical educator/general practice supervisor assists registrars in updating their learning plan to reflect any new learning needs and activities. Prior to leaving the practice, the visiting medical educator/general practice supervisor has a pre-scheduled, short feedback session with the practice supervisor. This provides an opportunity for the visiting medical educator/general practice supervisor to report on ECTV findings and seek the supervisor’s perception of registrar performance. A formative assessment report is completed by the visiting medical educator/general practice supervisor and is available to the registrar, supervisor and regional training organisation within one week of the visit.

The future

The ARCADO ECTV has yet to be widely used by any of the newly created regional training organisations. With greater experience, there are likely to be further refinements to the ARCADO ECTV and its adaptation to ensure it fits within the formative assessment blueprint of the individual regional training organisations.15 Although the ECTV is unique to Australia and the ARCADO tool is underpinned by the RACGP’s domains of general practice, the ARCADO ECTV could be used in the formative assessment of participants in other vocational training programs in Australia and internationally.

There is potential to develop the ARCADO ECTV beyond formative assessment and use it as a summative or medium-stakes, in-training ‘hurdle’ assessment.11,16 For example, it could be adapted for use by regional training organisations to determine if a registrar is ready to sit the RACGP’s fellowship exams. The common problem of unexpected exam failure is distressing for registrars and adds cost to the Australian general practice training program when it results in extension of the registrars’ training time. Research comparing the outcomes of an ARCADO ECTV with RACGP’s fellowship exam outcomes would investigate whether the ARCADO ECTV is a valid and reliable summative assessment. A reliable and valid pre-exam assessment could enable better targeting of regional training organisation educational resources to registrars in need of further assistance.

The ARCADO ECTV handbook is available from the lead author.

Authors

Gerard Ingham MBBS, FRACGP, DRANZCOG, Grad Cert HPE, General Practice Supervisor, Springs Medical Centre, Daylesford, Vic. drgingham@gmail.com

Jennifer Fry BAppSc(OccTher), MAppSc(OccHlthPrac), Manager Health Systems, Murray Primary Health Network, Bendigo, Vic

Bernadette Ward PhD, MHSc, BHSc, Senior Research Fellow, Monash Rural Health, Office of Research, Monash University, Bendigo, Vic

Competing interests: None.

Provenance and peer review: Not commissioned, externally peer reviewed.

Acknowledgements

The authors of this article would like to acknowledge doctors Sandra Mendel, Ewa Piejko and Natalie Radomski.


References
  1. Hays R, Peterson L. The external clinical teaching visit: A descriptive evaluation. Townsville, Qld: General Practice and Rural Health Unit, North Queensland Clinical School, 1994. Search PubMed
  2. Black F. External clinical teacher visits. A personal viewpoint. Aust Fam Physician 1996;25(10):1588–89. Search PubMed
  3. Black F, Faux S. ECT manual: Becoming an external clinical teacher – The Royal Australian College of General Practitioners Training Program. Melbourne: RACGP, 1996. Search PubMed
  4. Kefalas B, Usherwood T, Knight A. External clinical teaching visits in the regionalised environment. Aust Fam Physician 2005;34(Suppl 1):10–12. Search PubMed
  5. Starling C, Regan C, Lee C, Morgan S, Magin P. How useful are clinical teaching visits (CTVs) to registrars and supervisors. Melbourne: General Practice Education & Training, 2012. Search PubMed
  6. Fraser J. Registrar clinical teaching visits – Evaluation of an assessment tool. Aust Fam Physician 2007;36(12):1070–72. Search PubMed
  7. Lawson C, Schattner P. Evaluation of external clinical teaching visits (ECTVs) in The Royal Australian College of General Practitioners Training Program. Focus on Health Professional Education 2000;2(3):11–17. Search PubMed
  8. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29(9):855–71. Search PubMed
  9. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65(9 Suppl):S63–67. Search PubMed
  10. Norcini JJ. Current perspectives in assessment: The assessment of performance at work. Med Educ 2005;39(9):880–89. Search PubMed
  11. van der Vleuten CP, Schuwirth LW, Driessen EW, Govaerts MJ, Heeneman S. 12 Tips for programmatic assessment. Med Teach 2014:1–6. Search PubMed
  12. Morgan S, Ingham G. Random case analysis – A new framework for Australian general practice training. Aust Fam Physician 2013;42(1–2):69–73. Search PubMed
  13. Morgan S, Ingham G, Kinsman L, Fry J. Clinical supervision using random case analysis in general practice training. Educ Prim Care 2015;26(1):40–46. Search PubMed
  14. Ingham G, Fry J, Morgan S, Ward B. ARCADO – Adding random case analysis to direct observation in workplace-based formative assessment of general practice registrars. BMC Med Educ 2015;15:218. Search PubMed
  15. Hamdy H. Blueprinting for the assessment of healthcare professionals. Clin Teach 1996;3:175–79. Search PubMed
  16. Hays R, Wellard R. In-training assessment in postgraduate training for general practice. Med Educ 1998;32(5):507–13. Search PubMed
Download article PDF

Advertising

Australian Family Physician RACGP

Printed from Australian Family Physician - https://www.racgp.org.au/afp/2016/december/adding-random-case-analysis-to-direct-observat-2
© The Australian College of General Practitioners www.racgp.org.au