RACGP
Australian Family Physician
Australian Family Physician

Advertising

Volume 44, Issue 4, April 2015

Are GP supervisors confident they can assess registrar competence and safety, and what methods do they use?

Leigh Kinsman Jennifer Fry Gerard Ingham Simon Morgan
Download article
Cite this article    BIBTEX    REFER    RIS

Background
The new Royal Australian College of General Practitioners’ vocational training standards emphasise patient safety and matching the level of supervision to registrar competence.
Discussion
Supervisors reported confidence in their ability to assess their registrars’ competence, suggesting readiness for outcomes-based standards. The low frequency of using supervision methods considered more effective in targeting registrars’ ‘unknown un-knowns’ and the reduction in confidence with greater experiences as a supervisor raises concerns about the accuracy of this self-assessment.
Results
Of 84 supervisors, 92.8% expressed overall confidence in their ability to assess their registrars’ competence and 10.7% were totally confident. Consultation observation and audit techniques were infrequently used in their assessments.

Supervision has been defined as ‘the provision of guidance and feedback on matters of personal, professional and educational development in the context of a trainee’s experience of providing safe and appropriate patient care’.1 It comprises two distinct but overlapping elements – facilitating learning (educational supervision), and monitoring quality-of-care and patient safety (clinical supervision).

General practice training in Australia has traditionally been based on an apprenticeship model, where registrars see patients independently but under the supervision of accredited general practitioner (GP) supervisors. The new Royal Australian College of General Practice (RACGP) Vocational Training Standards (2013) are outcome based and emphasise the importance of matching supervision of registrars to their competence in order to ensure patient safety.2

The GP supervisor can use a variety of methods to undertake clinical supervision and assess competence in the practice (eg direct observation, video consultation review, random case analysis and patient feedback).3 Each method has particular strengths as a tool to assess competence and monitor patient safety. Furthermore, competence extends well beyond clinical skills, and assessment needs to extend to all facets of the GP’s role, as described in the five domains of general practice in the RACGP curriculum.4

The degree of GP supervisors’ confidence in their ability to assess their registrars’ competence and safety with patients has not previously been described. Similarly, the frequency of uptake of various clinical supervision methods and how this relates to supervisor confidence has not been explored in the Australian general practice setting. We sought to describe the confidence of GP supervisors in their understanding of their registrars’ competence and patient safety, and the methods they use to assess this.

Methods

This was a cross-sectional analysis of GP supervisors. All GP supervisors from one regional training provider (RTP) attending one of three annual education workshops in 2013 were invited to participate. The workshop was compulsory for one supervisor from every practice within the RTP.

GP supervisors were asked to complete a questionnaire before the workshop and consent was gained for participation in the research. To ensure participant anonymity, surveys were de-identified and coded.

Survey items included supervisor demographics and experience, and practice characteristics. Supervisors were asked about the frequency of use of selected clinical supervision methods and asked to describe other supervision methods used. Supervisors were asked to rate their confidence in their understanding of their most recent registrar’s competence across the five RACGP domains, clinical decision making and overall safe management of patients. This was done using a 5-point Likert scale. A score of 3 or 4 indicated confident or very confident and a score of 5 indicated totally confident.

We measured the relationships between supervisors’ self-reported overall confidence in their ability to assess their registrars’ patients’ safety with supervisor gender (t-test), country of training (ANOVA), years of general practice experience (Pearson correlation) and years working as a supervisor (Pearson correlation). Data analysis was completed using SPSS version 21.

Ethics approval for the study was obtained from the Monash Human Research Ethics Committee (CF13/663 – 2013000289).

Results

Participants

Ninety-three surveys were distributed and 91 surveys were returned (97.8% response rate). Those who identified that they were new supervisors or had not had registrars (n = 7) in the previous 6 months were excluded, leaving a total of 84 respondents. Overall, 81.0% of supervisors were male and the median age range was 51–60 years. The mean number of years supervising was 11.9 years, mean years practising as a GP was 22.8 years and 83.3% had other supervisors in their practice. Characteristics of the participating supervisors (n = 84) are detailed in Table 1.

Table 1. Characteristics of trainers and training practices (n = 84)

Characteristic

n (%)

Age (years)

≤40
41–50
51–60
≥61

 

10 (11.9%)
21 (25.0%)
32 (38.1%)
21 (25.0%)

Gender

Male
Female

 

68 (81.0%)
16 (19.0%)

Number of years as GP (mean 22.8 years)

0–9
10–19
20–29
30–39
≥40

 8 (9.5%)
18 (21.4%)
33 (39.3%)
22 (26.2%)
3 (3.6%)

Number of years as supervisor (mean 11.9 years)

0–5
6–10
11–20
≥21

 22 (26.2%)
18 (21.4%)
35 (41.7%)
9 (10.7%)

Location of primary medical degree

Australia
Other
Australia and other

 61 (72.6%)
22 (26.2%)
1 (1.2%)

Number of sessions per week (mean 7.8 sessions)

1–5
6–10
≥10

 11 (13.1%)
65 (77.4%)
7 (7.0%)*

Other supervisors in practice

0
1–3
4–6

 14 (16.7%)
57 (67.9%)
11 (13.1%)*

Practice location using the Australian Standard Geographical Classification – Remoteness Area (ASGC-RA) system(16)

RA2 – inner regional
RA3 – outer regional
RA4 - remote
RA5 – very remote
Both RA2 and RA3

41 (48.8%)
34 (40.5%)
8 (9.5%)
0 (0.0%)
1 (1.2%)

Currently teach other students

Yes
No

 77 (91.7%)
7 (8.3%)

Teach registrars in first 12 months of training (GPT1 and GPT2)

Yes
No

 73 (86.9%)
11 (13.1%)

*Data does not total 100% as missing data from questions not answered by participants were excluded

Supervisor confidence

Supervisors rated a high level of confidence in their ability to assess the competence of registrars in professionalism, communication skills and whole-person care, clinical knowledge and skills, clinical decision making, and legal and organisational issues (information management and records). There was less confidence in the ability to assess registrar competence in population health practice (public health and screening) with 22.6% of supervisors self-rating as less than confident and only 3.6% totally confident (Figure 1).

92.8% of supervisors expressed overall confidence (confident to totally confident) in their ability to assess their registrars’ patients’ safety and 10.7% stated that they were totally confident.

 Supervisor1
Figure 1. Supervisors’ self-rated confidence in assessing their registrars’ competence
Overall confidence and clarifying questions related to domains of general practice and clinical reasoning
(n = 84)

Supervision methods

Opportunistic case discussion was the most frequently used supervision method and 95.2% of supervisors reported using this method on a weekly basis. Other frequently used supervision methods (at least monthly) were formal teaching sessions (90.4%), staff feedback (66.7%) and review of medical records (66.7%).

Direct observation of consultations was used at least monthly by only one-third of trainers (32.1%) and 9.5% of supervisors stated they never used it. Video observation was used even less frequently and 38.1% stated that it was never used. Nearly half the trainers (42.9%) never used pathology audit (Table 2).

Table 2. Frequency of supervision method conducted (n = 84)

Supervision method

Weekly
n (%)

Monthly
n (%)

1–2 times
each term
n (%)

Never
n (%)

Direct consultation observation

11 (13.1%)

16 (19.0%)

47 (56.0%)

8 (9.5%)*

Review of videotaped consultations

2 (2.4%)

9 (10.7%)

41 (48.8%)

32 (38.1%)

Medical record review

20 (23.8%)

36 (42.9%)

18 (21.4%)

9 (10.7%)*

Review patient feedback

21 (25.0%)

25 (29.8%)

27 (32.1%)

10 (11.9%)*

Review staff feedback

36 (42.9%)

20 (23.8%)

19 (22.6%)

7 (8.3%)*

Formal teaching

59 (70.2%)

17 (20.2%)

5 (6.0%)

2 (2.4%)*

Opportunistic teaching

80 (95.2%)

3 (3.6%)

1 (1.2%)

0 (0.0%)

Audit pathology

11 (13.1%)

20 (23.8%)

17 (20.2%)

36 (42.9%)

Audit referrals

8 (9.5%)

9 (10.7%)

28 (33.3%)

39 (46.4%)

*Data does not total 100% as missing data from questions not answered by participants excluded

Supervisors also reported using a number of other supervision methods including critical incident reviews, use of simulated patients, procedural training, joint consulting and review of hospital admissions (Table 3).

Table 3. Other supervisory methods described by participants
  • Critical incident review
  • Retrospective notes review of registrar management prior to admission to hospital
  • Review of inpatient care
  • Hospital nursing staff feedback
  • Simulated patients
  • Procedural teaching
  • Role plays
  • Case discussion of supervisor’s patients’ feedback from registrars
  • Joint consulting
  • Performing short cases for exam preparation
  • Procedural assessment – anaesthesia, accident and emergency skills

The methods listed are a selection of the free text responses

Associations

There was a statistically significant negative association between years supervising and confidence in assessing registrars’ patient safety (r = –0.272; P = 0.015). There was no association with supervisor gender (P = 0.105), country of training (P = 0.170) or general practice years of experience (r = -0.201; P = 0.071).

Discussion

We have described the self-rated confidence of GP supervisors in their ability to assess registrar competence and patient safety, and the frequency of clinical supervision methods used in this assessment in a single Australian general practice training provider.

Patient safety is the cornerstone of quality care, and monitoring patient safety the key aspect of clinical supervision.5,6 Australian GPs self-report significant errors involving a threat to patient wellbeing in one in every 500 patients seen.7 There is a paucity of evidence on the impact of training in general practice on patient safety.8 There is no evidence for a greater frequency of error in general practice registrars, compared with established GPs, although higher rates of error have been reported in other disciplines for doctors in training.9

We found that, overall, supervisors are very confident in their ability to assess their registrars’ competence and patient safety. Confidence in ability to complete a task is termed ‘self-efficacy’ in the psychology literature.10 Self-efficacy has been shown to influence motivation and to undertake new tasks. It may be that supervisors who are more confident in their ability to assess registrars (high self-efficacy) will be less likely to see the need to supervise in a different manner to their current practice.

We found that with greater experience, supervisor confidence in their ability to assess their registrar decreased. This is a novel and counterintuitive result. Interpretation of this finding is difficult but might suggest a more discriminating and discerning approach with greater experience. Less experienced supervisors may be unaware of the inadequacy of some supervision methods until they encounter a significantly less competent registrar.

Supervisors expressed a high level of confidence in their ability to assess registrar competence across all five RACGP domains, but less so in population health practice. The uptake and use of the RACGP guidelines on preventive care by GPs has not been evaluated to date.11 It is possible that supervisors are not confident of their own competence in population health practice. Other explanations for the finding include supervisors having limited perceived methods to assess competence in the public health domain or supervisors misunderstanding the term.

Supervisors reported a high frequency of engagement (opportunistic case discussions and face-to-face teaching) with their registrars. This is likely to reflect the teaching requirements of the training program and RACGP standards, as well as the close relationship that exists in the apprenticeship model.12

We also identified a low frequency of the use of observation and audit techniques by the supervisors, which is a critical finding. There are known barriers to the use of direct observation of consultations and video consultation analysis,13 including the significant effort required, technology problems or registrar reluctance. However, it is known that supervision activities based on observation and audit, that is, review of medical records and audit of referral letters and pathology test ordering, are more likely to detect areas of unconscious incompetence, so called unknown unknowns.14 Audit techniques have been described as valuable for registrar patient safety.6

We believe, therefore, that we have identified a paradox in the clinical supervision of GP registrars – while supervisors highly rate their confidence in awareness of their registrars patient safety, they infrequently use specific supervision methods best suited to detect unconscious incompetence and therefore unknown knowledge and skills gaps.

Strengths and limitations

We had an excellent response rate of 97.8%. This is an exceptional response rate for surveys of GPs.15

The limitation of this study is that the sample was taken from only regional/rural GP supervisors and from one RTP, and hence may not be generalisable to other regions or metropolitan supervisors. Frequency of use of supervision methods is likely to reflect past training and emphasis by the specific RTP. The study is exploratory in nature and is not able to make associations between particular supervisor activities and confidence.

Self-assessment is known to be prone to bias, leading the incompetent to overestimate their competence and those with true ability to relatively underestimate it.16 This may be a factor in the overall confidence reported by supervisors and the decline in confidence with experience.

Implications for general practice training

Supervisors report high levels of confidence in their ability to assess their registrars’ competence and patient safety, suggesting readiness for the implementation of outcome-based standards. Our study did not assess the accuracy of their assessments (ie whether supervisors are truly competent assessors). The infrequent use of observation and audit activities associated with valid and reliable assessments of registrar competence and the reduction in supervisor confidence with experience raises concerns about the accuracy of their self-assessment.

The question of whether supervisors are truly competent assessors could be explored by comparing supervisor assessment with summative examination outcomes. Qualitative research interviews with experienced supervisors could be used to both investigate the comparative value of supervision and assessment methods and uncover the reasons for the decline with experience in supervisors’ confidence in their ability to assess registrar competence and safety.

The adoption of consultation observation and audit activities may be impeded by supervisors’ high level of confidence in their current performance as assessors. Supervisor professional development activities, highlighting adverse patient outcomes that occurred because of the absence of audit and consultation observation, are recommended to create the cognitive dissonance required for behaviour change.

Competing interests: Gerard Ingham and Jennifer Fry are employees of Beyond Medical Education. Simon Morgan is employed by GP Training Valley to Coast.
Provenance and peer review: Not commissioned, externally peer reviewed.


References
  1. Kilminster S, Cottrell D, Grant J, Jolly B. AMEE Guide No. 27: Effective educational and clinical supervision. Med Teach 2007;29:2–19. Search PubMed
  2. Royal Australian College of General Practioners. Vocational Training Standards. Melbourne: RACGP, 2013. Search PubMed
  3. Hays R, Wellard R. In-training assessment in postgraduate training for general practice. Medical Educ 1998;32:507–13. Search PubMed
  4. The Royal Australian College of General Practitioners. The RACGP Curriculum for Australian General Practice 2011. Melbourne: RACGP, 2011. Search PubMed
  5. Medical Board of Australia. Good medical practice: a code of conduct for doctors in Australia. Available at www.medicalboard.gov.au/Codes-Guidelines-Policies.aspx [Accessed 17 March 2014]. Search PubMed
  6. Byrnes PD, Crawford M, Wong B. Are they safe in there? Patient safety and trainees in the practice. Aust Fam Physician 2012;41:26–29. Search PubMed
  7. Makeham MA, Kidd MR, Saltman DC, et al. The Threats to Australian Patient Safety (TAPS) study: incidence of reported errors in general practice. Med J Aust 2006;185:95–98. Search PubMed
  8. Bowie P, McKay J, Kelly M. Maximising harm reduction in early specialty training for general practice: validation of a safety checklist. BMC Fam Pract 2012;13:62. Search PubMed
  9. Young JQ, Ranji SR, Wachter RM, Lee CM, Niehaus B, Auerbach AD. ‘July Effect’: impact of the academic year-end changeover on patient outcomes: a systematic review. Ann Intern Med 2011;155:309–15. Search PubMed
  10. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev 1977;84:191–215. Search PubMed
  11. Harris MF, Bailey L, Snowdon T, et al. Developing the guidelines for preventive care – two decades of experience. Aust Fam Physician 2010;39:63–65. Search PubMed
  12. Wearne S, Dornan T, Teunissen PW, Skinner T. General practitioners as supervisors in postgraduate clinical education: an integrative review. Med Educ 2012;46:1161–73. Search PubMed
  13. Evans A, Gask L, Singleton C, Bahrami J. Teaching consultation skills: a survey of general practice trainers. Med Educ 2001;35:222–24. Search PubMed
  14. Morgan S, Ingham G. Random case analysis – a new framework for Australian general practice training. Aust Fam Physician 2013;42:69–73. Search PubMed
  15. Bonevski B, Magin P, Horton G, Foster M, Girgis A. Response rates in GP surveys – trialling two recruitment strategies. Aust Fam Physician 2011;40:427–30. Search PubMed
  16. Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol 1999;77:1121–34. Search PubMed
Download article PDF

Advertising

Australian Family Physician RACGP

Printed from Australian Family Physician - https://www.racgp.org.au/afp/2015/april/are-gp-supervisors-confident-they-can-assess-r-2
© The Australian College of General Practitioners www.racgp.org.au