Supervision has been defined as ‘the provision of guidance and feedback on matters of personal, professional and educational development in the context of a trainee’s experience of providing safe and appropriate patient care’.1 It comprises two distinct but overlapping elements – facilitating learning (educational supervision), and monitoring quality-of-care and patient safety (clinical supervision).
General practice training in Australia has traditionally been based on an apprenticeship model, where registrars see patients independently but under the supervision of accredited general practitioner (GP) supervisors. The new Royal Australian College of General Practice (RACGP) Vocational Training Standards (2013) are outcome based and emphasise the importance of matching supervision of registrars to their competence in order to ensure patient safety.2
The GP supervisor can use a variety of methods to undertake clinical supervision and assess competence in the practice (eg direct observation, video consultation review, random case analysis and patient feedback).3 Each method has particular strengths as a tool to assess competence and monitor patient safety. Furthermore, competence extends well beyond clinical skills, and assessment needs to extend to all facets of the GP’s role, as described in the five domains of general practice in the RACGP curriculum.4
The degree of GP supervisors’ confidence in their ability to assess their registrars’ competence and safety with patients has not previously been described. Similarly, the frequency of uptake of various clinical supervision methods and how this relates to supervisor confidence has not been explored in the Australian general practice setting. We sought to describe the confidence of GP supervisors in their understanding of their registrars’ competence and patient safety, and the methods they use to assess this.
Methods
This was a cross-sectional analysis of GP supervisors. All GP supervisors from one regional training provider (RTP) attending one of three annual education workshops in 2013 were invited to participate. The workshop was compulsory for one supervisor from every practice within the RTP.
GP supervisors were asked to complete a questionnaire before the workshop and consent was gained for participation in the research. To ensure participant anonymity, surveys were de-identified and coded.
Survey items included supervisor demographics and experience, and practice characteristics. Supervisors were asked about the frequency of use of selected clinical supervision methods and asked to describe other supervision methods used. Supervisors were asked to rate their confidence in their understanding of their most recent registrar’s competence across the five RACGP domains, clinical decision making and overall safe management of patients. This was done using a 5-point Likert scale. A score of 3 or 4 indicated confident or very confident and a score of 5 indicated totally confident.
We measured the relationships between supervisors’ self-reported overall confidence in their ability to assess their registrars’ patients’ safety with supervisor gender (t-test), country of training (ANOVA), years of general practice experience (Pearson correlation) and years working as a supervisor (Pearson correlation). Data analysis was completed using SPSS version 21.
Ethics approval for the study was obtained from the Monash Human Research Ethics Committee (CF13/663 – 2013000289).
Results
Participants
Ninety-three surveys were distributed and 91 surveys were returned (97.8% response rate). Those who identified that they were new supervisors or had not had registrars (n = 7) in the previous 6 months were excluded, leaving a total of 84 respondents. Overall, 81.0% of supervisors were male and the median age range was 51–60 years. The mean number of years supervising was 11.9 years, mean years practising as a GP was 22.8 years and 83.3% had other supervisors in their practice. Characteristics of the participating supervisors (n = 84) are detailed in Table 1.
Table 1. Characteristics of trainers and training practices (n = 84)
|
|
Characteristic
|
n (%)
|
|---|
|
Age (years)
|
|
≤40
41–50
51–60
≥61
|
10 (11.9%)
21 (25.0%)
32 (38.1%)
21 (25.0%)
|
|
Gender
|
|
Male
Female
|
68 (81.0%)
16 (19.0%)
|
|
Number of years as GP (mean 22.8 years)
|
|
0–9
10–19
20–29
30–39
≥40
|
8 (9.5%)
18 (21.4%)
33 (39.3%)
22 (26.2%)
3 (3.6%)
|
|
Number of years as supervisor (mean 11.9 years)
|
|
0–5
6–10
11–20
≥21
|
22 (26.2%)
18 (21.4%)
35 (41.7%)
9 (10.7%)
|
|
Location of primary medical degree
|
|
Australia
Other
Australia and other
|
61 (72.6%)
22 (26.2%)
1 (1.2%)
|
|
Number of sessions per week (mean 7.8 sessions)
|
|
1–5
6–10
≥10
|
11 (13.1%)
65 (77.4%)
7 (7.0%)*
|
|
Other supervisors in practice
|
|
0
1–3
4–6
|
14 (16.7%)
57 (67.9%)
11 (13.1%)*
|
|
Practice location using the Australian Standard Geographical Classification – Remoteness Area (ASGC-RA) system(16)
|
|
RA2 – inner regional
RA3 – outer regional
RA4 - remote
RA5 – very remote
Both RA2 and RA3
|
41 (48.8%)
34 (40.5%)
8 (9.5%)
0 (0.0%)
1 (1.2%)
|
|
Currently teach other students
|
|
Yes
No
|
77 (91.7%)
7 (8.3%)
|
|
Teach registrars in first 12 months of training (GPT1 and GPT2)
|
|
Yes
No
|
73 (86.9%)
11 (13.1%)
|
|
*Data does not total 100% as missing data from questions not answered by participants were excluded
|
Supervisor confidence
Supervisors rated a high level of confidence in their ability to assess the competence of registrars in professionalism, communication skills and whole-person care, clinical knowledge and skills, clinical decision making, and legal and organisational issues (information management and records). There was less confidence in the ability to assess registrar competence in population health practice (public health and screening) with 22.6% of supervisors self-rating as less than confident and only 3.6% totally confident (Figure 1).
92.8% of supervisors expressed overall confidence (confident to totally confident) in their ability to assess their registrars’ patients’ safety and 10.7% stated that they were totally confident.
 |
Figure 1. Supervisors’ self-rated confidence in assessing their registrars’ competence
Overall confidence and clarifying questions related to domains of general practice and clinical reasoning
(n = 84) |
Supervision methods
Opportunistic case discussion was the most frequently used supervision method and 95.2% of supervisors reported using this method on a weekly basis. Other frequently used supervision methods (at least monthly) were formal teaching sessions (90.4%), staff feedback (66.7%) and review of medical records (66.7%).
Direct observation of consultations was used at least monthly by only one-third of trainers (32.1%) and 9.5% of supervisors stated they never used it. Video observation was used even less frequently and 38.1% stated that it was never used. Nearly half the trainers (42.9%) never used pathology audit (Table 2).
Table 2. Frequency of supervision method conducted (n = 84)
|
|
Supervision method
|
Weekly
n (%)
|
Monthly
n (%)
|
1–2 times
each term
n (%)
|
Never
n (%)
|
|---|
|
Direct consultation observation
|
11 (13.1%)
|
16 (19.0%)
|
47 (56.0%)
|
8 (9.5%)*
|
|
Review of videotaped consultations
|
2 (2.4%)
|
9 (10.7%)
|
41 (48.8%)
|
32 (38.1%)
|
|
Medical record review
|
20 (23.8%)
|
36 (42.9%)
|
18 (21.4%)
|
9 (10.7%)*
|
|
Review patient feedback
|
21 (25.0%)
|
25 (29.8%)
|
27 (32.1%)
|
10 (11.9%)*
|
|
Review staff feedback
|
36 (42.9%)
|
20 (23.8%)
|
19 (22.6%)
|
7 (8.3%)*
|
|
Formal teaching
|
59 (70.2%)
|
17 (20.2%)
|
5 (6.0%)
|
2 (2.4%)*
|
|
Opportunistic teaching
|
80 (95.2%)
|
3 (3.6%)
|
1 (1.2%)
|
0 (0.0%)
|
|
Audit pathology
|
11 (13.1%)
|
20 (23.8%)
|
17 (20.2%)
|
36 (42.9%)
|
|
Audit referrals
|
8 (9.5%)
|
9 (10.7%)
|
28 (33.3%)
|
39 (46.4%)
|
|
*Data does not total 100% as missing data from questions not answered by participants excluded
|
Supervisors also reported using a number of other supervision methods including critical incident reviews, use of simulated patients, procedural training, joint consulting and review of hospital admissions (Table 3).
Table 3. Other supervisory methods described by participants
|
- Critical incident review
- Retrospective notes review of registrar management prior to admission to hospital
- Review of inpatient care
- Hospital nursing staff feedback
- Simulated patients
- Procedural teaching
- Role plays
- Case discussion of supervisor’s patients’ feedback from registrars
- Joint consulting
- Performing short cases for exam preparation
- Procedural assessment – anaesthesia, accident and emergency skills
|
|
The methods listed are a selection of the free text responses
|
Associations
There was a statistically significant negative association between years supervising and confidence in assessing registrars’ patient safety (r = –0.272; P = 0.015). There was no association with supervisor gender (P = 0.105), country of training (P = 0.170) or general practice years of experience (r = -0.201; P = 0.071).
Discussion
We have described the self-rated confidence of GP supervisors in their ability to assess registrar competence and patient safety, and the frequency of clinical supervision methods used in this assessment in a single Australian general practice training provider.
Patient safety is the cornerstone of quality care, and monitoring patient safety the key aspect of clinical supervision.5,6 Australian GPs self-report significant errors involving a threat to patient wellbeing in one in every 500 patients seen.7 There is a paucity of evidence on the impact of training in general practice on patient safety.8 There is no evidence for a greater frequency of error in general practice registrars, compared with established GPs, although higher rates of error have been reported in other disciplines for doctors in training.9
We found that, overall, supervisors are very confident in their ability to assess their registrars’ competence and patient safety. Confidence in ability to complete a task is termed ‘self-efficacy’ in the psychology literature.10 Self-efficacy has been shown to influence motivation and to undertake new tasks. It may be that supervisors who are more confident in their ability to assess registrars (high self-efficacy) will be less likely to see the need to supervise in a different manner to their current practice.
We found that with greater experience, supervisor confidence in their ability to assess their registrar decreased. This is a novel and counterintuitive result. Interpretation of this finding is difficult but might suggest a more discriminating and discerning approach with greater experience. Less experienced supervisors may be unaware of the inadequacy of some supervision methods until they encounter a significantly less competent registrar.
Supervisors expressed a high level of confidence in their ability to assess registrar competence across all five RACGP domains, but less so in population health practice. The uptake and use of the RACGP guidelines on preventive care by GPs has not been evaluated to date.11 It is possible that supervisors are not confident of their own competence in population health practice. Other explanations for the finding include supervisors having limited perceived methods to assess competence in the public health domain or supervisors misunderstanding the term.
Supervisors reported a high frequency of engagement (opportunistic case discussions and face-to-face teaching) with their registrars. This is likely to reflect the teaching requirements of the training program and RACGP standards, as well as the close relationship that exists in the apprenticeship model.12
We also identified a low frequency of the use of observation and audit techniques by the supervisors, which is a critical finding. There are known barriers to the use of direct observation of consultations and video consultation analysis,13 including the significant effort required, technology problems or registrar reluctance. However, it is known that supervision activities based on observation and audit, that is, review of medical records and audit of referral letters and pathology test ordering, are more likely to detect areas of unconscious incompetence, so called unknown unknowns.14 Audit techniques have been described as valuable for registrar patient safety.6
We believe, therefore, that we have identified a paradox in the clinical supervision of GP registrars – while supervisors highly rate their confidence in awareness of their registrars patient safety, they infrequently use specific supervision methods best suited to detect unconscious incompetence and therefore unknown knowledge and skills gaps.
Strengths and limitations
We had an excellent response rate of 97.8%. This is an exceptional response rate for surveys of GPs.15
The limitation of this study is that the sample was taken from only regional/rural GP supervisors and from one RTP, and hence may not be generalisable to other regions or metropolitan supervisors. Frequency of use of supervision methods is likely to reflect past training and emphasis by the specific RTP. The study is exploratory in nature and is not able to make associations between particular supervisor activities and confidence.
Self-assessment is known to be prone to bias, leading the incompetent to overestimate their competence and those with true ability to relatively underestimate it.16 This may be a factor in the overall confidence reported by supervisors and the decline in confidence with experience.
Implications for general practice training
Supervisors report high levels of confidence in their ability to assess their registrars’ competence and patient safety, suggesting readiness for the implementation of outcome-based standards. Our study did not assess the accuracy of their assessments (ie whether supervisors are truly competent assessors). The infrequent use of observation and audit activities associated with valid and reliable assessments of registrar competence and the reduction in supervisor confidence with experience raises concerns about the accuracy of their self-assessment.
The question of whether supervisors are truly competent assessors could be explored by comparing supervisor assessment with summative examination outcomes. Qualitative research interviews with experienced supervisors could be used to both investigate the comparative value of supervision and assessment methods and uncover the reasons for the decline with experience in supervisors’ confidence in their ability to assess registrar competence and safety.
The adoption of consultation observation and audit activities may be impeded by supervisors’ high level of confidence in their current performance as assessors. Supervisor professional development activities, highlighting adverse patient outcomes that occurred because of the absence of audit and consultation observation, are recommended to create the cognitive dissonance required for behaviour change.
Competing interests: Gerard Ingham and Jennifer Fry are employees of Beyond Medical Education. Simon Morgan is employed by GP Training Valley to Coast.
Provenance and peer review: Not commissioned, externally peer reviewed.