The use of pathology and imaging tests is increasing worldwide.1 In Australia, the majority of testing occurs in primary care.2 While some of this increased testing is appropriate, reflecting advances in technology and clinical knowledge, a growing body of evidence suggests that over-testing is a significant problem.3 For example, Australian data suggest that 25–75% of pathology testing in general practice is not supported by evidence or consensus guidelines.2
Over-testing is especially problematic in general practice, a clinical setting characterised by a high prevalence of undifferentiated illness and a low pre-test probability of serious disease. This combination leads to a high likelihood of false positive results, even in tests with reasonable specificity.4 In addition, incidental findings of questionable significance are common.5 Both outcomes can lead to a cascade of further tests (so-called investigation momentum)6 and, in turn, to a greater risk of complications and patient harm. Over-testing may also lead to overdiagnosis and unnecessary treatment, further adding to the risk of adverse events.7
Vocational training is a critical period for developing patterns of future clinical practice for the general practitioner (GP). Compared with established GPs, general practice registrars (trainees) order more pathology tests per problem managed and in a higher proportion of consultations.8 General practice training in Australia is based on the apprenticeship model, where registrars work independently, but under the supervision of accredited general practice supervisors. Formal practice-based supervision sessions are a mandatory component of training, and general practice supervisors are required to undertake regular continuing professional development to support this activity.9
The general practice supervisor can use a wide variety of methods to supervise registrars, particularly to identify learning needs and monitor patient safety by ‘looking beyond the door’.10 One of the methods of supervising registrars is an audit of test ordering, which involves randomly reviewing pathology and radiology ordering by registrars.10 This teaching and supervision method has previously only been briefly described in the literature. No previous studies have investigated its utility in general practice supervisors’ continuing professional development or general practice training.
We sought to assess the uptake, acceptability and outcomes of an educational intervention to train general practice supervisors on the use of test result audit and feedback (TRAFk).
Methods
As part of a continuing professional development session for general practice supervisors, we developed a one-hour workshop session on TRAFk as a practice-based teaching and supervision method. The workshop supported a primary intervention delivered to the supervisors’ general practice registrars to address rational use of tests. The workshop consisted of large and small group teaching, and role-playing with mock case notes (Box 1). We developed a framework for TRAFk with a specific focus on exploring clinical reasoning and using hypothetical scenarios to extend the learners’ skills (Box 2). The workshop was delivered to separate groups of supervisors on one of two dates in March 2015.
Box 1. TRAFk workshop session
Learning objectives At the end of the session, the supervisor will be able to:
- describe the common ‘drivers’ for over-testing
- describe the risks associated with over-testing
- use TRAFk as a practical tool to teach rational use of investigations in the practice setting.
Introduction and rationale Critical use of investigations as core learning outcome of general practice training Over-testing and overdiagnosis
- Scope and issues
- Drivers for over-testing – doctor, patient, system factors
- Problems of over-testing – costs, issues with interpretation, over-diagnosis, patient harm
- General practice registrar test ordering data
TRAFk
- Description of process
- Framework for analysis of test ordering (Box 2)
Small group activity
- Role-playing general practice registrar
- Facilitator
- Supervisors role-play method
Strategies for rational test ordering Discussion of other methods of rational test ordering – patient-centred care, shared decision making, evidence-based guidelines and managing uncertainty Further resources
Conclusion
- Critical appraisal of supervisors’ own test ordering practice
- Encouragement to use the method in practice
|
Mock case notes example Bob, a male 53 years of age, presents stating that he has been sent in by his wife for a health check. His wife attends the practice but he has never been seen here before. Bob is asymptomatic, very fit, denies significant past medical history, takes no medications, drinks six standard drinks per week (never binges) and has no significant family history. He is not overweight. He has had no blood tests for at least five years. |
Tests ordered Full blood count (FBC), electrolytes, urea and creatinine (EUC), liver function test (LFT), iron studies, lipids, blood sugar level (BSL), prostate-specific antigen (PSA) His results are all normal except for the following: |
Iron studies |
Bob’s results |
Reference range |
Ferritin (ug/L) |
544 |
30–250 |
Iron (Umol/L) |
20 |
11–30 |
Transferrin (g/L) |
2.8 |
2.0–3.6 |
% saturation |
35 |
15–50 |
Box 2. Framework for analysis of test ordering
Explore the rationale for ordering the test Why did you order this test? How will the result alter your management? What are the risks of ordering/not ordering this test? What is the likelihood of a positive result? What is the prevalence of the provisional diagnosis? Did any other factors influence your decision to order the test? Pose hypothetical scenarios What if the test was positive/negative? What if the patient were older/younger/Aboriginal or Torres Strait Islander, etc? Discuss best practice Does this presentation have any guidelines for testing? Where might you seek guidance on best practice? |
General practice supervisors provided demographic details and consent to participate in a pre-workshop survey. Consenting supervisors were sent a post-workshop survey three months after the workshop, to be returned by fax or in a stamped, self-addressed envelope. Responses were matched by a unique identifier but the identity of the respondent remained anonymous to the researchers.
Outcome factors
Outcome factors included satisfaction with the workshop and uptake of TRAFk as a result of attending the workshop. We asked about the acceptability of TRAFk as a supervision method and value of TRAFk across a range of supervision areas (identifying learning needs, exploring clinical reasoning, improving test ordering behaviour, and assessing overall competence and safety with patients). These were measured using Likert scales.
Data analysis
Simple descriptive statistics were used to measure satisfaction with the workshop, and the uptake, acceptability and value of TRAFk. Proportions were calculated with 95% confidence interval (CI) and means with standard deviation (SD).
Ethical considerations
Formal ethics approval for the study was not required. The Human Research Ethics Committee at the University of Newcastle, Australia, deemed the evaluation to be a quality assurance project.
Results
Participants
A total of 124 supervisors attended the workshops and 70 completed a post-workshop survey (response rate 56.5%). Of these 70 respondents, 16 were excluded from the analysis as they had not actively trained a registrar since the workshop (Figure 1). Of the remaining 54 participating supervisors, 18 (34.6%; 95% CI: 22.7–48.9) were female, with a mean of 23.0 years working as a GP (SD: 10.2). General practice supervisors who gained their primary medical degree in Australia comprised 84.9% (95% CI: 72.1–92.4) of the sample. The supervisors’ mean number of clinical sessions worked was 7.8 (SD: 1.6) per week. Table 1 displays the characteristics of participating supervisors.
|
Figure 1. Flowchart of supervisor inclusion |
Table 1. Participating supervisor characteristics
Variable (n = 54) |
Class |
n (%; 95% CI) or mean (SD) |
Gender |
Male |
34 (65.4%; 51.1–77.3) |
|
Female |
18 (34.6%; 22.7–48.9) |
Age |
≤40 |
6 (11.1%; 4.9–23.1) |
|
41–50 |
15 (27.8%; 17.2–41.6) |
|
51–60 |
21 (38.9%; 26.6–52.8) |
|
≥61 |
12 (22.2%; 12.8–35.7) |
Years as a GP |
|
23.0 (10.2) |
Years as a supervisor |
|
8.9 (9.4) |
Medical degree |
Australia |
45 (84.9%; 72.1–92.4) |
|
Other |
8 (15.1%; 7.6–27.9) |
Number of clinical sessions |
|
7.8 (1.6) |
Other supervisors in practice |
|
1.6 (1.5) |
Evaluation and effectiveness of the workshop
Prior to the workshop, less than half (44.4%; 95% CI: 31.5–58.2) of the supervisors were using TRAFk as a supervision method with their registrars. This increased to 79.6% (95% CI: 66.4–88.6) after the workshop, with 97.7% (95% CI: 84.2–99.7) of supervisors using the approach taught during the workshop. Almost all (94.3%; 95% CI: 83.3–98.2) of the participants reported that the workshop improved skills in TRAFk either very well or extremely well.
Outcomes of TRAFk as a supervision method
Supervisors highly rated TRAFk as a method for supervision across a range of supervision areas (Figure 2). As a supervision method, TRAFk was rated as very or highly acceptable by 95.4% (95% CI: 82.4–98.9) of participants.
|
Figure 2. Supervisors’ ratings on the value of TRAFk across four components of registrar supervision |
Discussion
We have shown that a brief workshop on TRAFk increased the uptake of this teaching and supervision method, and increased supervisors’ self-reported skills. To our knowledge, this is the first study to describe general practice supervisor training in, and educational outcomes of, test result audit and feedback, in any clinical setting.
Chart audit and random case analysis (RCA) have previously been used for a variety of educational purposes, including identification of learning needs;11 assessment of clinical knowledge, skills and decision making;12 and assessment of overall clinical competence.13 These are core elements of the supervisors’ role.9 As a supervision and teaching method, RCA is time efficient, allows provision of targeted feedback and encourages reflective practice.
Auditing test results is a refinement of RCA whereby the test results are the starting point of discussion. It retains the strengths of RCA in terms of identifying unconscious incompetence and scope for posing hypothetical scenarios,14 but has an explicit focus on the rationale and consequences of individual test ordering. Critical use of investigations is one of the core skills of The Royal Australian College of General Practitioners’s (RACGP) Common training outcomes.15
Monitoring patient safety is the key aspect of clinical supervision, and assessment of clinical competence is a critical role of the supervisor.9 We found that supervisors highly rated TRAFk as a method of clinical supervision across a range of key areas, which is similar to a recent study of RCA.16 In particular, we found that 65.1% of supervisors rated TRAFk as a good or excellent method of identifying clinical competence, making it a highly valuable supervision tool.
Strengths and weaknesses
Our response rate of 56.5% was good in the context of a GP survey.17 Our workshop session was one hour in duration, a modest investment of time for supervisor professional development. One limitation was that we surveyed supervisors from only one regional training provider (RTP), which may limit the generalisability of our findings.
Implications for general practice
Registrars usually enter general practice after exclusive, hospital-based experience, a setting with a much greater focus on investigation and diagnostic certainty. Registrars, with their inexperience and unfamiliarity with managing undifferentiated illness, may be less tolerant of uncertainty. A low tolerance of uncertainty has been described as a causative factor in over-testing.18
A number of general and specific strategies for teaching and learning rational test ordering in the practice setting have been described, including test result audit and feedback.19 However, despite the use and interpretation of tests having been described as a particularly challenging area for Australian general practice registrars, there is evidence of a lack of training in quality use of pathology.20
We have developed a practical framework for TRAFk based on exploration of clinical reasoning and the use of hypothetical scenarios, similar to that used in RCA.14 As a result, we believe this model has educational utility for learners at all stages and across all levels of competence.
We have demonstrated that training general practice supervisors in the use of TRAFk resulted in a substantial uptake of the method and a high level of acceptability. More importantly, we have demonstrated that use of the method resulted in a significant increase in self-rated confidence as to whether their registrar is ‘safe in there’. These findings reinforce the educational utility of this supervision method in general practice training and highlight the opportunities for greater use in teaching and learning, and in both formative and summative assessments of general practice registrars.
We have shown elsewhere that the delivery of an external educational intervention for general practice registrars on rational test ordering, including the subsequent use of TRAFk in the practice setting, led to desirable changes in attitudes and intended practice in paper-based scenarios.21 Further research should aim to assess the acceptability of TRAFk to general practice registrars and measure the effect of TRAFk on the actual test ordering behaviour of registrars.
Authors
Simon Morgan MBBS FRACGP MPH&TM, GP and Medical Educator, General Practice Training Valley to Coast, Newcastle, NSW. lochswilly@gmail.com
Tony Saltis MBBS FRACGP, GP and Medical Educator, General Practice Training Valley to Coast, Newcastle, NSW
Justin Coleman MBBS FRACGP MPH, Senior Lecturer, Griffith University and University of Queensland, and General Practitioner, Inala Indigenous Health Service, Brisbane, Qld
Amanda Tapley BBiomed Sci (Hons), MMed, Statistics Research Officer, General Practice Training Valley to Coast, Newcastle, NSW
Parker Magin PhD FRACGP, GP, Professor and Medical Educator, General Practice Training Valley to Coast, Newcastle, NSW; Discipline of General Practice, University of Newcastle, NSW
Competing interests: None.
Provenance and peer review: Not commissioned, externally peer reviewed.