Sports medicine

July 2016

Research

Test result audit and feedback (TRAFk) as a supervision method for rational test ordering in general practice training

Volume 45, No.7, July 2016 Pages 518-522

Simon Morgan

Tony Saltis

Justin Coleman

Amanda Tapley

Parker Magin

Background

The use of medical investigations is increasing, with most testing occurring in primary care. The interpretation of test results is challenging for general practice registrars. Additionally, emerging evidence suggests that over-testing is a significant problem and has the potential for patient harm. Test result audit and feedback (TRAFk) is a teaching and supervision method in general practice training, but no previous studies have investigated its utility.

Objective/s

The objective of this article is to describe the outcomes of an educational intervention for general practice supervisors on TRAFk.

Methods

We developed and delivered a workshop to general practice supervisors and administered pre- and post-workshop surveys.

Results

Of the 54 supervisors who participated in the study, a substantial proportion (79.6%) used TRAFk after the workshop. Participants highly rated the method across a range of supervision areas, including clinical reasoning, test ordering quality and patient safety.

Discussion

Our findings reinforce the educational utility of this supervision method in general practice training for the teaching and assessment of registrars.

The use of pathology and imaging tests is increasing worldwide.1 In Australia, the majority of testing occurs in primary care.2 While some of this increased testing is appropriate, reflecting advances in technology and clinical knowledge, a growing body of evidence suggests that over-testing is a significant problem.3 For example, Australian data suggest that 25–75% of pathology testing in general practice is not supported by evidence or consensus guidelines.2

Over-testing is especially problematic in general practice, a clinical setting characterised by a high prevalence of undifferentiated illness and a low pre-test probability of serious disease. This combination leads to a high likelihood of false positive results, even in tests with reasonable specificity.4 In addition, incidental findings of questionable significance are common.5 Both outcomes can lead to a cascade of further tests (so-called investigation momentum)6 and, in turn, to a greater risk of complications and patient harm. Over-testing may also lead to overdiagnosis and unnecessary treatment, further adding to the risk of adverse events.7

Vocational training is a critical period for developing patterns of future clinical practice for the general practitioner (GP). Compared with established GPs, general practice registrars (trainees) order more pathology tests per problem managed and in a higher proportion of consultations.8 General practice training in Australia is based on the apprenticeship model, where registrars work independently, but under the supervision of accredited general practice supervisors. Formal practice-based supervision sessions are a mandatory component of training, and general practice supervisors are required to undertake regular continuing professional development to support this activity.9

The general practice supervisor can use a wide variety of methods to supervise registrars, particularly to identify learning needs and monitor patient safety by ‘looking beyond the door’.10 One of the methods of supervising registrars is an audit of test ordering, which involves randomly reviewing pathology and radiology ordering by registrars.10 This teaching and supervision method has previously only been briefly described in the literature. No previous studies have investigated its utility in general practice supervisors’ continuing professional development or general practice training.

We sought to assess the uptake, acceptability and outcomes of an educational intervention to train general practice supervisors on the use of test result audit and feedback (TRAFk).

Methods

As part of a continuing professional development session for general practice supervisors, we developed a one-hour workshop session on TRAFk as a practice-based teaching and supervision method. The workshop supported a primary intervention delivered to the supervisors’ general practice registrars to address rational use of tests. The workshop consisted of large and small group teaching, and role-playing with mock case notes (Box 1). We developed a framework for TRAFk with a specific focus on exploring clinical reasoning and using hypothetical scenarios to extend the learners’ skills (Box 2). The workshop was delivered to separate groups of supervisors on one of two dates in March 2015.

Box 1. TRAFk workshop session
Learning objectives
At the end of the session, the supervisor will be able to:
  • describe the common ‘drivers’ for over-testing
  • describe the risks associated with over-testing
  • use TRAFk as a practical tool to teach rational use of investigations in the practice setting.
Introduction and rationale
Critical use of investigations as core learning outcome of general practice training
Over-testing and overdiagnosis
  • Scope and issues
  • Drivers for over-testing – doctor, patient, system factors
  • Problems of over-testing – costs, issues with interpretation, over-diagnosis, patient harm
  • General practice registrar test ordering data
TRAFk
  • Description of process
  • Framework for analysis of test ordering (Box 2)
Small group activity
  • Role-playing general practice registrar
  • Facilitator
  • Supervisors role-play method
Strategies for rational test ordering
Discussion of other methods of rational test ordering – patient-centred care, shared decision making, evidence-based guidelines and managing uncertainty
Further resources Conclusion
  • Critical appraisal of supervisors’ own test ordering practice
  • Encouragement to use the method in practice
Mock case notes example
Bob, a male 53 years of age, presents stating that he has been sent in by his wife for a health check. His wife attends the practice but he has never been seen here before. Bob is asymptomatic, very fit, denies significant past medical history, takes no medications, drinks six standard drinks per week (never binges) and has no significant family history. He is not overweight. He has had no blood tests for at least five years.
Tests ordered
Full blood count (FBC), electrolytes, urea and creatinine (EUC), liver function test (LFT), iron studies, lipids, blood sugar level (BSL), prostate-specific antigen (PSA)
His results are all normal except for the following:
Iron studies Bob’s results Reference range
Ferritin (ug/L) 544 30–250
Iron (Umol/L) 20 11–30
Transferrin (g/L) 2.8 2.0–3.6
% saturation 35 15–50
Box 2. Framework for analysis of test ordering
Explore the rationale for ordering the test
Why did you order this test?
How will the result alter your management?
What are the risks of ordering/not ordering this test?
What is the likelihood of a positive result?
What is the prevalence of the provisional diagnosis?
Did any other factors influence your decision to order the test?
Pose hypothetical scenarios
What if the test was positive/negative?
What if the patient were older/younger/Aboriginal or Torres Strait Islander, etc?
Discuss best practice
Does this presentation have any guidelines for testing?
Where might you seek guidance on best practice?

General practice supervisors provided demographic details and consent to participate in a pre-workshop survey. Consenting supervisors were sent a post-workshop survey three months after the workshop, to be returned by fax or in a stamped, self-addressed envelope. Responses were matched by a unique identifier but the identity of the respondent remained anonymous to the researchers.

Outcome factors

Outcome factors included satisfaction with the workshop and uptake of TRAFk as a result of attending the workshop. We asked about the acceptability of TRAFk as a supervision method and value of TRAFk across a range of supervision areas (identifying learning needs, exploring clinical reasoning, improving test ordering behaviour, and assessing overall competence and safety with patients). These were measured using Likert scales.

Data analysis

Simple descriptive statistics were used to measure satisfaction with the workshop, and the uptake, acceptability and value of TRAFk. Proportions were calculated with 95% confidence interval (CI) and means with standard deviation (SD).

Ethical considerations

Formal ethics approval for the study was not required. The Human Research Ethics Committee at the University of Newcastle, Australia, deemed the evaluation to be a quality assurance project.

Results

Participants

A total of 124 supervisors attended the workshops and 70 completed a post-workshop survey (response rate 56.5%). Of these 70 respondents, 16 were excluded from the analysis as they had not actively trained a registrar since the workshop (Figure 1). Of the remaining 54 participating supervisors, 18 (34.6%; 95% CI: 22.7–48.9) were female, with a mean of 23.0 years working as a GP (SD: 10.2). General practice supervisors who gained their primary medical degree in Australia comprised 84.9% (95% CI: 72.1–92.4) of the sample. The supervisors’ mean number of clinical sessions worked was 7.8 (SD: 1.6) per week. Table 1 displays the characteristics of participating supervisors.

Figure 1. Flowchart of supervisor inclusion

 

Table 1. Participating supervisor characteristics
Variable
(n = 54)
Class n (%; 95% CI)
or mean (SD)
Gender Male 34 (65.4%; 51.1–77.3)
  Female 18 (34.6%; 22.7–48.9)
Age ≤40 6 (11.1%; 4.9–23.1)
  41–50 15 (27.8%; 17.2–41.6)
  51–60 21 (38.9%; 26.6–52.8)
  ≥61 12 (22.2%; 12.8–35.7)
Years as a GP   23.0 (10.2)
Years as a supervisor   8.9 (9.4)
Medical degree Australia 45 (84.9%; 72.1–92.4)
  Other 8 (15.1%; 7.6–27.9)
Number of clinical sessions   7.8 (1.6)
Other supervisors in practice   1.6 (1.5)

Evaluation and effectiveness of the workshop

Prior to the workshop, less than half (44.4%; 95% CI: 31.5–58.2) of the supervisors were using TRAFk as a supervision method with their registrars. This increased to 79.6% (95% CI: 66.4–88.6) after the workshop, with 97.7% (95% CI: 84.2–99.7) of supervisors using the approach taught during the workshop. Almost all (94.3%; 95% CI: 83.3–98.2) of the participants reported that the workshop improved skills in TRAFk either very well or extremely well.

Outcomes of TRAFk as a supervision method

Supervisors highly rated TRAFk as a method for supervision across a range of supervision areas (Figure 2). As a supervision method, TRAFk was rated as very or highly acceptable by 95.4% (95% CI: 82.4–98.9) of participants.

Figure 2. Supervisors’ ratings on the value of TRAFk across four components of registrar supervision

Discussion

We have shown that a brief workshop on TRAFk increased the uptake of this teaching and supervision method, and increased supervisors’ self-reported skills. To our knowledge, this is the first study to describe general practice supervisor training in, and educational outcomes of, test result audit and feedback, in any clinical setting.

Chart audit and random case analysis (RCA) have previously been used for a variety of educational purposes, including identification of learning needs;11 assessment of clinical knowledge, skills and decision making;12 and assessment of overall clinical competence.13 These are core elements of the supervisors’ role.9 As a supervision and teaching method, RCA is time efficient, allows provision of targeted feedback and encourages reflective practice.

Auditing test results is a refinement of RCA whereby the test results are the starting point of discussion. It retains the strengths of RCA in terms of identifying unconscious incompetence and scope for posing hypothetical scenarios,14 but has an explicit focus on the rationale and consequences of individual test ordering. Critical use of investigations is one of the core skills of The Royal Australian College of General Practitioners’s (RACGP) Common training outcomes.15

Monitoring patient safety is the key aspect of clinical supervision, and assessment of clinical competence is a critical role of the supervisor.9 We found that supervisors highly rated TRAFk as a method of clinical supervision across a range of key areas, which is similar to a recent study of RCA.16 In particular, we found that 65.1% of supervisors rated TRAFk as a good or excellent method of identifying clinical competence, making it a highly valuable supervision tool.

Strengths and weaknesses

Our response rate of 56.5% was good in the context of a GP survey.17 Our workshop session was one hour in duration, a modest investment of time for supervisor professional development. One limitation was that we surveyed supervisors from only one regional training provider (RTP), which may limit the generalisability of our findings.

Implications for general practice

Registrars usually enter general practice after exclusive, hospital-based experience, a setting with a much greater focus on investigation and diagnostic certainty. Registrars, with their inexperience and unfamiliarity with managing undifferentiated illness, may be less tolerant of uncertainty. A low tolerance of uncertainty has been described as a causative factor in over-testing.18

A number of general and specific strategies for teaching and learning rational test ordering in the practice setting have been described, including test result audit and feedback.19 However, despite the use and interpretation of tests having been described as a particularly challenging area for Australian general practice registrars, there is evidence of a lack of training in quality use of pathology.20

We have developed a practical framework for TRAFk based on exploration of clinical reasoning and the use of hypothetical scenarios, similar to that used in RCA.14 As a result, we believe this model has educational utility for learners at all stages and across all levels of competence.

We have demonstrated that training general practice supervisors in the use of TRAFk resulted in a substantial uptake of the method and a high level of acceptability. More importantly, we have demonstrated that use of the method resulted in a significant increase in self-rated confidence as to whether their registrar is ‘safe in there’. These findings reinforce the educational utility of this supervision method in general practice training and highlight the opportunities for greater use in teaching and learning, and in both formative and summative assessments of general practice registrars.

We have shown elsewhere that the delivery of an external educational intervention for general practice registrars on rational test ordering, including the subsequent use of TRAFk in the practice setting, led to desirable changes in attitudes and intended practice in paper-based scenarios.21 Further research should aim to assess the acceptability of TRAFk to general practice registrars and measure the effect of TRAFk on the actual test ordering behaviour of registrars.

Authors

Simon Morgan MBBS FRACGP MPH&TM, GP and Medical Educator, General Practice Training Valley to Coast, Newcastle, NSW. lochswilly@gmail.com

Tony Saltis MBBS FRACGP, GP and Medical Educator, General Practice Training Valley to Coast, Newcastle, NSW

Justin Coleman MBBS FRACGP MPH, Senior Lecturer, Griffith University and University of Queensland, and General Practitioner, Inala Indigenous Health Service, Brisbane, Qld

Amanda Tapley BBiomed Sci (Hons), MMed, Statistics Research Officer, General Practice Training Valley to Coast, Newcastle, NSW

Parker Magin PhD FRACGP, GP, Professor and Medical Educator, General Practice Training Valley to Coast, Newcastle, NSW; Discipline of General Practice, University of Newcastle, NSW

Competing interests: None.
Provenance and peer review: Not commissioned, externally peer reviewed.

References

  1. Smellie WS. Demand management and test request rationalization. Ann Clin Biochem 2012;49(4):323–36.
  2. Bayram C, Britt H, Miller G, Valenti L. Evidence-practice gap in GP pathology test ordering: A comparison of BEACH pathology data and recommended testing. Final report. Sydney: University of Sydney, Family Medicine Research Centre, 2009.
  3. McGregor MJ, Martin D. Testing 1,2,3: Is overtesting undermining patient and system health? Can Fam Physician 2012;58(11):1191–93.
  4. Hammett RJH, Harris RD. Halting the growth in diagnostic testing. Med J Aust 2002;177:124–25.
  5. Brito JP, Morris JC, Montori VM. Thyroid cancer: Zealous imaging has increased detection and treatment of low risk tumours. BMJ 2013;347:f4706.
  6. Sah S, Elias P, Ariely D. Investigation momentum: The relentless pursuit to resolve uncertainty. JAMA Intern Med 2013;173(10):932–33.
  7. Moynihan R, Doust J, Henry D. Preventing overdiagnosis: How to stop harming the healthy. BMJ 2012;344:e3502.
  8. Morgan S, Henderson KM, Tapley A, et al. Pathology test-ordering behaviour of Australian general practice trainees: A cross-sectional analysis. Int J Qual Health Care 2015;27:527–34.
  9. The Royal Australian College of General Practitioners. RACGP Vocational training standards. East Melbourne, Vic: RACGP, 2014. Available at www.racgp.org.au/education/rtp/vocational-training-standards [Accessed 16 November 2015].
  10. Byrnes PD, Crawford M, Wong B. Are they safe in there? – Patient safety and trainees in the practice. Aust Fam Physician 2012(1–2);41:26–29.
  11. Jennett P, Affleck L. Chart audit and chart stimulated recall as methods of needs assessment in continuing professional health education. J Contin Educ Health Prof 1998;18(3):163–71.
  12. Schipper S, Ross S. Structured teaching and assessment: A new chart-stimulated recall worksheet for family medicine residents. Can Fam Physician 2010;56(9):958–59.
  13. Jennett P. Chart stimulated recall: A technique to assess clinical competence and performance. Educ Gen Pract 1995;6:30–34.
  14. Morgan S, Ingham G. Random case analysis: A new framework for Australian general practice training. Aust Fam Physician 2013;42(1–2):69–73.
  15. The Royal Australian College of General Practitioners. Common training outcomes. East Melbourne, Vic: RACGP, 2011. Available at http://curriculum.racgp.org.au/statements/common-training-outcomes [Accessed 16 November 2015].
  16. Morgan S, Ingham G, Kinsman L, Fry J. Clinical supervision using random case analysis in general practice training. Educ Prim Care 2015;26(1):40–46.
  17. Bonevski B, Magin P, Horton G, Foster M, Girgis A. Response rates in GP surveys: Trialling two recruitment strategies. Aust Fam Physician 2011;40(6):427–30.
  18. van der Weijden T, van Velsen M, Dinant GJ, van Hasselt CM, Grol R. Unexplained complaints in general practice: Prevalence, patients’ expectations, and professionals’ test ordering behavior. Med Decis Making 2003;23(3):226–31.
  19. Morgan S, Coleman J. We live in testing times – Teaching rational test ordering in general practice. Aust Fam Physician 2014;43(3):273–76.
  20. Bubner T, Laurence C, Tirimacco R. Assessing pathology training needs – Results from a survey of general practice registrars. Aust Fam Physician 2012;41(9):721–24.
  21. Morgan S, Kerr R, Morgan A, Tapley A, Magin P. The effect of an educational intervention on the test ordering attitudes and intended practice of general practice trainees. Can Fam Physician (accepted for publication 11 January 2016).

Correspondence afp@racgp.org.au

Yes     No

Declaration of competing interests *

Yes No

Additional Author (remove)

Yes No

    

 

 

 

 

Competing Interests: 

Your comment is being submitted, please wait

 

Download citation in RIS format (EndNote, Zotero, RefMan, RefWorks)

Download citation in BIBTEX format (RefMan)

Download citation in REFER format (EndNote, Zotero, RefMan, RefWorks)

For more information see Wikipedia: Comparison of reference management software