Audit and feedback by medical students to improve the preventive care practices of general practice supervisors

Medical students in their clinical placements represent the most junior members of the healthcare team. Previous studies have assessed patients’ perspectives on having medical students at consultations and the observations are mostly positive. The benefits observed by patients include that the:

• consultations are longer¹
• quality of the consultation may be improved²
• general practitioner (GP) might learn more about their condition.³

However, little is known about the effect that medical students have on their general practice supervisors and on patient care.

This study was designed to measure the effect of an audit and feedback educational activity carried out by medical students on patient care. The educational activity consisted of clinical case note audits of preventive care markers. Medical students performed these audits during their general practice clinical placement.

We chose preventive care as the target of our study because its success in primary care is fundamental to the whole healthcare system. However, there are many barriers to its effective implementation in general practice, including limitations of time and human resources.⁴ A Cochrane review concluded that audit and feedback generally led to small but potentially important improvements in professional practice.⁵ Clinical audit is a time-consuming activity and, in the Australian fee-for-services healthcare system, there are limited incentives for clinical audit activities.

In order to test the audit and feedback activity on preventive care, we carried out a study where medical students performed the clinical audits for their general practice supervisors. The students gave feedback from the audit to their supervisors. In this study, we report the results of the changes in preventive care practices of general practice supervisors observed over a period of repeated audit and feedback cycles.

Methods

Design and setting

This study was a non-randomised, before–after interventional study that observed the effect of a cycle of repeated audit with feedback over time. Medical students performed audits on the preventive practices of their general practice preceptors. The results of all patients from all general practices who completed at least one cycle of audit were combined. A longitudinal log analysis was performed over four audit periods to determine if there were changes in the proportion of patients with certain preventive health parameters documented over time.

Procedures

Students were given training on the audit process before auditing the case notes of 20 patients who attended the practice during their eight-week general practice term. They then submitted their results through the internet survey tool SurveyMonkey. Students were given a copy of their audit results, which they discussed with their supervisor during their usual teaching session.

The GPs were prepared for the audit through distribution of written information, and were invited to an information session. The audit questionnaire included a wide range of markers of preventive health activities, including documentation of social history risk factors, assessment of cardiovascular risk factors and cancer screening tests. The preventive activities investigated in this study are recommended in The Royal Australian College of General Practitioners’ (RACGP’s) Guidelines for preventive activities in general practice,⁶ with the exception of prostate-specific antigen (PSA) testing.

Patients gave written consent for their data to be used in the audit. For each preventive health parameter, the combined data from the group of GPs were analysed to determine if there was a trend to change over time.

The effect of this cycle of audit with feedback over time was assessed using longitudinal log analysis. The results were compiled into three groups of preventive health activities:

• Cardiovascular risk factor assessment
• Recording social history
• Cancer screening tests.

Ethics approval was obtained from the University of Western Australia’s Human Research Ethics Committee (reference RA/4/1/2230).

Results

In total, 3076 patients were recruited by medical students. One hundred and fifty-six students performed the audit. A total of 58 GPs were involved in the audit over four audit periods (42 weeks). Thirty-five GPs completed more than one audit with feedback, and their data were included in the longitudinal analysis. This represents 2608 patient audit questionnaires, which is 85% of the total sample.

Table 1 provides a summary of the preventive health data collected from all patient case note audits. Patients’ demographics of age and gender were compared with Bettering the Evaluation and Care of Health (BEACH) data⁷ (Table 2). The results of the audit were also compared with known Australian Bureau of Statistics (ABS) health data for the Australian population to assess the fidelity of the data collection. The four health parameters examined were smoking, hypertension, alcohol consumption and mammogram screening (Table 2). Data from this study and nationally were not significantly different for the parameters of patients’ age and gender, patients who smoke, patients who drink alcohol above recommended levels, or patients with hypertension.

Table 3 shows data from the first audit for all GPs and the fourth-term results. The first audit period for all GPs is an indicator of baseline compliance. The fourth‑term results are an indicator of the final outcomes of the audit cycle.

There was a statistically and clinically significant increase in recording patients’ alcohol consumption, which increased from 24% to 36% (odds ratio [OR]: 1.19; 95% confidence interval [CI]: 1.10–1.29). While there was no increase in the proportion of patients who had a family history recorded overall, there was a significant increase in the proportion of patients who had a more detailed family history documented. GPs increased their recording of family histories of type 2 diabetes (23% to 32%), early ischaemic heart disease (24% to 33%), breast cancer (21% to 32%) and colorectal cancer (20% to 30%; Table 3).

There was no significant change in recording weight, cholesterol or blood glucose level over the period of the four audit cycles. There was a small, yet statistically significant reduction in blood pressure recording over the four audit periods (86% to 82%). There appeared to be a small increase in recording smoking status (56% to 60%), but this was not statistically significant (P = 0.051; Table 3) .

There was no significant change in recording of Papanicolaou (Pap) test or PSA testing. There was a significant reduction in the recording of mammogram rate over the course of the audit from 46% to 36% at the end of the audit.

Discussion

Our study has two main messages: audit and feedback by a medical student were feasible procedures, and these created clinically important increases in the general practice supervisors’ recordings of social and family history. The changes were significant in areas with the lowest baseline levels. Recording alcohol consumption increased from 24% to 36%. Similarly, recording patients’ family history of type 2 diabetes, early ischaemic heart disease, and breast or colorectal cancer increased by 10% on average. This result is consistent with the findings of the Cochrane review by Ivers et al on the effect of audit with feedback.⁵ These authors identified low baseline compliance as a factor associated with greater effect of audit with feedback.

Overall, the preventive health parameters measured varied widely in their compliance with guideline recommendations. Some areas of preventive care showed high compliance with guideline recommendations (eg recording the cardiovascular risk factors and measuring cholesterol in adults >50 years of age). In areas of high baseline compliance, we were not able to demonstrate improvement, which can be explained by a ceiling effect.⁸ In fact, rates of blood pressure recording dropped from 86% to 82%. The most likely explanation for this observation is a normal fluctuation of adherence around a peak of performance.

The results concerning cancer screening were diverse. We observed a significant reduction in mammogram rate over the course of the audit, which is difficult to explain. There are multiple external factors involved in the breast cancer screening program in Australia, and it is unlikely that the change occurred as a result of the GP audit. By contrast, recording social history is entirely within the influence of the GP and, intuitively, we can understand that identifying a low level of adherence in this area could result in an immediate change in practice.

Another interesting finding was the PSA testing rate, which was high (75% of men >50 years of age) despite the fact that it is not recommended in key general practice guidelines.⁶ This high rate of testing may be attributable to media coverage of PSA testing and increased patient expectations.^4,8

The strengths of our study include our study design, which provided a pragmatic, innovative approach to data collection in Australian general practice. A total of 3076 patient audits were completed while students were learning about preventive healthcare, with the only extra costs being academic time and postage costs. Many studies that have shown a significant change with audit required a very labour intensive, and therefore expensive, audit and feedback process.

Another strength of the study is the rather brave challenge given to medical students in asking them to break out of the hierarchical roles and critique the care given by their supervisors. This was potentially challenging for the students and GPs. However, all the GPs who responded to the post-audit questionnaire indicated that they were comfortable with students performing an audit on their patients, so the difficulties may be more a perception than a reality.

The pragmatic nature of this study created some methodological limitations. We were unable to have a control group as all students needed to complete this audit activity. We used very small cross-sectional samples of patients with the assumption that they are a representative sample of that GPs’ patient population.

An audit has minimal effect unless it is associated with feedback to the clinician.⁹ The analysis method used in this study assumed that all GPs had equal exposure to the audit and feedback. We were unable to determine the quality of the feedback given by the students. The post-audit questionnaire sent to GPs designed to help assess the quality of feedback had a very low response rate (15 out of 58; 26%).

We did not account for potential clustering effects in our analyses. As an observational study, the observed changes may have been due to other factors within the practices that are unrelated to the study. The small cross-sectional samples of patients may have included patients who also saw other doctors in the practice who were not audited and not given feedback.

This study has shown a small but measurable effect of a student educational activity on the quality of documentation of certain disease risk factors by their supervising GP. This differs from the studies by Gould et al¹⁰ and Doyle et al.¹¹ Their studies assessed the impact of more substantial student-led quality improvement activities, which represented a significant body of work for the students; most medical curricula would only expect students to perform one of these activities. This study showed the effect of a small, brief educational activity with multiple students participating at multiple time points. Medical curricula could expect students to perform multiple activities of this size if they were found to have educational validity. It is possible that many of the educational activities in which students participate by reviewing patients in detail and discussing their care with a treating doctor could trigger some reflection by the treating doctor on the quality of their care. This is an area that could deserve further study.

The limitation of time within the consultation and limited human resources are universally cited as barriers for delivering preventive care outcomes in primary care. We require a paradigm shift in the way we approach preventive health in general practice and should look to the whole practice team, including the medical students, to assist through measures such as improving practice systems to record preventive health information.

Implications for general practice

This study has shown that a student‑led audit with feedback can result in improvement in some parameters of preventive care, and that students could be considered as potentially useful members of the primary healthcare team in the campaign for health promotion and disease prevention in the Australian population.

Authors

Lucy A Gilkes MBBS, FRACGP, MClinRes, Senior Lecturer, School of Primary, Aboriginal and Rural Health Care, Faculty of Medicine, Dentistry and Health Sciences, The University of Western Australia, Crawley, WA. lucy.gilkes@uwa.edu.au

Helena Liira MD, PhD, MBA, Professor of General Practice, Head of School, School of Primary, Aboriginal and Rural Health Care, Faculty of Medicine, Dentistry and Health Sciences, The University of Western Australia. Crawley, WA

Jon D Emery DPhil, MA, MBBCh, BA, Herman Professor of Primary Care, Cancer Research, Department of General Practice, Melbourne Medical School, Faculty of Medicine, Dentistry & Health Sciences, The University of Melbourne and Western Health, Carlton, Vic

Competing interests: None.

Provenance and peer review: Not commissioned, externally peer reviewed

Acknowledgements

We would like to thank all the medical students and their general practice supervisors who collaborated with us for this study. With statistics we received highly appreciated help from Dr Sharon Evans.