In this study, we assessed how accurately one of the most commonly used audit tools extracted data from patient record systems. The study forms part of the Treatment Of cardiovascular Risk in Primary care using Electronic Decision SuppOrt (TORPEDO) study. TORPEDO is a cluster randomised controlled trial examining the effectiveness of a multi-faceted quality improvement intervention to improve cardiovascular disease (CVD) risk screening and management. Sixty health services are participating (40 general practices and 20 ACCHSs) in New South Wales and Queensland. Full details of the study protocol have been published elsewhere.12
Methods
TORPEDO involves extracting clinical data for all regularly attending patients (three visits in the previous 2 years and one visit in the previous 6 months) in whom national guidelines recommend CVD risk screening (>35 years if Aboriginal and Torres Strait Islander and >45 years for all others). Participating sites need to be exclusively using either Medical Director™ or Best Practice™ for their electronic health records (EHR) without any use of paper or hybrid paper/electronic recording. These software products currently comprise the majority of EHR systems used in Australia. Extractions are performed at baseline and end of study. Two de-identified data files are generated: (1) an individual patient data file that is sent to the coordinating research institute for trial analyses via a secure file transfer protocol; and (2) an aggregated data file that is sent to a web-based portal via an identical mechanism as that used in the APCC program. Intervention arm sites can perform these extractions monthly and can view peer-ranked performance feedback data.
Variables are extracted on patient demographics, recorded diagnoses of chronic diseases, chronic disease risk factors, pathology tests and medications. We assessed the validity of the data extraction tool in detecting these variables from the two previously described EHR systems via a two-stage process.
Stage 1: Pathology review
For extraction tools to accurately collect pathology data, laboratories must submit test results in a standard format (Health Level Seven [HL-7]) and assign a unique code per test (Logical Observation Identifiers Names and Codes [LOINC]). Prior to validating the tool itself, we reviewed pathology reporting for plasma cholesterol, creatinine and glycated haemoglobin (HbA1c), and urinary albumin to creatinine ratio (UACR). As a condition of participating in TORPEDO we mandated that all sites have their pathology reported in HL-7 format. We then reviewed pathology extracts from a sample of health services covering all the laboratories in the TORPEDO study. These extracts identified what LOINCs were being reported for the tests in question and from these data we determined whether they were concordant with those searched for with the data extraction tool.
Stage 2: Manual case record audit at two general practices
Two hundred records (100 per practice) from patients in the eligible age range for TORPEDO were randomly selected. The two sites were urban Sydney practices participating in the intervention arm of TORPEDO. One site used Medical Director™ and the other used Best Practice™. Audits were conducted as part of a 12-month follow-up visit. Both sites had shown some mild improvement in data quality over the intervention period. A research assistant reviewed the following sections of the record: demographic details, diagnoses, past medical history, physical risk factor measurements, current medications and pathology results, encompassing 23 chronic disease-related variables. Free text entries in progress notes were not reviewed. A fresh data extraction was simultaneously performed and securely sent to the research institute. Data entered from the manual record audit and that obtained from the individual patient data extract were compared. Kappa statistics were used for categorical variables and Bland–Altman plots were constructed for continuous variables with the mean differences and 95% limits of agreement reported.13 Analyses were conducted using SAS Enterprise Guide (v. 4.5; SAS Institute, Cary, NC, USA)
Results
Pathology audit
Thirty-three sites had pathology data audited covering nine laboratories in New South Wales and Queensland. All laboratories were using the correct LOINC codes for total cholesterol, high-density lipoprotein cholesterol, low-density lipoprotein cholesterol, triglycerides, creatinine and HbA1c. There were, however, marked variations in codes used for UACR. Four laboratories used incorrect codes, precluding accurate extraction of results for these tests. All pathology companies were notified and instructed on the correct codes to use for this test.
Manual record audit
Table 1 highlights the patient characteristics for the 200 records audited. Table 2 outlines the level of agreement/correlation for the 23 variables manually audited. Overall, the majority of variables achieved perfect or near perfect agreement/correlation. There was, however, one notable exception related to chronic kidney disease (CKD). Criteria for CKD were based on national guideline definitions and included a recorded diagnosis of CKD or proteinuria or an estimated glomerular filtration rate (eGFR) < 45ml/min/1.73m2. There were several cases where the extraction tool seemed to inappropriately assign a diagnosis of CKD where no such criteria could be identified from the manual record review. This was subsequently identified to be a problem in the way the tool extracted eGFR values from one pathology company. If the result reported a non-integer value (eg. >90 ml/min), this defaulted to a zero value and hence met one of the CKD criteria (eGFR <45 ml/min). The software company was notified and a fix was put in place. There was also low agreement with UACR tests. This is likely to not be a problem related to the tool per se, but related to incorrect LOINC coding by one of the pathology companies used at that practice.
Table 1. Patient characteristics obtained from the manual case record audit tool (n = 200)
| Risk factor measurement | Mean (SD) or n (%) in those with available information | No of records with available information (% total) |
|---|
| Demographic information |
|
|
| Age (years) |
66.2 (12.5) |
200 (100) |
| Male |
93 (47) |
200 (100) |
| Risk factor measurements |
|
|
| Current smoker |
29 (20) |
145 (73) |
| BMI (kg/m2) |
31.5 (8.8) |
97 (49) |
| Systolic blood pressure (mmHg) |
129.1 (15.4) |
179 (90) |
| Diastolic blood pressure (mmHg) |
79.1 (10.0) |
179 (90) |
| Pathology laboratory measurements |
|
|
| Total cholesterol (mmol/L) |
5.1 (1.2) |
157 (79) |
| HDL cholesterol (mmol/L) |
1.4 (0.4) |
155 (78) |
| LDL cholesterol (mmol/L) |
2.8 (0.9) |
153 (77) |
| Triglycerides (mmol/L) |
1.9 (1.2) |
157 (79) |
| Urinary albumin to creatinine ratio (mg/mmol) |
6.8 (24.3) |
93 (47) |
| Creatinine (umol/L) |
78.7 (28.3) |
147 (74) |
| HbA1c (%) |
6.4 (1.3) |
112 (56) |
| Past medical history |
|
n (%) |
| Coronary heart disease |
|
30 (15) |
| Ischaemic stroke/transient ischaemic attack |
|
15 (7.5) |
| Peripheral vascular disease |
|
6 (3) |
| Left ventricular hypertrophy |
|
0 (0) |
| Atrial fibrillation |
|
11 (5.5) |
| Heart failure |
|
11 (5.5) |
| Diabetes |
|
41 (20.5) |
| Abbreviations: BMI = body mass index; HbA1c = glycated haemoglobin; HDL = high-density lipoprotein; LDL = low-density lipoprotein; SD = standard deviation |
Table 2. Agreement/correlation statistics between the audit tool and manual case record audit for 200 patient records from two general practices
| Variable | Agreement (kappa) (95% CI where applicable) |
|---|
| Aboriginal status |
1.00 |
| Smoking status |
1.00 |
| Recorded diagnoses |
|
| Coronary heart disease |
0.95 (0.90–1.00) |
| Peripheral vascular disease |
1.00 |
| Stroke |
0.96 (0.89–1.00) |
| Atrial fibrillation |
1.00 |
| Left ventricular hypertrophy |
1.00 |
| Congestive heart failure |
0.95 (0.85–1.00 |
| Diabetes |
1.00 |
| Gestational diabetes |
1.00 |
| Chronic kidney disease |
0.24 (0.00–0.48) |
| Chronic disease risk factor measurements | Mean difference* (95% limits of agreement where applicable) |
No of readings where difference > 2SD (%)†
|
|---|
| Weight (kg) |
0.001 (–0.51–0.51) |
2 (1.0)
|
| Height (cm) |
0 |
0 (0)
|
| Waist circumference (cm) |
0 |
0 (0)
|
| Systolic blood pressure (mmHg) |
0.06 (–1.8–1.6) |
3 (1.5)
|
| Diastolic blood pressure (mmHg) |
0.14 (-4.9–5.2) |
2 (1)
|
| Total cholesterol (mmol/l) |
0.02 (–0.37–0.34) |
6 (3)
|
| HDL (mmol/l) |
0.002 (–0.03–0.04) |
0 (0)
|
| LDL (mmol/l) |
0.03 (–0.52–0.46) |
2 (1)
|
| Triglycerides (mmol/l) |
0.02 (–0.65–0.62) |
6 (3)
|
| Creatinine (µmol/l) |
0.7 (–10.1–8.8) |
5 (2.5)
|
| HbA1c (%) |
0.001 (–0.18–0.19) |
2 (1)
|
| Urinary albumin to creatinine ratio (mg/mmol) |
9.7 (–71.7–91.0) |
0 (0)
|
Abbreviations: SD = standard deviation
*The mean difference between the readings obtained from the manual audit vs the audit tool with zero indicating perfect agreement for all readings
†Derived from constructing Bland–Altman plots for each variable |
Discussion
In this study we found that data extraction using one of the most commonly used tools in Australia was highly consistent with manual reviews of the data for all variables except CKD recording which, as a result of this study, was identified to be a programming error and subsequently fixed. We also found wide discrepancies in the coding of CKD-related test results by various laboratories. This affects the ability of any data extraction tool to accurately detect pathology test results for this condition.
An important study limitation is that it was ‘fit for purpose’ for the chronic disease-related variables used in TORPEDO and for the population recommended for CVD risk screening. We cannot make any assumptions, therefore, about validity for other disease areas. Another limitation is that we looked at interaction between one extraction tool and two GP EHR systems. Although these are the most commonly used systems in Australia, we cannot extrapolate findings to other systems. A final limitation is that we only included data that could potentially be extracted from the EHR. If a practitioner is making free text entries rather than in codable sections of the record then neither the extraction tool nor our manual case record reviewer would have detected those entries.
The Australian health care system is rapidly evolving to incorporate an information communication technology (ICT) infrastructure that will enable data to be shared across multiple platforms. The primary health care sector is a leading driver of this change. GPs and ACCHSs were early adopters of EHRs and are now leading developments in use of data for quality improvement, key performance indicator reporting, secure messaging, and the personally controlled e-health record. The ability to extract data reliably and consistently is central to supporting these activities.
There are two clear recommendations from the study. First, there is a critical need for consistent reporting of pathology tests across all laboratories. Second, the lack of agreement for variables associated with CKD recording, although in retrospect was able to be explained, highlights the importance of establishing regular surveillance procedures that are independent of any validation work conducted by the software vendors themselves. Whenever new code is written, either for the extraction tools or within the EHR system itself, there is potential for errors in the data extraction process. National programs such as the APCC are well placed to establish such surveillance systems. This program receives monthly aggregated data from a large number of practices. Automated warning systems for unexpected deviations in these routinely collected data would enable early detection of potential problems. A sentinel network of practices that routinely extracted data could complement such a surveillance system. As we become increasingly reliant on ICT systems for both clinical care and system improvements, it is important that action be taken to ensure these systems are robust.
Implications for general practice
- The audit tool used in this study was reliable for most chronic disease-related measurements with the exception of CKD diagnoses.
- Inaccurate pathology data extracted from audit tools is due to laboratories assigning incorrect codes to test results.
- National initiatives to independently and regularly review the quality of data obtained from data extraction tools are needed.
Ethics approval: The study was approved by the University of Sydney Human Research Ethics Committee (Reference number: 2012/2/2183) and permission was obtained from all participating services to audit records.
Competing interests: None declared.
Provenance and peer review: Not commissioned; externally peer reviewed.
Acknowledgements
We gratefully acknowledge the support of Dr Cedric Meyerowitz and Dr John Hillman in the conduct of this study. David Peiris was supported by an NHMRC Translating Research into Practice Fellowship and now holds a NHMRC Early Career Fellowship. Maria Agaliotis is supported by a NHMRC Senior Research Fellowship.